bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Christian Johansson]

Hi!

~/.authinfo.gpg seems to be loaded automatically for TRAMP sometimes but 
not always, (dired) seems to be one command that doesn't load 
.authinfo.gpg automatically.

Steps to reproduce:
1. Open emacs with src/emacs -Q
2. Open ~/.authinfo.gpg
3. Make sure this line exists: machine ftp.dlptest.com login 
dlpuserdlpuser@dlptest.com port ftp password e73jzTRTNqCN9PYAAjjn
4. Open *scratch* buffer
5. Paste: (dired "/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")
6. Run M-x "eval-buffer"
7. Notice the request for password in the mini-buffer even though 
password is already available in .authinfo.gpg

Regards
Christian

In GNU Emacs 27.0.50 (build 3, x86_64-apple-darwin17.6.0, NS 
appkit-1561.40 Version 10.13.5 (Build 17F77))
  of 2018-06-30 built on Christians-MacBook-Air.local
Repository revision: ed65ea18152636500399a7b6b75c87bac7d4ef2b
Windowing system distributor 'Apple', version 10.3.1671
System Description:  Mac OS X 10.14

Recent messages:
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/mail/emacsbug.el’ newer 
than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/mail/sendmail.el’ newer 
than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/gnus/message.el’ newer 
than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/mail/mailabbrev.el’ 
newer than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/gnus/gnus-util.el’ newer 
than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/mail/rmail.el’ newer 
than byte-compiled file
Source file ‘/Users/christianjohansson/Documents/emacs/lisp/epg.el’ 
newer than byte-compiled file
Source file 
‘/Users/christianjohansson/Documents/emacs/lisp/epg-config.el’ newer 
than byte-compiled file
s-b is undefined [2 times]
funcall-interactively: End of buffer

Configured features:
RSVG IMAGEMAGICK DBUS NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS
NS THREADS

Important settings:
   value of $LC_CTYPE: UTF-8
   value of $LANG: en_SE.UTF-8
   locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
   shell-dirtrack-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny rfc822 mml easymenu
mml-sec epa derived epg epg-config gnus-util rmail rmail-loaddefs
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils conf-mode ange-ftp tramp-ftp tramp trampver tramp-compat
tramp-loaddefs shell pcomplete comint ansi-color ring parse-time
format-spec advice auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs elec-pair
time-date tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/ns-win ns-win ucs-normalize mule-util
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 222461 11024)
  (symbols 48 22158 1)
  (miscs 40 38 188)
  (strings 32 36026 1980)
  (string-bytes 1 1003870)
  (vectors 16 38793)
  (vector-slots 8 764344 15208)
  (floats 8 56 100)
  (intervals 56 244 0)
  (buffers 992 14))

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi!

Hi Christian,

> ~/.authinfo.gpg seems to be loaded automatically for TRAMP sometimes
> but not always, (dired) seems to be one command that doesn't load
> .authinfo.gpg automatically.

~/.authinfo (and its different versions) are taken by the auth-sourc.el
package. Tramp is aware of it, and uses its mechanisms.

> 5. Paste: (dired "/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")

As said before, "ftp" based methods are delegated via Tramp to
ange-ftp.el. That package does not know auth-source.el, instead it has
implemented its own parsing of ~/.netrc style files.

Set ange-ftp-netrc-filename to "~/.authinfo.gpg", this should work.

> Regards
> Christian

Best regards, Michael.

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Christian Johansson]

Hi!

Alright, that worked. Would it be possible to make ange-ftp.el 
automatically aware of .authinfo files?
I think that would improve user experience a lot.

I can try adding support for that if it's possible

Regards
Christian

On 2018-10-25 10:14, Michael Albinus wrote:
> Christian Johansson <christian@cvj.se> writes:
>
>> Hi!
> Hi Christian,
>
>> ~/.authinfo.gpg seems to be loaded automatically for TRAMP sometimes
>> but not always, (dired) seems to be one command that doesn't load
>> .authinfo.gpg automatically.
> ~/.authinfo (and its different versions) are taken by the auth-sourc.el
> package. Tramp is aware of it, and uses its mechanisms.
>
>> 5. Paste: (dired "/ftp:dlpuser@dlptest.com@ftp.dlptest.com:/")
> As said before, "ftp" based methods are delegated via Tramp to
> ange-ftp.el. That package does not know auth-source.el, instead it has
> implemented its own parsing of ~/.netrc style files.
>
> Set ange-ftp-netrc-filename to "~/.authinfo.gpg", this should work.
>
>> Regards
>> Christian
> Best regards, Michael.

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi!

Hi Christian,

> Alright, that worked. Would it be possible to make ange-ftp.el
> automatically aware of .authinfo files?
> I think that would improve user experience a lot.

Finally, it means that ange-ftp.el uses auth-source.el. Anything goes,
but somebody must implement it. And auth-source.el has its own
interpretation of the "port" element in .netrc/.authinfo lines, which is
kind of incompatible.

I'll mark this bug report as wishlist. If nobody beats me I'll could do
it, but it will take time (too many other open requests). And for the
time being, you'll be able to apply the workaround.

> I can try adding support for that if it's possible

You could try it yourself :-)

Otherwise, I will ask you at least for tests, when the implementation
happens.

> Regards
> Christian

Best regards, Michael.

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Christian,

>> Alright, that worked. Would it be possible to make ange-ftp.el
>> automatically aware of .authinfo files?
>> I think that would improve user experience a lot.
>
> Finally, it means that ange-ftp.el uses auth-source.el. Anything goes,
> but somebody must implement it. And auth-source.el has its own
> interpretation of the "port" element in .netrc/.authinfo lines, which is
> kind of incompatible.
>
> I'll mark this bug report as wishlist. If nobody beats me I'll could do
> it, but it will take time (too many other open requests). And for the
> time being, you'll be able to apply the workaround.

I have checked ange-ftp.el, and it does much more with the netrc file
than auth-sources.el. For example, it handles also the "account" token.

Therefore I have decided not to touch this. Instead, I have added the
following text to the Tramp manual, section "5.12.1 Using an
authentication file":

--8<---------------cut here---------------start------------->8---
   *Note* that ‘auth-source.el’ is not used for ‘ftp’ connections,
because TRAMP passes the work to Ange FTP.  If you want, for example,
use your ‘~/.authinfo.gpg’ authentication file, you must customize
‘ange-ftp-netrc-filename’:

     (customize-set-variable 'ange-ftp-netrc-filename "~/.authinfo.gpg")
--8<---------------cut here---------------end--------------->8---

If you agree, I would close this bug.

>> Regards
>> Christian

Best regards, Michael.

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Christian Johansson]

Hi Michael

Would it be a huge task extending auto-source.el to make it compatible 
with ange-ftp as well?

I think many people would appreciate that feature, but it's not really a 
bug rather a feature request

Sounds good to improve the documentation about the possible work-around

Regards
Christian

On 2018-11-08 12:56, Michael Albinus wrote:
> Michael Albinus <michael.albinus@gmx.de> writes:
>
> Hi Christian,
>
>>> Alright, that worked. Would it be possible to make ange-ftp.el
>>> automatically aware of .authinfo files?
>>> I think that would improve user experience a lot.
>> Finally, it means that ange-ftp.el uses auth-source.el. Anything goes,
>> but somebody must implement it. And auth-source.el has its own
>> interpretation of the "port" element in .netrc/.authinfo lines, which is
>> kind of incompatible.
>>
>> I'll mark this bug report as wishlist. If nobody beats me I'll could do
>> it, but it will take time (too many other open requests). And for the
>> time being, you'll be able to apply the workaround.
> I have checked ange-ftp.el, and it does much more with the netrc file
> than auth-sources.el. For example, it handles also the "account" token.
>
> Therefore I have decided not to touch this. Instead, I have added the
> following text to the Tramp manual, section "5.12.1 Using an
> authentication file":
>
> --8<---------------cut here---------------start------------->8---
>     *Note* that ‘auth-source.el’ is not used for ‘ftp’ connections,
> because TRAMP passes the work to Ange FTP.  If you want, for example,
> use your ‘~/.authinfo.gpg’ authentication file, you must customize
> ‘ange-ftp-netrc-filename’:
>
>       (customize-set-variable 'ange-ftp-netrc-filename "~/.authinfo.gpg")
> --8<---------------cut here---------------end--------------->8---
>
> If you agree, I would close this bug.
>
>>> Regards
>>> Christian
> Best regards, Michael.

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi Michael

Hi Christian,

> Would it be a huge task extending auto-source.el to make it compatible
> with ange-ftp as well?

It is more complex than I expected initially, that's why I haven't
started it. And honestly, I'm rather an auth-source.el user than a
developer; I'm not very familiar with that code. That's why it might be
better if somebody else does it.

Requirements from ange-ftp.el would be at least

- Support of the "account" token.
- Support of the "default" token.
- Check of proper file permissions of "~/.netrc" or whatever file is
  specified (see ange-ftp-disable-netrc-security-check).
- Everything I have forgotten to tell here ...

The "account" token is already parsed in auth-source.el, but it is not
divided from the "login" token. Both are handled via the common keyword
:user.

The "default" token is already parsed in auth-source.el, but I'm not
sure it is used in the sense ange-ftp.el expects it. In auth-source.el
this token is expanded to an entry of its own, with an unspecified
"machine". In ange-ftp.el, the tokens of "default" are used for all
other entries, replacing missing tokens. AFAIU.

Once this is prepared in auth-source.el, I would be able to integrate it
in ange-ftp.el. That is a more simple task; I've done this already for Tramp.

> I think many people would appreciate that feature, but it's not really
> a bug rather a feature request

Agreed. Let's keep this bug (wishlist) open, maybe somebody takes over
the task. I'm Cc'ing Ted Zlatanov, the author of auth-source.el.

> Regards
> Christian

Best regards, Michael.

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Ted Zlatanov]

On Fri, 09 Nov 2018 10:45:17 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Requirements from ange-ftp.el would be at least

MA> - Support of the "account" token.
MA> - Support of the "default" token.
MA> - Check of proper file permissions of "~/.netrc" or whatever file is
MA>   specified (see ange-ftp-disable-netrc-security-check).

MA> The "account" token is already parsed in auth-source.el, but it is not
MA> divided from the "login" token. Both are handled via the common keyword
MA> :user.

MA> The "default" token is already parsed in auth-source.el, but I'm not
MA> sure it is used in the sense ange-ftp.el expects it. In auth-source.el
MA> this token is expanded to an entry of its own, with an unspecified
MA> "machine". In ange-ftp.el, the tokens of "default" are used for all
MA> other entries, replacing missing tokens. AFAIU.

MA> Once this is prepared in auth-source.el, I would be able to integrate it
MA> in ange-ftp.el. That is a more simple task; I've done this already for Tramp.

Hi Michael,

if you or a user could define the desired behavior as tests against
auth-source.el (we already have that set up in the core), I'd be happy
to implement them. Something like "for this call, with this
authinfo/netrc content, I expect to get back this data."

I wasn't able to deduce the desired behavior on my own, unfortunately.

Checking the permissions of the netrc file sounds like a good feature.
For a GPG file it's not so important.

MA> - Everything I have forgotten to tell here ...

There's always a few of those... :)

Ted

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Christian Johansson]

Hi again

I stumbled on this alternative solution from the user-guide to John 
Wiegley async.el:
https://github.com/jwiegley/emacs-async

They instead suggest using:
(setq auth-sources '("~/.authinfo.gpg" "~/.netrc"))

instead of:
(setq ange-ftp-netrc-filename "~/.authinfo.gpg")

to avoid the problem of unnecessary user-interaction, especially in the 
case of async.el since it's spawned process doesn't support 
user-interaction.

Maybe both solutions are equivalent in solving the problem that authinfo 
doesn't automatically work with .netrc files and ange-ftp doesn't work 
with .authinfo files?

I think most people just use the tokens: machine, default, port, login, 
password and they seem to work already after setting either 
ange-ftp-netrc-filename or auth-sources variable.

Can you make one of those solution a default configuration and then 
perhaps add a override for people that want to use tokens that have 
special meaning the context of ange-ftp or authinfo it they are not 
fully compatible? Maybe there could be warnings occuring when authinfo 
stumbles upon a token it doesn't support and vice versa for ange-ftp?

Regards
Christian

On 2018-11-18 22:53, Ted Zlatanov wrote:
> On Fri, 09 Nov 2018 10:45:17 +0100 Michael Albinus <michael.albinus@gmx.de> wrote:
>
> MA> Requirements from ange-ftp.el would be at least
>
> MA> - Support of the "account" token.
> MA> - Support of the "default" token.
> MA> - Check of proper file permissions of "~/.netrc" or whatever file is
> MA>   specified (see ange-ftp-disable-netrc-security-check).
>
> MA> The "account" token is already parsed in auth-source.el, but it is not
> MA> divided from the "login" token. Both are handled via the common keyword
> MA> :user.
>
> MA> The "default" token is already parsed in auth-source.el, but I'm not
> MA> sure it is used in the sense ange-ftp.el expects it. In auth-source.el
> MA> this token is expanded to an entry of its own, with an unspecified
> MA> "machine". In ange-ftp.el, the tokens of "default" are used for all
> MA> other entries, replacing missing tokens. AFAIU.
>
> MA> Once this is prepared in auth-source.el, I would be able to integrate it
> MA> in ange-ftp.el. That is a more simple task; I've done this already for Tramp.
>
> Hi Michael,
>
> if you or a user could define the desired behavior as tests against
> auth-source.el (we already have that set up in the core), I'd be happy
> to implement them. Something like "for this call, with this
> authinfo/netrc content, I expect to get back this data."
>
> I wasn't able to deduce the desired behavior on my own, unfortunately.
>
> Checking the permissions of the netrc file sounds like a good feature.
> For a GPG file it's not so important.
>
> MA> - Everything I have forgotten to tell here ...
>
> There's always a few of those... :)
>
> Ted

bug#33146: .authinfo.gpg not automatically loaded when using (dired) on macOS

[Michael Albinus]

Christian Johansson <christian@cvj.se> writes:

> Hi again

Hi Christian,

> I stumbled on this alternative solution from the user-guide to John
> Wiegley async.el:
> https://github.com/jwiegley/emacs-async
>
> They instead suggest using:
> (setq auth-sources '("~/.authinfo.gpg" "~/.netrc"))
>
> instead of:
> (setq ange-ftp-netrc-filename "~/.authinfo.gpg")
>
> to avoid the problem of unnecessary user-interaction, especially in
> the case of async.el since it's spawned process doesn't support
> user-interaction.

That's the goal of making ange-ftp aware of auth-source.el.

> Maybe both solutions are equivalent in solving the problem that
> authinfo doesn't automatically work with .netrc files and ange-ftp
> doesn't work with .authinfo files?

Yes. In general, I believe the auth-source.el solution is superior,
because you could manage all your passwords wherever you want, even in
the Gnome Password Manager (in case you use the secrets.el backend of
auth-sources).

> I think most people just use the tokens: machine, default, port,
> login, password and they seem to work already after setting either
> ange-ftp-netrc-filename or auth-sources variable.

Right.

> Can you make one of those solution a default configuration and then
> perhaps add a override for people that want to use tokens that have
> special meaning the context of ange-ftp or authinfo it they are not
> fully compatible? Maybe there could be warnings occuring when authinfo
> stumbles upon a token it doesn't support and vice versa for ange-ftp?

In general, I don't want to support two different ways to handle
passwords in ange-ftp. That sounds good for trouble, because it
increases configuration complexity.

I prefer to switch to the auth-source.el solution in ange-ftp. With the
help of Ted, this shall be possible.

> Regards
> Christian

Best regards, Michael.