bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juan José García Ripoll]

Symptoms:
0. Install Gnu Privacy Guard for Windows or similar (non-msys) alternatives
1. Save login information in .authinfo.gpg
2. Launch Emacs on Windows
3. Launch Gnus (assumes an imap account is configured with a server in .authinfo.gpg)
4. Gnus cannot connect to the IMAP server.

What is happening:
1. Gnus calls nnimap-open-connection-1
2. nnimap-open-connection-1 binds (coding-system-for-read 'binary)
3. nnimap-credentials is invoked, which starts a chaing of calls ending
up in epg-config--make-gpg-configuration
4. This function uses call-process
5. Because the coding system is set to 'binary, the DOS ^M characters
are exposed and stored in the temporary buffer.
6. When epg-config--make-gpg-configuration parses the output from
gpg.exe, it stores those characters in the configuration and version
fields.
7. The system cannot parse those strings and concludes that there is no
suitable gpg.exe file in the system.

Workaround:
Invoke (epg-find-configuration 'OpenPGP) from .emacs. In this situation,
Emacs' call-process uses a suitable encoding to parse the output.

Fix:
a. Set a suitable value for the encoding around call-process in
epg-config--make-gpg-configuration. Unfortunately, I do not know which
value would work in non-Windows platforms.

b. Rewrite all regexps in epg-config--make-gpg-configuration to ignore
^M characters. Unfortunately I am unfamiliar with gpg.exe's output
format to do this with confidence.

c. Revert to previous behavior in epg-config.el

--- c:/Users/juanj/scoop/apps/emacs/27/share/emacs/27.0.90/lisp/epg-config.el	2020-03-26 23:37:05.096603400 +0100
+++ c:/Users/juanj/scoop/apps/emacs/27/share/emacs/27.0.90/lisp/epg-config-new.el	2020-03-27 00:06:26.138870700 +0100
@@ -181,7 +181,7 @@
 
 ;; Create an `epg-configuration' object for `gpg', using PROGRAM.
 (defun epg-config--make-gpg-configuration (program)
-  (let (config groups type args)
+  (let (config groups type args (coding-system-for-read nil))
     (with-temp-buffer
       (apply #'call-process program nil (list t nil) nil
 	     (append (if epg-gpg-home-directory



In GNU Emacs 27.0.90 (build 5, x86_64-w64-mingw32)
 of 2020-03-25 built on DESKTOP-3A8AAJ0
Repository revision: 4860530f3c130c6f854ea83dcc03f59e535a33ba
Repository branch: emacs-27
Windowing system distributor 'Microsoft Corp.', version 10.0.18363
System Description: Microsoft Windows 10 Pro for Workstations (v10.0.1909.18363.720)

Recent messages:
Sending email  
Sending email done
Sending...done
command-execute: Command attempted to use minibuffer while in minibuffer
e is undefined
p is undefined
Quit
Making completion list...

ESC M-x is undefined
Quit
Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: ESN
  locale-coding-system: cp1252

Major mode: Emacs-Lisp

Minor modes in effect:
  ido-vertical-mode: t
  save-place-mode: t
  savehist-mode: t
  gcmh-mode: t
  override-global-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(gnutls epa-file network-stream nsm smtpmail qp pp shadow sort vc-git
diff-mode mailalias bbdb-mua bbdb-com crm bbdb bbdb-site timezone
org-mime ox-org org-protocol ox-reveal cl ox-odt rng-loc rng-uri
rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox org-element avl-tree generator org org-macro org-footnote
org-pcomplete pcomplete org-list org-faces org-entities org-version
ob-python ob ob-tangle org-src ob-ref ob-lob ob-table ob-exp ob-comint
comint ansi-color ring ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat org-macs org-loaddefs noutline outline face-remap
mail-extr warnings emacsbug message rmc puny format-spec rfc822 mml
mml-sec epa derived epg epg-config mailabbrev gmm-utils mailheader
eieio-opt speedbar sb-image ezimage dframe cal-menu calendar
cal-loaddefs thingatpt help-fns radix-tree benchmark-init-modes
mm-decode mm-bodies mm-encode mail-parse rfc2231 debug backtrace
find-func mailcap pcase ido-vertical-mode ido gnus-win gnus nnheader
gnus-util rmail rmail-loaddefs text-property-search time-date sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils dired
dired-loaddefs grayscale-theme saveplace savehist edmacro kmacro
cus-edit cus-start cus-load wid-edit benchmark-init advice gcmh diminish
cl-extra help-mode use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key easy-mmode
use-package-core finder-inf tex-site info package easymenu browse-url
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads w32notify w32
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 411297 173966)
 (symbols 48 26521 16)
 (strings 32 148179 19593)
 (string-bytes 1 4021443)
 (vectors 16 50121)
 (vector-slots 8 1383982 345852)
 (floats 8 278 875)
 (intervals 56 1333 586)
 (buffers 1000 21))

-- 
Juan José García Ripoll

Quantum Information and Foundations Group
Institute of Fundamental Physics IFF-CSIC
Calle Serrano 113b, Madrid 28006 Spain
http://quinfog.hbar.es - http://juanjose.garcia.ripoll

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Juan José García Ripoll
>  <juanjose.garcia.ripoll@csic.es>
> Date: Fri, 27 Mar 2020 00:07:28 +0100
> 
> 1. Gnus calls nnimap-open-connection-1
> 2. nnimap-open-connection-1 binds (coding-system-for-read 'binary)
> 3. nnimap-credentials is invoked, which starts a chaing of calls ending
> up in epg-config--make-gpg-configuration
> 4. This function uses call-process

Can you provide more details regarding the chain of calls mentioned in
3 above?

> 5. Because the coding system is set to 'binary, the DOS ^M characters
> are exposed and stored in the temporary buffer.

If you use add-variable-watcher to watch any changes to
coding-system-for-read, does that allow to tell where does it get set
to 'binary' in this recipe?

> a. Set a suitable value for the encoding around call-process in
> epg-config--make-gpg-configuration. Unfortunately, I do not know which
> value would work in non-Windows platforms.

I don't think this is viable, and not only for Windows, because
there isn't sufficient context at this point to decide which encoding
to use.  This must be decided at a higher level, specifically where
coding-system-for-read is set to 'binary'.

> b. Rewrite all regexps in epg-config--make-gpg-configuration to ignore
> ^M characters.

That'd be a kludge in my book: working around our own bugs, instead of
fixing them.

> -  (let (config groups type args)
> +  (let (config groups type args (coding-system-for-read nil))

Setting coding-system-for-read to nil is not a good idea.  How do you
know what that means, and how it will affect the called process?

Thanks.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> Date: Fri, 27 Mar 2020 11:18:43 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 40248@debbugs.gnu.org

I understand you have some potentially useful backtraces which show
how epg-find-configuration is improperly called in this case.  If so,
please do provide any information about this you have.

Thanks.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
>> Date: Fri, 27 Mar 2020 11:18:43 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 40248@debbugs.gnu.org
>
> I understand you have some potentially useful backtraces which show
> how epg-find-configuration is improperly called in this case.  If so,
> please do provide any information about this you have.

epg-find-configuration is called via the backtrace below. I have edited
some personal information and domain names. The call chain looks
legit. The problems are in nnimap-open-connection-1 and upwards, where
the read encoding is set to binary.


  epg-config--make-gpg-configuration("c:/msys64/usr/bin/gpg.exe")
  epg-find-configuration(OpenPGP)
  epg-context--make(OpenPGP nil nil nil nil nil nil)
  epg-make-context()
  epa-file-insert-file-contents("c:/Users/me/OneDrive/Library/dot.authinfo.gpg" nil nil nil nil)
  apply(epa-file-insert-file-contents ("c:/Users/me/OneDrive/Library/dot.authinfo.gpg" nil nil nil nil))
  epa-file-handler(insert-file-contents "c:/Users/me/OneDrive/Library/dot.authinfo.gpg" nil nil nil nil)
  insert-file-contents("~/OneDrive/Library/dot.authinfo.gpg")
  auth-source-netrc-parse(:max 1 :require (:user :secret) :file "~/OneDrive/Library/dot.authinfo.gpg" :host ("mail" "mail.nowhere.org") :user t :port (993 "imaps" "imap" "993" "143"))
  auth-source-netrc-search(:backend #<auth-source-backend auth-source-backend-2dd74b0> :type netrc :max 1 :require (:user :secret) :create nil :delete nil :max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t)
  apply(auth-source-netrc-search :backend #<auth-source-backend auth-source-backend-2dd74b0> :type netrc :max 1 :require (:user :secret) :create nil :delete nil (:max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t))
  auth-source-search-backends((#<auth-source-backend auth-source-backend-2dd74b0>) (:max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t) 1 nil nil (:user :secret))
  auth-source-search(:max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t)
  nnimap-credentials(("mail" "mail.nowhere.org") (993 "imaps" "imap" "993" "143") nil)
  nnimap-open-connection-1(#<buffer  *nntpd*>)
  nnimap-open-connection(#<buffer  *nntpd*>)
  nnimap-open-server("mail" ((nnimap-address "mail.nowhere.org") (nnimap-server-port 993) (nnimap-stream ssl) (nnimap-fetch-partial-articles t) (nnmail-expiry-target "nnimap+mail:Trash") (nnir-search-engine imap)))
  gnus-open-server((nnimap "mail" (nnimap-address "mail.nowhere.org") (nnimap-server-port 993) (nnimap-stream ssl) (nnimap-fetch-partial-articles t) (nnmail-expiry-target "nnimap+mail:Trash") (nnir-search-engine imap)))
  gnus-get-unread-articles(nil nil)
  gnus-setup-news(nil nil nil)
  #f(compiled-function () #<bytecode 0x2dd4e65>)()
  gnus-1(nil nil nil)
  gnus(nil)
  funcall-interactively(gnus nil)
  call-interactively(gnus record nil)
  command-execute(gnus record)
  execute-extended-command(nil "gnus" "gnus")
  funcall-interactively(execute-extended-command nil "gnus" "gnus")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)


-- 
Juan José García Ripoll

Quantum Information and Foundations Group
Institute of Fundamental Physics IFF-CSIC
Calle Serrano 113b, Madrid 28006 Spain
http://quinfog.hbar.es - http://juanjose.garcia.ripoll

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garcia.ripoll@csic.es>
> Cc: 40248@debbugs.gnu.org
> Date: Fri, 27 Mar 2020 17:21:03 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Date: Fri, 27 Mar 2020 11:18:43 +0300
> >> From: Eli Zaretskii <eliz@gnu.org>
> >> Cc: 40248@debbugs.gnu.org
> >
> > I understand you have some potentially useful backtraces which show
> > how epg-find-configuration is improperly called in this case.  If so,
> > please do provide any information about this you have.
> 
> epg-find-configuration is called via the backtrace below. I have edited
> some personal information and domain names. The call chain looks
> legit. The problems are in nnimap-open-connection-1 and upwards, where
> the read encoding is set to binary.

Thanks, that is very useful.  Does the patch below give good results,
both in your Gnus scenario and when epg-find-configuration is called
in other contexts you are aware of and use?

Lars, can you or someone of the Gnus team tell why
nnimap-open-connection-1 binds coding-system-for-* to 'binary?  I
don't understand the rationale, as the code which uses this connection
seems to expect ASCII text in response.  What am I missing?  (I tried
to find the change(s) which introduced those bindings, but that seems
to have been done when Gnus was maintained on a separate Git
repository, and the relevant commits aren't visible in the Emacs
repository.)

diff --git a/lisp/epg-config.el b/lisp/epg-config.el
index 74ab651..daa9a5a 100644
--- a/lisp/epg-config.el
+++ b/lisp/epg-config.el
@@ -183,10 +183,18 @@ epg-find-configuration
 (defun epg-config--make-gpg-configuration (program)
   (let (config groups type args)
     (with-temp-buffer
-      (apply #'call-process program nil (list t nil) nil
-	     (append (if epg-gpg-home-directory
-			 (list "--homedir" epg-gpg-home-directory))
-		     '("--with-colons" "--list-config")))
+      ;; The caller might have bound coding-system-for-* to something
+      ;; like 'no-conversion, but the below needs to call PROGRAM
+      ;; expecting human-readable text in both directions (since we
+      ;; are going to parse the output as text), so let Emacs guess
+      ;; the encoding of that text by its usual encoding-detection
+      ;; machinery.
+      (let ((coding-system-for-read 'undecided)
+            (coding-system-for-write 'undecided))
+        (apply #'call-process program nil (list t nil) nil
+	       (append (if epg-gpg-home-directory
+			   (list "--homedir" epg-gpg-home-directory))
+		       '("--with-colons" "--list-config"))))
       (goto-char (point-min))
       (while (re-search-forward "^cfg:\\([^:]+\\):\\(.*\\)" nil t)
 	(setq type (intern (match-string 1))

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Lars, can you or someone of the Gnus team tell why
> nnimap-open-connection-1 binds coding-system-for-* to 'binary?  I
> don't understand the rationale, as the code which uses this connection
> seems to expect ASCII text in response.

The IMAP connections (as do most network protocols) operate on octets,
not ASCII.  It is common, though, for most email messages to use some
transfer encoding (so that it's mostly ASCII in practice), but anything
can be transferred via IMAP.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Juan José García-Ripoll
>  <juanjose.garcia.ripoll@csic.es>,
>   40248@debbugs.gnu.org
> Date: Sat, 28 Mar 2020 09:40:12 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Lars, can you or someone of the Gnus team tell why
> > nnimap-open-connection-1 binds coding-system-for-* to 'binary?  I
> > don't understand the rationale, as the code which uses this connection
> > seems to expect ASCII text in response.
> 
> The IMAP connections (as do most network protocols) operate on octets,
> not ASCII.

But then only the encoding of the network process started by
nnimap-open-connection-1 should be made no-conversion.  By contrast,
binding coding-system-for-* around the call to open-network-stream has
much wider consequences, as it affects _any_ code run as part of
open-network-stream, in particularly epg-find-configuration, which has
nothing to do with the network stream.  nnimap-open-connection-1 is a
large function, so that binding affects quite a lot more than just the
network process it creates.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> But then only the encoding of the network process started by
> nnimap-open-connection-1 should be made no-conversion.

I'm not sure I quite follow; if I misunderstand, I apologise:

Do you mean as a :coding parameter to make-network-process?
nnimap-open-connection-1 doesn't call that function directly, and the
functions it does call doesn't take any :coding parameters.

> By contrast, binding coding-system-for-* around the call to
> open-network-stream has much wider consequences, as it affects _any_
> code run as part of open-network-stream, in particularly
> epg-find-configuration, which has nothing to do with the network
> stream.

Yes, that's unfortunate -- the "bind variables as a way of passing in
parameters" thing that Emacs does a lot is a really bad one.  I think
the fix here would be to change open-network-stream, at least, to have a
:coding parameter (and pass it on).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: juanjose.garcia.ripoll@csic.es,  40248@debbugs.gnu.org
> Date: Sun, 29 Mar 2020 09:45:55 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But then only the encoding of the network process started by
> > nnimap-open-connection-1 should be made no-conversion.
> 
> I'm not sure I quite follow; if I misunderstand, I apologise:
> 
> Do you mean as a :coding parameter to make-network-process?
> nnimap-open-connection-1 doesn't call that function directly, and the
> functions it does call doesn't take any :coding parameters.

We could extend open-network-stream to accept :coding and pass that to
make-network-process.  Or we could call set-process-coding-system
right after open-network-stream returns.

> > By contrast, binding coding-system-for-* around the call to
> > open-network-stream has much wider consequences, as it affects _any_
> > code run as part of open-network-stream, in particularly
> > epg-find-configuration, which has nothing to do with the network
> > stream.
> 
> Yes, that's unfortunate -- the "bind variables as a way of passing in
> parameters" thing that Emacs does a lot is a really bad one.  I think
> the fix here would be to change open-network-stream, at least, to have a
> :coding parameter (and pass it on).

Yes, let's do that on master.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juan Jose Garcia Ripoll]

[-- Attachment #1: Sólo texto --]
[-- Type: text/plain, Size: 566 bytes --]

  Eli Zaretskii <eliz@gnu.org> escribió:

> Thanks, that is very useful.  Does the patch below give good results,
> both in your Gnus scenario and when epg-find-configuration is called
> in other contexts you are aware of and use?

To me it works perfectly. No errors reported, neither here, nor in  
separate invocations. Thanks!
   Juan José García Ripoll

Instituto de Física Fundamental, CSIC
Calle Serrano 113b, Madrid 28006, Spain
Phone: (+34) 915616800 (dial 943107 when hearing the voice)
http://quinfog.iff.csic.es / http://juanjose.garciaripoll.com

[-- Attachment #2: Mensaje HTML --]
[-- Type: text/html, Size: 1303 bytes --]

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 15:03:14 +0100
> From: Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es>
> Cc: 40248@debbugs.gnu.org
> 
>  Thanks, that is very useful.  Does the patch below give good results,
>  both in your Gnus scenario and when epg-find-configuration is called
>  in other contexts you are aware of and use?
> 
> To me it works perfectly. No errors reported, neither here, nor in separate invocations. Thanks!

Thanks for testing.  I will wait for Lars to respond to my last
message in this thread, and then decide what to do about this problem.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> +      ;; The caller might have bound coding-system-for-* to something
> +      ;; like 'no-conversion, but the below needs to call PROGRAM
> +      ;; expecting human-readable text in both directions (since we
> +      ;; are going to parse the output as text), so let Emacs guess
> +      ;; the encoding of that text by its usual encoding-detection
> +      ;; machinery.
> +      (let ((coding-system-for-read 'undecided)
> +            (coding-system-for-write 'undecided))

I think this probably is the right thing here, but I'm not 100% sure --
I seem to remember there being some issue of people putting non-ASCII
stuff in the name parts of the gpg data and then Emacs having a problem
of matching that up to the data we're looking for...

I may be misremembering, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Juan José García-Ripoll
>  <juanjose.garcia.ripoll@csic.es>,
>   40248@debbugs.gnu.org
> Date: Sun, 29 Mar 2020 09:48:59 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > +      ;; The caller might have bound coding-system-for-* to something
> > +      ;; like 'no-conversion, but the below needs to call PROGRAM
> > +      ;; expecting human-readable text in both directions (since we
> > +      ;; are going to parse the output as text), so let Emacs guess
> > +      ;; the encoding of that text by its usual encoding-detection
> > +      ;; machinery.
> > +      (let ((coding-system-for-read 'undecided)
> > +            (coding-system-for-write 'undecided))
> 
> I think this probably is the right thing here, but I'm not 100% sure --
> I seem to remember there being some issue of people putting non-ASCII
> stuff in the name parts of the gpg data and then Emacs having a problem
> of matching that up to the data we're looking for...

If that non-ASCII data is compatible with the default encoding, or if
Emacs will detect the encoding correctly given 'undecided', that
shouldn't be any problem.  And this function is only about getting the
gpg configuration, so what kind of non-ASCII data is expected there?
And how would using 'no-conversion' here help?

If you can find those cases where you saw non-ASCII data in this case,
by all means describe them or point to relevant discussions.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Sun, 29 Mar 2020 16:52:18 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Lars Ingebrigtsen <larsi@gnus.org>
    >> Cc: Juan José García-Ripoll
    >> <juanjose.garcia.ripoll@csic.es>,
    >> 40248@debbugs.gnu.org
    >> Date: Sun, 29 Mar 2020 09:48:59 +0200
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > +      ;; The caller might have bound coding-system-for-* to something
    >> > +      ;; like 'no-conversion, but the below needs to call PROGRAM
    >> > +      ;; expecting human-readable text in both directions (since we
    >> > +      ;; are going to parse the output as text), so let Emacs guess
    >> > +      ;; the encoding of that text by its usual encoding-detection
    >> > +      ;; machinery.
    >> > +      (let ((coding-system-for-read 'undecided)
    >> > +            (coding-system-for-write 'undecided))
    >> 
    >> I think this probably is the right thing here, but I'm not 100% sure --
    >> I seem to remember there being some issue of people putting non-ASCII
    >> stuff in the name parts of the gpg data and then Emacs having a problem
    >> of matching that up to the data we're looking for...

    Eli> If that non-ASCII data is compatible with the default encoding, or if
    Eli> Emacs will detect the encoding correctly given 'undecided', that
    Eli> shouldn't be any problem.  And this function is only about getting the
    Eli> gpg configuration, so what kind of non-ASCII data is expected there?
    Eli> And how would using 'no-conversion' here help?

    Eli> If you can find those cases where you saw non-ASCII data in this case,
    Eli> by all means describe them or point to relevant discussions.

My (admittedly fallible) memory is that gpg always uses UTF-8 for
non-ASCII data (except for some old versions that %-escape it instead).

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  juanjose.garcia.ripoll@csic.es,
>   40248@debbugs.gnu.org
> Date: Mon, 30 Mar 2020 11:37:25 +0200
> 
>     Eli> If that non-ASCII data is compatible with the default encoding, or if
>     Eli> Emacs will detect the encoding correctly given 'undecided', that
>     Eli> shouldn't be any problem.  And this function is only about getting the
>     Eli> gpg configuration, so what kind of non-ASCII data is expected there?
>     Eli> And how would using 'no-conversion' here help?
> 
>     Eli> If you can find those cases where you saw non-ASCII data in this case,
>     Eli> by all means describe them or point to relevant discussions.
> 
> My (admittedly fallible) memory is that gpg always uses UTF-8 for
> non-ASCII data (except for some old versions that %-escape it instead).

Is that true even on MS-Windows?  can someone please verify that?  If
gpg uses UTF-8 on all platforms, the 'undecided' isn't TRT, as in some
cases Emacs could mistakenly decide the encoding is the current system
codepage.  We should use 'utf-8' instead if UTF-8 is guaranteed.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Mon, 30 Mar 2020 16:10:15 +0300, Eli Zaretskii <eliz@gnu.org> said:
    Eli> If you can find those cases where you saw non-ASCII data in this case,
    Eli> by all means describe them or point to relevant discussions.
    >> 
    >> My (admittedly fallible) memory is that gpg always uses UTF-8 for
    >> non-ASCII data (except for some old versions that %-escape it instead).

    Eli> Is that true even on MS-Windows?  can someone please verify that?  If
    Eli> gpg uses UTF-8 on all platforms, the 'undecided' isn't TRT, as in some
    Eli> cases Emacs could mistakenly decide the encoding is the current system
    Eli> codepage.  We should use 'utf-8' instead if UTF-8 is guaranteed.

Having now re-checked it, I was wrong. gpg uses UTF-8 consistenly
_internally_, but converts to/from whatever it thinks the native
codepage is (on MS-Windows and unixy platforms).

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: larsi@gnus.org,  juanjose.garcia.ripoll@csic.es,  40248@debbugs.gnu.org
> Date: Mon, 30 Mar 2020 16:10:34 +0200
> 
>     >> My (admittedly fallible) memory is that gpg always uses UTF-8 for
>     >> non-ASCII data (except for some old versions that %-escape it instead).
> 
>     Eli> Is that true even on MS-Windows?  can someone please verify that?  If
>     Eli> gpg uses UTF-8 on all platforms, the 'undecided' isn't TRT, as in some
>     Eli> cases Emacs could mistakenly decide the encoding is the current system
>     Eli> codepage.  We should use 'utf-8' instead if UTF-8 is guaranteed.
> 
> Having now re-checked it, I was wrong. gpg uses UTF-8 consistenly
> _internally_, but converts to/from whatever it thinks the native
> codepage is (on MS-Windows and unixy platforms).

Thanks.  In that case, 'undecided' is exactly right.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  In that case, 'undecided' is exactly right.

Yeah, sounds like it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Robert Pluim <rpluim@gmail.com>,  juanjose.garcia.ripoll@csic.es,
>   40248@debbugs.gnu.org
> Date: Thu, 02 Apr 2020 13:11:27 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  In that case, 'undecided' is exactly right.
> 
> Yeah, sounds like it.

Thanks, pushed to the emacs-27 branch.

I'm not closing the bug because we still didn't make the more invasive
change on master, AFAIR.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Fri, 03 Apr 2020 14:31:43 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Lars Ingebrigtsen <larsi@gnus.org>
    >> Cc: Robert Pluim <rpluim@gmail.com>,  juanjose.garcia.ripoll@csic.es,
    >> 40248@debbugs.gnu.org
    >> Date: Thu, 02 Apr 2020 13:11:27 +0200
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > Thanks.  In that case, 'undecided' is exactly right.
    >> 
    >> Yeah, sounds like it.

    Eli> Thanks, pushed to the emacs-27 branch.

    Eli> I'm not closing the bug because we still didn't make the more invasive
    Eli> change on master, AFAIR.

I was waiting to see if Lars or anybody had any more comments. I can
post the complete patch over in emacs-devel.

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>     Eli> I'm not closing the bug because we still didn't make the more invasive
>     Eli> change on master, AFAIR.
>
> I was waiting to see if Lars or anybody had any more comments. I can
> post the complete patch over in emacs-devel.

I see that Robert added:

commit 6382e1330814ca4df20eeccd8b4ef9ca17b997af
Author: Robert Pluim <rpluim@gmail.com>
Date:   Thu Apr 2 18:41:33 2020 +0200

    Add :coding support to open-network-stream and open-gnutls-stream

I thought that was the "more invasive" change, but it's committed the
day before the final exchange in this bug report.

So I'm not quite sure after skimming this thread now...  has this all
been resolved, or were there any further work to be done?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  juanjose.garcia.ripoll@csic.es,
>   40248@debbugs.gnu.org
> Date: Sun, 19 Jul 2020 04:45:59 +0200
> 
> Robert Pluim <rpluim@gmail.com> writes:
> 
> >     Eli> I'm not closing the bug because we still didn't make the more invasive
> >     Eli> change on master, AFAIR.
> >
> > I was waiting to see if Lars or anybody had any more comments. I can
> > post the complete patch over in emacs-devel.
> 
> I see that Robert added:
> 
> commit 6382e1330814ca4df20eeccd8b4ef9ca17b997af
> Author: Robert Pluim <rpluim@gmail.com>
> Date:   Thu Apr 2 18:41:33 2020 +0200
> 
>     Add :coding support to open-network-stream and open-gnutls-stream
> 
> I thought that was the "more invasive" change, but it's committed the
> day before the final exchange in this bug report.

No, it was committed on

  CommitDate: Fri Apr 3 14:45:49 2020 +0200

which is 2 minutes _after_ that exchange.

The date you show above is AuthorDate, i.e. the date when Robert
committed this locally on his machine.  (This confusion is precisely
the reason why I customized "git log" to show both of the dates.)

> So I'm not quite sure after skimming this thread now...  has this all
> been resolved, or were there any further work to be done?

The "invasive" change is the above one; the less invasive one, which
was at the time done on emacs-27, was this one:

  commit fa823653ffb0e3e893d30daa5abf68e909934e2e
  Author:     Eli Zaretskii <eliz@gnu.org>
  AuthorDate: Fri Apr 3 14:29:49 2020 +0300
  Commit:     Eli Zaretskii <eliz@gnu.org>
  CommitDate: Fri Apr 3 14:29:49 2020 +0300

      Fix invocations of gpg from Gnus

      * lisp/epg-config.el (epg-config--make-gpg-configuration): Bind
      coding-system-for-read/write to 'undecided', to countermand
      possible values of 'no-conversion' or somesuch by the callers.
      (Bug#40248)

So I think this bug can be closed.  Robert?

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> No, it was committed on
>
>   CommitDate: Fri Apr 3 14:45:49 2020 +0200
>
> which is 2 minutes _after_ that exchange.

Heh.

> The date you show above is AuthorDate, i.e. the date when Robert
> committed this locally on his machine.  (This confusion is precisely
> the reason why I customized "git log" to show both of the dates.)

That sounds like a very useful thing to have.  How do you make vc-mode
(etc) show both dates?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rpluim@gmail.com,  juanjose.garcia.ripoll@csic.es,  40248@debbugs.gnu.org
> Date: Sun, 19 Jul 2020 16:29:44 +0200
> 
> > The date you show above is AuthorDate, i.e. the date when Robert
> > committed this locally on his machine.  (This confusion is precisely
> > the reason why I customized "git log" to show both of the dates.)
> 
> That sounds like a very useful thing to have.  How do you make vc-mode
> (etc) show both dates?

You don't need to tell VC anything, just say in your ~/.gitconfig

  [format]
	  pretty = fuller

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> You don't need to tell VC anything, just say in your ~/.gitconfig
>
>   [format]
> 	  pretty = fuller

Thanks!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juri Linkov]

>> That sounds like a very useful thing to have.  How do you make vc-mode
>> (etc) show both dates?
>
> You don't need to tell VC anything, just say in your ~/.gitconfig
>
>   [format]
> 	  pretty = fuller

This setting breaks vc-git-log-view-mode: typing 'D' in the log buffer
with "CommitDate" throws the error:

  fatal: bad revision 'Date'

This is because it searches for the commit number and finds "CommitDate"
as in this example:

  commit 365e01cc9f64ce6ca947ccfd8612d60763280a37
  CommitDate: 2020-01-01 00:59:52 +0000

The problem is that 'vc-git-log-view-mode' sets 'log-view-message-re'
to this regexp:

  "^commit *\\([0-9a-z]+\\)"

where "\\([0-9a-z]+\\)" matches "Date" in "CommitDate".

This patch fixes it, but I'm not sure if this is the correct regexp
for git commits:

diff --git a/lisp/vc/vc-git.el b/lisp/vc/vc-git.el
index 9b8151705f..2220a33188 100644
--- a/lisp/vc/vc-git.el
+++ b/lisp/vc/vc-git.el
@@ -1241,7 +1241,7 @@ vc-git-log-view-mode
   (set (make-local-variable 'log-view-message-re)
        (if (not (memq vc-log-view-type '(long log-search with-diff)))
 	   (cadr vc-git-root-log-format)
-	 "^commit *\\([0-9a-z]+\\)"))
+	 "^commit +\\([0-9a-z]+\\)"))
   ;; Allow expanding short log entries.
   (when (memq vc-log-view-type '(short log-outgoing log-incoming mergebase))
     (setq truncate-lines t)

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Dmitry Gutov]

On 23.07.2020 03:20, Juri Linkov wrote:
> This patch fixes it, but I'm not sure if this is the correct regexp
> for git commits:

It looks like an improvement either way. Please install.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Sun, 19 Jul 2020 17:23:40 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> 
    >> I thought that was the "more invasive" change, but it's committed the
    >> day before the final exchange in this bug report.

That was half of the more invasive change. The second half is getting
gnus and other users of 'open-network-stream' to pass :coding to it,
rather than binding 'coding-system-for-{read-write}'. Thatʼs currently
sitting in my workspace waiting for me to finish testing it.

    Eli> So I think this bug can be closed.  Robert?

Yes. This bug is fixed, I just need to finish off the gnus changes.

Robert