bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juan José García Ripoll]

Symptoms:
0. Install Gnu Privacy Guard for Windows or similar (non-msys) alternatives
1. Save login information in .authinfo.gpg
2. Launch Emacs on Windows
3. Launch Gnus (assumes an imap account is configured with a server in .authinfo.gpg)
4. Gnus cannot connect to the IMAP server.

What is happening:
1. Gnus calls nnimap-open-connection-1
2. nnimap-open-connection-1 binds (coding-system-for-read 'binary)
3. nnimap-credentials is invoked, which starts a chaing of calls ending
up in epg-config--make-gpg-configuration
4. This function uses call-process
5. Because the coding system is set to 'binary, the DOS ^M characters
are exposed and stored in the temporary buffer.
6. When epg-config--make-gpg-configuration parses the output from
gpg.exe, it stores those characters in the configuration and version
fields.
7. The system cannot parse those strings and concludes that there is no
suitable gpg.exe file in the system.

Workaround:
Invoke (epg-find-configuration 'OpenPGP) from .emacs. In this situation,
Emacs' call-process uses a suitable encoding to parse the output.

Fix:
a. Set a suitable value for the encoding around call-process in
epg-config--make-gpg-configuration. Unfortunately, I do not know which
value would work in non-Windows platforms.

b. Rewrite all regexps in epg-config--make-gpg-configuration to ignore
^M characters. Unfortunately I am unfamiliar with gpg.exe's output
format to do this with confidence.

c. Revert to previous behavior in epg-config.el

--- c:/Users/juanj/scoop/apps/emacs/27/share/emacs/27.0.90/lisp/epg-config.el	2020-03-26 23:37:05.096603400 +0100
+++ c:/Users/juanj/scoop/apps/emacs/27/share/emacs/27.0.90/lisp/epg-config-new.el	2020-03-27 00:06:26.138870700 +0100
@@ -181,7 +181,7 @@
 
 ;; Create an `epg-configuration' object for `gpg', using PROGRAM.
 (defun epg-config--make-gpg-configuration (program)
-  (let (config groups type args)
+  (let (config groups type args (coding-system-for-read nil))
     (with-temp-buffer
       (apply #'call-process program nil (list t nil) nil
 	     (append (if epg-gpg-home-directory



In GNU Emacs 27.0.90 (build 5, x86_64-w64-mingw32)
 of 2020-03-25 built on DESKTOP-3A8AAJ0
Repository revision: 4860530f3c130c6f854ea83dcc03f59e535a33ba
Repository branch: emacs-27
Windowing system distributor 'Microsoft Corp.', version 10.0.18363
System Description: Microsoft Windows 10 Pro for Workstations (v10.0.1909.18363.720)

Recent messages:
Sending email  
Sending email done
Sending...done
command-execute: Command attempted to use minibuffer while in minibuffer
e is undefined
p is undefined
Quit
Making completion list...

ESC M-x is undefined
Quit
Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: ESN
  locale-coding-system: cp1252

Major mode: Emacs-Lisp

Minor modes in effect:
  ido-vertical-mode: t
  save-place-mode: t
  savehist-mode: t
  gcmh-mode: t
  override-global-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(gnutls epa-file network-stream nsm smtpmail qp pp shadow sort vc-git
diff-mode mailalias bbdb-mua bbdb-com crm bbdb bbdb-site timezone
org-mime ox-org org-protocol ox-reveal cl ox-odt rng-loc rng-uri
rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox org-element avl-tree generator org org-macro org-footnote
org-pcomplete pcomplete org-list org-faces org-entities org-version
ob-python ob ob-tangle org-src ob-ref ob-lob ob-table ob-exp ob-comint
comint ansi-color ring ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat org-macs org-loaddefs noutline outline face-remap
mail-extr warnings emacsbug message rmc puny format-spec rfc822 mml
mml-sec epa derived epg epg-config mailabbrev gmm-utils mailheader
eieio-opt speedbar sb-image ezimage dframe cal-menu calendar
cal-loaddefs thingatpt help-fns radix-tree benchmark-init-modes
mm-decode mm-bodies mm-encode mail-parse rfc2231 debug backtrace
find-func mailcap pcase ido-vertical-mode ido gnus-win gnus nnheader
gnus-util rmail rmail-loaddefs text-property-search time-date sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils dired
dired-loaddefs grayscale-theme saveplace savehist edmacro kmacro
cus-edit cus-start cus-load wid-edit benchmark-init advice gcmh diminish
cl-extra help-mode use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key easy-mmode
use-package-core finder-inf tex-site info package easymenu browse-url
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads w32notify w32
lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 411297 173966)
 (symbols 48 26521 16)
 (strings 32 148179 19593)
 (string-bytes 1 4021443)
 (vectors 16 50121)
 (vector-slots 8 1383982 345852)
 (floats 8 278 875)
 (intervals 56 1333 586)
 (buffers 1000 21))

-- 
Juan José García Ripoll

Quantum Information and Foundations Group
Institute of Fundamental Physics IFF-CSIC
Calle Serrano 113b, Madrid 28006 Spain
http://quinfog.hbar.es - http://juanjose.garcia.ripoll

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Juan José García Ripoll
>  <juanjose.garcia.ripoll@csic.es>
> Date: Fri, 27 Mar 2020 00:07:28 +0100
> 
> 1. Gnus calls nnimap-open-connection-1
> 2. nnimap-open-connection-1 binds (coding-system-for-read 'binary)
> 3. nnimap-credentials is invoked, which starts a chaing of calls ending
> up in epg-config--make-gpg-configuration
> 4. This function uses call-process

Can you provide more details regarding the chain of calls mentioned in
3 above?

> 5. Because the coding system is set to 'binary, the DOS ^M characters
> are exposed and stored in the temporary buffer.

If you use add-variable-watcher to watch any changes to
coding-system-for-read, does that allow to tell where does it get set
to 'binary' in this recipe?

> a. Set a suitable value for the encoding around call-process in
> epg-config--make-gpg-configuration. Unfortunately, I do not know which
> value would work in non-Windows platforms.

I don't think this is viable, and not only for Windows, because
there isn't sufficient context at this point to decide which encoding
to use.  This must be decided at a higher level, specifically where
coding-system-for-read is set to 'binary'.

> b. Rewrite all regexps in epg-config--make-gpg-configuration to ignore
> ^M characters.

That'd be a kludge in my book: working around our own bugs, instead of
fixing them.

> -  (let (config groups type args)
> +  (let (config groups type args (coding-system-for-read nil))

Setting coding-system-for-read to nil is not a good idea.  How do you
know what that means, and how it will affect the called process?

Thanks.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> Date: Fri, 27 Mar 2020 11:18:43 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 40248@debbugs.gnu.org

I understand you have some potentially useful backtraces which show
how epg-find-configuration is improperly called in this case.  If so,
please do provide any information about this you have.

Thanks.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
>> Date: Fri, 27 Mar 2020 11:18:43 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 40248@debbugs.gnu.org
>
> I understand you have some potentially useful backtraces which show
> how epg-find-configuration is improperly called in this case.  If so,
> please do provide any information about this you have.

epg-find-configuration is called via the backtrace below. I have edited
some personal information and domain names. The call chain looks
legit. The problems are in nnimap-open-connection-1 and upwards, where
the read encoding is set to binary.


  epg-config--make-gpg-configuration("c:/msys64/usr/bin/gpg.exe")
  epg-find-configuration(OpenPGP)
  epg-context--make(OpenPGP nil nil nil nil nil nil)
  epg-make-context()
  epa-file-insert-file-contents("c:/Users/me/OneDrive/Library/dot.authinfo.gpg" nil nil nil nil)
  apply(epa-file-insert-file-contents ("c:/Users/me/OneDrive/Library/dot.authinfo.gpg" nil nil nil nil))
  epa-file-handler(insert-file-contents "c:/Users/me/OneDrive/Library/dot.authinfo.gpg" nil nil nil nil)
  insert-file-contents("~/OneDrive/Library/dot.authinfo.gpg")
  auth-source-netrc-parse(:max 1 :require (:user :secret) :file "~/OneDrive/Library/dot.authinfo.gpg" :host ("mail" "mail.nowhere.org") :user t :port (993 "imaps" "imap" "993" "143"))
  auth-source-netrc-search(:backend #<auth-source-backend auth-source-backend-2dd74b0> :type netrc :max 1 :require (:user :secret) :create nil :delete nil :max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t)
  apply(auth-source-netrc-search :backend #<auth-source-backend auth-source-backend-2dd74b0> :type netrc :max 1 :require (:user :secret) :create nil :delete nil (:max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t))
  auth-source-search-backends((#<auth-source-backend auth-source-backend-2dd74b0>) (:max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t) 1 nil nil (:user :secret))
  auth-source-search(:max 1 :host ("mail" "mail.nowhere.org") :port (993 "imaps" "imap" "993" "143") :user nil :require (:user :secret) :create t)
  nnimap-credentials(("mail" "mail.nowhere.org") (993 "imaps" "imap" "993" "143") nil)
  nnimap-open-connection-1(#<buffer  *nntpd*>)
  nnimap-open-connection(#<buffer  *nntpd*>)
  nnimap-open-server("mail" ((nnimap-address "mail.nowhere.org") (nnimap-server-port 993) (nnimap-stream ssl) (nnimap-fetch-partial-articles t) (nnmail-expiry-target "nnimap+mail:Trash") (nnir-search-engine imap)))
  gnus-open-server((nnimap "mail" (nnimap-address "mail.nowhere.org") (nnimap-server-port 993) (nnimap-stream ssl) (nnimap-fetch-partial-articles t) (nnmail-expiry-target "nnimap+mail:Trash") (nnir-search-engine imap)))
  gnus-get-unread-articles(nil nil)
  gnus-setup-news(nil nil nil)
  #f(compiled-function () #<bytecode 0x2dd4e65>)()
  gnus-1(nil nil nil)
  gnus(nil)
  funcall-interactively(gnus nil)
  call-interactively(gnus record nil)
  command-execute(gnus record)
  execute-extended-command(nil "gnus" "gnus")
  funcall-interactively(execute-extended-command nil "gnus" "gnus")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)


-- 
Juan José García Ripoll

Quantum Information and Foundations Group
Institute of Fundamental Physics IFF-CSIC
Calle Serrano 113b, Madrid 28006 Spain
http://quinfog.hbar.es - http://juanjose.garcia.ripoll

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garcia.ripoll@csic.es>
> Cc: 40248@debbugs.gnu.org
> Date: Fri, 27 Mar 2020 17:21:03 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> Date: Fri, 27 Mar 2020 11:18:43 +0300
> >> From: Eli Zaretskii <eliz@gnu.org>
> >> Cc: 40248@debbugs.gnu.org
> >
> > I understand you have some potentially useful backtraces which show
> > how epg-find-configuration is improperly called in this case.  If so,
> > please do provide any information about this you have.
> 
> epg-find-configuration is called via the backtrace below. I have edited
> some personal information and domain names. The call chain looks
> legit. The problems are in nnimap-open-connection-1 and upwards, where
> the read encoding is set to binary.

Thanks, that is very useful.  Does the patch below give good results,
both in your Gnus scenario and when epg-find-configuration is called
in other contexts you are aware of and use?

Lars, can you or someone of the Gnus team tell why
nnimap-open-connection-1 binds coding-system-for-* to 'binary?  I
don't understand the rationale, as the code which uses this connection
seems to expect ASCII text in response.  What am I missing?  (I tried
to find the change(s) which introduced those bindings, but that seems
to have been done when Gnus was maintained on a separate Git
repository, and the relevant commits aren't visible in the Emacs
repository.)

diff --git a/lisp/epg-config.el b/lisp/epg-config.el
index 74ab651..daa9a5a 100644
--- a/lisp/epg-config.el
+++ b/lisp/epg-config.el
@@ -183,10 +183,18 @@ epg-find-configuration
 (defun epg-config--make-gpg-configuration (program)
   (let (config groups type args)
     (with-temp-buffer
-      (apply #'call-process program nil (list t nil) nil
-	     (append (if epg-gpg-home-directory
-			 (list "--homedir" epg-gpg-home-directory))
-		     '("--with-colons" "--list-config")))
+      ;; The caller might have bound coding-system-for-* to something
+      ;; like 'no-conversion, but the below needs to call PROGRAM
+      ;; expecting human-readable text in both directions (since we
+      ;; are going to parse the output as text), so let Emacs guess
+      ;; the encoding of that text by its usual encoding-detection
+      ;; machinery.
+      (let ((coding-system-for-read 'undecided)
+            (coding-system-for-write 'undecided))
+        (apply #'call-process program nil (list t nil) nil
+	       (append (if epg-gpg-home-directory
+			   (list "--homedir" epg-gpg-home-directory))
+		       '("--with-colons" "--list-config"))))
       (goto-char (point-min))
       (while (re-search-forward "^cfg:\\([^:]+\\):\\(.*\\)" nil t)
 	(setq type (intern (match-string 1))

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Lars, can you or someone of the Gnus team tell why
> nnimap-open-connection-1 binds coding-system-for-* to 'binary?  I
> don't understand the rationale, as the code which uses this connection
> seems to expect ASCII text in response.

The IMAP connections (as do most network protocols) operate on octets,
not ASCII.  It is common, though, for most email messages to use some
transfer encoding (so that it's mostly ASCII in practice), but anything
can be transferred via IMAP.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Juan José García-Ripoll
>  <juanjose.garcia.ripoll@csic.es>,
>   40248@debbugs.gnu.org
> Date: Sat, 28 Mar 2020 09:40:12 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Lars, can you or someone of the Gnus team tell why
> > nnimap-open-connection-1 binds coding-system-for-* to 'binary?  I
> > don't understand the rationale, as the code which uses this connection
> > seems to expect ASCII text in response.
> 
> The IMAP connections (as do most network protocols) operate on octets,
> not ASCII.

But then only the encoding of the network process started by
nnimap-open-connection-1 should be made no-conversion.  By contrast,
binding coding-system-for-* around the call to open-network-stream has
much wider consequences, as it affects _any_ code run as part of
open-network-stream, in particularly epg-find-configuration, which has
nothing to do with the network stream.  nnimap-open-connection-1 is a
large function, so that binding affects quite a lot more than just the
network process it creates.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juan Jose Garcia Ripoll]

[-- Attachment #1: Sólo texto --]
[-- Type: text/plain, Size: 566 bytes --]

  Eli Zaretskii <eliz@gnu.org> escribió:

> Thanks, that is very useful.  Does the patch below give good results,
> both in your Gnus scenario and when epg-find-configuration is called
> in other contexts you are aware of and use?

To me it works perfectly. No errors reported, neither here, nor in  
separate invocations. Thanks!
   Juan José García Ripoll

Instituto de Física Fundamental, CSIC
Calle Serrano 113b, Madrid 28006, Spain
Phone: (+34) 915616800 (dial 943107 when hearing the voice)
http://quinfog.iff.csic.es / http://juanjose.garciaripoll.com

[-- Attachment #2: Mensaje HTML --]
[-- Type: text/html, Size: 1303 bytes --]

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 15:03:14 +0100
> From: Juan Jose Garcia Ripoll <juanjose.garcia.ripoll@csic.es>
> Cc: 40248@debbugs.gnu.org
> 
>  Thanks, that is very useful.  Does the patch below give good results,
>  both in your Gnus scenario and when epg-find-configuration is called
>  in other contexts you are aware of and use?
> 
> To me it works perfectly. No errors reported, neither here, nor in separate invocations. Thanks!

Thanks for testing.  I will wait for Lars to respond to my last
message in this thread, and then decide what to do about this problem.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> But then only the encoding of the network process started by
> nnimap-open-connection-1 should be made no-conversion.

I'm not sure I quite follow; if I misunderstand, I apologise:

Do you mean as a :coding parameter to make-network-process?
nnimap-open-connection-1 doesn't call that function directly, and the
functions it does call doesn't take any :coding parameters.

> By contrast, binding coding-system-for-* around the call to
> open-network-stream has much wider consequences, as it affects _any_
> code run as part of open-network-stream, in particularly
> epg-find-configuration, which has nothing to do with the network
> stream.

Yes, that's unfortunate -- the "bind variables as a way of passing in
parameters" thing that Emacs does a lot is a really bad one.  I think
the fix here would be to change open-network-stream, at least, to have a
:coding parameter (and pass it on).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> +      ;; The caller might have bound coding-system-for-* to something
> +      ;; like 'no-conversion, but the below needs to call PROGRAM
> +      ;; expecting human-readable text in both directions (since we
> +      ;; are going to parse the output as text), so let Emacs guess
> +      ;; the encoding of that text by its usual encoding-detection
> +      ;; machinery.
> +      (let ((coding-system-for-read 'undecided)
> +            (coding-system-for-write 'undecided))

I think this probably is the right thing here, but I'm not 100% sure --
I seem to remember there being some issue of people putting non-ASCII
stuff in the name parts of the gpg data and then Emacs having a problem
of matching that up to the data we're looking for...

I may be misremembering, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: juanjose.garcia.ripoll@csic.es,  40248@debbugs.gnu.org
> Date: Sun, 29 Mar 2020 09:45:55 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But then only the encoding of the network process started by
> > nnimap-open-connection-1 should be made no-conversion.
> 
> I'm not sure I quite follow; if I misunderstand, I apologise:
> 
> Do you mean as a :coding parameter to make-network-process?
> nnimap-open-connection-1 doesn't call that function directly, and the
> functions it does call doesn't take any :coding parameters.

We could extend open-network-stream to accept :coding and pass that to
make-network-process.  Or we could call set-process-coding-system
right after open-network-stream returns.

> > By contrast, binding coding-system-for-* around the call to
> > open-network-stream has much wider consequences, as it affects _any_
> > code run as part of open-network-stream, in particularly
> > epg-find-configuration, which has nothing to do with the network
> > stream.
> 
> Yes, that's unfortunate -- the "bind variables as a way of passing in
> parameters" thing that Emacs does a lot is a really bad one.  I think
> the fix here would be to change open-network-stream, at least, to have a
> :coding parameter (and pass it on).

Yes, let's do that on master.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Juan José García-Ripoll
>  <juanjose.garcia.ripoll@csic.es>,
>   40248@debbugs.gnu.org
> Date: Sun, 29 Mar 2020 09:48:59 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > +      ;; The caller might have bound coding-system-for-* to something
> > +      ;; like 'no-conversion, but the below needs to call PROGRAM
> > +      ;; expecting human-readable text in both directions (since we
> > +      ;; are going to parse the output as text), so let Emacs guess
> > +      ;; the encoding of that text by its usual encoding-detection
> > +      ;; machinery.
> > +      (let ((coding-system-for-read 'undecided)
> > +            (coding-system-for-write 'undecided))
> 
> I think this probably is the right thing here, but I'm not 100% sure --
> I seem to remember there being some issue of people putting non-ASCII
> stuff in the name parts of the gpg data and then Emacs having a problem
> of matching that up to the data we're looking for...

If that non-ASCII data is compatible with the default encoding, or if
Emacs will detect the encoding correctly given 'undecided', that
shouldn't be any problem.  And this function is only about getting the
gpg configuration, so what kind of non-ASCII data is expected there?
And how would using 'no-conversion' here help?

If you can find those cases where you saw non-ASCII data in this case,
by all means describe them or point to relevant discussions.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Sun, 29 Mar 2020 16:52:18 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Lars Ingebrigtsen <larsi@gnus.org>
    >> Cc: Juan José García-Ripoll
    >> <juanjose.garcia.ripoll@csic.es>,
    >> 40248@debbugs.gnu.org
    >> Date: Sun, 29 Mar 2020 09:48:59 +0200
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > +      ;; The caller might have bound coding-system-for-* to something
    >> > +      ;; like 'no-conversion, but the below needs to call PROGRAM
    >> > +      ;; expecting human-readable text in both directions (since we
    >> > +      ;; are going to parse the output as text), so let Emacs guess
    >> > +      ;; the encoding of that text by its usual encoding-detection
    >> > +      ;; machinery.
    >> > +      (let ((coding-system-for-read 'undecided)
    >> > +            (coding-system-for-write 'undecided))
    >> 
    >> I think this probably is the right thing here, but I'm not 100% sure --
    >> I seem to remember there being some issue of people putting non-ASCII
    >> stuff in the name parts of the gpg data and then Emacs having a problem
    >> of matching that up to the data we're looking for...

    Eli> If that non-ASCII data is compatible with the default encoding, or if
    Eli> Emacs will detect the encoding correctly given 'undecided', that
    Eli> shouldn't be any problem.  And this function is only about getting the
    Eli> gpg configuration, so what kind of non-ASCII data is expected there?
    Eli> And how would using 'no-conversion' here help?

    Eli> If you can find those cases where you saw non-ASCII data in this case,
    Eli> by all means describe them or point to relevant discussions.

My (admittedly fallible) memory is that gpg always uses UTF-8 for
non-ASCII data (except for some old versions that %-escape it instead).

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  juanjose.garcia.ripoll@csic.es,
>   40248@debbugs.gnu.org
> Date: Mon, 30 Mar 2020 11:37:25 +0200
> 
>     Eli> If that non-ASCII data is compatible with the default encoding, or if
>     Eli> Emacs will detect the encoding correctly given 'undecided', that
>     Eli> shouldn't be any problem.  And this function is only about getting the
>     Eli> gpg configuration, so what kind of non-ASCII data is expected there?
>     Eli> And how would using 'no-conversion' here help?
> 
>     Eli> If you can find those cases where you saw non-ASCII data in this case,
>     Eli> by all means describe them or point to relevant discussions.
> 
> My (admittedly fallible) memory is that gpg always uses UTF-8 for
> non-ASCII data (except for some old versions that %-escape it instead).

Is that true even on MS-Windows?  can someone please verify that?  If
gpg uses UTF-8 on all platforms, the 'undecided' isn't TRT, as in some
cases Emacs could mistakenly decide the encoding is the current system
codepage.  We should use 'utf-8' instead if UTF-8 is guaranteed.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Mon, 30 Mar 2020 16:10:15 +0300, Eli Zaretskii <eliz@gnu.org> said:
    Eli> If you can find those cases where you saw non-ASCII data in this case,
    Eli> by all means describe them or point to relevant discussions.
    >> 
    >> My (admittedly fallible) memory is that gpg always uses UTF-8 for
    >> non-ASCII data (except for some old versions that %-escape it instead).

    Eli> Is that true even on MS-Windows?  can someone please verify that?  If
    Eli> gpg uses UTF-8 on all platforms, the 'undecided' isn't TRT, as in some
    Eli> cases Emacs could mistakenly decide the encoding is the current system
    Eli> codepage.  We should use 'utf-8' instead if UTF-8 is guaranteed.

Having now re-checked it, I was wrong. gpg uses UTF-8 consistenly
_internally_, but converts to/from whatever it thinks the native
codepage is (on MS-Windows and unixy platforms).

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: larsi@gnus.org,  juanjose.garcia.ripoll@csic.es,  40248@debbugs.gnu.org
> Date: Mon, 30 Mar 2020 16:10:34 +0200
> 
>     >> My (admittedly fallible) memory is that gpg always uses UTF-8 for
>     >> non-ASCII data (except for some old versions that %-escape it instead).
> 
>     Eli> Is that true even on MS-Windows?  can someone please verify that?  If
>     Eli> gpg uses UTF-8 on all platforms, the 'undecided' isn't TRT, as in some
>     Eli> cases Emacs could mistakenly decide the encoding is the current system
>     Eli> codepage.  We should use 'utf-8' instead if UTF-8 is guaranteed.
> 
> Having now re-checked it, I was wrong. gpg uses UTF-8 consistenly
> _internally_, but converts to/from whatever it thinks the native
> codepage is (on MS-Windows and unixy platforms).

Thanks.  In that case, 'undecided' is exactly right.

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

(dropping the bug, redirecting to emacs-devel)

>>>>> On Sun, 29 Mar 2020 16:49:03 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >> Yes, that's unfortunate -- the "bind variables as a way of passing in
    >> parameters" thing that Emacs does a lot is a really bad one.  I think
    >> the fix here would be to change open-network-stream, at least, to have a
    >> :coding parameter (and pass it on).

    Eli> Yes, let's do that on master.

Having resisted the strong urge to rip out the gnutls-cli support in
open-network-stream, this turned out to be surprisingly invasive. It
works for nnimap with Gnus for me when passing :coding 'binary instead
of binding coding-system-for-{read-write}, but more testing is
definitely required.

diff --git c/doc/lispref/processes.texi i/doc/lispref/processes.texi
index 4a4f51474c..7d9ca46d94 100644
--- c/doc/lispref/processes.texi
+++ i/doc/lispref/processes.texi
@@ -2462,6 +2462,12 @@ Network
 @item :nowait @var{boolean}
 If non-@code{nil}, try to make an asynchronous connection.
 
+@item :coding @var{coding}
+Use this to set the coding systems used by the network process, in
+preference to binding @code{coding-system-for-read} or
+@code{coding-system-for-write}.  @xref{Network Processes} for
+details.
+
 @item :type @var{type}
 The type of connection.  Options are:
 
diff --git c/etc/NEWS i/etc/NEWS
index 8bbe2aee0b..14bdae3ea7 100644
--- c/etc/NEWS
+++ i/etc/NEWS
@@ -274,6 +274,11 @@ optional argument specifying whether to follow symbolic links.
 ** 'parse-time-string' can now parse ISO 8601 format strings,
 such as "2020-01-15T16:12:21-08:00".
 
+** 'open-network-stream' now accepts a :coding argument.
+This allows specifying the coding systems used by a network process
+for encoding and decoding without having to bind
+coding-system-for-{read,write} or call 'set-process-coding-system'.
+
 \f
 * Changes in Emacs 28.1 on Non-Free Operating Systems
 
diff --git c/lisp/net/gnutls.el i/lisp/net/gnutls.el
index 459156e6d2..171a829f5b 100644
--- c/lisp/net/gnutls.el
+++ i/lisp/net/gnutls.el
@@ -169,8 +169,9 @@ open-gnutls-stream
 Fourth arg SERVICE is the name of the service desired, or an integer
 specifying a port number to connect to.
 Fifth arg PARAMETERS is an optional list of keyword/value pairs.
-Only :client-certificate and :nowait keywords are recognized, and
-have the same meaning as for `open-network-stream'.
+Only :client-certificate, :nowait, and :coding keywords are
+recognized, and have the same meaning as for
+`open-network-stream'.
 For historical reasons PARAMETERS can also be a symbol, which is
 interpreted the same as passing a list containing :nowait and the
 value of that symbol.
@@ -199,16 +200,19 @@ open-gnutls-stream
          (cert (network-stream-certificate host service parameters))
          (keylist (and cert (list cert)))
          (nowait (plist-get parameters :nowait))
-         (process (open-network-stream
-                   name buffer host service
-                   :nowait nowait
-                   :tls-parameters
-                   (and nowait
-                        (cons 'gnutls-x509pki
-                              (gnutls-boot-parameters
-                               :type 'gnutls-x509pki
-                               :keylist keylist
-                               :hostname (puny-encode-domain host)))))))
+         (coding-p (plist-member parameters :coding))
+         (coding-val (plist-get parameters :coding))
+         (args (append (list name buffer host service
+                             :nowait nowait
+                             :tls-parameters
+                             (and nowait
+                                  (cons 'gnutls-x509pki
+                                        (gnutls-boot-parameters
+                                         :type 'gnutls-x509pki
+                                         :keylist keylist
+                                         :hostname (puny-encode-domain host)))))
+                       (when coding-p (list :coding coding-val))))
+         (process (apply #'open-network-stream args)))
     (if nowait
         process
       (gnutls-negotiate :process process
diff --git c/lisp/net/network-stream.el i/lisp/net/network-stream.el
index e99d7a372c..d7d0aedada 100644
--- c/lisp/net/network-stream.el
+++ i/lisp/net/network-stream.el
@@ -113,6 +113,10 @@ open-network-stream
   `ssl'      -- Equivalent to `tls'.
   `shell'    -- A shell connection.
 
+:coding is a symbol or a cons used to specify the coding systems
+used to decode and encode the data which the process reads and
+writes.  See `make-network-process' for details.
+
 :return-list specifies this function's return value.
   If omitted or nil, return a process object.  A non-nil means to
   return (PROC . PROPS), where PROC is a process object and PROPS
@@ -178,18 +182,21 @@ open-network-stream
   (unless (featurep 'make-network-process)
     (error "Emacs was compiled without networking support"))
   (let ((type (plist-get parameters :type))
-	(return-list (plist-get parameters :return-list)))
+	(return-list (plist-get parameters :return-list))
+        (coding-p (plist-member parameters :coding)))
     (if (and (not return-list)
 	     (or (eq type 'plain)
 		 (and (memq type '(nil network))
 		      (not (and (plist-get parameters :success)
 				(plist-get parameters :capability-command))))))
 	;; The simplest case: wrapper around `make-network-process'.
-	(make-network-process :name name :buffer buffer
-			      :host (puny-encode-domain host) :service service
-			      :nowait (plist-get parameters :nowait)
-                              :tls-parameters
-                              (plist-get parameters :tls-parameters))
+        (let ((args (append (list :name name :buffer buffer
+                                  :host (puny-encode-domain host) :service service
+                                  :nowait (plist-get parameters :nowait)
+                                  :tls-parameters
+                                  (plist-get parameters :tls-parameters))
+                            (when coding-p (list :coding (plist-get parameters :coding))))))
+	  (apply #'make-network-process args))
       (let ((work-buffer (or buffer
 			     (generate-new-buffer " *stream buffer*")))
 	    (fun (cond ((and (eq type 'plain)
@@ -245,11 +252,15 @@ 'open-protocol-stream
   "26.1")
 
 (defun network-stream-open-plain (name buffer host service parameters)
-  (let ((start (with-current-buffer buffer (point)))
-	(stream (make-network-process :name name :buffer buffer
-				      :host (puny-encode-domain host)
-                                      :service service
-				      :nowait (plist-get parameters :nowait))))
+  (let* ((start (with-current-buffer buffer (point)))
+         (coding-p (plist-member parameters :coding))
+         (coding-val (plist-get parameters :coding))
+         (args (append (list :name name :buffer buffer
+                             :host (puny-encode-domain host)
+                             :service service
+                             :nowait (plist-get parameters :nowait))
+                       (when coding-p (list :coding coding-val))))
+	 (stream (apply #'make-network-process args)))
     (when (plist-get parameters :warn-unless-encrypted)
       (setq stream (nsm-verify-connection stream host service nil t)))
     (list stream
@@ -267,10 +278,14 @@ network-stream-open-starttls
 	 (eoc                (plist-get parameters :end-of-command))
 	 (eo-capa            (or (plist-get parameters :end-of-capability)
 				 eoc))
+         (coding-p (plist-member parameters :coding))
+         (coding-val (plist-get parameters :coding))
+         (args (append (list :name name :buffer buffer
+                             :host (puny-encode-domain host)
+                             :service service)
+                       (when coding-p (list :coding coding-val))))
 	 ;; Return (STREAM GREETING CAPABILITIES RESULTING-TYPE)
-	 (stream (make-network-process :name name :buffer buffer
-				       :host (puny-encode-domain host)
-                                       :service service))
+         (stream (apply #'make-network-process args))
 	 (greeting (and (not (plist-get parameters :nogreeting))
 			(network-stream-get-response stream start eoc)))
 	 (capabilities (network-stream-command stream capability-command
@@ -348,9 +363,7 @@ network-stream-open-starttls
 	  ;; isn't demanded, reopen an unencrypted connection.
 	  (unless require-tls
 	    (setq stream
-		  (make-network-process :name name :buffer buffer
-					:host (puny-encode-domain host)
-                                        :service service))
+		  (apply #'make-network-process args))
 	    (network-stream-get-response stream start eoc)))
         (unless (process-live-p stream)
           (error "Unable to negotiate a TLS connection with %s/%s"
@@ -453,6 +466,8 @@ network-stream-open-shell
   (let* ((capability-command (plist-get parameters :capability-command))
 	 (eoc 		     (plist-get parameters :end-of-command))
 	 (start (with-current-buffer buffer (point)))
+         (coding-p (plist-member parameters :coding))
+         (coding-val (plist-get parameters :coding))
 	 (stream (let ((process-connection-type nil))
 		   (start-process name buffer shell-file-name
 				  shell-command-switch
@@ -461,6 +476,13 @@ network-stream-open-shell
 				   (format-spec-make
 				    ?s host
 				    ?p service))))))
+    (when coding-p (if (consp coding-val)
+                       (set-process-coding-system stream
+                                                  (car coding-val)
+                                                  (cdr coding-val))
+                     (set-process-coding-system stream
+                                                coding-val
+                                                coding-val)))
     (list stream
 	  (network-stream-get-response stream start eoc)
 	  (network-stream-command stream capability-command

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Andreas Schwab]

On Mär 31 2020, Robert Pluim wrote:

> @@ -245,11 +252,15 @@ 'open-protocol-stream
>    "26.1")
>  
>  (defun network-stream-open-plain (name buffer host service parameters)
> -  (let ((start (with-current-buffer buffer (point)))
> -	(stream (make-network-process :name name :buffer buffer
> -				      :host (puny-encode-domain host)
> -                                      :service service
> -				      :nowait (plist-get parameters :nowait))))
> +  (let* ((start (with-current-buffer buffer (point)))
> +         (coding-p (plist-member parameters :coding))
> +         (coding-val (plist-get parameters :coding))
> +         (args (append (list :name name :buffer buffer
> +                             :host (puny-encode-domain host)
> +                             :service service
> +                             :nowait (plist-get parameters :nowait))
> +                       (when coding-p (list :coding coding-val))))
> +	 (stream (apply #'make-network-process args)))

You don't need to use apply.  Since :coding nil is the same as the
default, you can just pass `:coding (plist-get parameters :coding)'
unconditionally.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Tue, 31 Mar 2020 11:53:36 +0200, Andreas Schwab <schwab@linux-m68k.org> said:

    Andreas> On Mär 31 2020, Robert Pluim wrote:
    >> @@ -245,11 +252,15 @@ 'open-protocol-stream
    >> "26.1")
    >> 
    >> (defun network-stream-open-plain (name buffer host service parameters)
    >> -  (let ((start (with-current-buffer buffer (point)))
    >> -	(stream (make-network-process :name name :buffer buffer
    >> -				      :host (puny-encode-domain host)
    >> -                                      :service service
    >> -				      :nowait (plist-get parameters :nowait))))
    >> +  (let* ((start (with-current-buffer buffer (point)))
    >> +         (coding-p (plist-member parameters :coding))
    >> +         (coding-val (plist-get parameters :coding))
    >> +         (args (append (list :name name :buffer buffer
    >> +                             :host (puny-encode-domain host)
    >> +                             :service service
    >> +                             :nowait (plist-get parameters :nowait))
    >> +                       (when coding-p (list :coding coding-val))))
    >> +	 (stream (apply #'make-network-process args)))

    Andreas> You don't need to use apply.  Since :coding nil is the same as the
    Andreas> default, you can just pass `:coding (plist-get parameters :coding)'
    Andreas> unconditionally.

My reading of set_network_socket_coding_system is that having a plist
in make-network-process with :coding nil overrides a non-nil
coding-system-for-{read-write}, which I donʼt think we want.

Robert

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Andreas Schwab]

On Mär 31 2020, Robert Pluim wrote:

> My reading of set_network_socket_coding_system is that having a plist
> in make-network-process with :coding nil overrides a non-nil
> coding-system-for-{read-write}, which I donʼt think we want.

You are right, make-network-process and make-serial-process handle that
differently than make-process and make-pipe-process.  I think this is a
bug, they should all handle :coding nil the same (same as absence).

But in any case, your patch can be simplyfied:

--- a/lisp/net/network-stream.el
+++ b/lisp/net/network-stream.el
@@ -246,10 +246,12 @@ gnutls-boot (as returned by `gnutls-boot-parameters')."
 
 (defun network-stream-open-plain (name buffer host service parameters)
   (let ((start (with-current-buffer buffer (point)))
-	(stream (make-network-process :name name :buffer buffer
-				      :host (puny-encode-domain host)
-                                      :service service
-				      :nowait (plist-get parameters :nowait))))
+	(stream (apply #'make-network-process :name name :buffer buffer
+		       :host (puny-encode-domain host)
+                       :service service
+		       :nowait (plist-get parameters :nowait)
+                       (if (plist-member parameters :coding)
+                           (list :coding (plist-get parameters :coding))))))
     (when (plist-get parameters :warn-unless-encrypted)
       (setq stream (nsm-verify-connection stream host service nil t)))
     (list stream

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Tue, 31 Mar 2020 12:31:04 +0200, Andreas Schwab <schwab@linux-m68k.org> said:

    Andreas> On Mär 31 2020, Robert Pluim wrote:
    >> My reading of set_network_socket_coding_system is that having a plist
    >> in make-network-process with :coding nil overrides a non-nil
    >> coding-system-for-{read-write}, which I donʼt think we want.

    Andreas> You are right, make-network-process and make-serial-process handle that
    Andreas> differently than make-process and make-pipe-process.  I think this is a
    Andreas> bug, they should all handle :coding nil the same (same as absence).

I think I agree with that. Basically this, initially (and the same for
make-serial-process)

diff --git a/src/process.c b/src/process.c
index e18a5aa538..b00d8e1cfe 100644
--- a/src/process.c
+++ b/src/process.c
@@ -3261,9 +3262,7 @@ set_network_socket_coding_system (Lisp_Object proc, Lisp_Object host,
   Lisp_Object coding_systems = Qt;
   Lisp_Object val;
 
-  tem = Fplist_member (contact, QCcoding);
-  if (!NILP (tem) && (!CONSP (tem) || !CONSP (XCDR (tem))))
-    tem = Qnil;  /* No error message (too late!).  */
+  tem = Fplist_get (contact, QCcoding);
 
   /* Setup coding systems for communicating with the network stream.  */
   /* Qt denotes we have not yet called Ffind_operation_coding_system.  */

    Andreas> But in any case, your patch can be simplyfied:

    Andreas> --- a/lisp/net/network-stream.el
    Andreas> +++ b/lisp/net/network-stream.el
    Andreas> @@ -246,10 +246,12 @@ gnutls-boot (as returned by `gnutls-boot-parameters')."
 
    Andreas>  (defun network-stream-open-plain (name buffer host service parameters)
    Andreas>    (let ((start (with-current-buffer buffer (point)))
    Andreas> -	(stream (make-network-process :name name :buffer buffer
    Andreas> -				      :host (puny-encode-domain host)
    Andreas> -                                      :service service
    Andreas> -				      :nowait (plist-get parameters :nowait))))
    Andreas> +	(stream (apply #'make-network-process :name name :buffer buffer
    Andreas> +		       :host (puny-encode-domain host)
    Andreas> +                       :service service
    Andreas> +		       :nowait (plist-get parameters :nowait)
    Andreas> +                       (if (plist-member parameters :coding)
    Andreas> +                           (list :coding (plist-get parameters :coding))))))
    Andreas>      (when (plist-get parameters :warn-unless-encrypted)
    Andreas>        (setq stream (nsm-verify-connection stream host service nil t)))
    Andreas>      (list stream

Sure.

Robert

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Tue, 31 Mar 2020 13:09:14 +0200, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Tue, 31 Mar 2020 12:31:04 +0200, Andreas Schwab <schwab@linux-m68k.org> said:
    Andreas> On Mär 31 2020, Robert Pluim wrote:
    >>> My reading of set_network_socket_coding_system is that having a plist
    >>> in make-network-process with :coding nil overrides a non-nil
    >>> coding-system-for-{read-write}, which I donʼt think we want.

    Andreas> You are right, make-network-process and make-serial-process handle that
    Andreas> differently than make-process and make-pipe-process.  I think this is a
    Andreas> bug, they should all handle :coding nil the same (same as absence).

    Robert> I think I agree with that. Basically this, initially (and the same for
    Robert> make-serial-process)

Now that Iʼve actually tested it, the change looks like this, making
make-network-process and make-serial process behave the same as
make-process and make-pipe-process. Iʼve looked at all uses of those
two functions in Emacs' sources, and none of them depend on the change
in semantics, in fact only a couple of them actually pass a :coding
keyword.

Having said all that, this does go back a looooong way (2002 at
least), so maybe we want to let sleeping dogs lie.

diff --git a/src/process.c b/src/process.c
index e18a5aa538..2db45b18b2 100644
--- a/src/process.c
+++ b/src/process.c
@@ -3205,14 +3206,12 @@ DEFUN ("make-serial-process", Fmake_serial_process, Smake_serial_process,
 		       BUF_ZV_BYTE (XBUFFER (buffer)));
     }
 
-  tem = Fplist_member (contact, QCcoding);
-  if (!NILP (tem) && (!CONSP (tem) || !CONSP (XCDR (tem))))
-    tem = Qnil;
+  tem = Fplist_get (contact, QCcoding);
 
   val = Qnil;
   if (!NILP (tem))
     {
-      val = XCAR (XCDR (tem));
+      val = tem;
       if (CONSP (val))
 	val = XCAR (val);
     }
@@ -3226,7 +3225,7 @@ DEFUN ("make-serial-process", Fmake_serial_process, Smake_serial_process,
   val = Qnil;
   if (!NILP (tem))
     {
-      val = XCAR (XCDR (tem));
+      val = tem;
       if (CONSP (val))
 	val = XCDR (val);
     }
@@ -3261,16 +3260,14 @@ set_network_socket_coding_system (Lisp_Object proc, Lisp_Object host,
   Lisp_Object coding_systems = Qt;
   Lisp_Object val;
 
-  tem = Fplist_member (contact, QCcoding);
-  if (!NILP (tem) && (!CONSP (tem) || !CONSP (XCDR (tem))))
-    tem = Qnil;  /* No error message (too late!).  */
+  tem = Fplist_get (contact, QCcoding);
 
   /* Setup coding systems for communicating with the network stream.  */
   /* Qt denotes we have not yet called Ffind_operation_coding_system.  */
 
   if (!NILP (tem))
     {
-      val = XCAR (XCDR (tem));
+      val = tem;
       if (CONSP (val))
 	val = XCAR (val);
     }
@@ -3304,7 +3301,7 @@ set_network_socket_coding_system (Lisp_Object proc, Lisp_Object host,
 
   if (!NILP (tem))
     {
-      val = XCAR (XCDR (tem));
+      val = tem;
       if (CONSP (val))
 	val = XCDR (val);
     }

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Lars Ingebrigtsen <larsi@gnus.org>,
>   emacs-devel@gnu.org
> Date: Tue, 31 Mar 2020 18:13:10 +0200
> 
> Now that Iʼve actually tested it, the change looks like this, making
> make-network-process and make-serial process behave the same as
> make-process and make-pipe-process. Iʼve looked at all uses of those
> two functions in Emacs' sources, and none of them depend on the change
> in semantics, in fact only a couple of them actually pass a :coding
> keyword.

Nevertheless, we should call out this change in NEWS.

> Having said all that, this does go back a looooong way (2002 at
> least), so maybe we want to let sleeping dogs lie.

I don't think this is a risky change.  Programs that assume what we
currently do rely on undefined behavior, IMO.

Thanks.

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Tue, 31 Mar 2020 20:59:01 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Eli Zaretskii <eliz@gnu.org>,  Lars Ingebrigtsen <larsi@gnus.org>,
    >> emacs-devel@gnu.org
    >> Date: Tue, 31 Mar 2020 18:13:10 +0200
    >> 
    >> Now that Iʼve actually tested it, the change looks like this, making
    >> make-network-process and make-serial process behave the same as
    >> make-process and make-pipe-process. Iʼve looked at all uses of those
    >> two functions in Emacs' sources, and none of them depend on the change
    >> in semantics, in fact only a couple of them actually pass a :coding
    >> keyword.

    Eli> Nevertheless, we should call out this change in NEWS.

That goes without saying. Probably put a note somewhere in the elisp
manual as well.

    >> Having said all that, this does go back a looooong way (2002 at
    >> least), so maybe we want to let sleeping dogs lie.

    Eli> I don't think this is a risky change.  Programs that assume what we
    Eli> currently do rely on undefined behavior, IMO.

Well, itʼs not undefined in the 'C' sense of undefined at least :-)

Iʼll finish it up, and then convert at least Gnus over to use it.

Robert

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Now that Iʼve actually tested it, the change looks like this, making
> make-network-process and make-serial process behave the same as
> make-process and make-pipe-process. Iʼve looked at all uses of those
> two functions in Emacs' sources, and none of them depend on the change
> in semantics, in fact only a couple of them actually pass a :coding
> keyword.
>
> Having said all that, this does go back a looooong way (2002 at
> least), so maybe we want to let sleeping dogs lie.

Yeah, it does sound slightly scary, but I think the current nil
behaviour for :coding is probably not something any code would rely
on...  I'm not quite sure what `nil' semantics for :coding here would
signify?

On the other hand, I generally think that functions should respect their
parameters, so if you say :coding nil, it might then be somewhat
surprising that `coding-system-for-{read,write}' are used instead?

So I'm not quite sure that the current make-network-process behaviour
here is a bug.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks.  In that case, 'undecided' is exactly right.

Yeah, sounds like it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Thu, 02 Apr 2020 13:10:37 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >> Now that Iʼve actually tested it, the change looks like this, making
    >> make-network-process and make-serial process behave the same as
    >> make-process and make-pipe-process. Iʼve looked at all uses of those
    >> two functions in Emacs' sources, and none of them depend on the change
    >> in semantics, in fact only a couple of them actually pass a :coding
    >> keyword.
    >> 
    >> Having said all that, this does go back a looooong way (2002 at
    >> least), so maybe we want to let sleeping dogs lie.

    Lars> Yeah, it does sound slightly scary, but I think the current nil
    Lars> behaviour for :coding is probably not something any code would rely
    Lars> on...  I'm not quite sure what `nil' semantics for :coding here would
    Lars> signify?

No code in Emacs that I could find relies on it. The new semantics
would be 'use coding-system-for-read', which I think is defensible.

    Lars> On the other hand, I generally think that functions should respect their
    Lars> parameters, so if you say :coding nil, it might then be somewhat
    Lars> surprising that `coding-system-for-{read,write}' are used instead?

    Lars> So I'm not quite sure that the current make-network-process behaviour
    Lars> here is a bug.

Itʼs different from make-process and make-pipe-process, itʼs
undocumented, and it was surprising to both me and Andreas. If it
turns out that in some obscure case somebody really needs the old
behaviour, then binding coding-system-for-{read,write} will still be
available, as will set-process-coding-sytem (or even passing :coding
'(nil nil), but thatʼs just evil :-) )

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Robert Pluim <rpluim@gmail.com>,  juanjose.garcia.ripoll@csic.es,
>   40248@debbugs.gnu.org
> Date: Thu, 02 Apr 2020 13:11:27 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks.  In that case, 'undecided' is exactly right.
> 
> Yeah, sounds like it.

Thanks, pushed to the emacs-27 branch.

I'm not closing the bug because we still didn't make the more invasive
change on master, AFAIR.

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Fri, 03 Apr 2020 14:31:43 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Lars Ingebrigtsen <larsi@gnus.org>
    >> Cc: Robert Pluim <rpluim@gmail.com>,  juanjose.garcia.ripoll@csic.es,
    >> 40248@debbugs.gnu.org
    >> Date: Thu, 02 Apr 2020 13:11:27 +0200
    >> 
    >> Eli Zaretskii <eliz@gnu.org> writes:
    >> 
    >> > Thanks.  In that case, 'undecided' is exactly right.
    >> 
    >> Yeah, sounds like it.

    Eli> Thanks, pushed to the emacs-27 branch.

    Eli> I'm not closing the bug because we still didn't make the more invasive
    Eli> change on master, AFAIR.

I was waiting to see if Lars or anybody had any more comments. I can
post the complete patch over in emacs-devel.

Robert

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Itʼs different from make-process and make-pipe-process, itʼs
> undocumented, and it was surprising to both me and Andreas.

Yeah, that's a good reason for changing it.

> If it turns out that in some obscure case somebody really needs the
> old behaviour, then binding coding-system-for-{read,write} will still
> be available, as will set-process-coding-sytem (or even passing
> :coding '(nil nil), but thatʼs just evil :-) )

Oh, yeah.  :-)  Then I don't have any quibbles about your patch.

I don't think that's evil at all, though -- it should be mentioned in
NEWS that if somebody (for some odd, unlikely reason) wants the old
:coding nil behaviour, then they should say :coding '(nil nil) now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Fri, 03 Apr 2020 13:52:06 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >> Itʼs different from make-process and make-pipe-process, itʼs
    >> undocumented, and it was surprising to both me and Andreas.

    Lars> Yeah, that's a good reason for changing it.

    >> If it turns out that in some obscure case somebody really needs the
    >> old behaviour, then binding coding-system-for-{read,write} will still
    >> be available, as will set-process-coding-sytem (or even passing
    >> :coding '(nil nil), but thatʼs just evil :-) )

    Lars> Oh, yeah.  :-)  Then I don't have any quibbles about your patch.

    Lars> I don't think that's evil at all, though -- it should be mentioned in
    Lars> NEWS that if somebody (for some odd, unlikely reason) wants the old
    Lars> :coding nil behaviour, then they should say :coding '(nil nil) now.

Ok, pushed to master as d08e81ce5a , with that in NEWS.

Iʼll hold off on the :coding change to open-network-stream et al for
the moment, just in case this does break something.

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>     Eli> I'm not closing the bug because we still didn't make the more invasive
>     Eli> change on master, AFAIR.
>
> I was waiting to see if Lars or anybody had any more comments. I can
> post the complete patch over in emacs-devel.

I see that Robert added:

commit 6382e1330814ca4df20eeccd8b4ef9ca17b997af
Author: Robert Pluim <rpluim@gmail.com>
Date:   Thu Apr 2 18:41:33 2020 +0200

    Add :coding support to open-network-stream and open-gnutls-stream

I thought that was the "more invasive" change, but it's committed the
day before the final exchange in this bug report.

So I'm not quite sure after skimming this thread now...  has this all
been resolved, or were there any further work to be done?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  juanjose.garcia.ripoll@csic.es,
>   40248@debbugs.gnu.org
> Date: Sun, 19 Jul 2020 04:45:59 +0200
> 
> Robert Pluim <rpluim@gmail.com> writes:
> 
> >     Eli> I'm not closing the bug because we still didn't make the more invasive
> >     Eli> change on master, AFAIR.
> >
> > I was waiting to see if Lars or anybody had any more comments. I can
> > post the complete patch over in emacs-devel.
> 
> I see that Robert added:
> 
> commit 6382e1330814ca4df20eeccd8b4ef9ca17b997af
> Author: Robert Pluim <rpluim@gmail.com>
> Date:   Thu Apr 2 18:41:33 2020 +0200
> 
>     Add :coding support to open-network-stream and open-gnutls-stream
> 
> I thought that was the "more invasive" change, but it's committed the
> day before the final exchange in this bug report.

No, it was committed on

  CommitDate: Fri Apr 3 14:45:49 2020 +0200

which is 2 minutes _after_ that exchange.

The date you show above is AuthorDate, i.e. the date when Robert
committed this locally on his machine.  (This confusion is precisely
the reason why I customized "git log" to show both of the dates.)

> So I'm not quite sure after skimming this thread now...  has this all
> been resolved, or were there any further work to be done?

The "invasive" change is the above one; the less invasive one, which
was at the time done on emacs-27, was this one:

  commit fa823653ffb0e3e893d30daa5abf68e909934e2e
  Author:     Eli Zaretskii <eliz@gnu.org>
  AuthorDate: Fri Apr 3 14:29:49 2020 +0300
  Commit:     Eli Zaretskii <eliz@gnu.org>
  CommitDate: Fri Apr 3 14:29:49 2020 +0300

      Fix invocations of gpg from Gnus

      * lisp/epg-config.el (epg-config--make-gpg-configuration): Bind
      coding-system-for-read/write to 'undecided', to countermand
      possible values of 'no-conversion' or somesuch by the callers.
      (Bug#40248)

So I think this bug can be closed.  Robert?

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> No, it was committed on
>
>   CommitDate: Fri Apr 3 14:45:49 2020 +0200
>
> which is 2 minutes _after_ that exchange.

Heh.

> The date you show above is AuthorDate, i.e. the date when Robert
> committed this locally on his machine.  (This confusion is precisely
> the reason why I customized "git log" to show both of the dates.)

That sounds like a very useful thing to have.  How do you make vc-mode
(etc) show both dates?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rpluim@gmail.com,  juanjose.garcia.ripoll@csic.es,  40248@debbugs.gnu.org
> Date: Sun, 19 Jul 2020 16:29:44 +0200
> 
> > The date you show above is AuthorDate, i.e. the date when Robert
> > committed this locally on his machine.  (This confusion is precisely
> > the reason why I customized "git log" to show both of the dates.)
> 
> That sounds like a very useful thing to have.  How do you make vc-mode
> (etc) show both dates?

You don't need to tell VC anything, just say in your ~/.gitconfig

  [format]
	  pretty = fuller

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> You don't need to tell VC anything, just say in your ~/.gitconfig
>
>   [format]
> 	  pretty = fuller

Thanks!

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Robert Pluim]

>>>>> On Sun, 19 Jul 2020 17:23:40 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> 
    >> I thought that was the "more invasive" change, but it's committed the
    >> day before the final exchange in this bug report.

That was half of the more invasive change. The second half is getting
gnus and other users of 'open-network-stream' to pass :coding to it,
rather than binding 'coding-system-for-{read-write}'. Thatʼs currently
sitting in my workspace waiting for me to finish testing it.

    Eli> So I think this bug can be closed.  Robert?

Yes. This bug is fixed, I just need to finish off the gnus changes.

Robert

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Juri Linkov]

>> That sounds like a very useful thing to have.  How do you make vc-mode
>> (etc) show both dates?
>
> You don't need to tell VC anything, just say in your ~/.gitconfig
>
>   [format]
> 	  pretty = fuller

This setting breaks vc-git-log-view-mode: typing 'D' in the log buffer
with "CommitDate" throws the error:

  fatal: bad revision 'Date'

This is because it searches for the commit number and finds "CommitDate"
as in this example:

  commit 365e01cc9f64ce6ca947ccfd8612d60763280a37
  CommitDate: 2020-01-01 00:59:52 +0000

The problem is that 'vc-git-log-view-mode' sets 'log-view-message-re'
to this regexp:

  "^commit *\\([0-9a-z]+\\)"

where "\\([0-9a-z]+\\)" matches "Date" in "CommitDate".

This patch fixes it, but I'm not sure if this is the correct regexp
for git commits:

diff --git a/lisp/vc/vc-git.el b/lisp/vc/vc-git.el
index 9b8151705f..2220a33188 100644
--- a/lisp/vc/vc-git.el
+++ b/lisp/vc/vc-git.el
@@ -1241,7 +1241,7 @@ vc-git-log-view-mode
   (set (make-local-variable 'log-view-message-re)
        (if (not (memq vc-log-view-type '(long log-search with-diff)))
 	   (cadr vc-git-root-log-format)
-	 "^commit *\\([0-9a-z]+\\)"))
+	 "^commit +\\([0-9a-z]+\\)"))
   ;; Allow expanding short log entries.
   (when (memq vc-log-view-type '(short log-outgoing log-incoming mergebase))
     (setq truncate-lines t)

bug#40248: 27.0.90; Failure open .authinfo.gpg from Gnus

[Dmitry Gutov]

On 23.07.2020 03:20, Juri Linkov wrote:
> This patch fixes it, but I'm not sure if this is the correct regexp
> for git commits:

It looks like an improvement either way. Please install.