Two errors in 27.* with Windows

Two errors in 27.* with Windows

[Juan José García-Ripoll]

The following errors rendered my Gnus installation unusable until I
found some workarounds. I believe they should be corrected before
release. However, I still have no clear solution for the second.

----
First problem: mailcap-user-data is emptied by
(mailcap-parse-mailcaps). Formerly, in 26.3 and earlier, the information
from .mailcap was added to the list. Now the variable is deleted even if
there is no .mailcap file around.

My configuration file relies on mailcap-add-mailcap-entry to add mailcap
entries that are not external programs, such as doc-view. This cannot be
configured neither with mailcap-user-mime-data nor with .mailcap.

Workaround: set mailcap-parsed-p to T
----

Second problem: output from gpg.exe is read in binary mode, and
corrupted because of ^M characters. I cannot find out what defines the
mode of the buffer where the output of CALL-PROCESSS is written to. Help
in debugging this is welcome.

In 27.0.90:
CALL-PROCESS:
Args: ("c:/Program Files (x86)/GnuPG/bin/gpg.exe" nil (t nil) nil "--with-colons" "--list-config")
Coding system ((coding-system-for-read binary) (buffer-file-coding-system utf-8-dos) (inhibit-eol-conversion nil))

In 26.3:
CALL-PROCESS:
Args: ("c:/Program Files (x86)/GnuPG/bin/gpg.exe" nil (t nil) nil "--with-colons" "--list-config")
Coding system ((coding-system-for-read nil) (buffer-file-coding-system iso-latin-1-dos) (inhibit-eol-conversion nil))

An important difference is that in 26.3, epg-find-configuration is
called by the package module, for signature verification. This
guarantees that pgp is ok when Emacs is booted, even if gpg has not yet
been used.

This initial call does not happen in 27.0.90. The first time
epg-find-configuration is used is from Gnus. This causes the coding
system to be improperly set and the output from gpg.exe also to be
improperly parsed.

Workaround: add (epg-find-configuration 'OpenPGP) in .emacs before
invoking Gnus.

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Two errors in 27.* with Windows

[Juan José García-Ripoll]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:
> ----
> First problem: mailcap-user-data is emptied by
> (mailcap-parse-mailcaps). Formerly, in 26.3 and earlier, the information
> from .mailcap was added to the list. Now the variable is deleted even if
> there is no .mailcap file around.
>
> My configuration file relies on mailcap-add-mailcap-entry to add mailcap
> entries that are not external programs, such as doc-view. This cannot be
> configured neither with mailcap-user-mime-data nor with .mailcap.
>
> Workaround: set mailcap-parsed-p to T

Actually I correct myself: there is no workaround for this, because in
some places the mime parser calls mailcap-user-mime-data with argument
/force/ equal to /t/, ignoring maicap-parsed-p.

Moreover, this happens in a context where it is hardly justified. I got
an email with a text/html alternative part. The mere display of the
message (not the attachment) forced calling mailcap-parse-mailcaps. I
would understand this for external viewers, but not for inlined
parts. And in no case should "force" be used.

--- Backtrace to illustrate where mailcap-mime-data is cleared---

Debugger entered--setting mailcap-mime-data to nil: 
  debug--implement-debug-watch(mailcap-mime-data nil set nil)
  mailcap-parse-mailcaps(nil t)
  mailcap-mime-info("text/html")
  mm-display-part((#<buffer  *mm*-258304> ("text/html" (charset . "utf-8")) quoted-printable nil nil nil nil nil) t)
  gnus-mime-display-single((#<buffer  *mm*-258304> ("text/html" (charset . "utf-8")) quoted-printable nil nil nil nil nil))
  gnus-mime-display-part((#<buffer  *mm*-258304> ("text/html" (charset . "utf-8")) quoted-printable nil nil nil nil nil))
  mapcar(gnus-mime-display-part ((#<buffer  *mm*-258304> ("text/html" (charset . "utf-8")) quoted-printable nil nil nil nil nil) (#<buffer  *mm*-939186> ("application/msword") base64 nil ("inline") nil nil nil)))
  gnus-mime-display-mixed(((#<buffer  *mm*-258304> ("text/html" (charset . "utf-8")) quoted-printable nil nil nil nil nil) (#<buffer  *mm*-939186> ("application/msword") base64 nil ("inline") nil nil nil)))
  gnus-mime-display-part((#("multipart/mixed" 0 15 (start nil from "t.ramos.delrio@gmail.com" buffer #<buffer  *mm*> boundary "_4839E48A-B119-4B60-99C6-209ACE439711_")) (#<buffer  *mm*-258304> ("text/html" (charset . "utf-8")) quoted-printable nil nil nil nil nil) (#<buffer  *mm*-939186> ("application/msword") base64 nil ("inline") nil nil nil)))
  gnus-display-mime()
  gnus-article-prepare-display()
  gnus-article-prepare(37215 nil)
  gnus-summary-display-article(37215 nil)
  gnus-summary-select-article(nil nil pseudo)
  gnus-summary-scroll-up(1)
  funcall-interactively(gnus-summary-scroll-up 1)
  call-interactively(gnus-summary-scroll-up nil nil)
  command-execute(gnus-summary-scroll-up)

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Two errors in 27.* with Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Thu, 26 Mar 2020 17:21:37 +0100
> 
> The following errors rendered my Gnus installation unusable until I
> found some workarounds. I believe they should be corrected before
> release. However, I still have no clear solution for the second.

Please report each problem separately using "M-x report-emacs-bug".
That will allow to deal with each one in a separate thread.  Also, the
bug tracker is the proper place for discussing bugs and looking for
their solutions.

> Second problem: output from gpg.exe is read in binary mode, and
> corrupted because of ^M characters. I cannot find out what defines the
> mode of the buffer where the output of CALL-PROCESSS is written to. Help
> in debugging this is welcome.
> 
> In 27.0.90:
> CALL-PROCESS:
> Args: ("c:/Program Files (x86)/GnuPG/bin/gpg.exe" nil (t nil) nil "--with-colons" "--list-config")
> Coding system ((coding-system-for-read binary) (buffer-file-coding-system utf-8-dos) (inhibit-eol-conversion nil))
> 
> In 26.3:
> CALL-PROCESS:
> Args: ("c:/Program Files (x86)/GnuPG/bin/gpg.exe" nil (t nil) nil "--with-colons" "--list-config")
> Coding system ((coding-system-for-read nil) (buffer-file-coding-system iso-latin-1-dos) (inhibit-eol-conversion nil))

What does gpg.exe output in this case?  Does it output human-readable
text, or does it output binary byte stream?  If the latter, how come
^M corruption is an issue?

Re: Two errors in 27.* with Windows

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
> Please report each problem separately using "M-x report-emacs-bug".
> That will allow to deal with each one in a separate thread.  Also, the
> bug tracker is the proper place for discussing bugs and looking for
> their solutions.

Sorry, being so close to release I wasn't sure whether that would be the
right place.

> What does gpg.exe output in this case?  Does it output human-readable
> text, or does it output binary byte stream?  If the latter, how come
> ^M corruption is an issue?

The problem is not gpg's output. That is parsed properly if I invoke
epg's routines before Gnus: they are parsed in a text buffer (not
binary) and the version number is identified (x.x.x). The problem is
that, because package.el now does not trigger the identification of gpg,
this happens in a binary buffer. The routines parse the output
correctly, but the protocols, version numbers and everything else get
parsed with an ending ^M -> "x.x.x^M" which breaks the identification of
version number. If I run epg-find-configuration _after_ or _before_
Gnus, the parsing is done in a text buffer and the ^M is not
appended. Everything is fine. I am unable to find where the switch from
'nil to 'binary happens in the new code.

Hope this is clear. I will report the bugs separately later on.

Cheers,

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Two errors in 27.* with Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Thu, 26 Mar 2020 20:01:37 +0100
> 
> > What does gpg.exe output in this case?  Does it output human-readable
> > text, or does it output binary byte stream?  If the latter, how come
> > ^M corruption is an issue?
> 
> The problem is not gpg's output. That is parsed properly if I invoke
> epg's routines before Gnus: they are parsed in a text buffer (not
> binary) and the version number is identified (x.x.x). The problem is
> that, because package.el now does not trigger the identification of gpg,
> this happens in a binary buffer. The routines parse the output
> correctly, but the protocols, version numbers and everything else get
> parsed with an ending ^M -> "x.x.x^M" which breaks the identification of
> version number. If I run epg-find-configuration _after_ or _before_
> Gnus, the parsing is done in a text buffer and the ^M is not
> appended. Everything is fine. I am unable to find where the switch from
> 'nil to 'binary happens in the new code.
> 
> Hope this is clear.

Sorry, no, it isn't.  When you report this problem to the bug tracker,
please make sure the description answers the following questions:

  . what do you mean by "binary buffer" (or "text buffer")?
  . I didn't say that gpg's output is the problem, I asked whether
    it's human-readable text or byte stream in this case.  I
    understand it's text (the gpg version number), in which case I
    don't understand why the code sets coding-system-for-read to
    'binary' -- it sounds like a mistake.  Can you show the place
    where this is done?
  . you are assuming that using nil for coding-system-for-read is
    correct, but I don't think it is, so if you can show where that is
    done, it might be the beginning of the solution.

Thanks.

Re: Two errors in 27.* with Windows

[Juan José García-Ripoll]

Apologies for following up here, but there are questions below that I
don't know how to answer:

Eli Zaretskii <eliz@gnu.org> writes:
> Sorry, no, it isn't.  When you report this problem to the bug tracker,
> please make sure the description answers the following questions:
>
>   . what do you mean by "binary buffer" (or "text buffer")?

I have no way to debug this problem clearly. Obviously the problem is in
call-process producing output in a buffer and that being parsed in two
different ways: outside Gnus, it is parsed in a way that does not ignore
^M characters; inside Gnus it sometimes happens and sometimes does not.

>   . I didn't say that gpg's output is the problem, I asked whether
>     it's human-readable text or byte stream in this case.  I
>     understand it's text (the gpg version number), in which case I
>     don't understand why the code sets coding-system-for-read to
>     'binary' -- it sounds like a mistake.  Can you show the place
>     where this is done?

No. That is what I was asking in my earlier email. I do not know how to
trace where this happens. The most I have managed is to set an "advice"
function that interferes CALL-PROCESS and output properties of the
buffer that is current when it is invoked, but I have no idea how to
find which part of the code selects those values. Man, this code base is huge.

>   . you are assuming that using nil for coding-system-for-read is
>     correct, but I don't think it is, so if you can show where that is
>     done, it might be the beginning of the solution.

No, I am not assuming it is correct: I am assuming it is the way it was
done before. It was so in 26.3 and it is so in 27.0.90 before Gnus is
invoked.

And I am not being lazy: I am totally lost at where this variable is
changed or how. I have spent 4 hours this morning just identifying what
prevented me from opening email, and just managed to get what I showed
before. If I know where the wrong assignment was taking place I would
have provided a patch, not a bug report.

But this is kind of a serious bug: it breaks opening of .authinfo.gpg.

-- 
Juan José García Ripoll

Quantum Information and Foundations Group
Institute of Fundamental Physics IFF-CSIC
Calle Serrano 113b, Madrid 28006 Spain
http://quinfog.hbar.es - http://juanjose.garcia.ripoll

Re: Two errors in 27.* with Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Cc: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>,  emacs-devel@gnu.org
> Date: Thu, 26 Mar 2020 23:09:08 +0100
> 
> >   . what do you mean by "binary buffer" (or "text buffer")?
> 
> I have no way to debug this problem clearly. Obviously the problem is in
> call-process producing output in a buffer and that being parsed in two
> different ways: outside Gnus, it is parsed in a way that does not ignore
> ^M characters; inside Gnus it sometimes happens and sometimes does not.
> 
> >   . I didn't say that gpg's output is the problem, I asked whether
> >     it's human-readable text or byte stream in this case.  I
> >     understand it's text (the gpg version number), in which case I
> >     don't understand why the code sets coding-system-for-read to
> >     'binary' -- it sounds like a mistake.  Can you show the place
> >     where this is done?
> 
> No. That is what I was asking in my earlier email. I do not know how to
> trace where this happens. The most I have managed is to set an "advice"
> function that interferes CALL-PROCESS and output properties of the
> buffer that is current when it is invoked, but I have no idea how to
> find which part of the code selects those values. Man, this code base is huge.

Lars (or anyone else of the Gnus developers), can you help?  We are
looking for the place that calls (perhaps indirectly) call-process to
run gpg in this scenario, and in particularly where does that code set
coding-system-for-read to 'binary'.

Juan, can you please describe the scenario (the commands you issue and
any relevant customizations you have) that causes this in detail?
Also, if this doesn't happen in "emacs -Q", please show any relevant
customizations in your init files, especially the ones that customize
the default coding-systems.

Thanks.

Re: Two errors in 27.* with Windows

[Eli Zaretskii]

> Date: Fri, 27 Mar 2020 10:08:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Juan, can you please describe the scenario (the commands you issue and
> any relevant customizations you have) that causes this in detail?
> Also, if this doesn't happen in "emacs -Q", please show any relevant
> customizations in your init files, especially the ones that customize
> the default coding-systems.

Oops, I see you already did describe the recipe in the bug report.
Then let's continue discussing this there.

Re: Two errors in 27.* with Windows

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
> Lars (or anyone else of the Gnus developers), can you help?  We are
> looking for the place that calls (perhaps indirectly) call-process to
> run gpg in this scenario, and in particularly where does that code set
> coding-system-for-read to 'binary'.

I just got a break from work.  It seems I discovered debug-watch works
with let-bound variables last night and got a useful backtrace with the
location that calls epg-find-configuration with the wrong buffer
configuration. If you need more information, please let me know by
replying to the bug report and I will do my best.

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es