bug#50285: 28.0.50; Mew dosen't work correctly if .el files of Mew are native compiled

bug#50285: 28.0.50; Mew dosen't work correctly if .el files of Mew are native compiled

[Yasuhiro Kimura]

Mew is a mail reader for Emacs (http://www.mew.org/). While testing
native compilation with FreeBSD amd64, I faced the problem that Mew
doesn't work correctly if .el files of Mew are native compiled.

I reported the problem to mailing list of Mew and recieved a reply
that Debian also faces it and works it around by setting
'no-native-compile' variable to 't' as file-local variable binding.
So it seems the problem is caused by native compilation itself and
isn't specific to some OS.

Mew is large package which consists of 65 .el files. So it is
difficult to narrow down the problem. To reproduce the problem you
need to install Mew and make minimal configration according the manual
of Mew (http://www.mew.org/en/info/release/).

Assuming that you completed the installation and configuration of Mew,
There are 2 cases that can be certainly reproduced.

[Case 1]

A. Steps to reproduce the problem

1. Start mew with 'M-x mew'.
2. Depending on the personal configuration, it changes which folder is
   visited just after the startup. Typically you see '+inbox'
   folder. If so you can skip this step. Otherwise switch to '+inbox'
   by typing "g", "+inbox" and return key.
3. At the '+inbox' folder, type 's' and return key.
   (Depending on the configuration you may need to type password.)

B. Expected behavior

Newly arrived message are displayed in "Mew: +inbox" buffer.
(If new message isn't arrived there is no change in "Mew: +inbox"
buffer.)

C. What really happens

Following error message is displayed in the echo area.

"error in process sentinel: Invalid function: mew-filter"

[Case 2]

A. Steps to reproduce the problem

1. Start mew with 'M-x mew'.
2. Type "w". Then buffer to compose new message is displayed.
3. Fill out To:, Subject:, message body, etc.
4. Type C-c C-m.

B. Expected behavior

Message such as following is displayed in the echo area.

"+draft/1 has been queued to +queue/1 (from Draft mode)"

And switch back to the folder before the step 2.

C. What really happens

"nil" is displayed in the echo area and don't switch back to the folder
before the step 2.


In GNU Emacs 28.0.50 (build 1, amd64-portbld-freebsd13.0)
Repository revision: 3572613
Repository branch: master
System Description: 13.0-RELEASE-p4

Configured using:
 'configure --disable-build-details --localstatedir=/var
 --without-libsystemd --without-selinux --with-sound=no
 --with-x-toolkit=no --without-cairo --without-dbus --without-gconf
 --without-gif --without-gsettings --without-imagemagick
 --without-jpeg --without-lcms2 --without-libotf --without-m17n-flt
 --without-png --without-rsvg --without-tiff
 --without-toolkit-scroll-bars --without-x --without-xim --without-xpm
 --without-xwidgets --enable-acl --without-cairo --without-dbus
 --without-gconf --without-gif --with-gnutls --without-gsettings
 --without-harfbuzz --without-jpeg --with-json
 --with-file-notification=kqueue --without-lcms2 --without-m17n-flt
 --without-imagemagick --without-mailutils --with-modules
 --with-native-compilation --without-libotf --without-png
 --without-toolkit-scroll-bars --without-rsvg --with-threads
 --without-tiff --without-xft --without-xim --with-xml2 --without-xpm
 --without-xwidgets --with-x-toolkit=no --prefix=/usr/local
 --mandir=/usr/local/man --disable-silent-rules
 --infodir=/usr/local/share/emacs/info/
 --build=amd64-portbld-freebsd13.0 'CFLAGS=-O2 -pipe
 -fstack-protector-strong -isystem /usr/local/include
 -fno-strict-aliasing ' 'CPPFLAGS=-isystem /usr/local/include'
 'LDFLAGS= -L/usr/local/lib/gcc11 -fstack-protector-strong
 -L/usr/local/lib ''

Configured features:
ACL GMP GNUTLS JSON LIBXML2 MODULES NATIVE_COMP NOTIFY KQUEUE PDUMPER
THREADS XIM ZLIB

Important settings:
  value of $LANG: ja_JP.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
/usr/local/share/emacs/28.0.50/site-lisp/transient hides /usr/local/share/emacs/28.0.50/lisp/transient
/home/yasu/share/emacs/lisp/ruby-mode hides /usr/local/share/emacs/28.0.50/lisp/progmodes/ruby-mode
/usr/local/share/emacs/28.0.50/site-lisp/flim/sasl hides /usr/local/share/emacs/28.0.50/lisp/net/sasl

Features:
(shadow emacsbug sendmail w3m-form w3m-symbol w3m-bookmark pp qp
network-stream mew-varsx mew-unix mew-w3m w3m doc-view image-mode exif
image timezone w3m-hist w3m-fb bookmark-w3m w3m-ems wid-edit
w3m-favicon w3m-image tab-line w3m-proc w3m-util mew-auth mew-config
mew-imap2 mew-imap mew-nntp2 mew-nntp mew-pop mew-smtp mew-ssl mew-ssh
mew-net mew-highlight mew-sort mew-fib mew-ext mew-refile mew-demo
mew-attach mew-draft mew-message mew-thread mew-virtual mew-summary4
mew-summary3 mew-summary2 mew-summary mew-search mew-pick mew-passwd
mew-scan mew-syntax mew-bq mew-smime mew-pgp mew-header mew-exec
mew-mark mew-mime mew-edit mew-decode mew-encode mew-cache mew-minibuf
mew-complete mew-addrbook mew-local mew-vars3 mew-vars2 mew-vars
mew-env mew-lang-jp mew-mule3 mew-mule mew-gemacs mew-key mew-func
mew-blvs mew-const mew pcase term/screen term/xterm xterm jka-compr
magit-version yaml-mode tmux mime-setup mail-mime-setup alist
semi-setup path-util apel-ver product semi-def rng-nxml rng-valid
rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util
rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap sgml-mode facemenu dom
nxml-util nxml-enc xmltok edmacro kmacro forge-list forge-commands
forge-semi forge-bitbucket buck forge-gogs gogs forge-gitea gtea
forge-gitlab glab forge-github ghub-graphql treepy gsexp ghub
let-alist gnutls forge-notify forge-revnote forge-pullreq forge-issue
forge-topic yaml parse-time iso8601 bug-reference forge-post
markdown-mode color noutline outline forge-repo forge forge-core
forge-db closql emacsql-sqlite advice emacsql emacsql-compiler
url-http url-auth url-gw nsm magit-submodule magit-obsolete
magit-blame magit-stash magit-reflog magit-bisect magit-push
magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
package browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse url-vars magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode diff diff-mode git-commit
log-edit easy-mmode message rmc puny dired dired-loaddefs rfc822 mml
mml-sec epa derived epg epg-config gnus-util rmail rmail-loaddefs
auth-source password-cache json map text-property-search time-date
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
mm-util ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils
mailheader pcvs-util add-log magit-core magit-autorevert autorevert
filenotify magit-margin magit-transient magit-process with-editor
shell pcomplete comint ring server ansi-color magit-mode transient
format-spec magit-git comp regexp-opt comp-cstr warnings cl-extra
help-mode tool-bar magit-section cl-seq magit-utils crm subr-x seq
eieio byte-opt bytecomp byte-compile cconv eieio-core cl-macs gv
eieio-loaddefs dash gitignore-mode gitconfig-mode conf-mode rx
gitattributes-mode thingatpt cl-loaddefs cl-lib japan-util iso-transl
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads kqueue multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 328603 25793)
 (symbols 48 26450 31)
 (strings 32 94994 4795)
 (string-bytes 1 2888107)
 (vectors 16 56476)
 (vector-slots 8 1017500 70312)
 (floats 8 297 411)
 (intervals 56 382 0)
 (buffers 992 15))

bug#50285: 28.0.50; Mew dosen't work correctly if .el files of Mew are native compiled

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Following error message is displayed in the echo area.
> "error in process sentinel: Invalid function: mew-filter"

This error message indicates that a call to `mew-filter` was compiled as
a function call, while `mew-filter` is a macro.  This happens when
`mew-filter` is not defined at the time the call is compiled by
`bytecomp.el`.

When this problem shows up in a `.elc` file it means the `.elc` file was
miscompiled and the fault is usually attributed to the source code's
failure to make sure the macro was defined (e.g. by using (require 'mew-func)
somewhere earlier in the file).

In the present case, IIUC the `.elc` file itself is correct because
Mew's makefile compensates for the lack of (require 'mew-func) by using
an ad-hoc compilation script that makes sure the right files
(e.g. `mew-func`) are loaded as needed before the compilation of the
relevant code.

But the `.eln` file is not because when generating the `.eln` our code
starts again from the `.el` file and passes it again to `bytecomp.el`
but this time it's done outside of the control of the makefile, and it
so happens that `mew-func.el` was not loaded before compiling (again)
that call to `mew-filter`.

So we really have 3 problems here:

1- The code in Mew should not rely on the makefile rule for
   correct compilation.
   Fixing this would also make it easier to distribute Mew via ELPA
   (where a similar problem shows up since package.el will compile the
   .el files in an unspecified order which could thus result in the
   same kind of miscompilation).

2- The generation of the `.eln` file should start from the `.elc` file
   rather than from the `.el` file because we sometimes lack the
   information needed to correctly re-generate the `.elc` file from the
   `.el` file.


-- Stefan

bug#50285: 28.0.50; Mew dosen't work correctly if .el files of Mew are native compiled

[Eli Zaretskii]

> Cc: 50285@debbugs.gnu.org
> Date: Mon, 30 Aug 2021 17:02:08 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> So we really have 3 problems here:
> 
> 1- The code in Mew should not rely on the makefile rule for
>    correct compilation.
>    Fixing this would also make it easier to distribute Mew via ELPA
>    (where a similar problem shows up since package.el will compile the
>    .el files in an unspecified order which could thus result in the
>    same kind of miscompilation).
> 
> 2- The generation of the `.eln` file should start from the `.elc` file
>    rather than from the `.el` file because we sometimes lack the
>    information needed to correctly re-generate the `.elc` file from the
>    `.el` file.

And 3?...

bug#50285: 28.0.50; Mew dosen't work correctly if .el files of Mew are native compiled

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii [2021-08-31 15:05:14] wrote:

>> Cc: 50285@debbugs.gnu.org
>> Date: Mon, 30 Aug 2021 17:02:08 -0400
>> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> So we really have 3 problems here:
>> 
>> 1- The code in Mew should not rely on the makefile rule for
>>    correct compilation.
>>    Fixing this would also make it easier to distribute Mew via ELPA
>>    (where a similar problem shows up since package.el will compile the
>>    .el files in an unspecified order which could thus result in the
>>    same kind of miscompilation).
>> 
>> 2- The generation of the `.eln` file should start from the `.elc` file
>>    rather than from the `.el` file because we sometimes lack the
>>    information needed to correctly re-generate the `.elc` file from the
>>    `.el` file.
>
> And 3?...

The third problem is the well-known off-by-one,


        Stefan

bug#50285: 28.0.50; Mew dosen't work correctly if .el files of Mew are native compiled

[Andrea Corallo]

Yasuhiro Kimura <yasu@utahime.org> writes:

> Mew is a mail reader for Emacs (http://www.mew.org/). While testing
> native compilation with FreeBSD amd64, I faced the problem that Mew
> doesn't work correctly if .el files of Mew are native compiled.
>
> I reported the problem to mailing list of Mew and recieved a reply
> that Debian also faces it and works it around by setting
> 'no-native-compile' variable to 't' as file-local variable binding.
> So it seems the problem is caused by native compilation itself and
> isn't specific to some OS.

Hi Yasuhiro,

was the missing require issue fixed upstream in the package?

Thanks

  Andrea

bug#50285: 28.0.50; Mew dosen't work correctly if .el files of Mew are native compiled

[Lars Ingebrigtsen]

Yasuhiro Kimura <yasu@utahime.org> writes:

> Following error message is displayed in the echo area.
>
> "error in process sentinel: Invalid function: mew-filter"

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

The problem here is that, as Stefan said, the Mew files can't be
compiled in a normal fashion, and that also break native-compiling.  The
problem seems to be that mew-local.el doesn't (require 'mew) before
using the `mew-filter' macro, and that breaks things.

So this should be fixed in Mew, and there's nothing to be done on the
Emacs side here, really, and I'm therefore closing this bug report.

(The discussion then went on to whether the native compiler should use
the .elc files instead of the .el files, but that's a different
discussion, I think.)