bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

[Chris Siebenmann]

The summary is that if you use one of the MH-E limit commands, normally
bound under / in MH-E folder mode, and you attempt to limit (narrow) to
something that isn't present, MH-E doesn't handle the situation properly
and winds up with a narrowed list of folder contents displaying nothing
useful instead of reporting that no messages matched.

To reproduce this, you will need NMH and MH-E configured. Then:

* Enter MH-E, for example with 'M-x mh-rmail'.
* widen the display to cover your entire inbox, '/ w'.
* search for something that is not present: 'C-u / s', and change the
  subject to search for to, for example, 'nosuchsubjectstring'.

When you do this, MH-E will first show an error message in a buffer
below the folder window and then narrow the main folder window to a
single line showing 'scan: no messages match specification'. ('/ w' will
then fix the situation.)

When this happens, my *MH-E Log* buffer shows (without the leading
indent):
	Errors when executing: mark +inbox -add -sequence header 0 0
	mark: no messages match specification

The NMH commands MH-E has run (for me) are:
	/usr/bin/mh/pick +inbox 1-382 -list -subject nosuchsubjectstring
	/usr/bin/mh/mark +inbox -add -sequence header 0 0

The output from this pick command if run by hand *without standard
output being a tty* are:
	$ pick +inbox 1-382 -list -subject nosuchsubjectstring | cat
	pick: no messages match specification
	0

The potentially surprising '0' is documented in the current NMH pick
manual page in "Output when no messages are matched". In brief, with
-list and standard output not being a tty, pick deliberately outputs as
'0' to standard output to signal error.

I believe the code problem is in mh-narrow-to-header-field in
mh-limit.el, where it processes pick output using 'string-to-number' on
each line and then pushes the nominal resulting number on to the
msg-list list with:
	(when num (push num msg-list))

I think that there are two problems with this code. The first is that
string-to-number doesn't return nil on non-numbers, it returns 0. The
second is that sometimes a real '0' is present, but should not be pushed
into the message list either. The two together explain why mark was
invoked with the odd '0 0'; one is from the pick error line and one is
from the actual '0' it output.

I am using the version of NMH packaged with Ubuntu 22.04, which is
listed as NMH 1.7.1. My Emacs is a locally compiled 29.1.

NOTE: This reproduces in 'emacs -Q', but this report is submitted from
inside my regular Emacs environment because life is too short. You can
ignore the list of third party packages, load-path shadows, and other
things.

In GNU Emacs 29.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.33,
 cairo version 1.16.0) of 2023-10-30 built on comps3
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Ubuntu 22.04.3 LTS

Configured using:
 'configure --prefix /u/cks/lib/x64-linux/emacs-29.1 --without-gsettings
 --with-x --with-x-toolkit=gtk3 --with-json --with-tree-sitter
 --with-native-compilation=aot'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM HARFBUZZ JPEG JSON LCMS2
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LC_COLLATE: C
  value of $LC_CTYPE: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  smartparens-mode: t
  global-corfu-mode: t
  corfu-mode: t
  marginalia-mode: t
  vertico-mouse-mode: t
  vertico-multiform-mode: t
  vertico-mode: t
  which-key-mode: t
  global-flycheck-mode: t
  flycheck-mode: t
  backward-forward-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/u/cks/.emacs.d/elpa/transient-20231121.1154/transient hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/transient
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package-core hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package-core
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package-diminish hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package-diminish
/u/cks/.emacs.d/elpa/bind-key-20230203.2004/bind-key hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/bind-key
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package-lint hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package-lint
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package-jump hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package-jump
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package-delight hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package-delight
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package-bind-key hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package-bind-key
/u/cks/.emacs.d/elpa/use-package-20230426.2324/use-package-ensure hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/use-package/use-package-ensure
/u/cks/.emacs.d/elpa/project-0.10.0/project hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/progmodes/project
/u/cks/.emacs.d/elpa/seq-2.24/seq hides /u/cks/lib/x64-linux/emacs-29.1/share/emacs/29.1/lisp/emacs-lisp/seq

Features:
(shadow mh-identity mh-letter emacsbug mule-util mh-limit shr-color
color textsec uni-scripts idna-mapping ucs-normalize uni-confusable
textsec-check mm-archive mail-extr mh-seq mh-inc hl-line mh-tool-bar
flyspell ispell mh-xface mh-cks mh-cks-speed mh-speed speedbar ezimage
dframe gnus-icalendar org-capture org-refile org ob ob-tangle ob-ref
ob-lob ob-table ob-exp org-macro org-src ob-comint org-pcomplete
pcomplete comint ansi-osc ring org-list org-footnote org-faces
org-entities noutline outline ob-emacs-lisp ob-core ob-eval org-cycle
org-table ol org-fold org-fold-core org-keys oc org-loaddefs org-version
org-compat org-macs format-spec icalendar diary-lib diary-loaddefs
cal-menu calendar cal-loaddefs mh-utils mh-mime mh-acros mh-show
gnus-cite gnus-art mm-uu mml2015 gnus-sum shr pixel-fill kinsoku
url-file svg dom gnus-group gnus-undo gnus-start gnus-dbus dbus xml
gnus-cloud nnimap nnmail mail-source utf7 nnoo parse-time iso8601
gnus-spec gnus-int message yank-media dired dired-loaddefs rfc822
mailabbrev gmm-utils mailheader gnus-win gnus-range gnus nnheader range
wid-edit mh-comp sendmail mail-utils mh-gnus mml mm-view mml-smime
mml-sec epa derived epg rfc6068 epg-config smime gnutls puny dig
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mh-folder which-func imenu gnus-util
text-property-search time-date mh-scan mh-e mh-buffers mh-loaddefs
goto-addr comp comp-cstr warnings icons smartparens advice loadhist
thingatpt .emacs corfu marginalia vertico-mouse vertico-multiform
vertico compat which-key flycheck ansi-color find-func rx dash
backward-forward edmacro kmacro cl-extra help-mode pp diminish
use-package use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
company-autoloads consult-flycheck-autoloads consult-flyspell-autoloads
consult-lsp-autoloads corfu-autoloads embark-consult-autoloads
consult-autoloads embark-autoloads evil-autoloads
expand-region-autoloads flycheck-golangci-lint-autoloads
flycheck-autoloads fold-this-autoloads go-mode-autoloads
goto-chg-autoloads lsp-ui-autoloads lsp-mode-autoloads magit-autoloads
pcase git-commit-autoloads magit-section-autoloads marginalia-autoloads
orderless-autoloads project-autoloads markdown-mode-autoloads
f-autoloads rust-mode-autoloads smartparens-autoloads
transient-autoloads try-autoloads vertico-autoloads vundo-autoloads
which-key-autoloads with-editor-autoloads info compat-autoloads
seq-autoloads yaml-mode-autoloads package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify lcms2 dynamic-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 677360 268275)
 (symbols 48 32601 280)
 (strings 32 175807 52415)
 (string-bytes 1 5114423)
 (vectors 16 78133)
 (vector-slots 8 1328934 545011)
 (floats 8 474 1371)
 (intervals 56 853 260)
 (buffers 984 19))

bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

[Eli Zaretskii]

> Cc: cks.emacsbugs-01@cs.toronto.edu
> From: Chris Siebenmann <cks.emacsbugs-01@cs.toronto.edu>
> Date: Tue, 21 Nov 2023 19:28:45 -0500
> 
> The summary is that if you use one of the MH-E limit commands, normally
> bound under / in MH-E folder mode, and you attempt to limit (narrow) to
> something that isn't present, MH-E doesn't handle the situation properly
> and winds up with a narrowed list of folder contents displaying nothing
> useful instead of reporting that no messages matched.
> 
> To reproduce this, you will need NMH and MH-E configured. Then:
> 
> * Enter MH-E, for example with 'M-x mh-rmail'.
> * widen the display to cover your entire inbox, '/ w'.
> * search for something that is not present: 'C-u / s', and change the
>   subject to search for to, for example, 'nosuchsubjectstring'.
> 
> When you do this, MH-E will first show an error message in a buffer
> below the folder window and then narrow the main folder window to a
> single line showing 'scan: no messages match specification'. ('/ w' will
> then fix the situation.)
> 
> When this happens, my *MH-E Log* buffer shows (without the leading
> indent):
> 	Errors when executing: mark +inbox -add -sequence header 0 0
> 	mark: no messages match specification
> 
> The NMH commands MH-E has run (for me) are:
> 	/usr/bin/mh/pick +inbox 1-382 -list -subject nosuchsubjectstring
> 	/usr/bin/mh/mark +inbox -add -sequence header 0 0
> 
> The output from this pick command if run by hand *without standard
> output being a tty* are:
> 	$ pick +inbox 1-382 -list -subject nosuchsubjectstring | cat
> 	pick: no messages match specification
> 	0
> 
> The potentially surprising '0' is documented in the current NMH pick
> manual page in "Output when no messages are matched". In brief, with
> -list and standard output not being a tty, pick deliberately outputs as
> '0' to standard output to signal error.
> 
> I believe the code problem is in mh-narrow-to-header-field in
> mh-limit.el, where it processes pick output using 'string-to-number' on
> each line and then pushes the nominal resulting number on to the
> msg-list list with:
> 	(when num (push num msg-list))
> 
> I think that there are two problems with this code. The first is that
> string-to-number doesn't return nil on non-numbers, it returns 0. The
> second is that sometimes a real '0' is present, but should not be pushed
> into the message list either. The two together explain why mark was
> invoked with the odd '0 0'; one is from the pick error line and one is
> from the actual '0' it output.
> 
> I am using the version of NMH packaged with Ubuntu 22.04, which is
> listed as NMH 1.7.1. My Emacs is a locally compiled 29.1.
> 
> NOTE: This reproduces in 'emacs -Q', but this report is submitted from
> inside my regular Emacs environment because life is too short. You can
> ignore the list of third party packages, load-path shadows, and other
> things.

Adding Bill.

I'm not sure if it is better to report MH-E-related bugs here or
submit the reports to https://sourceforge.net/p/mh-e/bugs/.

bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

[Mike Kupfer]

Eli Zaretskii wrote:

> I'm not sure if it is better to report MH-E-related bugs here or
> submit the reports to https://sourceforge.net/p/mh-e/bugs/.

We (MH-E developers) have talked about dropping the SourceForge bug
tracker and just using the Emacs debbugs, but I don't think we ever
reached a definite decision.  In the meantime, either should work, since
at least one of us (me) is subscribed to Emacs bugs.

> > From: Chris Siebenmann <cks.emacsbugs-01@cs.toronto.edu>
> > Date: Tue, 21 Nov 2023 19:28:45 -0500
[...]
> > When you do this, MH-E will first show an error message in a buffer
> > below the folder window and then narrow the main folder window to a
> > single line showing 'scan: no messages match specification'. ('/ w' will
> > then fix the situation.)

Yes, I can reproduce this.  

Chris, I'm not sure what you're expecting instead.  I'm guessing you'd
like to have the original message list in the folder buffer (no need to
type "/ w"), with the error text displayed like a normal error, rather
than in the folder buffer.  Do I understand correctly?

thanks,
mike

bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

[Chris Siebenmann]

> > > From: Chris Siebenmann <cks.emacsbugs-01@cs.toronto.edu>
> > > Date: Tue, 21 Nov 2023 19:28:45 -0500
> [...]
> > > When you do this, MH-E will first show an error message in a buffer
> > > below the folder window and then narrow the main folder window to a
> > > single line showing 'scan: no messages match specification'. ('/ w' will
> > > then fix the situation.)
>
> Yes, I can reproduce this.  
>
> Chris, I'm not sure what you're expecting instead.  I'm guessing you'd
> like to have the original message list in the folder buffer (no need to
> type "/ w"), with the error text displayed like a normal error, rather
> than in the folder buffer.  Do I understand correctly?

I think the folder should keep the original message list. I don't know
what should be done with the error message, since it's semi-redundant. I
say this because the current limit-to code appears to contain an attempt
to handle the case where no messages match. mh-narrow-to-header-field
has, at the end:

	(if (null msg-list)
	    (message "No matches")
	  [... regular narrowing code ...] )

but since its parsing of pick output puts 0's in msg-list instead of
leaving it empty here, you never get the 'No matches' message. So I
think the earlier code should arrange to leave msg-list empty by not
adding '0' message numbers to it (they're invalid anyway), and then this
code can act normally and people get the 'No matches' message and have
nothing happen.

On the one hand, showing pick's error message for this is redundant with
the 'No matches' message. On the other hand, there might be other pick
errors in odder cases that one wouldn't want to throw away.

(I ran into this issue because I wrote some general pick-based
limit-to-search functions for my own use, and sometimes what I was
checking for wasn't there; in that case I definitely want to leave the
message list intact. Many people probably use the / stuff without a
prefix argument to modify the search terms, or only search for things
that are there in at least one message.)

	- cks

bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

[Bill Wohler via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> wrote:

> Adding Bill.
> 
> I'm not sure if it is better to report MH-E-related bugs here or
> submit the reports to https://sourceforge.net/p/mh-e/bugs/.

Thanks Eli and Chris.

To add to what Mike said, I believe we decided to retire the SourceForge
bug tracker in favor of debbugs.gnu.org, but we haven't had the time to
move all of the issues and turn off the SourceForge tracker.

In the meantime, I'd suggest that folks use the Emacs bug tracker. We'll
find them there.

I'll reproduce this issue over the long and see if I can add 2 cents.

-- 
Bill Wohler <wohler@newt.com> aka <Bill.Wohler@nasa.gov>
http://www.newt.com/wohler/, GnuPG ID:610BD9AD

bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

[Eli Zaretskii]

> cc: Chris Siebenmann <cks.emacsbugs-01@cs.toronto.edu>,
>     67359@debbugs.gnu.org
> From: Bill Wohler <wohler@newt.com>
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Wed, 22 Nov 2023 16:43:09 +0200."
> Date: Wed, 22 Nov 2023 17:56:21 -0800
> 
> Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Adding Bill.
> > 
> > I'm not sure if it is better to report MH-E-related bugs here or
> > submit the reports to https://sourceforge.net/p/mh-e/bugs/.
> 
> Thanks Eli and Chris.
> 
> To add to what Mike said, I believe we decided to retire the SourceForge
> bug tracker in favor of debbugs.gnu.org, but we haven't had the time to
> move all of the issues and turn off the SourceForge tracker.
> 
> In the meantime, I'd suggest that folks use the Emacs bug tracker. We'll
> find them there.

Thanks, but this would mean someone on the MH-E team should be
subscribed to the bug-gnu-emacs list, and respond to MH-E related
issues raised on the bug tracker.  Is that the case?

bug#67359: 29.1; 29.1: MH-E limited display malfunctions if nothing is matched

[Mike Kupfer]

Eli Zaretskii wrote:

> > From: Bill Wohler <wohler@newt.com>
> > Date: Wed, 22 Nov 2023 17:56:21 -0800

> > In the meantime, I'd suggest that folks use the Emacs bug tracker. We'll
> > find them there.
> 
> Thanks, but this would mean someone on the MH-E team should be
> subscribed to the bug-gnu-emacs list, and respond to MH-E related
> issues raised on the bug tracker.  Is that the case?

Yes, I'm subscribed, and I do look for issues related to MH-E.

I haven't yet responded about #67360 (mangled subject line when
forwarding) because I haven't reproduced it yet.  But I think I know why
I haven't reproduced it.  I plan to continue working on it over the US
holiday weekend.

mike