bug#67954: 30.0.50; Gnus: notmuch results are not sorted properly

bug#67954: 30.0.50; Gnus: notmuch results are not sorted properly

[Torsten Bronger]

[-- Attachment #1: Type: text/plain, Size: 12115 bytes --]


If I search for mails using Gnus and notmuch.  My .gnus says:

(setq gnus-secondary-select-methods
      '((nnml "" (gnus-search-engine gnus-search-notmuch)))
      gnus-search-notmuch-config-file "/etc/notmuch/.notmuch-config")

Then I press ^ in group mode, move to the nnml: entry, and press G.
Then, I enter my query string, which starts the search.  Gnus asks how
many articles I want to have, and here the problem begins: If 1000 mails
have been found and I want to see 10, I expect to see the 10 most recent
ones.  However, they seem to be arbitrary.  At least, they are not the
10 first/latest delivered by notmuch (which works in newest-first mode).

If I retrieve all 1000 mails, they are also not sorted in the summary
buffer.  I can hit C-c C-s C-d without effect.  The sorting seems to be
random.

The only sort-related setting in my .gnus is:

(setq gnus-thread-sort-functions '(gnus-thread-sort-by-date))



In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.33, cairo version 1.16.0) of 2023-11-16 built on wolowitz
Repository revision: 4dc26a1e6e11416ea631121e46e2084d4fc29203
Repository branch: HEAD
System Description: Ubuntu 22.04.3 LTS

Configured using:
 'configure --prefix=/usr/ --mandir=/usr/share/man/
 --infodir=/usr/share/info/ --with-x-toolkit=no --with-xpm --with-jpeg
 --with-tiff --with-gif --with-png --without-x
 --with-file-notification=inotify --with-pgtk --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP
NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP XIM GTK3 ZLIB

Important settings:
  value of $EMACSLOADPATH: /home/bronger/.emacs.d/lisp:
  value of $LC_MONETARY: de_DE.UTF-8
  value of $LC_NUMERIC: de_DE.UTF-8
  value of $LC_TIME: de_DE.UTF-8
  value of $LANG: de_DE.UTF-8
  value of $XMODIFIERS: @im=fcitx
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  dap-tooltip-mode: t
  dap-ui-many-windows-mode: t
  dap-ui-controls-mode: t
  dap-ui-mode: t
  treemacs-filewatch-mode: t
  treemacs-follow-mode: t
  treemacs-git-mode: t
  treemacs-fringe-indicator-mode: t
  dap-auto-configure-mode: t
  dap-mode: t
  TeX-PDF-mode: t
  server-mode: t
  global-auto-revert-mode: t
  savehist-mode: t
  display-time-mode: t
  desktop-save-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  minibuffer-regexp-mode: t
  line-number-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/bronger/.emacs.d/elpa/transient-20231103.2312/transient hides /usr/share/emacs/30.0.50/lisp/transient

Features:
(shadow mailalias smtpmail bbdb-message emacsbug mule-util calc-alg
calc-ext calc-menu calc calc-loaddefs rect calc-macs misearch
multi-isearch shortdoc gnus-html help-fns radix-tree gnus-fun smiley
gnus-cite url-http url-gw url-auth gnus-gravatar gravatar dns url-cache
gnus-picon sort gnus-icalendar org-capture icalendar mm-archive
mail-extr textsec uni-scripts idna-mapping ucs-normalize uni-confusable
textsec-check gnus-async gnus-bcklg qp gnus-ml disp-table gnus-search
eieio-opt speedbar ezimage dframe pop3 nndraft nnmh gnus-agent gnus-srvr
gnus-score score-mode nnvirtual gnus-msg nnml nnfolder gnus-cache
bbdb-gnus bbdb-mua bbdb-com nntp vc-hg vc-bzr vc-src vc-sccs vc-svn
vc-cvs vc-rcs log-view vc bug-reference magit-extras magit-submodule
magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull
magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch
magit-reset magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log which-func magit-diff smerge-mode diff
git-commit log-edit pcvs-util add-log magit-core magit-autorevert
magit-margin magit-transient magit-process with-editor magit-mode
transient magit-git magit-base magit-section cursor-sensor js
c-ts-common cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs view lsp-jq lsp-zig lsp-tilt
lsp-steep lsp-svelte lsp-sqls lsp-solidity lsp-ruby-syntax-tree
lsp-ruby-lsp lsp-yaml lsp-xml lsp-vimscript lsp-vhdl lsp-volar lsp-vetur
lsp-html lsp-verilog lsp-vala lsp-v lsp-typeprof lsp-ttcn3 lsp-toml
lsp-terraform lsp-tex lsp-sorbet lsp-solargraph lsp-semgrep lsp-rust
lsp-rubocop lsp-rf lsp-ruff-lsp lsp-remark lsp-racket lsp-r
lsp-purescript lsp-pyright lsp-pylsp lsp-pyls lsp-pwsh lsp-php lsp-pls
lsp-perlnavigator lsp-perl lsp-openscad lsp-ocaml lsp-mojo lsp-magik
lsp-nix lsp-nim lsp-nginx lsp-move lsp-mint lsp-mdx lsp-marksman
lsp-markdown lsp-lua lsp-kotlin lsp-json lsp-javascript lsp-idris
lsp-haxe lsp-groovy lsp-hack lsp-graphql lsp-glsl lsp-gleam lsp-go
lsp-gdscript lsp-fsharp lsp-fortran lsp-eslint lsp-erlang lsp-emmet
lsp-elixir lsp-elm lsp-dockerfile lsp-dhall lsp-d lsp-css lsp-csharp
lsp-crystal lsp-credo lsp-cmake lsp-clojure lsp-semantic-tokens
lsp-clangd lsp-beancount lsp-bash lsp-astro lsp-awk lsp-ansible
lsp-angular lsp-ada lsp-actionscript dap-mouse dap-ui gdb-mi bindat gud
bui bui-list bui-info bui-entry bui-core bui-history bui-button
bui-utils cus-edit pp cus-start lsp-modeline lsp-diagnostics flycheck
lsp-completion cap-words superword subword go-mode find-file ffap
centered-cursor-mode darkroom face-remap rst flyspell ispell
dockerfile-mode dired-aux reftex-dcr reftex reftex-loaddefs reftex-vars
font-latex latexenc preview latex latex-flymake flymake tex-ispell
tex-style tex-mode shell python compat org-element org-persist org-id
org-refile avl-tree oc-basic ol-eww eww xdg url-queue mm-url ol-rmail
ol-mhe ol-irc ol-info ol-gnus nnselect gnus-art mm-uu mml2015 mm-view
mml-smime smime gnutls dig gnus-sum shr pixel-fill kinsoku url-file svg
gnus-group gnus-undo gnus-start gnus-dbus gnus-cloud nnimap nnmail
mail-source utf7 nnoo parse-time gnus-spec gnus-int gnus-range message
sendmail yank-media rfc822 mml mml-sec epa derived mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader gnus-win gnus nnheader gnus-util mail-utils range
mm-util mail-prsvr ol-docview doc-view jka-compr image-mode exif
ol-bibtex bibtex iso8601 ol-bbdb ol-w3m ol-doi org-link-doi org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-src ob-comint
org-pcomplete pcomplete org-list org-footnote org-faces org-entities
ob-emacs-lisp ob-core ob-eval org-cycle org-table ol org-fold
org-fold-core org-keys oc org-loaddefs find-func org-version org-compat
org-macs format-spec sh-script smie treesit executable yaml-mode
time-date vc-git diff-mode vc-dispatcher delight etags-table etags
fileloop generator tj3-mode advice lsp-dart lsp-dart-commands
lsp-dart-flutter-widget-guide lsp-dart-flutter-fringe-colors
lsp-dart-flutter-colors lsp-dart-outline lsp-dart-code-lens lsp-lens
lsp-dart-test-tree lsp-treemacs lsp-treemacs-generic lsp-treemacs-themes
treemacs-treelib treemacs treemacs-header-line treemacs-compatibility
treemacs-mode treemacs-interface treemacs-persistence
treemacs-filewatch-mode treemacs-follow-mode treemacs-rendering
treemacs-annotations treemacs-async treemacs-workspaces treemacs-dom
treemacs-visuals treemacs-fringe-indicator pulse treemacs-faces
treemacs-icons treemacs-scope treemacs-themes treemacs-core-utils
pfuture hl-line treemacs-logging treemacs-customization treemacs-macros
lsp-dart-test-output lsp-dart-test-support lsp-dart-dap
lsp-dart-devtools lsp-dart-flutter-daemon jsonrpc dap-utils dom dap-mode
dap-tasks dap-launch lsp-docker yaml posframe dap-overlays
lsp-dart-closing-labels lsp-dart-utils lsp-dart-protocol lsp-mode
lsp-protocol go-snippets yasnippet-snippets yasnippet xref project
tree-widget wid-edit spinner network-stream puny nsm markdown-mode color
thingatpt noutline outline lv inline imenu ht f f-shortdoc s ewoc epg
rfc6068 epg-config dash dired-x dired dired-loaddefs adaptive-wrap
edmacro kmacro mmm-mode mmm-univ mmm-class mmm-region mmm-auto mmm-vars
mmm-utils mmm-compat bbdb bbdb-site timezone cal-china lunar solar
cal-dst cal-bahai cal-islam cal-hebrew holidays holiday-loaddefs appt
diary-lib diary-loaddefs cal-menu calendar cal-loaddefs tex-fold
easy-mmode tex dbus compile text-property-search comint ansi-osc
ansi-color ring xml crm texmathp comp comp-cstr warnings icons comp-run
comp-common rx server cl-extra help-mode autorevert filenotify savehist
time desktop frameset cus-load auctex-autoloads tex-site bbdb-autoloads
browse-kill-ring-autoloads centered-cursor-mode-autoloads
deadgrep-autoloads dockerfile-mode-autoloads flycheck-autoloads
gnuplot-autoloads go-guru-autoloads go-mode-autoloads
graphviz-dot-mode-autoloads lsp-dart-autoloads dart-mode-autoloads
dap-mode-autoloads bui-autoloads lsp-docker-autoloads
lsp-pyright-autoloads lsp-treemacs-autoloads lsp-mode-autoloads
f-autoloads lua-mode-autoloads magit-autoloads pcase
git-commit-autoloads magit-section-autoloads markdown-mode-autoloads
mastodon-autoloads mmm-mode-autoloads persist-autoloads
list-utils-autoloads pcache-autoloads po-mode-autoloads
request-autoloads rust-mode-autoloads sparql-mode-autoloads
swift-mode-autoloads transient-autoloads treemacs-autoloads
cfrs-autoloads posframe-autoloads ht-autoloads hydra-autoloads
pfuture-autoloads ace-window-autoloads avy-autoloads s-autoloads
dash-autoloads ttl-mode-autoloads with-editor-autoloads info
compat-autoloads yaml-autoloads yaml-mode-autoloads
yasnippet-snippets-autoloads yasnippet-autoloads package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile url-vars cl-loaddefs cl-lib rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win
term/common-win pgtk-dnd touch-screen tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
lcms2 multi-tty move-toolbar make-network-process native-compile emacs)

Memory information:
((conses 16 1147841 306102) (symbols 48 64670 17) (strings 32 254677 34398)
 (string-bytes 1 8184601) (vectors 16 154713) (vector-slots 8 3637027 118152)
 (floats 8 1697 10343) (intervals 56 32800 630) (buffers 984 132))

-- 
Torsten Bronger
Forschungszentrum Jülich GmbH
Zentralbibliothek / Central Library
Fachbereich Open Science / Open Science Group
Forschungsdaten-Management / research data management
52425 Jülich
+49 2461 61-1758
https://www.fz-juelich.de/zb

Sitz der Gesellschaft: Jülich
Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Stefan Müller
Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende),
Karsten Beneke (stellv. Vorsitzender), Dr. Ir. Pieter Jansens

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5461 bytes --]

bug#67954: 30.0.50; Gnus: notmuch results are not sorted properly

[Eric Abrahamsen]

Torsten Bronger <bronger@physik.rwth-aachen.de> writes:

> If I search for mails using Gnus and notmuch.  My .gnus says:
>
> (setq gnus-secondary-select-methods
>       '((nnml "" (gnus-search-engine gnus-search-notmuch)))
>       gnus-search-notmuch-config-file "/etc/notmuch/.notmuch-config")
>
> Then I press ^ in group mode, move to the nnml: entry, and press G.
> Then, I enter my query string, which starts the search.  Gnus asks how
> many articles I want to have, and here the problem begins: If 1000 mails
> have been found and I want to see 10, I expect to see the 10 most recent
> ones.  However, they seem to be arbitrary.  At least, they are not the
> 10 first/latest delivered by notmuch (which works in newest-first mode).
>
> If I retrieve all 1000 mails, they are also not sorted in the summary
> buffer.  I can hit C-c C-s C-d without effect.  The sorting seems to be
> random.

Weird! Thanks for the report. I've piled up a few gnus-search bugs and
will work on them in the next few days, if necessary I'll come back for
more info then.

bug#67954: 30.0.50; Gnus: notmuch results are not sorted properly

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Torsten Bronger <bronger@physik.rwth-aachen.de> writes:
>
>> If I search for mails using Gnus and notmuch.  My .gnus says:
>>
>> (setq gnus-secondary-select-methods
>>       '((nnml "" (gnus-search-engine gnus-search-notmuch)))
>>       gnus-search-notmuch-config-file "/etc/notmuch/.notmuch-config")
>>
>> Then I press ^ in group mode, move to the nnml: entry, and press G.
>> Then, I enter my query string, which starts the search.  Gnus asks how
>> many articles I want to have, and here the problem begins: If 1000 mails
>> have been found and I want to see 10, I expect to see the 10 most recent
>> ones.  However, they seem to be arbitrary.  At least, they are not the
>> 10 first/latest delivered by notmuch (which works in newest-first mode).
>>
>> If I retrieve all 1000 mails, they are also not sorted in the summary
>> buffer.  I can hit C-c C-s C-d without effect.  The sorting seems to be
>> random.
>
> Weird! Thanks for the report. I've piled up a few gnus-search bugs and
> will work on them in the next few days, if necessary I'll come back for
> more info then.

I haven't been able to reproduce this. I don't have an nnml
installation, but I've tried it with nnmaildir. I set
`gnus-thread-sort-functions' to the same value as yours, and remove all
other sorting customizations, then run the search at both the group
level and the server level. In all cases the results are sorted by date,
both when selecting a smaller number of results, or showing everything.

A few questions:

`gnus-search-notmuch-remove-prefix' defaults to "~/Mail", is that where
your mail is?

What do you mean C-s C-s C-d has no effect? It should reconstruct the
*Summary* buffer at least -- do you mean it reconstructs the buffer with
the same random sorting? That would be very strange to me, because at
this point the sorting commands are just working on the articles in the
summary buffer, and it doesn't have anything to do with search anymore.

If no further suspects present themselves, I can try to construct an
nnml mail store to work on.

Thanks,
Eric