bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Jose A. Ortega Ruiz]

Hi,

I've been trying the recently added gnus-search with an nnimap local
server (dovecot) that i index with notmuch.  More concretely:

   - dovecot is saving my incoming email in maildirs located at
     ~/var/mail.
   - notmuch is indexing ~/var/mail.  that's working well: i can use
     notmuch without problems.
   - gnus has a secondary method for accessing dovecot using nnimap, but
     i would like searches on its groups to be performed using notmuch,
     not the built-in imap search.  so i add a secondary method that
     looks like this:

       (nnimap ""
              (nnimap-address "localhost")
              (nnimap-expunge immediately)
              (gnus-search-engine gnus-search-notmuch
                 (remove-prefix "/home/jao/var/mail/")))

   - if i now query any of my nnimap groups with a query that gives me
     back one or more results, i get an error with a backtrace that
     looks like this one ("nnimap:feeds.emacs" is the group name in this
     concrete case):

  --8<---------------cut here---------------start------------->8---
  gnus-compress-sequence((nil))
  nnselect-compress-artlist([["nnimap:feeds.emacs" nil 100]])
  nnselect-request-group("nnselect-87lffe8d08.fsf" "nnselect-ephemeral" t ("nnselect:nnselect-87lffe8d08.fsf" 3 nil nil (nnselect "nnselect-ephemeral" (nnselect-address "nnselect")) ((quit-config #<buffer *Group*> . group) (nnselect-specs (nnselect-function . gnus-search-run-query) (nnselect-args (search-query-spec (query . "Maxim") (raw)) (search-group-spec ("nnimap:" "nnimap:feeds/emacs")))) (nnselect-artlist))))
  gnus-request-group("nnselect:nnselect-87lffe8d08.fsf" t nil ("nnselect:nnselect-87lffe8d08.fsf" 3 nil nil (nnselect "nnselect-ephemeral" (nnselect-address "nnselect")) ((quit-config #<buffer *Group*> . group) (nnselect-specs (nnselect-function . gnus-search-run-query) (nnselect-args (search-query-spec (query . "Maxim") (raw)) (search-group-spec ("nnimap:" "nnimap:feeds/emacs")))) (nnselect-artlist))))
  gnus-select-newsgroup("nnselect:nnselect-87lffe8d08.fsf" t nil)
  gnus-summary-read-group-1("nnselect:nnselect-87lffe8d08.fsf" t t nil nil nil)
  gnus-summary-read-group("nnselect:nnselect-87lffe8d08.fsf" t t nil nil nil nil)
  gnus-group-read-group(t t "nnselect:nnselect-87lffe8d08.fsf" nil)
  gnus-group-read-ephemeral-group("nnselect-87lffe8d08.fsf" (nnselect "nnselect") nil (#<buffer *Group*> . group) nil nil ((nnselect-specs (nnselect-function . gnus-search-run-query) (nnselect-args (search-query-spec (query . "Maxim") (raw)) (search-group-spec ("nnimap:" "nnimap:feeds/emacs")))) (nnselect-artlist)))
  gnus-group-read-ephemeral-search-group(nil)
  funcall-interactively(gnus-group-read-ephemeral-search-group nil)
  call-interactively(gnus-group-read-ephemeral-search-group nil nil)
  command-execute(gnus-group-read-ephemeral-search-group)
  --8<---------------cut here---------------end--------------->8---

Maybe this trick of using notmuch on imap groups is not really supposed
to work? If i used the default gnus-search-imap engine, queries work
fine, but notmuch is so much faster that it'd be nice to use its engine
instead.


Thanks!



In GNU Emacs 28.0.50 (build 19, x86_64-pc-linux-gnu, cairo version 1.16.0)
 of 2020-11-06 built on osgiliath
Repository revision: 093a6bec52d21aba6b3b1318a6d7bafc2c870f25
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12009000
System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure --prefix=/usr/local/stow/emacs --with-x-toolkit=no
 --with-imagemagick'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO IMAGEMAGICK SOUND DBUS GSETTINGS GLIB
NOTIFY INOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
LIBOTF ZLIB OLDXMENU X11 XDBE XIM MODULES THREADS JSON PDUMPER LCMS2

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  semantic-minor-modes-format: ((:eval (if (or semantic-highlight-edits-mode semantic-show-unmatched-syntax-mode)  S)))
  gnus-agent-group-mode: t
  circe-lagmon-mode: t
  gnus-topic-mode: t
  telega-mode-line-mode: t
  ednc-mode: t
  global-diff-hl-mode: t
  eshell-syntax-highlighting-global-mode: t
  pdf-occur-global-minor-mode: t
  gnus-undo-mode: t
  winner-mode: t
  modern-fringes-mode: t
  show-paren-mode: t
  persistent-scratch-autosave-mode: t
  global-so-long-mode: t
  display-battery-mode: t
  global-company-mode: t
  ivy-rich-mode: t
  counsel-mode: t
  ivy-mode: t
  savehist-mode: t
  recentf-mode: t
  save-place-mode: t
  global-magit-file-mode: t
  magit-auto-revert-mode: t
  global-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t

Load-path shadows:
~/lib/elisp/telega hides /home/jao/.emacs.d/elpa.28/telega-20201101.1649/telega
/home/jao/.emacs.d/elpa.28/circe-20201101.1515/shorten hides /home/jao/.emacs.d/elpa.28/tracking-20201101.1045/shorten
/home/jao/.emacs.d/elpa.28/circe-20201101.1515/tracking hides /home/jao/.emacs.d/elpa.28/tracking-20201101.1045/tracking
/home/jao/lib/elisp/jao/bmk/dot-emacs hides /home/jao/.emacs.d/elpa.28/tuareg-20200518.1820/dot-emacs
/home/jao/.emacs.d/elpa.28/org-20201102/org-num hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-num
/home/jao/.emacs.d/elpa.28/org-20201102/ob-R hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-R
/home/jao/.emacs.d/elpa.28/org-20201102/ob-dot hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-dot
/home/jao/.emacs.d/elpa.28/org-20201102/org-feed hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-feed
/home/jao/.emacs.d/elpa.28/org-20201102/ob-clojure hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-clojure
/home/jao/.emacs.d/elpa.28/org-20201102/ob-sql hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-sql
/home/jao/.emacs.d/elpa.28/org-20201102/ob-latex hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-latex
/home/jao/.emacs.d/elpa.28/org-20201102/org-colview hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-colview
/home/jao/.emacs.d/elpa.28/org-20201102/ol-gnus hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-gnus
/home/jao/.emacs.d/elpa.28/org-20201102/ob-ref hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-ref
/home/jao/.emacs.d/elpa.28/org-20201102/org-archive hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-archive
/home/jao/.emacs.d/elpa.28/org-20201102/ob-forth hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-forth
/home/jao/.emacs.d/elpa.28/org-20201102/ob-hledger hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-hledger
/home/jao/.emacs.d/elpa.28/org-20201102/org-entities hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-entities
/home/jao/.emacs.d/elpa.28/org-20201102/org-attach hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-attach
/home/jao/.emacs.d/elpa.28/org-20201102/ox-odt hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-odt
/home/jao/.emacs.d/elpa.28/org-20201102/ob-table hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-table
/home/jao/.emacs.d/elpa.28/org-20201102/ox hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox
/home/jao/.emacs.d/elpa.28/org-20201102/ob-eval hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-eval
/home/jao/.emacs.d/elpa.28/org-20201102/ol-bbdb hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-bbdb
/home/jao/.emacs.d/elpa.28/org-20201102/ob-ledger hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-ledger
/home/jao/.emacs.d/elpa.28/org-20201102/ox-org hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-org
/home/jao/.emacs.d/elpa.28/org-20201102/ob-exp hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-exp
/home/jao/.emacs.d/elpa.28/org-20201102/ob-lisp hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-lisp
/home/jao/.emacs.d/elpa.28/org-20201102/ob-sqlite hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-sqlite
/home/jao/.emacs.d/elpa.28/org-20201102/ob-ebnf hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-ebnf
/home/jao/.emacs.d/elpa.28/org-20201102/ol-eshell hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-eshell
/home/jao/.emacs.d/elpa.28/org-20201102/ob-screen hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-screen
/home/jao/.emacs.d/elpa.28/org-20201102/ob-mscgen hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-mscgen
/home/jao/.emacs.d/elpa.28/org-20201102/ob-lob hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-lob
/home/jao/.emacs.d/elpa.28/org-20201102/ob-matlab hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-matlab
/home/jao/.emacs.d/elpa.28/org-20201102/org-protocol hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-protocol
/home/jao/.emacs.d/elpa.28/org-20201102/ob-makefile hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-makefile
/home/jao/.emacs.d/elpa.28/org-20201102/ob-awk hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-awk
/home/jao/.emacs.d/elpa.28/org-20201102/ob-scheme hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-scheme
/home/jao/.emacs.d/elpa.28/org-20201102/org-clock hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-clock
/home/jao/.emacs.d/elpa.28/org-20201102/ol-info hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-info
/home/jao/.emacs.d/elpa.28/org-20201102/ob hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob
/home/jao/.emacs.d/elpa.28/org-20201102/ob-shen hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-shen
/home/jao/.emacs.d/elpa.28/org-20201102/ox-ascii hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-ascii
/home/jao/.emacs.d/elpa.28/org-20201102/ob-C hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-C
/home/jao/.emacs.d/elpa.28/org-20201102/ob-core hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-core
/home/jao/.emacs.d/elpa.28/org-20201102/org-duration hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-duration
/home/jao/.emacs.d/elpa.28/org-20201102/ob-processing hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-processing
/home/jao/.emacs.d/elpa.28/org-20201102/org-datetree hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-datetree
/home/jao/.emacs.d/elpa.28/org-20201102/ob-haskell hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-haskell
/home/jao/.emacs.d/elpa.28/org-20201102/org-element hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-element
/home/jao/.emacs.d/elpa.28/org-20201102/org-crypt hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-crypt
/home/jao/.emacs.d/elpa.28/org-20201102/org-pcomplete hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-pcomplete
/home/jao/.emacs.d/elpa.28/org-20201102/ob-abc hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-abc
/home/jao/.emacs.d/elpa.28/org-20201102/ob-tangle hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-tangle
/home/jao/.emacs.d/elpa.28/org-20201102/ox-beamer hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-beamer
/home/jao/.emacs.d/elpa.28/org-20201102/org-keys hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-keys
/home/jao/.emacs.d/elpa.28/org-20201102/ol-eww hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-eww
/home/jao/.emacs.d/elpa.28/org-20201102/ob-ruby hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-ruby
/home/jao/.emacs.d/elpa.28/org-20201102/ob-java hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-java
/home/jao/.emacs.d/elpa.28/org-20201102/org-attach-git hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-attach-git
/home/jao/.emacs.d/elpa.28/org-20201102/ob-gnuplot hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-gnuplot
/home/jao/.emacs.d/elpa.28/org-20201102/org-ctags hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-ctags
/home/jao/.emacs.d/elpa.28/org-20201102/ob-picolisp hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-picolisp
/home/jao/.emacs.d/elpa.28/org-20201102/ob-perl hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-perl
/home/jao/.emacs.d/elpa.28/org-20201102/ol-bibtex hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-bibtex
/home/jao/.emacs.d/elpa.28/org-20201102/org-capture hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-capture
/home/jao/.emacs.d/elpa.28/org-20201102/ob-python hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-python
/home/jao/.emacs.d/elpa.28/org-20201102/org-list hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-list
/home/jao/.emacs.d/elpa.28/org-20201102/ol-mhe hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-mhe
/home/jao/.emacs.d/elpa.28/org-20201102/ox-html hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-html
/home/jao/.emacs.d/elpa.28/org-20201102/ox-latex hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-latex
/home/jao/.emacs.d/elpa.28/org-20201102/ob-stan hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-stan
/home/jao/.emacs.d/elpa.28/org-20201102/ob-coq hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-coq
/home/jao/.emacs.d/elpa.28/org-20201102/ob-js hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-js
/home/jao/.emacs.d/elpa.28/org-20201102/ob-css hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-css
/home/jao/.emacs.d/elpa.28/org-20201102/ob-sed hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-sed
/home/jao/.emacs.d/elpa.28/org-20201102/ol-docview hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-docview
/home/jao/.emacs.d/elpa.28/org-20201102/ox-md hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-md
/home/jao/.emacs.d/elpa.28/org-20201102/ob-ocaml hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-ocaml
/home/jao/.emacs.d/elpa.28/org-20201102/ob-ditaa hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-ditaa
/home/jao/.emacs.d/elpa.28/org-20201102/org-tempo hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-tempo
/home/jao/.emacs.d/elpa.28/org-20201102/org-table hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-table
/home/jao/.emacs.d/elpa.28/org-20201102/org-agenda hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-agenda
/home/jao/.emacs.d/elpa.28/org-20201102/org-indent hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-indent
/home/jao/.emacs.d/elpa.28/org-20201102/ol-rmail hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-rmail
/home/jao/.emacs.d/elpa.28/org-20201102/org-id hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-id
/home/jao/.emacs.d/elpa.28/org-20201102/ox-texinfo hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-texinfo
/home/jao/.emacs.d/elpa.28/org-20201102/ob-org hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-org
/home/jao/.emacs.d/elpa.28/org-20201102/org-src hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-src
/home/jao/.emacs.d/elpa.28/org-20201102/ob-J hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-J
/home/jao/.emacs.d/elpa.28/org-20201102/ox-icalendar hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-icalendar
/home/jao/.emacs.d/elpa.28/org-20201102/ob-lilypond hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-lilypond
/home/jao/.emacs.d/elpa.28/org-20201102/org-timer hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-timer
/home/jao/.emacs.d/elpa.28/org-20201102/ob-sass hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-sass
/home/jao/.emacs.d/elpa.28/org-20201102/ol hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol
/home/jao/.emacs.d/elpa.28/org-20201102/org-inlinetask hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-inlinetask
/home/jao/.emacs.d/elpa.28/org-20201102/ox-publish hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-publish
/home/jao/.emacs.d/elpa.28/org-20201102/ob-asymptote hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-asymptote
/home/jao/.emacs.d/elpa.28/org-20201102/ob-maxima hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-maxima
/home/jao/.emacs.d/elpa.28/org-20201102/org-lint hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-lint
/home/jao/.emacs.d/elpa.28/org-20201102/ob-fortran hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-fortran
/home/jao/.emacs.d/elpa.28/org-20201102/org hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org
/home/jao/.emacs.d/elpa.28/org-20201102/org-faces hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-faces
/home/jao/.emacs.d/elpa.28/org-20201102/ob-plantuml hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-plantuml
/home/jao/.emacs.d/elpa.28/org-20201102/org-install hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-install
/home/jao/.emacs.d/elpa.28/org-20201102/org-macs hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-macs
/home/jao/.emacs.d/elpa.28/org-20201102/org-mouse hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-mouse
/home/jao/.emacs.d/elpa.28/org-20201102/org-habit hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-habit
/home/jao/.emacs.d/elpa.28/org-20201102/ox-man hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ox-man
/home/jao/.emacs.d/elpa.28/org-20201102/ol-w3m hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-w3m
/home/jao/.emacs.d/elpa.28/org-20201102/ob-octave hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-octave
/home/jao/.emacs.d/elpa.28/org-20201102/org-footnote hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-footnote
/home/jao/.emacs.d/elpa.28/org-20201102/ol-irc hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ol-irc
/home/jao/.emacs.d/elpa.28/org-20201102/ob-calc hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-calc
/home/jao/.emacs.d/elpa.28/org-20201102/ob-comint hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-comint
/home/jao/.emacs.d/elpa.28/org-20201102/ob-groovy hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-groovy
/home/jao/.emacs.d/elpa.28/org-20201102/org-compat hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-compat
/home/jao/.emacs.d/elpa.28/org-20201102/ob-lua hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-lua
/home/jao/.emacs.d/elpa.28/org-20201102/org-mobile hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-mobile
/home/jao/.emacs.d/elpa.28/org-20201102/ob-eshell hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-eshell
/home/jao/.emacs.d/elpa.28/org-20201102/org-macro hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-macro
/home/jao/.emacs.d/elpa.28/org-20201102/org-plot hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-plot
/home/jao/.emacs.d/elpa.28/org-20201102/org-goto hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-goto
/home/jao/.emacs.d/elpa.28/org-20201102/ob-vala hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-vala
/home/jao/.emacs.d/elpa.28/org-20201102/ob-emacs-lisp hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-emacs-lisp
/home/jao/.emacs.d/elpa.28/org-20201102/ob-shell hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-shell
/home/jao/.emacs.d/elpa.28/org-20201102/org-version hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-version
/home/jao/.emacs.d/elpa.28/org-20201102/org-loaddefs hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/org-loaddefs
/home/jao/.emacs.d/elpa.28/org-20201102/ob-io hides /usr/local/stow/emacs/share/emacs/28.0.50/lisp/org/ob-io

Features:
(shadow gnus-fun bbdb-pgp mailalias bbdb-message flow-fill ffap
magit-extras url-cache lui-track circe-display-images circe-color-nicks
circe-lagmon circe lui-irc-colors irc lcs lui-format circe-compat
bash-completion em-unix em-script em-prompt em-ls em-hist em-pred
em-glob em-dirs esh-var em-cmpl em-basic em-banner em-alias gnus-cite qp
mm-archive mail-extr gnus-bcklg nnmaildir gnus-async gnus-dup gnus-ml
gnus-topic utf-7 bbdb-gnus gnus-delay gnus-draft gnus-agent gnus-srvr
gnus-score score-mode nnvirtual gnus-cache gnus-msg nndraft nnmh
gnus-demon nntp cal-move help-fns radix-tree cl-print debug backtrace
quail counsel-spotify counsel-spotify-messages
counsel-spotify-notifications counsel-spotify-backends
counsel-spotify-search ucs-normalize calc-bin calc-misc calc-menu
calc-aent w3m-cookie w3m-form w3m-symbol w3m-filter w3m-search
w3m-bookmark w3m-tabmenu w3m-session view cal-iso emojify apropos
tar-mode arc-mode archive-mode ht twittering-mode slack slack-company
slack-unread slack-websocket slack-thread-event slack-room-event
slack-star-event slack-reaction-event slack-reply-event slack-typing
slack-slash-commands slack-message-event slack-event
slack-dialog-edit-element-buffer slack-dialog-buffer slack-dialog
slack-stars-buffer slack-search-result-buffer
slack-thread-message-compose-buffer slack-file-list-buffer
slack-file-info-buffer slack-all-threads-buffer slack-message-buffer
slack-user-profile-buffer slack-pinned-items-buffer slack-pinned-item
slack-thread-message-buffer slack-room-info-buffer slack-room-buffer
slack-message-share-buffer slack-message-edit-buffer
slack-room-message-compose-buffer slack-message-compose-buffer
slack-message-attachment-preview-buffer slack-action slack-star
slack-reminder slack-search slack-message-reaction slack-message-editor
slack-message-sender slack-message-notification slack-buffer
slack-message-formatter slack-thread slack-im slack-channel slack-group
slack-conversations slack-create-message slack-attachment
slack-selectable slack-bot-message slack-user-message slack-file
slack-message slack-message-faces slack-unescape slack-block
slack-mrkdwn slack-usergroup slack-reaction slack-modeline slack-room
slack-counts slack-user slack-bot slack-dnd-status slack-emoji
slack-image slack-request slack-log request lui tracking shorten
flyspell ispell slack-team slack-team-ws slack-util websocket bindat
copyright sh-script executable sgml-mode paredit vc-mtn vc-hg vc-bzr
vc-src vc-sccs vc-svn vc-cvs vc-rcs mule-util ace-window avy jao-patches
bml-logs bml bml-misc bml-whizzml bml-clojure bml-clj-tests bml-python
info-look bml-skels bml-utils whizzml-skeletons skeleton whizzml-mode
exwm-systemtray xcb-systemtray xcb-xembed exwm-edit jao-afio exwm
exwm-input xcb-keysyms xcb-xkb exwm-manage exwm-floating xcb-cursor
xcb-render exwm-layout exwm-workspace exwm-core xcb-ewmh xcb-icccm xcb
xcb-xproto xcb-types xcb-debug kana kanji-mode jao-emms-random-album
ivy-emms jao-emms-lyrics jao-lyrics network-stream jao-emms-info-track
jao-emms jao-osd emms-librefm-stream emms-librefm-scrobbler
emms-playlist-limit emms-volume emms-volume-mixerctl emms-volume-pulse
emms-volume-amixer emms-i18n emms-history emms-score emms-stream-info
emms-metaplaylist-mode emms-bookmarks emms-cue emms-mode-line-icon
emms-browser sort emms-playlist-sort emms-last-played emms-player-xine
emms-player-mpd emms-playing-time emms-lyrics emms-url emms-streams
emms-show-all emms-tag-editor emms-mark emms-mode-line emms-cache
emms-info-exiftool emms-info-tinytag emms-info-metaflac
emms-info-opusinfo emms-info-ogginfo emms-info-mp3info emms-info
emms-later-do emms-playlist-mode emms-player-vlc emms-player-mpv
emms-player-mplayer emms-player-simple emms-source-playlist
emms-source-file locate emms-setup emms emms-compat spotify
telega-obsolete telega telega-tdlib-events telega-modes telega-webpage
telega-root telega-chat telega-company telega-user telega-notifications
telega-msg telega-tme telega-sticker telega-i18n telega-vvnote
telega-media telega-voip telega-ffplay telega-info telega-sort
telega-filter telega-ins telega-folders telega-inline telega-tdlib
telega-util rainbow-identifiers telega-server telega-core cursor-sensor
telega-customize emacsbug jao-ednc jao-minibuffer ednc jao-proton-utils
enwc enwc-backend bluetooth github-review deferred a gitconfig-mode
conf-mode git-timemachine diff-hl vc-dir ewoc json-mode json-reformat
json-snatcher js cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs merlin-company merlin-cap merlin utop
utop-minor-mode tuareg tuareg-opam flymake-proc flymake caml-help
caml-types caml-emacs find-file alchemist alchemist-macroexpand
alchemist-company alchemist-help alchemist-complete alchemist-refcard
alchemist-phoenix alchemist-compile alchemist-iex alchemist-message
alchemist-hooks alchemist-hex alchemist-mix alchemist-info
alchemist-goto alchemist-scope alchemist-eval alchemist-interact
alchemist-server alchemist-execute alchemist-report alchemist-test-mode
alchemist-project alchemist-file alchemist-key alchemist-utils
elixir-mode elixir-format pkg-info epl elixir-smie erlang
virtualenvwrapper gud ediprolog sesman vc vc-dispatcher clojure-mode
lisp-mnt geiser edit-list debbugs soap-client warnings rng-xsd
xsd-regexp flycheck eshell-autojump eshell-up git-ps1-mode em-term
eshell-syntax-highlighting vterm face-remap vterm-module term ehelp
jao-doc-view saveplace-pdf-view pdf-occur ibuf-ext ibuffer
ibuffer-loaddefs tablist tablist-filter semantic/wisent/comp
semantic/wisent semantic/wisent/wisent semantic/util-modes semantic/util
semantic semantic/tag semantic/lex semantic/fw mode-local cedet
pdf-isearch pdf-misc pdf-tools pdf-view magit-bookmark bookmark
pdf-cache pdf-info tq pdf-util bbdb-mua bbdb-anniv bbdb-com jao-frm
visual-fill-column counsel-notmuch notmuch hl-line notmuch-tree
notmuch-jump notmuch-hello notmuch-show notmuch-print notmuch-crypto
notmuch-mua notmuch-message notmuch-draft notmuch-maildir-fcc
notmuch-address notmuch-company notmuch-parser notmuch-wash coolj
notmuch-query goto-addr notmuch-tag notmuch-lib notmuch-compat
display-fill-column-indicator orgit-forge forge-list forge-commands
forge-semi forge-bitbucket buck forge-gogs gogs forge-gitea gtea
forge-gitlab glab forge-github ghub-graphql treepy gsexp ghub let-alist
forge-notify forge-revnote forge-pullreq forge-issue forge-topic
bug-reference forge-post forge-repo forge forge-core forge-db closql
emacsql-sqlite emacsql emacsql-compiler url-http url-auth url-gw nsm
cdlatex texmathp disp-table w3m w3m-hist w3m-fb bookmark-w3m w3m-ems
w3m-favicon w3m-image tab-line w3m-proc w3m-util bibtex bbdb bbdb-site
timezone smtpmail sendmail randomsig gnutls markdown-toc s markdown-mode
thingatpt org-static-blog htmlize jao-org-utils ol-info ol-docview
doc-view image-mode exif ol-bbdb ol-eshell esh-mode eshell esh-cmd
esh-ext esh-opt esh-proc esh-io esh-arg esh-module esh-groups esh-util
ol-w3m ol-gnus nnselect gnus-search eieio-opt speedbar ezimage dframe
gnus-icalendar org-capture gnus-art mm-uu mml2015 mm-view mml-smime
smime dig gnus-sum shr kinsoku svg dom gnus-group gnus-undo gnus-start
gnus-dbus gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo gnus-spec
gnus-int gnus-range gnus-win icalendar org-agenda org-refile gnus
nnheader epresent ob-shell ob-scheme ob-python python tramp-sh tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat
parse-time iso8601 ls-lisp ob-org ob-ocaml ob-makefile ob-haskell
ob-gnuplot ob-clojure ob-calc calc-store calc-trail calc-ext calc
calc-loaddefs calc-macs ob-prolog prolog smie align ob-elixir org-tempo
tempo ox-odt rng-loc rng-uri rng-parse rng-match rng-dt rng-util
rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar ox-html table ox-ascii ox-publish ox org-element avl-tree
org-fragtog winner bm modern-fringes jao-notify paren iscroll inform
find-dired dired-x jka-compr company-oddmuse company-keywords
company-etags etags fileloop generator company-gtags
company-dabbrev-code company-dabbrev company-files company-clang
company-capf company-cmake company-semantic company-template
company-bbdb persistent-scratch so-long epa-file battery cal-china lunar
solar cal-dst cal-bahai cal-islam cal-hebrew holidays hol-loaddefs
vc-git appt diary-lib diary-loaddefs company-quickhelp pos-tip
company-math math-symbol-lists company pcase alert log4e notifications
gntp ivy-rich wgrep-ag wgrep grep diminish counsel xdg xref project
compile swiper ivy flx delsel ivy-faces ivy-overlay colir autoinsert
time jao-greenish-theme jao-themes cl memory-usage jao-sleep dbus xml
undo-fu-session savehist recentf tree-widget saveplace
gnu-elpa-keyring-update poly-org orgit magit-submodule magit-obsolete
magit-blame magit-stash magit-reflog magit-bisect magit-push magit-pull
magit-fetch magit-clone magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-tag magit-merge magit-branch
magit-reset magit-files magit-refs magit-status magit magit-repos
magit-apply magit-wip magit-log which-func imenu magit-diff smerge-mode
diff diff-mode magit-core magit-autorevert autorevert filenotify
magit-margin magit-transient magit-process magit-mode git-commit
transient magit-git magit-section magit-utils crm log-edit message rmc
puny dired dired-loaddefs rfc822 mml mml-sec epa epg epg-config
gnus-util rmail rmail-loaddefs text-property-search mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log
with-editor async-bytecomp async shell server dash org ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src ob-comint
org-pcomplete pcomplete comint ansi-color org-list org-faces
org-entities time-date noutline outline org-version ob-emacs-lisp
org-table ol org-keys org-loaddefs find-func cal-menu calendar
cal-loaddefs polymode derived poly-lock polymode-base polymode-weave
polymode-export polymode-compat polymode-methods polymode-core
polymode-classes eieio-custom eieio-base color paradox paradox-menu
paradox-commit-list hydra ring lv cus-edit pp cus-start cus-load
wid-edit paradox-execute paradox-github paradox-core spinner cl-extra
help-mode use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key easy-mmode
use-package-core literate-elisp ob-core org-compat advice ob-eval
org-macs format-spec finder-inf tex-site gh-common marshal eieio-compat
rx edmacro kmacro w3m-load info package easymenu browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo x multi-tty make-network-process emacs)

Memory information:
((conses 16 3337525 350243)
 (symbols 48 95123 8)
 (strings 32 552087 43578)
 (string-bytes 1 18251913)
 (vectors 16 280839)
 (vector-slots 8 8179302 322493)
 (floats 8 3268 3751)
 (intervals 56 120431 16846)
 (buffers 992 183))

--
No amount of belief makes something a fact. -James Randi

--
A man has to live with himself, and he should see to it that he always has
good company. -Charles Evans Hughes, jurist (1862-1948)

-- 
Experience is not what happens to you; it is what you do with what
happens to you.
 - Aldous Huxley

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Eric Abrahamsen]

"Jose A. Ortega Ruiz" <mail@jao.io> writes:

> Hi,
>
> I've been trying the recently added gnus-search with an nnimap local
> server (dovecot) that i index with notmuch.  More concretely:

I'm curious if this used to work before gnus-search? My understanding is
that it never should have worked: notmuch returns search results as
filenames on your local system, while dovecot wants search results
returned as its own internal message UIDs. 

If this worked before gnus-search, I would very much like to know that,
and to know *how* it worked, given that it's two systems that aren't
meant to talk to one another.

Thanks,
Eric

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Jose A. Ortega Ruiz]

On Sat, Nov 07 2020, Eric Abrahamsen wrote:

> "Jose A. Ortega Ruiz" <mail@jao.io> writes:
>
>> Hi,
>>
>> I've been trying the recently added gnus-search with an nnimap local
>> server (dovecot) that i index with notmuch.  More concretely:
>
> I'm curious if this used to work before gnus-search? My understanding is
> that it never should have worked: notmuch returns search results as
> filenames on your local system, while dovecot wants search results
> returned as its own internal message UIDs.
>
> If this worked before gnus-search, I would very much like to know that,
> and to know *how* it worked, given that it's two systems that aren't
> meant to talk to one another.

I would have sworn it worked, but now you make me doubt it.  I think the
gist might be that I am telling dovecot to store its mails in maildirs,
and there the filenames are essentially the maildir followed by the
message id.

But if that doesn't make sense, i might be misremembering.  I'll have to
go back to a previous commit, recompile and check again.

Cheers,
jao
-- 
Journalist: "Who owns the patent on this vaccine?"
Jonas Salk: "Well, the people, I would say.  There is no patent. Could you
patent the sun?"
  -Jonas Salk, medical researcher and developer of polio vaccine (1914-1995)

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Eric Abrahamsen]

On 11/08/20 02:49 AM, Jose A. Ortega Ruiz wrote:
> On Sat, Nov 07 2020, Eric Abrahamsen wrote:
>
>> "Jose A. Ortega Ruiz" <mail@jao.io> writes:
>>
>>> Hi,
>>>
>>> I've been trying the recently added gnus-search with an nnimap local
>>> server (dovecot) that i index with notmuch.  More concretely:
>>
>> I'm curious if this used to work before gnus-search? My understanding is
>> that it never should have worked: notmuch returns search results as
>> filenames on your local system, while dovecot wants search results
>> returned as its own internal message UIDs.
>>
>> If this worked before gnus-search, I would very much like to know that,
>> and to know *how* it worked, given that it's two systems that aren't
>> meant to talk to one another.
>
> I would have sworn it worked, but now you make me doubt it.  I think the
> gist might be that I am telling dovecot to store its mails in maildirs,
> and there the filenames are essentially the maildir followed by the
> message id.
>
> But if that doesn't make sense, i might be misremembering.  I'll have to
> go back to a previous commit, recompile and check again.

It's something that's come up on gnus.general several times over the
years, but I think no one's ever sat down and tried to figure out
exactly how/if it's supposed to work. Thinking about it logically, I
just don't see how it could. On my filesystem (in the maildir maintained
by Dovecot), your message has the filename:

1604808232.M340024P65055.slip,S=3293,W=3369:2,a

And that's how notmuch will return it. If I search for the same message
via Dovecot's internal search, the UUID is 1116. Is there anything to go
from one to the other?

In a few days I can try to set this up and see if there's a way to make
it work.

(BTW have you turned on full text search in your local Dovecot? That
makes imap searches "fast enough" for me, though I doubt they're as fast
as notmuch.)

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Jose A. Ortega Ruiz]

On Sat, Nov 07 2020, Eric Abrahamsen wrote:

> It's something that's come up on gnus.general several times over the
> years, but I think no one's ever sat down and tried to figure out
> exactly how/if it's supposed to work. Thinking about it logically, I
> just don't see how it could. On my filesystem (in the maildir maintained
> by Dovecot), your message has the filename:
>
> 1604808232.M340024P65055.slip,S=3293,W=3369:2,a
>
> And that's how notmuch will return it. If I search for the same message
> via Dovecot's internal search, the UUID is 1116. Is there anything to go
> from one to the other?

Not to my knowledge.  What you say makes sense and i must be
misremembering... i have a helper function that knows how to find in
gnus a message shown by notmuch, and i use that sometimes to search
using notmuch (counsel-notmuch, actually) while keeping gnus as my main
interface.

> In a few days I can try to set this up and see if there's a way to make
> it work.

That'd be really great.  I'm unfortunately not familiar with
nnir/search, or i'd would have dived a bit into it by now, but i'd be
happy to try and test whatever you come up with.

That said, it's possible to tell notmuch to return message ids instead
of filenames.  For instance, one of the results of the query

     notmuch search --format=sexp messages from:Eric to:jao

looks like:

      ((:thread "000000000002e056" :timestamp 1604811194 :date_relative
      "Today 04:53" :matched 2 :total 5 :authors "Eric Abrahamsen| Jose
      A. Ortega Ruiz, help-debbugs@gnu.org" :subject "bug#44509:
      28.0.50; Error querying with new gnus-search and notmuch" :query
      ("id:87361kd1sg.fsf@ericabrahamsen.net
      id:874km0bgat.fsf@ericabrahamsen.net"
      "id:87imag7kkh.fsf@gnus.jao.io
      id:handler.44509.B.160479992727359.ack@debbugs.gnu.org
      id:87v9eg7ec3.fsf@gnus.jao.io") :tags ("inbox" "replied"
      "unread"))

and those values for :query seem to be the ids of this thread.
notmuch-search.el has actually a function notmuch-query-get-message-ids
that directly returns them, and also this one:

  (defun notmuch-query-get-threads (search-terms)
    "Return a list of threads of messages matching SEARCH-TERMS.

where possibly that list contains info from both the path and message
id.

> (BTW have you turned on full text search in your local Dovecot? That
> makes imap searches "fast enough" for me, though I doubt they're as
> fast as notmuch.)

No... i once read that the fastest way was via xapian (if i'm recalling
correctly), but then i saw it's a java thing and lost steam :)

Another reason is that notmuch knows how to index usenet messages just
fine, and i use leafnode to fetch my news to a local server, so i can
play the trick of for instance symlinking leafnode's gwene/gmane message
folders in my ~/var/mail and have a single index for all my mail and
news.  The same function that brings me from notmuch message buffers to
gnus nnimap groups nows how to bring me to nntp groups to.

Cheers,
jao
--
I don't necessarily agree with everything I say.
 -Marshall McLuhan (1911-1980)

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Jose A. Ortega Ruiz]

Hi again, Eric.

I was looking at the notmuch engine code and wondering why it wasn't
working for me in a leafnode directory, which puts its messages in a
nnmail-compatible format (as far as i can tell).  And i discovered what
looks like a possible bug in gnus-search-indexed (or a misunderstanding
on my side).  Concretely, on line 1363 of gnus-search, when implementing
gnus-search-indexed-parsed output, we're constructing a groups regexp
that looks like:

	(group-regexp (when groups
			(regexp-opt
			 (mapcar
			  (lambda (x) (gnus-group-real-name x))
			  groups))))

and then matching the returned list of files on that regexp (line 1377):

       (string-match-p group-regexp f-name)))

But gnus-group-real-name is giving me back a dot-separated path for
nested folders (e.g. gmane.bugs.gnus), while the file paths in the
results are using slashes (gmane/bugs/gnus/234 etc.), so the match
test always fails and no results are returned.

If i simply redefine group-regexp using:

    (replace-regexp-in-string "\\." "/" (gnus-group-real-name x)))

searches work fine for me.  I can do that with an around advice, so that
my tricks work in this case, but i was wondering if it's an oversight.

On other news, i was trying to find a way in Gnus to go from Message-ID
to article no. for IMAP groups or nnmaildirs (which would make using
notmuch with dovecot really trivial), but without luck: anyone knows of
an easy way?

Cheers,
jao
-- 
As I grow to understand life less and less, I learn to live it more and
more.
 -Jules Renard, writer (1864-1910)

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Eric Abrahamsen]

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> Hi again, Eric.
>
> I was looking at the notmuch engine code and wondering why it wasn't
> working for me in a leafnode directory, which puts its messages in a
> nnmail-compatible format (as far as i can tell).  And i discovered what
> looks like a possible bug in gnus-search-indexed (or a misunderstanding
> on my side).  Concretely, on line 1363 of gnus-search, when implementing
> gnus-search-indexed-parsed output, we're constructing a groups regexp
> that looks like:
>
> 	(group-regexp (when groups
> 			(regexp-opt
> 			 (mapcar
> 			  (lambda (x) (gnus-group-real-name x))
> 			  groups))))
>
> and then matching the returned list of files on that regexp (line 1377):
>
>        (string-match-p group-regexp f-name)))
>
> But gnus-group-real-name is giving me back a dot-separated path for
> nested folders (e.g. gmane.bugs.gnus), while the file paths in the
> results are using slashes (gmane/bugs/gnus/234 etc.), so the match
> test always fails and no results are returned.
>
> If i simply redefine group-regexp using:
>
>     (replace-regexp-in-string "\\." "/" (gnus-group-real-name x)))

This code that comes straight from the old nnir.el, and has always
looked a bit fragile to me. The problem is that it really just expects
each message to be in a regular file, with groups as folders.

So you're using notmuch as a search engine for a local leafnode nntp
server, and indexing its message store directly?

Is there any leafnode setting that could influence how it stores its
messages? Can it be convinced to store them in hierarchical folders?

I suppose I could change the group-regexp to munge periods, but that
could cause breakage in other cases, and I would be hesitant to do that.

Otherwise, all the indexed search engines have a
`gnus-search-indexed-extract' method that's used to actually return the
file name from the results buffer. Each one's got something slightly
different. It would be very easy to create a new notmuch search engine
subclass that only overrides this method. To wit:

(defclass gnus-search-leafnode-notmuch (gnus-search-notmuch))

(cl-defmethod gnus-search-indexed-extract ((_engine gnus-search-leafnode-notmuch))
  (let ((results-string (buffer-substring-no-properties
			 (line-beginning-position)
			 (line-end-position))))
    (prog1
	(list (replace-regexp-in-string
	       "periods" "forward slashes" results-string)
	      100)
      (forward-line))))

The `replace-regexp-in-string' changes the periods into forward
slashes. Then in your Gnus config:

'(nntp "your-local-leafnode"
   (gnus-search-engine gnus-search-leafnode-notmuch))

This is what I would do (I haven't tested the above, it might require
some tweaking). It takes advantage of all the benefits of the
generic-method approach, and lets you change behavior without messing
with the rest of the gnus-search code. You might also find it helpful to
tweak a few other slots or methods for this engine in particular.

> On other news, i was trying to find a way in Gnus to go from Message-ID
> to article no. for IMAP groups or nnmaildirs (which would make using
> notmuch with dovecot really trivial), but without luck: anyone knows of
> an easy way?

Come to think of it, nnimap can already accept article numbers
as message-ids. So if notmuch returns its results as message-ids, it
should work transparently.

The problem is that while the mechanism is there, it works by searching
each message-id and getting the proper article number that way. My guess
is that you'd be negating most of the speed advantage of using notmuch
like this, and you'd still be better off using dovecot's own full-text
indexing. You don't have to use xapian!

https://doc.dovecot.org/configuration_manual/fts/

BUT, if you really wanted to do this, it would be relatively easy to
check for "--output=messages" in 'switches, and use that instead of
"--output=files".

Eric

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:
>
>> Hi again, Eric.
>>
>> I was looking at the notmuch engine code and wondering why it wasn't
>> working for me in a leafnode directory, which puts its messages in a
>> nnmail-compatible format (as far as i can tell).  And i discovered what
>> looks like a possible bug in gnus-search-indexed (or a misunderstanding
>> on my side).  Concretely, on line 1363 of gnus-search, when implementing
>> gnus-search-indexed-parsed output, we're constructing a groups regexp
>> that looks like:
>>
>> 	(group-regexp (when groups
>> 			(regexp-opt
>> 			 (mapcar
>> 			  (lambda (x) (gnus-group-real-name x))
>> 			  groups))))
>>
>> and then matching the returned list of files on that regexp (line 1377):
>>
>>        (string-match-p group-regexp f-name)))
>>
>> But gnus-group-real-name is giving me back a dot-separated path for
>> nested folders (e.g. gmane.bugs.gnus), while the file paths in the
>> results are using slashes (gmane/bugs/gnus/234 etc.), so the match
>> test always fails and no results are returned.
>>
>> If i simply redefine group-regexp using:
>>
>>     (replace-regexp-in-string "\\." "/" (gnus-group-real-name x)))
>
> This code that comes straight from the old nnir.el, and has always
> looked a bit fragile to me. The problem is that it really just expects
> each message to be in a regular file, with groups as folders.
>
> So you're using notmuch as a search engine for a local leafnode nntp
> server, and indexing its message store directly?
>
> Is there any leafnode setting that could influence how it stores its
> messages? Can it be convinced to store them in hierarchical folders?
>
> I suppose I could change the group-regexp to munge periods, but that
> could cause breakage in other cases, and I would be hesitant to do that.
>
> Otherwise, all the indexed search engines have a
> `gnus-search-indexed-extract' method that's used to actually return the
> file name from the results buffer. Each one's got something slightly
> different. It would be very easy to create a new notmuch search engine
> subclass that only overrides this method. To wit:
>
> (defclass gnus-search-leafnode-notmuch (gnus-search-notmuch))

Sorry, this would need to be:

(defclass gnus-search-leafnode-notmuch (gnus-search-notmuch)
  nil)

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[jao]

On Wed, Nov 11 2020, Eric Abrahamsen wrote:

[...]

> This code that comes straight from the old nnir.el, and has always
> looked a bit fragile to me. The problem is that it really just expects
> each message to be in a regular file, with groups as folders.
>
> So you're using notmuch as a search engine for a local leafnode nntp
> server, and indexing its message store directly?

Yes.

> Is there any leafnode setting that could influence how it stores its
> messages? Can it be convinced to store them in hierarchical folders?

It is exactly what it is doing.  Leafnode messages in the nntp group
gmane.emacs.bugs are stored in /prefix/gmane/emac/bugs/<article no.>
exactly as nnmail would do.  And the result of the search by notmuch is
displaying those paths correctly.  The problem is that, in the code i
mentioned, when that output is processed, the code is expecting to see
in the results list gmane.emacs.bugs instead of pathnames.  So i don't
understand how that code has ever worked, even on the old nnir.

> I suppose I could change the group-regexp to munge periods, but that
> could cause breakage in other cases, and I would be hesitant to do that.

In may case, the problem is the other way around: the regexp should
accept /, but is only accepting periods (because it's constructed
directly from the group name, which, for nntp at least, contains
periods, not slashes).

> Otherwise, all the indexed search engines have a
> `gnus-search-indexed-extract' method that's used to actually return the
> file name from the results buffer.

See above.  It's really very easy for me to fix this with an around
method like this one:

     (cl-defmethod gnus-search-indexed-parse-output :around ((e gnus-search-notmuch) s q groups)
        (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
                          groups)))
          (cl-call-next-method e s q gs)))

because the problem in my case is the group name, not the search
results. Note that, for cases (if any) where the group names already
look like "gmane/emacs/bug" (which i suspect is what nnml and nnmaildir
are doing and that's why it works), that's a no-op.

So yes, for me it's a solved problem, even without defining a new
engine.

>> On other news, i was trying to find a way in Gnus to go from Message-ID
>> to article no. for IMAP groups or nnmaildirs (which would make using
>> notmuch with dovecot really trivial), but without luck: anyone knows of
>> an easy way?
>
> Come to think of it, nnimap can already accept article numbers
> as message-ids. So if notmuch returns its results as message-ids, it
> should work transparently.

Yes, i thought that at first.  i got stuck when i discovered that
nnselect apparently only accepts article numbers to build its groups.  i
guess one could extend nnselect to also accept message ids, and that
that would be useful also in other contexts.

> The problem is that while the mechanism is there, it works by searching
> each message-id and getting the proper article number that way. My guess
> is that you'd be negating most of the speed advantage of using notmuch
> like this, and you'd still be better off using dovecot's own full-text
> indexing. You don't have to use xapian!
>
> https://doc.dovecot.org/configuration_manual/fts/

Ah, thanks for that, i'll definitely take a look.  The nice thing about
using notmuch would be that is one less moving piece to maintain, and
that then one could move very, very transparently between notmuch's
emacs interface and gnus, using the same message files and indexes.

> BUT, if you really wanted to do this, it would be relatively easy to
> check for "--output=messages" in 'switches, and use that instead of
> "--output=files".

Yes, it's trivial to get a vector of results of the form 
[group message-id score] ... a full engine like that is just a few
lines:

    (require 'notmuch-query)

    (defclass jao-gnus-notmuch (gnus-search-notmuch) ())

    (cl-defmethod gnus-search-run-search ((engine jao-gnus-notmuch) srv query groups)
      (let ((qstring (gnus-search-make-query-string engine query))
            (res))
        (dolist (group groups)
          (let* ((folder (format "folder:%s"
                                 (replace-regexp-in-string "\\." "/" group)))
                 (ids (notmuch-query-get-message-ids qstring folder)))
            (dolist (id ids)
              (push (vector (gnus-group-full-name group srv) id 100) res))))
        res))

That's all (well, obviating thread seaches, but those are not hard
either).  Except that it doesn't work because nnselect wants 
[group article-no score].  If you tell me how to obtain article-no from
message-id, i'm done :)

Thanks,
jao
-- 
I don't necessarily agree with everything I say.
 -Marshall McLuhan (1911-1980)

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Eric Abrahamsen]

jao <jao@gnu.org> writes:

> On Wed, Nov 11 2020, Eric Abrahamsen wrote:
>
> [...]
>
>> This code that comes straight from the old nnir.el, and has always
>> looked a bit fragile to me. The problem is that it really just expects
>> each message to be in a regular file, with groups as folders.
>>
>> So you're using notmuch as a search engine for a local leafnode nntp
>> server, and indexing its message store directly?
>
> Yes.
>
>> Is there any leafnode setting that could influence how it stores its
>> messages? Can it be convinced to store them in hierarchical folders?
>
> It is exactly what it is doing.  Leafnode messages in the nntp group
> gmane.emacs.bugs are stored in /prefix/gmane/emac/bugs/<article no.>
> exactly as nnmail would do.  And the result of the search by notmuch is
> displaying those paths correctly.  The problem is that, in the code i
> mentioned, when that output is processed, the code is expecting to see
> in the results list gmane.emacs.bugs instead of pathnames.  So i don't
> understand how that code has ever worked, even on the old nnir.

I see, I had your problem backwards. My guess is that no one has tried
to do this before, so the issue has never come up.

BTW, while there's no longer a "gmane" search engine, it would be nice
to provide an engine for searching other nntp servers. In your case
you're running it locally, but do you know if a remote leafnode server
handles any sort of search functionality? (I might as well google this
myself.)

>> I suppose I could change the group-regexp to munge periods, but that
>> could cause breakage in other cases, and I would be hesitant to do that.
>
> In may case, the problem is the other way around: the regexp should
> accept /, but is only accepting periods (because it's constructed
> directly from the group name, which, for nntp at least, contains
> periods, not slashes).
>
>> Otherwise, all the indexed search engines have a
>> `gnus-search-indexed-extract' method that's used to actually return the
>> file name from the results buffer.
>
> See above.  It's really very easy for me to fix this with an around
> method like this one:
>
>      (cl-defmethod gnus-search-indexed-parse-output :around ((e gnus-search-notmuch) s q groups)
>         (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
>                           groups)))
>           (cl-call-next-method e s q gs)))
>
> because the problem in my case is the group name, not the search
> results. Note that, for cases (if any) where the group names already
> look like "gmane/emacs/bug" (which i suspect is what nnml and nnmaildir
> are doing and that's why it works), that's a no-op.
>
> So yes, for me it's a solved problem, even without defining a new
> engine.

Okay! Glad that's sorted. You might still want a new engine, if you have
other notmuch engines that shouldn't inherit this behavior.

>>> On other news, i was trying to find a way in Gnus to go from Message-ID
>>> to article no. for IMAP groups or nnmaildirs (which would make using
>>> notmuch with dovecot really trivial), but without luck: anyone knows of
>>> an easy way?
>>
>> Come to think of it, nnimap can already accept article numbers
>> as message-ids. So if notmuch returns its results as message-ids, it
>> should work transparently.
>
> Yes, i thought that at first.  i got stuck when i discovered that
> nnselect apparently only accepts article numbers to build its groups.  i
> guess one could extend nnselect to also accept message ids, and that
> that would be useful also in other contexts.

It looks like nnselect can accept message-ids when requesting individual
articles, but not when categorizing groups/articles, nor when requesting
headers. My guess is that might be quite a bit of work to implement,
particularly since the id->number routine will be different for each
backend. You can see that happening in `nnselect-request-article'.

>> The problem is that while the mechanism is there, it works by searching
>> each message-id and getting the proper article number that way. My guess
>> is that you'd be negating most of the speed advantage of using notmuch
>> like this, and you'd still be better off using dovecot's own full-text
>> indexing. You don't have to use xapian!
>>
>> https://doc.dovecot.org/configuration_manual/fts/
>
> Ah, thanks for that, i'll definitely take a look.  The nice thing about
> using notmuch would be that is one less moving piece to maintain, and
> that then one could move very, very transparently between notmuch's
> emacs interface and gnus, using the same message files and indexes.
>
>> BUT, if you really wanted to do this, it would be relatively easy to
>> check for "--output=messages" in 'switches, and use that instead of
>> "--output=files".
>
> Yes, it's trivial to get a vector of results of the form 
> [group message-id score] ... a full engine like that is just a few
> lines:
>
>     (require 'notmuch-query)
>
>     (defclass jao-gnus-notmuch (gnus-search-notmuch) ())
>
>     (cl-defmethod gnus-search-run-search ((engine jao-gnus-notmuch) srv query groups)
>       (let ((qstring (gnus-search-make-query-string engine query))
>             (res))
>         (dolist (group groups)
>           (let* ((folder (format "folder:%s"
>                                  (replace-regexp-in-string "\\." "/" group)))
>                  (ids (notmuch-query-get-message-ids qstring folder)))
>             (dolist (id ids)
>               (push (vector (gnus-group-full-name group srv) id 100) res))))
>         res))

Hmm, I guess it wouldn't be useful for me to implement this in the
general case, since the ids aren't going to be accepted by nnselect.

> That's all (well, obviating thread seaches, but those are not hard
> either).  Except that it doesn't work because nnselect wants 
> [group article-no score].  If you tell me how to obtain article-no from
> message-id, i'm done :)

For nnimap, it's `nnimap-find-article-by-message-id'.

Eric

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[jao]

On Wed, Nov 11 2020, Eric Abrahamsen wrote:

[...]

> I see, I had your problem backwards. My guess is that no one has tried
> to do this before, so the issue has never come up.
>
> BTW, while there's no longer a "gmane" search engine, it would be nice
> to provide an engine for searching other nntp servers. In your case
> you're running it locally, but do you know if a remote leafnode server
> handles any sort of search functionality? (I might as well google this
> myself.)

No, i don't think leafnode does anything of the sort (it's old, from
simpler times, and kind of opinionated on a minimal feature set).

[...]

> Okay! Glad that's sorted. You might still want a new engine, if you have
> other notmuch engines that shouldn't inherit this behavior.

Yes, and it's just adding a single line to the above code, so pretty cheap.


[...]

> Hmm, I guess it wouldn't be useful for me to implement this in the
> general case, since the ids aren't going to be accepted by nnselect.
>
>> That's all (well, obviating thread seaches, but those are not hard
>> either).  Except that it doesn't work because nnselect wants
>> [group article-no score].  If you tell me how to obtain article-no from
>> message-id, i'm done :)
>
> For nnimap, it's `nnimap-find-article-by-message-id'.

Well, that function is contacting back the imap server and asking for a
(simple) search, so it's going to be very roundabout even if it finally
works (i cannot manage to make it work for me, although i'm surely doing
something wrong and its implementation not trivial).  

fst-xapian doesn't look that bad, after all (actually, i was confusing
it with solr or elastic which need independent servers and are too
heavyweight for me.) :)

Thanks,
jao

bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Eric Abrahamsen]

jao <jao@gnu.org> writes:

> On Wed, Nov 11 2020, Eric Abrahamsen wrote:
>
> [...]
>
>> I see, I had your problem backwards. My guess is that no one has tried
>> to do this before, so the issue has never come up.
>>
>> BTW, while there's no longer a "gmane" search engine, it would be nice
>> to provide an engine for searching other nntp servers. In your case
>> you're running it locally, but do you know if a remote leafnode server
>> handles any sort of search functionality? (I might as well google this
>> myself.)
>
> No, i don't think leafnode does anything of the sort (it's old, from
> simpler times, and kind of opinionated on a minimal feature set).

That's what my googling indicated, too :)


[...]

>> For nnimap, it's `nnimap-find-article-by-message-id'.
>
> Well, that function is contacting back the imap server and asking for a
> (simple) search, so it's going to be very roundabout even if it finally
> works (i cannot manage to make it work for me, although i'm surely doing
> something wrong and its implementation not trivial).  

Right, that's what I meant by the search probably negating any speedups
you'd get from using notmuch.

> fst-xapian doesn't look that bad, after all (actually, i was confusing
> it with solr or elastic which need independent servers and are too
> heavyweight for me.) :)

I think it's great! You make sure a package is installed, then add a
single line to your dovecot config, then never think about it again.

Eric