nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[postmaster]

It seems that nnir has problems with mailboxes which have umlauts in
there names.

For my search tests I created these two mailboxes on my dovecot IMAP-server:
"dick" and "dünn".
When I tried to search a string in these boxes (with G G) I got this
"result" for the mailbox "dünn":
gnus-select-newsgroup: Couldn’t request group nnir:nnir-riek1dwxvv5.fsf: Search produced empty results.
(The same search in the mailbox "dick" works fine.)

I switched on the debug-on-entry for the function nnir-run-imap and got
this result (at the mailbox "dünn"):

Debugger entered--entering a function:
* nnir-run-imap(((query . "dick")) "nnimap:buckthorn's IMAP server" ("d\303\274nn"))
#f(compiled-function (x) #<bytecode 0x1aaf37d>)(("nnimap:buckthorn's IMAP server" ("d\303\274nn")))
mapcar(#f(compiled-function (x) #<bytecode 0x1aaf37d>) (("nnimap:buckthorn's IMAP server" ("d\303\274nn"))))
nnir-run-query(((nnir-query-spec (query . "dick")) (nnir-group-spec ("nnimap:buckthorn's IMAP server" ("d\303\274nn")))))
nnir-request-group("nnir-rieftokxud4.fsf" "nnir-ephemeral" t ("nnir:nnir-rieftokxud4.fsf" 3 nil nil (nnir "nnir-ephemeral" (nnir-address "nnir")) ((quit-config #<killed buffer> . group) (nnir-specs (nnir-query-spec (query . "dick")) (nnir-group-spec ("nnimap:buckthorn's IMAP server" ("d\303\274nn")))) (nnir-artlist))))
gnus-request-group("nnir:nnir-rieftokxud4.fsf" t nil ("nnir:nnir-rieftokxud4.fsf" 3 nil nil (nnir "nnir-ephemeral" (nnir-address "nnir")) ((quit-config #<killed buffer> . group) (nnir-specs (nnir-query-spec (query . "dick")) (nnir-group-spec ("nnimap:buckthorn's IMAP server" ("d\303\274nn")))) (nnir-artlist))))
gnus-select-newsgroup("nnir:nnir-rieftokxud4.fsf" t nil)
gnus-summary-read-group-1("nnir:nnir-rieftokxud4.fsf" t t nil nil nil)
gnus-summary-read-group("nnir:nnir-rieftokxud4.fsf" t t nil nil nil nil)
gnus-group-read-group(t t "nnir:nnir-rieftokxud4.fsf" nil)
gnus-group-read-ephemeral-group("nnir-rieftokxud4.fsf" (nnir "nnir") nil nil nil nil ((nnir-specs (nnir-query-spec (query . "dick")) (nnir-group-spec ("nnimap:buckthorn's IMAP server" ("d\303\274nn")))) (nnir-artlist)))
gnus-group-make-nnir-group(nil)
funcall-interactively(gnus-group-make-nnir-group nil)
call-interactively(gnus-group-make-nnir-group nil nil)
command-execute(gnus-group-make-nnir-group)

For me it seems that this is a problem in nnir because doveadm has no
problems with these searches:
doveadm search mailbox dick text "dick"
18b74815af68fb5c540900001b5d164c 1
doveadm search mailbox dünn text "dick"
18b74815af68fb5c540900001b5d164c 1

Am I right that this is a bug?

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[Robert Pluim]

>>>>> On Tue, 11 Jun 2019 17:14:09 +0200 (CEST), postmaster@faulbaum.in-berlin.de said:

    postmaster> It seems that nnir has problems with mailboxes which have umlauts in
    postmaster> there names.

    postmaster> Am I right that this is a bug?

Yes. You should do 'M-x report-emacs-bug' which will gather
information about your emacs, and create a bug report containing this
info.

Robert

PS There have been some changes in the latest master branch of Emacs
to better handle non-ascii group names, that might function better
(but you didnʼt tell us which version of emacs youʼre using :-) )

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[Eric Abrahamsen]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 11 Jun 2019 17:14:09 +0200 (CEST), postmaster@faulbaum.in-berlin.de said:
>
>     postmaster> It seems that nnir has problems with mailboxes which have umlauts in
>     postmaster> there names.
>
>     postmaster> Am I right that this is a bug?
>
> Yes. You should do 'M-x report-emacs-bug' which will gather
> information about your emacs, and create a bug report containing this
> info.
>
> Robert
>
> PS There have been some changes in the latest master branch of Emacs
> to better handle non-ascii group names, that might function better
> (but you didnʼt tell us which version of emacs youʼre using :-) )

Right, this sounds like the stuff I've been working on/breaking. Do tell
us your Emacs version, and in the meantime I'll check the relevant areas
of the code.

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>>>>>>> On Tue, 11 Jun 2019 17:14:09 +0200 (CEST), postmaster@faulbaum.in-berlin.de said:
>>
>>     postmaster> It seems that nnir has problems with mailboxes which have umlauts in
>>     postmaster> there names.
>>
>>     postmaster> Am I right that this is a bug?
>>
>> Yes. You should do 'M-x report-emacs-bug' which will gather
>> information about your emacs, and create a bug report containing this
>> info.
>>
>> Robert
>>
>> PS There have been some changes in the latest master branch of Emacs
>> to better handle non-ascii group names, that might function better
>> (but you didnʼt tell us which version of emacs youʼre using :-) )
>
> Right, this sounds like the stuff I've been working on/breaking. Do tell
> us your Emacs version, and in the meantime I'll check the relevant areas
> of the code.

I tested this with Emacs 26.1, and it looks like the same bug is present
there -- I'm surprised no one's run into it before. I'll look into a
fix, either as part of scratch/gnus-decoded or (probably better) its
own patch.

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[postmaster]

> On June 11, 2019 8:45 PM Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> 
>  
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> 
> > Robert Pluim <rpluim@gmail.com> writes:
> >
> >>>>>>> On Tue, 11 Jun 2019 17:14:09 +0200 (CEST), postmaster@faulbaum.in-berlin.de said:
> >>
> >>     postmaster> It seems that nnir has problems with mailboxes which have umlauts in
> >>     postmaster> there names.
> >>
> >>     postmaster> Am I right that this is a bug?
> >>
> >> Yes. You should do 'M-x report-emacs-bug' which will gather
> >> information about your emacs, and create a bug report containing this
> >> info.
> >>
> >> Robert
> >>
> >> PS There have been some changes in the latest master branch of Emacs
> >> to better handle non-ascii group names, that might function better
> >> (but you didnʼt tell us which version of emacs youʼre using :-) )
> >
> > Right, this sounds like the stuff I've been working on/breaking. Do tell
> > us your Emacs version, and in the meantime I'll check the relevant areas
> > of the code.
> 
> I tested this with Emacs 26.1, and it looks like the same bug is present
> there -- I'm surprised no one's run into it before. I'll look into a
> fix, either as part of scratch/gnus-decoded or (probably better) its
> own patch.

I use the last stable version of emacs (26.2) and added the one line of the patch into the line 973 of nnir.el (byte-compiled the file) but this doesn't work for me. Don't know what I made wrong.-(

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[postmaster]

> I tested this with Emacs 26.1, and it looks like the same bug is present
> there -- I'm surprised no one's run into it before. I'll look into a
> fix, either as part of scratch/gnus-decoded or (probably better) its
> own patch.

I patched the nnir.el of emcas 26.2 with the last diff found in the mailing-list and 'diff -w nnir.el.orig nnir.el' shows not a big change:
970c970
<           #'(lambda (group)
---
>           (lambda (group)
972a973,974
> 		  (progn
> 		    (setq group (nnimap-decode-gnus-group group))
995c997
<                     (message "Searching %s...done" group))
---
> 		      (message "Searching %s...done" group)))

And the patch doesn't work.-(
I still get the error:
gnus-select-newsgroup: Couldn’t request group nnir:nnir-riev9xaq51j.fsf: Search produced empty results.

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[Eric Abrahamsen]

postmaster@faulbaum.in-berlin.de writes:

>> I tested this with Emacs 26.1, and it looks like the same bug is present
>> there -- I'm surprised no one's run into it before. I'll look into a
>> fix, either as part of scratch/gnus-decoded or (probably better) its
>> own patch.
>
> I patched the nnir.el of emcas 26.2 with the last diff found in the
> mailing-list and 'diff -w nnir.el.orig nnir.el' shows not a big
> change:
> 970c970
> <           #'(lambda (group)
> ---
>>           (lambda (group)
> 972a973,974
>> 		  (progn
>> 		    (setq group (nnimap-decode-gnus-group group))
> 995c997
> <                     (message "Searching %s...done" group))
> ---
>> 		      (message "Searching %s...done" group)))
>
> And the patch doesn't work.-(
> I still get the error:
> gnus-select-newsgroup: Couldn’t request group
> nnir:nnir-riev9xaq51j.fsf: Search produced empty results.

Oh that's interesting, it worked for me after applying that patch. In
the message above ("Searching %s...") is the group name correctly
displayed? If you're familiar with edebug, can you step through
`nnir-run-imap' and see if anything looks funny?

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>>>>>>> On Tue, 11 Jun 2019 17:14:09 +0200 (CEST), postmaster@faulbaum.in-berlin.de said:
>>
>>     postmaster> It seems that nnir has problems with mailboxes which have umlauts in
>>     postmaster> there names.
>>
>>     postmaster> Am I right that this is a bug?
>>
>> Yes. You should do 'M-x report-emacs-bug' which will gather
>> information about your emacs, and create a bug report containing this
>> info.
>>
>> Robert
>>
>> PS There have been some changes in the latest master branch of Emacs
>> to better handle non-ascii group names, that might function better
>> (but you didnʼt tell us which version of emacs youʼre using :-) )
>
> Right, this sounds like the stuff I've been working on/breaking. Do tell
> us your Emacs version, and in the meantime I'll check the relevant areas
> of the code.

Looks like the attached diff is all we need. Please give this a whirl,
and I'll open a bug report for it. I think there's just better than a
snowball's chance in hell we could squeeze this into the next Emacs 26
bugfix release.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnir-imap-fix.diff --]
[-- Type: text/x-patch, Size: 479 bytes --]

diff --git a/lisp/gnus/nnir.el b/lisp/gnus/nnir.el
index 9d59a4db0d..8fe7ac8cd0 100644
--- a/lisp/gnus/nnir.el
+++ b/lisp/gnus/nnir.el
@@ -991,6 +991,7 @@ nnir-run-imap
           #'(lambda (group)
             (let (artlist)
               (condition-case ()
+		  (setq group (nnimap-decode-gnus-group group))
                   (when (nnimap-change-group
                          (gnus-group-short-name group) server)
                     (with-current-buffer (nnimap-buffer)

Re: nnir problem with mailboxes with umlauts (oh sorry, the last mail was HTML)

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Robert Pluim <rpluim@gmail.com> writes:
>>
>>>>>>>> On Tue, 11 Jun 2019 17:14:09 +0200 (CEST), postmaster@faulbaum.in-berlin.de said:
>>>
>>>     postmaster> It seems that nnir has problems with mailboxes which have umlauts in
>>>     postmaster> there names.
>>>
>>>     postmaster> Am I right that this is a bug?
>>>
>>> Yes. You should do 'M-x report-emacs-bug' which will gather
>>> information about your emacs, and create a bug report containing this
>>> info.
>>>
>>> Robert
>>>
>>> PS There have been some changes in the latest master branch of Emacs
>>> to better handle non-ascii group names, that might function better
>>> (but you didnʼt tell us which version of emacs youʼre using :-) )
>>
>> Right, this sounds like the stuff I've been working on/breaking. Do tell
>> us your Emacs version, and in the meantime I'll check the relevant areas
>> of the code.
>
> Looks like the attached diff is all we need. Please give this a whirl,
> and I'll open a bug report for it. I think there's just better than a
> snowball's chance in hell we could squeeze this into the next Emacs 26
> bugfix release.

Whoops, I meant this one. I wonder why that worked...


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnir-imap-fix.diff --]
[-- Type: text/x-patch, Size: 3422 bytes --]

diff --git a/lisp/gnus/nnir.el b/lisp/gnus/nnir.el
index 9d59a4db0d..2fb64c3d33 100644
--- a/lisp/gnus/nnir.el
+++ b/lisp/gnus/nnir.el
@@ -977,7 +977,7 @@ nnir-run-imap
   (save-excursion
     (let ((qstring (cdr (assq 'query query)))
           (server (cadr (gnus-server-to-method srv)))
-;;          (defs (nth 2 (gnus-server-to-method srv)))
+	  ;;          (defs (nth 2 (gnus-server-to-method srv)))
           (criteria (or (cdr (assq 'criteria query))
                         (cdr (assoc nnir-imap-default-search-key
                                     nnir-imap-search-arguments))))
@@ -988,34 +988,36 @@ nnir-run-imap
        'vconcat
        (catch 'found
          (mapcar
-          #'(lambda (group)
+          (lambda (group)
             (let (artlist)
-              (condition-case ()
-                  (when (nnimap-change-group
-                         (gnus-group-short-name group) server)
-                    (with-current-buffer (nnimap-buffer)
-                      (message "Searching %s..." group)
-                      (let ((arts 0)
-                            (result (nnimap-command "UID SEARCH %s"
-                                                    (if (string= criteria "")
-                                                        qstring
-                                                      (nnir-imap-make-query
-                                                       criteria qstring)))))
-                        (mapc
-                         (lambda (artnum)
-                           (let ((artn (string-to-number artnum)))
-                             (when (> artn 0)
-                               (push (vector group artn 100)
-                                     artlist)
-                               (when (assq 'shortcut query)
-                                 (throw 'found (list artlist)))
-                               (setq arts (1+ arts)))))
-                         (and (car result)
-			      (cdr (assoc "SEARCH" (cdr result)))))
-                        (message "Searching %s... %d matches" group arts)))
-                    (message "Searching %s...done" group))
+	      (condition-case ()
+		  (progn
+		    (setq group (nnimap-decode-gnus-group group))
+                    (when (nnimap-change-group
+                           (gnus-group-short-name group) server)
+		      (with-current-buffer (nnimap-buffer)
+			(message "Searching %s..." group)
+			(let ((arts 0)
+			      (result (nnimap-command "UID SEARCH %s"
+						      (if (string= criteria "")
+                                                          qstring
+							(nnir-imap-make-query
+							 criteria qstring)))))
+                          (mapc
+                           (lambda (artnum)
+                             (let ((artn (string-to-number artnum)))
+			       (when (> artn 0)
+				 (push (vector group artn 100)
+				       artlist)
+				 (when (assq 'shortcut query)
+                                   (throw 'found (list artlist)))
+				 (setq arts (1+ arts)))))
+                           (and (car result)
+				(cdr (assoc "SEARCH" (cdr result)))))
+                          (message "Searching %s... %d matches" group arts)))
+		      (message "Searching %s...done" group)))
                 (quit nil))
-              (nreverse artlist)))
+	      (nreverse artlist)))
           groups))))))
 
 (defun nnir-imap-make-query (criteria qstring)