bug#35383: 27.0.50; Complete process of decoding Gnus group names

[Eric Abrahamsen <eric@ericabrahamsen.net>]

[-- Attachment #1: Type: text/plain, Size: 2939 bytes --]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Mon 22 Apr 2019, Eric Abrahamsen wrote:
>
>> This is part two and the completion of bug#33653, which changed Gnus's
>> obarrays into hash tables, and group names from symbols to (encoded)
>> strings. The commits in the recently-pushed "scratch/gnus-decoded"
>> branch change group names to decoded strings.
>>
>> Bug 33653 was a mess for a few reasons: I partitioned the changes
>> poorly, didn't call for testers, and it turned out that I was locally
>> testing a mismash of that change plus a couple changes included in this
>> branch, which hid some bugs.
>>
>> This time around I'll keep it cleaner: I'll locally test only this
>> change in isolation, I'm writing a semi-interactive test suite for Gnus,
>> and in a few weeks I'll rope in three or four users to road-test the
>> changes. The upside is that, once these changes are stabilized and put
>> in, it will eliminate a whole class of potential bugs.
>>
>> In the meantime I'm hanging this here as a placeholder. If any brave
>> soul does decide to give it a test-run in the meantime, back up your
>> .newsrc.eld file first!
>
> There don;t seem to bnbe any problems with the .nersrc.eld, as I have
> used the same file in emacs-26, master and scratch/gnus-decoded branches
> without obvious problems.
>
> There is a problem in the server buffer in the scratch/gnus-decoded buffer
> with group names. In my setup:
> 1) Start gnus, and open the server buffer
> 2) Select the feedbase server. This is configured as:
>
>   (gnus-secondary-select-methods
>    '((nntp "feedbase"
>            (nntp-open-connection-function nntp-open-tls-stream)
>            (nntp-port-number 563)
>            (nntp-address "feedbase.org"))
>      ...
>
> 3) Note that some group names are not decoded correctly in the server
> buffer. The group names are decoded properly in the article buffer.
>
> For example, (showing only names with problems):
>
> server buffer (master branch):
>
> K    140: feedbase.blog.københavnsskiklub
> K    208: feedbase.news.brk.høringer
> K   2017: feedbase.news.dr.hovedstadsområdet
>
>
> server buffer (scratch/gnus-decoded branch):
>
> K    140: feedbase.blog.k\303\270benhavnsskiklub
> K    208: feedbase.news.brk.h\303\270ringer
> K   2017: feedbase.news.dr.hovedstadsomr\303\245det
>
> The raw bytes (shown as escapes above) match the UTF-8 for the expected
> characters, so the decoding is not quite right.

Aha! Good catch. The attached patch fixes things for me. At first I was
a little hesitant to assume all group names are decodable as
'utf-8-emacs, but then I figured that's all that
`gnus-group-decoded-name' does, and that's worked for years, so I guess
this is safe.

And thanks for the nntp server with non-ascii group names! I had looked
for one of those with no success, this will be very useful for testing.

Eric



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus-browse-srv-decoded.diff --]
[-- Type: text/x-patch, Size: 1655 bytes --]

diff --git a/lisp/gnus/gnus-srvr.el b/lisp/gnus/gnus-srvr.el
index 2328f1da16..da110c8bbf 100644
--- a/lisp/gnus/gnus-srvr.el
+++ b/lisp/gnus/gnus-srvr.el
@@ -784,11 +784,13 @@ gnus-browse-foreign-server
 	      (while (not (eobp))
 		(ignore-errors
 		  (push (cons
-			 (buffer-substring
-			  (point)
-			  (progn
-			    (skip-chars-forward "^ \t")
-			    (point)))
+			 (decode-coding-string
+			  (buffer-substring
+			   (point)
+			   (progn
+			     (skip-chars-forward "^ \t")
+			     (point)))
+			  'utf-8-emacs)
 			 (let ((last (read cur)))
 			   (cons (read cur) last)))
 			groups))
@@ -796,18 +798,20 @@ gnus-browse-foreign-server
 	    (while (not (eobp))
 	      (ignore-errors
 		(push (cons
-		       (if (eq (char-after) ?\")
-			   (read cur)
-			 (let ((p (point)) (name ""))
-			   (skip-chars-forward "^ \t\\\\")
-			   (setq name (buffer-substring p (point)))
-			   (while (eq (char-after) ?\\)
-			     (setq p (1+ (point)))
-			     (forward-char 2)
-			     (skip-chars-forward "^ \t\\\\")
-			     (setq name (concat name (buffer-substring
-						      p (point)))))
-			   name))
+		       (decode-coding-string
+			(if (eq (char-after) ?\")
+			    (read cur)
+			  (let ((p (point)) (name ""))
+			    (skip-chars-forward "^ \t\\\\")
+			    (setq name (buffer-substring p (point)))
+			    (while (eq (char-after) ?\\)
+			      (setq p (1+ (point)))
+			      (forward-char 2)
+			      (skip-chars-forward "^ \t\\\\")
+			      (setq name (concat name (buffer-substring
+						       p (point)))))
+			    name))
+			'utf-8-emacs)
 		       (let ((last (read cur)))
 			 (cons (read cur) last)))
 		      groups))