From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.bugs Subject: bug#38381: 27.0.50; Bump Gnus version to 5.14, properly encode group names on disk Date: Mon, 25 Nov 2019 15:56:55 -0800 Message-ID: <87blszy0fs.fsf@ericabrahamsen.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="219114"; mail-complaints-to="usenet@blaine.gmane.org" Cc: larsi@gnus.org To: 38381@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Nov 26 00:58:15 2019 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1iZOFC-000us4-5P for geb-bug-gnu-emacs@m.gmane.org; Tue, 26 Nov 2019 00:58:14 +0100 Original-Received: from localhost ([::1]:49224 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iZOFB-0003lx-2M for geb-bug-gnu-emacs@m.gmane.org; Mon, 25 Nov 2019 18:58:13 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:50322) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iZOF2-0003lr-RY for bug-gnu-emacs@gnu.org; Mon, 25 Nov 2019 18:58:06 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iZOF0-0008VS-IE for bug-gnu-emacs@gnu.org; Mon, 25 Nov 2019 18:58:04 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]:43994) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iZOF0-0008VN-DP for bug-gnu-emacs@gnu.org; Mon, 25 Nov 2019 18:58:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1iZOF0-0007vD-9E; Mon, 25 Nov 2019 18:58:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Eric Abrahamsen Original-Sender: "Debbugs-submit" Resent-CC: larsi@gnus.org, bug-gnu-emacs@gnu.org Resent-Date: Mon, 25 Nov 2019 23:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 38381 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org X-Debbugs-Original-Xcc: larsi@gnus.org Original-Received: via spool by submit@debbugs.gnu.org id=B.157472623230386 (code B ref -1); Mon, 25 Nov 2019 23:58:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 25 Nov 2019 23:57:12 +0000 Original-Received: from localhost ([127.0.0.1]:49966 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iZOEB-0007u1-9m for submit@debbugs.gnu.org; Mon, 25 Nov 2019 18:57:11 -0500 Original-Received: from lists.gnu.org ([209.51.188.17]:51449) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iZOE9-0007ts-8g for submit@debbugs.gnu.org; Mon, 25 Nov 2019 18:57:10 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:50263) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iZOE6-0003j9-Pg for bug-gnu-emacs@gnu.org; Mon, 25 Nov 2019 18:57:09 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iZOE4-0008CE-DY for bug-gnu-emacs@gnu.org; Mon, 25 Nov 2019 18:57:06 -0500 Original-Received: from ericabrahamsen.net ([52.70.2.18]:56120 helo=mail.ericabrahamsen.net) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iZOE4-000892-6g for bug-gnu-emacs@gnu.org; Mon, 25 Nov 2019 18:57:04 -0500 Original-Received: from localhost (unknown [205.175.106.123]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id EBA98FA0E4 for ; Mon, 25 Nov 2019 23:56:56 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:172403 Archived-At: --=-=-= Content-Type: text/plain As part of switching Gnus group names to fully-decoded strings, I left some code (really horrible code) in place to make sure that the group names stayed in their previous encoding in Gnus' save files, so that people running Emacs from git builds could switch back and forth without worrying about the save files becoming unreadable. Right now, there are several routines in place that decode non-ascii group names on file read, and re-encode them on file write. This only has an effect on non-ascii group names. Now Emacs 27.1 is on its way, and I would really like to make this change permanent. The attached patch moves all the ugly compatibility code into an upgrade routine as part of `gnus-convert-old-newsrc'. Users will be prompted once to upgrade, and then the various files will be fixed to use properly encoded group names, and you won't be able to go back. Again, this only affects users who had non-ascii group names to begin with. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-Encode-Gnus-group-names-properly-on-disk-bump-Gnus-v.patch >From 7a5194edff2d69cfcd8f5ee2bdec61bc12c1a11b Mon Sep 17 00:00:00 2001 From: Eric Abrahamsen Date: Sun, 24 Nov 2019 16:34:00 -0800 Subject: [PATCH] Encode Gnus group names properly on disk, bump Gnus version to 5.14 * lisp/gnus/gnus.el (gnus-version-number): Bump to 5.14 to trigger upgrade routines. * lisp/gnus/gnus-start.el (gnus-convert-old-newsrc): Check for new version. (gnus-convert-encoded-group-names, gnus-convert-encoded-category-group-names, gnus-convert-encoded-registry-group-names): Do a one-time conversion of encoded group names read from various persistence files. (gnus-read-newsrc-el-file, gnus-gnus-to-quick-newsrc-format): Stop doing the work here. * lisp/gnus/gnus-agent.el (gnus-category-read, gnus-category-write): Don't check group names. * lisp/gnus/gnus-registry.el (gnus-registry--munge-group-names): Remove function. (gnus-registry-save, gnus-registry-fixup-registry): Don't call it. --- lisp/gnus/gnus-agent.el | 46 ++++---------- lisp/gnus/gnus-registry.el | 50 +-------------- lisp/gnus/gnus-start.el | 124 +++++++++++++++++++++++++------------ lisp/gnus/gnus.el | 4 +- 4 files changed, 101 insertions(+), 123 deletions(-) diff --git a/lisp/gnus/gnus-agent.el b/lisp/gnus/gnus-agent.el index 1f25255278..768b3beaa6 100644 --- a/lisp/gnus/gnus-agent.el +++ b/lisp/gnus/gnus-agent.el @@ -2721,14 +2721,6 @@ gnus-category-read '(agent-predicate agent-score-file agent-groups)))) c) old-list))))))) - ;; Possibly decode group names. - (dolist (cat list) - (setf (alist-get 'agent-groups cat) - (mapcar (lambda (g) - (if (string-match-p "[^[:ascii:]]" g) - (decode-coding-string g 'utf-8-emacs) - g)) - (alist-get 'agent-groups cat)))) list) (list (gnus-agent-cat-make 'default 'short))))) @@ -2736,31 +2728,19 @@ gnus-category-write "Write the category alist." (setq gnus-category-predicate-cache nil gnus-category-group-cache nil) - ;; Temporarily encode non-ascii group names when saving to file, - ;; pending an upgrade of Gnus' file formats. - (let ((gnus-category-alist - (mapcar (lambda (cat) - (setf (alist-get 'agent-groups cat) - (mapcar (lambda (g) - (if (multibyte-string-p g) - (encode-coding-string g 'utf-8-emacs) - g)) - (alist-get 'agent-groups cat))) - cat) - (copy-tree gnus-category-alist)))) - (gnus-make-directory (nnheader-concat gnus-agent-directory "lib")) - (with-temp-file (nnheader-concat gnus-agent-directory "lib/categories") - ;; This prin1 is temporary. It exists so that people can revert - ;; to an earlier version of gnus-agent. - (prin1 (mapcar (lambda (c) - (list (car c) - (cdr (assoc 'agent-predicate c)) - (cdr (assoc 'agent-score-file c)) - (cdr (assoc 'agent-groups c)))) - gnus-category-alist) - (current-buffer)) - (newline) - (prin1 gnus-category-alist (current-buffer))))) + (gnus-make-directory (nnheader-concat gnus-agent-directory "lib")) + (with-temp-file (nnheader-concat gnus-agent-directory "lib/categories") + ;; This prin1 is temporary. It exists so that people can revert + ;; to an earlier version of gnus-agent. + (prin1 (mapcar (lambda (c) + (list (car c) + (cdr (assoc 'agent-predicate c)) + (cdr (assoc 'agent-score-file c)) + (cdr (assoc 'agent-groups c)))) + gnus-category-alist) + (current-buffer)) + (newline) + (prin1 gnus-category-alist (current-buffer)))) (defun gnus-category-edit-predicate (category) "Edit the predicate for CATEGORY." diff --git a/lisp/gnus/gnus-registry.el b/lisp/gnus/gnus-registry.el index e6fb382c2f..7691e3e916 100644 --- a/lisp/gnus/gnus-registry.el +++ b/lisp/gnus/gnus-registry.el @@ -266,47 +266,7 @@ gnus-registry-sort-by-creation-time ;; Remove this from the save routine (and fix it to only decode) at ;; next Gnus version bump. -(defun gnus-registry--munge-group-names (db &optional encode) - "Encode/decode group names in DB, before saving or after loading. -Encode names if ENCODE is non-nil, otherwise decode." - (let ((datahash (slot-value db 'data)) - (grouphash (registry-lookup-secondary db 'group)) - reset-pairs) - (when (hash-table-p grouphash) - (maphash - (lambda (group-name val) - (if encode - (when (multibyte-string-p group-name) - (remhash group-name grouphash) - (puthash (encode-coding-string group-name 'utf-8-emacs) - val grouphash)) - (when (string-match-p "[^[:ascii:]]" group-name) - (remhash group-name grouphash) - (puthash (decode-coding-string group-name 'utf-8-emacs) val grouphash)))) - grouphash)) - (maphash - (lambda (id data) - (let ((groups (cdr-safe (assq 'group data)))) - (when (seq-some (lambda (g) - (if encode - (multibyte-string-p g) - (string-match-p "[^[:ascii:]]" g))) - groups) - ;; Create a replacement DATA. - (push (list id (cons (cons 'group (mapcar - (lambda (g) - (funcall - (if encode - #'encode-coding-string - #'decode-coding-string) - g 'utf-8-emacs)) - groups)) - (assq-delete-all 'group data))) - reset-pairs)))) - datahash) - (pcase-dolist (`(,id ,data) reset-pairs) - (remhash id datahash) - (puthash id data datahash)))) + (defun gnus-registry-fixup-registry (db) (when db @@ -325,8 +285,7 @@ gnus-registry-fixup-registry '(mark group keyword))) (when (not (equal old (oref db tracked))) (gnus-message 9 "Reindexing the Gnus registry (tracked change)") - (registry-reindex db)) - (gnus-registry--munge-group-names db))) + (registry-reindex db)))) db) (defun gnus-registry-make-db (&optional file) @@ -410,11 +369,6 @@ gnus-registry-save (registry-size db) file) (registry-prune db gnus-registry-default-sort-function) - ;; Write a clone of the database with non-ascii group names - ;; encoded as 'utf-8. Let-bind `gnus-registry-db' so that - ;; functions in the munging process work on our clone. - (let ((gnus-registry-db clone)) - (gnus-registry--munge-group-names clone 'encode)) ;; TODO: call (gnus-string-remove-all-properties v) on all elements? (eieio-persistent-save clone file) (gnus-message 5 "Saving Gnus registry (size %d) to %s...done" diff --git a/lisp/gnus/gnus-start.el b/lisp/gnus/gnus-start.el index e142c438ee..776f60f972 100644 --- a/lisp/gnus/gnus-start.el +++ b/lisp/gnus/gnus-start.el @@ -36,6 +36,10 @@ (autoload 'gnus-agent-save-local "gnus-agent") (autoload 'gnus-agent-possibly-alter-active "gnus-agent") (declare-function gnus-group-decoded-name "gnus-group" (string)) +(declare-function eieio-object-p "eieio-core" (obj)) +(declare-function slot-value "eieio" (obj slot)) +(declare-function registry-lookup-secondary "registry" + (db tracksym &optional create)) (eval-when-compile (require 'cl-lib)) @@ -43,6 +47,9 @@ gnus-agent-covered-methods (defvar gnus-agent-file-loading-local) (defvar gnus-agent-file-loading-cache) (defvar gnus-topic-alist) +(defvar gnus-category-alist) +(defvar gnus-registry-enabled) +(defvar gnus-registry-db) (defcustom gnus-startup-file (nnheader-concat gnus-home-directory ".newsrc") "Your `.newsrc' file. @@ -1824,17 +1831,13 @@ gnus-make-hashtable-from-newsrc-alist gnus-newsrc-alist (cons (list "dummy.group" 0 nil) alist)))) (while alist - (setq info (car alist)) + (setq info (car alist) + gname (car info)) ;; Make the same select-methods identical Lisp objects. (when (setq method (gnus-info-method info)) (if (setq rest (member method methods)) (gnus-info-set-method info (car rest)) (push method methods))) - ;; Check for encoded group names and decode them. - (when (string-match-p "[^[:ascii:]]" (setq gname (car info))) - (let ((decoded (gnus-group-decoded-name gname))) - (setf gname decoded - (car info) decoded))) ;; Check for duplicates. (if (gethash gname gnus-newsrc-hashtb) ;; Remove this entry from the alist. @@ -2295,7 +2298,9 @@ gnus-convert-old-newsrc ("Gnus v5.10.7" "legacy-gnus-agent" gnus-agent-unlist-expire-days) ("Gnus v5.10.7" "legacy-gnus-agent" - gnus-agent-unhook-expire-days))) + gnus-agent-unhook-expire-days) + ("Gnus v5.14" nil + gnus-convert-encoded-group-names))) #'car-less-than-car))) ;; Skip converters older than the file version (while (and converters (>= fcv (caar converters))) @@ -2369,6 +2374,78 @@ gnus-convert-old-ticks (nconc (gnus-uncompress-range dormant) (gnus-uncompress-range ticked))))))))) +(defun gnus-convert-encoded-group-names (_converting-to) + "Decode encoded group names. +Non-ascii group names were previously stored on disk as encoded +bytes. This conversion makes sure names are encoded/decoded +properly." + (setq gnus-group-list + ;; Edit group names in `gnus-group-list', and incidentally + ;; edit them in `gnus-newsrc-hashtb' and `gnus-newsrc-alist', + ;; as well. + (mapcar + (lambda (gname) + (if (string-match-p "[^[:ascii:]]" gname) + (let ((decoded (gnus-group-decoded-name gname)) + (entry (gethash gname gnus-newsrc-hashtb))) + ;; First doctor the entry -- this will also touch + ;; `gnus-newsrc-alist'. + (setcar (nth 1 entry) decoded) + ;; Then re-hash. + (remhash gname gnus-newsrc-hashtb) + (puthash decoded entry gnus-newsrc-hashtb) + decoded) + gname)) + gnus-group-list)) + (add-hook 'gnus-agent-mode-hook #'gnus-convert-encoded-category-group-names) + (add-hook 'gnus-started-hook #'gnus-convert-encoded-registry-group-names)) + +(defun gnus-convert-encoded-category-group-names () + "Possibly decode group names stored in the agent category." + (when gnus-category-alist + (mapc (lambda (category) + (setf (alist-get 'agent-groups category) + (mapcar (lambda (g) + (if (string-match-p "[^[:ascii:]]" g) + (decode-coding-string g 'utf-8-emacs) + g)) + (alist-get 'agent-groups category)))) + gnus-category-alist))) + +(defun gnus-convert-encoded-registry-group-names () + "Possibly decode group names in the Gnus registry file." + (when (and (featurep 'gnus-registry) + gnus-registry-enabled + (eieio-object-p gnus-registry-db)) + (let ((datahash (slot-value gnus-registry-db 'data)) + (grouphash (registry-lookup-secondary gnus-registry-db 'group)) + reset-pairs) + (when (hash-table-p grouphash) + (maphash + (lambda (group-name val) + (when (string-match-p "[^[:ascii:]]" group-name) + (remhash group-name grouphash) + (puthash (decode-coding-string group-name 'utf-8-emacs) val grouphash))) + grouphash)) + (maphash + (lambda (id data) + (let ((groups (cdr-safe (assq 'group data)))) + (when (seq-some (lambda (g) + (string-match-p "[^[:ascii:]]" g)) + groups) + ;; Create a replacement DATA. + (push (list id (cons (cons 'group (mapcar + (lambda (g) + (decode-coding-string + g 'utf-8-emacs)) + groups)) + (assq-delete-all 'group data))) + reset-pairs)))) + datahash) + (pcase-dolist (`(,id ,data) reset-pairs) + (remhash id datahash) + (puthash id data datahash))))) + (defun gnus-load (file) "Load FILE, but in such a way that read errors can be reported." (with-temp-buffer @@ -2403,17 +2480,6 @@ gnus-read-newsrc-el-file (when gnus-newsrc-assoc (setq gnus-newsrc-alist gnus-newsrc-assoc)))) (gnus-make-hashtable-from-newsrc-alist) - (when gnus-topic-alist - (setq gnus-topic-alist - (mapcar - (lambda (elt) - (cons (car elt) - (mapcar (lambda (g) - (if (string-match-p "[^[:ascii:]]" g) - (gnus-group-decoded-name g) - g)) - (cdr elt)))) - gnus-topic-alist))) (when (file-newer-than-file-p file ding-file) ;; Old format quick file (gnus-message 5 "Reading %s..." file) @@ -2878,8 +2944,6 @@ gnus-gnus-to-quick-newsrc-format (delete "dummy.group" gnus-group-list))) (let* ((print-quoted t) (print-readably t) - (print-escape-multibyte nil) - (print-escape-nonascii t) (print-length nil) (print-level nil) (print-circle nil) @@ -2895,26 +2959,6 @@ gnus-gnus-to-quick-newsrc-format ;; Remove the `gnus-killed-list' from the list of variables ;; to be saved, if required. (delq 'gnus-killed-list (copy-sequence gnus-variable-list))))) - ;; Encode group names in `gnus-newsrc-alist' and - ;; `gnus-topic-alist' in order to keep newsrc.eld files - ;; compatible with older versions of Gnus. At some point, - ;; if/when a new version of Gnus is released, stop doing - ;; this and move the corresponding decode in - ;; `gnus-read-newsrc-el-file' into a conversion routine. - (gnus-newsrc-alist - (mapcar (lambda (info) - (cons (encode-coding-string (car info) 'utf-8-emacs) - (cdr info))) - gnus-newsrc-alist)) - (gnus-topic-alist - (when (memq 'gnus-topic-alist variables) - (mapcar (lambda (elt) - (cons (car elt) ; Topic name - (mapcar (lambda (g) - (encode-coding-string - g 'utf-8-emacs)) - (cdr elt)))) - gnus-topic-alist))) variable) ;; Insert the variables into the file. (while variables diff --git a/lisp/gnus/gnus.el b/lisp/gnus/gnus.el index 0673ac15f6..90b70e0923 100644 --- a/lisp/gnus/gnus.el +++ b/lisp/gnus/gnus.el @@ -6,7 +6,7 @@ ;; Author: Masanobu UMEDA ;; Lars Magne Ingebrigtsen ;; Keywords: news, mail -;; Version: 5.13 +;; Version: 5.14 ;; This file is part of GNU Emacs. @@ -292,7 +292,7 @@ gnus-fun :link '(custom-manual "(gnus)Exiting Gnus") :group 'gnus) -(defconst gnus-version-number "5.13" +(defconst gnus-version-number "5.14" "Version number for this version of Gnus.") (defconst gnus-version (format "Gnus v%s" gnus-version-number) -- 2.24.0 --=-=-=--