bug#21462: 25.0.50; Gnus thread gathering and sorting inverted

bug#21462: 25.0.50; Gnus thread gathering and sorting inverted

[Michael Welsh Duggan]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

I have been attempting to implement the following use case:

I have a gnus group that receives messages with subjects that look sort
of like this:

Some title, part 1, by X
Some title, part 2, by X
Some other title, part 32, by Y

I gather these into threads by title using my own simplification
function which I add to `gnus-simplify-subject-functions' in this
group.  I also set `gnus-thread-hide-subtree' to `t' in this group, so
each title takes up one Summary line.

I then want to sort these threads by the *latest* message in the
thread.  Thus, when a new part is received, the entire thread shows up
as "newer".  To do this, I created by own function to set
`gnus-thread-sort-functions' to.  This looks like this:

(defun md5i-thread-sort-by-most-recent-date-reverse (h1 h2)
  (<= (gnus-thread-latest-date h1) (gnus-thread-latest-date h2)))

Unfortunately, this fails in many cases due to the fact that
`gnus-summary-prepare' gathers thread after sorting threads.  In my case
I need the reverse, and I think that the reverse always makes sense.  I
have a patch which does this, and has been working for me for half a
year or so.  I suggest you add this or an equivalent change to Gnus.
Feel free to rewrite this patch completely.  I wrote it long enough ago
that I do not remember why I needed to create `gnus-make-threaded-sort'.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2138 bytes --]

ab920a06607c2ef5989fe3e5671daf294c0e25eb HEAD gnussort
Author: Michael Welsh Duggan <mwd@md5i.com>
Date:   Thu Jan 29 10:07:35 2015 -0500

    Sort threads after gathering them.
    
    Gnus sorts threads, and then gathers subjects into threads.  This can
    cause threads to be sorted improperly.

1 file changed, 11 insertions(+), 3 deletions(-)
 lisp/gnus/gnus-sum.el | 14 +++++++++++---

	Modified   lisp/gnus/gnus-sum.el
diff --git a/lisp/gnus/gnus-sum.el b/lisp/gnus/gnus-sum.el
index e4c144b..3f1d8c6 100644
--- a/lisp/gnus/gnus-sum.el
+++ b/lisp/gnus/gnus-sum.el
@@ -4149,8 +4149,8 @@ If SELECT-ARTICLES, only select those articles from GROUP."
       (gnus-summary-prepare-threads
        (if gnus-show-threads
 	   (gnus-sort-gathered-threads
-	    (funcall gnus-summary-thread-gathering-function
-		     (gnus-sort-threads
+            (gnus-sort-threads
+             (funcall gnus-summary-thread-gathering-function
 		      (gnus-cut-threads (gnus-make-threads)))))
 	 ;; Unthreaded display.
 	 (gnus-sort-articles gnus-newsgroup-headers))))
@@ -4874,6 +4874,12 @@ If LINE, insert the rebuilt thread starting on line LINE."
 	      (1+ (point-at-eol))
 	    (gnus-delete-line)))))))
 
+(defun gnus-make-threaded-sort (func)
+  (gnus-byte-compile
+   `(lambda (t1 t2)
+      (,func (if (stringp (car t1)) (cdr t1) t1)
+             (if (stringp (car t2)) (cdr t2) t2)))))
+
 (defun gnus-sort-threads-recursive (threads func)
   ;; Responsible for sorting the root articles of threads.
   (let ((subthread-sort-func (if (eq gnus-subthread-sort-functions
@@ -4920,7 +4926,9 @@ If LINE, insert the rebuilt thread starting on line LINE."
     (prog1
 	(condition-case nil
 	    (let ((max-lisp-eval-depth (max max-lisp-eval-depth 5000))
-		  (sort-func (gnus-make-sort-function gnus-thread-sort-functions)))
+		  (sort-func
+                   (gnus-make-threaded-sort
+                    (gnus-make-sort-function gnus-thread-sort-functions))))
 	      (gnus-sort-threads-recursive threads sort-func))
 	  ;; Even after binding max-lisp-eval-depth, the recursive
 	  ;; sorter might fail for very long threads.  In that case,


[-- Attachment #3: Type: text/plain, Size: 4733 bytes --]



In GNU Emacs 25.0.50.1 (x86_64-unknown-linux-gnu, X toolkit)
 of 2015-09-09
Repository revision: 54ea37308a5f6d7dc803dafaef0030ab5630f68c
Windowing system distributor 'The X.Org Foundation', version 11.0.11702000
System Description:	Debian GNU/Linux unstable (sid)

Configured using:
 'configure --without-toolkit-scroll-bars --with-x-toolkit=lucid
 'CFLAGS=-Og -ggdb3' --with-wide-int --with-gameuser=:staff'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GCONF GSETTINGS NOTIFY ACL
GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB LUCID X11

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  gnus-topic-mode: t
  cursor-sensor-mode: t
  gnus-undo-mode: t
  display-time-mode: t
  magit-auto-revert-mode: t
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t

Recent messages:
Reading active file from ~/Mail via nnml...done
nnimap read 0k from localhost
Reading active file via nndraft...done
Checking new news...done
nnimap read 0k from localhost
Don’t touch it!  It’s the History Eraser Button, you fool! [3 times]
Type "q" in help window to restore its previous buffer.
Undo! [2 times]
Auto-saving...done
Undo!

Load-path shadows:
/usr/share/emacs/site-lisp/rst hides /usr/local/share/emacs/25.0.50/lisp/textmodes/rst

Features:
(shadow emacsbug doc-view jka-compr image-mode url-util url-parse
url-vars shr dom subr-x browse-url flow-fill mm-archive sort gnus-cite
mail-extr gnus-bcklg misearch multi-isearch gnus-async gnus-dup qp
gnus-ml disp-table cus-start cus-load gnus-topic cursor-sensor utf-7
nndraft nnmh epa-file network-stream nsm starttls nnml gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu
mml2015 mm-view mml-smime smime dig gnus-cache gnus-demon nntp gnutls
nnir gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source tls utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range
gnus-win gnus gnus-ems nnheader flyspell ispell yaml-mode uptimes pp
descr-text time sieve-manage rng-loc rng-uri rng-parse rng-match rng-dt
rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util
sgml-mode psvn wid-edit cl magit-key-mode magit view tramp tramp-compat
auth-source eieio byte-opt bytecomp byte-compile cl-extra cconv
eieio-core cl-macs gv gnus-util time-date password-cache tramp-loaddefs
trampver shell pcomplete advice grep compile epa derived epg epg-config
diff-mode autorevert filenotify git-rebase-mode thingatpt
git-commit-mode server log-edit easy-mmode pcvs-util add-log mailcap
message sendmail format-spec rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mm-util help-fns
help-mode mail-prsvr mailabbrev mail-utils gmm-utils mailheader lua-mode
edmacro rx generated generic-x ediff-merg ediff-wind ediff-diff
ediff-mult ediff-help ediff-init ediff-util ediff debian-el
debian-el-loaddefs dired-x easymenu dired cc-styles cc-align cc-engine
cc-vars cc-defs ange-ftp comint ansi-color ring cl-seq kmacro
cl-loaddefs pcase cl-lib mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 417061 54251)
 (symbols 48 40364 130)
 (miscs 40 258 1042)
 (strings 32 79195 13526)
 (string-bytes 1 2526390)
 (vectors 16 34019)
 (vector-slots 8 693360 18838)
 (floats 8 513 554)
 (intervals 56 1478 480)
 (buffers 976 45)
 (heap 1024 42474 12467))

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#21462: 25.0.50; Gnus thread gathering and sorting inverted

[Lars Ingebrigtsen]

Michael Welsh Duggan <mwd@md5i.com> writes:

> I have been attempting to implement the following use case:
>
> I have a gnus group that receives messages with subjects that look sort
> of like this:
>
> Some title, part 1, by X
> Some title, part 2, by X
> Some other title, part 32, by Y
>
> I gather these into threads by title using my own simplification
> function which I add to `gnus-simplify-subject-functions' in this
> group.  I also set `gnus-thread-hide-subtree' to `t' in this group, so
> each title takes up one Summary line.
>
> I then want to sort these threads by the *latest* message in the
> thread.  Thus, when a new part is received, the entire thread shows up
> as "newer".  To do this, I created by own function to set
> `gnus-thread-sort-functions' to.  This looks like this:
>
> (defun md5i-thread-sort-by-most-recent-date-reverse (h1 h2)
>   (<= (gnus-thread-latest-date h1) (gnus-thread-latest-date h2)))
>
> Unfortunately, this fails in many cases due to the fact that
> `gnus-summary-prepare' gathers thread after sorting threads.  In my case
> I need the reverse, and I think that the reverse always makes sense.  I
> have a patch which does this, and has been working for me for half a
> year or so.  I suggest you add this or an equivalent change to Gnus.
> Feel free to rewrite this patch completely.  I wrote it long enough ago
> that I do not remember why I needed to create `gnus-make-threaded-sort'.

There is a separater `gnus-sort-gathered-threads-function' -- doesn't it
do what you want it to?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21462: 25.0.50; Gnus thread gathering and sorting inverted

[Michael Welsh Duggan]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <mwd@md5i.com> writes:
>
>> I have been attempting to implement the following use case:
>>
>> I have a gnus group that receives messages with subjects that look sort
>> of like this:
>>
>> Some title, part 1, by X
>> Some title, part 2, by X
>> Some other title, part 32, by Y
>>
>> I gather these into threads by title using my own simplification
>> function which I add to `gnus-simplify-subject-functions' in this
>> group.  I also set `gnus-thread-hide-subtree' to `t' in this group, so
>> each title takes up one Summary line.
>>
>> I then want to sort these threads by the *latest* message in the
>> thread.  Thus, when a new part is received, the entire thread shows up
>> as "newer".  To do this, I created by own function to set
>> `gnus-thread-sort-functions' to.  This looks like this:
>>
>> (defun md5i-thread-sort-by-most-recent-date-reverse (h1 h2)
>>   (<= (gnus-thread-latest-date h1) (gnus-thread-latest-date h2)))
>>
>> Unfortunately, this fails in many cases due to the fact that
>> `gnus-summary-prepare' gathers thread after sorting threads.  In my case
>> I need the reverse, and I think that the reverse always makes sense.  I
>> have a patch which does this, and has been working for me for half a
>> year or so.  I suggest you add this or an equivalent change to Gnus.
>> Feel free to rewrite this patch completely.  I wrote it long enough ago
>> that I do not remember why I needed to create `gnus-make-threaded-sort'.
>
> There is a separater `gnus-sort-gathered-threads-function' -- doesn't it
> do what you want it to?

I don't think so.  At least not according to how I understand the
documentation and how I read the sources.  gnus-sort-gathered-threads
looks like it sorts the articles _within_ a gathered thread.  The
process is:

From gnus-summary-prepare:

1) gnus-make-threads -> gathers articles into threads by ref
2) gnus-cut-threads  -> removes "uninteresting" articles from the
   threads
3) gnus-sort-threads -> sorts the threads (with respect to each other thread)
4) gnus-summary-thread-gathering-function -> gathers threads with
   similar characteristics (subject, reference) into a single thread
5) gnus-sort-gathered-threads -> sorts the articles within each thread

My argument is that steps 3 and 4 are backwards.  Step 4 is the place
where the final thread groupings are decided, and sorting these threads
with respect to each other should happen afterward.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#21462: 25.0.50; Gnus thread gathering and sorting inverted

[Lars Ingebrigtsen]

Michael Welsh Duggan <mwd@md5i.com> writes:

> 1) gnus-make-threads -> gathers articles into threads by ref
> 2) gnus-cut-threads  -> removes "uninteresting" articles from the
>    threads
> 3) gnus-sort-threads -> sorts the threads (with respect to each other thread)
> 4) gnus-summary-thread-gathering-function -> gathers threads with
>    similar characteristics (subject, reference) into a single thread
> 5) gnus-sort-gathered-threads -> sorts the articles within each thread
>
> My argument is that steps 3 and 4 are backwards.  Step 4 is the place
> where the final thread groupings are decided, and sorting these threads
> with respect to each other should happen afterward.

Yes, but can't you just sort the gathered threads in the way you would
have wanted to have them sorted if they had been sorted before
gathering?  I think I vaguely think that should kinda work...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21462: 25.0.50; Gnus thread gathering and sorting inverted

[Michael Welsh Duggan]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <mwd@md5i.com> writes:
>
>> 1) gnus-make-threads -> gathers articles into threads by ref
>> 2) gnus-cut-threads  -> removes "uninteresting" articles from the
>>    threads
>> 3) gnus-sort-threads -> sorts the threads (with respect to each other thread)
>> 4) gnus-summary-thread-gathering-function -> gathers threads with
>>    similar characteristics (subject, reference) into a single thread
>> 5) gnus-sort-gathered-threads -> sorts the articles within each thread
>>
>> My argument is that steps 3 and 4 are backwards.  Step 4 is the place
>> where the final thread groupings are decided, and sorting these threads
>> with respect to each other should happen afterward.
>
> Yes, but can't you just sort the gathered threads in the way you would
> have wanted to have them sorted if they had been sorted before
> gathering?  I think I vaguely think that should kinda work...

If I understand correctly, that would (in my case) require writing my
own version of gnus-gather-threads-by-subject.  All I can say to that
is, ugh.  If this sorting and gathering were done in what I consider to
be the more logical order, I have to write a two line function to do
what I want.  Instead I would have to mimic most of the functionality of
gnus-gather-threads-by-subject with the added restriction that I want to
maintain thread order by the latest gathered thread.  Complicated and
ugly.  Not that I wouldn't do it, if needed.  Is there a good reason for
step 3 and 4 above to be in the given order instead of the reverse?

-- 
Michael Welsh Duggan
(mwd@cert.org)

bug#21462: 25.0.50; Gnus thread gathering and sorting inverted

[Lars Ingebrigtsen]

Michael Welsh Duggan <mwd@cert.org> writes:

>>> 1) gnus-make-threads -> gathers articles into threads by ref
>>> 2) gnus-cut-threads  -> removes "uninteresting" articles from the
>>>    threads
>>> 3) gnus-sort-threads -> sorts the threads (with respect to each other thread)
>>> 4) gnus-summary-thread-gathering-function -> gathers threads with
>>>    similar characteristics (subject, reference) into a single thread
>>> 5) gnus-sort-gathered-threads -> sorts the articles within each thread

[...]

> Is there a good reason for step 3 and 4 above to be in the given order
> instead of the reverse?

I think the reason they are in that order is for historical (and
backwards compatible) reasons: The predicates that work on thread
sorting will normally fail on gathered threads (which are lists of
thread roots).  I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no