Gnus: Thread notes?

Gnus: Thread notes?

[Michael Heerdegen]

Hi,

do you know the problem that after some time, you have forgotten which
messages of a thread were important, and what you wanted to do after
some consent has been made?

I would like a simple feature in Gnus that could be described as "Thread
notes": when I click on a message, a file (maybe org) opens, maybe with
a link back to that message (the file itself would be attached to the
thread) where you could store some notes about the thread.  When you
later need to reconsider that thread, that file would serve as a summary
of the aspects that were important to you, with links to the messages
and text passages, so you spare the time to re-read the whole thread and
find the passages yourself.

Is something like that already possible, with means of Gnus or Org, that
is similarly convenient?  I'm not a heavy org-mode user, so I would
prefer a solution that is build up on Gnus instead of Org.

Or do people have other interesting workflows that are worth sharing
helping with that kind of problem?


Regards,

Michael.

Re: Gnus: Thread notes?

[Danny YUE]

Hi Michael,

I think you can do it in a reverse way:
org-capture a note when using Gnus and comment something, Org Mode will
*remember* the source/place where you capture a note, i.e. the ID of
email in this case.

Org Mode will attach a link to the original email that you can open and
jump to the email.

I think you need to somehow configure Org Mode to accomplish that.
(Please look at variable `org-capture-templates'.)

Is that sufficient enough for your needs?

Danny


On 2017-10-12 12:44, Michael Heerdegen <michael_heerdegen@web.de> wrote:
> Hi,
>
> do you know the problem that after some time, you have forgotten which
> messages of a thread were important, and what you wanted to do after
> some consent has been made?
>
> I would like a simple feature in Gnus that could be described as "Thread
> notes": when I click on a message, a file (maybe org) opens, maybe with
> a link back to that message (the file itself would be attached to the
> thread) where you could store some notes about the thread.  When you
> later need to reconsider that thread, that file would serve as a summary
> of the aspects that were important to you, with links to the messages
> and text passages, so you spare the time to re-read the whole thread and
> find the passages yourself.
>
> Is something like that already possible, with means of Gnus or Org, that
> is similarly convenient?  I'm not a heavy org-mode user, so I would
> prefer a solution that is build up on Gnus instead of Org.
>
> Or do people have other interesting workflows that are worth sharing
> helping with that kind of problem?
>
>
> Regards,
>
> Michael.

Re: Gnus: Thread notes?

[Göktuğ Kayaalp]

On 2017-10-12 14:44 +02, Michael Heerdegen <michael_heerdegen@web.de> wrote:
> I would like a simple feature in Gnus that could be described as "Thread
> notes": when I click on a message, a file (maybe org) opens, maybe with
> a link back to that message (the file itself would be attached to the
> thread) where you could store some notes about the thread.

IDK how you'd do this exact thing but, you can call ‘org-store-link’
when viewing a message in Gnus, and then in an Org mode file, when you
hit C-c C-l, a popup menu will list the stored link(s, in case you store
more than one), there you can choose the link you want.  Also, in Gnus
you can ‘tick’ articles with ‘!’, and they'll remain on the top of the
Summary listing until you untick them (M-u).

-- 
İ. Göktuğ Kayaalp	<http://www.gkayaalp.com/>
			024C 30DD 597D 142B 49AC
			40EB 465C D949 B101 2427

Re: Gnus: Thread notes?

[Robert Pluim]

Göktuğ Kayaalp <self@gkayaalp.com> writes:

> On 2017-10-12 14:44 +02, Michael Heerdegen <michael_heerdegen@web.de> wrote:
>> I would like a simple feature in Gnus that could be described as "Thread
>> notes": when I click on a message, a file (maybe org) opens, maybe with
>> a link back to that message (the file itself would be attached to the
>> thread) where you could store some notes about the thread.
>
> IDK how you'd do this exact thing but, you can call ‘org-store-link’
> when viewing a message in Gnus, and then in an Org mode file, when you
> hit C-c C-l, a popup menu will list the stored link(s, in case you store
> more than one), there you can choose the link you want.

Another way to do this is with 'org-capture', which lets you define
templates that will capture notes, email links, etc, and then file
them away in an org file.

Robert

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Hi,
>
> do you know the problem that after some time, you have forgotten which
> messages of a thread were important, and what you wanted to do after
> some consent has been made?
>
> I would like a simple feature in Gnus that could be described as "Thread
> notes": when I click on a message, a file (maybe org) opens, maybe with
> a link back to that message (the file itself would be attached to the
> thread) where you could store some notes about the thread.  When you
> later need to reconsider that thread, that file would serve as a summary
> of the aspects that were important to you, with links to the messages
> and text passages, so you spare the time to re-read the whole thread and
> find the passages yourself.
>
> Is something like that already possible, with means of Gnus or Org, that
> is similarly convenient?  I'm not a heavy org-mode user, so I would
> prefer a solution that is build up on Gnus instead of Org.
>
> Or do people have other interesting workflows that are worth sharing
> helping with that kind of problem?

I'll add a brief shameless plug for my package Gnorb (in Elpa) which
does something like this. Basically you can attach Gnus messages to an
Org heading (using the registry), and jump from the heading to an
ephemeral group holding all the related messages. Plus a bunch of other stuff.

Re: Gnus: Thread notes?

[Narendra Joshi]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Hi,
>
> do you know the problem that after some time, you have forgotten which
> messages of a thread were important, and what you wanted to do after
> some consent has been made?
>
> I would like a simple feature in Gnus that could be described as "Thread
> notes": when I click on a message, a file (maybe org) opens, maybe with
> a link back to that message (the file itself would be attached to the
> thread) where you could store some notes about the thread.  When you
> later need to reconsider that thread, that file would serve as a summary
> of the aspects that were important to you, with links to the messages
> and text passages, so you spare the time to re-read the whole thread and
> find the passages yourself.
>
> Is something like that already possible, with means of Gnus or Org, that
> is similarly convenient?  I'm not a heavy org-mode user, so I would
> prefer a solution that is build up on Gnus instead of Org.
>
> Or do people have other interesting workflows that are worth sharing
> helping with that kind of problem?
I have the following capture template for TODO's


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-org, Size: 484 bytes --]

#+name: org-capture-templates
#+begin_src org :session vicarious :exports both :results output
(setq org-capture-templates
        `(("i" "TODO" entry (file+headline "capture.org" "Tasks")
           ,(concat
             "* TODO %? %^g\n\n"
             "%(let ((link (plist-get org-store-link-plist :annotation)))"
             "     (if (string-empty-p link) "
             "         link "
             "       (concat \"Here: \" link))) ")
           :kill-buffer t)))
#+end_src

[-- Attachment #3: Type: text/plain, Size: 105 bytes --]


Every time I capture a TODO, a link to the buffer is saved in the TODO
entry.

Best,
-- 
Narendra Joshi

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I'll add a brief shameless plug for my package Gnorb (in Elpa) which
> does something like this. Basically you can attach Gnus messages to an
> Org heading (using the registry), and jump from the heading to an
> ephemeral group holding all the related messages. Plus a bunch of
> other stuff.

Thanks for all answers and ideas!  I don't want to answer to all, so
here is a summary:

I already knew org capture commands - but as sole util, they don't alone
fit what I want.  It could only be a part.

I also knew that I can tick messages.  I already have hundreds of ticked
messages so that I started to mark messages as "unread" as a flag for
myself meaning "read it again - important".  Not a good workflow.

I then learned about the Gnus registry, which is a thing I not knew
about and which fills a gap which I wondered why it existed.  It allows
custom flags (solving the above problem) and storing arbitrary data
attached to a message.  It's very low-level, however, but it would allow
to implement what I want.

Finally, "Gnorb" seems to fit perfectly, though I was a bit skeptical
first cause it seemed a bit complicated.  It allows me to capture notes/
todos etc from the Gnus summary buffer.  I can let Gnus flag all
messages that have an org headline attached in the summary buffer,
directly jump to the headline from the messages buffer and back etc -
all I wanted.  Seems nice - how could something with such a nice name as
"Gnorb" disappoint you... ;-)


Regards,

Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I'll add a brief shameless plug for my package Gnorb (in Elpa) which
>> does something like this. Basically you can attach Gnus messages to an
>> Org heading (using the registry), and jump from the heading to an
>> ephemeral group holding all the related messages. Plus a bunch of
>> other stuff.
>
> Thanks for all answers and ideas!  I don't want to answer to all, so
> here is a summary:
>
> I already knew org capture commands - but as sole util, they don't alone
> fit what I want.  It could only be a part.
>
> I also knew that I can tick messages.  I already have hundreds of ticked
> messages so that I started to mark messages as "unread" as a flag for
> myself meaning "read it again - important".  Not a good workflow.

That exact state of affairs is what led me to write Gnorb :(

[...]

> Finally, "Gnorb" seems to fit perfectly, though I was a bit skeptical
> first cause it seemed a bit complicated.  It allows me to capture notes/
> todos etc from the Gnus summary buffer.  I can let Gnus flag all
> messages that have an org headline attached in the summary buffer,
> directly jump to the headline from the messages buffer and back etc -
> all I wanted.  Seems nice - how could something with such a nice name as
> "Gnorb" disappoint you... ;-)

Glad it's working! It does seem complicated, partly because it *is*
complicated, but partly because it's hard to explain the workflow in
words. I keep meaning to make a screencast of the tracking feature, but
I haven't gone and looked into packages for doing that. There's
something called "screenkey" that looks okay, but I'd like something
more Emacs aware, where you can filter out everything but commands from
a single package.

Anyway, please report bugs/feature requests. I've gotten a bit
distracted from Gnorb while working on EBDB, and now attacking bits of
Gnus, but it's really all the same project.

Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Anyway, please report bugs/feature requests. I've gotten a bit
> distracted from Gnorb while working on EBDB, and now attacking bits of
> Gnus, but it's really all the same project.

Maybe you could improve "landing" a bit.  For example, there's an info
manual, but I discovered it only by accident.  Could you just mention
its existence in the README file?

And maybe "4.8 Likely Workflow" could be "4.1" instead.  Users hate to
read a complete section if they don't know if it contains stuff that's
useful for them.  I think it would be better if the examples would come
first in the given circumstances.  At least this was my impression when
learning about Gnorb...


Regards,

Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Anyway, please report bugs/feature requests. I've gotten a bit
>> distracted from Gnorb while working on EBDB, and now attacking bits of
>> Gnus, but it's really all the same project.
>
> Maybe you could improve "landing" a bit.  For example, there's an info
> manual, but I discovered it only by accident.  Could you just mention
> its existence in the README file?
>
> And maybe "4.8 Likely Workflow" could be "4.1" instead.  Users hate to
> read a complete section if they don't know if it contains stuff that's
> useful for them.  I think it would be better if the examples would come
> first in the given circumstances.  At least this was my impression when
> learning about Gnorb...

Good pointers! And easy to do.

Thanks,
Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Anyway, please report bugs/feature requests. I've gotten a bit
> distracted from Gnorb while working on EBDB, and now attacking bits of
> Gnus, but it's really all the same project.

Can I ask some more usage questions?

I have a thread where I want to mark some Emails as "to do" and store
some notes about them (about what's important in them, and what I have
to do).  The thread already had become longish before I decided to use
Gnorb on it.

Did I do it right?  At first, I used `org-capture' (on the
chronologically first message that seemed relevant) to create a new org
headline for the thread.  I think this message didn't get a "&" in the
summary before I explicitly added another note for it with
`gnorb-gnus-incoming-do-todo' - is this expected?

Then I `gnorb-gnus-incoming-do-todo' on the other messages in this
thread that seemed important to add notes about them.  That worked, and
I also got the "&", though I wished these notes where explicitly linked
with the corresponding message ids in the org buffer - can I let Gnorb
do this automatically?  I can C-c v on the note, but that doesn't show
me which note corresponds to which message, and vice versa.

When I opened the group again, the relevant messages were not visible
(because they now were "old").  I had to / o in the summary buffer (btw,
is there a way to let A T fetch and show also "old" messages so that I
get the complete thread without / o?), I then marked the relevant
messages "important" ("!") so that they were shown the next time.

From the org buffer, I get the C-c v (`gnorb-org-view') thing working,
but every time I want to use, seems I have to go to the server buffer
(^) to make it work - when I don't do this I get user-error: "Please add
a "nngnorb" backend to your gnus installation".  Do you have a clue why
this happens and how I can solve it?

Finally, is there a way to show the complete thread from the Gnorb nnir
group buffer?

Please excuse my ignorance, I'm not really a Gnus expert...


Thanks,

Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Anyway, please report bugs/feature requests. I've gotten a bit
>> distracted from Gnorb while working on EBDB, and now attacking bits of
>> Gnus, but it's really all the same project.
>
> Can I ask some more usage questions?

Of course!

> I have a thread where I want to mark some Emails as "to do" and store
> some notes about them (about what's important in them, and what I have
> to do).  The thread already had become longish before I decided to use
> Gnorb on it.
>
> Did I do it right?  At first, I used `org-capture' (on the
> chronologically first message that seemed relevant) to create a new org
> headline for the thread.  I think this message didn't get a "&" in the
> summary before I explicitly added another note for it with
> `gnorb-gnus-incoming-do-todo' - is this expected?

A few things: if you're adding several messages at once, you can "bulk
associate": use Gnus' process mark ("#") to mark all the relevant
messages (or "T #" for a whole thread), and then
`gnorb-gnus-incoming-do-todo'. I don't think this will work until you
_already_ have an Org heading in place -- if you mark a bunch of
messages and use `org-capture', I'm not sure it will do the right thing.
It should, though -- I'll see if I can add that.

The "&" might not be showing up simply because the gnorb function
doesn't re-draw the Gnus summary buffer. I've found that mildly annoying
in the past, I'll try to add that, too.

> Then I `gnorb-gnus-incoming-do-todo' on the other messages in this
> thread that seemed important to add notes about them.  That worked, and
> I also got the "&", though I wished these notes where explicitly linked
> with the corresponding message ids in the org buffer - can I let Gnorb
> do this automatically?  I can C-c v on the note, but that doesn't show
> me which note corresponds to which message, and vice versa.

This is something I wanted right from the beginning (to annotate
individual messages), but have never come up with a good UI for it.

Are you using a log drawer on the Org heading? That's my preferred
method of tracking TODO progress. What I could do is, when you choose
the "add a note" action (or the TODO state change prompts for a state
change log note), automatically insert the message id of the relevant
message into the log note text.

Alternately, if you wanted to keep the Gnorb notes separate, add the
note to the registry entry.

But then what? The only two ideas I can come up with are:

Provide a `tabulated-list-mode' buffer showing correspondences between
message headers and notes. But it seems a bit goofy to provide a whole
new view just for this information.

Provide a command that can be called on a message to see the relevant
note. There could then be three different marks: "¡" for relevance, "&"
for association, and "@" (or something) for association with note.

Any ideas?

> When I opened the group again, the relevant messages were not visible
> (because they now were "old").  I had to / o in the summary buffer (btw,
> is there a way to let A T fetch and show also "old" messages so that I
> get the complete thread without / o?), I then marked the relevant
> messages "important" ("!") so that they were shown the next time.

Avoiding the need to tick messages was one of the points of Gnorb! But I
see what you mean -- active Org projects are not automatically
visible...

Hmm, ideal behavior would be that all messages associated with TODOs not
in a "DONE" state would always be visible. That would be nice!

I don't think Gnus allows for this behavior by default, but what I could
do is provide a `gnus-summary-limit-to-gnorb-undone' command, bound to
"/ g". You'd enter the group, hit that key, and it would display all
currently-active tracked messages. That's not a bad idea. I'd still like
to make it do that by default, though.

"A T" ought to fetch old messages too, I use that all the time...
Threading in Gnus is arcane magic. You can almost always get it to do
exactly what you want, but it might take research and experimentation.

> From the org buffer, I get the C-c v (`gnorb-org-view') thing working,
> but every time I want to use, seems I have to go to the server buffer
> (^) to make it work - when I don't do this I get user-error: "Please add
> a "nngnorb" backend to your gnus installation".  Do you have a clue why
> this happens and how I can solve it?

This is a dumb artifact of how Gnus currently works, and this
requirement will go away if we can ever get the feature/gnus-select
branch to land. The backend is currently necessary so that nnir uses the
correct Gnorb function to retrieve the messages. Have you added a
"nngnorb" server to your `gnus-secondary-select-methods'? I'm not sure
why simply entering the server buffer would do anything.

Hmm, did you add the nngnorb server via "a" in the server buffer? I've
never tried doing that, probably safer to add it to
`gnus-secondary-select-methods'.

> Finally, is there a way to show the complete thread from the Gnorb nnir
> group buffer?

Not with Gnorb tools. "A T" is probably all you've got, but try playing
with `gnus-refer-thread-use-nnir' and surrounding code? 

> Please excuse my ignorance, I'm not really a Gnus expert...

No one is!

Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> The "&" might not be showing up simply because the gnorb function
> doesn't re-draw the Gnus summary buffer. I've found that mildly annoying
> in the past, I'll try to add that, too.

`gnus-summary-update-article' maybe? (at least, seems this is what
`gnus-registry--set/remove-mark' is doing)


> > Then I `gnorb-gnus-incoming-do-todo' on the other messages in this
> > thread that seemed important to add notes about them.  That worked, and
> > I also got the "&", though I wished these notes where explicitly linked
> > with the corresponding message ids in the org buffer - can I let Gnorb
> > do this automatically?  I can C-c v on the note, but that doesn't show
> > me which note corresponds to which message, and vice versa.
>
> This is something I wanted right from the beginning (to annotate
> individual messages), but have never come up with a good UI for it.

> Any ideas?

At least for the start, it would be enough I think if (1) adding a note
would insert the message id into the org buffer and (2) C-c v over a
message id would put the cursor over the according message.  Or
something like that...keep it simple.


> > When I opened the group again, the relevant messages were not
> > visible (because they now were "old").  I had to / o in the summary
> > buffer (btw, is there a way to let A T fetch and show also "old"
> > messages so that I get the complete thread without / o?), I then
> > marked the relevant messages "important" ("!") so that they were
> > shown the next time.
>
> Avoiding the need to tick messages was one of the points of Gnorb! But I
> see what you mean -- active Org projects are not automatically
> visible...
>
> Hmm, ideal behavior would be that all messages associated with TODOs not
> in a "DONE" state would always be visible. That would be nice!

FWIW, I would even find it acceptable if Gnorb would just silently tick
the relevant messages.  It's easy enough to untick them again, and maybe
the need to do that explicitly would integrate well with people's work
flow.  The idea is that messages you have operated on with Gnorb are all
important.

> > From the org buffer, I get the C-c v (`gnorb-org-view') thing working,
> > but every time I want to use, seems I have to go to the server buffer
> > (^) to make it work - when I don't do this I get user-error: "Please
> > add
> > a "nngnorb" backend to your gnus installation".  Do you have a clue why
> > this happens and how I can solve it?
>
> This is a dumb artifact of how Gnus currently works, and this
> requirement will go away if we can ever get the feature/gnus-select
> branch to land. The backend is currently necessary so that nnir uses the
> correct Gnorb function to retrieve the messages. Have you added a
> "nngnorb" server to your `gnus-secondary-select-methods'?

Yes.

> I'm not sure why simply entering the server buffer would do anything.
>
> Hmm, did you add the nngnorb server via "a" in the server buffer? I've
>  never tried doing that, probably safer to add it to
>  `gnus-secondary-select-methods'.

I think I did both.  AFAIR only adding to
`gnus-secondary-select-methods' did not work, and I didn't get it work
before adding a server for it.  FWIW, this is what I do:

#+begin_src emacs-lisp
(add-to-list 'gnus-secondary-select-methods '(nngnorb "my-nngnorb-server"))
#+end_src


Regards,

Michael.

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> > (btw, is there a way to let A T fetch and show also "old" messages
> > so that I get the complete thread without / o?)

> "A T" ought to fetch old messages too, I use that all the time...

It turned out that I had to increase `gnus-refer-thread-limit'.  FWIW,
you can specify it as numerical prefix arg (!) to A T.


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> The "&" might not be showing up simply because the gnorb function
>> doesn't re-draw the Gnus summary buffer. I've found that mildly annoying
>> in the past, I'll try to add that, too.
>
> `gnus-summary-update-article' maybe? (at least, seems this is what
> `gnus-registry--set/remove-mark' is doing)

Oh, weird, apparently when this annoyed me in the past I already added
this code. I don't know why it's not working...

>> > Then I `gnorb-gnus-incoming-do-todo' on the other messages in this
>> > thread that seemed important to add notes about them.  That worked, and
>> > I also got the "&", though I wished these notes where explicitly linked
>> > with the corresponding message ids in the org buffer - can I let Gnorb
>> > do this automatically?  I can C-c v on the note, but that doesn't show
>> > me which note corresponds to which message, and vice versa.
>>
>> This is something I wanted right from the beginning (to annotate
>> individual messages), but have never come up with a good UI for it.
>
>> Any ideas?
>
> At least for the start, it would be enough I think if (1) adding a note
> would insert the message id into the org buffer and (2) C-c v over a
> message id would put the cursor over the according message.  Or
> something like that...keep it simple.

Sounds good, I'll start with that. I'll add actual "gnus:" links to the
log note, that way you can make use of them in the normal way, as well.

>> > When I opened the group again, the relevant messages were not
>> > visible (because they now were "old").  I had to / o in the summary
>> > buffer (btw, is there a way to let A T fetch and show also "old"
>> > messages so that I get the complete thread without / o?), I then
>> > marked the relevant messages "important" ("!") so that they were
>> > shown the next time.
>>
>> Avoiding the need to tick messages was one of the points of Gnorb! But I
>> see what you mean -- active Org projects are not automatically
>> visible...
>>
>> Hmm, ideal behavior would be that all messages associated with TODOs not
>> in a "DONE" state would always be visible. That would be nice!
>
> FWIW, I would even find it acceptable if Gnorb would just silently tick
> the relevant messages.  It's easy enough to untick them again, and maybe
> the need to do that explicitly would integrate well with people's work
> flow.  The idea is that messages you have operated on with Gnorb are all
> important.

I'd be annoyed by that behavior, but I can provide a customization
option that will do it. I still like the idea of the Gnorb-specific
limit function, though.

[...]

>> This is a dumb artifact of how Gnus currently works, and this
>> requirement will go away if we can ever get the feature/gnus-select
>> branch to land. The backend is currently necessary so that nnir uses the
>> correct Gnorb function to retrieve the messages. Have you added a
>> "nngnorb" server to your `gnus-secondary-select-methods'?
>
> Yes.
>
>> I'm not sure why simply entering the server buffer would do anything.
>>
>> Hmm, did you add the nngnorb server via "a" in the server buffer? I've
>>  never tried doing that, probably safer to add it to
>>  `gnus-secondary-select-methods'.
>
> I think I did both.  AFAIR only adding to
> `gnus-secondary-select-methods' did not work, and I didn't get it work
> before adding a server for it.  FWIW, this is what I do:
>
> #+begin_src emacs-lisp
> (add-to-list 'gnus-secondary-select-methods '(nngnorb "my-nngnorb-server"))
> #+end_src

This has been a multi-year headache. Right now I'm checking the value of
`gnus-server-method-cache', and I don't know why your server isn't added
to that. I'll add an explicit check for `gnus-secondary-select-methods',
too.

Thanks,
Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> >> The "&" might not be showing up simply because the gnorb function
> >> doesn't re-draw the Gnus summary buffer. I've found that mildly
> >> annoying in the past, I'll try to add that, too.
> >
> > `gnus-summary-update-article' maybe? (at least, seems this is what
> > `gnus-registry--set/remove-mark' is doing)
>
> Oh, weird, apparently when this annoyed me in the past I already added
> this code. I don't know why it's not working...

AFAICT, you only do it for `gnorb-gnus-incoming-do-todo' (where it
works), but not when org-capturing in the first place.


Michael.

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> > FWIW, I would even find it acceptable if Gnorb would just silently
> > tick the relevant messages.  It's easy enough to untick them again,
> > and maybe the need to do that explicitly would integrate well with
> > people's work flow.  The idea is that messages you have operated on
> > with Gnorb are all important.
>
> I'd be annoyed by that behavior, but I can provide a customization
> option that will do it. I still like the idea of the Gnorb-specific
> limit function, though.

If you would get it all work conveniently without the need of background
ticking, it would be better and nicer, yes.


> This has been a multi-year headache. Right now I'm checking the value of
> `gnus-server-method-cache', and I don't know why your server isn't added
> to that. I'll add an explicit check for `gnus-secondary-select-methods',
> too.

I don't know, since I don't know the purpose of this cache.

If I replace that test so that it returns "my-gnorb-server" (hardcoded),
C-c v works, so it's indeed the wrong test.  I guess you could instead
check `gnus-server-alist'?


Regards,

Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> >> The "&" might not be showing up simply because the gnorb function
>> >> doesn't re-draw the Gnus summary buffer. I've found that mildly
>> >> annoying in the past, I'll try to add that, too.
>> >
>> > `gnus-summary-update-article' maybe? (at least, seems this is what
>> > `gnus-registry--set/remove-mark' is doing)
>>
>> Oh, weird, apparently when this annoyed me in the past I already added
>> this code. I don't know why it's not working...
>
> AFAICT, you only do it for `gnorb-gnus-incoming-do-todo' (where it
> works), but not when org-capturing in the first place.

Yes, I noticed that soon after...

>> > FWIW, I would even find it acceptable if Gnorb would just silently
>> > tick the relevant messages.  It's easy enough to untick them again,
>> > and maybe the need to do that explicitly would integrate well with
>> > people's work flow.  The idea is that messages you have operated on
>> > with Gnorb are all important.
>>
>> I'd be annoyed by that behavior, but I can provide a customization
>> option that will do it. I still like the idea of the Gnorb-specific
>> limit function, though.
>
> If you would get it all work conveniently without the need of background
> ticking, it would be better and nicer, yes.

It looks like automatic visibility isn't really feasible, but I've got
ticking running, and am working on the limit function.

>> This has been a multi-year headache. Right now I'm checking the value of
>> `gnus-server-method-cache', and I don't know why your server isn't added
>> to that. I'll add an explicit check for `gnus-secondary-select-methods',
>> too.
>
> I don't know, since I don't know the purpose of this cache.
>
> If I replace that test so that it returns "my-gnorb-server" (hardcoded),
> C-c v works, so it's indeed the wrong test.  I guess you could instead
> check `gnus-server-alist'?

There are several variables that hold servers:
`gnus-secondary-select-methods' for ones you've defined in your gnus.el
file, and `gnus-server-alist' for those you've defined using the Server
buffer interface. But they're supposed to all end up in
`gnus-server-method-cache'. Anyway, I've added an explicit check for
`gnus-secondary-select-methods'.

I'm testing these changes, and will push sometime this week.

Thanks,
Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> There are several variables that hold servers:
> `gnus-secondary-select-methods' for ones you've defined in your gnus.el
> file, and `gnus-server-alist' for those you've defined using the Server
> buffer interface. But they're supposed to all end up in
> `gnus-server-method-cache'.

Where should it be added?  `gnus-method-to-server' or
`gnus-server-to-method'?  Is it worth to debug it?


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> There are several variables that hold servers:
>> `gnus-secondary-select-methods' for ones you've defined in your gnus.el
>> file, and `gnus-server-alist' for those you've defined using the Server
>> buffer interface. But they're supposed to all end up in
>> `gnus-server-method-cache'.
>
> Where should it be added?  `gnus-method-to-server' or
> `gnus-server-to-method'?  Is it worth to debug it?

No, I don't think it's worth it. Gnus' own code is full of "how did we
get here" full scans, I think if we look in server-alist,
secondary-methods and the cache, that's as good as can be reasonably
expected. I understand Gnus better now than when I originally wrote this
code, so it shouldn't be too hard.

Re: Gnus: Thread notes?

[Michael Heerdegen]

Hi Eric,

thanks for uploading the new stuff.  Haven't yet tried it, but one
comment about this issue:

> > If you would get it all work conveniently without the need of
> > background ticking, it would be better and nicer, yes.
>
> It looks like automatic visibility isn't really feasible, but I've got
> ticking running, and am working on the limit function.


| +;;;###autoload
| +(defun gnorb-gnus-insert-tracked-messages (show-all)
| +  "Insert tracked messages into the Summary buffer.
| +Only inserts tracked messages belonging to this group.  If
| +SHOW-ALL (interactively, the prefix arg) is non-nil, insert all
| +messages; otherwise only insert messages that are tracked by a
| +heading in a non-DONE state."
| +  (interactive "P")
| +  (let ((old (sort (mapcar 'car gnus-newsgroup-data) '<))
| +	(tracked-messages
| +	 (registry-search gnus-registry-db
| +			  :regex `((gnorb-ids ".+"))
| +			  :member `((group ,gnus-newsgroup-name)))))
| +    (unless show-all
| +      (setq tracked-messages
| +	    (cl-remove-if
| +	     (lambda (msg-id)
| +	       (let ((id (car-safe (gnus-registry-get-id-key
| +				    msg-id 'gnorb-ids))))
| +		 (or (null id)
| +		     (save-window-excursion
| +		       (org-id-goto id)
| +		       (org-entry-is-done-p)))))
| +	     tracked-messages)))
| +    (if tracked-messages
| +	(progn
| +	  (setq tracked-messages
| +		(delq nil
| +		      (mapcar (lambda (id)
| +				(cdr (gnus-request-head id gnus-newsgroup-name)))
| +			      tracked-messages)))
| +	  (gnus-summary-insert-articles tracked-messages)
| +	  (gnus-summary-limit (gnus-sorted-nunion tracked-messages old))
| +	  (gnus-summary-position-point))
| +      (message "No tracked messages in this group"))))


I think you could factor this defun so that it would be possible to set
`gnus-alter-articles-to-read-function' to one factor, then AFAICT,
automatic listing of tracked messages should be possible.


If you care about Gnus, two things that came to my mind when fiddling
with this stuff:

(1) I think `gnus-alter-articles-to-read-function' should better default
to a function (lambda (_group-name article-list) article-list), not to
nil, so that one could use `add-function' on it.

(2) (info "(gnus) Store arbitrary data") is missing a function that can
be used to delete the entry of a key for an ID.  One can only associate
a key to nil, but that doesn't remove the association from the database;
it still contains the association key -> nil for ID.


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

[...]

> I think you could factor this defun so that it would be possible to set
> `gnus-alter-articles-to-read-function' to one factor, then AFAICT,
> automatic listing of tracked messages should be possible.

Interesting! I'd never looked at that option before. I'd still like to
keep the manual command, for those who don't want this to happen
automatically, but yes, it wouldn't be hard to refactor.

It would also require some sort of internal caching first -- right now
it's too slow to have it running each time you enter a group.

> If you care about Gnus, two things that came to my mind when fiddling
> with this stuff:
>
> (1) I think `gnus-alter-articles-to-read-function' should better default
> to a function (lambda (_group-name article-list) article-list), not to
> nil, so that one could use `add-function' on it.

Or the code could coerce the value to a list, and map all the functions.
Maybe that would be more intuitive than `add-function'?

> (2) (info "(gnus) Store arbitrary data") is missing a function that can
> be used to delete the entry of a key for an ID.  One can only associate
> a key to nil, but that doesn't remove the association from the database;
> it still contains the association key -> nil for ID.

The idea with the registry is that it's sort of a catch-all slush
bucket. I think it's expected that it will fill up rather quickly, and
then get pruned as necessary. Ie, I don't think it matters that there
are useless associations in there, only that "precious" associations
aren't pruned.

I've been thinking for a while about providing the option to have Gnorb
use a separate registry. The way the registry works now, there's so much
churn it's not really feasible to keep it in version control.

Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> > `gnus-alter-articles-to-read-function' [...]
>
> Interesting! I'd never looked at that option before. I'd still like to
> keep the manual command, for those who don't want this to happen
> automatically, but yes, it wouldn't be hard to refactor.

Yes, I agree we should keep it.

> It would also require some sort of internal caching first -- right now
> it's too slow to have it running each time you enter a group.

In my tests, I didn't see a delay.  It probably depends on how much you
used the registry.

> > (1) I think `gnus-alter-articles-to-read-function' should better
> >  default to a function (lambda (_group-name article-list)
> > article-list), not to nil, so that one could use `add-function' on
> > it.
>
> Or the code could coerce the value to a list, and map all the functions.
> Maybe that would be more intuitive than `add-function'?

But for a list, you can't control how the functions are combined.  It is
always the same, e.g., all the return values are appended.  Then it is
impossible to use the thing for limiting shown articles.  That's quite a
limitation.  With `add-function', there would not be such a restriction
- and one could use priorities (aka advice depth) to control the order
of processing.

But I know some people refrain from using `add-function'.  We could
support both mechanisms at the same time: If
`gnus-alter-articles-to-read-function' is `functionp', call it as a
function, else, treat it as a list (of functions).


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> > `gnus-alter-articles-to-read-function' [...]
>>
>> Interesting! I'd never looked at that option before. I'd still like to
>> keep the manual command, for those who don't want this to happen
>> automatically, but yes, it wouldn't be hard to refactor.
>
> Yes, I agree we should keep it.
>
>> It would also require some sort of internal caching first -- right now
>> it's too slow to have it running each time you enter a group.
>
> In my tests, I didn't see a delay.  It probably depends on how much you
> used the registry.

I use it pretty heavily! I'll implement this and test, and see how it
shakes out. A big slow-down comes in doing the
message-id-->article-number lookup each time. In the past I saved
article numbers to the registry, before realizing that was a bad idea,
but I've been considering re-introducing a per-Gnus-session cache, which
would be almost as helpful.

>> > (1) I think `gnus-alter-articles-to-read-function' should better
>> >  default to a function (lambda (_group-name article-list)
>> > article-list), not to nil, so that one could use `add-function' on
>> > it.
>>
>> Or the code could coerce the value to a list, and map all the functions.
>> Maybe that would be more intuitive than `add-function'?
>
> But for a list, you can't control how the functions are combined.  It is
> always the same, e.g., all the return values are appended.  Then it is
> impossible to use the thing for limiting shown articles.  That's quite a
> limitation.  With `add-function', there would not be such a restriction
> - and one could use priorities (aka advice depth) to control the order
> of processing.
>
> But I know some people refrain from using `add-function'.  We could
> support both mechanisms at the same time: If
> `gnus-alter-articles-to-read-function' is `functionp', call it as a
> function, else, treat it as a list (of functions).

Right, that's what I was thinking of. I do think it's bad in principle
to expect uses of `add-function'.

The following feels awkward to me, but I can't find any sort of "reverse
reduce" function in the libs. Is there a sexier way of doing this?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: alter-articles.diff --]
[-- Type: text/x-patch, Size: 693 bytes --]

diff --git a/lisp/gnus/gnus-sum.el b/lisp/gnus/gnus-sum.el
index 4dee306c81..eae0ebf130 100644
--- a/lisp/gnus/gnus-sum.el
+++ b/lisp/gnus/gnus-sum.el
@@ -5917,8 +5917,12 @@ gnus-articles-to-read
       (when gnus-alter-articles-to-read-function
 	(setq articles
 	      (sort
-	       (funcall gnus-alter-articles-to-read-function
-			gnus-newsgroup-name articles)
+	       (if (functionp gnus-alter-articles-to-read-function)
+		   (funcall gnus-alter-articles-to-read-function
+			    gnus-newsgroup-name articles)
+		 (let ((ret articles))
+		   (dolist (f gnus-alter-articles-to-read-function)
+		     (setq ret (funcall f gnus-newsgroup-name ret)))))
 	       '<)))
       articles)))
 

[-- Attachment #3: Type: text/plain, Size: 6 bytes --]


Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> The following feels awkward to me, but I can't find any sort of "reverse
> reduce" function in the libs. Is there a sexier way of doing this?

> +		 (let ((ret articles))
> +		   (dolist (f gnus-alter-articles-to-read-function)
> +		     (setq ret (funcall f gnus-newsgroup-name ret)))))

Hmm - you would need to reduce with a lambda.  That still counts as sexy
enough, I think.


Michael.

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:


> > In my tests, I didn't see a delay.  It probably depends on how much you
> > used the registry.
>
> I use it pretty heavily! I'll implement this and test, and see how it
> shakes out. A big slow-down comes in doing the
> message-id-->article-number lookup each time. In the past I saved
> article numbers to the registry, before realizing that was a bad idea,
> but I've been considering re-introducing a per-Gnus-session cache, which
> would be almost as helpful.

FWIW, I use the registry extremely seldom directly, but it already has
148,122 entries, and I get also a delay when existing Gnus.  I guess some
of these entries are not...really...necessary?


Michael.

Re: Gnus: Thread notes?

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> FWIW, I use the registry extremely seldom directly, but it already has
> 148,122 entries, and I get also a delay when existing Gnus.  I guess some
> of these entries are not...really...necessary?

Gnus seems to create an entry for every article it ever saw.  Is this
really necessary?


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

On 12/14/17 17:05 PM, Michael Heerdegen wrote:
> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> FWIW, I use the registry extremely seldom directly, but it already has
>> 148,122 entries, and I get also a delay when existing Gnus.  I guess some
>> of these entries are not...really...necessary?
>
> Gnus seems to create an entry for every article it ever saw.  Is this
> really necessary?

That's a lot of entries! You can set `gnus-registry-max-entries' to a
lower number, and play with `gnus-registry-ignored-groups'.

Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> That's a lot of entries! You can set `gnus-registry-max-entries' to a
> lower number

I think I only use the "3. Store custom flags and keywords" and
"4. Store arbitrary data" features of the registry.  I guess I don't
need thousands of registry entries therefore.  But
`gnus-registry-max-entries' doesn't seem to be the right tool - won't
pruning also delete data that an explicit action (like adding a registry
mark) created?

It would be better to prevent the entries I don't need to land in the
registry at all.  Should I just push a catchall rule to
gnus-registry-ignored-groups, or would then the "3." and "4." features
also stop to work?


Thanks,

Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

On 12/14/17 18:25 PM, Michael Heerdegen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> That's a lot of entries! You can set `gnus-registry-max-entries' to a
>> lower number
>
> I think I only use the "3. Store custom flags and keywords" and
> "4. Store arbitrary data" features of the registry.  I guess I don't
> need thousands of registry entries therefore.  But
> `gnus-registry-max-entries' doesn't seem to be the right tool - won't
> pruning also delete data that an explicit action (like adding a registry
> mark) created?

The registry also has the concept of "precious" keys -- any entry that
has a "precious" key won't be pruned at all (this is how Gnorb prevents
its data from being deleted). `gnus-registry-extra-entries-precious'
is '(mark) by default, so anything with a custom flag will be preserved.

> It would be better to prevent the entries I don't need to land in the
> registry at all.  Should I just push a catchall rule to
> gnus-registry-ignored-groups, or would then the "3." and "4." features
> also stop to work?

You can set marks on a message even if it's in an ignored group, so you
should fine ignoring more groups. The registry is also useful for other
things (tracking moved articles, etc), so I wouldn't choke it too badly.
At some point I will implement "quick searching", which would let you
use the registry to quickly recall articles you've viewed recently,
without having to mark groups, etc. In that case, you'd be happy to have
a larger registry.

My recommendation is to shrink the size of your registry until you don't
notice a slowdown, and then just relax :)

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> The following feels awkward to me, but I can't find any sort of "reverse
>> reduce" function in the libs. Is there a sexier way of doing this?
>
>> +		 (let ((ret articles))
>> +		   (dolist (f gnus-alter-articles-to-read-function)
>> +		     (setq ret (funcall f gnus-newsgroup-name ret)))))
>
> Hmm - you would need to reduce with a lambda.  That still counts as sexy
> enough, I think.

Erm, I don't quite know what you mean. Post a bit of code?

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Erm, I don't quite know what you mean. Post a bit of code?

I thought you were speaking about `reduce' as the Lisp function, like in

#+begin_src emacs-lisp
(cl-reduce (lambda (ret f) (funcall f gnus-newsgroup-name ret))
           gnus-alter-articles-to-read-function
           :initial-value articles)
#+end_src


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Erm, I don't quite know what you mean. Post a bit of code?
>
> I thought you were speaking about `reduce' as the Lisp function, like in
>
> #+begin_src emacs-lisp
> (cl-reduce (lambda (ret f) (funcall f gnus-newsgroup-name ret))
>            gnus-alter-articles-to-read-function
>            :initial-value articles)
> #+end_src

That's exactly what I meant! My brain was not cooperating. I was trying
to do it with `seq-reduce', which works exactly the same.

The only remaining issue is, I think it would be confusing to allow
`gnus-alter-articles-to-read-function' to be either a single function
value, or a list of functions. That makes it harder for consumers to
manipulate, as they have to check its current value first. What do you
think about requiring a list?

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> The only remaining issue is, I think it would be confusing to allow
>  `gnus-alter-articles-to-read-function' to be either a single function
>  value, or a list of functions. That makes it harder for consumers to
>  manipulate, as they have to check its current value first. What do
> you think about requiring a list?

I wouldn't find that good (it would be an unnecessary restriction for
users ability to configure stuff), and I think then also the variable
name would really not fit anymore the semantics.  I think it would then
be cleaner to introduce a new variable
`gnus-alter-articles-to-read-functions' (note the added "s").  Make it
so that

 - gnus-alter-articles-to-read-function (without "s") defaults to a new
 named function that would process the elements of the new variable
 which should be bound to a list (of functions), default nil.  That
 would be backward compatible.

 - People like me could use `add-function' on
 `gnus-alter-articles-to-read-function'.

 - People preferring a list could add their functions to the new
 variable binding.

I would still prefer a solution with only one variable, but given what
we currently have, and what you want, two variables may be better.  But
it's not really nice.

If I were you, I would tell people to use `add-function', it's not that
hard, and I heard most of Gnus users even use Emacs ;-)

BTW, I would expect that when the default value of
`gnus-alter-articles-to-read-function' is changed to a no-op function,
most people would just setq that variable to a function defined in their
config, there is no need to use `add-function' to configure things.  So
for users who don't like to use `add-function', nothing would change.
But OTOH, packages would be able to use `add-function' to change the
behavior (though, with a certain risk that the user inadvertently erases
that when setting the variable after a package has used `add-function'
on the binding).

Anyway, I expect that we are talking about very few users here.  But I
would hate a solution where I have to redefine a Gnus function just
because the provided means of configuration don't suffice.


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> The only remaining issue is, I think it would be confusing to allow
>>  `gnus-alter-articles-to-read-function' to be either a single function
>>  value, or a list of functions. That makes it harder for consumers to
>>  manipulate, as they have to check its current value first. What do
>> you think about requiring a list?
>
> I wouldn't find that good (it would be an unnecessary restriction for
> users ability to configure stuff), and I think then also the variable
> name would really not fit anymore the semantics.  I think it would then
> be cleaner to introduce a new variable
> `gnus-alter-articles-to-read-functions' (note the added "s").  Make it
> so that
>
>  - gnus-alter-articles-to-read-function (without "s") defaults to a new
>  named function that would process the elements of the new variable
>  which should be bound to a list (of functions), default nil.  That
>  would be backward compatible.
>
>  - People like me could use `add-function' on
>  `gnus-alter-articles-to-read-function'.
>
>  - People preferring a list could add their functions to the new
>  variable binding.
>
> I would still prefer a solution with only one variable, but given what
> we currently have, and what you want, two variables may be better.  But
> it's not really nice.
>
> If I were you, I would tell people to use `add-function', it's not that
> hard, and I heard most of Gnus users even use Emacs ;-)
>
> BTW, I would expect that when the default value of
> `gnus-alter-articles-to-read-function' is changed to a no-op function,
> most people would just setq that variable to a function defined in their
> config, there is no need to use `add-function' to configure things.  So
> for users who don't like to use `add-function', nothing would change.
> But OTOH, packages would be able to use `add-function' to change the
> behavior (though, with a certain risk that the user inadvertently erases
> that when setting the variable after a package has used `add-function'
> on the binding).
>
> Anyway, I expect that we are talking about very few users here.  But I
> would hate a solution where I have to redefine a Gnus function just
> because the provided means of configuration don't suffice.

Okay, you've convinced me. The only thing that seems weird is having it
run as as a no-op function by default, but I guess that's what
docstrings are for. I agree it would be a bad idea to change the
variable name (or make it inaccurate), or to introduce another variable
that does nearly the same thing as the first. I'll push this at some
point soon.

Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> The idea with the registry is that it's sort of a catch-all slush
> bucket. I think it's expected that it will fill up rather quickly, and
> then get pruned as necessary.

FWIW, pruning now kicked in the first time, and it errored with

Debugger entered--Lisp error: (wrong-type-argument listp quote)
  car(quote)
  (memq (car entry-key) precious)
  (cdr (memq (car entry-key) precious))
  (lambda (entry-key) (cdr (memq (car entry-key) precious)))(quote)
  cl-some((lambda (entry-key) (cdr (memq (car entry-key) precious))) '''''''''''''''''''''''''''''''''''''''''''''''''''''''((creation-time (23012 49774 793490 105000)) (group) (sender) (subject)))
  apply(cl-some (lambda (entry-key) (cdr (memq (car entry-key) precious))) '''''''''''''''''''''''''''''''''''''''''''''''''''''''((creation-time (23012 49774 793490 105000)) (group) (sender) (subject)) nil)
  cl-notany((lambda (entry-key) (cdr (memq (car entry-key) precious))) '''''''''''''''''''''''''''''''''''''''''''''''''''''''((creation-time (23012 49774 793490 105000)) (group) (sender) (subject)))
  (if (cl-notany precious-p v) (progn (setq #1=#:--cl-var-- (cons (cons k v) #1#))))
  (lambda (k v) (if (cl-notany precious-p v) (progn (setq #1=#:--cl-var-- (cons (cons k v) #1#)))))("" '''''''''''''''''''''''''''''''''''''''''''''''''''''''((creation-time (23012 49774 793490 105000)) (group) (sender) (subject)))
  maphash((lambda (k v) (if (cl-notany precious-p v) (progn (setq #1=#:--cl-var-- (cons (cons k v) #1#))))) #<hash-table equal 25000/40000 0x1bbf149>)
  (let* ((#1=#:--cl-var-- nil)) (maphash (function (lambda (k v) (if (cl-notany precious-p v) (progn (setq #1# (cons (cons k v) #1#)))))) data) (nreverse #1#))
  (let* ((precious (eieio-oref db 'precious)) (precious-p (function (lambda (entry-key) (cdr (memq (car entry-key) precious))))) (data (eieio-oref db 'data)) (candidates (let* ((#1=#:--cl-var-- nil)) (maphash (function (lambda (k v) (if (cl-notany precious-p v) (progn (setq #1# (cons (cons k v) #1#)))))) data) (nreverse #1#)))) (if sortfunc (progn (setq candidates (sort candidates sortfunc)))) (cl-subseq (mapcar (function car) candidates) 0 (min limit (length candidates))))
  (progn (let* ((precious (eieio-oref db 'precious)) (precious-p (function (lambda (entry-key) (cdr (memq (car entry-key) precious))))) (data (eieio-oref db 'data)) (candidates (let* ((#1=#:--cl-var-- nil)) (maphash (function (lambda (k v) (if (cl-notany precious-p v) (progn (setq #1# (cons (cons k v) #1#)))))) data) (nreverse #1#)))) (if sortfunc (progn (setq candidates (sort candidates sortfunc)))) (cl-subseq (mapcar (function car) candidates) 0 (min limit (length candidates)))))
  (lambda (db limit sortfunc) "Collects pruning candidates from the registry-db object DB.\n\nProposes only entries without the :precious keys, and attempts to\nreturn LIMIT such candidates.  If SORTFUNC is provided, sort\nentries first and return candidates from beginning of list." (progn (let* ((precious (eieio-oref db 'precious)) (precious-p (function (lambda (entry-key) (cdr (memq (car entry-key) precious))))) (data (eieio-oref db 'data)) (candidates (let* ((#1=#:--cl-var-- nil)) (maphash (function (lambda (k v) (if (cl-notany precious-p v) (progn (setq #1# (cons (cons k v) #1#)))))) data) (nreverse #1#)))) (if sortfunc (progn (setq candidates (sort candidates sortfunc)))) (cl-subseq (mapcar (function car) candidates) 0 (min limit (length candidates))))))(#<registry-db registry-db-195b
 804> 2500 gnus-registry-sort-by-creation-time)
  apply((lambda (db limit sortfunc) "Collects pruning candidates from the registry-db object DB.\n\nProposes only entries without the :precious keys, and attempts to\nreturn LIMIT such candidates.  If SORTFUNC is provided, sort\nentries first and return candidates from beginning of list." (progn (let* ((precious (eieio-oref db 'precious)) (precious-p (function (lambda (entry-key) (cdr (memq (car entry-key) precious))))) (data (eieio-oref db 'data)) (candidates (let* ((#1=#:--cl-var-- nil)) (maphash (function (lambda (k v) (if (cl-notany precious-p v) (progn (setq #1# (cons (cons k v) #1#)))))) data) (nreverse #1#)))) (if sortfunc (progn (setq candidates (sort candidates sortfunc)))) (cl-subseq (mapcar (function car) candidates) 0 (min limit (length candidates)))))) #<registry-db registry-d
 b-195b804> (2500 gnus-registry-sort-by-creation-time))
  registry-collect-prune-candidates(#<registry-db registry-db-195b804> 2500 gnus-registry-sort-by-creation-time)
  (setq candidates (registry-collect-prune-candidates db (- size target-size) sortfunc))
  (progn (setq candidates (registry-collect-prune-candidates db (- size target-size) sortfunc)) (length (registry-delete db candidates nil)))
  (if (registry-full db) (progn (setq candidates (registry-collect-prune-candidates db (- size target-size) sortfunc)) (length (registry-delete db candidates nil))) 0)
  (let ((size (registry-size db)) (target-size (floor (- (eieio-oref db 'max-size) (* (eieio-oref db 'max-size) (eieio-oref db 'prune-factor))))) candidates) (if (registry-full db) (progn (setq candidates (registry-collect-prune-candidates db (- size target-size) sortfunc)) (length (registry-delete db candidates nil))) 0))
  (progn (let ((size (registry-size db)) (target-size (floor (- (eieio-oref db 'max-size) (* (eieio-oref db 'max-size) (eieio-oref db 'prune-factor))))) candidates) (if (registry-full db) (progn (setq candidates (registry-collect-prune-candidates db (- size target-size) sortfunc)) (length (registry-delete db candidates nil))) 0)))
  (lambda (db &optional sortfunc) "Prunes the registry-db object DB.\n\nAttempts to prune the number of entries down to (*\n:max-size :prune-factor) less than the max-size limit, so\npruning doesn't need to happen on every save. Removes only\nentries without the :precious keys, so it may not be possible to\nreach the target limit.\n\nEntries to be pruned are first sorted using SORTFUNC.  Entries\nfrom the front of the list are deleted first.\n\nReturns the number of deleted entries." (progn (let ((size (registry-size db)) (target-size (floor (- (eieio-oref db 'max-size) (* (eieio-oref db 'max-size) (eieio-oref db 'prune-factor))))) candidates) (if (registry-full db) (progn (setq candidates (registry-collect-prune-candidates db (- size target-size) sortfunc)) (length (registry-delete db can
 didates nil))) 0))))(#<registry-db registry-db-195b804> gnus-registry-sort-by-creation-time)
  apply((lambda (db &optional sortfunc) "Prunes the registry-db object DB.\n\nAttempts to prune the number of entries down to (*\n:max-size :prune-factor) less than the max-size limit, so\npruning doesn't need to happen on every save. Removes only\nentries without the :precious keys, so it may not be possible to\nreach the target limit.\n\nEntries to be pruned are first sorted using SORTFUNC.  Entries\nfrom the front of the list are deleted first.\n\nReturns the number of deleted entries." (progn (let ((size (registry-size db)) (target-size (floor (- (eieio-oref db 'max-size) (* (eieio-oref db 'max-size) (eieio-oref db 'prune-factor))))) candidates) (if (registry-full db) (progn (setq candidates (registry-collect-prune-candidates db (- size target-size) sortfunc)) (length (registry-delete 
 db candidates nil))) 0)))) #<registry-db registry-db-195b804> gnus-registry-sort-by-creation-time)
  registry-prune(#<registry-db registry-db-195b804> gnus-registry-sort-by-creation-time)
  gnus-registry-save()
  run-hooks(gnus-save-newsrc-hook)
  apply(run-hooks gnus-save-newsrc-hook)
  gnus-run-hooks(gnus-save-newsrc-hook)
  gnus-save-newsrc-file()
  gnus-group-exit()
  funcall-interactively(gnus-group-exit)
  call-interactively(gnus-group-exit nil nil)
  command-execute(gnus-group-exit)

There seems to be an entry with a lot of quotes.  Any idea what could be
happening here?  An eieio related thing, maybe?  Where should I dig?


Thanks,

Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> The idea with the registry is that it's sort of a catch-all slush
>> bucket. I think it's expected that it will fill up rather quickly, and
>> then get pruned as necessary.
>
> FWIW, pruning now kicked in the first time, and it errored with
>
> Debugger entered--Lisp error: (wrong-type-argument listp quote)
>   car(quote)
>   (memq (car entry-key) precious)
>   (cdr (memq (car entry-key) precious))
>   (lambda (entry-key) (cdr (memq (car entry-key) precious)))(quote)
>   cl-some((lambda (entry-key) (cdr (memq (car entry-key) precious)))
>   '''''''''''''''''''''''''''''''''''''''''''''''''''''''((creation-time
>   (23012 49774 793490 105000)) (group) (sender) (subject)))

Well that's deeply weird. You have an entry in your registry that looks like:

("" '''''''''''''''''''''''''''''''''''''''''''''''''''''''\
    ((creation-time (23012 49774 793490 105000)) (group) (sender) (subject)))

Ie, no key at all, and who knows what's going on with all those quotes.

Are there many of those, or just one? And are you running an Emacs that
includes e1cc2037a918?

I looked at the code, and there's nothing that would prevent the
creation of an entry for a message that had no Message-ID, so presumably
the blank there is possible. That doesn't explain all the quotes,
though.

Ugh, now I'm seeing more bugs in there --
`gnus-registry-fetch-header-fast' looks broken...

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Well that's deeply weird. You have an entry in your registry that
> looks like:
>
> ("" '''''''''''''''''''''''''''''''''''''''''''''''''''''''\
>     ((creation-time (23012 49774 793490 105000)) (group) (sender)
> (subject)))
>
> Ie, no key at all

The empty string is the key, I think.

> , and who knows what's going on with all those quotes.
>
> Are there many of those, or just one? And are you running an Emacs that
> includes e1cc2037a918?

Yes.

I did

(maphash (lambda (key value) (message "%S\n\n" (list key value)))
         (oref gnus-registry-db data))

and nearly all entries have a lot of quotes.  I starts with entries with
81 quotes, and then the number of quotes seems to decrease slowly.  The
last entries printed only have one quote.  I guess there is a quote
added for every save or so.

But I only have one entry with the empty string as key.

> I looked at the code, and there's nothing that would prevent the
> creation of an entry for a message that had no Message-ID, so presumably
> the blank there is possible. That doesn't explain all the quotes,
> though.

Yes, it's horribly broken...


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Well that's deeply weird. You have an entry in your registry that
>> looks like:
>>
>> ("" '''''''''''''''''''''''''''''''''''''''''''''''''''''''\
>>     ((creation-time (23012 49774 793490 105000)) (group) (sender)
>> (subject)))
>>
>> Ie, no key at all
>
> The empty string is the key, I think.

Yes, in the sense that it's *supposed* to be the key. But not in the
sense that it's the source of the problem -- I just noticed that my
registry is doing the same thing, and I don't have empty-string keys.

>> , and who knows what's going on with all those quotes.
>>
>> Are there many of those, or just one? And are you running an Emacs that
>> includes e1cc2037a918?
>
> Yes.
>
> I did
>
> (maphash (lambda (key value) (message "%S\n\n" (list key value)))
>          (oref gnus-registry-db data))
>
> and nearly all entries have a lot of quotes.  I starts with entries with
> 81 quotes, and then the number of quotes seems to decrease slowly.  The
> last entries printed only have one quote.  I guess there is a quote
> added for every save or so.
>
> But I only have one entry with the empty string as key.
>
>> I looked at the code, and there's nothing that would prevent the
>> creation of an entry for a message that had no Message-ID, so presumably
>> the blank there is possible. That doesn't explain all the quotes,
>> though.
>
> Yes, it's horribly broken...

Actually, it's my fault, not the registry's. I patched the persistence
process in e1cc2037a918 so that objects inside hash tables were written
and read correctly. But it looks like the trick I was using writes a
call to `quote' where there shouldn't be one, and those accumulate.
Bummer. I will go back to the other bug thread with this.

Sorry!
Eric

Re: Gnus: Thread notes?

[Michael Heerdegen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Yes, in the sense that it's *supposed* to be the key. But not in the
> sense that it's the source of the problem -- I just noticed that my
> registry is doing the same thing, and I don't have empty-string keys.

Ok.  And I can't rule out that my first experiments with the registry
produced the empty key.

> Actually, it's my fault, not the registry's. I patched the persistence
> process in e1cc2037a918 so that objects inside hash tables were written
> and read correctly. But it looks like the trick I was using writes a
> call to `quote' where there shouldn't be one, and those accumulate.
> Bummer. I will go back to the other bug thread with this.

Thanks.  Which subject does that thread have?  Can you maybe also post a
tool for repairing the registry?


Michael.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Thanks.  Which subject does that thread have?  Can you maybe also post a
> tool for repairing the registry?

It's bug #29220. And yes, once that's fixed, I'll post something for
fixing the registry. Sorry for the mess.

Re: Gnus: Thread notes?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> The only remaining issue is, I think it would be confusing to allow
>>>  `gnus-alter-articles-to-read-function' to be either a single function
>>>  value, or a list of functions. That makes it harder for consumers to
>>>  manipulate, as they have to check its current value first. What do
>>> you think about requiring a list?
>>
>> I wouldn't find that good (it would be an unnecessary restriction for
>> users ability to configure stuff), and I think then also the variable
>> name would really not fit anymore the semantics.  I think it would then
>> be cleaner to introduce a new variable
>> `gnus-alter-articles-to-read-functions' (note the added "s").  Make it
>> so that
>>
>>  - gnus-alter-articles-to-read-function (without "s") defaults to a new
>>  named function that would process the elements of the new variable
>>  which should be bound to a list (of functions), default nil.  That
>>  would be backward compatible.
>>
>>  - People like me could use `add-function' on
>>  `gnus-alter-articles-to-read-function'.
>>
>>  - People preferring a list could add their functions to the new
>>  variable binding.
>>
>> I would still prefer a solution with only one variable, but given what
>> we currently have, and what you want, two variables may be better.  But
>> it's not really nice.
>>
>> If I were you, I would tell people to use `add-function', it's not that
>> hard, and I heard most of Gnus users even use Emacs ;-)
>>
>> BTW, I would expect that when the default value of
>> `gnus-alter-articles-to-read-function' is changed to a no-op function,
>> most people would just setq that variable to a function defined in their
>> config, there is no need to use `add-function' to configure things.  So
>> for users who don't like to use `add-function', nothing would change.
>> But OTOH, packages would be able to use `add-function' to change the
>> behavior (though, with a certain risk that the user inadvertently erases
>> that when setting the variable after a package has used `add-function'
>> on the binding).
>>
>> Anyway, I expect that we are talking about very few users here.  But I
>> would hate a solution where I have to redefine a Gnus function just
>> because the provided means of configuration don't suffice.
>
> Okay, you've convinced me. The only thing that seems weird is having it
> run as as a no-op function by default, but I guess that's what
> docstrings are for. I agree it would be a bad idea to change the
> variable name (or make it inaccurate), or to introduce another variable
> that does nearly the same thing as the first. I'll push this at some
> point soon.

Okay, there it goes.