Updating mumi on berlin

Updating mumi on berlin

[Arun Isaac]

Hi Maxim,

I would like to update the mumi running on berlin to the latest
version. I spoke to Ricardo off-list and he tells me that there are
issues running `guix system reconfigure' on the server. Are there any
updates on that?

Thanks,
Arun

Re: Updating mumi on berlin

[zimoun]

Hi,

On Fri, 22 Apr 2022 at 11:08, Arun Isaac <arunisaac@systemreboot.net> wrote:

> I would like to update the mumi running on berlin to the latest
> version.

The update of Mumi adds a GraphQL thing to issues.guix.gnu.org, IIUC.
Quoting Ricardo from #guix-hpc [1].

--8<---------------cut here---------------start------------->8---
<rekado> mumi’s API is very limited  [20:53]
<rekado> but Arun has recently added a graphql thing
<rekado> for example, you can write this query to a file “req”:  [21:01]
<rekado> {"query": "query { issues(search: \"tag:easy\") { title tags open
         submitter { name address } } }" }
<rekado> and then do: curl --data-binary @req
         https://issues.guix.gnu.org/graphql | jq  [21:02]
<rekado> that’ll return a JSON response formatted as the query prescribes,
         containing all issues that match the given search string “tag:easy”.
--8<---------------cut here---------------end--------------->8---

Therefore, such update of Mumi looks very promising. :-)

Thanks Arun for working on that.


>          I spoke to Ricardo off-list and he tells me that there are
> issues running `guix system reconfigure' on the server. Are there any
> updates on that?

What are the blockers (issues) for updating the server (Berlin and/or
Bayfront)?


Cheers,
simon


1: <https://logs.guix.gnu.org/guix-hpc/2022-04-26.log>

Re: Updating mumi on berlin

[Arun Isaac]

Hi zimoun,

> The update of Mumi adds a GraphQL thing to issues.guix.gnu.org, IIUC.

Yes, with the update, Mumi will have a GraphQL API. GraphQL APIs are
more flexible than traditional REST APIs. See https://graphql.org/ for
more.

At some point, I plan to add support for identifying a message just by
its Message-ID. IIRC, zimoun requested this feature once. With this, I
hope to write a simple 'mumi send-email' program or similar, that will
let us send a patchset to mumi all in one go without inadvertently
opening multiple issues.

Cheers,
Arun

Re: Updating mumi on berlin

[zimoun]

Hi Arun,

On Wed, 27 Apr 2022 at 16:44, Arun Isaac <arunisaac@systemreboot.net> wrote:

> Yes, with the update, Mumi will have a GraphQL API. GraphQL APIs are
> more flexible than traditional REST APIs. See https://graphql.org/ for
> more.

Is the update of Mumi done?


Well, this GraphQL API could ease the current workflow, IMHO.

Today, one has to subscribe and it is not easy to locally read the
archives or follow what is going in.  The Emacs front-end to Debbugs
helps but Emacs is not always popular.

Instead of subscription, public-inbox provides the archives as a Git
repo.  Therefore, being up to date just becomes “guix pull” and the Git
repo can be transformed to Maildir using e.g.,

--8<---------------cut here---------------start------------->8---
    $git rev-list <range> | while read sha; do
	$git show $sha:m > <maildir>/new/$sha
    done
--8<---------------cut here---------------end--------------->8---

(modulo some errors for deleted messaged, another story).


The main issue is to synchronize the status of the Debbugs database;
from the messages, we cannot deduce the status, etc.

This GraphQL API would help to bridge the gap… at least my gap. ;-)


> At some point, I plan to add support for identifying a message just by
> its Message-ID. IIRC, zimoun requested this feature once. With this, I
> hope to write a simple 'mumi send-email' program or similar, that will
> let us send a patchset to mumi all in one go without inadvertently
> opening multiple issues.

Really cool!

Debian has this CLI tool [1], allowing to request against Debbugs.  It
could be nice to have a similar tool based on the top of Mumi.

And for instance, using this ’bts’ Debian script, it is “easy“ to have
other scripts for importing [2] bugs to any mail reader using Maildir.
Given a bug number, it allows to download all the thread of that bug.

We could package these Debian scripts and tweak them to work with the
Guix instance… Or maybe we could have some Scheme scripts. ;-)


About Message-ID and Mumi, yeah… for example, it is just an instant
using notmuch for finding the patch where you tweak Xapian and ‘guix
search’.  But to publicly refer to it, I have to open my browser scroll
and scroll again, and found it; the 14th message in the thread.  Well,
since I can easily stash the Message-ID from notmuch and I can easily
build an URL with it, then I think that:

   https://issues.guix.gnu.org/20200227204150.30985-1-arunisaac@systemreboot.net

redirecting to,

   https://issues.guix.gnu.org/39258#14

would simplify my workflow. :-)  It appears to me weird that all is
built around email but core components as Message-Id is barely used.


1: <https://salsa.debian.org/debian/devscripts/-/blob/master/scripts/bts.pl>
2: <https://git.spwhitton.name/mailscripts/tree/maildir-import-patch.1.pod>


Cheers,
simon

Re: Updating mumi on berlin

[Arun Isaac]

Hi zimoun,

> Is the update of Mumi done?

Mumi has been patched with a proof-of-concept GraphQL API. See
https://git.elephly.net/gitweb.cgi?p=software/mumi.git;a=commit;h=f5232c49fe8a3b127c96f7b502775f16aebf3033
But, I don't know if mumi has been reconfigured on berlin yet. Still
waiting on Maxim for that one.

> Well, this GraphQL API could ease the current workflow, IMHO.
>
> Today, one has to subscribe and it is not easy to locally read the
> archives or follow what is going in.  The Emacs front-end to Debbugs
> helps but Emacs is not always popular.

Indeed, one of my motivations with the GraphQL API is to "liberate" some
of our power tools from Emacs' grip. As much as I love Emacs, I
understand that not everybody does. So, it's important to strengthen our
presence outside Emacs as well.

> Debian has this CLI tool [1], allowing to request against Debbugs.  It
> could be nice to have a similar tool based on the top of Mumi.
>
> And for instance, using this ’bts’ Debian script, it is “easy“ to have
> other scripts for importing [2] bugs to any mail reader using Maildir.
> Given a bug number, it allows to download all the thread of that bug.
>
> We could package these Debian scripts and tweak them to work with the
> Guix instance… Or maybe we could have some Scheme scripts. ;-)

I didn't know about bts. Thanks for the reference. We could always
support both but I daresay we might have more fun with Scheme scripts!
;-)

> About Message-ID and Mumi, yeah… for example, it is just an instant
> using notmuch for finding the patch where you tweak Xapian and ‘guix
> search’.  But to publicly refer to it, I have to open my browser scroll
> and scroll again, and found it; the 14th message in the thread.  Well,
> since I can easily stash the Message-ID from notmuch and I can easily
> build an URL with it, then I think that:
>
>    https://issues.guix.gnu.org/20200227204150.30985-1-arunisaac@systemreboot.net
>
> redirecting to,
>
>    https://issues.guix.gnu.org/39258#14
>
> would simplify my workflow. :-)  It appears to me weird that all is
> built around email but core components as Message-Id is barely used.

I totally agree. We should be able to support something based on
Message-ID like you suggested.

Cheers!
Arun

Re: Updating mumi on berlin

[Thiago Jung Bauermann]

Hello zimoun,

zimoun <zimon.toutoune@gmail.com> writes:

> Instead of subscription, public-inbox provides the archives as a Git
> repo.  Therefore, being up to date just becomes “guix pull” and the Git
> repo can be transformed to Maildir using e.g.,
>
>     $git rev-list <range> | while read sha; do
> 	$git show $sha:m > <maildir>/new/$sha
>     done
>
> (modulo some errors for deleted messaged, another story).

Just an aside about public-inbox: Starting with version 0.7 it ships
with a ‘lei’ tool that can be used to query a local or remote
public-inbox repo, and export the matching messages to a maildir.

So the above would become something like:

$ lei q -o <maildir> -I <path or URL to public-inbox repo> -t <query>

The maildir can even be updated later with newer messages matching the
query with:

$ lei up <maildir>

-- 
Thanks
Thiago

Re: Updating mumi on berlin

[zimoun]

Hi,

On Tue, 03 May 2022 at 17:42, Thiago Jung Bauermann <bauermann@kolabnow.com> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
>
>> Instead of subscription, public-inbox provides the archives as a Git
>> repo.  Therefore, being up to date just becomes “guix pull” and the Git
>> repo can be transformed to Maildir using e.g.,
>>
>>     $git rev-list <range> | while read sha; do
>> 	$git show $sha:m > <maildir>/new/$sha
>>     done
>>
>> (modulo some errors for deleted messaged, another story).
>
> Just an aside about public-inbox: Starting with version 0.7 it ships
> with a ‘lei’ tool that can be used to query a local or remote
> public-inbox repo, and export the matching messages to a maildir.

Thanks.  Maybe I am missing something, “guix show public-inbox” tells
version 1.6.1 but I do not find ’lei’.  Anyway.

That’s why I used this ugly loop. ;-)


> So the above would become something like:
>
> $ lei q -o <maildir> -I <path or URL to public-inbox repo> -t <query>
>
> The maildir can even be updated later with newer messages matching the
> query with:
>
> $ lei up <maildir>

Yes, I agree ’lei’ is cool.  For instance, it allows to filter (and thus
follow), e.g., let query for the patches submitted the last 3 months to
’gnu/packages/bioinformatics.scm’ touching R packages,

dfn:gnu/packages/bioinformatics AND b:r-build-system AND rt:3.months.ago..

where

   dfn:     match filename from diff
   b:       match within message body, including text attachments
   rt:      match date-time range, git "approxidate" formats supported
            Open-ended ranges such as `d:last.week..' and `d:..2.days.ago'
            are supported

<https://yhetil.org/guix/?q=dfn%3Agnu%2Fpackages%2Fbioinformatics+AND+b%3Ar-build-system+AND+rt%3A3.months.ago..>

Then, ’lei up’ will request the public-inbox server for this query and
will download the messages if any.


Cheers,
simon

Re: Updating mumi on berlin

[Kyle Meyer]

zimoun writes:

> On Tue, 03 May 2022 at 17:42, Thiago Jung Bauermann <bauermann@kolabnow.com> wrote:
[...]
>> Just an aside about public-inbox: Starting with version 0.7 it ships
>> with a ‘lei’ tool that can be used to query a local or remote
>> public-inbox repo, and export the matching messages to a maildir.
>
> Thanks.  Maybe I am missing something, “guix show public-inbox” tells
> version 1.6.1 but I do not find ’lei’.  Anyway.

Yes, Thiago must have meant to write 1.7, not 0.7.

I've tried sit down and update Guix's public-inbox definition a few
times, but there are various test failures that I didn't figure out how
to handle in the time I had.  (IIRC, I think they're related to the lei
tests expecting to be able to kill the lei-daemon process, which isn't
the case in Guix's build environment.)

Re: Updating mumi on berlin

[Thiago Jung Bauermann]

Kyle Meyer <kyle@kyleam.com> writes:

> zimoun writes:
>
>> On Tue, 03 May 2022 at 17:42, Thiago Jung Bauermann <bauermann@kolabnow.com> wrote:
> [...]
>>> Just an aside about public-inbox: Starting with version 0.7 it ships
>>> with a ‘lei’ tool that can be used to query a local or remote
>>> public-inbox repo, and export the matching messages to a maildir.
>>
>> Thanks.  Maybe I am missing something, “guix show public-inbox” tells
>> version 1.6.1 but I do not find ’lei’.  Anyway.
>
> Yes, Thiago must have meant to write 1.7, not 0.7.

Indeed, that's what I meant. Sorry about the confusion.

> I've tried sit down and update Guix's public-inbox definition a few
> times, but there are various test failures that I didn't figure out how
> to handle in the time I had.

Me too. What I did while I can't move this forward was install in a
separate Guix profile a local package definition of public-inbox with
tests disabled. Then I run lei with:

$ guix shell -p path/to/profile -- lei up --all

> (IIRC, I think they're related to the lei tests expecting to be able
> to kill the lei-daemon process, which isn't the case in Guix's build
> environment.)

Yes, that's what I'm seeing as well. The lei-daemon process is actually
killed, but because of bug 30948 it is left in a zombie state and so the
testsuite thinks that it didn't go away.

The testsuite checks whether lei-daemon is gone by doing a
“kill(<lei-daemon pid>, 0)”, which unfortunately succeeds for zombie processes.

I've been meaning to add child reaping to the Guix builder process, but
I'm moving very slowly due to time constraints and my unfamiliarity with
that part of Guix...

-- 
Thanks
Thiago

¹ https://issues.guix.gnu.org/30948

Re: Updating mumi on berlin

[zimoun]

Hi Kyle and Thiago,

Reading your both answers…

On Wed, 04 May 2022 at 21:49, Kyle Meyer <kyle@kyleam.com> wrote:

> I've tried sit down and update Guix's public-inbox definition a few
> times, but there are various test failures that I didn't figure out how
> to handle in the time I had.  (IIRC, I think they're related to the lei
> tests expecting to be able to kill the lei-daemon process, which isn't
> the case in Guix's build environment.)

On Thu, 05 May 2022 at 00:13, Thiago Jung Bauermann <bauermann@kolabnow.com> wrote:

> Yes, that's what I'm seeing as well. The lei-daemon process is actually
> killed, but because of bug 30948 it is left in a zombie state and so the
> testsuite thinks that it didn't go away.
>
> The testsuite checks whether lei-daemon is gone by doing a
> “kill(<lei-daemon pid>, 0)”, which unfortunately succeeds for zombie processes.
>
> I've been meaning to add child reaping to the Guix builder process, but
> I'm moving very slowly due to time constraints and my unfamiliarity with
> that part of Guix...

…why not just disable the specific test or if not possible, turn off the
test suite ’#:tests? #f’.  Whereas it is not the best, it would allow to
have ’lei’ while waiting the fix at the Guix build environment level.

WDYT?


> $ guix shell -p path/to/profile -- lei up --all

How do you query all new ones for a specific list?


Thiago, do you use ’piem’ [1]?

Kyle, the function ’piem-inject-thread-into-maildir’ [2] is really
handy.  But I would like this signature instead:

--8<---------------cut here---------------start------------->8---
  (defun piem-inject-thread-into-maildir (mid &optional inbox message-only)
--8<---------------cut here---------------end--------------->8---

i.e., passing an ’inbox’ argument and this argument would be also passed
to ’piem-inbox-maildir-directory’ and ’piem-mid-url’,

--8<---------------cut here---------------start------------->8---
  (let ((maildir-directory (piem-inbox-maildir-directory inbox)))
[...]
    (let ((url (concat (piem-mid-url mid inbox)
--8<---------------cut here---------------end--------------->8---


Then using ’piem-inboxes’ accordingly, I have,

--8<---------------cut here---------------start------------->8---
  (defun my/public-inbox-insert (number)
    (interactive "nBug number: ")
    (let* ((meta  (car (debbugs-get-status number)))
           (inbox (car (debbugs-get-attribute meta 'package))) ;Probably inaccurate for the general case
           (raw   (debbugs-get-attribute meta 'msgid))
           (msgid (replace-regexp-in-string "<\\|>" "" raw)))
      (message "Message-ID: %s from %s." msgid inbox)
      (my/piem-inject-thread-into-maildir msgid inbox))))
--8<---------------cut here---------------end--------------->8---

For sure, it could be improved but it fits my needs.

When I do old bug squashing, I use ’M-x debbugs-gnu’ and pick some.
Sometimes, when the thread is long and complex [3], I just drop it
because it requires from me more work to deal with just the reading;
other said, I inject the thread to my local inbox which allows me to
read it offline, but more importantly, read it (or navigate) with my
favorite tool not the Gnus-like interface.


1: <https://git.kyleam.com/piem/about/>
2: <https://git.kyleam.com/piem/tree/piem.el#n759>
3: <https://yhetil.org/guix-bugs/87wpg7ffbm.fsf@gnu.org/#r>


Cheers,
simon

Re: Updating mumi on berlin

[Maxim Cournoyer]

Hi,

Thiago Jung Bauermann <bauermann@kolabnow.com> writes:

[...]

>> (IIRC, I think they're related to the lei tests expecting to be able
>> to kill the lei-daemon process, which isn't the case in Guix's build
>> environment.)
>
> Yes, that's what I'm seeing as well. The lei-daemon process is actually
> killed, but because of bug 30948 it is left in a zombie state and so the
> testsuite thinks that it didn't go away.

> The testsuite checks whether lei-daemon is gone by doing a
> “kill(<lei-daemon pid>, 0)”, which unfortunately succeeds for zombie processes.
>
> I've been meaning to add child reaping to the Guix builder process, but
> I'm moving very slowly due to time constraints and my unfamiliarity with
> that part of Guix...

Yes, that would be the correct and general solution (I hope you get to
fix it, else I may look into it in some time).  Workarounds currently in
use can be seen in our mutter package definition (it's a bit convoluted,
having to fork a process in which we set the child reaping property and
invoke the test suite via tini, acting as a fake PID 1).

HTH,

Maxim

public-inbox v1.7 update (was: Updating mumi on berlin)

[Kyle Meyer]

zimoun writes:

> …why not just disable the specific test or if not possible, turn off the
> test suite ’#:tests? #f’.  Whereas it is not the best, it would allow to
> have ’lei’ while waiting the fix at the Guix build environment level.
>
> WDYT?

public-inbox has a good test suite, so it'd be nice to keep most of it
wired up.  No objections from me about disabling the lei tests (or a
subset of them), though I haven't looked into it, so perhaps it's tricky
to do cleanly.

>> $ guix shell -p path/to/profile -- lei up --all
>
> How do you query all new ones for a specific list?

Rather than pass --all, you can call `lei up OUTPUT', where OUTPUT is a
particular saved search generated by `lei q', so you could have a saved
search that's specific for a list or set of lists:

   $ lei ls-external | grep guix
   /home/kyle/inboxes/guix-bugs boost=0
   /home/kyle/inboxes/guix-devel boost=0
   /home/kyle/inboxes/guix-patches boost=0
   /home/kyle/inboxes/guix-science boost=0
   /home/kyle/inboxes/guix-user boost=0

   $ lei q -I 'guix-*' -o /tmp/zimoun-on-guix d:20.days.ago.. f:zimoun
   $ lei up /tmp/zimoun-on-guix

See https://public-inbox.org/lei-up.txt and
https://public-inbox.org/lei-q.txt

> Kyle, the function ’piem-inject-thread-into-maildir’ [2] is really
> handy.  But I would like this signature instead:
>
> --8<---------------cut here---------------start------------->8---
>   (defun piem-inject-thread-into-maildir (mid &optional inbox message-only)
> --8<---------------cut here---------------end--------------->8---
> [...]

Without thinking too much about it, that sounds fine to me.  I'll plan
to send a patch to piem's inbox with you cc'd within the next week or
two.

Re: Updating mumi on berlin

[Thiago Jung Bauermann]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi,
>
> Thiago Jung Bauermann <bauermann@kolabnow.com> writes:
>
> [...]
>
>>> (IIRC, I think they're related to the lei tests expecting to be able
>>> to kill the lei-daemon process, which isn't the case in Guix's build
>>> environment.)
>>
>> Yes, that's what I'm seeing as well. The lei-daemon process is actually
>> killed, but because of bug 30948 it is left in a zombie state and so the
>> testsuite thinks that it didn't go away.
>
>> The testsuite checks whether lei-daemon is gone by doing a
>> “kill(<lei-daemon pid>, 0)”, which unfortunately succeeds for zombie processes.
>>
>> I've been meaning to add child reaping to the Guix builder process, but
>> I'm moving very slowly due to time constraints and my unfamiliarity with
>> that part of Guix...
>
> Yes, that would be the correct and general solution (I hope you get to
> fix it, else I may look into it in some time).

I hope so too, but at this point it's more hope than anything else,
unfortunately. I think I'll be in a time crunch for a few more weeks...

> Workarounds currently in use can be seen in our mutter package
> definition (it's a bit convoluted, having to fork a process in which
> we set the child reaping property and invoke the test suite via tini,
> acting as a fake PID 1).

Ah, that's great to know! This is a good alternative for the main
branch, since the general solution would need to go to core-updates
since it would change every derivation in Guix.

> HTH,

It does, very much! Thank you for the mutter tip.

-- 
Thanks
Thiago

Re: Updating mumi on berlin

[Thiago Jung Bauermann]

Hello,

zimoun <zimon.toutoune@gmail.com> writes:

> On Wed, 04 May 2022 at 21:49, Kyle Meyer <kyle@kyleam.com> wrote:
>
>> I've tried sit down and update Guix's public-inbox definition a few
>> times, but there are various test failures that I didn't figure out how
>> to handle in the time I had.  (IIRC, I think they're related to the lei
>> tests expecting to be able to kill the lei-daemon process, which isn't
>> the case in Guix's build environment.)
>
> On Thu, 05 May 2022 at 00:13, Thiago Jung Bauermann <bauermann@kolabnow.com> wrote:
>
>> Yes, that's what I'm seeing as well. The lei-daemon process is actually
>> killed, but because of bug 30948 it is left in a zombie state and so the
>> testsuite thinks that it didn't go away.
>>
>> The testsuite checks whether lei-daemon is gone by doing a
>> “kill(<lei-daemon pid>, 0)”, which unfortunately succeeds for zombie processes.
>>
>> I've been meaning to add child reaping to the Guix builder process, but
>> I'm moving very slowly due to time constraints and my unfamiliarity with
>> that part of Guix...
>
> …why not just disable the specific test or if not possible, turn off the
> test suite ’#:tests? #f’.  Whereas it is not the best, it would allow to
> have ’lei’ while waiting the fix at the Guix build environment level.
>
> WDYT?

I didn't go that route because I don't like disabling tests without
understanding why they fail and it took me a while to understand the
zombie root cause. In addition to that, without the testsuite passing I
didn't feel confident that I would be proposing a good enough package
because virtually the only part of public-inbox that I use is lei
itself. What if there was a regression in other parts of it and I didn't
notice because I use so little?

But I wouldn't mind disabling the part that checks that lei-daemon is
truly gone, or even the whole testsuite if that is too complicated.

On the other hand, the preferred solution would be the workaround
pointed out by Maxim, where tini is used to run the testsuite and reap
its sub-processes. I can probably take a stab at it over the weekend if
no one beats me to it.

>> $ guix shell -p path/to/profile -- lei up --all
>
> How do you query all new ones for a specific list?
>
>
> Thiago, do you use ’piem’ [1]?

I don't, but it looks interesting! I'll try it out, thanks!

> 1: <https://git.kyleam.com/piem/about/>

-- 
Thanks
Thiago

Re: public-inbox v1.7 update (was: Updating mumi on berlin)

[zimoun]

Hi Kyle,

Thanks for the answer.


On Thu, 05 May 2022 at 22:24, Kyle Meyer <kyle@kyleam.com> wrote:
> zimoun writes:

>> How do you query all new ones for a specific list?
>
> Rather than pass --all, you can call `lei up OUTPUT', where OUTPUT is a
> particular saved search generated by `lei q', so you could have a saved
> search that's specific for a list or set of lists:
>
>    $ lei ls-external | grep guix
>    /home/kyle/inboxes/guix-bugs boost=0
>    /home/kyle/inboxes/guix-devel boost=0
>    /home/kyle/inboxes/guix-patches boost=0
>    /home/kyle/inboxes/guix-science boost=0
>    /home/kyle/inboxes/guix-user boost=0
>
>    $ lei q -I 'guix-*' -o /tmp/zimoun-on-guix d:20.days.ago.. f:zimoun
>    $ lei up /tmp/zimoun-on-guix
>
> See https://public-inbox.org/lei-up.txt and
> https://public-inbox.org/lei-q.txt

For people considering the volume of the lists too much, the “saved
search” is a perfect mechanism.  For instance, it is a nice improvement
to follow guix-patches, IMHO.

However, considering guix-science, for example, the volume is not too
high, and the “saved search” would be: all the messages.

Using the public-inbox Git repo, I just run “guix fetch”, and somehow
the commits in the range master..origin/master are all the new ones.
I can convert them to maildir or whatever.

Well, using the plain Git repo, it is easy to:

 1. get messages from a list starting at a date;
    using ’git clone --mirror --shallow-since=’

 2. get all the new messages;
    (using ’git pull)

Git becoming the way to transport the information.

However, I have to choose the storage format (Git vs Maildir) to avoid
unnecessary duplication on my poor laptop.  I use Maildir because I can
easily index and search (notmuch) and read (emacs) locally; by locally,
I mean when I am offline.  Therefore, I have to run “git gc” and prune
already imported messages.

IIUC, ’lei’ avoid this manual dance with the Git repo and do it for me.

Well, what I miss is the lei “saved search” query for all messages.
Other said,

    lei q d:<starting-date>..
    lei up

fits #1.  Then, how do I achieve #2?

The workflow using “saved search” is not clear for me.  Before investing
some time, especially when ’lei’ is not packaged in Guix, I would like
to be sure about how to run «my workflow».


Cheers,
simon

Re: Updating mumi on berlin

[zimoun]

Hi Thiago,

On Fri, 06 May 2022 at 00:22, Thiago Jung Bauermann <bauermann@kolabnow.com> wrote:

> I didn't go that route because I don't like disabling tests without
> understanding why they fail and it took me a while to understand the
> zombie root cause. In addition to that, without the testsuite passing I
> didn't feel confident that I would be proposing a good enough package
> because virtually the only part of public-inbox that I use is lei
> itself. What if there was a regression in other parts of it and I didn't
> notice because I use so little?
>
> But I wouldn't mind disabling the part that checks that lei-daemon is
> truly gone, or even the whole testsuite if that is too complicated.
>
> On the other hand, the preferred solution would be the workaround
> pointed out by Maxim, where tini is used to run the testsuite and reap
> its sub-processes. I can probably take a stab at it over the weekend if
> no one beats me to it.

For sure.  For it is worth, I try to keep in mind, :-)

        Now is better than never.
        Although never is often better than *right* now.

        – The Zen of Python, by Tim Peters –

and

        Perfect is the enemy of good

        – commonly attributed to Voltaire –


Cheers,
simon

Re: Updating mumi on berlin

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Yes, that would be the correct and general solution (I hope you get to
> fix it, else I may look into it in some time).  Workarounds currently in
> use can be seen in our mutter package definition (it's a bit convoluted,
> having to fork a process in which we set the child reaping property and
> invoke the test suite via tini, acting as a fake PID 1).

The Guile process running the build script (with the build phases) runs
as PID 1.  So how about doing one of these:

  1. Add a final phase that does (waitpid WAIT_ANY).

  2. Add a SIGCHLD handler that calls ‘waitpid’.

I suspect the former is enough.

Eventually, we should do one of these in gnu-build-system.scm, as in:

  https://issues.guix.gnu.org/30948#2

WDYT?

Ludo’.

Re: Updating mumi on berlin

[Thiago Jung Bauermann]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Yes, that would be the correct and general solution (I hope you get to
>> fix it, else I may look into it in some time).  Workarounds currently in
>> use can be seen in our mutter package definition (it's a bit convoluted,
>> having to fork a process in which we set the child reaping property and
>> invoke the test suite via tini, acting as a fake PID 1).
>
> The Guile process running the build script (with the build phases) runs
> as PID 1.  So how about doing one of these:
>
>   1. Add a final phase that does (waitpid WAIT_ANY).
>
>   2. Add a SIGCHLD handler that calls ‘waitpid’.
>
> I suspect the former is enough.

I don't think it is, unless I'm misunderstanding something. The
testsuite expects the daemon to die within a few seconds of sending it
the quit command (which it does several times over the course of a
number of the tests), so having it reaped at the end of the build
process won't help.

I tried the latter a couple of weeks ago and it didn't work either,
though it's highly likely that I did something wrong in my Scheme code.
I also didn't use the 1-second timer hack so perhaps that was it.

I just tested this (adapted from the mutter package) and it worked:

--8<---------------cut here---------------start------------->8---
@@ -4137,6 +4142,26 @@ (define-public public-inbox
              ;; the builder.
              (substitute* "t/httpd-unix.t"
                (("^SKIP: \\{") "SKIP: { skip('Guix');"))))
+         (replace 'check
+           (lambda* (#:key target
+                     (tests? (not target)) (test-flags '())
+                     #:allow-other-keys)
+             (if tests?
+                 (match (primitive-fork)
+                   (0                     ;child process
+                    (set-child-subreaper!)
+                    ;; Use tini so that signals are properly handled and
+                    ;; doubly-forked processes get reaped; otherwise,
+                    ;; lei-daemon is kept as a zombie and the testsuite
+                    ;; fails thinking that it didn't quit as it should.
+                    (apply execlp "tini" "--"
+                           "make" "check" test-flags))
+                   (pid
+                    (match (waitpid pid)
+                      ((_ . status)
+                       (unless (zero? status)
+                         (error "`./Build test' exited with status" status))))))
+                 (format #t "test suite not run~%"))))
          (add-after 'install 'wrap-programs
            (lambda* (#:key inputs outputs #:allow-other-keys)
              (let ((out (assoc-ref outputs "out")))
--8<---------------cut here---------------end--------------->8---

> Eventually, we should do one of these in gnu-build-system.scm, as in:
>
>   https://issues.guix.gnu.org/30948#2
>
> WDYT?

I agree, possibly with the 1-second timer hack as well.

-- 
Thanks
Thiago

Re: public-inbox v1.7 update (was: Updating mumi on berlin)

[Kyle Meyer]

zimoun writes:

> [...]
> Well, using the plain Git repo, it is easy to:
>
>  1. get messages from a list starting at a date;
>     using ’git clone --mirror --shallow-since=’
>
>  2. get all the new messages;
>     (using ’git pull)
>[...]
> IIUC, ’lei’ avoid this manual dance with the Git repo and do it for me.

It depends.  Yes, you can use it in a way that avoids needing to clone
down repos and index them; in that case you're relying on the remote
instance.

Or you can clone and index the repos yourself and point lei at them.
You can do this with `lei add-external --mirror', but you still have to
update the repos yourself (see example at
<https://yhetil.org/emacs-devel/87wnh22w7o.fsf@kyleam.com>).

If you go the clone/index route, however, I'd recommend using grokmirror
to fetch Git repositories from public-inbox instances.  (In
https://issues.guix.gnu.org/43637#1 <878scww903.fsf@kyleam.com/>, I
provided an example config that mirrors Guix-related inboxes at
yhetil.org.)

> Well, what I miss is the lei “saved search” query for all messages.
> Other said,
>
>     lei q d:<starting-date>..
>     lei up
>
> fits #1.  Then, how do I achieve #2?

For remote externals, `lei up' will pull in updates from the remote.
For local externals (things you've cloned down and indexed yourself),
you need to do something to pull in those update (e.g., a grok-pull or a
`git fetch').

> The workflow using “saved search” is not clear for me.  Before investing
> some time, especially when ’lei’ is not packaged in Guix, I would like
> to be sure about how to run «my workflow».

I don't regularly used saved searches myself, so I don't know that I can
be much help, even if I had a clear picture of what you wanted in your
workflow.

Fwiw my setup is:

 1) use nntp via Gnus to skim new messages on mailing lists

 2) if I want to participate in a thread, use
    piem-inject-thread-into-maildir to get the messages into my local
    mail store and my Notmuch index.

 3) If I want to search for messages, I use lei-q via piem-lei-q.

    Using grokmirror and a configured pull.post_update_hook, I fetch
    updates for and index a subset yhetil.org inboxes that I want to
    have available locally for searching.

    (I use piem-inject-thread-into-maildir here too.)

I'd eventually like to be able to follow list with lei via piem's
interface, but things aren't there yet.

Re: public-inbox v1.7 update (was: Updating mumi on berlin)

[zimoun]

Hi Kyle,

Thanks for the details and explanations.


On Sun, 08 May 2022 at 00:41, Kyle Meyer <kyle@kyleam.com> wrote:

> I'd eventually like to be able to follow list with lei via piem's
> interface, but things aren't there yet.

All in all, I would like that the public-inbox format would be indexed
by Notmuch [1] :-) Instead of rely on conversion forth and back.



Cheers,
simon

1: <https://yhetil.org/notmuch/87czgm6hie.fsf@gmail.com/>

Re: Updating mumi on berlin

[Ludovic Courtès]

Hi,

Thiago Jung Bauermann <bauermann@kolabnow.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi,
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>
>>> Yes, that would be the correct and general solution (I hope you get to
>>> fix it, else I may look into it in some time).  Workarounds currently in
>>> use can be seen in our mutter package definition (it's a bit convoluted,
>>> having to fork a process in which we set the child reaping property and
>>> invoke the test suite via tini, acting as a fake PID 1).
>>
>> The Guile process running the build script (with the build phases) runs
>> as PID 1.  So how about doing one of these:
>>
>>   1. Add a final phase that does (waitpid WAIT_ANY).
>>
>>   2. Add a SIGCHLD handler that calls ‘waitpid’.
>>
>> I suspect the former is enough.
>
> I don't think it is, unless I'm misunderstanding something. The
> testsuite expects the daemon to die within a few seconds of sending it
> the quit command (which it does several times over the course of a
> number of the tests), so having it reaped at the end of the build
> process won't help.

I see, makes sense.

> I tried the latter a couple of weeks ago and it didn't work either,
> though it's highly likely that I did something wrong in my Scheme code.
> I also didn't use the 1-second timer hack so perhaps that was it.
>
> I just tested this (adapted from the mutter package) and it worked:
>
> @@ -4137,6 +4142,26 @@ (define-public public-inbox
>               ;; the builder.
>               (substitute* "t/httpd-unix.t"
>                 (("^SKIP: \\{") "SKIP: { skip('Guix');"))))
> +         (replace 'check
> +           (lambda* (#:key target
> +                     (tests? (not target)) (test-flags '())
> +                     #:allow-other-keys)
> +             (if tests?
> +                 (match (primitive-fork)
> +                   (0                     ;child process
> +                    (set-child-subreaper!)
> +                    ;; Use tini so that signals are properly handled and
> +                    ;; doubly-forked processes get reaped; otherwise,
> +                    ;; lei-daemon is kept as a zombie and the testsuite
> +                    ;; fails thinking that it didn't quit as it should.
> +                    (apply execlp "tini" "--"
> +                           "make" "check" test-flags))

How does a basic SIGCHLD handler fare?

  (sigaction SIGCHLD
    (lambda _ (false-if-exception (waitpid WAIT_ANY WNOHANG))))

>> Eventually, we should do one of these in gnu-build-system.scm, as in:
>>
>>   https://issues.guix.gnu.org/30948#2
>>
>> WDYT?
>
> I agree, possibly with the 1-second timer hack as well.

Heh, I hate that one, we need to find a better solution.

Thanks,
Ludo’.

Re: Updating mumi on berlin

[Thiago Jung Bauermann]

Hello zimoun,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Thiago,
>
> On Fri, 06 May 2022 at 00:22, Thiago Jung Bauermann <bauermann@kolabnow.com> wrote:
>
>> I didn't go that route because I don't like disabling tests without
>> understanding why they fail and it took me a while to understand the
>> zombie root cause. In addition to that, without the testsuite passing I
>> didn't feel confident that I would be proposing a good enough package
>> because virtually the only part of public-inbox that I use is lei
>> itself. What if there was a regression in other parts of it and I didn't
>> notice because I use so little?
>>
>> But I wouldn't mind disabling the part that checks that lei-daemon is
>> truly gone, or even the whole testsuite if that is too complicated.
>>
>> On the other hand, the preferred solution would be the workaround
>> pointed out by Maxim, where tini is used to run the testsuite and reap
>> its sub-processes. I can probably take a stab at it over the weekend if
>> no one beats me to it.
>
> For sure.  For it is worth, I try to keep in mind, :-)
>
>         Now is better than never.
>         Although never is often better than *right* now.
>
>         – The Zen of Python, by Tim Peters –

I liked the plot twist in this one. :-)

> and
>
>         Perfect is the enemy of good
>
>         – commonly attributed to Voltaire –

Yeah, I can easily fall into the trap of trying to get things just right
and taking too long to finish my projects… Though in this case not
feeling confident enough about the updated package due to not being able
to run the tests was a real concern for me.

Since then I have been able to run the testsuite locally and it passes,
so I have just posted a patch series updating public-inbox:

https://issues.guix.gnu.org/55436

Though that one is still with tests disabled (in the spirit of the
quotes above!). I have a bit more polishing to do before I can send the
follow-up patches to enable tests.

For now, you can find them here if you're curious:

https://gitlab.com/bauermann/guix/-/commits/public-inbox-update

-- 
Thanks
Thiago

Re: Updating mumi on berlin

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

Ludovic Courtès schreef op zo 15-05-2022 om 22:59 [+0200]:
> How does a basic SIGCHLD handler fare?
> 
>   (sigaction SIGCHLD
>     (lambda _ (false-if-exception (waitpid WAIT_ANY WNOHANG))))

Will we get two SIGCHLD when two subprocesses exit at the same time, or
only a single one? If they are combined to a single SIGCHLD, maybe
'waitpid' needs to be called in a loop so no subprocess is missed. 
Also, I think only the ECHILD system-error and maybe EINTR needs to be
caught here.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Updating mumi on berlin

[Thiago Jung Bauermann]

Hello Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Thiago Jung Bauermann <bauermann@kolabnow.com> skribis:
>
>> I tried the latter a couple of weeks ago and it didn't work either,
>> though it's highly likely that I did something wrong in my Scheme code.
>> I also didn't use the 1-second timer hack so perhaps that was it.
>>
>> I just tested this (adapted from the mutter package) and it worked:
>>
>> @@ -4137,6 +4142,26 @@ (define-public public-inbox
>>               ;; the builder.
>>               (substitute* "t/httpd-unix.t"
>>                 (("^SKIP: \\{") "SKIP: { skip('Guix');"))))
>> +         (replace 'check
>> +           (lambda* (#:key target
>> +                     (tests? (not target)) (test-flags '())
>> +                     #:allow-other-keys)
>> +             (if tests?
>> +                 (match (primitive-fork)
>> +                   (0                     ;child process
>> +                    (set-child-subreaper!)
>> +                    ;; Use tini so that signals are properly handled and
>> +                    ;; doubly-forked processes get reaped; otherwise,
>> +                    ;; lei-daemon is kept as a zombie and the testsuite
>> +                    ;; fails thinking that it didn't quit as it should.
>> +                    (apply execlp "tini" "--"
>> +                           "make" "check" test-flags))
>
> How does a basic SIGCHLD handler fare?
>
>   (sigaction SIGCHLD
>     (lambda _ (false-if-exception (waitpid WAIT_ANY WNOHANG))))

I finally had a chance to try this, but it didn't work. I didn't try the
changes suggested by Maxime though.

Here's the full ‘check’ phase I tried:

--8<---------------cut here---------------start------------->8---
(replace 'check
  (lambda* (#:key target
            (tests? (not target)) (test-flags '())
            #:allow-other-keys)
    (if tests?
        (match (primitive-fork)
          (0                     ;child process
            (set-child-subreaper!)

            ;; lei tests build UNIX domain sockets in the temporary
            ;; directory, but the path of those sockets can be at most
            ;; 108 chars and Guix' default value for the variables
            ;; below already use 47 chars. Use the shortest temporary
            ;; path possible to avoid hitting the limit.
            (setenv "TEMP" "/tmp")
            (setenv "TEMPDIR" "/tmp")
            (setenv "TMP" "/tmp")
            (setenv "TMPDIR" "/tmp")

            (sigaction SIGCHLD
              (lambda _ (false-if-exception (waitpid WAIT_ANY WNOHANG))))
            (exit (status:exit-val
                    (apply system* "make" "check" test-flags))))
          (pid
            (match (waitpid pid)
              ((_ . status)
                (unless (zero? status)
                  (error "`make check' exited with status" status))))))
        (format #t "test suite not run~%"))))
--8<---------------cut here---------------end--------------->8---

-- 
Thanks
Thiago