unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
* This is how you import a mailing list archives (e.g. guix-patches) into mu4e
@ 2022-11-16 23:04 Mekeor Melire
  2022-11-16 23:22 ` jbranso
  2022-12-05  8:45 ` zimoun
  0 siblings, 2 replies; 15+ messages in thread
From: Mekeor Melire @ 2022-11-16 23:04 UTC (permalink / raw)
  To: help-guix

Hello Guix,

you might want to import a mailing-list-archive into mu4e. E.g. because
debbugs.el is slow, you might want to import the archives of the
guix-patches mailing list. This is how to do so.

First, download the archives as in the "mbox" format -- the only format
that Mailman offers:

--8<---------------cut here---------------start------------->8---
cd ~/some/where
for y in {2017..2022}; do
    for m in {01..12}; do
        curl --fail "https://lists.gnu.org/archive/mbox/guix-patches/$y-$m" > "$y-$m" || rm "$y-$m";
    done;
done
--8<---------------cut here---------------end--------------->8---

Now, download and run a perl-tool named "mb2md" to convert the mboxes
into maildirs:

--8<---------------cut here---------------start------------->8---
mkdir tool maildir
curl http://batleth.sapienti-sat.org/projects/mb2md/mb2md-3.20.pl.gz \
    --output tool/mb2md.pl.gz
gzip -d tool/mb2md.pl.gz
chmod 755 tool/mb2md.pl
guix shell perl perl-datetime perl-timedate
for f in $(realpath ./mbox)/*; do
    perl tool/mb2md.pl -s "$f" -d $(realpath mailbox);
done
exit
--8<---------------cut here---------------end--------------->8---

Next, you might want to mark all mails in that maildir as "read":

--8<---------------cut here---------------start------------->8---
guix shell mblaze
mlist -s ./mailbox | mflag -S
exit
--8<---------------cut here---------------end--------------->8---

Finally, move that maildir to its destination:

--8<---------------cut here---------------start------------->8---
mv maildir ~/path/to/maildirs/guix-patches
--8<---------------cut here---------------end--------------->8---

Enjoy!


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-11-16 23:04 This is how you import a mailing list archives (e.g. guix-patches) into mu4e Mekeor Melire
@ 2022-11-16 23:22 ` jbranso
  2022-11-16 23:41   ` Mekeor Melire
  2022-12-05  8:45 ` zimoun
  1 sibling, 1 reply; 15+ messages in thread
From: jbranso @ 2022-11-16 23:22 UTC (permalink / raw)
  To: Mekeor Melire, help-guix

November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote:

> Hello Guix,
> 
> you might want to import a mailing-list-archive into mu4e. E.g. because
> debbugs.el is slow, you might want to import the archives of the
> guix-patches mailing list. This is how to do so.

Hey thanks for sharing! How about we make this guide a bit more permanent?

Would you mind writing this up and including it in the guix cookbook?
Once you submit the patch, please CC me.  I'll review it for you.

Thanks again!

Joshua

> 
> First, download the archives as in the "mbox" format -- the only format
> that Mailman offers:
> 
> --8<---------------cut here---------------start------------->8---
> cd ~/some/where
> for y in {2017..2022}; do
> for m in {01..12}; do
> curl --fail "https://lists.gnu.org/archive/mbox/guix-patches/$y-$m" > "$y-$m" || rm "$y-$m";
> done;
> done
> --8<---------------cut here---------------end--------------->8---
> 
> Now, download and run a perl-tool named "mb2md" to convert the mboxes
> into maildirs:
> 
> --8<---------------cut here---------------start------------->8---
> mkdir tool maildir
> curl http://batleth.sapienti-sat.org/projects/mb2md/mb2md-3.20.pl.gz \
> --output tool/mb2md.pl.gz
> gzip -d tool/mb2md.pl.gz
> chmod 755 tool/mb2md.pl
> guix shell perl perl-datetime perl-timedate
> for f in $(realpath ./mbox)/*; do
> perl tool/mb2md.pl -s "$f" -d $(realpath mailbox);
> done
> exit
> --8<---------------cut here---------------end--------------->8---
> 
> Next, you might want to mark all mails in that maildir as "read":
> 
> --8<---------------cut here---------------start------------->8---
> guix shell mblaze
> mlist -s ./mailbox | mflag -S
> exit
> --8<---------------cut here---------------end--------------->8---
> 
> Finally, move that maildir to its destination:
> 
> --8<---------------cut here---------------start------------->8---
> mv maildir ~/path/to/maildirs/guix-patches
> --8<---------------cut here---------------end--------------->8---
> 
> Enjoy!


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-11-16 23:22 ` jbranso
@ 2022-11-16 23:41   ` Mekeor Melire
  2022-12-04 20:16     ` Mekeor Melire
  0 siblings, 1 reply; 15+ messages in thread
From: Mekeor Melire @ 2022-11-16 23:41 UTC (permalink / raw)
  To: jbranso; +Cc: help-guix

2022-11-16 / 23:22 / jbranso@dismail.de:

> November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote:
>
> > Hello Guix,
> >
> > you might want to import a mailing-list-archive into mu4e. E.g.
> > because debbugs.el is slow, you might want to import the archives of
> > the guix-patches mailing list. This is how to do so.
>
> Hey

Hello :)

> thanks for sharing! How about we make this guide a bit more permanent?

Good idea :)

> Would you mind writing this up and including it in the guix cookbook?

Alright, I'll try to send a patch. I think I should change the
instructions in order to replace the unpackaged mb2md perl-script in
favor of the packaged mblaze tool. Afterwards, it should be straight
forward.

> Once you submit the patch, please CC me. I'll review it for you.

Alright :)

> Thanks again!
> Joshua

Thank you, too.
Mekeor


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-11-16 23:41   ` Mekeor Melire
@ 2022-12-04 20:16     ` Mekeor Melire
  2022-12-05 15:39       ` Joshua Branson
  0 siblings, 1 reply; 15+ messages in thread
From: Mekeor Melire @ 2022-12-04 20:16 UTC (permalink / raw)
  To: Mekeor Melire; +Cc: jbranso, help-guix

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

2022-11-16 / 23:41 / mekeor@posteo.de:
> 2022-11-16 / 23:22 / jbranso@dismail.de:
> > November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote:

> > > you might want to import a mailing-list-archive into mu4e. E.g.
> > > because debbugs.el is slow, you might want to import the archives of
> > > the guix-patches mailing list. This is how to do so.

> > Would you mind writing this up and including it in the guix cookbook?
>
> Alright, I'll try to send a patch. I think I should change the
> instructions in order to replace the unpackaged mb2md perl-script in
> favor of the packaged mblaze tool. Afterwards, it should be straight
> forward.

I made a little shell script out of these instructions. You can find it
attached. The script allows to either import a specified GNU
mailing-list into a specified maildir, or update that maildir, beginning
either from a specified YYYY-MM date or from the date of the latest
message.

For example, I used this command to initialize my mailing-list-maildirs:

    guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix
        guix-devel guix-patches help-guix; do ~/path/to/ugmltmd
        ~/path/to/mail/$l $l 2022-01; done'

And I use the following command to update my mailing-list-maildirs:

    guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix
        guix-devel guix-patches help-guix; do ~/path/to/ugmltmd
        ~/path/to/mail/$l $l; done'

I wonder if it still makes sense to add some instructions into the Guix
cookbook. And if so, I wonder if this shell script should just be put
there or if it should live in some public git repository somewhere.

What do you think?


[-- Attachment #2: ugmltmd --]
[-- Type: application/x-sh, Size: 1831 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-11-16 23:04 This is how you import a mailing list archives (e.g. guix-patches) into mu4e Mekeor Melire
  2022-11-16 23:22 ` jbranso
@ 2022-12-05  8:45 ` zimoun
  2022-12-05 21:43   ` Mekeor Melire
  2022-12-06  1:03   ` Kyle Meyer
  1 sibling, 2 replies; 15+ messages in thread
From: zimoun @ 2022-12-05  8:45 UTC (permalink / raw)
  To: Mekeor Melire, help-guix

Hi,

On Wed, 16 Nov 2022 at 23:04, Mekeor Melire <mekeor@posteo.de> wrote:

> you might want to import a mailing-list-archive into mu4e. E.g. because
> debbugs.el is slow, you might want to import the archives of the
> guix-patches mailing list. This is how to do so.

Personally, I use “git clone” from a public-inbox instance [1].

    git clone --mirror https://yhetil.org/guix-patches/1 \
              guix-patches/git/1.git

where ’1’ can be also replace by ’0’ for the very old ones.

Then the conversion from Git commit to maildir is done by a small script
[2], where all the job reads:

--8<---------------cut here---------------start------------->8---
    # Extract the message from each commit in the range and store it
    # in the Maildir for notmuch to consume.
    $git rev-list $range | while read sha; do
        # XXXX: fatal: path 'm' does not exist in <commit>
        # and it can also raise issues with notmuch, as:
        # Note: Ignoring non-mail file: $maildir/new/$sha
	$git show $sha:m > $maildir/new/$sha
    done
--8<---------------cut here---------------end--------------->8---

(Maybe better could be done and more robust are around.)


Even, the same can be used for reading Guix mailing lists.  New messages
are gotten with “git fetch”.


1: <https://yhetil.org/guix-patches/>
2: <https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L60-67>

Cheers,
simon


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-04 20:16     ` Mekeor Melire
@ 2022-12-05 15:39       ` Joshua Branson
  0 siblings, 0 replies; 15+ messages in thread
From: Joshua Branson @ 2022-12-05 15:39 UTC (permalink / raw)
  To: Mekeor Melire; +Cc: help-guix

Mekeor Melire <mekeor@posteo.de> writes:

> 2022-11-16 / 23:41 / mekeor@posteo.de:
>> 2022-11-16 / 23:22 / jbranso@dismail.de:
>> > November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote:
>
>> > > you might want to import a mailing-list-archive into mu4e. E.g.
>> > > because debbugs.el is slow, you might want to import the archives of
>> > > the guix-patches mailing list. This is how to do so.
>
>> > Would you mind writing this up and including it in the guix cookbook?
>>
>> Alright, I'll try to send a patch. I think I should change the
>> instructions in order to replace the unpackaged mb2md perl-script in
>> favor of the packaged mblaze tool. Afterwards, it should be straight
>> forward.
>
> I made a little shell script out of these instructions. You can find it
> attached. The script allows to either import a specified GNU
> mailing-list into a specified maildir, or update that maildir, beginning
> either from a specified YYYY-MM date or from the date of the latest
> message.
>
> For example, I used this command to initialize my mailing-list-maildirs:
>
>     guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix
>         guix-devel guix-patches help-guix; do ~/path/to/ugmltmd
>         ~/path/to/mail/$l $l 2022-01; done'
>
> And I use the following command to update my mailing-list-maildirs:
>
>     guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix
>         guix-devel guix-patches help-guix; do ~/path/to/ugmltmd
>         ~/path/to/mail/$l $l; done'
>
> I wonder if it still makes sense to add some instructions into the Guix
> cookbook. And if so, I wonder if this shell script should just be put
> there or if it should live in some public git repository somewhere.
>
> What do you think?

I would reach out to guix devel/guix-patches and try to put this little
code snippet in guix's source's /etc directory.  Then add to the
cookbook explaining how to use it.  Something like this would be
tremendously useful!

Thanks for working on it!

Joshua


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-05  8:45 ` zimoun
@ 2022-12-05 21:43   ` Mekeor Melire
  2022-12-06 11:33     ` zimoun
  2022-12-06  1:03   ` Kyle Meyer
  1 sibling, 1 reply; 15+ messages in thread
From: Mekeor Melire @ 2022-12-05 21:43 UTC (permalink / raw)
  To: zimoun; +Cc: help-guix

2022-12-05, 09:45, zimon.toutoune@gmail.com:

> Personally, I use “git clone” from a public-inbox instance [1].
>
>     git clone --mirror https://yhetil.org/guix-patches/1 \
>               guix-patches/git/1.git
>
> where ’1’ can be also replace by ’0’ for the very old ones.
>
> Then the conversion from Git commit to maildir is done by a small script
> [2], where all the job reads:
>
>     # Extract the message from each commit in the range and store it
>     # in the Maildir for notmuch to consume.
>     $git rev-list $range | while read sha; do
>         # XXXX: fatal: path 'm' does not exist in <commit>
>         # and it can also raise issues with notmuch, as:
>         # Note: Ignoring non-mail file: $maildir/new/$sha
> 	$git show $sha:m > $maildir/new/$sha
>     done
>
> (Maybe better could be done and more robust are around.)
>
>
> Even, the same can be used for reading Guix mailing lists.  New messages
> are gotten with “git fetch”.
>
>
> 1: <https://yhetil.org/guix-patches/>
> 2:
> <https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L60-67>

This is an interesting approach. Thanks for sharing.

If I understand correctly, when using public-inbox, another approach
would be to use IMAP, right? Just like it's described e.g. at
<https://yhetil.org/guix-patches/_/text/mirror/>:

    “IMAP subfolder(s) are available under:
      imaps://;AUTH=ANONYMOUS@news.yhetil.org/yhetil.gnu.guix.patches
      # each subfolder (starting with `0') holds 50K messages at most”

I'll probably try that approach out soon.

I guess, a disadvantage of using public-inbox is that it's not an
official service of the GNU Guix community.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-05  8:45 ` zimoun
  2022-12-05 21:43   ` Mekeor Melire
@ 2022-12-06  1:03   ` Kyle Meyer
  2022-12-06 12:03     ` zimoun
  2022-12-07  2:19     ` Mekeor Melire
  1 sibling, 2 replies; 15+ messages in thread
From: Kyle Meyer @ 2022-12-06  1:03 UTC (permalink / raw)
  To: zimoun; +Cc: Mekeor Melire, help-guix

On 12/05/22 09:45:41 +0100, zimoun wrote:
> Personally, I use “git clone” from a public-inbox instance [1].
>
>     git clone --mirror https://yhetil.org/guix-patches/1 \
>               guix-patches/git/1.git
>
> where ’1’ can be also replace by ’0’ for the very old ones.

In the case of guix-patches, there's not a 0.git.  I started it at 1.git
to leave open the possibility of adding 0.git with the messages I was
missing from the beginning of the list's history (from Feb 12, 2017 to
March 27, 2017).  I'm not sure I'm ever going to do so at this point
and, even if I did, reserving 0.git doesn't have much advantage over
just adding the old messages on top of the existing epoch, so I probably
should have just started it at 0.git.

> Then the conversion from Git commit to maildir is done by a small script
> [2], where all the job reads:
>
> --8<---------------cut here---------------start------------->8---
>     # Extract the message from each commit in the range and store it
>     # in the Maildir for notmuch to consume.
>     $git rev-list $range | while read sha; do
>         # XXXX: fatal: path 'm' does not exist in <commit>
>         # and it can also raise issues with notmuch, as:
>         # Note: Ignoring non-mail file: $maildir/new/$sha

A tree can either have m or d ("deleted" messages):

  https://public-inbox.org/public-inbox-v2-format.html

So you should be able to avoid this error by skipping d's.

> 	$git show $sha:m > $maildir/new/$sha
>     done
> --8<---------------cut here---------------end--------------->8---
>
> (Maybe better could be done and more robust are around.)

No need to change what works, of course, but
https://public-inbox.org/clients.html mentions l2md and impibe as tools
for converting public-inbox archives to Maildir.

 * https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
 * https://leahneukirchen.org/dotfiles/bin/impibe

In terms of cloning archives, plain cloning and fetching with Git is
fine, but, if you have public-inbox locally, you can instead use
public-inbox-clone and public-inbox-fetch, which will handle some
details for you (e.g, cloning underlying epochs and recognizing that new
epochs have been added):

 $ public-inbox-clone https://yhetil.org/guix-patches

Another option for fetching that's nice if you're mirror multiple repos
is grokmirror:

 * https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/about/
 * example setup for guix: https://yhetil.org/guix-patches/878scww903.fsf@kyleam.com/

Both grokmirror and public-inbox-clone/fetch make use of the manifests
that are published for public-inbox archives:

  $ curl -fSsL https://yhetil.org/manifest.js.gz | gzip -d | \
    jq -r 'keys | .[] | select(contains("guix"))'
  /guix-bugs/git/0.git
  /guix-devel/git/0.git
  /guix-patches/git/1.git
  /guix-science/git/0.git
  /guix-user/git/0.git

Then there's of course also public-inbox's lei (local email interface).
I won't get into that, but, for anyone interested, here are messages
where I've given some examples:

 * https://yhetil.org/emacs-devel/87wnh22w7o.fsf@kyleam.com
 * https://yhetil.org/guix-devel/87y1zcljq3.fsf@kyleam.com


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-05 21:43   ` Mekeor Melire
@ 2022-12-06 11:33     ` zimoun
  2022-12-07  2:27       ` Mekeor Melire
  0 siblings, 1 reply; 15+ messages in thread
From: zimoun @ 2022-12-06 11:33 UTC (permalink / raw)
  To: Mekeor Melire; +Cc: help-guix

Hi,

On Mon, 05 Dec 2022 at 21:43, Mekeor Melire <mekeor@posteo.de> wrote:

> I guess, a disadvantage of using public-inbox is that it's not an
> official service of the GNU Guix community.

It is up to us to have an “official” instance. :-)  Well, I remember
discussing how to setup a public-inbox instance for Guix but I fail to
find the message (probably a discussion on guix-sysadmin@gnu.org :-)).

I mean the Guix project already serves https://issues.guix.gnu.org which
is an improvement, IMHO, over https://bugs.gnu.org.  We could also have
https://lists.guix.gnu.org serving ’public-inbox’es for the various
Guix mailing lists.

Cheers,
simon


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-06  1:03   ` Kyle Meyer
@ 2022-12-06 12:03     ` zimoun
  2022-12-07  2:02       ` Kyle Meyer
  2022-12-07  2:19     ` Mekeor Melire
  1 sibling, 1 reply; 15+ messages in thread
From: zimoun @ 2022-12-06 12:03 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: Mekeor Melire, help-guix

Hi Kyle,

On Mon, 05 Dec 2022 at 20:03, Kyle Meyer <kyle@kyleam.com> wrote:

> In terms of cloning archives, plain cloning and fetching with Git is
> fine, but, if you have public-inbox locally, you can instead use
> public-inbox-clone and public-inbox-fetch, which will handle some
> details for you (e.g, cloning underlying epochs and recognizing that new
> epochs have been added):
>
>  $ public-inbox-clone https://yhetil.org/guix-patches
>
> Another option for fetching that's nice if you're mirror multiple repos
> is grokmirror:
>
>  * https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/about/
>  * example setup for guix: https://yhetil.org/guix-patches/878scww903.fsf@kyleam.com/

I fetch from yhetil.org (Git) and then convert to MailDir because I use
Notmuch as indexer (and reader :-)).  And I do not want to duplicate the
storage.  Therefore, after fetching and converting, the Git repo is
garbage collected,

--8<---------------cut here---------------start------------->8---
	echo $head > $gitdir/shallow
	$git gc --quiet --prune=all
--8<---------------cut here---------------end--------------->8---

https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L94-99

For instance, the current size of guix-patches on my machine is 152K
compared to the complete guix-patches Git repo of 277M.


Well, I have not investigated much with ’grokmirror’ and
’public-inbox-fetch’ and maybe it would also work smoothly with the same
trick; avoiding many manual plumbing.

Thanks for sharing.

Cheers,
simon


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-06 12:03     ` zimoun
@ 2022-12-07  2:02       ` Kyle Meyer
  2022-12-07 10:38         ` zimoun
  0 siblings, 1 reply; 15+ messages in thread
From: Kyle Meyer @ 2022-12-07  2:02 UTC (permalink / raw)
  To: zimoun; +Cc: Mekeor Melire, help-guix

zimoun writes:

> I fetch from yhetil.org (Git) and then convert to MailDir because I use
> Notmuch as indexer (and reader :-)).  And I do not want to duplicate the
> storage.  Therefore, after fetching and converting, the Git repo is
> garbage collected,
>
> --8<---------------cut here---------------start------------->8---
> 	echo $head > $gitdir/shallow
> 	$git gc --quiet --prune=all
> --8<---------------cut here---------------end--------------->8---
>
> https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L94-99
>
> For instance, the current size of guix-patches on my machine is 152K
> compared to the complete guix-patches Git repo of 277M.

Nice.  Maintaining a shallow clone certainly makes sense.

> Well, I have not investigated much with ’grokmirror’ and
> ’public-inbox-fetch’ and maybe it would also work smoothly with the same
> trick; avoiding many manual plumbing.

I haven't used it myself (unsurprisingly I have an interest in keeping
yhetil.org's inboxes replicated on my local machine), but I think you
can keep grokmirror clones shallow by hooking up its grok-pi-piper to
the post-update hook. Here's a relevant bit from grokmirror's docs:

--8<---------------cut here---------------start------------->8---
[DEFAULT]
# To start piping public-inbox messages into your inbox, simply
# install procmail and add the following line to your ~/.procmailrc:
#     DEFAULT=$HOME/Maildir/
# You can now read your mail with "mutt -f ~/Maildir/"
pipe = /usr/bin/procmail
# Once you've successfully piped the messages, you generally
# don't need them any more. If you set shallow = yes, then
# the repository will be configured as "shallow" and all succesffully
# processed messages will be pruned from the repo.
# This will greatly reduce disk space usage, especially on large archives.
# You can always get any number of them back, e.g. by running:
# git fetch _grokmirror master --deepen 100
shallow = yes
--8<---------------cut here---------------end--------------->8---


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-06  1:03   ` Kyle Meyer
  2022-12-06 12:03     ` zimoun
@ 2022-12-07  2:19     ` Mekeor Melire
  1 sibling, 0 replies; 15+ messages in thread
From: Mekeor Melire @ 2022-12-07  2:19 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: zimoun, help-guix

[-- Attachment #1: Type: text/plain, Size: 3936 bytes --]

2022-12-05 20:03 kyle@kyleam.com:

> On 12/05/22 09:45:41 +0100, zimoun wrote:

> > Personally, I use “git clone” from a public-inbox instance [1].
> >
> >     git clone --mirror https://yhetil.org/guix-patches/1 \
> >               guix-patches/git/1.git
> >
> > where ’1’ can be also replace by ’0’ for the very old ones.
>
> In the case of guix-patches, there's not a 0.git.  I started it at 1.git
> to leave open the possibility of adding 0.git with the messages I was
> missing from the beginning of the list's history (from Feb 12, 2017 to
> March 27, 2017).  I'm not sure I'm ever going to do so at this point
> and, even if I did, reserving 0.git doesn't have much advantage over
> just adding the old messages on top of the existing epoch, so I probably
> should have just started it at 0.git.
>
> > Then the conversion from Git commit to maildir is done by a small script
> > [2], where all the job reads:
> >
> > --8<---------------cut here---------------start------------->8---
> >     # Extract the message from each commit in the range and store it
> >     # in the Maildir for notmuch to consume.
> >     $git rev-list $range | while read sha; do
> >         # XXXX: fatal: path 'm' does not exist in <commit>
> >         # and it can also raise issues with notmuch, as:
> >         # Note: Ignoring non-mail file: $maildir/new/$sha
>
> A tree can either have m or d ("deleted" messages):
>
>   https://public-inbox.org/public-inbox-v2-format.html
>
> So you should be able to avoid this error by skipping d's.
>
> > 	$git show $sha:m > $maildir/new/$sha
> >     done
> > --8<---------------cut here---------------end--------------->8---
> >
> > (Maybe better could be done and more robust are around.)
>
> No need to change what works, of course, but
> https://public-inbox.org/clients.html mentions l2md and impibe as tools
> for converting public-inbox archives to Maildir.
>
>  * https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
>  * https://leahneukirchen.org/dotfiles/bin/impibe
>
> In terms of cloning archives, plain cloning and fetching with Git is
> fine, but, if you have public-inbox locally, you can instead use
> public-inbox-clone and public-inbox-fetch, which will handle some
> details for you (e.g, cloning underlying epochs and recognizing that new
> epochs have been added):
>
>  $ public-inbox-clone https://yhetil.org/guix-patches
>
> Another option for fetching that's nice if you're mirror multiple repos
> is grokmirror:
>
>  * https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/about/
>  * example setup for guix: https://yhetil.org/guix-patches/878scww903.fsf@kyleam.com/
>
> Both grokmirror and public-inbox-clone/fetch make use of the manifests
> that are published for public-inbox archives:
>
>   $ curl -fSsL https://yhetil.org/manifest.js.gz | gzip -d | \
>     jq -r 'keys | .[] | select(contains("guix"))'
>   /guix-bugs/git/0.git
>   /guix-devel/git/0.git
>   /guix-patches/git/1.git
>   /guix-science/git/0.git
>   /guix-user/git/0.git
>
> Then there's of course also public-inbox's lei (local email interface).
> I won't get into that, but, for anyone interested, here are messages
> where I've given some examples:
>
>  * https://yhetil.org/emacs-devel/87wnh22w7o.fsf@kyleam.com
>  * https://yhetil.org/guix-devel/87y1zcljq3.fsf@kyleam.com

Thank you both for sharing your approaches.

As I personally use isync / mbsync to fetch the emails of my
email-accounts via IMAP into maildirs (which I then proceed to index
with mu and read with mu4e), I prefer to use the same tech stack to
fetch the mailing-list archives. And I now accomplished this with the
attached isync-configuration. Note that, as Kyle noted, I needed to use
the "0" path sometimes, and the "1" path other times. Also, you might
want to set and adapt MaxMessages to fit your needs.


[-- Attachment #2: isync-config --]
[-- Type: text/plain, Size: 2064 bytes --]

IMAPAccount guix-bugs
Host news.yhetil.org
User arbitrary
AuthMechs ANONYMOUS
SSLType IMAPS
SSLVersions TLSv1.2

IMAPStore guix-bugs-remote
Account guix-bugs

MaildirStore guix-bugs-local
Inbox /home/user/path/to/maildirs/guix-bugs/inbox
Path /home/user/path/to/maildirs/guix-bugs/
SubFolders Verbatim

Channel guix-bugs
Far :guix-bugs-remote:yhetil.gnu.guix.bugs.0
Near :guix-bugs-local:
CopyArrivalDate yes
Create Near
ExpireUnread yes
# MaxMessages 100000
Sync Pull
SyncState *

IMAPAccount guix-devel
Host news.yhetil.org
User arbitrary
AuthMechs ANONYMOUS
SSLType IMAPS
SSLVersions TLSv1.2

IMAPStore guix-devel-remote
Account guix-devel

MaildirStore guix-devel-local
Inbox /home/user/path/to/maildirs/guix-devel/inbox
Path /home/user/path/to/maildirs/guix-devel/
SubFolders Verbatim

Channel guix-devel
Far :guix-devel-remote:yhetil.gnu.guix.devel.1
Near :guix-devel-local:
CopyArrivalDate yes
Create Near
ExpireUnread yes
# MaxMessages 100000
Sync Pull
SyncState *

IMAPAccount guix-help
Host news.yhetil.org
User arbitrary
AuthMechs ANONYMOUS
SSLType IMAPS
SSLVersions TLSv1.2

IMAPStore guix-help-remote
Account guix-help

MaildirStore guix-help-local
Inbox /home/user/path/to/maildirs/guix-help/inbox
Path /home/user/path/to/maildirs/guix-help/
SubFolders Verbatim

Channel guix-help
Far :guix-help-remote:yhetil.gnu.guix.user.0
Near :guix-help-local:
CopyArrivalDate yes
Create Near
ExpireUnread yes
# MaxMessages 100000
Sync Pull
SyncState *

IMAPAccount guix-patches
Host news.yhetil.org
User arbitrary
AuthMechs ANONYMOUS
SSLType IMAPS
SSLVersions TLSv1.2

IMAPStore guix-patches-remote
Account guix-patches

MaildirStore guix-patches-local
Inbox /home/user/path/to/maildirs/guix-patches/inbox
Path /home/user/path/to/maildirs/guix-patches/
SubFolders Verbatim

Channel guix-patches
Far :guix-patches-remote:yhetil.gnu.guix.patches.1
Near :guix-patches-local:
CopyArrivalDate yes
Create Near
ExpireUnread yes
# MaxMessages 100000
Sync Pull
SyncState *

Group mailing-lists
Channel guix-bugs
Channel guix-devel
Channel guix-help
Channel guix-patches

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-06 11:33     ` zimoun
@ 2022-12-07  2:27       ` Mekeor Melire
  2022-12-07 10:33         ` zimoun
  0 siblings, 1 reply; 15+ messages in thread
From: Mekeor Melire @ 2022-12-07  2:27 UTC (permalink / raw)
  To: zimoun; +Cc: help-guix

2022-12-06 12:33 zimon.toutoune@gmail.com:

> On Mon, 05 Dec 2022 at 21:43, Mekeor Melire <mekeor@posteo.de> wrote:
>
> > I guess, a disadvantage of using public-inbox is that it's not an
> > official service of the GNU Guix community.
>
> It is up to us to have an “official” instance. :-)  Well, I remember
> discussing how to setup a public-inbox instance for Guix but I fail to
> find the message (probably a discussion on guix-sysadmin@gnu.org :-)).
>
> I mean the Guix project already serves https://issues.guix.gnu.org which
> is an improvement, IMHO, over https://bugs.gnu.org.  We could also have
> https://lists.guix.gnu.org serving ’public-inbox’es for the various
> Guix mailing lists.

Yes, it'd be lovely if there were official GNU Guix public-inboxes, I
agree. Personally, I just started to use yhetil.org as of today, for my
local copy of the mailing-lists. But it'd be much nicer to rely on an
official service. I should go and find that old thread in the
guix-sysadmin list.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-07  2:27       ` Mekeor Melire
@ 2022-12-07 10:33         ` zimoun
  0 siblings, 0 replies; 15+ messages in thread
From: zimoun @ 2022-12-07 10:33 UTC (permalink / raw)
  To: Mekeor Melire; +Cc: help-guix

Hi,

On Wed, 07 Dec 2022 at 02:27, Mekeor Melire <mekeor@posteo.de> wrote:

>                   I should go and find that old thread in the
> guix-sysadmin list.

This list guix-sysadmin is a private aliases, I guess.  Maybe it could
be worth to start a new thread in guix-devel.

Cheers,
simon


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e
  2022-12-07  2:02       ` Kyle Meyer
@ 2022-12-07 10:38         ` zimoun
  0 siblings, 0 replies; 15+ messages in thread
From: zimoun @ 2022-12-07 10:38 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: Mekeor Melire, help-guix

Hi Kyle,

On Tue, 06 Dec 2022 at 21:02, Kyle Meyer <kyle@kyleam.com> wrote:

> I haven't used it myself (unsurprisingly I have an interest in keeping
> yhetil.org's inboxes replicated on my local machine), but I think you
> can keep grokmirror clones shallow by hooking up its grok-pi-piper to
> the post-update hook. Here's a relevant bit from grokmirror's docs:
>
> --8<---------------cut here---------------start------------->8---
> [DEFAULT]
> # To start piping public-inbox messages into your inbox, simply
> # install procmail and add the following line to your ~/.procmailrc:
> #     DEFAULT=$HOME/Maildir/
> # You can now read your mail with "mutt -f ~/Maildir/"
> pipe = /usr/bin/procmail
> # Once you've successfully piped the messages, you generally
> # don't need them any more. If you set shallow = yes, then
> # the repository will be configured as "shallow" and all succesffully
> # processed messages will be pruned from the repo.
> # This will greatly reduce disk space usage, especially on large archives.
> # You can always get any number of them back, e.g. by running:
> # git fetch _grokmirror master --deepen 100
> shallow = yes
> --8<---------------cut here---------------end--------------->8---

Cool!  I will give a look and report if I am happy with it. :-)

Cheers,
simon


^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2022-12-07 11:25 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-16 23:04 This is how you import a mailing list archives (e.g. guix-patches) into mu4e Mekeor Melire
2022-11-16 23:22 ` jbranso
2022-11-16 23:41   ` Mekeor Melire
2022-12-04 20:16     ` Mekeor Melire
2022-12-05 15:39       ` Joshua Branson
2022-12-05  8:45 ` zimoun
2022-12-05 21:43   ` Mekeor Melire
2022-12-06 11:33     ` zimoun
2022-12-07  2:27       ` Mekeor Melire
2022-12-07 10:33         ` zimoun
2022-12-06  1:03   ` Kyle Meyer
2022-12-06 12:03     ` zimoun
2022-12-07  2:02       ` Kyle Meyer
2022-12-07 10:38         ` zimoun
2022-12-07  2:19     ` Mekeor Melire

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).