* This is how you import a mailing list archives (e.g. guix-patches) into mu4e @ 2022-11-16 23:04 Mekeor Melire 2022-11-16 23:22 ` jbranso 2022-12-05 8:45 ` zimoun 0 siblings, 2 replies; 15+ messages in thread From: Mekeor Melire @ 2022-11-16 23:04 UTC (permalink / raw) To: help-guix Hello Guix, you might want to import a mailing-list-archive into mu4e. E.g. because debbugs.el is slow, you might want to import the archives of the guix-patches mailing list. This is how to do so. First, download the archives as in the "mbox" format -- the only format that Mailman offers: --8<---------------cut here---------------start------------->8--- cd ~/some/where for y in {2017..2022}; do for m in {01..12}; do curl --fail "https://lists.gnu.org/archive/mbox/guix-patches/$y-$m" > "$y-$m" || rm "$y-$m"; done; done --8<---------------cut here---------------end--------------->8--- Now, download and run a perl-tool named "mb2md" to convert the mboxes into maildirs: --8<---------------cut here---------------start------------->8--- mkdir tool maildir curl http://batleth.sapienti-sat.org/projects/mb2md/mb2md-3.20.pl.gz \ --output tool/mb2md.pl.gz gzip -d tool/mb2md.pl.gz chmod 755 tool/mb2md.pl guix shell perl perl-datetime perl-timedate for f in $(realpath ./mbox)/*; do perl tool/mb2md.pl -s "$f" -d $(realpath mailbox); done exit --8<---------------cut here---------------end--------------->8--- Next, you might want to mark all mails in that maildir as "read": --8<---------------cut here---------------start------------->8--- guix shell mblaze mlist -s ./mailbox | mflag -S exit --8<---------------cut here---------------end--------------->8--- Finally, move that maildir to its destination: --8<---------------cut here---------------start------------->8--- mv maildir ~/path/to/maildirs/guix-patches --8<---------------cut here---------------end--------------->8--- Enjoy! ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-11-16 23:04 This is how you import a mailing list archives (e.g. guix-patches) into mu4e Mekeor Melire @ 2022-11-16 23:22 ` jbranso 2022-11-16 23:41 ` Mekeor Melire 2022-12-05 8:45 ` zimoun 1 sibling, 1 reply; 15+ messages in thread From: jbranso @ 2022-11-16 23:22 UTC (permalink / raw) To: Mekeor Melire, help-guix November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote: > Hello Guix, > > you might want to import a mailing-list-archive into mu4e. E.g. because > debbugs.el is slow, you might want to import the archives of the > guix-patches mailing list. This is how to do so. Hey thanks for sharing! How about we make this guide a bit more permanent? Would you mind writing this up and including it in the guix cookbook? Once you submit the patch, please CC me. I'll review it for you. Thanks again! Joshua > > First, download the archives as in the "mbox" format -- the only format > that Mailman offers: > > --8<---------------cut here---------------start------------->8--- > cd ~/some/where > for y in {2017..2022}; do > for m in {01..12}; do > curl --fail "https://lists.gnu.org/archive/mbox/guix-patches/$y-$m" > "$y-$m" || rm "$y-$m"; > done; > done > --8<---------------cut here---------------end--------------->8--- > > Now, download and run a perl-tool named "mb2md" to convert the mboxes > into maildirs: > > --8<---------------cut here---------------start------------->8--- > mkdir tool maildir > curl http://batleth.sapienti-sat.org/projects/mb2md/mb2md-3.20.pl.gz \ > --output tool/mb2md.pl.gz > gzip -d tool/mb2md.pl.gz > chmod 755 tool/mb2md.pl > guix shell perl perl-datetime perl-timedate > for f in $(realpath ./mbox)/*; do > perl tool/mb2md.pl -s "$f" -d $(realpath mailbox); > done > exit > --8<---------------cut here---------------end--------------->8--- > > Next, you might want to mark all mails in that maildir as "read": > > --8<---------------cut here---------------start------------->8--- > guix shell mblaze > mlist -s ./mailbox | mflag -S > exit > --8<---------------cut here---------------end--------------->8--- > > Finally, move that maildir to its destination: > > --8<---------------cut here---------------start------------->8--- > mv maildir ~/path/to/maildirs/guix-patches > --8<---------------cut here---------------end--------------->8--- > > Enjoy! ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-11-16 23:22 ` jbranso @ 2022-11-16 23:41 ` Mekeor Melire 2022-12-04 20:16 ` Mekeor Melire 0 siblings, 1 reply; 15+ messages in thread From: Mekeor Melire @ 2022-11-16 23:41 UTC (permalink / raw) To: jbranso; +Cc: help-guix 2022-11-16 / 23:22 / jbranso@dismail.de: > November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote: > > > Hello Guix, > > > > you might want to import a mailing-list-archive into mu4e. E.g. > > because debbugs.el is slow, you might want to import the archives of > > the guix-patches mailing list. This is how to do so. > > Hey Hello :) > thanks for sharing! How about we make this guide a bit more permanent? Good idea :) > Would you mind writing this up and including it in the guix cookbook? Alright, I'll try to send a patch. I think I should change the instructions in order to replace the unpackaged mb2md perl-script in favor of the packaged mblaze tool. Afterwards, it should be straight forward. > Once you submit the patch, please CC me. I'll review it for you. Alright :) > Thanks again! > Joshua Thank you, too. Mekeor ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-11-16 23:41 ` Mekeor Melire @ 2022-12-04 20:16 ` Mekeor Melire 2022-12-05 15:39 ` Joshua Branson 0 siblings, 1 reply; 15+ messages in thread From: Mekeor Melire @ 2022-12-04 20:16 UTC (permalink / raw) To: Mekeor Melire; +Cc: jbranso, help-guix [-- Attachment #1: Type: text/plain, Size: 1654 bytes --] 2022-11-16 / 23:41 / mekeor@posteo.de: > 2022-11-16 / 23:22 / jbranso@dismail.de: > > November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote: > > > you might want to import a mailing-list-archive into mu4e. E.g. > > > because debbugs.el is slow, you might want to import the archives of > > > the guix-patches mailing list. This is how to do so. > > Would you mind writing this up and including it in the guix cookbook? > > Alright, I'll try to send a patch. I think I should change the > instructions in order to replace the unpackaged mb2md perl-script in > favor of the packaged mblaze tool. Afterwards, it should be straight > forward. I made a little shell script out of these instructions. You can find it attached. The script allows to either import a specified GNU mailing-list into a specified maildir, or update that maildir, beginning either from a specified YYYY-MM date or from the date of the latest message. For example, I used this command to initialize my mailing-list-maildirs: guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix guix-devel guix-patches help-guix; do ~/path/to/ugmltmd ~/path/to/mail/$l $l 2022-01; done' And I use the following command to update my mailing-list-maildirs: guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix guix-devel guix-patches help-guix; do ~/path/to/ugmltmd ~/path/to/mail/$l $l; done' I wonder if it still makes sense to add some instructions into the Guix cookbook. And if so, I wonder if this shell script should just be put there or if it should live in some public git repository somewhere. What do you think? [-- Attachment #2: ugmltmd --] [-- Type: application/x-sh, Size: 1831 bytes --] ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-04 20:16 ` Mekeor Melire @ 2022-12-05 15:39 ` Joshua Branson 0 siblings, 0 replies; 15+ messages in thread From: Joshua Branson @ 2022-12-05 15:39 UTC (permalink / raw) To: Mekeor Melire; +Cc: help-guix Mekeor Melire <mekeor@posteo.de> writes: > 2022-11-16 / 23:41 / mekeor@posteo.de: >> 2022-11-16 / 23:22 / jbranso@dismail.de: >> > November 16, 2022 6:15 PM, "Mekeor Melire" <mekeor@posteo.de> wrote: > >> > > you might want to import a mailing-list-archive into mu4e. E.g. >> > > because debbugs.el is slow, you might want to import the archives of >> > > the guix-patches mailing list. This is how to do so. > >> > Would you mind writing this up and including it in the guix cookbook? >> >> Alright, I'll try to send a patch. I think I should change the >> instructions in order to replace the unpackaged mb2md perl-script in >> favor of the packaged mblaze tool. Afterwards, it should be straight >> forward. > > I made a little shell script out of these instructions. You can find it > attached. The script allows to either import a specified GNU > mailing-list into a specified maildir, or update that maildir, beginning > either from a specified YYYY-MM date or from the date of the latest > message. > > For example, I used this command to initialize my mailing-list-maildirs: > > guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix > guix-devel guix-patches help-guix; do ~/path/to/ugmltmd > ~/path/to/mail/$l $l 2022-01; done' > > And I use the following command to update my mailing-list-maildirs: > > guix shell mblaze html-xml-utils -- sh -c 'for l in bug-guix > guix-devel guix-patches help-guix; do ~/path/to/ugmltmd > ~/path/to/mail/$l $l; done' > > I wonder if it still makes sense to add some instructions into the Guix > cookbook. And if so, I wonder if this shell script should just be put > there or if it should live in some public git repository somewhere. > > What do you think? I would reach out to guix devel/guix-patches and try to put this little code snippet in guix's source's /etc directory. Then add to the cookbook explaining how to use it. Something like this would be tremendously useful! Thanks for working on it! Joshua ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-11-16 23:04 This is how you import a mailing list archives (e.g. guix-patches) into mu4e Mekeor Melire 2022-11-16 23:22 ` jbranso @ 2022-12-05 8:45 ` zimoun 2022-12-05 21:43 ` Mekeor Melire 2022-12-06 1:03 ` Kyle Meyer 1 sibling, 2 replies; 15+ messages in thread From: zimoun @ 2022-12-05 8:45 UTC (permalink / raw) To: Mekeor Melire, help-guix Hi, On Wed, 16 Nov 2022 at 23:04, Mekeor Melire <mekeor@posteo.de> wrote: > you might want to import a mailing-list-archive into mu4e. E.g. because > debbugs.el is slow, you might want to import the archives of the > guix-patches mailing list. This is how to do so. Personally, I use “git clone” from a public-inbox instance [1]. git clone --mirror https://yhetil.org/guix-patches/1 \ guix-patches/git/1.git where ’1’ can be also replace by ’0’ for the very old ones. Then the conversion from Git commit to maildir is done by a small script [2], where all the job reads: --8<---------------cut here---------------start------------->8--- # Extract the message from each commit in the range and store it # in the Maildir for notmuch to consume. $git rev-list $range | while read sha; do # XXXX: fatal: path 'm' does not exist in <commit> # and it can also raise issues with notmuch, as: # Note: Ignoring non-mail file: $maildir/new/$sha $git show $sha:m > $maildir/new/$sha done --8<---------------cut here---------------end--------------->8--- (Maybe better could be done and more robust are around.) Even, the same can be used for reading Guix mailing lists. New messages are gotten with “git fetch”. 1: <https://yhetil.org/guix-patches/> 2: <https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L60-67> Cheers, simon ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-05 8:45 ` zimoun @ 2022-12-05 21:43 ` Mekeor Melire 2022-12-06 11:33 ` zimoun 2022-12-06 1:03 ` Kyle Meyer 1 sibling, 1 reply; 15+ messages in thread From: Mekeor Melire @ 2022-12-05 21:43 UTC (permalink / raw) To: zimoun; +Cc: help-guix 2022-12-05, 09:45, zimon.toutoune@gmail.com: > Personally, I use “git clone” from a public-inbox instance [1]. > > git clone --mirror https://yhetil.org/guix-patches/1 \ > guix-patches/git/1.git > > where ’1’ can be also replace by ’0’ for the very old ones. > > Then the conversion from Git commit to maildir is done by a small script > [2], where all the job reads: > > # Extract the message from each commit in the range and store it > # in the Maildir for notmuch to consume. > $git rev-list $range | while read sha; do > # XXXX: fatal: path 'm' does not exist in <commit> > # and it can also raise issues with notmuch, as: > # Note: Ignoring non-mail file: $maildir/new/$sha > $git show $sha:m > $maildir/new/$sha > done > > (Maybe better could be done and more robust are around.) > > > Even, the same can be used for reading Guix mailing lists. New messages > are gotten with “git fetch”. > > > 1: <https://yhetil.org/guix-patches/> > 2: > <https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L60-67> This is an interesting approach. Thanks for sharing. If I understand correctly, when using public-inbox, another approach would be to use IMAP, right? Just like it's described e.g. at <https://yhetil.org/guix-patches/_/text/mirror/>: “IMAP subfolder(s) are available under: imaps://;AUTH=ANONYMOUS@news.yhetil.org/yhetil.gnu.guix.patches # each subfolder (starting with `0') holds 50K messages at most” I'll probably try that approach out soon. I guess, a disadvantage of using public-inbox is that it's not an official service of the GNU Guix community. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-05 21:43 ` Mekeor Melire @ 2022-12-06 11:33 ` zimoun 2022-12-07 2:27 ` Mekeor Melire 0 siblings, 1 reply; 15+ messages in thread From: zimoun @ 2022-12-06 11:33 UTC (permalink / raw) To: Mekeor Melire; +Cc: help-guix Hi, On Mon, 05 Dec 2022 at 21:43, Mekeor Melire <mekeor@posteo.de> wrote: > I guess, a disadvantage of using public-inbox is that it's not an > official service of the GNU Guix community. It is up to us to have an “official” instance. :-) Well, I remember discussing how to setup a public-inbox instance for Guix but I fail to find the message (probably a discussion on guix-sysadmin@gnu.org :-)). I mean the Guix project already serves https://issues.guix.gnu.org which is an improvement, IMHO, over https://bugs.gnu.org. We could also have https://lists.guix.gnu.org serving ’public-inbox’es for the various Guix mailing lists. Cheers, simon ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-06 11:33 ` zimoun @ 2022-12-07 2:27 ` Mekeor Melire 2022-12-07 10:33 ` zimoun 0 siblings, 1 reply; 15+ messages in thread From: Mekeor Melire @ 2022-12-07 2:27 UTC (permalink / raw) To: zimoun; +Cc: help-guix 2022-12-06 12:33 zimon.toutoune@gmail.com: > On Mon, 05 Dec 2022 at 21:43, Mekeor Melire <mekeor@posteo.de> wrote: > > > I guess, a disadvantage of using public-inbox is that it's not an > > official service of the GNU Guix community. > > It is up to us to have an “official” instance. :-) Well, I remember > discussing how to setup a public-inbox instance for Guix but I fail to > find the message (probably a discussion on guix-sysadmin@gnu.org :-)). > > I mean the Guix project already serves https://issues.guix.gnu.org which > is an improvement, IMHO, over https://bugs.gnu.org. We could also have > https://lists.guix.gnu.org serving ’public-inbox’es for the various > Guix mailing lists. Yes, it'd be lovely if there were official GNU Guix public-inboxes, I agree. Personally, I just started to use yhetil.org as of today, for my local copy of the mailing-lists. But it'd be much nicer to rely on an official service. I should go and find that old thread in the guix-sysadmin list. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-07 2:27 ` Mekeor Melire @ 2022-12-07 10:33 ` zimoun 0 siblings, 0 replies; 15+ messages in thread From: zimoun @ 2022-12-07 10:33 UTC (permalink / raw) To: Mekeor Melire; +Cc: help-guix Hi, On Wed, 07 Dec 2022 at 02:27, Mekeor Melire <mekeor@posteo.de> wrote: > I should go and find that old thread in the > guix-sysadmin list. This list guix-sysadmin is a private aliases, I guess. Maybe it could be worth to start a new thread in guix-devel. Cheers, simon ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-05 8:45 ` zimoun 2022-12-05 21:43 ` Mekeor Melire @ 2022-12-06 1:03 ` Kyle Meyer 2022-12-06 12:03 ` zimoun 2022-12-07 2:19 ` Mekeor Melire 1 sibling, 2 replies; 15+ messages in thread From: Kyle Meyer @ 2022-12-06 1:03 UTC (permalink / raw) To: zimoun; +Cc: Mekeor Melire, help-guix On 12/05/22 09:45:41 +0100, zimoun wrote: > Personally, I use “git clone” from a public-inbox instance [1]. > > git clone --mirror https://yhetil.org/guix-patches/1 \ > guix-patches/git/1.git > > where ’1’ can be also replace by ’0’ for the very old ones. In the case of guix-patches, there's not a 0.git. I started it at 1.git to leave open the possibility of adding 0.git with the messages I was missing from the beginning of the list's history (from Feb 12, 2017 to March 27, 2017). I'm not sure I'm ever going to do so at this point and, even if I did, reserving 0.git doesn't have much advantage over just adding the old messages on top of the existing epoch, so I probably should have just started it at 0.git. > Then the conversion from Git commit to maildir is done by a small script > [2], where all the job reads: > > --8<---------------cut here---------------start------------->8--- > # Extract the message from each commit in the range and store it > # in the Maildir for notmuch to consume. > $git rev-list $range | while read sha; do > # XXXX: fatal: path 'm' does not exist in <commit> > # and it can also raise issues with notmuch, as: > # Note: Ignoring non-mail file: $maildir/new/$sha A tree can either have m or d ("deleted" messages): https://public-inbox.org/public-inbox-v2-format.html So you should be able to avoid this error by skipping d's. > $git show $sha:m > $maildir/new/$sha > done > --8<---------------cut here---------------end--------------->8--- > > (Maybe better could be done and more robust are around.) No need to change what works, of course, but https://public-inbox.org/clients.html mentions l2md and impibe as tools for converting public-inbox archives to Maildir. * https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git * https://leahneukirchen.org/dotfiles/bin/impibe In terms of cloning archives, plain cloning and fetching with Git is fine, but, if you have public-inbox locally, you can instead use public-inbox-clone and public-inbox-fetch, which will handle some details for you (e.g, cloning underlying epochs and recognizing that new epochs have been added): $ public-inbox-clone https://yhetil.org/guix-patches Another option for fetching that's nice if you're mirror multiple repos is grokmirror: * https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/about/ * example setup for guix: https://yhetil.org/guix-patches/878scww903.fsf@kyleam.com/ Both grokmirror and public-inbox-clone/fetch make use of the manifests that are published for public-inbox archives: $ curl -fSsL https://yhetil.org/manifest.js.gz | gzip -d | \ jq -r 'keys | .[] | select(contains("guix"))' /guix-bugs/git/0.git /guix-devel/git/0.git /guix-patches/git/1.git /guix-science/git/0.git /guix-user/git/0.git Then there's of course also public-inbox's lei (local email interface). I won't get into that, but, for anyone interested, here are messages where I've given some examples: * https://yhetil.org/emacs-devel/87wnh22w7o.fsf@kyleam.com * https://yhetil.org/guix-devel/87y1zcljq3.fsf@kyleam.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-06 1:03 ` Kyle Meyer @ 2022-12-06 12:03 ` zimoun 2022-12-07 2:02 ` Kyle Meyer 2022-12-07 2:19 ` Mekeor Melire 1 sibling, 1 reply; 15+ messages in thread From: zimoun @ 2022-12-06 12:03 UTC (permalink / raw) To: Kyle Meyer; +Cc: Mekeor Melire, help-guix Hi Kyle, On Mon, 05 Dec 2022 at 20:03, Kyle Meyer <kyle@kyleam.com> wrote: > In terms of cloning archives, plain cloning and fetching with Git is > fine, but, if you have public-inbox locally, you can instead use > public-inbox-clone and public-inbox-fetch, which will handle some > details for you (e.g, cloning underlying epochs and recognizing that new > epochs have been added): > > $ public-inbox-clone https://yhetil.org/guix-patches > > Another option for fetching that's nice if you're mirror multiple repos > is grokmirror: > > * https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/about/ > * example setup for guix: https://yhetil.org/guix-patches/878scww903.fsf@kyleam.com/ I fetch from yhetil.org (Git) and then convert to MailDir because I use Notmuch as indexer (and reader :-)). And I do not want to duplicate the storage. Therefore, after fetching and converting, the Git repo is garbage collected, --8<---------------cut here---------------start------------->8--- echo $head > $gitdir/shallow $git gc --quiet --prune=all --8<---------------cut here---------------end--------------->8--- https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L94-99 For instance, the current size of guix-patches on my machine is 152K compared to the complete guix-patches Git repo of 277M. Well, I have not investigated much with ’grokmirror’ and ’public-inbox-fetch’ and maybe it would also work smoothly with the same trick; avoiding many manual plumbing. Thanks for sharing. Cheers, simon ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-06 12:03 ` zimoun @ 2022-12-07 2:02 ` Kyle Meyer 2022-12-07 10:38 ` zimoun 0 siblings, 1 reply; 15+ messages in thread From: Kyle Meyer @ 2022-12-07 2:02 UTC (permalink / raw) To: zimoun; +Cc: Mekeor Melire, help-guix zimoun writes: > I fetch from yhetil.org (Git) and then convert to MailDir because I use > Notmuch as indexer (and reader :-)). And I do not want to duplicate the > storage. Therefore, after fetching and converting, the Git repo is > garbage collected, > > --8<---------------cut here---------------start------------->8--- > echo $head > $gitdir/shallow > $git gc --quiet --prune=all > --8<---------------cut here---------------end--------------->8--- > > https://gitlab.com/zimoun/my-conf/-/blob/master/.local/bin/sync-public-inboxes.sh#L94-99 > > For instance, the current size of guix-patches on my machine is 152K > compared to the complete guix-patches Git repo of 277M. Nice. Maintaining a shallow clone certainly makes sense. > Well, I have not investigated much with ’grokmirror’ and > ’public-inbox-fetch’ and maybe it would also work smoothly with the same > trick; avoiding many manual plumbing. I haven't used it myself (unsurprisingly I have an interest in keeping yhetil.org's inboxes replicated on my local machine), but I think you can keep grokmirror clones shallow by hooking up its grok-pi-piper to the post-update hook. Here's a relevant bit from grokmirror's docs: --8<---------------cut here---------------start------------->8--- [DEFAULT] # To start piping public-inbox messages into your inbox, simply # install procmail and add the following line to your ~/.procmailrc: # DEFAULT=$HOME/Maildir/ # You can now read your mail with "mutt -f ~/Maildir/" pipe = /usr/bin/procmail # Once you've successfully piped the messages, you generally # don't need them any more. If you set shallow = yes, then # the repository will be configured as "shallow" and all succesffully # processed messages will be pruned from the repo. # This will greatly reduce disk space usage, especially on large archives. # You can always get any number of them back, e.g. by running: # git fetch _grokmirror master --deepen 100 shallow = yes --8<---------------cut here---------------end--------------->8--- ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-07 2:02 ` Kyle Meyer @ 2022-12-07 10:38 ` zimoun 0 siblings, 0 replies; 15+ messages in thread From: zimoun @ 2022-12-07 10:38 UTC (permalink / raw) To: Kyle Meyer; +Cc: Mekeor Melire, help-guix Hi Kyle, On Tue, 06 Dec 2022 at 21:02, Kyle Meyer <kyle@kyleam.com> wrote: > I haven't used it myself (unsurprisingly I have an interest in keeping > yhetil.org's inboxes replicated on my local machine), but I think you > can keep grokmirror clones shallow by hooking up its grok-pi-piper to > the post-update hook. Here's a relevant bit from grokmirror's docs: > > --8<---------------cut here---------------start------------->8--- > [DEFAULT] > # To start piping public-inbox messages into your inbox, simply > # install procmail and add the following line to your ~/.procmailrc: > # DEFAULT=$HOME/Maildir/ > # You can now read your mail with "mutt -f ~/Maildir/" > pipe = /usr/bin/procmail > # Once you've successfully piped the messages, you generally > # don't need them any more. If you set shallow = yes, then > # the repository will be configured as "shallow" and all succesffully > # processed messages will be pruned from the repo. > # This will greatly reduce disk space usage, especially on large archives. > # You can always get any number of them back, e.g. by running: > # git fetch _grokmirror master --deepen 100 > shallow = yes > --8<---------------cut here---------------end--------------->8--- Cool! I will give a look and report if I am happy with it. :-) Cheers, simon ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: This is how you import a mailing list archives (e.g. guix-patches) into mu4e 2022-12-06 1:03 ` Kyle Meyer 2022-12-06 12:03 ` zimoun @ 2022-12-07 2:19 ` Mekeor Melire 1 sibling, 0 replies; 15+ messages in thread From: Mekeor Melire @ 2022-12-07 2:19 UTC (permalink / raw) To: Kyle Meyer; +Cc: zimoun, help-guix [-- Attachment #1: Type: text/plain, Size: 3936 bytes --] 2022-12-05 20:03 kyle@kyleam.com: > On 12/05/22 09:45:41 +0100, zimoun wrote: > > Personally, I use “git clone” from a public-inbox instance [1]. > > > > git clone --mirror https://yhetil.org/guix-patches/1 \ > > guix-patches/git/1.git > > > > where ’1’ can be also replace by ’0’ for the very old ones. > > In the case of guix-patches, there's not a 0.git. I started it at 1.git > to leave open the possibility of adding 0.git with the messages I was > missing from the beginning of the list's history (from Feb 12, 2017 to > March 27, 2017). I'm not sure I'm ever going to do so at this point > and, even if I did, reserving 0.git doesn't have much advantage over > just adding the old messages on top of the existing epoch, so I probably > should have just started it at 0.git. > > > Then the conversion from Git commit to maildir is done by a small script > > [2], where all the job reads: > > > > --8<---------------cut here---------------start------------->8--- > > # Extract the message from each commit in the range and store it > > # in the Maildir for notmuch to consume. > > $git rev-list $range | while read sha; do > > # XXXX: fatal: path 'm' does not exist in <commit> > > # and it can also raise issues with notmuch, as: > > # Note: Ignoring non-mail file: $maildir/new/$sha > > A tree can either have m or d ("deleted" messages): > > https://public-inbox.org/public-inbox-v2-format.html > > So you should be able to avoid this error by skipping d's. > > > $git show $sha:m > $maildir/new/$sha > > done > > --8<---------------cut here---------------end--------------->8--- > > > > (Maybe better could be done and more robust are around.) > > No need to change what works, of course, but > https://public-inbox.org/clients.html mentions l2md and impibe as tools > for converting public-inbox archives to Maildir. > > * https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git > * https://leahneukirchen.org/dotfiles/bin/impibe > > In terms of cloning archives, plain cloning and fetching with Git is > fine, but, if you have public-inbox locally, you can instead use > public-inbox-clone and public-inbox-fetch, which will handle some > details for you (e.g, cloning underlying epochs and recognizing that new > epochs have been added): > > $ public-inbox-clone https://yhetil.org/guix-patches > > Another option for fetching that's nice if you're mirror multiple repos > is grokmirror: > > * https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git/about/ > * example setup for guix: https://yhetil.org/guix-patches/878scww903.fsf@kyleam.com/ > > Both grokmirror and public-inbox-clone/fetch make use of the manifests > that are published for public-inbox archives: > > $ curl -fSsL https://yhetil.org/manifest.js.gz | gzip -d | \ > jq -r 'keys | .[] | select(contains("guix"))' > /guix-bugs/git/0.git > /guix-devel/git/0.git > /guix-patches/git/1.git > /guix-science/git/0.git > /guix-user/git/0.git > > Then there's of course also public-inbox's lei (local email interface). > I won't get into that, but, for anyone interested, here are messages > where I've given some examples: > > * https://yhetil.org/emacs-devel/87wnh22w7o.fsf@kyleam.com > * https://yhetil.org/guix-devel/87y1zcljq3.fsf@kyleam.com Thank you both for sharing your approaches. As I personally use isync / mbsync to fetch the emails of my email-accounts via IMAP into maildirs (which I then proceed to index with mu and read with mu4e), I prefer to use the same tech stack to fetch the mailing-list archives. And I now accomplished this with the attached isync-configuration. Note that, as Kyle noted, I needed to use the "0" path sometimes, and the "1" path other times. Also, you might want to set and adapt MaxMessages to fit your needs. [-- Attachment #2: isync-config --] [-- Type: text/plain, Size: 2064 bytes --] IMAPAccount guix-bugs Host news.yhetil.org User arbitrary AuthMechs ANONYMOUS SSLType IMAPS SSLVersions TLSv1.2 IMAPStore guix-bugs-remote Account guix-bugs MaildirStore guix-bugs-local Inbox /home/user/path/to/maildirs/guix-bugs/inbox Path /home/user/path/to/maildirs/guix-bugs/ SubFolders Verbatim Channel guix-bugs Far :guix-bugs-remote:yhetil.gnu.guix.bugs.0 Near :guix-bugs-local: CopyArrivalDate yes Create Near ExpireUnread yes # MaxMessages 100000 Sync Pull SyncState * IMAPAccount guix-devel Host news.yhetil.org User arbitrary AuthMechs ANONYMOUS SSLType IMAPS SSLVersions TLSv1.2 IMAPStore guix-devel-remote Account guix-devel MaildirStore guix-devel-local Inbox /home/user/path/to/maildirs/guix-devel/inbox Path /home/user/path/to/maildirs/guix-devel/ SubFolders Verbatim Channel guix-devel Far :guix-devel-remote:yhetil.gnu.guix.devel.1 Near :guix-devel-local: CopyArrivalDate yes Create Near ExpireUnread yes # MaxMessages 100000 Sync Pull SyncState * IMAPAccount guix-help Host news.yhetil.org User arbitrary AuthMechs ANONYMOUS SSLType IMAPS SSLVersions TLSv1.2 IMAPStore guix-help-remote Account guix-help MaildirStore guix-help-local Inbox /home/user/path/to/maildirs/guix-help/inbox Path /home/user/path/to/maildirs/guix-help/ SubFolders Verbatim Channel guix-help Far :guix-help-remote:yhetil.gnu.guix.user.0 Near :guix-help-local: CopyArrivalDate yes Create Near ExpireUnread yes # MaxMessages 100000 Sync Pull SyncState * IMAPAccount guix-patches Host news.yhetil.org User arbitrary AuthMechs ANONYMOUS SSLType IMAPS SSLVersions TLSv1.2 IMAPStore guix-patches-remote Account guix-patches MaildirStore guix-patches-local Inbox /home/user/path/to/maildirs/guix-patches/inbox Path /home/user/path/to/maildirs/guix-patches/ SubFolders Verbatim Channel guix-patches Far :guix-patches-remote:yhetil.gnu.guix.patches.1 Near :guix-patches-local: CopyArrivalDate yes Create Near ExpireUnread yes # MaxMessages 100000 Sync Pull SyncState * Group mailing-lists Channel guix-bugs Channel guix-devel Channel guix-help Channel guix-patches ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2022-12-07 11:25 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-16 23:04 This is how you import a mailing list archives (e.g. guix-patches) into mu4e Mekeor Melire 2022-11-16 23:22 ` jbranso 2022-11-16 23:41 ` Mekeor Melire 2022-12-04 20:16 ` Mekeor Melire 2022-12-05 15:39 ` Joshua Branson 2022-12-05 8:45 ` zimoun 2022-12-05 21:43 ` Mekeor Melire 2022-12-06 11:33 ` zimoun 2022-12-07 2:27 ` Mekeor Melire 2022-12-07 10:33 ` zimoun 2022-12-06 1:03 ` Kyle Meyer 2022-12-06 12:03 ` zimoun 2022-12-07 2:02 ` Kyle Meyer 2022-12-07 10:38 ` zimoun 2022-12-07 2:19 ` Mekeor Melire
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.