Mumi, public-inbox and tools

Mumi, public-inbox and tools

[zimoun]

Hi,

Recently, we discussed [1] in «public-inbox/elfeed -> Maildir bridge
(was: Incentives for review)» some tips to ease the workflow dealing
with the various lists and their volume.

Kyle wrote some Emacs glue around public-inbox, named piem [2].

One cool feature exposed by Kyle in [1] is being able to inject the
complete thread to a Maildir.

For instead, let configure piem with just:

--8<---------------cut here---------------start------------->8---
(setq
 piem-inboxes '(("guix-bugs"
                  :url "https://yhetil.org/guix-bugs/"
                  :maildir "/tmp/mail/guix-bugs"
                  )
                ("guix-patches"
                  :url "https://yhetil.org/guix-patches/"
                  :maildir "/tmp/mail/guix-patches"
                  )))
(add-hook 'eww-mode-hook #'piem-eww-mode)                  
--8<---------------cut here---------------end--------------->8---

then, let create the maildirs:

    mkdir -p /tmp/mail/guix-{bugs,patches}/{new,cur,tmp}

Last, just search for a number id:

    M-x eww
    https://yhetil.org/guix-patches/?q=a:39258

then click to one, and

    M-x piem-inject-thread-into-maildir

paf!  The 126 messages of the patch submission #39258 are now in my
local maildir and I can read them using my favorite mail reader; say
Emacs-notmuch. ;-)

It completely replace my usage of the Debian scripts ’bts’ and
’mailscripts’.  Neat!  Because it was Perl… :-)


The issue is that it is going via a public-inbox query [4]:

    a:       match within the To, Cc, and From headers

where it could be nice to directly use Mumi with:

    M-x eww
    http://issues.guix.gnu.org/issue/39258

However, then ’piem-inject-thread-into-maildir’ cannot find the
Message-ID even if many are visible:

--8<---------------cut here---------------start------------->8---
Arun Isaac wrote on 23 Jan 2020 20:51
SVG Image
Recipients:(address . guix-patches@gnu.org)
Message-ID:cu7pnfaar36.fsf@systemreboot.net
--8<---------------cut here---------------end--------------->8---


Well, it could be nice if I could hook and directly import to my local
maildir the thread.


WDYT?  What is missing to bridge the current features of Mumi via
issues.guix.gnu.org, of public-inbox via yhetil.org/guix and the piem
glue?


It could be nice to:

 1. run “guix issues 12345” and get back something ready (as mbox or
 maildir) to go to a mail reader, or pipe to Git maybe.

 2. integrate such to Emacs-Guix.
 

1: <https://yhetil.org/guix-devel/87a6izsoio.fsf@kyleam.com/>
2: <https://git.kyleam.com/piem/>
3: <https://docs.kyleam.com/piem/Injecting-messages-into-a-Maildir-directory.html>
4: <https://yhetil.org/guix-bugs/_/text/help/>


Cheers,
simon

Re: Mumi, public-inbox and tools

[Arun Isaac]

Hi zimoun,

> WDYT?  What is missing to bridge the current features of Mumi via
> issues.guix.gnu.org,

mumi indexes the full text and selected headers (the date, for example)
of every email it receives. But, I don't know if it indexes the
Message-ID header. If it does, then referring to emails by their
Message-ID and retrieving all emails of a bug report should be easy to
implement. If not, we'll just have to patch mumi to index the
Message-ID. More work, but still very much doable.

> of public-inbox via yhetil.org/guix and the piem glue?

I haven't thought carefully about public-inbox integration. But, if we
can do everything in mumi, do we need public-inbox at all?

But, of course, no harm implementing public-inbox integration as an
optional add-on for those who prefer it.

> It could be nice to:
>
>  1. run “guix issues 12345” and get back something ready (as mbox or
>  maildir) to go to a mail reader, or pipe to Git maybe.

Agreed!

Cheers!
Arun

Re: Mumi, public-inbox and tools

[zimoun]

Hi Arun,

On Tue, 03 May 2022 at 23:20, Arun Isaac <arunisaac@systemreboot.net> wrote:

>> of public-inbox via yhetil.org/guix and the piem glue?
>
> I haven't thought carefully about public-inbox integration. But, if we
> can do everything in mumi, do we need public-inbox at all?

Well, some advantages I see after using a bit the public-inbox instance
<https://yhetil.org/> mirroring some lists.

1. The web interface,

  a) works well with “poor“ web browser as EWW
     (whereas I prefer Mumi with full featured browser ;-))
  b) show the complete thread
  c) powerful and quick search

2. The local mirror,

  a) ‘git pull’, so do not pollute my email inbox   
  b) I pull and so read whenever I want, not when messages are
    delivered
  c) I index and read using whatever fits my workflow
  d) the public-inbox server provides NNTP for Gnus folks

3. Bridge the present with the past

  assume I start to follow guix-devel on Wed, 30 Mar 2022

  and I locally read with my favorite Email reader this message:
  <https://yhetil.org/guix/877d8bwz1l.fsf@gnu.org>

  I am interested and I would like to know more:
  a) I can read using my web browser
  b) I can download the complete thread and read offline
     (it requires some glue code, as ’lei’ or ’piem’; IIUC)

Using the “official” interface, it is really boring.  For instance, I
read this message
<https://lists.gnu.org/archive/html/help-guix/2022-03/msg00039.html>,
then one needs some motivation to find the message of this reply; since
it had not been sent the same month.


Mumi is about bugs and another interface for Debbugs.  Therefore, we
cannot do everything with Mumi. ;-)

Maybe, what could help, IMHO, would to have the various Guix mailing
lists as public-inboxes under, say https://lists.guix.gnu.org and Mumi
could use this instance, eventually, or bridge.  The aim would be a
flexible interface but still uniform.

And I do not speak about other tools from the ecosystem as B4,

    <https://git.kernel.org/pub/scm/utils/b4/b4.git/tree/README.rst>


Cheers,
simon

Re: Mumi, public-inbox and tools

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

Hello!

The attached super-early-draft ‘guix review’ script fetches mboxes from
mumi—that part is OK.

What’s more difficult is the rest: determining what’s a patch and what’s
not (I gather this is something that b4 and Patchwork handle nicely),
getting the latest patch series, things like that.

Then there’s the big about applying patches in a branch, and optionally
finding out the package differences so the tool can provide feedback and
possibly start building things (I think the Guix Data Service has code
to help with that.)

I think we could have, as a first step, a command that allows us to run:

  ./pre-inst-env guix review 54874

and that fetches patches, applies them, and ideally reports on the
difference and what needs to be built.

It’s a relatively low-hanging fruit after all.

Thoughts?

Ludo’.


[-- Attachment #2: the code --]
[-- Type: text/plain, Size: 7188 bytes --]

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2022 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.


(define-module (guix scripts review)
  #:use-module (guix scripts)
  #:use-module (guix ui)
  #:use-module (guix i18n)
  #:use-module (guix colors)
  #:use-module (guix http-client)
  #:use-module (guix utils)
  #:use-module ((guix build utils) #:select (find-files invoke))
  #:autoload   (guix inferior) (open-inferior close-inferior
                                               inferior-available-packages)
  #:autoload   (gnu packages) (fold-available-packages)
  #:autoload   (email email) (parse-email email-headers email-body)
  #:use-module (ice-9 binary-ports)
  #:use-module (srfi srfi-1)
  #:use-module (web uri)
  #:use-module (web http)
  #:use-module (ice-9 match)
  #:use-module (ice-9 rdelim)
  #:use-module (ice-9 format)
  #:export (guix-review))

(define (patch-set-url number)
  (string-append "https://issues.guix.gnu.org/issue/"
                 (number->string number) "/patch-set"))

(define (monkey-patch!)
  ;; XXX: Currently mumi returns "Content-Type: text;...", which is considered
  ;; invalid.  Work around it.
  (declare-header! "Content-Type"
                   (lambda (str)
                     (let ((parts (string-split str #\;)))
                       (cons 'text/plain          ;<- this is the "fix"
                             (map (lambda (x)
                                    (let ((eq (string-index x #\=)))
                                      (cons
                                       (string->symbol
                                        (string-trim x char-set:whitespace 0 eq))
                                       (string-trim-right x char-set:whitespace (1+ eq)))))
                                  (cdr parts)))))
                   (header-validator 'content-type)
                   (header-writer 'content-type))
  (set! monkey-patch! (const #t)))

(define (download-patch-set directory number)
  "Download to DIRECTORY the patches of issue NUMBER."
  (define (patch-file n)
    (format #f "~a/~3,'0d.patch" directory n))

  (define (from-line? str)
    (string-prefix? "From " str))

  (monkey-patch!)
  (let ((port (http-fetch (string->uri (patch-set-url number)))))
    (let loop ((n 0)
               (output (%make-void-port "w0")))
      (match (read-line port 'concat)
        ((? eof-object?)
         (close-port output)
         n)
        ((? from-line? line)
         (close-port output)
         (let* ((n   (+ n 1))
                (new (open-output-file (patch-file n))))
           (display line new)
           (loop n new)))
        (line
         (display line output)
         (loop n output))))))

(define colorize-patch-line
  (color-rules
   ("^(---.*)$" BOLD)
   ("^(\\+\\+\\+.*)$" BOLD)
   ("^(-.*)$" RED)
   ("^(\\+.*)$" GREEN)
   ("^(@@ .* @@)" CYAN)))

(define (display-patch file)
  "Read the patch in FILE and display it to standard output."
  (define email
    (parse-email
     (call-with-input-file file
       (lambda (port)
         (read-line port)                         ;discard the "From " line
         (get-bytevector-all port)))))

  (with-paginated-output-port port
    (match (assoc-ref (email-headers email) 'from)
      ((from)
       (format port "From: ~a <~a>~%"
               (or (assoc-ref from 'name) "")
               (assoc-ref from 'address))
       (format port "Subject: ~a~%"
               (assoc-ref (email-headers email) 'subject))
       (newline port)))

    (call-with-input-string (email-body email)
      (lambda (input)
        (let loop ()
          (match (read-line input 'concat)
            ((? eof-object?)
             #t)
            (line
             (display (colorize-patch-line line) port)
             (loop))
            (_
             (loop))))))))

(define (apply-patch patch)
  (invoke "git" "am" "-s" patch))

\f
;;;
;;; Package differences.
;;;

(define (available-packages)
  (fold-available-packages
   (lambda* (name version result
                  #:key supported? deprecated?
                  #:allow-other-keys)
     (if (and supported? (not deprecated?))
         (alist-cons name version result)
         result))
   '()))

(define (new-available-packages)
  (define %top-builddir
    (dirname (search-path %load-compiled-path "guix.go")))

  (let* ((inferior (open-inferior %top-builddir #:command "scripts/guix"))
         (packages (inferior-available-packages inferior)))
    (close-inferior inferior)
    packages))

;; (define display-new/upgraded-packages
;;   (@@ (guix scripts pull) display-new/upgraded-packages))

\f
;;;
;;; Command-line options.
;;;

(define %default-options
  '((action . apply)))

(define %options
  '())

\f
;;;
;;; Entry point.
;;;

(define-command (guix-review . args)
  (category packaging)
  (synopsis "review patches")

  (with-error-handling
    (let* ((options (parse-command-line args %options
                                        (list %default-options)))
           (issue (match (assoc-ref options 'argument)
                    ((= string->number number)
                     number)
                    (_
                     (leave (G_ "missing issue number~%"))))))
      (call-with-temporary-directory
       (lambda (directory)
         (let ((patches (download-patch-set directory issue)))
           (info (N_ "downloaded ~a patch for issue #~a~%"
                     "downloaded ~a patches for issue #~a~%"
                     patches)
                 patches issue)
           (if (= 1 patches)
               (match (find-files directory)
                 ((patch)
                  (display-patch patch)
                  (info (G_ "hit enter to apply~%"))))
               (info (G_ "screen the patches under ~a and hit enter to apply~%")
                     directory))
           (read-char)

           (let ((old (available-packages)))
             (for-each apply-patch (find-files directory))
             (invoke "make")
             (let ((new (new-available-packages)))
               (pk 'diff (lset-difference equal? new old))
               (display-new/upgraded-packages old new
                                              #:concise? #f)))))))))

;; Local Variables:
;; eval: (put 'with-binding 'scheme-indent-function 2)
;; End:

Re: Mumi, public-inbox and tools

[Arun Isaac]

Hi Ludo,

> The attached super-early-draft ‘guix review’ script fetches mboxes from
> mumi—that part is OK.

I like the idea! But, could we put this in mumi instead of guix? That
way, other packages that use debbugs/mumi (skribilo comes to mind), can
benefit from it as well. Something like

$ guix shell mumi -- mumi review 54874

Maybe, we should read some config file, say mumi.scm, from the top-level
directory of the current git repo to determine settings (think package
name, debbugs/mumi web URI, post-processing function to provide feedback
about the patch, etc.) specific to that repo.

> What’s more difficult is the rest: determining what’s a patch and what’s
> not (I gather this is something that b4 and Patchwork handle nicely),
> getting the latest patch series, things like that.

At some point, we should patch mumi to determine these. Determining the
latest patch series shouldn't be hard if we use the Date
header. guile-email will let us read any email header.

WDYT?

Cheers!
Arun

Re: Mumi, public-inbox and tools

[zimoun]

Hi,

Really cool!

On Sun, 08 May 2022 at 00:52, Ludovic Courtès <ludo@gnu.org> wrote:

> I think we could have, as a first step, a command that allows us to run:
>
>   ./pre-inst-env guix review 54874
>
> and that fetches patches, applies them, and ideally reports on the
> difference and what needs to be built.

Currently, using the hhtps://yhetil.org/guix public-inbox instance,
coupled to the Emacs package ’piem’, I already have that. :-)

Basically, I use Emacs code from emacs-debbugs to query, then the Emacs
package ’piem’ [1] coupled [2] to the Python tool ’b4’ [3] and done.

Well, maybe using Scheme code for query Debbugs via Mumi, then instead
of,

> (define (apply-patch patch)
>   (invoke "git" "am" "-s" patch))

’invoke’ the B4 tool.  Other said, start to move all the hard work of
extracting the patches and application to an external tool and just glue
with Scheme (instead of Emacs lisp as ’piem’, for instance).

Then this Python B4 could be reimplemented (or adapted) using Scheme.

1: <https://docs.kyleam.com/piem/>
2: <https://docs.kyleam.com/piem/Applying-patches.html>
3: <https://git.kernel.org/pub/scm/utils/b4/b4.git>


Cheers,
simon

Re: Mumi, public-inbox and tools

[zimoun]

Hi Arun,

On Sun, 08 May 2022 at 11:43, Arun Isaac <arunisaac@systemreboot.net> wrote:
> Hi Ludo,
>
>> The attached super-early-draft ‘guix review’ script fetches mboxes from
>> mumi—that part is OK.
>
> I like the idea! But, could we put this in mumi instead of guix? That
> way, other packages that use debbugs/mumi (skribilo comes to mind), can
> benefit from it as well. Something like

Personally, I think such feature is a perfect candidate to Guix
extensions.


> Maybe, we should read some config file, say mumi.scm, from the top-level
> directory of the current git repo to determine settings (think package
> name, debbugs/mumi web URI, post-processing function to provide feedback
> about the patch, etc.) specific to that repo.

I agree that it deserves a config file.  For instance, piem [1] uses
something like,

        ("guix"
         :url "https://yhetil.org/guix"
         :address "guix-patches@gnu.org"
         :listid "guix-patches@gnu.org"
         :coderepo "~/src/guix/")

1:
<https://docs.kyleam.com/piem/Registering-inboxes.html#Registering-inboxes>


>> What’s more difficult is the rest: determining what’s a patch and what’s
>> not (I gather this is something that b4 and Patchwork handle nicely),
>> getting the latest patch series, things like that.
>
> At some point, we should patch mumi to determine these. Determining the
> latest patch series shouldn't be hard if we use the Date
> header. guile-email will let us read any email header.

Well, I think the general case is not easy.  For an example of a complex
thread and patch submission, consider this one:

    <https://yhetil.org/guix/cu7pnfaar36.fsf@systemreboot.net/#r>

I have not tried to use B4 [2] on that one though.  Somehow, I am not
convinced that the Date header would be enough to identify the last
patches.  For instance, the first submission is a patch attached, then
the others are inline, then comments are interleaved, etc.

2: <https://git.kernel.org/pub/scm/utils/b4/b4.git/about/>


IMHO, this determination requires some work. :-)

That’s said, we could also agree that “mumi review” would only work when
a set of rules (inline patches, base-commit, etc.) are applied.  And for
the other cases, the current manual patches application would be used.

Somehow, constraint a set of rule would ease the automation and it
appears to me an acceptable trade-off: it helps such automation and the
work by reviewer and in addition we can still deal with the submissions
not respecting such rules by reusing the “laborious” current method.


Cheers,
simon

Re: Mumi, public-inbox and tools

[Ricardo Wurmus]

Arun Isaac <arunisaac@systemreboot.net> writes:

> Hi Ludo,
>
>> The attached super-early-draft ‘guix review’ script fetches mboxes from
>> mumi—that part is OK.
>
> I like the idea! But, could we put this in mumi instead of guix? That
> way, other packages that use debbugs/mumi (skribilo comes to mind), can
> benefit from it as well. Something like
>
> $ guix shell mumi -- mumi review 54874

I also think it would be a good fit for mumi.

-- 
Ricardo

Re: Mumi, public-inbox and tools

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

Ludovic Courtès schreef op zo 08-05-2022 om 00:52 [+0200]:
>            (info (N_ "downloaded ~a patch for issue #~a~%"
>                      "downloaded ~a patches for issue #~a~%"

For i18n, could positional arguments be used here?
<https://www.mail-archive.com/guix-devel@gnu.org/msg61035.html>

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Mumi, public-inbox and tools

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Arun Isaac <arunisaac@systemreboot.net> writes:
>
>> Hi Ludo,
>>
>>> The attached super-early-draft ‘guix review’ script fetches mboxes from
>>> mumi—that part is OK.
>>
>> I like the idea! But, could we put this in mumi instead of guix? That
>> way, other packages that use debbugs/mumi (skribilo comes to mind), can
>> benefit from it as well. Something like
>>
>> $ guix shell mumi -- mumi review 54874
>
> I also think it would be a good fit for mumi.

Yes, there are good arguments for having it in mumi.  One of them is
that we can make the server and client-side work hand-in-hand.

An argument in favor of having such a tool in Guix is that we could have
Guix-specific tooling layered on top, such as listing the set of
new/upgraded packages, automatically building them, things like that.

But look: we could have a generic ‘mumi review’ that a hypothetical
‘guix review’ could use as a library and extend.

At any rate, anything will be better than the status quo, so I’m all for
starting with a simple ‘mumi review’!

Perhaps we should organize a hacking session or something eventually?

Thanks,
Ludo’.