generate commit messages for package updates

generate commit messages for package updates

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 3812 bytes --]

Hi Guix,

I’m currently working on upgrading all Bioconductor packages to the 3.11
release.  The hardest work here is to write commit messages for the 200+
packages that have changed.  Even with the “update” yasnippet and magit
this takes a very long time.

I wrote a little tool to reduce the amount of time needed to create
commits.  It is not fully automatic yet as it requires my input to stage
hunks, but the commit messages it produces take all input changes into
account, which is something the “update” yasnippet does not do.

After “guix refresh -t bioconductor -u”, manually verifying and
implementing the suggested changes, and building all packages, I run
“git add -p” to stage hunks that belong to the same package definition.
Then I run the attached tool to make a commit:

    ./committer.scm | git commit -F -

The tool works by looking at the unified diff in the staging area and
generating two S-expressions corresponding to the original package
definition and the changed package definition, respectively.  It then
looks at the inputs, propagated-inputs, and native-inputs fields and
generates a GNU ChangeLog-style commit message describing the changes.

Here’s an example diff:

--8<---------------cut here---------------start------------->8---
modified   gnu/packages/bioconductor.scm
@@ -2701,29 +2701,32 @@ gene and isoform level using RNA-seq data")
 (define-public r-karyoploter
   (package
     (name "r-karyoploter")
-    (version "1.12.4")
+    (version "1.14.0")
     (source (origin
               (method url-fetch)
               (uri (bioconductor-uri "karyoploteR" version))
               (sha256
                (base32
-                "03jmfgmw35hrgn3pc5lq6pblzhfx9fp4l6dx50rp303lr7kjxp9v"))))
+                "0h0gk4xd95k5phy6qcsv7j931d7gk3p24i2fg4mz5dsk110lpifs"))))
     (build-system r-build-system)
     (propagated-inputs
-     `(("r-regioner" ,r-regioner)
+     `(("r-annotationdbi" ,r-annotationdbi)
+       ("r-bamsignals" ,r-bamsignals)
+       ("r-bezier" ,r-bezier)
+       ("r-biovizbase" ,r-biovizbase)
+       ("r-digest" ,r-digest)
+       ("r-genomeinfodb" ,r-genomeinfodb)
+       ("r-genomicfeatures" ,r-genomicfeatures)
        ("r-genomicranges" ,r-genomicranges)
        ("r-iranges" ,r-iranges)
-       ("r-rsamtools" ,r-rsamtools)
        ("r-memoise" ,r-memoise)
+       ("r-regioner" ,r-regioner)
+       ("r-rsamtools" ,r-rsamtools)
        ("r-rtracklayer" ,r-rtracklayer)
-       ("r-genomeinfodb" ,r-genomeinfodb)
        ("r-s4vectors" ,r-s4vectors)
-       ("r-biovizbase" ,r-biovizbase)
-       ("r-digest" ,r-digest)
-       ("r-bezier" ,r-bezier)
-       ("r-bamsignals" ,r-bamsignals)
-       ("r-annotationdbi" ,r-annotationdbi)
        ("r-variantannotation" ,r-variantannotation)))
+    (native-inputs
+     `(("r-knitr" ,r-knitr)))
     (home-page "https://bioconductor.org/packages/karyoploteR/")
     (synopsis "Plot customizable linear genomes displaying arbitrary data")
     (description "This package creates karyotype plots of arbitrary genomes and
--8<---------------cut here---------------end--------------->8---

…and this is the generated commit message:

--8<---------------cut here---------------start------------->8---
gnu: r-karyoploter: Update to 1.14.0.

* gnu/packages/bioconductor.scm (r-karyoploter): Update to 1.14.0.
[propagated-inputs]: Add r-genomicfeatures.
[native-inputs]: Add r-knitr.
--8<---------------cut here---------------end--------------->8---

Obviously, this can be improved by avoiding the staging area and
operating on all hunks in all selected files, so that more than one
commit can be made at a time.  But I thought I’d share this hack anyway,
crude as it is.

-- 
Ricardo


[-- Attachment #2: committer.scm --]
[-- Type: application/x-scheme, Size: 5829 bytes --]

Re: generate commit messages for package updates

[Edouard Klein]

Thank you Ricardo !

Writing commit messages by hand is a major pain, especially for
newcommers like me. For simple commits it took me more time to write the
message than to write the code. The mistakes I made writing the commit
messages made the reviewers lose time correcting them. Automating this
task away is exactly what we need :)


Ricardo Wurmus writes:

> Hi Guix,
>
> I’m currently working on upgrading all Bioconductor packages to the 3.11
> release.  The hardest work here is to write commit messages for the 200+
> packages that have changed.  Even with the “update” yasnippet and magit
> this takes a very long time.
>
> I wrote a little tool to reduce the amount of time needed to create
> commits.  It is not fully automatic yet as it requires my input to stage
> hunks, but the commit messages it produces take all input changes into
> account, which is something the “update” yasnippet does not do.
>
> After “guix refresh -t bioconductor -u”, manually verifying and
> implementing the suggested changes, and building all packages, I run
> “git add -p” to stage hunks that belong to the same package definition.
> Then I run the attached tool to make a commit:
>
>     ./committer.scm | git commit -F -
>
> The tool works by looking at the unified diff in the staging area and
> generating two S-expressions corresponding to the original package
> definition and the changed package definition, respectively.  It then
> looks at the inputs, propagated-inputs, and native-inputs fields and
> generates a GNU ChangeLog-style commit message describing the changes.
>
> Here’s an example diff:
>
> --8<---------------cut here---------------start------------->8---
> modified   gnu/packages/bioconductor.scm
> @@ -2701,29 +2701,32 @@ gene and isoform level using RNA-seq data")
>  (define-public r-karyoploter
>    (package
>      (name "r-karyoploter")
> -    (version "1.12.4")
> +    (version "1.14.0")
>      (source (origin
>                (method url-fetch)
>                (uri (bioconductor-uri "karyoploteR" version))
>                (sha256
>                 (base32
> -                "03jmfgmw35hrgn3pc5lq6pblzhfx9fp4l6dx50rp303lr7kjxp9v"))))
> +                "0h0gk4xd95k5phy6qcsv7j931d7gk3p24i2fg4mz5dsk110lpifs"))))
>      (build-system r-build-system)
>      (propagated-inputs
> -     `(("r-regioner" ,r-regioner)
> +     `(("r-annotationdbi" ,r-annotationdbi)
> +       ("r-bamsignals" ,r-bamsignals)
> +       ("r-bezier" ,r-bezier)
> +       ("r-biovizbase" ,r-biovizbase)
> +       ("r-digest" ,r-digest)
> +       ("r-genomeinfodb" ,r-genomeinfodb)
> +       ("r-genomicfeatures" ,r-genomicfeatures)
>         ("r-genomicranges" ,r-genomicranges)
>         ("r-iranges" ,r-iranges)
> -       ("r-rsamtools" ,r-rsamtools)
>         ("r-memoise" ,r-memoise)
> +       ("r-regioner" ,r-regioner)
> +       ("r-rsamtools" ,r-rsamtools)
>         ("r-rtracklayer" ,r-rtracklayer)
> -       ("r-genomeinfodb" ,r-genomeinfodb)
>         ("r-s4vectors" ,r-s4vectors)
> -       ("r-biovizbase" ,r-biovizbase)
> -       ("r-digest" ,r-digest)
> -       ("r-bezier" ,r-bezier)
> -       ("r-bamsignals" ,r-bamsignals)
> -       ("r-annotationdbi" ,r-annotationdbi)
>         ("r-variantannotation" ,r-variantannotation)))
> +    (native-inputs
> +     `(("r-knitr" ,r-knitr)))
>      (home-page "https://bioconductor.org/packages/karyoploteR/")
>      (synopsis "Plot customizable linear genomes displaying arbitrary data")
>      (description "This package creates karyotype plots of arbitrary genomes and
> --8<---------------cut here---------------end--------------->8---
>
> …and this is the generated commit message:
>
> --8<---------------cut here---------------start------------->8---
> gnu: r-karyoploter: Update to 1.14.0.
>
> * gnu/packages/bioconductor.scm (r-karyoploter): Update to 1.14.0.
> [propagated-inputs]: Add r-genomicfeatures.
> [native-inputs]: Add r-knitr.
> --8<---------------cut here---------------end--------------->8---
>
> Obviously, this can be improved by avoiding the staging area and
> operating on all hunks in all selected files, so that more than one
> commit can be made at a time.  But I thought I’d share this hack anyway,
> crude as it is.

Re: generate commit messages for package updates

[Hartmut Goebel]

Hi Ricardo,
> I wrote a little tool to reduce the amount of time needed to create
> commits.  It is not fully automatic yet as it requires my input to stage
> hunks, but the commit messages it produces take all input changes into
> account, which is something the “update” yasnippet does not do.

Great! I wished such a tool since long!

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: generate commit messages for package updates

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 69 bytes --]

Same here!  Thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: generate commit messages for package updates

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]


Ricardo Wurmus <rekado@elephly.net> writes:

> It is not fully automatic yet as it requires my input to stage
> hunks, but the commit messages it produces take all input changes into
> account, which is something the “update” yasnippet does not do.

Attached is the fully automatic version.

You just make your changes and run the tool.  It will stage all diffs
that are associated with the same S-expression, generate the commit
message, and make the commit for each S-expression.

Note that it probably doesn’t do the right thing when something other
than a package expression is modified (e.g. a copyright line).

-- 
Ricardo


[-- Attachment #2: committer.scm --]
[-- Type: application/octet-stream, Size: 8286 bytes --]

#!/run/current-system/profile/bin/guile \
--no-auto-compile -s
!#

(import (sxml xpath)
        (srfi srfi-1)
        (srfi srfi-9)
        (ice-9 format)
        (ice-9 popen)
        (ice-9 match)
        (ice-9 rdelim)
        (ice-9 textual-ports))

(define (read-excursion port)
  "Read an expression from PORT and reset the port position before returning
the expression."
  (let ((start (ftell port))
        (result (read port)))
    (seek port start SEEK_SET)
    result))

(define (surrounding-sexp port line-no)
  "Return the top-level S-expression surrounding the change at line number
LINE-NO in PORT."
  (let loop ((i (1- line-no))
             (last-top-level-sexp #f))
    (if (zero? i)
        last-top-level-sexp
        (match (peek-char port)
          (#\(
           (let ((sexp (read-excursion port)))
             (read-line port)
             (loop (1- i) sexp)))
          (_
           (read-line port)
           (loop (1- i) last-top-level-sexp))))))

(define-record-type <hunk>
  (make-hunk file-name
             old-line-number
             new-line-number
             diff)
  hunk?
  (file-name       hunk-file-name)
  ;; Line number before the change
  (old-line-number hunk-old-line-number)
  ;; Line number after the change
  (new-line-number hunk-new-line-number)
  ;; The full diff to be used with "git apply --cached"
  (diff hunk-diff))

(define* (hunk->patch hunk #:optional (port (current-output-port)))
  (let ((file-name (hunk-file-name hunk)))
    (format port
            "diff --git a/~a b/~a~%--- a/~a~%+++ b/~a~%~a"
            file-name file-name file-name file-name
            (hunk-diff hunk))))

(define (diff-info)
  "Read the diff and return a list of <hunk> values."
  (let ((port (open-pipe* OPEN_READ
                          "git" "diff"
                          "--no-prefix"
                          ;; Do not include any context lines.  This makes it
                          ;; easier to find the S-expression surrounding the
                          ;; change.
                          "--unified=0")))
    (define (extract-line-number line-tag)
      (abs (string->number
            (car (string-split line-tag #\,)))))
    (define (read-hunk)
      (reverse
       (let loop ((lines '()))
         (let ((line (read-line port 'concat)))
           (cond
            ((eof-object? line) lines)
            ((or (string-prefix? "@@ " line)
                 (string-prefix? "diff --git" line))
             (unget-string port line)
             lines)
            (else (loop (cons line lines))))))))
    (define info
      (let loop ((acc '()) 
                 (file-name #f))
        (let ((line (read-line port)))
          (cond
           ((eof-object? line) acc)
           ((string-prefix? "--- " line)
            (match (string-split line #\space)
              ((_ file-name)
               (loop acc file-name))))
           ((string-prefix? "@@ " line)
            (match (string-split line #\space)
              ((_ old-start new-start . _)
               (loop (cons (make-hunk file-name
                                      (extract-line-number old-start)
                                      (extract-line-number new-start)
                                      (string-join (cons* line "\n"
                                                          (read-hunk)) ""))
                           acc)
                     file-name))))
           (else (loop acc file-name))))))
    (close-pipe port)
    info))

(define (old-sexp hunk)
  "Using the diff information in HUNK return the unmodified S-expression
corresponding to the top-level definition containing the staged changes."
  ;; TODO: We can't seek with a pipe port...
  (let* ((port (open-pipe* OPEN_READ
                           "git" "show" (string-append "HEAD:"
                                                       (hunk-file-name hunk))))
         (contents (get-string-all port)))
    (close-pipe port)
    (call-with-input-string contents
      (lambda (port)
        (surrounding-sexp port (hunk-old-line-number hunk))))))

(define (new-sexp hunk)
  "Using the diff information in HUNK return the modified S-expression
corresponding to the top-level definition containing the staged changes."
  (call-with-input-file (hunk-file-name hunk)
    (lambda (port)
      (surrounding-sexp port
                        (hunk-new-line-number hunk)))))

(define* (commit-message file-name old new #:optional (port (current-output-port)))
  "Print ChangeLog commit message for changes between OLD and NEW."
  (define (get-values expr field)
    (match ((sxpath `(// ,field quasiquote *)) expr)
      (() '())
      ((first . rest)
       (map cadadr first))))
  (define (listify items)
    (match items
      ((one) one)
      ((one two)
       (string-append one " and " two))
      ((one two . more)
       (string-append (string-join (drop-right items 1) ", ")
                      ", and " (first (take-right items 1))))))
  (define variable-name
    (second old))
  (define version
    (and=> ((sxpath '(// version *any*)) new)
           first))
  (format port
          "gnu: ~a: Update to ~a.~%~%* ~a (~a): Update to ~a.~%"
          variable-name version file-name variable-name version)
  (for-each (lambda (field)
              (let ((old-values (get-values old field))
                    (new-values (get-values new field)))
                (or (equal? old-values new-values)
                    (let ((removed (lset-difference eq? old-values new-values))
                          (added (lset-difference eq? new-values old-values)))
                      (format port
                              "[~a]: ~a~%" field
                              (match (list (map symbol->string removed)
                                           (map symbol->string added))
                                ((() added)
                                 (format #f "Add ~a."
                                         (listify added)))
                                ((removed ())
                                 (format #f "Remove ~a."
                                         (listify removed)))
                                ((removed added)
                                 (format #f "Remove ~a; add ~a."
                                         (listify removed)
                                         (listify added)))))))))
            '(inputs propagated-inputs native-inputs)))

(define (group-hunks-by-sexp hunks)
  "Return a list of pairs associating all hunks with the S-expression they are
modifying."
  (fold (lambda (sexp hunk acc)
          (match acc
            (((previous-sexp . hunks) . rest)
             (if (equal? sexp previous-sexp)
                 (cons (cons previous-sexp
                             (cons hunk hunks))
                       rest)
                 (cons (cons sexp (list hunk))
                       acc)))
            (_
             (cons (cons sexp (list hunk))
                   acc))))
        '()
        (map new-sexp hunks)
        hunks))

(define (main . args)
  (match (diff-info)
    (()
     (display "Nothing to be done." (current-error-port)))
    (hunks
     (for-each (match-lambda
                 ((new . hunks)
                  (let ((old (old-sexp (first hunks))))
                    (for-each (lambda (hunk)
                                (let ((port (open-pipe* OPEN_WRITE
                                                        "git" "apply"
                                                        "--cached"
                                                        "--unidiff-zero")))
                                  (hunk->patch hunk port)
                                  (unless (eqv? 0 (status:exit-val (close-pipe port)))
                                    (error "Cannot apply"))))
                              hunks)
                    (let ((port (open-pipe* OPEN_WRITE "git" "commit" "-F" "-")))
                      (commit-message (hunk-file-name (first hunks))
                                      old new
                                      port)
                      (unless (eqv? 0 (status:exit-val (close-pipe port)))
                        (error "Cannot commit"))))))
               (group-hunks-by-sexp hunks)))))

(main)

Re: generate commit messages for package updates

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 84 bytes --]

Is it worth including in guix/etc?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: generate commit messages for package updates

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Is it worth including in guix/etc?

In my biased opinion: yes.  I’d like to improve it a little more and
handle more changes, but I think the general approach isn’t going to
change much.

-- 
Ricardo

Re: generate commit messages for package updates

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Is it worth including in guix/etc?

I think it’d be nice!

Ludo’.

Re: generate commit messages for package updates

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
>> Is it worth including in guix/etc?
>
> I think it’d be nice!

I fixed a bug and added etc/committer.scm.in.

-- 
Ricardo

Re: generate commit messages for package updates

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>>
>>> Is it worth including in guix/etc?
>>
>> I think it’d be nice!
>
> I fixed a bug and added etc/committer.scm.in.

Yay!  Consider adding a line in “Contributing” to mention its existence.

Thank you,
Ludo’.

Re: generate commit messages for package updates

[Alex Sassmannshausen]

Hi Ricardo,

Just used your committer tool for the first time and it's bloody lovely.

Thank you for sharing it :-)

Alex

Ricardo Wurmus <rekado@elephly.net> writes:

> Hi Guix,
>
> I’m currently working on upgrading all Bioconductor packages to the 3.11
> release.  The hardest work here is to write commit messages for the 200+
> packages that have changed.  Even with the “update” yasnippet and magit
> this takes a very long time.
>
> I wrote a little tool to reduce the amount of time needed to create
> commits.  It is not fully automatic yet as it requires my input to stage
> hunks, but the commit messages it produces take all input changes into
> account, which is something the “update” yasnippet does not do.
>
> After “guix refresh -t bioconductor -u”, manually verifying and
> implementing the suggested changes, and building all packages, I run
> “git add -p” to stage hunks that belong to the same package definition.
> Then I run the attached tool to make a commit:
>
>     ./committer.scm | git commit -F -
>
> The tool works by looking at the unified diff in the staging area and
> generating two S-expressions corresponding to the original package
> definition and the changed package definition, respectively.  It then
> looks at the inputs, propagated-inputs, and native-inputs fields and
> generates a GNU ChangeLog-style commit message describing the changes.
>
> Here’s an example diff:
>
> --8<---------------cut here---------------start------------->8---
> modified   gnu/packages/bioconductor.scm
> @@ -2701,29 +2701,32 @@ gene and isoform level using RNA-seq data")
>  (define-public r-karyoploter
>    (package
>      (name "r-karyoploter")
> -    (version "1.12.4")
> +    (version "1.14.0")
>      (source (origin
>                (method url-fetch)
>                (uri (bioconductor-uri "karyoploteR" version))
>                (sha256
>                 (base32
> -                "03jmfgmw35hrgn3pc5lq6pblzhfx9fp4l6dx50rp303lr7kjxp9v"))))
> +                "0h0gk4xd95k5phy6qcsv7j931d7gk3p24i2fg4mz5dsk110lpifs"))))
>      (build-system r-build-system)
>      (propagated-inputs
> -     `(("r-regioner" ,r-regioner)
> +     `(("r-annotationdbi" ,r-annotationdbi)
> +       ("r-bamsignals" ,r-bamsignals)
> +       ("r-bezier" ,r-bezier)
> +       ("r-biovizbase" ,r-biovizbase)
> +       ("r-digest" ,r-digest)
> +       ("r-genomeinfodb" ,r-genomeinfodb)
> +       ("r-genomicfeatures" ,r-genomicfeatures)
>         ("r-genomicranges" ,r-genomicranges)
>         ("r-iranges" ,r-iranges)
> -       ("r-rsamtools" ,r-rsamtools)
>         ("r-memoise" ,r-memoise)
> +       ("r-regioner" ,r-regioner)
> +       ("r-rsamtools" ,r-rsamtools)
>         ("r-rtracklayer" ,r-rtracklayer)
> -       ("r-genomeinfodb" ,r-genomeinfodb)
>         ("r-s4vectors" ,r-s4vectors)
> -       ("r-biovizbase" ,r-biovizbase)
> -       ("r-digest" ,r-digest)
> -       ("r-bezier" ,r-bezier)
> -       ("r-bamsignals" ,r-bamsignals)
> -       ("r-annotationdbi" ,r-annotationdbi)
>         ("r-variantannotation" ,r-variantannotation)))
> +    (native-inputs
> +     `(("r-knitr" ,r-knitr)))
>      (home-page "https://bioconductor.org/packages/karyoploteR/")
>      (synopsis "Plot customizable linear genomes displaying arbitrary data")
>      (description "This package creates karyotype plots of arbitrary genomes and
> --8<---------------cut here---------------end--------------->8---
>
> …and this is the generated commit message:
>
> --8<---------------cut here---------------start------------->8---
> gnu: r-karyoploter: Update to 1.14.0.
>
> * gnu/packages/bioconductor.scm (r-karyoploter): Update to 1.14.0.
> [propagated-inputs]: Add r-genomicfeatures.
> [native-inputs]: Add r-knitr.
> --8<---------------cut here---------------end--------------->8---
>
> Obviously, this can be improved by avoiding the staging area and
> operating on all hunks in all selected files, so that more than one
> commit can be made at a time.  But I thought I’d share this hack anyway,
> crude as it is.