[outreachy] Walk through the Git history (guix git {authenticate, log})

[outreachy] Walk through the Git history (guix git {authenticate, log})

[zimoun]

Hi,

Discussing how to walk through the Git history
~/.cache/guix/checkouts/<hash>, there is 2 ways to do:

 1. Loop with commit-parents as it is done for ’commit-closure’ in
    guix/git.scm.

 2. Bind git_revwalk_* and use it instead.

WDYT?

Well, #1 is more straightforward but less efficient, IIUC.


All the best,
simon

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[Magali]

Hi,

On 11/12/2020 14:41, zimoun wrote:
> Hi,
>
> Discussing how to walk through the Git history
> ~/.cache/guix/checkouts/<hash>, there is 2 ways to do:
>
>  1. Loop with commit-parents as it is done for ’commit-closure’ in
>     guix/git.scm.
>
>  2. Bind git_revwalk_* and use it instead.
>
> WDYT?
>
> Well, #1 is more straightforward but less efficient, IIUC.
>
>
> All the best,
> simon

I think #2 is the way to go. It may be more difficult to implement right
now, but in the future it'll make things easier if we think about adding
an option that allows us to choose the order the commits will be shown,
for instance.


Magali

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[Mathieu Othacehe]

Hello zimoun,

>  1. Loop with commit-parents as it is done for ’commit-closure’ in
>     guix/git.scm.
>
>  2. Bind git_revwalk_* and use it instead.
>
> WDYT?
>
> Well, #1 is more straightforward but less efficient, IIUC.

Running something like:

--8<---------------cut here---------------start------------->8---
(let loop ((commit (commit-lookup r (string->oid "cf53ea79d")))
           (res '()))
     (let ((parents (commit-parents commit)))
       (if (null? parents)
           res
         (loop (car parents) (cons (car parents) res)))))
--8<---------------cut here---------------end--------------->8---

takes 1.45s to go over 53391 commits. The "revwalk" API may be more
efficient but I would say that what's currently implemented could be a
fine starting point.

Thanks,

Mathieu

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[zimoun]

Hi Mathieu,

On Sat, 12 Dec 2020 at 09:42, Mathieu Othacehe <othacehe@gnu.org> wrote:

>>  1. Loop with commit-parents as it is done for ’commit-closure’ in
>>     guix/git.scm.
>>
>>  2. Bind git_revwalk_* and use it instead.
>>
>> WDYT?
>>
>> Well, #1 is more straightforward but less efficient, IIUC.
>
> Running something like:
>
> --8<---------------cut here---------------start------------->8---
> (let loop ((commit (commit-lookup r (string->oid "cf53ea79d")))
>            (res '()))
>      (let ((parents (commit-parents commit)))
>        (if (null? parents)
>            res
>          (loop (car parents) (cons (car parents) res)))))
> --8<---------------cut here---------------end--------------->8---
>
> takes 1.45s to go over 53391 commits. The "revwalk" API may be more
> efficient but I would say that what's currently implemented could be a
> fine starting point.

It does not work for complex history where multiple branches are merged.
For example, the snippet excludes one of the master or core-updates at
the merge points; or I miss something.

Therefore, the loop should explore the different branches at the merge
points, keeping somehow a list of already visited commit, implementing
bit of graph stuff, and all in all, it is git_revwalk_*, IIUC.

I agree that looping with commit-parents is a good starting point.


All the best,
simon

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[zimoun]

Hi Magali,

On Fri, 11 Dec 2020 at 16:01, Magali <magalilemes00@gmail.com> wrote:

> I think #2 is the way to go. It may be more difficult to implement right
> now, but in the future it'll make things easier if we think about adding
> an option that allows us to choose the order the commits will be shown,
> for instance.

Awaiting how to walk the history tree, I am proposing to use the loop as
Mathieu said.  However, ’match’ is preferred over ’car’.  Something
along:

--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.4
Copyright (C) 1995-2020 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(guix git)
scheme@(guix-user)> ,use(git)
scheme@(guix-user)> ,use(guix channels)
scheme@(guix-user)> ,use(ice-9 match)
scheme@(guix-user)> ,use(srfi srfi-1)    
scheme@(guix-user)> (define cache (url-cache-directory (channel-url %default-guix-channel)))
scheme@(guix-user)> (define repo (repository-open cache))
scheme@(guix-user)> (define (commit-list repo)
  (let loop ((commit (commit-lookup repo
                                    (reference-target (repository-head repo))))
             (res '()))
    (match (commit-parents commit)
      (() (reverse res))
      ((head . tail)
       (loop head (cons head res))))))

scheme@(guix-user)> (define commits (commit-list repo))
scheme@(guix-user)> ,pp (take commits 5)
$1 = (#<git-commit 2639a99cf31e3ce2970645badf11f15679c893b5>
 #<git-commit f769ad12f1b7834a5fb6c870a6737d2eea495744>
 #<git-commit 8f330aebee504269d4d7188daa85e95503f71888>
 #<git-commit f765577dec0340d035021d030f17b54a3a5fd6b1>
 #<git-commit 72e3614666b8ffb7ecc568cbf4ba18874b9d8750>)
scheme@(guix-user)> cache
$2 = "/home/simon/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq"
scheme@(guix-user)> ,q

$ repo=/home/simon/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
$ git -C $repo log --oneline | head -n6
91e35e32a4 gnu: fcitx5-chinese-addons: Remove stale build steps.
2639a99cf3 gnu: fcitx5-lua: Remove stale build steps.
f769ad12f1 gnu: toybox: Cross compile.
8f330aebee gnu: busybox: Update to 1.32.0.
f765577dec gnu: Add emacs-webpaste.
72e3614666 gnu: emacs-elpher: Mention gemini in synopsis and description.
--8<---------------cut here---------------end--------------->8---

Well, ’commit-list’ is far from perfect because all the log is traversed
and the list reversed, aside that the first commit is not captured,
etc. But I hope you get the idea.  It is a starting point to implement a
draft of:

  guix git log --oneline


Cheers,
simon

...

[Magali]

Hey,

On 12/12/2020 09:24, zimoun wrote:
> Awaiting how to walk the history tree, I am proposing to use the loop as
> Mathieu said.  However, ’match’ is preferred over ’car’.


I got curious as to why it is preferred, hehe. Is there a special reason?


> Well, ’commit-list’ is far from perfect because all the log is traversed
> and the list reversed, aside that the first commit is not captured,
> etc. But I hope you get the idea.  It is a starting point to implement a
> draft of:
>
>   guix git log --oneline


Understood. I'll start working on it.


Magali

[outreachy] Walk through the Git history

[zimoun]

Hey,

On Sat, 12 Dec 2020 at 14:24, Magali <magalilemes00@gmail.com> wrote:

>> Awaiting how to walk the history tree, I am proposing to use the loop as
>> Mathieu said.  However, ’match’ is preferred over ’car’.
>
> I got curious as to why it is preferred, hehe. Is there a special
> reason?

I do not really know.  I am just taking in count previous Ludo’s
comments, for example [1].  From my understanding, it seems more
aesthetic.  Maybe Ludo or someone else would comment more. :-)

1: <https://yhetil.org/guix/87d0121iru.fsf@gnu.org>

>> Well, ’commit-list’ is far from perfect because all the log is traversed
>> and the list reversed, aside that the first commit is not captured,
>> etc. But I hope you get the idea.  It is a starting point to implement a
>> draft of:
>>
>>   guix git log --oneline
>
> Understood. I'll start working on it.

IMHO, the reward to this path will be easier; incremental improvements.
However, feel free to start whenever you want to give a look on how the
Guile-Git bindings are done; it will be implemented at one point or the
other. :-)

Cheers,
simon

Re: [outreachy] Walk through the Git history

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 12 Dec 2020 at 14:24, Magali <magalilemes00@gmail.com> wrote:
>
>>> Awaiting how to walk the history tree, I am proposing to use the loop as
>>> Mathieu said.  However, ’match’ is preferred over ’car’.
>>
>> I got curious as to why it is preferred, hehe. Is there a special
>> reason?
>
> I do not really know.  I am just taking in count previous Ludo’s
> comments, for example [1].  From my understanding, it seems more
> aesthetic.  Maybe Ludo or someone else would comment more. :-)

‘match’ is usually clearer and less error-prone: it checks that the list
actually has the right shape.

  https://guix.gnu.org/manual/en/html_node/Data-Types-and-Pattern-Matching.html
  https://www.gnu.org/software/guile/manual/html_node/Pattern-Matching.html

I hope this sheds some light on this!

Ludo’.

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[Magali]

Hello, I've been tweaking this piece of code below, trying to adapt it
so that it could get the latest commit into the commit list too, but I
haven't had success.

On 12/12/2020 09:24, zimoun wrote:
> --8<---------------cut here---------------start------------->8---
> $ guix repl
> GNU Guile 3.0.4
> Copyright (C) 1995-2020 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.
> scheme@(guix-user)> ,use(guix git)
> scheme@(guix-user)> ,use(git)
> scheme@(guix-user)> ,use(guix channels)
> scheme@(guix-user)> ,use(ice-9 match)
> scheme@(guix-user)> ,use(srfi srfi-1)    
> scheme@(guix-user)> (define cache (url-cache-directory (channel-url %default-guix-channel)))
> scheme@(guix-user)> (define repo (repository-open cache))
> scheme@(guix-user)> (define (commit-list repo)
>   (let loop ((commit (commit-lookup repo
>                                     (reference-target (repository-head repo))))
>              (res '()))
>     (match (commit-parents commit)
>       (() (reverse res))
>       ((head . tail)
>        (loop head (cons head res))))))
>
> scheme@(guix-user)> (define commits (commit-list repo))
> scheme@(guix-user)> ,pp (take commits 5)
> $1 = (#<git-commit 2639a99cf31e3ce2970645badf11f15679c893b5>
>  #<git-commit f769ad12f1b7834a5fb6c870a6737d2eea495744>
>  #<git-commit 8f330aebee504269d4d7188daa85e95503f71888>
>  #<git-commit f765577dec0340d035021d030f17b54a3a5fd6b1>
>  #<git-commit 72e3614666b8ffb7ecc568cbf4ba18874b9d8750>)
> scheme@(guix-user)> cache
> $2 = "/home/simon/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq"
> scheme@(guix-user)> ,q
>
> $ repo=/home/simon/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq
> $ git -C $repo log --oneline | head -n6
> 91e35e32a4 gnu: fcitx5-chinese-addons: Remove stale build steps.
> 2639a99cf3 gnu: fcitx5-lua: Remove stale build steps.
> f769ad12f1 gnu: toybox: Cross compile.
> 8f330aebee gnu: busybox: Update to 1.32.0.
> f765577dec gnu: Add emacs-webpaste.
> 72e3614666 gnu: emacs-elpher: Mention gemini in synopsis and description.
> --8<---------------cut here---------------end--------------->8---
>
> Well, ’commit-list’ is far from perfect because all the log is traversed
> and the list reversed, aside that the first commit is not captured,
> etc. But I hope you get the idea.  It is a starting point to implement a
> draft of:
>
>   guix git log --oneline


This is what I've been doing:

-------

$ guix repl
GNU Guile 3.0.4
Copyright (C) 1995-2020 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> ,use(guix git)
scheme@(guix-user)> ,use(git)
scheme@(guix-user)> ,use(guix channels)
scheme@(guix-user)> ,use(ice-9 match)
scheme@(guix-user)> (define cache (url-cache-directory (channel-url
%default-guix-channel)))
scheme@(guix-user)> cache
$1 =
"/home/magali/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq"
scheme@(guix-user)> (let* ((repo (repository-open cache))
         (latest-commit
          (commit-lookup repo (reference-target (repository-head repo)))))
    (let loop ((commit latest-commit)
               (res (list latest-commit)))
      (match (commit-parents commit)
             (() (reverse res))
             ((head . tail)
              (loop head (cons head res))))))
Segmentation fault (core dumped)
-------

I couldn't quite figure it out why I'm getting segfault. Any tips?


Magali

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[zimoun]

Hi Magali,

On Tue, 15 Dec 2020 at 12:20, Magali <magalilemes00@gmail.com> wrote:

> scheme@(guix-user)> ,use(guix git)
> scheme@(guix-user)> ,use(git)
> scheme@(guix-user)> ,use(guix channels)
> scheme@(guix-user)> ,use(ice-9 match)
> scheme@(guix-user)> (define cache (url-cache-directory (channel-url
> %default-guix-channel)))
> scheme@(guix-user)> cache
> $1 =
> "/home/magali/.cache/guix/checkouts/pjmkglp4t7znuugeurpurzikxq3tnlaywmisyr27shj7apsnalwq"
> scheme@(guix-user)> (let* ((repo (repository-open cache))
>          (latest-commit
>           (commit-lookup repo (reference-target (repository-head repo)))))
>     (let loop ((commit latest-commit)
>                (res (list latest-commit)))
>       (match (commit-parents commit)
>              (() (reverse res))
>              ((head . tail)
>               (loop head (cons head res))))))
> Segmentation fault (core dumped)
> -------

It rings this bell:

   <https://yhetil.org/guix-patches/87v9khjq3y.fsf@gnu.org>

and so maybe this bug: <https://gitlab.com/guile-git/guile-git/-/issues/21>. 

Mathieu, Ludo or any other Guile-Git masters, WDYT?

Therefore, let open the repo outside:

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (use-modules (guix git) (git) (guix channels) (ice-9 match) (srfi srfi-1))
scheme@(guix-user)> (define cache (url-cache-directory (channel-url %default-guix-channel)))
scheme@(guix-user)> (define repo (repository-open cache))
scheme@(guix-user)> (define commits (let* ((latest-commit
                             (commit-lookup repo (reference-target (repository-head repo)))))
     (let loop ((commit latest-commit)
                (res (list latest-commit)))
       (match (commit-parents commit)
              (() (reverse res))
              ((head . tail)
               (loop head (cons head res)))))))
scheme@(guix-user)> ,pp (take commits 5)
$1 = (#<git-commit b828057ea4c096898da8c9810e7a6607359f2aa3>
 #<git-commit ea50795d7f54582d5c439f60e836ff2ee5d8aed2>
 #<git-commit b7b9ee75885426a6ed27a78f351c6cdd9cf7f952>
 #<git-commit 4c327e025277dd59b864200d5ecf671b5656e7c9>
 #<git-commit a3a81a8703295b6ac2f0ffde5a2e8e72196178bb>)
--8<---------------cut here---------------end--------------->8---

Hope that helps,
simon

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[Ludovic Courtès]

Hi Magali,

Magali <magalilemes00@gmail.com> skribis:

> scheme@(guix-user)> (let* ((repo (repository-open cache))
>          (latest-commit
>           (commit-lookup repo (reference-target (repository-head repo)))))
>     (let loop ((commit latest-commit)
>                (res (list latest-commit)))
>       (match (commit-parents commit)
>              (() (reverse res))
>              ((head . tail)
>               (loop head (cons head res))))))
> Segmentation fault (core dumped)

I can reproduce the bug; the C backtrace looks like this:

--8<---------------cut here---------------start------------->8---
(gdb) bt
#0  0x00007fabfa9d31ee in git_oidmap_get ()
   from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
#1  0x00007fabfa98d516 in cache_get () from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
#2  0x00007fabfa9cbef7 in git_object_lookup_prefix ()
   from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
#3  0x00007fac01b8866d in ffi_call_unix64 ()
   from /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
#4  0x00007fac01b86ac0 in ffi_call_int () from /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
#5  0x00007fac01e54f2e in scm_i_foreign_call (cif_scm=<optimized out>, pointer_scm=<optimized out>, 
    errno_ret=errno_ret@entry=0x7ffe5fc1f95c, argv=0x7fabfe2e4980) at foreign.c:1073
#6  0x00007fac01ec3a84 in foreign_call (thread=0x7fac014a8d80, cif=<optimized out>, pointer=<optimized out>)
    at vm.c:1282
--8<---------------cut here---------------end--------------->8---

Could it be that, if you keep ‘repo’ in a global variable like in the
example zimoun posted, segfault no longer occurs?

I believe that what happens is a bug in Guile-Git: ‘repo’ is “finalized”
(freed) before Guile inspects the commit objects to print them, and when
Guile gets around to printing those commit objects, they now refer to a
repo that has been freed, hence the crash.

Until the bug is fixed, the workaround is to arrange your code so that
the repository object outlives commit objects.  We can discuss this
further here or on IRC if you want.

Hope this helps!

Ludo’.

Re: [outreachy] Walk through the Git history (guix git {authenticate,log})

[Magali]

Hi!

On 18/12/2020 12:34, Ludovic Courtès wrote:
> Hi Magali,
>
> Magali <magalilemes00@gmail.com> skribis:
>
>> scheme@(guix-user)> (let* ((repo (repository-open cache))
>>          (latest-commit
>>           (commit-lookup repo (reference-target (repository-head repo)))))
>>     (let loop ((commit latest-commit)
>>                (res (list latest-commit)))
>>       (match (commit-parents commit)
>>              (() (reverse res))
>>              ((head . tail)
>>               (loop head (cons head res))))))
>> Segmentation fault (core dumped)
> I can reproduce the bug; the C backtrace looks like this:
>
> --8<---------------cut here---------------start------------->8---
> (gdb) bt
> #0  0x00007fabfa9d31ee in git_oidmap_get ()
>    from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
> #1  0x00007fabfa98d516 in cache_get () from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
> #2  0x00007fabfa9cbef7 in git_object_lookup_prefix ()
>    from /gnu/store/zchrrs2zf4l06cszbadqsk18329q78sg-libgit2-1.1.0/lib/libgit2.so
> #3  0x00007fac01b8866d in ffi_call_unix64 ()
>    from /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
> #4  0x00007fac01b86ac0 in ffi_call_int () from /gnu/store/bw15z9kh9c65ycc2vbhl2izwfwfva7p1-libffi-3.3/lib/libffi.so.7
> #5  0x00007fac01e54f2e in scm_i_foreign_call (cif_scm=<optimized out>, pointer_scm=<optimized out>, 
>     errno_ret=errno_ret@entry=0x7ffe5fc1f95c, argv=0x7fabfe2e4980) at foreign.c:1073
> #6  0x00007fac01ec3a84 in foreign_call (thread=0x7fac014a8d80, cif=<optimized out>, pointer=<optimized out>)
>     at vm.c:1282
> --8<---------------cut here---------------end--------------->8---
>
> Could it be that, if you keep ‘repo’ in a global variable like in the
> example zimoun posted, segfault no longer occurs?

Yes, this seems to work just fine.


> I believe that what happens is a bug in Guile-Git: ‘repo’ is “finalized”
> (freed) before Guile inspects the commit objects to print them, and when
> Guile gets around to printing those commit objects, they now refer to a
> repo that has been freed, hence the crash.
>
> Until the bug is fixed, the workaround is to arrange your code so that
> the repository object outlives commit objects.  We can discuss this
> further here or on IRC if you want.

Nice! Thanks.


> Hope this helps!

It sure did :-)


Magali