Data Services: use cases

Data Services: use cases

[zimoun]

Hi Chris,

Since blog post about Data Services seems floating around, I describe
what I find useful as an end-user of the service.  And if you have not
showed me IRL at FOSDEM (and you showed me several times ;-)), then I
would not be using the service so often.


To me, the entry point is always:

  <https://data.guix.gnu.org/repository/1/branch/master/package/PACKAGE>

and to be exact <https://data.guix.gnu.org/…/PACKAGE/output-history>.

Even, I have this Emacs trivial helper function:

--8<---------------cut here---------------start------------->8---
(defun my/guix-data (package)
  "Add URL of PACKAGE to `kill-ring'.

Yankable result:
`https://data.guix.gnu.org/repository/1/branch/master/package/PACKAGE/output-history'.

With `universal-argument', load URL using `browse-url'."
  (interactive "sPackage: ")
  (let ((url
         (format
          "https://data.guix.gnu.org/repository/1/branch/master/package/%s/output-history" package)))
    (kill-new url)
    (when current-prefix-arg
      (browse-url url))
    (message (format "%s killed." url))))
--8<---------------cut here---------------end--------------->8---

(This dumb function could be improved using “guix-emacs” with package
completion etc.  But the time to implement such thing is not worth for
me because I generally exactly know which package I look up. :-))


Let’s take the recent example of the package ’python-umap-learn’ [1].
From the webpage, it is quick and easy to see which version build or
fail.  Then, once I know that:

       Version	Output	Builds	From	To

“From” means the date of the first commit corresponding to Version ––
specifically at the derivation.  And “To” resp. the last commit.

Then I can click to say the “To” date of the last “Succeeded” build and
it leads to the webpage:

 <https://data.guix.gnu.org/revision/cac674d99dc4a332e6210c57ec7f1b8164f66642>

which provides the commit hash.  Now, it is easy to use the
time-machine:

   guix time-machine –commit=cac674d99dc4a332e6210c57ec7f1b8164f66642 \
        – install python-umap-learn


Time to time, I click to “Failed Dependency” but I do not know how this
information is accurate.


Well, I could describe other use cases, but I think this one is really
useful for the end-user:

 - list all the versions available (since Dec. 2019 I guess)
 - know which build and which not
 - easily find the commit for “guix time-machine”


Cheers,
simon

1: <https://lists.gnu.org/archive/html/help-guix/2020-09/msg00140.html>

Re: Data Services: use cases

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

Hi!

Sorry if I'm stating the obvious, but ideally it'd be nice if the
interface would be so straightforward it didn't need any instructions to
use :)

In particular, the "Version From To" table is hard to grasp to a
newcomer.

During the Guix Days we talked about how to make it a bit more
approachable.  From what I recall:

- Maybe make the timeline bars more visible.

- The timeline is not usable and does not convey much information beside
  an approximate lifespan of versions, which I'm not sure many users
  need.  Consider removing it, or displaying it in a separate graph?

  Ultimately, we need a list of versions with their Guix builds.  

- "From" and "To" should probably be renamed "From Guix build" and "To
  Guix build" respectively.

- (More information) should be renamed to something like "Package
  information".

There are probably a few more adjustments we could make.  Any idea?

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: Data Services: use cases

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> In particular, the "Version From To" table is hard to grasp to a
> newcomer.

I’m not a newcomer but I find it limpid: I think it immediately shows
that this is about version changes and when they happened.  Maybe it
could be decluttered a bit by using shorter date strings (like
“Oct. 1st” instead of “2020-10-01 18:42:00”).

BTW, when looking at /output-history, the commits in “From” and “To” can
be a bit confusing.  Take:

  https://data.guix.gnu.org/repository/1/branch/master/package/openmpi/output-history

I see “To” pointing to
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=ec2a8f602ee0fd2fac2210eeae6d1da091bb47e4>,
a commit that changes ‘mpd’, which ‘openmpi’ doesn’t depend on; this is
completely bogus.  However, the next “From” points to the right commit:
<https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d623ece56624fd7d611cdaf42f728b21d89fec5c>.

It’s surprising that “To” and the immediately following “From” are not
the same.

WDYT, Chris?

Thanks,
Ludo’.

Re: Data Services: use cases

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

Hi Ludo!

Ludovic Courtès <ludo@gnu.org> writes:

> I’m not a newcomer but I find it limpid: I think it immediately shows
> that this is about version changes and when they happened.

What about the 4 points I listed?  I find that they make a case for what
could be improved in terms of clarity.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: Data Services: use cases

[zimoun]

Hi Ludo,

On Wed, 21 Oct 2020 at 11:27, Ludovic Courtès <ludo@gnu.org> wrote:

> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
> > In particular, the "Version From To" table is hard to grasp to a
> > newcomer.
>
> I’m not a newcomer but I find it limpid: I think it immediately shows
> that this is about version changes and when they happened.  Maybe it
> could be decluttered a bit by using shorter date strings (like
> “Oct. 1st” instead of “2020-10-01 18:42:00”).

On my side, short date and short commit hash appearing (avoiding the
click) would be nice. :-)


> BTW, when looking at /output-history, the commits in “From” and “To” can
> be a bit confusing.  Take:
>
>   https://data.guix.gnu.org/repository/1/branch/master/package/openmpi/output-history
>
> I see “To” pointing to
> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=ec2a8f602ee0fd2fac2210eeae6d1da091bb47e4>,
> a commit that changes ‘mpd’, which ‘openmpi’ doesn’t depend on; this is
> completely bogus.  However, the next “From” points to the right commit:
> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d623ece56624fd7d611cdaf42f728b21d89fec5c>.
>
> It’s surprising that “To” and the immediately following “From” are not
> the same.

I recently have a very similar issue [1].  And based on my
investigations [2], the issue seems "commit date vs author date";
especially when merges are floating around.  Because of that, from
"From" to "To" is not always accurate.  Chris, could you confirm?

1: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00084.html>
2: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html>


All the best,
simon

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> Since blog post about Data Services seems floating around, I describe
> what I find useful as an end-user of the service.  And if you have not
> showed me IRL at FOSDEM (and you showed me several times ;-)), then I
> would not be using the service so often.

...

> Well, I could describe other use cases, but I think this one is really
> useful for the end-user:
>
>  - list all the versions available (since Dec. 2019 I guess)
>  - know which build and which not
>  - easily find the commit for “guix time-machine”

Thanks for putting this together Simon.

Something I've been thinking about in relation to this is making this
information more available. The Guix Data Service tries to structure the
data in a sensible manor, which doesn't always best fit how people most
often want to use it.

Maybe there could be a packages.guix.gnu.org service which pulls
together information about packages to meet the common needs that users
of Guix have. Things like:

 - Searching for packages by name
 - Looking at version availability
 - Looking at build success/substitute availability

Thanks again,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Sorry if I'm stating the obvious, but ideally it'd be nice if the
> interface would be so straightforward it didn't need any instructions to
> use :)
>
> In particular, the "Version From To" table is hard to grasp to a
> newcomer.
>
> During the Guix Days we talked about how to make it a bit more
> approachable.  From what I recall:
>
> - Maybe make the timeline bars more visible.

I've finally got around to looking at this now. I've changed them to a
slightly darker grey.

> - The timeline is not usable and does not convey much information beside
>   an approximate lifespan of versions, which I'm not sure many users
>   need.  Consider removing it, or displaying it in a separate graph?
>
>   Ultimately, we need a list of versions with their Guix builds.

Personally, that what I designed that page to convey, at least the grey
bars, they're just to give a sense of how the
versions/derivations/outputs have changed over time.

> - "From" and "To" should probably be renamed "From Guix build" and "To
>   Guix build" respectively.

I've made the commit hash more prominent now, which might help with
this.

> - (More information) should be renamed to something like "Package
>   information".

I've made this change now.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Pierre Neidhardt <mail@ambrevar.xyz> skribis:
>
>> In particular, the "Version From To" table is hard to grasp to a
>> newcomer.
>
> I’m not a newcomer but I find it limpid: I think it immediately shows
> that this is about version changes and when they happened.  Maybe it
> could be decluttered a bit by using shorter date strings (like
> “Oct. 1st” instead of “2020-10-01 18:42:00”).
>
> BTW, when looking at /output-history, the commits in “From” and “To” can
> be a bit confusing.  Take:
>
>   https://data.guix.gnu.org/repository/1/branch/master/package/openmpi/output-history
>
> I see “To” pointing to
> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=ec2a8f602ee0fd2fac2210eeae6d1da091bb47e4>,
> a commit that changes ‘mpd’, which ‘openmpi’ doesn’t depend on; this is
> completely bogus.  However, the next “From” points to the right commit:
> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d623ece56624fd7d611cdaf42f728b21d89fec5c>.
>
> It’s surprising that “To” and the immediately following “From” are not
> the same.
>
> WDYT, Chris?

So, I think the commits, plus the date at which that commit hit master
(which is the date shown) is the main source of the confusion here.

I think the data in the table makes sense when viewed as
revisions/commits, the "To" is the last revision with that
version/derivation/output.

The date by the "To" bit doesn't map neatly, as it's when the revision
was first on master, not the last moment when it was on master. Maybe it
would be clearer with that date removed.

Anyway, I've made a few changes to the display in these tables now.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

>> BTW, when looking at /output-history, the commits in “From” and “To” can
>> be a bit confusing.  Take:
>>
>>   https://data.guix.gnu.org/repository/1/branch/master/package/openmpi/output-history
>>
>> I see “To” pointing to
>> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=ec2a8f602ee0fd2fac2210eeae6d1da091bb47e4>,
>> a commit that changes ‘mpd’, which ‘openmpi’ doesn’t depend on; this is
>> completely bogus.  However, the next “From” points to the right commit:
>> <https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d623ece56624fd7d611cdaf42f728b21d89fec5c>.
>>
>> It’s surprising that “To” and the immediately following “From” are not
>> the same.
>
> I recently have a very similar issue [1].  And based on my
> investigations [2], the issue seems "commit date vs author date";
> especially when merges are floating around.  Because of that, from
> "From" to "To" is not always accurate.  Chris, could you confirm?
>
> 1: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00084.html>
> 2: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html>

The Guix Data Service stores the date in the email from the Git hook
that runs when one pushes to Savannah. That's the date you see for a
revision, and it should be pretty close to when it actually showed up in
the Git repository. It's not perfectly accurate, but it should be better
than either the commit date or author date from the Git metadata.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[zimoun]

Hi Chris,

On Mon, 16 Nov 2020 at 19:28, Christopher Baines <mail@cbaines.net> wrote:

>> I recently have a very similar issue [1].  And based on my
>> investigations [2], the issue seems "commit date vs author date";
>> especially when merges are floating around.  Because of that, from
>> "From" to "To" is not always accurate.  Chris, could you confirm?
>>
>> 1: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00084.html>
>> 2: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html>
>
> The Guix Data Service stores the date in the email from the Git hook
> that runs when one pushes to Savannah. That's the date you see for a
> revision, and it should be pretty close to when it actually showed up in
> the Git repository. It's not perfectly accurate, but it should be better
> than either the commit date or author date from the Git metadata.

Re-reading [2] above, I am confused.  Because the Data Service seems to
show the author date rather than the commit date or the header email
date.

The email database from where the Data Service is fetching is
guix-commits, right?

If yes, there is different dates that are appearing; for instance the
first I find: commit 5627bfe45ce46f498979b4ad2deab1fdfed22b6c authored
by Jason Conroy and pushed by mothacehe:

--8<---------------cut here---------------start------------->8---
From MAILER-DAEMON Thu Oct 01 03:28:02 2020
Date: Thu, 01 Oct 2020 03:27:57 -0400
X-List-Received-Date: Thu, 01 Oct 2020 07:28:00 -0000
--
AuthorDate: Sun Sep 27 13:16:39 2020 -0400
--8<---------------cut here---------------end--------------->8---

Message-ID: <20201001072757.23203.96292@vcs0.savannah.gnu.org>


And from my Git checkout: CommitDate: Thu Oct 1 09:27:33 2020 +0200.

(Modulo some timezone conversion, I guess.)


Maybe my investigations in [2] above are wrong but which date from the
email the Data Service is using?


I agree the most accurate is the date from the email header of
guix-commits.


Cheers,
simon

Re: Data Services: use cases

[zimoun]

Hi Chris,

On Mon, 16 Nov 2020 at 19:17, Christopher Baines <mail@cbaines.net> wrote:

>>  - list all the versions available (since Dec. 2019 I guess)
>>  - know which build and which not
>>  - easily find the commit for “guix time-machine”

[...]

> Maybe there could be a packages.guix.gnu.org service which pulls
> together information about packages to meet the common needs that users
> of Guix have. Things like:
>
>  - Searching for packages by name
>  - Looking at version availability
>  - Looking at build success/substitute availability

I do not know if the comparison makes sense: To me, the Data Service is
more-or-less the equivalent of:

   https://tracker.debian.org/pkg/guix

and what you are suggesting would be the equivalent of:

   https://packages.debian.org/experimental/guix

Are we on the same wavelength?  In other words, the Data Service is
devoted to Guix dev. and Package Service to Guix user.

In this picture, the usage that I described fits to the Package Service,
right?


Cheers,
simon

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2429 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> On Mon, 16 Nov 2020 at 19:28, Christopher Baines <mail@cbaines.net> wrote:
>
>>> I recently have a very similar issue [1].  And based on my
>>> investigations [2], the issue seems "commit date vs author date";
>>> especially when merges are floating around.  Because of that, from
>>> "From" to "To" is not always accurate.  Chris, could you confirm?
>>>
>>> 1: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00084.html>
>>> 2: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html>
>>
>> The Guix Data Service stores the date in the email from the Git hook
>> that runs when one pushes to Savannah. That's the date you see for a
>> revision, and it should be pretty close to when it actually showed up in
>> the Git repository. It's not perfectly accurate, but it should be better
>> than either the commit date or author date from the Git metadata.
>
> Re-reading [2] above, I am confused.  Because the Data Service seems to
> show the author date rather than the commit date or the header email
> date.
>
> The email database from where the Data Service is fetching is
> guix-commits, right?
>
> If yes, there is different dates that are appearing; for instance the
> first I find: commit 5627bfe45ce46f498979b4ad2deab1fdfed22b6c authored
> by Jason Conroy and pushed by mothacehe:
>
> --8<---------------cut here---------------start------------->8---
> From MAILER-DAEMON Thu Oct 01 03:28:02 2020
> Date: Thu, 01 Oct 2020 03:27:57 -0400
> X-List-Received-Date: Thu, 01 Oct 2020 07:28:00 -0000
> --
> AuthorDate: Sun Sep 27 13:16:39 2020 -0400
> --8<---------------cut here---------------end--------------->8---
>
> Message-ID: <20201001072757.23203.96292@vcs0.savannah.gnu.org>
>
>
> And from my Git checkout: CommitDate: Thu Oct 1 09:27:33 2020 +0200.
>
> (Modulo some timezone conversion, I guess.)
>
>
> Maybe my investigations in [2] above are wrong but which date from the
> email the Data Service is using?
>
>
> I agree the most accurate is the date from the email header of
> guix-commits.

So the date given by data.guix.gnu.org for this revision is:
2020-10-01 07:27:57 [1]

1: https://data.guix.gnu.org/revision/5627bfe45ce46f498979b4ad2deab1fdfed22b6c

Unfortunately, it's not explicit about the timezone, but it's UTC. This
matches the date in the email, as you include it above:

It's just expressed in a different timezone.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi Chris,
>
> On Mon, 16 Nov 2020 at 19:17, Christopher Baines <mail@cbaines.net> wrote:
>
>>>  - list all the versions available (since Dec. 2019 I guess)
>>>  - know which build and which not
>>>  - easily find the commit for “guix time-machine”
>
> [...]
>
>> Maybe there could be a packages.guix.gnu.org service which pulls
>> together information about packages to meet the common needs that users
>> of Guix have. Things like:
>>
>>  - Searching for packages by name
>>  - Looking at version availability
>>  - Looking at build success/substitute availability
>
> I do not know if the comparison makes sense: To me, the Data Service is
> more-or-less the equivalent of:
>
>    https://tracker.debian.org/pkg/guix
>
> and what you are suggesting would be the equivalent of:
>
>    https://packages.debian.org/experimental/guix
>
> Are we on the same wavelength?  In other words, the Data Service is
> devoted to Guix dev. and Package Service to Guix user.
>
> In this picture, the usage that I described fits to the Package Service,
> right?

Yeah, I think you're on the right track. I'm not set on mirroring the
exact Debian services though.

What I'm suggesting here is that there's some web pages that sit outside
of the Guix Data Service, and try and help users of Guix do things like
searching for packages, and looking at versions through time. Those
pages could be powered by data from data.guix.gnu.org.

A similar idea to the package search prototype here [1], that uses data
from data.guix.gnu.org.

1: http://guix-website-test.cbaines.net/packages/search

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[zimoun]

Cool!  I have seen that you added the commit!  Great.

On Mon, 16 Nov 2020 at 20:54, Christopher Baines <mail@cbaines.net> wrote:

>>>> 2: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html>

>>> The Guix Data Service stores the date in the email from the Git hook
>>> that runs when one pushes to Savannah. That's the date you see for a
>>> revision, and it should be pretty close to when it actually showed up in
>>> the Git repository. It's not perfectly accurate, but it should be better
>>> than either the commit date or author date from the Git metadata.

>> Maybe my investigations in [2] above are wrong but which date from the
>> email the Data Service is using?

I do not know what is wrong, but some thing is.  From

  https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html

the commit ranges:

https://data.guix.gnu.org/repository/1/branch/master/package/ghc-haddock/output-history

does not match with the Git history if I am not wrong.

The first commit producing j5llsz…-ghc-haddock is not
3922208091 but 12dc9f58c4.  And then compare:

                author date              commit date
3922208091|Thu Jul 23 14:03:19 2020|Fri Jul 24 14:21:31 2020
12dc9f58c4|Fri Jul 24 09:56:25 2020|Fri Jul 24 14:21:30 2020

I have not checked what guix-commits says.  Maybe the mess is
overthere.


All the best,
simon

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2478 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Cool!  I have seen that you added the commit!  Great.
>
> On Mon, 16 Nov 2020 at 20:54, Christopher Baines <mail@cbaines.net> wrote:
>
>>>>> 2: <https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html>
>
>>>> The Guix Data Service stores the date in the email from the Git hook
>>>> that runs when one pushes to Savannah. That's the date you see for a
>>>> revision, and it should be pretty close to when it actually showed up in
>>>> the Git repository. It's not perfectly accurate, but it should be better
>>>> than either the commit date or author date from the Git metadata.
>
>>> Maybe my investigations in [2] above are wrong but which date from the
>>> email the Data Service is using?
>
> I do not know what is wrong, but some thing is.  From
>
>   https://lists.gnu.org/archive/html/guix-devel/2020-10/msg00096.html
>
> the commit ranges:
>
> https://data.guix.gnu.org/repository/1/branch/master/package/ghc-haddock/output-history
>
> does not match with the Git history if I am not wrong.
>
> The first commit producing j5llsz…-ghc-haddock is not
> 3922208091 but 12dc9f58c4.  And then compare:
>
>                 author date              commit date
> 3922208091|Thu Jul 23 14:03:19 2020|Fri Jul 24 14:21:31 2020
> 12dc9f58c4|Fri Jul 24 09:56:25 2020|Fri Jul 24 14:21:30 2020
>
> I have not checked what guix-commits says.  Maybe the mess is
> overthere.

The author date and commit date won't necessarily match the date
whatever branch is updated (and it's when that branch was updated that
the Guix Data Service tries to store).

Generally, the author date will match or precede the commit date, which
will match or precede the date the commit hit whatever branch.

Taking 3922208091 as an example, I pulled the commit dates in UTC [1]:

AuthorDate:        Thu Jul 23 12:03:19 2020
CommitDate:        Fri Jul 24 12:21:31 2020
Guix Data Service: 2020-07-24 12:22:11 [2]


1: TZ=UTC git show --pretty=fuller 3922208091
2: https://data.guix.gnu.org/revision/39222080911eaf3d7f74effe4467c1a04464aef3

I think the author date says something about when the commit was
originally created. Rebasing and adding the Signed-off-by by bit will
update the commit date (but not the author date). Finally, the Guix Data
Service will store when the branch was updated (it uses the date of the
email as a proxy for this).

Does that clear things up?

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[zimoun]

On Mon, 16 Nov 2020 at 21:50, Christopher Baines <mail@cbaines.net> wrote:

>> The first commit producing j5llsz…-ghc-haddock is not
>> 3922208091 but 12dc9f58c4.  And then compare:
>>
>>                 author date              commit date
>> 3922208091|Thu Jul 23 14:03:19 2020|Fri Jul 24 14:21:31 2020
>> 12dc9f58c4|Fri Jul 24 09:56:25 2020|Fri Jul 24 14:21:30 2020
>>
>> I have not checked what guix-commits says.  Maybe the mess is
>> overthere.
>
> The author date and commit date won't necessarily match the date
> whatever branch is updated (and it's when that branch was updated that
> the Guix Data Service tries to store).

Yes.

> Generally, the author date will match or precede the commit date, which
> will match or precede the date the commit hit whatever branch.

Yes.


> Taking 3922208091 as an example, I pulled the commit dates in UTC [1]:
>
> AuthorDate:        Thu Jul 23 12:03:19 2020
> CommitDate:        Fri Jul 24 12:21:31 2020
> Guix Data Service: 2020-07-24 12:22:11 [2]

My point was: 3922208091 could not be the “correct“ and the correct one
could be 12dc9f58c4.  By correct, read the one introducing the output
j5llsz…-ghc-haddock.  And git show says:

                 author date              commit date
 3922208091|Thu Jul 23 14:03:19 2020|Fri Jul 24 14:21:31 2020
 12dc9f58c4|Fri Jul 24 09:56:25 2020|Fri Jul 24 14:21:30 2020

In the final Git tree history, the commit introducing
j5llsz…-ghc-haddock, i.e., changing from one row to another in the
output-history page, is the commit 12dc9f58c4.  Because of this 1 sec.
difference.

The question is what is the header email date?  But…

> 1: TZ=UTC git show --pretty=fuller 3922208091
> 2:
> https://data.guix.gnu.org/revision/39222080911eaf3d7f74effe4467c1a04464aef3

…the revision 12dc9f58c4 is not available.  See:

http://data.guix.gnu.org/revision/12dc9f58c422c06bf9950f21c54ca3df1dc40af1

Maybe that the explanation.  I do not know.


> I think the author date says something about when the commit was
> originally created. Rebasing and adding the Signed-off-by by bit will
> update the commit date (but not the author date). Finally, the Guix
> Data

Exact.

> Service will store when the branch was updated (it uses the date of the
> email as a proxy for this).

Now, after this discussion, I am not convinced anymore that the email
header date is the right thing because we (user) do not have this
information when debugging locally; only the Git checkout.  Anyway.


All is clear and I am ready to move on. :-) Thanks for all the explanations.


Cheers,
simon

Re: Data Services: use cases

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> I've finally got around to looking at this now. I've changed them to a
> slightly darker grey.

It's better indeed!

> I've made the commit hash more prominent now, which might help with
> this.

Good!

>> - (More information) should be renamed to something like "Package
>>   information".
>
> I've made this change now.

Great!

Something else: I couldn't figure out how to access

https://data.guix.gnu.org/repository/1/branch/master/package/emacs

From the home page, or even from the Emacs derivation page:

https://data.guix.gnu.org/revision/dc14cbda517a4d7f5c76f4b8ee969a6b0bb728cc/package/emacs/27.1?locale=en_US.UTF-8

Missing link?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>>> - (More information) should be renamed to something like "Package
>>>   information".
>>
>> I've made this change now.
>
> Great!
>
> Something else: I couldn't figure out how to access
>
> https://data.guix.gnu.org/repository/1/branch/master/package/emacs

I you click "Latest processed revision" on the branch page:

http://data.guix.gnu.org/repository/1/branch/master

Then "View packages", then click through to Emacs.

http://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package/emacs/27.1?locale=en_US.UTF-8

You should see a "Version history" link. That'll get you to that page.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

Thanks!  Maybe this could be added to the View packages page as well to
avoid too many page loads?  This is one of the most useful pages in my
experience, and it's like 3-4 pages away.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Linking to the data service

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> What I'm suggesting here is that there's some web pages that sit outside
> of the Guix Data Service, and try and help users of Guix do things like
> searching for packages, and looking at versions through time. Those
> pages could be powered by data from data.guix.gnu.org.

I added links to the version history in package pages at
<https://hpc.guix.info/browse> (which currently needs an update.)

Perhaps we could add links to the version history and package
reproducibility bits at <https://guix.gnu.org/packages>?  Any takers?

Thanks,
Ludo’.

Re: Data Services: use cases

[zimoun]

Hi Chris,

On Mon, 16 Nov 2020 at 21:08, Christopher Baines <mail@cbaines.net> wrote:

> Yeah, I think you're on the right track. I'm not set on mirroring the
> exact Debian services though.

Yeah, for sure.  Just to fix the idea by approximated examples. :-)


> A similar idea to the package search prototype here [1], that uses data
> from data.guix.gnu.org.

Cool!

> 1: http://guix-website-test.cbaines.net/packages/search

From:

  http://guix-website-test.cbaines.net/packages/search?search=gmsh

points to:

  http://guix-website-test.cbaines.net/packages/gmsh-4.6.0

which does not exist.  That’s because it is a prototype I guess.


Cheers,
simon

Re: Linking to the data service

[zimoun]

Hi Ludo,

On Tue, 17 Nov 2020 at 14:19, Ludovic Courtès <ludo@gnu.org> wrote:
> Hi!
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> What I'm suggesting here is that there's some web pages that sit outside
>> of the Guix Data Service, and try and help users of Guix do things like
>> searching for packages, and looking at versions through time. Those
>> pages could be powered by data from data.guix.gnu.org.
>
> I added links to the version history in package pages at
> <https://hpc.guix.info/browse> (which currently needs an update.)

From <https://hpc.guix.info/browse> I type gmsh and then click to:

   <https://hpc.guix.info/package/gmsh>

        Uh-oh...

        The package is gone!

Guix <https://hpc.guix.info/package/guix> is also gone.


> Perhaps we could add links to the version history and package
> reproducibility bits at <https://guix.gnu.org/packages>?  Any takers?

Yeah, this could be helpful to have a button which points to:

<https://data.guix.gnu.org/repository/1/branch/master/package/gmsh/output-history>

The reproducibility seems still a bit more experimental to me.


All the best,
simon

Re: Data Services: use cases

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]


Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Thanks!  Maybe this could be added to the View packages page as well to
> avoid too many page loads?  This is one of the most useful pages in my
> experience, and it's like 3-4 pages away.

Do you mean link to something else on this [1] page?

1: http://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Data Services: use cases

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

Christopher Baines <mail@cbaines.net> writes:

> 1: http://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages

Exactly, for instance below the "More information" link.

It would be an opportunity to rename "More information" to something
more explicit ;)

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: Data Services: use cases

[zimoun]

Hi Chris

>> Service will store when the branch was updated (it uses the date of the
>> email as a proxy for this).
>
> Now, after this discussion, I am not convinced anymore that the email
> header date is the right thing because we (user) do not have this
> information when debugging locally; only the Git checkout.  Anyway.

Just another example to show the issues about the dates.  Let’s give a
look at Gmsh (which I know a bit).

From this page (not matter the failure, which is another story):

<https://data.guix.gnu.org/repository/1/branch/master/package/gmsh/output-history>

the Data Service that the commit changing the output is b5e51a06.  I
claim the Data Service is wrong. :-) And I suspect issue with the dates;
as we have already talked about.


From the Data Service, the range commit is:

   b5e51a06 (first)
   3b026eb3 (last) 

Give a look at:

--8<---------------cut here---------------start------------->8---
$ git --no-pager log --pretty=format:"%h %ad|%cd %s" 3b026eb3^..b5e51a06 --date=local
b5e51a06de Thu Oct 22 16:14:07 2020|Fri Oct 23 16:35:39 2020 gnu: python-black: Update to 20.8b1.
7128174201 Sun Sep 20 23:42:04 2020|Fri Oct 23 16:35:39 2020 gnu: linux-libre-with-bpf: Update to 5.8.
1893c80fc1 Mon Oct 12 18:08:47 2020|Fri Oct 23 16:35:39 2020 gnu: kodi: Add nfs functionality.
c5372cd488 Fri Oct 23 12:12:34 2020|Fri Oct 23 16:35:39 2020 gnu: hdf5: Default to 1.10.
c9a656a5d0 Fri Oct 23 12:11:41 2020|Fri Oct 23 16:35:38 2020 gnu: imp: Update to 2.13.0.
bfb65a6768 Fri Oct 23 12:10:32 2020|Fri Oct 23 16:35:38 2020 installer: Make the network management selection window tighter.
3b026eb3b1 Fri Oct 23 15:34:50 2020|Fri Oct 23 15:40:36 2020 Revert "gnu: u-boot: Update to 2020.10."
--8<---------------cut here---------------end--------------->8---

I am sure because I know the Gmsh internals that the offending is:
“c5372cd488 gnu: hdf5: Default to 1.10.”  which is verified by:

        $ guix graph --path gmsh hdf5
        gmsh@4.6.0
        hdf5@1.10.7

and note that using the commit b5e51a06de pointed by the Data Service,
the culprit should be python-black.  But this package does not appear in the
graph of gmsh:

        $ guix graph --path gmsh python-black -t bag
        guix graph: error: no path from 'gmsh@4.6.0' to 'python-black@20.8b1'


Ok, so then let examine guix-commits and the mbox file, since the Data
Serive uses that.


From the message ID <20201023143628.6130.60435@vcs0.savannah.gnu.org> it reads,

  Date: Fri, 23 Oct 2020 10:36:29 -0400 (EDT)

which is the date reported by the Data Service.  So far so good. :-)

The message says:

  civodul pushed a change to branch master
  in repository guix.

      from 3b026eb  Revert "gnu: u-boot: Update to 2020.10."
       new bfb65a6  installer: Make the network management selection window tighter.
       new c9a656a  gnu: imp: Update to 2.13.0.
       new c5372cd  gnu: hdf5: Default to 1.10.
       new 1893c80  gnu: kodi: Add nfs functionality.
       new 7128174  gnu: linux-libre-with-bpf: Update to 5.8.
       new b5e51a0  gnu: python-black: Update to 20.8b1.

Hum, it is because civodul probably sent a series by batch.  But this
should not be used since it is not a real commit. IMHO.


Now, let compare c5372cd (good) and b5e51a0 (bad):

c5372cd gnu: hdf5: Default to 1.10.

        From MAILER-DAEMON Fri Oct 23 10:36:34 2020
        Date: Fri, 23 Oct 2020 10:36:30 -0400 (EDT)
        X-List-Received-Date: Fri, 23 Oct 2020 14:36:32 -0000

b5e51a0  gnu: python-black: Update to 20.8b1.

        From MAILER-DAEMON Fri Oct 23 10:36:41 2020
        Date: Fri, 23 Oct 2020 10:36:32 -0400 (EDT
        X-List-Received-Date: Fri, 23 Oct 2020 14:36:34 -0000


Therefore, the Data Service somehow uses incorrect dates.

Well, the Data Service should filter out the messages only containing
the header X-Git-Rev (commit hash) and use the date (Date) of the
message, which is, IMHO, the most accurate way to do.  Instead of rely
on the date of message containing such headers:

        X-Git-Oldrev: 3b026eb3b1dd92c653d1a640b1280b96cfea3084
        X-Git-Newrev: b5e51a06de08f446a07f678ff806cd71a355f7

which can miss the correct commit introducing a change of derivation.


Hope that helps,
simon

Fix incorrect date and commit in Data Service

[zimoun]

Hi again,

On Tue, 17 Nov 2020 at 22:43, zimoun <zimon.toutoune@gmail.com> wrote:

> Well, the Data Service should filter out the messages only containing
> the header X-Git-Rev (commit hash) and use the date (Date) of the
> message, which is, IMHO, the most accurate way to do.  Instead of rely
> on the date of message containing such headers:
>
>         X-Git-Oldrev: 3b026eb3b1dd92c653d1a640b1280b96cfea3084
>         X-Git-Newrev: b5e51a06de08f446a07f678ff806cd71a355f7
>
> which can miss the correct commit introducing a change of derivation.

The file data-service/branch-updated-emails.scm contains:

         (x-git-newrev  (assq-ref headers 'x-git-newrev)))

which is the issue.  Instead, it should be

         (x-git-rev  (assq-ref headers 'x-git-rev)))

but maybe some filtering should be applied.

Well, to be continued…


All the best,
simon

Re: Fix incorrect date and commit in Data Service

[zimoun]

Hi again again,

On Tue, 17 Nov 2020 at 22:57, zimoun <zimon.toutoune@gmail.com> wrote:

>> Well, the Data Service should filter out the messages only containing
>> the header X-Git-Rev (commit hash) and use the date (Date) of the
>> message, which is, IMHO, the most accurate way to do.  Instead of rely
>> on the date of message containing such headers:
>>
>>         X-Git-Oldrev: 3b026eb3b1dd92c653d1a640b1280b96cfea3084
>>         X-Git-Newrev: b5e51a06de08f446a07f678ff806cd71a355f7
>>
>> which can miss the correct commit introducing a change of derivation.
>
> The file data-service/branch-updated-emails.scm contains:
>
>          (x-git-newrev  (assq-ref headers 'x-git-newrev)))
>
> which is the issue.  Instead, it should be
>
>          (x-git-rev  (assq-ref headers 'x-git-rev)))
>
> but maybe some filtering should be applied.

Following the discussion we add on IRC, if process all the commits is
too much for the Data Service, undetermined yet, processing all the
“state” could be only done for failing packages.  Because it is the
concrete case where I (maybe we?) want  exactness.

Thank you for all the explanations.  Things clearer now. :-)


All the best,
simon