Can we find a better idiom for unversioned packages?

Can we find a better idiom for unversioned packages?

[Sarah Morgensen]

Hello Guix,

Currently, there are about 1500 packages defined like this:

--8<---------------cut here---------------start------------->8---
(define-public sbcl-feeder
  (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
        (revision "1"))
    (package
      [...])))
--8<---------------cut here---------------end--------------->8---

I feel like there are some issues with this idiom (in no particular
order):

1. When converting between this idiom and regularly versioned packages,
the git diff shows the whole package changing because of the indentation
change.

2. We cannot get at the source location for the definition of 'commit' or
'revision'.  This would be useful for updating these packages with `guix
refresh -u`.  There is a proposed patch [0] to work around this, but it
*is* a workaround.

3. Packages inheriting from it lose the definitions.  For actual fields,
we have e.g. `(package-version this-package)`, but we have no equivalent
for these.

4. Horizontal space is at a premium, and an extra two spaces here and
there add up.  (Personally, I think we could do with a
define-public-package macro to save another two spaces, but that's for
another day...)

5. The closest thing we have to a standardized way of generating
versions for these packages is `(version (git-version "0.0.0" revision
commit))`.  We can do better than that boilerplate.

6. Not a direct complaint, but I feel like the overall package interface
was designed before straight-from-vcs unversioned packages were so
common, and so this idiom developed organically to work around that.

I do not have a specific solution in mind, but I think there must be
one.  I do have a few half-baked ideas, but I'm curious what we can all
come up with together.  Or maybe you'll just tell me I'm just being
awfully picky :)

[0] https://issues.guix.gnu.org/50286

--
Sarah

Re: Can we find a better idiom for unversioned packages?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 3018 bytes --]

Sarah Morgensen schreef op di 31-08-2021 om 12:57 [-0700]:
> Hello Guix,
> 
> Currently, there are about 1500 packages defined like this:
> 
> --8<---------------cut here---------------start------------->8---
> (define-public sbcl-feeder
>   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
>         (revision "1"))
>     (package
>       [...])))
> --8<---------------cut here---------------end--------------->8---
> 
> I feel like there are some issues with this idiom (in no particular
> order):
> 
> 1. When converting between this idiom and regularly versioned packages,
> the git diff shows the whole package changing because of the indentation
> change.
> 
> 2. We cannot get at the source location for the definition of 'commit' or
> 'revision'.  This would be useful for updating these packages with `guix
> refresh -u`.  There is a proposed patch [0] to work around this, but it
> *is* a workaround.
> 
> 3. Packages inheriting from it lose the definitions.  For actual fields,
> we have e.g. `(package-version this-package)`, but we have no equivalent
> for these.
> 
> 4. Horizontal space is at a premium, and an extra two spaces here and
> there add up.  (Personally, I think we could do with a
> define-public-package macro to save another two spaces, but that's for
> another day...)
> 
> 5. The closest thing we have to a standardized way of generating
> versions for these packages is `(version (git-version "0.0.0" revision
> commit))`.  We can do better than that boilerplate.

Suggestion: extend the 'version' field.  More specifically,
introduce a new record <full-version>, like this:

(define-record-type* <extended-version> extended-version make-extended-version
  extended-version? this-version
  ;; something like 1.2.3 (TODO better name)
  (base extended-version-base)
  (revision extended-version-revision)
  (commit extended-version-commit))

(define (version->string version)
  (match version
    ((? string?) version)
    (($ <extended-version> ...) code from original git-version  and hg-version)))

;; TODO:
;; adjust git-file-name and hg-file-name to accept <extended-version> records
;; (as well as the ‘old style’ for compatibility)

To be used like:

(define-public sbcl-feeder
  (name "sbcl-feeder")
  (version (extended-version
             (base "1.0.0")
             (revision 1)
             (commit "b05f517d7729564575cc809e086c262646a94d34")))
 (source
  (origin
    (method git-fetch)
    (uri (git-reference ...)
           (url ...)
           ;; git-reference needs to be extended to retrieve the commit from the version
           (version version)))
    (file-name (git-file-name "feeder" version))
    (sha256 ...)))
 [...])

That should address 1,2,3,4 and 5.

One problem with this approach is that most users of 'package-version' expect
it to return a string.  Maybe adding a keyword argument '#:full-version? #t/#f'
defaulting to #f would work?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 3043 bytes --]

On Tue, Aug 31 2021, Maxime Devos wrote:

> Sarah Morgensen schreef op di 31-08-2021 om 12:57 [-0700]:
>> Hello Guix,
>> 
>> Currently, there are about 1500 packages defined like this:
>> 
>> --8<---------------cut here---------------start------------->8---
>> (define-public sbcl-feeder
>>   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
>>         (revision "1"))
>>     (package
>>       [...])))
>> --8<---------------cut here---------------end--------------->8---
>> 
>> I feel like there are some issues with this idiom (in no particular
>> order):
>> 
>> 1. When converting between this idiom and regularly versioned packages,
>> the git diff shows the whole package changing because of the indentation
>> change.
>> 
>> 2. We cannot get at the source location for the definition of 'commit' or
>> 'revision'.  This would be useful for updating these packages with `guix
>> refresh -u`.  There is a proposed patch [0] to work around this, but it
>> *is* a workaround.
>> 
>> 3. Packages inheriting from it lose the definitions.  For actual fields,
>> we have e.g. `(package-version this-package)`, but we have no equivalent
>> for these.
>> 
>> 4. Horizontal space is at a premium, and an extra two spaces here and
>> there add up.  (Personally, I think we could do with a
>> define-public-package macro to save another two spaces, but that's for
>> another day...)
>> 
>> 5. The closest thing we have to a standardized way of generating
>> versions for these packages is `(version (git-version "0.0.0" revision
>> commit))`.  We can do better than that boilerplate.
>
> Suggestion: extend the 'version' field.  More specifically,
> introduce a new record <full-version>, like this:
>
> (define-record-type* <extended-version> extended-version make-extended-version
>   extended-version? this-version
>   ;; something like 1.2.3 (TODO better name)
>   (base extended-version-base)
>   (revision extended-version-revision)
>   (commit extended-version-commit))
>
> (define (version->string version)
>   (match version
>     ((? string?) version)
>     (($ <extended-version> ...) code from original git-version  and hg-version)))
>
> ;; TODO:
> ;; adjust git-file-name and hg-file-name to accept <extended-version> records
> ;; (as well as the ‘old style’ for compatibility)
>
> To be used like:
>
> (define-public sbcl-feeder
>   (name "sbcl-feeder")
>   (version (extended-version
>              (base "1.0.0")
>              (revision 1)
>              (commit "b05f517d7729564575cc809e086c262646a94d34")))
>  (source
>   (origin
>     (method git-fetch)
>     (uri (git-reference ...)
>            (url ...)
>            ;; git-reference needs to be extended to retrieve the commit from the version
>            (version version)))
>     (file-name (git-file-name "feeder" version))
>     (sha256 ...)))
>  [...])

How will this work for SVN and CVS?  I am not familiar with either, but
I know that SVN has its own ‘revision’ thing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1314 bytes --]

> > (define-public sbcl-feeder
> >   (name "sbcl-feeder")
> >   (version (extended-version
> >              (base "1.0.0")
> >              (revision 1)
> >              (commit "b05f517d7729564575cc809e086c262646a94d34")))
> >  (source
> >   (origin
> >     (method git-fetch)
> >     (uri (git-reference ...)
> >            (url ...)
> >            ;; git-reference needs to be extended to retrieve the commit from the version
> >            (version version)))
> >     (file-name (git-file-name "feeder" version))
> >     (sha256 ...)))
> >  [...])
> 
> How will this work for SVN and CVS?  I am not familiar with either, but
> I know that SVN has its own ‘revision’ thing.

Maybe:

(extended-version
  (base-version "14.12.0")
  (revision 123)  ; what guix considers a revision
  (commit 38938)) ; what svn considers a revision (TODO figure out non-confusing terminology)

(define (version->string v)
  (cond ((string? v) v)
        ;; for SVN
        (REVISION is set but COMMIT isn't
         (string-append base-version "-" (number->string revision)))
        ;; for git and mercurial
        (both REVISION and COMMIT are set
         the code from git-version)))

Or have separate types <git-version>, <mercurial-version>, <svn-version> ...

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Liliana Marie Prikler]

Hi

Am Dienstag, den 31.08.2021, 23:20 +0200 schrieb Maxime Devos:
> Sarah Morgensen schreef op di 31-08-2021 om 12:57 [-0700]:
> > Hello Guix,
> > 
> > Currently, there are about 1500 packages defined like this:
> > 
> > --8<---------------cut here---------------start------------->8---
> > (define-public sbcl-feeder
> >   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
> >         (revision "1"))
> >     (package
> >       [...])))
> > --8<---------------cut here---------------end--------------->8---
> > 
> > I feel like there are some issues with this idiom (in no particular
> > order):
> > 
> > 1. When converting between this idiom and regularly versioned
> > packages, the git diff shows the whole package changing because of
> > the indentation change.
If you are worried about that in a frequently changing package, you
could set both to *unspecified* or #f instead, which would cause any
reference to them in a string manipulation context to fail.  I don't
think that such transitions are too frequent, though, as the point is
rather to discourage them where not absolutely necessary and to use
upstream releases instead.

> > 2. We cannot get at the source location for the definition of
> > 'commit' or 'revision'.  This would be useful for updating these
> > packages with `guix refresh -u`.  There is a proposed patch [0] to
> > work around this, but it *is* a workaround.
Other versioning idioms would also be workarounds, wouldn't they?

> > 3. Packages inheriting from it lose the definitions.  For actual
> > fields, we have e.g. `(package-version this-package)`, but we have
> > no equivalent for these.
What purpose would extracting those serve however? 

> > 4. Horizontal space is at a premium, and an extra two spaces here
> > and there add up.  (Personally, I think we could do with a
> > define-public-package macro to save another two spaces, but that's
> > for another day...)
The worst offenders in horizontal space typically come from the
packages themselves and going from 79 to 81 in select outliers is AFAIU
acceptable.

> > 5. The closest thing we have to a standardized way of generating
> > versions for these packages is `(version (git-version "0.0.0"
> > revision commit))`.  We can do better than that boilerplate.
This concerns packages without a public release, many of which never
make one in the belief that commit hashes are valid versions (they are
not).

> Suggestion: extend the 'version' field.  More specifically,
> introduce a new record <full-version>, like this:
> 
> (define-record-type* <extended-version> extended-version make-
> extended-version
>   extended-version? this-version
>   ;; something like 1.2.3 (TODO better name)
>   (base extended-version-base)
>   (revision extended-version-revision)
>   (commit extended-version-commit))
> 
> (define (version->string version)
>   (match version
>     ((? string?) version)
>     (($ <extended-version> ...) code from original git-version  and
> hg-version)))
> 
> ;; TODO:
> ;; adjust git-file-name and hg-file-name to accept <extended-version> 
> records
> ;; (as well as the ‘old style’ for compatibility)
> 
> To be used like:
> 
> (define-public sbcl-feeder
>   (name "sbcl-feeder")
>   (version (extended-version
>              (base "1.0.0")
>              (revision 1)
>              (commit "b05f517d7729564575cc809e086c262646a94d34")))
>  (source
>   (origin
>     (method git-fetch)
>     (uri (git-reference ...)
>            (url ...)
>            ;; git-reference needs to be extended to retrieve the
> commit from the version
>            (version version)))
>     (file-name (git-file-name "feeder" version))
>     (sha256 ...)))
>  [...])
> 
> That should address 1,2,3,4 and 5.
> 
> One problem with this approach is that most users of 'package-
> version' expect it to return a string.  Maybe adding a keyword
> argument '#:full-version? #t/#f' defaulting to #f would work?
I think the bigger problem here is that you're moving bits meant for
the origin into the version only to be able to point to the version
from the origin.  Even accepting that you could use "commit" or a
separate field to encode SVN/CVS revision numbers instead of hashes,
everything beyond the revision number is basically pointless from a
versioning scheme POV and only really useful to fetch the source code. 
As Xinglu Chen points out, a commit hash encodes remarkably little on
its own.

Perhaps we could make the point that origins can be versioned just like
packages can and should think about where the version field really
belongs.  I think having a per-origin version field and making it so
that the package defaults to the origin version unless the package
itself has a version specified, might be a better solution
philosophically speaking.  Technically, it would only solve 1, 4 and 5,
but there's a separate workaround for 2 and I don't really think 3 to
be that big of an issue.

Regards

Re: Can we find a better idiom for unversioned packages?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 3856 bytes --]

Liliana Marie Prikler schreef op wo 01-09-2021 om 15:33 [+0200]:
> Hi
> 
> Am Dienstag, den 31.08.2021, 23:20 +0200 schrieb Maxime Devos:
> > Sarah Morgensen schreef op di 31-08-2021 om 12:57 [-0700]:
> > > Hello Guix,
> > > 
> > > Currently, there are about 1500 packages defined like this:
> > > 
> > > --8<---------------cut here---------------start------------->8---
> > > (define-public sbcl-feeder
> > >   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
> > >         (revision "1"))
> > >     (package
> > >       [...])))
> > > --8<---------------cut here---------------end--------------->8---
> > > 
> > > I feel like there are some issues with this idiom (in no particular
> > > order):
> > > 
> > > 1. When converting between this idiom and regularly versioned
> > > packages, the git diff shows the whole package changing because of
> > > the indentation change.
> If you are worried about that in a frequently changing package, you
> could set both to *unspecified* or #f instead, which would cause any
> reference to them in a string manipulation context to fail.  I don't
> think that such transitions are too frequent, though, as the point is
> rather to discourage them where not absolutely necessary and to use
> upstream releases instead.
> 
> > > 2. We cannot get at the source location for the definition of
> > > 'commit' or 'revision'.  This would be useful for updating these
> > > packages with `guix refresh -u`.  There is a proposed patch [0] to
> > > work around this, but it *is* a workaround.
> Other versioning idioms would also be workarounds, wouldn't they?
> 
> > > 3. Packages inheriting from it lose the definitions.  For actual
> > > fields, we have e.g. `(package-version this-package)`, but we have
> > > no equivalent for these.
> What purpose would extracting those serve however? 

Not losing the revision is useful for things like <https://issues.guix.gnu.org/50072>,
to be able to determine the old revision.  (That's not about inheriting
packages though.)
> [...]

> > To be used like:
> > 
> > (define-public sbcl-feeder
> >   (name "sbcl-feeder")
> >   (version (extended-version
> >              (base "1.0.0")
> >              (revision 1)
> >              (commit "b05f517d7729564575cc809e086c262646a94d34")))
> >  (source
> >   (origin
> >     (method git-fetch)
> >     (uri (git-reference ...)
> >            (url ...)
> >            ;; git-reference needs to be extended to retrieve the
> > commit from the version
> >            (version version)))
> >     (file-name (git-file-name "feeder" version))
> >     (sha256 ...)))
> >  [...])
> > 
> > That should address 1,2,3,4 and 5.
> > 
> > One problem with this approach is that most users of 'package-
> > version' expect it to return a string.  Maybe adding a keyword
> > argument '#:full-version? #t/#f' defaulting to #f would work?
> I think the bigger problem here is that you're moving bits meant for
> the origin into the version only to be able to point to the version
> from the origin.  Even accepting that you could use "commit" or a
> separate field to encode SVN/CVS revision numbers instead of hashes,
> everything beyond the revision number is basically pointless from a
> versioning scheme POV and only really useful to fetch the source code. 
> As Xinglu Chen points out, a commit hash encodes remarkably little on
> its own.

The commit is largely useless, ok.  If the (first few characters of) the
git commit/svn revision are removed from the version strings, it can be
removed from the proposed extended-version.

Otherwise, it would seem you wouldn't mind extended-version if it
only had the 'base version' and 'revision' field (in the guix sense,
not the SVN sense of revision), or am I misunderstanding here?

Geetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Liliana Marie Prikler]

Am Mittwoch, den 01.09.2021, 18:39 +0200 schrieb Maxime Devos:
> Liliana Marie Prikler schreef op wo 01-09-2021 om 15:33 [+0200]:
> > Hi
> > 
> > Am Dienstag, den 31.08.2021, 23:20 +0200 schrieb Maxime Devos:
> > > Sarah Morgensen schreef op di 31-08-2021 om 12:57 [-0700]:
> > > > Hello Guix,
> > > > 
> > > > Currently, there are about 1500 packages defined like this:
> > > > 
> > > > --8<---------------cut here---------------start------------->8-
> > > > --
> > > > (define-public sbcl-feeder
> > > >   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
> > > >         (revision "1"))
> > > >     (package
> > > >       [...])))
> > > > --8<---------------cut here---------------end--------------->8-
> > > > --
> > > > 
> > > > I feel like there are some issues with this idiom (in no
> > > > particular
> > > > order):
> > > > 
> > > > 1. When converting between this idiom and regularly versioned
> > > > packages, the git diff shows the whole package changing because
> > > > of
> > > > the indentation change.
> > If you are worried about that in a frequently changing package, you
> > could set both to *unspecified* or #f instead, which would cause
> > any
> > reference to them in a string manipulation context to fail.  I
> > don't
> > think that such transitions are too frequent, though, as the point
> > is
> > rather to discourage them where not absolutely necessary and to use
> > upstream releases instead.
> > 
> > > > 2. We cannot get at the source location for the definition of
> > > > 'commit' or 'revision'.  This would be useful for updating
> > > > these
> > > > packages with `guix refresh -u`.  There is a proposed patch [0]
> > > > to
> > > > work around this, but it *is* a workaround.
> > Other versioning idioms would also be workarounds, wouldn't they?
> > 
> > > > 3. Packages inheriting from it lose the definitions.  For
> > > > actual
> > > > fields, we have e.g. `(package-version this-package)`, but we
> > > > have
> > > > no equivalent for these.
> > What purpose would extracting those serve however? 
> 
> Not losing the revision is useful for things like 
> <https://issues.guix.gnu.org/50072>, to be able to determine the old
> revision.  (That's not about inheriting packages though.)
Isn't that addressed by addressing the second point, though?  Like, if
you know the source location of the revision, you can read it back to
get the value itself (or possibly even access it as-is), no?
> > [...]
> > > To be used like:
> > > 
> > > (define-public sbcl-feeder
> > >   (name "sbcl-feeder")
> > >   (version (extended-version
> > >              (base "1.0.0")
> > >              (revision 1)
> > >              (commit
> > > "b05f517d7729564575cc809e086c262646a94d34")))
> > >  (source
> > >   (origin
> > >     (method git-fetch)
> > >     (uri (git-reference ...)
> > >            (url ...)
> > >            ;; git-reference needs to be extended to retrieve the
> > > commit from the version
> > >            (version version)))
> > >     (file-name (git-file-name "feeder" version))
> > >     (sha256 ...)))
> > >  [...])
> > > 
> > > That should address 1,2,3,4 and 5.
> > > 
> > > One problem with this approach is that most users of 'package-
> > > version' expect it to return a string.  Maybe adding a keyword
> > > argument '#:full-version? #t/#f' defaulting to #f would work?
> > I think the bigger problem here is that you're moving bits meant
> > for
> > the origin into the version only to be able to point to the version
> > from the origin.  Even accepting that you could use "commit" or a
> > separate field to encode SVN/CVS revision numbers instead of
> > hashes,
> > everything beyond the revision number is basically pointless from a
> > versioning scheme POV and only really useful to fetch the source
> > code. 
> > As Xinglu Chen points out, a commit hash encodes remarkably little
> > on its own.
> 
> The commit is largely useless, ok.  If the (first few characters of)
> the git commit/svn revision are removed from the version strings, it
> can be removed from the proposed extended-version.
> 
> Otherwise, it would seem you wouldn't mind extended-version if it
> only had the 'base version' and 'revision' field (in the guix sense,
> not the SVN sense of revision), or am I misunderstanding here?
That was not my suggestion, but let's entertain the idea, shall we?  In
that case, we would discard the commit part from the version field,
which might not be everyone's tea, but I'm more or less indifferent as
to whether to include the hash there or not – after all, even if it was
lacking, we'd quickly get it through inspecting the package
description.  If we simply didn't capture the hash at all except inside
the commit field of the origin, we would gain 1, 4 and 5 so the
question is whether we should have an extended version so as to update
the revision more easily...

My personal answer to this might be a disappointing one, as in that
case I believe we wouldn't even need procedures like git-version to
form them, but could instead use <upstream-version>-<guix-revision> as
a mere convention like many more popular distros already do.  If the
dash is overused for that, we could also use a different symbol, though
perhaps there's not that many on a typical US keyboard to reserve one
as a revision delimiter.

Making our rando commit git versions look like such other distro
versions does come at a disadvantage though, particularly when we look
at it through the lense of someone not used to Guix' versioning scheme.
Instead of telling us "yeah, this is the Nth time we picked a rando
commit since the last release and this time it's de4db3ef", users
coming from such distros would assume "oh well, this is still good ol'
1.0 with some more patches applied".  So while the commit itself does
not give us any particularly useful information (unless you're that
person who uses this part of the version string to look the commit up
on hubbucket), especially not when thinking in the context of
versioning scheme, it does provide the existential information of "hold
on, this is not a release commit, it's something else" and might thus
direct users to be a little more attentive when they'd otherwise go
"yep, upstream considers this solid and Guix considers it even more
solid, so it's the solidest".  Note, that this can be overcome both by
teaching/learning about it and by using a special sigil as mentioned
above.

All in all, I don't think putting too much "opinion" in the version
field by storing it as a record is a good idea.  It's fine if it's just
a string that can be parsed/version-compared.  We could also make it a
list like Emacs does and like we use internally, though I'm not too
certain of what the benefit of that would be at the cost of breaking
pretty much everything (and probably putting in some opinions as to
what is to be delimited by dots and what by dashes).

Regards

Re: Can we find a better idiom for unversioned packages?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

> > > > > 2. We cannot get at the source location for the definition of
> > > > > 'commit' or 'revision'.  This would be useful for updating
> > > > > these
> > > > > packages with `guix refresh -u`.  There is a proposed patch [0]
> > > > > to
> > > > > work around this, but it *is* a workaround.
> > > Other versioning idioms would also be workarounds, wouldn't they?
> > > 
> > > > > 3. Packages inheriting from it lose the definitions.  For
> > > > > actual
> > > > > fields, we have e.g. `(package-version this-package)`, but we
> > > > > have
> > > > > no equivalent for these.
> > > What purpose would extracting those serve however? 
> > 
> > Not losing the revision is useful for things like 
> > <https://issues.guix.gnu.org/50072>;, to be able to determine the old
> > revision.  (That's not about inheriting packages though.)

> Isn't that addressed by addressing the second point, though?  Like, if
> you know the source location of the revision, you can read it back to
> get the value itself (or possibly even access it as-is), no?

Indeed!  The patch [0] addresses the second point.  With that patch,
the proposed <extension-version> isn't required.  But also: some people
(at least Sarah?) consider [0] a work-around, and if guix used something
like <extended-version>, [0] wouldn't be necessary.

It doesn't really matter to me what we'll end up using in guix
in the long term, though in the short term, I would like something
like [0] to be merged, as it is used by the (not-yet submitted, needs some
cleanup, testing & rebasing) minetest updater, and it makes
<https://issues.guix.gnu.org/50072> work in more cases.

[0]: <https://issues.guix.gnu.org/50072>

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Liliana Marie Prikler]

Am Donnerstag, den 02.09.2021, 16:09 +0200 schrieb Maxime Devos:
> > > > > > 2. We cannot get at the source location for the definition
> > > > > > of
> > > > > > 'commit' or 'revision'.  This would be useful for updating
> > > > > > these
> > > > > > packages with `guix refresh -u`.  There is a proposed patch
> > > > > > [0]
> > > > > > to
> > > > > > work around this, but it *is* a workaround.
> > > > Other versioning idioms would also be workarounds, wouldn't
> > > > they?
> > > > 
> > > > > > 3. Packages inheriting from it lose the definitions.  For
> > > > > > actual fields, we have e.g. `(package-version this-
> > > > > > package)`, but we have no equivalent for these.
> > > > What purpose would extracting those serve however? 
> > > 
> > > Not losing the revision is useful for things like 
> > > <https://issues.guix.gnu.org/50072>;;, to be able to determine
> > > the old
> > > revision.  (That's not about inheriting packages though.)
> > Isn't that addressed by addressing the second point, though?  Like,
> > if
> > you know the source location of the revision, you can read it back
> > to
> > get the value itself (or possibly even access it as-is), no?
> 
> Indeed!  The patch [0] addresses the second point.  With that patch,
> the proposed <extension-version> isn't required.  But also: some
> people (at least Sarah?) consider [0] a work-around, and if guix used
> something like <extended-version>, [0] wouldn't be necessary.
> 
> It doesn't really matter to me what we'll end up using in guix
> in the long term, though in the short term, I would like something
> like [0] to be merged, as it is used by the (not-yet submitted, needs
> some cleanup, testing & rebasing) minetest updater, and it makes
> <https://issues.guix.gnu.org/50072> work in more cases.
That's not quite my point.  Sarah said that "inheriting definitions"
loses those values, which is true regardless of the merge of [0].  What
you said to answer my question w.r.t. why that matters was to repeat
the second point, which is addressed by [0].  In other words, what I'm
asking is why specifically inheriting (as in record inheriting) is made
to be such a big deal that it deserves its own point when I would
personally argue that it is not at all that important.

Greetings

Re: Can we find a better idiom for unversioned packages?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2590 bytes --]

Liliana Marie Prikler schreef op do 02-09-2021 om 16:20 [+0200]:
> Am Donnerstag, den 02.09.2021, 16:09 +0200 schrieb Maxime Devos:
> > > > > > > 2. We cannot get at the source location for the definition
> > > > > > > of
> > > > > > > 'commit' or 'revision'.  This would be useful for updating
> > > > > > > these
> > > > > > > packages with `guix refresh -u`.  There is a proposed patch
> > > > > > > [0]
> > > > > > > to
> > > > > > > work around this, but it *is* a workaround.
> > > > > Other versioning idioms would also be workarounds, wouldn't
> > > > > they?
> > > > > 
> > > > > > > 3. Packages inheriting from it lose the definitions.  For
> > > > > > > actual fields, we have e.g. `(package-version this-
> > > > > > > package)`, but we have no equivalent for these.
> > > > > What purpose would extracting those serve however? 
> > > > 
> > > > Not losing the revision is useful for things like 
> > > > <https://issues.guix.gnu.org/50072>;;;, to be able to determine
> > > > the old
> > > > revision.  (That's not about inheriting packages though.)
> > > Isn't that addressed by addressing the second point, though?  Like,
> > > if
> > > you know the source location of the revision, you can read it back
> > > to
> > > get the value itself (or possibly even access it as-is), no?
> > 
> > Indeed!  The patch [0] addresses the second point.  With that patch,
> > the proposed <extension-version> isn't required.  But also: some
> > people (at least Sarah?) consider [0] a work-around, and if guix used
> > something like <extended-version>, [0] wouldn't be necessary.
> > 
> > It doesn't really matter to me what we'll end up using in guix
> > in the long term, though in the short term, I would like something
> > like [0] to be merged, as it is used by the (not-yet submitted, needs
> > some cleanup, testing & rebasing) minetest updater, and it makes
> > <https://issues.guix.gnu.org/50072> work in more cases.
> That's not quite my point.  Sarah said that "inheriting definitions"
> loses those values, which is true regardless of the merge of [0].  What
> you said to answer my question w.r.t. why that matters was to repeat
> the second point, which is addressed by [0].

[extra newlines for clarity]

> In other words, what I'm
> asking is why specifically inheriting (as in record inheriting) is made
> to be such a big deal that it deserves its own point when I would
> personally argue that it is not at all that important.

Ok, I understand!  Though I don't have an answer to your question.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Jonathan McHugh]

September 1, 2021 8:35 PM, "Liliana Marie Prikler" <leo.prikler@student.tugraz.at> wrote

> Making our rando commit git versions look like such other distro
> versions does come at a disadvantage though, particularly when we look
> at it through the lense of someone not used to Guix' versioning scheme.
> Instead of telling us "yeah, this is the Nth time we picked a rando
> commit since the last release and this time it's de4db3ef", users
> coming from such distros would assume "oh well, this is still good ol'
> 1.0 with some more patches applied". So while the commit itself does
> not give us any particularly useful information (unless you're that
> person who uses this part of the version string to look the commit up
> on hubbucket), especially not when thinking in the context of
> versioning scheme, it does provide the existential information of "hold
> on, this is not a release commit, it's something else" and might thus
> direct users to be a little more attentive when they'd otherwise go
> "yep, upstream considers this solid and Guix considers it even more
> solid, so it's the solidest". Note, that this can be overcome both by
> teaching/learning about it and by using a special sigil as mentioned
> above.

Perhaps a function revealing metadata based upon the version string would allow more people get an overview without visiting hubbucket^1?

Would that be any weirder and awkward for workflows than the command `guix download'?
=> https://guix.gnu.org/manual/en/html_node/Invoking-guix-download.html

Even better, highlighting the part of the string and launching an appropriate context in Emacs-Hyperbole

> My personal answer to this might be a disappointing one, as in that
> case I believe we wouldn't even need procedures like git-version to
> form them, but could instead use <upstream-version>-<guix-revision> as
> a mere convention like many more popular distros already do. If the
> dash is overused for that, we could also use a different symbol, though
> perhaps there's not that many on a typical US keyboard to reserve one
> as a revision delimiter.

# Apologies for being off topic
The inclusion of that character '£' on keyboards bothers me - Ive never seen anybody use it (though maybe I have some fuzzy memory with regards to the Commodore 64).

If it unfortunately is on an international band of keyboard classes consider it as a delimiter. Otherwise Im ripping out that button and never interfacing the number 3 again.

^1 Is that pronounced bouquet? 
=> https://keepingupappearances.fandom.com/wiki/Hyacinth_Bucket


====================
Jonathan McHugh
indieterminacy@libre.brussels


> Am Mittwoch, den 01.09.2021, 18:39 +0200 schrieb Maxime Devos:
> 
>> Liliana Marie Prikler schreef op wo 01-09-2021 om 15:33 [+0200]:
>> Hi
>> 
>> Am Dienstag, den 31.08.2021, 23:20 +0200 schrieb Maxime Devos:
>>> Sarah Morgensen schreef op di 31-08-2021 om 12:57 [-0700]:
>>>> Hello Guix,
>>>> 
>>>> Currently, there are about 1500 packages defined like this:
>>>> 
>>>> --8<---------------cut here---------------start------------->8-
>>>> --
>>>> (define-public sbcl-feeder
>>>> (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
>>>> (revision "1"))
>>>> (package
>>>> [...])))
>>>> --8<---------------cut here---------------end--------------->8-
>>>> --
>>>> 
>>>> I feel like there are some issues with this idiom (in no
>>>> particular
>>>> order):
>>>> 
>>>> 1. When converting between this idiom and regularly versioned
>>>> packages, the git diff shows the whole package changing because
>>>> of
>>>> the indentation change.
>> If you are worried about that in a frequently changing package, you
>> could set both to *unspecified* or #f instead, which would cause
>> any
>> reference to them in a string manipulation context to fail. I
>> don't
>> think that such transitions are too frequent, though, as the point
>> is
>> rather to discourage them where not absolutely necessary and to use
>> upstream releases instead.
>> 
>>>> 2. We cannot get at the source location for the definition of
>>>> 'commit' or 'revision'. This would be useful for updating
>>>> these
>>>> packages with `guix refresh -u`. There is a proposed patch [0]
>>>> to
>>>> work around this, but it *is* a workaround.
>> Other versioning idioms would also be workarounds, wouldn't they?
>> 
>>>> 3. Packages inheriting from it lose the definitions. For
>>>> actual
>>>> fields, we have e.g. `(package-version this-package)`, but we
>>>> have
>>>> no equivalent for these.
>> What purpose would extracting those serve however?
>> 
>> Not losing the revision is useful for things like
>> <https://issues.guix.gnu.org/50072>, to be able to determine the old
>> revision. (That's not about inheriting packages though.)
> 
> Isn't that addressed by addressing the second point, though? Like, if
> you know the source location of the revision, you can read it back to
> get the value itself (or possibly even access it as-is), no?
> 
>> [...]
>>> To be used like:
>>> 
>>> (define-public sbcl-feeder
>>> (name "sbcl-feeder")
>>> (version (extended-version
>>> (base "1.0.0")
>>> (revision 1)
>>> (commit
>>> "b05f517d7729564575cc809e086c262646a94d34")))
>>> (source
>>> (origin
>>> (method git-fetch)
>>> (uri (git-reference ...)
>>> (url ...)
>>> ;; git-reference needs to be extended to retrieve the
>>> commit from the version
>>> (version version)))
>>> (file-name (git-file-name "feeder" version))
>>> (sha256 ...)))
>>> [...])
>>> 
>>> That should address 1,2,3,4 and 5.
>>> 
>>> One problem with this approach is that most users of 'package-
>>> version' expect it to return a string. Maybe adding a keyword
>>> argument '#:full-version? #t/#f' defaulting to #f would work?
>> I think the bigger problem here is that you're moving bits meant
>> for
>> the origin into the version only to be able to point to the version
>> from the origin. Even accepting that you could use "commit" or a
>> separate field to encode SVN/CVS revision numbers instead of
>> hashes,
>> everything beyond the revision number is basically pointless from a
>> versioning scheme POV and only really useful to fetch the source
>> code.
>> As Xinglu Chen points out, a commit hash encodes remarkably little
>> on its own.
>> 
>> The commit is largely useless, ok. If the (first few characters of)
>> the git commit/svn revision are removed from the version strings, it
>> can be removed from the proposed extended-version.
>> 
>> Otherwise, it would seem you wouldn't mind extended-version if it
>> only had the 'base version' and 'revision' field (in the guix sense,
>> not the SVN sense of revision), or am I misunderstanding here?
> 
> That was not my suggestion, but let's entertain the idea, shall we? In
> that case, we would discard the commit part from the version field,
> which might not be everyone's tea, but I'm more or less indifferent as
> to whether to include the hash there or not – after all, even if it was
> lacking, we'd quickly get it through inspecting the package
> description. If we simply didn't capture the hash at all except inside
> the commit field of the origin, we would gain 1, 4 and 5 so the
> question is whether we should have an extended version so as to update
> the revision more easily...
> 
> My personal answer to this might be a disappointing one, as in that
> case I believe we wouldn't even need procedures like git-version to
> form them, but could instead use <upstream-version>-<guix-revision> as
> a mere convention like many more popular distros already do. If the
> dash is overused for that, we could also use a different symbol, though
> perhaps there's not that many on a typical US keyboard to reserve one
> as a revision delimiter.
> 
> Making our rando commit git versions look like such other distro
> versions does come at a disadvantage though, particularly when we look
> at it through the lense of someone not used to Guix' versioning scheme.
> Instead of telling us "yeah, this is the Nth time we picked a rando
> commit since the last release and this time it's de4db3ef", users
> coming from such distros would assume "oh well, this is still good ol'
> 1.0 with some more patches applied". So while the commit itself does
> not give us any particularly useful information (unless you're that
> person who uses this part of the version string to look the commit up
> on hubbucket), especially not when thinking in the context of
> versioning scheme, it does provide the existential information of "hold
> on, this is not a release commit, it's something else" and might thus
> direct users to be a little more attentive when they'd otherwise go
> "yep, upstream considers this solid and Guix considers it even more
> solid, so it's the solidest". Note, that this can be overcome both by
> teaching/learning about it and by using a special sigil as mentioned
> above.
> 
> All in all, I don't think putting too much "opinion" in the version
> field by storing it as a record is a good idea. It's fine if it's just
> a string that can be parsed/version-compared. We could also make it a
> list like Emacs does and like we use internally, though I'm not too
> certain of what the benefit of that would be at the cost of breaking
> pretty much everything (and probably putting in some opinions as to
> what is to be delimited by dots and what by dashes).
> 
> Regards

Re: Can we find a better idiom for unversioned packages?

[Liliana Marie Prikler]

Am Mittwoch, den 01.09.2021, 19:48 +0000 schrieb Jonathan McHugh:
> September 1, 2021 8:35 PM, "Liliana Marie Prikler" <
> leo.prikler@student.tugraz.at> wrote
> 
> > Making our rando commit git versions look like such other distro
> > versions does come at a disadvantage though, particularly when we
> > look
> > at it through the lense of someone not used to Guix' versioning
> > scheme.
> > Instead of telling us "yeah, this is the Nth time we picked a rando
> > commit since the last release and this time it's de4db3ef", users
> > coming from such distros would assume "oh well, this is still good
> > ol'
> > 1.0 with some more patches applied". So while the commit itself
> > does
> > not give us any particularly useful information (unless you're that
> > person who uses this part of the version string to look the commit
> > up
> > on hubbucket), especially not when thinking in the context of
> > versioning scheme, it does provide the existential information of
> > "hold
> > on, this is not a release commit, it's something else" and might
> > thus
> > direct users to be a little more attentive when they'd otherwise go
> > "yep, upstream considers this solid and Guix considers it even more
> > solid, so it's the solidest". Note, that this can be overcome both
> > by
> > teaching/learning about it and by using a special sigil as
> > mentioned
> > above.
> 
> Perhaps a function revealing metadata based upon the version string
> would allow more people get an overview without visiting hubbucket^1?
I don't think that information is actually encoded anywhere nor
immediately obvious unless you're vaguely familiar with said software,
but it's still important e.g. if reading the documentation.  Does this
feature mentioned in the frobnicatorinator 1.44 docs apply to 1.36 or
not?  Do the examples from some book written in the early 70s work if I
compile them with GCC 12?  And so on and so forth.

> Would that be any weirder and awkward for workflows than the command
> `guix download'?
> => 
> https://guix.gnu.org/manual/en/html_node/Invoking-guix-download.html
Imo the only thing awkard about guix download is that it only handles
tarballs when a large chunk of packages use some sort of version
control.  We might want to look over that at some point and specify
revisions/commits to download on the command line.

> Even better, highlighting the part of the string and launching an
> appropriate context in Emacs-Hyperbole

> > My personal answer to this might be a disappointing one, as in that
> > case I believe we wouldn't even need procedures like git-version to
> > form them, but could instead use <upstream-version>-<guix-revision> 
> > as a mere convention like many more popular distros already do. If
> > the dash is overused for that, we could also use a different
> > symbol, though perhaps there's not that many on a typical US
> > keyboard to reserve one as a revision delimiter.
> 
> # Apologies for being off topic
> The inclusion of that character '£' on keyboards bothers me - Ive
> never seen anybody use it (though maybe I have some fuzzy memory with
> regards to the Commodore 64).
Pretty sure people in the UK used it for their currency even pre-
Brexit, but if you're used to $ for everything it might be
understandable why you're confused.

> If it unfortunately is on an international band of keyboard classes
> consider it as a delimiter. Otherwise Im ripping out that button and
> never interfacing the number 3 again.
Are you okay?

> ^1 Is that pronounced bouquet? 
> => https://keepingupappearances.fandom.com/wiki/Hyacinth_Bucket
I regret clicking on that.  While bucket and bouquet are etymologically
related, at least according to Wiktionary, I'd assume the generally
accepted pronunciation would be /hʌbˈbʌkɪt/, though there may be local
variations.

Re: Can we find a better idiom for unversioned packages?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

Liliana Marie Prikler schreef op wo 01-09-2021 om 23:47 [+0200]:
> > https://guix.gnu.org/manual/en/html_node/Invoking-guix-download.html
> Imo the only thing awkard about guix download is that it only handles
> tarballs when a large chunk of packages use some sort of version
> control.  We might want to look over that at some point and specify
> revisions/commits to download on the command line.

jgart sent a patch to implement this for git:
<https://issues.guix.gnu.org/50274#0>.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Jonathan McHugh]

Hi Liliana,

Given your examples I expect improving upstream CHANGELOG (or third party) files would be too much of a burden in order to solve the aforementioned problems.


Jonathan

September 1, 2021 11:47 PM, "Liliana Marie Prikler" <leo.prikler@student.tugraz.at> wrote:

> Am Mittwoch, den 01.09.2021, 19:48 +0000 schrieb Jonathan McHugh:
> 
>> September 1, 2021 8:35 PM, "Liliana Marie Prikler" <
>> leo.prikler@student.tugraz.at> wrote
>> 
>> Making our rando commit git versions look like such other distro
>> versions does come at a disadvantage though, particularly when we
>> look
>> at it through the lense of someone not used to Guix' versioning
>> scheme.
>> Instead of telling us "yeah, this is the Nth time we picked a rando
>> commit since the last release and this time it's de4db3ef", users
>> coming from such distros would assume "oh well, this is still good
>> ol'
>> 1.0 with some more patches applied". So while the commit itself
>> does
>> not give us any particularly useful information (unless you're that
>> person who uses this part of the version string to look the commit
>> up
>> on hubbucket), especially not when thinking in the context of
>> versioning scheme, it does provide the existential information of
>> "hold
>> on, this is not a release commit, it's something else" and might
>> thus
>> direct users to be a little more attentive when they'd otherwise go
>> "yep, upstream considers this solid and Guix considers it even more
>> solid, so it's the solidest". Note, that this can be overcome both
>> by
>> teaching/learning about it and by using a special sigil as
>> mentioned
>> above.
>> 
>> Perhaps a function revealing metadata based upon the version string
>> would allow more people get an overview without visiting hubbucket^1?
> 
> I don't think that information is actually encoded anywhere nor
> immediately obvious unless you're vaguely familiar with said software,
> but it's still important e.g. if reading the documentation. Does this
> feature mentioned in the frobnicatorinator 1.44 docs apply to 1.36 or
> not? Do the examples from some book written in the early 70s work if I
> compile them with GCC 12? And so on and so forth.

Re: Can we find a better idiom for unversioned packages?

[Liliana Marie Prikler]

Am Donnerstag, den 02.09.2021, 07:53 +0000 schrieb Jonathan McHugh:
> Hi Liliana,
> 
> Given your examples I expect improving upstream CHANGELOG (or third
> party) files would be too much of a burden in order to solve the
> aforementioned problems.
ChangeLogs are generally not installed as part of packages, though some
packages might include them as documentation.  In other cases, they
might even be part of the source files themselves, Emacs packages
sometimes function like that.  In either case, they only "address" the
problems insofar as reading them might give a clue about what goes into
the resulting binary, but they do not help that much with versioning.

> 
> Jonathan
> 
> September 1, 2021 11:47 PM, "Liliana Marie Prikler" <
> leo.prikler@student.tugraz.at> wrote:
> 
> > Am Mittwoch, den 01.09.2021, 19:48 +0000 schrieb Jonathan McHugh:
> > 
> > > September 1, 2021 8:35 PM, "Liliana Marie Prikler" <
> > > leo.prikler@student.tugraz.at> wrote
> > > 
> > > Making our rando commit git versions look like such other distro
> > > versions does come at a disadvantage though, particularly when we
> > > look
> > > at it through the lense of someone not used to Guix' versioning
> > > scheme.
> > > Instead of telling us "yeah, this is the Nth time we picked a
> > > rando
> > > commit since the last release and this time it's de4db3ef", users
> > > coming from such distros would assume "oh well, this is still
> > > good
> > > ol'
> > > 1.0 with some more patches applied". So while the commit itself
> > > does
> > > not give us any particularly useful information (unless you're
> > > that
> > > person who uses this part of the version string to look the
> > > commit
> > > up
> > > on hubbucket), especially not when thinking in the context of
> > > versioning scheme, it does provide the existential information of
> > > "hold
> > > on, this is not a release commit, it's something else" and might
> > > thus
> > > direct users to be a little more attentive when they'd otherwise
> > > go
> > > "yep, upstream considers this solid and Guix considers it even
> > > more
> > > solid, so it's the solidest". Note, that this can be overcome
> > > both
> > > by
> > > teaching/learning about it and by using a special sigil as
> > > mentioned
> > > above.
> > > 
> > > Perhaps a function revealing metadata based upon the version
> > > string
> > > would allow more people get an overview without visiting
> > > hubbucket^1?
> > 
> > I don't think that information is actually encoded anywhere nor
> > immediately obvious unless you're vaguely familiar with said
> > software,
> > but it's still important e.g. if reading the documentation. Does
> > this
> > feature mentioned in the frobnicatorinator 1.44 docs apply to 1.36
> > or
> > not? Do the examples from some book written in the early 70s work
> > if I
> > compile them with GCC 12? And so on and so forth.

Re: Can we find a better idiom for unversioned packages?

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 3094 bytes --]

On Tue, Aug 31 2021, Sarah Morgensen wrote:

> Hello Guix,
>
> Currently, there are about 1500 packages defined like this:
>
> --8<---------------cut here---------------start------------->8---
> (define-public sbcl-feeder
>   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
>         (revision "1"))
>     (package
>       [...])))
> --8<---------------cut here---------------end--------------->8---
>
> I feel like there are some issues with this idiom (in no particular
> order):
>
> 1. When converting between this idiom and regularly versioned packages,
> the git diff shows the whole package changing because of the indentation
> change.
>
> 2. We cannot get at the source location for the definition of 'commit' or
> 'revision'.  This would be useful for updating these packages with `guix
> refresh -u`.  There is a proposed patch [0] to work around this, but it
> *is* a workaround.
>
> 3. Packages inheriting from it lose the definitions.  For actual fields,
> we have e.g. `(package-version this-package)`, but we have no equivalent
> for these.
>
> 4. Horizontal space is at a premium, and an extra two spaces here and
> there add up.  (Personally, I think we could do with a
> define-public-package macro to save another two spaces, but that's for
> another day...)
>
> 5. The closest thing we have to a standardized way of generating
> versions for these packages is `(version (git-version "0.0.0" revision
> commit))`.  We can do better than that boilerplate.
>
> 6. Not a direct complaint, but I feel like the overall package interface
> was designed before straight-from-vcs unversioned packages were so
> common, and so this idiom developed organically to work around that.
>
> I do not have a specific solution in mind, but I think there must be
> one.  I do have a few half-baked ideas, but I'm curious what we can all
> come up with together.  Or maybe you'll just tell me I'm just being
> awfully picky :)
>
> [0] https://issues.guix.gnu.org/50286

I never felt like including the commit id in the version of a package
was useful; e.g., just seeing the first seven characters of the commit
id doesn’t really tell me anything about the version.  I think it is
more useful to put the date of the commit in the version; Nixpkgs does
something like this[1].  I have started to put the date of the commit in
a comment, just to make it easier for people to know how old it commit
is; otherwise, one would have to find the VCS repo and find the commit
just to see how old the commit is.

--8<---------------cut here---------------start------------->8---
(define-public foo
  ;; No tags; commit from YYYY-MM-DD.
  (let ((commit "COMMIT-ID")
        (revision "1"))
    (package
      ...)))
--8<---------------cut here---------------end--------------->8---

Instead of having showing something like

  foo-1.0.0-1.cabba9e

users would see

  foo-1.0.0-1.2021-01-31


Just my 2€  :-)

[1]: <https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/accessibility/dasher/default.nix>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

On Wed, Sep 1, 2021, at 06:55, Xinglu Chen wrote:
> I never felt like including the commit id in the version of a package
> was useful; e.g., just seeing the first seven characters of the commit
> id doesn’t really tell me anything about the version.  I think it is
> more useful to put the date of the commit in the version; Nixpkgs does
> something like this[1].  I have started to put the date of the commit in
> a comment, just to make it easier for people to know how old it commit
> is; otherwise, one would have to find the VCS repo and find the commit
> just to see how old the commit is.

The issue I see with only using the date is that Git dates are not unique, in order, or even meaningful in a clear way.

Commit dates don't have a consistent meaning: are they the time of first revision of a commit? Final revision of a commit? Time of signing? Pushing? They are often useful to estimate a timeline, but it's common for a Git "timeline" to jump back and forth by months or a year due to long-running development branches being merged in, or due to a "commit and then polish by rebasing" workflow.

Using the revision ID (of sufficient length) gives an unambiguous reference to the upstream source of a package and its artifacts in the store. How would you describe a package version to upstream when reporting a bug, except by revision ID? You can't tell them a timestamp and expect them to know which code a buggy package is based on.

We could certainly add a timestamp to our version strings for VCS-based packages, but we should keep the revision ID too.

Re: Can we find a better idiom for unversioned packages?

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 2550 bytes --]

On Wed, Sep 01 2021, Leo Famulari wrote:

> On Wed, Sep 1, 2021, at 06:55, Xinglu Chen wrote:
>> I never felt like including the commit id in the version of a package
>> was useful; e.g., just seeing the first seven characters of the commit
>> id doesn’t really tell me anything about the version.  I think it is
>> more useful to put the date of the commit in the version; Nixpkgs does
>> something like this[1].  I have started to put the date of the commit in
>> a comment, just to make it easier for people to know how old it commit
>> is; otherwise, one would have to find the VCS repo and find the commit
>> just to see how old the commit is.
>
> The issue I see with only using the date is that Git dates are not
> unique, in order, or even meaningful in a clear way.

Well, seeing

  foo-1.0.0-1.2021-01-31

gives a user more useful information than something like

  foo-1.0.0-1.cabba9e

With the former, I can quickly see that the version is from 2021-01-31,
whereas with the latter, I would have to either find the VCS repo online
or go to my local checkout of it and browse the logs.

> Commit dates don't have a consistent meaning: are they the time of
> first revision of a commit? Final revision of a commit? Time of
> signing? Pushing? They are often useful to estimate a timeline, but
> it's common for a Git "timeline" to jump back and forth by months or a
> year due to long-running development branches being merged in, or due
> to a "commit and then polish by rebasing" workflow.

I would say the the time of the final commit would be the best option,
but I agree that it can be ambiguous.

> Using the revision ID (of sufficient length) gives an unambiguous
> reference to the upstream source of a package and its artifacts in the
> store. How would you describe a package version to upstream when
> reporting a bug, except by revision ID? You can't tell them a
> timestamp and expect them to know which code a buggy package is based
> on.

One can get the commit id of a package by simply running ‘guix edit
PACKAGE’ and copying the commit id from the package definition.  I don’t
think hiding the commit id from the package version would be a problem
for this situation.

> We could certainly add a timestamp to our version strings for
> VCS-based packages, but we should keep the revision ID too.

I haven’t really found the commit id that useful when looking at the
version of a package; adding a timestamp would certainly be better than
the status quo.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

On Wed, Sep 01, 2021 at 06:50:36PM +0200, Xinglu Chen wrote:
> With the former, I can quickly see that the version is from 2021-01-31,
> whereas with the latter, I would have to either find the VCS repo online
> or go to my local checkout of it and browse the logs.
> 
> > Commit dates don't have a consistent meaning: are they the time of
> > first revision of a commit? Final revision of a commit? Time of
> > signing? Pushing? They are often useful to estimate a timeline, but
> > it's common for a Git "timeline" to jump back and forth by months or a
> > year due to long-running development branches being merged in, or due
> > to a "commit and then polish by rebasing" workflow.
> 
> I would say the the time of the final commit would be the best option,
> but I agree that it can be ambiguous.

My point was that Git does not provide you a timestamp to use for this.
There are multiple timestamps embedded into Git commits and none of them
reflect anything meaningful.

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

On Thu, Sep 02, 2021 at 12:51:58PM -0400, Leo Famulari wrote:
> On Wed, Sep 01, 2021 at 06:50:36PM +0200, Xinglu Chen wrote:
> > > Commit dates don't have a consistent meaning: are they the time of
> > > first revision of a commit? Final revision of a commit? Time of
> > > signing? Pushing? They are often useful to estimate a timeline, but
> > > it's common for a Git "timeline" to jump back and forth by months or a
> > > year due to long-running development branches being merged in, or due
> > > to a "commit and then polish by rebasing" workflow.
> > 
> > I would say the the time of the final commit would be the best option,
> > but I agree that it can be ambiguous.

Reading your message again, I think you misunderstood what I wrote.

I wasn't asking what date we should choose to include in our package
versions. I was asking, "What does the Git commit timestamp describe?"
And the answer is that there is not a clear answer, and it depends on
the workflow of the person who made the Git commit. My point being that
a Git repo does not offer us meaningful information about when anything
happened.

Re: Can we find a better idiom for unversioned packages?

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 2115 bytes --]

On Thu, Sep 02 2021, Leo Famulari wrote:

> On Thu, Sep 02, 2021 at 12:51:58PM -0400, Leo Famulari wrote:
>> On Wed, Sep 01, 2021 at 06:50:36PM +0200, Xinglu Chen wrote:
>> > > Commit dates don't have a consistent meaning: are they the time of
>> > > first revision of a commit? Final revision of a commit? Time of
>> > > signing? Pushing? They are often useful to estimate a timeline, but
>> > > it's common for a Git "timeline" to jump back and forth by months or a
>> > > year due to long-running development branches being merged in, or due
>> > > to a "commit and then polish by rebasing" workflow.
>> > 
>> > I would say the the time of the final commit would be the best option,
>> > but I agree that it can be ambiguous.
>
> Reading your message again, I think you misunderstood what I wrote.
>
> I wasn't asking what date we should choose to include in our package
> versions. I was asking, "What does the Git commit timestamp describe?"
> And the answer is that there is not a clear answer, and it depends on
> the workflow of the person who made the Git commit. My point being that
> a Git repo does not offer us meaningful information about when anything
> happened.

The date does give an idea of how old the version is, compare that to a
random string of 7 characters.  If a user wants to know the exact
commit, they can always just run ‘guix edit PACKAGE’ and check the
‘commit’ field in the source of the package.

From a Guix developer’s perspective, one can get an idea of roughly how
old a package is when looking at the package definition.  E.g., a few
months ago, when Magit hadn’t a release for around 2 years or so, I
wanted to see how old our current ‘emacs-magit’ package was.  To do that
I had to ‘guix edit emacs-magit’, then find the commit-id and copy it,
then go to my local checkout of Magit and run ‘git log COMMIT-ID’, just
to see how old the version was.  If the date of the commit was encoded
in the version string, I would immediately see that its 6 months old or
something; no need to manually look up the commit-id.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

On Fri, Sep 03, 2021 at 06:11:46PM +0200, Xinglu Chen wrote:
> The date does give an idea of how old the version is, compare that to a
> random string of 7 characters.  If a user wants to know the exact
> commit, they can always just run ‘guix edit PACKAGE’ and check the
> ‘commit’ field in the source of the package.

That's true.

However, it's not easy to figure out which revision of guix.git produced
a package that is installed in a profile.

The transformation from package definition to built package is lossy.
You cannot take a built package and ask Guix "which guix.git commit was
used for this?"

So, if I have a package foo-0.0.0-1-2021-12-31 in my profile, it's
impossible to reliably trace that back to the correct Guix package
definition and then discover what upstream source code it was built
with. [0] On the other hand, if the package's version includes the Git
commit, it's trivial.

So, I think we each have different needs that we want satisfied:

1) To know how old the package's source code is
2) To know what source code a built package is based on

The second item is something I do often, and I think a lot of Guix
developers do it too. And in general, it's imporant for store paths to
include meaningful "version" information; a date is not meaningful in
this sense. So let's be careful not to lose that ability by removing
commit IDs from the package versions.

We can satisfy both of these by adding the date to the version string,
although we should be careful not to risk exceeding Linux's shebang
length limit (127), which sometimes will crop up if packages provide
shebang-able interpreters. The store path needs to be short enough that
the "bin/foo" part does not make the absolute path exceed 127
characters. This is something that was addressed while designing the
current versioning for VCS snapshots.

I'm still skeptical of the utility of adding a date, given the lack of
useful time information conveyed by Git, but if people really want it
and it can be made to work, then we should go ahead.

[0] Complications arise when a package origin includes patches or
snippets, but anyways...

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

On Fri, Sep 03, 2021 at 12:35:21PM -0400, Leo Famulari wrote:
> 1) To know how old the package's source code is

Oh, some more thoughts:

What about extending `guix refresh` to help with your use case, assuming
that it doesn't already?

And what about packages that aren't based on VCS, but on tarballs / zip
files? Do those need a timestamp too?

Re: Can we find a better idiom for unversioned packages?

[Xinglu Chen]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

On Fri, Sep 03 2021, Leo Famulari wrote:

> On Fri, Sep 03, 2021 at 06:11:46PM +0200, Xinglu Chen wrote:
>> The date does give an idea of how old the version is, compare that to a
>> random string of 7 characters.  If a user wants to know the exact
>> commit, they can always just run ‘guix edit PACKAGE’ and check the
>> ‘commit’ field in the source of the package.
>
> That's true.
>
> However, it's not easy to figure out which revision of guix.git produced
> a package that is installed in a profile.

‘guix describe’ would show the commit of the guix.git repo used,
wouldn’t it?

> The transformation from package definition to built package is lossy.
> You cannot take a built package and ask Guix "which guix.git commit was
> used for this?"
>
> So, if I have a package foo-0.0.0-1-2021-12-31 in my profile, it's
> impossible to reliably trace that back to the correct Guix package
> definition and then discover what upstream source code it was built
> with. [0] On the other hand, if the package's version includes the Git
> commit, it's trivial.

Good point, I don’t really have a good solution to that.

> So, I think we each have different needs that we want satisfied:
>
> 1) To know how old the package's source code is
> 2) To know what source code a built package is based on
>
> The second item is something I do often, and I think a lot of Guix
> developers do it too. And in general, it's imporant for store paths to
> include meaningful "version" information; a date is not meaningful in
> this sense. So let's be careful not to lose that ability by removing
> commit IDs from the package versions.
>
> We can satisfy both of these by adding the date to the version string,
> although we should be careful not to risk exceeding Linux's shebang
> length limit (127), which sometimes will crop up if packages provide
> shebang-able interpreters. The store path needs to be short enough that
> the "bin/foo" part does not make the absolute path exceed 127
> characters. This is something that was addressed while designing the
> current versioning for VCS snapshots.

I didn’t know Linux had this weird(?) restriction, but yeah, it will be
something to think about.

> I'm still skeptical of the utility of adding a date, given the lack of
> useful time information conveyed by Git, but if people really want it
> and it can be made to work, then we should go ahead.

If the date is not part of the version string, I think that there should
at least be a comment saying the the date of the commit, just to let
other people quickly see how old the version is.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

On Fri, Sep 03, 2021 at 10:03:47PM +0200, Xinglu Chen wrote:
> ‘guix describe’ would show the commit of the guix.git repo used,
> wouldn’t it?

It shows the commit of currently effective revision of Guix, but it can't
tell you what revision of Guix a built package came from.

> If the date is not part of the version string, I think that there should
> at least be a comment saying the the date of the commit, just to let
> other people quickly see how old the version is.

To clarify, I'm not saying that we shouldn't include the date! But we
should keep the commit IDs too. And be careful about the shebang
lengths. I am sympathetic to your use case of "how old is this thing?"
and that it's currently annoying to satisfy.

Re: Can we find a better idiom for unversioned packages?

[Ludovic Courtès]

Hi,

Xinglu Chen <public@yoctocell.xyz> skribis:

> Well, seeing
>
>   foo-1.0.0-1.2021-01-31
>
> gives a user more useful information than something like
>
>   foo-1.0.0-1.cabba9e
>
> With the former, I can quickly see that the version is from 2021-01-31,
> whereas with the latter, I would have to either find the VCS repo online
> or go to my local checkout of it and browse the logs.

I agree with Leo here.  IMO what matters with version strings is that
they be comparable: users (and UIs) should be able to tell which string
denotes a newer version.  That’s exactly what the ‘revision’ bit does in
the example above.  The commit shorthand is an additional bit to
uniquely identify the snapshot.

Knowing the date of a release, or that of a commit, may also be useful,
but that’s not what version strings are about generally.

Thanks,
Ludo’.

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

On Tue, Aug 31, 2021 at 12:57:23PM -0700, Sarah Morgensen wrote:
> I do not have a specific solution in mind, but I think there must be
> one.  I do have a few half-baked ideas, but I'm curious what we can all
> come up with together.  Or maybe you'll just tell me I'm just being
> awfully picky :)

Thanks for starting this discussion. In order to avoid repeating /
rediscovering the reasons why the current idiom was chosen, I recommend
reading the discussion that led to it:

https://lists.gnu.org/archive/html/guix-devel/2016-01/msg00335.html

Re: Can we find a better idiom for unversioned packages?

[Ludovic Courtès]

Hello!

Sarah Morgensen <iskarian@mgsn.dev> skribis:

> Currently, there are about 1500 packages defined like this:
>
> (define-public sbcl-feeder
>   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
>         (revision "1"))
>     (package
>       [...])))
>
> I feel like there are some issues with this idiom (in no particular
> order):

I’m late to the party but I’ll complement previous answers.  :-)

> 1. When converting between this idiom and regularly versioned packages,
> the git diff shows the whole package changing because of the indentation
> change.

One can use ‘git diff -w’ to work around that (or the newfangled
Diffstatic tool.)

> 3. Packages inheriting from it lose the definitions.  For actual fields,
> we have e.g. `(package-version this-package)`, but we have no equivalent
> for these.

Right, these pieces of information are not “first-class”, except in the
‘git-reference’ record (or similar) for the commit ID.  Do you have
examples in mind where it’s insufficient?

[...]

> 5. The closest thing we have to a standardized way of generating
> versions for these packages is `(version (git-version "0.0.0" revision
> commit))`.  We can do better than that boilerplate.

I can sympathize with the feeling, but I’m not sure what to do.  A
‘vcs-version’ record as Maxime proposes seems a bit overkill to me (and
it would probably have an impact on performance, build times, and
whatnot.)

> 6. Not a direct complaint, but I feel like the overall package interface
> was designed before straight-from-vcs unversioned packages were so
> common, and so this idiom developed organically to work around that.

Sure, though “straight-from-vcs” and “unversioned” are two different
things: I’m fine with the former, but the latter equates to upstream
telling its users “go find a revision that works for you”.  I think
releases still make sense for any non-trivial piece of software.

As noted in the manual (info "(guix) Version Numbers"), packages built
from arbitrary commits were supposed to be exceptional.  Perhaps the
reason we’re having this conversation now is that development practices
are evolving towards what looks like chaos.  :-)

Thanks,
Ludo’.

Re: Can we find a better idiom for unversioned packages?

[Jonathan McHugh]

Hi Ludo,

Just checking:

Is Diffstatic a real tool? It wasnt quite clear to me (and I fancy finding a new diff tool).

====================
Jonathan McHugh
indieterminacy@libre.brussels

September 8, 2021 11:31 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:

> Hello!
> 
> Sarah Morgensen <iskarian@mgsn.dev> skribis:
> 
>> Currently, there are about 1500 packages defined like this:
>> 
>> (define-public sbcl-feeder
>> (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
>> (revision "1"))
>> (package
>> [...])))
>> 
>> I feel like there are some issues with this idiom (in no particular
>> order):
> 
> I’m late to the party but I’ll complement previous answers. :-)
> 
>> 1. When converting between this idiom and regularly versioned packages,
>> the git diff shows the whole package changing because of the indentation
>> change.
> 
> One can use ‘git diff -w’ to work around that (or the newfangled
> Diffstatic tool.)
> 
>> 3. Packages inheriting from it lose the definitions. For actual fields,
>> we have e.g. `(package-version this-package)`, but we have no equivalent
>> for these.
> 
> Right, these pieces of information are not “first-class”, except in the
> ‘git-reference’ record (or similar) for the commit ID. Do you have
> examples in mind where it’s insufficient?
> 
> [...]
> 
>> 5. The closest thing we have to a standardized way of generating
>> versions for these packages is `(version (git-version "0.0.0" revision
>> commit))`. We can do better than that boilerplate.
> 
> I can sympathize with the feeling, but I’m not sure what to do. A
> ‘vcs-version’ record as Maxime proposes seems a bit overkill to me (and
> it would probably have an impact on performance, build times, and
> whatnot.)
> 
>> 6. Not a direct complaint, but I feel like the overall package interface
>> was designed before straight-from-vcs unversioned packages were so
>> common, and so this idiom developed organically to work around that.
> 
> Sure, though “straight-from-vcs” and “unversioned” are two different
> things: I’m fine with the former, but the latter equates to upstream
> telling its users “go find a revision that works for you”. I think
> releases still make sense for any non-trivial piece of software.
> 
> As noted in the manual (info "(guix) Version Numbers"), packages built
> from arbitrary commits were supposed to be exceptional. Perhaps the
> reason we’re having this conversation now is that development practices
> are evolving towards what looks like chaos. :-)
> 
> Thanks,
> Ludo’.

Re: Can we find a better idiom for unversioned packages?

[Leo Famulari]

I think Ludo meant difftastic:

https://github.com/Wilfred/difftastic

On Wed, Sep 8, 2021, at 18:21, Jonathan McHugh wrote:
> Hi Ludo,
> 
> Just checking:
> 
> Is Diffstatic a real tool? It wasnt quite clear to me (and I fancy 
> finding a new diff tool).
> 
> ====================
> Jonathan McHugh
> indieterminacy@libre.brussels
> 
> September 8, 2021 11:31 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:
> 
> > Hello!
> > 
> > Sarah Morgensen <iskarian@mgsn.dev> skribis:
> > 
> >> Currently, there are about 1500 packages defined like this:
> >> 
> >> (define-public sbcl-feeder
> >> (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
> >> (revision "1"))
> >> (package
> >> [...])))
> >> 
> >> I feel like there are some issues with this idiom (in no particular
> >> order):
> > 
> > I’m late to the party but I’ll complement previous answers. :-)
> > 
> >> 1. When converting between this idiom and regularly versioned packages,
> >> the git diff shows the whole package changing because of the indentation
> >> change.
> > 
> > One can use ‘git diff -w’ to work around that (or the newfangled
> > Diffstatic tool.)
> > 
> >> 3. Packages inheriting from it lose the definitions. For actual fields,
> >> we have e.g. `(package-version this-package)`, but we have no equivalent
> >> for these.
> > 
> > Right, these pieces of information are not “first-class”, except in the
> > ‘git-reference’ record (or similar) for the commit ID. Do you have
> > examples in mind where it’s insufficient?
> > 
> > [...]
> > 
> >> 5. The closest thing we have to a standardized way of generating
> >> versions for these packages is `(version (git-version "0.0.0" revision
> >> commit))`. We can do better than that boilerplate.
> > 
> > I can sympathize with the feeling, but I’m not sure what to do. A
> > ‘vcs-version’ record as Maxime proposes seems a bit overkill to me (and
> > it would probably have an impact on performance, build times, and
> > whatnot.)
> > 
> >> 6. Not a direct complaint, but I feel like the overall package interface
> >> was designed before straight-from-vcs unversioned packages were so
> >> common, and so this idiom developed organically to work around that.
> > 
> > Sure, though “straight-from-vcs” and “unversioned” are two different
> > things: I’m fine with the former, but the latter equates to upstream
> > telling its users “go find a revision that works for you”. I think
> > releases still make sense for any non-trivial piece of software.
> > 
> > As noted in the manual (info "(guix) Version Numbers"), packages built
> > from arbitrary commits were supposed to be exceptional. Perhaps the
> > reason we’re having this conversation now is that development practices
> > are evolving towards what looks like chaos. :-)
> > 
> > Thanks,
> > Ludo’.
> 
> 

Re: Can we find a better idiom for unversioned packages?

[Sarah Morgensen]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> writes:

>> > Not losing the revision is useful for things like 
>> > <https://issues.guix.gnu.org/50072>;, to be able to determine the old
>> > revision.  (That's not about inheriting packages though.)
>
>> Isn't that addressed by addressing the second point, though?  Like, if
>> you know the source location of the revision, you can read it back to
>> get the value itself (or possibly even access it as-is), no?
>
> Indeed!  The patch [0] addresses the second point.  With that patch,
> the proposed <extension-version> isn't required.  But also: some people
> (at least Sarah?) consider [0] a work-around, and if guix used something
> like <extended-version>, [0] wouldn't be necessary.
>
> It doesn't really matter to me what we'll end up using in guix
> in the long term, though in the short term, I would like something
> like [0] to be merged, as it is used by the (not-yet submitted, needs some
> cleanup, testing & rebasing) minetest updater, and it makes
> <https://issues.guix.gnu.org/50072> work in more cases.
>
> [0]: <https://issues.guix.gnu.org/50072>

Despite starting this thread, I do agree that your patch would be
helpful, at least in the short term, since I'd like to get git updating
working sooner rather than later :)

(There are a couple other usages of 'let' forms in packages, and your
patch could either help or hinder getting source locations for those,
depending on what semantics we want.  Very much an edge case, though.)

--
Sarah

Re: Can we find a better idiom for unversioned packages?

[Sarah Morgensen]

Liliana,

Thanks for the criticism!

Liliana Marie Prikler <leo.prikler@student.tugraz.at> writes:

> Hi
>
> Am Dienstag, den 31.08.2021, 23:20 +0200 schrieb Maxime Devos:
>> Sarah Morgensen schreef op di 31-08-2021 om 12:57 [-0700]:
>> > Hello Guix,
>> > 
>> > Currently, there are about 1500 packages defined like this:
>> > 
>> > --8<---------------cut here---------------start------------->8---
>> > (define-public sbcl-feeder
>> >   (let ((commit "b05f517d7729564575cc809e086c262646a94d34")
>> >         (revision "1"))
>> >     (package
>> >       [...])))
>> > --8<---------------cut here---------------end--------------->8---
>> > 
>> > I feel like there are some issues with this idiom (in no particular
>> > order):
>> > 
>> > 1. When converting between this idiom and regularly versioned
>> > packages, the git diff shows the whole package changing because of
>> > the indentation change.
> If you are worried about that in a frequently changing package, you
> could set both to *unspecified* or #f instead, which would cause any
> reference to them in a string manipulation context to fail.  I don't
> think that such transitions are too frequent, though, as the point is
> rather to discourage them where not absolutely necessary and to use
> upstream releases instead.

To be clear, this point was regarding "git blame", not any programmatic
manipulation.

>> > 2. We cannot get at the source location for the definition of
>> > 'commit' or 'revision'.  This would be useful for updating these
>> > packages with `guix refresh -u`.  There is a proposed patch [0] to
>> > work around this, but it *is* a workaround.
> Other versioning idioms would also be workarounds, wouldn't they?
>
>> > 3. Packages inheriting from it lose the definitions.  For actual
>> > fields, we have e.g. `(package-version this-package)`, but we have
>> > no equivalent for these.
> What purpose would extracting those serve however?
>
>> > 4. Horizontal space is at a premium, and an extra two spaces here
>> > and there add up.  (Personally, I think we could do with a
>> > define-public-package macro to save another two spaces, but that's
>> > for another day...)
> The worst offenders in horizontal space typically come from the
> packages themselves and going from 79 to 81 in select outliers is AFAIU
> acceptable.
>
>> > 5. The closest thing we have to a standardized way of generating
>> > versions for these packages is `(version (git-version "0.0.0"
>> > revision commit))`.  We can do better than that boilerplate.
> This concerns packages without a public release, many of which never
> make one in the belief that commit hashes are valid versions (they are
> not).

You're largely correct, in that all these reasons are minor.  I feel
that they do add up, though... and, IMO, the current usage of 'let' just
feels inelegant.

> Perhaps we could make the point that origins can be versioned just like
> packages can and should think about where the version field really
> belongs.  I think having a per-origin version field and making it so
> that the package defaults to the origin version unless the package
> itself has a version specified, might be a better solution
> philosophically speaking.  Technically, it would only solve 1, 4 and 5,
> but there's a separate workaround for 2 and I don't really think 3 to
> be that big of an issue.

I like this idea.  It makes sense, in that the source does have an
inherent version, whether that's a blessed version number or some
identifier picked by the packager.  It keeps related values together.

(As you predicted, this does break down when multiple units of software
are packaged together and their versions don't necessarily match the
version of the source-as-a-whole, but being able to override the version
(as you suggested) should be sufficient.)

A minor downside of this method is that developers may have to look at
the source field to determine the package version if it's not directly
specified in the package (programmatic access would still work).
Similarly, anything which parses the raw sexps (such as committer.scm)
would have to be modified to accommodate.

Some ideas....

1. Version-first

(origin
  (version (vcs-version (base "2.13.3")))
  ;; or: (version "2.13.3")
  ;; (would require special-casing, but url-fetch uris would not need to
  ;; be modified)
  (method git-fetch)
  (uri (git-reference
        (url "https://github.com/git-lfs/git-lfs")
        (commit (string-append "v" version))))
        ;; or: (commit (version->tag version #:prefix "v"))
  (file-name (git-file-name name version))
  (sha256
   (base32
    "0r7dmqhkhz91d3n7qfpny483x8f1n88yya22j2fvx75rgg33z2sg")))

Or, if we encode the information a la #50359 [0], but in 'vcs-version':

(origin
  (version (vcs-version
            (base "2.13.3")
            (tag-prefix "v")))
  (method git-fetch)
  (uri (git-reference
        (url "https://github.com/git-lfs/git-lfs")
        (commit (vcs-version->git-commit version))))
  (file-name (git-file-name name version))
  (sha256
   (base32
    "0r7dmqhkhz91d3n7qfpny483x8f1n88yya22j2fvx75rgg33z2sg")))

In some ways it makes more sense to put e.g. tag-prefix in vcs-version,
rather than package properties, because the transformation can
occasionally vary between versions.

(origin
  (method git-fetch)
  (version (vcs-version
            (base "0.12.9")
            (identifier "e41b504dca90a25e9be27f296da7ce22e5782893")
            (revision "1")))
  (uri (git-reference
        (url "https://github.com/nosarthur/gita")
        (commit (vcs-version->git-commit version))))
  (file-name (git-file-name name version))
  (sha256
   (base32
    "1k03zgcbhl91cgyh4k7ywyjp00y63q4bqbimncqh5b3lni8l8j5l")))

Another advantage to this approach is that a potential git updater only
has to change the 'version' record.

2. Reference-first

(define* (git-ref->version ref #:optional base (revision "0")
                           #:key prefix suffix)
  (if base
    (vcs-version
     (base base)
     (identifier (git-reference-commit ref))
     (revision revision))
    (let ((prefix-len (or (and=> prefix string-length) 0))
          (suffix-len (or (and=> suffix string-length) 0)))
      (vcs-version
       (base (string-drop (string-drop-right ref suffix-len) prefix-len))
       (tag-prefix prefix)
       (tag-suffix suffix)))))

(origin
  (method git-fetch)
  (uri (git-reference
        (url "https://github.com/git-lfs/git-lfs")
        (commit "v2.13.3")))
  (version (git-ref->version uri #:prefix "v"))
  (file-name (git-file-name name version))
  (sha256
   (base32
    "0r7dmqhkhz91d3n7qfpny483x8f1n88yya22j2fvx75rgg33z2sg")))

(origin
  (method git-fetch)
  (uri (git-reference
        (url "https://github.com/nosarthur/gita")
        (commit "e41b504dca90a25e9be27f296da7ce22e5782893")))
  (version (git-ref->version uri "0.12.9" "1"))
  (file-name (git-file-name name version))
  (sha256
   (base32
    "1k03zgcbhl91cgyh4k7ywyjp00y63q4bqbimncqh5b3lni8l8j5l")))

Hmmm. I think this method falls short of the previous because we're
forced to work backwards from the tag, and it splits the information
relevant for updating between 'version' and 'uri'.

Perhaps 'vcs-version' should actually be separated into 'tagged-version'
and 'improper-version' or similar?

[0] <https://issues.guix.gnu.or/50359> Add 'generic-git' updater.

--
Sarah

Re: Can we find a better idiom for unversioned packages?

[Liliana Marie Prikler]

Hi Sarah,

Am Freitag, den 03.09.2021, 14:14 -0700 schrieb Sarah Morgensen:
> [...]
> 
> > If you are worried about that in a frequently changing package, you
> > could set both to *unspecified* or #f instead, which would cause
> > any reference to them in a string manipulation context to fail.  I
> > don't think that such transitions are too frequent, though, as the
> > point is rather to discourage them where not absolutely necessary
> > and to use upstream releases instead.
> 
> To be clear, this point was regarding "git blame", not any
> programmatic manipulation.
Well, let #f or *unspecified* would also solve the git blame issue, but
again I don't think the recursive git blame implemented e.g. by magit
is broken too easily by those transitions.  I might be wrong about
that, though.

> [...]
> You're largely correct, in that all these reasons are minor.  I feel
> that they do add up, though... and, IMO, the current usage of 'let'
> just feels inelegant.
The point is not so much to say "no" because all those points are
minor, but rather to establish which properties a "solution" would
require and which ones are simply nice to have.  For example, we need
to be able to update the expression in-place, but referencing its value
for the sake of inheritance is not that important.

> [...]
> 
> A minor downside of this method is that developers may have to look
> at the source field to determine the package version if it's not
> directly specified in the package (programmatic access would still
> work). Similarly, anything which parses the raw sexps (such as
> committer.scm) would have to be modified to accommodate.
Point taken.

> Some ideas....
> 
> 1. Version-first
> 
> (origin
>   (version (vcs-version (base "2.13.3")))
>   ;; or: (version "2.13.3")
>   ;; (would require special-casing, but url-fetch uris would not need
> to
>   ;; be modified)
>   (method git-fetch)
>   (uri (git-reference
>         (url "https://github.com/git-lfs/git-lfs")
>         (commit (string-append "v" version))))
>         ;; or: (commit (version->tag version #:prefix "v"))
>   (file-name (git-file-name name version))
>   (sha256
>    (base32
>     "0r7dmqhkhz91d3n7qfpny483x8f1n88yya22j2fvx75rgg33z2sg")))
> 
> Or, if we encode the information a la #50359 [0], but in 'vcs-
> version':
> 
> (origin
>   (version (vcs-version
>             (base "2.13.3")
>             (tag-prefix "v")))
>   (method git-fetch)
>   (uri (git-reference
>         (url "https://github.com/git-lfs/git-lfs")
>         (commit (vcs-version->git-commit version))))
>   (file-name (git-file-name name version))
>   (sha256
>    (base32
>     "0r7dmqhkhz91d3n7qfpny483x8f1n88yya22j2fvx75rgg33z2sg")))
> 
> In some ways it makes more sense to put e.g. tag-prefix in vcs-
> version, rather than package properties, because the transformation
> can occasionally vary between versions.
I don't agree with your assessment.  tag-prefix is not a raw string, it
can be a regexp, so if upstream jumps from "v1.2.3" to "release-1.2.4", 
you can encode that in one regexp.  You can't do that with version
records.

> (origin
>   (method git-fetch)
>   (version (vcs-version
>             (base "0.12.9")
>             (identifier "e41b504dca90a25e9be27f296da7ce22e5782893")
>             (revision "1")))
>   (uri (git-reference
>         (url "https://github.com/nosarthur/gita")
>         (commit (vcs-version->git-commit version))))
>   (file-name (git-file-name name version))
>   (sha256
>    (base32
>     "1k03zgcbhl91cgyh4k7ywyjp00y63q4bqbimncqh5b3lni8l8j5l")))
> 
> Another advantage to this approach is that a potential git updater
> only
> has to change the 'version' record.
> 
> 2. Reference-first
> 
> (define* (git-ref->version ref #:optional base (revision "0")
>                            #:key prefix suffix)
>   (if base
>     (vcs-version
>      (base base)
>      (identifier (git-reference-commit ref))
>      (revision revision))
>     (let ((prefix-len (or (and=> prefix string-length) 0))
>           (suffix-len (or (and=> suffix string-length) 0)))
>       (vcs-version
>        (base (string-drop (string-drop-right ref suffix-len) prefix-
> len))
>        (tag-prefix prefix)
>        (tag-suffix suffix)))))
> 
> (origin
>   (method git-fetch)
>   (uri (git-reference
>         (url "https://github.com/git-lfs/git-lfs")
>         (commit "v2.13.3")))
>   (version (git-ref->version uri #:prefix "v"))
>   (file-name (git-file-name name version))
>   (sha256
>    (base32
>     "0r7dmqhkhz91d3n7qfpny483x8f1n88yya22j2fvx75rgg33z2sg")))
> 
> (origin
>   (method git-fetch)
>   (uri (git-reference
>         (url "https://github.com/nosarthur/gita")
>         (commit "e41b504dca90a25e9be27f296da7ce22e5782893")))
>   (version (git-ref->version uri "0.12.9" "1"))
>   (file-name (git-file-name name version))
>   (sha256
>    (base32
>     "1k03zgcbhl91cgyh4k7ywyjp00y63q4bqbimncqh5b3lni8l8j5l")))
> 
> Hmmm. I think this method falls short of the previous because we're
> forced to work backwards from the tag, and it splits the information
> relevant for updating between 'version' and 'uri'.
You know, that you could alternate between version-first for tagged
commits and reference-first for untagged ones, assuming you don't do
the reasonable thing of wrapping the origin in a let&.

Then you'd have

(origin
  (method git-fetch)
  (version "2.13.3")
  (uri (git-reference
          (url "https://github.com/git-lfs/git-lfs")
          (commit (string-append "v" version))
  [...])

and 

(origin
  (method git-fetch)
  (uri (git-reference
          (url "https://github.com/git-lfs/git-lfs")
          (commit "e41b504dca90a25e9be27f296da7ce22e5782893")
  (version (git-version "0.12.9" (git-reference-commit uri)))
  [...])

Of course, you could do something similar with packages as is, but
(git-reference-commit (origin-uri origin)) might be a bit much.

Regards

Re: Can we find a better idiom for unversioned packages?

[Taylan Kammer]

On 04.09.2021 00:11, Liliana Marie Prikler wrote:
>
> Well, let #f or *unspecified* would also solve the git blame issue, [...]
> 

Only partially on-topic but: I believe *unspecified* should not be used
explicitly in code.  It's an implementation detail, and by avoiding it in
user code, we lessen the burden on the Guile maintainers to stick to this
particular implementation detail for backwards compatibility.

I'd advise #f or (values) for a "no meaningful return value" situation.

-- 
Taylan