Rebuilding a package after removing a build step

Rebuilding a package after removing a build step

[Konrad Hinsen]

Hi everyone,

Until today, I thought that all parts of a package definition that could
make a difference to the outputs enter somehow into the package hash,
such that any relevant change to a package definition causes a rebuild.

Today's experience: I built a package, then removed an "add-after" form
from "modify-phases", effectively removing a build step, and ran "guix
build" again. It built nothing, and returned the same path (same hash)
as before.

Is this a bug or a feature? If it's a feature, how do I get Guix to
rebuild the package after removing a build step?

Cheers,
  Konrad.

Re: Rebuilding a package after removing a build step

[Tobias Geerinckx-Rice]

Hi Konrad,

'That is not possible.'  You must somehow not have measured what you thought you measured.

Probably not the answer you were looking for, but reassuring in its own way.

Can you 'reproduce' this?  It might still be enlightening to find out where things went wrong.

Kind regards,

T G-R

Sent on the go.  Excuse or enjoy my brevity.

Re: Rebuilding a package after removing a build step

[Konrad Hinsen]

Hi Tobias,

 Geerinckx-Rice <me@tobias.gr> writes:

> 'That is not possible.'  You must somehow not have measured what you thought you measured.

Thanks for restoring my faith in Guix!

You were in fact right, I was not measuring what I thought you measured,
but for an unusual reason.

> Can you 'reproduce' this?  It might still be enlightening to find out
> where things went wrong.

Yes, the behavior is perfectly reproducible. But it was tied to one
package definition. I tried modifying a few other definitions, with
expected outcomes.

So I set out to explore what is special about this package. It's a
definition I had started last week and came back to for finishing
touches. In the meantime, I had done a "guix pull", and with that update
I had retrieved an almost identical definition in the "guix" channel.
Someone else had beaten me to submitting the package!

Unfortunately, when there are several packages with identical name and
version number in two channels, Guix silently chooses one of them.
It would probably be more useful to emit a warning. So I ended up
editing a package definition which would undergo basic checking
by Guile, but then be ignored.

Cheers,
  Konrad.

Re: Rebuilding a package after removing a build step

[Suhail Singh]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Unfortunately, when there are several packages with identical name and
> version number in two channels, Guix silently chooses one of them.

Ah, I had wondered what it did.  Thank you for reporting.

> It would probably be more useful to emit a warning.

Agreed.

On a related note, when multiple channels are enabled, is there a way to
specify a specific package from a particular channel?

-- 
Suhail

Re: Rebuilding a package after removing a build step

[Konrad Hinsen]

Suhail Singh <suhailsingh247@gmail.com> writes:

> On a related note, when multiple channels are enabled, is there a way to
> specify a specific package from a particular channel?

Yes, using the -e option followed by the Guile module name. See

  https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html

You have to know the module and not just the channel.

Cheers,
  Konrad.

Re: Rebuilding a package after removing a build step

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi,

On Tue, Sep 17 2024, Tobias Geerinckx-Rice wrote:

> 'That is not possible.'

I believe I experienced it when removing or adding propagated or native
inputs (not sure) to my very own guile-pam [1] while using 'guix shell'.

Unfortunately, I have not had time to investigate.

Kind regards
Felix

[1] https://issues.guix.gnu.org/72316

P.S. Please excuse or enjoy my brevity.

Re: Rebuilding a package after removing a build step

[Suhail Singh]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

>> On a related note, when multiple channels are enabled, is there a way to
>> specify a specific package from a particular channel?
>
> Yes, using the -e option followed by the Guile module name. See
>
>   https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html
>
> You have to know the module and not just the channel.

Thank you for the reference.

-- 
Suhail

Re: Rebuilding a package after removing a build step

[Simon Tournier]

Hi Konrad,

On mer., 18 sept. 2024 at 10:33, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Unfortunately, when there are several packages with identical name and
> version number in two channels, Guix silently chooses one of them.

Choose when doing what? :-)

When running “guix shell” or “guix package”, it should warn.  It
doesn’t?  Ah?  I thought it does…  Hum?

When packaging, you choose: you use one module or the other, or both.
Guile tells you if there is a symbol clash, so then you resolve it by
using some #:prefix or #:hide at #:use-module.  Else you choose one
symbol or the other inside the package definition.

Cheers,
simon

Re: Rebuilding a package after removing a build step

[Konrad Hinsen]

Hi Simon,

>> Unfortunately, when there are several packages with identical name and
>> version number in two channels, Guix silently chooses one of them.
>
> Choose when doing what? :-)

Building, installing, ... whatever you can do with a package.

> When running “guix shell” or “guix package”, it should warn.  It
> doesn’t?  Ah?  I thought it does…  Hum?

It doesn't.

> When packaging, you choose: you use one module or the other, or both.

Right, at the level of Guile code, there is no problem. It's the lookup
by name/version at the command line that doesn't let me choose nor tells
me that there is a need to choose.

Cheers,
  Konrad.

Re: Rebuilding a package after removing a build step

[Kaelyn]

On Sunday, September 22nd, 2024 at 1:34 AM, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> 
> 
> Hi Simon,
> 
> > > Unfortunately, when there are several packages with identical name and
> > > version number in two channels, Guix silently chooses one of them.
> > 
> > Choose when doing what? :-)
> 
> 
> Building, installing, ... whatever you can do with a package.
> 
> > When running “guix shell” or “guix package”, it should warn. It
> > doesn’t? Ah? I thought it does… Hum?
> 
> 
> It doesn't.

Whether it does or doesn't depends on the versions of the duplicate packages in question. If both packages have the same version, then the command line tools will warn about the ambiguous package specification. If the version numbers are different, then the tools will quietly choose the "newer" version (the larger version number).

For example, I have a local channel that included an updated version of yt-dlp, and the version in Guix has since been updated to the same version:

$ guix package -A yt-dlp
yt-dlp	2024.08.06	out	local/packages/updated.scm:157:2
yt-dlp	2024.08.06	out	gnu/packages/video.scm:3163:2

$ guix build yt-dlp
guix build: warning: ambiguous package specification `yt-dlp'
guix build: warning: choosing yt-dlp@2024.08.06 from gnu/packages/video.scm:3163:2

But if I change the local version to be different (say the fictitious "2024.09.06"), I no longer get a warning:

$ guix build -L ~/guix/local yt-dlp
substitute: updating substitutes from...

> 
> > When packaging, you choose: you use one module or the other, or both.
> 
> 
> Right, at the level of Guile code, there is no problem. It's the lookup
> by name/version at the command line that doesn't let me choose nor tells
> me that there is a need to choose.

Assuming the packages with the same name have different versions, you'd need to specify at least part of the version on the command line to get the older version. In Guix, both GTK+ 2/3 and QT 5/6 are examples where that is needed:

$ guix package -A qtbase
qtbase	6.6.3  	out,debug	gnu/packages/qt.scm:716:2
qtbase	5.15.10	out,debug	gnu/packages/qt.scm:450:2

$ guix build -n --no-substitutes qtbase
The following derivations would be built:
  /gnu/store/49vfjglgi4myb4c2sa59gjypa3sr05il-qtbase-6.6.3.drv

$ guix build -n --no-substitutes qtbase@5
The following derivations would be built:
  /gnu/store/9bpsq64ljwx2yf1gvw9kby2wda0bcx6a-qtbase-5.15.10.drv

However, if there are two packages with the same name and version, then to select a specific one you'd have to use the "-e" flag with a Scheme expression that evaluates to the desired package. Using the previous yt-dlp example, to specify the one in my local channel instead of the one from the guix channel that it warned it was choosing, I'd run:

   guix build -e '(@ (local packages updated) yt-dlp)'

The reason using an expression is needed is because the package specifications like "yt-dlp" or "qtbase@5" are not expressive enough to handle a distinction when the full package name and version--like "yt-dlp@2024.08.06"--match more than one package definition.

This also raises a question that I don't think has been asked yet: is the local package you are modifying the same version as the package of the same name in the guix channel?

HTH!

Cheers,
Kaelyn

Re: Rebuilding a package after removing a build step

[Konrad Hinsen]

Hi Kaelyn,

> Whether it does or doesn't depends on the versions of the duplicate
> packages in question. If both packages have the same version, then the
> command line tools will warn about the ambiguous package
> specification. If the version numbers are different, then the tools
> will quietly choose the "newer" version (the larger version number).

My case was in neither of these well-defined situations. I had different
version number strings without an obvious order relation between them.
Not obvious to me at least. Here is an example (edited for focus):

  $ guix show -L . sbcl-websocket-driver
  name: sbcl-websocket-driver
  version: 0.2.0-0.df94496
  location: gnu/packages/lisp-xyz.scm:30847:4

  name: sbcl-websocket-driver
  version: 0.2.0-0.17ba553
  location: ./kh/packages/lisp.scm:53:4

  $ guix build -L . sbcl-websocket-driver
  /gnu/store/nd9ajz9ni395792f03ggf3jprf44cgz2-sbcl-websocket-driver-0.2.0-0.df94496

There are two definitions for sbcl-websocket-driver, differing only in
the commit id. How did "guix build" choose the first one? The second one
is newer, but figuring this out requires checking out the repository
and analyzing its DAG. It looks like Guix picked the larger one by
alphanumeric order, which is not a reasonable choice in a development
context.

Cheers,
  Konrad.

Re: Rebuilding a package after removing a build step

[Andreas Enge]

Am Mon, Sep 23, 2024 at 10:22:28AM +0200 schrieb Konrad Hinsen:
>   $ guix show -L . sbcl-websocket-driver
>   name: sbcl-websocket-driver
>   version: 0.2.0-0.df94496
>   location: gnu/packages/lisp-xyz.scm:30847:4
> It looks like Guix picked the larger one by
> alphanumeric order, which is not a reasonable choice in a development
> context.

This is exactly the role of the number "-0." in front of the git commit;
one needs to specify by hand the order of the (non-)releases, as coming
from here:
(define-public sbcl-websocket-driver
  (let ((commit "df94496ecb525d086eeada4f5875975515b7212e")
        (revision "0"))
    (package
     (name "sbcl-websocket-driver")
     (version (git-version "0.2.0" revision commit))

Andreas

Re: Rebuilding a package after removing a build step

[Konrad Hinsen]

Hi Andreas,

>> It looks like Guix picked the larger one by
>> alphanumeric order, which is not a reasonable choice in a development
>> context.
>
> This is exactly the role of the number "-0." in front of the git commit;
> one needs to specify by hand the order of the (non-)releases, as coming

Good to know, thanks! I had been wondering what those revision numbers
were good for.

Manual ordering looks fine for updating packages in a given channel. But
in this case, the two versions come from different channels. It's a bit
hard to globally coordinate manual revision numbers across all possible
channels!

Cheers,
  Konrad.

Re: Rebuilding a package after removing a build step

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

On 2024-09-23, Andreas Enge wrote:
> Am Mon, Sep 23, 2024 at 10:22:28AM +0200 schrieb Konrad Hinsen:
>>   $ guix show -L . sbcl-websocket-driver
>>   name: sbcl-websocket-driver
>>   version: 0.2.0-0.df94496
>>   location: gnu/packages/lisp-xyz.scm:30847:4
>> It looks like Guix picked the larger one by
>> alphanumeric order, which is not a reasonable choice in a development
>> context.
>
> This is exactly the role of the number "-0." in front of the git commit;
> one needs to specify by hand the order of the (non-)releases, as coming
> from here:
> (define-public sbcl-websocket-driver
>   (let ((commit "df94496ecb525d086eeada4f5875975515b7212e")
>         (revision "0"))
>     (package
>      (name "sbcl-websocket-driver")
>      (version (git-version "0.2.0" revision commit))

Rather than picking an arbitrary incremental number, I have in the past
used something based on the results from git describe... e.g. in my
current checkout of guix:

  $ git describe --match=v1.4'*'
  v1.4.0-142685-gfc059c66cf

e.g. 142685 commits past v1.4.0, with the commit fc059c66cf

That should *usually* end up in the correct order, although sometimes
there are surprises. For example, I had to specify --match otherwise it
picked v1.3.0 for some inscrutible git merge-ordering reason.

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Rebuilding a package after removing a build step

[Tobias Geerinckx-Rice]

While others have pointed out the workflow fix,

On 2024-09-23 10:22, Konrad Hinsen wrote:
>> If both packages have the same version, then the
>> command line tools will warn about the ambiguous package
>> specification. If the version numbers are different, then the tools
>> will quietly choose the "newer" version (the larger version number).
> 
> My case was in neither of these well-defined situations.

It was.  The second.  This behaviour is stable and documented, at least 
under (guix)Invoking guix package, although it applies to the entire 
CLI.

Kind regards,

T G-R

Sent from a Web browser.  Excuse or enjoy my brevity.

Re: Rebuilding a package after removing a build step

[Suhail Singh]

Vagrant Cascadian <vagrant@debian.org> writes:

> Rather than picking an arbitrary incremental number, I have in the past
> used something based on the results from git describe... e.g. in my
> current checkout of guix:
>
>   $ git describe --match=v1.4'*'
>   v1.4.0-142685-gfc059c66cf
>
> e.g. 142685 commits past v1.4.0, with the commit fc059c66cf
>
> That should *usually* end up in the correct order, although sometimes
> there are surprises. For example, I had to specify --match otherwise it
> picked v1.3.0 for some inscrutible git merge-ordering reason.

That seems like a useful and fairly generally applicable strategy.  Any
reason as to why something like that (which allows for additional
optional arguments such as "--match-v1.4'*'") shouldn't be used as the
default (as opposed to manual revision numbers)?

-- 
Suhail

Re: Rebuilding a package after removing a build step

[Kaelyn]

On Monday, September 23rd, 2024 at 11:17 AM, Suhail Singh <suhailsingh247@gmail.com> wrote:

> 
> 
> Vagrant Cascadian vagrant@debian.org writes:
> 
> > Rather than picking an arbitrary incremental number, I have in the past
> > used something based on the results from git describe... e.g. in my
> > current checkout of guix:
> > 
> > $ git describe --match=v1.4'*'
> > v1.4.0-142685-gfc059c66cf
> > 
> > e.g. 142685 commits past v1.4.0, with the commit fc059c66cf
> > 
> > That should usually end up in the correct order, although sometimes
> > there are surprises. For example, I had to specify --match otherwise it
> > picked v1.3.0 for some inscrutible git merge-ordering reason.
> 
> 
> That seems like a useful and fairly generally applicable strategy. Any
> reason as to why something like that (which allows for additional
> optional arguments such as "--match-v1.4'*'") shouldn't be used as the
> default (as opposed to manual revision numbers)?

There are a few problems with using that as the default:
* Generating that version number requires the git source repository; a package definition requires a fixed version number that can be computed without ever downloading the source code, since it is part of the package metadata (though such a strategy could potentially be used by e.g. "guix refresh -u" though that would require a consistent way of setting the revision in the package definition such that the tool knows what string to change).
* Using something like "git describe" to compute the revision would either be specific to packages which are built from git checkouts (i.e. built into the git-version function) or recipes for obtaining similar output would have to be written for the other version control systems for which support is desired. Closely related (but optional if the actual value of the revision is treated more loosely) is that some packages would need the "--match" flag to generate revision numbers from the correct point and the match expression would have to be updated when the "base" version for the revision counting changes.
* Within git repos, "git describe" only works if there are (annotated) tags for "git describe" to base the revision number on. If the source repo doesn't use tags, the approach won't work and manual revision numbers would still be needed. Likewise, if the repo doesn't use annotated tags or uses annotated tags for purposes other than tagging official releases, the chosen tag used for the revision (even at a given commit, if an older commit is given a new tag) may not be consistent without using a sufficiently specific "--match" flag.

I also want to say that the above points are primarily about automatically using "git describe" to dynamically generate a revision. I do not want to rule out or discourage ideas for ways to make use of "git describe" more convenient as a source of truth when determining what value to use for the revision. I think it could be quite useful especially for updating packages or doing local development.

Cheers,
Kaelyn

> 
> --
> Suhail

Re: Rebuilding a package after removing a build step

[Konrad Hinsen]

Hi Vagrant,

> Rather than picking an arbitrary incremental number, I have in the past
> used something based on the results from git describe... e.g. in my
> current checkout of guix:
>
>   $ git describe --match=v1.4'*'
>   v1.4.0-142685-gfc059c66cf
>
> e.g. 142685 commits past v1.4.0, with the commit fc059c66cf

That looks very good!

Not suitable for automation, as Kaelyn explained, but it could be
codified as a recommendation for developers. Possibly with tool support
for package developers, e.g. "guix derive-revision".

Cheers,
  Konrad.

Re: Rebuilding a package after removing a build step

[Ludovic Courtès]

Hi,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Unfortunately, when there are several packages with identical name and
> version number in two channels, Guix silently chooses one of them.
> It would probably be more useful to emit a warning.

I believe that’s already the case.

Specifically, if there are several packages with the same name and
different versions, the newest one is picked up:

--8<---------------cut here---------------start------------->8---
$ guix build gcc-toolchain -n
/gnu/store/9h8cbavj4q7pq1590xsnjw71ww0d06gk-gcc-toolchain-14.2.0-debug
/gnu/store/x2kv3zw2k7ql211m5kvb6yw401gab0x9-gcc-toolchain-14.2.0
/gnu/store/9h5kczcvfxs2fgn9708cs70xi0xpjrw6-gcc-toolchain-14.2.0-static
--8<---------------cut here---------------end--------------->8---

When there are several matching packages with the given name and
version, you get a warning:

--8<---------------cut here---------------start------------->8---
$ guix build guile@2.2 -n
guix build: warning: ambiguous package specification `guile@2.2'
guix build: warning: choosing guile@2.2.7 from gnu/packages/guile.scm:287:2
The following files would be downloaded:
  /gnu/store/3qj7zmfq5cw5k37zf8qz9dr4kld8rsnd-guile-2.2.7-debug
  /gnu/store/0g5cbaf3xczis6x11j3h2xc1x1dpp3xk-guile-2.2.7
--8<---------------cut here---------------end--------------->8---

In Guix itself, there are no packages with the exact same name/version
pair though.  The example above is because there’s 2.2.7 and 2.2.4.

Ludo’.

Re: Rebuilding a package after removing a build step

[Konrad Hinsen]

Hi Ludo,

>> Unfortunately, when there are several packages with identical name and
>> version number in two channels, Guix silently chooses one of them.
>> It would probably be more useful to emit a warning.
>
> I believe that’s already the case.

We already had some discussion, the conclusion being that there are
indeed tests for equal and for "higher" version, but that these tests do
not always correspond to the intention of "choosing the newest version".

In particular, Guix uses version *strings* rather than version
*numbers*, with alphabetical ordering for the non-numerical parts of
these strings. In the common case of version strings containing git
commit hashes, alphabetical order does not correspond to "newest
version". So you can consider yourself safe when working on the most
recent commit of a package, and yet end up building a older but
alphabetically higher version from another channel.

Andreas pointed out that this is the reason for adding a manually
curated revision number before such hashes. But there is no coordination
mechanism for such revision number across channels.

Vagrant suggested deriving a version number from the commit history
using "git describe". That sounds like a good solution, but it's not
used in Guix as far as I can tell.

Cheers,
  Konrad.