Building and installing packages with modifications

Building and installing packages with modifications

[白い熊]

Hi Guix: 

This is a question derived from my current trouble with running Guix on Android, as seen in: http://lists.gnu.org/archive/html/help-guix/2019-02/msg00151.html

Basically: 
— I have Guix running on an armhf Android phone 
— “guix pull” won't complete as it needs to build curl locally, since a substitute is unavailable 
— curl local build fails on checking internet connection in one of its tests, as is common, as I found out on the web 
— Thus “guix pull” never finishes, and I can't install any packages etc, since even without pulling a new version, they also fall into building curl 

I'd like to overcome this, am not sufficiently advanced in Guix, so would like to receive ideas / tips on how to best / easiest do this. 

Currently, I'm thinking of two options. 

① Build curl locally, without the failing tests. 
② Install an older substitute version of curl already built. 

Ad ①: 
— I've found there is an environment variable (INTERNET_TEST or something) you can set so the test is not run during auto build run. It's ignored if “guix build curl” is run with it. 
— Can environment variables be passed to “guix build” so they are then honored during build? 
— If not, can I instruct “guix build” to not run “make test”? 
— If both not, can I then carry out the individual stages of a package build in Guix, so I can skip the test phase? 
— If I build it in any of these ways, will “guix package -i curl” then install this version in the profile? 
— If not, can I install it any other way then, so I can commence with “guix pull” without it being rebuilt? 

Ad ②: 
— Is there any way I can search the substitute servers for the latest built substitute of a package? 
— How can I then install it? 

If none of these, is there another way to overcome my problem? 

Many thanks for any advice! :@) 
--
白い熊

Re: Building and installing packages with modifications

[ison]

Since this hasn't received any replies I'll give my solution, although I
apologize in advance if this isn't the "best" way to do it.

When I want to modify a package that I install via "guix package -i" (as opposed
to using the GuixSD system configuration) I just create a new file for each such
package that inherits the default package and then override any options I need.
For example, in the case of curl you could make a file with the contents:
--------------------
(use-modules ((guix licences) #:prefix license:)
              (gnu packages)
              (gnu packages curl)
              (guix packages)
              (guix utils)))

(package
  (inherit curl)
  (substitute-keyword-arguments (package-arguments curl)
    ((#:tests? #f #f)
      #f)))
--------------------
which will just inherit curl and only modify the flag "tests?" to disable tests.
Everything else will be kept the same. Then if you save that to a file called
my-curl.scm you could install it to your guix profile with
"guix package -f my-curl.scm"

Re: Building and installing packages with modifications

[白い熊＠相撲道]

On February 24, 2019 6:46:20 PM UTC, ison <ison@airmail.cc> wrote:
>Since this hasn't received any replies I'll give my solution, although
>I
>apologize in advance if this isn't the "best" way to do it.
>
… 
>which will just inherit curl and only modify the flag "tests?" to
>disable tests.
>Everything else will be kept the same. Then if you save that to a file
>called
>my-curl.scm you could install it to your guix profile with
>"guix package -f my-curl.scm"

Thanks for this — it's definitely a big help. Now though, this would mean I'd then have “my-curl” installed in my profile and on the next “guix pull” it would still try to build “curl” with the tests and fail… 

Do you think, if I save it as curl.scm and then install it — it would have the desired effect of installing it in my profile as “curl” and then the “guix pull” could proceed further as it would have a working “curl” installed? 
--
白い熊＠相撲道

Re: Building and installing packages with modifications

[ison]

So basically you're saying curl is a dependency of other packages and it's
failing to build due to the tests? That makes it more complicated. You could
always try renaming it to "curl.scm" and testing it, but I'm failry certain it
won't work.

Again, I'm really not sure if my way is the best way. I'm still learning a lot
myself and making new packages that inherit existing packages is just a neat
trick I've used myself and thought it could help you. So the best answer I could
give to your predicament is to just keep using the same trick. Find out what
package is pulling in "curl" as a dependency and then define a new package for
that one (using the same trick) and override its "curl" input with "my-curl"

Although it seems like this can get very ugly very quickly, so I'm also curious
now if anyone else knows a better way to handle this. It's almost as if you need
to make "curl" a global (and then modify it) so that all packages will use your
modified curl instead of the one they pull in from use-modules.
I'm not sure.

Re: Building and installing packages with modifications

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1163 bytes --]

ison <ison@airmail.cc> writes:

> Although it seems like this can get very ugly very quickly, so I'm also curious
> now if anyone else knows a better way to handle this. It's almost as if you need
> to make "curl" a global (and then modify it) so that all packages will use your
> modified curl instead of the one they pull in from use-modules.
> I'm not sure.

I don't know of a better way, myself.  Guix insists on running the
tests.  We've talked about ways we might be able to relax that
restriction [1], but for now it seems difficult to do.  In addition, if
substitutes are more frequently available and you have authorized them,
then it isn't as big of a problem.

In the meantime, modifying packages can be a good work-around.  In the
case of a package that gets used by many other packages, it would be
difficult because you would want to replace practically every reference
(maybe less, if you only need a few packages).  An intrepid hacker might
be able to whip something up by using (guix discovery), though.

Hope that helps!

Footnotes: 
[1]  https://lists.gnu.org/archive/html/guix-devel/2018-04/msg00071.html

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Building and installing packages with modifications

[Wayne]

Quoting ison <ison@airmail.cc>:

> Since this hasn't received any replies I'll give my solution, although I
> apologize in advance if this isn't the "best" way to do it.
>
> When I want to modify a package that I install via "guix package -i"  
> (as opposed
> to using the GuixSD system configuration) I just create a new file  
> for each such
> package that inherits the default package and then override any  
> options I need.
> For example, in the case of curl you could make a file with the contents:
> --------------------
> (use-modules ((guix licences) #:prefix license:)
>               (gnu packages)
>               (gnu packages curl)
>               (guix packages)
>               (guix utils)))
>
> (package
>   (inherit curl)
>   (substitute-keyword-arguments (package-arguments curl)
>     ((#:tests? #f #f)
>       #f)))
> --------------------
> which will just inherit curl and only modify the flag "tests?" to  
> disable tests.
> Everything else will be kept the same. Then if you save that to a file called
> my-curl.scm you could install it to your guix profile with
> "guix package -f my-curl.scm"

   thank you for sharing, a good clear example of the power of using  
Guile for package descriptions.  i am very interested in this as a  
long time Gentoo user and had assumed that in Guix there was a nice  
way to achieve something similar to Gentoo's handy USE flag feature  
[1], as hinted at by this reddit post [2].  this helps :)

   my follow up question is whether there is (or a plan for) any  
support for sharing these derived custom build binaries with other  
users, especially given the reproducability and verification of these  
builds already built into Guix e.g. if my custom package,  
architecture, etc. matches one already built and shared, substitute  
with that binary instead of rebuilding?  while as useful (pun  
intended!) as USE flags can be, the never-ending compilation of  
packages in Gentoo (and its associated waste of time, CPU and most  
importantly energy) has me looking for alternatives which has led to  
GuixSD, seeminglyg a good compromise .. with lots of added features!   
currently in Gentoo there is little support for sharing binaries,  
particularly when not using the default USE flags of a package ..  
though the feature is there to support sharing of even these custom  
USE flag binaries [3].

   thus basically are there any plans for sharing custom package  
definition binaries, perhaps even peer-to-peer?  i thought that i had  
read something about this on this list but cannot find it currently in  
the archives.

thanks, w

PS: kudos to all the FOSDEM presenters!  i just finished watching  
quite a few lectures and found them all not only informative but  
enjoyable :)


[1] https://wiki.gentoo.org/wiki/USE_flag
[2] https://www.reddit.com/r/GUIX/comments/58xvjx/guixsd_vs_gentoo/
[3] https://bugs.gentoo.org/150031


-- 
https://waynedpj.ingiro.xyz

Re: Building and installing packages with modifications

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 5887 bytes --]

Hi,

ison <ison@airmail.cc> writes:

> Since this hasn't received any replies I'll give my solution, although I
> apologize in advance if this isn't the "best" way to do it.

No need to apologize!  There are many ways to solve problems, and often
I find it very helpful to see the different ways others have solved the
same problem.  Thank you for taking the time to share your example!

> (package
>   (inherit curl)
>   (substitute-keyword-arguments (package-arguments curl)
>     ((#:tests? #f #f)
>       #f)))

That's a great example!  Unfortunately, I'm not sure it will work out of
the box.  I think you would need to write it like this (note the
"arguments" field):

  (package
    (inherit curl)
    (arguments
      (substitute-keyword-arguments (package-arguments curl)
      ((#:tests? #f #f)
        #f))))

Furthermore, although that will successfully disable tests for the curl
package because it lacks a #:tests? package argument, I don't think it
will work in general.  For example, suppose you try the same trick for
the emacs-dash package, which has the following arguments:

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,pp (package-arguments emacs-dash)
$12 = (#:tests? #t #:test-command '("./run-tests.sh"))
scheme@(guix-user)> (define emacs-dash-no-tests (package (inherit emacs-dash) (arguments (substitute-keyword-arguments (package-arguments emacs-dash) ((#:tests? #f #f) #f)))))
scheme@(guix-user)> ,pp (package-arguments emacs-dash-no-tests)
$13 = (#:tests? #t #:test-command '("./run-tests.sh"))
scheme@(guix-user)> 
--8<---------------cut here---------------end--------------->8---

The arguments are unchanged.  I think this happens because the
substitute-keyword-arguments form you've used looks specifically for a
#:tests? keyword argument with value #f (the Guile manual explains that
the #f pattern will match only #f, in (guile) Pattern Matching), so it
doesn't match in the case where #:tests? is given the value #t
explicitly.  This results in no substitution.

My understanding is that you intended to disable the tests.  I think you
can still accomplish that if you change the (#:tests? #f #f) part to
(#:tests? _ #f), like this:

  (package
    (inherit curl)
    (arguments
      (substitute-keyword-arguments (package-arguments curl)
      ((#:tests? _ #f)
        #f))))

Here, the underscore in (#:tests? _ #f) is a variable that will be bound
to the existing value of #:tests? (and then ignored).  This _will_ match
#t, so the substitution will occur even when #:tests? was originally set
to #t.

> [...] if you save that to a file called my-curl.scm you could install
> it to your guix profile with "guix package -f my-curl.scm"

Maybe you already know about this, bu one can also use GUIX_PACKAGE_PATH
or set up a local channel (i.e., a channel for which the url is simply a
path to a local Git repository, such as
"/home/marusich/my-guix-channel").  I've found that putting custom
package definitions in a single place makes it much easier to manage,
especially because then I can install any of the custom packages in a
manifest file via "guix package -m my-manifest.scm".

Wayne <waynedpj@ingiro.xyz> writes:

>   my follow up question is whether there is (or a plan for) any
> support for sharing these derived custom build binaries with other
> users

Yes!  For starters, see (guix) Channels in the manual.  Here's an online
copy:

https://www.gnu.org/software/guix/manual/en/html_node/Channels.html

Basically, channels provide an easy way for people to share custom
package definitions - but not binaries.  It's very nice!  Here's a blog
post showcasing them in more detail:

https://www.gnu.org/software/guix/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/

>   thus basically are there any plans for sharing custom package
> definition binaries, perhaps even peer-to-peer?  i thought that i had
> read something about this on this list but cannot find it currently in
> the archives.

The task of sharing package definitions is orthogonal to the task of
sharing pre-built binaries (called "substitutes" in the Guix world).
Sharing package definitions is easy to do now, in a decentralized
fashion, thanks to channels and the distributed nature of Git.

Sharing substitutes is also possible, but less practical.  To start
with, you can manually ship around binaries with "guix copy" or "guix
archive".  And anyone (with root access) can publish substitutes on a
network for others to use with the "--substitute-urls" build option
simply by invoking "guix publish".  You can even set up a build farm
like the one running at berlin.guixsd.org if you want.  For details on
how to do that, see the Guix maintenance repository here:

https://savannah.gnu.org/git/?group=guix

However, getting people to trust your Guix server's key, providing the
substitutes with high availability, and continuously building all the
packages is a non-trivial task.  Ultimately, it still feels pretty
centralized.  In the future, we hope people will be interested in
contributing to and using a more distributed solution.  For example,
integration with something like IPFS or Gnunet would be great!  But we
need help from you and others to make it a reality.

For more information, you can check the email lists.  I found a few
likely results via the following searches:

https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=ipfs&submit=Search%21&idxname=guix-devel&max=20&result=normal&sort=score

https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=gnunet+publish&submit=Search%21&idxname=guix-devel&max=20&result=normal&sort=score

If you're interested in helping out or know people who might be
interested, please let us know.

Happy hacking,

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Building and installing packages with modifications

[Wayne]

Quoting Chris Marusich <cmmarusich@gmail.com>:
>
> Wayne <waynedpj@ingiro.xyz> writes:
>
>>   my follow up question is whether there is (or a plan for) any
>> support for sharing these derived custom build binaries with other
>> users
>
> Yes!  For starters, see (guix) Channels in the manual.  Here's an online
> copy:
>
> https://www.gnu.org/software/guix/manual/en/html_node/Channels.html
>
> Basically, channels provide an easy way for people to share custom
> package definitions - but not binaries.  It's very nice!  Here's a blog
> post showcasing them in more detail:
>
> https://www.gnu.org/software/guix/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/
>
>>   thus basically are there any plans for sharing custom package
>> definition binaries, perhaps even peer-to-peer?  i thought that i had
>> read something about this on this list but cannot find it currently in
>> the archives.
>
> The task of sharing package definitions is orthogonal to the task of
> sharing pre-built binaries (called "substitutes" in the Guix world).
> Sharing package definitions is easy to do now, in a decentralized
> fashion, thanks to channels and the distributed nature of Git.
>
> Sharing substitutes is also possible, but less practical.  To start
> with, you can manually ship around binaries with "guix copy" or "guix
> archive".  And anyone (with root access) can publish substitutes on a
> network for others to use with the "--substitute-urls" build option
> simply by invoking "guix publish".  You can even set up a build farm
> like the one running at berlin.guixsd.org if you want.  For details on
> how to do that, see the Guix maintenance repository here:
>
> https://savannah.gnu.org/git/?group=guix
>
> However, getting people to trust your Guix server's key, providing the
> substitutes with high availability, and continuously building all the
> packages is a non-trivial task.  Ultimately, it still feels pretty
> centralized.  In the future, we hope people will be interested in
> contributing to and using a more distributed solution.  For example,
> integration with something like IPFS or Gnunet would be great!  But we
> need help from you and others to make it a reality.
>
> For more information, you can check the email lists.  I found a few
> likely results via the following searches:
>
> https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=ipfs&submit=Search%21&idxname=guix-devel&max=20&result=normal&sort=score
>
> https://lists.gnu.org/archive/cgi-bin/namazu.cgi?query=gnunet+publish&submit=Search%21&idxname=guix-devel&max=20&result=normal&sort=score
>
> If you're interested in helping out or know people who might be
> interested, please let us know.

   i am definitely interested in creating this reality, as Guix seems  
like a great foundation for these features.  though i still need to  
first install Guix System and learn Scheme, so it might be some time  
;)  thanks for the reply and resources, will start digesting those  
first.

peace, w


-- 
https://waynedpj.ingiro.xyz

Re: Building and installing packages with modifications

[Gábor Boskovits]

Hello,

白い熊 <help-guix_gnu.org@sumou.com> ezt írta (időpont: 2019. febr. 16.,
Szo, 16:12):
>
> Hi Guix:
>
> This is a question derived from my current trouble with running Guix on Android, as seen in: http://lists.gnu.org/archive/html/help-guix/2019-02/msg00151.html
>
> Basically:
> — I have Guix running on an armhf Android phone
> — “guix pull” won't complete as it needs to build curl locally, since a substitute is unavailable
> — curl local build fails on checking internet connection in one of its tests, as is common, as I found out on the web

There is one thing I can't understand. How is that this test does not
fail otherwise? Is it only done on Android?
I ask this, as none of the guix builds have internet access. If there
is any such test in a package, then the most
usual way is to disable the test in our package, as it won't be able to succeed.

> — Thus “guix pull” never finishes, and I can't install any packages etc, since even without pulling a new version, they also fall into building curl
>
> I'd like to overcome this, am not sufficiently advanced in Guix, so would like to receive ideas / tips on how to best / easiest do this.
>
> Currently, I'm thinking of two options.
>
> ① Build curl locally, without the failing tests.
> ② Install an older substitute version of curl already built.
>
> Ad ①:
> — I've found there is an environment variable (INTERNET_TEST or something) you can set so the test is not run during auto build run. It's ignored if “guix build curl” is run with it.
> — Can environment variables be passed to “guix build” so they are then honored during build?
> — If not, can I instruct “guix build” to not run “make test”?
These would involve modifying the package, I believe using this
environment variable would be the best way, maybe
adding it to configure-flags/make-flags...
> — If both not, can I then carry out the individual stages of a package build in Guix, so I can skip the test phase?
> — If I build it in any of these ways, will “guix package -i curl” then install this version in the profile?
It will install the latest version known by the guix called. One way
to solve this is build guix from a checkout,
modify the package, bump the version to make sure it gets updated, and
the pre-inst-env guix package -i it.
> — If not, can I install it any other way then, so I can commence with “guix pull” without it being rebuilt?
Guix pull might need some tweaking, as you can't build to the same
hash, so it won't be picked up anyways.
You can pull from another version of guix, for example the one build
from the chekout, even using
pre-inst-env guix pull... Then the modified package will be picked up.
>
> Ad ②:
> — Is there any way I can search the substitute servers for the latest built substitute of a package?
> — How can I then install it?
>
> If none of these, is there another way to overcome my problem?
>
> Many thanks for any advice! :@)
> --
> 白い熊
>

Best regards,
g_bor