Building many packages in order

Building many packages in order

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

Hi,

testing the new python build system I’m seeking for some way to build
*many* packages in order.

Of course I can just run `guix build` with the list of packages to be
build. But when building all python packages, the list gets rather long
and does not fit into one call of `guix build` (due to the number of
arguments xargs passes to the command is restricted). Thus only
partitions of the packages are build in order.
 
So I need some way for passing *all* packages to be build to `guix build`.

Any hints?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |
man	


[-- Attachment #2: 0xBF773B65.asc --]
[-- Type: application/pgp-keys, Size: 14855 bytes --]

Re: Building many packages in order

[Roel Janssen]

Hartmut Goebel writes:

> Hi,
>
> testing the new python build system I’m seeking for some way to build
> *many* packages in order.
>
> Of course I can just run `guix build` with the list of packages to be
> build. But when building all python packages, the list gets rather long
> and does not fit into one call of `guix build` (due to the number of
> arguments xargs passes to the command is restricted). Thus only
> partitions of the packages are build in order.
>  
> So I need some way for passing *all* packages to be build to `guix build`.
>
> Any hints?

You could create a meta-package (just a package recipe) that has all
python packages as inputs.

The @code{gnome} package is an example of such a meta-package.

Kind regards,
Roel Janssen

Re: Building many packages in order

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

On Fri, Oct 21, 2016 at 12:56:08PM +0200, Hartmut Goebel wrote:
> Hi,
> 
> testing the new python build system I’m seeking for some way to build
> *many* packages in order.
> 
> Of course I can just run `guix build` with the list of packages to be
> build. But when building all python packages, the list gets rather long
> and does not fit into one call of `guix build` (due to the number of
> arguments xargs passes to the command is restricted). Thus only
> partitions of the packages are build in order.
>  
> So I need some way for passing *all* packages to be build to `guix build`.
> 
> Any hints?
> 

If you want to build all packages that are python- or python2-, then you
can call 'guix build $(guix package -A ^python | cut -f1)'

If you want all the packages that rely on python or python-2, then you
can call 'guix build $(guix refresh -l python python@2 | cut -d':' -f2)'

Both of these as written will stop when they encounter a package that
doesn't build. I'm sure there's also a scheme-y way of doing it.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: Building many packages in order

[Hartmut Goebel]

Am 21.10.2016 um 13:18 schrieb Roel Janssen:
> You could create a meta-package (just a package recipe) that has all
> python packages as inputs.

Thanks for sharing this idea. Unfortunately this make it complicated to
pass the version of the package (if there are several versions
available), since I would need to know the correct variable name.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Building many packages in order

[Hartmut Goebel]

Am 21.10.2016 um 13:53 schrieb Efraim Flashner:
> If you want to build all packages that are python- or python2-, then you
> can call 'guix build $(guix package -A ^python | cut -f1)'

I was about to answer that this will not work if the length of the
arguments string exceeds some limit. This made me re-check my premises
and found they are wrong: Both the length of the commands line and the
number of arguments to be passed to a new process are *really* big. See [1].

So I checked again and my command line is about 18,000 characters long
and even xargs calls `guix build` only once.

So my premises was irrelevant and my "mass rebuild" script works as
expected.

> If you want all the packages that rely on python or python-2, then you
> can call 'guix build $(guix refresh -l python python@2 | cut -d':' -f2)'

Thanks for this trick. I'm not used to "guix refresh" and my code for
selecting the packages is much more complicated. (I select them by
name.) I really have to get more common with these guix subcommands.


[1]
http://unix.stackexchange.com/questions/120642/what-defines-the-maximum-size-for-a-command-single-argument

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Building many packages in order

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

On Fri, Oct 21, 2016 at 02:53:02PM +0300, Efraim Flashner wrote:
> On Fri, Oct 21, 2016 at 12:56:08PM +0200, Hartmut Goebel wrote:
> > Hi,
> > 
> > testing the new python build system I’m seeking for some way to build
> > *many* packages in order.
> > 
> > Of course I can just run `guix build` with the list of packages to be
> > build. But when building all python packages, the list gets rather long
> > and does not fit into one call of `guix build` (due to the number of
> > arguments xargs passes to the command is restricted). Thus only
> > partitions of the packages are build in order.
> >  
> > So I need some way for passing *all* packages to be build to `guix build`.
> > 
> > Any hints?
> > 
> 
> If you want to build all packages that are python- or python2-, then you
> can call 'guix build $(guix package -A ^python | cut -f1)'
> 
> If you want all the packages that rely on python or python-2, then you
> can call 'guix build $(guix refresh -l python python@2 | cut -d':' -f2)'
> 
> Both of these as written will stop when they encounter a package that
> doesn't build. I'm sure there's also a scheme-y way of doing it.

I've been doing this to build all users of pytest, and I passed
--keep-going to `guix build`. I think that should work here, right?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Building many packages in order

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1876 bytes --]

On Fri, Oct 21, 2016 at 03:44:47PM -0400, Leo Famulari wrote:
> On Fri, Oct 21, 2016 at 02:53:02PM +0300, Efraim Flashner wrote:
> > On Fri, Oct 21, 2016 at 12:56:08PM +0200, Hartmut Goebel wrote:
> > > Hi,
> > > 
> > > testing the new python build system I’m seeking for some way to build
> > > *many* packages in order.
> > > 
> > > Of course I can just run `guix build` with the list of packages to be
> > > build. But when building all python packages, the list gets rather long
> > > and does not fit into one call of `guix build` (due to the number of
> > > arguments xargs passes to the command is restricted). Thus only
> > > partitions of the packages are build in order.
> > >  
> > > So I need some way for passing *all* packages to be build to `guix build`.
> > > 
> > > Any hints?
> > > 
> > 
> > If you want to build all packages that are python- or python2-, then you
> > can call 'guix build $(guix package -A ^python | cut -f1)'
> > 
> > If you want all the packages that rely on python or python-2, then you
> > can call 'guix build $(guix refresh -l python python@2 | cut -d':' -f2)'
> > 
> > Both of these as written will stop when they encounter a package that
> > doesn't build. I'm sure there's also a scheme-y way of doing it.
> 
> I've been doing this to build all users of pytest, and I passed
> --keep-going to `guix build`. I think that should work here, right?

I suppose it depends on if you want to fix each build error as it
happens. There's not a way that I can think of to say "only build the
ones that build successfully on hydra" or "print me a list of successes
and failures at the end"


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: Building many packages in order

[Hartmut Goebel]

Am 22.10.2016 um 17:47 schrieb Efraim Flashner:
> There's not a way that I can think of to say […] "print me a list of successes
> and failures at the end"

I did not find one the last time I checked. So I wrote myself a wrapper
script, tee-ing the whole output to a logfile and grep that log-file
after the build for failed build.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Building many packages in order

[Ricardo Wurmus]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Hi,
>
> testing the new python build system I’m seeking for some way to build
> *many* packages in order.
>
> Of course I can just run `guix build` with the list of packages to be
> build. But when building all python packages, the list gets rather long
> and does not fit into one call of `guix build` (due to the number of
> arguments xargs passes to the command is restricted). Thus only
> partitions of the packages are build in order.
>  
> So I need some way for passing *all* packages to be build to `guix build`.

Here’s what I do to rebuild all bioconductor packages:


guix build -e '((@@ (gnu packages) fold-packages)
                (lambda (pkg acc)
                  (if ((@@ (guix import cran) bioconductor-package?) pkg)
                      (cons pkg acc)
                      acc))
                (list))'

“fold-packages” takes a procedure that accumulates a list of packages.
It takes the current package and the list of accumulated packages so
far.  Here I’m using “bioconductor-package?” as a predicate, but you
could use something else in its place (e.g. a procedure testing the
package name).

~~ Ricardo

Re: Building many packages in order

[Hartmut Goebel]

Am 24.10.2016 um 15:56 schrieb Ricardo Wurmus:
> Here’s what I do to rebuild all bioconductor packages:
>
>
> guix build -e '((@@ (gnu packages) fold-packages)
>                 (lambda (pkg acc)
>                   (if ((@@ (guix import cran) bioconductor-package?) pkg)
>                       (cons pkg acc)
>                       acc))
>                 (list))'
>
> “fold-packages” takes a procedure that accumulates a list of packages.
> It takes the current package and the list of accumulated packages so
> far.  Here I’m using “bioconductor-package?” as a predicate, but you
> could use something else in its place (e.g. a procedure testing the
> package name).

Thanks for sharing this. Since I'm not a guile hacker, I'll stay with my
bash-script, even if it is not that performant :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |