proposal: make build failures explicit results in the store

proposal: make build failures explicit results in the store

[Florian Paul Schmidt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Hi,

currently I'm trying my hands on building _all_ available packages on
the x86_64 platform in a qemu-vm (with the intended aim to later
publish them to maybe take some load off hydra), but I came across an
issue that might have an easy solution but I'm not sure exactly what's
the right way to proceed.

Basically what I did right now was this:

for n in `guix package -A | cut -f1`; do guix build --no-substitutes\
- --max-silent-time=10 "$n" || true; done

Watching it run I noticed something though: Some package, let's call
it "foo1", seemed to have package "bar" as depencency (the concrete
example for bar was Qt5 which is probably why I noticed ;)), but the
bar build failed. So building foo1 failed, too. So far so good, aside
from foo1's and bar's failure everything is fine.

Now another package, foo2, comes along and it, too, depends on bar.
Since bar failed to build previously guix thinks it's a swell idea to
retry building bar. This, of course, is silly, and in this concrete
example, Qt5, a massive waste of cpu cycles and the only good thing it
does is to bring the universe closer to the heat death.

Afaict the real problem is that the functional approach is slightly
broken here (being functional is fine, just the implementation is
broken). The attempt to build bar _had_ a result. That result was a
build failure and as such it should be explicitly represented in the
store such that when another attempt to build bar comes along, the
result is already known: build failure. And this should make the build
of foo2 fail immediately, too, since one of its dependencies failed to
build.

Once the package definition of bar changes, there's a new store path
for its result and the build will be retried.

Do I miss something here? What do you think?

Have fun,
Flo

- -- 
https://fps.io
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWVW37AAoJEA5f4Coltk8ZpDsIAJNXNMs73RJvPa+XMwjLLnIb
zckxdP12xDzq4CH+P4ehqSip+6evTfnaW+9VlYLjTdw7maD6VNk1CFIaOUObAxSI
wwaX1JsHEzoJgaymHOSa+ZSoa8U8BczZeOkoiupJVQevSTryC0nHvX5iWOxJdNC5
/AzGo/VfvYVIciHtDwzcq7cExS5NRBZSvwoaeITdtFk+/XwkQJ8d78eH+EUBYoQK
saoy4pklnpesr4jofmI32yZ9lxONkmYd1etpwIyirIuEnxsqzGk9EHTTBO4MmrJg
B3OQnh/fNoqwToQblWIn/tEAJZO7h0a2clmaWNDf0wwMIfO2NNfMotAWYzXBj1A=
=6KOG
-----END PGP SIGNATURE-----

Re: proposal: make build failures explicit results in the store

[Florian Paul Schmidt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/25/2015 09:14 AM, Florian Paul Schmidt wrote:

> Afaict the real problem is that the functional approach is slightly
> broken here (being functional is fine, just the implementation is
> broken). The attempt to build bar _had_ a result. That result was a
> build failure and as such it should be explicitly represented in
> the store such that when another attempt to build bar comes along,
> the result is already known: build failure. And this should make
> the build of foo2 fail immediately, too, since one of its
> dependencies failed to build.

Actually it seems I just misinterpreted my observations. Never mind
the noise :)

Flo

- -- 
https://fps.io
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWVXEpAAoJEA5f4Coltk8ZFgQH/R4XwG1XPKh+dEHHw83guZTh
bd2QfkYT3MZiFIu/JVcIbXqT04Kgqn663kqgDd9vQCG52DK0EA/fdEKiSqbKdAy0
+XS+JpnXecqsT3mLGCqdU/3khY+IzfvIkRuHofR5x8HGSQ10PYqGiHZwA8Guuwz6
PgTYWn2pd9TPO9YMemK1gsHtSEbwcMOih10qoOyThKMgmj63dEY7GK/eKTLakEpR
g/h2KUlblIV46p0j0hdkmLIMxHYZvJphCJTu9R0V8uW5INta1M1MI4QhvvVGmto8
l517JiGX1H3eaLo18B2xdOhQnQbjWF4XSEAh5Ps+pmw0syHZrEEeGTVSNtXW6L8=
=P5YV
-----END PGP SIGNATURE-----

Re: proposal: make build failures explicit results in the store

[Taylan Ulrich Bayırlı/Kammer]

Florian Paul Schmidt <mista.tapas@gmx.net> writes:

> Hi,
>
> currently I'm trying my hands on building _all_ available packages on
> the x86_64 platform in a qemu-vm (with the intended aim to later
> publish them to maybe take some load off hydra), but I came across an
> issue that might have an easy solution but I'm not sure exactly what's
> the right way to proceed.
>
> Basically what I did right now was this:
>
> for n in `guix package -A | cut -f1`; do guix build --no-substitutes\
> --max-silent-time=10 "$n" || true; done
>
> Watching it run I noticed something though: Some package, let's call
> it "foo1", seemed to have package "bar" as depencency (the concrete
> example for bar was Qt5 which is probably why I noticed ;)), but the
> bar build failed. So building foo1 failed, too. So far so good, aside
> from foo1's and bar's failure everything is fine.
>
> Now another package, foo2, comes along and it, too, depends on bar.
> Since bar failed to build previously guix thinks it's a swell idea to
> retry building bar. This, of course, is silly, and in this concrete
> example, Qt5, a massive waste of cpu cycles and the only good thing it
> does is to bring the universe closer to the heat death.

Hi,

You can run the Guix daemon with the switch --cache-failures and it will
do pretty much exactly what you want. :-)

(I'm not sure whether it actually puts the failures into /gnu/store in
some format, so "pretty much.")

Don't know why this isn't enabled by default.

Taylan

Re: proposal: make build failures explicit results in the store

[Ludovic Courtès]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> You can run the Guix daemon with the switch --cache-failures and it will
> do pretty much exactly what you want. :-)

Yep.

> (I'm not sure whether it actually puts the failures into /gnu/store in
> some format, so "pretty much.")

Technically, nothing is left behind in /gnu/store, but the database (by
default /var/guix/db/db.sqlite) records the derivation as failed.  You
can use ‘guix gc --clear-failures xxx’ to remove that mark.

> Don't know why this isn't enabled by default.

Because we’re not confident enough about build determinism?  :-)

Ludo’.

Re: proposal: make build failures explicit results in the store

[Florian Paul Schmidt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/25/2015 02:22 PM, Ludovic Courtès wrote:
> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
> 
>> You can run the Guix daemon with the switch --cache-failures and
>> it will do pretty much exactly what you want. :-)
> 
> Yep.

Thanks for your helpful comments, Taylan and Ludovic.. I changed the
guix-service's configuration on the machine to have the
- --cache-failures option then and restarted the whole build procedure.

> 
>> (I'm not sure whether it actually puts the failures into
>> /gnu/store in some format, so "pretty much.")
> 
> Technically, nothing is left behind in /gnu/store, but the database
> (by default /var/guix/db/db.sqlite) records the derivation as
> failed.  You can use ‘guix gc --clear-failures xxx’ to remove that
> mark.

Oh, I wasn't even aware there was a DB. I had a more functional
picture in mind where there was a map

package definition -> entr(y|ies) in store

where "package definition" transitively also encompassed all inputs,
etc, where "entry in store" was a somewhat loosely defined type (a
directory with some things in it) and that's "it" ;) In this picture I
thought it would make sense to make the output a variant type: Either
a directory with some things it XOR a build failure. I suppose the
entry in the DB achieves this albeit a little obscured.

Thanks for the heads up.

>> Don't know why this isn't enabled by default.
> 
> Because we’re not confident enough about build determinism?  :-)

;)

Regards,
Flo

- -- 
https://fps.io
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWVcONAAoJEA5f4Coltk8Zh90H/R4NCNWQjSvjKdgdp3ZL7pzd
ARalvopgHLxISAX4zE6Fd+Xq6hu1lD4K8G9UW9htiaxetgA58Jt7I1ESmCbMnZf5
ZgDWuRe2xXjXx+nXrTJWDA37nLQmOH7k81FiA+RCwSJ/fGQVO4md6731QCR0O6zP
S1GMbADeh3kBGrJbn6+eidv0XrEzPRg/u2gBkKuOqBddbM4eeqKGKqrP4xzDD9rn
XMJ6RA3P3YgIUH60wNdDPjkc15lEDtgzV6V2DfFUVdoHB5vydQ1UN9JH2TP4mSPN
sITCeIJJ5CpIBmO55YrRhuuL4fSKuozgiwmASexK4BKNX4Ku5vvOz9oWJM6I+cY=
=5JId
-----END PGP SIGNATURE-----

Re: proposal: make build failures explicit results in the store

[Florian Paul Schmidt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/25/2015 03:19 PM, Florian Paul Schmidt wrote:

> Oh, I wasn't even aware there was a DB. I had a more functional 
> picture in mind where there was a map
> 

I guess I should have written "simpler" instead of "more functional"..

Flo

- -- 
https://fps.io
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWVcrkAAoJEA5f4Coltk8ZjLwH/1OQUUSjw0WBZz9wZDFA6q3z
FauWAMOMJFMlR6Un9HLtFAQpnNh6hXKa9bl2LI2soJDZ7bxsNL/MXstYFWGCoAg6
6NtCEgPKLdZPD+pRUk/nVbYnk71VrEwh7PPmSOyg6RhNKJSHEaj4busH/RTnflTZ
sbAea9zXTAiyhpaiz1MLyo5CXQ6AZOBCbSqKmP4jNhxSOGuA3g3Y2RNrcAMtWWN6
bHGU8nwx3aNi9vSdT1WV6eoAyeqbVsoMSBM4kjjW3Omh1KwMLMJz5Db+uauI4Vyq
LDxRNxxdjOFijTSMe4sRV/3tf2fVvBHTAJ9YZ3A4lNz3SbvHILzJYEXKY4Mlh5w=
=Kvmu
-----END PGP SIGNATURE-----

Re: proposal: make build failures explicit results in the store

[Ludovic Courtès]

Florian Paul Schmidt <mista.tapas@gmx.net> skribis:

> Oh, I wasn't even aware there was a DB. I had a more functional
> picture in mind where there was a map

See <http://www.gnu.org/software/guix/manual/html_node/The-Store.html>.

> package definition -> entr(y|ies) in store

The store is at a lower level, it doesn’t know about packages.

Ludo’.

Re: proposal: make build failures explicit results in the store

[Florian Paul Schmidt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/25/2015 10:49 PM, Ludovic Courtès wrote:
> Florian Paul Schmidt <mista.tapas@gmx.net> skribis:
> 
>> Oh, I wasn't even aware there was a DB. I had a more functional 
>> picture in mind where there was a map
> 
> See
> <http://www.gnu.org/software/guix/manual/html_node/The-Store.html>.
>
> 
>> package definition -> entr(y|ies) in store
> 
> The store is at a lower level, it doesn’t know about packages.

I never meant to imply that it did.

If you have a map (a function)

f: A -> B

where A is the input type and B is the result type, how could B "know
about" A? It's a completely separate type from A and the map f.

This is completely off topic from the original question, but I do
wonder how the impression that I might have implied that the store
"knows about packages" came about :)

Oh, sure, the map f might be composed of other maps like taking a
package definition to (a) derivation(s) first and then taking the
derivation(s) to (a) store entr(y|ies).

All I wanted to express was that I had a lack of knowledge about what
gets mapped to what :) The output of the map is not just one or more
store entries, but also possibly, depending on the configuration, a DB
entry which records the failure, which indeed then is a variant type
in some sense, albeit a little obscured :)

Flo

- -- 
https://fps.io
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWVviKAAoJEA5f4Coltk8ZrE4IAIbyLUODQ32JGdfVPemOgVuq
X5816EfFywW3h/ChSVOHbI6q/wZ9oPv84+R4FvGORDnGq23cGqNMVmpG0isBqzsi
hQujWMwCIDDO3TA+ih7Fm826I4eVkWeQq0fjh03FcZ+xJ370/LvHZhAnUP3g+zfx
1q4gbbgVG+IeiOkeCqXEI15v8veIPLjU30IN9LRW7ZL3K3pfofgYilQlQ2b/DPNn
4m0hRsobI9VcJY9QsySQd07QpgJ58z7ZJ/vrxed/MsZhdn/mK7xI3w4JpBy0h86J
TQW5seCRX5V1w7EkuYCT1zwtUAzMxEsuhmpGAGZJ2wUCg6YevCEE6N6M948aUq8=
=EAj3
-----END PGP SIGNATURE-----