unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Test failure when building libarchive-3.1.2
@ 2015-11-13 11:33 Jan Synáček
  2015-11-14 11:40 ` Ludovic Courtès
  0 siblings, 1 reply; 15+ messages in thread
From: Jan Synáček @ 2015-11-13 11:33 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 4934 bytes --]

Hello,

I have a relatively recent guix version (ran guix pull yesterday) and I
can't build emacs, as libarchive, one of its dependencies, fails to build.
Here is a snippet of the error message:

"
Reference files will be read from:
/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/tar/test
Running tests on:
"/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/./bsdtar"
Exercising: bsdtar 3.1.2 - libarchive 3.1.2

  0: test_0                                            ok
  1: test_basic                                        ok
  2: test_copy                                         ok
  3: test_empty_mtree                                  ok
  4: test_extract_tar_bz2                              ok
  5: test_extract_tar_grz                              ok
  6: test_extract_tar_gz                               ok
  7: test_extract_tar_lrz                              ok
  8: test_extract_tar_lz                               ok
  9: test_extract_tar_lzma                             ok
 10: test_extract_tar_lzo                              ok
 11: test_extract_tar_xz                               ok
 12: test_extract_tar_Z                                ok
 13: test_format_newc                                  ok
 14: test_help                                         ok
 15: test_option_a                                     ok
 16: test_option_b64encode                             ok
 17: test_option_b                                     FAIL
 18: test_option_C_upper                               ok
 19: test_option_exclude                               ok
 20: test_option_gid_gname                             ok
 21: test_option_grzip                                 ok
 22: test_option_H_upper                               ok
 23: test_option_j                                     ok
 24: test_option_k                                     ok
 25: test_option_keep_newer_files                      ok
 26: test_option_lrzip                                 ok
 27: test_option_L_upper                               ok
 28: test_option_lzma                                  ok
 29: test_option_lzop                                  ok
 30: test_option_n                                     ok
 31: test_option_newer_than                            ok
 32: test_option_nodump                                ok
 33: test_option_older_than                            ok
 34: test_option_O_upper                               ok
 35: test_option_q                                     ok
 36: test_option_r                                     ok
 37: test_option_s                                     ok
 38: test_option_T_upper                               ok
 39: test_option_uid_uname                             ok
 40: test_option_uuencode                              ok
 41: test_option_U_upper                               ok
 42: test_option_X_upper                               ok
 43: test_option_xz                                    ok
 44: test_option_z                                     ok
 45: test_patterns                                     ok
 46: test_print_longpath                               ok
 47: test_stdio                                        ok
 48: test_strip_components                             ok
 49: test_symlink_dir                                  ok
 50: test_version                                      ok
 51: test_windows                                      ok

Totals:
  Tests run:               52
  Tests failed:             1
  Assertions checked:    7715
  Assertions failed:        2
  Skips reported:           6

Failing tests:
  17: test_option_b (2 failures)

Details for failing tests:
/tmp/nix-build-libarchive-3.1.2.drv-0/bsdtar_test.2015-11-13T09.52.46-000

phase `check' failed after 77.9 seconds
builder for
`/gnu/store/671ccl9qibwzzaizq3g9rjdinisqspiz-libarchive-3.1.2.drv' failed
with exit code 1
cannot build derivation
`/gnu/store/laphmpnyxy3aiwn6s2s2hmk6a2kb110l-cmake-3.3.2.drv': 1
dependencies couldn't be built
cannot build derivation
`/gnu/store/aw9721lq55hzgy3ld8hw0jvxq04pc7mn-graphite2-1.3.3.drv': 1
dependencies couldn't be built
cannot build derivation
`/gnu/store/p2yxxb0xn8yrmrnbh3y1v6crlx8bxxjc-openjpeg-1.5.2.drv': 1
dependencies couldn't be built
cannot build derivation
`/gnu/store/rsbdafy8zjb8iyx035cd35x28pd295lk-emacs-24.5.drv': 1
dependencies couldn't be built
cannot build derivation
`/gnu/store/w6ifcj7iq7g23rx8xy55325kxavpp65y-gtk+-3.18.2.drv': 1
dependencies couldn't be built
cannot build derivation
`/gnu/store/sbamsn3qcwc6ny09ix51swagczd6wmrl-profile.drv': 1 dependencies
couldn't be built
guix package: error: build failed: build of
`/gnu/store/sbamsn3qcwc6ny09ix51swagczd6wmrl-profile.drv' failed
"

Let me know if I can provide more details. I'm running guix on Fedora 23.

-- 
Jan Synáček

[-- Attachment #2: Type: text/html, Size: 7102 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-13 11:33 Test failure when building libarchive-3.1.2 Jan Synáček
@ 2015-11-14 11:40 ` Ludovic Courtès
  2015-11-14 12:47   ` Jan Synáček
  2015-11-20 13:44   ` Jan Synáček
  0 siblings, 2 replies; 15+ messages in thread
From: Ludovic Courtès @ 2015-11-14 11:40 UTC (permalink / raw)
  To: Jan Synáček; +Cc: guix-devel

Jan Synáček <jan.synacek@gmail.com> skribis:

> Reference files will be read from:
> /tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/tar/test
> Running tests on:
> "/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/./bsdtar"
> Exercising: bsdtar 3.1.2 - libarchive 3.1.2

[...]

>  17: test_option_b                                     FAIL

Ricardo reported the same issue a while back:

  https://lists.gnu.org/archive/html/guix-devel/2015-03/msg00182.html

What platform is this on, i686?

It would be nice to see if this systematically fails.  If it is
non-deterministic, we should build it with --keep-failed until it fails
(removing successful builds with ‘guix gc -d’), collect useful info from
the build tree, and debug.

(You can also work around it by enabling substitutes since Hydra had no
problems building it.)

Ludo’.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-14 11:40 ` Ludovic Courtès
@ 2015-11-14 12:47   ` Jan Synáček
  2015-11-20 13:51     ` Ludovic Courtès
  2015-11-20 13:44   ` Jan Synáček
  1 sibling, 1 reply; 15+ messages in thread
From: Jan Synáček @ 2015-11-14 12:47 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Sat, Nov 14, 2015 at 12:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Jan Synáček <jan.synacek@gmail.com> skribis:
>
>> Reference files will be read from:
>> /tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/tar/test
>> Running tests on:
>> "/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/./bsdtar"
>> Exercising: bsdtar 3.1.2 - libarchive 3.1.2
>
> [...]
>
>>  17: test_option_b                                     FAIL
>
> Ricardo reported the same issue a while back:
>
>   https://lists.gnu.org/archive/html/guix-devel/2015-03/msg00182.html
>
> What platform is this on, i686?
>
> It would be nice to see if this systematically fails.  If it is
> non-deterministic, we should build it with --keep-failed until it fails
> (removing successful builds with ‘guix gc -d’), collect useful info from
> the build tree, and debug.
>
> (You can also work around it by enabling substitutes since Hydra had no
> problems building it.)
>
> Ludo’.

In my case the build fails always. I'm running guix on Fedora 23, x86_64.

-- 
Jan Synáček

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-14 11:40 ` Ludovic Courtès
  2015-11-14 12:47   ` Jan Synáček
@ 2015-11-20 13:44   ` Jan Synáček
  2015-11-20 13:48     ` Jan Synáček
                       ` (2 more replies)
  1 sibling, 3 replies; 15+ messages in thread
From: Jan Synáček @ 2015-11-20 13:44 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Sat, Nov 14, 2015 at 12:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Jan Synáček <jan.synacek@gmail.com> skribis:
>
>> Reference files will be read from:
>> /tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/tar/test
>> Running tests on:
>> "/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/./bsdtar"
>> Exercising: bsdtar 3.1.2 - libarchive 3.1.2
>
> [...]
>
>>  17: test_option_b                                     FAIL
>
> Ricardo reported the same issue a while back:
>
>   https://lists.gnu.org/archive/html/guix-devel/2015-03/msg00182.html
>
> What platform is this on, i686?
>
> It would be nice to see if this systematically fails.  If it is
> non-deterministic, we should build it with --keep-failed until it fails
> (removing successful builds with ‘guix gc -d’), collect useful info from
> the build tree, and debug.
>
> (You can also work around it by enabling substitutes since Hydra had no
> problems building it.)
>
> Ludo’.

I debugged this and the upstream patch
https://github.com/libarchive/libarchive/commit/b539b2e597b566fe3c4b49cb61c9eef83e5e052d
fixes the problem. You will probably want to add it to the build process.

Cheers,
-- 
Jan Synáček

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-20 13:44   ` Jan Synáček
@ 2015-11-20 13:48     ` Jan Synáček
  2015-11-20 22:45       ` Ludovic Courtès
  2015-11-21 15:38     ` guix refresh -l Ludovic Courtès
  2015-12-09 22:55     ` Test failure when building libarchive-3.1.2 Ludovic Courtès
  2 siblings, 1 reply; 15+ messages in thread
From: Jan Synáček @ 2015-11-20 13:48 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Fri, Nov 20, 2015 at 2:44 PM, Jan Synáček <jan.synacek@gmail.com> wrote:
> On Sat, Nov 14, 2015 at 12:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Jan Synáček <jan.synacek@gmail.com> skribis:
>>
>>> Reference files will be read from:
>>> /tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/tar/test
>>> Running tests on:
>>> "/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/./bsdtar"
>>> Exercising: bsdtar 3.1.2 - libarchive 3.1.2
>>
>> [...]
>>
>>>  17: test_option_b                                     FAIL
>>
>> Ricardo reported the same issue a while back:
>>
>>   https://lists.gnu.org/archive/html/guix-devel/2015-03/msg00182.html
>>
>> What platform is this on, i686?
>>
>> It would be nice to see if this systematically fails.  If it is
>> non-deterministic, we should build it with --keep-failed until it fails
>> (removing successful builds with ‘guix gc -d’), collect useful info from
>> the build tree, and debug.
>>
>> (You can also work around it by enabling substitutes since Hydra had no
>> problems building it.)
>>
>> Ludo’.
>
> I debugged this and the upstream patch
> https://github.com/libarchive/libarchive/commit/b539b2e597b566fe3c4b49cb61c9eef83e5e052d
> fixes the problem. You will probably want to add it to the build process.

I just realized I need to do this locally as well to fix the
compilation...:) How do I modify the build process and add the patch
to the store?

Cheers,
-- 
Jan Synáček

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-14 12:47   ` Jan Synáček
@ 2015-11-20 13:51     ` Ludovic Courtès
  2015-11-20 14:10       ` Jan Synáček
  0 siblings, 1 reply; 15+ messages in thread
From: Ludovic Courtès @ 2015-11-20 13:51 UTC (permalink / raw)
  To: Jan Synáček; +Cc: guix-devel

Jan Synáček <jan.synacek@gmail.com> skribis:

> On Sat, Nov 14, 2015 at 12:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Jan Synáček <jan.synacek@gmail.com> skribis:
>>
>>> Reference files will be read from:
>>> /tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/tar/test
>>> Running tests on:
>>> "/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/./bsdtar"
>>> Exercising: bsdtar 3.1.2 - libarchive 3.1.2
>>
>> [...]
>>
>>>  17: test_option_b                                     FAIL
>>
>> Ricardo reported the same issue a while back:
>>
>>   https://lists.gnu.org/archive/html/guix-devel/2015-03/msg00182.html
>>
>> What platform is this on, i686?
>>
>> It would be nice to see if this systematically fails.  If it is
>> non-deterministic, we should build it with --keep-failed until it fails
>> (removing successful builds with ‘guix gc -d’), collect useful info from
>> the build tree, and debug.
>>
>> (You can also work around it by enabling substitutes since Hydra had no
>> problems building it.)
>>
>> Ludo’.
>
> In my case the build fails always. I'm running guix on Fedora 23, x86_64.

What file system is this on?

I’ve run several builds on my x86_64 GuixSD, ext4, but I’ve failed to
reproduce the test failure.

I noticed that libarchive uses ‘readdir’ calls as-is, without sorting
directory entries afterwards.  Thus, the order of directory entries is
effectively non-deterministic and may change depending on the phase of
the moon.

This has been reported at:

  https://github.com/libarchive/libarchive/issues/602

Could you add the patch that’s given at that URL to the ‘patches’ field
or libarchive’s ‘origin’ form and see if the problem shows up again,
preferably building several times in a row?

At any rate we’ll probably apply the patch.

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-20 13:51     ` Ludovic Courtès
@ 2015-11-20 14:10       ` Jan Synáček
  2015-11-20 22:47         ` Ludovic Courtès
  0 siblings, 1 reply; 15+ messages in thread
From: Jan Synáček @ 2015-11-20 14:10 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Fri, Nov 20, 2015 at 2:51 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Jan Synáček <jan.synacek@gmail.com> skribis:
>
>> On Sat, Nov 14, 2015 at 12:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>>> Jan Synáček <jan.synacek@gmail.com> skribis:
>>>
>>>> Reference files will be read from:
>>>> /tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/tar/test
>>>> Running tests on:
>>>> "/tmp/nix-build-libarchive-3.1.2.drv-0/libarchive-3.1.2/./bsdtar"
>>>> Exercising: bsdtar 3.1.2 - libarchive 3.1.2
>>>
>>> [...]
>>>
>>>>  17: test_option_b                                     FAIL
>>>
>>> Ricardo reported the same issue a while back:
>>>
>>>   https://lists.gnu.org/archive/html/guix-devel/2015-03/msg00182.html
>>>
>>> What platform is this on, i686?
>>>
>>> It would be nice to see if this systematically fails.  If it is
>>> non-deterministic, we should build it with --keep-failed until it fails
>>> (removing successful builds with ‘guix gc -d’), collect useful info from
>>> the build tree, and debug.
>>>
>>> (You can also work around it by enabling substitutes since Hydra had no
>>> problems building it.)
>>>
>>> Ludo’.
>>
>> In my case the build fails always. I'm running guix on Fedora 23, x86_64.
>
> What file system is this on?

The test itself seems to always run in /tmp, which is tmpfs in my case. I tried
patching it to run in /var/tmp, which is ext4 (on LVM), but that failed as well.

> I’ve run several builds on my x86_64 GuixSD, ext4, but I’ve failed to
> reproduce the test failure.

I could reproduce the issue even outside of the guix build process.
I downloaded the tarball, extracted it and ran configure + make + make check.
The latest git version worked fine, even though the asserts in test_option_b.c
didn't change.

> I noticed that libarchive uses ‘readdir’ calls as-is, without sorting
> directory entries afterwards.  Thus, the order of directory entries is
> effectively non-deterministic and may change depending on the phase of
> the moon.
>
> This has been reported at:
>
>   https://github.com/libarchive/libarchive/issues/602
>
> Could you add the patch that’s given at that URL to the ‘patches’ field
> or libarchive’s ‘origin’ form and see if the problem shows up again,
> preferably building several times in a row?

I built it once and it passed (note that it failed *everytime* I
wanted to build it).
Maybe a dumb question, but how do I force a rebuild of an already
built package?:)

> At any rate we’ll probably apply the patch.
>
> Thanks,
> Ludo’.

Cheers,
-- 
Jan Synáček

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-20 13:48     ` Jan Synáček
@ 2015-11-20 22:45       ` Ludovic Courtès
  0 siblings, 0 replies; 15+ messages in thread
From: Ludovic Courtès @ 2015-11-20 22:45 UTC (permalink / raw)
  To: Jan Synáček; +Cc: guix-devel

Jan Synáček <jan.synacek@gmail.com> skribis:

> On Fri, Nov 20, 2015 at 2:44 PM, Jan Synáček <jan.synacek@gmail.com> wrote:

[...]

>> I debugged this and the upstream patch
>> https://github.com/libarchive/libarchive/commit/b539b2e597b566fe3c4b49cb61c9eef83e5e052d
>> fixes the problem. You will probably want to add it to the build process.
>
> I just realized I need to do this locally as well to fix the
> compilation...:) How do I modify the build process and add the patch
> to the store?

Just list the patch in the ‘patches’ field of libarchive.  See for
instance ‘grep’ in gnu/packages/base.scm for an example.

Ludo’.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-20 14:10       ` Jan Synáček
@ 2015-11-20 22:47         ` Ludovic Courtès
  2015-11-21  7:58           ` Jan Synáček
  0 siblings, 1 reply; 15+ messages in thread
From: Ludovic Courtès @ 2015-11-20 22:47 UTC (permalink / raw)
  To: Jan Synáček; +Cc: guix-devel

Jan Synáček <jan.synacek@gmail.com> skribis:

> On Fri, Nov 20, 2015 at 2:51 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Jan Synáček <jan.synacek@gmail.com> skribis:
>>
>>> On Sat, Nov 14, 2015 at 12:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>>>> It would be nice to see if this systematically fails.  If it is
>>>> non-deterministic, we should build it with --keep-failed until it fails
>>>> (removing successful builds with ‘guix gc -d’), collect useful info from
>>>> the build tree, and debug.

[...]

>> I noticed that libarchive uses ‘readdir’ calls as-is, without sorting
>> directory entries afterwards.  Thus, the order of directory entries is
>> effectively non-deterministic and may change depending on the phase of
>> the moon.
>>
>> This has been reported at:
>>
>>   https://github.com/libarchive/libarchive/issues/602
>>
>> Could you add the patch that’s given at that URL to the ‘patches’ field
>> or libarchive’s ‘origin’ form and see if the problem shows up again,
>> preferably building several times in a row?
>
> I built it once and it passed (note that it failed *everytime* I
> wanted to build it).

So this patch appears to solve the issue?

> Maybe a dumb question, but how do I force a rebuild of an already
> built package?:)

You can’t rebuild packages because build processes are assumed to be
deterministic.  However, you can delete a build result with ‘gc -d’, as
mentioned above, and rebuild it afterwards.

HTH,
Ludo’.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-20 22:47         ` Ludovic Courtès
@ 2015-11-21  7:58           ` Jan Synáček
  2015-11-21 10:17             ` Ludovic Courtès
  0 siblings, 1 reply; 15+ messages in thread
From: Jan Synáček @ 2015-11-21  7:58 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

On Fri, Nov 20, 2015 at 11:47 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Jan Synáček <jan.synacek@gmail.com> skribis:
>
>> On Fri, Nov 20, 2015 at 2:51 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>>> Jan Synáček <jan.synacek@gmail.com> skribis:
>>>
>>>> On Sat, Nov 14, 2015 at 12:40 PM, Ludovic Courtès <ludo@gnu.org> wrote:
>
> [...]
>
>>>>> It would be nice to see if this systematically fails.  If it is
>>>>> non-deterministic, we should build it with --keep-failed until it fails
>>>>> (removing successful builds with ‘guix gc -d’), collect useful info from
>>>>> the build tree, and debug.
>
> [...]
>
>>> I noticed that libarchive uses ‘readdir’ calls as-is, without sorting
>>> directory entries afterwards.  Thus, the order of directory entries is
>>> effectively non-deterministic and may change depending on the phase of
>>> the moon.
>>>
>>> This has been reported at:
>>>
>>>   https://github.com/libarchive/libarchive/issues/602
>>>
>>> Could you add the patch that’s given at that URL to the ‘patches’ field
>>> or libarchive’s ‘origin’ form and see if the problem shows up again,
>>> preferably building several times in a row?
>>
>> I built it once and it passed (note that it failed *everytime* I
>> wanted to build it).
>
> So this patch appears to solve the issue?
>
>> Maybe a dumb question, but how do I force a rebuild of an already
>> built package?:)
>
> You can’t rebuild packages because build processes are assumed to be
> deterministic.  However, you can delete a build result with ‘gc -d’, as
> mentioned above, and rebuild it afterwards.

So I've built the package several times and it worked every time. I don't even
think the build was failing non-deterministically, because, as the
commit message
of the patch explains, the unpatched bsdtar was trying to put things
like SELinux
contexts into the archive, which resulted in an archive with bigger size than
anticipated. And since I'm running guix on Fedora, I probably hit the
SELinux problem.

Thank you for helping out!

Cheers,
-- 
Jan Synáček

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-21  7:58           ` Jan Synáček
@ 2015-11-21 10:17             ` Ludovic Courtès
  0 siblings, 0 replies; 15+ messages in thread
From: Ludovic Courtès @ 2015-11-21 10:17 UTC (permalink / raw)
  To: Jan Synáček; +Cc: guix-devel

Jan Synáček <jan.synacek@gmail.com> skribis:

> So I've built the package several times and it worked every time. I don't even
> think the build was failing non-deterministically, because, as the
> commit message
> of the patch explains, the unpatched bsdtar was trying to put things
> like SELinux
> contexts into the archive, which resulted in an archive with bigger size than
> anticipated. And since I'm running guix on Fedora, I probably hit the
> SELinux problem.

Oh, I see.  Thanks for investigating!

Now to apply the patch…

Ludo’.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* guix refresh -l
  2015-11-20 13:44   ` Jan Synáček
  2015-11-20 13:48     ` Jan Synáček
@ 2015-11-21 15:38     ` Ludovic Courtès
  2015-11-21 16:06       ` Mathieu Lirzin
  2015-11-21 20:31       ` Jan Synáček
  2015-12-09 22:55     ` Test failure when building libarchive-3.1.2 Ludovic Courtès
  2 siblings, 2 replies; 15+ messages in thread
From: Ludovic Courtès @ 2015-11-21 15:38 UTC (permalink / raw)
  To: Jan Synáček; +Cc: guix-devel, Eric Bavier

Jan Synáček <jan.synacek@gmail.com> skribis:

> I debugged this and the upstream patch
> https://github.com/libarchive/libarchive/commit/b539b2e597b566fe3c4b49cb61c9eef83e5e052d
> fixes the problem. You will probably want to add it to the build process.

To determine how to handle the mass rebuild, I wanted to see how many
packages depend on libarchive.  However, ‘guix refresh -l’ would
underestimate that number because it ignores implicit dependencies, and
it turns out that libarchive is a dependency of CMake, which in turn is
an implicit dependency of many things.

So I thought (guix scripts graph) has the necessary infrastructure to
manipulate these graphs, including using the “bag” representation, so
why not use it.

Hence, commit 8fb5837 adds a generic (guix graph), 923d846 adds a couple
of helper procedures, and a51cbec rewrites ‘guix refresh -l’ in terms of
(guix graph).

Before the change:

--8<---------------cut here---------------start------------->8---
$ time guix refresh -l libarchive
Building the following 11 packages would ensure 18 dependent packages are rebuilt: ath9k-htc-firmware-1.4.0 totem-3.16.1 rhythmbox-3.2.1 diffoscope-34 rdup-1.1.14 nestopia-ue-1.46.2 file-roller-3.10.0 gvfs-1.24.1 claws-mail-3.13.0 zathura-cb-0.1.4 python-libarchive-c-2.1

real	0m13.443s
user	0m14.556s
sys	0m0.064s
--8<---------------cut here---------------end--------------->8---

After:

--8<---------------cut here---------------start------------->8---
$ time ./pre-inst-env guix refresh -l libarchive
Building the following 279 packages would ensure 620 dependent packages are rebuilt: python-libarchive-c-2.1 i3status-2.9 libwebsockets-1.3 vtk-6.1.0 taskwarrior-2.4.3 synergy-1.7.4 guile-ssh-0.8.0 diffoscope-34 python-tlsh-3.4.1 libpano13-2.9.19 slepc-complex-3.6.2 slepc-3.6.2 plink-1.07 apl-1.5 mumps-5.0.1 mumps-metis-5.0.1 gmsh-2.8.4 dealii-8.2.1 julia-0.3.10 c-reduce-2.3.0 gcc-4.9.3 clang-3.5.0 unionfs-fuse-0.26 unionfs-fuse-static-0.26 pflask-0.2 avrdude-6.1 flashrom-0.9.7 oxygen-icons-4.14.2 r-servr-0.2 r-data.table-1.9.6 r-htmlwidgets-0.5 r-dplyr-0.4.3 r-devtools-1.9.1 python2-rpy2-2.6.0 python-rpy2-2.6.0 rsem-1.2.20 r-qtl-1.37-11 slepc-complex-openmpi-3.6.2 mumps-openmpi-5.0.1 superlu-dist-3.3 dealii-openmpi-8.2.1 bless-1p02 fftw-openmpi-3.3.4 guile-charting-0.2.0 accountsservice-0.6.40 zynaddsubfx-2.5.2 non-sequencer-1.9.5-1d9bd576 emacs-pdf-tools-0.60 mupdf-1.6 openimageio-1.5.18 ctl-1.5.2 slim-1.3.6 python2-scikit-image-0.11.3 raincat-1.1.1.3 guile-sly-0.1 mesa-utils-8.2.0 i3-wm-4.10.3 xnee-3.19 racket-6.2.1 sawfish-1.11 lxappearance-0.6.1 lxtask-0.1.6 pcmanfm-1.2.3 lxrandr-0.3.0 sra-tools-2.5.4 htsjdk-1.129 ruby-atoulme-antwrap-0.7.5 pspp-0.8.5 arandr-0.1.8 wicd-1.7.3 gourmet-0.17.4 gajim-0.16.3 patchage-1.0.0 gpscorrelate-1.6.1.365f6e1b3f pinentry-0.9.5 xournal-0.4.8 glade-3.8.4 lxterminal-0.2.0 gkrellm-2.3.5 geeqie-1.1 geda-gaf-1.8.2 dvdisaster-0.72.6 azr3-1.2.3 calf-0.0.60 ir-1.3.2 gnubik-2.4.2 pcb-20140316 gst-plugins-ugly-1.6.1 guix-0.9.0.5c36edc skribilo-0.9.2 a2ps-4.14 emacs-w3m-1.4.483+0.20120614 orpheus-1.6 ripperx-2.8.0 emms-4.0 abcde-2.7 mpd-mpc-0.27 mpdscribble-0.22 ncmpcpp-0.6.7 ncmpc-0.24 pidgin-otr-4.0.1 libstdc++-doc-5.2.0 libstdc++-doc-4.9.3 manaplus-1.5.10.24 wayland-1.9.0 fish-2.2.0 openbox-3.5.2 guile-present-0.3.0 amule-2.3.1 gmtp-1.3.9 tuxguitar-1.2 conkeror-1.0pre1.20150730 lablgtk-2.18.3 gnubg-1.02 inklingreader-0.8 gxmessage-3.4.3 pavucontrol-3.0 zathura-cb-0.1.4 zathura-ps-0.2.2 zathura-pdf-poppler-0.2.5 zathura-djvu-0.2.4 vte-0.36.5 tilda-1.3.0 gnome-keyring-3.16.0 gsegrafix-1.0.6 d-feet-0.3.10 seahorse-3.16.0 gnome-terminal-3.16.0 gedit-3.18.1 gjs-1.44.0 xfce-4.12.0 devhelp-3.16.1 yelp-3.16.1 shotwell-0.22.0 hexchat-2.10.1 claws-mail-3.13.0 ibus-libpinyin-1.7.2 network-manager-applet-1.0.6 file-roller-3.10.0 gnome-session-3.18.1.2 epiphany-3.16.3 nestopia-ue-1.46.2 gamine-1.4 sfxr-1.2.1 transmission-2.84 gnome-mines-3.16.0 aisleriot-3.16.1 gnucash-2.6.9 key-mon-1.17 eog-3.16.2 gnome-themes-standard-3.18.0 gnome-klotski-3.16.1 gnome-settings-daemon-3.16.0 gnumeric-1.12.17 xboard-4.8.0 fvwm-2.6.5 hop-2.4.0 patches-0.0.26d7dbc magit-svn-2.1.1 emacs-typo-1.1 emacs-flycheck-0.23 emacs-butler-0.2.4 emacs-auctex-11.88.6 emacs-debbugs-0.7 emacs-ob-ipython-20150704.8807064693 bigloo-really-sucks-4.2a.20141017 guile-emacs-20150512.41120e0 emacs-no-x-toolkit-24.5 abiword-2.8.6 emacs-foo-bar-patched-42 evince-3.18.1 rhythmbox-3.2.1 caribou-0.4.19 totem-3.16.1 libchamplain-0.12.10 mutter-3.18.1 gimp-2.8.14 wesnoth-1.12.4 gvfs-1.24.1 python2-ipython-3.2.1 python-numexpr-2.4.4 python-h5py-2.4.0 python-biopython-1.65 python-statsmodels-0.6.1 python-scikit-image-0.11.3 python-scikit-learn-0.16.1 python-seaborn-0.5.1 idr-2.0.0 python-ipython-3.2.1 python2-numexpr-2.4.4 macs-2.1.0.20140616 seqmagick-0.6.1 crossmap-0.2.1 rseqc-2.6.1 python2-statsmodels-0.6.1 miso-0.5.3 deeptools-1.5.11 python2-warpedlmm-0.21 clipper-0.3.0 pbtranscript-tofu-2.2.3.8f5467fe6 grit-2.0.2 couger-1.8.2 proof-general-4.2 enblend-enfuse-4.1.3 shogun-4.0.0 xsensors-0.70 wxmaxima-15.04.0 fastcap-2.0-18Sep92 simple-scan-3.17.4 hydra-20150407.4c0e3e4 gerbv-2.6.1 dunst-1.1.0 evas-generic-loaders-1.16.0 terminology-0.9.1 rage-0.1.4 enlightenment-0.19.13 ath9k-htc-firmware-1.4.0 fcitx-4.2.8.6 soprano-2.9.4 python2-pyqt-4.11.4 polkit-qt-1-0.112.0 frescobaldi-2.18.1 keepassx-2.0-beta2 hydrogen-0.9.5.1 phonon-4.8.3 attica-0.4.2 libdbusmenu-qt-0.9.2 brdf-explorer-17 gpsbabel-1.5.0 librecad-2.0.6-rc alsa-modular-synth-2.1.1 jalv-1.4.6 ardour-4.4 avidemux-2.6.10 scribus-1.5.0 python-pyqt-5.5 pumpa-0.9.1 owncloud-client-2.0.2 lxqt-common-0.9.1 lxqt-session-0.9.0 tiled-0.13.1 bitcoin-core-0.11.0 calibre-2.41.0 qsynth-0.4.0 bind-utils-9.10.2-P2 mariadb-10.0.20 conky-1.10.0 powertabeditor-2.0.0-alpha8 libreoffice-5.0.0.2 evolution-data-server-3.18.2 flexbar-2.5 express-1.5.1 solfege-3.22.2 wine-1.7.52 mplayer-1.2 emotion-generic-players-1.16.0 cmus-2.7.1 mpd-0.19.10 strigi-0.7.8 gst-libav-1.6.1 guile-gnunet-0.0.383eac2 audacity-2.1.0 synfigstudio-1.0.2 retroarch-1.2.2 minetest-0.4.13 mars-0.7.5.1.c855d04409 love-0.9.2 mpv-0.13.0 emulation-station-2.0.1 guitarix-0.34.0 alta-minimal-0.0 alta-clang-0.0 dfc-3.0.4 alta-0.0 alta-gcc5-0.0 alta-gcc4.8-0.0 alta-gcc4.7-0.0 libwebsockets-1.3.dbg rdup-1.1.14

real	0m0.979s
user	0m1.152s
sys	0m0.016s
--8<---------------cut here---------------end--------------->8---

Eric, I would welcome your feedback on this.  I would think the graph
helpers in (gnu packages) that were used by ‘guix refresh -l’ can now be
removed; WDYT?

There’s still room for improvement: ‘guix refresh -l’ could look at the
graph that includes origin objects and count the number of dependents of
the origin, rather than the number of dependents of one package that
uses it.  (Well, that’s the information we’re after when we want to
upgrade, but when we just want to change the package, that’s not what we
want.)

Cheers,
Ludo’.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: guix refresh -l
  2015-11-21 15:38     ` guix refresh -l Ludovic Courtès
@ 2015-11-21 16:06       ` Mathieu Lirzin
  2015-11-21 20:31       ` Jan Synáček
  1 sibling, 0 replies; 15+ messages in thread
From: Mathieu Lirzin @ 2015-11-21 16:06 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel, Eric Bavier

ludo@gnu.org (Ludovic Courtès) writes:

> Hence, commit 8fb5837 adds a generic (guix graph), 923d846 adds a couple
> of helper procedures, and a51cbec rewrites ‘guix refresh -l’ in terms of
> (guix graph).

This looks really nice!

--
Mathieu Lirzin

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: guix refresh -l
  2015-11-21 15:38     ` guix refresh -l Ludovic Courtès
  2015-11-21 16:06       ` Mathieu Lirzin
@ 2015-11-21 20:31       ` Jan Synáček
  1 sibling, 0 replies; 15+ messages in thread
From: Jan Synáček @ 2015-11-21 20:31 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel, Eric Bavier

On Sat, Nov 21, 2015 at 4:38 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Building the following 279 packages would ensure 620 dependent packages are rebuilt: [...]
> [...]
> real    0m0.979s
> user    0m1.152s
> sys     0m0.016s

I'm running guix-daemon with --no-substitutes, but didn't realize that
libarchive actually had so many dependent packages when I was building
it:)

Anyway, the new version looks great!

Cheers,
-- 
Jan Synáček

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: Test failure when building libarchive-3.1.2
  2015-11-20 13:44   ` Jan Synáček
  2015-11-20 13:48     ` Jan Synáček
  2015-11-21 15:38     ` guix refresh -l Ludovic Courtès
@ 2015-12-09 22:55     ` Ludovic Courtès
  2 siblings, 0 replies; 15+ messages in thread
From: Ludovic Courtès @ 2015-12-09 22:55 UTC (permalink / raw)
  To: Jan Synáček; +Cc: guix-devel

Jan Synáček <jan.synacek@gmail.com> skribis:

> I debugged this and the upstream patch
> https://github.com/libarchive/libarchive/commit/b539b2e597b566fe3c4b49cb61c9eef83e5e052d
> fixes the problem. You will probably want to add it to the build process.

I’ve finally pushed it to ‘core-updates’.  Thanks!

Ludo’.

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2015-12-09 22:55 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-13 11:33 Test failure when building libarchive-3.1.2 Jan Synáček
2015-11-14 11:40 ` Ludovic Courtès
2015-11-14 12:47   ` Jan Synáček
2015-11-20 13:51     ` Ludovic Courtès
2015-11-20 14:10       ` Jan Synáček
2015-11-20 22:47         ` Ludovic Courtès
2015-11-21  7:58           ` Jan Synáček
2015-11-21 10:17             ` Ludovic Courtès
2015-11-20 13:44   ` Jan Synáček
2015-11-20 13:48     ` Jan Synáček
2015-11-20 22:45       ` Ludovic Courtès
2015-11-21 15:38     ` guix refresh -l Ludovic Courtès
2015-11-21 16:06       ` Mathieu Lirzin
2015-11-21 20:31       ` Jan Synáček
2015-12-09 22:55     ` Test failure when building libarchive-3.1.2 Ludovic Courtès

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).