Thursday 18th: ‘core-updates-frozen’ sprint!

Thursday 18th: ‘core-updates-frozen’ sprint!

[Ludovic Courtès]

Hello Guix!

I hereby declare tomorrow, Thursday Nov. 18th, day of the on-line
‘core-updates-frozen’ sprint!

The idea is that you join #guix on IRC, you try to build your system or
profile from ‘core-updates-frozen’, and you report (and fix!) any issues
you may have, in the warmth of a collective of friendly human beings all
sharing the same goal and helping each other.

Nov. 23th will mark Guix’s 9th birthday.  Wouldn’t it be great to have
that branch finally merged in ‘master’ by then?

See you on Thursday!  :-)

Ludo’.

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> I hereby declare tomorrow, Thursday Nov. 18th, day of the 
> on-line
> ‘core-updates-frozen’ sprint!
>
> The idea is that you join #guix on IRC, you try to build your 
> system or
> profile from ‘core-updates-frozen’, and you report (and fix!) 
> any issues
> you may have, in the warmth of a collective of friendly human 
> beings all
> sharing the same goal and helping each other.

I’ll be there!

To give everyone a head start: we have some annoying errors:

1) 
/gnu/store/aw65rwl2c50ckwghafv0p98hpg4wqvxm-vigra-1.11.1-0.9b514fa.drv 
has a bunch of test failures; this blocks libreoffice.  Looks like 
this and was caused by the most recent boost upgrade:

--8<---------------cut here---------------start------------->8---
Boost.Python.ArgumentError: Python argument types in
    vigra.vigranumpycore.constructArrayFromAxistags(type, tuple, 
    numpy.dtype[float32], AxisTags, bool)
did not match C++ signature:
    constructArrayFromAxistags(boost::python::api::object, 
    vigra::ArrayVector<long, std::allocator<long> >, NPY_TYPES, 
    vigra::AxisTags, bool)
--8<---------------cut here---------------end--------------->8---


2) python-notebook has failing tests.  Same for a more recent 
version (6.4.3).  The failing tests *all* relate to deleting files 
with send2trash.  Upgrading send2trash does not help.  These tests 
also fail outside the build container environment.  The errors 
look like this:

--8<---------------cut here---------------start------------->8---
notebook-6.4.3/notebook/services/contents/manager.py:279: in 
delete
    self.delete_file(path)
notebook-6.4.3/notebook/services/contents/filemanager.py:533: in 
delete_file
    send2trash(os_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

path = 
'/tmp/guix-build-python-notebook-6.4.3.drv-0/tmpkwgdpuk_/Untitled.ipynb'

    def send2trash(path):
        try:
            f = Gio.File.new_for_path(path)
            f.trash(cancellable=None)
        except GObject.GError as e:
            if e.code == Gio.IOErrorEnum.NOT_SUPPORTED:
                # We get here if we can't create a trash directory 
                on the same
                # device. I don't know if other errors can result 
                in NOT_SUPPORTED.
                raise TrashPermissionError('')
>           raise OSError(e.message)
E           OSError: Error trashing file 
/tmp/guix-build-python-notebook-6.4.3.drv-0/tmpkwgdpuk_/Untitled.ipynb: 
No such file or directory
--8<---------------cut here---------------end--------------->8---


3) Building Gnome systems fails because of a package conflict. 
Commit 781f475bbac4e73848f68cb9f420a7283ec17c16 is the direct 
cause; different variants of at-spi2-atk end up in the same 
profile.


> Nov. 23th will mark Guix’s 9th birthday.  Wouldn’t it be great 
> to have
> that branch finally merged in ‘master’ by then?

Yes!

-- 
Ricardo

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[Ludovic Courtès]

Hi there!

Ludovic Courtès <ludo@gnu.org> skribis:

> I hereby declare tomorrow, Thursday Nov. 18th, day of the on-line
> ‘core-updates-frozen’ sprint!

I’d like to congratulate ourselves and call it a success!  A dedicated
group worked all day long anywhere-on-earth and pushed 79 commits fixing
a variety of things—thumbs up!

We’re almost done but we have yet to get past the finish line.

Please give it a spin either by testing your system:

  sudo guix time-machine --branch=core-updates-frozen -- \
    system reconfigure /…/config.scm

or by upgrading your profile:

  guix time-machine --branch=core-updates-frozen -- upgrade

Expect rebuilds as things in the GNOME/Freedesktop stack are still
receiving fixes, or wait until ci.guix has caught up.

The issue Ricardo mentioned about ‘python-jupyter’ has yet to be fixed
too, and there’s probably more.

If you email bug-guix or guix-patches with “core-updates-frozen” in the
subject and a clear description of what the problem/fix is, your
friendly committer collective will be sure to fast-track your patch
(right? :-)).

> Nov. 23th will mark Guix’s 9th birthday.  Wouldn’t it be great to have
> that branch finally merged in ‘master’ by then?

Sounds doable, let’s keep up the good work!

Ludo’.

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> … there’s probably more.

I noticed that we’re still using PYTHONPATH in a bunch of places, 
even though we should now be using GUIX_PYTHONPATH.  I’ve been 
slowly hacking away at the remaining offenders.

That said, it’s amazing how much progress we made in so little 
time!  I’ll continue to prioritize core-updates-frozen for the 
next few days.

>> Nov. 23th will mark Guix’s 9th birthday.  Wouldn’t it be great 
>> to have
>> that branch finally merged in ‘master’ by then?
>
> Sounds doable, let’s keep up the good work!

I hope we can make it!

-- 
Ricardo

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[Ludovic Courtès]

Hi!

Ricardo Wurmus <rekado@elephly.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> … there’s probably more.
>
> I noticed that we’re still using PYTHONPATH in a bunch of places, even
> though we should now be using GUIX_PYTHONPATH.  I’ve been slowly
> hacking away at the remaining offenders.

I see you fixed a bunch of them, yay!

> That said, it’s amazing how much progress we made in so little time!
> I’ll continue to prioritize core-updates-frozen for the next few days.

Yes, let’s do that.  I’ve fixed things I’ll probably never ever use,
like trytond.  A call to everyone though: make sure your favorite
packages work because you can’t necessarily count on others to test it
and fix it.  :-)

I’ve noticed Python packages still failing so if you rely on Python,
better be safe than sorry.

>>> Nov. 23th will mark Guix’s 9th birthday.  Wouldn’t it be great to
>>> have
>>> that branch finally merged in ‘master’ by then?
>>
>> Sounds doable, let’s keep up the good work!
>
> I hope we can make it!

Yup, let’s see!

Ludo’.

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[zimoun]

Hi Ludo,

On Mon, 22 Nov 2021 at 14:21, Ludovic Courtès <ludo@gnu.org> wrote:

> >> … there’s probably more.

Julia is annoying,  Because the test suite sometimes passes, sometimes
not.  Well, I am not even sure it is the same part that fails.  I
notice that "guix build julia --without-tests=julia" then "guix build
julia" tends to works. :-)

Any idea to debug this?


Cheers,
simon

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[Maxim Cournoyer]

Hello,

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Ludo,
>
> On Mon, 22 Nov 2021 at 14:21, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> >> … there’s probably more.
>
> Julia is annoying,  Because the test suite sometimes passes, sometimes
> not.  Well, I am not even sure it is the same part that fails.  I
> notice that "guix build julia --without-tests=julia" then "guix build
> julia" tends to works. :-)
>
> Any idea to debug this?

No idea how to debug this, but when faced with nondeterministic tests,
here's how I proceed:

1. Report them upstream.
2. Disable the problematic tests.  You may need to disable a whole test suite of
similarly faulted tests, if they all behave the same.

Thanks,

Maxim

Re: Thursday 18th: ‘core-updates-frozen’ sprint!

[zimoun]

Hi,

On Mon, 22 Nov 2021 at 15:13, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> 2. Disable the problematic tests.  You may need to disable a whole test suite of
> similarly faulted tests, if they all behave the same.

I am becoming crazy…  For instance this test randomly* fails:

    @test mul!(C, vf, transpose(vf), 2, 3) == 2vf*vf' .+ 3C0

<https://github.com/JuliaLang/julia/blob/v1.6.3/stdlib/LinearAlgebra/test/matmul.jl#L155>

Therefore, I add,

--8<---------------cut here---------------start------------->8---
            (substitute* "stdlib/LinearAlgebra/test/matmul.jl" 
              (("@test mul\\!\\(C, vf, transpose\\(vf\\), 2, 3\\) == 2vf\\*vf' \\.+ 3C0") 
               "@test_broken @test mul!(C, vf, transpose(vf), 2, 3) == 2vf*vf' .+ 3C0")
            (substitute* "test/math.jl" 
              (("@test isinf\\(log1p\\(-one\\(T\\)\\)\\)")
               "@test_broken isinf(log1p(-one(T)))"))))
--8<---------------cut here---------------end--------------->8---

So far, so good.  Then I get:

--8<---------------cut here---------------start------------->8---
Error During Test at /tmp/guix-build-julia-1.6.3.drv-0/julia-1.6.3/test/math.jl:278
 Unexpected Pass     
Expression: isinf(log1p(-(one(T))))
 Got correct result, please change to @test if no longer broken.
--8<---------------cut here---------------end--------------->8---

and thus the build is reported as failed.  Argh!!  Ok, because these 2
tests are random, I replace the snippet above by:

--8<---------------cut here---------------start------------->8---
            (substitute* "stdlib/LinearAlgebra/test/matmul.jl" 
              (("@test mul\\!\\(C, vf, transpose\\(vf\\), 2, 3\\) == 2vf\\*vf' \\.+ 3C0") 
               "")
            (substitute* "test/math.jl" 
              (("@test isinf\\(log1p\\(-one\\(T\\)\\)\\)")
               ""))))
--8<---------------cut here---------------end--------------->8---

And now, something magical happens, I get this error with parenthesis:

--8<---------------cut here---------------start------------->8---
Test Failed at /tmp/guix-build-julia-1.6.3.drv-0/julia-1.6.3/usr/share/julia/stdlib/v1.6/LinearAlgebra/test/matmul.jl:155
Expression: mul!(C, vf, transpose(vf), 2, 3) == (2vf) * vf' .+ 3C0
--8<---------------cut here---------------end--------------->8---

What?!  And AFAIU, it is not in the test suite.

Do I miss something with ’substitute*’?


Cheers,
simon

*randomly: because I do not know why, nothing is random with
 computer. ;-)

PS: Note that current Julia master contains this test:

        @test mul!(C, vf, transpose(vf), 2, 3) ≈ 2vf*vf' .+ 3C0

https://github.com/JuliaLang/julia/blob/master/stdlib/LinearAlgebra/test/matmul.jl#L155

I am trying to ’substitute*’ with that.  Then I give up for now.

‘core-updates-frozen’: julia status

[zimoun]

Hi,

On Mon, 22 Nov 2021 at 15:13, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>> Julia is annoying,  Because the test suite sometimes passes, sometimes
>> not.  Well, I am not even sure it is the same part that fails.  I
>> notice that "guix build julia --without-tests=julia" then "guix build
>> julia" tends to works. :-)
>>
>> Any idea to debug this?
>
> No idea how to debug this, but when faced with nondeterministic tests,
> here's how I proceed:
>
> 1. Report them upstream.
> 2. Disable the problematic tests.  You may need to disable a whole test suite of
> similarly faulted tests, if they all behave the same.

After this hacking day, Julia builds deterministically* on
core-updates-frozen.  Along the way, now the test suite runs in parallel
which reduces a lot the feedback loop. :-) And a variant of the current
patch of julia source is upstream PR.

*deterministically: at least on 2 machines. ;-)

One or two failing tests randomly failing should be investigated and
probably reported upstream.

The test suite of julia-* packages supports also (more or less) some
parallelism.  From my understanding, the limitation comes from the
script 'test/runtests.jl' from each package; each implements more or
less some parallelism.  Now, the environment variable JULIA_CPU_THREADS
is set but Julia does not always honor it as its doc specifies – see the
patch fixing the parallelism of Julia test suite ;-).  Therefore, the
test suite of each package is launched with
'--procs=(parallel-job-count)' and Julia exploits at best the available
workers.

I am currently rebuilding all the julia-* packages.  It is high probable
some are now broken.  But it appears to me not a blocker for the merge.

On a side note, Julia is not currently reproducible [1] and
'julia-build-system' neither [2].  Therefore, probable non-reproducible
build introduced by parallelism is currently not the point, IMHO.



Thanks Maxim for the help and the good vibes to tackle that.

Cheers,
simon

1: <http://issues.guix.gnu.org/issue/22304>
2: <http://issues.guix.gnu.org/issue/47354>

Re: ‘core-updates-frozen’: julia status update

[zimoun]

Hi,

On Thu, 25 Nov 2021 at 03:13, zimoun <zimon.toutoune@gmail.com> wrote:

> I am currently rebuilding all the julia-* packages.  It is high probable
> some are now broken.  But it appears to me not a blocker for the
> merge.

All is fixed in patch#52117 [1].  After this hacking session,

 1. The package julia uses parallelism to build, i.e., it requires much
    much less time to build; something like 5x speedup on Berlin.

 2. The julia-build-system runs the tests with some degree of
    parallelism.  The speedup depends on each ’test/runtests.jl’ script
    provided by the package julia-* itself.

The strong issues to tackle are:

 a) bug#22304: build julia package with reproducible
 b) bug#47354: build julia-* packages reproducible


1: <https://issues.guix.gnu.org/52117>
2: <http://issues.guix.gnu.org/issue/22304>
3: <http://issues.guix.gnu.org/issue/47354>


Cheers,
simon