Status update on reproducible builds in Guix

Status update on reproducible builds in Guix

[Ludovic Courtès]

Hello Guix!

Here’s an update on reproducibility in Guix:

  https://www.gnu.org/software/guix/news/reproducible-builds-a-status-update.html

Happy hacking!  :-)

Ludo’.

Re: Status update on reproducible builds in Guix

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> Here’s an update on reproducibility in Guix:
>
>   https://www.gnu.org/software/guix/news/reproducible-builds-a-status-update.html

At least 78% to possibly 91% reproduciblility of packages is not bad.

Is there a (small) core that is already 100% reprocucible, like the
installation binaries/USB installer, bare-bones.tmpl or even
lightweight-desktop.tmpl?

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Status update on reproducible builds in Guix

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
>> Here’s an update on reproducibility in Guix:
>>
>>   https://www.gnu.org/software/guix/news/reproducible-builds-a-status-update.html
>
> At least 78% to possibly 91% reproduciblility of packages is not bad.
>
> Is there a (small) core that is already 100% reprocucible, like the
> installation binaries/USB installer, bare-bones.tmpl or even
> lightweight-desktop.tmpl?

Good question!  I think as soon as you have Python .pyc files in the
dependency graph (reference graph), there are non-reproducible bits.  We
certainly have Python stuff in the base system reference graph, so
that’s one thing we should fix most urgently I guess.

Ludo’.

Re: Status update on reproducible builds in Guix

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1733 bytes --]

Yesteday we had a discussion about that on irc.
Here it goes:


[15:15:16] <g_bor> hello guix!
[15:16:01] <g_bor> do we have a proposed way to build pyc files
reproducibly?
[15:16:50] <g_bor> I've read in the report, that we are not there yet, but
is someone working on it?
[15:17:58] <lfam> g_bor: This is the report you mention? <
https://bugs.gnu.org/22533>
[15:18:10] <lfam> I'm not sure if anyone has been working on it since the
last message
[15:20:26] * Guest74 has joined #guix
[15:23:05] <g_bor> thx, just what i was looking for.
[15:23:26] <lfam> It's possible that some work in this area is pending on
the 'core-updates' Git branch, but I'm not sure
[15:37:41] <mb[m]1> At this stage we might as well wait for this to land
upstream: https://www.python.org/dev/peps/pep-0552/

So, it seems, that we are waiting for this pep to land upstream.


2017-11-05 16:49 GMT+01:00 Ludovic Courtès <ludo@gnu.org>:

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
> > Ludovic Courtès writes:
> >
> >> Here’s an update on reproducibility in Guix:
> >>
> >>   https://www.gnu.org/software/guix/news/reproducible-builds-
> a-status-update.html
> >
> > At least 78% to possibly 91% reproduciblility of packages is not bad.
> >
> > Is there a (small) core that is already 100% reprocucible, like the
> > installation binaries/USB installer, bare-bones.tmpl or even
> > lightweight-desktop.tmpl?
>
> Good question!  I think as soon as you have Python .pyc files in the
> dependency graph (reference graph), there are non-reproducible bits.  We
> certainly have Python stuff in the base system reference graph, so
> that’s one thing we should fix most urgently I guess.
>
> Ludo’.
>
>

[-- Attachment #2: Type: text/html, Size: 3322 bytes --]

Re: Status update on reproducible builds in Guix

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

Am 05.11.2017 um 20:04 schrieb Gábor Boskovits:
> [15:37:41]<mb[m]1>At this stage we might as well wait for this to land
> upstream: https://www.python.org/dev/peps/pep-0552/

Bad news: PEP 552 proposes a new file-format for .pyc file, so this
change can not be back-ported to Python 3.6 and older.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1264 bytes --]

Re: Status update on reproducible builds in Guix

[ng0]

[-- Attachment #1: Type: text/plain, Size: 2292 bytes --]

Gábor Boskovits transcribed 5.3K bytes:
> Yesteday we had a discussion about that on irc.
> Here it goes:
> 
> 
> [15:15:16] <g_bor> hello guix!
> [15:16:01] <g_bor> do we have a proposed way to build pyc files
> reproducibly?
> [15:16:50] <g_bor> I've read in the report, that we are not there yet, but
> is someone working on it?
> [15:17:58] <lfam> g_bor: This is the report you mention? <
> https://bugs.gnu.org/22533>
> [15:18:10] <lfam> I'm not sure if anyone has been working on it since the
> last message
> [15:20:26] * Guest74 has joined #guix
> [15:23:05] <g_bor> thx, just what i was looking for.
> [15:23:26] <lfam> It's possible that some work in this area is pending on
> the 'core-updates' Git branch, but I'm not sure
> [15:37:41] <mb[m]1> At this stage we might as well wait for this to land
> upstream: https://www.python.org/dev/peps/pep-0552/
> 
> So, it seems, that we are waiting for this pep to land upstream.

PEP Status: Accepted.
I'm not familar with PEPs, other than that their sometimes very
long discussions on the mailinglists. Does this mean that the
missing pieces now are:

implementation
inclusion of this implementation in a new python release

> 2017-11-05 16:49 GMT+01:00 Ludovic Courtès <ludo@gnu.org>:
> 
> > Jan Nieuwenhuizen <janneke@gnu.org> skribis:
> >
> > > Ludovic Courtès writes:
> > >
> > >> Here’s an update on reproducibility in Guix:
> > >>
> > >>   https://www.gnu.org/software/guix/news/reproducible-builds-
> > a-status-update.html
> > >
> > > At least 78% to possibly 91% reproduciblility of packages is not bad.
> > >
> > > Is there a (small) core that is already 100% reprocucible, like the
> > > installation binaries/USB installer, bare-bones.tmpl or even
> > > lightweight-desktop.tmpl?
> >
> > Good question!  I think as soon as you have Python .pyc files in the
> > dependency graph (reference graph), there are non-reproducible bits.  We
> > certainly have Python stuff in the base system reference graph, so
> > that’s one thing we should fix most urgently I guess.
> >
> > Ludo’.
> >
> >

-- 
ng0
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://dist.ng0.infotropique.org/dist/keys/
https://www.infotropique.org https://ng0.infotropique.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Status update on reproducible builds in Guix

[Ludovic Courtès]

Hello,

Gábor Boskovits <boskovits@gmail.com> skribis:

> Yesteday we had a discussion about that on irc.
> Here it goes:
>
>
> [15:15:16] <g_bor> hello guix!
> [15:16:01] <g_bor> do we have a proposed way to build pyc files
> reproducibly?
> [15:16:50] <g_bor> I've read in the report, that we are not there yet, but
> is someone working on it?
> [15:17:58] <lfam> g_bor: This is the report you mention? <
> https://bugs.gnu.org/22533>
> [15:18:10] <lfam> I'm not sure if anyone has been working on it since the
> last message
> [15:20:26] * Guest74 has joined #guix
> [15:23:05] <g_bor> thx, just what i was looking for.
> [15:23:26] <lfam> It's possible that some work in this area is pending on
> the 'core-updates' Git branch, but I'm not sure
> [15:37:41] <mb[m]1> At this stage we might as well wait for this to land
> upstream: https://www.python.org/dev/peps/pep-0552/
>
> So, it seems, that we are waiting for this pep to land upstream.

Cool, thanks for the notification.

Now, I’d be in favor of incorporating a patch in the meantime, rather
than wait for the PEP to materialize.

I can’t find Debian’s patches in
<https://alioth.debian.org/scm/loggerhead/pkg-python/python3-defaults-debian/files/head:/debian/>,
but they probably have something.

Ludo’.

Re: Status update on reproducible builds in Guix

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

Am 06.11.2017 um 09:52 schrieb Ludovic Courtès:
>> [15:37:41] <mb[m]1> At this stage we might as well wait for this to land
>> upstream: https://www.python.org/dev/peps/pep-0552/
>>
>> So, it seems, that we are waiting for this pep to land upstream.
> Cool, thanks for the notification.

Seems like my message did not go through:

This change will not solve out problem!

PEP 552 proposes a new file-format for .pyc file and a hash-based
mechanism for checking if the .pyc file is recent. This means, you can
not backport this changes to Python 3.6 or older. Even if you manage to
backport, this would seriously break all tools working on .pyc files.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1656 bytes --]

Re: Status update on reproducible builds in Guix

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]

Ok, I think we should try the patch the bytecode complier way.
WDYT?

Rekado mentioned that setting the times would be an easier way to go, but
breaks some tests...
I guess they were also discussing options on irc.
You are right, backporting does not seem to be a good option here.

Regarding message of ng0:
Debian does not ship pyc files, they compile  them at install time, so this
reproducibility issue does not affect them.

2017-11-06 11:19 GMT+01:00 Hartmut Goebel <h.goebel@crazy-compilers.com>:

> Am 06.11.2017 um 09:52 schrieb Ludovic Courtès:
>
> [15:37:41] <mb[m]1> At this stage we might as well wait for this to land
> upstream: https://www.python.org/dev/peps/pep-0552/
>
> So, it seems, that we are waiting for this pep to land upstream.
>
> Cool, thanks for the notification.
>
> Seems like my message did not go through:
>
> This change will not solve out problem!
>
> PEP 552 proposes a new file-format for .pyc file and a hash-based
> mechanism for checking if the .pyc file is recent. This means, you can not
> backport this changes to Python 3.6 or older. Even if you manage to
> backport, this would seriously break all tools working on .pyc files.
>
> --
> Regards
> Hartmut Goebel
>
> | Hartmut Goebel          | h.goebel@crazy-compilers.com               |
> | www.crazy-compilers.com | compilers which you thought are impossible |
>
>

[-- Attachment #2: Type: text/html, Size: 2570 bytes --]

Re: Status update on reproducible builds in Guix

[Eric Myhre]

iirc there is even an env var you can set -- PYTHONDONTWRITEBYTECODE=anyvalue -- which will prevent these .pyc files from ever being generated.

There are no ill effects to this I have ever noticed in several years of having it set in my bashrc.

On 6 November 2017 12:47:03 CET, "Gábor Boskovits" <boskovits@gmail.com> wrote:
>
>Regarding message of ng0:
>Debian does not ship pyc files, they compile  them at install time, so
>this
>reproducibility issue does not affect them.
>
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Status update on reproducible builds in Guix

[Hartmut Goebel]

Am 07.11.2017 um 10:55 schrieb Eric Myhre:
> iirc there is even an env var you can set -- PYTHONDONTWRITEBYTECODE=anyvalue -- which will prevent these .pyc files from ever being generated.
>
> There are no ill effects to this I have ever noticed in several years of having it set in my bashrc.

This env-var disables byt-code caching (.pyc files), thus every module
needs to be recompiled every time is is used.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Status update on reproducible builds in Guix

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 06.11.2017 um 09:52 schrieb Ludovic Courtès:
>>> [15:37:41] <mb[m]1> At this stage we might as well wait for this to land
>>> upstream: https://www.python.org/dev/peps/pep-0552/
>>>
>>> So, it seems, that we are waiting for this pep to land upstream.
>> Cool, thanks for the notification.
>
> Seems like my message did not go through:
>
> This change will not solve out problem!
>
> PEP 552 proposes a new file-format for .pyc file and a hash-based
> mechanism for checking if the .pyc file is recent. This means, you can
> not backport this changes to Python 3.6 or older. Even if you manage to
> backport, this would seriously break all tools working on .pyc files.

We only support Python 2 and the latest (in Guix, anyway!) Python 3
interpreter.  So while we will indeed need a solution for Python2, we
don't have to care about older 3.x variants.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Status update on reproducible builds in Guix

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

Am 06.11.2017 um 22:29 schrieb Marius Bakke:
>> Seems like my message did not go through:
>>
>> This change will not solve out problem!
>>
>> PEP 552 proposes a new file-format for .pyc file and a hash-based
>> mechanism for checking if the .pyc file is recent. This means, you can
>> not backport this changes to Python 3.6 or older. Even if you manage to
>> backport, this would seriously break all tools working on .pyc files.
> We only support Python 2 and the latest (in Guix, anyway!) Python 3
> interpreter.  So while we will indeed need a solution for Python2, we
> don't have to care about older 3.x variants.

This depends on what you call "older 3.5 variants". :-)

The current version is Python 3.6. Guix has 3.5 (and 2.7).

Python 3.7 (which will materialize PEP 552 ) is planed for mid of 2018.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 1695 bytes --]

Re: Status update on reproducible builds in Guix

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 06.11.2017 um 22:29 schrieb Marius Bakke:
>>> Seems like my message did not go through:
>>>
>>> This change will not solve out problem!
>>>
>>> PEP 552 proposes a new file-format for .pyc file and a hash-based
>>> mechanism for checking if the .pyc file is recent. This means, you can
>>> not backport this changes to Python 3.6 or older. Even if you manage to
>>> backport, this would seriously break all tools working on .pyc files.
>> We only support Python 2 and the latest (in Guix, anyway!) Python 3
>> interpreter.  So while we will indeed need a solution for Python2, we
>> don't have to care about older 3.x variants.
>
> This depends on what you call "older 3.5 variants". :-)
>
> The current version is Python 3.6. Guix has 3.5 (and 2.7).
>
> Python 3.7 (which will materialize PEP 552 ) is planed for mid of 2018.

Ah right, I thought 3.7 was much closer.  Then it makes a lot of sense
to pick the reproducibility patch for 3.6 (which is in 'core-updates').

Any volunteers? :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Status update on reproducible builds in Guix

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1939 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Gábor Boskovits <boskovits@gmail.com> skribis:
>
>> Yesteday we had a discussion about that on irc.
>> Here it goes:
>>
>>
>> [15:15:16] <g_bor> hello guix!
>> [15:16:01] <g_bor> do we have a proposed way to build pyc files
>> reproducibly?
>> [15:16:50] <g_bor> I've read in the report, that we are not there yet, but
>> is someone working on it?
>> [15:17:58] <lfam> g_bor: This is the report you mention? <
>> https://bugs.gnu.org/22533>
>> [15:18:10] <lfam> I'm not sure if anyone has been working on it since the
>> last message
>> [15:20:26] * Guest74 has joined #guix
>> [15:23:05] <g_bor> thx, just what i was looking for.
>> [15:23:26] <lfam> It's possible that some work in this area is pending on
>> the 'core-updates' Git branch, but I'm not sure
>> [15:37:41] <mb[m]1> At this stage we might as well wait for this to land
>> upstream: https://www.python.org/dev/peps/pep-0552/
>>
>> So, it seems, that we are waiting for this pep to land upstream.
>
> Cool, thanks for the notification.
>
> Now, I’d be in favor of incorporating a patch in the meantime, rather
> than wait for the PEP to materialize.

I tried cherry-picking this patch at some point, but hit some difficult
test failures and haven't had time to follow up on it:

https://github.com/python/cpython/pull/296

According to FRidh from NixOS in that thread, this fix is also needed:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/cpython/3.6/default.nix#L75

Help in this area very welcome!

> I can’t find Debian’s patches in
> <https://alioth.debian.org/scm/loggerhead/pkg-python/python3-defaults-debian/files/head:/debian/>,
> but they probably have something.

Debian is "cheating" in that they compile the Python code at install
time.  So the build artifact is reproducible, but not the installed
product.  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]