all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files
@ 2017-05-01 13:57 Clément Lassieur
  2017-05-02 13:00 ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Clément Lassieur @ 2017-05-01 13:57 UTC (permalink / raw)
  To: 26734

Please consider the following package:

--8<---------------cut here---------------start------------->8---
(define-public hello
  (package
    (name "hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))
              (modules '((guix build utils)))
              (snippet '(begin #t))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
--8<---------------cut here---------------end--------------->8---

It is the 'hello' package with two extra lines that add an empty
snippet.

--8<---------------cut here---------------start------------->8---
(modules '((guix build utils)))
(snippet '(begin #t))
--8<---------------cut here---------------end--------------->8---

Now here is the output of 'ls -l' within the original (extracted)
'hello':

--8<---------------cut here---------------start------------->8---
-rw-r--r-- 1 clement users  93787 Nov 16  2014 ABOUT-NLS
-rw-r--r-- 1 clement users  43830 Nov 16  2014 aclocal.m4
-rw-r--r-- 1 clement users    593 Jul 19  2014 AUTHORS
drwxr-xr-x 3 clement users   4096 Nov 16  2014 build-aux/
-rw-r--r-- 1 clement users  12988 Nov 16  2014 ChangeLog
-rw-r--r-- 1 clement users  30632 Jul 19  2014 ChangeLog.O
-rw-r--r-- 1 clement users  32871 Nov 16  2014 config.in
-rwxr-xr-x 1 clement users 449922 Nov 16  2014 configure
-rw-r--r-- 1 clement users   2408 Oct  5  2014 configure.ac
drwxr-xr-x 2 clement users   4096 Nov 16  2014 contrib/
-rw-r--r-- 1 clement users  35147 Dec 12  2013 COPYING
drwxr-xr-x 2 clement users   4096 Nov 16  2014 doc/
-rw-r--r-- 1 clement users   4573 Nov 16  2014 GNUmakefile
-r--r--r-- 1 clement users   1400 Nov 16  2014 hello.1
-rw-r--r-- 1 clement users  15752 Dec 29  2013 INSTALL
drwxr-xr-x 2 clement users   4096 Nov 16  2014 lib/
drwxr-xr-x 2 clement users   4096 Nov 16  2014 m4/
-rw-r--r-- 1 clement users  63223 Nov 16  2014 maint.mk
-rw-r--r-- 1 clement users   4367 Aug  5  2014 Makefile.am
-rw-r--r-- 1 clement users 143282 Nov 16  2014 Makefile.in
drwxr-xr-x 2 clement users   4096 Nov 16  2014 man/
-rw-r--r-- 1 clement users   4023 Nov 16  2014 NEWS
drwxr-xr-x 2 clement users   4096 Nov 16  2014 po/
-rw-r--r-- 1 clement users   3582 Jul 19  2014 README
-rw-r--r-- 1 clement users   1571 Jul 19  2014 README-dev
-rw-r--r-- 1 clement users   3014 Nov 16  2014 README-release
drwxr-xr-x 2 clement users   4096 Nov 16  2014 src/
drwxr-xr-x 2 clement users   4096 Nov 16  2014 tests/
-rw-r--r-- 1 clement users    898 Jul 19  2014 THANKS
-rw-r--r-- 1 clement users     75 Jul 19  2014 TODO
--8<---------------cut here---------------end--------------->8---

And here is the same output within the modified 'hello' (with the empty
snippet):

--8<---------------cut here---------------start------------->8---
-rw-r--r-- 1 clement users  93787 Jan  1  1970 ABOUT-NLS
-rw-r--r-- 1 clement users  43830 Jan  1  1970 aclocal.m4
-rw-r--r-- 1 clement users    593 Jan  1  1970 AUTHORS
drwxr-xr-x 3 clement users   4096 Jan  1  1970 build-aux/
-rw-r--r-- 1 clement users  12988 Jan  1  1970 ChangeLog
-rw-r--r-- 1 clement users  30632 Jan  1  1970 ChangeLog.O
-rw-r--r-- 1 clement users  32871 Jan  1  1970 config.in
-rwxr-xr-x 1 clement users 449922 Jan  1  1970 configure
-rw-r--r-- 1 clement users   2408 Jan  1  1970 configure.ac
drwxr-xr-x 2 clement users   4096 Jan  1  1970 contrib/
-rw-r--r-- 1 clement users  35147 Jan  1  1970 COPYING
drwxr-xr-x 2 clement users   4096 Jan  1  1970 doc/
-rw-r--r-- 1 clement users   4573 Jan  1  1970 GNUmakefile
-r--r--r-- 1 clement users   1400 Jan  1  1970 hello.1
-rw-r--r-- 1 clement users  15752 Jan  1  1970 INSTALL
drwxr-xr-x 2 clement users   4096 Jan  1  1970 lib/
drwxr-xr-x 2 clement users   4096 Jan  1  1970 m4/
-rw-r--r-- 1 clement users  63223 Jan  1  1970 maint.mk
-rw-r--r-- 1 clement users   4367 Jan  1  1970 Makefile.am
-rw-r--r-- 1 clement users 143282 Jan  1  1970 Makefile.in
drwxr-xr-x 2 clement users   4096 Jan  1  1970 man/
-rw-r--r-- 1 clement users   4023 Jan  1  1970 NEWS
drwxr-xr-x 2 clement users   4096 Jan  1  1970 po/
-rw-r--r-- 1 clement users   3582 Jan  1  1970 README
-rw-r--r-- 1 clement users   1571 Jan  1  1970 README-dev
-rw-r--r-- 1 clement users   3014 Jan  1  1970 README-release
drwxr-xr-x 2 clement users   4096 Jan  1  1970 src/
drwxr-xr-x 2 clement users   4096 Jan  1  1970 tests/
-rw-r--r-- 1 clement users    898 Jan  1  1970 THANKS
-rw-r--r-- 1 clement users     75 Jan  1  1970 TODO
--8<---------------cut here---------------end--------------->8---

As you can see, the timestamps are not the same, although there is
nothing in the snippet.

I think the code is in guix/packages.scm (patch-and-repack).  Here is my
understanding of what happens: the presence of the snippet leads to 3
things:

1. extraction of the archive,
2. modification of some files (because of patches or snippet),
3. compression of the archive.

Step 3 sets all timestamps to zero so to avoid non-determinism in the
archive.  We obviously don't want archives to depend on the time of
their creation.  But I believe we should only reset the timestamps of
the files we modified in step 2.  Unmodified files should stay
unmodified.

What's the point?  Well, while working on the 0ad package, I realized
that building with 32 cores would always fail with a snippet (even if it
is empty) and always succeed without a snippet.  It took me a few hours
to understand this, and I put a comment in the 0ad package.  Maybe it is
a 0ad bug (some tricky race condition that depends of file timestamps),
but I believe it is also a Guix bug: an empty snippet should not change
in any way the binary output of a package.

I tried to patch 'patch-and-repack', but it triggers a full
rebuild...  WDYT?

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files
  2017-05-01 13:57 bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files Clément Lassieur
@ 2017-05-02 13:00 ` Ludovic Courtès
  2017-05-02 13:17   ` Clément Lassieur
  0 siblings, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2017-05-02 13:00 UTC (permalink / raw)
  To: Clément Lassieur; +Cc: 26734

Hi!

Clément Lassieur <clement@lassieur.org> skribis:

> I think the code is in guix/packages.scm (patch-and-repack).  Here is my
> understanding of what happens: the presence of the snippet leads to 3
> things:
>
> 1. extraction of the archive,
> 2. modification of some files (because of patches or snippet),
> 3. compression of the archive.
>
> Step 3 sets all timestamps to zero so to avoid non-determinism in the
> archive.  We obviously don't want archives to depend on the time of
> their creation.  But I believe we should only reset the timestamps of
> the files we modified in step 2.  Unmodified files should stay
> unmodified.

I agree.

> What's the point?  Well, while working on the 0ad package, I realized
> that building with 32 cores would always fail with a snippet (even if it
> is empty) and always succeed without a snippet.  It took me a few hours
> to understand this, and I put a comment in the 0ad package.  Maybe it is
> a 0ad bug (some tricky race condition that depends of file timestamps),
> but I believe it is also a Guix bug: an empty snippet should not change
> in any way the binary output of a package.

Yeah I was bitten by the same problem recently.

> I tried to patch 'patch-and-repack', but it triggers a full
> rebuild...  WDYT?

Right, it’s expected to trigger a full rebuild, so this should be fixed
in ‘core-updates’.

I guess we’ll have to collect the timestamps of all non-symlink files¹
in step #1 and to reapply them with ‘set-file-time’ from (guix build
utils) after step #2.

Thoughts?  Would you like to do that?

Thanks,
Ludo’.

¹ Because Guile provides bindings for ‘utime’, which does not support
  setting timestamps on symlinks.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files
  2017-05-02 13:00 ` Ludovic Courtès
@ 2017-05-02 13:17   ` Clément Lassieur
  2017-05-03  8:58     ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Clément Lassieur @ 2017-05-02 13:17 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 26734

Ludovic Courtès <ludo@gnu.org> writes:

> Clément Lassieur <clement@lassieur.org> skribis:
>
>> I tried to patch 'patch-and-repack', but it triggers a full
>> rebuild...  WDYT?
>
> Right, it’s expected to trigger a full rebuild, so this should be fixed
> in ‘core-updates’.

Yes, but is there a way to test the patch on one package without having
first to rebuild everything?

> I guess we’ll have to collect the timestamps of all non-symlink files¹
> in step #1 and to reapply them with ‘set-file-time’ from (guix build
> utils) after step #2.

Does that mean that symlinks will still have their timestamps changed?
To me that is a half-solution...  Wouldn't it be easier to collect all
recently modified files (those modified by snippet and patches), and set
their timestamp to "1 January 1970", without changing the other files?
That means removing the --mtime option from tar at step 3.

> Thoughts?  Would you like to do that?

Sure :-)

> ¹ Because Guile provides bindings for ‘utime’, which does not support
>   setting timestamps on symlinks.

If the guile binding doesn't support setting timestamps on symlinks, I
guess we can still use another way, like a system touch.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files
  2017-05-02 13:17   ` Clément Lassieur
@ 2017-05-03  8:58     ` Ludovic Courtès
  2017-05-03 21:45       ` Mark H Weaver
  0 siblings, 1 reply; 6+ messages in thread
From: Ludovic Courtès @ 2017-05-03  8:58 UTC (permalink / raw)
  To: Clément Lassieur; +Cc: 26734

Clément Lassieur <clement@lassieur.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Clément Lassieur <clement@lassieur.org> skribis:
>>
>>> I tried to patch 'patch-and-repack', but it triggers a full
>>> rebuild...  WDYT?
>>
>> Right, it’s expected to trigger a full rebuild, so this should be fixed
>> in ‘core-updates’.
>
> Yes, but is there a way to test the patch on one package without having
> first to rebuild everything?

I would add a snippet in ‘gnu-make-boot0’, which is the first package
being built, and then run:

  ./pre-inst-env guix build -S --rounds=2 \
    -e '(@@ (gnu packages commencement) gnu-make-boot0)'

>> I guess we’ll have to collect the timestamps of all non-symlink files¹
>> in step #1 and to reapply them with ‘set-file-time’ from (guix build
>> utils) after step #2.
>
> Does that mean that symlinks will still have their timestamps changed?

No, that means symlinks will still have their timestamps unchanged.  :-)

BTW, what timestamps to we put on the modified files?  We want that to
be deterministic so we cannot use the build time.  We cannot use a date
in the future, either.  We cannot use Jan. 1 1970 either because that
means that modified files may now be older than the unmodified files,
which may break build systems; for the same reason, we cannot leave the
mtime of modified files unchanged.

Now that I think about it, it’s not clear to me what can be done without
breaking something.

Thoughts?

>> ¹ Because Guile provides bindings for ‘utime’, which does not support
>>   setting timestamps on symlinks.
>
> If the guile binding doesn't support setting timestamps on symlinks, I
> guess we can still use another way, like a system touch.

Or we could add bindings for ‘futimes’.

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files
  2017-05-03  8:58     ` Ludovic Courtès
@ 2017-05-03 21:45       ` Mark H Weaver
  2017-05-03 22:01         ` Ludovic Courtès
  0 siblings, 1 reply; 6+ messages in thread
From: Mark H Weaver @ 2017-05-03 21:45 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 26734, Clément Lassieur

ludo@gnu.org (Ludovic Courtès) writes:

> BTW, what timestamps to we put on the modified files?  We want that to
> be deterministic so we cannot use the build time.  We cannot use a date
> in the future, either.  We cannot use Jan. 1 1970 either because that
> means that modified files may now be older than the unmodified files,
> which may break build systems; for the same reason, we cannot leave the
> mtime of modified files unchanged.
>
> Now that I think about it, it’s not clear to me what can be done without
> breaking something.
>
> Thoughts?

We could set the timestamp of modified files to be 1 second newer than
the newest file in the original source archive.

      Mark

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files
  2017-05-03 21:45       ` Mark H Weaver
@ 2017-05-03 22:01         ` Ludovic Courtès
  0 siblings, 0 replies; 6+ messages in thread
From: Ludovic Courtès @ 2017-05-03 22:01 UTC (permalink / raw)
  To: Mark H Weaver; +Cc: 26734, Clément Lassieur

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> BTW, what timestamps to we put on the modified files?  We want that to
>> be deterministic so we cannot use the build time.  We cannot use a date
>> in the future, either.  We cannot use Jan. 1 1970 either because that
>> means that modified files may now be older than the unmodified files,
>> which may break build systems; for the same reason, we cannot leave the
>> mtime of modified files unchanged.
>>
>> Now that I think about it, it’s not clear to me what can be done without
>> breaking something.
>>
>> Thoughts?
>
> We could set the timestamp of modified files to be 1 second newer than
> the newest file in the original source archive.

Sounds like a good idea.

Thanks,
Ludo’.

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2017-05-03 22:02 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-05-01 13:57 bug#26734: Snippets (even empty ones) of tar sources reset the timestamps of all files Clément Lassieur
2017-05-02 13:00 ` Ludovic Courtès
2017-05-02 13:17   ` Clément Lassieur
2017-05-03  8:58     ` Ludovic Courtès
2017-05-03 21:45       ` Mark H Weaver
2017-05-03 22:01         ` Ludovic Courtès

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.