unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Rust reprodubility -- .rmeta and shadow-rs
@ 2022-06-26  7:54 Maxime Devos
  2022-06-30 11:35 ` Ludovic Courtès
  0 siblings, 1 reply; 3+ messages in thread
From: Maxime Devos @ 2022-06-26  7:54 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2189 bytes --]

Hi,

There was some mail about irreproducibility in Rust, but I couldn't
find it anymore.  Anyway, I found a potential cause: rust-shadow-rs
embeds timestamps (even though it nominally respects
SOURCE_DATE_EPOCH???) and the ordering of definitions it generates is
based on a hash map (and hence, irreproducible).

The crate id is based on a hash over the source code, so this
irreproducibility can cause build failures if substitutes are used.

By removing the time stamp and sorting the definitions, 'nushell'
successfully built on ci.guix.gnu.org whereas it previously failed to
build on ci.guix.gnu.org but built successfully locally (with
antioxidant), and IIRC the (antioxidated) 'rust-nu-command' is now
reproducible:

Anyway, here the patch I used:

    ("rust-shadow-rs"
     ,#~((add-after 'unpack 'fixup-source-date-epoch
	   (lambda _
	     ;; TODO: it nominally supports SOURCE_DATE_EPOCH, yet something things go wrong,
	     ;; as the shadow.rs still contains the unnormalised time stamp ...
	     ;; For now, do a work-around.
	     (substitute* '("src/lib.rs" "src/env.rs")
	       (("BuildTime::Local\\(Local::now\\(\\)\\)\\.human_format\\(\\)")
		(object->string "[timestamp expunged for reproducibility]"))
	       (("time\\.human_format\\(\\)")
		"\"[timestamp expunged for reproducibility]\".to_string()")
	       (("time\\.to_rfc3339_opts\\(SecondsFormat::Secs, true)")
		"\"[timestamp expunged for reproducibility]\".to_string()")
	       (("time\\.to_rfc2822\\(\\)")
		"\"[timestamp expunged for reproducibility]\".to_string()"))))
	 (add-after 'unpack 'more-reproducibility ;; by default, it uses a hashmap, leading to an irreproducible ordering in shadow.rs and hence an irreproducible .rmeta (TODO: upstream?)
	   (lambda _
	     (substitute* "src/lib.rs" ; sort
	       (("\\(k, v\\) in self\\.map\\.clone\\(\\)")
		"(k, v) in std::collections::BTreeMap::from_iter(self.map.clone().iter())")
	       (("self\\.write_const\\(k, v\\)") "self.write_const(k, v.clone())")
	       (("self\\.map\\.keys\\(\\)") "std::collections::BTreeSet::from_iter(self.map.keys())"))))))

Maybe that was the cause?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Rust reprodubility -- .rmeta and shadow-rs
  2022-06-26  7:54 Rust reprodubility -- .rmeta and shadow-rs Maxime Devos
@ 2022-06-30 11:35 ` Ludovic Courtès
  2022-06-30 11:45   ` Maxime Devos
  0 siblings, 1 reply; 3+ messages in thread
From: Ludovic Courtès @ 2022-06-30 11:35 UTC (permalink / raw)
  To: Maxime Devos; +Cc: guix-devel

Hello!

Maxime Devos <maximedevos@telenet.be> skribis:

> There was some mail about irreproducibility in Rust, but I couldn't
> find it anymore.  Anyway, I found a potential cause: rust-shadow-rs
> embeds timestamps (even though it nominally respects
> SOURCE_DATE_EPOCH???) and the ordering of definitions it generates is
> based on a hash map (and hence, irreproducible).

I found these:

  https://issues.guix.gnu.org/50015
  https://issues.guix.gnu.org/55928

> The crate id is based on a hash over the source code, so this
> irreproducibility can cause build failures if substitutes are used.
>
> By removing the time stamp and sorting the definitions, 'nushell'
> successfully built on ci.guix.gnu.org whereas it previously failed to
> build on ci.guix.gnu.org but built successfully locally (with
> antioxidant), and IIRC the (antioxidated) 'rust-nu-command' is now
> reproducible:
>
> Anyway, here the patch I used:
>
>     ("rust-shadow-rs"
>      ,#~((add-after 'unpack 'fixup-source-date-epoch
> 	   (lambda _
> 	     ;; TODO: it nominally supports SOURCE_DATE_EPOCH, yet something things go wrong,
> 	     ;; as the shadow.rs still contains the unnormalised time stamp ...
> 	     ;; For now, do a work-around.
> 	     (substitute* '("src/lib.rs" "src/env.rs")
> 	       (("BuildTime::Local\\(Local::now\\(\\)\\)\\.human_format\\(\\)")
> 		(object->string "[timestamp expunged for reproducibility]"))
> 	       (("time\\.human_format\\(\\)")
> 		"\"[timestamp expunged for reproducibility]\".to_string()")
> 	       (("time\\.to_rfc3339_opts\\(SecondsFormat::Secs, true)")
> 		"\"[timestamp expunged for reproducibility]\".to_string()")
> 	       (("time\\.to_rfc2822\\(\\)")
> 		"\"[timestamp expunged for reproducibility]\".to_string()"))))
> 	 (add-after 'unpack 'more-reproducibility ;; by default, it uses a hashmap, leading to an irreproducible ordering in shadow.rs and hence an irreproducible .rmeta (TODO: upstream?)
> 	   (lambda _
> 	     (substitute* "src/lib.rs" ; sort
> 	       (("\\(k, v\\) in self\\.map\\.clone\\(\\)")
> 		"(k, v) in std::collections::BTreeMap::from_iter(self.map.clone().iter())")
> 	       (("self\\.write_const\\(k, v\\)") "self.write_const(k, v.clone())")
> 	       (("self\\.map\\.keys\\(\\)") "std::collections::BTreeSet::from_iter(self.map.keys())"))))))
>
> Maybe that was the cause?

You mean this issue you identified could have been the cause of
reproducibility issues found in other Rust packages?

Anyway, it looks like the snippet above should be applied to
‘rust-shadow-rs’ in current ‘master’, no?

Thanks,
Ludo’.


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Rust reprodubility -- .rmeta and shadow-rs
  2022-06-30 11:35 ` Ludovic Courtès
@ 2022-06-30 11:45   ` Maxime Devos
  0 siblings, 0 replies; 3+ messages in thread
From: Maxime Devos @ 2022-06-30 11:45 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

> https://issues.guix.gnu.org/50015

I don't think that shadow-rs changes Cargo.toml, so I think that one is
a separate issue ...

> https://issues.guix.gnu.org/55928

Maybe related, though .rmeta isn't mentioned there so maybe not.
Seems debugging information related, so maybe the report at (and fix
at)
https://github.com/rust-lang/rust/issues/34902#issuecomment-565557076
is important there.

Ludovic Courtès schreef op do 30-06-2022 om 13:35 [+0200]:
> > Anyway, here the patch I used:
> > 
> > [...]
> > 
> > Maybe that was the cause?
> 
> You mean this issue you identified could have been the cause of
> reproducibility issues found in other Rust packages?

I don't know if it applies to leaf packages or only to dependencies,
but perhaps! 

> Anyway, it looks like the snippet above should be applied to
> ‘rust-shadow-rs’ in current ‘master’, no?

Yes -- also, upstream has a patch now!
<https://github.com/baoyachi/shadow-rs/pull/97>
That's only for the ordering, not the timestamps, though.
The timestamp issue might be caused due to how antioxidant replaces
some inputs and fiddles with ‘features’ so I haven't reported that yet
....

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2022-06-30 12:26 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-26  7:54 Rust reprodubility -- .rmeta and shadow-rs Maxime Devos
2022-06-30 11:35 ` Ludovic Courtès
2022-06-30 11:45   ` Maxime Devos

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).