Rust reprodubility -- .rmeta and shadow-rs

Rust reprodubility -- .rmeta and shadow-rs

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2189 bytes --]

Hi,

There was some mail about irreproducibility in Rust, but I couldn't
find it anymore.  Anyway, I found a potential cause: rust-shadow-rs
embeds timestamps (even though it nominally respects
SOURCE_DATE_EPOCH???) and the ordering of definitions it generates is
based on a hash map (and hence, irreproducible).

The crate id is based on a hash over the source code, so this
irreproducibility can cause build failures if substitutes are used.

By removing the time stamp and sorting the definitions, 'nushell'
successfully built on ci.guix.gnu.org whereas it previously failed to
build on ci.guix.gnu.org but built successfully locally (with
antioxidant), and IIRC the (antioxidated) 'rust-nu-command' is now
reproducible:

Anyway, here the patch I used:

    ("rust-shadow-rs"
     ,#~((add-after 'unpack 'fixup-source-date-epoch
	   (lambda _
	     ;; TODO: it nominally supports SOURCE_DATE_EPOCH, yet something things go wrong,
	     ;; as the shadow.rs still contains the unnormalised time stamp ...
	     ;; For now, do a work-around.
	     (substitute* '("src/lib.rs" "src/env.rs")
	       (("BuildTime::Local\\(Local::now\\(\\)\\)\\.human_format\\(\\)")
		(object->string "[timestamp expunged for reproducibility]"))
	       (("time\\.human_format\\(\\)")
		"\"[timestamp expunged for reproducibility]\".to_string()")
	       (("time\\.to_rfc3339_opts\\(SecondsFormat::Secs, true)")
		"\"[timestamp expunged for reproducibility]\".to_string()")
	       (("time\\.to_rfc2822\\(\\)")
		"\"[timestamp expunged for reproducibility]\".to_string()"))))
	 (add-after 'unpack 'more-reproducibility ;; by default, it uses a hashmap, leading to an irreproducible ordering in shadow.rs and hence an irreproducible .rmeta (TODO: upstream?)
	   (lambda _
	     (substitute* "src/lib.rs" ; sort
	       (("\\(k, v\\) in self\\.map\\.clone\\(\\)")
		"(k, v) in std::collections::BTreeMap::from_iter(self.map.clone().iter())")
	       (("self\\.write_const\\(k, v\\)") "self.write_const(k, v.clone())")
	       (("self\\.map\\.keys\\(\\)") "std::collections::BTreeSet::from_iter(self.map.keys())"))))))

Maybe that was the cause?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Rust reprodubility -- .rmeta and shadow-rs

[Ludovic Courtès]

Hello!

Maxime Devos <maximedevos@telenet.be> skribis:

> There was some mail about irreproducibility in Rust, but I couldn't
> find it anymore.  Anyway, I found a potential cause: rust-shadow-rs
> embeds timestamps (even though it nominally respects
> SOURCE_DATE_EPOCH???) and the ordering of definitions it generates is
> based on a hash map (and hence, irreproducible).

I found these:

  https://issues.guix.gnu.org/50015
  https://issues.guix.gnu.org/55928

> The crate id is based on a hash over the source code, so this
> irreproducibility can cause build failures if substitutes are used.
>
> By removing the time stamp and sorting the definitions, 'nushell'
> successfully built on ci.guix.gnu.org whereas it previously failed to
> build on ci.guix.gnu.org but built successfully locally (with
> antioxidant), and IIRC the (antioxidated) 'rust-nu-command' is now
> reproducible:
>
> Anyway, here the patch I used:
>
>     ("rust-shadow-rs"
>      ,#~((add-after 'unpack 'fixup-source-date-epoch
> 	   (lambda _
> 	     ;; TODO: it nominally supports SOURCE_DATE_EPOCH, yet something things go wrong,
> 	     ;; as the shadow.rs still contains the unnormalised time stamp ...
> 	     ;; For now, do a work-around.
> 	     (substitute* '("src/lib.rs" "src/env.rs")
> 	       (("BuildTime::Local\\(Local::now\\(\\)\\)\\.human_format\\(\\)")
> 		(object->string "[timestamp expunged for reproducibility]"))
> 	       (("time\\.human_format\\(\\)")
> 		"\"[timestamp expunged for reproducibility]\".to_string()")
> 	       (("time\\.to_rfc3339_opts\\(SecondsFormat::Secs, true)")
> 		"\"[timestamp expunged for reproducibility]\".to_string()")
> 	       (("time\\.to_rfc2822\\(\\)")
> 		"\"[timestamp expunged for reproducibility]\".to_string()"))))
> 	 (add-after 'unpack 'more-reproducibility ;; by default, it uses a hashmap, leading to an irreproducible ordering in shadow.rs and hence an irreproducible .rmeta (TODO: upstream?)
> 	   (lambda _
> 	     (substitute* "src/lib.rs" ; sort
> 	       (("\\(k, v\\) in self\\.map\\.clone\\(\\)")
> 		"(k, v) in std::collections::BTreeMap::from_iter(self.map.clone().iter())")
> 	       (("self\\.write_const\\(k, v\\)") "self.write_const(k, v.clone())")
> 	       (("self\\.map\\.keys\\(\\)") "std::collections::BTreeSet::from_iter(self.map.keys())"))))))
>
> Maybe that was the cause?

You mean this issue you identified could have been the cause of
reproducibility issues found in other Rust packages?

Anyway, it looks like the snippet above should be applied to
‘rust-shadow-rs’ in current ‘master’, no?

Thanks,
Ludo’.

Re: Rust reprodubility -- .rmeta and shadow-rs

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

> https://issues.guix.gnu.org/50015

I don't think that shadow-rs changes Cargo.toml, so I think that one is
a separate issue ...

> https://issues.guix.gnu.org/55928

Maybe related, though .rmeta isn't mentioned there so maybe not.
Seems debugging information related, so maybe the report at (and fix
at)
https://github.com/rust-lang/rust/issues/34902#issuecomment-565557076
is important there.

Ludovic Courtès schreef op do 30-06-2022 om 13:35 [+0200]:
> > Anyway, here the patch I used:
> > 
> > [...]
> > 
> > Maybe that was the cause?
> 
> You mean this issue you identified could have been the cause of
> reproducibility issues found in other Rust packages?

I don't know if it applies to leaf packages or only to dependencies,
but perhaps! 

> Anyway, it looks like the snippet above should be applied to
> ‘rust-shadow-rs’ in current ‘master’, no?

Yes -- also, upstream has a patch now!
<https://github.com/baoyachi/shadow-rs/pull/97>
That's only for the ordering, not the timestamps, though.
The timestamp issue might be caused due to how antioxidant replaces
some inputs and fiddles with ‘features’ so I haven't reported that yet
....

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]