git-fetch origin output is read-only - and reproducibility

git-fetch origin output is read-only - and reproducibility

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

Hi,

currently, when using the git-fetch origin, the resulting source will have
all files read-only.  Further phases trying to do reproducibility patches have
to manually chmod in order to be able to patch.

Can we change that?  For example change what this origin does or change what
the build systems do by default?

We have to rebuild core-updates anyway--and adding a chmod should not change
anything to the worse there.

If it's practical to do in master, that would be nice, too.

(The tarball origin doesn't do that btw, so the git-fetch origin is the
odd one out)

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: git-fetch origin output is read-only - and reproducibility

[Ricardo Wurmus]

Hi Danny,

> currently, when using the git-fetch origin, the resulting source will have
> all files read-only.  Further phases trying to do reproducibility patches have
> to manually chmod in order to be able to patch.
>
> Can we change that?  For example change what this origin does or change what
> the build systems do by default?

I think this would be good.  It’s a little annoying when switching from
a tarball to git-fetch because you’ll have to add a phase that makes all
files writable.

This would probably have to be changed in the unpack phase of the
gnu-build-system, which copies files recursively when the sources are a
directory (instead of a tarball).

I don’t know if that’s safe to do.  Are other fetch methods affected as
well?  hg-fetch or svn-fetch, for example?

-- 
Ricardo

Re: git-fetch origin output is read-only - and reproducibility

[Robert Vollmert]

> On 29. Jul 2019, at 17:49, Ricardo Wurmus <rekado@elephly.net> wrote:
>> currently, when using the git-fetch origin, the resulting source will have
>> all files read-only.  Further phases trying to do reproducibility patches have
>> to manually chmod in order to be able to patch.
>> 
>> Can we change that?  For example change what this origin does or change what
>> the build systems do by default?
> 
> I think this would be good.  It’s a little annoying when switching from
> a tarball to git-fetch because you’ll have to add a phase that makes all
> files writable.
> 
> This would probably have to be changed in the unpack phase of the
> gnu-build-system, which copies files recursively when the sources are a
> directory (instead of a tarball).

At the risk of getting slightly off-topic, I was very surprised when I figured
out that such things were left to the build system. Couldn’t the source/origin
abstraction consistently refer a directory tree *or* a tarball?

Cheers
Robert

Re: git-fetch origin output is read-only - and reproducibility

[Ricardo Wurmus]

Hi Robert,

>> On 29. Jul 2019, at 17:49, Ricardo Wurmus <rekado@elephly.net> wrote:
>>> currently, when using the git-fetch origin, the resulting source will have
>>> all files read-only.  Further phases trying to do reproducibility patches have
>>> to manually chmod in order to be able to patch.
>>>
>>> Can we change that?  For example change what this origin does or change what
>>> the build systems do by default?
>>
>> I think this would be good.  It’s a little annoying when switching from
>> a tarball to git-fetch because you’ll have to add a phase that makes all
>> files writable.
>>
>> This would probably have to be changed in the unpack phase of the
>> gnu-build-system, which copies files recursively when the sources are a
>> directory (instead of a tarball).
>
> At the risk of getting slightly off-topic, I was very surprised when I figured
> out that such things were left to the build system. Couldn’t the source/origin
> abstraction consistently refer a directory tree *or* a tarball?

Almost anything can be a source.  (Even #F is a valid source.)  The
origin methods only operate on their arguments and fetch whatever has
been requested.  This might be a single file (a tarball) or it could be
the result of running “svn checkout” or anything else.

These fixed output derivations that are generated from origins only care
about obtaining *something* that has the provided hash.  It is up to the
builder to do something with whatever the source really is.

Most build systems inherit from the gnu-build-system, so they’ll get to
reuse the “unpack” phase, which conveniently checks if the source is a
tarball.  In the case of Java archives it doesn’t do the right thing,
because it doesn’t know about Jars, so the ant-build-system overrides
that phase, etc.

Dealing with sources sometimes requires special knowledge and the build
system might be best equipped to employ that knowledge.

What would you suggest the fetchers implement to guarantee that the
sources will always be of some expected form?

--
Ricardo

Re: git-fetch origin output is read-only - and reproducibility

[Robert Vollmert]

Hi,

On 29. Jul 2019, at 18:10, Ricardo Wurmus <rekado@elephly.net> wrote:
> Most build systems inherit from the gnu-build-system, so they’ll get to
> reuse the “unpack” phase, which conveniently checks if the source is a
> tarball.  In the case of Java archives it doesn’t do the right thing,
> because it doesn’t know about Jars, so the ant-build-system overrides
> that phase, etc.
> 
> Dealing with sources sometimes requires special knowledge and the build
> system might be best equipped to employ that knowledge.
> 
> What would you suggest the fetchers implement to guarantee that the
> sources will always be of some expected form?

I would suggest that they specify the archive type, and either

- repack the archive to a standard format, e.g. .tar.gz (this should then also
  apply to sources that are local directory trees)
- unpack the archive to a directory tree

An alternative change that would make the whole setup a bit less confusing
would be to factor all the “standard” build system stuff out of gnu-build-system
and into a base-build-system that provides source unpacking and phase handling
and nothing else.

I originally ran into this surprise by basing things on trivial-build-system,
getting things to work with a local directory, then seeing that my apparently
functioning build system failed once I referred to a git checkout because that
packs to a tarball.

Robert

Re: git-fetch origin output is read-only - and reproducibility

[Ludovic Courtès]

Hi,

Robert Vollmert <rob@vllmrt.net> skribis:

> On 29. Jul 2019, at 18:10, Ricardo Wurmus <rekado@elephly.net> wrote:
>> Most build systems inherit from the gnu-build-system, so they’ll get to
>> reuse the “unpack” phase, which conveniently checks if the source is a
>> tarball.  In the case of Java archives it doesn’t do the right thing,
>> because it doesn’t know about Jars, so the ant-build-system overrides
>> that phase, etc.
>> 
>> Dealing with sources sometimes requires special knowledge and the build
>> system might be best equipped to employ that knowledge.
>> 
>> What would you suggest the fetchers implement to guarantee that the
>> sources will always be of some expected form?
>
> I would suggest that they specify the archive type, and either
>
> - repack the archive to a standard format, e.g. .tar.gz (this should then also
>   apply to sources that are local directory trees)
> - unpack the archive to a directory tree

Note that ‘git-fetch’ & co. currently produced fixed-output derivations.

If ‘git-fetch’ were to systematically repack to a tarball, we’d
introduce a dependency on tar + some compressor, which is not always
desirable.

> An alternative change that would make the whole setup a bit less confusing
> would be to factor all the “standard” build system stuff out of gnu-build-system
> and into a base-build-system that provides source unpacking and phase handling
> and nothing else.

I agree that this would be an improvement.

Ludo’.