Bootstrap a GNU source distribution from git

Bootstrap a GNU source distribution from git

[Vivien Kraus]

[-- Attachment #1: Type: text/plain, Size: 2271 bytes --]

Dear Guix,

GNU sources are usually shipped as tarballs with some pre-compiled
sources included. This can be a bit scary at times, so the question now
is, can we skip the tarballs and build everything from human-authored
sources?

I first want to emphasize that distributing generated sources is an
important feature of a build system. It is necessary for it to boast
cross-compilation support, and for packages doing simple forms of
introspection (such as using help2man or gobject introspection) it may
even be sufficient. There are other nice things as well, so with that
in mind, let us open the black box and see what the maintainers are
hiding from us.

I chose Hello. After a bit of guixing around, it works:

guix build -f hello.scm

shows the build, and

guix build --source -f hello.scm

shows the full source distribution (although it is not in a tar because
it would have additional reproducibility problems).

So, what are the hello maintainers hiding from us?
— they use gnulib;
— the translations are not in the git repository, so their release
process involves fetching stuff from the web and packing it in;
— the manpage generation depends on the presence of a ".git" folder;
— they have perl installed;
— they have a bunch of autotools, included code generators (gperf);
— they work in the git repository to have a meaningful version number
and commit log (for the ChangeLog);
— the manual displays its revision date, but we don’t know how it is
computed (I suspect the revision date is the last time the maintainer
who did the release edited or git-pulled the texinfo source, which,
let’s agree on that, is not very precise).

I could not fully recover the release source (I am not attempting bit-
reproducibility) because:
— the translations were lost;
— the ChangeLog was lost;
— the manual revision date was lost;
— guix’ patched shebangs leaked in the sources.

This is a mixed responsibility problem. The big point here is
translations. I understand that using translationproject.org means that
committing the translations would storm the repository with automated
commits, which would be undesirable. Maybe Software Heritage could
help?

Happy hacking!

Vivien

[-- Attachment #2: hello.scm --]
[-- Type: text/x-scheme, Size: 5685 bytes --]

(use-modules
 ((guix licenses) #:prefix license:)
 (guix build-system gnu)
 (guix gexp)
 (guix packages)
 (guix git-download)
 (gnu packages autotools)
 (gnu packages base)
 (gnu packages gettext)
 (gnu packages gperf)
 (gnu packages man)
 (gnu packages perl)
 (gnu packages texlive)
 (gnu packages texinfo))

(define hello-dist
  (package
    (name "hello-dist")
    (version "2.12.1")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://git.savannah.gnu.org/git/hello.git")
                    (commit (string-append "v" version))
                    ;; hello uses gnulib as a submodule
                    (recursive? #t)))
              (file-name (string-append "hello-" version "-checkout"))
              (sha256
               (base32
                "0k8ibhlj7v6i0ynzldkfg5r8p78m1mkczci0drr6h9z7n85vfacy"))))
    (build-system gnu-build-system)
    (arguments
     (list
      #:phases
      #~(modify-phases %standard-phases
          (replace 'bootstrap
            (lambda* (#:key native-inputs inputs #:allow-other-keys)
              (substitute* "Makefile.am"
                (("\\$\\(top_srcdir\\)/build-aux/gitlog-to-changelog")
                 ;; This script requires a full git checkout, which we
                 ;; do not provide. FIXME: The source ChangeLog will
                 ;; not be relevant.
                 "true"))
              (for-each
               patch-shebang
               ;; These scripts should be patched early because they
               ;; are invoked by the bootstrap script (many more will
               ;; be patched after the bootstrap phase).
               (list "bootstrap" "gnulib/gnulib-tool" "gnulib/build-aux/prefix-gnulib-mk"
                     "build-aux/git-version-gen"))
              ;; hello will only build the manual pages if it is run
              ;; from its git repository (read: if .git is an existing
              ;; directory, see configure.ac) because the maintainer
              ;; thinks no cross-compiling can happen in this case
              ;; presumably. It would be cool if the hello maintainer
              ;; could precisely identify the dependencies of the
              ;; manpage so that a manpage template could be
              ;; distributed.
              (mkdir-p ".git")
              (call-with-output-file ".tarball-version"
                ;; hello would compute the version number from state
                ;; of the git repository if .tarball-version did not
                ;; exist.
                (lambda (port)
                  (format port "~a\n" #$version)))
              (invoke "sh" "bootstrap"
                      "--no-git"
                      "--gnulib-srcdir=gnulib"
                      ;; FIXME: package the po files separately and
                      ;; change the po_download_command_format
                      ;; definition in bootstrap. For now there is no
                      ;; way to have po files.
                      "--skip-po")
              (patch-makefile-SHELL "po/Makefile.in.in")))
          (replace 'check
            (lambda* (#:key native-inputs inputs #:allow-other-keys)
              (invoke "make" "distcheck"
                      (string-append "DISTCHECK_CONFIGURE_FLAGS = SHELL="
                                     (search-input-file (or native-inputs inputs) "bin/sh")))))
          (replace 'install
            (lambda _
              ;; FIXME: the manual page is dated for version released 1970-01-01
              ;; FIXME: the guix shebangs leak in the source code
              ;; distribution. I need some sort of reverse-patch-shebangs and
              ;; reverse-patch-makefile-SHELL.
              (mkdir-p #$output)
              ;; Can’t run "make distdir=#$output distdir because the
              ;; po build system does not expect the distdir to be
              ;; anything other than ../$(PACKAGE)-$(VERSION)/po.
              (invoke "tar"
                      "--extract"
                      (string-append "--file=hello-" #$version ".tar.gz")
                      (string-append "--directory=" #$output)
                      "--strip-components=1"))))))
    (native-inputs
     ;; All these things are used by the hello maintainer to make a
     ;; distribution tarball and run `make distcheck'.
     ;; FIXME: do not use the full texlive.
     (list autoconf autoconf-archive automake libtool gnu-gettext perl gperf help2man tar texinfo texlive))
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license license:gpl3+)))

(package
  ;; FIXME:
  ;; - the ChangeLog is incomplete
  ;; - no translations
  ;; - wrong info manual revision date
  ;;
  ;; Future works:
  ;; - regarding the ChangeLog: guix git-download could maybe run a
  ;;   form of gitlog-to-changelog and save the output
  ;; - regarding the info manual revision date: guix git-download
  ;;   could maybe find the latest commit date touching the manual and save
  ;;   it (bootstrap would then touch the texinfo source at this date)
  ;; - regarding the translations: propagate the Guix philosophy that
  ;;   you should not download things from the web at build time. Check the
  ;;   translations in to git on releases.
  (inherit hello-dist)
  (name "hello")
  (source hello-dist)
  (build-system gnu-build-system)
  (arguments '())
  (native-inputs '()))

Re: Bootstrap a GNU source distribution from git

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 2506 bytes --]

Hi,

On Sun, 29 Sep 2024 19:52:13 +0200
Vivien Kraus <vivien@planete-kraus.eu> wrote:

> GNU sources are usually shipped as tarballs with some pre-compiled
> sources included. This can be a bit scary at times, so the question
> now is, can we skip the tarballs and build everything from
> human-authored sources?
In some cases it is also possible to build from the tarballs and also
build everything from human-authored sources.

The solution to that is to treat the generated files as things that are
not wanted during the build and just delete them before starting to
build. It's similar to tarballs which also contain the binaries of some
of their dependencies (example: .so files, often with their
corresponding source code as well for licensing compliance).

In such case the answer of the distributions is simply to delete such
binaries before the build.

The advantage of these bundled binaries is that the same tarball makes
it possible for more people to more easily build the software but it
also increase the maintenance from the distributions as they need to
find and delete such files, so to me it doesn't look ideal. Having an
easy way to automatically delete such generated files safely could be a
good idea for source tarballs releases as both uses cases could be
covered with a single tarball release.

But this also brings an interesting questions that I also had myself but
I never really had the opportunity to ask them: what is Guix policy with
regard to source code when there are multiple providers (typically git
vs tarball)?

As I understand source from version control gives us the ability to
make some package transformation option work out of the box. For
instance we have:
> $ guix build \
> --with-commit=hello=2633763362586903cf6506f4c4d708727a981025 hello
> guix build: error: the source of hello@2.12.1 is not a Git reference

So is there some (implicit or explicit) rule to always prefer the
source from version control unless it's not possible or practical (no
version control being used, too complicated to build like with GNU
hello, or the package is required for the version control to work) ?.

Since Guix also checks the hash of the source code an idea to improve
things could also be to modify Guix to allow the use of external tools
to bootstrap the download of source code through version control and
for instance download git from git. Though that could require some
substantial work and discussions.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Bootstrap a GNU source distribution from git

[Tobias Geerinckx-Rice]

Hi Denis,

On 30 September 2024 21:43:06 UTC, Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> wrote:
> what is Guix policy with
>regard to source code when there are multiple providers (typically git
>vs tarball)?

There's never been an explicit policy as far as I'm aware.

In the past decade the consensus has shifted from release tarballs towards both, tarballs and VCS repository checkouts, as long as they represent an upstream release.

For the reasons you point out, I expect the tides to shift further in favour of VCS checkouts as time moves on.  I'm in favour of this.

If you're interested in formalising this 'policy', it would be a good test of this 'RFC' business that's seen little use so far.

>Since Guix also checks the hash of the source code an idea to improve
>things could also be to modify Guix to allow the use of external tools
>to bootstrap the download of source code through version control and
>for instance download git from git.

I don't understand what you mean by this, or what 'modify Guix' means and why it would be needed?



Kind regards,

T G-R

Sent on the go.  Excuse or enjoy my brevity.

Re: Bootstrap a GNU source distribution from git

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 2535 bytes --]

On Tue, 01 Oct 2024 10:20:54 +0000
Tobias Geerinckx-Rice <me@tobias.gr> wrote:
> >Since Guix also checks the hash of the source code an idea to improve
> >things could also be to modify Guix to allow the use of external
> >tools to bootstrap the download of source code through version
> >control and for instance download git from git.
> 
> I don't understand what you mean by this, or what 'modify Guix' means
> and why it would be needed?

We currently have something like that:
> (define-public git-minimal
>   (package
>     (name "git-minimal")
>     (version "2.46.0")
>     (source (origin
>              (method url-fetch)
>              (uri (string-append
> "mirror://kernel.org/software/scm/git/git-" version ".tar.xz"))
>              (sha256
>               (base32
>                "15bzq9m6c033qiz5q5gw1nqw4m452vvqax30wbms6z4bl9i384kz"))))
> [...]

If we replace with something like that:
> (define-public git-minimal
>   (package
>     (name "git-minimal")
>     (version "2.46.0")
>     (source
>       (origin
>         (method git-fetch)
>         (uri 
>           (git-reference
>             (url "https://git.kernel.org/pub/scm/git/git.git"))
>             (commit "<some hash>")))
>         (file-name (git-file-name name version))
>         (sha256
>           (base32
>             "15bzq9m6c033qiz5q5gw1nqw4m452vvqax30wbms6z4bl9i384kz"))))
> [...]

Then we have at least 2 issues.

The first one is that we might end up with circular dependencies inside
the Guix source code somehow that creates issues when building packages
and/or guix, etc. But that might be fixable with some work.

However if I understand well, that circular dependency would not create
any security/reproducibility issue since we would already have a base32
hash of the source code of "git-minimal".

And so if for instance someone packages Guix on a foreign distribution,
we could imagine some system(s) where the the git source code is somehow
provided to Guix as a dependency, and so once built, Guix would be able
to use that provided source code by verifying its hash and then using
it to build git, and enabling Guix to download subsequent packages
using git.

This could then be extended to all the packages that git depend on, and
with that we'd then be able to use git a lot more without security
issues.

The downside is that as always someone needs to be interested in it,
and find the time to work on it. It also might make building Guix
harder.

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Bootstrap a GNU source distribution from git

[Janneke Nieuwenhuizen]

Vivien Kraus writes:

Hi Vivien,

> I could not fully recover the release source (I am not attempting bit-
> reproducibility) because:
> — the translations were lost;
> — the ChangeLog was lost;
> — the manual revision date was lost;
> — guix’ patched shebangs leaked in the sources.
>
> This is a mixed responsibility problem.

If you would ask me, this is a bug in GNU hello.  Since this spring
the Guix source tarball reproduces bit-by-bit by running: `make dist',
see <https://lists.gnu.org/archive/html/guix-patches/2024-04/msg01157.html>.

Greetings,
Janneke

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

Re: Bootstrap a GNU source distribution from git

[Ludovic Courtès]

Hi,

Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org> skribis:

> If we replace with something like that:
>> (define-public git-minimal
>>   (package
>>     (name "git-minimal")
>>     (version "2.46.0")
>>     (source
>>       (origin
>>         (method git-fetch)
>>         (uri 
>>           (git-reference
>>             (url "https://git.kernel.org/pub/scm/git/git.git"))
>>             (commit "<some hash>")))
>>         (file-name (git-file-name name version))
>>         (sha256
>>           (base32
>>             "15bzq9m6c033qiz5q5gw1nqw4m452vvqax30wbms6z4bl9i384kz"))))
>> [...]
>
> Then we have at least 2 issues.
>
> The first one is that we might end up with circular dependencies inside
> the Guix source code somehow that creates issues when building packages
> and/or guix, etc. But that might be fixable with some work.

There’s no circular dependency *if* the daemon is recent enough and
supports the ‘git-download’ derivation builder.  See
<https://issues.guix.gnu.org/65866> for context.

HTH,
Ludo’.