Creating a reliable bootstrap for building from source

Creating a reliable bootstrap for building from source

[Pjotr Prins]

Thinking about it, what I want to achieve is that we can take the
latest git tree and bootstrap by building guix and packages. This
should be easy, since I have guix running, but it is not. And the main
trouble is that the underlying build packages can differ over time. I
am looking at gcc versions and guile versions.  I.e., we are building
on shifting sands. How unguixy!

The combination of 'guix pull' held a promise, were it not that pull is
also iffy. Probably for pretty much the same reason.

The bootstrap+configure scripts try to work that, but actually
address a wider case. I.e. people who want to bootstrap in Debian etc.
I don't think we need al that. I write Makefile.guix for my projects
and they tend to be simple! Once you can assume Guix is there life
gets simple as a developer - except when you try to bootstrap :0

The instruction I would like to write for others is:

1. Install the latest bootstrap-guix-from-source package after a guix pull
2. git clone guix && cd guix
3. run make -f Makefile.guix  

(no configure is needed in guix!)

4. ./pre-inst guix etc. etc.

Now the bootstrap-guix-from-source package is easy to create and needs
one single test: build guix from the git tree. It should have a
dependency on the git tree itself, so it gets forced every time on the
build farm.

Once we have that we can even bootstrap using guix pack and such.

I realise potluck could be an alternative, but a working bootstrap
would make guix pull also reliable and we can test it effectively on
the build farm.

Pj. 

(unguixy, I like that. It applies to many deployment statements. The
world is an unguixy place)

RE: Creating a reliable bootstrap for building from source

[Jeremiah]

> Thinking about it, what I want to achieve is that we can take the
> latest git tree and bootstrap by building guix and packages. This
> should be easy, since I have guix running, but it is not. And the main
> trouble is that the underlying build packages can differ over time. I
> am looking at gcc versions and guile versions.  I.e., we are building
> on shifting sands. How unguixy!

It is worse than that, Janneke and I are still trying to build out a
full source bootstrap.

Now mind you we have gotten quite a bit down that rabbit hole.
( I've build from a hex monitor to a Lexically scoped garbage
compacting/collecting lisp and Janneke built his rather impressive MES
which already supports large parts of the C language and enough to
bootstrap some rather important pieces)

But there still are many gaps left to close (how to bootstrap a 280 byte
hex monitor without a hex monitor or hex assembler, stage0-vm
downstrapping, MES tinycc bootstrapping, MES lisp bootstrapping, etc)

but ultimately shifting sands are the only grounds we can be certain
will be there.

So we better get comfortable minimizing our assumptions.

-Jeremiah

Re: Creating a reliable bootstrap for building from source

[Pjotr Prins]

On Sun, May 14, 2017 at 03:54:27PM -0400, Jeremiah@pdp10.guru wrote:
> > Thinking about it, what I want to achieve is that we can take the
> > latest git tree and bootstrap by building guix and packages. This
> > should be easy, since I have guix running, but it is not. And the main
> > trouble is that the underlying build packages can differ over time. I
> > am looking at gcc versions and guile versions.  I.e., we are building
> > on shifting sands. How unguixy!
> 
> It is worse than that, Janneke and I are still trying to build out a
> full source bootstrap.
> 
> Now mind you we have gotten quite a bit down that rabbit hole.
> ( I've build from a hex monitor to a Lexically scoped garbage
> compacting/collecting lisp and Janneke built his rather impressive MES
> which already supports large parts of the C language and enough to
> bootstrap some rather important pieces)
> 
> But there still are many gaps left to close (how to bootstrap a 280 byte
> hex monitor without a hex monitor or hex assembler, stage0-vm
> downstrapping, MES tinycc bootstrapping, MES lisp bootstrapping, etc)
> 
> but ultimately shifting sands are the only grounds we can be certain
> will be there.
> 
> So we better get comfortable minimizing our assumptions.

What you are trying to do is even more heroic - bootstrapping all of
guix :)

What I want is achievable, simply have the build system as a tested
binary available. A tested builder package that is know to work
against the current tree. We pretty much have it, it is only not
formalized and tested on the build farm.

Pj.
-- 

Re: Creating a reliable bootstrap for building from source

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> The combination of 'guix pull' held a promise, were it not that pull is
> also iffy. Probably for pretty much the same reason.
>
> The bootstrap+configure scripts try to work that, but actually
> address a wider case. I.e. people who want to bootstrap in Debian etc.
> I don't think we need al that. I write Makefile.guix for my projects
> and they tend to be simple! Once you can assume Guix is there life
> gets simple as a developer - except when you try to bootstrap :0
>
> The instruction I would like to write for others is:
>
> 1. Install the latest bootstrap-guix-from-source package after a guix pull
> 2. git clone guix && cd guix
> 3. run make -f Makefile.guix  
>
> (no configure is needed in guix!)
>
> 4. ./pre-inst guix etc. etc.

I think there are two very different use cases.

As a user I want something like ‘apt-get update’, which is what ‘guix
pull’ tries to do.

For Guix developers, I think it’s reasonable to have a traditional GNU
build system.  After all, Guix is also a regular software package that
people can build from source with “./configure && make && make install”.

My 2¢,
Ludo’.

Re: Creating a reliable bootstrap for building from source

[Jeremiah]

> What you are trying to do is even more heroic - bootstrapping all of
> guix :)

The real heros are those that check our work, report bugs and give us
insight into solving our most difficult problems.

And we LOVE patches and pull requests ;-D

> What I want is achievable, simply have the build system as a tested
> binary available. A tested builder package that is know to work
> against the current tree. We pretty much have it, it is only not
> formalized and tested on the build farm

If what you want is achievable and going to help, then please do it.
No need to ask, we always love people who take time to make things
better.

-Jeremiah

Re: Creating a reliable bootstrap for building from source

[David Pirotte]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

Heya,

> As a user I want something like ‘apt-get update’, which is what ‘guix
> pull’ tries to do.

For end-users, cool stuff would be:

	nguix	[ guix-ncurse

		u	[ update the aptitude cached list of all packages,
			[ installed or not... just package descriptions, nothing
			[ has been downloaded yet
		U	[ mark as 'wish to update'
		g	[ list the above marked for update packages, then fine tune
			[ using =, -, + ...
		g	ok, download what I really selected to be updated
			and update...

		note that the most important aptitude ncurse menu is 'Cancel pending
		actions' :)

	or using emacs, 'à la magit'

	M-x guix-status
		u
		U
		...

> For Guix developers, I think it’s reasonable to have a traditional GNU
> build system.  After all, Guix is also a regular software package that
> people can build from source with “./configure && make && make install”.

If I could just grab guile-gnutls and  “./configure && make && make install”
then I could compile, play, use, learn, contribute with/to guix...

Cheers,
David

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Creating a reliable bootstrap for building from source

[Pjotr Prins]

On Sun, May 14, 2017 at 11:32:06PM +0200, Ludovic Court??s wrote:
> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
> 
> > The combination of 'guix pull' held a promise, were it not that pull is
> > also iffy. Probably for pretty much the same reason.
> >
> > The bootstrap+configure scripts try to work that, but actually
> > address a wider case. I.e. people who want to bootstrap in Debian etc.
> > I don't think we need al that. I write Makefile.guix for my projects
> > and they tend to be simple! Once you can assume Guix is there life
> > gets simple as a developer - except when you try to bootstrap :0
> >
> > The instruction I would like to write for others is:
> >
> > 1. Install the latest bootstrap-guix-from-source package after a guix pull
> > 2. git clone guix && cd guix
> > 3. run make -f Makefile.guix  
> >
> > (no configure is needed in guix!)
> >
> > 4. ./pre-inst guix etc. etc.
> 
> I think there are two very different use cases.
> 
> As a user I want something like 'apt-get update', which is what 'guix
> pull' tries to do.

Sure. But from my previous E-mail you can see we are effectively using
pull to bootstrap the source tree build.

> For Guix developers, I think it's reasonable to have a traditional GNU
> build system.  After all, Guix is also a regular software package that
> people can build from source with './configure && make && make install'.

My point is that we can simplify. I like simple. Simple is good.

We can have both the configure and a simple Makefile.guix option. That
is what I do with my projects.

We do not need bootstrap, autoconf and configure on a running Guix
system. We do need it for other distributions.

Anyway, feel free to ignore this idea.

Pj.
-- 

Re: Creating a reliable bootstrap for building from source

[Ludovic Courtès]

Hi David,

David Pirotte <david@altosw.be> skribis:

>> As a user I want something like ‘apt-get update’, which is what ‘guix
>> pull’ tries to do.
>
> For end-users, cool stuff would be:
>
> 	nguix	[ guix-ncurse
>
> 		u	[ update the aptitude cached list of all packages,
> 			[ installed or not... just package descriptions, nothing
> 			[ has been downloaded yet
> 		U	[ mark as 'wish to update'
> 		g	[ list the above marked for update packages, then fine tune
> 			[ using =, -, + ...
> 		g	ok, download what I really selected to be updated
> 			and update...
>
> 		note that the most important aptitude ncurse menu is 'Cancel pending
> 		actions' :)
>
> 	or using emacs, 'à la magit'
>
> 	M-x guix-status
> 		u
> 		U
> 		...

How would that differ from what Emacs-Guix provides?

>> For Guix developers, I think it’s reasonable to have a traditional GNU
>> build system.  After all, Guix is also a regular software package that
>> people can build from source with “./configure && make && make install”.
>
> If I could just grab guile-gnutls and  “./configure && make && make install”
> then I could compile, play, use, learn, contribute with/to guix...

David, if your distro does not provide a guile-gnutls package, you can
always install Guix via the binary installation method:

  https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html

HTH!

Ludo’.

Re: Creating a reliable bootstrap for building from source

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Sun, May 14, 2017 at 11:32:06PM +0200, Ludovic Court??s wrote:
>> Pjotr Prins <pjotr.public12@thebird.nl> skribis:
>> 
>> > The combination of 'guix pull' held a promise, were it not that pull is
>> > also iffy. Probably for pretty much the same reason.
>> >
>> > The bootstrap+configure scripts try to work that, but actually
>> > address a wider case. I.e. people who want to bootstrap in Debian etc.
>> > I don't think we need al that. I write Makefile.guix for my projects
>> > and they tend to be simple! Once you can assume Guix is there life
>> > gets simple as a developer - except when you try to bootstrap :0
>> >
>> > The instruction I would like to write for others is:
>> >
>> > 1. Install the latest bootstrap-guix-from-source package after a guix pull
>> > 2. git clone guix && cd guix
>> > 3. run make -f Makefile.guix  
>> >
>> > (no configure is needed in guix!)
>> >
>> > 4. ./pre-inst guix etc. etc.
>> 
>> I think there are two very different use cases.
>> 
>> As a user I want something like 'apt-get update', which is what 'guix
>> pull' tries to do.
>
> Sure. But from my previous E-mail you can see we are effectively using
> pull to bootstrap the source tree build.

Sure.

>> For Guix developers, I think it's reasonable to have a traditional GNU
>> build system.  After all, Guix is also a regular software package that
>> people can build from source with './configure && make && make install'.
>
> My point is that we can simplify. I like simple. Simple is good.
>
> We can have both the configure and a simple Makefile.guix option. That
> is what I do with my projects.
>
> We do not need bootstrap, autoconf and configure on a running Guix
> system. We do need it for other distributions.

I agree.  build-aux/build-self.scm, which is what ‘guix pull’ runs to
build Guix, is close to what you’re suggesting, IIUC: a pure-Guile build
script.  WDYT?

Ludo’.

Re: Creating a reliable bootstrap for building from source

[Pjotr Prins]

On Mon, May 15, 2017 at 03:28:58PM +0200, Ludovic Court??s wrote:
> >> For Guix developers, I think it's reasonable to have a traditional GNU
> >> build system.  After all, Guix is also a regular software package that
> >> people can build from source with './configure && make && make install'.
> >
> > My point is that we can simplify. I like simple. Simple is good.
> >
> > We can have both the configure and a simple Makefile.guix option. That
> > is what I do with my projects.
> >
> > We do not need bootstrap, autoconf and configure on a running Guix
> > system. We do need it for other distributions.
> 
> I agree.  build-aux/build-self.scm, which is what ???guix pull??? runs to
> build Guix, is close to what you???re suggesting, IIUC: a pure-Guile build
> script.  WDYT?

It is interesting - especially the hoops jumping around guile
versioning ;) - but it still looks like it invokes the traditional
gnu-build-system. I think we can do without that too. 

Pj.
-- 

Re: Creating a reliable bootstrap for building from source

[Pjotr Prins]

On Sun, May 14, 2017 at 11:32:06PM +0200, Ludovic Court??s wrote:
> For Guix developers, I think it???s reasonable to have a traditional GNU
> build system.  After all, Guix is also a regular software package that
> people can build from source with ???./configure && make && make install???.

This is the procedure I have now for creating a reliable bootstrap
from source:

The safest route is by using guix environment after starting
a clean shell

  screen -S guix-build # I tend to build in screen
  env -i /bin/bash --login --noprofile --norc
  ~/.guix-profile/bin/guix environment guix --ad-hoc help2man git strace \
    pkg-config less vim binutils coreutils grep --no-grafts
  bash # you may want this shell

In fact pick the most recent guix you have got, see 'ls
/gnu/store/*guix*/bin/guix' and run that command. Use the --no-grafts
switch if you have built packages that way before.

Note that you can start guix by installing the binary tar ball, or
copying it from another machine using the rather useful guix archive
or [[https://www.gnu.org/software/guix/news/creating-bundles-with-guix-pack.html][guix pack]] commands.

You may want to take a note of these running versions

  gcc --version
  guile --version

Next in the source tree

  rm -rf autom4te.cache/ # to be sure
  make clean
  ./bootstrap
  ./configure --localstatedir=/var
  make clean # to be really sure
  time make

It works reliably on all my setups :). Note that typically gcc is v4 or v5 and guile 2.0.x,
but never the same.

I am pleased with this until it stops working ... I think it should go in the manual and
we should have an automated test on this procedure.

Pj.
-- 

Re: Creating a reliable bootstrap for building from source

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Mon, May 15, 2017 at 03:28:58PM +0200, Ludovic Court??s wrote:
>> >> For Guix developers, I think it's reasonable to have a traditional GNU
>> >> build system.  After all, Guix is also a regular software package that
>> >> people can build from source with './configure && make && make install'.
>> >
>> > My point is that we can simplify. I like simple. Simple is good.
>> >
>> > We can have both the configure and a simple Makefile.guix option. That
>> > is what I do with my projects.
>> >
>> > We do not need bootstrap, autoconf and configure on a running Guix
>> > system. We do need it for other distributions.
>> 
>> I agree.  build-aux/build-self.scm, which is what ???guix pull??? runs to
>> build Guix, is close to what you???re suggesting, IIUC: a pure-Guile build
>> script.  WDYT?
>
> It is interesting - especially the hoops jumping around guile
> versioning ;) - but it still looks like it invokes the traditional
> gnu-build-system.

No, (guix scripts pull) really just traverses the list of .scm files and
compiles them.

Ludo’.