hydra.gnu.org migrates to /gnu/store!

hydra.gnu.org migrates to /gnu/store!

[Ludovic Courtès]

Hello!

hydra.gnu.org is under maintenance for a couple of days; the web
front-end remains available, but builds don’t get queued.

Among other things, it’s migrating from /nix/store to /gnu/store.
Stay tuned! :-)

Ludo’.

Re: hydra.gnu.org migrates to /gnu/store!

[Sree Harsha Totakura]

On 02/28/2014 11:28 PM, Ludovic Courtès wrote:
> Among other things, it’s migrating from /nix/store to /gnu/store.

Is this already present in the master branch?  If so, will the store be
rebuilt when I update master?

Sree

Re: hydra.gnu.org migrates to /gnu/store!

[Ludovic Courtès]

Sree Harsha Totakura <sreeharsha@totakura.in> skribis:

> On 02/28/2014 11:28 PM, Ludovic Courtès wrote:
>> Among other things, it’s migrating from /nix/store to /gnu/store.
>
> Is this already present in the master branch?

No, currently you have to explicitly run:

  ./configure --with-store-dir=/gnu/store.

Ludo’.

Re: hydra.gnu.org migrates to /gnu/store!

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:
> Among other things, it’s migrating from /nix/store to /gnu/store.
> Stay tuned! :-)

Can you give some advice on how best to transition from /nix/store to
/gnu/store on an existing Guix system?  Can $PREFIX/var/nix and the
sqlite database be shared, or do I have to start with a clean slate?
Any other suggestions?

     Thanks,
       Mark

Transition to /gnu/store

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Can you give some advice on how best to transition from /nix/store to
> /gnu/store on an existing Guix system?  Can $PREFIX/var/nix and the
> sqlite database be shared, or do I have to start with a clean slate?
> Any other suggestions?

Sorry, I was planning to mention that eventually.

(I have not yet changed the default store directory that ./configure
chooses, but will do so in the near future.)

You don’t have to migrate to /gnu/store now.  You can keep using
/nix/store on your machine.

The only possible downside is that hydra.gnu.org will no longer provide
binaries for /nix/store, so everything will have to be built locally.
Also, if you re-configure Guix, you’ll have to make sure to pass
--with-store-dir=/nix/store when the default has changed to /gnu/store.

Since moving to /gnu/store involves a full re-build or re-download, I
recommend doing that once we’ve merged core-updates (hopefully within a
couple of weeks.)


I would also like to change the database directory to $PREFIX/var/guix.
This change is more intrusive: if you want to keep using the store
database that’s under $PREFIX/var/nix, you’ll have to manually change
the value in Makefile.am.

HTH,
Ludo’.

Re: Transition to /gnu/store

[Ludovic Courtès]

So!  Commit 834129e changes the default store directory to /gnu/store.
Again, if you don’t plan to change right now, make sure to configure
with --with-store-dir=/nix/store next time!

The other change is commit f5768af, which changes the place where the
store’s database and build logs are stored.  If you don’t plan to switch
to /gnu/store right now, then do something like:

  cd $localstatedir && ln -s nix guix
  cd $localstatedir/log && ln -s nix guix

If you do plan to switch to /gnu/store now, then just run ./configure
and forget about the rest.

Ludo’.

Re: Transition to /gnu/store

[Andreas Enge]

On Mon, Mar 10, 2014 at 12:09:25AM +0100, Ludovic Courtès wrote:
> If you do plan to switch to /gnu/store now, then just run ./configure
> and forget about the rest.

Just a quick comment, you will also have to restart the guix daemon.
For instance, if you installed Sree's init script:
   /etc/init.d/guixd restart
as root.

Andreas

Re: Transition to /gnu/store

[Andreas Enge]

Would it not be the time now to merge core-updates back into master, as
anyway everything will be rebuilt?

Andreas

Re: Transition to /gnu/store

[Cyril Roelandt]

On 03/10/2014 03:18 PM, Andreas Enge wrote:
> Would it not be the time now to merge core-updates back into master, as
> anyway everything will be rebuilt?
>

It would be nice to know that a bit in advance, so we have time to push 
pending patches to core-updates before it's merged into master :)

Cyril.

Re: Transition to /gnu/store

[Thompson, David]

On Mon, Mar 10, 2014 at 10:18 AM, Andreas Enge <andreas@enge.fr> wrote:
> Would it not be the time now to merge core-updates back into master, as
> anyway everything will be rebuilt?
>
> Andreas
>

I think that we want to wait to merge core-updates until Guile 2.0.10
is released.

- Dave

Re: Transition to /gnu/store

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Would it not be the time now to merge core-updates back into master, as
> anyway everything will be rebuilt?

We need to discuss with Mark whether to release Guile 2.0.10, or just
patch it the way Cyril proposed a while back.  But that’s basically the
only thing left to figure out, yes.

Thanks,
Ludo’.

Re: Transition to /gnu/store

[Cyril Roelandt]

On 03/10/2014 09:14 PM, Ludovic Courtès wrote:
> Andreas Enge <andreas@enge.fr> skribis:
>
>> Would it not be the time now to merge core-updates back into master, as
>> anyway everything will be rebuilt?
>
> We need to discuss with Mark whether to release Guile 2.0.10, or just
> patch it the way Cyril proposed a while back.  But that’s basically the
> only thing left to figure out, yes.
>

I'd sort of want my Python patches in the next merge, too, and they are 
currently being discussed with upstream.


Cyril.

Re: Transition to /gnu/store

[Ludovic Courtès]

Cyril Roelandt <tipecaml@gmail.com> skribis:

> On 03/10/2014 09:14 PM, Ludovic Courtès wrote:
>> Andreas Enge <andreas@enge.fr> skribis:
>>
>>> Would it not be the time now to merge core-updates back into master, as
>>> anyway everything will be rebuilt?
>>
>> We need to discuss with Mark whether to release Guile 2.0.10, or just
>> patch it the way Cyril proposed a while back.  But that’s basically the
>> only thing left to figure out, yes.
>>
>
> I'd sort of want my Python patches in the next merge, too, and they
> are currently being discussed with upstream.

Right, good point.

Ludo’.

Re: Transition to /gnu/store

[Andreas Enge]

Nothing prevents us from merging core-updates several times before a release.
As I see it, core-updates is there so that people tracking git master are not
forced to recompile too often. But a change to master that forces recompilat-
ion should be linked with a merge of core-updates into master, which comes
"for free" at this moment.

Andreas

Re: Transition to /gnu/store

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Nothing prevents us from merging core-updates several times before a release.
> As I see it, core-updates is there so that people tracking git master are not
> forced to recompile too often.

Too often and too much.  Also, changes in core-updates can potentially
break more stuff.

> But a change to master that forces recompilat- ion should be linked
> with a merge of core-updates into master, which comes "for free" at
> this moment.

Right, but in master we don’t change “core” stuff like build/utils.scm,
glibc, etc.  Sometimes there are changes that trigger a lot of rebuild,
but never as much as what we do in core-updates.

Now, I agree we must find a way to avoid core-updates to last for too
long.  That probably means we need to agree on a time-based policy.

Ideally, we’d make a new release every two months (we’re late this
time), and thus have a core-updates branch living in between two
releases.

How does that sound?

Thanks,
Ludo’.

Re: Transition to /gnu/store

[Andreas Enge]

On Wed, Mar 12, 2014 at 12:06:10PM +0100, Ludovic Courtès wrote:
> Right, but in master we don’t change “core” stuff like build/utils.scm,
> glibc, etc.  Sometimes there are changes that trigger a lot of rebuild,
> but never as much as what we do in core-updates.

Sorry, but I am still lost; we just moved to /gnu/store, which forces a
complete rebuild, (with hydra not being available, which implies full
rebuilds on all guix machines individually), no? So why not merge core-updates
at the same time?

Andreas

Re: Transition to /gnu/store

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Wed, Mar 12, 2014 at 12:06:10PM +0100, Ludovic Courtès wrote:
>> Right, but in master we don’t change “core” stuff like build/utils.scm,
>> glibc, etc.  Sometimes there are changes that trigger a lot of rebuild,
>> but never as much as what we do in core-updates.
>
> Sorry, but I am still lost; we just moved to /gnu/store, which forces a
> complete rebuild, (with hydra not being available, which implies full
> rebuilds on all guix machines individually), no? So why not merge core-updates
> at the same time?

Because you don’t have to move to /gnu/store right now; you can do that
any time.  And also because there are still a couple changes pending for
core-updates anyway.

Does that clarify things?

Ludo’.

Re: Transition to /gnu/store

[Ludovic Courtès]

FWIW I’ve migrated my laptop to /gnu/store, and also switched to a
different $localstatedir (meaning that the store’s DB, build log
directory, profiles, etc. are stored in a different place.)

After that, I ended up starting from an empty profile (because the old
one was elsewhere.)  So I recreated my profile with:

  guix package -i $(guix package -I . \
    -p /nix/var/nix/profiles/per-user/ludo/guix-profile | \
    cut -f1,3 | sed -e's/\t/:/g')

where /nix/var/../guix-profile is the old profile.

Ludo’.

Re: Transition to /gnu/store

[Andreas Enge]

Currently, the substituter fails for me. For instance,
   guix build pari-gp -n
yields the following message:
download-using-manifests.pl: error: executing `/usr/local/guix-git/libexec/nix/substituters/download-using-manifests.pl': No such file or directory
guix build: error: build failed: substituter `download-using-manifests.pl' died unexpectedly

In fact, the directory /usr/local/guix-git/libexec/nix/ does not exist
(any more?). There is /usr/local/guix-git/libexec/guix/, which does not
contain the perl script.

The culprit seems to be nix/libstore/globals.cc; or the script would have
to be installed.

Andreas

Re: Transition to /gnu/store

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Currently, the substituter fails for me. For instance,
>    guix build pari-gp -n
> yields the following message:
> download-using-manifests.pl: error: executing `/usr/local/guix-git/libexec/nix/substituters/download-using-manifests.pl': No such file or directory
> guix build: error: build failed: substituter `download-using-manifests.pl' died unexpectedly

download-using-manifests.pl is a Nix thing; the equivalent in Guix in
‘guix substitute-binary’ (no .pl here ;-)).

Normally guix-daemon.cc sets the NIX_SUBSTITUTERS environment variable
so that the daemon’s code uses our substituter.

Could you make sure guix build is talking to guix-daemon (and not
nix-daemon)?

Otherwise, can you run guix build with --verbosity=20?  Here the first
line I get is:

  starting substituter program `/home/ludo/src/guix/nix/scripts/substitute-binary'

Thanks,
Ludo’.

Re: Transition to /gnu/store

[Andreas Enge]

On Thu, Mar 20, 2014 at 07:05:35PM +0100, Ludovic Courtès wrote:
> download-using-manifests.pl is a Nix thing; the equivalent in Guix in
> ‘guix substitute-binary’ (no .pl here ;-)).
> 
> Normally guix-daemon.cc sets the NIX_SUBSTITUTERS environment variable
> so that the daemon’s code uses our substituter.
> 
> Could you make sure guix build is talking to guix-daemon (and not
> nix-daemon)?

There is no nix-daemon on my machine.

> Otherwise, can you run guix build with --verbosity=20?  Here the first
> line I get is:
>   starting substituter program `/home/ludo/src/guix/nix/scripts/substitute-binary'

I get:
starting substituter program `/usr/local/guix-git/libexec/nix/substituters/download-using-manifests.pl'
download-using-manifests.pl: error: executing `/usr/local/guix-git/libexec/nix/substituters/download-using-manifests.pl': No such file or directory

"strings guix-daemon" contains /nix/substituters/download-using-manifests.pl.

I just did a "git pull; ./sync-with-upstream" in the nix subdirectory, then a
"make distclean", bootstrap, configure and make. The result is the same, there
is still a reference to
   /usr/local/guix-git/libexec/nix/substituters/download-using-manifests.pl

Andreas

Re: Transition to /gnu/store

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> I get:
> starting substituter program `/usr/local/guix-git/libexec/nix/substituters/download-using-manifests.pl'
> download-using-manifests.pl: error: executing `/usr/local/guix-git/libexec/nix/substituters/download-using-manifests.pl': No such file or directory

OK, I believe this is fixed with 89faa5c.

Can you confirm?

Thanks,
Ludo’.

Re: Transition to /gnu/store

[Andreas Enge]

On Thu, Mar 20, 2014 at 11:07:34PM +0100, Ludovic Courtès wrote:
> OK, I believe this is fixed with 89faa5c.

Indeed, it is working now. Thanks!

Andreas

Re: hydra.gnu.org migrates to /gnu/store!

[Ludovic Courtès]

I’ve restarted Hydra but it’ll be some time before it catches up.

It now builds for /gnu/store, and offloads builds to hydra.gnunet.org
(the other new build machine will come next.)  There were networking
issue with that machine so we may still get transient errors.  I’ll keep
monitoring and I hope we can fix that.

Thanks,
Ludo’.

Re: hydra.gnu.org migrates to /gnu/store!

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> I’ve restarted Hydra but it’ll be some time before it catches up.

As I was replying to Andreas I realized Hydra’s web interface was only
available through hydra.gnu.org:3000, so I’ve reinstated the iptables
rules so that hydra.gnu.org (port 80) works.

Sorry about that!

Ludo’.