installing a mirror for offline installation

installing a mirror for offline installation

[Quiliro]

I finally could install bare-bones.scm GuixSD working machine. Now i am interested in installing a mirror [of hydra] for offline installation. There is no telecommunications where I live. So I could bring the server to download substitutes and then take it back to my network for local download. I was told that an nginx mirror of hydra would cache only substitutes after they have been requested through it. Nevertheless, for those that need that, here is the config:
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf

Since that solution is not for offline use, please suggest what I should do. 

-- 

Re: installing a mirror for offline installation

[Ricardo Wurmus]

Quiliro <quiliro@riseup.net> writes:

> I finally could install bare-bones.scm GuixSD working machine. Now i
> am interested in installing a mirror [of hydra] for offline
> installation. There is no telecommunications where I live. So I could
> bring the server to download substitutes and then take it back to my
> network for local download. I was told that an nginx mirror of hydra
> would cache only substitutes after they have been requested through
> it.

This is a little complicated.  We build software continuously with
Hydra, so what you want would only really work for one particular
version of Guix.

Hydra provides binaries not only for a single version of Guix, so you
cannot just copy everything.  What you *can* do, however, is this:

* provision a portable computer with lots of disk space

* check out the version of Guix you plan to install elsewhere

* build *every* package that you know you will need at the remote site.
  (This could be done by building a system configuration, for example.)
  This will lead to every needed store item to be in your local
  /gnu/store.  Note that you don’t have to build everything from source;
  you can download substitutes for most things.

* set up “guix publish” to share items from your store with others.
  Also make sure to export your local key and authorize it on the
  machines that should install Guix “offline”.

* On the machines that are to be installed override the substitute
  server to be the local address of your portable computer running “guix
  publish”.  This could be a local IP address.

* Make sure that the machines use the exact same version of Guix,
  e.g. the latest release.

As the machines are installed they will download substitutes from the
portable computer instead of trying to go online to talk to hydra.
Since the portable computer has all needed binaries in its store,
nothing has to be built from source.

All you need for this on the remote offline site is a LAN where your
portable computer acts as a substitute server.

Does this make sense?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: installing a mirror for offline installation

[Quiliro]

El Fri, 26 May 2017 13:20:36 +0200
Ricardo Wurmus <rekado@elephly.net> escribió:

> 
> Quiliro <quiliro@riseup.net> writes:
> 
> > I finally could install bare-bones.scm GuixSD working machine. Now i
> > am interested in installing a mirror [of hydra] for offline
> > installation. There is no telecommunications where I live. So I could
> > bring the server to download substitutes and then take it back to my
> > network for local download. I was told that an nginx mirror of hydra
> > would cache only substitutes after they have been requested through
> > it.
> 
> This is a little complicated.  We build software continuously with
> Hydra, so what you want would only really work for one particular
> version of Guix.
> 
> Hydra provides binaries not only for a single version of Guix, so you
> cannot just copy everything.  What you *can* do, however, is this:
> 
> * provision a portable computer with lots of disk space

Will 100GB be enough? 1TB?

> * check out the version of Guix you plan to install elsewhere
> 
> * build *every* package that you know you will need at the remote site.
>   (This could be done by building a system configuration, for example.)

I have no idea how to do this. I know how to install a group of packages to make a desktop for example. But is there a group where all packages are installed? Could there be a conflict where one package is not downloaded because it will conflict with another? (I know this is usually not the case in GuixSD but all cases have been considered?)

>   This will lead to every needed store item to be in your local
>   /gnu/store.  Note that you don’t have to build everything from source;
>   you can download substitutes for most things.
> 
> * set up “guix publish” to share items from your store with others.
>   Also make sure to export your local key and authorize it on the
>   machines that should install Guix “offline”.
> 
> * On the machines that are to be installed override the substitute
>   server to be the local address of your portable computer running “guix
>   publish”.  This could be a local IP address.
> 
> * Make sure that the machines use the exact same version of Guix,
>   e.g. the latest release.
> 
> As the machines are installed they will download substitutes from the
> portable computer instead of trying to go online to talk to hydra.
> Since the portable computer has all needed binaries in its store,
> nothing has to be built from source.
> 
> All you need for this on the remote offline site is a LAN where your
> portable computer acts as a substitute server.
> 
> Does this make sense?

It does. Thank you.

Re: installing a mirror for offline installation

[Ricardo Wurmus]

Quiliro <quiliro@riseup.net> writes:

> El Fri, 26 May 2017 13:20:36 +0200
> Ricardo Wurmus <rekado@elephly.net> escribió:
>
>>
>> Quiliro <quiliro@riseup.net> writes:
>>
>> > I finally could install bare-bones.scm GuixSD working machine. Now i
>> > am interested in installing a mirror [of hydra] for offline
>> > installation. There is no telecommunications where I live. So I could
>> > bring the server to download substitutes and then take it back to my
>> > network for local download. I was told that an nginx mirror of hydra
>> > would cache only substitutes after they have been requested through
>> > it.
>>
>> This is a little complicated.  We build software continuously with
>> Hydra, so what you want would only really work for one particular
>> version of Guix.
>>
>> Hydra provides binaries not only for a single version of Guix, so you
>> cannot just copy everything.  What you *can* do, however, is this:
>>
>> * provision a portable computer with lots of disk space
>
> Will 100GB be enough? 1TB?

Do you need *all* packages?  Note that it would only be for *one*
version of Guix, so you’d have to do this again if you want to update
the machines at some point.

>> * check out the version of Guix you plan to install elsewhere
>>
>> * build *every* package that you know you will need at the remote site.
>>   (This could be done by building a system configuration, for example.)
>
> I have no idea how to do this. I know how to install a group of
> packages to make a desktop for example. But is there a group where all
> packages are installed? Could there be a conflict where one package is
> not downloaded because it will conflict with another? (I know this is
> usually not the case in GuixSD but all cases have been considered?)

There will not be any conflicts.  You don’t install these packages into
a profile, you just add them to the store where every item has its own
directory.  Only when installing packages into a single profile there
can be file name conflicts.

Not all packages can actually be built.  There’s a small percentage of
packages that fails to build from source and you would probably waste a
lot of time trying to build them locally.  I don’t know if there’s an
easy way to exclude those packages and refuse to build anything locally
(maybe with a smart offloading setup).

It might be easier to just build a single complete system configuration,
export it (e.g. with “guix archive” or “guix copy”) and import it on the
other machine.  I don’t think it’s a good idea to repeatedly build and
copy store items over.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: installing a mirror for offline installation

[Quiliro]

El Fri, 26 May 2017 20:36:50 +0200
Ricardo Wurmus <rekado@elephly.net> escribió:

> > Will 100GB be enough? 1TB?
> 
> Do you need *all* packages?  Note that it would only be for *one*
> version of Guix, so you’d have to do this again if you want to update
> the machines at some point.

Yes. I will want all packages. I will work with all packages for an extended period. When I am desperate for an update, I would take the _server_ for update.

> Not all packages can actually be built.  There’s a small percentage of
> packages that fails to build from source and you would probably waste a
> lot of time trying to build them locally.  I don’t know if there’s an
> easy way to exclude those packages and refuse to build anything locally
> (maybe with a smart offloading setup).

How is this?

> It might be easier to just build a single complete system configuration,
> export it (e.g. with “guix archive” or “guix copy”) and import it on the
> other machine.

I cannot know what packages will interest me or others in the network until the next update.

> I don’t think it’s a good idea to repeatedly build and
> copy store items over.

It could be updated every 2 months.

Why is it necesary to build? I only need to download.

-- 
Example of the problems in top posting:

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I leave quotations after my reply?

Saluton,
Quiliro
0987631031

Re: installing a mirror for offline installation

[Ricardo Wurmus]

(+ help-guix)

Quiliro <quiliro@riseup.net> writes:

> Yes. I will want all packages. I will work with all packages for an
> extended period. When I am desperate for an update, I would take the
> _server_ for update.

Okay.  Maybe it would be better then to set up an instance of cuirass on
your portable server and specifically enable the use of substitutes.  It
would build / download all packages according to its configuration.  You
can limit the configuration such that it only provides binaries for your
desired architecture.

I must admit, though, that I’ve been trying to do something just like
that, but I haven’t fully succeeded yet, because Cuirass doesn’t handle
errors well.  But it wouldn’t hurt to have more users and contributors
to Cuirass.

>> Not all packages can actually be built.  There’s a small percentage of
>> packages that fails to build from source and you would probably waste a
>> lot of time trying to build them locally.  I don’t know if there’s an
>> easy way to exclude those packages and refuse to build anything locally
>> (maybe with a smart offloading setup).
>
> How is this?

Not all software builds at all time.  That’s pretty normal.  All
software has bugs and some software requires patching before it can be
built with particular versions of other pieces of software.  That’s what
we’re working on collectively as package maintainers.

>> I don’t think it’s a good idea to repeatedly build and
>> copy store items over.
>
> It could be done every 2 months.

If the machines are not connected to the internet this is okay.  If they
are used to connect to the internet, however, it would be a bad idea to
let them use outdated software.  Patches come in at irregular intervals
that are shorter than a 2 month period.

Hardly a week goes by without some new vulnerability.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: installing a mirror for offline installation

[Quiliro]

El Thu, 25 May 2017 18:19:02 -0500
Quiliro <quiliro@riseup.net> escribió:

> I finally could install bare-bones.scm GuixSD working machine. Now i am interested in installing a mirror [of hydra] for offline installation. There is no telecommunications where I live. So I could bring the server to download substitutes and then take it back to my network for local download. I was told that an nginx mirror of hydra would cache only substitutes after they have been requested through it. Nevertheless, for those that need that, here is the config:
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf
> 
> Since that solution is not for offline use, please suggest what I should do. 
> 

I would still want to test installing a hydra mirror. Is there a howto for this?

-- 
Saluton,
Quiliro
0987631031

Ekzemploj de la problemo de sur-respondigi:

R: Ĉar ĝi malutilas la ordon ke homoj legas la tekston normale.
> D: Kial estas sur-respondi tiel malbone?
>> R: Sur-respondi.
>>> D: Kio estas la pli ĝenanto en poŝto elektronika?

R: Ne.
> D: Ĉu mi devas lasi komentariojn post mia respondo?


Ejemplos de los problemas con el top-posting:

Respuesta: Porque desordena la forma en la que las personas normalmente leen texto.
> Pregunta: ¿Por qué es tan malo el top-posting?
>> Respuesta: Top-posting.
>>> Pregunta: ¿Qué es lo más molesto en el correo electrónico?

Respuesta: No.
> Pregunta: ¿Debo dejar las citas luego de mi respuesta?


Examples of the problems in top posting:

A: Because it messes up the order in which people normally read text.
>Q: Why is top-posting such a bad thing?
>>A: Top-posting.
>>>Q: What is the most annoying thing in e-mail?

A: No.
> Q: Should I leave quotations after my reply?

Re: installing a mirror for offline installation

[Quiliro]

El Fri, 26 May 2017 21:57:30 +0200
Ricardo Wurmus <rekado@elephly.net> escribió:

> (+ help-guix)
> 
> Quiliro <quiliro@riseup.net> writes:
> 
> > Yes. I will want all packages. I will work with all packages for an
> > extended period. When I am desperate for an update, I would take the
> > _server_ for update.
> 
> Okay.  Maybe it would be better then to set up an instance of cuirass on
> your portable server and specifically enable the use of substitutes.  It
> would build / download all packages according to its configuration.  You
> can limit the configuration such that it only provides binaries for your
> desired architecture.
> 
> I must admit, though, that I’ve been trying to do something just like
> that, but I haven’t fully succeeded yet, because Cuirass doesn’t handle
> errors well.  But it wouldn’t hurt to have more users and contributors
> to Cuirass.

Is there a how to? I will be online in about 42 hours. I can bring my "portable" desktop case which is going to be my server. I can stay for about 8 hours.

> >> I don’t think it’s a good idea to repeatedly build and
> >> copy store items over.
> >
> > It could be done every 2 months.
> 
> If the machines are not connected to the internet this is okay.

Machines will not be connected to at all. They will connect only to the local network. Only the "portable" server will be online every month or two.

Re: installing a mirror for offline installation

[Ricardo Wurmus]

Quiliro <quiliro@riseup.net> writes:

> El Thu, 25 May 2017 18:19:02 -0500
> Quiliro <quiliro@riseup.net> escribió:
>
>> I finally could install bare-bones.scm GuixSD working machine. Now i am interested in installing a mirror [of hydra] for offline installation. There is no telecommunications where I live. So I could bring the server to download substitutes and then take it back to my network for local download. I was told that an nginx mirror of hydra would cache only substitutes after they have been requested through it. Nevertheless, for those that need that, here is the config:
>> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf
>>
>> Since that solution is not for offline use, please suggest what I should do.
>>
>
> I would still want to test installing a hydra mirror. Is there a howto for this?

I have already replied to your previous email on the 26th.  Have you
tried those suggestions?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: installing a mirror for offline installation

[Quiliro]

El Mon, 5 Jun 2017 14:41:00 -0500
Quiliro <quiliro@riseup.net> escribió:

> El Fri, 26 May 2017 21:57:30 +0200
> Ricardo Wurmus <rekado@elephly.net> escribió:
> 
> > (+ help-guix)
> > 
> > Quiliro <quiliro@riseup.net> writes:
> > 
> > > Yes. I will want all packages. I will work with all packages for an
> > > extended period. When I am desperate for an update, I would take the
> > > _server_ for update.
> > 
> > Okay.  Maybe it would be better then to set up an instance of cuirass on
> > your portable server and specifically enable the use of substitutes.  It
> > would build / download all packages according to its configuration.  You
> > can limit the configuration such that it only provides binaries for your
> > desired architecture.
> > 
> > I must admit, though, that I’ve been trying to do something just like
> > that, but I haven’t fully succeeded yet, because Cuirass doesn’t handle
> > errors well.  But it wouldn’t hurt to have more users and contributors
> > to Cuirass.
> 
> Is there a how to? I will be online in about 42 hours. I can bring my "portable" desktop case which is going to be my server. I can stay for about 8 hours.
> 
> > >> I don’t think it’s a good idea to repeatedly build and
> > >> copy store items over.
> > >
> > > It could be done every 2 months.
> > 
> > If the machines are not connected to the internet this is okay.
> 
> Machines will not be connected to at all. They will connect only to the local network. Only the "portable" server will be online every month or two.
> 

I am currently online. Will stay for about 6 or 7 hours. Is it possible to test?

-- 
Saluton,
Quiliro
0987631031

Ekzemploj de la problemo de sur-respondigi:

R: Ĉar ĝi malutilas la ordon ke homoj legas la tekston normale.
> D: Kial estas sur-respondi tiel malbone?
>> R: Sur-respondi.
>>> D: Kio estas la pli ĝenanto en poŝto elektronika?

R: Ne.
> D: Ĉu mi devas lasi komentariojn post mia respondo?


Ejemplos de los problemas con el top-posting:

Respuesta: Porque desordena la forma en la que las personas normalmente leen texto.
> Pregunta: ¿Por qué es tan malo el top-posting?
>> Respuesta: Top-posting.
>>> Pregunta: ¿Qué es lo más molesto en el correo electrónico?

Respuesta: No.
> Pregunta: ¿Debo dejar las citas luego de mi respuesta?


Examples of the problems in top posting:

A: Because it messes up the order in which people normally read text.
>Q: Why is top-posting such a bad thing?
>>A: Top-posting.
>>>Q: What is the most annoying thing in e-mail?

A: No.
> Q: Should I leave quotations after my reply?

Re: installing a mirror for offline installation

[Quiliro]

El Fri, 26 May 2017 21:57:30 +0200
Ricardo Wurmus <rekado@elephly.net> escribió:

> (+ help-guix)
> 
> Quiliro <quiliro@riseup.net> writes:
> 
> > Yes. I will want all packages. I will work with all packages for an
> > extended period. When I am desperate for an update, I would take the
> > _server_ for update.
> 
> Okay.  Maybe it would be better then to set up an instance of cuirass on
> your portable server and specifically enable the use of substitutes.  It
> would build / download all packages according to its configuration.  You
> can limit the configuration such that it only provides binaries for your
> desired architecture.
> 
> I must admit, though, that I’ve been trying to do something just like
> that, but I haven’t fully succeeded yet, because Cuirass doesn’t handle
> errors well.  But it wouldn’t hurt to have more users and contributors
> to Cuirass.

OK. Is there a configuration file to set up this hydra mirror with Cuirass?

-- 
Saluton,
Quiliro
0987631031

Ekzemploj de la problemo de sur-respondigi:

R: Ĉar ĝi malutilas la ordon ke homoj legas la tekston normale.
> D: Kial estas sur-respondi tiel malbone?
>> R: Sur-respondi.
>>> D: Kio estas la pli ĝenanto en poŝto elektronika?

R: Ne.
> D: Ĉu mi devas lasi komentariojn post mia respondo?


Ejemplos de los problemas con el top-posting:

Respuesta: Porque desordena la forma en la que las personas normalmente leen texto.
> Pregunta: ¿Por qué es tan malo el top-posting?
>> Respuesta: Top-posting.
>>> Pregunta: ¿Qué es lo más molesto en el correo electrónico?

Respuesta: No.
> Pregunta: ¿Debo dejar las citas luego de mi respuesta?


Examples of the problems in top posting:

A: Because it messes up the order in which people normally read text.
>Q: Why is top-posting such a bad thing?
>>A: Top-posting.
>>>Q: What is the most annoying thing in e-mail?

A: No.
> Q: Should I leave quotations after my reply?

Re: installing a mirror for offline installation

[ng0]

Quiliro transcribed 1.8K bytes:
> El Thu, 25 May 2017 18:19:02 -0500
> Quiliro <quiliro@riseup.net> escribió:
> 
> > I finally could install bare-bones.scm GuixSD working machine. Now i am interested in installing a mirror [of hydra] for offline installation. There is no telecommunications where I live. So I could bring the server to download substitutes and then take it back to my network for local download. I was told that an nginx mirror of hydra would cache only substitutes after they have been requested through it. Nevertheless, for those that need that, here is the config:
> > https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf
> > 
> > Since that solution is not for offline use, please suggest what I should do. 
> > 
> 
> I would still want to test installing a hydra mirror. Is there a howto for this?
> 

It's quiet bad that this low connectivity case, which is not that rare,
is not included. But we can work on it. In the meantime, you mentioned you
have one place with connectivity.
Could you create a mid-, short-, or long-distance mesh network with
it so that you reach it? Similar to what Freifunk with BATMAN does
and other international meshnetwork communities.

I think I know at least one more person (besides myself) who's
interested in solving problems like these.
-- 
ng0
OpenPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
https://krosos.org/~/ng0/ https://www.infotropique.org

Re: installing a mirror for offline installation

[Quiliro]

El Fri, 16 Jun 2017 17:05:39 +0000
ng0 <ng0@infotropique.org> escribió:

> Quiliro transcribed 1.8K bytes:
> > El Thu, 25 May 2017 18:19:02 -0500
> > Quiliro <quiliro@riseup.net> escribió:
> > 
> > > I finally could install bare-bones.scm GuixSD working machine. Now i am interested in installing a mirror [of hydra] for offline installation. There is no telecommunications where I live. So I could bring the server to download substitutes and then take it back to my network for local download. I was told that an nginx mirror of hydra would cache only substitutes after they have been requested through it. Nevertheless, for those that need that, here is the config:
> > > https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/mirror.conf
> > > 
> > > Since that solution is not for offline use, please suggest what I should do. 
> > > 
> > 
> > I would still want to test installing a hydra mirror. Is there a howto for this?
> > 
> 
> It's quiet bad that this low connectivity case, which is not that rare,
> is not included. But we can work on it. In the meantime, you mentioned you
> have one place with connectivity.
> Could you create a mid-, short-, or long-distance mesh network with
> it so that you reach it? Similar to what Freifunk with BATMAN does
> and other international meshnetwork communities.
> 
> I think I know at least one more person (besides myself) who's
> interested in solving problems like these.

There is no connection possibility on the remote site. The only solution is to take a device back and forth from the connected site to the non-connected site.

-- 
Saluton,
Quiliro
0987631031

Ekzemploj de la problemo de sur-respondigi:

R: Ĉar ĝi malutilas la ordon ke homoj legas la tekston normale.
> D: Kial estas sur-respondi tiel malbone?
>> R: Sur-respondi.
>>> D: Kio estas la pli ĝenanto en poŝto elektronika?

R: Ne.
> D: Ĉu mi devas lasi komentariojn post mia respondo?


Ejemplos de los problemas con el top-posting:

Respuesta: Porque desordena la forma en la que las personas normalmente leen texto.
> Pregunta: ¿Por qué es tan malo el top-posting?
>> Respuesta: Top-posting.
>>> Pregunta: ¿Qué es lo más molesto en el correo electrónico?

Respuesta: No.
> Pregunta: ¿Debo dejar las citas luego de mi respuesta?


Examples of the problems in top posting:

A: Because it messes up the order in which people normally read text.
>Q: Why is top-posting such a bad thing?
>>A: Top-posting.
>>>Q: What is the most annoying thing in e-mail?

A: No.
> Q: Should I leave quotations after my reply?