Re: How to create a substitute server with substitutes that failed?

Re: How to create a substitute server with substitutes that failed?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]


Hi,

I’d like to create a small server that provides substitutes for packages
for which I often have to build a fallback (mostly ungoogled Chromium).

How can I do that? Is there already a simply way?

I tried guix copy, but without luck:

$ LANG=C guix copy ungoogled-chromium --to=guixtmp@localhost
guix copy: error: failed to connect to `#<input-output: channel (open)7effb4657a80>': Protocol error

What I want is just to put the required substitute archives on a server
for others to download. Ungoogled Chromium takes so long that only one
person should ever have to build it (and I need it for work, so I can be
that one person).

It would be ideal if I could put it on a dumb public host with only
HTTPS access and FTP for upload (since I already have that and it has
unlimited bandwidth).

Best wishes,
Arne


-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: How to create a substitute server with substitutes that failed?

[Julien Lepiller]

The best would be to run "guix publish" on your server, as it will allow others to use your server as a substitute server, instead of having to manually download archives of the package and dependencies.

guix copy copies an item from one store to the other, so it doesn't make sense to send from localhost to localhost :). Maybe you wanted to use guix archive instead, to create an archive that can later be imported?

Le 26 août 2020 07:18:15 GMT-04:00, "Dr. Arne Babenhauserheide" <arne_bab@web.de> a écrit :
>
>Hi,
>
>I’d like to create a small server that provides substitutes for
>packages
>for which I often have to build a fallback (mostly ungoogled Chromium).
>
>How can I do that? Is there already a simply way?
>
>I tried guix copy, but without luck:
>
>$ LANG=C guix copy ungoogled-chromium --to=guixtmp@localhost
>guix copy: error: failed to connect to `#<input-output: channel
>(open)7effb4657a80>': Protocol error
>
>What I want is just to put the required substitute archives on a server
>for others to download. Ungoogled Chromium takes so long that only one
>person should ever have to build it (and I need it for work, so I can
>be
>that one person).
>
>It would be ideal if I could put it on a dumb public host with only
>HTTPS access and FTP for upload (since I already have that and it has
>unlimited bandwidth).
>
>Best wishes,
>Arne
>
>
>-- 
>Unpolitisch sein
>heißt politisch sein
>ohne es zu merken

Re: How to create a substitute server with substitutes that failed?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

Julien Lepiller <julien@lepiller.eu> writes:
> The best would be to run "guix publish" on your server, as it will allow others to use your server as a substitute server, instead of having to manually download archives of the package and dependencies.
Thank you for your answer!

Is it possible to do the equivalent of guix publish on a dumb server?

The webhosts I have are pretty dumb servers that definitely have no
Guix, and I have no root-access (and currently don’t want it, because I
don’t like the maintenance burden that comes with it).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: How to create a substitute server with substitutes that failed?

[Julien Lepiller]

I suppose, but that's going to be a bit of work. I'm not very knowledgeable on this part of guix. I think you'll need to generate a narinfo and a nar for each store item you want to be available. Maybe you can run guix publish locally and mirror the files it serves on your server?

You'll need to publish /<store hash of output>.narifo and the other files it references (the nar archives themselves).

So you'd download them from localhost and send them to your server. Your guix daemon must trust the builder's public key, or the substitute will not be downloaded.

Le 26 août 2020 10:56:26 GMT-04:00, "Dr. Arne Babenhauserheide" <arne_bab@web.de> a écrit :
>Julien Lepiller <julien@lepiller.eu> writes:
>> The best would be to run "guix publish" on your server, as it will
>allow others to use your server as a substitute server, instead of
>having to manually download archives of the package and dependencies.
>Thank you for your answer!
>
>Is it possible to do the equivalent of guix publish on a dumb server?
>
>The webhosts I have are pretty dumb servers that definitely have no
>Guix, and I have no root-access (and currently don’t want it, because I
>don’t like the maintenance burden that comes with it).
>
>Best wishes,
>Arne
>-- 
>Unpolitisch sein
>heißt politisch sein
>ohne es zu merken

Re: How to create a substitute server with substitutes that failed?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1529 bytes --]


Julien Lepiller <julien@lepiller.eu> writes:

> I suppose, but that's going to be a bit of work. I'm not very knowledgeable on this part of guix. I think you'll need to generate a narinfo and a nar for each store item you want to be available. Maybe you can run guix publish locally and mirror the files it serves on your server?

That sounds like a viable option.

> You'll need to publish /<store hash of output>.narifo and the other files it references (the nar archives themselves).

So I only need a way to get all the hashes referenced for a given
package. Do you know the command for that?

> So you'd download them from localhost and send them to your server. Your guix daemon must trust the builder's public key, or the substitute will not be downloaded.
>
> Le 26 août 2020 10:56:26 GMT-04:00, "Dr. Arne Babenhauserheide" <arne_bab@web.de> a écrit :
>>Julien Lepiller <julien@lepiller.eu> writes:
>>> The best would be to run "guix publish" on your server, as it will
>>allow others to use your server as a substitute server, instead of
>>having to manually download archives of the package and dependencies.
>>Thank you for your answer!
>>
>>Is it possible to do the equivalent of guix publish on a dumb server?
>>
>>The webhosts I have are pretty dumb servers that definitely have no
>>Guix, and I have no root-access (and currently don’t want it, because I
>>don’t like the maintenance burden that comes with it).


-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: How to create a substitute server with substitutes that failed?

[Julien Lepiller]

guix build <package> will output all the store names for the outputs of <package> (usually only one). The hash in the directory names is what you want as the narinfo filename.

Le 26 août 2020 12:12:44 GMT-04:00, "Dr. Arne Babenhauserheide" <arne_bab@web.de> a écrit :
>
>Julien Lepiller <julien@lepiller.eu> writes:
>
>> I suppose, but that's going to be a bit of work. I'm not very
>knowledgeable on this part of guix. I think you'll need to generate a
>narinfo and a nar for each store item you want to be available. Maybe
>you can run guix publish locally and mirror the files it serves on your
>server?
>
>That sounds like a viable option.
>
>> You'll need to publish /<store hash of output>.narifo and the other
>files it references (the nar archives themselves).
>
>So I only need a way to get all the hashes referenced for a given
>package. Do you know the command for that?
>
>> So you'd download them from localhost and send them to your server.
>Your guix daemon must trust the builder's public key, or the substitute
>will not be downloaded.
>>
>> Le 26 août 2020 10:56:26 GMT-04:00, "Dr. Arne Babenhauserheide"
><arne_bab@web.de> a écrit :
>>>Julien Lepiller <julien@lepiller.eu> writes:
>>>> The best would be to run "guix publish" on your server, as it will
>>>allow others to use your server as a substitute server, instead of
>>>having to manually download archives of the package and dependencies.
>>>Thank you for your answer!
>>>
>>>Is it possible to do the equivalent of guix publish on a dumb server?
>>>
>>>The webhosts I have are pretty dumb servers that definitely have no
>>>Guix, and I have no root-access (and currently don’t want it, because
>I
>>>don’t like the maintenance burden that comes with it).
>
>
>-- 
>Unpolitisch sein
>heißt politisch sein
>ohne es zu merken

Re: How to create a substitute server with substitutes that failed?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2317 bytes --]


Dr. Arne Babenhauserheide <arne_bab@web.de> writes:

> I’d like to create a small server that provides substitutes for
> packages for which I often have to build a fallback (mostly ungoogled
> Chromium).
>
> How can I do that? Is there already a simply way?

It's not simple (yet), but I'm hoping to get there eventually with the
Guix Build Coordinator.

1: https://git.cbaines.net/guix/build-coordinator/about/

> It would be ideal if I could put it on a dumb public host with only
> HTTPS access and FTP for upload (since I already have that and it has
> unlimited bandwidth).

Using guix publish along with something to build the packages is
simpler, providing you want to serve them from the store. However, one
of the things on my mind when designing the Guix Build Coordinator was
specifically this use case. I was actually more thinking about
scalability, I didn't want to be limited by the storage of a single
machine.

Again, it's not simple (yet), but how this works currently is through
hooks [2]. You have the Guix Build Coordinator run some Guile code when
a build is successful, and this can do something with the outputs.

2: https://git.cbaines.net/guix/build-coordinator/tree/guix-build-coordinator/hooks.scm

For https://guix.cbaines.net/ I use the build-success-s3-publish-hook as
I use a hosted S3 service for the storage of the outputs. You could very
similarly implement a build-success-ftp-publish-hook and use this to
upload nars. Like guix publish, the Guix Build Coordinator can generate
the narinfo files you need to serve substitutes.

The Guix Build Coordinator is also close (but not quite there) to being
able to easily build a subset of Guix packages in an automated
way. Currently the
guix-build-coordinator-queue-builds-from-guix-data-service [3] script
will queue builds for all packages, and the channel instance
derivations, but it could and should in the future allow controlling
what to build.

3: https://git.cbaines.net/guix/build-coordinator/tree/scripts/guix-build-coordinator-queue-builds-from-guix-data-service.in

I realise the promise of being able to do this more easily in the future
isn't that helpful, but I still wanted to mention the Guix Build
Coordinator in case it's something you're interested in.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: How to create a substitute server with substitutes that failed?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2211 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Dr. Arne Babenhauserheide <arne_bab@web.de> writes:
>
>> I’d like to create a small server that provides substitutes for
>> packages for which I often have to build a fallback (mostly ungoogled
>> Chromium).
>>
>> How can I do that? Is there already a simply way?
>
> It's not simple (yet), but I'm hoping to get there eventually with the
> Guix Build Coordinator.
>
> 1: https://git.cbaines.net/guix/build-coordinator/about/
…
> I realise the promise of being able to do this more easily in the future
> isn't that helpful, but I still wanted to mention the Guix Build
> Coordinator in case it's something you're interested in.

It sounds promising and I’m not in so much of a hurry that I can’t wait
a bit more.

Thank you!

What I need is likely much simpler, though: I need a hook which triggers
whenever a package had to be built locally. That way we could have
dynamic substitute-servers.

And seeing guix publish run locally got me wishing more that I’d already
be finished with my download-mesh implementation, because guix would be
a perfect usecase for that: We can actually add headers and people have
long-running servers. Parts of the content-addressed storage and
chunk-assembly are already done, but I’m still missing the validation of
chunks and making currently running downloads available.

Conceptually it’s simple: Users could tell guix to serve all derivations
they downloaded and tell the server to mark their guix-daemon as
content-addressed alternate source (X-Alt), then when someone downloaded
something like the 828MiB chromium derivation, the original server would
hand out a list of X-Alts, the hash for content-addressed retrieval and
a merkle-tree of chunk hashes for chunk-validation, so clients could
download from other clients without having to trust them.

The current state of my implementation of such a server and downloader
is at https://hg.sr.ht/~arnebab/wispserve/browse/wispserve/serve.w?rev=tip

And while writing this message I implemented serving while downloading.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: How to create a substitute server with substitutes that failed?

[zimoun]

Dear Arne,

On Wed, 26 Aug 2020 at 13:18, "Dr. Arne Babenhauserheide" <arne_bab@web.de> wrote:

> What I want is just to put the required substitute archives on a server
> for others to download. Ungoogled Chromium takes so long that only one
> person should ever have to build it (and I need it for work, so I can be
> that one person).

Since “guix publish” does not fit your needs, this old draft [1] using
IPFS to distribute substitutes probably neither.  But maybe your
use-case could be an interesting test for the patch set. :-)

[1]: http://issues.guix.gnu.org/issue/33899


All the best,
simon

Re: How to create a substitute server with substitutes that failed?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> On Wed, 26 Aug 2020 at 13:18, "Dr. Arne Babenhauserheide" <arne_bab@web.de> wrote:
>
>> What I want is just to put the required substitute archives on a server
>> for others to download. Ungoogled Chromium takes so long that only one
>> person should ever have to build it (and I need it for work, so I can be
>> that one person).
>
> Since “guix publish” does not fit your needs, this old draft [1] using
> IPFS to distribute substitutes probably neither.  But maybe your
> use-case could be an interesting test for the patch set. :-)
>
> [1]: http://issues.guix.gnu.org/issue/33899

It looks really interesting, but I’m surprised at the amount of
complexity that goes into that. I had thought that just putting files by
key would be easier.

So even with this I still haven’t seen anything to date that’s
conceptually simpler than the Gnutella Download-Mesh for swarming
downloads.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]