guix is the guildhall that we always wanted!

guix is the guildhall that we always wanted!

[Andy Wingo]

Hi all!

I've always wondered about the right solution for getting Guile users to
share code with each other.  At one point I thought that we would have a
Guile-specific packaging system like CPAN or NPM, but one with GNU
stow-like characteristics.  We had problems with C extensions though:
how do you build one?  Where do you get the compilers?  Where do you get
the libraries?

Guix (https://guixsd.org/) solves this in a comprehensive way.  From the
four or five bootstrap binaries, Guix can download and build the world
from source, for any of its supported architectures.  The result is a
farm of weirdly-named files in /gnu/store, but the transitive closure of
a store item works on any distribution of that architecture.

I got the feeling that Guix was what the guildhall should be long ago,
but something recent made it clear to me.  For background, consider
https://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html,
the binary installation instructions for Guix.  These are directions for
how to install Guix on a "foreign" distro like Fedora.  Basically you
just untar a /gnu/store onto your system, make sure that your /var/guix
has some metadata so that Guix knows what's in the store, and there you
are.

The process of building that weird binary installation tarball was
always a bit ad-hoc though, geared to Guix's installation needs.

But it turns out that we can use the same strategy to distribute
reproducible binaries for any package that Guix includes.  Notably, if
you run:

  guix pack -S /opt/guile-fibers-1.0.0=/ guile-next guile-fibers glibc-utf8-locales

Then what you get is a tarball that has Guile, Fibers, and everything
they depend on.  It's safe to extract in / because it only adds files to
/gnu/store, and then it adds a symlink for /opt/guile-fibers-1.0.0.

Crucially, this "guix pack" is really an SDK for Guile and Fibers.
Fibers is a library.  You can now make an SDK for anything!!!!!!!!!!!  I
mean, I know, that's a lot of exclamation marks, but check it: you can
develop locally, say using

  /opt/guile-fibers-1.0.0/bin/guile -L /path/to/my/code

and it just works.  If you package up your code, you can make a Guix
package out of it and make a "guix pack" out of that closure and
distribute it to your users: REGARDLESS OF WHAT OPERATING SYSTEM THEY
RUN, AND WITH FULL SOURCE REPRODUCIBILITY.  It's huge.  It's the missing
piece of docker: not only do you get space-sharing (installing
guile-fibers-1.0.0 and my-app-built-on-guile-fibers-2.0 can share common
libraries as needed, instead of many huge image files), you also have
the keys to rebuild that image in a standard way.

So:

  * local development using packaged libraries: check.
  * distribution of your work to people without $package-manager: check.
  * easy addition of your code to a common NPM-like registry: ?

For this last bit, we have two options.  One is to add your package to
Guix.  It's relatively easy; here's what I added for Fibers:

    (define-public guile-fibers
      (package
        (name "guile-fibers")
        (version "1.0.0")
        (source (origin
                  (method url-fetch)
                  (uri (string-append "https://wingolog.org/pub/fibers/fibers-"
                                      version ".tar.gz"))
                  (sha256
                   (base32
                    "0vjkg72ghgdgphzbjz9ig8al8271rq8974viknb2r1rg4lz92ld0"))))
        (build-system gnu-build-system)
        (native-inputs
         `(("texinfo" ,texinfo)
           ("pkg-config" ,pkg-config)))
        (inputs
         `(("guile" ,guile-next)))
        (synopsis "Lightweight concurrency facility for Guile")
        (description
         "Fibers is a Guile library that implements a a lightweight concurrency
    facility, inspired by systems like Concurrent ML, Go, and Erlang.  A fiber is
    like a \"goroutine\" from the Go language: a lightweight thread-like
    abstraction.  Systems built with Fibers can scale up to millions of concurrent
    fibers, tens of thousands of concurrent socket connections, and many parallel
    cores.  The Fibers library also provides Concurrent ML-like channels for
    communication between fibers.

    Note that Fibers makes use of some Guile 2.1/2.2-specific features and
    is not available for Guile 2.0.")
        (home-page "https://github.com/wingo/fibers")
        (license license:lgpl3+)))

OK, so this uses gnu-build-system, which requires a tarball; we need to
extend this to have a "guile-build-system" for modules that are just
Scheme, in which we just "guild compile" everything.  That way you can
have a repo on gitlab or whatever and you just specify the URL and the
revision and you are done.  I don't know if we can get around specifying
the sha256 when we specify the git revision; probably not a good idea in
light of the SHA1 breakage.  Anyway, that's a thing.

But while getting your package into guix is easier than you think, it's
not automatic.  I think there's room for a middle ground that's a bit
more decentralized than Guix, but more centralized than people using
GUIX_PACKAGE_PATH to add random directories of Guix package files.

So!  My proposal for this new "guildhall" would be:

    1. a web service

    2. on which users registers projects

    3. a project is a name + a git repository with a /package.scm file

    4. the package.scm contains Guix package definitions for that project

    5. the web service administers a git repository collecting those
       packages
       - without any hydra.gnu.org overhead
       - without any manual checks
       - in a form that you can just check out once and add to GUIX_PACKAGE_PATH

    6. adding a new tag to a project's git repo of the form vX makes a
       release X and updates the guildhall package
       - it could be the web service has to poll git repos
       - or maybe you have to invoke some command to update guildhall

    7. probably we need to steal many ideas from npm.org

You might ask, why is this better than the current guildhall?  It does
exist after all.  I respond in the following way:

    1. adding projects to guildhall currently takes manual intervention

    2. guildhall does not include C extensions, much less Guile itself

    3. guildhall has no story for deploying on systems without guildhall
       or even Guile

    4. guildhall currently misses some compilation phases (AFAIU?)

Open questions would be, what about other targets like macOS or Windows
or whatever?  There I don't know.  I suspect that if Guix becomes
popular enough, someone will make a thing whereby you can cross-compile
to macOS from GNU/Linux at least.  I know you can do that already with
MinGW.  As for native "guildhall" operation on these targets, I don't
know.  But I think that is a cost that people will pay for in time.

I think there's still room and value for pushing Guildhall packages into
Guix proper.  I don't know the right relationship, but the barriers to
Guix contribution are really low so I have a feeling this should just
work out; and that otherwise, the web service can automate a lot of
review on the specific packages in consideration.  Dunno.

But!  Who is going to build the guildhall v2.2? :-)

Andy

Re: guix is the guildhall that we always wanted!

[Amirouche Boubekki]

Héllo wingo!

On 2017-03-16 19:25, Andy Wingo wrote:
> Hi all!
> 

[...]

> But it turns out that we can use the same strategy to distribute
> reproducible binaries for any package that Guix includes.  Notably, if
> you run:
> 
>   guix pack -S /opt/guile-fibers-1.0.0=/ guile-next guile-fibers
> glibc-utf8-locales
> 
> Then what you get is a tarball that has Guile, Fibers, and everything
> they depend on.  It's safe to extract in / because it only adds files 
> to
> /gnu/store, and then it adds a symlink for /opt/guile-fibers-1.0.0.
> 

cf. http://git.savannah.gnu.org/cgit/guix.git/tree/doc/guix.texi#n2392

> 
> So:
> 
>   * local development using packaged libraries: check.
>   * distribution of your work to people without $package-manager: 
> check.
>   * easy addition of your code to a common NPM-like registry: ?
> 
> For this last bit, we have two options.  One is to add your package to
> Guix.  It's relatively easy; here's what I added for Fibers:
> 
>     (define-public guile-fibers
>       (package
>         (name "guile-fibers")
>         (version "1.0.0")
>         (source (origin
>                   (method url-fetch)
>                   (uri (string-append 
> "https://wingolog.org/pub/fibers/fibers-"
>                                       version ".tar.gz"))
>                   (sha256
>                    (base32
>                     
> "0vjkg72ghgdgphzbjz9ig8al8271rq8974viknb2r1rg4lz92ld0"))))
>         (build-system gnu-build-system)
>         (native-inputs
>          `(("texinfo" ,texinfo)
>            ("pkg-config" ,pkg-config)))
>         (inputs
>          `(("guile" ,guile-next)))
>         (synopsis "Lightweight concurrency facility for Guile")
>         (description
>          "Fibers is a Guile library that implements a a lightweight 
> concurrency
>     facility, inspired by systems like Concurrent ML, Go, and Erlang.
> A fiber is
>     like a \"goroutine\" from the Go language: a lightweight 
> thread-like
>     abstraction.  Systems built with Fibers can scale up to millions
> of concurrent
>     fibers, tens of thousands of concurrent socket connections, and
> many parallel
>     cores.  The Fibers library also provides Concurrent ML-like 
> channels for
>     communication between fibers.
> 
>     Note that Fibers makes use of some Guile 2.1/2.2-specific features 
> and
>     is not available for Guile 2.0.")
>         (home-page "https://github.com/wingo/fibers")
>         (license license:lgpl3+)))
> 
> OK, so this uses gnu-build-system, which requires a tarball; we need to
> extend this to have a "guile-build-system" for modules that are just
> Scheme, in which we just "guild compile" everything.  That way you can
> have a repo on gitlab or whatever and you just specify the URL and the
> revision and you are done.  I don't know if we can get around 
> specifying
> the sha256 when we specify the git revision; probably not a good idea 
> in
> light of the SHA1 breakage.  Anyway, that's a thing.
> 
> But while getting your package into guix is easier than you think, it's
> not automatic.  I think there's room for a middle ground that's a bit
> more decentralized than Guix, but more centralized than people using
> GUIX_PACKAGE_PATH to add random directories of Guix package files.

Ok, I like the idea!

> 
> So!  My proposal for this new "guildhall" would be:
> 
>     1. a web service
> 

What would be the cli interface associated with that web service?
Some thing like the following will do:

   $ guildhall register

     Register a package against the guildhall repository.
     You must have a package.scm in the current working
     directory that follows package specification of guix.
     http://link/to/guix/doc/that/I/dont/know

Does it require login/password?

That is all I can think of.

>     2. on which users registers projects
> 
>     3. a project is a name + a git repository with a /package.scm file

We can have that using guile-git: current commit + remote origin.

>     4. the package.scm contains Guix package definitions for that 
> project
> 
>     5. the web service administers a git repository collecting those
>        packages
>        - without any hydra.gnu.org overhead

What do you mean, by no hydra.gnu.org overhead? Where are
done the 'guix pack' command?

>        - without any manual checks
>        - in a form that you can just check out once and add to 
> GUIX_PACKAGE_PATH
> 

Ok, the 'guix pack' can be done by the maintainer and
'guix register' does upload the lz file to guildhall
website.

>     6. adding a new tag to a project's git repo of the form vX makes a
>        release X and updates the guildhall package
>        - it could be the web service has to poll git repos
>        - or maybe you have to invoke some command to update guildhall

It's painful to have to "watch" git repositories, except if the git
repository is a remote of the develloper's git. Basically it requires
a manual intervention from the maintainer. Otherwise, guildhall website
need to pull regularly all git repositories.

>     7. probably we need to steal many ideas from npm.org

I think about their dependency resolution algorithm. I think
civodul should chime in.


> You might ask, why is this better than the current guildhall?  It does
> exist after all.  I respond in the following way:
> 

[...]

> 
> But!  Who is going to build the guildhall v2.2? :-)
> 

idk ;)

I have also a question, for guix community:

What do you think of guix web, how one can improve it.
The idea I have behind that question, is that maybe we
should target to create a web ui for guix to improve
guile experience.


Amirouche ~ amz3 ~ http://hyperdev.fr

Re: guix is the guildhall that we always wanted!

[Andy Wingo]

On Thu 16 Mar 2017 20:26, Amirouche Boubekki <amirouche@hypermove.net> writes:

>> So!  My proposal for this new "guildhall" would be:
>>
>>     1. a web service
>>
>
> What would be the cli interface associated with that web service?
> Some thing like the following will do:
>
>   $ guildhall register

So I think we could re-use the "guild" command; see the original
"guildhall" on how this is done.  I dunno though, maybe that's not
desireable.  If it's not, maybe guildhall isn't the right name?  I don't
know.

>     Register a package against the guildhall repository.
>     You must have a package.scm in the current working
>     directory that follows package specification of guix.
>     http://link/to/guix/doc/that/I/dont/know

I think you would have to specify a git repo and it would have to be
public, I think.  (Should the web service mirror the repo?  I don't
know.)

> Does it require login/password?

Yes I think.  Or at least email/token, where the service sends you the
token to your email address.  Dunno.  We could use GPG but that is
complicated too.

>>     2. on which users registers projects
>>
>>     3. a project is a name + a git repository with a /package.scm file
>
> We can have that using guile-git: current commit + remote origin.

Yes :)

>>     4. the package.scm contains Guix package definitions for that
>> project
>>
>>     5. the web service administers a git repository collecting those
>>        packages
>>        - without any hydra.gnu.org overhead
>
> What do you mean, by no hydra.gnu.org overhead? Where are
> done the 'guix pack' command?

I mean that when a package is part of Guix it becomes part of Guix's
continuous integration and build process, and Hydra will build it all
the time, and it's a burden of sorts onto Guix.  A value as well of
course but a burden too.  So what I am saying here is that we avoid
that.

>>        - without any manual checks
>>        - in a form that you can just check out once and add to
>> GUIX_PACKAGE_PATH
>>
>
> Ok, the 'guix pack' can be done by the maintainer and
> 'guix register' does upload the lz file to guildhall
> website.

"Guix pack" is orthogonal to this; it's just a nice way to deliver
software.  But for registering a package with the guildhall and its
dependencies and arranging that you can just "git pull" the guildhall
git repo which contains all the package.scm files in the guildhall --
that's what I mean.

>> But!  Who is going to build the guildhall v2.2? :-)
>
> idk ;)

:)

Andy

Re: guix is the guildhall that we always wanted!

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]


Andy Wingo <wingo@pobox.com> writes:

> On Thu 16 Mar 2017 20:26, Amirouche Boubekki <amirouche@hypermove.net> writes:
>
>>> So!  My proposal for this new "guildhall" would be:
>>>
>>>     1. a web service
>>>
>>
>> What would be the cli interface associated with that web service?
>> Some thing like the following will do:
>>
>>   $ guildhall register
>
> So I think we could re-use the "guild" command; see the original
> "guildhall" on how this is done.  I dunno though, maybe that's not
> desireable.  If it's not, maybe guildhall isn't the right name?  I don't
> know.

I’d think

$ guild register # register with the guild

What I long wanted to do:

$ guild hall # show and change the defined package sources

Adding guild sub-commands within guildhall is pretty easy. See
https://github.com/ArneBab/guildhall/commit/3c9b78a91088cc66ecfc9ca7b62cf2a1d8f90781

I’d assume it’s similarly easy when starting fresh.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: guix is the guildhall that we always wanted!

[Mark H Weaver]

Andy Wingo <wingo@pobox.com> writes:

> So!  My proposal for this new "guildhall" would be:
>
>     1. a web service
>
>     2. on which users registers projects
>
>     3. a project is a name + a git repository with a /package.scm file
>
>     4. the package.scm contains Guix package definitions for that project

We need to keep all Guix package definitions within Guix itself, for the
same reason that Linux (the kernel) developers insist on keeping all
device drivers within a single monolithic tree.  If we start encouraging
a decentralized approach, that would result in strong pressure on us to
freeze our API, which includes even such details as which module each
package is exported from.  This would drastically reduce the freedom
Guix has to evolve the way its packages are specified.

       Mark

Re: guix is the guildhall that we always wanted!

[Ludovic Courtès]

Hello!

Mark H Weaver <mhw@netris.org> skribis:

> Andy Wingo <wingo@pobox.com> writes:
>
>> So!  My proposal for this new "guildhall" would be:
>>
>>     1. a web service
>>
>>     2. on which users registers projects
>>
>>     3. a project is a name + a git repository with a /package.scm file
>>
>>     4. the package.scm contains Guix package definitions for that project
>
> We need to keep all Guix package definitions within Guix itself, for the
> same reason that Linux (the kernel) developers insist on keeping all
> device drivers within a single monolithic tree.  If we start encouraging
> a decentralized approach, that would result in strong pressure on us to
> freeze our API, which includes even such details as which module each
> package is exported from.  This would drastically reduce the freedom
> Guix has to evolve the way its packages are specified.

I think having repos maintained elsewhere is OKish, but it’s true that
it requires people who maintain those repos to follow closely what’s
going on in Guix proper because we’re not guaranteeing API stability.

There’s a couple of ways to mitigate potential API instability: using
‘specification->package’ instead of referring to packages by variable
name, and using (guix) and (gnu) instead of more specific modules.

So I think having an external repo for guildhall would be possible, but
we need to make sure the connection to Guix APIs is loose enough.

Also, guildhall would be a privileged “customer” I suppose.  :-)

Ludo’.

Re: guix is the guildhall that we always wanted!

[Florian Paul Schmidt]

On 03/16/2017 11:24 PM, Ludovic Courtès wrote:
> I think having repos maintained elsewhere is OKish, but it’s true that
> it requires people who maintain those repos to follow closely what’s
> going on in Guix proper because we’re not guaranteeing API stability.

Wouldn't taking the functional/reproducibility aspect one step further 
migitate this issue? I.e. maintainers of decentralized repos just need 
to have as input the precise guix version they are using?

For me using guix as a function to produce a piece of software just 
gains another argument: the guix version. If ignoring all the gory 
details and forgiving my ignorance is

guix-build: guix-version -> package-spec (+closure) -> binaries

applied to a particular guix-version gives

guix-build(guix-version): package (+closure) -> binaries

(It's "just" currying)

This of course introduces the problem of possibly having separate guix 
versions on a single system, possibly sharing a store, but it's an 
entertaining thought nonetheless, no?

Flo

-- 
https://fps.io

Re: guix is the guildhall that we always wanted!

[Andy Wingo]

On Fri 17 Mar 2017 10:01, Florian Paul Schmidt <mista.tapas@gmx.net> writes:

> On 03/16/2017 11:24 PM, Ludovic Courtès wrote:
>> I think having repos maintained elsewhere is OKish, but it’s true that
>> it requires people who maintain those repos to follow closely what’s
>> going on in Guix proper because we’re not guaranteeing API stability.
>
> Wouldn't taking the functional/reproducibility aspect one step further
> migitate this issue? I.e. maintainers of decentralized repos just need
> to have as input the precise guix version they are using?

This is certainly possible.  Currently the Guix revision is an implicit
input to everything; a guix package "bar" may depend on "foo@2.1", but
the precise built derivation of "foo" on which you depend will evolve
over time as the dependencies change (glibc for example).

Relying on a fixed Guix version fixes this, but at a cost of adding a
new dimension of maintenance complexity that is not reducible in an
automatic way at run-time.  I explain.

For complexity, in the Guix project itself we have one git repo with N
packages; right now packages for a given revision of git, packages only
depend on each other in that revision.  But if you start to build
systems that have packages depending on packages *at M specific
revisions in Git*, you no longer have N nodes in your graph, you have
N*M.  That's a serious increase in maintenance cost for the system as a
whole (if indeed you do decide to maintain it).

However when you actually instantiate a given application (process) it
is only going to depend on one libc, and one "libfoo".  (I know Rust's
cargo has some facility to import multiple versions of libraries.  It
does so by making the types that those versions work on distinct.  But
in Guile there's no easy facility here.)  So in practice you do have to
resolve the M revisions of Guix into a single revision.  How do you do
that in a system that depends on multiple libc versions?  In many ways
it is better to avoid the issue and have one system that evolves over
time.

When you are building an operating system as a collection of somewhat
independent packages that communicate over shared standard protocols and
not shared ABI, version skew is more manageable.  But when building
composite processes that need to agree on API and ABI internally, the
problem is hard.  A guildhall module needs to be able to compose with
other modules and for that reason I think specifying the Guix revision
is not the right thing.

Just my humble opinion of course :)

Andy

Re: guix is the guildhall that we always wanted!

[Ludovic Courtès]

Hi,

Florian Paul Schmidt <mista.tapas@gmx.net> skribis:

> On 03/16/2017 11:24 PM, Ludovic Courtès wrote:
>> I think having repos maintained elsewhere is OKish, but it’s true that
>> it requires people who maintain those repos to follow closely what’s
>> going on in Guix proper because we’re not guaranteeing API stability.
>
> Wouldn't taking the functional/reproducibility aspect one step further
> migitate this issue? I.e. maintainers of decentralized repos just need
> to have as input the precise guix version they are using?

It’s an interesting thought and may be useful in some scenarios.
The “channels” proposal by Pjotr and Leo looks at this issue:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629#19

However, as Andy wrote, there are situations where you’d rather avoid
having separate dependency graphs since that hinders composition.  For
example, you could end up with a situation where your guildhall repo is
stuck with Guile 2.2.0 and libc 2.24, even though the main Guix repo has
Guile 2.2.3 and libc 2.25.

Ludo’.

Re: guix is the guildhall that we always wanted!

[Marko Rauhamaa]

Mark H Weaver <mhw@netris.org>:

> We need to keep all Guix package definitions within Guix itself, for
> the same reason that Linux (the kernel) developers insist on keeping
> all device drivers within a single monolithic tree.

I think that's one of the most unfortunate aspects of Linux.

Just saying.


Marko

Re: guix is the guildhall that we always wanted!

[Andy Wingo]

On Thu 16 Mar 2017 23:01, Mark H Weaver <mhw@netris.org> writes:

> If [Guix] starts encouraging a decentralized approach, that would
> result in strong pressure on us to freeze our API, which includes even
> such details as which module each package is exported from.  This
> would drastically reduce the freedom Guix has to evolve the way its
> packages are specified.

I get what you are saying.  I think that if a future guildhall is
decentralized but uses Guix it needs to minimize its burden on Guix.
That could mean that the packages are actually specified in a different
DSL with different stability characteristics -- for example that DSL
could call specification->package under the hood for example, like
Ludovic mentions.  (I should mention that this idea of using Guix and
especially all its errors are my own -- haven't talked to others about
it yet!)

Which module a package definition is in is a good example of something
not to depend on.

Basically I think Guix should be able to do what it wants to.  The
stability characteristics that Guix already has are sufficient for a
Guildhall -- no additional maintenance burden intended and I hope no
additional burden imposed.

WDYT?

Andy

Re: guix is the guildhall that we always wanted!

[Christopher Allan Webber]

Andy Wingo writes:

> On Thu 16 Mar 2017 23:01, Mark H Weaver <mhw@netris.org> writes:
>
>> If [Guix] starts encouraging a decentralized approach, that would
>> result in strong pressure on us to freeze our API, which includes even
>> such details as which module each package is exported from.  This
>> would drastically reduce the freedom Guix has to evolve the way its
>> packages are specified.
>
> I get what you are saying.  I think that if a future guildhall is
> decentralized but uses Guix it needs to minimize its burden on Guix.
> That could mean that the packages are actually specified in a different
> DSL with different stability characteristics -- for example that DSL
> could call specification->package under the hood for example, like
> Ludovic mentions.  (I should mention that this idea of using Guix and
> especially all its errors are my own -- haven't talked to others about
> it yet!)
>
> Which module a package definition is in is a good example of something
> not to depend on.

This makes sense to me... if it really is true that our scheme'y
Guildhall-style packages are so simple they're more data than code,
maybe we could even restrict them to... just data.  Just a list of what
files are being provided, etc.  That could easily be stored in some
minimal database.

I guess I'm saying +1.

Re: guix is the guildhall that we always wanted!

[Andy Wingo]

On Fri 17 Mar 2017 14:54, Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Andy Wingo writes:
>
>> On Thu 16 Mar 2017 23:01, Mark H Weaver <mhw@netris.org> writes:
>>
>>> If [Guix] starts encouraging a decentralized approach, that would
>>> result in strong pressure on us to freeze our API, which includes even
>>> such details as which module each package is exported from.  This
>>> would drastically reduce the freedom Guix has to evolve the way its
>>> packages are specified.
>>
>> I get what you are saying.  I think that if a future guildhall is
>> decentralized but uses Guix it needs to minimize its burden on Guix.
>> That could mean that the packages are actually specified in a different
>> DSL with different stability characteristics -- for example that DSL
>> could call specification->package under the hood for example, like
>> Ludovic mentions.  (I should mention that this idea of using Guix and
>> especially all its errors are my own -- haven't talked to others about
>> it yet!)
>>
>> Which module a package definition is in is a good example of something
>> not to depend on.
>
> This makes sense to me... if it really is true that our scheme'y
> Guildhall-style packages are so simple they're more data than code,
> maybe we could even restrict them to... just data.  Just a list of what
> files are being provided, etc.  That could easily be stored in some
> minimal database.

Concretly I would propose something like this in a package.scm in the
git repo:

  (use-modules (guildhall)) ; or whatever we call it
  (import-guix-packages ((a "a")
                         (b "b@5.2")))

  (define foo
    (guildhall-package
      (name "foo")
      (inputs
       `(("a" ,a)
         ("b" ,b)))
      (git-url "https://github.com/foo/foo")
      (home-page "https://github.com/foo/foo")
      (synopsis "Foo is a thing")
      (build-system simple-guile-build-system)
      (description
       "It's a thing")
      (license license:expat)))

I guess you would not want to load this file from the web service as it
has arbitrary Scheme code inside it.  I could see solutions where only
the end-user tool loads this file and exports data to the server, and
the server creates an appropriate "normal" package definition using just
the Guix API.  It can write those definitions out to disk.  It would
export a progressing git repository of Guix package definitions.  As
Guix API changes, that server could re-render those package
descriptions.  Of course that only works in some limited cases.
Otherwise you as a user could do this rendering process for packages you
are developing.  Maybe it is a good thing to encourage nontrivial
packages to go upstream to Guix.

There's not a whole lot that is Guile-specific there of course.  Maybe
this is just an exurb of Guix; I don't know.  Probably "foo" might need
a local build story as well and this doesn't naturally solve that.  More
thinking to do here!

Andy

Re: guix is the guildhall that we always wanted!

[Ludovic Courtès]

Andy Wingo <wingo@igalia.com> skribis:

> Concretly I would propose something like this in a package.scm in the
> git repo:
>
>   (use-modules (guildhall)) ; or whatever we call it
>   (import-guix-packages ((a "a")
>                          (b "b@5.2")))
>
>   (define foo
>     (guildhall-package
>       (name "foo")
>       (inputs
>        `(("a" ,a)
>          ("b" ,b)))
>       (git-url "https://github.com/foo/foo")
>       (home-page "https://github.com/foo/foo")
>       (synopsis "Foo is a thing")
>       (build-system simple-guile-build-system)
>       (description
>        "It's a thing")
>       (license license:expat)))
>
> I guess you would not want to load this file from the web service as it
> has arbitrary Scheme code inside it.  I could see solutions where only
> the end-user tool loads this file and exports data to the server, and
> the server creates an appropriate "normal" package definition using just
> the Guix API.  It can write those definitions out to disk.  It would
> export a progressing git repository of Guix package definitions.  As
> Guix API changes, that server could re-render those package
> descriptions.  Of course that only works in some limited cases.
> Otherwise you as a user could do this rendering process for packages you
> are developing.  Maybe it is a good thing to encourage nontrivial
> packages to go upstream to Guix.

Sounds like a reasonable approach.

BTW, there’s a simple actionable item here, which is to write that
‘guile-build-system’ in Guix.  Currently, for Scheme-only packages, we
often more or less copy/pasted stuff like this:

  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/guile.scm#n733

If this were packaged in a ‘guile-build-system’¹, the barrier to
submitting Guile packages would already be lower.

Ludo’.

¹ See <https://gnu.org/software/guix/manual/html_node/Build-Systems.html>.

Re: guix is the guildhall that we always wanted!

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> skribis:

> Crucially, this "guix pack" is really an SDK for Guile and Fibers.
> Fibers is a library.  You can now make an SDK for anything!!!!!!!!!!!  I
> mean, I know, that's a lot of exclamation marks, but check it: you can
> develop locally, say using
>
>   /opt/guile-fibers-1.0.0/bin/guile -L /path/to/my/code
>
> and it just works.  If you package up your code, you can make a Guix
> package out of it and make a "guix pack" out of that closure and
> distribute it to your users: REGARDLESS OF WHAT OPERATING SYSTEM THEY
> RUN, AND WITH FULL SOURCE REPRODUCIBILITY.  It's huge.

IT’S AWESOME!!!!  ;-)

> But while getting your package into guix is easier than you think, it's
> not automatic.  I think there's room for a middle ground that's a bit
> more decentralized than Guix, but more centralized than people using
> GUIX_PACKAGE_PATH to add random directories of Guix package files.
>
> So!  My proposal for this new "guildhall" would be:
>
>     1. a web service
>
>     2. on which users registers projects
>
>     3. a project is a name + a git repository with a /package.scm file
>
>     4. the package.scm contains Guix package definitions for that project
>
>     5. the web service administers a git repository collecting those
>        packages
>        - without any hydra.gnu.org overhead
>        - without any manual checks
>        - in a form that you can just check out once and add to GUIX_PACKAGE_PATH
>
>     6. adding a new tag to a project's git repo of the form vX makes a
>        release X and updates the guildhall package
>        - it could be the web service has to poll git repos
>        - or maybe you have to invoke some command to update guildhall
>
>     7. probably we need to steal many ideas from npm.org

So are you suggesting that the new guildhall would provide
Guix-generated packs for those who don’t use Guix, and otherwise regular
Guix packages?

> Open questions would be, what about other targets like macOS or Windows
> or whatever?  There I don't know.  I suspect that if Guix becomes
> popular enough, someone will make a thing whereby you can cross-compile
> to macOS from GNU/Linux at least.  I know you can do that already with
> MinGW.  As for native "guildhall" operation on these targets, I don't
> know.  But I think that is a cost that people will pay for in time.

We could surely create cross-compiled packs for MinGW, though I suspect
a number of packages do not support cross-compilation.

Thanks for starting this thread!

Ludo’.

Re: guix is the guildhall that we always wanted!

[Andy Wingo]

On Fri 17 Mar 2017 12:30, ludo@gnu.org (Ludovic Courtès) writes:

>> So!  My proposal for this new "guildhall" would be:
>>
>>     1. a web service
>>
>>     2. on which users registers projects
>>
>>     3. a project is a name + a git repository with a /package.scm file
>>
>>     4. the package.scm contains Guix package definitions for that project
>>
>>     5. the web service administers a git repository collecting those
>>        packages
>>        - without any hydra.gnu.org overhead
>>        - without any manual checks
>>        - in a form that you can just check out once and add to GUIX_PACKAGE_PATH
>>
>>     6. adding a new tag to a project's git repo of the form vX makes a
>>        release X and updates the guildhall package
>>        - it could be the web service has to poll git repos
>>        - or maybe you have to invoke some command to update guildhall
>>
>>     7. probably we need to steal many ideas from npm.org
>
> So are you suggesting that the new guildhall would provide
> Guix-generated packs for those who don’t use Guix, and otherwise regular
> Guix packages?

No, I think the guildhall is simply another source of package
definitions for Guix.  Pack creation would be done by the user as
needed -- if I want to ship packages A, B, and C, I can just do that.
Some of those packages might be defined in Guix proper and some from
guildhall.

The interesting bit I think is simply being able to add to the set of
guildhall packages in a decentralized way, to complement the set of
packages in core Guix.  I imagine the package flow over time would be
from guildhall to core guix.

Andy

Re: guix is the guildhall that we always wanted!

[Pjotr Prins]

Hi Andy,

It does look a lot like the channels we propose.

Another thing to keep in mind is that it is possible to relocate guix
binaries into any prefix so you do not actually need root privileges. 

It is something I am using for testing packages on HPC without root
and I am going to make binaries available for the bioconda project.

Pj.

Re: guix is the guildhall that we always wanted!

[Mike Gran]

Hi all,

While I don't have too much experience with the Guix,
I have been around long enough to see multiple experiments

with package management come and go.  (I remember life
before RPM. haha)

As a general principle, any package manager will fail
if it doesn't have developer excitement.  And to be
developer friendly, it has to allow software in all
of its phases from a chaotic pre-alpha to a reliable
release.

I do rather like the idea of an easy push for new updates
to alpha and beta-level packages.  I also
like the idea of some curating or labeling or rating
of packages that describe themselves as stable or

complete.  Actually, I should be more forceful: without
allowing for both alpha-level chaos and a stable central
corpus of packages, any new package management experiment
will fail.


My two cents,
Thanks,
Mike Gran

Re: guix is the guildhall that we always wanted!

[Eli Zaretskii]

> From: ludo@gnu.org (Ludovic Courtès)
> Date: Fri, 17 Mar 2017 12:30:57 +0100
> Cc: guix-devel@gnu.org, guile-user@gnu.org
> 
> > Open questions would be, what about other targets like macOS or Windows
> > or whatever?  There I don't know.  I suspect that if Guix becomes
> > popular enough, someone will make a thing whereby you can cross-compile
> > to macOS from GNU/Linux at least.  I know you can do that already with
> > MinGW.  As for native "guildhall" operation on these targets, I don't
> > know.  But I think that is a cost that people will pay for in time.
> 
> We could surely create cross-compiled packs for MinGW, though I suspect
> a number of packages do not support cross-compilation.

The MinGW build doesn't support multithreading in Guile, so any such
cross-compiled packages should be aware of that.  Also, there are some
Posix functionality not supported, like primitive-fork, so if Guix
uses that, those parts won't work.

Re: guix is the guildhall that we always wanted!

[Ludovic Courtès]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> skribis:

>> From: ludo@gnu.org (Ludovic Courtès)
>> Date: Fri, 17 Mar 2017 12:30:57 +0100
>> Cc: guix-devel@gnu.org, guile-user@gnu.org
>> 
>> > Open questions would be, what about other targets like macOS or Windows
>> > or whatever?  There I don't know.  I suspect that if Guix becomes
>> > popular enough, someone will make a thing whereby you can cross-compile
>> > to macOS from GNU/Linux at least.  I know you can do that already with
>> > MinGW.  As for native "guildhall" operation on these targets, I don't
>> > know.  But I think that is a cost that people will pay for in time.
>> 
>> We could surely create cross-compiled packs for MinGW, though I suspect
>> a number of packages do not support cross-compilation.
>
> The MinGW build doesn't support multithreading in Guile, so any such
> cross-compiled packages should be aware of that.  Also, there are some
> Posix functionality not supported, like primitive-fork, so if Guix
> uses that, those parts won't work.

The goal is not to run Guix on MinGW.

I think Andy was referring to the possibility of cross-compiling
packages that use Guile to MinGW.  That is already possible thanks to
the work of Jan Jan Nieuwenhuizen, and since yesterday, we can create
“packs” that contain binaries cross-compiled for MinGW:

  https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00525.html

Ludo’.

Re: guix is the guildhall that we always wanted!

[Eli Zaretskii]

> From: ludo@gnu.org (Ludovic Courtès)
> Cc: wingo@pobox.com,  guix-devel@gnu.org,  guile-user@gnu.org
> Date: Sat, 18 Mar 2017 15:04:52 +0100
> 
> I think Andy was referring to the possibility of cross-compiling
> packages that use Guile to MinGW.  That is already possible thanks to
> the work of Jan Jan Nieuwenhuizen, and since yesterday, we can create
> “packs” that contain binaries cross-compiled for MinGW:
> 
>   https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00525.html

I'm saying that when cross-compiling, it's easy to produce binaries
that are unusable on Windows because Guile is not run natively during
the build, and so problems inherent to the Windows port are not
revealed during the build.  Multithreading is one such issue, quite a
significant one.  Unless I disable multithreading, I cannot build
Guile natively, because every non-trivial Guile program crashes (I
reported that years ago, but no one was able to explain why that
happens or point me to a workaround).

So if the above binaries include threads, I'm afraid they might not be
very useful on Windows.

Re: guix is the guildhall that we always wanted!

[Ludovic Courtès]

Eli Zaretskii <eliz@gnu.org> skribis:

>> From: ludo@gnu.org (Ludovic Courtès)
>> Cc: wingo@pobox.com,  guix-devel@gnu.org,  guile-user@gnu.org
>> Date: Sat, 18 Mar 2017 15:04:52 +0100
>> 
>> I think Andy was referring to the possibility of cross-compiling
>> packages that use Guile to MinGW.  That is already possible thanks to
>> the work of Jan Jan Nieuwenhuizen, and since yesterday, we can create
>> “packs” that contain binaries cross-compiled for MinGW:
>> 
>>   https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00525.html
>
> I'm saying that when cross-compiling, it's easy to produce binaries
> that are unusable on Windows because Guile is not run natively during
> the build, and so problems inherent to the Windows port are not
> revealed during the build.

Right; that would be a bug in Guile’s build system.  Is pthread support
detection broken when cross-compiling to MinGW?

Thanks,
Ludo’.

Re: guix is the guildhall that we always wanted!

[Eli Zaretskii]

> From: ludo@gnu.org (Ludovic Courtès)
> Cc: wingo@pobox.com,  guix-devel@gnu.org,  guile-user@gnu.org
> Date: Sun, 19 Mar 2017 16:57:22 +0100
> 
> > I'm saying that when cross-compiling, it's easy to produce binaries
> > that are unusable on Windows because Guile is not run natively during
> > the build, and so problems inherent to the Windows port are not
> > revealed during the build.
> 
> Right; that would be a bug in Guile’s build system.  Is pthread support
> detection broken when cross-compiling to MinGW?

Not sure what you are asking here.  I never cross-built Guile, I
always build the MinGW port natively.  And pthreads detection is not
broken, the build process correctly detects that I have pthreads
installed.  But if I want to produce a working Guile (or even get the
build to run to completion, since that involves running Guile to
compile Scheme files), I need to disable threads.