Guix as a Guile package manager

Guix as a Guile package manager

[Fabio Pesari]

Package managers have been immensely successful in increasing the
popularity of programming languages - think about Perl's CPAN or Ruby's
Gem. But Guile doesn't a package manager, and that in my opinion slows
down its adoption.

The Guix repos distribute a lot of useful Guile libraries (like
guile-json or guile-opengl) which can't be found on most distro
repositories and it already provides Guile APIs and package management
capabilities...my question is, can Guix be forked into a full-blown
Guile package manager like gem from Ruby?

I know that an argument could be made that Guix can already be used in
this way, but there are many Scheme coders who don't need a system-wide
package manager and would rather use a program that can manage Guile
packages under a user root like ~/.guile and allow them to easily
distribute their packages (something like Python's virtualenvs would
also be useful).

Perhaps some of the Guix code can be moved to a library, so that both
the Guix and the Guile package manager binaries can reuse the same code.
Moving Guix' core to a library would also facilitate its inclusion in
things like PackageKit, as well as make it easier to create front-ends.

I'm not a package management expert so I'm not sure this idea is
feasible but I would really like Guile to become more
popular, and this I think would be a step in the right direction.

Re: Guix as a Guile package manager

[Amirouche Boubekki]

Héllo,

On 2016-01-09 11:35, Fabio Pesari wrote:
> Package managers have been immensely successful in increasing the
> popularity of programming languages - think about Perl's CPAN or Ruby's
> Gem. But Guile doesn't a package manager, and that in my opinion slows
> down its adoption.

There is a package manager https://github.com/ijp/guildhall with a 
package
repository with automatic package publishing without review.

> The Guix repos distribute a lot of useful Guile libraries (like
> guile-json or guile-opengl) which can't be found on most distro
> repositories and it already provides Guile APIs and package management
> capabilities...

> my question is, can Guix be forked into a full-blown Guile package 
> manager
> like gem from Ruby?

I'm not a core developer, but I don't think it makes sens to fork. Most
if not all features of guix are required by language package manager.
This includes virtualenv since guix can do them via 'guix environment'.

I think what could/should/must be done is settle on guix being the 
package
manager of Guile *and* document how to create package recipes for pure 
and
non-pure guile non autotools modules. Maybe declare guix the optional 
official
package manager something like that..

> I know that an argument could be made that Guix can already be used in
> this way, but there are many Scheme coders who don't need a system-wide
> package manager

This is not an issue since IIRC you can install guix as a user. Not sure
what the status of this mode of operation is.

Also binary installation of guix is trivial, so it's not like something
that can forbid people using it. Having guix in other distros would be
of great help too.

> and would rather use a program that can manage Guile
> packages under a user root like ~/.guile and allow them to easily
> distribute their packages

Again there is guildhall even if it doesn't have the polish (nice web 
ui)
of pypi or cpan, it's said to work.

> (something like Python's virtualenvs would also be useful).

This is supported by guix.

> Perhaps some of the Guix code can be moved to a library, so that both
> the Guix and the Guile package manager binaries can reuse the same 
> code.
> Moving Guix' core to a library would also facilitate its inclusion in
> things like PackageKit, as well as make it easier to create front-ends.
> 
> I'm not a package management expert so I'm not sure this idea is
> feasible but I would really like Guile to become more
> popular, and this I think would be a step in the right direction.

To repeat myself, I think we need more guidance and incentive on how to
include guile modules that are not autotools ready. For my usecase and a
lot of other modules I don't think it makes sens to have autotools if 
you
have guix. People who happen to not want to use/install guix, can 
resolve
dependencies themself and I don't see how autotools help anyway.

Somewhat related: even if I never saw a package rejected in guix, I 
think
most contributors have some expectations regarding the quality of 
packages
included in guix *main* repository. Otherwise said, I don't mind pushing 
a
alpha software or snippet on pypi, but this is not the case with guix.

So maybe, it will be nice to have a guix repository dedicated to guile
modules where the expectations will much lower and where guilers
can freely share their small and not so small contributions.

Also, this will be a visible example of how to extend guix with third 
party
package repository which is a significant asset is some commercial 
situations.

-- 
Amirouche ~ amz3 ~ http://www.hypermove.net

Re: Guix as a Guile package manager

[Fabio Pesari]

On 01/09/2016 02:05 PM, Amirouche Boubekki wrote:
> Héllo,

Hi!

> There is a package manager https://github.com/ijp/guildhall with a 
> package
> repository with automatic package publishing without review.

There are many package managers actually, and most of them were
abandoned by their maintainers and/or never gained any traction in the
community.

This one doesn't look like it's any different - last commit goes back to
June 22nd, 2015.

The reason CPAN, pip and gem are so successful is that they are bundled
with their language's interpreters and are officially maintained.

Asking users to install a separate package manager might work in some
cases (Leiningen, Composer) but it usually leads to fragmentation and
confusion (as it was the case with many Lisps, especially CLs).

> I'm not a core developer, but I don't think it makes sens to fork. Most
> if not all features of guix are required by language package manager.
> This includes virtualenv since guix can do them via 'guix environment'.

I called for a fork because having Guix as both a general-purpose
package manager and a Guile-specific package manager is very confusing
and doesn't follow the UNIX philosophy.

As I said before, there doesn't need to be any code duplication - Guix'
core can be abstracted into a library (e.g. guile-guix) and both Guix
and the Guile package manager can be implemented on top of it with a
minimal amount of code (since nearly all functionalities are implemented
in the library).

> I think what could/should/must be done is settle on guix being the 
> package
> manager of Guile *and* document how to create package recipes for pure 
> and
> non-pure guile non autotools modules. Maybe declare guix the optional 
> official
> package manager something like that..

That's still better than nothing, I suppose, so I'm not against it if
it's the only way to make it happen (at least in the short term).

> This is not an issue since IIRC you can install guix as a user. Not sure
> what the status of this mode of operation is.
> 
> Also binary installation of guix is trivial, so it's not like something
> that can forbid people using it. Having guix in other distros would be
> of great help too.

Yes, Guix should be in the official repos of other distros as soon as
possible. I understand that the devs are still working on Guix but
increasing the userbase (and the pool of potential testers) wouldn't hurt.

> Again there is guildhall even if it doesn't have the polish (nice web 
> ui)
> of pypi or cpan, it's said to work.

The lack of a web UI is a huge problem, and so is the fact that "it's
said to work".

The official repository server should also be hosted on GNU servers and
not by an independent developer like guildhall's:

http://shift-reset.com/doro/

how can I know for sure all those packages are 100% free?

User should be able to upload packages but each package should be
carefully reviewed (possibly by the community itself).

> Somewhat related: even if I never saw a package rejected in guix, I 
> think
> most contributors have some expectations regarding the quality of 
> packages
> included in guix *main* repository. Otherwise said, I don't mind pushing 
> a
> alpha software or snippet on pypi, but this is not the case with guix.
> 
> So maybe, it will be nice to have a guix repository dedicated to guile
> modules where the expectations will much lower and where guilers
> can freely share their small and not so small contributions.

I agree with you that users should be able to submit packages easily -
that's why I called for a fork, so that the standards for package
inclusion can be much lower (except for freedom, which is imperative)
and the Guile packages are by default separated from all other software.

This could also be achieved with a separate repo (like "guile-contrib"
or something along these lines) for Guix, sure, but I'd still like a
separate repo with a separate database and site, so that important
things like user ratings can be implemented independently from the other
Guix repos.

> Also, this will be a visible example of how to extend guix with third 
> party
> package repository which is a significant asset is some commercial 
> situations.

I'm not against the idea of third party package repositories (I see no
reason why this functionality should not be implemented) but Guix should
focus on having every decent quality free program in its repositories,
so that people are not encouraged to use third-party repos.

I find it self-defeating that in distros like Parabola (or upstream,
Arch), fully functional and semi-popular programs like OpenArena,
pngquant and yuicompressor can only be found in the user repository (the
AUR), which also distribute proprietary software.

If people are encouraged to include third-party repos, freedom goes out
of the window pretty easily, so the official repositories should be as
complete as possible (I know it's easier said than done, but it should
be much easier for Guix compared to other package managers).

Re: Guix as a Guile package manager

[Amirouche Boubekki]

On 2016-01-09 15:06, Fabio Pesari wrote:
> On 01/09/2016 02:05 PM, Amirouche Boubekki wrote:
>> There is a package manager https://github.com/ijp/guildhall with a
>> package
>> repository with automatic package publishing without review.
> 
> Asking users to install a separate package manager might work in some
> cases (Leiningen, Composer) but it usually leads to fragmentation and
> confusion (as it was the case with many Lisps, especially CLs).
> 

That's one of the reason why I advocate for guix as package manager of 
guile.

>> I'm not a core developer, but I don't think it makes sens to fork. 
>> Most
>> if not all features of guix are required by language package manager.
>> This includes virtualenv since guix can do them via 'guix 
>> environment'.
> 
> I called for a fork because having Guix as both a general-purpose
> package manager and a Guile-specific package manager is very confusing
> and doesn't follow the UNIX philosophy.
> 

Can you explain what a Guile specific fork of guix will bring over guix?

> User should be able to upload packages but each package should be
> carefully reviewed (possibly by the community itself).

This is not how pypi works for instance.

>> Somewhat related: even if I never saw a package rejected in guix, I
>> think
>> most contributors have some expectations regarding the quality of
>> packages
>> included in guix *main* repository. Otherwise said, I don't mind 
>> pushing
>> a
>> alpha software or snippet on pypi, but this is not the case with guix.
>> 
>> So maybe, it will be nice to have a guix repository dedicated to guile
>> modules where the expectations will much lower and where guilers
>> can freely share their small and not so small contributions.
> 
> I agree with you that users should be able to submit packages easily -
> that's why I called for a fork, so that the standards for package
> inclusion can be much lower (except for freedom, which is imperative)
> and the Guile packages are by default separated from all other 
> software.
> 
> This could also be achieved with a separate repo (like "guile-contrib"
> or something along these lines) for Guix, sure, but I'd still like a
> separate repo with a separate database and site, so that important
> things like user ratings can be implemented independently from the 
> other
> Guix repos.
> 

I just checked the documentation [1] and it's possible to have third 
party
repositories but the policy is to not fork the effort and package guile
softwares in guix.

[1] 
https://www.gnu.org/software/guix/manual/guix.html#index-GUIX_005fPACKAGE_005fPATH

>> Also, this will be a visible example of how to extend guix with third
>> party
>> package repository which is a significant asset is some commercial
>> situations.
> 
> I'm not against the idea of third party package repositories (I see no
> reason why this functionality should not be implemented) but Guix 
> should
> focus on having every decent quality free program in its repositories,
> so that people are not encouraged to use third-party repos.
> 
> I find it self-defeating that in distros like Parabola (or upstream,
> Arch), fully functional and semi-popular programs like OpenArena,
> pngquant and yuicompressor can only be found in the user repository 
> (the
> AUR), which also distribute proprietary software.
> 
> If people are encouraged to include third-party repos, freedom goes out
> of the window pretty easily, so the official repositories should be as
> complete as possible (I know it's easier said than done, but it should
> be much easier for Guix compared to other package managers).

My question is: what must do a guix fork that guix doesn't have already?

Re: Guix as a Guile package manager

[Fabio Pesari]

On 01/09/2016 03:35 PM, Amirouche Boubekki wrote:
>
> Can you explain what a Guile specific fork of guix will bring over guix?

See the last part of this post.

>> User should be able to upload packages but each package should be
>> carefully reviewed (possibly by the community itself).
> 
> This is not how pypi works for instance.

But that doesn't mean Guix should follow suit. Implementing approval
like Savannah does is a bad idea, because packages get stuck there for
ages and there's a lot of centralized power, but putting packages in a
public queue which users can vote and allow packages to be reported for
being nonfree and let maintainers be moderators is a better idea.

> I just checked the documentation [1] and it's possible to have third 
> party
> repositories but the policy is to not fork the effort and package guile
> softwares in guix.
> 
> [1] 
> https://www.gnu.org/software/guix/manual/guix.html#index-GUIX_005fPACKAGE_005fPATH

Where's the bit about forking? The page is too long and I couldn't find it.

> My question is: what must do a guix fork that guix doesn't have already?

It's not about functionality, it's about purpose.

The "fork" is not really a fork, but rather a separate program which
uses the same core library as Guix (which I defined as guile-guix
earlier). In practical terms, it would use a different repo and have
different defaults tuned to Guile development rather than system
maintenance.

If the Guile devs decided to adopt Guix as it is right now as the
official package manager like Perl's CPAN or Ruby's Gem, they would have
to ship Guix with Guile, but that doesn't sound right in my opinion
because Guix has a much larger scope and different purpose than CPAN and
Gem and users could perceive this as adding bloat and/or a tool that is
largely unrelated to Guile development - much like bundling Emacs
because it has good Guile editing capabilities.

It would also make more sense to distribute Guix as a separate program
rather than require users to run a command like "pacman -S guile" to
install Guix, but the same does not apply to the Guile package manager
obviously because package management for Guile can be seen as part of
the Guile environment (like it is for Ruby, Perl, Python, Go, Chicken,
Racket, Emacs, etc.)

The way I see it:

* Guile ships with guile-guix (a library that implements much of Guix'
  package management capabilities, but not the Guix program) and a
  (very small) Guile package manager which depends on guile-guix

* Guix is distributed separately as a (very small) program which
  depends on guile-guix

Just to be clear, Guile does not depend on guile-guix, it just is a
library that is shipped along with Guile and on which the package
manager (also shipped with Guile) depends.

The only side effect from doing this is that Guile's size in bytes would
go up a bit, however it'd still be smaller than Python and it would gain
package management capabilities. If that is not acceptable,
I think that distributing the Guile package manager as a standalone
package (like npm or composer) would still be a good thing, but I'd make
it a separate program from Guix anyway for the reasons mentioned above.

Re: Guix as a Guile package manager

[Thompson, David]

On Sat, Jan 9, 2016 at 5:35 AM, Fabio Pesari <fabiop@gnu.org> wrote:
> Package managers have been immensely successful in increasing the
> popularity of programming languages - think about Perl's CPAN or Ruby's
> Gem. But Guile doesn't a package manager, and that in my opinion slows
> down its adoption.

Many of us in the Guile community, including myself, are not very
interested in language-specific package managers because they are
problematic:

- They put burden on the user to make sure the right toolchain is
  available to build/install things (GCC and co.)

- They cannot describe the full dependency graph for things that
  aren't completely written in that language or use a foreign-function
  interface (for example, a Guile library that talks to a C library
  via the dynamic FFI, or one that uses the Guile C API)

- They encourage new users to conflate the build system with the
  package manager, leading to software that cannot be built without a
  network connection and a specific package manager.  Java software is
  notorious for this, as are Ruby and Python to a lesser degree.

That said, I see the desire to have a tool for easily sharing pure
Guile modules.

> The Guix repos distribute a lot of useful Guile libraries (like
> guile-json or guile-opengl) which can't be found on most distro
> repositories and it already provides Guile APIs and package management
> capabilities...my question is, can Guix be forked into a full-blown
> Guile package manager like gem from Ruby?

You can install Guix right now and use it to help you develop with
Guile, or any other language.  I use it for all of my Guile projects
and it works great.

> I know that an argument could be made that Guix can already be used in
> this way, but there are many Scheme coders who don't need a system-wide
> package manager and would rather use a program that can manage Guile
> packages under a user root like ~/.guile and allow them to easily
> distribute their packages (something like Python's virtualenvs would
> also be useful).

This is what the Guildhall[0] is for, but it needs some love.  I
really do not encourage the use of language-specific package managers
for the above-mentioned problems, but if you *really* want that,
contributing to the Guildhall seems to be the best thing to do.
People ask about it somewhat frequently, but so far no one has seemed
willing to roll up their sleeves and make it work for them.

> Perhaps some of the Guix code can be moved to a library, so that both
> the Guix and the Guile package manager binaries can reuse the same code.
> Moving Guix' core to a library would also facilitate its inclusion in
> things like PackageKit, as well as make it easier to create front-ends.

This would be a bad idea, because it creates a divide between the Guix
core and the rest of Guix, which would force us to make currently
internal APIs into external, public APIs and worry about backwards
compatibility and other issues.  Our development of Guix models that
of Linux.  In Linux, the kernel is maintained in conjuction with the
kernel modules, in the same source tree.  In Guix, the core modules
are maintained in conjunction with the user interfaces and package
recipes, in the same source tree.  Thus, if we change the packaging
API, or something else, we can update the relevant code in the source
tree and be done with it.  We don't have to worry about breaking
compatibility with an older version of libguix or something.  This was
a deliberate decision made to deal with problems that the Nix
community have when their version of Nix doesn't work with their
version of nixpkgs.

On the topic of PackageKit, it's been discussed in the past, but
PackageKit is incompatible with functional package managers because it
does not support important features like unprivileged package
management. [1]

> I'm not a package management expert so I'm not sure this idea is
> feasible but I would really like Guile to become more
> popular, and this I think would be a step in the right direction.

I don't think Guix is a good fit here, because there's simply no way
around the fact that Guix packages *must* contain the full dependency
graph for build-time and runtime dependencies.  There's no way that
you could use, say, the Guile or GCC from your host system because
doing so invalidates the entire purpose and design of Guix.  Thus, a
Debian user that tried to use a Guile-specific package manager based
on Guix would be surprised when it starts compiling its own glibc,
GCC, Guile, etc.  So, I don't see anything worth extracting from Guix
to use to build a Guile-specific package manager.

I encourage you to hack on Guildhall to make it more usable for your
needs.

Happy hacking,

- Dave

[0] https://github.com/ijp/guildhall
[1] https://lists.gnu.org/archive/html/bug-guix/2014-03/msg00018.html

Re: Guix as a Guile package manager

[Fabio Pesari]

On 01/09/2016 04:30 PM, Thompson, David wrote:
>
> I don't think Guix is a good fit here, because there's simply no way
> around the fact that Guix packages *must* contain the full dependency
> graph for build-time and runtime dependencies.  There's no way that
> you could use, say, the Guile or GCC from your host system because
> doing so invalidates the entire purpose and design of Guix.  Thus, a
> Debian user that tried to use a Guile-specific package manager based
> on Guix would be surprised when it starts compiling its own glibc,
> GCC, Guile, etc.  So, I don't see anything worth extracting from Guix
> to use to build a Guile-specific package manager.

Hello David,

and thank you for your answer!

As a simple user, I didn't know enough about the finer points of Guix in
this context so I really appreciate your answer. I thought some of Guix'
package management functionalities could be worked into a less specific
program but if it can't be done, it can't be done!

> I encourage you to hack on Guildhall to make it more usable for your
> needs.

I've seen people work on dozens of package managers over the years and
honestly, I don't feel like working on one unless it's officially part
of the language or it already has mainstream adoption.

It's a Catch 22, I know, but those projects all amount to re-inventing
the wheel and trying to solve the same problems over and over again
until they get abandoned because nobody uses them (like it happened with
Guildhall, last updated in June).

I know Guile was designed as an extension language so it is excused from
not having a package manager, but with the latest developments and Guix
in particular I thought the language was heading in a more
general-purpose direction and that's why I posted this proposal.

Re: Guix as a Guile package manager

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> Many of us in the Guile community, including myself, are not very
> interested in language-specific package managers because they are
> problematic:
>
> - They put burden on the user to make sure the right toolchain is
>   available to build/install things (GCC and co.)
>
> - They cannot describe the full dependency graph for things that
>   aren't completely written in that language or use a foreign-function
>   interface (for example, a Guile library that talks to a C library
>   via the dynamic FFI, or one that uses the Guile C API)
>
> - They encourage new users to conflate the build system with the
>   package manager, leading to software that cannot be built without a
>   network connection and a specific package manager.  Java software is
>   notorious for this, as are Ruby and Python to a lesser degree.
>
> That said, I see the desire to have a tool for easily sharing pure
> Guile modules.

To complement this, I would say that, despite these limitations, there’s
probably room for Guildhall because it addresses the needs of
non-GNU/Linux users and GNU/Linux users who cannot run Guix for some
reason.

That said, we already have a number of Guile packages in Guix, and I
would love to see more of them.  The fact that Guix can make it easy to
work not only with pure Scheme libraries but also with things like the
Guile-GNOME stack makes it a very good option.

Thanks,
Ludo’.

Re: Guix as a Guile package manager

[Ricardo Wurmus]

Fabio Pesari <fabiop@gnu.org> writes:

> The way I see it:
>
> * Guile ships with guile-guix (a library that implements much of Guix'
>   package management capabilities, but not the Guix program) and a
>   (very small) Guile package manager which depends on guile-guix

The package management capabilities provided by Guix only make sense in
the world of functional package management where the complete graph of
dependencies is captured, all the way down to bootstrap libraries.

If you take away functional package management (e.g. because it’s
inconvenient in a simple language-specific package manager), you won’t
need the guix-daemon anymore, nor do you need to talk to it anymore, nor
do you need to actually be able to represent the complete graph of
inputs ... and you end up throwing away pretty much everything that Guix
provides.

> * Guix is distributed separately as a (very small) program which
>   depends on guile-guix

The commands provided by Guix are tiny Guile scripts that use the bulk
of Guix as a library.  In a sense “guile-guix” already exists: it’s
called Guix :)  But it’s the wrong library for a language-specific
package manager.

~~ Ricardo

Re: Guix as a Guile package manager

[Leo Famulari]

On Sat, Jan 09, 2016 at 11:35:13AM +0100, Fabio Pesari wrote:
> Package managers have been immensely successful in increasing the
> popularity of programming languages - think about Perl's CPAN or Ruby's
> Gem. But Guile doesn't a package manager, and that in my opinion slows
> down its adoption.
> 
> The Guix repos distribute a lot of useful Guile libraries (like
> guile-json or guile-opengl) which can't be found on most distro
> repositories and it already provides Guile APIs and package management
> capabilities...my question is, can Guix be forked into a full-blown
> Guile package manager like gem from Ruby?
> 
> I know that an argument could be made that Guix can already be used in
> this way, but there are many Scheme coders who don't need a system-wide
> package manager and would rather use a program that can manage Guile
> packages under a user root like ~/.guile and allow them to easily
> distribute their packages (something like Python's virtualenvs would
> also be useful).
> 
> Perhaps some of the Guix code can be moved to a library, so that both
> the Guix and the Guile package manager binaries can reuse the same code.
> Moving Guix' core to a library would also facilitate its inclusion in
> things like PackageKit, as well as make it easier to create front-ends.

I think the main problem with this idea is that the whole point of Guix
/ Nix is to manage the entire dependency graph — from the package you
are trying to install and use all the way down the tar used to unpack
the sources, the GCC used to compile everything, etc.

There is not much point in using Guix to only manage the very top-level
of the graph (the Guile modules you'd want to install) because by not
managing the entire graph from top to bottom, the Guix system would just
break as the unmanaged dependencies are changed the OS's package manager
or the user.

There really isn't anything left when you try to split up Guix in the
way you are requesting.

> 
> I'm not a package management expert so I'm not sure this idea is
> feasible but I would really like Guile to become more
> popular, and this I think would be a step in the right direction.
>