Replacing Bower with "guix environment"

Replacing Bower with "guix environment"

[Christopher Allan Webber]

Hello all,

I'm interested in "guix environment" as a universal solution for
virtualenv/bower/etc.  There's good reasons to want this; aside from
Guix being awesome, packaging for web applications is getting more and
more insane and often involves many layers of package management, often
*multiple language package managers at once*.  It's a sad state of
affairs.  Why not have a universal solution?  "guix environment" could
be that, I hope?  But there are some challenges.

The virtualenv side (having libraries set up for development on the
Python end) is really easy with "guix environment" assuming those files
are there: you have your PYTHONPATH set up, done.

But the "bower" side of things is not so easy.  Most web applications
need to serve many assets to the browser: javascript like jquery, css
like bootstrap, fonts like fontawesome or really any other
fonts, not to mention any icon sets, etc.  Web applications need to
serve these in both a "development" way, usually using some simple
development server, in which case these have to be easily discovered and
served.  For production, these need to be served by
nginx/apache/whatever.  Describing to users how to deploy web
applications tends to be pretty hard, so for this last one you want to
make the configuration requirement as much of a one-liner as possible.
This usually means then that the static assets (jquery, fonts, etc) are
referenced from somewhere inside the package, usually in an extlib/
directory or such.

There are two common ways to do this:

 - The "classic" method of doing this is pretty gross: just check all
   those assets directly into your project, maybe in some sort of
   "extlib/" directory.  I don't need to tell guix-devel just how bad of
   a solution this is!  And yet, most web developers (including me
   *shameface*) have done it, even in our free software projects.

 - The "hip new way" of doing things is to use Bower.  Bower is a
   package manager, but it's made specifically for static assets served
   to the user, such as css files, fonts, javascript like jquery, etc.
   Bower also puts these in an extlib/ or whatever, but it puts them in
   that place *for* you.

It's possible in Guix or Debian packages or whatever to rely on a system
installed jquery or etc, just a minor pain in the butt, linking in the
jquery packages and etc manually into that extlib place during building
the package/derivation.  Okay, not so bad.

But what about "guix environment", where we're directly checked out
inside the package?  What we want to do is this:

  ~/devel/mediagoblin/$ guix environment -l package.scm

And this will set up the python dependencies for sure, and maybe it can
even install the jquery package and stuff, but if I run:

  # put your dev server command here
  ~/devel/mediagoblin/$ ./lazyserver.sh

Oh no!  jquery is installed, but it's not linked into extlib/jquery/
where I expected it.  So suddenly I'm looking at my development instance
and none of my javascript is working and I'm very confused!

It could be that we have jquery packaged, that it's in my
~/.guix-profile/ or etc, but my application doesn't know about it.  This
is "guix environment" for development, so there's no munging the package
during install, I'm hacking *in* the project.

So what can we do?  I still think serving an extlib/ directory or some
easily findable directory is the best way to keep deployment clean, and
anyway, "guix package" can handle setting up the symlinks for that, it's
just "guix environment" that has no way to play along.

What can we do?  My first reaction on IRC is that "maybe there could be
some sort of hook that's run after 'guix environment' that could link
these things into the package".  This was not well received, and for
good reason: it makes "guix environment" non-functional.

Dave Thompson has made what I think is probably a good suggestion: there
should be some sort of environment variable set with something
searchable.  This way, during the "./configure --link-extlib && make" or
"python setup.py develop" or whatever such setup-for-development
process, there's a way to see, aha here's the jquery, symlink that here;
here's the font-awesome, symlink that here; here's the bootstrap,
symlink that here...

I don't know what this environment variable would be called.
WEB_ASSETS_PATH?  The other tricky thing is, it's easy to say "put CSS
files and jquery in that thing", but what about fonts?  Those have
their own location already.  Should both be added to the path?

Anyay, I'd love to have a way to do this with Guix.  If someone can find
a nice solution, I'll make it an option for development with GNU
MediaGoblin instead of using virtualenv/bower. ;)  But we need to know
the right path first!

Looking forward to your ideas,
 - Chris

Re: Replacing Bower with "guix environment"

[David Thompson]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Hello all,
>
> I'm interested in "guix environment" as a universal solution for
> virtualenv/bower/etc.  There's good reasons to want this; aside from
> Guix being awesome, packaging for web applications is getting more and
> more insane and often involves many layers of package management, often
> *multiple language package managers at once*.  It's a sad state of
> affairs.

Agreed, it's terrible.  A modern web application typically requires
*four* separate package managers:

* The system package manager (apt, yum, guix, etc.)
* The server-side language package manager (pip, gem, etc.)
* The static asset package manager (bower)
* The server-side JavaScript package manager (npm)

> Why not have a universal solution?  "guix environment" could be that,
> I hope?  But there are some challenges.
>
> The virtualenv side (having libraries set up for development on the
> Python end) is really easy with "guix environment" assuming those files
> are there: you have your PYTHONPATH set up, done.
>
> But the "bower" side of things is not so easy.  Most web applications
> need to serve many assets to the browser: javascript like jquery, css
> like bootstrap, fonts like fontawesome or really any other
> fonts, not to mention any icon sets, etc.  Web applications need to
> serve these in both a "development" way, usually using some simple
> development server, in which case these have to be easily discovered and
> served.  For production, these need to be served by
> nginx/apache/whatever.  Describing to users how to deploy web
> applications tends to be pretty hard, so for this last one you want to
> make the configuration requirement as much of a one-liner as possible.
> This usually means then that the static assets (jquery, fonts, etc) are
> referenced from somewhere inside the package, usually in an extlib/
> directory or such.
>
> There are two common ways to do this:
>
>  - The "classic" method of doing this is pretty gross: just check all
>    those assets directly into your project, maybe in some sort of
>    "extlib/" directory.  I don't need to tell guix-devel just how bad of
>    a solution this is!  And yet, most web developers (including me
>    *shameface*) have done it, even in our free software projects.

If you take a look at the guix-web repo you'll see that this is what
I've been doing. :X

>  - The "hip new way" of doing things is to use Bower.  Bower is a
>    package manager, but it's made specifically for static assets served
>    to the user, such as css files, fonts, javascript like jquery, etc.
>    Bower also puts these in an extlib/ or whatever, but it puts them in
>    that place *for* you.
>
> It's possible in Guix or Debian packages or whatever to rely on a system
> installed jquery or etc, just a minor pain in the butt, linking in the
> jquery packages and etc manually into that extlib place during building
> the package/derivation.  Okay, not so bad.
>
> But what about "guix environment", where we're directly checked out
> inside the package?  What we want to do is this:
>
>   ~/devel/mediagoblin/$ guix environment -l package.scm
>
> And this will set up the python dependencies for sure, and maybe it can
> even install the jquery package and stuff, but if I run:
>
>   # put your dev server command here
>   ~/devel/mediagoblin/$ ./lazyserver.sh
>
> Oh no!  jquery is installed, but it's not linked into extlib/jquery/
> where I expected it.  So suddenly I'm looking at my development instance
> and none of my javascript is working and I'm very confused!
>
> It could be that we have jquery packaged, that it's in my
> ~/.guix-profile/ or etc, but my application doesn't know about it.  This
> is "guix environment" for development, so there's no munging the package
> during install, I'm hacking *in* the project.
>
> So what can we do?  I still think serving an extlib/ directory or some
> easily findable directory is the best way to keep deployment clean, and
> anyway, "guix package" can handle setting up the symlinks for that, it's
> just "guix environment" that has no way to play along.

To reiterate a bit of what I said on IRC for everyone else: I think this
problem is bigger than 'guix environment', which just sets some
environment variables and runs the command you tell it to run and
nothing more.  The real problem is that there's no notion of
$JAVASCRIPT_PATH or $CSS_PATH that build systems can use to find static
assets dependencies, thus deduplicating them system-wide.  Instead,
every web application just bundles all of the dependencies.  We all know
that bundling is bad for C libraries and such, and Chris and I think
it's clear that bundling is also bad for web assets.  Bower solves this
problem, at the expense of being yet another package manager and
duplicating the assets for each application on your system that uses
Bower.

> What can we do?  My first reaction on IRC is that "maybe there could be
> some sort of hook that's run after 'guix environment' that could link
> these things into the package".  This was not well received, and for
> good reason: it makes "guix environment" non-functional.

Not only is it imperative, but I think it's out of scope for 'guix
environment'.

> Dave Thompson has made what I think is probably a good suggestion: there
> should be some sort of environment variable set with something
> searchable.  This way, during the "./configure --link-extlib && make" or
> "python setup.py develop" or whatever such setup-for-development
> process, there's a way to see, aha here's the jquery, symlink that here;
> here's the font-awesome, symlink that here; here's the bootstrap,
> symlink that here...

Yes, something like that.  I think the most important part of any
solution is that it not be dependent upon Guix tools at all.  Just like
Guix isn't needed to find C libraries at configure time.  I wouldn't
want Guix integrated into web application build systems like Bower is.

Even if we come up with a good solution to this problem, Bower is still
deeply embedded into so many projects.  I wonder if we can be compatible
with it in any way.

> I don't know what this environment variable would be called.
> WEB_ASSETS_PATH?  The other tricky thing is, it's easy to say "put CSS
> files and jquery in that thing", but what about fonts?  Those have
> their own location already.  Should both be added to the path?

I need to think about this more.  I should take Bower for a spin and see
how people are integrating it into their build systems.  How do they
handle fonts?

> Anyay, I'd love to have a way to do this with Guix.  If someone can find
> a nice solution, I'll make it an option for development with GNU
> MediaGoblin instead of using virtualenv/bower. ;)  But we need to know
> the right path first!

Thanks for starting this discussion.  Let's keep brainstorming!

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Replacing Bower with "guix environment"

[Christopher Allan Webber]

David Thompson writes:

> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>
>> Hello all,
>>
>> I'm interested in "guix environment" as a universal solution for
>> virtualenv/bower/etc.  There's good reasons to want this; aside from
>> Guix being awesome, packaging for web applications is getting more and
>> more insane and often involves many layers of package management, often
>> *multiple language package managers at once*.  It's a sad state of
>> affairs.
>
> Agreed, it's terrible.  A modern web application typically requires
> *four* separate package managers:
>
> * The system package manager (apt, yum, guix, etc.)
> * The server-side language package manager (pip, gem, etc.)
> * The static asset package manager (bower)
> * The server-side JavaScript package manager (npm)

Yes you're right... the situation is worse than I even painted it to be!

[ ... snip ... ]

>> Dave Thompson has made what I think is probably a good suggestion: there
>> should be some sort of environment variable set with something
>> searchable.  This way, during the "./configure --link-extlib && make" or
>> "python setup.py develop" or whatever such setup-for-development
>> process, there's a way to see, aha here's the jquery, symlink that here;
>> here's the font-awesome, symlink that here; here's the bootstrap,
>> symlink that here...
>
> Yes, something like that.  I think the most important part of any
> solution is that it not be dependent upon Guix tools at all.

This is a really important point!

> Just like Guix isn't needed to find C libraries at configure time.  I
> wouldn't want Guix integrated into web application build systems like
> Bower is.

I agree.  It would be nice for it to be an option, and a highly
desirable one, for development, but that's just because "guix
environment" hopefully could solve this and other problems for
developers so well.  But if someone already had packages installed via
apt/yum/whatever, that should be good enough.

> Even if we come up with a good solution to this problem, Bower is still
> deeply embedded into so many projects.  I wonder if we can be compatible
> with it in any way.

It's possible we could try to read Bower's json description files
somehow, but I doubt it both because package names may be different and
becuase you have the option of calling out with http requests and etc to
pull down files, etc.  The only way to do that in Guix is to read the
file and help figure out how to make pulling down those files happen so
they can become inputs.  I suspect we'd rather link them to standard
files anyway, where possible.

>> I don't know what this environment variable would be called.
>> WEB_ASSETS_PATH?  The other tricky thing is, it's easy to say "put CSS
>> files and jquery in that thing", but what about fonts?  Those have
>> their own location already.  Should both be added to the path?
>
> I need to think about this more.  I should take Bower for a spin and see
> how people are integrating it into their build systems.  How do they
> handle fonts?

Same as any other static asset, they get pulled down and installed
inside the package directory itself.

>> Anyay, I'd love to have a way to do this with Guix.  If someone can find
>> a nice solution, I'll make it an option for development with GNU
>> MediaGoblin instead of using virtualenv/bower. ;)  But we need to know
>> the right path first!
>
> Thanks for starting this discussion.  Let's keep brainstorming!

Yes, it's a good conversation so far, I hope we can find some solutions!

Re: Replacing Bower with "guix environment"

[David Thompson]

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> David Thompson writes:
>
>> Christopher Allan Webber <cwebber@dustycloud.org> writes:
>>

[SNIP]

>>> Dave Thompson has made what I think is probably a good suggestion: there
>>> should be some sort of environment variable set with something
>>> searchable.  This way, during the "./configure --link-extlib && make" or
>>> "python setup.py develop" or whatever such setup-for-development
>>> process, there's a way to see, aha here's the jquery, symlink that here;
>>> here's the font-awesome, symlink that here; here's the bootstrap,
>>> symlink that here...
>>
>> Yes, something like that.  I think the most important part of any
>> solution is that it not be dependent upon Guix tools at all.
>
> This is a really important point!
>
>> Just like Guix isn't needed to find C libraries at configure time.  I
>> wouldn't want Guix integrated into web application build systems like
>> Bower is.
>
> I agree.  It would be nice for it to be an option, and a highly
> desirable one, for development, but that's just because "guix
> environment" hopefully could solve this and other problems for
> developers so well.  But if someone already had packages installed via
> apt/yum/whatever, that should be good enough.

Yes, people should use Guix for development because it's awesome, not
because software doesn't build without it. ;)

>> Even if we come up with a good solution to this problem, Bower is still
>> deeply embedded into so many projects.  I wonder if we can be compatible
>> with it in any way.
>
> It's possible we could try to read Bower's json description files
> somehow, but I doubt it both because package names may be different and
> becuase you have the option of calling out with http requests and etc to
> pull down files, etc.  The only way to do that in Guix is to read the
> file and help figure out how to make pulling down those files happen so
> they can become inputs.  I suspect we'd rather link them to standard
> files anyway, where possible.

Seems like you're talking about importing.  A 'guix import bower' tool
could be easily written, I think.  We have guile-json, so we can read
the 'bower.json' files and DTRT.  The biggest issue I see is that Bower
doesn't build from source, and people are just checking in minified
JavaScript files to the Git repos that Bower clones.  Our
'bower-build-system' would have to work around this and build from
source.  However, in order to do that, we need a 'node-build-system' so
that we can build the common build tools like Grunt and Uglify.  Are you
still with me or am I going down this rabbit hole alone? ;)

As for build system compatibility, I think that the 'bower_components'
directory structure is quite sane, so it could be a valid
$WEB_ASSET_PATH.

>>> I don't know what this environment variable would be called.
>>> WEB_ASSETS_PATH?  The other tricky thing is, it's easy to say "put CSS
>>> files and jquery in that thing", but what about fonts?  Those have
>>> their own location already.  Should both be added to the path?
>>
>> I need to think about this more.  I should take Bower for a spin and see
>> how people are integrating it into their build systems.  How do they
>> handle fonts?
>
> Same as any other static asset, they get pulled down and installed
> inside the package directory itself.

Thanks.  I confirmed this with a 'bower install bootstrap' (which is a
total mess, btw).

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Replacing Bower with "guix environment"

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

>  - The "hip new way" of doing things is to use Bower.  Bower is a
>    package manager, but it's made specifically for static assets served
>    to the user, such as css files, fonts, javascript like jquery, etc.
>    Bower also puts these in an extlib/ or whatever, but it puts them in
>    that place *for* you.

Interesting.

(Thinking out lout.)

Just like ‘guix system vm’ returns a script that runs QEMU with the
right arguments, one could imagine generating a script that copies
dependencies in the right place maybe?

  (define (make-installer assets)
    (gexp->script "copy-assets"
                  #~(begin
                      (for-each copy-file '#$@assets)
                      ...)))

(This could/should be turned into a package object so that adding it as
an input would drop it in $PATH.)

The developer would have to explicitly run that script to have the files
copied under extlib/.

Alternately one could generate a script that directly runs some http
server with the right parameters so that it finds CSS files, JS files,
etc.

Does that make sense?

Ludo’.

Re: Replacing Bower with "guix environment"

[Christopher Allan Webber]

Ludovic Courtès writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>>  - The "hip new way" of doing things is to use Bower.  Bower is a
>>    package manager, but it's made specifically for static assets served
>>    to the user, such as css files, fonts, javascript like jquery, etc.
>>    Bower also puts these in an extlib/ or whatever, but it puts them in
>>    that place *for* you.
>
> Interesting.
>
> (Thinking out lout.)
>
> Just like ‘guix system vm’ returns a script that runs QEMU with the
> right arguments, one could imagine generating a script that copies
> dependencies in the right place maybe?
>
>   (define (make-installer assets)
>     (gexp->script "copy-assets"
>                   #~(begin
>                       (for-each copy-file '#$@assets)
>                       ...)))
>
> (This could/should be turned into a package object so that adding it as
> an input would drop it in $PATH.)
>
> The developer would have to explicitly run that script to have the files
> copied under extlib/.
>
> Alternately one could generate a script that directly runs some http
> server with the right parameters so that it finds CSS files, JS files,
> etc.
>
> Does that make sense?
>
> Ludo’.

That's a very interesting idea!  (Conveniently, I just gave "guix system
vm" a test yesterday! :))

It makes sense to me, I think.  I'd like to give it a try... I need to
understand gexps better I suppose :)

Re: Replacing Bower with "guix environment"

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>>  - The "hip new way" of doing things is to use Bower.  Bower is a
>>    package manager, but it's made specifically for static assets served
>>    to the user, such as css files, fonts, javascript like jquery, etc.
>>    Bower also puts these in an extlib/ or whatever, but it puts them in
>>    that place *for* you.
>
> Interesting.
>
> (Thinking out lout.)
>
> Just like ‘guix system vm’ returns a script that runs QEMU with the
> right arguments, one could imagine generating a script that copies
> dependencies in the right place maybe?
>
>   (define (make-installer assets)
>     (gexp->script "copy-assets"
>                   #~(begin
>                       (for-each copy-file '#$@assets)
>                       ...)))
>
> (This could/should be turned into a package object so that adding it as
> an input would drop it in $PATH.)
>
> The developer would have to explicitly run that script to have the files
> copied under extlib/.

That is a really neat use of gexps, and I guess running the script
manually would be akin to running 'bower install', so that should work.

I envision the package recipe below, is this approximately what you were
describing?

    (package
      (name "mediagoblin")
      (version "0.8.0")
      ...
      (inputs
       `(("python" ,python)
         ("assets" ,(web-assets jquery
                                videojs
                                bootstrap))))
      ...)

> Alternately one could generate a script that directly runs some http
> server with the right parameters so that it finds CSS files, JS files,
> etc.

I could see that being convenient, but it doesn't help in this
particular case of displacing Bower.

Thanks for your thoughts!

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Replacing Bower with "guix environment"

[Christopher Allan Webber]

David Thompson writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>>
>>>  - The "hip new way" of doing things is to use Bower.  Bower is a
>>>    package manager, but it's made specifically for static assets served
>>>    to the user, such as css files, fonts, javascript like jquery, etc.
>>>    Bower also puts these in an extlib/ or whatever, but it puts them in
>>>    that place *for* you.
>>
>> Interesting.
>>
>> (Thinking out lout.)
>>
>> Just like ‘guix system vm’ returns a script that runs QEMU with the
>> right arguments, one could imagine generating a script that copies
>> dependencies in the right place maybe?
>>
>>   (define (make-installer assets)
>>     (gexp->script "copy-assets"
>>                   #~(begin
>>                       (for-each copy-file '#$@assets)
>>                       ...)))
>>
>> (This could/should be turned into a package object so that adding it as
>> an input would drop it in $PATH.)

The idea has me really excited.  But how do I make this happen, so that
copy-assets appears in $PATH?  I am trying to do something with
"guix environment" that does this, but I'm pretty confused as to how it
could be done.

How would you turn a gexp into a package object?  Would it just be a lot
of package object stuff stubbed out?

>> The developer would have to explicitly run that script to have the files
>> copied under extlib/.
>
> That is a really neat use of gexps, and I guess running the script
> manually would be akin to running 'bower install', so that should work.
>
> I envision the package recipe below, is this approximately what you were
> describing?
>
>     (package
>       (name "mediagoblin")
>       (version "0.8.0")
>       ...
>       (inputs
>        `(("python" ,python)
>          ("assets" ,(web-assets jquery
>                                 videojs
>                                 bootstrap))))
>       ...)

Does web-assets generate that package object I assume?

I tried looking for examples of other package definitions that include
derivations from gexps, and couldn't find anything clear...

It also isn't clear to me: is it okay for (inputs) to include things
that aren't packages?

I'd really like to explore this, though!

Re: Replacing Bower with "guix environment"

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> David Thompson writes:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>>>
>>>>  - The "hip new way" of doing things is to use Bower.  Bower is a
>>>>    package manager, but it's made specifically for static assets served
>>>>    to the user, such as css files, fonts, javascript like jquery, etc.
>>>>    Bower also puts these in an extlib/ or whatever, but it puts them in
>>>>    that place *for* you.
>>>
>>> Interesting.
>>>
>>> (Thinking out lout.)
>>>
>>> Just like ‘guix system vm’ returns a script that runs QEMU with the
>>> right arguments, one could imagine generating a script that copies
>>> dependencies in the right place maybe?
>>>
>>>   (define (make-installer assets)
>>>     (gexp->script "copy-assets"
>>>                   #~(begin
>>>                       (for-each copy-file '#$@assets)
>>>                       ...)))
>>>
>>> (This could/should be turned into a package object so that adding it as
>>> an input would drop it in $PATH.)
>
> The idea has me really excited.  But how do I make this happen, so that
> copy-assets appears in $PATH?  I am trying to do something with
> "guix environment" that does this, but I'm pretty confused as to how it
> could be done.

You would use a ‘web-assets’ procedure as Dave suggests, which would
return a package object that does the above and has the script in its
bin/ directory.

> How would you turn a gexp into a package object?  Would it just be a lot
> of package object stuff stubbed out?

Currently this is not automatic: one basically has to write a package,
typically using ‘trivial-build-system’ which is a thin wrapper around
gexps.

We should add a ‘gexp->package’ procedure.

>>> The developer would have to explicitly run that script to have the files
>>> copied under extlib/.
>>
>> That is a really neat use of gexps, and I guess running the script
>> manually would be akin to running 'bower install', so that should work.
>>
>> I envision the package recipe below, is this approximately what you were
>> describing?
>>
>>     (package
>>       (name "mediagoblin")
>>       (version "0.8.0")
>>       ...
>>       (inputs
>>        `(("python" ,python)
>>          ("assets" ,(web-assets jquery
>>                                 videojs
>>                                 bootstrap))))
>>       ...)
>
> Does web-assets generate that package object I assume?

Yes.

> I tried looking for examples of other package definitions that include
> derivations from gexps, and couldn't find anything clear...

Look for ‘trivial-build-system’.

> It also isn't clear to me: is it okay for (inputs) to include things
> that aren't packages?

Inputs can include packages, origins, derivations, and file names.

HTH!

Ludo’.