Guix Goals: One Hackable Developer Tool To Rule Them All

Guix Goals: One Hackable Developer Tool To Rule Them All

[jgart]

Hi Guixers,

What do you think if we were to implement "Guix Goals"?

The various language ecosystems have their own linters, formatters,
typecheckers, repls, test runners, etc.

What if Guix could manage them all?

`guix lint` in a python project would run mypy.
`guix lint` in a haskell project would run hlint.
`guix lint` in an erlang project would run dialyzer.

`guix fmt` in a python project would run black.
`guix fmt` in a haskell project would run ormolu.
`guix fmt` in a erlang project would run erlfmt.

`guix repl` in a python project would run ptpython or some other configured repl.
`guix repl` in a haskell project would run ghci.
`guix repl` in a erlang project would run erl.

`guix test` in a python project would run pytest or unittest.
`guix test` in a haskell project would run hunit.
`guix test` in a erlang project would run eunit.

`guix run` in a python project could start a flask app listening on a particular port.
`guix run` in a ruby project could start a puma server.
`guix run` in a haskell project could run a pre-configured script or Main.hs
`guix run` in a erlang project could start a cowboy server.

The idea would be to have Guix provide a configurable CLI wrapper
subcommand around all language ecosystem developer tools. In other words
it's the same Guixy thesis applied to developer tooling. I think this
could take the Guix developer experience to the next level.

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[zimoun]

Hi,
On jeu., 13 oct. 2022 at 01:07, jgart <jgart@dismail.de> wrote:

> `guix lint` in a python project would run mypy.

[...]

> `guix fmt` in a python project would run black.

Following your logic, it should be “guix style”.

> `guix repl` in a python project would run ptpython or some other configured repl.

[...]


[...]

> `guix run` in a python project could start a flask app listening on a particular port.

[...]

> The idea would be to have Guix provide a configurable CLI wrapper
> subcommand around all language ecosystem developer tools. In other words
> it's the same Guixy thesis applied to developer tooling. I think this
> could take the Guix developer experience to the next level.

I think it is a bad idea to mix generic Guix commands with per specific
project ones.  For instance, considering a Python project, then if I run
“guix repl”, it launches some Python interpreter, but what if I really
want to start Guix REPL.  Instead, it would appears a better design to
have:

    guix project lint    
    guix project style
    guix project repl
    guix project test
    guix project run

Well, for it is worth, I often deal with some multi-languages projects,
and what I prefer is to explicitly run the per language tool, e.g.,

    guixify ipython

launching some Python interpreter.  I personally prefer an explicit
reference of the tools.

*guixify is currently a tiny Bash script similar to the one pasted here
[1].  But we could imagine a Guile script. :-)

1: <https://yhetil.org/guix/86wn9puqj7.fsf@gmail.com>

Cheers,
simon

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[pinoaffe]

Hi,

I think that (if done well) this would greatly simplify my workflow in
many software projects, so I think it's a good idea.

In the rest of my email, I more or less assume that there is a
one-to-one correspondence between software projects and guix packages,
even though this is clearly not the case.  I nonetheless make this
assumption since it is the case (or can be made the case) for many of
the software projects on which one might want to work using guix.

jgart <jgart@dismail.de> writes:
> The various language ecosystems have their own linters, formatters,
> typecheckers, repls, test runners, etc.
>
> What if Guix could manage them all?
>
> `guix lint` in a python project would run mypy.
> `guix lint` in a haskell project would run hlint.
> `guix lint` in an erlang project would run dialyzer.

These would require some sort of configurability, since most
languages/projects have not just one linter.  Even if there's a
"monopoly" there might be different settings/versions of said linter.

Do you propose that such configuration is added to packages?  Or should
this go somewhere else?

Furthermore, in many projects there are various different languages at
once (e.g., a python webapp with client-side javascript and css), so it
would probably be useful to mirror the build-phases approach in guix
packaging.  It might even be possible to set up (somewhat) sane defaults
based on the build-system used by the local project.

> `guix fmt` in a python project would run black.
> `guix fmt` in a haskell project would run ormolu.
> `guix fmt` in a erlang project would run erlfmt.

My comments regarding linters apply here as well

> `guix repl` in a python project would run ptpython or some other configured repl.
> `guix repl` in a haskell project would run ghci.
> `guix repl` in a erlang project would run erl.

This should be fairly easy to implement, assuming that there is just one
repl associated to each project one might want to use this command with.
It should (imo) be possible to configure/pass arguments to the repl on a
per-project basis and whenever you run `guix repl`

> `guix test` in a python project would run pytest or unittest.
> `guix test` in a haskell project would run hunit.
> `guix test` in a erlang project would run eunit.

Should I read this as a proposal to add an extra command to build (and
test) the local project's guix package, or do you propose to make this
do something different?  Because I'm not sure the latter would be a good
idea, if there is a need to do testing in some other way than currently
possible I'd think it would be more valuable (and more clean) to extend
the current testing facilities to incorporate this.

> `guix run` in a python project could start a flask app listening on a particular port.
> `guix run` in a ruby project could start a puma server.
> `guix run` in a haskell project could run a pre-configured script or Main.hs
> `guix run` in a erlang project could start a cowboy server.

I'm not sure how widely this may be applicable, since many projects or
packages don't have just one way to run them, they may even contain
several different executables or modules.

> The idea would be to have Guix provide a configurable CLI wrapper
> subcommand around all language ecosystem developer tools. In other words
> it's the same Guixy thesis applied to developer tooling. I think this
> could take the Guix developer experience to the next level.

Kind regards,
pinoaffe

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[indieterminacy]

Sorry if this comes off as facetious (because this is an interesting 
proposal) but hasnt Make cornered a lot of these usecases well enough?

On 13-10-2022 15:56, pinoaffe wrote:
> Hi,
> 
> I think that (if done well) this would greatly simplify my workflow in
> many software projects, so I think it's a good idea.
> 
> In the rest of my email, I more or less assume that there is a
> one-to-one correspondence between software projects and guix packages,
> even though this is clearly not the case.  I nonetheless make this
> assumption since it is the case (or can be made the case) for many of
> the software projects on which one might want to work using guix.
> 
> jgart <jgart@dismail.de> writes:
>> The various language ecosystems have their own linters, formatters,
>> typecheckers, repls, test runners, etc.
>> 
>> What if Guix could manage them all?
>> 
>> `guix lint` in a python project would run mypy.
>> `guix lint` in a haskell project would run hlint.
>> `guix lint` in an erlang project would run dialyzer.
> 
> These would require some sort of configurability, since most
> languages/projects have not just one linter.  Even if there's a
> "monopoly" there might be different settings/versions of said linter.
> 
> Do you propose that such configuration is added to packages?  Or should
> this go somewhere else?
> 
> Furthermore, in many projects there are various different languages at
> once (e.g., a python webapp with client-side javascript and css), so it
> would probably be useful to mirror the build-phases approach in guix
> packaging.  It might even be possible to set up (somewhat) sane 
> defaults
> based on the build-system used by the local project.
> 
>> `guix fmt` in a python project would run black.
>> `guix fmt` in a haskell project would run ormolu.
>> `guix fmt` in a erlang project would run erlfmt.
> 
> My comments regarding linters apply here as well
> 
>> `guix repl` in a python project would run ptpython or some other 
>> configured repl.
>> `guix repl` in a haskell project would run ghci.
>> `guix repl` in a erlang project would run erl.
> 
> This should be fairly easy to implement, assuming that there is just 
> one
> repl associated to each project one might want to use this command 
> with.
> It should (imo) be possible to configure/pass arguments to the repl on 
> a
> per-project basis and whenever you run `guix repl`
> 
>> `guix test` in a python project would run pytest or unittest.
>> `guix test` in a haskell project would run hunit.
>> `guix test` in a erlang project would run eunit.
> 
> Should I read this as a proposal to add an extra command to build (and
> test) the local project's guix package, or do you propose to make this
> do something different?  Because I'm not sure the latter would be a 
> good
> idea, if there is a need to do testing in some other way than currently
> possible I'd think it would be more valuable (and more clean) to extend
> the current testing facilities to incorporate this.
> 
>> `guix run` in a python project could start a flask app listening on a 
>> particular port.
>> `guix run` in a ruby project could start a puma server.
>> `guix run` in a haskell project could run a pre-configured script or 
>> Main.hs
>> `guix run` in a erlang project could start a cowboy server.
> 
> I'm not sure how widely this may be applicable, since many projects or
> packages don't have just one way to run them, they may even contain
> several different executables or modules.
> 
>> The idea would be to have Guix provide a configurable CLI wrapper
>> subcommand around all language ecosystem developer tools. In other 
>> words
>> it's the same Guixy thesis applied to developer tooling. I think this
>> could take the Guix developer experience to the next level.
> 
> Kind regards,
> pinoaffe

-- 
Jonathan McHugh
indieterminacy@libre.brussels

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[pinoaffe]

indieterminacy <indieterminacy@libre.brussels> writes:

> Sorry if this comes off as facetious (because this is an interesting
> proposal) but hasnt Make cornered a lot of these usecases well enough?

It has covered some of these usecases to a certain degree, but in my
opinion there's a lot of room for improvement.

As an example, setting up make to run a particular command in a
particular environment (e.g., with certain packages available) is (imo)
quite annoying.  It would be possible to shell out to guix from within a
makefile, but at that point, it's (imo) way cleaner if this can be
handled by guix.

Furthermore, doing this ckind of configuration in a makefile means that
the same or very similar make commands would need to be put in many
projects' makefiles, this could be implemented a lot cleaner within guix.

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[jgart]

On Thu, 13 Oct 2022 16:55:33 +0200 pinoaffe <pinoaffe@airmail.cc> wrote:

What I proposed is not an original idea but what the pants project
does. See this doc:

https://www.pantsbuild.org/v1.29/docs/goals

Pants fully supports Python but they are quickly working on golang,
java, and other languages.

zimoun, I'm mostly just brainstorming the general idea at the moment. I don't
mind if it takes the form of `guix project ...` or some other form.

pinoaffe, I agree that the developer commands should be completely
Guile configurable and hackable to our hacker heart's content, of
course. That's why we're all here ;()

Jonathan,

> but hasnt Make cornered a lot of these usecases well enough?

That didn't stop Guixers from making the gnu-build-system that paren
hugs Make. Why stop there? If you want to call mypy from Make that's
fine but I'd like to call mypy from Make from guix or why not cut out
the middleman altogether? In the end I just want to see the same unifying
guix ui experience transfered to dev tools across the polyglot stack.

This infrastructure should be flexible enough to allow us to
easily go unix plumbing if need be.

Let's wrap the world in parens!

paren hugs,

jgart 

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[Liliana Marie Prikler]

Am Donnerstag, dem 13.10.2022 um 01:07 -0500 schrieb jgart:
> Hi Guixers,
> 
> What do you think if we were to implement "Guix Goals"?
> 
> The various language ecosystems have their own linters, formatters,
> typecheckers, repls, test runners, etc.
> 
> What if Guix could manage them all?
> 
> `guix lint` in a python project would run mypy.
> `guix lint` in a haskell project would run hlint.
> `guix lint` in an erlang project would run dialyzer.
> 
> `guix fmt` in a python project would run black.
> `guix fmt` in a haskell project would run ormolu.
> `guix fmt` in a erlang project would run erlfmt.
> 
> `guix repl` in a python project would run ptpython or some other
> configured repl.
> `guix repl` in a haskell project would run ghci.
> `guix repl` in a erlang project would run erl.
> 
> `guix test` in a python project would run pytest or unittest.
> `guix test` in a haskell project would run hunit.
> `guix test` in a erlang project would run eunit.
> 
> `guix run` in a python project could start a flask app listening on a
> particular port.
> `guix run` in a ruby project could start a puma server.
> `guix run` in a haskell project could run a pre-configured script or
> Main.hs
> `guix run` in a erlang project could start a cowboy server.
> 
> The idea would be to have Guix provide a configurable CLI wrapper
> subcommand around all language ecosystem developer tools. In other
> words it's the same Guixy thesis applied to developer tooling. I
> think this could take the Guix developer experience to the next
> level.
Completely unnecessary.  You can already do this with guix shell or gwl
if you prefer.

To elaborate, note that only the last variable in a file or manifest
needs to evaluate to a list of packages or manifest.  That is, you can
write a self-reading manifest/script wrapper relatively easy.

  #!/bin/sh
  exec guix shell -m "$0" -- guile -e main -s "$0" "$@" 
  !#

  (define (main args) ...)

  ...

  (specifications->manifest ...)

Of course, it'd be hell of a lot cleaner to separate manifest and
business logic, but who am I to stop you?

Cheers

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[jgart]

On Sun, 16 Oct 2022 15:11:38 +0200 Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:
> Am Donnerstag, dem 13.10.2022 um 01:07 -0500 schrieb jgart:
>   #!/bin/sh
>   exec guix shell -m "$0" -- guile -e main -s "$0" "$@" 
>   !#
> 
>   (define (main args) ...)
> 
>   ...
> 
>   (specifications->manifest ...)
> 
> Of course, it'd be hell of a lot cleaner to separate manifest and
> business logic, but who am I to stop you?

Hi lilyp,

Yup, the latter is what I'm proposing instead of implementing a goal
for each lang à"ala carte" as a script for each project.

That would be like implementing part of a build-system everytime
when needed like we currently do when we needing to run pytest
instead their being first class support like was being worked on in:
https://github.com/guix-mirror/guix/tree/wip-python-pep517.

I'd like for "goals" to have first class support like guix home configs,
etc. do. But thanks for reminding us that we can just implement it
directly as needed in a script for each project. That's a testament to
the extensibility of Guix!

> Of course, it'd be hell of a lot cleaner to separate manifest and
business logic, but who am I to stop you?

I just need to find the time to implement the framework/system and
propose it again with a patch series ;() Not sure when that will be yet...

all best,

jgart

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[jgart]

On Sun, 16 Oct 2022 12:54:42 -0500 jgart <jgart@dismail.de> wrote:

That said thanks for suggesting a way to start hacking on this before it exists.
I'll use your suggestion for testing things out.

all best,

jgart

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[Liliana Marie Prikler]

Am Sonntag, dem 16.10.2022 um 12:54 -0500 schrieb jgart:
> On Sun, 16 Oct 2022 15:11:38 +0200 Liliana Marie Prikler
> <liliana.prikler@gmail.com> wrote:
> > Am Donnerstag, dem 13.10.2022 um 01:07 -0500 schrieb jgart:
> >   #!/bin/sh
> >   exec guix shell -m "$0" -- guile -e main -s "$0" "$@" 
> >   !#
> > 
> >   (define (main args) ...)
> > 
> >   ...
> > 
> >   (specifications->manifest ...)
> > 
> > Of course, it'd be hell of a lot cleaner to separate manifest and
> > business logic, but who am I to stop you?
> 
> Hi lilyp,
> 
> Yup, the latter is what I'm proposing instead of implementing a goal
> for each lang à"ala carte" as a script for each project.
This à la carte approach will definitely break down in polyglot
applications – hence needing a more generic solution for the generic
case – and even when there's just one language, the fact that multiple
compilers, linters, etc. exist and people are very opinionated on which
ones to use with which options makes your goal a somewhat stupid one if
you don't actually specify what it has to do on a per-project basis.

> That would be like implementing part of a build-system everytime
> when needed like we currently do when we needing to run pytest
> instead their being first class support like was being worked on in:
> https://github.com/guix-mirror/guix/tree/wip-python-pep517.
> 
> I'd like for "goals" to have first class support like guix home
> configs, etc. do. But thanks for reminding us that we can just
> implement it directly as needed in a script for each project. That's
> a testament to the extensibility of Guix!
I don't think it'd be that.  Note that you have the full power of
Guix/Guile at your disposal, so you can encapsulate much of it in
channels.  Is it a good idea to do so?  Hell, no.  Refusing to use the
tools your colleagues are using to instead hack on your own is not a
recipe for success.  There are good reasons why generic build systems
like make and ninja are more widely adopted than their language-focused
counterparts.  You're not doing anyone a favour here by forcing them to
adopt a solution that has parentheses.

Instead of aiming for world domination, you should try writing a tool
that helps folks collaborate.  Adding a guix.scm alone already does
wonders in this regard, but if you really want to go beyond, you could
define a set of workflows with gwl or fall back to ye good olde make
and ninja.  And if you don't like either, there's potato-make [1].

Cheers

[1] https://github.com/spk121/potato-make

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[jgart]

On Sun, 16 Oct 2022 20:50:47 +0200 Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

> makes your goal a somewhat stupid one if you don't actually specify
> what it has to do on a per-project basis.

hi lilyp,

That's a bit harsh to say that my idea is "stupid" even in light of
your qualifier.

If you read the previous thread you can see some of my proposals on a
per language basis.

There's absolutely no requirement for me to work out the whole solution
just yet on this email thread or to offer per project details. I'm not at
the stage of proposing per-project ideas. I'm just casually brainstorming
with those colleagues who want to. If that's too stressful to do then
anyone is free to opt out of the conversation. There's no need to call it
"stupid". Instead, ask me to expound on it in a friendly way instead if
something is not clear or not yet stated.

> I don't think it'd be that.  Note that you have the full power of
> Guix/Guile at your disposal, so you can encapsulate much of it in
> channels.  Is it a good idea to do so?  Hell, no.    

If Guix channels don't give users enough freedom/hackability/extensibility
to do what they want then maybe that's something that should be looked
into. I would consider it a bug, and a limitation of the system. There
should be enough trust in our community to allow users to hack channels
the way they want and to contribute back to GNU Guix proper if they
so choose.  Restrictions shouldn't be built into the system to prevent
users from doing x, y, z with regards to extending it.  There should
only be trust regarding extensibility. That's what made Guix stand out
to me from the start. If that is not the case going forward then I will
lose hope in Guix and would rather invest my time chasing that dream
elsewhere. The extensibility thesis that Guix subscribes to (atleast
to my understanding) involves trust in its users instead of giving them
a limiting DSL and telling them what they are and are not allowed to do
with it.

I came to GNU Guix because of this freedom of extensibility and these
ideals that I perceived in channels and in the system as a whole.
Contributing back to upstream not a limiting feature that is "designed in"
to Guix in order to enforce it. I don't want to be limited that way. This
trust has to exist if it wants to cooperate with the Schemer spirit.

> Refusing to use the tools your colleagues are using to instead hack on your own is not a
> recipe for success.

I'm currently using my colleagues tools extensively. Not sure what you
mean here...

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi,

On Sun, Oct 16, 2022 at 1:10 PM jgart <jgart@dismail.de> wrote:
>
> That's a bit harsh.

I also found it a bit stiff, but sometimes folks don't find the right
words—hence my own recent apology.

Kind regards
Felix Lechner

Re: Guix Goals: One Hackable Developer Tool To Rule Them All

[Liliana Marie Prikler]

Am Sonntag, dem 16.10.2022 um 15:03 -0500 schrieb jgart:
> On Sun, 16 Oct 2022 20:50:47 +0200 Liliana Marie Prikler
> <liliana.prikler@gmail.com> wrote:
> 
> > makes your goal a somewhat stupid one if you don't actually specify
> > what it has to do on a per-project basis.
> 
> hi lilyp,
> 
> That's a bit harsh to say that my idea is "stupid" even in light of
> your qualifier.
> 
> If you read the previous thread you can see some of my proposals on a
> per language basis.
For context, the "goal" here refers to the individual goal, e.g.
"format", as specified in the initial post.  If it only ever covers the
most common case per programming language, that is a very stupid tool
indeed.

> There's absolutely no requirement for me to work out the whole
> solution just yet on this email thread or to offer per project
> details. I'm not at the stage of proposing per-project ideas. I'm
> just casually brainstorming with those colleagues who want to. If
> that's too stressful to do then anyone is free to opt out of the
> conversation. There's no need to call it "stupid". Instead, ask me to
> expound on it in a friendly way instead if something is not clear or
> not yet stated.
I am insisting on this notion of per-projectness, because it is key to
you understanding that you are indeed trying to construct a build
system.¹  Perhaps a very crude one, but a build system still.  Your
logical foundation are the following statements:

1. Every project written in L uses T for a specific task.
2. P is written in L.
3. Therefore P uses T for that task.

By constructing a per-task mapping of L ~> T, you can offer a unified
solution.

The truth, however, is more complicated.

1. Every sufficiently mature programming language L has at least one
task such that at least two tools T, T', T != T' are used in different
projects to achieve it.
2. There exist projects that are not limited to a single programming
language.

You can of course try to be smarter than that and try to figure out
which tools to invoke based on the existence and contents of certain
magic files, but note that at this point you're doing significantly
more work than in the declarative approach you rejected.

> > I don't think it'd be that.  Note that you have the full power of
> > Guix/Guile at your disposal, so you can encapsulate much of it in
> > channels.  Is it a good idea to do so?  Hell, no.    
> 
> If Guix channels don't give users enough
> freedom/hackability/extensibility
> to do what they want then maybe that's something that should be
> looked into. I would consider it a bug, and a limitation of the
> system. There should be enough trust in our community to allow users
> to hack channels the way they want and to contribute back to GNU Guix
> proper if they so choose.  Restrictions shouldn't be built into the
> system to prevent users from doing x, y, z with regards to extending
> it.  There should only be trust regarding extensibility. That's what
> made Guix stand out to me from the start. If that is not the case
> going forward then I will lose hope in Guix and would rather invest
> my time chasing that dream elsewhere. The extensibility thesis that
> Guix subscribes to (atleast to my understanding) involves trust in
> its users instead of giving them a limiting DSL and telling them what
> they are and are not allowed to do with it.
> 
> I came to GNU Guix because of this freedom of extensibility and these
> ideals that I perceived in channels and in the system as a whole.
> Contributing back to upstream not a limiting feature that is
> "designed in" to Guix in order to enforce it. I don't want to be
> limited that way. This trust has to exist if it wants to cooperate
> with the Schemer spirit.
I believe you're attacking a straw man here.  I'm not telling you that
you can't run with scissors, only that it'd be unwise if you did.

> > Refusing to use the tools your colleagues are using to instead hack
> > on your own is not a recipe for success.
> 
> I'm currently using my colleagues tools extensively. Not sure what
> you mean here...
This is calling back to the main motivation outlined in the initial
post; rather than invoking the linters, formatters, etc., you simply
wrap them in Guix.  It is almost always preferable to provide a
solution that works even for those who don't use Guix – for example a
"make indent" rule, etc.

Cheers

¹ Also note that pants, which seems to be a source of your inspiration,
describes itself as a build system.  Compare the already mentioned GWL
for a build system already based on Guix, albeit focused on scientific
workflows, and potato-make for a simplistic Guile-based build system.