Trying to contribute packages: ./pre-inst-env doesn't work

Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

Hey all,

Coming from Arch Linux, I'm trying to add new packages and contribute
them to the Guix project. However, I have a few doubts as for dictating
a workflow and most importantly I can't seem to be able to actually test
my new packages' definitions with `./pre-inst-env`.

First of all, I'm not sure whether I should define custom channels or
not. After reading the section Channels in the manual, it seemed
reasonable for me that in order to test my fork of Guix, I should create
a branch with my new packages and then test it with `guix pull` and
`guix build`. But `guix search` and `guix build` don't seem to be able
to find the new packages I defined and I couldn't find a way to debug
this.

To be more specific, I've tried this idea by writing the following in
`~/.config/guix/channels.scm`:

(list (channel
        (name 'guix)
        (url "file:///var/code/doron/guix")))

I ran `guix pull` and it seems it checked out my version of Guix but it
make those new packages available.

-----

Failing to achieve my target with custom channels, I've disabled the
custom channels I configured above and I tried following the
Contributing section in the manual. I jumped straight to the Packaging
Guidelines section and I've done the following:

- I ran ./configure inside /var/code/doron/guix
- I killed the systemd service of guix-daemon and I ran this command:
  `sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild`
- In another shell, I tried running:
  `./pre-inst-env guix build my-new-package`

But it reports that the package `my-new-package` is unknown.

-----

Failing to do that and after looking for alternative ways to create new
packages, I read about the `--load-path` option of `guix build`. Is it
possible to use it for new packages testings? I tried to run the
following but I got all kinds of warnings and errors:

guix build -L/var/code/doron/guix my-new-package

Here is all of the output:
https://gist.github.com/doronbehar/38d31142522fd01a67e7da412b0cf5ed

Since there were all of these "source file newer than compiled"
messages, I tried putting back my channels configuration from above and
run `guix pull` again but I got pretty much the same errors and
warnings.

-----

Any way, I'm very frustrated. Although I'm very excited as for the
genius design of Guix and it's flexibility and extensibility, I'm very
disappointed by it's UX for a simple thing as package contribution. but
this should be addressed in a separate discussion.

I hope someone could help me.

Thanks, Doron.

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Gábor Boskovits]

Hello Doron!

Doron Behar <doron.behar@gmail.com> ezt írta (időpont: 2019. máj. 15., Sze,
10:09):

> Hey all,
>
> Coming from Arch Linux, I'm trying to add new packages and contribute
> them to the Guix project. However, I have a few doubts as for dictating
> a workflow and most importantly I can't seem to be able to actually test
> my new packages' definitions with `./pre-inst-env`.
>
> First of all, I'm not sure whether I should define custom channels or
> not. After reading the section Channels in the manual, it seemed
> reasonable for me that in order to test my fork of Guix, I should create
> a branch with my new packages and then test it with `guix pull` and
> `guix build`. But `guix search` and `guix build` don't seem to be able
> to find the new packages I defined and I couldn't find a way to debug
> this.
>
> To be more specific, I've tried this idea by writing the following in
> `~/.config/guix/channels.scm`:
>
> (list (channel
>         (name 'guix)
>         (url "file:///var/code/doron/guix")))
>
> I ran `guix pull` and it seems it checked out my version of Guix but it
> make those new packages available.
>
>
It might be easier to first set up a channel with your packages only. You
can find
several examples out there.
Here is one of my channels:
https://gitlab.com/g_bor/nominatim-channel


> -----
>
> Failing to achieve my target with custom channels, I've disabled the
> custom channels I configured above and I tried following the
> Contributing section in the manual. I jumped straight to the Packaging
> Guidelines section and I've done the following:
>
> - I ran ./configure inside /var/code/doron/guix
>

Did you specify your localstatedir to configure?
Did you also run make?

Are your packages in a separate file?
If yes, did you add it to local.mk?

- I killed the systemd service of guix-daemon and I ran this command:
>   `sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild`
> - In another shell, I tried running:
>   `./pre-inst-env guix build my-new-package`
>
> But it reports that the package `my-new-package` is unknown.
>
> -----
>
> Failing to do that and after looking for alternative ways to create new
> packages, I read about the `--load-path` option of `guix build`. Is it
> possible to use it for new packages testings? I tried to run the
> following but I got all kinds of warnings and errors:
>
> guix build -L/var/code/doron/guix my-new-package
>
>
I would not do that, the channel mechanism superseded this.

Here is all of the output:
> https://gist.github.com/doronbehar/38d31142522fd01a67e7da412b0cf5ed
>
> Since there were all of these "source file newer than compiled"
>

These are actually harmless, they indicate that your .go files should be
regenerated, as they are older than the .scm files. You can run make to
get rid of these. (Have not looked at the error, tough)


> messages, I tried putting back my channels configuration from above and
> run `guix pull` again but I got pretty much the same errors and
> warnings.
>
> -----
>
> Any way, I'm very frustrated. Although I'm very excited as for the
> genius design of Guix and it's flexibility and extensibility, I'm very
> disappointed by it's UX for a simple thing as package contribution. but
> this should be addressed in a separate discussion

.
>

I am very sorry to hear that. I hope you can point us to the right
directions
to make this easier. Suggestions about documentation are also very welcome.


>
> I hope someone could help me.
>
> Thanks, Doron.
>
>
Best regards,
g_bor

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[pelzflorian (Florian Pelz)]

On Wed, May 15, 2019 at 11:06:54AM +0300, Doron Behar wrote:
> To be more specific, I've tried this idea by writing the following in
> `~/.config/guix/channels.scm`:
> 
> (list (channel
>         (name 'guix)
>         (url "file:///var/code/doron/guix")))
> 

It could be (url "/var/code/doron/guix").

> I ran `guix pull` and it seems it checked out my version of Guix but it
> make those new packages available.
>

Maybe you have not done `git add gnu/packages/your-package.scm && git
commit`.

Regards,
Florian

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

Thanks for the quick reply!

On Wed, May 15, 2019 at 10:46:29AM +0200, Gábor Boskovits wrote:
> >
> It might be easier to first set up a channel with your packages only. You
> can find
> several examples out there.
> Here is one of my channels:
> https://gitlab.com/g_bor/nominatim-channel

Oh right, thanks. That's an excellent reference.

> >
> > - I ran ./configure inside /var/code/doron/guix
> >
> 
> Did you specify your localstatedir to configure?

Yes, I've given it the same flags I gave it when I've build it at first.

> Did you also run make?

No! I was hoping I could avoid that as that's extremely CPU intensive..
Is there a way to run make with only the relevant targets? That could be
very helpful...

Plus, do I have to run make after every change to a package definition!?
Will it compile everything from the ground up after every such change? I
wish these technical details would have been documented...

I'm running make while I'm writing this email and I must say that not
only it takes a lot of time and uses a lot of CPU power, it handles way
too much irrelevant things such as translations.

I also noticed it runs GUILEC - is it really necessary? I'm no Guile
expert and I wish I could avoid diving to the whole ecosystem of it just
because I want to use Guix and contribute some packages!

I talked about this subject in a different thread: The design choice of
putting all package definitions in the same repository as the package
manager it self is very uncommon in the Linux. I understand that almost
all of Guix is written in Guile so it must be linked somehow to the
package definitions but I think it makes it very hard to actually
contribute something to the packages definitions! It seems that we,
package maintainers, or to put it simply - the community, need to be too
much involved in the source code of everything in order to help the
distributions. That's rather unfortunate IMHO. I'd like to be convinced
I'm wrong at least in this subject because as I dive more and more into
Guix I feel there have been made a lot of poor design choices in it..

> 
> Are your packages in a separate file?
> If yes, did you add it to local.mk?

No, I've added new definitions to existing files.

> > -----
> >
> > Failing to do that and after looking for alternative ways to create new
> > packages, I read about the `--load-path` option of `guix build`. Is it
> > possible to use it for new packages testings? I tried to run the
> > following but I got all kinds of warnings and errors:
> >
> > guix build -L/var/code/doron/guix my-new-package
> >
> >
> I would not do that, the channel mechanism superseded this.

I see, makes sense.

> > Here is all of the output:
> > https://gist.github.com/doronbehar/38d31142522fd01a67e7da412b0cf5ed
> >
> > Since there were all of these "source file newer than compiled"
> >
> 
> These are actually harmless, they indicate that your .go files should be
> regenerated, as they are older than the .scm files. You can run make to
> get rid of these. (Have not looked at the error, tough)

Oh right, makes sense in accordance to your previous comment on running
make.

> 
> 
> > messages, I tried putting back my channels configuration from above and
> > run `guix pull` again but I got pretty much the same errors and
> > warnings.
> >
> > -----
> >
> > Any way, I'm very frustrated. Although I'm very excited as for the
> > genius design of Guix and it's flexibility and extensibility, I'm very
> > disappointed by it's UX for a simple thing as package contribution. but
> > this should be addressed in a separate discussion
> 
> .
> >
> 
> I am very sorry to hear that. I hope you can point us to the right
> directions
> to make this easier. Suggestions about documentation are also very welcome.

Oh right. When I'll actually understand everything hopefully with your
guidance, I hope I'll find the time to send some patches.

Doron.

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

On Wed, May 15, 2019 at 11:08:20AM +0200, pelzflorian (Florian Pelz) wrote:
> On Wed, May 15, 2019 at 11:06:54AM +0300, Doron Behar wrote:
> > To be more specific, I've tried this idea by writing the following in
> > `~/.config/guix/channels.scm`:
> > 
> > (list (channel
> >         (name 'guix)
> >         (url "file:///var/code/doron/guix")))
> > 
> 
> It could be (url "/var/code/doron/guix").
> 
> > I ran `guix pull` and it seems it checked out my version of Guix but it
> > make those new packages available.
> >
> 
> Maybe you have not done `git add gnu/packages/your-package.scm && git
> commit`.

I did. Please assume that I'm very experienced with Git in general, just
not much experienced with Guile and the Gnu Build system.

> 
> Regards,
> Florian
> 

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

To add to my previous message, I'll note that after I `make` finally
finished, I've encountered the same issue:

I ran the guix-daemon command:

sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild

Afterwards, in a different shell, I ran

guix environment guix

And afterwards, finally:

./pre-inst-env guix build my-new-package

But that returned with the same error that this package doesn't exist.

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Gábor Boskovits]

Hello,

Doron Behar <doron.behar@gmail.com> ezt írta (időpont: 2019. máj. 15., Sze,
11:47):

> Thanks for the quick reply!
>
> On Wed, May 15, 2019 at 10:46:29AM +0200, Gábor Boskovits wrote:
> > >
> > It might be easier to first set up a channel with your packages only. You
> > can find
> > several examples out there.
> > Here is one of my channels:
> > https://gitlab.com/g_bor/nominatim-channel
>
> Oh right, thanks. That's an excellent reference.
>
> > >
> > > - I ran ./configure inside /var/code/doron/guix
> > >
> >
> > Did you specify your localstatedir to configure?
>
> Yes, I've given it the same flags I gave it when I've build it at first.
>
> > Did you also run make?
>
> No! I was hoping I could avoid that as that's extremely CPU intensive..
> Is there a way to run make with only the relevant targets? That could be
> very helpful...
>
> Plus, do I have to run make after every change to a package definition!?
> Will it compile everything from the ground up after every such change? I
> wish these technical details would have been documented...
>
>
No, you don't need to do that. Guix will interpret the changed sources, and
it will add a warning that the source file is newer.


> I'm running make while I'm writing this email and I must say that not
> only it takes a lot of time and uses a lot of CPU power, it handles way
> too much irrelevant things such as translations.
>
> I also noticed it runs GUILEC - is it really necessary? I'm no Guile
> expert and I wish I could avoid diving to the whole ecosystem of it just
> because I want to use Guix and contribute some packages!
>
> I talked about this subject in a different thread: The design choice of
> putting all package definitions in the same repository as the package
> manager it self is very uncommon in the Linux. I understand that almost
> all of Guix is written in Guile so it must be linked somehow to the
> package definitions but I think it makes it very hard to actually
>

The channels mechanism actually makes it possible to split off any
part of the packages. So there is no limitations in moving most of
the packages definitions out of guix. I am not familiar enough to
tell if all package definitions could be moved into channels, but for
most of them this could be done.

Could you elaborate on the benefits of doing that?
I guess this could be a great topic of discussion.


> contribute something to the packages definitions! It seems that we,
> package maintainers, or to put it simply - the community, need to be too
> much involved in the source code of everything in order to help the
> distributions. That's rather unfortunate IMHO. I'd like to be convinced
> I'm wrong at least in this subject because as I dive more and more into
> Guix I feel there have been made a lot of poor design choices in it..
>
> >
> > Are your packages in a separate file?
> > If yes, did you add it to local.mk?
>
> No, I've added new definitions to existing files.
>

Ok, then that is not the problem.


>
> > > -----
> > >
> > > Failing to do that and after looking for alternative ways to create new
> > > packages, I read about the `--load-path` option of `guix build`. Is it
> > > possible to use it for new packages testings? I tried to run the
> > > following but I got all kinds of warnings and errors:
> > >
> > > guix build -L/var/code/doron/guix my-new-package
> > >
> > >
> > I would not do that, the channel mechanism superseded this.
>
> I see, makes sense.
>
> > > Here is all of the output:
> > > https://gist.github.com/doronbehar/38d31142522fd01a67e7da412b0cf5ed
> > >
> > > Since there were all of these "source file newer than compiled"
> > >
> >
> > These are actually harmless, they indicate that your .go files should be
> > regenerated, as they are older than the .scm files. You can run make to
> > get rid of these. (Have not looked at the error, tough)
>
> Oh right, makes sense in accordance to your previous comment on running
> make.
>
> >
> >
> > > messages, I tried putting back my channels configuration from above and
> > > run `guix pull` again but I got pretty much the same errors and
> > > warnings.
> > >
> > > -----
> > >
> > > Any way, I'm very frustrated. Although I'm very excited as for the
> > > genius design of Guix and it's flexibility and extensibility, I'm very
> > > disappointed by it's UX for a simple thing as package contribution. but
> > > this should be addressed in a separate discussion
> >
> > .
> > >
> >
> > I am very sorry to hear that. I hope you can point us to the right
> > directions
> > to make this easier. Suggestions about documentation are also very
> welcome.
>
> Oh right. When I'll actually understand everything hopefully with your
> guidance, I hope I'll find the time to send some patches.
>
> Doron.
>
>
Best regards,
g_bor

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Gábor Boskovits]

Hello Doron,

Doron Behar <doron.behar@gmail.com> ezt írta (időpont: 2019. máj. 15., Sze,
12:23):

> To add to my previous message, I'll note that after I `make` finally
> finished, I've encountered the same issue:
>
> I ran the guix-daemon command:
>
> sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild
>

This should only be needed if you don't run it from a service manager.
Changes to the daemon are rare, and most of the times you should be
able to use the old daemon. But this should not cause a problem either.

>
> Afterwards, in a different shell, I ran
>
> guix environment guix
>
> I also believe this is not strictly needed, most of the times it works for
me without this,
once make finished.

> And afterwards, finally:
>
> ./pre-inst-env guix build my-new-package
>
> But that returned with the same error that this package doesn't exist.
>
>
Could you share the diff?

I don't have any more idea without seeing what you were trying to do.

Best regards,
g_bor

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

On Wed, May 15, 2019 at 12:41:45PM +0200, Gábor Boskovits wrote:
> > > Did you also run make?
> >
> > No! I was hoping I could avoid that as that's extremely CPU intensive..
> > Is there a way to run make with only the relevant targets? That could be
> > very helpful...
> >
> > Plus, do I have to run make after every change to a package definition!?
> > Will it compile everything from the ground up after every such change? I
> > wish these technical details would have been documented...
> >
> >
> No, you don't need to do that. Guix will interpret the changed sources, and
> it will add a warning that the source file is newer.

I've finally succeeded in setting up my environment for building and
contributing packages. Thanks for making it clear that `make` is needed.
This is certainly not explicitly mentioned in the docs as far as I can
see.

> 
> 
> > I'm running make while I'm writing this email and I must say that not
> > only it takes a lot of time and uses a lot of CPU power, it handles way
> > too much irrelevant things such as translations.
> >
> > I also noticed it runs GUILEC - is it really necessary? I'm no Guile
> > expert and I wish I could avoid diving to the whole ecosystem of it just
> > because I want to use Guix and contribute some packages!
> >
> > I talked about this subject in a different thread: The design choice of
> > putting all package definitions in the same repository as the package
> > manager it self is very uncommon in the Linux. I understand that almost
> > all of Guix is written in Guile so it must be linked somehow to the
> > package definitions but I think it makes it very hard to actually
> >
> 
> The channels mechanism actually makes it possible to split off any
> part of the packages. So there is no limitations in moving most of
> the packages definitions out of guix. I am not familiar enough to
> tell if all package definitions could be moved into channels, but for
> most of them this could be done.
> 
> Could you elaborate on the benefits of doing that?
> I guess this could be a great topic of discussion.

Well I guess that wouldn't have mattered that much if there was a known
way to run `make` only for the packages and not for the whole code base
which is huge.

I'm not sure. The problem for me was that I ran `make clean` under the
source tree and then I couldn't figure out why packages weren't loaded
anymore from my tree. I still don't understand really why these scheme
files aren't loaded dynamically. Do I need these `.go` files to exist?

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

On Wed, May 15, 2019 at 12:46:30PM +0200, Gábor Boskovits wrote:
> > I ran the guix-daemon command:
> >
> > sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild
> >
> 
> This should only be needed if you don't run it from a service manager.
> Changes to the daemon are rare, and most of the times you should be
> able to use the old daemon. But this should not cause a problem either.

This what the docs say. There's so much room for improvements...

> > Afterwards, in a different shell, I ran
> >
> > guix environment guix
>
> I also believe this is not strictly needed, most of the times it works for
> me without this,
> once make finished.

O.K, good to know.

> > ./pre-inst-env guix build my-new-package
> >
> > But that returned with the same error that this package doesn't exist.
> >
> >
> Could you share the diff?
> 
> I don't have any more idea without seeing what you were trying to do.

Sorry about that, I've forgot to `git checkout` the branch I was working
on. As I noted in one of my other replies, that's finally working :)

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Ricardo Wurmus]

Doron Behar <doron.behar@gmail.com> writes:

> On Wed, May 15, 2019 at 12:46:30PM +0200, Gábor Boskovits wrote:
>> > I ran the guix-daemon command:
>> >
>> > sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild
>> >
>>
>> This should only be needed if you don't run it from a service manager.
>> Changes to the daemon are rare, and most of the times you should be
>> able to use the old daemon. But this should not cause a problem either.
>
> This what the docs say.

The Contributing section only says that you *can* run all of the tools
even before installation.  It’s not wrong and in cases when you *don’t*
have a daemon running at all it is the right thing to do, but I agree
that in the more common case where you already *have* Guix it is
probably more confusing.

> There's so much room for improvements...

Do you have a constructive suggestion for improving this section of the
manual?

--
Ricardo

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

> > There's so much room for improvements...
> 
> Do you have a constructive suggestion for improving this section of the
> manual?

Not yet. I need to gain experience with the ecosystem and I hope to be
able then to think of better ways to organize the documentation. When
I'll be ready, I'd be happy to send patches or discuss the subject at
guix-devel or guix-patches.

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Amirouche]

Welcome Dordon,


On 2019-05-15 11:46, Doron Behar wrote:

> On Wed, May 15, 2019 at 10:46:29AM +0200, Gábor Boskovits wrote:
>> >
>> It might be easier to first set up a channel with your packages only. 
>> You
>> can find
>> several examples out there.
>> Here is one of my channels:
>> https://gitlab.com/g_bor/nominatim-channel

Here is mine: https://git.sr.ht/~amz3/guix-amz3-channel

> 
>> >
>> > - I ran ./configure inside /var/code/doron/guix
>> >
>> 
>> Did you specify your localstatedir to configure?
> 
> Yes, I've given it the same flags I gave it when I've build it at 
> first.
> 
>> Did you also run make?
> 
> No! I was hoping I could avoid that as that's extremely CPU intensive..
> Is there a way to run make with only the relevant targets? That could 
> be
> very helpful...

That is up to you. Look up:

compression.scm
1454:         (replace 'build
1455-           (lambda* (#:key (make-flags '()) #:allow-other-keys)
1456-             (apply invoke "make" "generic_gcc" make-flags)))

If you mean running a specific target in guix tree, I don't know about
such thing.

> Plus, do I have to run make after every change to a package 
> definition!?
> Will it compile everything from the ground up after every such change?

There is several way around that but I do not do it often (or at all):

1) You can run guix build -K -e "(@ (doron module) software-package)"

Several things here:

The -K flag will keep the build if it fails, lookup the
documentation about it. From there you can recreate the environment and
retry manually the build (almost?) as done by the builder.

The -e will install the package that is returned by the eval'ed string
that follow the flag. That thing:

   (@ (doron module) software-package)

Read as "import 'software-package' from (doron module)". Usually you
also rely on --load-path=DIR aka. -L so the complete command might look
like the following:

   guix build -L $(pwd) -K -e "(@ (doron module) software-package)"

There is also the option -f not as nice as -e once you get it.

2) The above technic is nice and good to know but it will not help
to tweak and trick a lot of stuff. Another way to go, if you want to
keep around the build and experiment with various settings. You can
setup a profile that will gather all and only the dependencies for
the build your are working on. Once you are confident with the steps
required to execute the build your can *translate* that into a guix
package. I never do that so I can not help much more.

> I wish these technical details would have been documented...

Maybe slow but sure thing :)

> I'm running make while I'm writing this email and I must say that not
> only it takes a lot of time and uses a lot of CPU power,

I think, It use all the CPU available (maybe minus one) except if you
disable parallel build in the package definition #:parallel-build? #f

> it handles way too much irrelevant things such as translations.

I don't understand why they show up in git status all the time.
Last time I published a patch with those.

> I also noticed it runs GUILEC - is it really necessary? I'm no Guile
> expert and I wish I could avoid diving to the whole ecosystem of it 
> just
> because I want to use Guix and contribute some packages!

You can avoid that package definition compilation step, but it is not
worthwhile. the way guix works is that, I think, it needs to lookup all
definitions to be able to figure the one you want to install. So, 
anyway,
guile/guix will go through all the packages that are defined.

The compilation itself is linked to the fact that GNU Guile use a 
bytecode
VM.


> I talked about this subject in a different thread: The design choice of
> putting all package definitions in the same repository as the package
> manager it self is very uncommon in the Linux.

monorepo(s) for the win!

> I understand that almost
> all of Guix is written in Guile so it must be linked somehow to the
> package definitions but I think it makes it very hard to actually
> contribute something to the packages definitions! It seems that we,
> package maintainers, or to put it simply - the community, need to be 
> too
> much involved in the source code of everything in order to help the
> distributions.

Not really, with channels you are free to do whatever you want. Most 
likely
you will helpers defined in guix/ or refer to existing package 
definitions
with guix edit sqlite3 or else... so in the end I think a monorepo is 
good
for the time being, given there is channels.

> I'd like to be convinced I'm wrong at least in this subject because as
> I dive more and more into Guix I feel there have been made a lot of 
> poor
> design choices in it..

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Amirouche]

On 2019-05-15 18:33, Amirouche wrote:
> Welcome Dordon,
> 
> 
> On 2019-05-15 11:46, Doron Behar wrote:
> 

[...]

> 
>> I'd like to be convinced I'm wrong at least in this subject because as
>> I dive more and more into Guix I feel there have been made a lot of 
>> poor
>> design choices in it..

That the usual feeling one get when one start a project. That's good, it
means you have ideas!!

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Ricardo Wurmus]

Doron Behar <doron.behar@gmail.com> writes:

> Plus, do I have to run make after every change to a package definition!?
> Will it compile everything from the ground up after every such change? I
> wish these technical details would have been documented...

“make” does not cause recompilation of unchanged things.  This is
not particular to Guix, so it’s not documented as part of Guix.

> I also noticed it runs GUILEC - is it really necessary?

It compiles Guile modules.  Using compiled modules is faster than having
to interpret everything.  You can try without compiling changed modules
but the performance penalty when you have lots of interpreted modules
will likely be hard to tolerate.

> It seems that we, package maintainers, or to put it simply - the
> community, need to be too much involved in the source code of
> everything in order to help the distributions.

Packages in Guix are Scheme values bound to variable names.  Packaging
is a simple form of programming.

> because as I dive more and more into
> Guix I feel there have been made a lot of poor design choices in it..

This is a surprising statement and naturally I disagree with it :)  I
suggest to take a step back when the frustration of learning something
new makes you want to say harsh things.

--
Ricardo

Re: Trying to contribute packages: ./pre-inst-env doesn't work

[Doron Behar]

On Wed, May 15, 2019 at 11:26:08PM +0200, Ricardo Wurmus wrote:
> It compiles Guile modules.  Using compiled modules is faster than having
> to interpret everything.  You can try without compiling changed modules
> but the performance penalty when you have lots of interpreted modules
> will likely be hard to tolerate.
> 
> 
> This is a surprising statement and naturally I disagree with it :)  I
> suggest to take a step back when the frustration of learning something
> new makes you want to say harsh things.

You are right. I'm not frustrated anymore. I have been fighting this for
many hours without taking a break and this probably made me angry. I'm
ashamed now that I wasn't aware of that I felt that way because I was
tired and hungry..

And please don't get me wrong. Overall, with all my heart, I find
Guix's design magnificent! So you wrote:

> Packages in Guix are Scheme values bound to variable names.  Packaging
> is a simple form of programming.

This is definitely the right way to implement a functional package
manager. TBH, I've had this kind of idea in the past for a package
manager with very similar feature set as Guix's and while it was cooking
in my head I've had no idea GNU implemented it already so well.
Therefor, I'm thrilled that I get the chance to contribute to it. I just
never expected it to be so hard right at the beginning. Also, I've had
no idea how complicated the implementation for this idea might become
but I'm certain it is justified.

> > Plus, do I have to run make after every change to a package definition!?
> > Will it compile everything from the ground up after every such change? I
> > wish these technical details would have been documented...
> 
> “make” does not cause recompilation of unchanged things.  This is
> not particular to Guix, so it’s not documented as part of Guix.
> 

This is a good demonstration for a lack of technical details in the
documentation I think needs to be addressed so that hopefully, common
and simple help requests in this list won't be as common.

I've quiet a lot of experience with *using* the Gnu build system (not
with *developing* with it) and the whole `./configure && make && make
install` procedure. Notably, I've gained that experience from packaging
software for Arch Linux. Therefor, I knew that `make` doesn't recompile
everything and that it checks the modification date of compiled files vs
the source files etc.

However, I didn't assume that this characteristic of `make` applies to a
project that uses Guile. This might seem obvious to Guile experts like
you but for newbies / newcomers like me, Guile is a strange creature
with a hard to learn syntax, not to mention my unfamiliarity with it's
ecosystem. Therefor, TBH, it's quiet terrifying. As a hobbyist
programmer like me who've had experience only with common languages like
Python, Go, Bash, Lua, C and C++, Guile has a very steeping learning
curve for me.

Perhaps this design choice is not bad from a technical point of view,
but it is certainly not inviting contributers. Therefor, especially if
we profess that "Packaging is a simple form of programming", packages
contributers need to be introduced officially, inside the Guix
documentation to Guile.

The following is a rough description of the way I have in mind for how
to improve the documentation pretty much from the ground up. It doesn't
consist only of adding more information but also changing the order in
which the subjects are presented.

------

Under section 14 (Contributing), the first subsection should address
package contributers because this is the most common type of
contributers. This section (14.1) should assume no programming
experience but only certain system administration experience, which will
include:

- Command line usage.
- Editing text files with a command line editor.
- Awareness of how environment variables might influence a program's
  behaviour.

After this assumption stated, an introduction to the following should be
given[1]:

- What is a build systems?
- How Guix uses the common characteristics of all Build systems to build
  packages using all used build systems in a similar manner (phases..).
- Because Guix uses it, introducing the Gnu build system in particular:
  * What does `./configure`?
  * How `make` works?
  * Why does sometimes `./configure` is not available and `./bootstrap`
    (or sometimes `autoreconf -vfi`) is needed.

That said, I think parts of section 6 (Programming Interface) relevant
to package definitions, should be merged with the new version of the new
subsection 14.1; and the rest of it should be moved to the bottom of the
whole manual so packaging guidelines should be as concise as possible.

The following part of subsection 14.1 should explain concisely how to
setup an environment suitable for packages contribution. This should
include stating clearly that it might take a while and a lot of CPU but
it is unavoidable and it will only be needed to be done once.

The instructions should be accompanied with an explanation of what each
command does and exactly why it is needed and why they will be needed or
not in the future when you'll add new packages. This accompanying
information should assume the reader has read the introduction to build
systems from above[1]. Summarizing:

- guix environment guix
- ./bootstrap 
- ./configure - using the flags given to it when Guix (from wherever you
  got it) was built - very important to make this standout.
- make - important to note that this takes a while but since make
  doesn't recompile everything every time this should be done only once.

Afterwards, an introduction to Guile in general and particularly for
programming new packages should be given, with the following subjects:

- What is functional programming.
- A Syntax Crash course with links to the official reference.
- Guile's ecosystem:
  * The compiler's usage syntax.
  * What is it's "bytecode".
  * Environment variables used by the compiler.
- How Guile is used in combination with the Gnu build system in Guix's
  repo.

------

This is an rough summarize of what I've cooked in mined as for how to
improve the documentation. I hope this brings back some of the respect I
lost after my impolite rants in my previous replies (I hope I'm forgiven
now :)).

Please let me know what do you think Ricardo. I can forward or Cc this
to guix-devel but I'd be happy to know your opinion first.

Regards, Doron.