Development shell for diffutils does not appear to work - what am I doing wrong?

Development shell for diffutils does not appear to work - what am I doing wrong?

[Daniel Littlewood]

Hi guix help,

I want to try out making a simple change to the program `diff`, which
is part of GNU diffutils:
https://packages.guix.gnu.org/packages/diffutils/3.8/
I'd like to set up a dev environment, patch diff.c, rebuild it and try
out the new binary. Maybe install it globally later, but I'm not there
yet.
I cloned the diffutils repo from
https://git.savannah.gnu.org/git/diffutils.git, and in that directory
ran
`guix shell git vim nnn -D diffutils` (but I think it's just the -D
diffutils I'm having trouble with). I believe that the
-D/--development argument should produce a shell within which I can
build `diff`.
The README says that the first step is to run `./bootstrap`, but that
fails because of several missing packages. I don't have the full list
right now, but I think autoconf was one, and texi2pdf was another.

The packaging for diffutils clearly works (since I can install it), so
I wonder if it does something different from what I'm attempting? I
couldn't find the scheme file that defines diffutils, but I'm not sure
I'd be able to read it anyway (I'm really trying out guix for the
first time).

Thanks for reading, please let me know if I can provide more info.
Dan

Re: Development shell for diffutils does not appear to work - what am I doing wrong?

[Ekaitz Zarraga]

ElenQ Technology


------- Original Message -------
On Tuesday, September 26th, 2023 at 08:08, Daniel Littlewood <danielittlewood@gmail.com> wrote:


> Hi guix help,
> 
> I want to try out making a simple change to the program `diff`, which
> is part of GNU diffutils:
> https://packages.guix.gnu.org/packages/diffutils/3.8/
> I'd like to set up a dev environment, patch diff.c, rebuild it and try
> out the new binary. Maybe install it globally later, but I'm not there
> yet.
> I cloned the diffutils repo from
> https://git.savannah.gnu.org/git/diffutils.git, and in that directory
> ran
> `guix shell git vim nnn -D diffutils` (but I think it's just the -D
> diffutils I'm having trouble with). I believe that the
> -D/--development argument should produce a shell within which I can
> build `diff`.
> The README says that the first step is to run `./bootstrap`, but that
> fails because of several missing packages. I don't have the full list
> right now, but I think autoconf was one, and texi2pdf was another.
> 
> The packaging for diffutils clearly works (since I can install it), so
> I wonder if it does something different from what I'm attempting? I
> couldn't find the scheme file that defines diffutils, but I'm not sure
> I'd be able to read it anyway (I'm really trying out guix for the
> first time).
> 
> Thanks for reading, please let me know if I can provide more info.
> Dan



Dan,

I don't think you are doing anything wrong. I don't know why but
when doing `-D package` guix is often not adding all the development
dependencies as it doesn't load autotools and related things to
the shell.

You have to add them by hand.

Also, diffutils downloads a tar.xz which probably has the bootstrap
step already done because it is considered a release source code.

You are working from development code I expect, which probably needs
some extra tools.

What I would do: go adding them to the shell one by one as the build
system complains until it doesn't complain anymore. They will
probably be `texinfo`, `automake`, `autoconf`, `libtool` and maybe
I'm missing something... If you add them as you go you shouldn't 
leave anything out.

Also, this is an interesting call. It might be cool to have a way to
add those directly... I don't know what is best but probably with
some kind of flag we should add all the deps from the build-system
too.
And also, have all the `autotools` in just one package because I
always forget some of them.

I may start another thread with that...

Thanks for your question, it is a very valid one! It happened to me
before, too, and it's pretty annoying.

Hope this helps,
Ekaitz

Re: Development shell for diffutils does not appear to work - what am I doing wrong?

[Daniel Littlewood]

Hi Ekaitz, thanks for you reply!

Reading the docs a second time, I think I did misunderstand the
--development argument. The docs say
> Cause guix shell to include in the environment the dependencies of the following package rather than the package itself.
and I took "dependencies" to mean "build dependencies". But perhaps it
just means the runtime ones?
So as you say, you could download the release tarball of diffutils and
be able to run it, but not necessarily to build it from the source.
I wonder why the flag is called "development" when it doesn't allow
you to develop the package. Perhaps it refers to developing guix?

I did exactly as you suggest yesterday and did manage to find all the
dependencies (although I didn't manage to build it in the end, but I
don't know the root cause yet). In fact the very next documented
config option looks like it may be able to build autotools. They give
an example for including the base packages:
guix shell -e '(@ (gnu) %base-packages)'
Perhaps there is a similar expression for importing everything in a
specific build system?

Since I was mixed up between building and installing, I also wonder
whether this is what `guix build` is for. Or, equally, if I copy the
definition of diffutils in base.scm and then do `guix shell -f
my-diffutils.scm` maybe it will build itself? It certainly seems to
have all the necessary information to do so.

I'll test those options later, thanks!
Dan

On Wed, Sep 27, 2023 at 11:08 AM Ekaitz Zarraga <ekaitz@elenq.tech> wrote:
>
>
>
>
>
> ElenQ Technology
>
>
> ------- Original Message -------
> On Tuesday, September 26th, 2023 at 08:08, Daniel Littlewood <danielittlewood@gmail.com> wrote:
>
>
> > Hi guix help,
> >
> > I want to try out making a simple change to the program `diff`, which
> > is part of GNU diffutils:
> > https://packages.guix.gnu.org/packages/diffutils/3.8/
> > I'd like to set up a dev environment, patch diff.c, rebuild it and try
> > out the new binary. Maybe install it globally later, but I'm not there
> > yet.
> > I cloned the diffutils repo from
> > https://git.savannah.gnu.org/git/diffutils.git, and in that directory
> > ran
> > `guix shell git vim nnn -D diffutils` (but I think it's just the -D
> > diffutils I'm having trouble with). I believe that the
> > -D/--development argument should produce a shell within which I can
> > build `diff`.
> > The README says that the first step is to run `./bootstrap`, but that
> > fails because of several missing packages. I don't have the full list
> > right now, but I think autoconf was one, and texi2pdf was another.
> >
> > The packaging for diffutils clearly works (since I can install it), so
> > I wonder if it does something different from what I'm attempting? I
> > couldn't find the scheme file that defines diffutils, but I'm not sure
> > I'd be able to read it anyway (I'm really trying out guix for the
> > first time).
> >
> > Thanks for reading, please let me know if I can provide more info.
> > Dan
>
>
>
> Dan,
>
> I don't think you are doing anything wrong. I don't know why but
> when doing `-D package` guix is often not adding all the development
> dependencies as it doesn't load autotools and related things to
> the shell.
>
> You have to add them by hand.
>
> Also, diffutils downloads a tar.xz which probably has the bootstrap
> step already done because it is considered a release source code.
>
> You are working from development code I expect, which probably needs
> some extra tools.
>
> What I would do: go adding them to the shell one by one as the build
> system complains until it doesn't complain anymore. They will
> probably be `texinfo`, `automake`, `autoconf`, `libtool` and maybe
> I'm missing something... If you add them as you go you shouldn't
> leave anything out.
>
> Also, this is an interesting call. It might be cool to have a way to
> add those directly... I don't know what is best but probably with
> some kind of flag we should add all the deps from the build-system
> too.
> And also, have all the `autotools` in just one package because I
> always forget some of them.
>
> I may start another thread with that...
>
> Thanks for your question, it is a very valid one! It happened to me
> before, too, and it's pretty annoying.
>
> Hope this helps,
> Ekaitz

Re: Development shell for diffutils does not appear to work - what am I doing wrong?

[Tobias Geerinckx-Rice]

Hi (again) Dan,

On 2023-09-27 12:58, Daniel Littlewood wrote:
> --development argument. The docs say
>> Cause guix shell to include in the environment the dependencies of the 
>> following package rather than the package itself.
> and I took "dependencies" to mean "build dependencies".

Correct.

> But perhaps it just means the runtime ones?

I'm not aware of an option for this even if you wanted to.

> So as you say, you could download the release tarball of diffutils and
> be able to run it, but not necessarily to build it from the source.

The release tarball *is* the source — of the ‘diffutils’ package.  The 
only additional native (‘build’) input needed to build it is ‘perl’, 
which is also provided by ‘guix shell -D diffutils’.

> I wonder why the flag is called "development" when it doesn't allow
> you to develop the package.

It's true that it doesn't provide a full-featured ‘development 
environment’, including someone's favourite editor, etc.  However, I 
don't think it's poorly named.  Language is just hard.  Calling it 
‘--build’ would confuse people who think Guix has a notion of ‘build 
dependencies’, of which there are already too many.

The point of ‘guix shell -D’ is to provide an environment where you can 
build a package, as packaged in Guix, from source, as packaged in Guix.  
Hence:

   - *All* build system (‘implicit’) inputs such as a compiler/toolchain, 
GNU coreutils, etc.
   - *All* native package inputs, if any (in diffutils' case: perl).

This is all you need to build diffutils as packaged: 
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/base.scm#n292

If you BYO source, different from that used by Guix, you'll have to BYO 
extra development inputs it requires.

Alternatively, you could rewrite Guix's diffutils package to build from 
Git.  You'd then add extra packages like autoconf, automake, etc., to 
native-inputs, which would then be picked up by ‘guix package -D 
diffutils’.

> Perhaps it refers to developing guix?

When used in ‘guix shell -D guix’, maybe… :-)

> Perhaps there is a similar expression for importing everything in a
> specific build system?

That's what ‘-D’ does.  It's the assumption that autotools are part of 
any build-system that's muddying the waters here.  AFAIK they are not.

Grep Guix: if a package bootstraps ./configure and friends from scratch, 
it is expected to declare autoconf & automake (and probably more) as 
inputs.  Diffutils doesn't.  If you do, that's fine, but then you're 
operating outside of the Guix diffutils package and need to add your own 
extra inputs on the command line.

> Since I was mixed up between building and installing, I also wonder
> whether this is what `guix build` is for. Or, equally, if I copy the
> definition of diffutils in base.scm and then do `guix shell -f
> my-diffutils.scm` maybe it will build itself? It certainly seems to
> have all the necessary information to do so.

That's identical to ‘guix build diffutils’, no?

Kind regards,

T G-R

Sent from a Web browser.  Excuse or enjoy my brevity.

Re: Development shell for diffutils does not appear to work - what am I doing wrong?

[Tobias Geerinckx-Rice]

On 2023-09-27 14:43, Tobias Geerinckx-Rice wrote:
>   - *All* native package inputs, if any (in diffutils' case: perl).

s/native //, but I wanted to emphasise native, since they seldom show up 
as references in the output and are hence often build-time-only.

Kind regards,

T G-R

Sent from a Web browser.  Excuse or enjoy my brevity.

Re: Development shell for diffutils does not appear to work - what am I doing wrong?

[wolf]

[-- Attachment #1: Type: text/plain, Size: 6233 bytes --]

On 2023-09-27 11:58:32 +0100, Daniel Littlewood wrote:
> Hi Ekaitz, thanks for you reply!
> 
> Reading the docs a second time, I think I did misunderstand the
> --development argument. The docs say
> > Cause guix shell to include in the environment the dependencies of the following package rather than the package itself.
> and I took "dependencies" to mean "build dependencies". But perhaps it
> just means the runtime ones?
> So as you say, you could download the release tarball of diffutils and
> be able to run it, but not necessarily to build it from the source.
> I wonder why the flag is called "development" when it doesn't allow
> you to develop the package.

I think your confusion here stems from not understanding how (not only, not all)
GNU project in general operate as far as release management goes.  Most GNU
project do provide something called release tarball, which is an archive
intended for consumption by distributions/end-users.  This archive is (for
autotools projects) created by running `make dist'.  This is the archive
distributions (including Guix) use for building the package.

The advantage of the archive is that it contains some already generated files.
For example, when building the archive, you usually start by running
`./configure', since that file is provided as part of the archive.  When
building from VCS, you usually start with some variant of `./bootstrap'.

The implication (and source of your problem) is that less software is required
for the build from the archive compared to the VCS.  And that is a good thing!
For example autotools can be sensitive quite a bit to the version (you need to
have the correct one).  However the generated configure script is insanely
portable, so you can (for example) build the project even on distributions that
do not package autotools 2.69 anymore.

So if you download the archive, and use the -D, you will be able to build it.
However, for VCS checkout you need to provide the additional packages yourself
(as you already figured out).

Hope this sheds some light on the cause.

> Perhaps it refers to developing guix?
> 
> I did exactly as you suggest yesterday and did manage to find all the
> dependencies (although I didn't manage to build it in the end, but I
> don't know the root cause yet). In fact the very next documented
> config option looks like it may be able to build autotools. They give
> an example for including the base packages:
> guix shell -e '(@ (gnu) %base-packages)'
> Perhaps there is a similar expression for importing everything in a
> specific build system?
> 
> Since I was mixed up between building and installing, I also wonder
> whether this is what `guix build` is for. Or, equally, if I copy the
> definition of diffutils in base.scm and then do `guix shell -f
> my-diffutils.scm` maybe it will build itself? It certainly seems to
> have all the necessary information to do so.
> 
> I'll test those options later, thanks!
> Dan
> 
> On Wed, Sep 27, 2023 at 11:08 AM Ekaitz Zarraga <ekaitz@elenq.tech> wrote:
> >
> >
> >
> >
> >
> > ElenQ Technology
> >
> >
> > ------- Original Message -------
> > On Tuesday, September 26th, 2023 at 08:08, Daniel Littlewood <danielittlewood@gmail.com> wrote:
> >
> >
> > > Hi guix help,
> > >
> > > I want to try out making a simple change to the program `diff`, which
> > > is part of GNU diffutils:
> > > https://packages.guix.gnu.org/packages/diffutils/3.8/
> > > I'd like to set up a dev environment, patch diff.c, rebuild it and try
> > > out the new binary. Maybe install it globally later, but I'm not there
> > > yet.
> > > I cloned the diffutils repo from
> > > https://git.savannah.gnu.org/git/diffutils.git, and in that directory
> > > ran
> > > `guix shell git vim nnn -D diffutils` (but I think it's just the -D
> > > diffutils I'm having trouble with). I believe that the
> > > -D/--development argument should produce a shell within which I can
> > > build `diff`.
> > > The README says that the first step is to run `./bootstrap`, but that
> > > fails because of several missing packages. I don't have the full list
> > > right now, but I think autoconf was one, and texi2pdf was another.
> > >
> > > The packaging for diffutils clearly works (since I can install it), so
> > > I wonder if it does something different from what I'm attempting? I
> > > couldn't find the scheme file that defines diffutils, but I'm not sure
> > > I'd be able to read it anyway (I'm really trying out guix for the
> > > first time).
> > >
> > > Thanks for reading, please let me know if I can provide more info.
> > > Dan
> >
> >
> >
> > Dan,
> >
> > I don't think you are doing anything wrong. I don't know why but
> > when doing `-D package` guix is often not adding all the development
> > dependencies as it doesn't load autotools and related things to
> > the shell.
> >
> > You have to add them by hand.
> >
> > Also, diffutils downloads a tar.xz which probably has the bootstrap
> > step already done because it is considered a release source code.
> >
> > You are working from development code I expect, which probably needs
> > some extra tools.
> >
> > What I would do: go adding them to the shell one by one as the build
> > system complains until it doesn't complain anymore. They will
> > probably be `texinfo`, `automake`, `autoconf`, `libtool` and maybe
> > I'm missing something... If you add them as you go you shouldn't
> > leave anything out.
> >
> > Also, this is an interesting call. It might be cool to have a way to
> > add those directly... I don't know what is best but probably with
> > some kind of flag we should add all the deps from the build-system
> > too.
> > And also, have all the `autotools` in just one package because I
> > always forget some of them.
> >
> > I may start another thread with that...
> >
> > Thanks for your question, it is a very valid one! It happened to me
> > before, too, and it's pretty annoying.
> >
> > Hope this helps,
> > Ekaitz
>

Have a nice day,
W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Development shell for diffutils does not appear to work - what am I doing wrong?

[Daniel Littlewood]

Hi wolf,

I have now tried that, and I was able to build from the tarball and
test my local changes. Indeed, I was confused previously - I always
assumed that the release tarball was a binary; I had no idea there was
a distinction between "building from VCS" and "building from the
archive". Thank you for explaining!

I think that's fixed now, thanks everyone.
Dan

On Wed, Sep 27, 2023 at 5:25 PM wolf <wolf@wolfsden.cz> wrote:
>
> On 2023-09-27 11:58:32 +0100, Daniel Littlewood wrote:
> > Hi Ekaitz, thanks for you reply!
> >
> > Reading the docs a second time, I think I did misunderstand the
> > --development argument. The docs say
> > > Cause guix shell to include in the environment the dependencies of the following package rather than the package itself.
> > and I took "dependencies" to mean "build dependencies". But perhaps it
> > just means the runtime ones?
> > So as you say, you could download the release tarball of diffutils and
> > be able to run it, but not necessarily to build it from the source.
> > I wonder why the flag is called "development" when it doesn't allow
> > you to develop the package.
>
> I think your confusion here stems from not understanding how (not only, not all)
> GNU project in general operate as far as release management goes.  Most GNU
> project do provide something called release tarball, which is an archive
> intended for consumption by distributions/end-users.  This archive is (for
> autotools projects) created by running `make dist'.  This is the archive
> distributions (including Guix) use for building the package.
>
> The advantage of the archive is that it contains some already generated files.
> For example, when building the archive, you usually start by running
> `./configure', since that file is provided as part of the archive.  When
> building from VCS, you usually start with some variant of `./bootstrap'.
>
> The implication (and source of your problem) is that less software is required
> for the build from the archive compared to the VCS.  And that is a good thing!
> For example autotools can be sensitive quite a bit to the version (you need to
> have the correct one).  However the generated configure script is insanely
> portable, so you can (for example) build the project even on distributions that
> do not package autotools 2.69 anymore.
>
> So if you download the archive, and use the -D, you will be able to build it.
> However, for VCS checkout you need to provide the additional packages yourself
> (as you already figured out).
>
> Hope this sheds some light on the cause.
>
> > Perhaps it refers to developing guix?
> >
> > I did exactly as you suggest yesterday and did manage to find all the
> > dependencies (although I didn't manage to build it in the end, but I
> > don't know the root cause yet). In fact the very next documented
> > config option looks like it may be able to build autotools. They give
> > an example for including the base packages:
> > guix shell -e '(@ (gnu) %base-packages)'
> > Perhaps there is a similar expression for importing everything in a
> > specific build system?
> >
> > Since I was mixed up between building and installing, I also wonder
> > whether this is what `guix build` is for. Or, equally, if I copy the
> > definition of diffutils in base.scm and then do `guix shell -f
> > my-diffutils.scm` maybe it will build itself? It certainly seems to
> > have all the necessary information to do so.
> >
> > I'll test those options later, thanks!
> > Dan
> >
> > On Wed, Sep 27, 2023 at 11:08 AM Ekaitz Zarraga <ekaitz@elenq.tech> wrote:
> > >
> > >
> > >
> > >
> > >
> > > ElenQ Technology
> > >
> > >
> > > ------- Original Message -------
> > > On Tuesday, September 26th, 2023 at 08:08, Daniel Littlewood <danielittlewood@gmail.com> wrote:
> > >
> > >
> > > > Hi guix help,
> > > >
> > > > I want to try out making a simple change to the program `diff`, which
> > > > is part of GNU diffutils:
> > > > https://packages.guix.gnu.org/packages/diffutils/3.8/
> > > > I'd like to set up a dev environment, patch diff.c, rebuild it and try
> > > > out the new binary. Maybe install it globally later, but I'm not there
> > > > yet.
> > > > I cloned the diffutils repo from
> > > > https://git.savannah.gnu.org/git/diffutils.git, and in that directory
> > > > ran
> > > > `guix shell git vim nnn -D diffutils` (but I think it's just the -D
> > > > diffutils I'm having trouble with). I believe that the
> > > > -D/--development argument should produce a shell within which I can
> > > > build `diff`.
> > > > The README says that the first step is to run `./bootstrap`, but that
> > > > fails because of several missing packages. I don't have the full list
> > > > right now, but I think autoconf was one, and texi2pdf was another.
> > > >
> > > > The packaging for diffutils clearly works (since I can install it), so
> > > > I wonder if it does something different from what I'm attempting? I
> > > > couldn't find the scheme file that defines diffutils, but I'm not sure
> > > > I'd be able to read it anyway (I'm really trying out guix for the
> > > > first time).
> > > >
> > > > Thanks for reading, please let me know if I can provide more info.
> > > > Dan
> > >
> > >
> > >
> > > Dan,
> > >
> > > I don't think you are doing anything wrong. I don't know why but
> > > when doing `-D package` guix is often not adding all the development
> > > dependencies as it doesn't load autotools and related things to
> > > the shell.
> > >
> > > You have to add them by hand.
> > >
> > > Also, diffutils downloads a tar.xz which probably has the bootstrap
> > > step already done because it is considered a release source code.
> > >
> > > You are working from development code I expect, which probably needs
> > > some extra tools.
> > >
> > > What I would do: go adding them to the shell one by one as the build
> > > system complains until it doesn't complain anymore. They will
> > > probably be `texinfo`, `automake`, `autoconf`, `libtool` and maybe
> > > I'm missing something... If you add them as you go you shouldn't
> > > leave anything out.
> > >
> > > Also, this is an interesting call. It might be cool to have a way to
> > > add those directly... I don't know what is best but probably with
> > > some kind of flag we should add all the deps from the build-system
> > > too.
> > > And also, have all the `autotools` in just one package because I
> > > always forget some of them.
> > >
> > > I may start another thread with that...
> > >
> > > Thanks for your question, it is a very valid one! It happened to me
> > > before, too, and it's pretty annoying.
> > >
> > > Hope this helps,
> > > Ekaitz
> >
>
> Have a nice day,
> W.
>
> --
> There are only two hard things in Computer Science:
> cache invalidation, naming things and off-by-one errors.

Re: Development shell for diffutils does not appear to work - what am I doing wrong?

[Carlo Zancanaro]

Hi Daniel,

On Tue, Sep 26 2023, Daniel Littlewood wrote:
> I want to try out making a simple change to the program `diff`, which is part of GNU diffutils: https://packages.guix.gnu.org/packages/diffutils/3.8/ I'd like to set up a dev environment, patch diff.c, rebuild it and try out the new binary. Maybe install it globally later, but I'm not there yet.

There's a few emails in this thread about how to get your git clone working, but I thought it might also be helpful to mention a potentially easier workflow here.

The --development argument gives you an environment that is suitable for hacking on the source that "guix build --source" gives you. With that in mind, I just checked that this works:

  $ tar xf $(guix build --source diffutils)
  $ cd diffutils-3.8/
  $ guix shell --development diffutils -- ./configure
  ... lots of output ...
  $ guix shell --development diffutils -- make
  ... lots of output ...

Then you can change the source in the diffutils-3.8 directory and recompile it with "guix shell --development diffutils -- make".

If you want to install it to a profile later, you can use something like "guix install diffutils --with-source=path/to/diffutils-3.8/".

If you then want to upstream your changes, you can manually move your fixes into a git checkout.

Carlo