Problem with `direnv` package definition

Problem with `direnv` package definition

[Tanguy Le Carrour]

Dear Guix

As I'm still in the process of setting up my environment,
I am not (yet) able to write and submit a patch for package `direnv`,
so I'm sending this report instead…

As mentioned on the `direnv` homepage, "direnv is compiled into
a single static executable" [1]. As I understand it, this means that
Go is required to build it, but not to run it.

[1]: https://github.com/direnv/direnv

However, in the package definition [2], 3 Go packages are listed as
"inputs" whereas they should be listed as "native-inputs". Is this
correct?

[2]: gnu/packages/shellutils.scm

As I said, I'm still learning. But I've tried, and here is what I've
done so far…

I've `git clone`d the repo, ran the `guix env… guix` then `bootstrap`
`configure` and `make` and everything seems to be fine.
Just to make sure, I did `./pre-… guix env… direnv` then
ran `./pre-… guix build direnv` and everything went well.
Then I moved the direnv dependencies from inputs to native-inputs and
build it again… and it unsurprisingly failed! The error message [3]
does not say much… at least to me! ^_^'

[3]: output of `./pre-inst-env guix build --keep-failed direnv`
  direnv-2.15.2/version.txt
  direnv-2.15.2/xdg.go
  phase `unpack' succeeded after 0.0 seconds
  starting phase `bootstrap'
  no 'configure.ac' or anything like that, doing nothing
  phase `bootstrap' succeeded after 0.0 seconds
  starting phase `patch-usr-bin-file'
  phase `patch-usr-bin-file' succeeded after 0.0 seconds
  starting phase `patch-source-shebangs'
  phase `patch-source-shebangs' succeeded after 0.0 seconds
  starting phase `patch-generated-file-shebangs'
  phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
  starting phase `build'
  make: *** No targets specified and no makefile found.  Stop.
  Backtrace:
  4 (primitive-load
      "/gnu/store/7rrxdai48si293dihzf55zn3svn…")
  In ice-9/eval.scm:
     191:35  3 (_ #f)
  In srfi/srfi-1.scm:
     863:16  2 (every1 #<procedure 71bfc0 at /gnu/store/i5ip2vy29fqpp…>
  …)
  In
   /gnu/store/i5ip2vy29fqppjb4pq5isq36gqd42d89-module-import/guix/build/gn
   u-build-system.scm:
   799:28  1 (_ _)
  In
   /gnu/store/i5ip2vy29fqppjb4pq5isq36gqd42d89-module-import/guix/build/utils.scm:
   616:6  0 (invoke _ . _)
   /gnu/store/i5ip2vy29fqppjb4pq5isq36gqd42d89-module-import/guix/build/utils.scm:616:6: In procedure invoke:
  Throw to key `srfi-34' with args `(#<condition &invoke-error[program: "make" arguments: ("-j" "8"
  "DESTDIR=/gnu/store/dcvs8cq0ll4hcxc4x3mlcf24y1cw4ckm-direnv-2.15.2")
  exit-status: 2 term-signal: #f stop-signal: #f] 869e80>)'.


I've tried to read the rest of the package definition, but as I'm still
attending Guile 1-0-1, that didn't help much!

I would very much appreciate it if someone could point out my mistake!

Regards,

-- 
Tanguy

Re: Problem with `direnv` package definition

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3580 bytes --]


Tanguy Le Carrour <tanguy@bioneland.org> writes:

> Dear Guix
>
> As I'm still in the process of setting up my environment,
> I am not (yet) able to write and submit a patch for package `direnv`,
> so I'm sending this report instead…
>
> As mentioned on the `direnv` homepage, "direnv is compiled into
> a single static executable" [1]. As I understand it, this means that
> Go is required to build it, but not to run it.
>
> [1]: https://github.com/direnv/direnv
>
> However, in the package definition [2], 3 Go packages are listed as
> "inputs" whereas they should be listed as "native-inputs". Is this
> correct?
>
> [2]: gnu/packages/shellutils.scm

Hi Tanguy,

That sounds right to me, although there have been issues with binaries
generated with Go still referencing the Go compiler, and that's
seemingly the case with the direnv package in Guix at the moment.

There's also an important aspect of cross-compilation to these fields,
which you can read about here:

  https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html

> As I said, I'm still learning. But I've tried, and here is what I've
> done so far…
>
> I've `git clone`d the repo, ran the `guix env… guix` then `bootstrap`
> `configure` and `make` and everything seems to be fine.
> Just to make sure, I did `./pre-… guix env… direnv` then
> ran `./pre-… guix build direnv` and everything went well.
> Then I moved the direnv dependencies from inputs to native-inputs and
> build it again… and it unsurprisingly failed! The error message [3]
> does not say much… at least to me! ^_^'
>
> [3]: output of `./pre-inst-env guix build --keep-failed direnv`
>   direnv-2.15.2/version.txt
>   direnv-2.15.2/xdg.go
>   phase `unpack' succeeded after 0.0 seconds
>   starting phase `bootstrap'
>   no 'configure.ac' or anything like that, doing nothing
>   phase `bootstrap' succeeded after 0.0 seconds
>   starting phase `patch-usr-bin-file'
>   phase `patch-usr-bin-file' succeeded after 0.0 seconds
>   starting phase `patch-source-shebangs'
>   phase `patch-source-shebangs' succeeded after 0.0 seconds
>   starting phase `patch-generated-file-shebangs'
>   phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
>   starting phase `build'
>   make: *** No targets specified and no makefile found.  Stop.
>   Backtrace:
>   4 (primitive-load
>       "/gnu/store/7rrxdai48si293dihzf55zn3svn…")
>   In ice-9/eval.scm:
>      191:35  3 (_ #f)
>   In srfi/srfi-1.scm:
>      863:16  2 (every1 #<procedure 71bfc0 at /gnu/store/i5ip2vy29fqpp…>
>   …)
>   In
>    /gnu/store/i5ip2vy29fqppjb4pq5isq36gqd42d89-module-import/guix/build/gn
>    u-build-system.scm:
>    799:28  1 (_ _)
>   In
>    /gnu/store/i5ip2vy29fqppjb4pq5isq36gqd42d89-module-import/guix/build/utils.scm:
>    616:6  0 (invoke _ . _)
>    /gnu/store/i5ip2vy29fqppjb4pq5isq36gqd42d89-module-import/guix/build/utils.scm:616:6: In procedure invoke:
>   Throw to key `srfi-34' with args `(#<condition &invoke-error[program: "make" arguments: ("-j" "8"
>   "DESTDIR=/gnu/store/dcvs8cq0ll4hcxc4x3mlcf24y1cw4ckm-direnv-2.15.2")
>   exit-status: 2 term-signal: #f stop-signal: #f] 869e80>)'.
>
>
> I've tried to read the rest of the package definition, but as I'm still
> attending Guile 1-0-1, that didn't help much!
>
> I would very much appreciate it if someone could point out my mistake!

I tried changing the inputs to native-inputs, and the package built for
me. Could you share the exact changes you made?

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Problem with `direnv` package definition

[Tanguy Le Carrour]

Le 04/20, Christopher Baines a écrit :
> Tanguy Le Carrour <tanguy@bioneland.org> writes:
> > However, in the package definition [2], 3 Go packages are listed as
> > "inputs" whereas they should be listed as "native-inputs". Is this
> > correct?
> That sounds right to me, although there have been issues with binaries
> generated with Go […]
> There's also an important aspect of cross-compilation to these fields,
> which you can read about here:
> 
>   https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html

Thanks for the link.

I do realize that, so close to the release of Guix System 1.0,
this can sound trivial, but I find it a little annoying to have
`guix size direnv` reporting 594 MiB when the actual binary size is
2.9M. :-(


> > As I said, I'm still learning. But I've tried, and here is what I've
> > done so far […]
> 
> I tried changing the inputs to native-inputs, and the package built for
> me. Could you share the exact changes you made?

I haven't done much:

--- a/gnu/packages/shellutils.scm
+++ b/gnu/packages/shellutils.scm
@@ -129,12 +129,11 @@ are already there.")
          ;; Help the build scripts find the Go language dependencies.
          (add-before 'unpack 'setup-go-environment
            (assoc-ref go:%standard-phases 'setup-go-environment)))))
-    (inputs
-     `(("go" ,go)
-       ("go-github-com-burntsushi-toml" ,go-github-com-burntsushi-toml)
-       ("go-github-com-direnv-go-dotenv" ,go-github-com-direnv-go-dotenv)))
     (native-inputs
-      `(("which" ,which)))
+      `(("go" ,go)
+        ("go-github-com-burntsushi-toml" ,go-github-com-burntsushi-toml)
+        ("go-github-com-direnv-go-dotenv" ,go-github-com-direnv-go-dotenv)
+        ("which" ,which)))
     (home-page "https://direnv.net/")
     (synopsis "Environment switcher for the shell")
     (description


Thanks again for your time!


-- 
Tanguy

Re: Problem with `direnv` package definition

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3885 bytes --]


Tanguy Le Carrour <tanguy@bioneland.org> writes:

> Le 04/20, Christopher Baines a écrit :
>> Tanguy Le Carrour <tanguy@bioneland.org> writes:
>> > However, in the package definition [2], 3 Go packages are listed as
>> > "inputs" whereas they should be listed as "native-inputs". Is this
>> > correct?
>> That sounds right to me, although there have been issues with binaries
>> generated with Go […]
>> There's also an important aspect of cross-compilation to these fields,
>> which you can read about here:
>>
>>   https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html
>
> Thanks for the link.
>
> I do realize that, so close to the release of Guix System 1.0,
> this can sound trivial, but I find it a little annoying to have
> `guix size direnv` reporting 594 MiB when the actual binary size is
> 2.9M. :-(

So, in contrast to some other package management systems, the runtime
dependencies, or references of the output(s) in the case of Guix are not
explicitly set in the package definition.

This is still something I have a little difficulty understanding, but
the inputs/native-inputs distinction is more about architecture than
runtime references.

That's not to say that making the inputs, native-inputs in this case is
wrong, quite the opposite. But, to reduce the size of the package, all
that's required is to remove the reference from the direnv binary to the
go compiler.

It turns out that there's a specific phase of the go-build-system in
Guix that does this. Currently the direnv package is using the
gnu-build-system, as it's one of the older go packages in Guix, however
it does now pull in some phases from the go-build-system.

I've just pushed a commit that pulls in the relevant phase, and makes
the inputs, native-inputs [1]. With that change, the size of the package
does drop [2].

1: http://git.savannah.gnu.org/cgit/guix.git/commit/?id=9729b6ea2fe89d5b8cfd180ef2dfca367c7417e5
2:
→ guix size /gnu/store/64m1v6smprhzmsbdvrwvd5gsrc5lirr5-direnv-2.15.2
store item                                                       total    self
/gnu/store/64m1v6smprhzmsbdvrwvd5gsrc5lirr5-direnv-2.15.2            4.9     2.9  59.0%
/gnu/store/izyk3kppj42pa8i2cq29bw3bnr1607ps-tzdata-2018i             2.0     2.0  41.0%
total: 4.9 MiB

>> > As I said, I'm still learning. But I've tried, and here is what I've
>> > done so far […]
>>
>> I tried changing the inputs to native-inputs, and the package built for
>> me. Could you share the exact changes you made?
>
> I haven't done much:
>
> --- a/gnu/packages/shellutils.scm
> +++ b/gnu/packages/shellutils.scm
> @@ -129,12 +129,11 @@ are already there.")
>           ;; Help the build scripts find the Go language dependencies.
>           (add-before 'unpack 'setup-go-environment
>             (assoc-ref go:%standard-phases 'setup-go-environment)))))
> -    (inputs
> -     `(("go" ,go)
> -       ("go-github-com-burntsushi-toml" ,go-github-com-burntsushi-toml)
> -       ("go-github-com-direnv-go-dotenv" ,go-github-com-direnv-go-dotenv)))
>      (native-inputs
> -      `(("which" ,which)))
> +      `(("go" ,go)
> +        ("go-github-com-burntsushi-toml" ,go-github-com-burntsushi-toml)
> +        ("go-github-com-direnv-go-dotenv" ,go-github-com-direnv-go-dotenv)
> +        ("which" ,which)))
>      (home-page "https://direnv.net/")
>      (synopsis "Environment switcher for the shell")
>      (description
>
>
> Thanks again for your time!

What you've put above looks ok, I'm unsure why that wouldn't work...

Anyway, while the size of direnv should now be improved, there's still
more improvements to be made to the direnv package if you're interested!

I think it would be better to switch to using the go build system, and
the package in Guix is quite a few versions behind.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Problem with `direnv` package definition

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2538 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Tanguy Le Carrour <tanguy@bioneland.org> writes:
>
>> Le 04/20, Christopher Baines a écrit :
>>> Tanguy Le Carrour <tanguy@bioneland.org> writes:
>>> > As I said, I'm still learning. But I've tried, and here is what I've
>>> > done so far […]
>>>
>>> I tried changing the inputs to native-inputs, and the package built for
>>> me. Could you share the exact changes you made?
>>
>> I haven't done much:
>>
>> --- a/gnu/packages/shellutils.scm
>> +++ b/gnu/packages/shellutils.scm
>> @@ -129,12 +129,11 @@ are already there.")
>>           ;; Help the build scripts find the Go language dependencies.
>>           (add-before 'unpack 'setup-go-environment
>>             (assoc-ref go:%standard-phases 'setup-go-environment)))))
>> -    (inputs
>> -     `(("go" ,go)
>> -       ("go-github-com-burntsushi-toml" ,go-github-com-burntsushi-toml)
>> -       ("go-github-com-direnv-go-dotenv" ,go-github-com-direnv-go-dotenv)))
>>      (native-inputs
>> -      `(("which" ,which)))
>> +      `(("go" ,go)
>> +        ("go-github-com-burntsushi-toml" ,go-github-com-burntsushi-toml)
>> +        ("go-github-com-direnv-go-dotenv" ,go-github-com-direnv-go-dotenv)
>> +        ("which" ,which)))
>>      (home-page "https://direnv.net/")
>>      (synopsis "Environment switcher for the shell")
>>      (description
>>
>>
>> Thanks again for your time!
>
> What you've put above looks ok, I'm unsure why that wouldn't work...

Just to follow up, as I've learned some more.

So... last night I had this same failure on another machine. Turns out
there is a bigger issue with the direnv package. The particular
combination of the 'setup-go-environment phase from the go-build-system,
and the 'unpack phase from the gnu-build-system, leads to
non-deterministic behaviour, where dependent on the order in which the
filesystem returns two directories, the package either builds, or fails
to build.

> I think it would be better to switch to using the go build system, and
> the package in Guix is quite a few versions behind.

As far as I understand, switching entirely to the go-build-system will
avoid this issue we've both now encountered, so I've sent a patch up
about this [1]. I've also created an issue to track the problem in the
gnu-build-system, as I think it could check for this circumstance and
error, to avoid this issue happening in the future.

1: https://issues.guix.info/issue/35386
2: https://issues.guix.info/issue/35387

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Problem with `direnv` package definition

[Tanguy Le Carrour]

Le 04/21, Christopher Baines a écrit :
> Tanguy Le Carrour <tanguy@bioneland.org> writes:
> > Le 04/20, Christopher Baines a écrit :
> >> Tanguy Le Carrour <tanguy@bioneland.org> writes:
> >> > However, in the package definition [2], 3 Go packages are listed as
> >> > "inputs" whereas they should be listed as "native-inputs". Is this
> >> > correct?
> >> That sounds right to me, although there have been issues with binaries
> >> generated with Go […]
> […]
> So, in contrast to some other package management systems, the runtime
> dependencies, or references of the output(s) in the case of Guix are not
> explicitly set in the package definition.
> 
> This is still something I have a little difficulty understanding, but
> the inputs/native-inputs distinction is more about architecture than
> runtime references.

Ouch?! If you have difficulty understanding it, imagine for me! ^_^'


> I've just pushed a commit that pulls in the relevant phase, and makes
> the inputs, native-inputs [1]. With that change, the size of the package
> does drop [2].

Thank you so much for that!


> That's not to say that making the inputs, native-inputs in this case is
> wrong, quite the opposite. […] Currently the direnv package is using the
> gnu-build-system, as it's one of the older go packages in Guix, however
> it does now pull in some phases from the go-build-system.
> […]
> Anyway, while the size of direnv should now be improved, there's still
> more improvements to be made to the direnv package if you're interested!
> I think it would be better to switch to using the go build system, and
> the package in Guix is quite a few versions behind.

So this might be something I could be investigating in a near future!
But I guess I'll have to read some Go package definitions first…
Which won't be a waste of time, for `fzf` is also written in Go!


Regards


-- 
Tanguy