Golang check phase skipping some tests?

Golang check phase skipping some tests?

[Troy Figiel]

[-- Attachment #1.1.1: Type: text/plain, Size: 1196 bytes --]

Hi everyone,

When looking into the Go build system, I noticed the default check phase
runs (invoke "go" "test" import-path), which only runs the tests in the
root directory of the source. Running the tests in all subdirectories
would require something like this instead:

--8<---------------cut here---------------start------------->8---
(define* (check #:key tests? import-path #:allow-other-keys)
  "Run the tests for the package named by IMPORT-PATH."
  (when tests?
    (invoke "go" "test" (string-append import-path "/...")))
  #t)
--8<---------------cut here---------------end--------------->8---

For example, if the -v flag is added (which might be a better default?)
to the check phase of go-github-stretchr-testify, it can be seen that
only `TestImports' runs, none of the tests in assert, http, etc.
However, the source code in these subdirectories is still recursively
copied to out during the install phase.

Is this desired behaviour? I assumed it isn't, because it looks like we
are skipping a lot of tests during the check phase. However, I might
also simply be overlooking something here as I am new to packaging
Golang with Guix.

Best wishes,

Troy

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 6367 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Golang check phase skipping some tests?

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

On 2024-01-14 22:12:38 +0100, Troy Figiel wrote:
> Hi everyone,
> 
> When looking into the Go build system, I noticed the default check phase
> runs (invoke "go" "test" import-path), which only runs the tests in the
> root directory of the source. Running the tests in all subdirectories
> would require something like this instead:
> 
> --8<---------------cut here---------------start------------->8---
> (define* (check #:key tests? import-path #:allow-other-keys)
>   "Run the tests for the package named by IMPORT-PATH."
>   (when tests?
>     (invoke "go" "test" (string-append import-path "/...")))
>   #t)
> --8<---------------cut here---------------end--------------->8---
> 
> For example, if the -v flag is added (which might be a better default?)
> to the check phase of go-github-stretchr-testify, it can be seen that
> only `TestImports' runs, none of the tests in assert, http, etc.
> However, the source code in these subdirectories is still recursively
> copied to out during the install phase.
> 
> Is this desired behaviour? I assumed it isn't, because it looks like we
> are skipping a lot of tests during the check phase. However, I might
> also simply be overlooking something here as I am new to packaging
> Golang with Guix.

I tend to agree (on both accounts).  I am not aware of anything Guix specific
here that would prevent it, but maybe someone from golang team will chime in.

Have a nice day,
Tomas Volf

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Golang check phase skipping some tests?

[Simon Tournier]

Hi,

CC:
    $ ./etc/teams.scm list-members go
    Katherine Cox-Buday <cox.katherine.e+guix@gmail.com>
    Sharlatan Hellseher <sharlatanus@gmail.com>


On Sun, 14 Jan 2024 at 22:12, Troy Figiel <troy@troyfigiel.com> wrote:

> --8<---------------cut here---------------start------------->8---
> (define* (check #:key tests? import-path #:allow-other-keys)
>   "Run the tests for the package named by IMPORT-PATH."
>   (when tests?
>     (invoke "go" "test" (string-append import-path "/...")))
>   #t)
> --8<---------------cut here---------------end--------------->8---
>
> For example, if the -v flag is added (which might be a better default?)
> to the check phase of go-github-stretchr-testify, it can be seen that
> only `TestImports' runs, none of the tests in assert, http, etc.
> However, the source code in these subdirectories is still recursively
> copied to out during the install phase.
>
> Is this desired behaviour? I assumed it isn't, because it looks like we
> are skipping a lot of tests during the check phase. However, I might
> also simply be overlooking something here as I am new to packaging
> Golang with Guix.

From your description, it seems a good idea.  What do Go “experts“
think?

Cheers,
simon

Re: Golang check phase skipping some tests?

[Maxim Cournoyer]

Hi Tomas,

Tomas Volf <~@wolfsden.cz> writes:

> On 2024-01-14 22:12:38 +0100, Troy Figiel wrote:
>> Hi everyone,
>> 
>> When looking into the Go build system, I noticed the default check phase
>> runs (invoke "go" "test" import-path), which only runs the tests in the
>> root directory of the source. Running the tests in all subdirectories
>> would require something like this instead:
>> 
>> --8<---------------cut here---------------start------------->8---
>> (define* (check #:key tests? import-path #:allow-other-keys)
>>   "Run the tests for the package named by IMPORT-PATH."
>>   (when tests?
>>     (invoke "go" "test" (string-append import-path "/...")))
>>   #t)
>> --8<---------------cut here---------------end--------------->8---
>> 
>> For example, if the -v flag is added (which might be a better default?)
>> to the check phase of go-github-stretchr-testify, it can be seen that
>> only `TestImports' runs, none of the tests in assert, http, etc.
>> However, the source code in these subdirectories is still recursively
>> copied to out during the install phase.
>> 
>> Is this desired behaviour? I assumed it isn't, because it looks like we
>> are skipping a lot of tests during the check phase. However, I might
>> also simply be overlooking something here as I am new to packaging
>> Golang with Guix.
>
> I tend to agree (on both accounts).  I am not aware of anything Guix specific
> here that would prevent it, but maybe someone from golang team will chime in.

I'm not much of a Go person, but your suggestions sound reasonable.
They'd have to be tested on the go-team branch to see whether it breaks
many things (it probably would, since many more tests would be run).

-- 
Thanks,
Maxim

Re: Golang check phase skipping some tests?

[Sharlatan Hellseher]

Hi,

I can't say that I'm an expert in Golang :-), but I've got some experience in
building and deploying Glang daily for some time.

First things first the default behaviour of `go test ./...` is like this:

--8<---------------cut here---------------start------------->8---
find * -type f -name *_test.go | <golang-internal-assertion>
--8<---------------cut here---------------start------------->8---

Internally Golang tries to find any files with _test.go suffix in
provided module's
location and evaluate defined tests in them. If the directory of the module does
not have test files it's just ignored.

Now in actions.

If we investigate which test files we have for `go-github-com-stretchr-testify`
we'll see that for the current version packed in Guix there are some:

--8<---------------cut here---------------start------------->8---
guix describe
Generation 503  Jan 16 2024 13:15:09    (current)
  guix 8520f00
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 8520f00d6989ff5ab22755a97dfcfd0375f6b5ca

guix time-machine --commit=8520f00d6989ff5ab22755a97dfcfd0375f6b5ca -- \
shell findutils -- \
find $(guix build go-github-com-stretchr-testify  --no-substitutes)
-type f -name "*_test.go"
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/suite/suite_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/suite/stats_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/mock/mock_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/package_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/require/requirements_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/require/forward_requirements_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/assert/http_assertions_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/assert/forward_assertions_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/assert/assertion_order_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/assert/assertion_compare_test.go
/gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/assert/assertions_test.go
--8<---------------cut here---------------start------------->8---

Checking current implementation of the go-build-system.scm I realized that it
only runs test available in `import-path` and does not scan any farther like
the `./...` option does. In our case it is just one file, rest are ignored.

> /gnu/store/jpizwxbhr93wmday0raz0vk0ny5s5jiv-go-github-com-stretchr-testify-1.7.0/src/github.com/stretchr/testify/package_test.go

> guix/guix/build/go-build-system.scm
--8<---------------cut here---------------start------------->8---
;; Can this also install commands???
(define* (check #:key tests? import-path #:allow-other-keys)
  "Run the tests for the package named by IMPORT-PATH."
  (when tests?
    (invoke "go" "test" import-path))
  #t)
--8<---------------cut here---------------start------------->8---

With small adjustment of the invok line, I could manage to trigger all tests to
be run, but it brings other issue of some not packed modules required for the
check phase.

--8<---------------cut here---------------start------------->8---
@@ -275,7 +275,7 @@ (define* (build #:key import-path build-flags
#:allow-other-keys)
 (define* (check #:key tests? import-path #:allow-other-keys)
   "Run the tests for the package named by IMPORT-PATH."
   (when tests?
-    (invoke "go" "test" import-path))
+    (invoke "go" "test" (string-append import-path "/...")))
   #t)

 (define* (install #:key install-source? outputs import-path
unpack-path #:allow-other-keys)
--8<---------------cut here---------------start------------->8---

As a quick ad-hoc to run all tests for some new package you may add a custom
check phase with the snippet you provided.

I'm currently in review and split some packages from (gnu packages golang) into
(gnu packages golang-crypto) to simplify the maintenance. I try to play with
that option and see which packages are missed to satisfy passing all tests.

Thanks,
Oleg

On Wed, 17 Jan 2024 at 20:25, Simon Tournier <zimon.toutoune@gmail.com> wrote:
>
> Hi,
>
> CC:
>     $ ./etc/teams.scm list-members go
>     Katherine Cox-Buday <cox.katherine.e+guix@gmail.com>
>     Sharlatan Hellseher <sharlatanus@gmail.com>
>
>
> On Sun, 14 Jan 2024 at 22:12, Troy Figiel <troy@troyfigiel.com> wrote:
>
> > --8<---------------cut here---------------start------------->8---
> > (define* (check #:key tests? import-path #:allow-other-keys)
> >   "Run the tests for the package named by IMPORT-PATH."
> >   (when tests?
> >     (invoke "go" "test" (string-append import-path "/...")))
> >   #t)
> > --8<---------------cut here---------------end--------------->8---
> >
> > For example, if the -v flag is added (which might be a better default?)
> > to the check phase of go-github-stretchr-testify, it can be seen that
> > only `TestImports' runs, none of the tests in assert, http, etc.
> > However, the source code in these subdirectories is still recursively
> > copied to out during the install phase.
> >
> > Is this desired behaviour? I assumed it isn't, because it looks like we
> > are skipping a lot of tests during the check phase. However, I might
> > also simply be overlooking something here as I am new to packaging
> > Golang with Guix.
>
> From your description, it seems a good idea.  What do Go “experts“
> think?
>
> Cheers,
> simon



-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang check phase skipping some tests?

[Maxim Cournoyer]

Hi Oleg,

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> Hi,
>
> I can't say that I'm an expert in Golang :-), but I've got some experience in
> building and deploying Glang daily for some time.
>
> First things first the default behaviour of `go test ./...` is like this:
>
> find * -type f -name *_test.go | <golang-internal-assertion>
>
>
> Internally Golang tries to find any files with _test.go suffix in
> provided module's
> location and evaluate defined tests in them. If the directory of the module does
> not have test files it's just ignored.
>

[...]

> I'm currently in review and split some packages from (gnu packages golang) into
> (gnu packages golang-crypto) to simplify the maintenance. I try to play with
> that option and see which packages are missed to satisfy passing all tests.

A simple thank you for doing this!

-- 
Maxim

Re: Golang check phase skipping some tests?

[Troy Figiel]

[-- Attachment #1.1.1: Type: text/plain, Size: 1361 bytes --]

Hi Oleg and others,

On 2024-01-18 11:25, Sharlatan Hellseher wrote:
> With small adjustment of the invok line, I could manage to trigger all tests to
> be run, but it brings other issue of some not packed modules required for the
> check phase.

Thanks for the update! I noticed the same with `go-github-com-kr-text'.
It was actually missing a propagated-input that has not been packaged
yet, so I couldn't easily fix it.

On 2024-01-18 11:25, Sharlatan Hellseher wrote:
> As a quick ad-hoc to run all tests for some new package you may add a custom
> check phase with the snippet you provided.
> 
> I'm currently in review and split some packages from (gnu packages golang) into
> (gnu packages golang-crypto) to simplify the maintenance. I try to play with
> that option and see which packages are missed to satisfy passing all tests.

Once the migration is over, what would you recommend for new packages? I
could see two options here:

1. Change the default check phase and only replace it back to the
previous one for packages that fail to build or

2. Replace the check phase for all packages one-by-one.

I noticed this behaviour when I was packaging gotenberg and as with any
reasonably sized Golang package, this one also has a gazillion
dependencies... I would love to start on the right track :-)

Best wishes,

Troy

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 6367 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Golang check phase skipping some tests?

[Sharlatan Hellseher]

Hi,

There are not too many Golang packages in Guix comparing to other
language spectific modules:

--8<---------------cut here---------------start------------->8---
grep -r "build-system go-build-system" gnu/packages | awk '{print $1}'
| sort | uniq -c | sort -rn
    382 gnu/packages/golang.scm:
     47 gnu/packages/golang-web.scm:
     34 gnu/packages/syncthing.scm:
     17 gnu/packages/golang-check.scm:
      9 gnu/packages/web.scm:
      8 gnu/packages/version-control.scm:
      7 gnu/packages/databases.scm:
      5 gnu/packages/ipfs.scm:
      5 gnu/packages/bioinformatics.scm:
      4 gnu/packages/virtualization.scm:
      4 gnu/packages/networking.scm:
      4 gnu/packages/mail.scm:
      4 gnu/packages/games.scm:
      4 gnu/packages/docker.scm:
      4 gnu/packages/check.scm:
      3 gnu/packages/file-systems.scm:
      3 gnu/packages/admin.scm:
      2 gnu/packages/time.scm:
      2 gnu/packages/textutils.scm:
      2 gnu/packages/terminals.scm:
      2 gnu/packages/password-utils.scm:
      2 gnu/packages/messaging.scm:
      2 gnu/packages/irc.scm:
      2 gnu/packages/geo.scm:
      2 gnu/packages/education.scm:
      2 gnu/packages/curl.scm:
      2 gnu/packages/containers.scm:
      2 gnu/packages/backup.scm:
      1 gnu/packages/xdisorg.scm:
      1 gnu/packages/web-browsers.scm:
      1 gnu/packages/weather.scm:
      1 gnu/packages/vpn.scm:
      1 gnu/packages/tls.scm:
      1 gnu/packages/terraform.scm:
      1 gnu/packages/tcl.scm:
      1 gnu/packages/task-runners.scm:
      1 gnu/packages/task-management.scm:
      1 gnu/packages/sync.scm:
      1 gnu/packages/shellutils.scm:
      1 gnu/packages/radio.scm:
      1 gnu/packages/pulseaudio.scm:
      1 gnu/packages/music.scm:
      1 gnu/packages/monitoring.scm:
      1 gnu/packages/linux.scm:
      1 gnu/packages/image-viewers.scm:
      1 gnu/packages/hyperledger.scm:
      1 gnu/packages/high-availability.scm:
      1 gnu/packages/finance.scm:
      1 gnu/packages/disk.scm:
      1 gnu/packages/debug.scm:
      1 gnu/packages/crypto.scm:
      1 gnu/packages/configuration-management.scm:
      1 gnu/packages/compression.scm:
      1 gnu/packages/calendar.scm:
      1 gnu/packages/authentication.scm:
      1 gnu/packages/android.scm:
--8<---------------cut here---------------start------------->8---

We may enable it globally and rebuild each package and pack and place missing in
native inputs/propagated inputs depending on the purpose.

I would love to investigate the count of packages in  `34
gnu/packages/syncthing.scm:` :-)

Thanks,
Oleg

On Thu, 18 Jan 2024 at 21:31, Troy Figiel <troy@troyfigiel.com> wrote:
>
> Hi Oleg and others,
>
> On 2024-01-18 11:25, Sharlatan Hellseher wrote:
> > With small adjustment of the invok line, I could manage to trigger all tests to
> > be run, but it brings other issue of some not packed modules required for the
> > check phase.
>
> Thanks for the update! I noticed the same with `go-github-com-kr-text'.
> It was actually missing a propagated-input that has not been packaged
> yet, so I couldn't easily fix it.
>
> On 2024-01-18 11:25, Sharlatan Hellseher wrote:
> > As a quick ad-hoc to run all tests for some new package you may add a custom
> > check phase with the snippet you provided.
> >
> > I'm currently in review and split some packages from (gnu packages golang) into
> > (gnu packages golang-crypto) to simplify the maintenance. I try to play with
> > that option and see which packages are missed to satisfy passing all tests.
>
> Once the migration is over, what would you recommend for new packages? I
> could see two options here:
>
> 1. Change the default check phase and only replace it back to the
> previous one for packages that fail to build or
>
> 2. Replace the check phase for all packages one-by-one.
>
> I noticed this behaviour when I was packaging gotenberg and as with any
> reasonably sized Golang package, this one also has a gazillion
> dependencies... I would love to start on the right track :-)
>
> Best wishes,
>
> Troy



-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang check phase skipping some tests?

[Maxim Cournoyer]

Hi Sharlatan,

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> Hi Maxim,
>
> You mentioned go-team branch which I tried to find :-)
>
> Is there any formal procedure to push new branches?
> I might need a branch to push changes from the split task
> instead of sending patches.

The branch workflow for teams is to use a *-team branch that is short
lived, e.g. for the time needed to do the integration work; with an
associated job spec in Cuirass (ci.guix.gnu.org) to build it.

Patches should still ideally be reviewed on the guix-patches mailing
list before merging to the branch, especially if multiple people are
working on the branch.

A request for merge message is then sent to guix-patches@gnu.org as
explained in (info "(guix) Managing Patches and Branches"), and we wait
for the branch to be scheduled and built by QA, which should hopefully
catch any problems with it before it's merged.

At least that's the spirit; when the QA is lagging too much behind and
nothing happens for a long time, we can fallback to a heavier local
testing as we were doing previously, back when we didn't have these
handy tools.  I use some shell procedures you can find here [0] that may
be of use to build dependent packages and such.

[0]  https://notabug.org/apteryx/guix-api-examples/src/master/command-line-hacks.sh

-- 
Thanks,
Maxim

Re: Golang check phase skipping some tests?

[Sharlatan Hellseher]

Hi Maxim,

Thank you for detailed replay.

> The branch workflow for teams is to use a *-team branch that is short
> lived, e.g. for the time needed to do the integration work; with an
> associated job spec in Cuirass (ci.guix.gnu.org) to build it.

Am I ok to push new go-team branch? And I'll push patches related to
split v3 soon this week.

Thanks,
Oleg

-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang check phase skipping some tests?

[Maxim Cournoyer]

Hi Sharlatan,

Sharlatan Hellseher <sharlatanus@gmail.com> writes:

> Hi Maxim,
>
> Thank you for detailed replay.
>
>> The branch workflow for teams is to use a *-team branch that is short
>> lived, e.g. for the time needed to do the integration work; with an
>> associated job spec in Cuirass (ci.guix.gnu.org) to build it.
>
> Am I ok to push new go-team branch? And I'll push patches related to
> split v3 soon this week.

I'm not in the go-team, so the safest bet to get everyone onboard would
be to send your series to guix-devel first via 'git send-email'; go team
members will be notified.

After a two weeks (or before that if you get it reviewed by other
members), you could then merge to go-team and have it built.

-- 
Thanks,
Maxim

Re: Golang check phase skipping some tests?

[Troy Figiel]

[-- Attachment #1.1.1: Type: text/plain, Size: 3861 bytes --]

Hi Oleg,

Small update to give you some numbers. Looking through gnu/packages/ I
found a total of 593 packages using the go-build-system. With guix
refresh -l, I find 1449 dependent packages. I am currently at commit
bed3a0b547a59f63b545db165e76ddd9af8bba1a. Take this as a rough estimate,
since I searched back for "define-public" for every occurrence of
"build-system go-build-system".

Without the proposed change in the check phase only a single package
fails to build for me (chezmoi, using x86_64-linux). With the change
this number goes up to 214.

Quite a few failures can be solved by adding missing inputs, but there
is still a fair amount of failures that is harder to solve (cyclic
dependencies, unclear test failures).

All in all, as to be expected I would say. A go-team branch as mentioned
in the other email chain, would be very helpful. In the meantime I will
see how far I get solving some of the build failures locally and sending
those patches to guix-patches.

Best wishes,

Troy

On 2024-01-18 22:45, Sharlatan Hellseher wrote:
> Hi,
> 
> There are not too many Golang packages in Guix comparing to other
> language spectific modules:
> 
> --8<---------------cut here---------------start------------->8---
> grep -r "build-system go-build-system" gnu/packages | awk '{print $1}'
> | sort | uniq -c | sort -rn
>     382 gnu/packages/golang.scm:
>      47 gnu/packages/golang-web.scm:
>      34 gnu/packages/syncthing.scm:
>      17 gnu/packages/golang-check.scm:
>       9 gnu/packages/web.scm:
>       8 gnu/packages/version-control.scm:
>       7 gnu/packages/databases.scm:
>       5 gnu/packages/ipfs.scm:
>       5 gnu/packages/bioinformatics.scm:
>       4 gnu/packages/virtualization.scm:
>       4 gnu/packages/networking.scm:
>       4 gnu/packages/mail.scm:
>       4 gnu/packages/games.scm:
>       4 gnu/packages/docker.scm:
>       4 gnu/packages/check.scm:
>       3 gnu/packages/file-systems.scm:
>       3 gnu/packages/admin.scm:
>       2 gnu/packages/time.scm:
>       2 gnu/packages/textutils.scm:
>       2 gnu/packages/terminals.scm:
>       2 gnu/packages/password-utils.scm:
>       2 gnu/packages/messaging.scm:
>       2 gnu/packages/irc.scm:
>       2 gnu/packages/geo.scm:
>       2 gnu/packages/education.scm:
>       2 gnu/packages/curl.scm:
>       2 gnu/packages/containers.scm:
>       2 gnu/packages/backup.scm:
>       1 gnu/packages/xdisorg.scm:
>       1 gnu/packages/web-browsers.scm:
>       1 gnu/packages/weather.scm:
>       1 gnu/packages/vpn.scm:
>       1 gnu/packages/tls.scm:
>       1 gnu/packages/terraform.scm:
>       1 gnu/packages/tcl.scm:
>       1 gnu/packages/task-runners.scm:
>       1 gnu/packages/task-management.scm:
>       1 gnu/packages/sync.scm:
>       1 gnu/packages/shellutils.scm:
>       1 gnu/packages/radio.scm:
>       1 gnu/packages/pulseaudio.scm:
>       1 gnu/packages/music.scm:
>       1 gnu/packages/monitoring.scm:
>       1 gnu/packages/linux.scm:
>       1 gnu/packages/image-viewers.scm:
>       1 gnu/packages/hyperledger.scm:
>       1 gnu/packages/high-availability.scm:
>       1 gnu/packages/finance.scm:
>       1 gnu/packages/disk.scm:
>       1 gnu/packages/debug.scm:
>       1 gnu/packages/crypto.scm:
>       1 gnu/packages/configuration-management.scm:
>       1 gnu/packages/compression.scm:
>       1 gnu/packages/calendar.scm:
>       1 gnu/packages/authentication.scm:
>       1 gnu/packages/android.scm:
> --8<---------------cut here---------------start------------->8---
> 
> We may enable it globally and rebuild each package and pack and place missing in
> native inputs/propagated inputs depending on the purpose.
> 
> I would love to investigate the count of packages in  `34
> gnu/packages/syncthing.scm:` :-)
> 
> Thanks,
> Oleg
> 

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 6367 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Golang check phase skipping some tests?

[Troy Figiel]

[-- Attachment #1.1.1: Type: text/plain, Size: 4692 bytes --]

Hi Oleg (and the go-team),

As I have spent some time with the go-build-system, I have a couple of
extra suggestions/ideas. Obviously, as the person suggesting these
changes, I would be more than happy to help implement them. This will
unfortunately be a rather lengthy and very information-dense email due
to the specifics of the build system, so here goes:

It seems originally the intention of the Go build system was to have
Guix packages per module. For example, to package
"golang.org/x/net/ipv4" as a separate Guix package
`go-golang-org-x-net-ipv4`, we would set `unpack-path` to
"golang.org/x/net/ipv4" and `import-path` to the base of the Git
repository, i.e. to "golang.org/x/net". See for example the comments in
"guix/build/go-build-system.scm":

--8<---------------cut here---------------start------------->8---
;; In general, Go software is built using a standardized build mechanism
;; that does not require any build scripts like Makefiles.  This means
;; that all modules of modular libraries cannot be built with a single
;; command.  Each module must be built individually.  This complicates
;; certain cases, and these issues are currently resolved by creating a
;; file system union of the required modules of such libraries.  I think
;; this could be improved in future revisions of the go-build-system.
--8<---------------cut here---------------end--------------->8---

There is also a note in the TODOs:

--8<---------------cut here---------------start------------->8---
;; * Remove module packages, only offering the full Git repos? This is
;; more idiomatic, I think, because Go downloads Git repos, not modules.
;; What are the trade-offs?
--8<---------------cut here---------------end--------------->8---

I agree with the statement that this split is not idiomatic. To answer
the question, I see at least several downsides of our current situation:

- Cross-module dependencies for the same overarching Go package/tree are
only specified in the files they are used in, not in "go.mod". To
clarify my use of "Go tree", "golang.org/x/net/ipv4" and
"golang.org/x/net/ipv6" would belong to the same "golang.org/x/net"
tree. There would be no feasible, systematic way to figure out all
dependencies for a large enough package, besides failing to build and
adding missing dependencies on the fly.
- Since there is no internal versioning inside a single Go tree, all
modules of a single Go tree would need to be upgraded together or you
might face an unexpected breakage.
- The module-based naming scheme has not been followed consistently. I
think this is mainly due to Go importer not working in this way. This
led to some annoyances when I was packaging some Go package, since I
assumed a module was included when in actuality it was not (for example
`go-github-com-kylelemons-godebug`). I have also noticed some duplicate
packages, such as `go-etcd-io-bbolt` and `go-go-etcd-io-bbolt`.

From what I can tell, the idiomatic way would be to append "/..." to the
import path, both for `go install` and for `go test`. This would
recursively install any binaries and test all packages.

Unfortunately though, neither change can easily be done incrementally.
Quite a few Go package builds are missing inputs and this is not caught
by the current build system. I have started adding some missing inputs
here and there, but from my point of view, we cannot reasonably proceed
without a dedicated go-team branch.

Some additional and miscellaneous suggestions in no particular order:

- With Python I can run something like `guix shell python python-pandas`
and immediately start development. With Go this is not as simple and at
least some environmental variables need to be set beforehand. I had to
figure out how this works from go-build-system.scm, so I think it would
be good to explicitly document how to use Guix for local Go development
for anyone who wants to give it a try.
- I find the build phase hard to debug with the current `go install`
flags. The combination "-v -x" that is set by default gives way too much
information in my opinion.
- Some builds fail due to the default `go test` timeout of 10 minutes.
For example, I was not able to build go-etcd-io-bbolt locally at all. I
would suggest we turn off the timeout to ensure local reproducibility.
- As far as I can tell, there is no way to skip single tests with the
current build system. This is easy to implement, because `go test` comes
with a "-skip" flag. There is no equivalent of `build-flags` for the
check phase yet, so I think it would make a lot of sense to add a
`test-flags` parameter.

Best wishes,

Troy

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 6367 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Golang check phase skipping some tests?

[Simon Tournier]

Hi,

Late to the party. :-)  Processing my backlog…

On jeu., 18 janv. 2024 at 10:25, Sharlatan Hellseher <sharlatanus@gmail.com> wrote:

> I'm currently in review and split some packages from (gnu packages golang) into
> (gnu packages golang-crypto) to simplify the maintenance. I try to play with
> that option and see which packages are missed to satisfy passing all tests.

What is the status of this?  Is all fine?

Cheers,
simon

Re: Golang check phase skipping some tests?

[Sharlatan Hellseher]

Hi Simon!

> What is the status of this?  Is all fine?

There are new modules available which I use for moving packages from golang.scm
and other places e.g. syncthing.scm.

- golang-build.scm
- golang-check.scm
- golang-compression.scm
- golang-crypto.scm
- golang-web.scm
- golang-xyz.scm

I would push go-team branch to check some lower level modifications to
go-build-system which are queued right now. I need someone with admin access to
set the branch on CI as well ;-)

Thanks,
Oleg

-- 
VCS: https://github.incerto.xyz/; https://git.sr.ht/~hellseher/
GPG: 9847 81DE 689C 21C2 6418 0867 76D7 27BF F62C D2B5

… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

Re: Golang check phase skipping some tests?

[Simon Tournier]

Hi,

On jeu., 15 févr. 2024 at 10:10, Sharlatan Hellseher <sharlatanus@gmail.com> wrote:

> I would push go-team branch to check some lower level modifications to
> go-build-system which are queued right now. I need someone with admin access to
> set the branch on CI as well ;-)

Cool!  Thank you.

Cheers,
simon