Shepherd in Debian

Shepherd in Debian

[Simon Josefsson via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 1898 bytes --]

Hi

I've packaged Shepherd 1.0.0 and guile-fibers 1.3.1 for Debian, and they
are in the NEW queue pending copyright review.  Meanwhile you can test
the Salsa amd64 and i386 builds like this:

echo 'deb [trusted=yes] https://salsa.debian.org/debian/guile-fibers/-/jobs/6733535/artifacts/raw/aptly unstable main' | sudo tee --append /etc/apt/sources.list.d/add.list
echo 'deb [trusted=yes] https://salsa.debian.org/debian/shepherd/-/jobs/6737847/artifacts/raw/aptly unstable main' | sudo tee --append /etc/apt/sources.list.d/add.list
sudo apt-get update
sudo apt-get install shepherd

It was possible to install it on my Trisquel aramo laptop, so I suspect
it will work on a varied mix of not-necessarily latest Debian, Ubuntu,
PureOS, Trisquel, Devuan, Gnuinos etc.  Feel free to build the packages
yourself if you don't trust the pre-built binaries, sources are here:

https://salsa.debian.org/debian/guile-fibers
https://salsa.debian.org/debian/shepherd

The package provides /usr/bin/herd and /usr/bin/shepherd, bash
completion, translations, info manual and man pages.

The halt, reboot and shutdown tools are not installed as that felt like
a bad idea.

There is no integration into any system services, the packaging is
minimal.

I've not tested it beyond running 'herd --help' and 'shepherd --help',
which is the reason for this e-mail: could someone test and give some
example of a useful thing to use this for on Debian?

Patches, merge requests, and co-maintainers on the packaging are
welcome.

Btw, running 'herd --help' prints a lot of warnings like below.  Any
ideas where these come from and/or how to silence them?  Salsa used
Guile 3.0.9 and my laptop has Guile 3.0.7, if that matters.

;;; WARNING: loading compiled file /usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache/shepherd/scripts/herd.go failed:
;;; In procedure load-thunk-from-memory: incompatible bytecode version

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Shepherd in Debian

[Tobias Alexandra Platen]

This is definitively interesting. Currenltly my plan is to use Shepherd
on my smartphone which has both Debian and Android installed. On top of
Debian I have a working guix install and guix is also shared on
Android. I also plan to use guix-android:
https://framagit.org/tyreunom/guix-android

On Thu, 2024-12-12 at 17:10 +0100, Simon Josefsson via Development of
GNU Guix and the GNU System distribution. wrote:
> Hi
> 
> I've packaged Shepherd 1.0.0 and guile-fibers 1.3.1 for Debian, and
> they
> are in the NEW queue pending copyright review.  Meanwhile you can
> test
> the Salsa amd64 and i386 builds like this:
> 
> echo 'deb [trusted=yes]
> https://salsa.debian.org/debian/guile-fibers/-/jobs/6733535/artifacts/raw/aptly
>  unstable main' | sudo tee --append /etc/apt/sources.list.d/add.list
> echo 'deb [trusted=yes]
> https://salsa.debian.org/debian/shepherd/-/jobs/6737847/artifacts/raw/aptly
>  unstable main' | sudo tee --append /etc/apt/sources.list.d/add.list
> sudo apt-get update
> sudo apt-get install shepherd
> 
> It was possible to install it on my Trisquel aramo laptop, so I
> suspect
> it will work on a varied mix of not-necessarily latest Debian,
> Ubuntu,
> PureOS, Trisquel, Devuan, Gnuinos etc.  Feel free to build the
> packages
> yourself if you don't trust the pre-built binaries, sources are here:
> 
> https://salsa.debian.org/debian/guile-fibers
> https://salsa.debian.org/debian/shepherd
> 
> The package provides /usr/bin/herd and /usr/bin/shepherd, bash
> completion, translations, info manual and man pages.
> 
> The halt, reboot and shutdown tools are not installed as that felt
> like
> a bad idea.
> 
> There is no integration into any system services, the packaging is
> minimal.
> 
> I've not tested it beyond running 'herd --help' and 'shepherd --
> help',
> which is the reason for this e-mail: could someone test and give some
> example of a useful thing to use this for on Debian?
> 
> Patches, merge requests, and co-maintainers on the packaging are
> welcome.
> 
> Btw, running 'herd --help' prints a lot of warnings like below.  Any
> ideas where these come from and/or how to silence them?  Salsa used
> Guile 3.0.9 and my laptop has Guile 3.0.7, if that matters.
> 
> ;;; WARNING: loading compiled file /usr/lib/x86_64-linux-
> gnu/guile/3.0/site-ccache/shepherd/scripts/herd.go failed:
> ;;; In procedure load-thunk-from-memory: incompatible bytecode
> version
> 
> /Simon

Re: Shepherd in Debian

[Ludovic Courtès]

Hi Simon,

Simon Josefsson via "Development of GNU Guix and the GNU System
distribution." <guix-devel@gnu.org> skribis:

> I've packaged Shepherd 1.0.0 and guile-fibers 1.3.1 for Debian, and they
> are in the NEW queue pending copyright review.  Meanwhile you can test
> the Salsa amd64 and i386 builds like this:

Nice!

> The halt, reboot and shutdown tools are not installed as that felt like
> a bad idea.

That would have been fun.  :-)

> Btw, running 'herd --help' prints a lot of warnings like below.  Any
> ideas where these come from and/or how to silence them?  Salsa used
> Guile 3.0.9 and my laptop has Guile 3.0.7, if that matters.
>
> ;;; WARNING: loading compiled file /usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache/shepherd/scripts/herd.go failed:
> ;;; In procedure load-thunk-from-memory: incompatible bytecode version

That indicates a discrepancy between the Guile used to build the
Shepherd and the one used at run time.  Object files built by 3.0.9 (as
is the case here) cannot be understood by 3.0.7.

How’s that handled for other Guile packages in Debian?  Vagrant must
know. :-)

Ludo’.

Re: Shepherd in Debian

[Simon Josefsson via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

>> Btw, running 'herd --help' prints a lot of warnings like below.  Any
>> ideas where these come from and/or how to silence them?  Salsa used
>> Guile 3.0.9 and my laptop has Guile 3.0.7, if that matters.
>>
>> ;;; WARNING: loading compiled file /usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache/shepherd/scripts/herd.go failed:
>> ;;; In procedure load-thunk-from-memory: incompatible bytecode version
>
> That indicates a discrepancy between the Guile used to build the
> Shepherd and the one used at run time.  Object files built by 3.0.9 (as
> is the case here) cannot be understood by 3.0.7.
>
> How’s that handled for other Guile packages in Debian?  Vagrant must
> know. :-)

Maybe that's just my local problem: I was taking those packages from a
Debian unstable build and running them on a Trisquel aramo machine.
Still, if there is some common magic that Debian Guile packages could do
to improve things, I want to learn.  Build them using an old Guile
perhaps?  Assuming object files built by 3.0.7 work fine with 3.0.9.

Btw, is there a Debian Guile packaging wiki page or group?  I looked at
a couple of different guile packages in Debian and they all seem to do
some similar things wrt dwz, shlibs, etc; having this documented
somewhere would have helped me.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Shepherd in Debian

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]

On 2024-12-15, Ludovic Courtès wrote:
> Simon Josefsson via "Development of GNU Guix and the GNU System
> distribution." <guix-devel@gnu.org> skribis:
>> Btw, running 'herd --help' prints a lot of warnings like below.  Any
>> ideas where these come from and/or how to silence them?  Salsa used
>> Guile 3.0.9 and my laptop has Guile 3.0.7, if that matters.
>>
>> ;;; WARNING: loading compiled file /usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache/shepherd/scripts/herd.go failed:
>> ;;; In procedure load-thunk-from-memory: incompatible bytecode version
>
> That indicates a discrepancy between the Guile used to build the
> Shepherd and the one used at run time.  Object files built by 3.0.9 (as
> is the case here) cannot be understood by 3.0.7.
>
> How’s that handled for other Guile packages in Debian?  Vagrant must
> know. :-)

In short, poorly. :(

To avoid noise like that, it may require rebuilding the package, and the
usual detection mechanisms in Debian for things that require rebuilds
(e.g. ABI bumps in C libraries) do not appear to trigger in any way with
guile packages.

This seems especially important with packages that have guile bindings
to C libraries (e.g. guile-git, guile-ssh, guile-zlib, etc.) which may
break in even more significant ways.

So, the only answer I have come up with is that I periodically rebuild
stuff when it seems like it breaks.

I suspect a fair number of the test suite failures in the Debian
packages of Guix might be due to issues along these lines. Many of the
tests are disabled for good reason (e.g. network-dependent tests or
tests that assume bootstrap binaries are available) but honestly, many
of them are just disabled to keep me from pulling my hair out. :/

On a good day I am happy to keep Guix and the many necessary
guile-specific packages working in Debian! On a bad day I wonder pretty
hard if it is worth the effort. I maintain quite a few guile packages in
Debian, despite being at best ambivalent to guile itself. :)

There was a big change in Debian recently where guile was downgraded
from 3.10.x back to 3.9.x. That was a little bit exciting.

I am not sure how to better automate or detect that sort of thing...
Guix avoids ABI issues entirely by rebuilding everything whenever
anything in the dependency chain changes, and is also perhaps a
significant driver of Guile usage ... so I can see how this might be
deprioritized in cases where ABI actually matters, though most
distributions still do largely use ABI to limit rebuilds to only when
"necessary".


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Shepherd in Debian

[Simon Josefsson via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 4185 bytes --]

Vagrant Cascadian <vagrant@debian.org> writes:

> On 2024-12-15, Ludovic Courtès wrote:
>> Simon Josefsson via "Development of GNU Guix and the GNU System
>> distribution." <guix-devel@gnu.org> skribis:
>>> Btw, running 'herd --help' prints a lot of warnings like below.  Any
>>> ideas where these come from and/or how to silence them?  Salsa used
>>> Guile 3.0.9 and my laptop has Guile 3.0.7, if that matters.
>>>
>>> ;;; WARNING: loading compiled file /usr/lib/x86_64-linux-gnu/guile/3.0/site-ccache/shepherd/scripts/herd.go failed:
>>> ;;; In procedure load-thunk-from-memory: incompatible bytecode version
>>
>> That indicates a discrepancy between the Guile used to build the
>> Shepherd and the one used at run time.  Object files built by 3.0.9 (as
>> is the case here) cannot be understood by 3.0.7.
>>
>> How’s that handled for other Guile packages in Debian?  Vagrant must
>> know. :-)
>
> In short, poorly. :(
>
> To avoid noise like that, it may require rebuilding the package, and the
> usual detection mechanisms in Debian for things that require rebuilds
> (e.g. ABI bumps in C libraries) do not appear to trigger in any way with
> guile packages.

I think my error can be written off because I installed the Debian
packages on my Trisquel laptop.  The packages are still in the NEW queue
but I'll look into if any of these errors remain later on.

> This seems especially important with packages that have guile bindings
> to C libraries (e.g. guile-git, guile-ssh, guile-zlib, etc.) which may
> break in even more significant ways.
>
> So, the only answer I have come up with is that I periodically rebuild
> stuff when it seems like it breaks.
>
> I suspect a fair number of the test suite failures in the Debian
> packages of Guix might be due to issues along these lines. Many of the
> tests are disabled for good reason (e.g. network-dependent tests or
> tests that assume bootstrap binaries are available) but honestly, many
> of them are just disabled to keep me from pulling my hair out. :/
>
> On a good day I am happy to keep Guix and the many necessary
> guile-specific packages working in Debian! On a bad day I wonder pretty
> hard if it is worth the effort. I maintain quite a few guile packages in
> Debian, despite being at best ambivalent to guile itself. :)
>
> There was a big change in Debian recently where guile was downgraded
> from 3.10.x back to 3.9.x. That was a little bit exciting.
>
> I am not sure how to better automate or detect that sort of thing...
> Guix avoids ABI issues entirely by rebuilding everything whenever
> anything in the dependency chain changes, and is also perhaps a
> significant driver of Guile usage ... so I can see how this might be
> deprioritized in cases where ABI actually matters, though most
> distributions still do largely use ABI to limit rebuilds to only when
> "necessary".

For what it's worth, I am happy you do maintain Guix in Debian and find
it useful, and on all Debian-derived machines I run I install Guix from
packaging.

When I packaged guile-fibers and shepherd one idea to me was that it
would be nice with a Debian Guile Maintainers team to collaborate on
Guile-related packages.  All these packages share similar quirky hacks
in debian/rules and I noticed the error messages and cut'n'pasted the
same workarounds for them that everyone else has discovered.  Would this
help to bridge Debian and Guix a bit?  If I had seen about the guile
3.0.10 32-bit bug earlier I could have some ideas, I'm struggling with
fixing some low-level packages like priv-wrapper and socket-wrapper for
i386/armhf too.  It is really unfortunate if guile is stuck on 3.0.9 in
Debian just because of i386/armhf issues.

Or is there a Debian Guile group already that I don't know about?

It would be nice if the subset of people who know Guile and Debian
packaging could collaborate more easily..  is talking about that on
guile-devel appropriate, or should we ask for a
debian-guile@lists.debian.org?  Assuming we can do an inventory of
guile-* maintainers in Debian and large number of them think it is a
good idea.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Shepherd in Debian

[Pjotr Prins]

On Sun, Dec 29, 2024 at 11:41:44PM +0100, Simon Josefsson via Development of GNU Guix and the GNU System distribution. wrote:
> For what it's worth, I am happy you do maintain Guix in Debian and find
> it useful, and on all Debian-derived machines I run I install Guix from
> packaging.

Totally agree and thank you Vagrant. We run some 30 machines with
Debian and Guix on top and it is a rock solid setup with Guix VMs and
containers. Recently, Collin has successfully added a Guix on bare metal
on a Genoa Dell server for a Guix build farm (more on that soon), so
we may have more of that coming.

Pj.

Re: Shepherd in Debian

[Rob Browning]

Simon Josefsson <simon@josefsson.org> writes:

> It is really unfortunate if guile is stuck on 3.0.9 in Debian just
> because of i386/armhf issues.

For what it's worth, it was all the 32-bit architectures, and as
mentioned, I'm hoping that we'll have a 3.0.11 before trixie freezes
(still not set, but I'd imagine it will be "soon" --
 https://release.debian.org/testing/freeze_policy.html).

> Or is there a Debian Guile group already that I don't know about?

There's a debian-scheme list, though, personally, I haven't kept up with
it the way I might like, and that might not be quite what you're looking
for:

  https://lists.debian.org/debian-scheme/

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Shepherd in Debian

[Simon Josefsson via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Rob Browning <rlb@defaultvalue.org> writes:

> Simon Josefsson <simon@josefsson.org> writes:
>
>> It is really unfortunate if guile is stuck on 3.0.9 in Debian just
>> because of i386/armhf issues.
>
> For what it's worth, it was all the 32-bit architectures, and as
> mentioned, I'm hoping that we'll have a 3.0.11 before trixie freezes
> (still not set, but I'd imagine it will be "soon" --
>  https://release.debian.org/testing/freeze_policy.html).

Is a 3.0.11 release likely for Guile in the next few days/week?  I
suspect the Debian freeze may not be that long away, and I think getting
3.0.11 into unstable and then rebuilding all Guile packages in Debian
would be nice.  I must confess I don't understand if rebuilding guile
packages will be necessary, but at least we should test that the warning
message aren't shown if somehow we manage to get 3.0.11 into trixie
without rebuilt guile-foo packages.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Shepherd in Debian

[Rob Browning]

Vagrant Cascadian <vagrant@debian.org> writes:

> To avoid noise like that, it may require rebuilding the package, and the
> usual detection mechanisms in Debian for things that require rebuilds
> (e.g. ABI bumps in C libraries) do not appear to trigger in any way with
> guile packages.

While I haven't thought hard about this with respect to Guile yet, and
even ignoring anything involving per-architecture binary ABIs
(i.e. per-arch shared libs), I suspect the problem is "reasonably
difficult", (though note that I'm mostly speaking here from a Debian
perspective, and based on similar experiences with Emacs), and I would
be very happy to be wrong.

One option, of course, is to just omit .go files from "add on" (Guile
dependent) packages, and let autocompilation (per user) handle
everything.  This is expensive (and who cleans up, when?), but
side-steps a lot of other issues.

For Guile at least, you could also ship a full set of .go files in each
package (i.e. all four size and endianness combinations) with the
attendant size increases, but I believe this may be insufficient, even
if you were to include the "go-ABI" in the package dependencies (which
would require either versioned Guile packages per go-ABI change[1], or
"flag days" in unstable whenever the go-ABI changes).

It may be insufficient unless every add-on package carefully versions
its Scheme API, and remembers to include relevant changes to any of its
public macros as part of of that API, because of course macro changes
could require rebuilding all of the package's reverse dependencies.

Alternately, instead of shipping go files, you could decide to just
build/rebuild/remove them during package installation/removal for the
installed Guile versions (guile-X.Ys), and Debian's Emacs policy tries
to for emacsen, but that's also fraught (in Debian at least), given what
the install/remove/upgrade maintainer script invocations allow[2].

A potential additional problem this approach creates is that you might
want have just one "add on" package, say guile-foo, work with multiple
Guile versions (guile-3.1, guile-3.2, etc. as Emacs policy does), but
then guile-foo *and* all of its guile-related reverse dependencies need
to be rebuilt/removed for a given Guile "flavor" opportunistically,
whenever any guile-X.Y is installed/upgraded/removed.

And *that* raises the bar substantially (again in Debian, at least),
because of [2] coupled with the fact that you might need to run the
rebuilds/removals in proper inter-package dependency order (unless
upstream has sufficient built-in tooling for that) outside the normal
apt/dpkg process.

And since any given apt/dpkg run could be installing or removing various
things, including guile-X.Y flavors in the same run with add-on package
(e.g. guile-foo) installs/removals/upgrades, "it's complicated".

[1] ...which Debian already has via the guile-X.Y versioning, *if* Guile
    never changes the ABI in a Z version.

[2] The current Emacs policy is broken, and coming up with a fix has
    been difficult.  The problem arose when we realized that (roughly)
    the only places *Debian* policy says you can count on your
    dependencies being available/ready are in your "postinst configure
    ..."  and your "postinst triggered ..." (and it appears that
    triggers don't promise any ordering wrt deps).  So, for example, you
    can't rely on anything outside your package or the Debian base tools
    (perl-base, ...) to clean up (in the preinst/prerm/postrm).

    Debian's current Emacs policy, had assumed (and requires) add-on
    packages to depend on and make calls to emacsen-common package tools
    from their prerm, for example, and that can (and as we saw, does)
    just break sometimes when emacsen-common, dependency or not, isn't
    properly installed.

> There was a big change in Debian recently where guile was downgraded
> from 3.10.x back to 3.9.x. That was a little bit exciting.

Right, we had do do that because 3.0.10 was just broken on 32-bit
architectures.  I'm hoping we'll have a 3.0.11 we can move to for
trixie.

Thanks, and I hope I'm just wrong somehow regarding the complexity here.
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4