Strategy for Zig packages

Strategy for Zig packages

[mcsinyx]

Hello Guix,

Coming from Nix, I am considering switching to Guix in long term.
So far, my biggest gripe with nixpkgs culture, i.e. package vendoring,
seems to be well objected by Guix philosophy.

My current goal is to package software I use and/or maintain,
some of which are written in Zig.  Since Zig is still immature and
there is no consensus on downstream packaging, most user applications,
e.g. ncdu or mepo, resorted to only link with C ABI libraries.

Like Go, Zig code is meant to be statically compiled (it is possible
to write libraries with C ABI in Zig, though they fall under the former
category).  Since the language-specific package managers are still
experimental, some projects, e.g. river or blackshades, choose to vendor
libraries as version-control submodules and rely on relative imports.
Zig packages are specified (recursively) via

zig build-* --pkg-begin name path --pkg-end ...

so I suppose the Zig build system can wrap the command appropriately.

The third category is the software using a Zig package manager.
Dependency information should be retrievable from the manifests.

In case this has not been discussed before, what shall be the plan
for interdependent Zig packages in Guix?  On top of my head there are
a few options:

1. Wait until Zig reaches 1.0; it's too soon to decide now.
2. Work with Zig maintainers for a standard way to install
   Zig libraries as source code.  It could be something like
   ZIG_PKGS where package name is at $ZIG_PKGS/$name.zig,
   or a file containing all the mappings.
3. Wrap the zig command and feed it declared dependency information
   while waiting for standardization.

I would love to hear what Guix maintainers' thought on this issue.

All the bests,
McSinyx

Re: Strategy for Zig packages

[Liliana Marie Prikler]

Am Dienstag, dem 26.07.2022 um 19:56 +0900 schrieb mcsinyx@disroot.org:
> In case this has not been discussed before, what shall be the plan
> for interdependent Zig packages in Guix?  On top of my head there are
> a few options:
> 
> 1. Wait until Zig reaches 1.0; it's too soon to decide now.
> 2. Work with Zig maintainers for a standard way to install
>    Zig libraries as source code.  It could be something like
>    ZIG_PKGS where package name is at $ZIG_PKGS/$name.zig,
>    or a file containing all the mappings.
> 3. Wrap the zig command and feed it declared dependency information
>    while waiting for standardization.
4. Convince Zig maintainers to perhaps maybe not join the ranks of Rust
et al. and produce reusable shared libraries?  

I don't think there's a good case to make for reuse via source code
archives (as the only possible way to reuse).  Technically, there's a
similar construct in C++ called header-only library, but if you need
more than one header to implement one, I'd highly question your design
choices.

Cheers

Re: Strategy for Zig packages

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 979 bytes --]


On 26-07-2022 20:48, Liliana Marie Prikler wrote:
> 4. Convince Zig maintainers to perhaps maybe not join the ranks of Rust
> et al. and produce reusable shared libraries?

I'd like to clarify that Rust supports shared libraries (*) just fine, 
it's Cargo that insists on source code.

Looking at 
<https://gist.github.com/matu3ba/92e5df1166c51b3725dbd04f7ff1cb4e>, it 
might be similar with Zig, as that page is about 'the Zig caching 
system' and mentions Zir code, ABIs and objects.

(*) or, at least, rustc's version of static libraries, I haven't tested 
out shared libraries yet except for the first failed attempt; there 
might be problems with monomorphisation and macros or somehing. Some 
libraries, e.g. recent versions of rust-nom, also are written in such a 
way that appears to require LTO optimisation to be efficient, which 
appears to be non-trivial to be compatible with shared libraries without 
extra disk space.

Greetings,
Maxime.


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Strategy for Zig packages

[Mája Tomášek]

<mcsinyx@disroot.org> writes:

> a few options:
>
> 1. Wait until Zig reaches 1.0; it's too soon to decide now.
> 2. Work with Zig maintainers for a standard way to install
>    Zig libraries as source code.  It could be something like
>    ZIG_PKGS where package name is at $ZIG_PKGS/$name.zig,
>    or a file containing all the mappings.
> 3. Wrap the zig command and feed it declared dependency information
>    while waiting for standardization.

Hi,

iirc zig does support the shared library model,
but it by default links all objects statically.

I have been thinking about zig packages recently. How could guix say
this is the way to create packages for guix.

My idea that came to me, would be to support guix packages inside vendor
repository that could be exported as rpms/debs etc. etc. And all build
actions made by the mantainers.

But I understand it wouldn't be popular, locking an independent language
into the guix ecosystem.

More realistic (imo) is that zig should be encouraged to build
dynamically linked packages, not static ones, and allow the ability
(with their future package manager) for the distribution to distribute
it's libraries C-style. If there's a newer version required, it should
be up to the distribution mantainers to fix that.

With hope, that I haven't spattered utter nonsense and with love,
Maya

PS. I apologize for sending you the email twice McSinyx, I forgot to add
the mailing list in the copy.

Re: Strategy for Zig packages

[mcsinyx]

On Mon Aug 1, 2022 at 10:43 PM +0200, Mája Tomášek wrote:
> More realistic (imo) is that zig should be encouraged to build
> dynamically linked packages, not static ones, and allow the ability
> (with their future package manager) for the distribution to distribute
> it's libraries C-style.

Technically this is not always possible since Zig relies
on compile-time execution for generic.

> If there's a newer version required, it should
> be up to the distribution mantainers to fix that.

Zig source files could be handled in the same manner as C headers
however, and be used as native inputs, so downstream can still
update a library for all dependees at once.

Re: Strategy for Zig packages

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 511 bytes --]


On 01-08-2022 22:43, Mája Tomášek wrote:
> But I understand it wouldn't be popular, locking an independent language
> into the guix ecosystem.

They could consider multiple package managers and distros (e.g.: Guix 
support, Debian support, vcpkg support, ...) according to what users of 
Zig are using, then they aren't locked into a particular distribution or 
package manager.

A bit more work upfront than rolling their own, but it will have to be 
done anyway later.

Greetings,
Maxime


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Strategy for Zig packages

[Maxime Devos]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 1386 bytes --]


On 02-08-2022 07:21, mcsinyx@disroot.org wrote:
> On Mon Aug 1, 2022 at 10:43 PM +0200, Mája Tomášek wrote:
>> More realistic (imo) is that zig should be encouraged to build
>> dynamically linked packages, not static ones, and allow the ability
>> (with their future package manager) for the distribution to distribute
>> it's libraries C-style.
> Technically this is not always possible since Zig relies
> on compile-time execution for generic.
>
I don't think this is a problem, as Guile has macros (*) yet it supports 
compilation (and shared libraries -- .go files are ELF shared libraries) 
-- macros are, in a sense, merely procedures that transform syntax into 
syntax, which can be compiled, and generics could be considered a 
special case of macros.

Likewise, Rust has generics (and macros (*)) and its compiler does 
static compilation, so I don't see why it would be impossible for Zig to 
do static libraries.

(Shared libraries might be harder, but you could just monomorphise 
generics in the 'user' of the shared library, so shared libraries appear 
technically possible to me as well.)

(*) Including macros that can run arbitrary Rust code, see 
<https://doc.rust-lang.org/book/ch19-06-macros.html#declarative-macros-with-macro_rules-for-general-metaprogramming>, 
so there you have compile-time execution too!

Greetings,
Maxime.


[-- Attachment #1.1.1.2: Type: text/html, Size: 2251 bytes --]

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Strategy for Zig packages

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 533 bytes --]

On 02-08-2022 07:21, mcsinyx@disroot.org wrote:

> Zig source files could be handled in the same manner as C headers
> however, and be used as native inputs, so downstream can still
> update a library for all dependees at once.

Going for native-inputs and not non-native sounds incorrect when 
cross-compiling.  We might need to substitute* a "executable" -> 
"/gnu/store/.../bin/executable", and the latter file name is 
architecture-dependent.  (This holds generally, including for C headers).

Greetings,
Maxime


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Strategy for Zig packages

[mcsinyx]

On Tue Aug 2, 2022 at 11:09 AM +0200, Maxime Devos wrote:
> On 02-08-2022 07:21, mcsinyx@disroot.org wrote:
> > On Mon Aug 1, 2022 at 10:43 PM +0200, Mája Tomášek wrote:
> > > More realistic (imo) is that zig should be encouraged
> > > to build dynamically linked packages, not static ones,
> > > and allow the ability (with their future package manager)
> > > for the distribution to distribute it's libraries C-style.
> >
> > Technically this is not always possible since Zig relies
> > on compile-time execution for generic.
>
> Rust has generics (and macros (*)) and its compiler does
> static compilation, so I don't see why it would be impossible
> for Zig to do static libraries.
>
> (*) Including macros that can run arbitrary Rust code, see
> <https://doc.rust-lang.org/book/ch19-06-macros.html#declarative-macros-with-macro_rules-for-general-metaprogramming>,
> so there you have compile-time execution too!

FWIW I was commenting on the impossibility
of dynamically linking Zig libraries that uses comptime.

On Tue Aug 2, 2022 at 11:09 AM +0200, Maxime Devos wrote:
> I don't think this is a problem, as Guile has macros (*) yet
> it supports compilation (and shared libraries -- .go files
> are ELF shared libraries) -- macros are, in a sense,
> merely procedures that transform syntax into syntax,
> which can be compiled, and generics could be considered
> a special case of macros.
>
> (Shared libraries might be harder, but you could just monomorphise
> generics in the 'user' of the shared library, so shared libraries
> appear technically possible to me as well.)

Compile-time execution must be done at, well, compilation time,
thus macros in shared objects are just glorified embedded
source code that are not used at runtime.

That being said, there may be a Zig ABI in the future, with the cost
of missing certain features (like comptime) and optimizations:
https://github.com/ziglang/zig/issues/3786

On Tue Aug 2, 2022 at 11:11 AM +0200, Maxime Devos wrote:
> On 02-08-2022 07:21, mcsinyx@disroot.org wrote:
> > Zig source files could be handled in the same manner as C headers
> > however, and be used as native inputs, so downstream can still
> > update a library for all dependees at once.
>
> Going for native-inputs and not non-native sounds incorrect
> when cross-compiling.  We might need to substitute* a "executable"
> -> "/gnu/store/.../bin/executable", and the latter file name
> is architecture-dependent.  (This holds generally, including
> for C headers).

Thanks, coming from Nix I forgot that Guix does not make use of PATH.

Re: Strategy for Zig packages

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 1868 bytes --]

> FWIW I was commenting on the impossibility
> of dynamically linking Zig libraries that uses comptime.
It doesn't make a difference, AFAICT my explanation on why it can work 
holds equally for static as for shared libraries.

On 03-08-2022 05:35, mcsinyx@disroot.org wrote:
> Compile-time execution must be done at, well, compilation time,
> thus macros in shared objects are just glorified embedded
> source code that are not used at runtime.

My point was that macros and dynamic (or static) linking is compatible 
with generics and macros. Of course, if your Zig code consists purely of 
generics and macros, it's kind of pointless, but presumably there's 
non-generic code too and non-generic library code that chooses a 
particular instance of a generic (monomorphisation?) that could benefit 
from compiling only once instead of being recompiled for every leaf (+- 
application).

> That being said, there may be a Zig ABI in the future, with the cost
> of missing certain features (like comptime) and optimizations:
> https://github.com/ziglang/zig/issues/3786

A fixed, stable, ABI or documented ABI is not required for Guix because 
we have the store and RPATH (or RUNPATH, I always forget which one we 
use) and the store model instead of putting things in /usr/lib and 
searching for them there, so occasional ABI breaks are not a problem for 
Guix.

Distros like, say, Debian, are another matter of course, though they 
could resort to build-from-scratch like done with Rust currently.

You are referring to optimisations, but at least LTO is perfectly 
combinable with static libraries (see, e.g., Rust). Shared libraries 
should be equally possible if it is accepted that it won't be always 
possible anymore to simply replace the shared library with another 
(because of things like inlining).

Greetings,
Maxime

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]