Re: Improving importers best investment for growing gnu/packages/

[Liliana Marie Prikler <liliana.prikler@gmail.com>]

Am Dienstag, dem 29.03.2022 um 10:50 +1100 schrieb Brendan Tildesley:
> On 25/3/22 18:05, Paul Alesius wrote:
> > Rust analyzer is a language server for the Rust programming
> > language.
> > 
> > I've tried to produce a patch for the latest version, and it has
> > probably hundreds of dependencies that need to be updated.
> [...]
> > I'd suggest that a Guix package for rust-analyzer is not needed,
> > especially due to the excessive time required to update its package
> > definition and all of the vendored dependency crates, and focus
> > should instead be on rust and rust-cargo.
> 
> I understand how defeating it feels when looking at the countless
> hundreds of dependencies that get pulled in, however, I believe this
> can be solved by improving the importers in Guix.  I think it
> requires too much manual labour to maintain packages curretly,
> particularly when refresh -u will only update the
> package version and not dependencies.
> Therefore I'm working on making use of
> https://github.com/rust-lang/crates.io-index to fully import base
> definitions all required crates rather than pulling metadata from the
> internet with each refresh.
> Maintaining rust packages then becomes mostly
> 1. Refresh generated package definitions
> 2. In a separate file edit any hacks/fix needed argument fields etc
> 3. Edit cargo-build-system to account for the most common hacks like
> setting
> LIBCLANG_PATH.
I'm not sure how this is for Rust packages, but in general, I'd
consider over-reliance on `guix refresh' more harmful than not.  There
has recently been a paper published, which shows this to be the case
for spreadsheet debugging – give people a smell checker without
appropriate hints where it might be flawed and they will be worse at
finding flaws than the control group.  I make a claim, that it's the
same for packaging software.  Our importers should come with a warning
sign attached that they perform the best possible approximation based
on available metadata, but that they might be missing important
information or contain bugs, and that the output needs human
verification.

In general, even though one might claim that this conclusion has been
largely forgone in the case of cargo, npm, et al., human intelligence
is a requirement in human computer interaction.  Trying to take too
much of it away leads to adverse effects rather than "smart systems".

> It should be easy for a guix user to get all the latest crates that
> they would get using cargo.
> I think importers can be taken to the next level by going beyond
> metadata and actually inspecting the contents of source files. For
> example grepping build files it can be determined if pkg-config is
> used and add it to inputs.
That'd probably fine for pkg-config as a single package, but to get
packages to a working state, you'll have to do so for more than just
pkg-config (you list dependencies referenced by pkg-config below, for
example).  Speaking more generally, short of actually running the build
(repeatedly) against all possible combinations of inputs, you will not
obtain a complete or accurate input set (particularly also considering
that some of said inputs might not even be packaged).  And even if you
do, all your machine could then tell you, is that it builds; not
whether it functions as intended.  This also assumes that the source
won't need to be patched to remove icky stuff, no phase needs to be
adjusted, ...

> An index of .pc files can automatically detect and add the inputs a
> package is most likely looking for. One complaint about Guix is that
> typically just running ./configure; make on a project will not work,
> but if a guix can have a Universal Importer where running guix wave-
> magic-wand in a directory can inspect files, determine the build
> system, required inputs etc and build a program, would that
> not be wonderful for programmers? Working on KDE made me feel like I
> was wasting too much time manually adding inputs and would be better
> of writing an importer.
To be fair, scanning pkg-config files sounds like a good idea until you
consider it is but a heuristic and pkg-config names do not easily map
to package names.

> Using rust in a hypothetical, imagine if a rust developer that is
> otherwise uninterested in Guix, they just want to build their
> software and distribute actually chooses to use Guix not due to
> idealism but because it:
> A. Handles their rust dependecies at least as well as cargo.
> B. Handles their non-rust dependencies even better.
> C. Improved Guix pack/build can produce distributable packages,
> stripped of unneeded files that actually runs on other distributions.
> 
> Would that not be awesome?
Hypothetically speaking, apart from the fact that rust inputs are not
inputs because the Rust ecosystem is awful, maintaining a guix.scm on
the side of your Rust package should not be that much more difficult
than maintaining a cargo.toml.  And better yet, you don't need those
god-awful lock files, you could just commit a channels.scm as well if
bit-for-bit reproducibility in CI is a requirement.  You could even
implement a scheme-based toml parser, that establishes a 1:1 mapping
between toml and guix package.  Ironically, Javascript is ahead in that
regard, as someone has already written a Guix package reader in
ECMAScript.  Wisp is also an option for those experiencing terminal
parenthephobia.

IMHO the onus is on Rust to create an ecosystem that does not collapse
if someone removes leftpad.