Re: How did you handle making a GNU/Linux distribution?

[Maxime Devos <maximedevos@telenet.be>]

[-- Attachment #1: Type: text/plain, Size: 17310 bytes --]

Sage Gerard schreef op ma 23-08-2021 om 18:24 [+0000]:
> Thank you for the links!
> 
> > I miss which problem Xiden is solving and how it does. 
> 
> You are not the first to say so, and I'm happy to learn more about why.
> 
> I'll try to explain in a different way here, so forgive the text wall.
> I'll incorporate any feedback here to improve the white paper. I'd also like
> to hear more about Guix's philosophy on what I'm going to talk about, because
> I had reasons to end up with a different model.

Maybe the publications at <https://guix.gnu.org/en/publications/> are informative.

> --
> 
> If we each use a program to distribute software, then those programs are probably
> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses this by
> decoupling the subjective elements of software distribution from the objective ones.
>   That means modeling software distribution as (partly) a semantics problem. Racket didn't
>  do this with its own package managers. Which is ironic, because it's hard to guarantee
>  that that an arbitrary Racket program will get the dependencies they mean to use.
>
>  I've written about this at length [1][2][3],

About polyglot: packages in Guix can specify which version of a package they use.
E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a specific
version of 'autoconf' (2.13) and 'rust' (1.41)

    (native-inputs
     `(("autoconf" ,autoconf-2.13)
       ("automake" ,automake)
       ("llvm" ,llvm)                   ;for llvm-objdump
       ("perl" ,perl)
       ("pkg-config" ,pkg-config)
       ("python" ,python-3)
       ("rust" ,rust-1.41)
       ("cargo" ,rust-1.41 "cargo")))

Is something missing from Guix here?  About [3]: this doesn't seem a problem in Guix,
if you're referring to the ability to define multiple versions of a package at the same
time (it's an ‘official policy’ to usually try to only keep the latest version of a
package though, but channels can ignore this completely).

>  and spoke about it last year [4]. Xiden has changed a bit since the speech to be more flexible, and it is no longer limited for use in Racket projects. Apologies if you see something that looks contradictory.
> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
> 
> Alice
> 
> trusts SHA-384, but only if its implementation uses Keccak.

SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
SHA-3: a version of Keccak.  By definition, implementations of SHA-384
have no reason to use Keccak.

> defines "sha384" as the canonical name of the CHF, and treat variants like "SHA-384" as aliases.
> delegates trust in artifact signatures to GPG subprocesses.

Guix allows changing the hash algorithm used, search for ‘content-hash’ in the guix manual.
Changing the hash algorithm globally unavoidably requires modifying every package definition
though, but that could be automated.

> downloads content from her employer's S3 buckets.

The ‘official’ substitute servers can be replaced:
"guix build --substitute-urls=http://s3.bucket.employer"
(after authorising that substitute server).

> unconditionally prefers cached installations, such that every defined artifact
> has exactly one stored copy on disk.

Coming from Guix, I don't know what this means.  Does this mean only one version
of a package can be in the store at the same time, and building a newer version
deletes the older version?  How does this compare with ‘content deduplication’?

> All dependents access the stored artifact
> using symbolic links or Windows junctions.


> prefers declaring versions with edition and revision names.
> Bob trusts SHA-1,

See ‘content-hash’ above.  Also, why should Xiden let Bob trust SHA-1 in the first place?
(See <https://en.wikipedia.org/wiki/SHA-1#SHAttered_%E2%80%93_first_public_collision>.)

>  but only for artifacts from services authenticated by his operating system's certificates

This is rather vague to me, unless you're referring to substitutes.
What does this mean for origin specifications?

    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256 ; imagine this was SHA-1 instead.
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))

Bob got the origin specification from Xiden (or something like a Guix channel
but for Xiden).  How would this ‘artifact’ authenticated by ‘operating system
certificates’?  I mean, when you let your operating system build that origin
(via guix (or Xiden?), which you could consider part of the OS), it will check
that hash, and it will be considered valid (‘authenticated?’) if the hash matches.
There are no certificates at play here.

I suppose the OS (guix or Xiden?) could then ‘sign’ it, but what value does that
provide to Bob?  (Unless Bob runs a substitute server on their computer.)

> .
> trusts digest creation and signature verification as implemented by his host's dynamically--linked `libcrypto` library for use in the same process.
> downloads content from GitHub packages

AFAIK there is no such thing a ‘GitHub packages’.  What is a ‘GitHub package’,
precisely?  When I hear ‘X package’, I think of "X = python, haskell, ...’ which
have a standardised build sytem (represented by python-build-system
and haskell-build-system in Guix), a standardised location to download them from
(pypi.org and hackage.haskell.org, represented by the 'pypi' and 'hackage' importers
an refreshers in Guix) and are separate languages.

The only thing things on GitHub have in common are:

 * they have git repositories on GitHub
 * some repositories on GitHub have ‘releases’, which can automatically be discovered

Notice dependency information and build system information are absent.
 
> unconditionally prefers SxS installations,

I searched for ‘SxS’, and found this: <https://en.wikipedia.org/wiki/SxS>.
I don't think you meant that.  What is ‘SxS’?

>  such that packages cannot conflict in a shared namespace 

‘packages cannot conflict in a shared namespace’ is a bit vague.
Are you referring to ‘profile collisions’?  About having multiple versions
of the same package on the same system?  Guix (and nix) can handle the latter.
When the ‘propagated-inputs’ are limited, installing multiple packages each using
a different version of a package is possible.

Is something missing from Guix here (please be very concrete)?

>  at the cost of defeating an internal cache.

I'm not sure what you mean here, can you illustrate this very concretely?
And is something missing in Guix here?

> prefers Semantic Versions

If upstream doesn't do ‘semver’, and instead uses a versioning system like
used by TeX (3.14, 3.141, 3.1415, 3.14159, ...), then Xiden cannot somehow
let the package use ‘semver’ instead.  Unless you want to teach Xiden to
convince upstream to change their versioning system?

> These preferences are valid in Xiden's DSL for writing launchers.

What is a launcher (compared to packages in Guix), and why would I want one?
Why shouldn't I just install packages and run them ("guix package -i texmacs",
start texmacs from some application chooser menu of the desktop environment).

>  The invariants for each launcher can differ as much as Cargo and Vortex differ.
>  What I wanted to do was allow users to declare their own invariants explicitly,
>  but only when those invariants are wanted for subjective reasons.

What are these invariants you're speaking of?  I don't think you're referring
to, say, translation invariance (‘the laws of physics remain the same under a
translation of the coordinate system’).

Or do you mean something like ‘Invariant (computer science), an expression whose
value doesn't change during program execution’ (<https://en.wikipedia.org/wiki/Invariant>)?
But what are the ‘expressions’ in this case?

>  I can't just write an overly-opinionated tool in this space because opinions
>  have shelf-lives, and they can't interrupt our ability to get the content we
>  need on demand, with our own safety guarentees.

Is Guix too opinionated about something in particular?

> Xiden launchers have advantages over shell scripts in that we can both still download
> and install dependencies from the same servers, and create reproducible builds in terms
> of the same (bit-identical) state on disk. We just differ on details that shouldn't impact
> how we work. It also helps us patch holes faster, since if (say) a catalog maintainer tells

What is a ‘catalog maintainer’ (maybe it's like someone with commit access to a ‘guix channel’?)
Also, what ‘shell scripts’ are you referring to?  I've been doing well without writing any shell
scripts.

> us that a CHF was compromised, we can immediately revoke trust in the CHF independently

Maybe guix could gain a command "guix style transition-hash OLD NEW" to automagically
rewrite package definitions to use the new hash algorithm instead of the old.  It will
need to download plenty of source code however, and it will entail a world-rebuild.

Note that, in practice, signs of the brokenness of hashes appear well in advance of
actual exploits.

> in our own clients.

What are these ‘clients’ you're speaking of?  I haven't been needing any ‘clients’ lately.

>  The package definitions are all expressed in terms of what the launchers allow.

If 'package definition' = (package definition as in Guix), then this is false,
Guix doesn't have a notion of ‘launchers’ and doesn't depend on ‘Xiden’.
What are you meaning, exactly?  Also, I haven't found any package definitions in Xiden,
could you point me to any?

> So what problem does this solve? I'm trying to preserve an element of walk-away power
> in any tech community. Maybe your community goes in a questionable direction. Maybe your
> desired software is in a different ecosystem.

Guix channels can function independently of the main ‘guix’, augmenting ‘guix’ with
additional packages, even if they contravene official Guix policies (e.g. non-free,
bad quality ...), or simply if putting it in ‘guix proper’ takes to much time.
There exist a few relatively well-known channels like that.

>  Maybe the maintainers are abusive jerks.

Technically, Guix has ‘maintainers’, but I wouldn't know where I could find the list.
Guix (the package manager) doesn't have its own infrastructure.
It does have official substitute servers, but you can ignore them.
So I don't see ‘maintainers are abusive jerks’ as a plausible threat to Guix.
If they become existent, they can easily be replaced.

> Maybe you have a `leftpad`/`event-stream` incident and the developer is unable/unwilling
> to patch the problem.

In Guix, if someone pushed malware (*) to, say, a package on pypi.org, then we (Guix)
would simply stop using new versions of that package from pypi.org.  If there's a healthy
fork somewhere, we could switch over to that for new versions.

(*) I could have confused the ‘`leftpad`/`event-stream` incident’ for another incident.

> You'd probably want a way to escape to a new community or distribution model
>  without

‘Escape to a new community’ --> only if the community as a whole are ‘abusive jerks’.
If it's merely some maintainers, they can be removed or replaced.  If it's all the
maintainers, I'm sure a few well-regarded members of the community could come together
for a ‘coup’.  If it's the community as a whole, I don't see how Xiden could help,
unless you mean I would move from Guix to Xiden? 

Why would I want to ‘escape to a new distribution model’?  Writing something like Guix
from scratch seems a lot of work to me, I would rather fork Guix (with a few
likewise-minded collaborators) than to start over again.  Or move to Debian maybe.
I don't see what ‘Xiden’ could offer here.

>  giving up content or doing complex integration work. If Xiden can do that, users
>  don't have to depend on the same middlemen to share work.

Is ‘middlemen’ = guix committers here?  Do you have concrete scenario to illustrate this,

>  This is also better for
>  developers because it doesn't oblige them to do everything their users want,

Are we talking about upstream developers here, or guix(/Xiden) committers?
How does this ‘obligation’ exist in the first place?

>  while knowing users will adapt to inevitable distribution problems.

I would rather not have to ‘adapt to problems’, I would rather that upstream has something
that works (though I can understand if occasionally mistakes are made).

>  Everyone's consent
>  becomes explicit, such that Xiden's "real" invariant is that consent between any
>  two parties be mutually compatible.

Consent about what?

> I hope to improve user freedom from this angle,
>  and my approach is biased by my experience with Racket's limitations in this space.

Why are you comparing it with ‘racket’ instead of, say, Debian or Guix?
‘racket’ is a Scheme implementation, not a package manager.

> The subjective parts I talk about go further into details like what name canons,
> versioning schemes, trusted certificate chains, and even specific approaches to an exact
> diamond dependency. The objective parts include integrity checking, signature verification,
> safety limits, automatic dependency resolution, patches, protocols, and other concerns that
> a decent package manager normally cares about. By keeping these separate, all of the below
> features become cheap to write, without sacrificing trust & safety in the abstract.
> 
> Download archived GitHub repositories (source on Racket Discord atm)

It can download GitHub repositories.  Why are you writing ‘archived’ here?
Guix supports git repositories (see 'git-reference' in the guix manual), so
I don't see what additional value Xiden brings here.  Can it also automagically build
the software?  Also, why ‘GitHub repositories’ specifically?  Why not, say,
the git repositories at <https://git.ngyro.com/>?

> Manage Racket installations like how `nvm` manages Node.js installations [5]

It would seem that your ‘racket-installation-manager’ [5] cannot resist a compromise
of https://downloads.racket-lang.org/releases/.  Also, I don't see how it would be
possibe to use ‘racket-installation-manager’ offline.  And how does it compile the
racket packages?  If we're talking about ‘trust’ and ‘escaping to a new community’,
maybe provide an option to override 'host-url-prefix'?  Currently, it's hardcoding
the ‘official’ racket community.

> Produce a self-hosted Xiden installations, with implicit trust for the running Xiden instance [6]
> Control which exact artifacts mnust be produced deterministically [7]

Can I reproduce the ‘artifact’ (= store item?) on a separate machine?

> Force generated bindings to be `eq?` (This one haunts Racket's package managers today) [8]

That seems like a Scheme thing to me, not a package manager thing.

> Source package definitions from Pastebin to download content, digests,
> and signatures from a public S3 bucket (This does not exist today. I'm spitballing because I know
>  it would work)

How can this be secure?

> I hypothesize that Xiden can "abstract over" package managers in the same way that Racket can
> abstract over languages. I think people are used to package managers deciding things for them. 

Please provide some sane defaults.  I would rather not have to decide everything myself,
and be able to trust Guix(/Xiden?) to do something reasonable.

Also, if you ‘abstract over package managers’, wouldn't you just end up with one ‘super
package manager’?  Why not ‘manage’ everything with a single package manager,
like e.g. guix?

> So what is Xiden for? Well, it's not for deciding an SOP for you.

Searching for SOP, I find <https://en.wikipedia.org/wiki/Same-origin_policy>.
What were you referring to here?

>  It's for giving everyone the option to change the SOP for themselves with minimal politics.

What are these changes you're referring to?

> > what you would like to bootstrap?
> 
> All binaries related to creating a GNU/Linux distribution, such that I can reproduce an exact
> OS, Racket installation, and Xiden instance. I want a trusted initial state on GNU/Linux.
> 
> To be clear, I don't need to do this for functional reasons. Xiden's already operational. I
> just can't claim that you can trust a given Xiden instance with the same confidence as a
>  Guix instance right now.

If you want to get package definitions from guix, take a look at (guix inferior).
Maybe you could port the relevant code of (guix inferior) to Racket, to let Xiden
talk to Guix?   In particular, 'inferior-eval' sends an S-exp to the inferior Guix,
which is evaluated there, and the result is sent back.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]