Re: How did you handle making a GNU/Linux distribution?

[Sage Gerard <sage@sagegerard.com>]

I'll answer your questions in a different thread to limit the noise in
this one. Look for the "Xiden/Guix design comparison" subject line
within the next 24h.

On 8/24/21 12:42 PM, Maxime Devos wrote:
> Sage Gerard schreef op ma 23-08-2021 om 18:24 [+0000]:
>> Thank you for the links!
>>
>>> I miss which problem Xiden is solving and how it does.
>> You are not the first to say so, and I'm happy to learn more about why.
>>
>> I'll try to explain in a different way here, so forgive the text wall.
>> I'll incorporate any feedback here to improve the white paper. I'd also like
>> to hear more about Guix's philosophy on what I'm going to talk about, because
>> I had reasons to end up with a different model.
> Maybe the publications at <https://guix.gnu.org/en/publications/> are informative.
>
>> --
>>
>> If we each use a program to distribute software, then those programs are probably
>> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses this by
>> decoupling the subjective elements of software distribution from the objective ones.
>>    That means modeling software distribution as (partly) a semantics problem. Racket didn't
>>   do this with its own package managers. Which is ironic, because it's hard to guarantee
>>   that that an arbitrary Racket program will get the dependencies they mean to use.
>>
>>   I've written about this at length [1][2][3],
> About polyglot: packages in Guix can specify which version of a package they use.
> E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a specific
> version of 'autoconf' (2.13) and 'rust' (1.41)
>
>      (native-inputs
>       `(("autoconf" ,autoconf-2.13)
>         ("automake" ,automake)
>         ("llvm" ,llvm)                   ;for llvm-objdump
>         ("perl" ,perl)
>         ("pkg-config" ,pkg-config)
>         ("python" ,python-3)
>         ("rust" ,rust-1.41)
>         ("cargo" ,rust-1.41 "cargo")))
>
> Is something missing from Guix here?  About [3]: this doesn't seem a problem in Guix,
> if you're referring to the ability to define multiple versions of a package at the same
> time (it's an ‘official policy’ to usually try to only keep the latest version of a
> package though, but channels can ignore this completely).
>
>>   and spoke about it last year [4]. Xiden has changed a bit since the speech to be more flexible, and it is no longer limited for use in Racket projects. Apologies if you see something that looks contradictory.
>> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
>>
>> Alice
>>
>> trusts SHA-384, but only if its implementation uses Keccak.
> SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
> SHA-3: a version of Keccak.  By definition, implementations of SHA-384
> have no reason to use Keccak.
>
>> defines "sha384" as the canonical name of the CHF, and treat variants like "SHA-384" as aliases.
>> delegates trust in artifact signatures to GPG subprocesses.
> Guix allows changing the hash algorithm used, search for ‘content-hash’ in the guix manual.
> Changing the hash algorithm globally unavoidably requires modifying every package definition
> though, but that could be automated.
>
>> downloads content from her employer's S3 buckets.
> The ‘official’ substitute servers can be replaced:
> "guix build --substitute-urls=http://s3.bucket.employer"
> (after authorising that substitute server).
>
>> unconditionally prefers cached installations, such that every defined artifact
>> has exactly one stored copy on disk.
> Coming from Guix, I don't know what this means.  Does this mean only one version
> of a package can be in the store at the same time, and building a newer version
> deletes the older version?  How does this compare with ‘content deduplication’?
>
>> All dependents access the stored artifact
>> using symbolic links or Windows junctions.
>
>> prefers declaring versions with edition and revision names.
>> Bob trusts SHA-1,
> See ‘content-hash’ above.  Also, why should Xiden let Bob trust SHA-1 in the first place?
> (See <https://en.wikipedia.org/wiki/SHA-1#SHAttered_%E2%80%93_first_public_collision>.)
>
>>   but only for artifacts from services authenticated by his operating system's certificates
> This is rather vague to me, unless you're referring to substitutes.
> What does this mean for origin specifications?
>
>      (source (origin
>                (method url-fetch)
>                (uri (string-append "mirror://gnu/hello/hello-" version
>                                    ".tar.gz"))
>                (sha256 ; imagine this was SHA-1 instead.
>                 (base32
>                  "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
>
> Bob got the origin specification from Xiden (or something like a Guix channel
> but for Xiden).  How would this ‘artifact’ authenticated by ‘operating system
> certificates’?  I mean, when you let your operating system build that origin
> (via guix (or Xiden?), which you could consider part of the OS), it will check
> that hash, and it will be considered valid (‘authenticated?’) if the hash matches.
> There are no certificates at play here.
>
> I suppose the OS (guix or Xiden?) could then ‘sign’ it, but what value does that
> provide to Bob?  (Unless Bob runs a substitute server on their computer.)
>
>> .
>> trusts digest creation and signature verification as implemented by his host's dynamically--linked `libcrypto` library for use in the same process.
>> downloads content from GitHub packages
> AFAIK there is no such thing a ‘GitHub packages’.  What is a ‘GitHub package’,
> precisely?  When I hear ‘X package’, I think of "X = python, haskell, ...’ which
> have a standardised build sytem (represented by python-build-system
> and haskell-build-system in Guix), a standardised location to download them from
> (pypi.org and hackage.haskell.org, represented by the 'pypi' and 'hackage' importers
> an refreshers in Guix) and are separate languages.
>
> The only thing things on GitHub have in common are:
>
>   * they have git repositories on GitHub
>   * some repositories on GitHub have ‘releases’, which can automatically be discovered
>
> Notice dependency information and build system information are absent.
>
>> unconditionally prefers SxS installations,
> I searched for ‘SxS’, and found this: <https://en.wikipedia.org/wiki/SxS>.
> I don't think you meant that.  What is ‘SxS’?
>
>>   such that packages cannot conflict in a shared namespace
> ‘packages cannot conflict in a shared namespace’ is a bit vague.
> Are you referring to ‘profile collisions’?  About having multiple versions
> of the same package on the same system?  Guix (and nix) can handle the latter.
> When the ‘propagated-inputs’ are limited, installing multiple packages each using
> a different version of a package is possible.
>
> Is something missing from Guix here (please be very concrete)?
>
>>   at the cost of defeating an internal cache.
> I'm not sure what you mean here, can you illustrate this very concretely?
> And is something missing in Guix here?
>
>> prefers Semantic Versions
> If upstream doesn't do ‘semver’, and instead uses a versioning system like
> used by TeX (3.14, 3.141, 3.1415, 3.14159, ...), then Xiden cannot somehow
> let the package use ‘semver’ instead.  Unless you want to teach Xiden to
> convince upstream to change their versioning system?
>
>> These preferences are valid in Xiden's DSL for writing launchers.
> What is a launcher (compared to packages in Guix), and why would I want one?
> Why shouldn't I just install packages and run them ("guix package -i texmacs",
> start texmacs from some application chooser menu of the desktop environment).
>
>>   The invariants for each launcher can differ as much as Cargo and Vortex differ.
>>   What I wanted to do was allow users to declare their own invariants explicitly,
>>   but only when those invariants are wanted for subjective reasons.
> What are these invariants you're speaking of?  I don't think you're referring
> to, say, translation invariance (‘the laws of physics remain the same under a
> translation of the coordinate system’).
>
> Or do you mean something like ‘Invariant (computer science), an expression whose
> value doesn't change during program execution’ (<https://en.wikipedia.org/wiki/Invariant>)?
> But what are the ‘expressions’ in this case?
>
>>   I can't just write an overly-opinionated tool in this space because opinions
>>   have shelf-lives, and they can't interrupt our ability to get the content we
>>   need on demand, with our own safety guarentees.
> Is Guix too opinionated about something in particular?
>
>> Xiden launchers have advantages over shell scripts in that we can both still download
>> and install dependencies from the same servers, and create reproducible builds in terms
>> of the same (bit-identical) state on disk. We just differ on details that shouldn't impact
>> how we work. It also helps us patch holes faster, since if (say) a catalog maintainer tells
> What is a ‘catalog maintainer’ (maybe it's like someone with commit access to a ‘guix channel’?)
> Also, what ‘shell scripts’ are you referring to?  I've been doing well without writing any shell
> scripts.
>
>> us that a CHF was compromised, we can immediately revoke trust in the CHF independently
> Maybe guix could gain a command "guix style transition-hash OLD NEW" to automagically
> rewrite package definitions to use the new hash algorithm instead of the old.  It will
> need to download plenty of source code however, and it will entail a world-rebuild.
>
> Note that, in practice, signs of the brokenness of hashes appear well in advance of
> actual exploits.
>
>> in our own clients.
> What are these ‘clients’ you're speaking of?  I haven't been needing any ‘clients’ lately.
>
>>   The package definitions are all expressed in terms of what the launchers allow.
> If 'package definition' = (package definition as in Guix), then this is false,
> Guix doesn't have a notion of ‘launchers’ and doesn't depend on ‘Xiden’.
> What are you meaning, exactly?  Also, I haven't found any package definitions in Xiden,
> could you point me to any?
>
>> So what problem does this solve? I'm trying to preserve an element of walk-away power
>> in any tech community. Maybe your community goes in a questionable direction. Maybe your
>> desired software is in a different ecosystem.
> Guix channels can function independently of the main ‘guix’, augmenting ‘guix’ with
> additional packages, even if they contravene official Guix policies (e.g. non-free,
> bad quality ...), or simply if putting it in ‘guix proper’ takes to much time.
> There exist a few relatively well-known channels like that.
>
>>   Maybe the maintainers are abusive jerks.
> Technically, Guix has ‘maintainers’, but I wouldn't know where I could find the list.
> Guix (the package manager) doesn't have its own infrastructure.
> It does have official substitute servers, but you can ignore them.
> So I don't see ‘maintainers are abusive jerks’ as a plausible threat to Guix.
> If they become existent, they can easily be replaced.
>
>> Maybe you have a `leftpad`/`event-stream` incident and the developer is unable/unwilling
>> to patch the problem.
> In Guix, if someone pushed malware (*) to, say, a package on pypi.org, then we (Guix)
> would simply stop using new versions of that package from pypi.org.  If there's a healthy
> fork somewhere, we could switch over to that for new versions.
>
> (*) I could have confused the ‘`leftpad`/`event-stream` incident’ for another incident.
>
>> You'd probably want a way to escape to a new community or distribution model
>>   without
> ‘Escape to a new community’ --> only if the community as a whole are ‘abusive jerks’.
> If it's merely some maintainers, they can be removed or replaced.  If it's all the
> maintainers, I'm sure a few well-regarded members of the community could come together
> for a ‘coup’.  If it's the community as a whole, I don't see how Xiden could help,
> unless you mean I would move from Guix to Xiden?
>
> Why would I want to ‘escape to a new distribution model’?  Writing something like Guix
> from scratch seems a lot of work to me, I would rather fork Guix (with a few
> likewise-minded collaborators) than to start over again.  Or move to Debian maybe.
> I don't see what ‘Xiden’ could offer here.
>
>>   giving up content or doing complex integration work. If Xiden can do that, users
>>   don't have to depend on the same middlemen to share work.
> Is ‘middlemen’ = guix committers here?  Do you have concrete scenario to illustrate this,
>
>>   This is also better for
>>   developers because it doesn't oblige them to do everything their users want,
> Are we talking about upstream developers here, or guix(/Xiden) committers?
> How does this ‘obligation’ exist in the first place?
>
>>   while knowing users will adapt to inevitable distribution problems.
> I would rather not have to ‘adapt to problems’, I would rather that upstream has something
> that works (though I can understand if occasionally mistakes are made).
>
>>   Everyone's consent
>>   becomes explicit, such that Xiden's "real" invariant is that consent between any
>>   two parties be mutually compatible.
> Consent about what?
>
>> I hope to improve user freedom from this angle,
>>   and my approach is biased by my experience with Racket's limitations in this space.
> Why are you comparing it with ‘racket’ instead of, say, Debian or Guix?
> ‘racket’ is a Scheme implementation, not a package manager.
>
>> The subjective parts I talk about go further into details like what name canons,
>> versioning schemes, trusted certificate chains, and even specific approaches to an exact
>> diamond dependency. The objective parts include integrity checking, signature verification,
>> safety limits, automatic dependency resolution, patches, protocols, and other concerns that
>> a decent package manager normally cares about. By keeping these separate, all of the below
>> features become cheap to write, without sacrificing trust & safety in the abstract.
>>
>> Download archived GitHub repositories (source on Racket Discord atm)
> It can download GitHub repositories.  Why are you writing ‘archived’ here?
> Guix supports git repositories (see 'git-reference' in the guix manual), so
> I don't see what additional value Xiden brings here.  Can it also automagically build
> the software?  Also, why ‘GitHub repositories’ specifically?  Why not, say,
> the git repositories at <https://git.ngyro.com/>?
>
>> Manage Racket installations like how `nvm` manages Node.js installations [5]
> It would seem that your ‘racket-installation-manager’ [5] cannot resist a compromise
> of https://downloads.racket-lang.org/releases/.  Also, I don't see how it would be
> possibe to use ‘racket-installation-manager’ offline.  And how does it compile the
> racket packages?  If we're talking about ‘trust’ and ‘escaping to a new community’,
> maybe provide an option to override 'host-url-prefix'?  Currently, it's hardcoding
> the ‘official’ racket community.
>
>> Produce a self-hosted Xiden installations, with implicit trust for the running Xiden instance [6]
>> Control which exact artifacts mnust be produced deterministically [7]
> Can I reproduce the ‘artifact’ (= store item?) on a separate machine?
>
>> Force generated bindings to be `eq?` (This one haunts Racket's package managers today) [8]
> That seems like a Scheme thing to me, not a package manager thing.
>
>> Source package definitions from Pastebin to download content, digests,
>> and signatures from a public S3 bucket (This does not exist today. I'm spitballing because I know
>>   it would work)
> How can this be secure?
>
>> I hypothesize that Xiden can "abstract over" package managers in the same way that Racket can
>> abstract over languages. I think people are used to package managers deciding things for them.
> Please provide some sane defaults.  I would rather not have to decide everything myself,
> and be able to trust Guix(/Xiden?) to do something reasonable.
>
> Also, if you ‘abstract over package managers’, wouldn't you just end up with one ‘super
> package manager’?  Why not ‘manage’ everything with a single package manager,
> like e.g. guix?
>
>> So what is Xiden for? Well, it's not for deciding an SOP for you.
> Searching for SOP, I find <https://en.wikipedia.org/wiki/Same-origin_policy>.
> What were you referring to here?
>
>>   It's for giving everyone the option to change the SOP for themselves with minimal politics.
> What are these changes you're referring to?
>
>>> what you would like to bootstrap?
>> All binaries related to creating a GNU/Linux distribution, such that I can reproduce an exact
>> OS, Racket installation, and Xiden instance. I want a trusted initial state on GNU/Linux.
>>
>> To be clear, I don't need to do this for functional reasons. Xiden's already operational. I
>> just can't claim that you can trust a given Xiden instance with the same confidence as a
>>   Guix instance right now.
> If you want to get package definitions from guix, take a look at (guix inferior).
> Maybe you could port the relevant code of (guix inferior) to Racket, to let Xiden
> talk to Guix?   In particular, 'inferior-eval' sends an S-exp to the inferior Guix,
> which is evaluated there, and the result is sent back.
>
> Greetings,
> Maxime.