From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id oDACHrwpJWEuGAEAgWs5BA (envelope-from ) for ; Tue, 24 Aug 2021 19:17:48 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id AFKAGbwpJWHlMgAAbx9fmQ (envelope-from ) for ; Tue, 24 Aug 2021 17:17:48 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DF424B51F for ; Tue, 24 Aug 2021 19:17:47 +0200 (CEST) Received: from localhost ([::1]:52880 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIa3W-0008SQ-UT for larch@yhetil.org; Tue, 24 Aug 2021 13:17:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIa3E-0008SI-Ge for guix-devel@gnu.org; Tue, 24 Aug 2021 13:17:28 -0400 Received: from mail1.protonmail.ch ([185.70.40.18]:12438) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIa39-0005cq-1T for guix-devel@gnu.org; Tue, 24 Aug 2021 13:17:25 -0400 Date: Tue, 24 Aug 2021 17:17:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sagegerard.com; s=protonmail; t=1629825436; bh=9ANYeavMQ5YEmlfZdMMA1neph/R82vZIn2/r5ZcoId8=; h=Date:To:From:Reply-To:Subject:In-Reply-To:References:From; b=Yu9MeYKfGOPnzDOBhZxLr2sMBBuBCwexEJgsUfSPfAIHNr/AlpTfjFC4qogr+BGrE qvVdGcGyJxA5MH2AzzP3JNFqwF2RiBQ7rssvDdTaoJ/jMH3KA7Oob1OEswmxzTO8mo V2StgYfkyBLVDYeiqkbTk4MPF+tjx4VuCE+UyTZU= To: Maxime Devos , guix-devel@gnu.org From: Sage Gerard Subject: Re: How did you handle making a GNU/Linux distribution? Message-ID: In-Reply-To: References: <73f02b09-6983-7535-b71d-69ff0b0124f4@sagegerard.com> <67e15919-c943-d805-218e-2fec6060984a@sagegerard.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=185.70.40.18; envelope-from=sage@sagegerard.com; helo=mail1.protonmail.ch X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Sage Gerard Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1629825468; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=9ANYeavMQ5YEmlfZdMMA1neph/R82vZIn2/r5ZcoId8=; b=Ll+t/xin3seGweyyJ3ONVwl1v0ucZ6FK1ANVe7rskJ7YZfGw2CYA0VoEkAXh1FAFZdTmA/ 1ER6cps9sYxoxoYgvPZA2/ldqeNPHZOo9phF9ezfpExFwIGLdfHz7PeQN3503IKaYCShjQ cCgctz0nEKriNT3fHJfWHgAqYuZuJp+8HvA2gDTY7ujJLFWTwivyN8oJEZLLg4iSneR209 oULRIKKx0VZy9hkAP8EbaaxDagIDvkciF3o9yFtt130f36udKI+7ALUC2Bre7/M4SDtmwk a15K+6TqY+QOXw7IJu/7UBduXpjsifIMiQyvBVIW54eeZky16o1PMfqKzcD01A== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1629825468; a=rsa-sha256; cv=none; b=DildZZPo7MFdip1+WysHlOM1sYVgR0GG0Dv/sXfg4DELakKYl9jTEpNxyJ7IA1MwHzI25y HxNcBBQdSTUQVrrsdtNtiAcDYnNsX2qppSejvM+1YgyXE1/dcW1rGarBM6e417trGS+kS2 f9OckkcLCoLVeboLXBvNYQ1zZCF85kQkt47nmqZRRNFqiT3rWAwrsms/E2hJG31Z0ICJBO qmNkRMHMfMj0fBKiFOV0rz5kksN1tsK7iFID8qCRnW+TcyKTxpwh3OZ+4l8h05hOMEX6zY x/rN/yfhOi9oHScRUH1dFDSn3RvBobMygJj61PZgg95eq85IagbPELyJ19NTGQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=sagegerard.com header.s=protonmail header.b=Yu9MeYKf; dmarc=pass (policy=none) header.from=sagegerard.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -2.12 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=sagegerard.com header.s=protonmail header.b=Yu9MeYKf; dmarc=pass (policy=none) header.from=sagegerard.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: DF424B51F X-Spam-Score: -2.12 X-Migadu-Scanner: scn0.migadu.com X-TUID: AyPWpksEfIKB I'll answer your questions in a different thread to limit the noise in this one. Look for the "Xiden/Guix design comparison" subject line within the next 24h. On 8/24/21 12:42 PM, Maxime Devos wrote: > Sage Gerard schreef op ma 23-08-2021 om 18:24 [+0000]: >> Thank you for the links! >> >>> I miss which problem Xiden is solving and how it does. >> You are not the first to say so, and I'm happy to learn more about why. >> >> I'll try to explain in a different way here, so forgive the text wall. >> I'll incorporate any feedback here to improve the white paper. I'd also = like >> to hear more about Guix's philosophy on what I'm going to talk about, be= cause >> I had reasons to end up with a different model. > Maybe the publications at are inf= ormative. > >> -- >> >> If we each use a program to distribute software, then those programs are= probably >> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses= this by >> decoupling the subjective elements of software distribution from the obj= ective ones. >> That means modeling software distribution as (partly) a semantics pro= blem. Racket didn't >> do this with its own package managers. Which is ironic, because it's h= ard to guarantee >> that that an arbitrary Racket program will get the dependencies they m= ean to use. >> >> I've written about this at length [1][2][3], > About polyglot: packages in Guix can specify which version of a package t= hey use. > E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a s= pecific > version of 'autoconf' (2.13) and 'rust' (1.41) > > (native-inputs > `(("autoconf" ,autoconf-2.13) > ("automake" ,automake) > ("llvm" ,llvm) ;for llvm-objdump > ("perl" ,perl) > ("pkg-config" ,pkg-config) > ("python" ,python-3) > ("rust" ,rust-1.41) > ("cargo" ,rust-1.41 "cargo"))) > > Is something missing from Guix here? About [3]: this doesn't seem a prob= lem in Guix, > if you're referring to the ability to define multiple versions of a packa= ge at the same > time (it's an =E2=80=98official policy=E2=80=99 to usually try to only ke= ep the latest version of a > package though, but channels can ignore this completely). > >> and spoke about it last year [4]. Xiden has changed a bit since the sp= eech to be more flexible, and it is no longer limited for use in Racket pro= jects. Apologies if you see something that looks contradictory. >> What does that mean? Let's say Alice and Bob each write a Xiden launcher= . >> >> Alice >> >> trusts SHA-384, but only if its implementation uses Keccak. > SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2, > SHA-3: a version of Keccak. By definition, implementations of SHA-384 > have no reason to use Keccak. > >> defines "sha384" as the canonical name of the CHF, and treat variants li= ke "SHA-384" as aliases. >> delegates trust in artifact signatures to GPG subprocesses. > Guix allows changing the hash algorithm used, search for =E2=80=98content= -hash=E2=80=99 in the guix manual. > Changing the hash algorithm globally unavoidably requires modifying every= package definition > though, but that could be automated. > >> downloads content from her employer's S3 buckets. > The =E2=80=98official=E2=80=99 substitute servers can be replaced: > "guix build --substitute-urls=3Dhttp://s3.bucket.employer" > (after authorising that substitute server). > >> unconditionally prefers cached installations, such that every defined ar= tifact >> has exactly one stored copy on disk. > Coming from Guix, I don't know what this means. Does this mean only one = version > of a package can be in the store at the same time, and building a newer v= ersion > deletes the older version? How does this compare with =E2=80=98content d= eduplication=E2=80=99? > >> All dependents access the stored artifact >> using symbolic links or Windows junctions. > >> prefers declaring versions with edition and revision names. >> Bob trusts SHA-1, > See =E2=80=98content-hash=E2=80=99 above. Also, why should Xiden let Bob= trust SHA-1 in the first place? > (See .) > >> but only for artifacts from services authenticated by his operating sy= stem's certificates > This is rather vague to me, unless you're referring to substitutes. > What does this mean for origin specifications? > > (source (origin > (method url-fetch) > (uri (string-append "mirror://gnu/hello/hello-" version > ".tar.gz")) > (sha256 ; imagine this was SHA-1 instead. > (base32 > "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))= )) > > Bob got the origin specification from Xiden (or something like a Guix cha= nnel > but for Xiden). How would this =E2=80=98artifact=E2=80=99 authenticated = by =E2=80=98operating system > certificates=E2=80=99? I mean, when you let your operating system build = that origin > (via guix (or Xiden?), which you could consider part of the OS), it will = check > that hash, and it will be considered valid (=E2=80=98authenticated?= =E2=80=99) if the hash matches. > There are no certificates at play here. > > I suppose the OS (guix or Xiden?) could then =E2=80=98sign=E2=80=99 it, b= ut what value does that > provide to Bob? (Unless Bob runs a substitute server on their computer.) > >> . >> trusts digest creation and signature verification as implemented by his = host's dynamically--linked `libcrypto` library for use in the same process. >> downloads content from GitHub packages > AFAIK there is no such thing a =E2=80=98GitHub packages=E2=80=99. What i= s a =E2=80=98GitHub package=E2=80=99, > precisely? When I hear =E2=80=98X package=E2=80=99, I think of "X =3D py= thon, haskell, ...=E2=80=99 which > have a standardised build sytem (represented by python-build-system > and haskell-build-system in Guix), a standardised location to download th= em from > (pypi.org and hackage.haskell.org, represented by the 'pypi' and 'hackage= ' importers > an refreshers in Guix) and are separate languages. > > The only thing things on GitHub have in common are: > > * they have git repositories on GitHub > * some repositories on GitHub have =E2=80=98releases=E2=80=99, which ca= n automatically be discovered > > Notice dependency information and build system information are absent. > >> unconditionally prefers SxS installations, > I searched for =E2=80=98SxS=E2=80=99, and found this: . > I don't think you meant that. What is =E2=80=98SxS=E2=80=99? > >> such that packages cannot conflict in a shared namespace > =E2=80=98packages cannot conflict in a shared namespace=E2=80=99 is a bit= vague. > Are you referring to =E2=80=98profile collisions=E2=80=99? About having = multiple versions > of the same package on the same system? Guix (and nix) can handle the la= tter. > When the =E2=80=98propagated-inputs=E2=80=99 are limited, installing mult= iple packages each using > a different version of a package is possible. > > Is something missing from Guix here (please be very concrete)? > >> at the cost of defeating an internal cache. > I'm not sure what you mean here, can you illustrate this very concretely? > And is something missing in Guix here? > >> prefers Semantic Versions > If upstream doesn't do =E2=80=98semver=E2=80=99, and instead uses a versi= oning system like > used by TeX (3.14, 3.141, 3.1415, 3.14159, ...), then Xiden cannot someho= w > let the package use =E2=80=98semver=E2=80=99 instead. Unless you want to= teach Xiden to > convince upstream to change their versioning system? > >> These preferences are valid in Xiden's DSL for writing launchers. > What is a launcher (compared to packages in Guix), and why would I want o= ne? > Why shouldn't I just install packages and run them ("guix package -i texm= acs", > start texmacs from some application chooser menu of the desktop environme= nt). > >> The invariants for each launcher can differ as much as Cargo and Vorte= x differ. >> What I wanted to do was allow users to declare their own invariants ex= plicitly, >> but only when those invariants are wanted for subjective reasons. > What are these invariants you're speaking of? I don't think you're refer= ring > to, say, translation invariance (=E2=80=98the laws of physics remain the = same under a > translation of the coordinate system=E2=80=99). > > Or do you mean something like =E2=80=98Invariant (computer science), an e= xpression whose > value doesn't change during program execution=E2=80=99 ()? > But what are the =E2=80=98expressions=E2=80=99 in this case? > >> I can't just write an overly-opinionated tool in this space because op= inions >> have shelf-lives, and they can't interrupt our ability to get the cont= ent we >> need on demand, with our own safety guarentees. > Is Guix too opinionated about something in particular? > >> Xiden launchers have advantages over shell scripts in that we can both s= till download >> and install dependencies from the same servers, and create reproducible = builds in terms >> of the same (bit-identical) state on disk. We just differ on details tha= t shouldn't impact >> how we work. It also helps us patch holes faster, since if (say) a catal= og maintainer tells > What is a =E2=80=98catalog maintainer=E2=80=99 (maybe it's like someone w= ith commit access to a =E2=80=98guix channel=E2=80=99?) > Also, what =E2=80=98shell scripts=E2=80=99 are you referring to? I've be= en doing well without writing any shell > scripts. > >> us that a CHF was compromised, we can immediately revoke trust in the CH= F independently > Maybe guix could gain a command "guix style transition-hash OLD NEW" to a= utomagically > rewrite package definitions to use the new hash algorithm instead of the = old. It will > need to download plenty of source code however, and it will entail a worl= d-rebuild. > > Note that, in practice, signs of the brokenness of hashes appear well in = advance of > actual exploits. > >> in our own clients. > What are these =E2=80=98clients=E2=80=99 you're speaking of? I haven't b= een needing any =E2=80=98clients=E2=80=99 lately. > >> The package definitions are all expressed in terms of what the launche= rs allow. > If 'package definition' =3D (package definition as in Guix), then this is= false, > Guix doesn't have a notion of =E2=80=98launchers=E2=80=99 and doesn't dep= end on =E2=80=98Xiden=E2=80=99. > What are you meaning, exactly? Also, I haven't found any package definit= ions in Xiden, > could you point me to any? > >> So what problem does this solve? I'm trying to preserve an element of wa= lk-away power >> in any tech community. Maybe your community goes in a questionable direc= tion. Maybe your >> desired software is in a different ecosystem. > Guix channels can function independently of the main =E2=80=98guix= =E2=80=99, augmenting =E2=80=98guix=E2=80=99 with > additional packages, even if they contravene official Guix policies (e.g.= non-free, > bad quality ...), or simply if putting it in =E2=80=98guix proper= =E2=80=99 takes to much time. > There exist a few relatively well-known channels like that. > >> Maybe the maintainers are abusive jerks. > Technically, Guix has =E2=80=98maintainers=E2=80=99, but I wouldn't know = where I could find the list. > Guix (the package manager) doesn't have its own infrastructure. > It does have official substitute servers, but you can ignore them. > So I don't see =E2=80=98maintainers are abusive jerks=E2=80=99 as a plaus= ible threat to Guix. > If they become existent, they can easily be replaced. > >> Maybe you have a `leftpad`/`event-stream` incident and the developer is = unable/unwilling >> to patch the problem. > In Guix, if someone pushed malware (*) to, say, a package on pypi.org, th= en we (Guix) > would simply stop using new versions of that package from pypi.org. If t= here's a healthy > fork somewhere, we could switch over to that for new versions. > > (*) I could have confused the =E2=80=98`leftpad`/`event-stream` incident= =E2=80=99 for another incident. > >> You'd probably want a way to escape to a new community or distribution m= odel >> without > =E2=80=98Escape to a new community=E2=80=99 --> only if the community as = a whole are =E2=80=98abusive jerks=E2=80=99. > If it's merely some maintainers, they can be removed or replaced. If it'= s all the > maintainers, I'm sure a few well-regarded members of the community could = come together > for a =E2=80=98coup=E2=80=99. If it's the community as a whole, I don't = see how Xiden could help, > unless you mean I would move from Guix to Xiden? > > Why would I want to =E2=80=98escape to a new distribution model=E2=80= =99? Writing something like Guix > from scratch seems a lot of work to me, I would rather fork Guix (with a = few > likewise-minded collaborators) than to start over again. Or move to Debi= an maybe. > I don't see what =E2=80=98Xiden=E2=80=99 could offer here. > >> giving up content or doing complex integration work. If Xiden can do t= hat, users >> don't have to depend on the same middlemen to share work. > Is =E2=80=98middlemen=E2=80=99 =3D guix committers here? Do you have con= crete scenario to illustrate this, > >> This is also better for >> developers because it doesn't oblige them to do everything their users= want, > Are we talking about upstream developers here, or guix(/Xiden) committers= ? > How does this =E2=80=98obligation=E2=80=99 exist in the first place? > >> while knowing users will adapt to inevitable distribution problems. > I would rather not have to =E2=80=98adapt to problems=E2=80=99, I would r= ather that upstream has something > that works (though I can understand if occasionally mistakes are made). > >> Everyone's consent >> becomes explicit, such that Xiden's "real" invariant is that consent b= etween any >> two parties be mutually compatible. > Consent about what? > >> I hope to improve user freedom from this angle, >> and my approach is biased by my experience with Racket's limitations i= n this space. > Why are you comparing it with =E2=80=98racket=E2=80=99 instead of, say, D= ebian or Guix? > =E2=80=98racket=E2=80=99 is a Scheme implementation, not a package manage= r. > >> The subjective parts I talk about go further into details like what name= canons, >> versioning schemes, trusted certificate chains, and even specific approa= ches to an exact >> diamond dependency. The objective parts include integrity checking, sign= ature verification, >> safety limits, automatic dependency resolution, patches, protocols, and = other concerns that >> a decent package manager normally cares about. By keeping these separate= , all of the below >> features become cheap to write, without sacrificing trust & safety in th= e abstract. >> >> Download archived GitHub repositories (source on Racket Discord atm) > It can download GitHub repositories. Why are you writing =E2=80=98archiv= ed=E2=80=99 here? > Guix supports git repositories (see 'git-reference' in the guix manual), = so > I don't see what additional value Xiden brings here. Can it also automag= ically build > the software? Also, why =E2=80=98GitHub repositories=E2=80=99 specifical= ly? Why not, say, > the git repositories at ? > >> Manage Racket installations like how `nvm` manages Node.js installations= [5] > It would seem that your =E2=80=98racket-installation-manager=E2=80=99 [5]= cannot resist a compromise > of https://downloads.racket-lang.org/releases/. Also, I don't see how it= would be > possibe to use =E2=80=98racket-installation-manager=E2=80=99 offline. An= d how does it compile the > racket packages? If we're talking about =E2=80=98trust=E2=80=99 and = =E2=80=98escaping to a new community=E2=80=99, > maybe provide an option to override 'host-url-prefix'? Currently, it's h= ardcoding > the =E2=80=98official=E2=80=99 racket community. > >> Produce a self-hosted Xiden installations, with implicit trust for the r= unning Xiden instance [6] >> Control which exact artifacts mnust be produced deterministically [7] > Can I reproduce the =E2=80=98artifact=E2=80=99 (=3D store item?) on a sep= arate machine? > >> Force generated bindings to be `eq?` (This one haunts Racket's package m= anagers today) [8] > That seems like a Scheme thing to me, not a package manager thing. > >> Source package definitions from Pastebin to download content, digests, >> and signatures from a public S3 bucket (This does not exist today. I'm s= pitballing because I know >> it would work) > How can this be secure? > >> I hypothesize that Xiden can "abstract over" package managers in the sam= e way that Racket can >> abstract over languages. I think people are used to package managers dec= iding things for them. > Please provide some sane defaults. I would rather not have to decide eve= rything myself, > and be able to trust Guix(/Xiden?) to do something reasonable. > > Also, if you =E2=80=98abstract over package managers=E2=80=99, wouldn't y= ou just end up with one =E2=80=98super > package manager=E2=80=99? Why not =E2=80=98manage=E2=80=99 everything wi= th a single package manager, > like e.g. guix? > >> So what is Xiden for? Well, it's not for deciding an SOP for you. > Searching for SOP, I find . > What were you referring to here? > >> It's for giving everyone the option to change the SOP for themselves w= ith minimal politics. > What are these changes you're referring to? > >>> what you would like to bootstrap? >> All binaries related to creating a GNU/Linux distribution, such that I c= an reproduce an exact >> OS, Racket installation, and Xiden instance. I want a trusted initial st= ate on GNU/Linux. >> >> To be clear, I don't need to do this for functional reasons. Xiden's alr= eady operational. I >> just can't claim that you can trust a given Xiden instance with the same= confidence as a >> Guix instance right now. > If you want to get package definitions from guix, take a look at (guix in= ferior). > Maybe you could port the relevant code of (guix inferior) to Racket, to l= et Xiden > talk to Guix? In particular, 'inferior-eval' sends an S-exp to the infe= rior Guix, > which is evaluated there, and the result is sent back. > > Greetings, > Maxime.