unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* How did you handle making a GNU/Linux distribution?
@ 2021-08-21 16:43 Sage Gerard
  2021-08-21 21:18 ` Leo Famulari
                   ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Sage Gerard @ 2021-08-21 16:43 UTC (permalink / raw)
  To: guix-devel

Hi folks,

My name is Sage. I wrote a cross-platform Guix-like package manager
called Xiden. It applies functional package management to the Racket
ecosystem. It is also free software under the GPLv3. The source is
available at zyrolasting/xiden on Github, pending migration to a new host.

I'm at the point where users are requesting a GNU/Linux distribution for
Xiden, such that Racket is the primary language for day-to-day
operation. I'm ignorant of the scope of work, and am unsure if I can do
it alone.  My understanding is that you had to bootstrap your entire
toolchain and address Ken Thompson's compiler hack from a different
angle. Is that right?

I'd greatly appreciate learning how you all built Guix's GNU/Linux
distribution so that I can prepare a realistic roadmap and recruit help
where necessary. I'm bad at both of those things, but if there is any
opportunity to collaborate on implementation details based on Xiden's
progress, I am happy to give back.

Thank you for Guix, and thank you for any replies.

~slg





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-21 16:43 How did you handle making a GNU/Linux distribution? Sage Gerard
@ 2021-08-21 21:18 ` Leo Famulari
  2021-08-22 21:53   ` Sage Gerard
  2021-08-23 10:43 ` Maxime Devos
  2021-08-23 15:38 ` zimoun
  2 siblings, 1 reply; 17+ messages in thread
From: Leo Famulari @ 2021-08-21 21:18 UTC (permalink / raw)
  To: Sage Gerard; +Cc: guix-devel

On Sat, Aug 21, 2021 at 04:43:34PM +0000, Sage Gerard wrote:
> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. It is also free software under the GPLv3. The source is
> available at zyrolasting/xiden on Github, pending migration to a new host.

Hi! Cool!

> I'm at the point where users are requesting a GNU/Linux distribution for
> Xiden, such that Racket is the primary language for day-to-day
> operation. I'm ignorant of the scope of work, and am unsure if I can do
> it alone.  My understanding is that you had to bootstrap your entire
> toolchain and address Ken Thompson's compiler hack from a different
> angle. Is that right?

At the time that Guix was founded, it was not considered possible to
bootstrap a GNU/Linux system from source; on which OS would you perform
the first compilation? How would you bootstrap that OS? Ad infinitum.
Stage0 and mes and friends are fixing that now. [0]

So, some binary "seeds" were created, which are compiled binaries of
core tools such as GCC and binutils, and the system was based on those
seeds. And we gave instructions for reproducing these seeds; you could
use any commonly available distro as a host. Maybe that sounds icky in
2021, since the situation has improved radically, but it's how every OS
was created since decades ago [1], except that most operating systems (and
even language ecosystems) did not keep track of their seeds or give
instructions for reproducing them. So Guix was better than all the rest
from its beginning :)

https://guix.gnu.org/manual/en/html_node/Bootstrapping.html

I was not involved at that stage of Guix. I'm sure others can give more
detailed answers.

> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. I'm bad at both of those things, but if there is any
> opportunity to collaborate on implementation details based on Xiden's
> progress, I am happy to give back.

[0] https://bootstrappable.org/
https://bootstrappable.org/projects/mes.html
https://www.gnu.org/software/mes/
Also visit the #bootstrappable channel on Libera.chat

[1] It would be interesting to know the last truly bootstrapped OS


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-21 21:18 ` Leo Famulari
@ 2021-08-22 21:53   ` Sage Gerard
  2021-08-22 22:54     ` Philip McGrath
  0 siblings, 1 reply; 17+ messages in thread
From: Sage Gerard @ 2021-08-22 21:53 UTC (permalink / raw)
  To: guix-devel

Thanks for the detailed answer!

It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead of
Guile to bring GNU's benefits to more Scheme and Racket programmers. Has
someone already tried something like that?

On 8/21/21 5:18 PM, Leo Famulari wrote:
> On Sat, Aug 21, 2021 at 04:43:34PM +0000, Sage Gerard wrote:
>> My name is Sage. I wrote a cross-platform Guix-like package manager
>> called Xiden. It applies functional package management to the Racket
>> ecosystem. It is also free software under the GPLv3. The source is
>> available at zyrolasting/xiden on Github, pending migration to a new host.
> Hi! Cool!
>
>> I'm at the point where users are requesting a GNU/Linux distribution for
>> Xiden, such that Racket is the primary language for day-to-day
>> operation. I'm ignorant of the scope of work, and am unsure if I can do
>> it alone.  My understanding is that you had to bootstrap your entire
>> toolchain and address Ken Thompson's compiler hack from a different
>> angle. Is that right?
> At the time that Guix was founded, it was not considered possible to
> bootstrap a GNU/Linux system from source; on which OS would you perform
> the first compilation? How would you bootstrap that OS? Ad infinitum.
> Stage0 and mes and friends are fixing that now. [0]
>
> So, some binary "seeds" were created, which are compiled binaries of
> core tools such as GCC and binutils, and the system was based on those
> seeds. And we gave instructions for reproducing these seeds; you could
> use any commonly available distro as a host. Maybe that sounds icky in
> 2021, since the situation has improved radically, but it's how every OS
> was created since decades ago [1], except that most operating systems (and
> even language ecosystems) did not keep track of their seeds or give
> instructions for reproducing them. So Guix was better than all the rest
> from its beginning :)
>
> https://guix.gnu.org/manual/en/html_node/Bootstrapping.html
>
> I was not involved at that stage of Guix. I'm sure others can give more
> detailed answers.
>
>> I'd greatly appreciate learning how you all built Guix's GNU/Linux
>> distribution so that I can prepare a realistic roadmap and recruit help
>> where necessary. I'm bad at both of those things, but if there is any
>> opportunity to collaborate on implementation details based on Xiden's
>> progress, I am happy to give back.
> [0] https://bootstrappable.org/
> https://bootstrappable.org/projects/mes.html
> https://www.gnu.org/software/mes/
> Also visit the #bootstrappable channel on Libera.chat
>
> [1] It would be interesting to know the last truly bootstrapped OS



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-22 21:53   ` Sage Gerard
@ 2021-08-22 22:54     ` Philip McGrath
  2021-09-13  3:09       ` Christine Lemmer-Webber
  0 siblings, 1 reply; 17+ messages in thread
From: Philip McGrath @ 2021-08-22 22:54 UTC (permalink / raw)
  To: Sage Gerard, guix-devel

Hi Sage,

On 8/22/21 5:53 PM, Sage Gerard wrote:
> Thanks for the detailed answer!
> 
> It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead of
> Guile to bring GNU's benefits to more Scheme and Racket programmers. Has
> someone already tried something like that?

I haven't tried Xiden yet, and I haven't done any concrete work toward 
this (I have been working on managing Racket packages with Guix), but 
Christine Lemmer-Webber had floated the idea at some point of trying to 
integrate Racket and Guile.

IIRC, I think what she's had in mind was trying to make a Guile backend 
for Racket along the lines of the Chez Scheme backend (or the BC 
backend, or experimental backends like Pycket).

As I said, I haven't actually tried any of this, but, as I've thought 
about what might be involved, there are two things that have struck me 
as downsides:

  1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
     Report)" (§6, p. 13) that, "If our task were to compile Racket to an
     existing target, then we would not have achieved such a high degree
     of compatibility. … we have taken the liberty of modifying Chez
     Scheme to make it an easier target for Racket."

     https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf

     Presumably a Racket-on-Guile project would face the same trade-off,
     where modifications to Guild, if Racket CS is a guide, could require
     hard work over many years, and lesser compatibility would make the
     result less useful.

  2. As you probably know, Racket programs can't generally use
     Chez Scheme implemented libraries, because Chez Scheme effectively
     is the "unsafe" layer of the Racket VM. For example, not all Racket
     procedures are Chez Scheme procedures, and Racket's continuations
     wrap Chez Scheme's to implement delimited and composable control,
     threads, parameters, full continuation marks, etc.

     For Racket CS, this isn't a great loss (there aren't so many
     Chez-specific libraries, and portable libraries can run in Racket's
     R6RS language), but, for a hypothetical Racket-on-Guile,
     bidirectional interoperability would be a big attraction: imagine
     Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
     working together.

If I were going to work on this, I'd start by looking at having Racket 
and Guile coexist as siblings with interoperability through their FFIs 
level. Even better, eventually you could compile Guile to Racket 
linklets, so the two could coexist in the same primitive module system. 
There would probably always need to be something along the lines of 
require/typed to interoperate between the languages, since Guile has 
mutable pairs and an unusual approach to falsehood and nullity to let 
Scheme's #f and '() coexist with Emacs List's nil. (See 
https://www.gnu.org/software/guile/manual/html_node/Nil.html).

>>> I'm at the point where users are requesting a GNU/Linux distribution for
>>> Xiden, such that Racket is the primary language for day-to-day
>>> operation. I'm ignorant of the scope of work, and am unsure if I can do
>>> it alone.

To me, at least, the scope of the work in creating a new GNU/Linux 
distribution seems daunting. My hope would be that bringing Racket and 
Guile closer together would let most if not all of the effort be shared.

Hope some of this is useful, and best of luck!

-Philip



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-21 16:43 How did you handle making a GNU/Linux distribution? Sage Gerard
  2021-08-21 21:18 ` Leo Famulari
@ 2021-08-23 10:43 ` Maxime Devos
  2021-08-23 12:48   ` Sage Gerard
  2021-08-23 15:38 ` zimoun
  2 siblings, 1 reply; 17+ messages in thread
From: Maxime Devos @ 2021-08-23 10:43 UTC (permalink / raw)
  To: Sage Gerard, guix-devel

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

Sage Gerard schreef op za 21-08-2021 om 16:43 [+0000]:
> Hi folks,
> 
> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. [...]

> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. [...]

I looked at the Xiden documentation and source code a bit,
and I couldn't find any actual package definitions, or how to make
any package definitions.  Where are the 'coreutils' and some text editors
like 'nano' or 'emacs' or 'vi' packaged for example?

Hyperlinks in the manual aren't working:

https://download.racket-lang.org/releases/8.2/doc/local-redirect/index.html?doc=xiden-reference&rel=index.html&version=8.2

reads

This page is intended to redirect to the result of a search request. Since you’re reading this, it seems that the redirection did not work.

I would recommend working on the documentation.

> It is also free software under the GPLv3

Is this GPL-version-3-only or GPL-version-3-or-later?  If it's the latter,
Guix and Xiden could copy code and package definitions from each other
(after making some adaptations for Guile vs Racket and different interfaces),
though I wouldn't know how realistic that is in advancce.

Greetings,
Maxxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-23 10:43 ` Maxime Devos
@ 2021-08-23 12:48   ` Sage Gerard
  0 siblings, 0 replies; 17+ messages in thread
From: Sage Gerard @ 2021-08-23 12:48 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2871 bytes --]

> I looked at the Xiden documentation and source code a bit,
> and I couldn't find any actual package definitions, or how to make
> any package definitions.  Where are the 'coreutils' and some text editors
> like 'nano' or 'emacs' or 'vi' packaged for example? [...] Guix and Xiden could copy code and package definitions from each other
> (after making some adaptations for Guile vs Racket and different interfaces),
> though I wouldn't know how realistic that is in advancce.

I put these bits of your email together because I think they are related.

The system-level examples you are looking don't exist because Xiden is not fully bootstrapped. What I can do in the short-term is write a launcher that follows LFS, then verifies any artifacts used for the purpose. I wasn't satisfied with publishing that approach since it involves implicit trust in system level dependencies, but you can lean on the zero-trust default config and use ACLs on the small attack surface to mitigate risks there.

Until a Guile/Racket bridge is formed as Phillip described, or until GNU Mes is adapted to target Racket, I do not feel comfortable distributing system-level binaries using Xiden. It's easy for a user to write a launcher to use them, though. That's why I put so much focus on writing launchers that define a contract between your chosen data sources and your subjective expression of dependencies. Package definitions are subservient to this approach, because as an abstraction, they sit between the subsystem that fetches and verifies data, and the subsystem that understands what a user means when they ask to install a dependency. The latter is the launcher's job, and I often use one to generate package definitions, find integrity information, trust certs, etc.

The end of the guide directs you to example launchers, which set rules for how package definitions work in the context of those launchers. The early examples are just toy demos, but later ones handle things like a particular self-hosting approach, and managing multiple versions of Racket installations.

> Hyperlinks in the manual aren't working:

Let me know which ones in a Github issue? The Racket community shares a Racket installation for online docs, and I occasionally run into problems with the generated links. Sometimes its CI related, sometimes its because I didn't type a Scribble form exactly right. The docs are all present at least. Entry point links follow for convenience.

https://docs.racket-lang.org/xiden-guide/index.html
https://docs.racket-lang.org/xiden-white-paper/index.html
https://docs.racket-lang.org/xiden-reference/index.html

> Is this GPL-version-3-only or GPL-version-3-or-later?

Currently 3. I didn't add "or later" (yet) because I don't understand the social and legal consequences of doing so, even when new license versions are compatible. Happy to learn more here.

[-- Attachment #2: Type: text/html, Size: 4159 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-21 16:43 How did you handle making a GNU/Linux distribution? Sage Gerard
  2021-08-21 21:18 ` Leo Famulari
  2021-08-23 10:43 ` Maxime Devos
@ 2021-08-23 15:38 ` zimoun
  2021-08-23 17:30   ` Philip McGrath
  2021-08-23 18:24   ` Sage Gerard
  2 siblings, 2 replies; 17+ messages in thread
From: zimoun @ 2021-08-23 15:38 UTC (permalink / raw)
  To: Sage Gerard; +Cc: Guix Devel

Hi Sage,

On Sat, 21 Aug 2021 at 22:18, Sage Gerard <sage@sagegerard.com> wrote:

> My name is Sage. I wrote a cross-platform Guix-like package manager
> called Xiden. It applies functional package management to the Racket
> ecosystem. It is also free software under the GPLv3. The source is
> available at zyrolasting/xiden on Github, pending migration to a new host.

Oh thanks! :-)

> I'm at the point where users are requesting a GNU/Linux distribution for
> Xiden, such that Racket is the primary language for day-to-day
> operation. I'm ignorant of the scope of work, and am unsure if I can do
> it alone.  My understanding is that you had to bootstrap your entire
> toolchain and address Ken Thompson's compiler hack from a different
> angle. Is that right?

Well, from my understanding, Guix (package manager) and Guix System
(OS on the top) are not about bootstrap.  If you give a look at one of
the first public presentations:

<https://people.bordeaux.inria.fr/lcourtes/doc/guix-ghm-2012.20120721.pdf>

then bootstrap the whole toolchain was not on topic (at least not
explicitly listed).  Reducing the bootstrap seed came later and
gradually, for instance see:

<https://archive.fosdem.org/2017/schedule/event/guixsdbootstrap/>
<https://archive.fosdem.org/2019/schedule/event/gnumes/>

> I'd greatly appreciate learning how you all built Guix's GNU/Linux
> distribution so that I can prepare a realistic roadmap and recruit help
> where necessary. I'm bad at both of those things, but if there is any
> opportunity to collaborate on implementation details based on Xiden's
> progress, I am happy to give back.

I have read the White Paper and the doc.  I am sorry but I miss which
problem Xiden is solving and how it does.  For instance, I do not
understand what a "launcher" is and what its job is.  Maybe Xiden is a
"workflow" engine?

The bootstrap path of Racket in Guix is not clear to me.  I miss if a
Racket interpreter or compiler in its binary format is used (as
Haskell for instance) or if all is compiled from source using tools
already bootstrapped.  Anyway, I miss what you would like bootstrap?
If you mean a trusted seed in order to start a package collection, you
could use the current Guix binaires---as a starting point.

Well, from my understanding, as Philip said, compile Guile to Racket
linklets seems something promising. :-)

<https://docs.racket-lang.org/reference/linklets.html>

All the best,
simon


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-23 15:38 ` zimoun
@ 2021-08-23 17:30   ` Philip McGrath
  2021-08-24 10:24     ` zimoun
  2021-08-23 18:24   ` Sage Gerard
  1 sibling, 1 reply; 17+ messages in thread
From: Philip McGrath @ 2021-08-23 17:30 UTC (permalink / raw)
  To: zimoun, Sage Gerard; +Cc: Guix Devel

Hi Simon,

On 8/23/21 11:38 AM, zimoun wrote:
> The bootstrap path of Racket in Guix is not clear to me.  I miss if a
> Racket interpreter or compiler in its binary format is used (as
> Haskell for instance) or if all is compiled from source using tools
> already bootstrapped.  Anyway, I miss what you would like bootstrap?
> If you mean a trusted seed in order to start a package collection, you
> could use the current Guix binaires---as a starting point.

I wrote a long comment at the top of gnu/packages/racket.scm that 
explains the current state of affairs. (For anyone unfamiliar with the 
Guix source tree: 
<https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/racket.scm>)

In brief, Racket is bootstrapped from source using only a C 
compiler---including bootstrapping Racket's fork of Chez Scheme without 
an existing Chez Scheme compiler---with one exception: the "expander" 
subsystem (which implements the reader, module system, and macro 
expander) is implemented in Racket, and the older C implementation was 
completely replaced as of Racket 7.0. (One of the motivations was to fix 
many problems with the old implementation.) For bootstrapping, an 
intermediate representation as a "schemified" linklet is checked into 
the Racket source repository.

This impact is mitigated because, by design, the intermediate 
representation is "human-readable and -editable Scheme code". 
Anecdotally, when I read diffs of changes to the Racket repository, I 
sometimes read tens of lines of code before I realize that I'm looking 
at the generated version of the file, not the source. So the expander is 
not bootstrappable in the most rigorous sense, but it is readily auditable.

If someone wanted to tie the knot, the task would be to use an existing 
bootstappable language to write a minimal expander that can expand the 
canonical Racket implementation: if your output is identical to the 
checked-in generated files, you've succeeded. (Using Guile could avoid 
the heavy maintenance burden of the old C expander implementation, and 
indeed you could write just enough of an adapter to load the Racket 
source files into Guile, which is the technique Racket uses to bootstrap 
the Chez Scheme compiler.) I expect Racket would welcome a contribution 
along those lines; it just hasn't been a priority for anyone yet.

-Philip


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-23 15:38 ` zimoun
  2021-08-23 17:30   ` Philip McGrath
@ 2021-08-23 18:24   ` Sage Gerard
  2021-08-23 19:23     ` Ryan Prior
  2021-08-24 16:42     ` Maxime Devos
  1 sibling, 2 replies; 17+ messages in thread
From: Sage Gerard @ 2021-08-23 18:24 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 7668 bytes --]

Thank you for the links!

> I miss which problem Xiden is solving and how it does.

You are not the first to say so, and I'm happy to learn more about why.

I'll try to explain in a different way here, so forgive the text wall. I'll incorporate any feedback here to improve the white paper. I'd also like to hear more about Guix's philosophy on what I'm going to talk about, because I had reasons to end up with a different model.

--

If we each use a program to distribute software, then those programs are probably incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses this by decoupling the subjective elements of software distribution from the objective ones. That means modeling software distribution as (partly) a semantics problem. Racket didn't do this with its own package managers. Which is ironic, because it's hard to guarantee that that an arbitrary Racket program will get the dependencies they mean to use. I've written about this at length [1][2][3], and spoke about it last year [4]. Xiden has changed a bit since the speech to be more flexible, and it is no longer limited for use in Racket projects. Apologies if you see something that looks contradictory.

What does that mean? Let's say Alice and Bob each write a Xiden launcher.

Alice

- trusts SHA-384, but only if its implementation uses Keccak.
- defines "sha384" as the canonical name of the CHF, and treat variants like "SHA-384" as aliases.
- delegates trust in artifact signatures to GPG subprocesses.
- downloads content from her employer's S3 buckets.
- unconditionally prefers cached installations, such that every defined artifact has exactly one stored copy on disk. All dependents access the stored artifact using symbolic links or Windows junctions.
- prefers declaring versions with edition and revision names.

Bob

- trusts SHA-1, but only for artifacts from services authenticated by his operating system's certificates.
- trusts digest creation and signature verification as implemented by his host's dynamically--linked `libcrypto` library for use in the same process.
- downloads content from GitHub packages
- unconditionally prefers SxS installations, such that packages cannot conflict in a shared namespace at the cost of defeating an internal cache.
- prefers Semantic Versions

These preferences are valid in Xiden's DSL for writing launchers. The invariants for each launcher can differ as much as Cargo and Vortex differ. What I wanted to do was allow users to declare their own invariants explicitly, but only when those invariants are wanted for subjective reasons. I can't just write an overly-opinionated tool in this space because opinions have shelf-lives, and they can't interrupt our ability to get the content we need on demand, with our own safety guarentees.

Xiden launchers have advantages over shell scripts in that we can both still download and install dependencies from the same servers, and create reproducible builds in terms of the same (bit-identical) state on disk. We just differ on details that shouldn't impact how we work. It also helps us patch holes faster, since if (say) a catalog maintainer tells us that a CHF was compromised, we can immediately revoke trust in the CHF independently in our own clients. The package definitions are all expressed in terms of what the launchers allow.

So what problem does this solve? I'm trying to preserve an element of walk-away power in any tech community. Maybe your community goes in a questionable direction. Maybe your desired software is in a different ecosystem. Maybe the maintainers are abusive jerks. Maybe you have a `leftpad`/`event-stream` incident and the developer is unable/unwilling to patch the problem. You'd probably want a way to escape to a new community or distribution model without giving up content or doing complex integration work. If Xiden can do that, users don't have to depend on the same middlemen to share work. This is also better for developers because it doesn't oblige them to do everything their users want, while knowing users will adapt to inevitable distribution problems. Everyone's consent becomes explicit, such that Xiden's "real" invariant is that consent between any two parties be mutually compatible. I hope to improve user freedom from this angle, and my approach is biased by my experience with Racket's limitations in this space.

The subjective parts I talk about go further into details like what name canons, versioning schemes, trusted certificate chains, and even specific approaches to an exact diamond dependency. The objective parts include integrity checking, signature verification, safety limits, automatic dependency resolution, patches, protocols, and other concerns that a decent package manager normally cares about. By keeping these separate, all of the below features become cheap to write, without sacrificing trust & safety in the abstract.

- Download archived GitHub repositories (source on Racket Discord atm)
- Manage Racket installations like how `nvm` manages Node.js installations [5]
- Produce a self-hosted Xiden installations, with implicit trust for the running Xiden instance [6]
- Control which exact artifacts must be produced deterministically [7]
- Force generated bindings to be `eq?` (This one haunts Racket's package managers today) [8]
- Source package definitions from Pastebin to download content, digests, and signatures from a public S3 bucket (This does not exist today. I'm spitballing because I know it would work)

I hypothesize that Xiden can "abstract over" package managers in the same way that Racket can abstract over languages. I think people are used to package managers deciding things for them.

So what is Xiden for? Well, it's not for deciding an SOP for you. It's for giving everyone the option to change the SOP for themselves with minimal politics.

> what you would like to bootstrap?

All binaries related to creating a GNU/Linux distribution, such that I can reproduce an exact OS, Racket installation, and Xiden instance. I want a trusted initial state on GNU/Linux.

To be clear, I don't need to do this for functional reasons. Xiden's already operational. I just can't claim that you can trust a given Xiden instance with the same confidence as a Guix instance right now.

Caveats:

- I have no idea how to do this yet, so thanks to all of you for the leads.
- Xiden is cross-platform, and it already runs anywhere Racket can (Windows, macOS, Unix-likes). I can't make the best claims re: trustworthiness without GNU software, but Xiden is at least operational everywhere. That's why I ended up defining a small attack surface that's consistent across systems, to make it easy to set up host-level protections.

> If you mean a trusted seed in order to start a package collection, you could use the current Guix binaires---as a starting point.

I'm reading GNU Mes' manual. I am confident that it will help me find the answers I'm looking for, including all of your helpful replies!

[1]: https://sagegerard.com/polyglot3-package-nightmare.html
[2]: https://www.reddit.com/r/GUIX/comments/p8v5xd/how_did_you_handle_making_a_gnulinux_distribution/h9zpyhn?utm_source=share&utm_medium=web2x&context=3
[3]: https://sagegerard.com/new-racket-pkg-releases.html
[4]: https://www.youtube.com/watch?t=2330&v=bIi-tUzOwdw&feature=youtu.be
[5]: https://github.com/zyrolasting/xiden/tree/master/examples/racket-installation-manager
[6]: https://github.com/zyrolasting/xiden/tree/master/examples/self-hosting
[7]: https://github.com/zyrolasting/xiden/tree/master/examples/determinism
[8]: https://github.com/zyrolasting/xiden/tree/master/examples/generated-racket-bindings

[-- Attachment #2: Type: text/html, Size: 10974 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-23 18:24   ` Sage Gerard
@ 2021-08-23 19:23     ` Ryan Prior
  2021-08-23 20:00       ` Sage Gerard
  2021-08-24 16:42     ` Maxime Devos
  1 sibling, 1 reply; 17+ messages in thread
From: Ryan Prior @ 2021-08-23 19:23 UTC (permalink / raw)
  To: Sage Gerard; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

> [I want to bootstrap] all binaries related to creating a GNU/Linux distribution, such that I can reproduce an exact OS, Racket installation, and Xiden instance. I want a trusted initial state on GNU/Linux.

Seems like the easy path for you is to package Xiden for Guix, and then construct a system definition (using Guix System) that includes Racket and Xiden in its default set of packages, and finally use Guix to build system-images for that target system.

To do this does not require that you can run Racket on the Guile VM. The target system-image doesn't even need to have Guile or Guix installed on it at all; it can just serve as a bootstrapped base and you can build from there to your final desired state.

I'm happy to explain or assist with any part of this process, and I imagine others on this list would be as well.

Designing your own bootstrap and rewriting everything in Racket, or waiting for a suitable Racket-on-Guile implementation, do not seem necessary at all to reach your stated goals. Hope this helps!

[-- Attachment #2: Type: text/html, Size: 1175 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-23 19:23     ` Ryan Prior
@ 2021-08-23 20:00       ` Sage Gerard
  0 siblings, 0 replies; 17+ messages in thread
From: Sage Gerard @ 2021-08-23 20:00 UTC (permalink / raw)
  To: rprior; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

Sounds like a great ROI, thank you for the suggestion!

-------- Original Message --------
On Aug 23, 2021, 3:23 PM, Ryan Prior wrote:

>> [I want to bootstrap] all binaries related to creating a GNU/Linux distribution, such that I can reproduce an exact OS, Racket installation, and Xiden instance. I want a trusted initial state on GNU/Linux.
>
> Seems like the easy path for you is to package Xiden for Guix, and then construct a system definition (using Guix System) that includes Racket and Xiden in its default set of packages, and finally use Guix to build system-images for that target system.
>
> To do this does not require that you can run Racket on the Guile VM. The target system-image doesn't even need to have Guile or Guix installed on it at all; it can just serve as a bootstrapped base and you can build from there to your final desired state.
>
> I'm happy to explain or assist with any part of this process, and I imagine others on this list would be as well.
>
> Designing your own bootstrap and rewriting everything in Racket, or waiting for a suitable Racket-on-Guile implementation, do not seem necessary at all to reach your stated goals. Hope this helps!

[-- Attachment #2: Type: text/html, Size: 1390 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-23 17:30   ` Philip McGrath
@ 2021-08-24 10:24     ` zimoun
  0 siblings, 0 replies; 17+ messages in thread
From: zimoun @ 2021-08-24 10:24 UTC (permalink / raw)
  To: Philip McGrath, Sage Gerard; +Cc: Guix Devel

Hi Philip,

On Mon, 23 Aug 2021 at 13:30, Philip McGrath <philip@philipmcgrath.com> wrote:
> On 8/23/21 11:38 AM, zimoun wrote:
>> The bootstrap path of Racket in Guix is not clear to me.  I miss if a
>> Racket interpreter or compiler in its binary format is used (as
>> Haskell for instance) or if all is compiled from source using tools
>> already bootstrapped.  Anyway, I miss what you would like bootstrap?
>> If you mean a trusted seed in order to start a package collection, you
>> could use the current Guix binaires---as a starting point.
>
> I wrote a long comment at the top of gnu/packages/racket.scm that 
> explains the current state of affairs. (For anyone unfamiliar with the 
> Guix source tree: 
> <https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/racket.scm>)

Oh sorry!  I did incorrectly my homework… although I remember the
discussion with Ludo when you submitted the patch set. :-)

> In brief, Racket is bootstrapped from source using only a C 
> compiler---including bootstrapping Racket's fork of Chez Scheme without 
> an existing Chez Scheme compiler---with one exception: the "expander" 
> subsystem (which implements the reader, module system, and macro 
> expander) is implemented in Racket, and the older C implementation was 
> completely replaced as of Racket 7.0. (One of the motivations was to fix 
> many problems with the old implementation.) For bootstrapping, an 
> intermediate representation as a "schemified" linklet is checked into 
> the Racket source repository.

Thanks for explaining.  All is clear.

> This impact is mitigated because, by design, the intermediate 
> representation is "human-readable and -editable Scheme code". 
> Anecdotally, when I read diffs of changes to the Racket repository, I 
> sometimes read tens of lines of code before I realize that I'm looking 
> at the generated version of the file, not the source. So the expander is 
> not bootstrappable in the most rigorous sense, but it is readily auditable.
>
> If someone wanted to tie the knot, the task would be to use an existing 
> bootstappable language to write a minimal expander that can expand the 
> canonical Racket implementation: if your output is identical to the 
> checked-in generated files, you've succeeded. (Using Guile could avoid 
> the heavy maintenance burden of the old C expander implementation, and 
> indeed you could write just enough of an adapter to load the Racket 
> source files into Guile, which is the technique Racket uses to bootstrap 
> the Chez Scheme compiler.) I expect Racket would welcome a contribution 
> along those lines; it just hasn't been a priority for anyone yet.

Guile everywhere! ;-)

Thanks again the details.

Cheers,
simon


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-23 18:24   ` Sage Gerard
  2021-08-23 19:23     ` Ryan Prior
@ 2021-08-24 16:42     ` Maxime Devos
  2021-08-24 17:17       ` Sage Gerard
  1 sibling, 1 reply; 17+ messages in thread
From: Maxime Devos @ 2021-08-24 16:42 UTC (permalink / raw)
  To: Sage Gerard, guix-devel

[-- Attachment #1: Type: text/plain, Size: 17310 bytes --]

Sage Gerard schreef op ma 23-08-2021 om 18:24 [+0000]:
> Thank you for the links!
> 
> > I miss which problem Xiden is solving and how it does. 
> 
> You are not the first to say so, and I'm happy to learn more about why.
> 
> I'll try to explain in a different way here, so forgive the text wall.
> I'll incorporate any feedback here to improve the white paper. I'd also like
> to hear more about Guix's philosophy on what I'm going to talk about, because
> I had reasons to end up with a different model.

Maybe the publications at <https://guix.gnu.org/en/publications/> are informative.

> --
> 
> If we each use a program to distribute software, then those programs are probably
> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses this by
> decoupling the subjective elements of software distribution from the objective ones.
>   That means modeling software distribution as (partly) a semantics problem. Racket didn't
>  do this with its own package managers. Which is ironic, because it's hard to guarantee
>  that that an arbitrary Racket program will get the dependencies they mean to use.
>
>  I've written about this at length [1][2][3],

About polyglot: packages in Guix can specify which version of a package they use.
E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a specific
version of 'autoconf' (2.13) and 'rust' (1.41)

    (native-inputs
     `(("autoconf" ,autoconf-2.13)
       ("automake" ,automake)
       ("llvm" ,llvm)                   ;for llvm-objdump
       ("perl" ,perl)
       ("pkg-config" ,pkg-config)
       ("python" ,python-3)
       ("rust" ,rust-1.41)
       ("cargo" ,rust-1.41 "cargo")))

Is something missing from Guix here?  About [3]: this doesn't seem a problem in Guix,
if you're referring to the ability to define multiple versions of a package at the same
time (it's an ‘official policy’ to usually try to only keep the latest version of a
package though, but channels can ignore this completely).

>  and spoke about it last year [4]. Xiden has changed a bit since the speech to be more flexible, and it is no longer limited for use in Racket projects. Apologies if you see something that looks contradictory.
> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
> 
> Alice
> 
> trusts SHA-384, but only if its implementation uses Keccak.

SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
SHA-3: a version of Keccak.  By definition, implementations of SHA-384
have no reason to use Keccak.

> defines "sha384" as the canonical name of the CHF, and treat variants like "SHA-384" as aliases.
> delegates trust in artifact signatures to GPG subprocesses.

Guix allows changing the hash algorithm used, search for ‘content-hash’ in the guix manual.
Changing the hash algorithm globally unavoidably requires modifying every package definition
though, but that could be automated.

> downloads content from her employer's S3 buckets.

The ‘official’ substitute servers can be replaced:
"guix build --substitute-urls=http://s3.bucket.employer"
(after authorising that substitute server).

> unconditionally prefers cached installations, such that every defined artifact
> has exactly one stored copy on disk.

Coming from Guix, I don't know what this means.  Does this mean only one version
of a package can be in the store at the same time, and building a newer version
deletes the older version?  How does this compare with ‘content deduplication’?

> All dependents access the stored artifact
> using symbolic links or Windows junctions.


> prefers declaring versions with edition and revision names.
> Bob trusts SHA-1,

See ‘content-hash’ above.  Also, why should Xiden let Bob trust SHA-1 in the first place?
(See <https://en.wikipedia.org/wiki/SHA-1#SHAttered_%E2%80%93_first_public_collision>.)

>  but only for artifacts from services authenticated by his operating system's certificates

This is rather vague to me, unless you're referring to substitutes.
What does this mean for origin specifications?

    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256 ; imagine this was SHA-1 instead.
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))

Bob got the origin specification from Xiden (or something like a Guix channel
but for Xiden).  How would this ‘artifact’ authenticated by ‘operating system
certificates’?  I mean, when you let your operating system build that origin
(via guix (or Xiden?), which you could consider part of the OS), it will check
that hash, and it will be considered valid (‘authenticated?’) if the hash matches.
There are no certificates at play here.

I suppose the OS (guix or Xiden?) could then ‘sign’ it, but what value does that
provide to Bob?  (Unless Bob runs a substitute server on their computer.)

> .
> trusts digest creation and signature verification as implemented by his host's dynamically--linked `libcrypto` library for use in the same process.
> downloads content from GitHub packages

AFAIK there is no such thing a ‘GitHub packages’.  What is a ‘GitHub package’,
precisely?  When I hear ‘X package’, I think of "X = python, haskell, ...’ which
have a standardised build sytem (represented by python-build-system
and haskell-build-system in Guix), a standardised location to download them from
(pypi.org and hackage.haskell.org, represented by the 'pypi' and 'hackage' importers
an refreshers in Guix) and are separate languages.

The only thing things on GitHub have in common are:

 * they have git repositories on GitHub
 * some repositories on GitHub have ‘releases’, which can automatically be discovered

Notice dependency information and build system information are absent.
 
> unconditionally prefers SxS installations,

I searched for ‘SxS’, and found this: <https://en.wikipedia.org/wiki/SxS>.
I don't think you meant that.  What is ‘SxS’?

>  such that packages cannot conflict in a shared namespace 

‘packages cannot conflict in a shared namespace’ is a bit vague.
Are you referring to ‘profile collisions’?  About having multiple versions
of the same package on the same system?  Guix (and nix) can handle the latter.
When the ‘propagated-inputs’ are limited, installing multiple packages each using
a different version of a package is possible.

Is something missing from Guix here (please be very concrete)?

>  at the cost of defeating an internal cache.

I'm not sure what you mean here, can you illustrate this very concretely?
And is something missing in Guix here?

> prefers Semantic Versions

If upstream doesn't do ‘semver’, and instead uses a versioning system like
used by TeX (3.14, 3.141, 3.1415, 3.14159, ...), then Xiden cannot somehow
let the package use ‘semver’ instead.  Unless you want to teach Xiden to
convince upstream to change their versioning system?

> These preferences are valid in Xiden's DSL for writing launchers.

What is a launcher (compared to packages in Guix), and why would I want one?
Why shouldn't I just install packages and run them ("guix package -i texmacs",
start texmacs from some application chooser menu of the desktop environment).

>  The invariants for each launcher can differ as much as Cargo and Vortex differ.
>  What I wanted to do was allow users to declare their own invariants explicitly,
>  but only when those invariants are wanted for subjective reasons.

What are these invariants you're speaking of?  I don't think you're referring
to, say, translation invariance (‘the laws of physics remain the same under a
translation of the coordinate system’).

Or do you mean something like ‘Invariant (computer science), an expression whose
value doesn't change during program execution’ (<https://en.wikipedia.org/wiki/Invariant>)?
But what are the ‘expressions’ in this case?

>  I can't just write an overly-opinionated tool in this space because opinions
>  have shelf-lives, and they can't interrupt our ability to get the content we
>  need on demand, with our own safety guarentees.

Is Guix too opinionated about something in particular?

> Xiden launchers have advantages over shell scripts in that we can both still download
> and install dependencies from the same servers, and create reproducible builds in terms
> of the same (bit-identical) state on disk. We just differ on details that shouldn't impact
> how we work. It also helps us patch holes faster, since if (say) a catalog maintainer tells

What is a ‘catalog maintainer’ (maybe it's like someone with commit access to a ‘guix channel’?)
Also, what ‘shell scripts’ are you referring to?  I've been doing well without writing any shell
scripts.

> us that a CHF was compromised, we can immediately revoke trust in the CHF independently

Maybe guix could gain a command "guix style transition-hash OLD NEW" to automagically
rewrite package definitions to use the new hash algorithm instead of the old.  It will
need to download plenty of source code however, and it will entail a world-rebuild.

Note that, in practice, signs of the brokenness of hashes appear well in advance of
actual exploits.

> in our own clients.

What are these ‘clients’ you're speaking of?  I haven't been needing any ‘clients’ lately.

>  The package definitions are all expressed in terms of what the launchers allow.

If 'package definition' = (package definition as in Guix), then this is false,
Guix doesn't have a notion of ‘launchers’ and doesn't depend on ‘Xiden’.
What are you meaning, exactly?  Also, I haven't found any package definitions in Xiden,
could you point me to any?

> So what problem does this solve? I'm trying to preserve an element of walk-away power
> in any tech community. Maybe your community goes in a questionable direction. Maybe your
> desired software is in a different ecosystem.

Guix channels can function independently of the main ‘guix’, augmenting ‘guix’ with
additional packages, even if they contravene official Guix policies (e.g. non-free,
bad quality ...), or simply if putting it in ‘guix proper’ takes to much time.
There exist a few relatively well-known channels like that.

>  Maybe the maintainers are abusive jerks.

Technically, Guix has ‘maintainers’, but I wouldn't know where I could find the list.
Guix (the package manager) doesn't have its own infrastructure.
It does have official substitute servers, but you can ignore them.
So I don't see ‘maintainers are abusive jerks’ as a plausible threat to Guix.
If they become existent, they can easily be replaced.

> Maybe you have a `leftpad`/`event-stream` incident and the developer is unable/unwilling
> to patch the problem.

In Guix, if someone pushed malware (*) to, say, a package on pypi.org, then we (Guix)
would simply stop using new versions of that package from pypi.org.  If there's a healthy
fork somewhere, we could switch over to that for new versions.

(*) I could have confused the ‘`leftpad`/`event-stream` incident’ for another incident.

> You'd probably want a way to escape to a new community or distribution model
>  without

‘Escape to a new community’ --> only if the community as a whole are ‘abusive jerks’.
If it's merely some maintainers, they can be removed or replaced.  If it's all the
maintainers, I'm sure a few well-regarded members of the community could come together
for a ‘coup’.  If it's the community as a whole, I don't see how Xiden could help,
unless you mean I would move from Guix to Xiden? 

Why would I want to ‘escape to a new distribution model’?  Writing something like Guix
from scratch seems a lot of work to me, I would rather fork Guix (with a few
likewise-minded collaborators) than to start over again.  Or move to Debian maybe.
I don't see what ‘Xiden’ could offer here.

>  giving up content or doing complex integration work. If Xiden can do that, users
>  don't have to depend on the same middlemen to share work.

Is ‘middlemen’ = guix committers here?  Do you have concrete scenario to illustrate this,

>  This is also better for
>  developers because it doesn't oblige them to do everything their users want,

Are we talking about upstream developers here, or guix(/Xiden) committers?
How does this ‘obligation’ exist in the first place?

>  while knowing users will adapt to inevitable distribution problems.

I would rather not have to ‘adapt to problems’, I would rather that upstream has something
that works (though I can understand if occasionally mistakes are made).

>  Everyone's consent
>  becomes explicit, such that Xiden's "real" invariant is that consent between any
>  two parties be mutually compatible.

Consent about what?

> I hope to improve user freedom from this angle,
>  and my approach is biased by my experience with Racket's limitations in this space.

Why are you comparing it with ‘racket’ instead of, say, Debian or Guix?
‘racket’ is a Scheme implementation, not a package manager.

> The subjective parts I talk about go further into details like what name canons,
> versioning schemes, trusted certificate chains, and even specific approaches to an exact
> diamond dependency. The objective parts include integrity checking, signature verification,
> safety limits, automatic dependency resolution, patches, protocols, and other concerns that
> a decent package manager normally cares about. By keeping these separate, all of the below
> features become cheap to write, without sacrificing trust & safety in the abstract.
> 
> Download archived GitHub repositories (source on Racket Discord atm)

It can download GitHub repositories.  Why are you writing ‘archived’ here?
Guix supports git repositories (see 'git-reference' in the guix manual), so
I don't see what additional value Xiden brings here.  Can it also automagically build
the software?  Also, why ‘GitHub repositories’ specifically?  Why not, say,
the git repositories at <https://git.ngyro.com/>?

> Manage Racket installations like how `nvm` manages Node.js installations [5]

It would seem that your ‘racket-installation-manager’ [5] cannot resist a compromise
of https://downloads.racket-lang.org/releases/.  Also, I don't see how it would be
possibe to use ‘racket-installation-manager’ offline.  And how does it compile the
racket packages?  If we're talking about ‘trust’ and ‘escaping to a new community’,
maybe provide an option to override 'host-url-prefix'?  Currently, it's hardcoding
the ‘official’ racket community.

> Produce a self-hosted Xiden installations, with implicit trust for the running Xiden instance [6]
> Control which exact artifacts mnust be produced deterministically [7]

Can I reproduce the ‘artifact’ (= store item?) on a separate machine?

> Force generated bindings to be `eq?` (This one haunts Racket's package managers today) [8]

That seems like a Scheme thing to me, not a package manager thing.

> Source package definitions from Pastebin to download content, digests,
> and signatures from a public S3 bucket (This does not exist today. I'm spitballing because I know
>  it would work)

How can this be secure?

> I hypothesize that Xiden can "abstract over" package managers in the same way that Racket can
> abstract over languages. I think people are used to package managers deciding things for them. 

Please provide some sane defaults.  I would rather not have to decide everything myself,
and be able to trust Guix(/Xiden?) to do something reasonable.

Also, if you ‘abstract over package managers’, wouldn't you just end up with one ‘super
package manager’?  Why not ‘manage’ everything with a single package manager,
like e.g. guix?

> So what is Xiden for? Well, it's not for deciding an SOP for you.

Searching for SOP, I find <https://en.wikipedia.org/wiki/Same-origin_policy>.
What were you referring to here?

>  It's for giving everyone the option to change the SOP for themselves with minimal politics.

What are these changes you're referring to?

> > what you would like to bootstrap?
> 
> All binaries related to creating a GNU/Linux distribution, such that I can reproduce an exact
> OS, Racket installation, and Xiden instance. I want a trusted initial state on GNU/Linux.
> 
> To be clear, I don't need to do this for functional reasons. Xiden's already operational. I
> just can't claim that you can trust a given Xiden instance with the same confidence as a
>  Guix instance right now.

If you want to get package definitions from guix, take a look at (guix inferior).
Maybe you could port the relevant code of (guix inferior) to Racket, to let Xiden
talk to Guix?   In particular, 'inferior-eval' sends an S-exp to the inferior Guix,
which is evaluated there, and the result is sent back.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-24 16:42     ` Maxime Devos
@ 2021-08-24 17:17       ` Sage Gerard
  0 siblings, 0 replies; 17+ messages in thread
From: Sage Gerard @ 2021-08-24 17:17 UTC (permalink / raw)
  To: Maxime Devos, guix-devel

I'll answer your questions in a different thread to limit the noise in
this one. Look for the "Xiden/Guix design comparison" subject line
within the next 24h.

On 8/24/21 12:42 PM, Maxime Devos wrote:
> Sage Gerard schreef op ma 23-08-2021 om 18:24 [+0000]:
>> Thank you for the links!
>>
>>> I miss which problem Xiden is solving and how it does.
>> You are not the first to say so, and I'm happy to learn more about why.
>>
>> I'll try to explain in a different way here, so forgive the text wall.
>> I'll incorporate any feedback here to improve the white paper. I'd also like
>> to hear more about Guix's philosophy on what I'm going to talk about, because
>> I had reasons to end up with a different model.
> Maybe the publications at <https://guix.gnu.org/en/publications/> are informative.
>
>> --
>>
>> If we each use a program to distribute software, then those programs are probably
>> incompatible in some way (e.g. Cargo vs. Nexus' Vortex). Xiden addresses this by
>> decoupling the subjective elements of software distribution from the objective ones.
>>    That means modeling software distribution as (partly) a semantics problem. Racket didn't
>>   do this with its own package managers. Which is ironic, because it's hard to guarantee
>>   that that an arbitrary Racket program will get the dependencies they mean to use.
>>
>>   I've written about this at length [1][2][3],
> About polyglot: packages in Guix can specify which version of a package they use.
> E.g., Guix has multiple versions of 'mozjs', and 'mozjs@78.10.1' uses a specific
> version of 'autoconf' (2.13) and 'rust' (1.41)
>
>      (native-inputs
>       `(("autoconf" ,autoconf-2.13)
>         ("automake" ,automake)
>         ("llvm" ,llvm)                   ;for llvm-objdump
>         ("perl" ,perl)
>         ("pkg-config" ,pkg-config)
>         ("python" ,python-3)
>         ("rust" ,rust-1.41)
>         ("cargo" ,rust-1.41 "cargo")))
>
> Is something missing from Guix here?  About [3]: this doesn't seem a problem in Guix,
> if you're referring to the ability to define multiple versions of a package at the same
> time (it's an ‘official policy’ to usually try to only keep the latest version of a
> package though, but channels can ignore this completely).
>
>>   and spoke about it last year [4]. Xiden has changed a bit since the speech to be more flexible, and it is no longer limited for use in Racket projects. Apologies if you see something that looks contradictory.
>> What does that mean? Let's say Alice and Bob each write a Xiden launcher.
>>
>> Alice
>>
>> trusts SHA-384, but only if its implementation uses Keccak.
> SHA-384 and Keccak are separate things? SHA-384: a version of SHA-2,
> SHA-3: a version of Keccak.  By definition, implementations of SHA-384
> have no reason to use Keccak.
>
>> defines "sha384" as the canonical name of the CHF, and treat variants like "SHA-384" as aliases.
>> delegates trust in artifact signatures to GPG subprocesses.
> Guix allows changing the hash algorithm used, search for ‘content-hash’ in the guix manual.
> Changing the hash algorithm globally unavoidably requires modifying every package definition
> though, but that could be automated.
>
>> downloads content from her employer's S3 buckets.
> The ‘official’ substitute servers can be replaced:
> "guix build --substitute-urls=http://s3.bucket.employer"
> (after authorising that substitute server).
>
>> unconditionally prefers cached installations, such that every defined artifact
>> has exactly one stored copy on disk.
> Coming from Guix, I don't know what this means.  Does this mean only one version
> of a package can be in the store at the same time, and building a newer version
> deletes the older version?  How does this compare with ‘content deduplication’?
>
>> All dependents access the stored artifact
>> using symbolic links or Windows junctions.
>
>> prefers declaring versions with edition and revision names.
>> Bob trusts SHA-1,
> See ‘content-hash’ above.  Also, why should Xiden let Bob trust SHA-1 in the first place?
> (See <https://en.wikipedia.org/wiki/SHA-1#SHAttered_%E2%80%93_first_public_collision>.)
>
>>   but only for artifacts from services authenticated by his operating system's certificates
> This is rather vague to me, unless you're referring to substitutes.
> What does this mean for origin specifications?
>
>      (source (origin
>                (method url-fetch)
>                (uri (string-append "mirror://gnu/hello/hello-" version
>                                    ".tar.gz"))
>                (sha256 ; imagine this was SHA-1 instead.
>                 (base32
>                  "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
>
> Bob got the origin specification from Xiden (or something like a Guix channel
> but for Xiden).  How would this ‘artifact’ authenticated by ‘operating system
> certificates’?  I mean, when you let your operating system build that origin
> (via guix (or Xiden?), which you could consider part of the OS), it will check
> that hash, and it will be considered valid (‘authenticated?’) if the hash matches.
> There are no certificates at play here.
>
> I suppose the OS (guix or Xiden?) could then ‘sign’ it, but what value does that
> provide to Bob?  (Unless Bob runs a substitute server on their computer.)
>
>> .
>> trusts digest creation and signature verification as implemented by his host's dynamically--linked `libcrypto` library for use in the same process.
>> downloads content from GitHub packages
> AFAIK there is no such thing a ‘GitHub packages’.  What is a ‘GitHub package’,
> precisely?  When I hear ‘X package’, I think of "X = python, haskell, ...’ which
> have a standardised build sytem (represented by python-build-system
> and haskell-build-system in Guix), a standardised location to download them from
> (pypi.org and hackage.haskell.org, represented by the 'pypi' and 'hackage' importers
> an refreshers in Guix) and are separate languages.
>
> The only thing things on GitHub have in common are:
>
>   * they have git repositories on GitHub
>   * some repositories on GitHub have ‘releases’, which can automatically be discovered
>
> Notice dependency information and build system information are absent.
>
>> unconditionally prefers SxS installations,
> I searched for ‘SxS’, and found this: <https://en.wikipedia.org/wiki/SxS>.
> I don't think you meant that.  What is ‘SxS’?
>
>>   such that packages cannot conflict in a shared namespace
> ‘packages cannot conflict in a shared namespace’ is a bit vague.
> Are you referring to ‘profile collisions’?  About having multiple versions
> of the same package on the same system?  Guix (and nix) can handle the latter.
> When the ‘propagated-inputs’ are limited, installing multiple packages each using
> a different version of a package is possible.
>
> Is something missing from Guix here (please be very concrete)?
>
>>   at the cost of defeating an internal cache.
> I'm not sure what you mean here, can you illustrate this very concretely?
> And is something missing in Guix here?
>
>> prefers Semantic Versions
> If upstream doesn't do ‘semver’, and instead uses a versioning system like
> used by TeX (3.14, 3.141, 3.1415, 3.14159, ...), then Xiden cannot somehow
> let the package use ‘semver’ instead.  Unless you want to teach Xiden to
> convince upstream to change their versioning system?
>
>> These preferences are valid in Xiden's DSL for writing launchers.
> What is a launcher (compared to packages in Guix), and why would I want one?
> Why shouldn't I just install packages and run them ("guix package -i texmacs",
> start texmacs from some application chooser menu of the desktop environment).
>
>>   The invariants for each launcher can differ as much as Cargo and Vortex differ.
>>   What I wanted to do was allow users to declare their own invariants explicitly,
>>   but only when those invariants are wanted for subjective reasons.
> What are these invariants you're speaking of?  I don't think you're referring
> to, say, translation invariance (‘the laws of physics remain the same under a
> translation of the coordinate system’).
>
> Or do you mean something like ‘Invariant (computer science), an expression whose
> value doesn't change during program execution’ (<https://en.wikipedia.org/wiki/Invariant>)?
> But what are the ‘expressions’ in this case?
>
>>   I can't just write an overly-opinionated tool in this space because opinions
>>   have shelf-lives, and they can't interrupt our ability to get the content we
>>   need on demand, with our own safety guarentees.
> Is Guix too opinionated about something in particular?
>
>> Xiden launchers have advantages over shell scripts in that we can both still download
>> and install dependencies from the same servers, and create reproducible builds in terms
>> of the same (bit-identical) state on disk. We just differ on details that shouldn't impact
>> how we work. It also helps us patch holes faster, since if (say) a catalog maintainer tells
> What is a ‘catalog maintainer’ (maybe it's like someone with commit access to a ‘guix channel’?)
> Also, what ‘shell scripts’ are you referring to?  I've been doing well without writing any shell
> scripts.
>
>> us that a CHF was compromised, we can immediately revoke trust in the CHF independently
> Maybe guix could gain a command "guix style transition-hash OLD NEW" to automagically
> rewrite package definitions to use the new hash algorithm instead of the old.  It will
> need to download plenty of source code however, and it will entail a world-rebuild.
>
> Note that, in practice, signs of the brokenness of hashes appear well in advance of
> actual exploits.
>
>> in our own clients.
> What are these ‘clients’ you're speaking of?  I haven't been needing any ‘clients’ lately.
>
>>   The package definitions are all expressed in terms of what the launchers allow.
> If 'package definition' = (package definition as in Guix), then this is false,
> Guix doesn't have a notion of ‘launchers’ and doesn't depend on ‘Xiden’.
> What are you meaning, exactly?  Also, I haven't found any package definitions in Xiden,
> could you point me to any?
>
>> So what problem does this solve? I'm trying to preserve an element of walk-away power
>> in any tech community. Maybe your community goes in a questionable direction. Maybe your
>> desired software is in a different ecosystem.
> Guix channels can function independently of the main ‘guix’, augmenting ‘guix’ with
> additional packages, even if they contravene official Guix policies (e.g. non-free,
> bad quality ...), or simply if putting it in ‘guix proper’ takes to much time.
> There exist a few relatively well-known channels like that.
>
>>   Maybe the maintainers are abusive jerks.
> Technically, Guix has ‘maintainers’, but I wouldn't know where I could find the list.
> Guix (the package manager) doesn't have its own infrastructure.
> It does have official substitute servers, but you can ignore them.
> So I don't see ‘maintainers are abusive jerks’ as a plausible threat to Guix.
> If they become existent, they can easily be replaced.
>
>> Maybe you have a `leftpad`/`event-stream` incident and the developer is unable/unwilling
>> to patch the problem.
> In Guix, if someone pushed malware (*) to, say, a package on pypi.org, then we (Guix)
> would simply stop using new versions of that package from pypi.org.  If there's a healthy
> fork somewhere, we could switch over to that for new versions.
>
> (*) I could have confused the ‘`leftpad`/`event-stream` incident’ for another incident.
>
>> You'd probably want a way to escape to a new community or distribution model
>>   without
> ‘Escape to a new community’ --> only if the community as a whole are ‘abusive jerks’.
> If it's merely some maintainers, they can be removed or replaced.  If it's all the
> maintainers, I'm sure a few well-regarded members of the community could come together
> for a ‘coup’.  If it's the community as a whole, I don't see how Xiden could help,
> unless you mean I would move from Guix to Xiden?
>
> Why would I want to ‘escape to a new distribution model’?  Writing something like Guix
> from scratch seems a lot of work to me, I would rather fork Guix (with a few
> likewise-minded collaborators) than to start over again.  Or move to Debian maybe.
> I don't see what ‘Xiden’ could offer here.
>
>>   giving up content or doing complex integration work. If Xiden can do that, users
>>   don't have to depend on the same middlemen to share work.
> Is ‘middlemen’ = guix committers here?  Do you have concrete scenario to illustrate this,
>
>>   This is also better for
>>   developers because it doesn't oblige them to do everything their users want,
> Are we talking about upstream developers here, or guix(/Xiden) committers?
> How does this ‘obligation’ exist in the first place?
>
>>   while knowing users will adapt to inevitable distribution problems.
> I would rather not have to ‘adapt to problems’, I would rather that upstream has something
> that works (though I can understand if occasionally mistakes are made).
>
>>   Everyone's consent
>>   becomes explicit, such that Xiden's "real" invariant is that consent between any
>>   two parties be mutually compatible.
> Consent about what?
>
>> I hope to improve user freedom from this angle,
>>   and my approach is biased by my experience with Racket's limitations in this space.
> Why are you comparing it with ‘racket’ instead of, say, Debian or Guix?
> ‘racket’ is a Scheme implementation, not a package manager.
>
>> The subjective parts I talk about go further into details like what name canons,
>> versioning schemes, trusted certificate chains, and even specific approaches to an exact
>> diamond dependency. The objective parts include integrity checking, signature verification,
>> safety limits, automatic dependency resolution, patches, protocols, and other concerns that
>> a decent package manager normally cares about. By keeping these separate, all of the below
>> features become cheap to write, without sacrificing trust & safety in the abstract.
>>
>> Download archived GitHub repositories (source on Racket Discord atm)
> It can download GitHub repositories.  Why are you writing ‘archived’ here?
> Guix supports git repositories (see 'git-reference' in the guix manual), so
> I don't see what additional value Xiden brings here.  Can it also automagically build
> the software?  Also, why ‘GitHub repositories’ specifically?  Why not, say,
> the git repositories at <https://git.ngyro.com/>?
>
>> Manage Racket installations like how `nvm` manages Node.js installations [5]
> It would seem that your ‘racket-installation-manager’ [5] cannot resist a compromise
> of https://downloads.racket-lang.org/releases/.  Also, I don't see how it would be
> possibe to use ‘racket-installation-manager’ offline.  And how does it compile the
> racket packages?  If we're talking about ‘trust’ and ‘escaping to a new community’,
> maybe provide an option to override 'host-url-prefix'?  Currently, it's hardcoding
> the ‘official’ racket community.
>
>> Produce a self-hosted Xiden installations, with implicit trust for the running Xiden instance [6]
>> Control which exact artifacts mnust be produced deterministically [7]
> Can I reproduce the ‘artifact’ (= store item?) on a separate machine?
>
>> Force generated bindings to be `eq?` (This one haunts Racket's package managers today) [8]
> That seems like a Scheme thing to me, not a package manager thing.
>
>> Source package definitions from Pastebin to download content, digests,
>> and signatures from a public S3 bucket (This does not exist today. I'm spitballing because I know
>>   it would work)
> How can this be secure?
>
>> I hypothesize that Xiden can "abstract over" package managers in the same way that Racket can
>> abstract over languages. I think people are used to package managers deciding things for them.
> Please provide some sane defaults.  I would rather not have to decide everything myself,
> and be able to trust Guix(/Xiden?) to do something reasonable.
>
> Also, if you ‘abstract over package managers’, wouldn't you just end up with one ‘super
> package manager’?  Why not ‘manage’ everything with a single package manager,
> like e.g. guix?
>
>> So what is Xiden for? Well, it's not for deciding an SOP for you.
> Searching for SOP, I find <https://en.wikipedia.org/wiki/Same-origin_policy>.
> What were you referring to here?
>
>>   It's for giving everyone the option to change the SOP for themselves with minimal politics.
> What are these changes you're referring to?
>
>>> what you would like to bootstrap?
>> All binaries related to creating a GNU/Linux distribution, such that I can reproduce an exact
>> OS, Racket installation, and Xiden instance. I want a trusted initial state on GNU/Linux.
>>
>> To be clear, I don't need to do this for functional reasons. Xiden's already operational. I
>> just can't claim that you can trust a given Xiden instance with the same confidence as a
>>   Guix instance right now.
> If you want to get package definitions from guix, take a look at (guix inferior).
> Maybe you could port the relevant code of (guix inferior) to Racket, to let Xiden
> talk to Guix?   In particular, 'inferior-eval' sends an S-exp to the inferior Guix,
> which is evaluated there, and the result is sent back.
>
> Greetings,
> Maxime.



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-08-22 22:54     ` Philip McGrath
@ 2021-09-13  3:09       ` Christine Lemmer-Webber
  2021-10-02  3:32         ` Philip McGrath
  0 siblings, 1 reply; 17+ messages in thread
From: Christine Lemmer-Webber @ 2021-09-13  3:09 UTC (permalink / raw)
  To: Philip McGrath; +Cc: guix-devel

Philip McGrath <philip@philipmcgrath.com> writes:

> Hi Sage,
>
> On 8/22/21 5:53 PM, Sage Gerard wrote:
>> Thanks for the detailed answer!
>> It seems wise to adapt GNU Mes towards Racket or Chez Scheme instead
>> of
>> Guile to bring GNU's benefits to more Scheme and Racket programmers. Has
>> someone already tried something like that?
>
> I haven't tried Xiden yet, and I haven't done any concrete work toward
> this (I have been working on managing Racket packages with Guix), but 
> Christine Lemmer-Webber had floated the idea at some point of trying
> to integrate Racket and Guile.
>
> IIRC, I think what she's had in mind was trying to make a Guile
> backend for Racket along the lines of the Chez Scheme backend (or the
> BC backend, or experimental backends like Pycket).

Yes that's what I had in mind :)

> As I said, I haven't actually tried any of this, but, as I've thought
> about what might be involved, there are two things that have struck me 
> as downsides:
>
>  1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
>     Report)" (§6, p. 13) that, "If our task were to compile Racket to an
>     existing target, then we would not have achieved such a high degree
>     of compatibility. … we have taken the liberty of modifying Chez
>     Scheme to make it an easier target for Racket."
>
>     https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf
>
>     Presumably a Racket-on-Guile project would face the same trade-off,
>     where modifications to Guild, if Racket CS is a guide, could require
>     hard work over many years, and lesser compatibility would make the
>     result less useful.

At one point when I spoke to Matthew, he was very optimistic that the
work on porting on top of Chez would open Racket to running on top of
many other backends.  But yes... since there have been so many "custom"
modifications to Chez, it's easier to be skeptical about that these
days...

>  2. As you probably know, Racket programs can't generally use
>     Chez Scheme implemented libraries, because Chez Scheme effectively
>     is the "unsafe" layer of the Racket VM. For example, not all Racket
>     procedures are Chez Scheme procedures, and Racket's continuations
>     wrap Chez Scheme's to implement delimited and composable control,
>     threads, parameters, full continuation marks, etc.
>
>     For Racket CS, this isn't a great loss (there aren't so many
>     Chez-specific libraries, and portable libraries can run in Racket's
>     R6RS language), but, for a hypothetical Racket-on-Guile,
>     bidirectional interoperability would be a big attraction: imagine
>     Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
>     working together.
>
> If I were going to work on this, I'd start by looking at having Racket
> and Guile coexist as siblings with interoperability through their FFIs 
> level. Even better, eventually you could compile Guile to Racket
> linklets, so the two could coexist in the same primitive module
> system. There would probably always need to be something along the
> lines of require/typed to interoperate between the languages, since
> Guile has mutable pairs and an unusual approach to falsehood and
> nullity to let Scheme's #f and '() coexist with Emacs List's nil. (See 
> https://www.gnu.org/software/guile/manual/html_node/Nil.html).
>
>>>> I'm at the point where users are requesting a GNU/Linux distribution for
>>>> Xiden, such that Racket is the primary language for day-to-day
>>>> operation. I'm ignorant of the scope of work, and am unsure if I can do
>>>> it alone.
>
> To me, at least, the scope of the work in creating a new GNU/Linux
> distribution seems daunting. My hope would be that bringing Racket and 
> Guile closer together would let most if not all of the effort be shared.
>
> Hope some of this is useful, and best of luck!
>
> -Philip



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-09-13  3:09       ` Christine Lemmer-Webber
@ 2021-10-02  3:32         ` Philip McGrath
  2021-10-02 12:59           ` Christine Lemmer-Webber
  0 siblings, 1 reply; 17+ messages in thread
From: Philip McGrath @ 2021-10-02  3:32 UTC (permalink / raw)
  To: Christine Lemmer-Webber; +Cc: guix-devel

On 9/12/21 11:09 PM, Christine Lemmer-Webber wrote:
> Philip McGrath <philip@philipmcgrath.com> writes:
>> Christine Lemmer-Webber had floated the idea at some point of trying
>> to integrate Racket and Guile.
>>
>> IIRC, I think what she's had in mind was trying to make a Guile
>> backend for Racket along the lines of the Chez Scheme backend (or the
>> BC backend, or experimental backends like Pycket).
> 
> Yes that's what I had in mind :)

A few stray thoughts, all with the caveat that I haven't actually tried 
any of this ...

>> there are two things that have struck me
>> as downsides:
>>
>>   1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
>>      Report)" (§6, p. 13) that, "If our task were to compile Racket to an
>>      existing target, then we would not have achieved such a high degree
>>      of compatibility. … we have taken the liberty of modifying Chez
>>      Scheme to make it an easier target for Racket."
>>
>>      https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf
>>
>>      Presumably a Racket-on-Guile project would face the same trade-off,
>>      where modifications to Guild, if Racket CS is a guide, could require
>>      hard work over many years, and lesser compatibility would make the
>>      result less useful.
> 
> At one point when I spoke to Matthew, he was very optimistic that the
> work on porting on top of Chez would open Racket to running on top of
> many other backends.  But yes... since there have been so many "custom"
> modifications to Chez, it's easier to be skeptical about that these
> days...

I think there a few possible senses for what "running on top of many 
other backends" could mean. My impression overall is that it has gotten 
easier, but may not necessarily be easy.

The most clearly demonstrated is that it seems to be easier to port Chez 
Scheme to new architectures than to port Racket BC's VM. Racket CS 
supports the Apple M1 chip natively (and hopefully will support the 
Linux kernel on the M1 when that stabilizes), which Racket BC does not. 
Racket CS also fully supports some platforms (ARM in general, IIRC) on 
which Racket BC lacks support for futures, places, or the JIT. The most 
promising route to Racket on WASM also seems to be adding a WASM backend 
to the Chez Scheme compiler. (In fairness, there are also some 
architectures, like ppc64le, to which no one has ported Racket CS yet, 
for which Racket BC can offer at least some support via C.)

More generally, the "linklet" abstraction[1] seems to provide a much 
more clear boundary between the responsibilities of a backend 
implementation and the common frontend than existed before Racket 7.0. 
For example, the backend doesn't have to deal with macro expansion or 
even shadowing of variables: it just need to have some way to 
instantiate linklets (likely with further backend-specific compilation) 
and to provide a whole bunch of primitives. I believe I've heard Sam 
Tobin-Hochstadt say that linklets have made it possible for Pycket (the 
experimental Racket implementation in Reticulated Python) to run most 
Racket code.

[1]: https://docs.racket-lang.org/reference/linklets.html

Another benefit of linklets is that they've defined a clear path for 
implementing parts of Racket in Racket itself, like the regexp 
implementation on Racket CS. This seems like it could help a lot with 
supplying that very large number of primitives.

So I expect it would be entirely possible to implement a linklet-based 
Racket backend in Guile. I do suspect that getting production-quality 
performance and compatibility could be a lot of work. But my bigger 
concern is ...

>>   2. As you probably know, Racket programs can't generally use
>>      Chez Scheme implemented libraries, because Chez Scheme effectively
>>      is the "unsafe" layer of the Racket VM. For example, not all Racket
>>      procedures are Chez Scheme procedures, and Racket's continuations
>>      wrap Chez Scheme's to implement delimited and composable control,
>>      threads, parameters, full continuation marks, etc.
>>
>>      For Racket CS, this isn't a great loss (there aren't so many
>>      Chez-specific libraries, and portable libraries can run in Racket's
>>      R6RS language), but, for a hypothetical Racket-on-Guile,
>>      bidirectional interoperability would be a big attraction: imagine
>>      Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
>>      working together.

I don't think a Racket-on-Guile that worked like Racket-on-Chez would 
achieve the most interesting benefit (IMO) of bringing the two languages 
together: safe, bidirectional interoperability.

Probably there are many ways to achieve that goal. In principle, one 
could write a Racket-on-Guile implementation where arbitrary Guile could 
run without breaking Racket's invariants, but I expect that would make 
the task even harder: presumably that's why Racket-on-Chez doesn't work 
that way.

But if linklets are as good as an abstraction as they seem to be for 
compiling code in the presence of modules and macros---all of the 
complications of Scheme---then a more exciting direction, it seems to 
me, would be to adapt Guile's compiler tower[2] to, at some layer, 
produce linklets. If the existing Guile backend (which does have some 
strengths) became a linklet engine, it could run linklets generated by 
the Racket front-end. Linklets from the Guile frontend could likewise 
run on other Racket backends: Racket-and-Guile-on-Chez, Pycket, or any 
exciting backend someone creates in the future.

[2]: https://www.gnu.org/software/guile/manual/html_node/Compiler-Tower.html

It would probably still be a fair bit of work, particularly in figuring 
out exactly how the semantics of Guile modules correspond to Racket 
modules and in making interoperability convenient. I have no particular 
plans to work on this myself. But I think it would be cool :)

-Philip


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: How did you handle making a GNU/Linux distribution?
  2021-10-02  3:32         ` Philip McGrath
@ 2021-10-02 12:59           ` Christine Lemmer-Webber
  0 siblings, 0 replies; 17+ messages in thread
From: Christine Lemmer-Webber @ 2021-10-02 12:59 UTC (permalink / raw)
  To: Philip McGrath; +Cc: guix-devel

Philip McGrath <philip@philipmcgrath.com> writes:

> On 9/12/21 11:09 PM, Christine Lemmer-Webber wrote:
>> Philip McGrath <philip@philipmcgrath.com> writes:
>>> Christine Lemmer-Webber had floated the idea at some point of trying
>>> to integrate Racket and Guile.
>>>
>>> IIRC, I think what she's had in mind was trying to make a Guile
>>> backend for Racket along the lines of the Chez Scheme backend (or the
>>> BC backend, or experimental backends like Pycket).
>> Yes that's what I had in mind :)
>
> A few stray thoughts, all with the caveat that I haven't actually
> tried any of this ...
>
>>> there are two things that have struck me
>>> as downsides:
>>>
>>>   1. Flatt et al. say in "Rebuilding Racket on Chez Scheme (Experience
>>>      Report)" (§6, p. 13) that, "If our task were to compile Racket to an
>>>      existing target, then we would not have achieved such a high degree
>>>      of compatibility. … we have taken the liberty of modifying Chez
>>>      Scheme to make it an easier target for Racket."
>>>
>>>      https://www.cs.utah.edu/plt/publications/icfp19-fddkmstz.pdf
>>>
>>>      Presumably a Racket-on-Guile project would face the same trade-off,
>>>      where modifications to Guild, if Racket CS is a guide, could require
>>>      hard work over many years, and lesser compatibility would make the
>>>      result less useful.
>> At one point when I spoke to Matthew, he was very optimistic that
>> the
>> work on porting on top of Chez would open Racket to running on top of
>> many other backends.  But yes... since there have been so many "custom"
>> modifications to Chez, it's easier to be skeptical about that these
>> days...
>
> I think there a few possible senses for what "running on top of many
> other backends" could mean. My impression overall is that it has
> gotten easier, but may not necessarily be easy.
>
> The most clearly demonstrated is that it seems to be easier to port
> Chez Scheme to new architectures than to port Racket BC's VM. Racket
> CS supports the Apple M1 chip natively (and hopefully will support the 
> Linux kernel on the M1 when that stabilizes), which Racket BC does
> not. Racket CS also fully supports some platforms (ARM in general,
> IIRC) on which Racket BC lacks support for futures, places, or the
> JIT. The most promising route to Racket on WASM also seems to be
> adding a WASM backend to the Chez Scheme compiler. (In fairness, there
> are also some architectures, like ppc64le, to which no one has ported
> Racket CS yet, for which Racket BC can offer at least some support via
> C.)

Who will get to WASM first, Racket or Guile?  :)

> More generally, the "linklet" abstraction[1] seems to provide a much
> more clear boundary between the responsibilities of a backend 
> implementation and the common frontend than existed before Racket
> 7.0. For example, the backend doesn't have to deal with macro
> expansion or even shadowing of variables: it just need to have some
> way to instantiate linklets (likely with further backend-specific
> compilation) and to provide a whole bunch of primitives. I believe
> I've heard Sam Tobin-Hochstadt say that linklets have made it possible
> for Pycket (the experimental Racket implementation in Reticulated
> Python) to run most Racket code.
>
> [1]: https://docs.racket-lang.org/reference/linklets.html
>
> Another benefit of linklets is that they've defined a clear path for
> implementing parts of Racket in Racket itself, like the regexp 
> implementation on Racket CS. This seems like it could help a lot with
> supplying that very large number of primitives.
>
> So I expect it would be entirely possible to implement a linklet-based
> Racket backend in Guile. I do suspect that getting production-quality 
> performance and compatibility could be a lot of work. But my bigger
> concern is ...
>
>>>   2. As you probably know, Racket programs can't generally use
>>>      Chez Scheme implemented libraries, because Chez Scheme effectively
>>>      is the "unsafe" layer of the Racket VM. For example, not all Racket
>>>      procedures are Chez Scheme procedures, and Racket's continuations
>>>      wrap Chez Scheme's to implement delimited and composable control,
>>>      threads, parameters, full continuation marks, etc.
>>>
>>>      For Racket CS, this isn't a great loss (there aren't so many
>>>      Chez-specific libraries, and portable libraries can run in Racket's
>>>      R6RS language), but, for a hypothetical Racket-on-Guile,
>>>      bidirectional interoperability would be a big attraction: imagine
>>>      Guix, Shepherd, Mcron, syntax/parse, racket/contract, and more all
>>>      working together.
>
> I don't think a Racket-on-Guile that worked like Racket-on-Chez would
> achieve the most interesting benefit (IMO) of bringing the two
> languages together: safe, bidirectional interoperability.
>
> Probably there are many ways to achieve that goal. In principle, one
> could write a Racket-on-Guile implementation where arbitrary Guile
> could run without breaking Racket's invariants, but I expect that
> would make the task even harder: presumably that's why Racket-on-Chez
> doesn't work that way.
>
> But if linklets are as good as an abstraction as they seem to be for
> compiling code in the presence of modules and macros---all of the 
> complications of Scheme---then a more exciting direction, it seems to
> me, would be to adapt Guile's compiler tower[2] to, at some layer, 
> produce linklets. If the existing Guile backend (which does have some
> strengths) became a linklet engine, it could run linklets generated by 
> the Racket front-end. Linklets from the Guile frontend could likewise
> run on other Racket backends: Racket-and-Guile-on-Chez, Pycket, or any 
> exciting backend someone creates in the future.
>
> [2]: https://www.gnu.org/software/guile/manual/html_node/Compiler-Tower.html
>
> It would probably still be a fair bit of work, particularly in
> figuring out exactly how the semantics of Guile modules correspond to
> Racket modules and in making interoperability convenient. I have no
> particular plans to work on this myself. But I think it would be cool
> :)
>
> -Philip

Lots of intersesting stuff in here.  The compiler tower direction is
indeed something I've hoped could be explored.  But it's also something
not on my agenda immediately either. :)


^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2021-10-02 15:40 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-21 16:43 How did you handle making a GNU/Linux distribution? Sage Gerard
2021-08-21 21:18 ` Leo Famulari
2021-08-22 21:53   ` Sage Gerard
2021-08-22 22:54     ` Philip McGrath
2021-09-13  3:09       ` Christine Lemmer-Webber
2021-10-02  3:32         ` Philip McGrath
2021-10-02 12:59           ` Christine Lemmer-Webber
2021-08-23 10:43 ` Maxime Devos
2021-08-23 12:48   ` Sage Gerard
2021-08-23 15:38 ` zimoun
2021-08-23 17:30   ` Philip McGrath
2021-08-24 10:24     ` zimoun
2021-08-23 18:24   ` Sage Gerard
2021-08-23 19:23     ` Ryan Prior
2021-08-23 20:00       ` Sage Gerard
2021-08-24 16:42     ` Maxime Devos
2021-08-24 17:17       ` Sage Gerard

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).