From 912a30fa0e60495cb14b28ad5cf2ea9c3994f477 Mon Sep 17 00:00:00 2001 From: unmush Date: Sat, 28 Dec 2024 04:24:07 Subject: [PATCH] website: add bootstrapping-mono blog post * website/posts/bootstrapping-mono.md: new file. --- website/posts/bootstrapping-mono.md | 369 ++++++++++++++++++++++++++++ 1 file changed, 369 insertions(+) create mode 100644 website/posts/bootstrapping-mono.md diff --git a/website/posts/bootstrapping-mono.md b/website/posts/bootstrapping-mono.md new file mode 100644 index 0000000..a5deff4 --- /dev/null +++ b/website/posts/bootstrapping-mono.md @@ -0,0 +1,369 @@ +title: Adding a fully-bootstrapped Mono +date: 2024-12-28 04:17 +author: unmush +tags: Bootstrapping, Reproducible builds, C#, .NET, Security +--- + +We used to have a Mono package. It was introduced on August 8 2016 by commit +763b3d50b6249b43fceda51445bbeb1f5f5fd7d0, at Mono version 4.4.1.0, but it was +later [discovered](https://issues.guix.gnu.org/55026) in April of 2022 that the +release tarball that it was built from included prebuilt binaries. Further +research revealed that these binaries were not optional. Due to this, a +decision was made to remove the Mono package, carried out on September 1, 2022. + +We now once again have a Mono package, due to a [patch +series](https://issues.guix.gnu.org/74609) I submitted on November 29, which +after some revisions was committed on December 22. This patch series introduced +a full, 17-mono-package sequence that takes us from a mono-1.2.6 built fully +from source to mono-6.12.0 built fully from source, using only packages that +already have full bootstrap paths. I make no promise that this is the shortest +or most optimal path, but it exists and I have verified it works. + +As I've spent what is probably an unreasonable amount of time working toward +this, I thought I'd share some of my thoughts, experiences, and commentary. +Sorry in advance if it gets a bit rambly or lecture-ish. + +# Prologue + +I started down this road because someone I'm working on a project with decided +to depend on a C# package that requires C# 12.0 features, and my personal Mono +package based on the tarball releases (which include bootstrap binaries) only +went up to C# 7.0. This meant that the C# package in question de facto required +strictly Microsoft's (er, I mean, "the .NET foundation"'s) .NET implementation - +hereafter referred to as "dotnet" - and a very recent version no less. The +bootstrapping story with dotnet is [very +bad](https://github.com/dotnet/source-build/issues/1930); even beginning to +untangle it would probably require a relatively modern C# compiler, and +something that at least sort of understands MSBuild. And there's not much point +to "bootstrapping" dotnet from something that isn't bootstrapped itself. So I +figured I may as well start with Mono. + +# History + +While Mono is today probably the most well-known alternative to Microsoft's .NET +offerings, it is not the only one. Indeed, in the early 2000s there were at +least 2 competing free software implementations: Mono, and DotGNU's Portable.NET +(abbreviated pnet). They differed in goals, licenses, and methods: Portable.NET +was a GNU project concerned with, among other things, limiting the ability of +Microsoft to impose vendor lock-in via its proprietary .NET implementation and +software patents. As a GNU project, it used the GPL for its runtime and +compiler, and the GPL with a linking exception for its standard library, +pnetlib. Mono, on the other hand, used a mix of many copyleft and permissive +licenses: X11 for the standard library, GPL for the compiler (later +dual-licensed to add an X11 option), and LGPL for the runtime, with GPL and LGPL +code also offered "under commercial terms for when the GPL and the LGPL are not +suitable". In 2016 after its acquisition by Microsoft, the runtime was +relicensed to use the Expat (MIT) license. + +But perhaps most importantly to us, while Mono opted to write its C# compiler, +mcs, in... C#, Portable.NET's runtime and C# compiler were both written in C. +Portable.NET along with the entire DotGNU project (except for LibJIT) was +decommissioned in 2012, but the source is still available, and it still works +fine (with a few modifications for compatibility with newer versions of its +dependencies). In September of 2022, Adam Faiz [submitted +patches](https://issues.guix.gnu.org/57625) to package pnet and pnetlib, along +with one of their dependencies named treecc. These packages were based on the +last release of Portable.NET, version 0.8.0, released in 2007. I initially used +these packages as the basis for my bootstrap efforts, and even managed to get +mono-1.2.6 built using them, but later discovered that using a more recent +version from git made it much easier. For example, while pnet-0.8.0 can do +pointer arithmetic inside unsafe code blocks, it doesn't support the `+=` or +`-=` operators specifically, which requires lots of patching (after all, who +would use `x = x + y` when you could do `x += y`?). There are many other +similar improvements in the git version, so for this patch series I've decided +to go with pnet-git. + +# The start + +After building mono-1.2.6, I tried a few later versions, and the third or fourth +one would always fail with errors about missing methods. It turns out that the +reason for this is that, contrary to what their marketing suggests, C# and Java +are not "write once, run everywhere". This is because their compilers rely on +the details of the libraries that the program will be run with at compile-time. +This is used, for example, to do overload resolution. Suppose, for example, +that a certain implementation of the `==` operator is present in version 1.0 of +a library, and then in version 2.0 of a library a more specific implementation +is introduced. Now code that is compiled against version 2.0 may instead +automatically reference the more-specific implementation, as is in accordance +with the rules of C#. But when it is run with version 1.0, it will fail because +that implementation doesn't exist. In my case, for some reason the initial mcs +and core libraries being built to compile the rest of Mono were being compiled +against a 2.0 library and then run with a 1.0 library. It turns out that this +was because mcs uses mono's code for producing assemblies (.NET dlls and exes), +and mono decides which version to put in an assembly it writes based on "which +runtime version" is being used, and that version is decided at startup based +on... the version that was put in the assembly it is running. So for example, +mono-1.9.1 would produce 2.0 assemblies because mono-1.2.6 produced 2.0 +assemblies because pnet produced 2.0 assemblies. So I modified Mono's runtime +in mono-1.9.1 to allow for this version to be overridden via environment +variable, and set it to `v1.1.4322`, and things went a lot more smoothly after +that. + +From there on it was mostly the usual trial-and-error process of identifying +where things had bitrotted. I made sure to unvendor libgc wherever possible, +though eventually by mono-4.9.0 they explicitly dropped support in their +configure script for using any libgc other than what was bundled, so at that +point I switched to using their homebrewed sgen garbage collector. + +# A concerning development + +Once I got to mono-2.11.4, though, things took a turn for the interesting: Mono +started using git submodules, and the `(recursive? #t)` clones were all failing. +It turns out that this is because their submodules reference github.com using +the `git://` protocol. + +This is notable for a few reasons. + +First, github dropped support for the `git://` protocol in 2021, so +recursive clones won't work now. This means I have to explicitly list +out every submodule, its commit, and its sha256 hash, for every Mono +version until they switched to using http or https. mono-2.11.4 has +only 4 submodules, but that doesn't last for long: by mono-4.9.0 it has +14 submodules. A significant portion of these patches is just listing +these submodules and their hashes. It's a bit annoying. + +The more concerning reason, though, is *why* github dropped support for the +`git://` protocol: it is unencrypted and unauthenticated. This is mitigated +somewhat by the use of sha-1 hashes to identify commits in the referenced +submodules, putting a significant computational burden on anyone who would try +to alter what was fetched corresponding to a given submodule. Significantly +more risky, though, is the process of *updating* submodules that use `git://` +URLs. It is quite unlikely that a developer is going to independently clone one +of the submodules over https, navigate to a desirable commit, copy the sha-1 +hash, and manually update the submodule reference's commit. They're far more +likely to run `cd submodule; git pull; cd ..; git add submodule; git commit ...` +or an equivalent. + +Of course, any changes a network man-in-the-middle might try to make here would +still be reflected in the commit history, so even if a developer did that, they +or any of their fellow committers could spot anything strange or malicious and +point it out. Also, the changes couldn't be propagated to others trying to pull +them who weren't on a network path containing the MITM because the +potentially-malicious commit wouldn't be present in the real submodule's +repository. So the transparency of git clearly showing changes to text files, +combined with the fact that surely no git hosting platform would just allow +arbitrary entities to make whatever commits they want accessible under any +arbitrary repository URL, rather mitigate this security issue. + +This usage of `git://` URLs lasted all the way until September 28, 2021, when +github's removal of support for it forced the developers to change them to +https. + +# Meanwhile, in reality + +On November 28, 2016, Mono added a submodule named roslyn-binaries. +Unsurprisingly, it included binary blobs for Microsoft's Roslyn compiler (which +I believe had been open-sourced shortly prior). From here on, Mono's build +system would default to using these binaries for building on little-endian +systems (though another compiler could be specified with the `--with-csc` +configure flag). I happen to know that it is extremely unlikely that many Mono +developers used this configure flag. I know this because the 5.0 series is an +absolute pain in the neck to build from source, because they consistently depend +on new C# features *before* they implement them. + +To go on a brief tangent: does anyone remember back when youtube-dl was +temporarily taken down from github due to the RIAA's DMCA request? Many were +unhappy about that. One such unhappy person made news when they [made the full +contents of youtube-dl's repository available to access through the DMCA request +repository](https://gist.github.com/lrvick/02088ee5466ca51116bdaf1e709ddd7c). +It turns out that there are many actions that one can take on github that will +make arbitrary commits available under arbitrary repository URLs. + +So, in reality, for the span of time from November 28, 2016 to +September 28, 2021, anybody sitting on the network path between github +and any Mono developer updating the roslyn-binaries submodule could +decide on any arbitrary new commit to be used. Of course, merely +inspecting the diff for the commit will reveal nothing of use, because +the contents are binary blobs. And not only are these blobs those of a +compiler, they are the blobs of a compiler that is sure to be used to +compile another compiler, which will then be redistributed as an opaque, +non-bootstrappable binary blob to be used for compiling other compilers. + +You would be hard-pressed to find a more fertile breeding ground for [Ken +Thompson / Trusting Trust +attacks](https://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf). +If every agent of the NSA (and whatever other agencies, including those of other +countries, had access to the appropriate network traffic) somehow failed to +capitalize on 6 years of opportunity to compromise an entire software ecosystem +using only a basic MITM of unencrypted traffic, they deserve to be sacked. +Whether such an attack actually occurred or not, this is a case study in +carelessness and why bootstrappability is so important; discovering all this +made me quite worried about having used a Mono version built from blobs +previously, and has convinced me that, as time-wasting and tedious as this +project has been, it is nevertheless probably an important one. + +# Another note on roslyn-binaries + +If you're going to write a self-hosting compiler, the least you can do is keep +it self-hosting. Deciding to write a self-hosting compiler is a valid choice, +of course, with its own merits and demerits, but there is something bitterly +poetic about Mono starting out requiring specifically Microsoft's C# compiler in +order to build (Mono did its initial bootstrapping using Microsoft's proprietary +csc), achieving independence through self-hosting, being acquired by Microsoft, +and thereafter coming crawling back to Microsoft's C# compiler once more before +eventually dying. + +The funny thing is that it's not even necessary. The dependencies on new C# +features are all in Mono's standard library (which increasingly borrowed code +from Microsoft's corefx library), not in Mono's compiler. + +# More binary submodules? + +Even before roslyn-binaries, there was binary-reference-assemblies, which +contained prebuilt "reference" blobs for the various versions of the standard +libraries. These exist, I assume, precisely because of the library +incompatibility problems regarding overloading that I mentioned earlier. While +later versions of Mono included sources and a build system for producing these +reference binaries, mono-4.9.0 and earlier did not. Mono's build system still +demanded *something* to install, though, so I told it to use the real standard +library of the input Mono version. When I did get to a Mono version that at +least claimed to support regenerating the reference binaries, I found that it +didn't work with mcs due to differences in which libraries had to be referenced, +so I had to patch it to add a bunch of references determined through trial and +error. + +The xunit-binaries submodule was also added sometime before mono-5.1.0. This +dependency makes it impossible to run the full test suite without binary blobs. +Presumably for this reason, Debian elects to only run tests within the +`mono/mini/` and `mono/tests/` subdirectories. For my part, I've disabled all +tests except for those of mono-6.12.0, the final version, limited to the two +aforementioned subdirectories. This is because it would take extra time for the +builds, because several of the tests depend on binary blobs bundled into the +Mono repository itself (which my thorough cleaning of all dlls and exes from the +sources removes), because a large chunk of the tests depend on binary blobs in +xunit-binaries in later versions, and because "expect some test failures" is +part of the Mono documentation and I don't have the time to figure out for the +Mono developers every reason why each of 17 versions of their test suite is +broken. + +# The long march through the 5.0s + +The 5.0 series was when Microsoft acquired Mono, and it shows. You'll notice I +needed to introduce `pre-` packages for various versions because in several +cases a tagged release could not build the following tagged release. For that +matter, they couldn't build the `pre-` package either, but it at least took +fewer patches to get them working. The reason for this is that Mono added a +dependency on Microsoft's corefx library source code, and it usually started +using C# features well before mcs was able to compile them. Because of this, +despite taking 8 versions to get from 1.2.6 to 4.9.0, it took another 8 versions +to get through the 5.0 series, and 5 of them required nontrivial patching to +massage the source into a form compilable by mcs. + +# The final stretch + +Eventually I realized that the dependencies on new features were all coming from +corefx, not from Mono's compiler. Consequently, the only reason for this +particular bootstrap-hostile ordering of builds is that it happened to be the +order the Mono devs committed things. So I just cherry-picked every commit I +could find touching `mcs/mcs` (magit was quite useful for this) and applied it +to 5.10.0 to produce what is essentially the 6.12.0 compiler, then used it to +jump straight to building 6.12.0. + +Use of this technique earlier on in the bootstrap process may be of interest to +anyone looking to shorten the chain of packages. + +# The finishing touches + +My initial goal was to package dotnet, and I had tried to progress toward that +from mono-4.9.0 for a period, but with no success. During that time, though, I +did encounter a bug in Mono's xbuild condition parser, which I wrote a patch +for, and included in mono-6.12.0. + +I also discovered that xbuild would wrongly complain about missing references +even when the proper assemblies were in `MONO_PATH` or `MONO_GAC_PREFIX`, +because xbuild would erroneously only consider the path +`/gnu/store/...mono-6.12.0/lib/mono/gac` when looking for global assembly +caches, completely ignoring `MONO_GAC_PREFIX`. So I wrote a patch to fix that, +and included it in mono-6.12.0. + +Having witnessed how much nicer it is to package things that use rpath / runpath +than things that use environment variables (like python) and therefore require +constant wrapping of executables and use of propagated-inputs, I devised a patch +that would extend Mono's per-assembly config files to support a `` +element. For example, if you have a file `/tmp/dir2/test2.exe`, and there is +also a file `/tmp/dir2/test2.exe.config`, and its contents are + +``` xml + + + + +``` + +and it references `test1.dll`, it will first look for it at +`/tmp/dir1/test1.dll`. Note that, of course, `test1.dll` still needs to be +accessible to the compiler at compile-time through `MONO_PATH` or an +explicitly-specified path passed on the mcs command line. + +It is my hope that this feature will be of use to anybody interested in +developing a build system. + +# Future work + +Mono had several difficult points in bootstrapping and packaging, but at the end +of the day it still met the basic description of a software package: +well-defined environment-supplied inputs and sources, a user-supplied install +prefix, and files installed under that prefix. + +The dotnet world is an entirely different beast. The first step of most build +systems I have encountered from that realm is downloading an entire toolchain, +among other dependencies, as a binary blob. They heavily depend on the exact +packages they specify being available exactly where they say to install them. +There is no "install", there are no "install directories" to my knowledge. A +build that doesn't contact nuget.org is an aberration. I am at a loss how to +build these things, much less package them. I badly need help. + +There are also some portability issues with the current bootstrap path. While +Portable.NET can fall back to an interpreter written in C where LibJIT isn't +supported, old versions of Mono have no such capability. Strictly speaking, +there is some bitrotted code for an interpreter that used to work, but has +stopped working by mono-1.2.6. It was left unmaintained until it was eventually +removed in 2014, only to be [revived in +2017](https://www.mono-project.com/news/2017/11/13/mono-interpreter/). This +poses a dilemma for anybody wanting to bootstrap Mono on a platform that wasn't +supported by mono-1.2.6's JIT compiler. There are a number of possible ways to +try resolving this, ranging from backporting the new interpreter, to fixing up +the old one for every version prior to the new interpreter, to forward-porting +the old compiler and class libraries to the new interpreter, etc. + +The most interesting option, though, in my opinion, would be to port mcs to +Portable.NET. This would achieve the intended portability, while also allowing +individual builds to be much faster since we're only building mcs, not the +runtime and class library each time. It would also allow us to make much bigger +version jumps, since, as we discovered earlier, many of the new C# feature +dependencies in Mono come from the class library rather than the compiler. Such +a shortened bootstrap could also make the bootstrap path more appealing for +other distributions to use instead of binaries. + +# Closing thoughts + +> "You wish now that our places had been exchanged. That I had died, and DotGNU +> had lived?" +> +> "... Yes. I wish that." + +Maintenance of Mono was recently transferred over to WineHQ. With that +announcement this statement was placed at https://www.mono-project.com: + +> "We want to recognize that the Mono Project was the first .NET implementation +> on Android, iOS, Linux, and other operating systems. The Mono Project was a +> trailblazer for the .NET platform across many operating systems. It helped +> make cross-platform .NET a reality and enabled .NET in many new places and we +> appreciate the work of those who came before us." + +I would like to clarify that, [according to Miguel de Icaza +himself](https://www.mono-project.com/archived/mailpostearlystory/), DotGNU +"started working on the system about the same time". According to [this DotGNU +newsletter](https://lwn.net/2002/0103/a/dotgnu.php3), Portable.NET began "in +January 2001". While it's unclear exactly when Portable.NET reached various +milestones, and the significance of the various milestones varies somewhat (for +example, Mono probably does not care that Portable.NET also includes a Java and +C compiler), I think that there is cause to dispute the claim that Mono was "the +first" .NET implementation on Linux. + +On a related note, if we haven't looked at the possibility of using Portable.NET +in the Java bootstrap process, it may be worth visiting at some point. + +Thank you to the DotGNU project, for the .NET implementation that made this +bootstrap possible, Adam Faiz, for the initial packaging of it that let me jump +straight in, the Mono project, for... Mono, and you, for your time. -- 2.45.2