From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Nieuwenhuizen Subject: Re: bootstrap: i686-linux now builds without binutils, gcc seeds Date: Mon, 17 Sep 2018 21:47:21 +0200 Message-ID: <8736u752k6.fsf@gnu.org> References: <87h8jats4r.fsf@gnu.org> <87k1nlfdp7.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35345) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g1zUc-0008JR-RI for guix-devel@gnu.org; Mon, 17 Sep 2018 15:47:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g1zUb-0005Hi-5M for guix-devel@gnu.org; Mon, 17 Sep 2018 15:47:34 -0400 In-Reply-To: <87k1nlfdp7.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Sun, 16 Sep 2018 21:24:20 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Ludovic Court=C3=A8s writes: > It=E2=80=99s taken me a while but I successfully built /gnu/store/58q6748= dydqxrhrw8xn2gsf60djl6gvv-hello-2.10 > for i686 from commit 5d8b7131c23d2285dd3546c59022dd5953508943 of > =E2=80=98wip-bootstrap=E2=80=99. :-) That's great! > $ guix hash -r /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10 > 005zi21nvhmygjam6vsvkgmw9awsmyacz6a65naxcblrpbra1g94 And...you found the lucky numbers ;-) > Previously we discussed that =E2=80=9C-s i686-linux=E2=80=9D on x86_64 wo= uld lead to a > different graph compared to a native i686-linux run. Is it still the > case? It looks like 80bd4a995 does the right thing in that respect. Almost...that is to say for all packages up to gnu-make-boot0 and all packages mentioned below: `Remove bootstrap leaks.' As discussed on IRC, I created bug https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D32749 > The =E2=80=9Cinteresting commits=E2=80=9D are these: > > 5d8b7131c * gnu: mes: Oops, use mirror://gnu download url. > 1ff6d26ac * bootstrap: Replace GNU toolchain seeds with Mes for i686-li= nux. > ca8895bdb * bootstrap: Add Mes bootstrap. > be65bf958 * gnu: Add Mes bootstrap seeds. > 45856f886 * gnu: Add linux-libre-headers-bootstrap-tarball. > cd9830fcd * guix: copy-linux-headers: Prepare for Mes bootstrap. > 6265040e7 * guix: package-from-tarball: Allow PROGRAM-TO-TEST to be #f. > 8d9c16ed3 * gnu: texinfo-boot0: Remove bootstrap leaks. > f9a76a79c * gnu: bison-boot0: Remove bootstrap leaks. > 12394cc83 * gnu: m4-boot0: New variable. > 6891da335 * gnu: perl-boot0: Remove bootstrap leaks. > f933b162c * gnu: file-boot0: Remove bootstrap leaks. > 9c0912a20 * gnu: findutils-boot0: Remove bootstrap leaks. > 53616d896 * gnu: diffutils-boot0: Remove bootstrap leaks. > 7503cee76 * bootstrap: %bootstrap-inputs+toolchain: Prepare for Mes boo= tstrap. > 80bd4a995 * bootstrap: %bootstrap-inputs: Prepare for Mes bootstrap. So we'll probably want to look into that bug and either remove all the `Remove bootstrap leaks' patches, or rewrite the *-boot0 packages from getext-boot0 onward (gcc-boot0-wrapped ld-wrapper-boot3 glibc-final binutils-final zlib-final gcc-final bash-final guile-final glibc-utf8-locales-final gnu-make-final coreutils-final grep-final sed-final). > I looked at the result and overall it LGTM! So I think the next step is > to rebase the branch on =E2=80=98core-updates=E2=80=99 (or merge it) and = rename it > =E2=80=98core-updates-next=E2=80=99. WDYT? Ricardo? Amazing news! I think it's best to "wait" until the `bootstrap leak' thing is resolved; wip-bootstrap once more? > Some comments on things that I think could be improved, in no particular > order: > > =E2=80=A2 It would be good to update the =E2=80=9CBootstrapping=E2=80= =9D section of the > manual=E2=80=94no urgency though. It would be useful both to Guix > developers and to outsiders interested in bootstrapping > (Bootstrappable, Reproducible Builds, maybe projects like GNU/Linux > From Scratch, etc.) I have added this text to wip-bootstrap, image not shown inline here, WDYT? --8<---------------cut here---------------start------------->8--- diff --git a/doc/guix.texi b/doc/guix.texi index 19a497c74..5a2cc8155 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -210,6 +210,7 @@ GNU Distribution * Package Modules:: Packages from the programmer's viewpoint. * Packaging Guidelines:: Growing the distribution. * Bootstrapping:: GNU/Linux built from scratch. +* Reduced Binary Seed Bootstrap:: A Bootstrap worthy of GNU. * Porting:: Targeting another platform or kernel. =20 System Installation @@ -8657,6 +8658,7 @@ For information on porting to other architectures or = kernels, * Package Modules:: Packages from the programmer's viewpoint. * Packaging Guidelines:: Growing the distribution. * Bootstrapping:: GNU/Linux built from scratch. +* Reduced Binary Seed Bootstrap:: A Bootstrap worthy of GNU. * Porting:: Targeting another platform or kernel. @end menu =20 @@ -23447,6 +23449,9 @@ Binutils, libc, and the other packages mentioned ab= ove---the These bootstrap binaries are ``taken for granted'', though we can also re-create them if needed (more on that later). =20 +For @code{i686-linux} the Guix bootstrap process is more elaborate, +@pxref{Reduced Binary Seed Bootstrap}. + @unnumberedsubsec Preparing to Use the Bootstrap Binaries =20 @c As of Emacs 24.3, Info-mode displays the image, but since it's a @@ -23600,6 +23605,70 @@ bootstrap GCC with a sequence of assemblers, inter= preters, and compilers of increasing complexity, which could be built from source starting from a simple and auditable assembler. Your help is welcome! =20 +@node Reduced Binary Seed Bootstrap +@section The Reduced Binary Seed Bootstrap + +Guix---like other GNU/Linux distributions---is traditionally bootstrapped = from +from a set of bootstrap binaries: Bourne shell, command-line tools provide= d by +GNU Coreutils, Awk, Findutils, `sed', and `grep' and Guile, GCC, Binutils,= and +the GNU C Library (@pxref{Bootstrapping}). Usually, these bootstrap-binar= ies +are ``taken for granted.'' + +What does this mean, really? By taking these binaries for granted, trusti= ng +Guix depends on the trusting these binaries to be correct and clean. Ther= ein +lies a problem: the current combined size of these bootstrap-binaries is a= bout +250MB (@pxref{Bootstrappable Builds,,, mes, Mes Reference Manual}). Audit= ing +or even inspecting these is next to impossible. + +For @code{i686-linux}, Guix now features a ``Reduced Binary Seed'' bootstr= ap +@footnote{We would like to say: ``Full Source Bootstrap'' and while we are +working towards that it would be a hyperbole to use that term for what we = do +now.}. + +The Reduced Binary Seed bootstrap removes the most critical tools---from a +trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU= C +Library are replaced by: @code{mescc-tools-seed} (a tiny assembler and lin= ker) +@code{mes-seed} (a small Scheme Interpreter and a C compiler writen in Sch= eme) +and @code{tinycc-seed} (the Mes C Library, built for TinyCC). Using these= new +binary seeds and a new set of +@c +package descriptions@footnote{@c +mescc-tools-boot, +nyacc-boot, +mes-boot, +tcc-boot0, +tcc-boot, +make-mesboot0, +diffutils-mesboot, +binutils-mesboot0, +gcc-core-mesboot, +mesboot-headers, +glibc-mesboot0, +gcc-mesboot0, +binutils-mesboot, +make-mesboot, +gcc-mesboot1, +gcc-mesboot1-wrapper, +glibc-headers-mesboot, +glibc-mesboot, +gcc-mesboot, +gcc-mesboot-wrapper. +} +@c +the ``missing'' Binutils, and the GNU C Library are built, from source. F= rom +here on the more traditional bootstrap process resumes. This approach has +reduced the bootstrap binaries in size to about 130MB. Work is ongoing to +reduce this further. If you are interested, join us on @code{#boottrappab= le} +on the Freenode IRC network. + +@c ./pre-inst-env guix graph --type=3Dbag -e '(begin (use-modules (guix pa= ckages)) (%current-system "i686-linux") (@@ (gnu packages commencement) gcc= -mesboot))' > doc/images/gcc-mesboot-bag-graph.dot +@c dot -T png doc/images/gcc-mesboot-bag-graph.dot > doc/images/gcc-mesboo= t-bag-graph.png + +Below is the generated dependency graph to for @code{gcc-mesboot} that bui= lds +the rest of GuixSD. + +@image{images/gcc-mesboot-bag-graph,6in,,Dependency graph of the gcc-mesbo= ot} + =20 @node Porting @section Porting to a New Platform --8<---------------cut here---------------end--------------->8--- > =E2=80=A2 I think all the =E2=80=9Cmesboot=E2=80=9D & co. packages in c= ommencement.scam can be > made private=E2=80=93i.e., changing =E2=80=98define-public=E2=80=99 t= o =E2=80=98define=E2=80=99. I agree; I kept gcc-mesboot (the final gcc) public, is that OK? > =E2=80=A2 There=E2=80=99s a couple of tests (for example in tests/debug= -link.scm) that > rely on %gcc-bootstrap, on the assumption that building it is > cheap. We should double-check that these are still okay on i686. > > =E2=80=A2 IWBN to add comments for gcc-mesboot1-wrapper, > glibc-headers-mesboot, %bootstrap-inputs+toolchain, just a line or > two giving some context. Done. Adding the comment for gcc-mesboot1-wrapper, I wonder if this package is currently necessary. I "played" (ahum) a lot with building gcc-mesboot1 (the first 4.7.4) already using --enable-shared. At the moment it's built using --disable-shared and gcc-mesboot (4.9.4) is the first gcc built using --enable shared. I probably kept gcc-mesboot1-wrapper even if it was no longer needed, because adding/removing it is also cumbersome. I'm testing right now to see if we can remove gcc-mesboot1-wrapper (and will do so if it's not currently needed); that information will help getting the comment more informational if it's still needed. > =E2=80=A2 I saw a couple of hanging parens in commencement.scm, they wa= nt to > be next to their friends. ;-) Oh, how sad :-( Fixed! > =E2=80=A2 Minor style issues: > `(,@(substitute-keyword-arguments =E2=80=A6)) =E2=86=92 (substitute-k= eyword-arguments =E2=80=A6) Sure, removed. > (zero? (system* =E2=80=A6)) =E2=86=92 (invoke =E2=80=A6) Done, ah...I still overlooked some of these (wip-bootstrap began before `invoke' I think). Still have some (zero? (system " ....")). > =E2=80=A2 Could you add a couple of lines of explanation at the top of = the new > gnu/packages/patches/*.patch files, as we do for other patches? > Some of them could also be simplified; for instance > =E2=80=98glibc-boot-2.2.5.patch=E2=80=99 contains the diff of what lo= oks like a > leftover file. I'm looking into this. I tried to "document" the problems that I encountered by capturing the build errors in a file called `adiff'. That was an ugly but convenient hack, I agree this should removed and put into words. > =E2=80=A2 For =E2=80=98binutils-mesboot0=E2=80=99: package/inherit =E2= =86=92 package (inherit =E2=80=A6) > since replacement for =E2=80=98binutils=E2=80=99 certainly won=E2=80= =99t apply to > =E2=80=98binutils-mesboot0=E2=80=99. OK. > =E2=80=A2 For commit 80bd4a995 for instance, instead of =E2=80=9CPrepar= e for Mes > bootstrap=E2=80=9D, the message could be =E2=80=9CWrap input lists in= to thunks.=E2=80=9D or > something like that to clarify what the actual change is. Nice. Put that in a TODO for the next squash of wip-bootstrap. > I noticed that: > > ./pre-inst-env guix graph -e '(begin (use-modules (guix utils)) (%curre= nt-system "i686-linux")(@@ (gnu packages commencement) gnu-make-boot0))' | = dot -Tps > t.ps > > shows lots of repetitions, which defeats eq? memoization, but the > problem is not new; I=E2=80=99ll try to look into fixing this. Great, thanks. Yes, I wondered about that! > As we were discussing on IRC, we=E2=80=99ll have to figure out what the n= ext > step should be, once this branch has become =E2=80=98core-updates-next=E2= =80=99. One > interesting bit is to remove the mes.M1 seed, as you wrote. This is certainly something that Jeremiah Orians will be working on, even if not directly by helping with the mes.M2 rewrite, certainly indirectly by creating M2-Planet v2.0. This has a timeframe of about half a year. > Another one might be to do x86_64 bootstrapping (using i686 binaries > at the beginning of the graph, until we reach gcc@4.9, at which point > we should be able to build x86_64 binaries; though maybe I=E2=80=99m just= too > naive here :-)). Yes, I think G=C3=A1bor wanted to look into this, he had the same kind of remark on IRC, G=C3=A1bor? I'm also working on a wip-x86_64 branch for Mes; this has a timeframe of at least a couple of months (well, unless our resources magically multiply :-). Even when Mes is 64 bits, we would need to revise the whole bootstrap chain (think: patch tcc, replace gcc-2.95 with gcc-3.x/4.x?). > Another low-hanging fruit (I think!) is using Gash instead of Bash, > though that=E2=80=99s completely orthogonal. Yes! I'd love to get this discussion going too, and start to do some planning. I integrated most of bournish into Gash, and then also Timothy's Geesh came on my radar. Interesting puzzle this...Timothy? > Thank you for the amazing work, and again apologies for taking so long > to get back to you! Thanks! To be fair, although wip-bootstrap has been "mostly working" for quite some time now, I only got wip-bootstrap in a somewhat reviewable state and fixed the integration with Guix since a week. janneke --=20 Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar=C2=AE http://AvatarAcademy.com