all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [bug#33038] bootstrap: Regeneration of Mes bootstrap seeds.
@ 2018-10-14  8:50 Jan Nieuwenhuizen
  2018-10-14  8:58 ` [bug#33038] [PATCH 1/6] doc: Move `Reduced Binary Seed Bootstrap' into `Bootstrapping' Jan Nieuwenhuizen
  0 siblings, 1 reply; 49+ messages in thread
From: Jan Nieuwenhuizen @ 2018-10-14  8:50 UTC (permalink / raw)
  To: 33038

Hi!

In order to take care of these TODOs

TODO:
  * regenerate Mes bootstrap seeds, put them on ftp.gnu.org
  * slow tests: "gnu-build-system", "union-build"
  * doc: make "Reduced Binary Seed Bootstrap" subsection in Bootstrapping
  * doc: update “Preparing to Use the Bootstrap Binaries”
  * doc: update “Reducing the Set of Bootstrap Binaries”

I replaced the opaque, manually created %mes-seed and %tinycc-seed by
new pacckages: mes-stripped, mes-stripped-tarball, and %bootstrap-mes.

That allowed me to modify the remaining documentation sections too.
Sending a patch series, also available from

    https://gitlab.com/janneke/guix @core-updates-next
    
WDYT?

Greetings,
janneke

^ permalink raw reply	[flat|nested] 49+ messages in thread
* Re: Preparing the reduced bootstrap tarballs
@ 2018-11-15 20:39 Jeremiah
  2018-11-16 18:27 ` Ludovic Courtès
  0 siblings, 1 reply; 49+ messages in thread
From: Jeremiah @ 2018-11-15 20:39 UTC (permalink / raw)
  To: guix-devel



> Indeed.  "mes-seed" and "tinycc-seed" are remnants of the past; the only
> things we need are
>
>> What we would need here is something to build the things listed in
>> ‘%bootstrap-inputs’, namely:
>> ‘linux-libre-headers-stripped-4.14.26-i686-linux.tar.xz’ (easy :-)),
>> ‘mescc-tools-seed-XYZ.tar.gz’, and
>> ‘mes-stripped-0.18-0.08f04f5-i686-linux.tar.xz’

That is correct (With mescc-tools-seed adding steps and removing
binaries over time)

> So if you like, please make that change.  There is only one little
> thing: I have no (scripted) recipe to create mescc-tools-seed-XYZ.  But
> wait: I have a great excuse for that...I was too lazy or too sloppy.

I do, in mescc-tools-seed; the script bootstrap.sh when run with the
option "sin" will build the mescc-tools-seed binaries using mescc-tools.
The .M1 files are always generated by cc_x86.s using the C source files.


> The thing is, I used to build mescc-tools-seed, mes-seed and tinycc-seed
> manually from the mes+mescc+tinycc source trees.
I've been building them from the stage0 bootstrap tree (which as you can
see is rather trivial:
https://github.com/oriansj/talk-notes/blob/master/Current%20bootstrap%20map.dot
)


> Jeremiah Orians is
> working to remove any need for mescc-tools-seed (esp. the forward
> dependency on Mes) but I don't think we're there yet.
We have eliminated the forward dependency on Mes for the creation of the
mescc-tools-seed already


> Anyway, I think we/I will have to put some work into scripting
> mescc-tools-seed or otherwise changing the mescc-tools-boot build.
Already done in bash and kaem but not in guix yet (Should be trivial)

> WDYT?
I think we will end up having several versions of mescc-tools-seed; as
each architecture guix supports will end up needing a variant if we plan
on keeping them small. (I also have no idea how to make a multi-arch fat
elf binary)

I am also curious if there is any demand for the stripped versions of
mescc-tools-seed as those binaries are nearly half the size.

>> (do we really need an x86_64 version of this Mes?).
> No, I don't think so.  I added it esp. to get a preview and enable
> future development of pure x86_64 bootstrap; but dependency-wise we
> should be able to drop it!
also, AMD64 does support i386 binaries without issue

- Jeremiah

^ permalink raw reply	[flat|nested] 49+ messages in thread
* Re: Preparing the reduced bootstrap tarballs
@ 2018-11-17 14:27 Jeremiah
  2018-11-17 23:14 ` Mark H Weaver
  2018-11-19 18:54 ` Giovanni Biscuolo
  0 siblings, 2 replies; 49+ messages in thread
From: Jeremiah @ 2018-11-17 14:27 UTC (permalink / raw)
  To: guix-devel

> I think it's important that the new bootstrap-tarballs be
> bit-reproducible, such that they can be independently verified by anyone
> who wishes to do so.

Every Piece below M2-Planet has always been bit-reproducible.
In fact, each piece is designed in a way that you could by hand predict
what the resulting binary must be after any change.

and once I finally complete stage0; you would also have the blueprints
for making the virtual machine in hardware, hand toggle in the bits for
the hex0-monitor and have absolute proof that no trusting trust or Nexus
Intruder Class attacks have occurred in the creation of the binaries.

Every issue anyone is willing to bring, I will publicly address until
all bootstrap roots (even on arbitrary hardware) lead to the proof that
these binaries are perfectly reproducible and that they only behave in
the manner explicitly specified by the standards to which they conform.


> In particular, *I* would like to independently verify them, on my own
> laptops where I have avoided using binary substitutes for a long time,
> and which I keep with me at all times.
Already done; here are the steps currently for bootstrapping the
mescc-tools-seed and M2-Planet seed.M1:
git clone 'https://git.savannah.nongnu.org/git/stage0.git'
cd stage0
make test
cd ..
git clone 'https://git.savannah.nongnu.org/git/mescc-tools.git'
cd mescc-tools
make test
cd ..
git clone 'https://github.com/oriansj/mescc-tools-seed.git'
cd mescc-tools-seed
./bootstrap.sh sin

To generate the M2-Planet seed.M1 you need to either export
mescc-tools-seed's blood-elf, M1 and hex2 or mescc-tools (via copying
into your path or doing make install) then the steps to generate are as
follows:
git clone 'https://github.com/oriansj/M2-Planet.git'
cd M2-Planet
./bootstrap.sh refresh

Now you are done


> My hope until now is that when we generated our existing bootstrap
> binaries in 2013, Guix was too marginal a project to attract the
> attention of hackers who might wish to compromise our bootstrap.  In
> 2018, as Guix has become more popular, we might well be considered a
> worthy target of such efforts.

I like to go with the assumption that every binary is already
compromised; but by going back to the basics we can find and rip out
every single hook until we are finally secure.

I don't trust any hardware I can't or didn't make myself.
And the only root of trust we have is the ability to work as a
community, giving every member the ability to independently check our
assumptions and point out our mistakes. We will have false starts and
failures of imagination but we by working together will make us all a
dream that is too hard to achieve alone but easy now that we have each
other helping us all strive to a brighter future.

-Jeremiah

^ permalink raw reply	[flat|nested] 49+ messages in thread
* Re: Preparing the reduced bootstrap tarballs
@ 2018-11-18 12:56 Jeremiah
  2018-11-18 18:27 ` Mark H Weaver
  0 siblings, 1 reply; 49+ messages in thread
From: Jeremiah @ 2018-11-18 12:56 UTC (permalink / raw)
  To: guix-devel

> However, my impression (correct me if I'm wrong) is that we are not yet
> able to bootstrap Guix exclusively from M2-Planet.
That is correct as the step of bootstrapping MesCC from M2-Planet is not
yet complete.
However once that is done, we can leverage Mes.c and gash to complete
the bootstrap of guix from that trusted reproducible source in a
reproducible fashion.

> For example, unless
> I'm mistaken, we still need Guile in our bootstrap, and I'm guessing
> that we are not yet able to build Guile exclusively from M2-Planet.
> Is that right?
We don't need it, so much as it is people wishing to avoid tedious work.
We already can bootstrap kaem without any shells or interpreters and it
can be used to run shell scripts that can perform the rest of the
bootstrap of a lisp or a proper shell.

I think because that work is less of a technical challenge that it has
been skipped.


> My only point is that if we cannot yet avoid blindly trusting
> precompiled binaries,

Depends on how restricted of an environment you ware willing to work in

> I have higher confidence in our 2013 binaries than
> in binaries we would produce today, because (1) we are more likely to be
> a target today because Guix has become far more popular, (2) I expect
> that intelligence agencies have far more advanced tools today than they
> did in 2013, and (3) I expect that governmental policies have become far
> more favorable to permitting such attacks against projects such as ours.

1) Granted
2) Not exactly; simply because the most advanced attack tool ever
invented was the Nexus Intruder Program in 1958. (Hardware that subverts
software that later subverts hardware designs and more software
[firmware, microcode, etc]). The tools might get more expensive but the
actual quality of attack tools depends on the teams and the market's
demand for pumping out vulnerable products and bugs. (Like the recent
Hard drive firmware attack which leveraged the vendor's cost cutting
process to hijack the drives and then lock out future attempts at
recovery.
3) Actually Government agencies are depending more and more on "Open
source tools" (Their words not mine) as software budgets have gotten
tighter and third party vendors integrate them more and more into their
commercial offerings purchased by Goverment agencies. Putting a backdoor
in the software most Government agencies depend upon, invites
vulnerabilities in our own Intelligence Agencies infrastructure and
increase the probablity that Spies will be identified before their
flight to their target country leaves the ground. To do such would not
only be suicidal for those Intelligence Agencies but also ensure
Cyberwarfare against the Countries they work for that much more
effective.

Now that isn't to say they consider that an extranality and doom us all
but nothing stays hidden when we can read the source and can DDC our
entire bootstrap across arbitrary hardware/operating system
combinations.

-Jeremiah

^ permalink raw reply	[flat|nested] 49+ messages in thread
* Re: Preparing the reduced bootstrap tarballs
@ 2018-11-20  0:26 jeremiah
  2018-11-20  8:28 ` Ricardo Wurmus
  0 siblings, 1 reply; 49+ messages in thread
From: jeremiah @ 2018-11-20  0:26 UTC (permalink / raw)
  To: g, guix-devel

> so, if I don't get it wrong, every skilled engineer will be able to
> build an "almost analogic" (zero bit of software preloaded) computing
> machine ad use stage0/mes [1] as the "metre" [2] to calibrate all other
> computing machines (thanks to reproducible builds)?

well, I haven't thought of it in those terms but yes I guess that is one
of the properties of the plan.

> the first bit of code have to be "manually" introduced in the machine,
> right?
Correct, otherwise you'll have to deal with firmware/bios as a trust
vector to be concerned about.

> for the lazyer like me, what about a punched card? :-)
If someone is willing to figure out how to read a deck of punched cards
without software, I'd be interested in learning more.

> I didn't know about Nexus Intruder attacks: could you please give me
> some links to the relevant bibliography?
I'll see if I can dig those up for you.

> so, having the scientific proof that binary conforms to source, there
> will be noo need to trust (the untrastable)
Well, that is what someone else could do with it but not a direct goal
of the work.

-Jeremiah

^ permalink raw reply	[flat|nested] 49+ messages in thread
* Re: Preparing the reduced bootstrap tarballs
@ 2018-11-21  3:36 Jeremiah
  0 siblings, 0 replies; 49+ messages in thread
From: Jeremiah @ 2018-11-21  3:36 UTC (permalink / raw)
  To: rekado, samplet; +Cc: guix-devel

> Now that’s a project for a free hardware design that I’d happily work
> on :)
Sweet

> Reading and transmitting of the bitstream can be done in hardware alone
> (shift register, MUX, photoresistors, …), though I’d need to know what
> kind of interface there is on the receiving side.  Just flash memory /
> TTL shift register?
Well ultimately that would need to be a memory bus that is connected to
their TTL DRAM (64KB chips) or TTL SRAM (so we wouldn't have to worry
about memory refresh or other sort of issues like that)
and setting the memory address block starting from Zero until end of
deck.
The Memory is Big Endian Bit, Byte and Word Addressed.

The number address pins needs to be atleast 9 (otherwise the bootstrap
binaries would exceed the address space the reader could write to) and
not to exceed 64 (As I don't plan on implementing a 128bit ALU with
Floating point and Vector instructions needed to comply with the 128bit
version of the spec). I am planning on having the memory data bus to be
8bits wide as all instructions and datatypes are multiples of 8bits.

I can provide a clock signal for when the next value is to be loaded or
a line for you to directly drive the memory chips.

Or we could go the uart route and simply leverage a classic serial
connector.


> I?ve thought about this a bit.  Would it be possible to go even more
> mechanical, and implement the SPI protocol in switches toggled by holes
> cut in some kind of insulating tape?
Well that mechanical aspect would be what you would expect in punched
cards or paper tape (I was planning on doing a paper tape reader to
allow easier manual inspection of outputs, inputs and binaries)

> You could have a couple of steel
> balls (bearings?) balanced on top of each other, connected to a circuit
> using brushes or springs.  Then, you place the tape in between the
> bearings.  When there?s a hole in the insulating tape, they make
> contact, and when there is no hole, no contact.
Yep, classical through-hole reader technology. Not as reliable as
optical sensors but much more low tech and no room for anyone to insert
anything funny with the sensors.


> Then, you have a row of
> holes for the SPI clock, and a row of holes for the SPI data.  All you
> would have to do is hook up the right voltage to it, and you could write
> to a flash ROM by pulling manually punched tape through it.  (You could
> use a crank or motor if the timing needs to be more precise.)
Well you could have a hole aligned with the values being read (say a 9th
bit) which would trigger a clock pulse to indicate reading of the next byte.


> Then, you could take an x200 and setup the bootrom to be whatever you
> wanted with no software whatsoever.

Not possible, there is a 700+KB microcode blob in that CPU, without
which the CPU fails to power on for more than 10 clock cycles.

> If you adjusted stage0 to set up a
> ?cache-as-RAM? environment (like they do in coreboot), you could do
> quite a bit of manoeuvring before having to initialize RAM.
Well the 4MB of space (or 6MB if you have Penryn chip) is more than
enough to bootstrap cc_x86 and be able to compile a self-hosting C compiler


> That is,
> you could probably set up a tiny Scheme environment and do more hardware
> initialization from there.

Well the hardware initialization isn't a concern as that is currently
well documented in libreboot's code. The hard part is those computers
are only going to be working for a few more years before tin whiskers
kill the whole lot of them (the transistion to lead-free solder occurred
during that time) and then we will have a working solution to hardware
that is no longer available, no longer works and we can't build more
that will exactly match those specs without considerable effort.

> Obviously all of this would be easier on a simpler computer, but then
> presumably you have to move from a simpler computer to a more
> complicated computer at some point (depending on your needs).
I assume a 64bit processor with 16+GB of memory, a SATA interface and a
network interface should be sufficiently complicated to bootstrap
everything we will need to conern ourselves with in the next couple
years. (We can always enhance a design of our own making)

The Low end will be a 16bit processor with only 64KB of RAM and will
probably require ugly coding to fit some of the programs in that address size.

> The nice
> thing about booting the x200 as a simple computer is that you can just
> expand its boundaries until it is full-featured.
One needs to be precise by what one means by full-featured. Would you
consider Minix or Linux 1.0 a full-featured kernel? Would the ability to
build GCC and guile be considered full-featured enough? Because I can
build productive environments down to a hex0-monitor in a few hundred
bytes of total memory.

> Sorry for derailing a bit, I just wanted to share some idle thoughts.
I actively encourage alternative perspectives and I love being told
how I can do this better. I look forward to more ideas and suggestions
from you on this subject later ^_^

Just an open reminder our #bootstrappable channel is always looking for
people intersted in these sorts of topics and we love hearing about what
you have created in this regard.

-Jeremiah

^ permalink raw reply	[flat|nested] 49+ messages in thread
end of thread, other threads:[~2018-11-26 18:49 UTC | newest]

Thread overview: 49+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-14  8:50 [bug#33038] bootstrap: Regeneration of Mes bootstrap seeds Jan Nieuwenhuizen
2018-10-14  8:58 ` [bug#33038] [PATCH 1/6] doc: Move `Reduced Binary Seed Bootstrap' into `Bootstrapping' Jan Nieuwenhuizen
2018-10-14  8:58   ` [bug#33038] [PATCH 2/6] gnu: mes: Update to 0.18.0-08f04f55 Jan Nieuwenhuizen
2018-10-19 21:23     ` Ludovic Courtès
2018-10-20  6:41       ` Jan Nieuwenhuizen
2018-10-14  8:58   ` [bug#33038] [PATCH 3/6] bootstrap: Add %bootstrap-mes Jan Nieuwenhuizen
2018-10-19 21:31     ` Ludovic Courtès
2018-10-20  7:35       ` Jan Nieuwenhuizen
2018-10-20 18:31         ` Jan Nieuwenhuizen
2018-10-21 20:37         ` Marius Bakke
2018-10-21 21:04           ` Ludovic Courtès
2018-10-21 21:28             ` Jan Nieuwenhuizen
2018-10-21 21:09         ` Ludovic Courtès
2018-10-21 21:32           ` Jan Nieuwenhuizen
2018-10-23 21:00             ` bug#33038: " Jan Nieuwenhuizen
2018-10-24 16:18               ` [bug#33038] " Marius Bakke
2018-10-24 16:58                 ` Jan Nieuwenhuizen
2018-11-15  9:06               ` Preparing the reduced bootstrap tarballs Ludovic Courtès
2018-11-15 15:44                 ` Jan Nieuwenhuizen
2018-11-16 18:22                   ` Ludovic Courtès
2018-11-16 20:52                     ` Jan Nieuwenhuizen
2018-10-14  8:58   ` [bug#33038] [PATCH 4/6] gnu: mescc-tools: Update to 0.5.2-0.bb062b0d Jan Nieuwenhuizen
2018-10-19 21:32     ` Ludovic Courtès
2018-10-20  7:41       ` Jan Nieuwenhuizen
2018-10-14  8:58   ` [bug#33038] [PATCH 5/6] bootstrap: Build with %bootstrap-mes Jan Nieuwenhuizen
2018-10-19 21:34     ` Ludovic Courtès
2018-10-14  8:58   ` [bug#33038] [PATCH 6/6] doc: Update Preparing to Use the Bootstrap Binaries Jan Nieuwenhuizen
2018-10-19 21:37     ` Ludovic Courtès
2018-10-20  8:23       ` Jan Nieuwenhuizen
2018-10-19 21:22   ` [bug#33038] [PATCH 1/6] doc: Move `Reduced Binary Seed Bootstrap' into `Bootstrapping' Ludovic Courtès
  -- strict thread matches above, loose matches on Subject: below --
2018-11-15 20:39 Preparing the reduced bootstrap tarballs Jeremiah
2018-11-16 18:27 ` Ludovic Courtès
2018-11-16 20:44   ` Jan Nieuwenhuizen
2018-11-17 14:05     ` Ludovic Courtès
2018-11-18  7:32       ` Jan Nieuwenhuizen
2018-11-18 10:02         ` Jan Nieuwenhuizen
2018-11-17  3:49   ` Mark H Weaver
2018-11-17 14:27 Jeremiah
2018-11-17 23:14 ` Mark H Weaver
2018-11-19 18:54 ` Giovanni Biscuolo
2018-11-18 12:56 Jeremiah
2018-11-18 18:27 ` Mark H Weaver
2018-11-18 18:39   ` Jan Nieuwenhuizen
2018-11-20 15:45     ` Timothy Sample
2018-11-21 20:32       ` Jan Nieuwenhuizen
2018-11-26 18:49         ` Timothy Sample
2018-11-20  0:26 jeremiah
2018-11-20  8:28 ` Ricardo Wurmus
2018-11-21  3:36 Jeremiah

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.