[opinion] CVE-patching is not sufficient for package security patching

[opinion] CVE-patching is not sufficient for package security patching

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1661 bytes --]

Hello!

I would like to share some opinion I have on CVE-patching for non-
rolling release GNU/Linux distributions and why we should strive to
always update to the latest available releases or always follow
upstream supported release series and never backport patches ourselves
in most cases (some upstreams may have really good practices but these
are rare).

A lot of security issues are patched silently in upstream projects
without ever getting a CVE, security issues may not be labeled as such
by upstreams for various reasons (fear of shame, belief to patch
something with no security impact while it has, bizarre security
through obscurity policy, ..).

For these reasons, I suggest that we always strive to update packages
to their latest versions and that I think it is security relevant to
always do so. Of course, new code could *introduce* new vulnerabilities
but I am not trying to debate this, it's that to the best of the
upstream's knowledge chances are that the latest version will contain
more security fixes than older versions (if that upstream is actually
maintaining the project).

In many cases, browsing through the commit history of some popular
projects can uncover security issues not publicized through any
security mailing lists or CVEs anywhere, this is unfortunately quite
common. We cannot possibly monitor the commit history (and code) of
every single project to backport fixes when we would need to. It is
better for us to always strive to use the latest versions even when it
requires us to do more far-reaching changes because of
dependents/dependencies.

Let me know what you think!

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [opinion] CVE-patching is not sufficient for package security patching

[Jonathan Brielmaier]

On 16.03.21 12:10, Léo Le Bouter wrote:
> For these reasons, I suggest that we always strive to update packages
> to their latest versions and that I think it is security relevant to
> always do so. Of course, new code could *introduce* new vulnerabilities
> but I am not trying to debate this, it's that to the best of the
> upstream's knowledge chances are that the latest version will contain
> more security fixes than older versions (if that upstream is actually
> maintaining the project).

I think the only two reasons against that are: time and CI/rebuilding. I
think thats the reason why stuff like Gnome and others lower in the
dependency tree are lacking behind... Being non-FHS and non-systemd
makes updates for those stuff not easier and is maybe the third
reason/root issue...

Re: [opinion] CVE-patching is not sufficient for package security patching

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

On Tue, 2021-03-16 at 12:17 +0100, Jonathan Brielmaier wrote:
> I think the only two reasons against that are: time and
> CI/rebuilding. I
> think thats the reason why stuff like Gnome and others lower in the
> dependency tree are lacking behind... Being non-FHS and non-systemd
> makes updates for those stuff not easier and is maybe the third
> reason/root issue...

I agree with all 3 points. I have hope however that we can develop
better tooling over time to ease the burden on us so we can devote more
time to tasks that actually absolutely **require** our human oversight
to be done. And then even without an increase in the contributor base
we can avoid lagging behind on these updates.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [opinion] CVE-patching is not sufficient for package security patching

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 3198 bytes --]

On Tue, Mar 16, 2021 at 12:10:26PM +0100, Léo Le Bouter wrote:
> For these reasons, I suggest that we always strive to update packages
> to their latest versions and that I think it is security relevant to
> always do so. Of course, new code could *introduce* new vulnerabilities
> but I am not trying to debate this, it's that to the best of the
> upstream's knowledge chances are that the latest version will contain
> more security fixes than older versions (if that upstream is actually
> maintaining the project).

I agree that every new release can be considered to have fixed security
problems.

Please read the rest of my message while keeping in mind that I have
spent *a lot* of time working on security in Guix over the years.

We must keep in mind that there are other values besides security.
Additionally, this kind of "security" mindset is a somewhat narrow way
of considering the problem of secure computing. It's important to
remember that security can be modeled with 3 factors: confidentiality,
integrity, and availability. The 3rd factor is often overlooked.

In terms of making a distro, there is a spectrum of approaches.

At one end of the spectrum, there is something like PyPi, which is just
a clearinghouse for upstream projects to distribute their code.
Everything is always updated to the latest version. It does not provide
a working system, even within the narrow world of "just Python"; there
are broad incompatibilities among the latest versions of Python
programs.

On the other end is the approach of Red Hat and Debian. They laboriously
filter the upstream software to provide stable operating systems.  They
do fix security bugs, but only after extensive validation that
functionality is not changed. The result is useful but the cost is very
high.

Guix has always been in the middle, along with other rolling release
distros, and I think that's a good place for us to be. With our superior
tooling we can be "more stable than rolling release" while also "moving
faster than stable".

It's instructive to consider the Linux kernel. They release about once a
week, and every release fixes serious but unpublicized bugs. At the time
they are announcing the release, they are already aware of other serious
bugs, that might be fixed in the next release. It sounds terrible, and
yet Linux is by far the most popular and useful general-purpose
operating system. The world of computing, which is based on Linux,
continues to serve our civilization well. That's because the most
important thing value in Linux development is to not break anything for
users; security is not the top priority, but just another important
thing to consider.

I think that, as an operating system distro, we must adopt a similar
mindset, and be careful not to sacrifice too much for an abstract sense
of security based on fixing CVEs, which are an arbitrary system that
have little bearing on utility or safety in the real world, which is
where security matters. Of course we should fix CVEs, but we must also
recognize that rushing too much reduces stability and availability. We
have to weigh the costs and benefits every time.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [opinion] CVE-patching is not sufficient for package security patching

[Mark H Weaver]

Hi Léo,

Léo Le Bouter <lle-bout@zaclys.net> writes:

> I would like to share some opinion I have on CVE-patching for non-
> rolling release GNU/Linux distributions and why we should strive to
> always update to the latest available releases or always follow
> upstream supported release series and never backport patches ourselves
> in most cases (some upstreams may have really good practices but these
> are rare).
>
> A lot of security issues are patched silently in upstream projects
> without ever getting a CVE, security issues may not be labeled as such
> by upstreams for various reasons (fear of shame, belief to patch
> something with no security impact while it has, bizarre security
> through obscurity policy, ..).

... and I'll add that it can be a lot of work to evaluate, for a given
bug, whether or not that bug is exploitable.

Anyway, I agree that bugs fixed upstream are sometimes exploitable, even
when they have not been explicitly identified as security flaws, and
that this is a valid argument in favor of keeping our packages updated
to the latest release.

That said, I strongly disagree that we should "never backport patches
ourselves in most cases".  The only way to do that, while addressing
security flaws, would be to promptly update even our lowest-level
libraries in response to CVEs, of which there is a steady stream.

Anyone with experience working on the 'staging' or 'core-updates'
branches in Guix, or in the release process of Debian, will immediately
recognize this proposal to be unrealistic.  In practice, updating
low-level or even mid-level libraries tends to cause breakage.  This
kind of integration breakage happens quite frequently, even on
x86_64-linux, the architecture that most developers work on.

It's *much* worse on other architectures.  New upstream releases quite
regularly cause breakage on less popular architectures.  It is often
left to distros such as Debian to fix these problems.

Since you're interested in security, I'll now remind you that *all*
modern Intel systems include another little computer inside them called
the Management Engine, which is always on when the machine is plugged in
(even when the computer is "off"), has it's own memory that the main CPU
cannot see, runs a proprietary OS that the user cannot replace, has full
access to the RAM and disk of the machine, and can talk to the network
without the main CPU even seeing those packets.

Are you comfortable with this?

If not, it would be good to work toward the goal of making Guix usable
on non-Intel systems.  I'm sorry to say that, in my opinion, your
proposal would move us in the wrong direction to achieve that goal.

In my experience, Guix is already moving far too fast to be usable on
less popular architectures.  I have some knowledge of this.  Years ago,
I made a serious effort to make Guix usable on non-Intel systems.  When
Guix was young, I initiated its first two ports to non-Intel
architectures: mips64el-linux and armhf-linux, and I tried to actually
use Guix on those systems in practice.  I found that my system was very
frequently broken by upstream updates, and that we didn't have nearly
enough developer energy to keep up with fixing those problems.

I've come to believe that having Guix work well on non-Intel systems is,
in practice, incompatible with the rate at which we update our packages.
I'm not sure that even Debian would have enough energy to keep less
popular architecures working well, given our practices.  I raised this
issue on guix-devel a few times over the years, but it became clear that
the desire in this community to keep packages aggressively updated far
outweighs any interest in supporting non-Intel systems.

Ultimately, I gave up.  In my opinion, Guix has never achieved usability
as a desktop system on non-Intel systems.  Therefore, the Guix community
is unable to attract many developers who want a distro that supports
non-Intel systems well.  Our community has thus become dominated by
Intel users, and there's unsufficient political will to adopt policies
that would enable us to provide a usable system for non-Intel users.

What do you think?

    Regards,
      Mark

Re: [opinion] CVE-patching is not sufficient for package security patching

[Leo Famulari]

On Tue, Mar 16, 2021 at 07:19:59PM -0400, Mark H Weaver wrote:
> Ultimately, I gave up.  In my opinion, Guix has never achieved usability
> as a desktop system on non-Intel systems.  Therefore, the Guix community
> is unable to attract many developers who want a distro that supports
> non-Intel systems well.  Our community has thus become dominated by
> Intel users, and there's unsufficient political will to adopt policies
> that would enable us to provide a usable system for non-Intel users.
> 
> What do you think?

Thanks, as always, for your well-reasoned message. Your story of your
experience here, and what it means for Guix, computing security, and
free software in general, is really valuable. I still think it's really
unfortunate for the rest of us that you gave up, but I don't see how it
could have gone differently.

I agree with you that Guix moves too fast to support non-Intel
architectures in its current state. My hope is that, within the next two
years, there will be workstation aarch64 machines more widely available
at comparable prices to Intel/AMD, and this will translate into more
developer support for aarch64 in the years after that. Time will tell.

Guix moving too fast?

[zimoun]

Hi,

Thanks Mark for your words.  Interestingly, using the angle of
scientific software, I agree with your general analysis.


On Tue, 16 Mar 2021 at 19:49, Leo Famulari <leo@famulari.name> wrote:
> On Tue, Mar 16, 2021 at 07:19:59PM -0400, Mark H Weaver wrote:

>> Ultimately, I gave up.  In my opinion, Guix has never achieved usability
>> as a desktop system on non-Intel systems.  Therefore, the Guix community
>> is unable to attract many developers who want a distro that supports
>> non-Intel systems well.  Our community has thus become dominated by
>> Intel users, and there's unsufficient political will to adopt policies
>> that would enable us to provide a usable system for non-Intel users.
>> 
>> What do you think?
>
> Thanks, as always, for your well-reasoned message. Your story of your
> experience here, and what it means for Guix, computing security, and
> free software in general, is really valuable. I still think it's really
> unfortunate for the rest of us that you gave up, but I don't see how it
> could have gone differently.

Moving less fast? :-)


> I agree with you that Guix moves too fast to support non-Intel
> architectures in its current state. My hope is that, within the next two
> years, there will be workstation aarch64 machines more widely available
> at comparable prices to Intel/AMD, and this will translate into more
> developer support for aarch64 in the years after that. Time will tell.

Moving too fast, i.e., pushing a lot of changes, has various other
consequences: a lot of rebuilds, even if the build farm is really
improving these days, there is a high probability that “guix pull”
intensively computes––hopefully ’channel-with-substitutes-available’ [1]
avoids that for limited resource machines––then “guix install” right
after generally builds the package locally.  For example, the package
’gmsh’ [2], there is no substitute for 373c7b5 (pulled on March 13th
couple of days ago) but builds fine, and this ’gmsh’ packages had been
last updated on Oct 8th 2020.

If you look at the Data Service [2], there is 7 different “versions” for
the same 4.6.0.  Therefore, it had been implied by some unrelated
changes.  That’s fine and that’s why Guix is so great: 4.6.0 is not
enough for the binary versioning.

This can be really worse, for instance the package ’openfoam’ [3].  It
is a complex package and for the same 4.1, the output version changes
every 5 days on average.  How is it possible that the build farm follows
such rate?

Another example, the package ’freecad’ [4].

Even worse, what happens if an unrelated change breaks the package?  The
time spent at fixing it is not spent at adding other nice-to-have
packages or fixing bugs.  Or we prefer to add/update other packages or
add features and this case, because we do not care, this broken package
should be removed: from a user perspective, nothing is worse that broken
packages and from the build farm perspective, it saves resources.

Currently, there is ~5% (~1000 packages) of broken packages.  My guess
is, considering the same rate of changes and an increase of the number
of packages, this percentage would be the same, i.e., the number of
broken packages would be higher.

Well, because scientific software are often complex, meaning with a huge
dependency graph, it makes hard to maintain them with such change rate.

And I am not speaking about third-party channels.  They are also hard to
maintain.

To that, let multiply by the number of architectures.


All in all, maybe the 3 branches model does not scale well.  It should
not be possible that broken packages are in the default branch (now
master).  It is unavoidable with the current model.  Chris initiated
discussions and works about QA with patchwork, see [5,6].  Somehow, what
is “production” should be distinguished from what is “test”.  It is not
currently the case, every updates is pushed to master and we cross the
fingers—–it works well most of the time though, the rare broken cases
are just full annoyance and too visible to my taste.


The default branch could be “stable”, which receives only security
fixes, i.e., grafts, also bug fixes and patches touching ’guix/’.  All
other patches, i.e., touching ’gnu/’, could go to “next”.  And massive
rebuilds could go to “massive”.  Each time a graft is added, it is also
ungrafted to “next” (or “massive” if it is a real deep change),
therefore, each release becomes (almost) ungrafted.

The branches “stable” and “next” are continuously built whereas
“massive” only built before the merge.

Every 3 months, “next” is merged to “stable”.
Every 3 months, “massive” is merged to “next”.
Every 6 months, “stable” is released.

Like a metronome.  For instance, if the “massive” merge is missed for
whatever reason, it is delayed to 3 months but “next” is still merged on
time and the release is on time too.

I do not have a clear view about other architectures than x86_64, but
since “stable” is moving slower, it could be discussed at the “next”
merge time some changes specific to these other architectures, i.e.,
exclude changes because they are failing or accept it fails for these
architectures.

And “stable” never receives changes that breaks packages (at least for
x86_64), it lets us couple of weeks to fix or revert offending commits.
Substitutes are always there for “stable”.  The rolling release becomes
“next” and not “stable”.

Wait, wait, it looks like “master”, “staging” and “core-updates”. ;-) It
just changes where to push.  And such model needs time to transition if
we agree.  We could start after the next release 1.2.1 and we could
expect a smooth schedule as described 2 or 3 releases later, I guess.


Cheers,
simon


1: <https://guix.gnu.org/manual/devel/en/guix.html#Channels-with-Substitutes>
2: <https://data.guix.gnu.org/repository/1/branch/master/package/gmsh/output-history>
3: <https://data.guix.gnu.org/repository/1/branch/master/package/openfoam/output-history>
4: <https://data.guix.gnu.org/repository/1/branch/master/package/freecad/output-history>
5: <https://yhetil.org/guix/87v9dos547.fsf@cbaines.net>
6: <https://yhetil.org/guix/877drg3lmi.fsf@cbaines.net/>

Re: [opinion] CVE-patching is not sufficient for package security patching

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

On Tue, 2021-03-16 at 19:19 -0400, Mark H Weaver wrote:
> That said, I strongly disagree that we should "never backport patches
> ourselves in most cases".  The only way to do that, while addressing
> security flaws, would be to promptly update even our lowest-level
> libraries in response to CVEs, of which there is a steady stream.

Fortunately I think that lots of these core package upstreams also have
good CVE-issuance practices. For the Glib care in particular, I think
they are good, I consider acceptable to backport patches, everyone is
doing it, upstream is cooperative and works towards that same goal.

To everyone else in general, I understand we have to ship a working
system, and I want that too, that's why I said we should "strive" to,
but it doesnt mean we should break things, of course. By that I mean
that we shouldnt leave packages unmaintained without updates for too
long even without CVEs or other security notices issued. At some point,
if a package is of no use, no users show up and it's painful to update,
we should also consider removing the package or archiving it in a third
party channel we could create like "guix-archive", "guix-ugly" or
"guix-love-me-please".

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [opinion] CVE-patching is not sufficient for package security patching

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 2992 bytes --]

Sorry for duplicated email..

On Tue, 2021-03-16 at 19:19 -0400, Mark H Weaver wrote:
> If not, it would be good to work toward the goal of making Guix
> usable
> on non-Intel systems.  I'm sorry to say that, in my opinion, your
> proposal would move us in the wrong direction to achieve that goal.

We have been working with Chris Marusich, Efraim and numerous others on
porting GNU Guix to PowerPC 64-bits, I think both are not
contradictory. I have a Talos II desktop at home which once GNU Guix
works on it will be the main machine I will use to contribute to GNU
Guix (along with offloading to other machines for testing on other
archs), so count on me to at the same time keep up on GNU Guix and test
things on PowerPC 64-bits.

I am of course concerned with any blob doing things I don't need (and
introducing security risk) under the hood, that's why (along with
strong software freedom imaginaries) I pre-ordered my RaptorCS Talos II
machine in 2017 and that I have been trying since 2 years to bring
PowerPC 64-bits to GNU Guix (also with numerous other folks who joined
efforts most recently Chris Marusich who've been enormous help!).

But I also want to be realistic that the major security risk in most
computers today probably isnt the Intel ME or Intel AMT and that we
also can do many other things in the system itself that reduces risk
greatly. I'll be honest also, the IBM POWER chips have gotten much less
security review than the Intel or AMD chips recently, so it's not
because there's not as much security drama on IBM POWER that it doesnt
have (maybe even more severe) issues :-)

About the overall security of GNU Guix and the things we can do that
don't involve keeping a fast-paced rythm to updating packages I see few
things, right now GNU Guile is the center of all's GNU Guix security, I
am not sure it received lots of security auditing, it's also written in
part in a memory-unsafe language that is C, so there's probably some
low hanging fruits there once some starts fuzzing it, I'm no big expert
in fuzzing but I may try at some point.

I think we can do many things complementary, prevention (sandboxing),
mitigation (enabling hardening compiler flags, ..), AND code security
patching. The first two don't require we keep a fast paced update
rythm, also we may do the first two especially because we realize we
can't do the latter at all time and I realize that, I just think we
should always try to, at least, that's all.

I am also a bit concerned with the idea that GNU Guix, GNU Shepherd
etc. execute code from arbitrary files in many places, I am not sure
all the security details have been reviewed here, it seems risky to me
to have configuration files that allow executing arbitrary code, also
GNU Guile seems to have a sandboxed evaluation mode, that's good. I
like the freedom of arbitrary code evaluation anywhere gives us, but I
also want to make sure it's actually secure to do so.

Léo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [opinion] CVE-patching is not sufficient for package security patching

[Ludovic Courtès]

Hi,

Mark H Weaver <mhw@netris.org> skribis:

> Ultimately, I gave up.  In my opinion, Guix has never achieved usability
> as a desktop system on non-Intel systems.  Therefore, the Guix community
> is unable to attract many developers who want a distro that supports
> non-Intel systems well.  Our community has thus become dominated by
> Intel users, and there's unsufficient political will to adopt policies
> that would enable us to provide a usable system for non-Intel users.

A practical problem that’s been mentioned repeatedly is lack of ci.guix
hardware for non-Intel architectures: please everyone, consider helping
the project find either sponsors or companies that sell fitting
hardware, along with a plan to host it and maintain it over time!

Ludo’.

Re: [opinion] CVE-patching is not sufficient for package security patching

[raingloom]

On Sat, 20 Mar 2021 12:19:11 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

> Hi,
> 
> Mark H Weaver <mhw@netris.org> skribis:
> 
> > Ultimately, I gave up.  In my opinion, Guix has never achieved
> > usability as a desktop system on non-Intel systems.  Therefore, the
> > Guix community is unable to attract many developers who want a
> > distro that supports non-Intel systems well.  Our community has
> > thus become dominated by Intel users, and there's unsufficient
> > political will to adopt policies that would enable us to provide a
> > usable system for non-Intel users.  
> 
> A practical problem that’s been mentioned repeatedly is lack of
> ci.guix hardware for non-Intel architectures: please everyone,
> consider helping the project find either sponsors or companies that
> sell fitting hardware, along with a plan to host it and maintain it
> over time!
> 
> Ludo’.
> 

What about a Liberapay for Guix? Could also be used to pay developers.

Re: [opinion] CVE-patching is not sufficient for package security patching

[Joshua Branson]

raingloom <raingloom@riseup.net> writes:
>
> What about a Liberapay for Guix? Could also be used to pay developers.
>

I'd be game for something like this.  We could have a guix membership.
Drew Devault has a "secret irc" channel for paying patreons.  Perhaps we
could advertise a guix membership on the guix site.  When someone signs
up, it actually makes them an FSF member. I believe membership with the
FSF includes perks such as: 5 email aliases, a federated XMPP account,
and some other things...

--
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar

Re: [opinion] CVE-patching is not sufficient for package security patching

[Mark H Weaver]

Joshua Branson <jbranso@dismail.de> writes:

> raingloom <raingloom@riseup.net> writes:
>>
>> What about a Liberapay for Guix? Could also be used to pay developers.
>>
>
> I'd be game for something like this.  We could have a guix membership.
> Drew Devault has a "secret irc" channel for paying patreons.  Perhaps we
> could advertise a guix membership on the guix site.

Then, the Guix leadership would have to decide which Guix developers are
worthy of funding, and how much.  Those who are excluded may feel that
their contributions are insufficiently valued, and therefore feel
alienated.  Sounds to me like it would open up a huge can of worms, so
to speak.

     Regards,
       Mark

Re: [opinion] CVE-patching is not sufficient for package security patching

[Leo Famulari]

On Mon, Mar 22, 2021 at 02:44:04PM +0100, raingloom wrote:
> What about a Liberapay for Guix? Could also be used to pay developers.

Some of us already have Liberapay accounts.

Re: [opinion] CVE-patching is not sufficient for package security patching

[Ricardo Wurmus]

raingloom <raingloom@riseup.net> writes:

> On Sat, 20 Mar 2021 12:19:11 +0100
> Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Hi,
>> 
>> Mark H Weaver <mhw@netris.org> skribis:
>> 
>> > Ultimately, I gave up.  In my opinion, Guix has never achieved
>> > usability as a desktop system on non-Intel systems.  Therefore, the
>> > Guix community is unable to attract many developers who want a
>> > distro that supports non-Intel systems well.  Our community has
>> > thus become dominated by Intel users, and there's unsufficient
>> > political will to adopt policies that would enable us to provide a
>> > usable system for non-Intel users.  
>> 
>> A practical problem that’s been mentioned repeatedly is lack of
>> ci.guix hardware for non-Intel architectures: please everyone,
>> consider helping the project find either sponsors or companies that
>> sell fitting hardware, along with a plan to host it and maintain it
>> over time!
>> 
>> Ludo’.
>> 
>
> What about a Liberapay for Guix? Could also be used to pay developers.

This seems to be a misunderstanding.  The first step is to use the money
we already have but cannot exchange for hardware, because

- finding appropriate hardware that you can actually buy is not easy
- hosting needs to be considered because we can’t just dump them in the
  MDC data centre where most of the Intel servers are hosted.

We bought a handful of Overdrive 1000 in the past (they are no longer
sold), and hosting was always an obstacle.

While looking for ways to get the project some more money is certainly
worthwhile, it’s really not the pressing issue here.

-- 
Ricardo

Re: [opinion] CVE-patching is not sufficient for package security patching

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Tue, Mar 23, 2021 at 11:54:54PM +0100, Ricardo Wurmus wrote:
> This seems to be a misunderstanding.  The first step is to use the money
> we already have but cannot exchange for hardware, because
> 
> - finding appropriate hardware that you can actually buy is not easy
> - hosting needs to be considered because we can’t just dump them in the
>   MDC data centre where most of the Intel servers are hosted.
> 
> We bought a handful of Overdrive 1000 in the past (they are no longer
> sold), and hosting was always an obstacle.
> 
> While looking for ways to get the project some more money is certainly
> worthwhile, it’s really not the pressing issue here.

I volunteer to host one or two workstation-type 64-bit ARM machines.

Concretely, this would mean a Honeycomb LX2 or Ampere ALTRA workstation,
since I don't believe there are any other aarch64 workstations available
for sale.

https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/
https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [opinion] CVE-patching is not sufficient for package security patching

[Vincent Legoll]

Hello,

On Wed, Mar 24, 2021 at 8:51 PM Leo Famulari <leo@famulari.name> wrote:
> > We bought a handful of Overdrive 1000 in the past (they are no longer
> > sold), and hosting was always an obstacle.
>
> I volunteer to host one or two workstation-type 64-bit ARM machines.

I already volunteered (privately) to host the same (1 or 2 WS power-class),
currently on ADSL uplink (so not for substitute distribution, only building),
FTTH in the future, no UPS though.

-- 
Vincent Legoll

Re: [opinion] CVE-patching is not sufficient for package security patching

[Léo Le Bouter]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Wed, 2021-03-24 at 21:24 +0100, Vincent Legoll wrote:
> Hello,
> 
> On Wed, Mar 24, 2021 at 8:51 PM Leo Famulari <leo@famulari.name>
> wrote:
> > > We bought a handful of Overdrive 1000 in the past (they are no
> > > longer
> > > sold), and hosting was always an obstacle.
> > 
> > I volunteer to host one or two workstation-type 64-bit ARM
> > machines.
> 
> I already volunteered (privately) to host the same (1 or 2 WS power-
> class),
> currently on ADSL uplink (so not for substitute distribution, only
> building),
> FTTH in the future, no UPS though.
> 

I also have space at home, ~3500Mbps down, 600Mbps up, no UPS (yet?).

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [opinion] CVE-patching is not sufficient for package security patching

[Leo Famulari]

On Wed, Mar 24, 2021 at 09:24:40PM +0100, Vincent Legoll wrote:
> I already volunteered (privately) to host the same (1 or 2 WS power-class),
> currently on ADSL uplink (so not for substitute distribution, only building),
> FTTH in the future, no UPS though.

The architecture of the build arm is such that substitutes are
distributed centrally, so you wouldn't have to worry about slow uplink,
as long as it's fast enough to pragmatically send build artifacts to the
central node.

Re: [opinion] CVE-patching is not sufficient for package security patching

[Mathieu Othacehe]

Hello,

> Concretely, this would mean a Honeycomb LX2 or Ampere ALTRA workstation,
> since I don't believe there are any other aarch64 workstations available
> for sale.
>
> https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/
> https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html

I recently added a new metric in Cuirass: "Builds count per machine
during the last day". Turns out the overdrive1 with its two workers
seems to outperform the hydra-guix-X running emulated builds on four
workers.

As soon as the other overdrives are back online, the situation will
hopefully be a tiny bit better. Buying and hosting other machines such
as the ones you mentioned could also help here.

The Wireguard tunnel between berlin an the overdrive1 works fine and
configuring those machines with something similar to
"hydra/modules/sysadmin/overdrive.scm" should be enough to add them as
Cuirass workers.

Thanks,

Mathieu

Re: [opinion] CVE-patching is not sufficient for package security patching

[Leo Famulari]

On Thu, Mar 25, 2021 at 03:22:16PM +0100, Mathieu Othacehe wrote:
> I recently added a new metric in Cuirass: "Builds count per machine
> during the last day". Turns out the overdrive1 with its two workers
> seems to outperform the hydra-guix-X running emulated builds on four
> workers.

That's good to know, thanks!

Buying AArch64 hardware?

[Ludovic Courtès]

Hi Leo,

Leo Famulari <leo@famulari.name> skribis:

> I volunteer to host one or two workstation-type 64-bit ARM machines.
>
> Concretely, this would mean a Honeycomb LX2 or Ampere ALTRA workstation,
> since I don't believe there are any other aarch64 workstations available
> for sale.
>
> https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/
> https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html

Let’s do that!  We can already buy one or two and ship it to your place.
If it works well, we can later expand and ship it to the other people
who volunteers.

We can work out practical details on guix-sysadmin, such as finding who
does the initial payment (currently the process to access money held at
the FSF is through reimbursement, with an invoice).

Thanks,
Ludo’.