What ARM hardware should we buy and where should we host it?

What ARM hardware should we buy and where should we host it?

[Ricardo Wurmus]

Hi Guix,

currently, we have a very small number of ARM build machines that
provide some substitutes for armhf and aarch64 systems.  (See
https://www.gnu.org/software/guix/donate/ for details.)

We would like to increase this number by purchasing more systems, but we
need your help to get this right.

The first question is what ARM hardware to buy; the second is where to
host that hardware.  Here are some considerations:

- The new systems should be easy to host in a data centre; this might
  mean that we need rack-mounted servers, or that we need a co-location
  data centre that accepts machines with unusual form factors.

- The system should be able to build substitutes for both armhf and
  aarch64.  Not all aarch64 CPUs support building for armhf, for
  example.

- We need to be able to restart the machine remotely.  If members of the
  Guix project have no physical access to the hosting site, this can be
  done via remote-controlled power supply or similar.

We are looking for volunteers who could either host a machine at home or
in a data centre that they could get emergency access to.

One system that looks promising for co-location is the Softiron
Overdrive 3000 (a rack-mounted ARM server.)  A system suitable for
hosting at home is the Softiron Overdrive 1000.  If you know of other
machines that would be suitable for our purposes, please let us know!

(The cost for the machines and for hosting them in a data centre would
be covered by the Guix project, but we are also willing to accept
machine donations if we can also find a volunteer to provide hosting.)

--
Ricardo

Re: What ARM hardware should we buy and where should we host it?

[Tobias Geerinckx-Rice]

Ricardo,

Ricardo Wurmus wrote:
> We are looking for volunteers who could either host a machine at 
> home or
> in a data centre that they could get emergency access to.

I've got plenty of cool, dry space for homeless boxen. No racks, 
though.

Kind regards,

T G-R

Re: What ARM hardware should we buy and where should we host it?

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]

On 2018-09-03, Ricardo Wurmus wrote:
> The first question is what ARM hardware to buy; the second is where to
> host that hardware.  Here are some considerations:
>
> - The new systems should be easy to host in a data centre; this might
>   mean that we need rack-mounted servers, or that we need a co-location
>   data centre that accepts machines with unusual form factors.
>
> - The system should be able to build substitutes for both armhf and
>   aarch64.  Not all aarch64 CPUs support building for armhf, for
>   example.
>
> - We need to be able to restart the machine remotely.  If members of the
>   Guix project have no physical access to the hosting site, this can be
>   done via remote-controlled power supply or similar.

Debian also has similar needs for build machines and is currently
exploring new hardware. There was a pretty good summary a few months
back:

  https://lists.debian.org/debian-arm/2018/06/msg00062.html


One of the most promising seems to be the SynQuacer:

  https://www.96boards.org/product/developerbox/

With a 24-core processor, SATA, PCIe, USB 3.0, micro-atx form-factor,
and 4 ram slots (up to 64GB, in theory, but may be picky about
ram).

Sounds like they're working with Debian's 4.17.x kernels, so should have
reasonable mainline linux support.

Steve McIntyre (CCed) has been rebuilding debian with them to test them
out, and my impression is it's been going pretty well. Might be more
details in the Debian ARM ports BoF video:

  https://debconf18.debconf.org/talks/93-arm-ports-bof/


> One system that looks promising for co-location is the Softiron
> Overdrive 3000 (a rack-mounted ARM server.)  A system suitable for
> hosting at home is the Softiron Overdrive 1000.

It does sound like the processors used in the Overdrive systems are
officially EOL, but maybe there are still enough of them that SoftIron
will continue to support them for a while. I'll see if I can follow-up
on that, too.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: What ARM hardware should we buy and where should we host it?

[Steve McIntyre]

Hey Vagrant!

On Mon, Sep 03, 2018 at 10:59:38AM -0700, Vagrant Cascadian wrote:
>On 2018-09-03, Ricardo Wurmus wrote:
>> The first question is what ARM hardware to buy; the second is where to
>> host that hardware.  Here are some considerations:
>>
>> - The new systems should be easy to host in a data centre; this might
>>   mean that we need rack-mounted servers, or that we need a co-location
>>   data centre that accepts machines with unusual form factors.
>>
>> - The system should be able to build substitutes for both armhf and
>>   aarch64.  Not all aarch64 CPUs support building for armhf, for
>>   example.
>>
>> - We need to be able to restart the machine remotely.  If members of the
>>   Guix project have no physical access to the hosting site, this can be
>>   done via remote-controlled power supply or similar.
>
>Debian also has similar needs for build machines and is currently
>exploring new hardware. There was a pretty good summary a few months
>back:
>
>  https://lists.debian.org/debian-arm/2018/06/msg00062.html
>
>One of the most promising seems to be the SynQuacer:
>
>  https://www.96boards.org/product/developerbox/
>
>With a 24-core processor, SATA, PCIe, USB 3.0, micro-atx form-factor,
>and 4 ram slots (up to 64GB, in theory, but may be picky about
>ram).
>
>Sounds like they're working with Debian's 4.17.x kernels, so should have
>reasonable mainline linux support.
>
>Steve McIntyre (CCed) has been rebuilding debian with them to test them
>out, and my impression is it's been going pretty well. Might be more
>details in the Debian ARM ports BoF video:
>
>  https://debconf18.debconf.org/talks/93-arm-ports-bof/
>
>
>> One system that looks promising for co-location is the Softiron
>> Overdrive 3000 (a rack-mounted ARM server.)  A system suitable for
>> hosting at home is the Softiron Overdrive 1000.
>
>It does sound like the processors used in the Overdrive systems are
>officially EOL, but maybe there are still enough of them that SoftIron
>will continue to support them for a while. I'll see if I can follow-up
>on that, too.

Other people I've been talking to have been struggling a lot to get
hold of any Overdrive-based systems. I'm honestly surprised there are
any left, to be honest!

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
Dance like no one's watching. Encrypt like everyone is.
 - @torproject

Re: What ARM hardware should we buy and where should we host it?

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Mon, Sep 03, 2018 at 10:59:38AM -0700, Vagrant Cascadian wrote:
> One of the most promising seems to be the SynQuacer:
> 
>   https://www.96boards.org/product/developerbox/
> 
> With a 24-core processor, SATA, PCIe, USB 3.0, micro-atx form-factor,
> and 4 ram slots (up to 64GB, in theory, but may be picky about
> ram).

It has 24 cores of Cortex-A53 CPUs, which are intended for "mobile"
applications (smartphones and similar). IMO, it would be great to get
more powerful processors. But, if this thing is working for Debian and
can handle sustained load on all 24 cores, that's already better than
some of the armhf boards we have. According to that Debian mailing list
post, this can do both 32 and 64-bit.

Steve, do you find the SynQuacer can handle being fully loaded for
extended periods of time?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: What ARM hardware should we buy and where should we host it?

[Steve McIntyre]

On Tue, Sep 04, 2018 at 02:03:18PM -0400, Leo Famulari wrote:
>On Mon, Sep 03, 2018 at 10:59:38AM -0700, Vagrant Cascadian wrote:
>> One of the most promising seems to be the SynQuacer:
>> 
>>   https://www.96boards.org/product/developerbox/
>> 
>> With a 24-core processor, SATA, PCIe, USB 3.0, micro-atx form-factor,
>> and 4 ram slots (up to 64GB, in theory, but may be picky about
>> ram).
>
>It has 24 cores of Cortex-A53 CPUs, which are intended for "mobile"
>applications (smartphones and similar). IMO, it would be great to get
>more powerful processors. But, if this thing is working for Debian and
>can handle sustained load on all 24 cores, that's already better than
>some of the armhf boards we have. According to that Debian mailing list
>post, this can do both 32 and 64-bit.

Correct.

>Steve, do you find the SynQuacer can handle being fully loaded for
>extended periods of time?

I've been doing rebuilds of the entire Debian archive for the last few
weeks on a Synquacer and some other machines. It's not shown any
issues at all during that period. A53 cores are not all that fast for
single-threaded code, but the CPU is claimed to only hit 5W flat out
on all 24 cores. It's got a large passive heat sink and a small fan in
the PSU. No issues seen.

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
  Getting a SCSI chain working is perfectly simple if you remember that there
  must be exactly three terminations: one on one end of the cable, one on the
  far end, and the goat, terminated over the SCSI chain with a silver-handled
  knife whilst burning *black* candles. --- Anthony DeBoer

Re: What ARM hardware should we buy and where should we host it?

[Andreas Enge]

Hello,

On Mon, Sep 03, 2018 at 01:37:54PM +0200, Ricardo Wurmus wrote:
> The first question is what ARM hardware to buy; the second is where to
> host that hardware.  Here are some considerations:

concerning hosting, I could easily host one smaller machine (like the
Overdrive 1000) in my living room, Tobias has stepped up, and I suppose
other members of the Guix community could do so as well. So we would end
up with a somewhat distributed effort, which would be nice for resilience
and spreading system administration load. Currently the web page for
the Overdrive 1000 mentions 26 available units for 600U US$ plus shipping
and tax. The Overdrive 3000 is out of stock.

For more centralised options, we could host machines at Aquilenet; in
addition to the rack in the datacentre where they host bayfront, they
just opened a "free form hosting option". Prices are here:
   https://atelier.aquilenet.fr/projects/services/wiki/Hergementlibre/7/diff
Prices are more or less proportional to electricity consumption; they state
that for 30W on average we would need to pay 20€/month. Over a few years
of lifetime this would add up quite considerably, compared to the distributed
solution.

Andreas

Re: What ARM hardware should we buy and where should we host it?

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

On 2018-09-03, Ricardo Wurmus wrote:
> - We need to be able to restart the machine remotely.  If members of the
>   Guix project have no physical access to the hosting site, this can be
>   done via remote-controlled power supply or similar.
>
> We are looking for volunteers who could either host a machine at home or
> in a data centre that they could get emergency access to.
>
> One system that looks promising for co-location is the Softiron
> Overdrive 3000 (a rack-mounted ARM server.)  A system suitable for
> hosting at home is the Softiron Overdrive 1000.  If you know of other
> machines that would be suitable for our purposes, please let us know!

For what it's worth, in my experience I don't think the Overdrive 1000
can be remotely power-cycled without physical button-pushing, at least
out of the box... maybe there's a workaround or getting a power supply
that somehow supports this...


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: What ARM hardware should we buy and where should we host it?

[Ricardo Wurmus]

Hey Tobias,

> Ricardo Wurmus wrote:
>> We are looking for volunteers who could either host a machine at
>> home or
>> in a data centre that they could get emergency access to.
>
> I've got plenty of cool, dry space for homeless boxen. No racks,
> though.

This is good!  Do you have somewhat convenient access to the space in
case the machines become unresponsive?

Would you be willing to host an Overdrive 1000 box or two?  If so, I
think we should be looking for a vendor that can deliver to your place.

--
Ricardo

Re: What ARM hardware should we buy and where should we host it?

[Ricardo Wurmus]

Vagrant Cascadian <vagrant@debian.org> writes:

> On 2018-09-03, Ricardo Wurmus wrote:
>> - We need to be able to restart the machine remotely.  If members of the
>>   Guix project have no physical access to the hosting site, this can be
>>   done via remote-controlled power supply or similar.
>>
>> We are looking for volunteers who could either host a machine at home or
>> in a data centre that they could get emergency access to.
>>
>> One system that looks promising for co-location is the Softiron
>> Overdrive 3000 (a rack-mounted ARM server.)  A system suitable for
>> hosting at home is the Softiron Overdrive 1000.  If you know of other
>> machines that would be suitable for our purposes, please let us know!
>
> For what it's worth, in my experience I don't think the Overdrive 1000
> can be remotely power-cycled without physical button-pushing, at least
> out of the box... maybe there's a workaround or getting a power supply
> that somehow supports this...

The two Overdrive 1000 machines that we have are currently hosted by
volunteers in their homes, so this is not generally a problem for us in
practice.  It would be inconvenient for co-location hosting, though.

--
Ricardo

Re: What ARM hardware should we buy and where should we host it?

[Ricardo Wurmus]

Hi Andreas,

> On Mon, Sep 03, 2018 at 01:37:54PM +0200, Ricardo Wurmus wrote:
>> The first question is what ARM hardware to buy; the second is where to
>> host that hardware.  Here are some considerations:
>
> concerning hosting, I could easily host one smaller machine (like the
> Overdrive 1000) in my living room, Tobias has stepped up, and I suppose
> other members of the Guix community could do so as well. So we would end
> up with a somewhat distributed effort, which would be nice for resilience
> and spreading system administration load.

Yes, this is good.

Close to the end of this year I could probably also start hosting one
Overdrive 1000 at my home.  (After moving to a new place.)

> For more centralised options, we could host machines at Aquilenet; in
> addition to the rack in the datacentre where they host bayfront, they
> just opened a "free form hosting option". Prices are here:
>    https://atelier.aquilenet.fr/projects/services/wiki/Hergementlibre/7/diff

Thanks for sharing this.  In my opinion, it would be good if we could
increase the number of build machines first before we allow too much of
the funds to be regularly consumed by hosting costs.

--
Ricardo

Re: What ARM hardware should we buy and where should we host it?

[Tobias Geerinckx-Rice]

Ricardo,

Ricardo Wurmus wrote:
>> Ricardo Wurmus wrote:
>>> We are looking for volunteers who could either host a machine 
>>> at
>>> home or
>>> in a data centre that they could get emergency access to.
>>
>> I've got plenty of cool, dry space for homeless boxen. No 
>> racks,
>> though.
>
> This is good!  Do you have somewhat convenient access to the 
> space in
> case the machines become unresponsive?

I tend to flee the city during week-ends and holidays, of course, 
but usually, yes.

> Would you be willing to host an Overdrive 1000 box or two?

Sure! Let me know what you decide and what I need to do.

> If so, I think we should be looking for a vendor that can 
> deliver to your place.

Righty-ho. If all else fails there's always the FOSDEM sneakernet.

Kind regards,

T G-R

Re: What ARM hardware should we buy and where should we host it?

[Andreas Enge]

On Fri, Sep 14, 2018 at 11:37:25PM +0200, Ricardo Wurmus wrote:
> Thanks for sharing this.  In my opinion, it would be good if we could
> increase the number of build machines first before we allow too much of
> the funds to be regularly consumed by hosting costs.

Indeed, I just mentioned it to give a data point, and would personally
prefer to spread out hosting for free inside the community.

Andreas