bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux

bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux

[Leo Famulari]

I see that ci.guix.gnu.org's builders seem to run out of memory while
building kernel headers for i686-linux:

------
xz: (stdin): Cannot allocate memory
/gnu/store/ns71xxkb3fzr37934bim9l8xiv68kc7w-tar-1.34/bin/tar: /gnu/store/536ifp75wv8i1kb1k0szv7zd57ygpg0n-linux-libre-6.5.13-guix.tar.xz: Wrote only 2048 of 10240 bytes
/gnu/store/ns71xxkb3fzr37934bim9l8xiv68kc7w-tar-1.34/bin/tar: Child returned status 1
/gnu/store/ns71xxkb3fzr37934bim9l8xiv68kc7w-tar-1.34/bin/tar: Error is not recoverable: exiting now
------

https://ci.guix.gnu.org/build/2736161/details

bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

On Wed, Nov 29, 2023 at 03:55:15PM -0500, Leo Famulari wrote:
> I see that ci.guix.gnu.org's builders seem to run out of memory while
> building kernel headers for i686-linux:
> 
> ------
> xz: (stdin): Cannot allocate memory
> /gnu/store/ns71xxkb3fzr37934bim9l8xiv68kc7w-tar-1.34/bin/tar: /gnu/store/536ifp75wv8i1kb1k0szv7zd57ygpg0n-linux-libre-6.5.13-guix.tar.xz: Wrote only 2048 of 10240 bytes
> /gnu/store/ns71xxkb3fzr37934bim9l8xiv68kc7w-tar-1.34/bin/tar: Child returned status 1
> /gnu/store/ns71xxkb3fzr37934bim9l8xiv68kc7w-tar-1.34/bin/tar: Error is not recoverable: exiting now
> ------
> 
> https://ci.guix.gnu.org/build/2736161/details

This looks like more of the too-many-cores while decompressing tarballs
issues we've had in the past on i686-linux.  Can we change that phase to
use a maximum of 4 cores or would that cause everything to rebuild?

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Leo Famulari]

For a long time we've not been able to build linux-libre on i686-linux
because the source unpacking process runs out of memory.

I'm forwarding this bug to guix-devel to get more attention.

Is anybody actually using i686-linux anymore? Or should we begin to
officially remove support for it?

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Kaelyn]

Hi,

On Friday, July 26th, 2024 at 11:51 AM, Leo Famulari <leo@famulari.name> wrote:

> 
> 
> For a long time we've not been able to build linux-libre on i686-linux
> because the source unpacking process runs out of memory.
> 
> I'm forwarding this bug to guix-devel to get more attention.
> 
> Is anybody actually using i686-linux anymore? Or should we begin to
> officially remove support for it?

I'm not sure about i686-linux's usage for a complete system (and I know several other distributions either already have or have plans to drop support for booting 32-bit x86 systems), but at least the "multi-lib" portion of i686-linux packages are needed for the wine and wine64 packages (and their "-staging" variants) on x86_64-linux systems.

Cheers,
Kaelyn

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

On Fri, Jul 26, 2024 at 02:51:49PM -0400, Leo Famulari wrote:
> For a long time we've not been able to build linux-libre on i686-linux
> because the source unpacking process runs out of memory.

I believe if we limit the unpacking process to not more than 8 cores we
can avoid that problem.

> I'm forwarding this bug to guix-devel to get more attention.
> 
> Is anybody actually using i686-linux anymore? Or should we begin to
> officially remove support for it?

Keeping this to i686-linux specifically, what generation of hardware
supports i686 but not x86_64? Some (very) quick checking on wikipedia
suggests that the x60 from 2006 was either 32-bit or 64-bit, and I
believe there was an atom chip from 2015 that was 32-bit. Specifically,
that makes the newest hardware (at least from the CPU perspective) 10
years old at least.

Perhaps a different question, what software _available in Guix_ is
supported by i686 that isn't supported by x86_64?

In terms of side-stepping the question, do we have enough x86_64
hardware to continue to support i686 without degrading support for
x86_64? (I ask this seriously, although I'm pretty certain the answer is
we're well covered on that front.)

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Ricardo Wurmus]

Efraim Flashner <efraim@flashner.co.il> writes:

> On Fri, Jul 26, 2024 at 02:51:49PM -0400, Leo Famulari wrote:
>> For a long time we've not been able to build linux-libre on i686-linux
>> because the source unpacking process runs out of memory.
>
> I believe if we limit the unpacking process to not more than 8 cores we
> can avoid that problem.
>
>> I'm forwarding this bug to guix-devel to get more attention.
>> 
>> Is anybody actually using i686-linux anymore? Or should we begin to
>> officially remove support for it?
>
> Keeping this to i686-linux specifically, what generation of hardware
> supports i686 but not x86_64? Some (very) quick checking on wikipedia
> suggests that the x60 from 2006 was either 32-bit or 64-bit, and I
> believe there was an atom chip from 2015 that was 32-bit. Specifically,
> that makes the newest hardware (at least from the CPU perspective) 10
> years old at least.

FWIW, I'm using one of those Atom chips in a netbook for an installation
of Sugar Desktop.  I upgrade it every few months or so.  If I'm the only
user of i686-linux I would not want to condemn the project to supporting
the architecture for my sake.

We have quite a few package failures on i686-linux, often because of
failing precision tests.  It may not be worth attempting to fix these
problems, e.g. for R, because it is very unlikely that people use R on
i686 machines.

> In terms of side-stepping the question, do we have enough x86_64
> hardware to continue to support i686 without degrading support for
> x86_64? (I ask this seriously, although I'm pretty certain the answer is
> we're well covered on that front.)

Support does not just mean dedicating build cycles to doomed builds, but
also dedicating people's time to wade through hundreds of failures for
little gain.  Perhaps our time is better spent supporting architectures
that still have a future.

-- 
Ricardo

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Richard Sent]

>> In terms of side-stepping the question, do we have enough x86_64
>> hardware to continue to support i686 without degrading support for
>> x86_64? (I ask this seriously, although I'm pretty certain the answer is
>> we're well covered on that front.)
>
>Support does not just mean dedicating build cycles to doomed builds, but
>also dedicating people's time to wade through hundreds of failures for
>little gain.  Perhaps our time is better spent supporting architectures
>that still have a future.
>

For consideration, I know at least one 3rd-party channel relies on being able to create a multiarch container containing i686 packages. I'll refrain from linking since it packages nonfree software. This is an example where keeping an old architecture around is more complicated than simply counting the number of active machines using said architecture.

Perhaps we could tally the number of substitutes served for supported architectures and use that as our metric for liveliness.

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Leo Famulari]

On Mon, Jul 29, 2024 at 11:00:36AM -0400, Richard Sent wrote:
> For consideration, I know at least one 3rd-party channel relies on being able to create a multiarch container containing i686 packages. I'll refrain from linking since it packages nonfree software. This is an example where keeping an old architecture around is more complicated than simply counting the number of active machines using said architecture.

People have presented some good reasons for keeping at least some level
of i686 support.

But unfortunately, 3rd party channels cannot be one of them, whether or
not they follow the FSDG.

Of course, we won't deliberately make their work more difficult, and
maybe we consider their needs if it's easy, but I think they shouldn't
be considered to present compelling arguments for us to make decisions
within GNU Guix, especially if it involves us doing extra work.

> Perhaps we could tally the number of substitutes served for supported architectures and use that as our metric for liveliness.

I'd love this! Not just for deciding when to remove support, but to
measure if adding support gains more users.

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Richard Sent]

Leo Famulari <leo@famulari.name> writes:

> People have presented some good reasons for keeping at least some level
> of i686 support.
> 
> But unfortunately, 3rd party channels cannot be one of them, whether or
> not they follow the FSDG.
>
> Of course, we won't deliberately make their work more difficult, and
> maybe we consider their needs if it's easy, but I think they shouldn't
> be considered to present compelling arguments for us to make decisions
> within GNU Guix, especially if it involves us doing extra work.

That's true enough! I don't mean to say that 3rd party channels using
i686 is sufficient reason alone to support it. I just consider it worth
keeping in mind.

In my opinion, when we ask questions like "Does anyone use X", it
doesn't really matter if that answer is "Yes, in my custom config" vs.
"Yes, in this 3rd party channel my custom config uses". The primary
distinction between the two is if the code is shared publicly. I don't
see that line in the sand being helpful when asking about usage.

To phrase this another way, if I instead said "I use multiarch
environments containing i686-linux Guix packages to run software that
can't be ported to x64" without mentioning 3rd-party channels at all,
would that suddenly become valid usage? Why?

i686 multiarch environments are useful in certain cases. Regardless of
whether those environments are provided in Guix proper, in a custom
config, or a 3rd party channel, user-facing functionality will be lost
if we remove them.

Breaking changes are okay, and if we consider this too niche of a use
case or too high of a maintenance burden it should be dropped. I do
believe it should progress into the consideration stage instead of being
discarded outright.

Thanks! :)

-- 
Take it easy,
Richard Sent
Making my computer weirder one commit at a time.

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Leo Famulari]

I basically agree with you :)

On Mon, Jul 29, 2024 at 09:21:57PM -0400, Richard Sent wrote:
> Breaking changes are okay, and if we consider this too niche of a use
> case or too high of a maintenance burden it should be dropped. I do
> believe it should progress into the consideration stage instead of being
> discarded outright.

It's not really a maintenance burden, at least for me as a person that
helps with the kernel packages. They get updated automatically as part
of our work updating the kernels for more popular systems.

But after years of watching the i686 kernel packages fail to build, I'm
wondering if the project should be attempting these builds.

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

[dropped address for defect report]

Hi Leo,

On Tue, Jul 30 2024, Leo Famulari wrote:

> After years of watching the i686 kernel packages fail to build

Sorry to burden the list.  A brief note about your and Wilko's tireless
but possibly under-appreciated commitment to kernels seemed in order.
Thank you!

Kind regards
Felix

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3843 bytes --]

On Mon, Jul 29, 2024 at 09:21:57PM -0400, Richard Sent wrote:
> Leo Famulari <leo@famulari.name> writes:
> 
> > People have presented some good reasons for keeping at least some level
> > of i686 support.
> > 
> > But unfortunately, 3rd party channels cannot be one of them, whether or
> > not they follow the FSDG.
> >
> > Of course, we won't deliberately make their work more difficult, and
> > maybe we consider their needs if it's easy, but I think they shouldn't
> > be considered to present compelling arguments for us to make decisions
> > within GNU Guix, especially if it involves us doing extra work.
> 
> That's true enough! I don't mean to say that 3rd party channels using
> i686 is sufficient reason alone to support it. I just consider it worth
> keeping in mind.
> 
> In my opinion, when we ask questions like "Does anyone use X", it
> doesn't really matter if that answer is "Yes, in my custom config" vs.
> "Yes, in this 3rd party channel my custom config uses". The primary
> distinction between the two is if the code is shared publicly. I don't
> see that line in the sand being helpful when asking about usage.
> 
> To phrase this another way, if I instead said "I use multiarch
> environments containing i686-linux Guix packages to run software that
> can't be ported to x64" without mentioning 3rd-party channels at all,
> would that suddenly become valid usage? Why?
> 
> i686 multiarch environments are useful in certain cases. Regardless of
> whether those environments are provided in Guix proper, in a custom
> config, or a 3rd party channel, user-facing functionality will be lost
> if we remove them.
> 
> Breaking changes are okay, and if we consider this too niche of a use
> case or too high of a maintenance burden it should be dropped. I do
> believe it should progress into the consideration stage instead of being
> discarded outright.
> 
> Thanks! :)

I would argue that some of the bootstrapping effort which is i686
specifically and can't be easily ported to x86_64 (such as compilers)
are a perfectly fine reason to need something to be built natively vs
cross-compiled. Another email mentioned wine, which, while I don't
believe it is currently possible to cross-compile in guix, may or may
not work correctly when used cross-compiled as an input for wine64.

Without directly answering the question of "is the phrasing wrong" vs
"is the burden too high", IMO there's not really a difference between a
package in a separate channel vs a custom package in someone's config,
other than how easy it is to share. If we said, despite the move to Qt6
and upstream chromium dropping support for 32-bit architectures and thus
affecting i686 support in qtwebengine, that it was imperative that i686
keep a working qtwebengine and that we couldn't upgrade it unless we
knew it worked on i686 that might be a problem due to "The Dangers of
the Internets", but ongoing work to update patches to keep it working
would be good. Or I suppose another example is if we froze Gnome at a
version that supported the old librsvg because the new one depends on
rust, instead we've worked around it so that those that can't use the
new one use the old one, and those packages which can't use the old one
specifically use the new one, with the side effect that gnome isn't
supported on all architectures.

I would not be against selecting some scientific packages and marking
them as 64bit only with a note that although they might build on 32bit
architectures, they would never be used there and there is no reason to
try to even build them.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#67535: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[André Batista]

Hi!

seg 29 jul 2024 às 14:33:59 (1722274439), rekado@elephly.net enviou:
> Efraim Flashner <efraim@flashner.co.il> writes:
> 
> > On Fri, Jul 26, 2024 at 02:51:49PM -0400, Leo Famulari wrote:
> >> For a long time we've not been able to build linux-libre on i686-linux
> >> because the source unpacking process runs out of memory.
> >
> > I believe if we limit the unpacking process to not more than 8 cores we
> > can avoid that problem.
> >
> >> I'm forwarding this bug to guix-devel to get more attention.
> >> 
> >> Is anybody actually using i686-linux anymore? Or should we begin to
> >> officially remove support for it?
> >
> > Keeping this to i686-linux specifically, what generation of hardware
> > supports i686 but not x86_64? Some (very) quick checking on wikipedia
> > suggests that the x60 from 2006 was either 32-bit or 64-bit, and I
> > believe there was an atom chip from 2015 that was 32-bit. Specifically,
> > that makes the newest hardware (at least from the CPU perspective) 10
> > years old at least.
> 
> FWIW, I'm using one of those Atom chips in a netbook for an installation
> of Sugar Desktop.  I upgrade it every few months or so.  If I'm the only
> user of i686-linux I would not want to condemn the project to supporting
> the architecture for my sake.

For the record, I'm another one still using those atom netbooks. Most
software that I use on that machine still builds and runs fine, with the
occasional hiccup.

But even though I use the arch, I also don't feel particularly inclined
to fix the occasional errors and can understand if people here decide to
drop support to it.

bug#67535: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Leo Famulari]

On Tue, Jul 30, 2024 at 06:02:23PM -0300, André Batista wrote:
> seg 29 jul 2024 às 14:33:59 (1722274439), rekado@elephly.net enviou:
> > FWIW, I'm using one of those Atom chips in a netbook for an installation
> > of Sugar Desktop.  I upgrade it every few months or so.  If I'm the only
> > user of i686-linux I would not want to condemn the project to supporting
> > the architecture for my sake.
> 
> For the record, I'm another one still using those atom netbooks. Most
> software that I use on that machine still builds and runs fine, with the
> occasional hiccup.
> 
> But even though I use the arch, I also don't feel particularly inclined
> to fix the occasional errors and can understand if people here decide to
> drop support to it.

Thanks for chiming in Ricardo and André. Do you build your own kernels
for these machines? Or wait for the occasional successful build from CI?
Download substitutes from a different build farm?

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> On Tue, Jul 30, 2024 at 06:02:23PM -0300, André Batista wrote:
>> seg 29 jul 2024 às 14:33:59 (1722274439), rekado@elephly.net enviou:
>> > FWIW, I'm using one of those Atom chips in a netbook for an installation
>> > of Sugar Desktop.  I upgrade it every few months or so.  If I'm the only
>> > user of i686-linux I would not want to condemn the project to supporting
>> > the architecture for my sake.
>> 
>> For the record, I'm another one still using those atom netbooks. Most
>> software that I use on that machine still builds and runs fine, with the
>> occasional hiccup.
>> 
>> But even though I use the arch, I also don't feel particularly inclined
>> to fix the occasional errors and can understand if people here decide to
>> drop support to it.
>
> Thanks for chiming in Ricardo and André. Do you build your own kernels
> for these machines? Or wait for the occasional successful build from CI?
> Download substitutes from a different build farm?

I used to get the kernel from ci.guix.gnu.org.  I haven't updated that
system in at least 6 months, though.  (It's not networked and used for
the occasional game.)

I use "guix deploy" for all weak machines at home and build whatever
might be missing on the targets on my x86_64 laptop.  For aarch64 this
means a regular build of a custom kernel.  For i686 I use the stock
kernel.

-- 
Ricardo

bug#67535: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[André Batista]

Hi

qui 01 ago 2024 às 16:12:18 (1722539538), leo@famulari.name enviou:
> On Tue, Jul 30, 2024 at 06:02:23PM -0300, André Batista wrote:
> > seg 29 jul 2024 às 14:33:59 (1722274439), rekado@elephly.net enviou:
> > > FWIW, I'm using one of those Atom chips in a netbook for an installation
> > > of Sugar Desktop.  I upgrade it every few months or so.  If I'm the only
> > > user of i686-linux I would not want to condemn the project to supporting
> > > the architecture for my sake.
> > 
> > For the record, I'm another one still using those atom netbooks. Most
> > software that I use on that machine still builds and runs fine, with the
> > occasional hiccup.
> > 
> > But even though I use the arch, I also don't feel particularly inclined
> > to fix the occasional errors and can understand if people here decide to
> > drop support to it.
> 
> Thanks for chiming in Ricardo and André. Do you build your own kernels
> for these machines? Or wait for the occasional successful build from CI?
> Download substitutes from a different build farm?

I build my own kernels tailored for that machine so I did not notice that
substitutes were not available. I usually keep pace with whatever is the
latest stable kernel until it goes eol or, if it is a lts, until the
latest stable reaches x.x.3 or x.x.4 minor version. Currently it is on
v. 6.9.12.

I've not had any issues building kernels to it in a long time, but I do
use two local offload builders that are x84_64. None of them have more
than 8 cores though.

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Ricardo Wurmus]

André Batista <nandre@riseup.net> writes:

>> > Keeping this to i686-linux specifically, what generation of hardware
>> > supports i686 but not x86_64? Some (very) quick checking on wikipedia
>> > suggests that the x60 from 2006 was either 32-bit or 64-bit, and I
>> > believe there was an atom chip from 2015 that was 32-bit. Specifically,
>> > that makes the newest hardware (at least from the CPU perspective) 10
>> > years old at least.
>> 
>> FWIW, I'm using one of those Atom chips in a netbook for an installation
>> of Sugar Desktop.  I upgrade it every few months or so.  If I'm the only
>> user of i686-linux I would not want to condemn the project to supporting
>> the architecture for my sake.
>
> For the record, I'm another one still using those atom netbooks.

I just noticed that my Atom-powered netbook should also be able to run
an x86_64 system.  I'll try to upgrade today; if this is successful I
won't have any i686 system left at home.

-- 
Ricardo

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[André Batista]

qui 05 set 2024 às 11:42:41 (1725547361), rekado@elephly.net enviou:
> André Batista <nandre@riseup.net> writes:
> 
> >> > Keeping this to i686-linux specifically, what generation of hardware
> >> > supports i686 but not x86_64? Some (very) quick checking on wikipedia
> >> > suggests that the x60 from 2006 was either 32-bit or 64-bit, and I
> >> > believe there was an atom chip from 2015 that was 32-bit. Specifically,
> >> > that makes the newest hardware (at least from the CPU perspective) 10
> >> > years old at least.
> >> 
> >> FWIW, I'm using one of those Atom chips in a netbook for an installation
> >> of Sugar Desktop.  I upgrade it every few months or so.  If I'm the only
> >> user of i686-linux I would not want to condemn the project to supporting
> >> the architecture for my sake.
> >
> > For the record, I'm another one still using those atom netbooks.
> 
> I just noticed that my Atom-powered netbook should also be able to run
> an x86_64 system.  I'll try to upgrade today; if this is successful I
> won't have any i686 system left at home.

Lucky you! Mine is an Intel Atom N270 which is 32-bit only...

Re: Does anyone use i686-linux? [was Re: bug#67535: ci.guix.gnu.org 'Cannot allocate memory' while building for i686-linux]

[Ludovic Courtès]

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Fri, Jul 26, 2024 at 02:51:49PM -0400, Leo Famulari wrote:
>> For a long time we've not been able to build linux-libre on i686-linux
>> because the source unpacking process runs out of memory.
>
> I believe if we limit the unpacking process to not more than 8 cores we
> can avoid that problem.

Also, this is very much a defect of xz; on ‘core-updates’,
‘patch-and-repack’ uses zstd, which is much less memory-hungry and more
predictable.

Ludo’.