all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* bug#53696: Integer overflow on Guix GC size calculation
@ 2022-02-01 14:06 Ekaitz Zarraga
  2022-02-01 21:20 ` Liliana Marie Prikler
  2022-02-02 10:05 ` Maxime Devos
  0 siblings, 2 replies; 13+ messages in thread
From: Ekaitz Zarraga @ 2022-02-01 14:06 UTC (permalink / raw)
  To: 53696

Hi,

I noticed there's some kind of a wierd integer overflow on the size calculation on `guix gc`:

[17592186042896 MiB] deleting '/gnu/store/j2s6kva8l20m6rjj10bnknq99jc9rg6w-ghc-random-1.2.0-builder'
[17592186042896 MiB] deleting '/gnu/store/8nx7zzj629qvv1533c48bl19wrkwcjh2-curl-7.79.1-doc'
[17592186042897 MiB] deleting '/gnu/store/dcsi13588yyjws76s1wjc7h5spnjd2vn-ghc-kan-extensions-5.2.3-builder'
[17592186042897 MiB] deleting '/gnu/store/5zrhw6kvb8wd3n6lazpblqzsg92y320b-ghc-sop-core-0.5.0.1-builder'
[17592186042897 MiB] deleting '/gnu/store/l2ya1z3la9qfdj9139f09q3djs36lw3l-ghc-aeson-pretty-0.8.9-builder'
[17592186042897 MiB] deleting '/gnu/store/8a8nbfxq508r7qywkhaw8jj8hicpfjh8-ghc-prelude-extras-0.4.0.3-builder'
[17592186042897 MiB] deleting '/gnu/store/wbz6vkiz7cq8c531xvb31lxm28nz332i-ghc-8.10.7'
[19 MiB] deleting '/gnu/store/i5np7ifiabg333g2l8ycmvbhimnrrx8k-ghc-8.10.7-doc'
[170 MiB] deleting '/gnu/store/yx9zjw9118gzfcx33adfwy6kghrzxkys-ghc-pem-0.2.4-builder'
[170 MiB] deleting '/gnu/store/pinvkg1x5rpsgm95zhn50l6xq7rly43f-perl-test-output-1.033.drv'
[170 MiB] deleting '/gnu/store/k1bdc950d62g1pw4yw1khgmfr32m3rpm-perl-sub-exporter-0.988.drv'





^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-01 14:06 bug#53696: Integer overflow on Guix GC size calculation Ekaitz Zarraga
@ 2022-02-01 21:20 ` Liliana Marie Prikler
  2022-02-01 21:54   ` Ekaitz Zarraga
  2022-02-02 10:05 ` Maxime Devos
  1 sibling, 1 reply; 13+ messages in thread
From: Liliana Marie Prikler @ 2022-02-01 21:20 UTC (permalink / raw)
  To: Ekaitz Zarraga, 53696

Am Dienstag, dem 01.02.2022 um 14:06 +0000 schrieb Ekaitz Zarraga:
> Hi,
> 
> I noticed there's some kind of a wierd integer overflow on the size
> calculation on `guix gc`:
> 
> [17592186042896 MiB] deleting
> '/gnu/store/j2s6kva8l20m6rjj10bnknq99jc9rg6w-ghc-random-1.2.0-
> builder'
> [17592186042896 MiB] deleting
> '/gnu/store/8nx7zzj629qvv1533c48bl19wrkwcjh2-curl-7.79.1-doc'
> [17592186042897 MiB] deleting
> '/gnu/store/dcsi13588yyjws76s1wjc7h5spnjd2vn-ghc-kan-extensions-
> 5.2.3-builder'
> [17592186042897 MiB] deleting
> '/gnu/store/5zrhw6kvb8wd3n6lazpblqzsg92y320b-ghc-sop-core-0.5.0.1-
> builder'
> [17592186042897 MiB] deleting
> '/gnu/store/l2ya1z3la9qfdj9139f09q3djs36lw3l-ghc-aeson-pretty-0.8.9-
> builder'
> [17592186042897 MiB] deleting
> '/gnu/store/8a8nbfxq508r7qywkhaw8jj8hicpfjh8-ghc-prelude-extras-
> 0.4.0.3-builder'
> [17592186042897 MiB] deleting
> '/gnu/store/wbz6vkiz7cq8c531xvb31lxm28nz332i-ghc-8.10.7'
> [19 MiB] deleting '/gnu/store/i5np7ifiabg333g2l8ycmvbhimnrrx8k-ghc-
> 8.10.7-doc'
> [170 MiB] deleting '/gnu/store/yx9zjw9118gzfcx33adfwy6kghrzxkys-ghc-
> pem-0.2.4-builder'
> [170 MiB] deleting '/gnu/store/pinvkg1x5rpsgm95zhn50l6xq7rly43f-perl-
> test-output-1.033.drv'
> [170 MiB] deleting '/gnu/store/k1bdc950d62g1pw4yw1khgmfr32m3rpm-perl-
> sub-exporter-0.988.drv'
I find it somewhat concerning that you've accumulated more than 2^64
bytes of garbage.  Are some items counted double here?
Other than that, that's pretty normal size_t wraparound semantics.  I
don't think that number is used for anything other than displaying.

Cheers




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-01 21:20 ` Liliana Marie Prikler
@ 2022-02-01 21:54   ` Ekaitz Zarraga
  2022-02-02 19:45     ` Liliana Marie Prikler
  0 siblings, 1 reply; 13+ messages in thread
From: Ekaitz Zarraga @ 2022-02-01 21:54 UTC (permalink / raw)
  To: Liliana Marie Prikler; +Cc: 53696


Hi,

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Tuesday, February 1st, 2022 at 10:20 PM, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:
>
> I find it somewhat concerning that you've accumulated more than 2^64
> bytes of garbage.

I'm a dirty boy.

> Are some items counted double here?

The number started growing from 0 and then that appeared and it continued
smoothly from the previous. It's like something went bad in the middle.

> Other than that, that's pretty normal size_t wraparound semantics. I
> don't think that number is used for anything other than displaying.

Showing wrong information to the people that use the program is pretty
weird. The program still works but showing wrong data is worse than
not showing it in my opinion.
I'll take a look and try to see if I can fix it.

> Cheers

Best,
Ekaitz




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-01 14:06 bug#53696: Integer overflow on Guix GC size calculation Ekaitz Zarraga
  2022-02-01 21:20 ` Liliana Marie Prikler
@ 2022-02-02 10:05 ` Maxime Devos
  2022-02-02 12:04   ` Bengt Richter
  1 sibling, 1 reply; 13+ messages in thread
From: Maxime Devos @ 2022-02-02 10:05 UTC (permalink / raw)
  To: Ekaitz Zarraga, 53696

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Ekaitz Zarraga schreef op di 01-02-2022 om 14:06 [+0000]:
> [17592186042897 MiB] deleting '/gnu/store/wbz6vkiz7cq8c531xvb31lxm28nz332i-ghc-8.10.7'

For comparison, this is about 16 exbibyte.
According to <https://en.wikipedia.org/wiki/Byte#Multiple-byte_units>,
that's more than the global monthly Internet traffic in 2004.

According to <https://what-if.xkcd.com/31/>, 16 exbibyte would be about
17 million solid-state disks.  Even though this ignores deduplication,
this seems rather expensive. 

My guess is that the size of a store item was misrecorded somewhere.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-02 10:05 ` Maxime Devos
@ 2022-02-02 12:04   ` Bengt Richter
  2022-02-02 22:46     ` Bengt Richter
  2022-02-08 10:01     ` Ludovic Courtès
  0 siblings, 2 replies; 13+ messages in thread
From: Bengt Richter @ 2022-02-02 12:04 UTC (permalink / raw)
  To: Maxime Devos; +Cc: 53696, Ekaitz Zarraga

Hi Maxime, Ekaitz, et al,

On +2022-02-02 11:05:31 +0100, Maxime Devos wrote:
> Ekaitz Zarraga schreef op di 01-02-2022 om 14:06 [+0000]:
> > [17592186042897 MiB] deleting '/gnu/store/wbz6vkiz7cq8c531xvb31lxm28nz332i-ghc-8.10.7'
> 
> For comparison, this is about 16 exbibyte.
> According to <https://en.wikipedia.org/wiki/Byte#Multiple-byte_units>,
> that's more than the global monthly Internet traffic in 2004.
> 
> According to <https://what-if.xkcd.com/31/>, 16 exbibyte would be about
> 17 million solid-state disks.  Even though this ignores deduplication,
> this seems rather expensive. 
> 
> My guess is that the size of a store item was misrecorded somewhere.
> 
> Greetings,
> Maxime.

s/misrecorded/mis-defined-in-record/ ?
Wild guessing follows:

--8<---------------cut here---------------start------------->8---
$ guile --no-auto-compile -c '(use-modules (ice-9 format))(format #t "~20x\n~20x\n~20d\n" (* 17592186042897 (expt 2 20)) #xa1100000 #xa1100000)';
    ffffffffa1100000
            a1100000
          2702180352
--8<---------------cut here---------------end--------------->8---

It looks to me like a 32-bit unsigned int should have been turned to 64-bit unsigned long or bigint
but somehow got cast/interpreted as signed, becoming signed 64-bit long,
which then in turn was seen by the print as 64-bit unsigned long.

I don't know, but if records are being used, perhaps some slot integer-widening logic
might be involved? Or a mis-defined int slot that should have been long to accomodate
big > 31-bit  positive integers?

Just guessing wildly -- I think I saw something about records and defining their fields
as fixed C ints or longs.

-- 
Regards,
Bengt Richter




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-01 21:54   ` Ekaitz Zarraga
@ 2022-02-02 19:45     ` Liliana Marie Prikler
  2022-02-02 19:51       ` Ekaitz Zarraga
  0 siblings, 1 reply; 13+ messages in thread
From: Liliana Marie Prikler @ 2022-02-02 19:45 UTC (permalink / raw)
  To: Ekaitz Zarraga; +Cc: 53696

Hi,

Am Dienstag, dem 01.02.2022 um 21:54 +0000 schrieb Ekaitz Zarraga:
> 
> Hi,
> 
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> 
> On Tuesday, February 1st, 2022 at 10:20 PM, Liliana Marie Prikler
> <liliana.prikler@gmail.com> wrote:
> > 
> > I find it somewhat concerning that you've accumulated more than
> > 2^64 bytes of garbage.
> 
> I'm a dirty boy.
> 
> > Are some items counted double here?
> 
> The number started growing from 0 and then that appeared and it
> continued smoothly from the previous. It's like something went bad in
> the middle.
WDYM by the middle?  Do you mean the jump back to 0 or do you mean
something before those lines?  If you did encounter a "self-correcting"
bit-flip that'd be one thing, but to me it appears as though you have
some very large storage on your hands.  Would you mind me asking where
you purchased that disk 😜

> > Other than that, that's pretty normal size_t wraparound semantics.
> > I don't think that number is used for anything other than
> > displaying.
> 
> Showing wrong information to the people that use the program is
> pretty weird. The program still works but showing wrong data is worse
> than not showing it in my opinion.
> I'll take a look and try to see if I can fix it.
I mean we could switch to GMP for those numbers, but it doesn't make
sense.  Ext4 volume size is capped at 2^60, which is still pretty well
below 2^64.  Even BTRFS can't get larger than that.  So unless you have
a distributed store, I'd hazard a guess that such numbers ought not to
even appear.

Cheers




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-02 19:45     ` Liliana Marie Prikler
@ 2022-02-02 19:51       ` Ekaitz Zarraga
  2022-02-02 20:04         ` Liliana Marie Prikler
  0 siblings, 1 reply; 13+ messages in thread
From: Ekaitz Zarraga @ 2022-02-02 19:51 UTC (permalink / raw)
  To: Liliana Marie Prikler; +Cc: 53696

Hi,

On Wednesday, February 2nd, 2022 at 8:45 PM, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

> Hi,
>
> Am Dienstag, dem 01.02.2022 um 21:54 +0000 schrieb Ekaitz Zarraga:
>
> > Hi,
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >
> > On Tuesday, February 1st, 2022 at 10:20 PM, Liliana Marie Prikler
> >
> > liliana.prikler@gmail.com wrote:
> >
> > > I find it somewhat concerning that you've accumulated more than
> > >
> > > 2^64 bytes of garbage.
> >
> > I'm a dirty boy.
> >
> > > Are some items counted double here?
> >
> > The number started growing from 0 and then that appeared and it
> > continued smoothly from the previous. It's like something went bad in
> > the middle.
>
> WDYM by the middle? Do you mean the jump back to 0 or do you mean
> something before those lines? If you did encounter a "self-correcting"
> bit-flip that'd be one thing, but to me it appears as though you have
> some very large storage on your hands. Would you mind me asking where
> you purchased that disk 😜

I mean something like:

0
1
2
4
8
10
12
HUGE_NUMBER
HUGE_NUMBER
...
HUGE_NUMBER
15
20
...

It's like it corrected itself. It happened in "low numbers" (less than a
hundred).

I just say this if it helps in the correction. It's very funny, still :3




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-02 19:51       ` Ekaitz Zarraga
@ 2022-02-02 20:04         ` Liliana Marie Prikler
  0 siblings, 0 replies; 13+ messages in thread
From: Liliana Marie Prikler @ 2022-02-02 20:04 UTC (permalink / raw)
  To: Ekaitz Zarraga; +Cc: 53696

Hi,

Am Mittwoch, dem 02.02.2022 um 19:51 +0000 schrieb Ekaitz Zarraga:
> I mean something like:
> 
> 0
> 1
> 2
> 4
> 8
> 10
> 12
> HUGE_NUMBER
> HUGE_NUMBER
> ...
> HUGE_NUMBER
> 15
> 20
> ...
> 
> It's like it corrected itself. It happened in "low numbers" (less
> than a
> hundred).
> 
> I just say this if it helps in the correction. It's very funny, still
> :3
Thanks, that wasn't clear from your original report.  As I hinted at in
my previous message, I think you'd get such results through well-placed
bit flips.  I'm not aware of Guix itself intentionally or otherwise
causing those, but bit flips are a problem on any modern hardware and
thus I'm sure such glitches will be encountered.




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-02 12:04   ` Bengt Richter
@ 2022-02-02 22:46     ` Bengt Richter
  2022-02-08 10:01     ` Ludovic Courtès
  1 sibling, 0 replies; 13+ messages in thread
From: Bengt Richter @ 2022-02-02 22:46 UTC (permalink / raw)
  To: Maxime Devos; +Cc: 53696, Ekaitz Zarraga

Sorry for following up my own post, but maybe it wasn't clear
why I printed (* 17592186042897 (expt 2 20)) in hex ? 

That is the value of [17592186042897 MiB] that you've been discussing.
(expt 2 20) is one MiB

Does that make
> --8<---------------cut here---------------start------------->8---
> $ guile --no-auto-compile -c '(use-modules (ice-9 format))(format #t "~20x\n~20x\n~20d\n" (* 17592186042897 (expt 2 20)) #xa1100000 #xa1100000)';
>     ffffffffa1100000
>             a1100000
>           2702180352
> --8<---------------cut here---------------end--------------->8---
a little clearer?

The discussion seems to be continuing, but no mention of the above.
How come?

Feeling ignored, and top-posting in desperation ;/

CC-ing ludo, who will instantly know where to fix it, if he hasn't already.


On +2022-02-02 13:04:41 +0100, Bengt Richter wrote:
> Hi Maxime, Ekaitz, et al,
> 
> On +2022-02-02 11:05:31 +0100, Maxime Devos wrote:
> > Ekaitz Zarraga schreef op di 01-02-2022 om 14:06 [+0000]:
> > > [17592186042897 MiB] deleting '/gnu/store/wbz6vkiz7cq8c531xvb31lxm28nz332i-ghc-8.10.7'
> > 
> > For comparison, this is about 16 exbibyte.
> > According to <https://en.wikipedia.org/wiki/Byte#Multiple-byte_units>,
> > that's more than the global monthly Internet traffic in 2004.
> > 
> > According to <https://what-if.xkcd.com/31/>, 16 exbibyte would be about
> > 17 million solid-state disks.  Even though this ignores deduplication,
> > this seems rather expensive. 
> > 
> > My guess is that the size of a store item was misrecorded somewhere.
> > 
> > Greetings,
> > Maxime.
> 
> s/misrecorded/mis-defined-in-record/ ?
> Wild guessing follows:
> 
> --8<---------------cut here---------------start------------->8---
> $ guile --no-auto-compile -c '(use-modules (ice-9 format))(format #t "~20x\n~20x\n~20d\n" (* 17592186042897 (expt 2 20)) #xa1100000 #xa1100000)';
>     ffffffffa1100000
>             a1100000
>           2702180352
> --8<---------------cut here---------------end--------------->8---
> 
> It looks to me like a 32-bit unsigned int should have been turned to 64-bit unsigned long or bigint
> but somehow got cast/interpreted as signed, becoming signed 64-bit long,
> which then in turn was seen by the print as 64-bit unsigned long.
> 
> I don't know, but if records are being used, perhaps some slot integer-widening logic
> might be involved? Or a mis-defined int slot that should have been long to accomodate
> big > 31-bit  positive integers?
> 
> Just guessing wildly -- I think I saw something about records and defining their fields
> as fixed C ints or longs.
> 
> -- 
> Regards,
> Bengt Richter
> 
> 
> 




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-02 12:04   ` Bengt Richter
  2022-02-02 22:46     ` Bengt Richter
@ 2022-02-08 10:01     ` Ludovic Courtès
  2022-02-08 10:52       ` Ekaitz Zarraga
  1 sibling, 1 reply; 13+ messages in thread
From: Ludovic Courtès @ 2022-02-08 10:01 UTC (permalink / raw)
  To: Bengt Richter; +Cc: 53696, Ekaitz Zarraga

Hi!

Bengt Richter <bokr@bokr.com> skribis:

> It looks to me like a 32-bit unsigned int should have been turned to 64-bit unsigned long or bigint
> but somehow got cast/interpreted as signed, becoming signed 64-bit long,
> which then in turn was seen by the print as 64-bit unsigned long.

That could be the explanation; there were a couple of bugs along these
lines fixed recently:

  https://issues.guix.gnu.org/51983#4

It would have been good to see which store item that came right before
the huge number.  Could it be that it was texlive-texmf, or flightgear,
or repeat-masker, the three packages whose store item size exceeds 2^31?

Ekaitz, what guix-daemon version are you running?

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-08 10:01     ` Ludovic Courtès
@ 2022-02-08 10:52       ` Ekaitz Zarraga
  2022-02-10 20:25         ` Ludovic Courtès
  0 siblings, 1 reply; 13+ messages in thread
From: Ekaitz Zarraga @ 2022-02-08 10:52 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 53696


> Ekaitz, what guix-daemon version are you running?

$ guix --version
guix (GNU Guix) c5000dcc375229ff42727f090d4243107d3a04a6
Copyright (C) 2022 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

TY!




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-08 10:52       ` Ekaitz Zarraga
@ 2022-02-10 20:25         ` Ludovic Courtès
  2022-02-10 20:27           ` Ekaitz Zarraga
  0 siblings, 1 reply; 13+ messages in thread
From: Ludovic Courtès @ 2022-02-10 20:25 UTC (permalink / raw)
  To: Ekaitz Zarraga; +Cc: 53696

Ekaitz Zarraga <ekaitz@elenq.tech> skribis:

>> Ekaitz, what guix-daemon version are you running?
>
> $ guix --version

What about the daemon though?  As in:

  sudo /proc/$(pidof guix-daemon |cut -f1 -d ' ')/exe --version

Thanks,
Ludo’.




^ permalink raw reply	[flat|nested] 13+ messages in thread

* bug#53696: Integer overflow on Guix GC size calculation
  2022-02-10 20:25         ` Ludovic Courtès
@ 2022-02-10 20:27           ` Ekaitz Zarraga
  0 siblings, 0 replies; 13+ messages in thread
From: Ekaitz Zarraga @ 2022-02-10 20:27 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 53696


> Ekaitz Zarraga ekaitz@elenq.tech skribis:
>
> > > Ekaitz, what guix-daemon version are you running?
> >
> > $ guix --version
>
> What about the daemon though? As in:
>
> sudo /proc/$(pidof guix-daemon |cut -f1 -d ' ')/exe --version
>
> Thanks,
>
> Ludo’.

ouch... sorry!

$   sudo /proc/$(pidof guix-daemon |cut -f1 -d ' ')/exe --version
Password:
guix-daemon (GNU Guix) 1.3.0-23.a27e47f




^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2022-02-10 20:29 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-02-01 14:06 bug#53696: Integer overflow on Guix GC size calculation Ekaitz Zarraga
2022-02-01 21:20 ` Liliana Marie Prikler
2022-02-01 21:54   ` Ekaitz Zarraga
2022-02-02 19:45     ` Liliana Marie Prikler
2022-02-02 19:51       ` Ekaitz Zarraga
2022-02-02 20:04         ` Liliana Marie Prikler
2022-02-02 10:05 ` Maxime Devos
2022-02-02 12:04   ` Bengt Richter
2022-02-02 22:46     ` Bengt Richter
2022-02-08 10:01     ` Ludovic Courtès
2022-02-08 10:52       ` Ekaitz Zarraga
2022-02-10 20:25         ` Ludovic Courtès
2022-02-10 20:27           ` Ekaitz Zarraga

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.