unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / Atom feed
* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
@ 2021-01-31 14:47 Christopher Baines
  2021-01-31 15:20 ` Ludovic Courtès
  2021-02-01 19:57 ` Christopher Baines
  0 siblings, 2 replies; 9+ messages in thread
From: Christopher Baines @ 2021-01-31 14:47 UTC (permalink / raw)
  To: 46212; +Cc: Mathieu Othacehe

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

I noticed through the Guix Data Service that some narinfo files from
ci.guix.gnu.org have an excessive NarSize.

These are the three that I've found, but there could be more.


/gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
NarSize: 18446744072099351584

/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
NarSize: 18446744071612438544

/gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
NarSize: 18446744071612438544


There's additional information on IRC: http://logs.guix.gnu.org/guix/2021-01-31.log#152751

Cc'ing Mathieu in case this is related to the new offloading mechanism.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-01-31 14:47 bug#46212: ci.guix.gnu.org narinfos with excessive NarSize Christopher Baines
@ 2021-01-31 15:20 ` Ludovic Courtès
  2021-02-01  9:15   ` Ludovic Courtès
  2021-02-19 15:11   ` Ludovic Courtès
  2021-02-01 19:57 ` Christopher Baines
  1 sibling, 2 replies; 9+ messages in thread
From: Ludovic Courtès @ 2021-01-31 15:20 UTC (permalink / raw)
  To: Christopher Baines; +Cc: Mathieu Othacehe, 46212

Christopher Baines <mail@cbaines.net> skribis:

> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
> NarSize: 18446744072099351584
>
> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
> NarSize: 18446744071612438544
>
> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
> NarSize: 18446744071612438544

The key point here is that ‘narSize’ in the database is negative:

--8<---------------cut here---------------start------------->8---
sqlite> select * from validpaths where path = '/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1';
43262123|/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1|sha256:33328e16d8d83dcf1a6e031598dbc517aff18e6c7ccd55f7894102bab55fcdb9|1611849907|/gnu/store/rr532q5fmwj1gafdgk6nhxg9khnbsw3z-repeat-masker-4.1.1.drv|-2097113072
--8<---------------cut here---------------end--------------->8---

The actual nar size in this case is just above 2³¹ so most likely we’re
seeing a signed integer wrapping error.

I believe this is a very recent regression (the registration data for
the item above is Jan. 28th); we have older store items with a correct
‘narSize’, such as
<https://ci.guix.gnu.org/nm6w84c9zj3yiylal3dk1sqzxq11sjzw.narinfo>.

Thoughts?

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-01-31 15:20 ` Ludovic Courtès
@ 2021-02-01  9:15   ` Ludovic Courtès
  2021-02-19 15:11   ` Ludovic Courtès
  1 sibling, 0 replies; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-01  9:15 UTC (permalink / raw)
  To: Christopher Baines; +Cc: Mathieu Othacehe, 46212

Ludovic Courtès <ludo@gnu.org> skribis:

> The key point here is that ‘narSize’ in the database is negative:

With commit 13a7d2a538b00aa0a8cf9b999f1a4ff3e5959af9, ‘register-items’ &
co. will now detect invalid nar size values early on.

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-01-31 14:47 bug#46212: ci.guix.gnu.org narinfos with excessive NarSize Christopher Baines
  2021-01-31 15:20 ` Ludovic Courtès
@ 2021-02-01 19:57 ` Christopher Baines
  2021-02-02 21:48   ` Ludovic Courtès
  1 sibling, 1 reply; 9+ messages in thread
From: Christopher Baines @ 2021-02-01 19:57 UTC (permalink / raw)
  To: 46212

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> I noticed through the Guix Data Service that some narinfo files from
> ci.guix.gnu.org have an excessive NarSize.
>
> These are the three that I've found, but there could be more.
>
>
> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
> NarSize: 18446744072099351584
>
> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
> NarSize: 18446744071612438544
>
> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
> NarSize: 18446744071612438544

Guix gives the following error when it encounters one of these bad
narinfos:

  error: integer expected from stream

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-02-01 19:57 ` Christopher Baines
@ 2021-02-02 21:48   ` Ludovic Courtès
  0 siblings, 0 replies; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-02 21:48 UTC (permalink / raw)
  To: Christopher Baines; +Cc: 46212

Christopher Baines <mail@cbaines.net> skribis:

> Christopher Baines <mail@cbaines.net> writes:
>
>> I noticed through the Guix Data Service that some narinfo files from
>> ci.guix.gnu.org have an excessive NarSize.
>>
>> These are the three that I've found, but there could be more.
>>
>>
>> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
>> NarSize: 18446744072099351584
>>
>> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>>
>> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>
> Guix gives the following error when it encounters one of these bad
> narinfos:
>
>   error: integer expected from stream

I guess ‘guix substitute --query’ reads the narinfo, passes the negative
integer as is to the daemon (for ‘query-path-info’ RPCs), which
rightfully complains.

It would be nice for ‘guix publish’ to not emit broken narinfos, and
perhaps we can also add extra checks in the (guix narinfo) reader.

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-01-31 15:20 ` Ludovic Courtès
  2021-02-01  9:15   ` Ludovic Courtès
@ 2021-02-19 15:11   ` Ludovic Courtès
  2021-02-22  8:59     ` Mathieu Othacehe
  1 sibling, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-19 15:11 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: 46212

Hi Mathieu,

Did you eventually find out where the negative size comes from?

  https://issues.guix.gnu.org/46212

What should we do in your opinion with database entries that have a
negative size?

Thanks,
Ludo’.

Ludovic Courtès <ludo@gnu.org> skribis:

> Christopher Baines <mail@cbaines.net> skribis:
>
>> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
>> NarSize: 18446744072099351584
>>
>> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>>
>> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>
> The key point here is that ‘narSize’ in the database is negative:
>
> sqlite> select * from validpaths where path = '/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1';
> 43262123|/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1|sha256:33328e16d8d83dcf1a6e031598dbc517aff18e6c7ccd55f7894102bab55fcdb9|1611849907|/gnu/store/rr532q5fmwj1gafdgk6nhxg9khnbsw3z-repeat-masker-4.1.1.drv|-2097113072
>
> The actual nar size in this case is just above 2³¹ so most likely we’re
> seeing a signed integer wrapping error.
>
> I believe this is a very recent regression (the registration data for
> the item above is Jan. 28th); we have older store items with a correct
> ‘narSize’, such as
> <https://ci.guix.gnu.org/nm6w84c9zj3yiylal3dk1sqzxq11sjzw.narinfo>.
>
> Thoughts?
>
> Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-02-19 15:11   ` Ludovic Courtès
@ 2021-02-22  8:59     ` Mathieu Othacehe
  2021-02-22 13:03       ` Ludovic Courtès
  0 siblings, 1 reply; 9+ messages in thread
From: Mathieu Othacehe @ 2021-02-22  8:59 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 46212


Hey Ludo,

> Did you eventually find out where the negative size comes from?
>
>   https://issues.guix.gnu.org/46212
>
> What should we do in your opinion with database entries that have a
> negative size?

I didn't look closely to this problem yet. However, I fixed an issue
with locales in the remote building mechanism that caused publish server
crashes:
https://lists.gnu.org/archive/html/bug-guix/2021-02/msg00231.html.

That's maybe somehow related.

Thanks,

Mathieu




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-02-22  8:59     ` Mathieu Othacehe
@ 2021-02-22 13:03       ` Ludovic Courtès
  2021-02-24 10:26         ` Mathieu Othacehe
  0 siblings, 1 reply; 9+ messages in thread
From: Ludovic Courtès @ 2021-02-22 13:03 UTC (permalink / raw)
  To: Mathieu Othacehe; +Cc: 46212

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> Did you eventually find out where the negative size comes from?
>>
>>   https://issues.guix.gnu.org/46212
>>
>> What should we do in your opinion with database entries that have a
>> negative size?
>
> I didn't look closely to this problem yet. However, I fixed an issue
> with locales in the remote building mechanism that caused publish server
> crashes:
> https://lists.gnu.org/archive/html/bug-guix/2021-02/msg00231.html.

Hmm I don’t think so.

The bug here is likely due to 32-bit signed integer wrapping.  That can
only happen in C code, so to me possible culprits would be
guile-simple-zmq or the layer above it (if there’s a binary protocol
involved) or the postgresql interface.  Only a vague intuition, though.

Ludo’.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
  2021-02-22 13:03       ` Ludovic Courtès
@ 2021-02-24 10:26         ` Mathieu Othacehe
  0 siblings, 0 replies; 9+ messages in thread
From: Mathieu Othacehe @ 2021-02-24 10:26 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: 46212


Hey,

> The bug here is likely due to 32-bit signed integer wrapping.  That can
> only happen in C code, so to me possible culprits would be
> guile-simple-zmq or the layer above it (if there’s a binary protocol
> involved) or the postgresql interface.  Only a vague intuition, though.

Hmm, looks like you're right! There's a memory corruption in the
remote-server process that's really hard to reproduce. I suspect the ZMQ
library or its Guile bindings.

I'm trying to valgrind the process to identify the issue, without much
success for now.

Thanks,

Mathieu




^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-02-24 10:27 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-31 14:47 bug#46212: ci.guix.gnu.org narinfos with excessive NarSize Christopher Baines
2021-01-31 15:20 ` Ludovic Courtès
2021-02-01  9:15   ` Ludovic Courtès
2021-02-19 15:11   ` Ludovic Courtès
2021-02-22  8:59     ` Mathieu Othacehe
2021-02-22 13:03       ` Ludovic Courtès
2021-02-24 10:26         ` Mathieu Othacehe
2021-02-01 19:57 ` Christopher Baines
2021-02-02 21:48   ` Ludovic Courtès

unofficial mirror of bug-guix@gnu.org 

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/guix-bugs/0 guix-bugs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 guix-bugs guix-bugs/ https://yhetil.org/guix-bugs \
		bug-guix@gnu.org
	public-inbox-index guix-bugs

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.gnu.guix.bugs
	nntp://news.gmane.io/gmane.comp.gnu.guix.bugs


AGPL code for this site: git clone http://ou63pmih66umazou.onion/public-inbox.git