Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

[Nils Gillmann]

Hi folks,

we did some changes in GNUnet where my part of the job reminded me again
of the path length Linux has as default.

I know there's at least (only?) 1 file in the Linux kernel which defines
the path length. Where else would we need to apply changes as a distro?

I think if there are really no serious implications of extending the length,
we should make the path longer. Not just for GNUnet. To avoid future annoyance.

What do you think? And if you are against it, what should I look into
that supports keeping the arbitrary short number we have now?


----- Forwarded message from Christian Grothoff -----

> On 05/06/2018 08:34 PM, Nils Gillmann wrote:
> > Christian Grothoff transcribed 2.9K bytes:
> >> On 05/05/2018 09:58 PM, Nils Gillmann wrote:
[snip]
> > I've changed the config files, now the problem for us in Nix and Guix
> > will be that the silly maximum unix path length will be hit quiet often...
> > at least for the Guix build chroot when the chroot is in /tmp/guix-build-gnunet-git1.drv-0/
> > 
> > The $GNUNET_TMP then gives you diretories like /tmp/guix-build-gnunet-git1.drv-0/gnunet/WAHTEVER-HERE/GOES/NOTHING
> > where previously they just landed in /tmp iirc.
> > 
> > We should make it possible to set the value of GNUNET_TMP before make check,
> > for example via the equivalent of export GNUNET_TMP="/tmp"; make check.
> > 
> > It's quiet unfortunate with the length..
> 
> Have you considered re-defining the maximum path length? Given that you
> are a distro, you should be able to recompile everything necessary
> (kernel, libc) with simply a larger value, right?

----- End forwarded message -----

Re: Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

On Sun, May 06, 2018 at 06:59:23PM +0000, Nils Gillmann wrote:
> we did some changes in GNUnet where my part of the job reminded me again
> of the path length Linux has as default.
> 
> I know there's at least (only?) 1 file in the Linux kernel which defines
> the path length. Where else would we need to apply changes as a distro?
> 
> I think if there are really no serious implications of extending the length,
> we should make the path longer. Not just for GNUnet. To avoid future annoyance.
> 
> What do you think? And if you are against it, what should I look into
> that supports keeping the arbitrary short number we have now?

If we are interested in discussing this change, we need some more
information for the discussion to be fruitful.

Specifically, we need to know how the path limit is set and used in the
Linux source code [0]. From there, we will be more able to evaluate the
feasibility and impact of the change.

This will also help us to understand why the limit exists. Perhaps the
limitation is required by Linux across the full set of platforms (libc
and filesystems) it aims to support, but not for the platforms supported
by Guix.

Finally, it will be helpful for the rest of guix-devel to understand why
we'd want to make this change. So far, the path length limit 
of 4096 characters seems long enough for us...

[0] My understanding is that the PATH_MAX macro from limits.h sets a
4096 "character" limit.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

[Nils Gillmann]

Leo Famulari transcribed 2.5K bytes:
> On Sun, May 06, 2018 at 06:59:23PM +0000, Nils Gillmann wrote:
> > we did some changes in GNUnet where my part of the job reminded me again
> > of the path length Linux has as default.
> > 
> > I know there's at least (only?) 1 file in the Linux kernel which defines
> > the path length. Where else would we need to apply changes as a distro?
> > 
> > I think if there are really no serious implications of extending the length,
> > we should make the path longer. Not just for GNUnet. To avoid future annoyance.
> > 
> > What do you think? And if you are against it, what should I look into
> > that supports keeping the arbitrary short number we have now?
> 
> If we are interested in discussing this change, we need some more
> information for the discussion to be fruitful.
> 
> Specifically, we need to know how the path limit is set and used in the
> Linux source code [0]. From there, we will be more able to evaluate the
> feasibility and impact of the change.
> 
> This will also help us to understand why the limit exists. Perhaps the
> limitation is required by Linux across the full set of platforms (libc
> and filesystems) it aims to support, but not for the platforms supported
> by Guix.
> 
> Finally, it will be helpful for the rest of guix-devel to understand why
> we'd want to make this change. So far, the path length limit 
> of 4096 characters seems long enough for us...

Hi Leo,

I have inserted a build log of a failing test from
GNUnet commit f13af7e7281064380def70d0b4392b4351211655

This is the limitation I experience, when the tests are not running
in /tmp but in $TMPDIR. $GNUNET_TMP's first choice is to select $TMPDIR
when it exists.


I'll look more into the rest of your email and the problem itself next
week. I can imagine that it is platform specific but guessing won't
help.

> [0] My understanding is that the PATH_MAX macro from limits.h sets a
> 4096 "character" limit.


==================================================
   gnunet 0.11.0: src/statistics/test-suite.log
==================================================

# TOTAL: 5
# PASS:  4
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test_gnunet_statistics.py
===============================

Preparing: Starting service...
TEST: Bad argument checking...PASS
TEST: Set value...PASS
TEST: Set another value...PASS
TEST: Viewing all stats...FAIL: unexpected output:
      osystem                                              other:               43
    subsystem                                               test:               42
!test-statistics-api                                             test-3:                3

FAIL test_gnunet_statistics.py (exit status: 1)






Human addition:
make[4]: *** [Makefile:1074: test-suite.log] Error 1
make[4]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
make[3]: *** [Makefile:1182: check-TESTS] Error 2
make[3]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
make[2]: *** [Makefile:1281: check-am] Error 2
make[2]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
make[1]: *** [Makefile:550: check-recursive] Error 1
make[1]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src'
make: *** [Makefile:591: check-recursive] Error 1
phase `set-path-for-check' failed after 29.2 seconds
note: keeping build directory `/tmp/guix-build-gnunet-git1.drv-0'
builder for `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed with exit code 1
@ build-failed /gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv - 1 builder for `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed with exit code 1
guix build: error: build failed: build of `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed




With this we get filenames like:
/tmp/guix-build-gnunet-git1.drv-0/gnunet/test-gnunet-statistics/.local/share/gnunet/statistics.dat
which is an average length. When you just do it in /tmp, you won't experience any difficulties.

I need to spin up a Debian testmachine, but I guess that other chroots won't hit the limit early
and often.

running guix build -f guix-env.scm in the source of gnunet, on commit 964f78df05ce848b96908ad0b931dfec1e182348
will enable you to reproduce my examle case.

Re: Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2051 bytes --]

On Sun, May 06, 2018 at 07:27:46PM +0000, Nils Gillmann wrote:
> I have inserted a build log of a failing test from
> GNUnet commit f13af7e7281064380def70d0b4392b4351211655

[...]

> TEST: Viewing all stats...FAIL: unexpected output:
>       osystem                                              other:               43
>     subsystem                                               test:               42
> !test-statistics-api                                             test-3:                3
> 
> FAIL test_gnunet_statistics.py (exit status: 1)

[...]

> Human addition:
> make[4]: *** [Makefile:1074: test-suite.log] Error 1
> make[4]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
> make[3]: *** [Makefile:1182: check-TESTS] Error 2
> make[3]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
> make[2]: *** [Makefile:1281: check-am] Error 2
> make[2]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
> make[1]: *** [Makefile:550: check-recursive] Error 1
> make[1]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src'
> make: *** [Makefile:591: check-recursive] Error 1
> phase `set-path-for-check' failed after 29.2 seconds
> note: keeping build directory `/tmp/guix-build-gnunet-git1.drv-0'
> builder for `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed with exit code 1
> @ build-failed /gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv - 1 builder for `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed with exit code 1
> guix build: error: build failed: build of `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed

[...]

> With this we get filenames like:
> /tmp/guix-build-gnunet-git1.drv-0/gnunet/test-gnunet-statistics/.local/share/gnunet/statistics.dat
> which is an average length. When you just do it in /tmp, you won't experience any difficulties.

That path is only 98 characters long. What limit do you think it is
exceeding?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

[Nils Gillmann]

Leo Famulari transcribed 3.0K bytes:
> On Sun, May 06, 2018 at 07:27:46PM +0000, Nils Gillmann wrote:
> > I have inserted a build log of a failing test from
> > GNUnet commit f13af7e7281064380def70d0b4392b4351211655
> 
> [...]
> 
> > TEST: Viewing all stats...FAIL: unexpected output:
> >       osystem                                              other:               43
> >     subsystem                                               test:               42
> > !test-statistics-api                                             test-3:                3
> > 
> > FAIL test_gnunet_statistics.py (exit status: 1)
> 
> [...]
> 
> > Human addition:
> > make[4]: *** [Makefile:1074: test-suite.log] Error 1
> > make[4]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
> > make[3]: *** [Makefile:1182: check-TESTS] Error 2
> > make[3]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
> > make[2]: *** [Makefile:1281: check-am] Error 2
> > make[2]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src/statistics'
> > make[1]: *** [Makefile:550: check-recursive] Error 1
> > make[1]: Leaving directory '/tmp/guix-build-gnunet-git1.drv-0/source/src'
> > make: *** [Makefile:591: check-recursive] Error 1
> > phase `set-path-for-check' failed after 29.2 seconds
> > note: keeping build directory `/tmp/guix-build-gnunet-git1.drv-0'
> > builder for `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed with exit code 1
> > @ build-failed /gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv - 1 builder for `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed with exit code 1
> > guix build: error: build failed: build of `/gnu/store/daqbvvd3zq2w1aw6w4yz5qm7s5k247ap-gnunet-git1.drv' failed
> 
> [...]
> 
> > With this we get filenames like:
> > /tmp/guix-build-gnunet-git1.drv-0/gnunet/test-gnunet-statistics/.local/share/gnunet/statistics.dat
> > which is an average length. When you just do it in /tmp, you won't experience any difficulties.
> 
> That path is only 98 characters long. What limit do you think it is
> exceeding?

Right, I provided too little context of the log.

Okay, the error is elsewhere in this case, not the path length..

But on topic, I've run into the UNIX_PATH_LENGTH error a couple of times
before when the package name was longer, but not unusual long (let's say
gnunet-variant-flavor-bla..  A little room to wiggle for applications that
do not consider folder naming with the checksums, would be good.

Obviously so far these are corner cases, but even if we decide that we
would not need it today, a discussion if this is something that can be
done, etc etc would be good.

From memory, I think I've only ever encountered this in C based applications
test suites. Everything else just worked.

Re: Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

[Pjotr Prins]

On Sun, May 06, 2018 at 08:17:20PM +0000, Nils Gillmann wrote:
> > That path is only 98 characters long. What limit do you think it is
> > exceeding?

It may be a shell restriction rather than a kernel restriction. I run
into the same thing with my binary path rewriting. My solution is to
truncate the hash. There is a limit to that...

Pj.

Re: Redefining the maximum path length [Fwd: Re: [GNUnet-developers] using $TMPDIR instead of /tmp]

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Sun, May 06, 2018 at 08:17:20PM +0000, Nils Gillmann wrote:
>> > That path is only 98 characters long. What limit do you think it is
>> > exceeding?
>
> It may be a shell restriction rather than a kernel restriction.

It’s a kernel restriction.

I think the main reason for these arbitrary limits is that in C it’s
easier to deal with fixed-size arrays than with dynamically-allocated
arrays.  :-)

(A valid argument to limiting the length of file names is that it could
make it easy to DoS a system.  But then, that limit could very well be
configurable at run time, pretty much like rlimits, if it were the only
reason…)

Ludo’.