bug#21410: Environment containers

bug#21410: Environment containers

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]

Hi everyone,

On 27/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
> Alex Vong <alexvong1995@gmail.com> skribis:
>
>> On 26/10/2015, Thompson, David <dthompson2@worcester.edu> wrote:
>>> On Mon, Oct 26, 2015 at 10:37 AM, Taylan Ulrich Bayırlı/Kammer
>>> <taylanbayirli@gmail.com> wrote:
>>>
>>>> Also, for Debian 8 users and maybe others, this might help:
>>>>
>>>>     sudo sysctl -w kernel.unprivileged_userns_clone=1
>>>
>>> Yes, user namespaces are a must-have for this to work.  I will prepare
>>> patches that mention this in the manual and add a test to 'guix
>>> environment' that can detect if user namespaces are unavailable and
>>> display a more helpful error message.
>>>
>> I think the patch could be considered a fix for
>> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21410> as well. How do
>> you think?
>
> Do you still experience the test failures mentioned in that report?  If
> not, could you email 21410@debbugs.gnu.org, specifying which commit
> works for you?
>
Yes, there are 4 tests still failing with the latest master branch
without unprivileged container.
But there is a new problem, tests/guix-environment-container.sh fails
even when running the tests as root. The test log is in the
attachment. I am running Debian 8, could anyone verify this? I would
also like to try it on Debian unstable, but currently my PC cannot
boot, I am using my old laptop.

> I suspect 0e3cc31 helped.
>
> Thanks,
> Ludo’.
>

Cheers,
Alex

[-- Attachment #2: guix-environment-container.log --]
[-- Type: text/x-log, Size: 2326 bytes --]

+ set -e
+ guix environment --version
warning: daemon is running as root, so using `--build-users-group' is highly recommended
guix environment (GNU Guix) 0.9.0
Copyright (C) 2015 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ tmpdir=t-guix-environment-9613
+ trap 'rm -r "$tmpdir"' EXIT
+ mkdir t-guix-environment-9613
+ guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '(exit 42)'
accepted connection from pid 9621, user root (trusted)
+ test 42 = 42
+ mount_test_code='
(use-modules (ice-9 rdelim)
             (ice-9 match)
             (srfi srfi-1))

(define mappings
  (filter-map (lambda (line)
                (match (string-split line #\space)
                  ;; Empty line.
                  (("") #f)
                  ;; Ignore these types of file systems.
                  ((_ _ (or "tmpfs" "proc" "sysfs" "devtmpfs"
                            "devpts" "cgroup" "mqueue") _ _ _)
                   #f)
                  ((_ mount _ _ _ _)
                   mount)))
              (string-split (call-with-input-file "/proc/mounts" read-string)
                            #\newline)))

(for-each (lambda (mount)
            (display mount)
            (newline))
          mappings)'
+ guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '
(use-modules (ice-9 rdelim)
             (ice-9 match)
             (srfi srfi-1))

(define mappings
  (filter-map (lambda (line)
                (match (string-split line #\space)
                  ;; Empty line.
                  (("") #f)
                  ;; Ignore these types of file systems.
                  ((_ _ (or "tmpfs" "proc" "sysfs" "devtmpfs"
                            "devpts" "cgroup" "mqueue") _ _ _)
                   #f)
                  ((_ mount _ _ _ _)
                   mount)))
              (string-split (call-with-input-file "/proc/mounts" read-string)
                            #\newline)))

(for-each (lambda (mount)
            (display mount)
            (newline))
          mappings)'
accepted connection from pid 9627, user root (trusted)
++ wc -l
+ test 4 -eq 3
+ rm -r t-guix-environment-9613

bug#21410: Environment containers

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Alex Vong <alexvong1995@gmail.com> skribis:

> On 27/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> Do you still experience the test failures mentioned in that report?  If
>> not, could you email 21410@debbugs.gnu.org, specifying which commit
>> works for you?
>>
> Yes, there are 4 tests still failing with the latest master branch
> without unprivileged container.

Which tests?  Does tests/container.scm pass?

> But there is a new problem, tests/guix-environment-container.sh fails
> even when running the tests as root. The test log is in the
> attachment.

[...]

> (for-each (lambda (mount)
>             (display mount)
>             (newline))
>           mappings)'
> accepted connection from pid 9627, user root (trusted)
> ++ wc -l
> + test 4 -eq 3

Could you apply the patch below, run:

  make check TESTS=tests/guix-environment-container.sh

and send ‘guix-environment-container.log’?

Also, what does:

  make check TESTS=tests/container.scm

report?

TIA!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 356 bytes --]

--- a/tests/guix-environment-container.sh
+++ b/tests/guix-environment-container.sh
@@ -66,6 +66,7 @@ mount_test_code="
 guix environment --container --ad-hoc --bootstrap guile-bootstrap \
      -- guile -c "$mount_test_code" > $tmpdir/mounts
 
+cat $tmpdir/mounts
 test `wc -l < $tmpdir/mounts` -eq 3
 
 grep -e "$PWD$" $tmpdir/mounts # current directory

bug#21410: Environment containers

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]

On 28/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
> Alex Vong <alexvong1995@gmail.com> skribis:
>
>> On 27/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>
> [...]
>
>>> Do you still experience the test failures mentioned in that report?  If
>>> not, could you email 21410@debbugs.gnu.org, specifying which commit
>>> works for you?
>>>
>> Yes, there are 4 tests still failing with the latest master branch
>> without unprivileged container.
>
> Which tests?  Does tests/container.scm pass?
>
It doesn't pass if I run as unprivileged user. It passes if I run as
root. I will be mailing the test logs on another mail.

>> But there is a new problem, tests/guix-environment-container.sh fails
>> even when running the tests as root. The test log is in the
>> attachment.
>
> [...]
>
>> (for-each (lambda (mount)
>>             (display mount)
>>             (newline))
>>           mappings)'
>> accepted connection from pid 9627, user root (trusted)
>> ++ wc -l
>> + test 4 -eq 3
>
> Could you apply the patch below, run:
>
>   make check TESTS=tests/guix-environment-container.sh
>
> and send ‘guix-environment-container.log’?
>
Sure! It is in the attachment. (Note that I run the test as root to
bypass the unprivileged container issue.)

> Also, what does:
>
>   make check TESTS=tests/container.scm
>
> report?
>
I will send the test log in another mail.

> TIA!
>
> Ludo’.
>
>

Cheers,
Alex

[-- Attachment #2: guix-environment-container.log --]
[-- Type: text/x-log, Size: 2564 bytes --]

+ set -e
+ guix environment --version
warning: daemon is running as root, so using `--build-users-group' is highly recommended
guix environment (GNU Guix) 0.9.0
Copyright (C) 2015 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ tmpdir=t-guix-environment-12182
+ trap 'rm -r "$tmpdir"' EXIT
+ mkdir t-guix-environment-12182
+ guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '(exit 42)'
accepted connection from pid 12190, user root (trusted)
+ test 42 = 42
+ mount_test_code='
(use-modules (ice-9 rdelim)
             (ice-9 match)
             (srfi srfi-1))

(define mappings
  (filter-map (lambda (line)
                (match (string-split line #\space)
                  ;; Empty line.
                  (("") #f)
                  ;; Ignore these types of file systems.
                  ((_ _ (or "tmpfs" "proc" "sysfs" "devtmpfs"
                            "devpts" "cgroup" "mqueue") _ _ _)
                   #f)
                  ((_ mount _ _ _ _)
                   mount)))
              (string-split (call-with-input-file "/proc/mounts" read-string)
                            #\newline)))

(for-each (lambda (mount)
            (display mount)
            (newline))
          mappings)'
+ guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '
(use-modules (ice-9 rdelim)
             (ice-9 match)
             (srfi srfi-1))

(define mappings
  (filter-map (lambda (line)
                (match (string-split line #\space)
                  ;; Empty line.
                  (("") #f)
                  ;; Ignore these types of file systems.
                  ((_ _ (or "tmpfs" "proc" "sysfs" "devtmpfs"
                            "devpts" "cgroup" "mqueue") _ _ _)
                   #f)
                  ((_ mount _ _ _ _)
                   mount)))
              (string-split (call-with-input-file "/proc/mounts" read-string)
                            #\newline)))

(for-each (lambda (mount)
            (display mount)
            (newline))
          mappings)'
accepted connection from pid 12196, user root (trusted)
+ cat t-guix-environment-12182/mounts
/
/home/alexvong1995/guix
/home/alexvong1995/guix/test-tmp/store/bdg31cb28ki1b0xi4rqrc721hadn3lfw-bash
/home/alexvong1995/guix/test-tmp/store/92vjphkx545ndw82vn4fp9m3xxkv3mzd-guile-bootstrap-2.0
++ wc -l
+ test 4 -eq 3
+ rm -r t-guix-environment-12182

bug#21410: Environment containers

[Thompson, David]

On Wed, Oct 28, 2015 at 11:14 AM, Alex Vong <alexvong1995@gmail.com> wrote:
> On 28/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>> Alex Vong <alexvong1995@gmail.com> skribis:
>>
>>> On 27/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>>
>> [...]
>>
>>>> Do you still experience the test failures mentioned in that report?  If
>>>> not, could you email 21410@debbugs.gnu.org, specifying which commit
>>>> works for you?
>>>>
>>> Yes, there are 4 tests still failing with the latest master branch
>>> without unprivileged container.
>>
>> Which tests?  Does tests/container.scm pass?
>>
> It doesn't pass if I run as unprivileged user. It passes if I run as
> root. I will be mailing the test logs on another mail.

This is because Debian doesn't let unprivileged users create user
namespaces without explicitly overriding some configuration.

- Dave

bug#21410: Environment containers

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Wed, Oct 28, 2015 at 11:14 AM, Alex Vong <alexvong1995@gmail.com> wrote:
>> On 28/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>>> Alex Vong <alexvong1995@gmail.com> skribis:
>>>
>>>> On 27/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>>>
>>> [...]
>>>
>>>>> Do you still experience the test failures mentioned in that report?  If
>>>>> not, could you email 21410@debbugs.gnu.org, specifying which commit
>>>>> works for you?
>>>>>
>>>> Yes, there are 4 tests still failing with the latest master branch
>>>> without unprivileged container.
>>>
>>> Which tests?  Does tests/container.scm pass?
>>>
>> It doesn't pass if I run as unprivileged user. It passes if I run as
>> root. I will be mailing the test logs on another mail.
>
> This is because Debian doesn't let unprivileged users create user
> namespaces without explicitly overriding some configuration.

How could we determine whether this restriction is in place?  That would
allow us to skip the test on these systems.

Ludo’.

bug#21410: Environment containers

[Thompson, David]

On Wed, Oct 28, 2015 at 11:56 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> On Wed, Oct 28, 2015 at 11:14 AM, Alex Vong <alexvong1995@gmail.com> wrote:
>>> On 28/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>>>> Alex Vong <alexvong1995@gmail.com> skribis:
>>>>
>>>>> On 27/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>>>>
>>>> [...]
>>>>
>>>>>> Do you still experience the test failures mentioned in that report?  If
>>>>>> not, could you email 21410@debbugs.gnu.org, specifying which commit
>>>>>> works for you?
>>>>>>
>>>>> Yes, there are 4 tests still failing with the latest master branch
>>>>> without unprivileged container.
>>>>
>>>> Which tests?  Does tests/container.scm pass?
>>>>
>>> It doesn't pass if I run as unprivileged user. It passes if I run as
>>> root. I will be mailing the test logs on another mail.
>>
>> This is because Debian doesn't let unprivileged users create user
>> namespaces without explicitly overriding some configuration.
>
> How could we determine whether this restriction is in place?  That would
> allow us to skip the test on these systems.

I think it is /proc/sys/kernel/unprivileged_userns_clone, but I don't
know what the contents are exactly.  0 when off, 1 when on?  Can
someone on Debian confirm?

If we can get the test suite passing, I'd like to extract these user
namespace presence tests to a procedure that 'guix environment' can
use to give the user an informative error message in these cases.

- Dave

bug#21410: Environment containers

[Alex Vong]

On 29/10/2015, Thompson, David <dthompson2@worcester.edu> wrote:
> On Wed, Oct 28, 2015 at 11:56 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>> "Thompson, David" <dthompson2@worcester.edu> skribis:
>>
>>> On Wed, Oct 28, 2015 at 11:14 AM, Alex Vong <alexvong1995@gmail.com>
>>> wrote:
>>>> On 28/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>>>>> Alex Vong <alexvong1995@gmail.com> skribis:
>>>>>
>>>>>> On 27/10/2015, Ludovic Courtès <ludo@gnu.org> wrote:
>>>>>
>>>>> [...]
>>>>>
>>>>>>> Do you still experience the test failures mentioned in that report?
>>>>>>> If
>>>>>>> not, could you email 21410@debbugs.gnu.org, specifying which commit
>>>>>>> works for you?
>>>>>>>
>>>>>> Yes, there are 4 tests still failing with the latest master branch
>>>>>> without unprivileged container.
>>>>>
>>>>> Which tests?  Does tests/container.scm pass?
>>>>>
>>>> It doesn't pass if I run as unprivileged user. It passes if I run as
>>>> root. I will be mailing the test logs on another mail.
>>>
>>> This is because Debian doesn't let unprivileged users create user
>>> namespaces without explicitly overriding some configuration.
>>
>> How could we determine whether this restriction is in place?  That would
>> allow us to skip the test on these systems.
>
> I think it is /proc/sys/kernel/unprivileged_userns_clone, but I don't
> know what the contents are exactly.  0 when off, 1 when on?  Can
> someone on Debian confirm?
>
Yes, I think that's the case.
Before I run `$ sysctl -w kernel.unprivileged_userns_clone=1',
`$ cat /proc/sys/kernel/unprivileged_userns_clone' returns 0.
After I run `$ sysctl -w kernel.unprivileged_userns_clone=1',
`$ cat /proc/sys/kernel/unprivileged_userns_clone' returns 1.

> If we can get the test suite passing, I'd like to extract these user
> namespace presence tests to a procedure that 'guix environment' can
> use to give the user an informative error message in these cases.
>
> - Dave
>

bug#21410: Environment containers

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> I think it is /proc/sys/kernel/unprivileged_userns_clone, but I don't
> know what the contents are exactly.  0 when off, 1 when on?  Can
> someone on Debian confirm?
>
> If we can get the test suite passing, I'd like to extract these user
> namespace presence tests to a procedure that 'guix environment' can
> use to give the user an informative error message in these cases.

That would be perfect.

The test machinery would still need to test them explicitly, though, to
mark tests as skipped instead of failed.

Ludo’.

bug#21410: Environment containers

[Thompson, David]

On Thu, Oct 29, 2015 at 3:24 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> I think it is /proc/sys/kernel/unprivileged_userns_clone, but I don't
>> know what the contents are exactly.  0 when off, 1 when on?  Can
>> someone on Debian confirm?
>>
>> If we can get the test suite passing, I'd like to extract these user
>> namespace presence tests to a procedure that 'guix environment' can
>> use to give the user an informative error message in these cases.
>
> That would be perfect.
>
> The test machinery would still need to test them explicitly, though, to
> mark tests as skipped instead of failed.

Yes, of course.  I have no intention of removing those checks from the tests.

- Dave

bug#21410: Environment containers

[Ludovic Courtès]

What’s the status of this now?  I would think b7d48312 addressed at
least part of it?

Alex: Are you still getting test failures with current ‘master’?

TIA,
Ludo’.

bug#21410: Environment containers

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 376 bytes --]

Hi,

On 20/11/2015, Ludovic Courtès <ludo@gnu.org> wrote:
> What’s the status of this now?  I would think b7d48312 addressed at
> least part of it?
>
> Alex: Are you still getting test failures with current ‘master’?
>
> TIA,
> Ludo’.
>

Yes, I still get a test failure with the latest master branch, the
test log is in the attachment.

Cheers,
Alex

[-- Attachment #2: test-suite.log --]
[-- Type: text/x-log, Size: 1342 bytes --]

======================================
   GNU Guix 0.9.0: ./test-suite.log
======================================

# TOTAL: 54
# PASS:  52
# SKIP:  1
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

SKIP: tests/containers
======================

;;; note: source file ./srfi/srfi-64.scm
;;;       newer than compiled /gnu/store/5i87jzm90nw8j692y7z1j2qfx16h6ni3-guile-2.0.11/lib/guile/2.0/ccache/srfi/srfi-64.go
SKIP tests/containers.scm (exit status: 77)

FAIL: tests/guix-environment-container
======================================

+ set -e
+ guix environment --version
guix environment (GNU Guix) 0.9.0
Copyright (C) 2015 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ tmpdir=t-guix-environment-29930
+ trap 'rm -r "$tmpdir"' EXIT
+ mkdir t-guix-environment-29930
+ guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '(exit 42)'
guix environment: error: cannot create container: unprivileged user cannot create user namespaces
guix environment: error: please set /proc/sys/kernel/unprivileged_userns_clone to "1"
+ test 1 = 42
+ rm -r t-guix-environment-29930
FAIL tests/guix-environment-container.sh (exit status: 1)

bug#21410: Environment containers

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

Alex Vong <alexvong1995@gmail.com> skribis:

> FAIL: tests/guix-environment-container
> ======================================
>
> + set -e
> + guix environment --version
> guix environment (GNU Guix) 0.9.0
> Copyright (C) 2015 the Guix authors
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> + tmpdir=t-guix-environment-29930
> + trap 'rm -r "$tmpdir"' EXIT
> + mkdir t-guix-environment-29930
> + guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '(exit 42)'
> guix environment: error: cannot create container: unprivileged user cannot create user namespaces
> guix environment: error: please set /proc/sys/kernel/unprivileged_userns_clone to "1"

Oh I see, that part fell through the cracks.

Could you confirm that the test is skipped with the attached patch?

TIA!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 365 bytes --]

--- a/tests/guix-environment-container.sh
+++ b/tests/guix-environment-container.sh
@@ -24,6 +24,12 @@ set -e
 
 guix environment --version
 
+if ! guile -c '((@@ (guix scripts environment) assert-container-features))'
+then
+    # User containers are not supported; skip this test.
+    exit 77
+fi
+
 tmpdir="t-guix-environment-$$"
 trap 'rm -r "$tmpdir"' EXIT
 

bug#21410: Environment containers

[Mathieu Lirzin]

ludo@gnu.org (Ludovic Courtès) writes:

>
>> FAIL: tests/guix-environment-container
>> ======================================
>>
>> + set -e
>> + guix environment --version
>> guix environment (GNU Guix) 0.9.0
>> Copyright (C) 2015 the Guix authors
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.
>> + tmpdir=t-guix-environment-29930
>> + trap 'rm -r "$tmpdir"' EXIT
>> + mkdir t-guix-environment-29930
>> + guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '(exit 42)'
>> guix environment: error: cannot create container: unprivileged user cannot create user namespaces
>> guix environment: error: please set /proc/sys/kernel/unprivileged_userns_clone to "1"
>
> Oh I see, that part fell through the cracks.
>
> Could you confirm that the test is skipped with the attached patch?

this test was failing on my system too, and it is skipped successfully
with this patch.

--
Mathieu Lirzin

bug#21410: Environment containers

[Ludovic Courtès]

Mathieu Lirzin <mthl@gnu.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>>
>>> FAIL: tests/guix-environment-container
>>> ======================================
>>>
>>> + set -e
>>> + guix environment --version
>>> guix environment (GNU Guix) 0.9.0
>>> Copyright (C) 2015 the Guix authors
>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.
>>> + tmpdir=t-guix-environment-29930
>>> + trap 'rm -r "$tmpdir"' EXIT
>>> + mkdir t-guix-environment-29930
>>> + guix environment --container --ad-hoc --bootstrap guile-bootstrap -- guile -c '(exit 42)'
>>> guix environment: error: cannot create container: unprivileged user cannot create user namespaces
>>> guix environment: error: please set /proc/sys/kernel/unprivileged_userns_clone to "1"
>>
>> Oh I see, that part fell through the cracks.
>>
>> Could you confirm that the test is skipped with the attached patch?
>
> this test was failing on my system too, and it is skipped successfully
> with this patch.

Thanks, pushed as 6493fd0.

Ludo’.