Re: 01/01: hydra: Work around import problem.

Re: 01/01: hydra: Work around import problem.

[Ludovic Courtès]

Hi Danny,

dannym@scratchpost.org (Danny Milosavljevic) skribis:

> commit ebd6a6445fa7d3db64795840a607815abbd56880
> Author: Danny Milosavljevic <dannym@scratchpost.org>
> Date:   Mon Jan 29 13:23:07 2018 +0100
>
>     hydra: Work around import problem.
>     
>     * build-aux/hydra/gnu-system.scm: qemu-jobs: Work around import problem.
>     
>     Follow-up to 1fe805b290f26db477bcd49be91af54f6c5d061f.
> ---
>  build-aux/hydra/gnu-system.scm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/build-aux/hydra/gnu-system.scm b/build-aux/hydra/gnu-system.scm
> index 46f09d1..0886d5f 100644
> --- a/build-aux/hydra/gnu-system.scm
> +++ b/build-aux/hydra/gnu-system.scm
> @@ -166,7 +166,7 @@ system.")
>    (define (adjust-bootloader os)
>      (if (member system %u-boot-systems)
>        (operating-system (inherit os)
> -        (bootloader (bootloader-configuration
> +        (bootloader ((@ (gnu bootloader) bootloader-configuration)
>                       (bootloader u-boot-bootloader)
>                       (target "/dev/null"))))

What was the problem exactly?  It’s best to add:

  (use-modules (gnu bootloader))

than to use ‘@’.

Ludo’.

Re: 01/01: hydra: Work around import problem.

[Danny Milosavljevic]

Hi Ludo,

On Mon, 29 Jan 2018 14:53:03 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> dannym@scratchpost.org (Danny Milosavljevic) skribis:
> 
> > commit ebd6a6445fa7d3db64795840a607815abbd56880
> > Author: Danny Milosavljevic <dannym@scratchpost.org>
> > Date:   Mon Jan 29 13:23:07 2018 +0100
> >
> >     hydra: Work around import problem.
> >     
> >     * build-aux/hydra/gnu-system.scm: qemu-jobs: Work around import problem.
> >     
> >     Follow-up to 1fe805b290f26db477bcd49be91af54f6c5d061f.
> > ---
> >  build-aux/hydra/gnu-system.scm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/build-aux/hydra/gnu-system.scm b/build-aux/hydra/gnu-system.scm
> > index 46f09d1..0886d5f 100644
> > --- a/build-aux/hydra/gnu-system.scm
> > +++ b/build-aux/hydra/gnu-system.scm
> > @@ -166,7 +166,7 @@ system.")
> >    (define (adjust-bootloader os)
> >      (if (member system %u-boot-systems)
> >        (operating-system (inherit os)
> > -        (bootloader (bootloader-configuration
> > +        (bootloader ((@ (gnu bootloader) bootloader-configuration)
> >                       (bootloader u-boot-bootloader)
> >                       (target "/dev/null"))))  
> 
> What was the problem exactly?  It’s best to add:
> 
>   (use-modules (gnu bootloader))
> 
> than to use ‘@’.
> 
> Ludo’.

Previously, I had added (gnu bootloader) to the existing (use-modules) form
in the file.  That got me this:

evaluating for 'i686-linux' (heap size: 743.0 MiB)...
evaluating for 'armhf-linux' (heap size: 759.0 MiB)...
Backtrace:
In ice-9/boot-9.scm:
    713:2 19 (call-with-prompt _ _ #<procedure default-prompt-handle?>)
In ice-9/eval.scm:
    619:8 18 (_ #(#(#<directory (guile-user) 14c80a0>)))
In ice-9/command-line.scm:
   181:18 17 (_ #<input: string 1bb57e0>)
In unknown file:
          16 (eval (apply (module-ref (resolve-interface (?)) #) #) #)
In /usr/local/bin/hydra-eval-guile-jobs:
   242:18 15 (eval-guile-jobs . _)
In ice-9/boot-9.scm:
    152:2 14 (with-fluid* _ _ _)
In srfi/srfi-1.scm:
   679:15 13 (append-map _ _ . _)
   592:29 12 (map1 _)
   592:29 11 (map1 _)
   592:17 10 (map1 ("armhf-linux"))
In ice-9/eval.scm:
   173:47  9 (_ #(#(#(#(#<module (#{ g859}#) 1748320> #<?> ?) ?) ?) ?))
    159:9  8 (_ #(#(#(#<module (#{ g859}#) 1748320>) #<build-d?> ?) ?))
   196:43  7 (_ #(#(#(#<module (#{ g859}#) 1748320>) #<build-d?> ?) ?))
In guix/store.scm:
  1443:24  6 (run-with-store _ _ #:guile-for-build _ #:system _ # _)
In ice-9/eval.scm:
   191:27  5 (_ #(#(#<module (#{ g859}#) 1748320> #<procedure 6?>) ?))
    163:9  4 (_ #(#(#<module (#{ g859}#) 1748320> #<variable 1?> ?) ?))
   293:34  3 (_ #(#(#<module (#{ g859}#) 1748320> "armhf-linux") #))
   182:19  2 (proc #(#(#<module (#{ g859}#) 1748320> "armhf-lin?") #))
   142:16  1 (compile-top-call _ (7 . bootloader-configuration) (# #))
In unknown file:
           0 (%resolve-variable (7 . bootloader-configuration) #<mod?>)

ERROR: In procedure %resolve-variable:
ERROR: bootloader-configuration: unbound variable

Some deprecated features have been used.  Set the environment
variable GUILE_WARN_DEPRECATED to "detailed" and rerun the
program to get more information.  Set it to "no" to suppress
this message.

Re: 01/01: hydra: Work around import problem.

[Mark H Weaver]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> writes:

> On Mon, 29 Jan 2018 14:53:03 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> dannym@scratchpost.org (Danny Milosavljevic) skribis:
>> 
>> > commit ebd6a6445fa7d3db64795840a607815abbd56880
>> > Author: Danny Milosavljevic <dannym@scratchpost.org>
>> > Date:   Mon Jan 29 13:23:07 2018 +0100
>> >
>> >     hydra: Work around import problem.
>> >     
>> >     * build-aux/hydra/gnu-system.scm: qemu-jobs: Work around import problem.
>> >     
>> >     Follow-up to 1fe805b290f26db477bcd49be91af54f6c5d061f.
>> > ---
>> >  build-aux/hydra/gnu-system.scm | 2 +-
>> >  1 file changed, 1 insertion(+), 1 deletion(-)
>> >
>> > diff --git a/build-aux/hydra/gnu-system.scm b/build-aux/hydra/gnu-system.scm
>> > index 46f09d1..0886d5f 100644
>> > --- a/build-aux/hydra/gnu-system.scm
>> > +++ b/build-aux/hydra/gnu-system.scm
>> > @@ -166,7 +166,7 @@ system.")
>> >    (define (adjust-bootloader os)
>> >      (if (member system %u-boot-systems)
>> >        (operating-system (inherit os)
>> > -        (bootloader (bootloader-configuration
>> > +        (bootloader ((@ (gnu bootloader) bootloader-configuration)
>> >                       (bootloader u-boot-bootloader)
>> >                       (target "/dev/null"))))  
>> 
>> What was the problem exactly?  It’s best to add:
>> 
>>   (use-modules (gnu bootloader))
>> 
>> than to use ‘@’.
>> 
>> Ludo’.
>
> Previously, I had added (gnu bootloader) to the existing (use-modules) form
> in the file.  That got me this:
[...]
> In unknown file:
>            0 (%resolve-variable (7 . bootloader-configuration) #<mod?>)
>
> ERROR: In procedure %resolve-variable:
> ERROR: bootloader-configuration: unbound variable

I strongly suspect that this evaluation error occurred for a commit
_before_ you added (use-modules (gnu bootloader)), unless you have
reason to believe otherwise.

      Mark

Re: 01/01: hydra: Work around import problem.

[Danny Milosavljevic]

Hi Mark,

On Mon, 29 Jan 2018 09:47:01 -0500
Mark H Weaver <mhw@netris.org> wrote:

> I strongly suspect that this evaluation error occurred for a commit
> _before_ you added (use-modules (gnu bootloader)), unless you have
> reason to believe otherwise.

Ah, might be.  How do I find out which commit has been evaluated?

Re: 01/01: hydra: Work around import problem.

[Mark H Weaver]

Danny Milosavljevic <dannym@scratchpost.org> writes:

> On Mon, 29 Jan 2018 09:47:01 -0500
> Mark H Weaver <mhw@netris.org> wrote:
>
>> I strongly suspect that this evaluation error occurred for a commit
>> _before_ you added (use-modules (gnu bootloader)), unless you have
>> reason to believe otherwise.
>
> Ah, might be.  How do I find out which commit has been evaluated?

The commit of _successful_ Hydra evaluations is recorded, but I don't
know of any record of the commits of failed evaluations.  However, from
my own memory of the timeline, I feel reasonably confident that the
failed Hydra evaluation was started in the ~10 hour period between
commit de2bc8d6d (hydra: add arm image builder) and commit 1fe805b29
(hydra: Add (gnu bootloader) import.)

I think it would be okay to simply revert commit ebd6a6445 (hydra: Work
around import problem.) and see what happens.

However, it also seems that the current 'flash-image' job isn't working.
Its first build on Hydra got stuck at a guile prompt:

  https://hydra.gnu.org/build/2481717/nixlog/6/tail-reload
  https://hydra.gnu.org/build/2481717

The error message was "ERROR: In procedure copy-file: Success".  I
believe this was caused by the fact that Guile's 'copy-file' procedure
incorrectly assumes that an error occurred if 'write' writes fewer bytes
than we asked it to.  This can happen if interrupted by a signal, or if
the file system was full.  I'll look into fixing this in Guile, although
in this case it seems likely that the file system was full.

      Mark

System calls interrupted by signals - or not

[Danny Milosavljevic]

Hi Mark,

>This can happen if interrupted by a signal, 

If only BSD-style signals (SA_RESTART) are used, as is very likely, this
should not happen.

It only happens with old "PC-losering" style signal handlers - which even
signal(2) doesn't install anymore (and I know of no reason why anyone
should ever use those).

If an OK design limitation is to support only SA_RESTART handlers, I think it
suffices to forbid anyone from using msgrcv and msgsnd.  That's all.  No EINTR
loops or buffer position adjustments necessary.

Otherwise I made a table some years ago when this bit me, I'll paste it here:

Affected Linux syscalls in glibc 2.19 (each lists the newest version only):
E     futex
E     semop
E     semtimedop
E     pause
f     ptrace (FIXME)
E     rt_sigsuspend
E   T rt_sigtimedwait
E     sigsuspend
E     waitid
E     waitpid
E     creat
E     open
E     openat
E B   read
E B   write
      close (DO NOT retry on EINTR)
E   U pselect6
E     dup
E     dup2
E     dup3
E   T epoll_pwait
E   T epoll_wait
E     fallocate
E     fcntl64
E     flock
E     ftruncate64
E     truncate64
E   T poll
E   T ppoll
E   T io_getevents (timeout not modified) (libaio) (only for the direct syscall! otherwise F)
E     fstatfs64
E     statfs64
E     accept4
E     connect
E B   recv
E B   recvfrom
E B   recvmsg
E B   send
E B   sendmsg
E B   sendto
E     request_key
e   t clock_nanosleep
E   t nanosleep
E B t mq_timedreceive
E B t mq_timedsend
E B   msgrcv [IGNORES SA_RESTART]
E B   msgsnd [IGNORES SA_RESTART]
E B   preadv
E B   readv
E B   pwritev
E B   writev
E   U newselect
E B   getrandom

Meaning:
        E ... returns (-1), sets errno == EINTR
        e ... returns EINTR directly.
        f ... returns (-1) whenever it feels like it, but sets errno on errors.
        F ... returns less than min_nr on error.
        B ... have to adjust buffer
        T ... have to adjust timeout. Note that timeout is optional usually.
        t ... have to adjust timeout, but that's easy. Note that timeout is optional usually.
        U ... have to adjust timeout, but "timeout is undefined". Note that timeout is optional usually.

Note: read() returns EFAULT on SEGV instead of doing SIGSEGV.

>or if the file system was full.

Yeah, I think I had this too some time ago.

Re: System calls interrupted by signals - or not

[Mark H Weaver]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> writes:

>>This can happen if interrupted by a signal, 
>
> If only BSD-style signals (SA_RESTART) are used, as is very likely, this
> should not happen.

Guile is a library meant for use within existing applications, and
therefore needs to be able to cope with whatever signal handling policy
those applications have chosen.  We certainly cannot assume that all
kinds of signals will be configured for SA_RESTART.

> It only happens with old "PC-losering" style signal handlers - which even
> signal(2) doesn't install anymore (and I know of no reason why anyone
> should ever use those).

Although SA_RESTART is more convenient, less error prone, and arguably
the sensible default, it is also strictly less powerful than what you
can do without SA_RESTART.  There are cases where you may want the
ability to interrupt a system call without killing the thread.  Suppose
you are waiting for a large I/O operation to complete over a slow
network or device.  Signals are the only way I know of in POSIX to
interrupt a system call, but it can only be done if there's at least one
kind of signal that's not configured for SA_RESTART.

At least that's my understanding.  Please correct me if I'm wrong.

Note that EINTR is the default behavior in GNU libc when using the
advanced signal handling interfaces:

<https://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html>

  "When you don’t specify with ‘sigaction’ or ‘siginterrupt’ what a
   particular handler should do, it uses a default choice.  The default
   choice in the GNU C Library is to make primitives fail with ‘EINTR’."

      Mark

Re: System calls interrupted by signals - or not

[Danny Milosavljevic]

Hi Mark,

On Thu, 01 Feb 2018 03:13:33 -0500
Mark H Weaver <mhw@netris.org> wrote:

> Guile is a library meant for use within existing applications, and
> therefore needs to be able to cope with whatever signal handling policy
> those applications have chosen.  We certainly cannot assume that all
> kinds of signals will be configured for SA_RESTART.

That's too bad.  It can't be helped, then.
Then see the table for a little overview.

(I've tried very hard to sensibly handle EINTR in the past and it's
surprisingly difficult - that table is what I got out of it - and
the decision to avoid PC-losering signals whenever I can)

And even for the purported use of EINTR (so that you can have complicated
signal-unsafe handler actions after an "if (errno == EINTR)" block)
it's difficult to get right.  That's because you only get EINTR when a system
call has been interrupted by a signal.  It can happen that you aren't yet in
the system call, the signal handler runs (to completion), and then you enter a
system call.
You *don't* get EINTR for the missed signal then.
I don't know what they were thinking.

> There are cases where you may want the
> ability to interrupt a system call without killing the thread.  Suppose
> you are waiting for a large I/O operation to complete over a slow
> network or device.  Signals are the only way I know of in POSIX to
> interrupt a system call, but it can only be done if there's at least one
> kind of signal that's not configured for SA_RESTART.

That's a good point.

But many people just use

  do { syscall } while (errno == EINTR);

in random libraries and then you can't interrupt the system call after all
in your user program.

Also, there's a race because you can be right before entering a system call,
your signal handler runs, and then the system call isn't interrupted after
all (because there was nothing to interrupt - and now you can't branch
on it anymore.  That's how I started to enter this EINTR rabbit hole -
one of my programs had such a bug).

>   "When you don’t specify with ‘sigaction’ or ‘siginterrupt’ what a
>    particular handler should do, it uses a default choice.  The default
>    choice in the GNU C Library is to make primitives fail with ‘EINTR’."

:-(