Re: heads-up: Haskell updates

[Mark H Weaver <mhw@netris.org>]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> writes:

> Hi Mark,
> Hi Ricardo,
>
> On Wed, 14 Feb 2018 20:39:12 +0100
> Ricardo Wurmus <rekado@elephly.net> wrote:
>
>> Nor do I see this message:
>> 
>>     ghc-pkg: unable to decommit memory: Invalid argument
>
> Which Linux kernel version does this run on?

Last I knew, Hydra's x86 build slaves were all running kernels older
than 4.5, but I'm not sure if that's still the case.

>> I don’t know what this message means, but the messages about requiring a
>
> If there's large address space support [1], ghc 8 does its own allocation in a
> 1 TB address space.  That means it has to tell the kernel when it doesn't
> need some chunk anymore - otherwise you're gonna run out of memory.
>
> It does that using madvise(2).  There's two ways it tries to do that:
>
> (1) MADV_FREE: Signals that "I don't need that range at all anymore".
> It usually means Linux will mark those pages free.
>
> (2) MADV_DONTNEED: Signals that "I don't need that range in the NEAR FUTURE".
> It usually means Linux will swap those pages out.
>
> MADV_FREE was added in Linux 4.5.  Haskell uses a #ifdef to detect it.

Given that it's using #ifdef to detect this, I'd expect the result to
depend only on the _headers_ being used to compile, and not the actual
kernel running the build.  Is that right?  If so, this doesn't explain
why it works for Ricardo but not for Andreas and Hydra.

Note that on our 'master' branch we are using Linux-libre-4.4 kernel
headers, but on 'core-updates' we're using 4.9 kernel headers.

The recently created evaluation 109913 on Hydra is for core-updates with
the recent Haskell changes included.  It'll be interesting to see if the
same failure happens there.  I gave the 'ghc-resourcet' jobs a high
priority.  Here are the 'ghc-resourcet' jobs for evaluation 109913:

  https://hydra.gnu.org/build/2497604  (x86_64)
  https://hydra.gnu.org/build/2498874  (i686)

> Newer glibc (such as the one in core-updates) unconditionally define MADV_FREE
> to prevent programs from depending on a specific Linux kernel in this way [2].
>
> There's a patch to ghc that falls back to (2) if (1) doesn't work:
>
> https://git.haskell.org/ghc.git/commitdiff/6576bf83cdf4eac05eb88a24aa934a736c91e3da
>
> ... but ghc 8.0.2 which we have on core-updates doesn't use it.
> It uses either MADV_FREE or MADV_DONTNEED, determined at compile time.
>
> So if the Linux kernel is < 4.5 that's gonna end very badly.

Thanks for the detailed analysis!

I suppose if one reads the error message literally:

  Control/Monad/Trans/Resource/Internal.hs:302:10: error:
      • Could not deduce (MonadBase IO (Strict.StateT s m))
          arising from the superclasses of an instance declaration
        from the context: MonadResource m
          bound by the instance declaration
          at Control/Monad/Trans/Resource/Internal.hs:302:10-63
      • In the instance declaration for
          ‘MonadResource (Strict.StateT s m)’

"Could not deduce" might be interpreted to include the case where we
failed for lack of sufficient memory.  They might have written the code
to catch any error at all, including out of memory errors, and report
the failure with this error message.

What do you think?

      Mark