bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Christopher Wellons]

The cl-random generator was not written for bignums, so it misbehaves in
Emacs 27. After generating a few million numbers from any particular
state, it will only signal "Arithmetic overflow error". The generator
relies on fixnums wrapping via two's complement. In Emacs 27, fixnums
turn into bignums rather than wrap, so the state grows until reaching
the bignum limit, then breaks.

The cl-random function is a lagged Fibonacci generator. The output is
the difference of two integers at special "taps" in the state vector,
and one tap is replaced with the output. The output is properly
truncated using logand, but state update is not, and it soon fills with
growing bignums.

The fix is trivial: Move the logand truncation so that it applies to
both the output and the state update. The truncated bits are never used
so the output of the generator remains unchanged.

diff --git a/lisp/emacs-lisp/cl-extra.el b/lisp/emacs-lisp/cl-extra.el
index 7e9d8fe870..2e0b37c14d 100644
--- a/lisp/emacs-lisp/cl-extra.el
+++ b/lisp/emacs-lisp/cl-extra.el
@@ -469,7 +469,7 @@ cl-random
 	  (while (< (setq i (1+ i)) 200) (cl-random 2 state))))
     (let* ((i (cl-callf (lambda (x) (% (1+ x) 55)) (cl--random-state-i state)))
 	   (j (cl-callf (lambda (x) (% (1+ x) 55)) (cl--random-state-j state)))
-	   (n (logand 8388607 (aset vec i (- (aref vec i) (aref vec j))))))
+	   (n (aset vec i (logand 8388607 (- (aref vec i) (aref vec j))))))
       (if (integerp lim)
 	  (if (<= lim 512) (% n lim)
 	    (if (> lim 8388607) (setq n (+ (ash n 9) (cl-random 512 state))))

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Mattias Engdegård]

Thank you! The proposed fix indeed looks correct; applied to emacs-27.

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Pip Cet]

On Thu, Dec 26, 2019 at 5:13 PM Christopher Wellons
<wellons@nullprogram.com> wrote:
> The cl-random generator was not written for bignums, so it misbehaves in
> Emacs 27.

Good catch.

I might be wrong, but it looks to me like cl-random and Frandom are
currently both limited to small integers (< 31 bits and fixnum range,
respectively) and we don't have a random number generator that will do
the right thing for
(random 100000000000000000000000000000000000000000000)

Any idea how easy it would be to fix either of them, or both?

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Philipp Stephani]

Am So., 29. Dez. 2019 um 14:33 Uhr schrieb Pip Cet <pipcet@gmail.com>:
>
> On Thu, Dec 26, 2019 at 5:13 PM Christopher Wellons
> <wellons@nullprogram.com> wrote:
> > The cl-random generator was not written for bignums, so it misbehaves in
> > Emacs 27.
>
> Good catch.
>
> I might be wrong, but it looks to me like cl-random and Frandom are
> currently both limited to small integers (< 31 bits and fixnum range,
> respectively) and we don't have a random number generator that will do
> the right thing for
> (random 100000000000000000000000000000000000000000000)
>
> Any idea how easy it would be to fix either of them, or both?


I haven't tried it out, but mpz_urandomm should be what we need. We'd
need to find a replacement for mini-gmp though.

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Mattias Engdegård]

> I might be wrong, but it looks to me like cl-random and Frandom are currently both limited to small integers (< 31 bits and fixnum range, respectively) and we don't have a random number generator that will do the right thing for
> (random 100000000000000000000000000000000000000000000) 

Thank you for noticing this! It definitely merits a separate bug (or two).

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Christopher Wellons]

> Any idea how easy it would be to fix either of them, or both?

The reason I found that bug is because I was experimenting with 
addressing exactly that problem:

https://github.com/skeeto/lcg128

It's got the right features, including support for arbitrary limits (per 
your example), but I'm not satisfied with the performance. For arbitrary 
limits it uses the same rejection algorithm as Python — generate (logb 
lim) bits and reject if out of range — and cl-random could be updated to 
use the same technique. The LCG around 6x slower than cl-random in the 
common case (generating small numbers). Bignums aren't ideal for this 
since every operation requires an allocation, and the LCG throws away 
half of the work (the upper half of the multiplication result).

The lagged Fibonacci generator really does hit a sweet spot for Emacs 
Lisp, where, before bignums, even 32-bit integers weren't a reliable 
option. So it fits within Emacs' worst fixnum limitations and requires 
few bytecode instructions to generate a number.

The random built-in ought to simply be pcg32 across all platforms:

http://www.pcg-random.org/download.html

Currently it's whatever crummy host-provided PRNG it happens to find at 
compile time. It's certainly possible to implement pcg32 using Emacs 27 
bignums, but it would be even slower than my LCG, especially on 32-bit 
platforms. In C it's very, very fast.

If the pcg32 state, just one 64-bit integer, was an optional parameter, 
then cl-random could build on it. However, the state would nearly always 
be a bignum, even on 64-bit platforms, and that would really slow it 
down. I haven't yet thought of any obviously good options for this.

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Pip Cet]

On Sun, Dec 29, 2019 at 2:27 PM Christopher Wellons
<wellons@nullprogram.com> wrote:
> > Any idea how easy it would be to fix either of them, or both?
> The reason I found that bug is because I was experimenting with
> addressing exactly that problem:
>
> https://github.com/skeeto/lcg128
>
> It's got the right features, including support for arbitrary limits (per
> your example), but I'm not satisfied with the performance.

That looks excellent. Maybe we should be using a cryptographic RNG by
default, though? I really don't know the tradeoffs, though.

At first glance, there are two questions I have about lcg128:
 - should random floats have more than 53 significant bits if they're
< 0.5? I think the answer is yes, even though that will make the
implementation harder.
 - should (lcg128-range 2^n) call the generator twice, on average? I
think it's an important special case in which we only want to call the
generator once.

bug#38753: 27.0.60; cl--random-state uncontrolled growth due to bignums

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Date: Sun, 29 Dec 2019 17:33:59 +0000
> Cc: 38753@debbugs.gnu.org
> 
> That looks excellent. Maybe we should be using a cryptographic RNG by
> default, though?

Are you aware that we already init the random seed using system
entropy and/or cryptographic RNG?  See init_random.