bug#48114: Disarchive occasionally fails tests

[Bengt Richter <bokr@bokr.com>]

Hi Timothy, Ludo,

On +2021-05-03 00:02:09 -0400, Timothy Sample wrote:
> Timothy Sample <samplet@ngyro.com> writes:
> 
> > I’m still looking into this, but I wanted to quickly post this
> > reproducer for the Guile bug:
> >
> >     (use-modules (ice-9 regex))
> >     (define str
> > "\U101514\U103ab0\U0f6e6e\U02e278\U01d9eb\U10b996\U1089b5\uea15\U0fa074\U101e41\U02e330\u0177\u2492")
> >     (match:substring (string-match "[0-8]+" str))
> >
> > This triggers the out-of-range error when run with “LC_ALL=C”.
> 
> It turns out that all that’s needed is the last code point, which is
> “Number Eleven Full Stop”, or ‘⒒’.  When Guile converts this to an ASCII
> C string using ‘u32_conv_from_encoding’, it becomes “11.”.  The regex
> (“[0-8]+”) matches the “11” part with start index 0 and end index 2.
> The ‘fixup_multibyte_match’ function does nothing (it only matters when
> the locale encoding is multibyte) [1].  Guile then builds the match
> vector with the original string but keeps the ASCII offsets.  In other
> words, it thinks the match substring goes from 0 to 2 in a single code
> point string:
> 
>     ,use (ice-9 regex)
>     (string-match "11" "\u2492")
>     => #("\u2492" (0 . 2))
> 
> I’m not sure there’s any way to solve this nicely in Guile.  It would be
> clearer if the match vector included the string as libc matched it, but
> it’s still surprising that the match happens with a different string.
> 
> In Disarchive, I can rewrite the generator without regex.  I’ll do that
> and see what I can do about the “Gave up!” issue.
> 
> [1] It works on the converted-to-ASCII C string, which means that the
> byte offsets and code point offsets are the same.  Hence, it has nothing
> to do.
> 
> 
> -- Tim
>

> 
> 
What happens with these?
(code ppoints in decimal)

    8554 _Ⅺ_ "ROMAN NUMERAL ELEVEN"
    8570 _ⅺ_ "SMALL ROMAN NUMERAL ELEVEN"
    9322 _⑪_ "CIRCLED NUMBER ELEVEN"
    9342 _⑾_ "PARENTHESIZED NUMBER ELEVEN"
    9362 _⒒_ "NUMBER ELEVEN FULL STOP"
    9451 _⓫_ "NEGATIVE CIRCLED NUMBER ELEVEN"
   13155 _㍣_ "IDEOGRAPHIC TELEGRAPH SYMBOL FOR HOUR ELEVEN"
   13290 _㏪_ "IDEOGRAPHIC TELEGRAPH SYMBOL FOR DAY ELEVEN"

I would argue that none of these should be "decoded" into ascii polyglyphs
since they are atomic character glyphs. IMO It is over-eager transformation
to make them into ascii polyglyphs.

/Super/sub/-script placement metadata is another thing to consider --
"decode" to ascii art?? ;-)

Unicode characters representing mathematical values in
other languages are different. Those are subject to natural language
translation with locale-dependent semantics.

These might be candidates for that?:
(code points in decimal)

    8544 _Ⅰ_ "ROMAN NUMERAL ONE"
    8545 _Ⅱ_ "ROMAN NUMERAL TWO"
    8546 _Ⅲ_ "ROMAN NUMERAL THREE"
    8547 _Ⅳ_ "ROMAN NUMERAL FOUR"
    8548 _Ⅴ_ "ROMAN NUMERAL FIVE"
    8549 _Ⅵ_ "ROMAN NUMERAL SIX"
    8550 _Ⅶ_ "ROMAN NUMERAL SEVEN"
    8551 _Ⅷ_ "ROMAN NUMERAL EIGHT"
    8552 _Ⅸ_ "ROMAN NUMERAL NINE"
    8553 _Ⅹ_ "ROMAN NUMERAL TEN"
    8554 _Ⅺ_ "ROMAN NUMERAL ELEVEN"
    8555 _Ⅻ_ "ROMAN NUMERAL TWELVE"
    8556 _Ⅼ_ "ROMAN NUMERAL FIFTY"
    8557 _Ⅽ_ "ROMAN NUMERAL ONE HUNDRED"
    8558 _Ⅾ_ "ROMAN NUMERAL FIVE HUNDRED"
    8559 _Ⅿ_ "ROMAN NUMERAL ONE THOUSAND"
    8560 _ⅰ_ "SMALL ROMAN NUMERAL ONE"
    8561 _ⅱ_ "SMALL ROMAN NUMERAL TWO"
    8562 _ⅲ_ "SMALL ROMAN NUMERAL THREE"
    8563 _ⅳ_ "SMALL ROMAN NUMERAL FOUR"
    8564 _ⅴ_ "SMALL ROMAN NUMERAL FIVE"
    8565 _ⅵ_ "SMALL ROMAN NUMERAL SIX"
    8566 _ⅶ_ "SMALL ROMAN NUMERAL SEVEN"
    8567 _ⅷ_ "SMALL ROMAN NUMERAL EIGHT"
    8568 _ⅸ_ "SMALL ROMAN NUMERAL NINE"
    8569 _ⅹ_ "SMALL ROMAN NUMERAL TEN"
    8570 _ⅺ_ "SMALL ROMAN NUMERAL ELEVEN"
    8571 _ⅻ_ "SMALL ROMAN NUMERAL TWELVE"
    8572 _ⅼ_ "SMALL ROMAN NUMERAL FIFTY"
    8573 _ⅽ_ "SMALL ROMAN NUMERAL ONE HUNDRED"
    8574 _ⅾ_ "SMALL ROMAN NUMERAL FIVE HUNDRED"
    8575 _ⅿ_ "SMALL ROMAN NUMERAL ONE THOUSAND"
    8576 _ↀ_ "ROMAN NUMERAL ONE THOUSAND C D"
    8577 _ↁ_ "ROMAN NUMERAL FIVE THOUSAND"
    8578 _ↂ_ "ROMAN NUMERAL TEN THOUSAND"
    8579 _Ↄ_ "ROMAN NUMERAL REVERSED ONE HUNDRED"
    8581 _ↅ_ "ROMAN NUMERAL SIX LATE FORM"
    8582 _ↆ_ "ROMAN NUMERAL FIFTY EARLY FORM"
    8583 _ↇ_ "ROMAN NUMERAL FIFTY THOUSAND"
    8584 _ↈ_ "ROMAN NUMERAL ONE HUNDRED THOUSAND"
   12321 _〡_ "HANGZHOU NUMERAL ONE"
   12322 _〢_ "HANGZHOU NUMERAL TWO"
   12323 _〣_ "HANGZHOU NUMERAL THREE"
   12324 _〤_ "HANGZHOU NUMERAL FOUR"
   12325 _〥_ "HANGZHOU NUMERAL FIVE"
   12326 _〦_ "HANGZHOU NUMERAL SIX"
   12327 _〧_ "HANGZHOU NUMERAL SEVEN"
   12328 _〨_ "HANGZHOU NUMERAL EIGHT"
   12329 _〩_ "HANGZHOU NUMERAL NINE"
   12344 _〸_ "HANGZHOU NUMERAL TEN"
   12345 _〹_ "HANGZHOU NUMERAL TWENTY"
   12346 _〺_ "HANGZHOU NUMERAL THIRTY"

Just my intuitive reaction, no academic creds to back it up ;)

-- 
Regards,
Bengt Richter