Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

Hi Robert,

> branch: emacs-29
> commit 2f94f6de9d64f9fd89284dac171e166e7d721dcd
> Author: Robert Pluim <rpluim@gmail.com>
> Commit: Robert Pluim <rpluim@gmail.com>
>
>     Make VS-15 and VS-16 compositions work correctly
>
>     There is a conflict between forward matching and backward matching
>     composition rules involving the same codepoint, which can cause the
>     backward matching ones not to be invoked.  Ensure that VS-15 (U+FE0E)
>     and VS-16 (U+FE0F) are composed by forward matching rules instead in
>     order to avoid this issue.
>
>     * admin/unidata/emoji-zwj.awk: Add rules for CHAR+VS-15 and CHAR+VS-16.
>     * lisp/composite.el: remove backward matching rule for VS-15.  (Bug#63731)
> ---
>  admin/unidata/emoji-zwj.awk | 3 ++-
>  lisp/composite.el           | 2 +-
>  2 files changed, 3 insertions(+), 2 deletions(-)

On emba, we see (in make bootstrap)

--8<---------------cut here---------------start------------->8---
#12 559.7   GEN      ../../lisp/international/emoji-zwj.el
#12 559.8 mawk: program limit exceeded: sprintf buffer size=8192
#12 559.8 	FILENAME="emoji-zwj.awk" FNR=141 NR=3046
#12 559.8 make[4]: *** [Makefile:120: ../../lisp/international/emoji-zwj.el] Error 2
--8<---------------cut here---------------end--------------->8---

See <https://emba.gnu.org/emacs/emacs/-/jobs/68356>

Best regards, Michael.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Robert Pluim <rpluim@gmail.com>
> Date: Fri, 02 Jun 2023 20:22:32 +0200
> 
> > commit 2f94f6de9d64f9fd89284dac171e166e7d721dcd
> > Author: Robert Pluim <rpluim@gmail.com>
> > Commit: Robert Pluim <rpluim@gmail.com>
> >
> >     Make VS-15 and VS-16 compositions work correctly
> >
> >     There is a conflict between forward matching and backward matching
> >     composition rules involving the same codepoint, which can cause the
> >     backward matching ones not to be invoked.  Ensure that VS-15 (U+FE0E)
> >     and VS-16 (U+FE0F) are composed by forward matching rules instead in
> >     order to avoid this issue.
> >
> >     * admin/unidata/emoji-zwj.awk: Add rules for CHAR+VS-15 and CHAR+VS-16.
> >     * lisp/composite.el: remove backward matching rule for VS-15.  (Bug#63731)
> > ---
> >  admin/unidata/emoji-zwj.awk | 3 ++-
> >  lisp/composite.el           | 2 +-
> >  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> On emba, we see (in make bootstrap)
> 
> --8<---------------cut here---------------start------------->8---
> #12 559.7   GEN      ../../lisp/international/emoji-zwj.el
> #12 559.8 mawk: program limit exceeded: sprintf buffer size=8192
> #12 559.8 	FILENAME="emoji-zwj.awk" FNR=141 NR=3046
> #12 559.8 make[4]: *** [Makefile:120: ../../lisp/international/emoji-zwj.el] Error 2
> --8<---------------cut here---------------end--------------->8---
> 
> See <https://emba.gnu.org/emacs/emacs/-/jobs/68356>

Maybe we should require Gawk for building Emacs.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> On emba, we see (in make bootstrap)
>>
>> --8<---------------cut here---------------start------------->8---
>> #12 559.7   GEN      ../../lisp/international/emoji-zwj.el
>> #12 559.8 mawk: program limit exceeded: sprintf buffer size=8192
>> #12 559.8 	FILENAME="emoji-zwj.awk" FNR=141 NR=3046
>> #12 559.8 make[4]: *** [Makefile:120: ../../lisp/international/emoji-zwj.el] Error 2
>> --8<---------------cut here---------------end--------------->8---
>>
>> See <https://emba.gnu.org/emacs/emacs/-/jobs/68356>
>
> Maybe we should require Gawk for building Emacs.

I've added this for emba. But likely, an additional check in
configure.ac would be helpful.

Best regards, Michael.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Mattias Engdegård]

3 juni 2023 kl. 09.57 skrev Michael Albinus <michael.albinus@gmx.de>:

>> Maybe we should require Gawk for building Emacs.
> 
> I've added this for emba. But likely, an additional check in
> configure.ac would be helpful.

Standard (BSD) awk on macOS seems to work nicely, so please don't require Gawk in configure.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Sat, 3 Jun 2023 10:38:13 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org, rpluim@gmail.com
> 
> 3 juni 2023 kl. 09.57 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> >> Maybe we should require Gawk for building Emacs.
> > 
> > I've added this for emba. But likely, an additional check in
> > configure.ac would be helpful.
> 
> Standard (BSD) awk on macOS seems to work nicely, so please don't require Gawk in configure.

Patches to support mawk are welcome.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

Hi Mattias,

>>> Maybe we should require Gawk for building Emacs.
>> 
>> I've added this for emba. But likely, an additional check in
>> configure.ac would be helpful.
>
> Standard (BSD) awk on macOS seems to work nicely, so please don't require Gawk in configure.

As usual, configure shouldn't check for a given (g)awk product. It
should check, whether "awk -f admin/unidata/emoji-zwj.awk" returns a
zero exit code.

Best regards, Michael.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Mattias Engdegård]

3 juni 2023 kl. 10.48 skrev Michael Albinus <michael.albinus@gmx.de>:

> As usual, configure shouldn't check for a given (g)awk product. It
> should check, whether "awk -f admin/unidata/emoji-zwj.awk" returns a
> zero exit code.

I tried generating emoji-zwj.el using a newly built mawk (1.3.4) and there were no errors. Not sure what part should have exceeded the sprintf buffer limit.

(By the way, the Makefile feeds the emoji-zwj.awk script itself into the script as data which is a bit silly.)

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

Hi Mattias,

>> As usual, configure shouldn't check for a given (g)awk product. It
>> should check, whether "awk -f admin/unidata/emoji-zwj.awk" returns a
>> zero exit code.
>
> I tried generating emoji-zwj.el using a newly built mawk (1.3.4) and there were no errors. Not sure what part should have exceeded the sprintf buffer limit.

emba runs a debian:bullseye container with mawk 1.3.4.

--8<---------------cut here---------------start------------->8---
oot@51ef3df5a6c1:/checkout# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@51ef3df5a6c1:/checkout# apt-cache show mawk
Package: mawk
Status: install ok installed
Priority: required
Section: interpreters
Installed-Size: 242
Maintainer: Boyuan Yang <byang@debian.org>
Architecture: amd64
Multi-Arch: foreign
Version: 1.3.4.20200120-2
...
--8<---------------cut here---------------end--------------->8---

Perhaps the restriction depends on the underlying os. A check in
configure would be needed even more.

Best regards, Michael.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Mattias Engdegård]

3 juni 2023 kl. 11.44 skrev Michael Albinus <michael.albinus@gmx.de>:

> emba runs a debian:bullseye container with mawk 1.3.4.

> Version: 1.3.4.20200120-2

I used the most recent version, 1.3.4-20230525, but I cannot reproduce the error (on macOS) even with 1.3.4-20200120. These two versions produce identical output.

> Perhaps the restriction depends on the underlying os. A check in
> configure would be needed even more.

mawk -Wversion prints:

mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs:       unknown
regex-funcs:        internal
compiled limits:
sprintf buffer      8192
maximum-integer     2147483647

Note the sprintf buffer size, the same as in the error message when mawk fails on emba.

There may be something more than just a buffer limitation here. I don't know if the sprintf buffer is used for more than literal calls to sprintf, but they all seem to be small and well-behaved.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

Hi Mattias,

>> Perhaps the restriction depends on the underlying os. A check in
>> configure would be needed even more.
>
> mawk -Wversion prints:
>
> mawk 1.3.4 20200120
> Copyright 2008-2019,2020, Thomas E. Dickey
> Copyright 1991-1996,2014, Michael D. Brennan
>
> random-funcs:       unknown
> regex-funcs:        internal
> compiled limits:
> sprintf buffer      8192
> maximum-integer     2147483647
>
> Note the sprintf buffer size, the same as in the error message when
> mawk fails on emba.
>
> There may be something more than just a buffer limitation here. I
> don't know if the sprintf buffer is used for more than literal calls
> to sprintf, but they all seem to be small and well-behaved.

Just to make sure: the problem happens in the emacs-29 branch only. Eli
has merged it into master just some minutes ago.

Or it depends on the way awk is invoked when generating
lisp/international/emoji-zwj.el.

If you like to check the emba environment on your local machine, see
admin/notes/emba. It contains instructions how to build and run the
respective container emacs-inotify. Not that you need to remove gawk
from the build recipe (line 32 in Dockerfile.emba), which I have added
this morning.

Best regards, Michael.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Mattias Engdegård]

3 juni 2023 kl. 12.41 skrev Michael Albinus <michael.albinus@gmx.de>:

> Just to make sure: the problem happens in the emacs-29 branch only. Eli
> has merged it into master just some minutes ago.

Thank you, that was very helpful. Now I can reproduce the mawk error and have fixed it on master; no configuration change or gawk should be needed.

If the change works as expected, maybe we want to backport it to emacs-29.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Sat, 3 Jun 2023 13:01:02 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org, rpluim@gmail.com
> 
> 3 juni 2023 kl. 12.41 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> > Just to make sure: the problem happens in the emacs-29 branch only. Eli
> > has merged it into master just some minutes ago.
> 
> Thank you, that was very helpful. Now I can reproduce the mawk error and have fixed it on master; no configuration change or gawk should be needed.
> 
> If the change works as expected, maybe we want to backport it to emacs-29.

Why on earth did you not fix that on emacs-29 in the first place?
Bugs introduced by changes on emacs-29 should be fixed there.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

Hi Mattias,

> If the change works as expected, maybe we want to backport it to emacs-29.

Thanks. I'll remove gawk from emba once your patch has arrived the
emacs-29 branch.

Best regards, Michael.

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Robert Pluim]

>>>>> On Sat, 3 Jun 2023 13:01:02 +0200, Mattias Engdegård <mattiase@acm.org> said:

    Mattias> 3 juni 2023 kl. 12.41 skrev Michael Albinus <michael.albinus@gmx.de>:
    >> Just to make sure: the problem happens in the emacs-29 branch only. Eli
    >> has merged it into master just some minutes ago.

    Mattias> Thank you, that was very helpful. Now I can reproduce the mawk error
    Mattias> and have fixed it on master; no configuration change or gawk should be
    Mattias> needed.

    Mattias> If the change works as expected, maybe we want to backport it to emacs-29.

Yes, it should go to emacs-29

Robert
-- 

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Robert Pluim]

>>>>> On Sat, 3 Jun 2023 11:26:46 +0200, Mattias Engdegård <mattiase@acm.org> said:

    Mattias> (By the way, the Makefile feeds the emoji-zwj.awk script itself into the script as data which is a bit silly.)

It is, Iʼll queue a patch for that on master (blocks.awk has the same
problem), unless you beat me to it.

Robert
-- 

Re: emacs-29 2f94f6de9d6: Make VS-15 and VS-16 compositions work correctly

[Mattias Engdegård]

3 juni 2023 kl. 13.32 skrev Robert Pluim <rpluim@gmail.com>:

>    Mattias> (By the way, the Makefile feeds the emoji-zwj.awk script itself into the script as data which is a bit silly.)
> 
> It is, Iʼll queue a patch for that on master (blocks.awk has the same
> problem), unless you beat me to it.

Go for it. So far the scripts haven't matched any line in themselves but this luck may not last forever.