bug#37036: [PATCH] Inconsistent ASCII and Latin char categories

[Eli Zaretskii <eliz@gnu.org>]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Thu, 15 Aug 2019 19:37:49 +0200
> Cc: 37036@debbugs.gnu.org
> 
> 15 aug. 2019 kl. 18.59 skrev Eli Zaretskii <eliz@gnu.org>:
> > 
> > What about "abcdef^A^B"?  Does M-f stop before the control characters?
> 
> Yes. Does forward-word use categories?

No.  Sorry, it was my faulty memory.  It uses char-script-table
instead.

> The more categories are arbitrary, the less useful they are.

I think they should become entirely useless, i.e. we should stop using
them.  We have the entire Unicode database with all the character
properties for quite some time now, and should favor using that
instead.  Categories are an old kludgey hack, which goes back to
pre-Unicode Emacs; it can never be anything but arbitrary, and we will
never be able to fix that anywhere near completely.

> Why would anyone use categories to discriminate characters if they do not have a sensible, useful and predictable structure?

I don't know why anyone should.  My recommendation is to just say no.

> Consider the function fill-polish-nobreak-p. It is clearly written with the assumption of a reasonable definition of the Latin category, and it doesn't work as expected because of that.

Can you tell the details of where this function doesn't work?  I'd
like to understand why fixing it needs to change the categories.

> It is perfectly clear that categories have been introduced in an ad-hoc way to solve problems as they arose, but that doesn't mean that no mistakes were made even for those narrow purposes.

I don't think we should fix those mistakes, because that's an
impossible goal.  We should instead gradually stop using categories
for anything serious, certainly for any new code.  We should use the
UCD properties and the various char-tables built upon that instead.