bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

As of 2016, the latest versions of Unicode (as of 2016) have now
formally deprecated and removed the vowel+oxia combinations from the
Greek extended range, leaving only the vowel+tonos from the basic Greek
and Coptic range.

As a result of this deprecation, the sixteen characters found in
greek.el (Quail package for inputting Greek) that use extended
codepoints should be replaced with those that use basic codepoints.  All
affected characters can be found here:
-->
https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels#Affected_characters

Although most Unicode Greek fonts display both versions identically, in
some cases, not using basic codepoints can break advanced features such
as alternate forms in Greek script.  To take an example, if some feature
is supposed to distinguish between regular and `curly' *beta* (β/ϐ) so
as to print the `curly' shape if the *beta* is found in medial position,
the substitution will succeed in βάρβαρος, but fail in λάβρος just
because of the extended codepoint of ά that is used by `greek.el`.

Best,

Robert

---------------------------------------------------------------
In GNU Emacs 25.3.1 (x86_64-pc-linux-gnu, GTK+ Version 3.22.12)
 of 2018-05-26 built on wsc_andre
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description:	Hyperbola GNU/Linux-libre

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 MODULES

Important settings:
  value of $LANG: fr_FR.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  global-linum-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Loading linum...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...
Quit
Making completion list...
report-emacs-bug-insert-to-mailer: Subject, To or body not found
delete-backward-char: Text is read-only
completing-read-default: Command attempted to use minibuffer while in minibuffer
Quit [3 times]

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message idna dired format-spec rfc822
mml mml-sec password-cache epg gnus-util mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util help-fns mail-prsvr mail-utils linum
cus-start cus-load edmacro kmacro reftex reftex-vars iso-transl server
finder-inf tex-site info package epg-config seq byte-opt gv bytecomp
byte-compile cl-extra help-mode easymenu cconv cl-loaddefs pcase cl-lib
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel x-win term/common-win x-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 186681 12333)
 (symbols 48 25396 0)
 (miscs 40 148 144)
 (strings 32 46038 6600)
 (string-bytes 1 1175040)
 (vectors 16 17760)
 (vector-slots 8 518374 6741)
 (floats 8 202 250)
 (intervals 56 294 17)
 (buffers 976 20))

-- 
Robert Alessi
CNRS UMR 8167 «Orient & Méditerranée» (Paris)

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Thu, 18 Jul 2019 11:03:10 +0200, Robert Alessi <alessi@robertalessi.net> said:

    Robert> As of 2016, the latest versions of Unicode (as of 2016) have now
    Robert> formally deprecated and removed the vowel+oxia combinations from the
    Robert> Greek extended range, leaving only the vowel+tonos from the basic Greek
    Robert> and Coptic range.

    Robert> As a result of this deprecation, the sixteen characters found in
    Robert> greek.el (Quail package for inputting Greek) that use extended
    Robert> codepoints should be replaced with those that use basic codepoints.  All
    Robert> affected characters can be found here:
    --> 
    Robert> https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels#Affected_characters

I took a look at greek.el, that shouldn't be difficult. What about
GREEK OXIA vs GREEK TONOS as standalone characters? Should we replace
the former with the latter?

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Thu, Jul 18, 2019 at 04:54:45PM +0200, Robert Pluim wrote:
> >>>>> On Thu, 18 Jul 2019 11:03:10 +0200, Robert Alessi <alessi@robertalessi.net> said:
> 
>     Robert> As of 2016, the latest versions of Unicode (as of 2016) have now
>     Robert> formally deprecated and removed the vowel+oxia combinations from the
>     Robert> Greek extended range, leaving only the vowel+tonos from the basic Greek
>     Robert> and Coptic range.
> 
>     Robert> As a result of this deprecation, the sixteen characters found in
>     Robert> greek.el (Quail package for inputting Greek) that use extended
>     Robert> codepoints should be replaced with those that use basic codepoints.  All
>     Robert> affected characters can be found here:
>     --> 
>     Robert> https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels#Affected_characters
> 
> I took a look at greek.el, that shouldn't be difficult. What about
> GREEK OXIA vs GREEK TONOS as standalone characters? Should we replace
> the former with the latter?

Very well spotted.  Actually, the latter is the right one—that is:
Greek Tonos, U+0384—while the former, Greek Oxia (U+1FFD) belongs to
the group that was deprecated.  So it is the other way round: keep the
former, and make the latter go.

Thank you,

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Thu, 18 Jul 2019 19:32:52 +0200, Robert Alessi <alessi@robertalessi.net> said:
    >> I took a look at greek.el, that shouldn't be difficult. What about
    >> GREEK OXIA vs GREEK TONOS as standalone characters? Should we replace
    >> the former with the latter?

    Robert> Very well spotted.  Actually, the latter is the right one—that is:
                                              ^^^^^^
    Robert> Greek Tonos, U+0384—while the former, Greek Oxia (U+1FFD) belongs to
    Robert> the group that was deprecated.  So it is the other way round: keep the
    Robert> former, and make the latter go.
            ^^^^^^

Now Iʼm confused :-)

Currently greek.el input methods insert \u1ffd, rather than
\u0384. Which one do you want?

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Thu, Jul 18, 2019 at 08:06:13PM +0200, Robert Pluim wrote:
> >>>>> On Thu, 18 Jul 2019 19:32:52 +0200, Robert Alessi <alessi@robertalessi.net> said:
>     Robert> Very well spotted.  Actually, the latter is the right one—that is:
>                                               ^^^^^^
>     Robert> Greek Tonos, U+0384—while the former, Greek Oxia (U+1FFD) belongs to
>     Robert> the group that was deprecated.  So it is the other way round: keep the
>     Robert> former, and make the latter go.
>             ^^^^^^
> 
> Now Iʼm confused :-)
> 
> Currently greek.el input methods insert \u1ffd, rather than
> \u0384. Which one do you want?

That is not entirely true, I daresay, for I did find a couple of
\u0384 in this file.  But I got confused at some point in my previous
email!  What I meant is: keep Greek Tonos (\u0384), and replace Greek
Oxia with Greek Tonos.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Thu, 18 Jul 2019 20:47:00 +0200, Robert Alessi <alessi@robertalessi.net> said:

    Robert> On Thu, Jul 18, 2019 at 08:06:13PM +0200, Robert Pluim wrote:
    >> >>>>> On Thu, 18 Jul 2019 19:32:52 +0200, Robert Alessi <alessi@robertalessi.net> said:
    Robert> Very well spotted.  Actually, the latter is the right one—that is:
    >> ^^^^^^
    Robert> Greek Tonos, U+0384—while the former, Greek Oxia (U+1FFD) belongs to
    Robert> the group that was deprecated.  So it is the other way round: keep the
    Robert> former, and make the latter go.
    >> ^^^^^^
    >> 
    >> Now Iʼm confused :-)
    >> 
    >> Currently greek.el input methods insert \u1ffd, rather than
    >> \u0384. Which one do you want?

    Robert> That is not entirely true, I daresay, for I did find a couple of
    Robert> \u0384 in this file.

Indeed, some of the methods already had a way to insert tonos.

    Robert> But I got confused at some point in my previous
    Robert> email!  What I meant is: keep Greek Tonos (\u0384), and replace Greek
    Robert> Oxia with Greek Tonos.

But those methods that have tonos are for 'modern' greek, whereas the
ones that have oxia are for classical greek, as Basil pointed out, so
perhaps thereʼs no need to change anything (unless thereʼs an edict
from the Unicode people that tonos must be used even when writing
classical greek).

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Thu, Jul 18, 2019 at 08:57:16PM +0200, Robert Pluim wrote:
> 
> But those methods that have tonos are for 'modern' greek, whereas the
> ones that have oxia are for classical greek, as Basil pointed out, so
> perhaps thereʼs no need to change anything (unless thereʼs an edict
> from the Unicode people that tonos must be used even when writing
> classical greek).

I was about to write that, as a classicist, if it were up to me I
wouldn't change anything myself.  So I would agree with Basil's
argument.  However, to take this one example, if one tries to use
Alexey Kryukov's beautiful Old Standard font with emacs and XeLaTeX or
LuaLaTeX, and activates the `ss06` feature which is supposed to
automatically make the distinction between initial and medial beta, he
will see that ὁ βάρβαρος typed with emacs succeeds while λάβρος fails,
just because of ά with oxia instead of ά with tonos.

Yesterday, I tried to get to the bottom of this: I cloned the source
files of Old Standard (https://github.com/akryukov/oldstand) and
edited the source file of the regular shape with FontForge.  I
included all of the seven letters with oxia that were missing in the
substitution rules and generated a new .otf file, but to no avail.
Actually, I discovered that even if one selects those letters,
FontForge automatically undoes this selection to select the
corresponding letters with tonos!  Here is the condition: as far as I
can tell, there is no way to get this kind of feature to work using
the letters with oxia. I myself consider this preposterous.

Then I came across digitalclassicist.org.  I modified my own greek.el
and got a Greek text perfectly typeset with all the required
substitutions.

In my opinion, this is a serious problem.  I will try to proceed
further on this line of inquiry.  An interesting question is: why
(unless I am mistaken) did FontForge deprecrate oxia in favor of
tonos?  I am asking just because I am experiencing this regression.
That said, I will take some time to go through
https://www.unicode.org/versions/ and report back if I find anything
of relevance.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Thu, Jul 18, 2019 at 08:57:16PM +0200, Robert Pluim wrote:
> Indeed, some of the methods already had a way to insert tonos.
> 
>     Robert> But I got confused at some point in my previous
>     Robert> email!  What I meant is: keep Greek Tonos (\u0384), and replace Greek
>     Robert> Oxia with Greek Tonos.
> 
> But those methods that have tonos are for 'modern' greek, whereas the
> ones that have oxia are for classical greek, as Basil pointed out, so
> perhaps thereʼs no need to change anything (unless thereʼs an edict
> from the Unicode people that tonos must be used even when writing
> classical greek).

Good point.  I will try to look for such an edict!

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> Date: Thu, 18 Jul 2019 22:32:03 +0200
> From: Robert Alessi <alessi@robertalessi.net>
> Cc: 36717@debbugs.gnu.org
> 
> On Thu, Jul 18, 2019 at 08:57:16PM +0200, Robert Pluim wrote:
> > Indeed, some of the methods already had a way to insert tonos.
> > 
> >     Robert> But I got confused at some point in my previous
> >     Robert> email!  What I meant is: keep Greek Tonos (\u0384), and replace Greek
> >     Robert> Oxia with Greek Tonos.
> > 
> > But those methods that have tonos are for 'modern' greek, whereas the
> > ones that have oxia are for classical greek, as Basil pointed out, so
> > perhaps thereʼs no need to change anything (unless thereʼs an edict
> > from the Unicode people that tonos must be used even when writing
> > classical greek).
> 
> Good point.  I will try to look for such an edict!

We could ask on the Unicode mailing list.  There are Unicode experts
there, and they are quite friendly.  If someone can come up with a
comprehensive description of our situation and the issues we are
trying to resolve, please write to unicode@unicode.org, and ask the
questions.

Thanks.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Fri, 19 Jul 2019 09:57:38 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> We could ask on the Unicode mailing list.  There are Unicode experts
    Eli> there, and they are quite friendly.  If someone can come up with a
    Eli> comprehensive description of our situation and the issues we are
    Eli> trying to resolve, please write to unicode@unicode.org, and ask the
    Eli> questions.

I think reading <https://www.unicode.org/faq/greek.html> helps
some. My understanding of the situation is that the basic Greek block
should be used, rather than the extended Greek block, for the LETTER +
OXIA/TONOS combinations (and the extended block versions all decompose
to characters in the basic block + a combining mark).

To me that implies that the Greek input methods should use GREEK TONOS
(\u384) consistently rather then GREEK OXIA (\u1ffd), but I couldn't
see any explicit mention of that, and at least in my font they're
visually distinct.

Thereʼs also <http://www.opoudjis.net/unicode/unicode.html>, but
thatʼs much longer, so I only read the bit about oxia vs tonos, and it
also has nothing to say on which to use when inserting only the
accenting character itself.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 10:27:35AM +0200, Robert Pluim wrote:
> >>>>> On Fri, 19 Jul 2019 09:57:38 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> We could ask on the Unicode mailing list.  There are Unicode experts
>     Eli> there, and they are quite friendly.  If someone can come up with a
>     Eli> comprehensive description of our situation and the issues we are
>     Eli> trying to resolve, please write to unicode@unicode.org, and ask the
>     Eli> questions.
> 
> I think reading <https://www.unicode.org/faq/greek.html> helps
> some. My understanding of the situation is that the basic Greek block
> should be used, rather than the extended Greek block, for the LETTER +
> OXIA/TONOS combinations (and the extended block versions all decompose
> to characters in the basic block + a combining mark).
> 
> To me that implies that the Greek input methods should use GREEK TONOS
> (\u384) consistently rather then GREEK OXIA (\u1ffd), but I couldn't
> see any explicit mention of that, and at least in my font they're
> visually distinct.
> 
> Thereʼs also <http://www.opoudjis.net/unicode/unicode.html>, but
> thatʼs much longer, so I only read the bit about oxia vs tonos, and it
> also has nothing to say on which to use when inserting only the
> accenting character itself.

It is quite an intricate business!  I have read this page yesterday
and also went through some of the references that are given.

What is very confusing in my opinion is that since the Greek
government allegedly decreed to mix up tonos and oxia, in a way, just
because tonos (as distinct from oxia) was the result of an earlier
reform, this may be interpreted as a deprecation of tonos and simple
vowels with tonos, and not the other way round...

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Robert Alessi <alessi@robertalessi.net>,  36717@debbugs.gnu.org
> Date: Fri, 19 Jul 2019 10:27:35 +0200
> 
> >>>>> On Fri, 19 Jul 2019 09:57:38 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> We could ask on the Unicode mailing list.  There are Unicode experts
>     Eli> there, and they are quite friendly.  If someone can come up with a
>     Eli> comprehensive description of our situation and the issues we are
>     Eli> trying to resolve, please write to unicode@unicode.org, and ask the
>     Eli> questions.
> 
> I think reading <https://www.unicode.org/faq/greek.html> helps
> some.

It didn't help me, FWIW.

> My understanding of the situation is that the basic Greek block
> should be used, rather than the extended Greek block, for the LETTER +
> OXIA/TONOS combinations (and the extended block versions all decompose
> to characters in the basic block + a combining mark).

That's unrelated to the issue at hand, AFAIU.  It is relevant to how
you set up your fonts, but not how our input methods should work.  The
point there is that by using the Greek Extended block, you request the
precomposed glyphs from the font, which may or may not be according to
what you want; whereas by using base characters followed by combining
marks you leave it to the rendering system to select the glyph.  But
we should always keep in mind that the shaping engine we use can (and
usually does) decide to use a precomposed glyph even when we type the
character in its decomposed form.  So this is not really relevant to
our issue here.

> To me that implies that the Greek input methods should use GREEK TONOS
> (\u384) consistently rather then GREEK OXIA (\u1ffd), but I couldn't
> see any explicit mention of that, and at least in my font they're
> visually distinct.

I don't actually understand this assertion, because we currently
provide both forms.  So I fail to see a problem in our input methods.
What did I miss?

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Fri, 19 Jul 2019 15:49:31 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Robert Alessi <alessi@robertalessi.net>,  36717@debbugs.gnu.org
    >> Date: Fri, 19 Jul 2019 10:27:35 +0200
    >> 
    >> >>>>> On Fri, 19 Jul 2019 09:57:38 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    Eli> We could ask on the Unicode mailing list.  There are Unicode experts
    Eli> there, and they are quite friendly.  If someone can come up with a
    Eli> comprehensive description of our situation and the issues we are
    Eli> trying to resolve, please write to unicode@unicode.org, and ask the
    Eli> questions.
    >> 
    >> I think reading <https://www.unicode.org/faq/greek.html> helps
    >> some.

    Eli> It didn't help me, FWIW.

    >> My understanding of the situation is that the basic Greek block
    >> should be used, rather than the extended Greek block, for the LETTER +
    >> OXIA/TONOS combinations (and the extended block versions all decompose
    >> to characters in the basic block + a combining mark).

Iʼm wrong about that bit in the parentheses, at least within Emacs. I
should have said

"decompose to characters in the basic block, which in turn decompose
to a base character + a combining mark", ie

GREEK SMALL LETTER IOTA WITH OXIA has decomposition
GREEK SMALL LETTER IOTA WITH TONOS which has decomposition
GREEK SMALL LETTER IOTA + COMBINING ACUTE ACCENT

    Eli> That's unrelated to the issue at hand, AFAIU.  It is relevant to how
    Eli> you set up your fonts, but not how our input methods should work.  The
    Eli> point there is that by using the Greek Extended block, you request the
    Eli> precomposed glyphs from the font, which may or may not be according to
    Eli> what you want; whereas by using base characters followed by combining
    Eli> marks you leave it to the rendering system to select the glyph.  But
    Eli> we should always keep in mind that the shaping engine we use can (and
    Eli> usually does) decide to use a precomposed glyph even when we type the
    Eli> character in its decomposed form.  So this is not really relevant to
    Eli> our issue here.

Now you've confused me (or Iʼve been unclear). The relevant characters
in the Greek basic block do not result in glyph composition:

C-u C-x = on ί =>

             position: 2893 of 3607 (80%), restriction: <411-3608>, column: 13
            character: ί (displayed as ί) (codepoint 943, #o1657, #x3af)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x03AF
               script: greek
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), g:Greek, j:Japanese
             to input: type "C-x 8 RET 3af" or "C-x 8 RET GREEK SMALL LETTER IOTA WITH TONOS"
          buffer code: #xCE #xAF
            file code: #xCE #xAF (encoded by coding system utf-8-emacs)
              display: by this font (glyph code)
    mac-ct:-*-Hack-normal-normal-normal-*-14-*-*-*-m-0-iso10646-1 (#x5DB)

Character code properties: customize what to show
  name: GREEK SMALL LETTER IOTA WITH TONOS
  old-name: GREEK SMALL LETTER IOTA TONOS
  general-category: Ll (Letter, Lowercase)
  decomposition: (953 769) ('ι' '́')

    >> To me that implies that the Greek input methods should use GREEK TONOS
    >> (\u384) consistently rather then GREEK OXIA (\u1ffd), but I couldn't
    >> see any explicit mention of that, and at least in my font they're
    >> visually distinct.

    Eli> I don't actually understand this assertion, because we currently
    Eli> provide both forms.  So I fail to see a problem in our input methods.
    Eli> What did I miss?

We donʼt provide both forms within the same input method. The question
is whether we should provide both, or provide only tonos, since letter
+ oxia were allegedly added by mistake, hence standalone oxia should
not be produced. Or we could change nothing.

Perhaps we should ask the unicode people :-)

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: 36717@debbugs.gnu.org,  alessi@robertalessi.net
> Date: Fri, 19 Jul 2019 15:27:45 +0200
> 
>     >> My understanding of the situation is that the basic Greek block
>     >> should be used, rather than the extended Greek block, for the LETTER +
>     >> OXIA/TONOS combinations (and the extended block versions all decompose
>     >> to characters in the basic block + a combining mark).
> 
> Iʼm wrong about that bit in the parentheses, at least within Emacs. I
> should have said
> 
> "decompose to characters in the basic block, which in turn decompose
> to a base character + a combining mark", ie
> 
> GREEK SMALL LETTER IOTA WITH OXIA has decomposition
> GREEK SMALL LETTER IOTA WITH TONOS which has decomposition
> GREEK SMALL LETTER IOTA + COMBINING ACUTE ACCENT

That's OK, but it's not really relevant, AFAIU.  We don't force users
to type decomposed characters, primarily because that's inconvenient,
and because most users wouldn't know what a given precomposed
character decomposes into.  A user who _wants_ to type decomposed
characters can (or at least should be able to) do that, including by
using our input methods.

>     Eli> That's unrelated to the issue at hand, AFAIU.  It is relevant to how
>     Eli> you set up your fonts, but not how our input methods should work.  The
>     Eli> point there is that by using the Greek Extended block, you request the
>     Eli> precomposed glyphs from the font, which may or may not be according to
>     Eli> what you want; whereas by using base characters followed by combining
>     Eli> marks you leave it to the rendering system to select the glyph.  But
>     Eli> we should always keep in mind that the shaping engine we use can (and
>     Eli> usually does) decide to use a precomposed glyph even when we type the
>     Eli> character in its decomposed form.  So this is not really relevant to
>     Eli> our issue here.
> 
> Now you've confused me (or Iʼve been unclear). The relevant characters
> in the Greek basic block do not result in glyph composition:
> 
> C-u C-x = on ί =>
> 
>              position: 2893 of 3607 (80%), restriction: <411-3608>, column: 13
>             character: ί (displayed as ί) (codepoint 943, #o1657, #x3af)
>               charset: unicode (Unicode (ISO10646))
> code point in charset: 0x03AF
>                script: greek
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), g:Greek, j:Japanese
>              to input: type "C-x 8 RET 3af" or "C-x 8 RET GREEK SMALL LETTER IOTA WITH TONOS"
>           buffer code: #xCE #xAF
>             file code: #xCE #xAF (encoded by coding system utf-8-emacs)
>               display: by this font (glyph code)
>     mac-ct:-*-Hack-normal-normal-normal-*-14-*-*-*-m-0-iso10646-1 (#x5DB)
> 
> Character code properties: customize what to show
>   name: GREEK SMALL LETTER IOTA WITH TONOS
>   old-name: GREEK SMALL LETTER IOTA TONOS
>   general-category: Ll (Letter, Lowercase)
>   decomposition: (953 769) ('ι' '́')

As hinted by the last line above, you should try the reverse: type
"C-x 8 RET 3b9" followed by "C-x 8 RET 301".  Then compare that with
what you get for "C-x 8 RET 3af".

IOW, I wasn't talking about "glyph decomposition" -- there's no such
thing AFAIK -- I was talking about the shaping engine displaying 2
characters as a single glyph using the font glyph for the precomposed
character.

>     >> To me that implies that the Greek input methods should use GREEK TONOS
>     >> (\u384) consistently rather then GREEK OXIA (\u1ffd), but I couldn't
>     >> see any explicit mention of that, and at least in my font they're
>     >> visually distinct.
> 
>     Eli> I don't actually understand this assertion, because we currently
>     Eli> provide both forms.  So I fail to see a problem in our input methods.
>     Eli> What did I miss?
> 
> We donʼt provide both forms within the same input method.

I don't see why this would be a problem.  We should provide the
variant expected by modern Greek in the input methods which target
modern Greek, and the variants for Classic Greek in input methods
which target that.  Bonus points for providing at least some of the
other variants in each type of the input methods.

> The question is whether we should provide both, or provide only
> tonos, since letter + oxia were allegedly added by mistake, hence
> standalone oxia should not be produced. Or we could change nothing.

I don't see that everyone agrees it was a mistake, and in any case
both variants are still in usage.  So I see no need to change
anything.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Fri, 19 Jul 2019 17:26:59 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> As hinted by the last line above, you should try the reverse: type
    Eli> "C-x 8 RET 3b9" followed by "C-x 8 RET 301".  Then compare that with
    Eli> what you get for "C-x 8 RET 3af".

OK, thatʼs clear now. No sane person would ever do it that way, though :-)

>> We donʼt provide both forms within the same input method.

    Eli> I don't see why this would be a problem.  We should provide the
    Eli> variant expected by modern Greek in the input methods which target
    Eli> modern Greek, and the variants for Classic Greek in input methods
    Eli> which target that.  Bonus points for providing at least some of the
    Eli> other variants in each type of the input methods.

That would be convenient, but should perhaps be a new, separate, input
method.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: 36717@debbugs.gnu.org,  alessi@robertalessi.net
> Date: Fri, 19 Jul 2019 16:41:07 +0200
> 
> >>>>> On Fri, 19 Jul 2019 17:26:59 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> As hinted by the last line above, you should try the reverse: type
>     Eli> "C-x 8 RET 3b9" followed by "C-x 8 RET 301".  Then compare that with
>     Eli> what you get for "C-x 8 RET 3af".
> 
> OK, thatʼs clear now. No sane person would ever do it that way, though :-)

Exactly my point, wrt the "advice" to use characters from the Basic
Greek block + diacriticals instead of the precomposed characters.

> >> We donʼt provide both forms within the same input method.
> 
>     Eli> I don't see why this would be a problem.  We should provide the
>     Eli> variant expected by modern Greek in the input methods which target
>     Eli> modern Greek, and the variants for Classic Greek in input methods
>     Eli> which target that.  Bonus points for providing at least some of the
>     Eli> other variants in each type of the input methods.
> 
> That would be convenient, but should perhaps be a new, separate, input
> method.

Could be.  However, the existing input methods already provide some of
the characters in both variants, and they certainly provide both oxia
and tonos as separate diacriticals.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 04:41:07PM +0200, Robert Pluim wrote:
>     Eli> I don't see why this would be a problem.  We should provide the
>     Eli> variant expected by modern Greek in the input methods which target
>     Eli> modern Greek, and the variants for Classic Greek in input methods
>     Eli> which target that.  Bonus points for providing at least some of the
>     Eli> other variants in each type of the input methods.
> 
> That would be convenient, but should perhaps be a new, separate, input
> method.

Hard to say.  I am thinking of some of my Greek colleagues who produce
in modern Greek editions and/or commentaries on classic texts.  I am
pretty sure that they use tonoi in every instance, but should they
wish to keep the distinction between tonos and oxeia, what would be
the more comfortable way to work with emacs?

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> Date: Fri, 19 Jul 2019 16:52:00 +0200
> From: Robert Alessi <alessi@robertalessi.net>
> Cc: Eli Zaretskii <eliz@gnu.org>, 36717@debbugs.gnu.org
> 
> On Fri, Jul 19, 2019 at 04:41:07PM +0200, Robert Pluim wrote:
> >     Eli> I don't see why this would be a problem.  We should provide the
> >     Eli> variant expected by modern Greek in the input methods which target
> >     Eli> modern Greek, and the variants for Classic Greek in input methods
> >     Eli> which target that.  Bonus points for providing at least some of the
> >     Eli> other variants in each type of the input methods.
> > 
> > That would be convenient, but should perhaps be a new, separate, input
> > method.
> 
> Hard to say.  I am thinking of some of my Greek colleagues who produce
> in modern Greek editions and/or commentaries on classic texts.  I am
> pretty sure that they use tonoi in every instance, but should they
> wish to keep the distinction between tonos and oxeia, what would be
> the more comfortable way to work with emacs?

They could switch between input methods back and forth.  Or we could
add support for the other variants to existing methods, to let users
use a single input method.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 06:00:55PM +0300, Eli Zaretskii wrote:
> > Hard to say.  I am thinking of some of my Greek colleagues who produce
> > in modern Greek editions and/or commentaries on classic texts.  I am
> > pretty sure that they use tonoi in every instance, but should they
> > wish to keep the distinction between tonos and oxeia, what would be
> > the more comfortable way to work with emacs?
> 
> They could switch between input methods back and forth.

This is what I do, and it's fine for me.

> Or we could add support for the other variants to existing methods,
> to let users use a single input method.

Surely, this would be more comfortable for some users, but how many?
I can't say.  But we were talking earlier about purists who could
argue that this addition would 'pollute' such or such input method.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 05:26:59PM +0300, Eli Zaretskii wrote:
> > We donʼt provide both forms within the same input method.
> 
> I don't see why this would be a problem.  We should provide the
> variant expected by modern Greek in the input methods which target
> modern Greek, and the variants for Classic Greek in input methods
> which target that.  Bonus points for providing at least some of the
> other variants in each type of the input methods.
> 
> > The question is whether we should provide both, or provide only
> > tonos, since letter + oxia were allegedly added by mistake, hence
> > standalone oxia should not be produced. Or we could change nothing.
> 
> I don't see that everyone agrees it was a mistake, and in any case
> both variants are still in usage.  So I see no need to change
> anything.

In my opinion, providing oxeia in the first place for classic Greek
was very right.  The distinction must be kept as some fonts render
tonoi and oxeiai very differently.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

Hi Eli,

On Fri, Jul 19, 2019 at 09:57:38AM +0300, Eli Zaretskii wrote:
> > > But those methods that have tonos are for 'modern' greek, whereas the
> > > ones that have oxia are for classical greek, as Basil pointed out, so
> > > perhaps thereʼs no need to change anything (unless thereʼs an edict
> > > from the Unicode people that tonos must be used even when writing
> > > classical greek).
> > 
> > Good point.  I will try to look for such an edict!
> 
> We could ask on the Unicode mailing list.  There are Unicode experts
> there, and they are quite friendly.  If someone can come up with a
> comprehensive description of our situation and the issues we are
> trying to resolve, please write to unicode@unicode.org, and ask the
> questions.

As of this writing, I am doing some research on this topic.  At least,
I went through almost all the documents that are listed here:
https://unicode.org/versions/ (in reverse chronological order), and
couldn't find any statement of Greek oxia being deprecated in favor of
tonos, contrary to what is claimed here:
https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels

Strictly speaking, tonos is not to be mistaken for oxia: that is for
sure, as tonos was introduced as a result of a reform to denote a
tone, namely a stress on some vowels, and not a pitch, namely a rising
and falling voice on accented vowels.  Confusion began in 1986 (from
what I learned) when the Greek government decreed that tonos shall be
the acute.  In addition to this, Unicode initially served both Greek
and Coptic on the same block, starting at U+0370.  At this time, only
monotonic Greek was served.  Later, an additional block called Greek
Extended was added at U+1F00, let alone further additions being
encoded in the Supplementary Multilingual Plane...

From what I could see, many Greek fonts originally reflected the
distinction between tonos and oxia.  But nowadays, they simply mix
them up.  This is precisely why I was not able to instruct FontForge
to take into account vowels with oxia in some of the substitution
rules of Old Standard which is a very fine Greek font (see
https://ctan.org/pkg/oldstandard): vowels with oxia were simply
missing from the Greek Extended Block, and what I did not see is a
rule that instructed to absorb vowels with oxia into vowels with tonos
as the glyphs are all strictly the same.

One question remains—and I wish to express my gratitude to all of you,
Robert, Basil and Eli: since assigning vowels with tonos and vowels
with oxia to the same code points is clearly unacceptable even if the
glyphs may be identical, is there a way to input tonos and vowels with
tonos with emacs?  I use greek-ibycus4, but if other input methods
can handle these letters, I would consider any change unnecessary.

Many thanks again to all of you!

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Fri, 19 Jul 2019 10:58:24 +0200, Robert Alessi <alessi@robertalessi.net> said:

    Robert> Robert, Basil and Eli: since assigning vowels with tonos and vowels
    Robert> with oxia to the same code points is clearly unacceptable even if the
    Robert> glyphs may be identical, is there a way to input tonos and vowels with
    Robert> tonos with emacs?  I use greek-ibycus4, but if other input methods
    Robert> can handle these letters, I would consider any change unnecessary.

1. You can locally derive an input method from greek-ibycus4, and
replace the entries for letter + oxia with letter with tonos (or we
can make that change in emacs)

2. Same, but we allow for entering both variants, so that eg

i' => iota with oxia
i; => iota with tonos (this is how eg greek-postfix does it)

3. You change input methods betweend greek-ibycus4 and greek postfix,
enter the character + tonos using i;, then switch back

4. Last resort: enter the required character directly using

C-x 8 RET 3af
or
C-x 8 RET GREEK SMALL LETTER IOTA WITH TONOS

Iʼd probably lean towards [2], but a purist might says that weʼre
polluting an input method intended for classical greek with a 'modern'
set of characters. OTOH Unicode seems to say that we should be using
the tonos variants anyway.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 11:26:57AM +0200, Robert Pluim wrote:
> >>>>> On Fri, 19 Jul 2019 10:58:24 +0200, Robert Alessi <alessi@robertalessi.net> said:
> 
>     Robert> Robert, Basil and Eli: since assigning vowels with tonos and vowels
>     Robert> with oxia to the same code points is clearly unacceptable even if the
>     Robert> glyphs may be identical, is there a way to input tonos and vowels with
>     Robert> tonos with emacs?  I use greek-ibycus4, but if other input methods
>     Robert> can handle these letters, I would consider any change unnecessary.
> 
> 1. You can locally derive an input method from greek-ibycus4, and
> replace the entries for letter + oxia with letter with tonos (or we
> can make that change in emacs)

This is what I did in the first place.  More precisely, I replaced
greek.el with a new greek.el which included the substitutions, but I
reverted to the original greek.el after realizing that the statements
of digitalclassicist.org are in fact questionable.

> 2. Same, but we allow for entering both variants, so that eg
> 
> i' => iota with oxia
> i; => iota with tonos (this is how eg greek-postfix does it)
 
Interesting option.

> 3. You change input methods betweend greek-ibycus4 and greek postfix,
> enter the character + tonos using i;, then switch back

This is what I would prefer (see below).

> 4. Last resort: enter the required character directly using
> 
> C-x 8 RET 3af
> or
> C-x 8 RET GREEK SMALL LETTER IOTA WITH TONOS

Provided one has not many characters to enter!

> Iʼd probably lean towards [2], but a purist might says that weʼre
> polluting an input method intended for classical greek with a 'modern'
> set of characters.

Yes [2] would be interesting, I agree, but the purist also has a point.

> OTOH Unicode seems to say that we should be using
> the tonos variants anyway.

Is that to say that the oxia variants are actually not recommended by
Unicode?  Did you find information on this form unicode.org?

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Fri, 19 Jul 2019 11:42:26 +0200, Robert Alessi <alessi@robertalessi.net> said:
    >> Iʼd probably lean towards [2], but a purist might says that weʼre
    >> polluting an input method intended for classical greek with a 'modern'
    >> set of characters.

    Robert> Yes [2] would be interesting, I agree, but the purist also has a point.

The purist can never prevent you from modifying your local copy of
greek.el :-)

    >> OTOH Unicode seems to say that we should be using
    >> the tonos variants anyway.

    Robert> Is that to say that the oxia variants are actually not recommended by
    Robert> Unicode?  Did you find information on this form unicode.org?

<https://www.unicode.org/faq/greek.html> says:

    Q: Which block of Greek characters should I use?

    A: The answer to that is that it depends what you are doing. But
    generally, the basic Greek block plus the use of the generic combining
    marks in the Combining Diacritical Marks block (U+0300..U+036F) is the
    best approach to polytonic Greek support.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 11:49:47AM +0200, Robert Pluim wrote:
>     Robert> Yes [2] would be interesting, I agree, but the purist also has a point.
> 
> The purist can never prevent you from modifying your local copy of
> greek.el :-)

He surely can't!  I have one question: do I have to modify the
greek.el that is in /usr/share/emacs/xx.y/lisp/leim/quail or is there
some place were to copy it under my own ~/.emacs.d?

>     Robert> Is that to say that the oxia variants are actually not recommended by
>     Robert> Unicode?  Did you find information on this form unicode.org?
> 
> <https://www.unicode.org/faq/greek.html> says:
> 
>     Q: Which block of Greek characters should I use?
> 
>     A: The answer to that is that it depends what you are doing. But
>     generally, the basic Greek block plus the use of the generic combining
>     marks in the Combining Diacritical Marks block (U+0300..U+036F) is the
>     best approach to polytonic Greek support.

Thank you.  So I was not totally wrong in the first place.  This
statement also makes clear that what is said on digitalclassicist.org
needs to be qualified.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Fri, 19 Jul 2019 12:03:04 +0200, Robert Alessi <alessi@robertalessi.net> said:

    Robert> On Fri, Jul 19, 2019 at 11:49:47AM +0200, Robert Pluim wrote:
    Robert> Yes [2] would be interesting, I agree, but the purist also has a point.
    >> 
    >> The purist can never prevent you from modifying your local copy of
    >> greek.el :-)

    Robert> He surely can't!  I have one question: do I have to modify the
    Robert> greek.el that is in /usr/share/emacs/xx.y/lisp/leim/quail or is there
    Robert> some place were to copy it under my own ~/.emacs.d?

Iʼd copy the relevant code into my .emacs (and perhaps even change the
name of the input method), just to be sure thereʼs no conflict or
accidental loading of the wrong file.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 01:49:00PM +0200, Robert Pluim wrote:
>     Robert> He surely can't!  I have one question: do I have to modify the
>     Robert> greek.el that is in /usr/share/emacs/xx.y/lisp/leim/quail or is there
>     Robert> some place were to copy it under my own ~/.emacs.d?
> 
> Iʼd copy the relevant code into my .emacs (and perhaps even change the
> name of the input method), just to be sure thereʼs no conflict or
> accidental loading of the wrong file.
> 
Thank you.

R.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  36717@debbugs.gnu.org
> Date: Fri, 19 Jul 2019 11:49:47 +0200
> 
> <https://www.unicode.org/faq/greek.html> says:
> 
>     Q: Which block of Greek characters should I use?
> 
>     A: The answer to that is that it depends what you are doing. But
>     generally, the basic Greek block plus the use of the generic combining
>     marks in the Combining Diacritical Marks block (U+0300..U+036F) is the
>     best approach to polytonic Greek support.

To me, the most important part of this is in the very first sentence.
It is not our business to second-guess "what the user is doing".  We
should just make sure they can produce both variants of the characters
involved, and let the user decide what they want/need.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

Bottom line, after reading this thread and related material on the
net, I must say I don't understand what might be wrong with our
current Greek input methods.  It sounds like we allow users to type
both characters with tonos and with oxia, and leave it to them to
decide which one(s) they need.  I see nothing wrong with that, because
only the user can know what they need.

Can someone please point out what am I missing?  Preferably with
concrete examples of characters and input methods used to type them.

TIA

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 04:00:33PM +0300, Eli Zaretskii wrote:
> Bottom line, after reading this thread and related material on the
> net, I must say I don't understand what might be wrong with our
> current Greek input methods.  It sounds like we allow users to type
> both characters with tonos and with oxia, and leave it to them to
> decide which one(s) they need.  I see nothing wrong with that, because
> only the user can know what they need.
> 
> Can someone please point out what am I missing?  Preferably with
> concrete examples of characters and input methods used to type them.
> 
> TIA

This is only my opinion, but since I am the one who started this
discussion, I must emphasize that the approach described above sounds
me right.  At first, being unable to get some substitution rules to
work in a Greek font using the oxia variants, and without knowing
tonos variants existed, I thought that emacs was using deprecated
variants after I came across the page of digitalclassicist.org on
``duplicated vowels''.  Further reading showed that this page leads to
some confusion.

As said in an earlier email, tonos and oxia refer to very different
systems, and only the latter should be used for classical Greek, even
if nowadays there is no visual distinction between the two.

If I understand correctly, greek-ibycus4 which I use is exclusively
designed for classical Greek input, but emacs also provides other
input methods should one wishes to input modern Greek with tonoi.

So as far as I can tell, there is nothing to change.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> Date: Fri, 19 Jul 2019 15:31:22 +0200
> From: Robert Alessi <alessi@robertalessi.net>
> Cc: rpluim@gmail.com, 36717@debbugs.gnu.org
> 
> If I understand correctly, greek-ibycus4 which I use is exclusively
> designed for classical Greek input, but emacs also provides other
> input methods should one wishes to input modern Greek with tonoi.
> 
> So as far as I can tell, there is nothing to change.

I agree.  I think we should close this issue.

Thanks.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 19 Jul 2019 15:31:22 +0200
>> From: Robert Alessi <alessi@robertalessi.net>
>> Cc: rpluim@gmail.com, 36717@debbugs.gnu.org
>> 
>> If I understand correctly, greek-ibycus4 which I use is exclusively
>> designed for classical Greek input, but emacs also provides other
>> input methods should one wishes to input modern Greek with tonoi.
>> 
>> So as far as I can tell, there is nothing to change.
>
> I agree.  I think we should close this issue.

Closed.

Best regards,
Stefan Kangas

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Fri, 19 Jul 2019 15:54:27 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Eli Zaretskii <eliz@gnu.org>,  36717@debbugs.gnu.org
    >> Date: Fri, 19 Jul 2019 11:49:47 +0200
    >> 
    >> <https://www.unicode.org/faq/greek.html> says:
    >> 
    >> Q: Which block of Greek characters should I use?
    >> 
    >> A: The answer to that is that it depends what you are doing. But
    >> generally, the basic Greek block plus the use of the generic combining
    >> marks in the Combining Diacritical Marks block (U+0300..U+036F) is the
    >> best approach to polytonic Greek support.

    Eli> To me, the most important part of this is in the very first sentence.
    Eli> It is not our business to second-guess "what the user is doing".  We
    Eli> should just make sure they can produce both variants of the characters
    Eli> involved, and let the user decide what they want/need.

That pleads for us doing nothing, since we have that today (just not
within the same input method).

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 03:29:53PM +0200, Robert Pluim wrote:
>     >> Q: Which block of Greek characters should I use?
>     >> 
>     >> A: The answer to that is that it depends what you are doing. But
>     >> generally, the basic Greek block plus the use of the generic combining
>     >> marks in the Combining Diacritical Marks block (U+0300..U+036F) is the
>     >> best approach to polytonic Greek support.
> 
>     Eli> To me, the most important part of this is in the very first sentence.
>     Eli> It is not our business to second-guess "what the user is doing".  We
>     Eli> should just make sure they can produce both variants of the characters
>     Eli> involved, and let the user decide what they want/need.
> 
> That pleads for us doing nothing, since we have that today (just not
> within the same input method).

Agreed.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> Date: Fri, 19 Jul 2019 11:42:26 +0200
> From: Robert Alessi <alessi@robertalessi.net>
> Cc: Eli Zaretskii <eliz@gnu.org>, 36717@debbugs.gnu.org
> 
> > OTOH Unicode seems to say that we should be using
> > the tonos variants anyway.
> 
> Is that to say that the oxia variants are actually not recommended by
> Unicode?

I don't think this matters for the issue at hand.  What variant is to
be used is entirely up to the user.  We should only be sure to allow
users to produce both variants.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> Date: Fri, 19 Jul 2019 10:58:24 +0200
> From: Robert Alessi <alessi@robertalessi.net>
> Cc: rpluim@gmail.com, 36717@debbugs.gnu.org
> 
> I went through almost all the documents that are listed here:
> https://unicode.org/versions/ (in reverse chronological order), and
> couldn't find any statement of Greek oxia being deprecated in favor of
> tonos, contrary to what is claimed here:
> https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels

So the basic claim that started this issue is no longer valid?  IOW,
this assertion:

  As of 2016, the latest versions of Unicode (as of 2016) have now
  formally deprecated and removed the vowel+oxia combinations from the
  Greek extended range, leaving only the vowel+tonos from the basic Greek
  and Coptic range.

is not really accurate?

> One question remains—and I wish to express my gratitude to all of you,
> Robert, Basil and Eli: since assigning vowels with tonos and vowels
> with oxia to the same code points is clearly unacceptable even if the
> glyphs may be identical, is there a way to input tonos and vowels with
> tonos with emacs?  I use greek-ibycus4, but if other input methods
> can handle these letters, I would consider any change unnecessary.

"C-x RET 8" would be the immediate answer, IIUC.

But I'm still studying the issue, so maybe I'm missing something.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 12:33:52PM +0300, Eli Zaretskii wrote:
> > I went through almost all the documents that are listed here:
> > https://unicode.org/versions/ (in reverse chronological order), and
> > couldn't find any statement of Greek oxia being deprecated in favor of
> > tonos, contrary to what is claimed here:
> > https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels
> 
> So the basic claim that started this issue is no longer valid?  IOW,
> this assertion:
> 
>   As of 2016, the latest versions of Unicode (as of 2016) have now
>   formally deprecated and removed the vowel+oxia combinations from the
>   Greek extended range, leaving only the vowel+tonos from the basic Greek
>   and Coptic range.
> 
> is not really accurate?

I would say so, to say the least, but I am still investigating.  What
is sure is that tonos originally does not encode the same as oxia.
The former encodes a stress, while the latter encodes a pitch.  This
is undisputable.  That said, the fact that the Greek government did
decree that tonos shall be the same as oxia (to be taken cautiously, I
am not a specialist of modern Greek) surely introduced a lot of
confusion.

For example, if one makes no distinction between the two, then it
becomes harder to analyse large corpuses with a computer.

> > One question remains—and I wish to express my gratitude to all of you,
> > Robert, Basil and Eli: since assigning vowels with tonos and vowels
> > with oxia to the same code points is clearly unacceptable even if the
> > glyphs may be identical, is there a way to input tonos and vowels with
> > tonos with emacs?  I use greek-ibycus4, but if other input methods
> > can handle these letters, I would consider any change unnecessary.
> 
> "C-x RET 8" would be the immediate answer, IIUC.

Well that is not very friendly...

> But I'm still studying the issue, so maybe I'm missing something.

Just the same on my side.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> Date: Fri, 19 Jul 2019 11:54:07 +0200
> From: Robert Alessi <alessi@robertalessi.net>
> Cc: rpluim@gmail.com, 36717@debbugs.gnu.org
> 
> >   As of 2016, the latest versions of Unicode (as of 2016) have now
> >   formally deprecated and removed the vowel+oxia combinations from the
> >   Greek extended range, leaving only the vowel+tonos from the basic Greek
> >   and Coptic range.
> > 
> > is not really accurate?
> 
> I would say so, to say the least, but I am still investigating.  What
> is sure is that tonos originally does not encode the same as oxia.
> The former encodes a stress, while the latter encodes a pitch.  This
> is undisputable.  That said, the fact that the Greek government did
> decree that tonos shall be the same as oxia (to be taken cautiously, I
> am not a specialist of modern Greek) surely introduced a lot of
> confusion.
> 
> For example, if one makes no distinction between the two, then it
> becomes harder to analyse large corpuses with a computer.

What do you mean by "makes no distinction"?  Those are different
codepoints, regardless of how they look on display.  So we definitely
_can_ distinguish between them.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Fri, Jul 19, 2019 at 03:55:51PM +0300, Eli Zaretskii wrote:
> > For example, if one makes no distinction between the two, then it
> > becomes harder to analyse large corpuses with a computer.
> 
> What do you mean by "makes no distinction"?  Those are different
> codepoints, regardless of how they look on display.  So we definitely
> _can_ distinguish between them.

I meant ``mixes up'': suppose you are dealing with large corpuses with
passages both in modern and ancient Greek and you only use one of the
two existing variants, either tonos or oxia: then if you wish to
analyse your texts with a computer, you are in trouble.  More
generally, any normalization leads to confusion.  This page will give
you some examples:
https://jktauber.com/articles/python-unicode-ancient-greek/

My point is that one should never use tonos variants in ancient Greek.
And so never use oxia variants in modern Greek.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Basil L. Contovounesios]

Robert Alessi <alessi@robertalessi.net> writes:

> On Thu, Jul 18, 2019 at 04:54:45PM +0200, Robert Pluim wrote:
>> >>>>> On Thu, 18 Jul 2019 11:03:10 +0200, Robert Alessi <alessi@robertalessi.net> said:
>> 
>>     Robert> As of 2016, the latest versions of Unicode (as of 2016) have now
>>     Robert> formally deprecated and removed the vowel+oxia combinations from the
>>     Robert> Greek extended range, leaving only the vowel+tonos from the basic Greek
>>     Robert> and Coptic range.
>> 
>>     Robert> As a result of this deprecation, the sixteen characters found in
>>     Robert> greek.el (Quail package for inputting Greek) that use extended
>>     Robert> codepoints should be replaced with those that use basic codepoints.  All
>>     Robert> affected characters can be found here:
>>     --> 
>>     Robert> https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels#Affected_characters
>> 
>> I took a look at greek.el, that shouldn't be difficult. What about
>> GREEK OXIA vs GREEK TONOS as standalone characters? Should we replace
>> the former with the latter?
>
> Very well spotted.  Actually, the latter is the right one—that is:
> Greek Tonos, U+0384—while the former, Greek Oxia (U+1FFD) belongs to
> the group that was deprecated.  So it is the other way round: keep the
> former, and make the latter go.

AFAICT the link you provided does not say anything about oxia the
standalone character.

I'm still interested in seeing some documentation on this deprecation
that is not from digitalclassicist.org.

Thanks,

-- 
Basil

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Thu, Jul 18, 2019 at 07:19:04PM +0100, Basil L. Contovounesios wrote:
> Robert Alessi <alessi@robertalessi.net> writes:
> > Very well spotted.  Actually, the latter is the right one—that is:
> > Greek Tonos, U+0384—while the former, Greek Oxia (U+1FFD) belongs to
> > the group that was deprecated.  So it is the other way round: keep the
> > former, and make the latter go.
> 
> AFAICT the link you provided does not say anything about oxia the
> standalone character.

You are perfectly right.  If it were up to me I would recommend to
keep all those characters that were considered as duplicates.  I quote
here from this page[^1]:

     A false distinction was introduced to Unicode between the oxia
     (acute) and tonos, resulting in wrongly duplicated code
     points. See Greek Unicode duplicated vowels for a full
     discussion.

So strictly speaking, oxia alone is not yet removed from Unicode.  But
as alpha, epsilon, eta, iota, omicron upsilon omega with oxia were
allegedly deprecated *and removed* from the Greek extended range, I
would say that oxia alone does not have much to live.

[^1]: https://wiki.digitalclassicist.org/Unicode_for_ancient_languages
> 
> I'm still interested in seeing some documentation on this deprecation
> that is not from digitalclassicist.org.

That, according to digitalclassicist.org, of course.  I will try to
see if I find anything of relevance on unicode.org and report back.

Robert

> Thanks,
> 
> -- 
> Basil

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Basil L. Contovounesios]

Robert Alessi <alessi@robertalessi.net> writes:

> On Thu, Jul 18, 2019 at 07:19:04PM +0100, Basil L. Contovounesios wrote:
>> Robert Alessi <alessi@robertalessi.net> writes:
>> > Very well spotted.  Actually, the latter is the right one—that is:
>> > Greek Tonos, U+0384—while the former, Greek Oxia (U+1FFD) belongs to
>> > the group that was deprecated.  So it is the other way round: keep the
>> > former, and make the latter go.
>> 
>> AFAICT the link you provided does not say anything about oxia the
>> standalone character.
>
> You are perfectly right.  If it were up to me I would recommend to
> keep all those characters that were considered as duplicates.  I quote
> here from this page[^1]:
>
>      A false distinction was introduced to Unicode between the oxia
>      (acute) and tonos, resulting in wrongly duplicated code
>      points. See Greek Unicode duplicated vowels for a full
>      discussion.
>
> So strictly speaking, oxia alone is not yet removed from Unicode.

[Are characters ever removed from Unicode?]

> But as alpha, epsilon, eta, iota, omicron upsilon omega with oxia were
> allegedly deprecated *and removed* from the Greek extended range, I
> would say that oxia alone does not have much to live.

Without being a Unicode expert, I doubt these vowel compositions or the
oxia itself are going anywhere.  The only question is whether it is safe
and/or encouraged to replace oxia with tonos in all contexts, regardless
of whether classical or modern, and how Quail might do this.  At least,
that's what I'm wondering.

> [^1]: https://wiki.digitalclassicist.org/Unicode_for_ancient_languages
>> 
>> I'm still interested in seeing some documentation on this deprecation
>> that is not from digitalclassicist.org.
>
> That, according to digitalclassicist.org, of course.  I will try to
> see if I find anything of relevance on unicode.org and report back.

Thanks for looking into this!

-- 
Basil

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Thu, 18 Jul 2019 11:03:10 +0200, Robert Alessi <alessi@robertalessi.net> said:
>
>     Robert> As of 2016, the latest versions of Unicode (as of 2016) have now
>     Robert> formally deprecated and removed the vowel+oxia combinations from the
>     Robert> Greek extended range, leaving only the vowel+tonos from the basic Greek
>     Robert> and Coptic range.
>
>     Robert> As a result of this deprecation, the sixteen characters found in
>     Robert> greek.el (Quail package for inputting Greek) that use extended
>     Robert> codepoints should be replaced with those that use basic codepoints.  All
>     Robert> affected characters can be found here:
>     --> 
>     Robert> https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels#Affected_characters
>
> I took a look at greek.el, that shouldn't be difficult.

Indeed, it's just a simple search+replace:


[-- Attachment #2: 0001-Replace-Greek-vowel-oxia-with-vowel-tonos-in-Quail.patch --]
[-- Type: text/x-diff, Size: 8570 bytes --]

From 1e7e52b25d70f827644e179f2c921adda31306fc Mon Sep 17 00:00:00 2001
From: "Basil L. Contovounesios" <contovob@tcd.ie>
Date: Thu, 18 Jul 2019 15:52:33 +0100
Subject: [PATCH] Replace Greek vowel+oxia with vowel+tonos in Quail

* lisp/leim/quail/greek.el (greek-mizuochi, greek-babel)
(greek-ibycus4):
* lisp/leim/quail/rfc1345.el (rfc1345): Replace vowel+oxia
characters from the Greek Extended block with their equivalent
vowel+tonos characters from the Greek and Coptic block (bug#36717).
---
 lisp/leim/quail/greek.el   | 82 +++++++++++++++++++-------------------
 lisp/leim/quail/rfc1345.el | 30 +++++++-------
 2 files changed, 56 insertions(+), 56 deletions(-)

diff --git a/lisp/leim/quail/greek.el b/lisp/leim/quail/greek.el
index 66a17a29f5..e5d3100390 100644
--- a/lisp/leim/quail/greek.el
+++ b/lisp/leim/quail/greek.el
@@ -264,7 +264,7 @@ "greek-mizuochi"
 
  ("i`" ?ἱ) ("iV" ?ἱ)
  ("i'" ?ἰ) ("iv" ?ἰ)
- ("i/" ?ί)
+ ("i/" ?ί)
  ("i`/" ?ἵ) ("iV/" ?ἵ) ("i/`" ?ἵ) ("i/V" ?ἵ)
  ("i'/" ?ἴ) ("iv/" ?ἴ) ("i/'" ?ἴ) ("i/v" ?ἴ)
  ("i?" ?ὶ)
@@ -276,7 +276,7 @@ "greek-mizuochi"
  ("i'^"  ?ἶ) ("i'\\" ?ἶ) ("iv^"  ?ἶ) ("iv\\" ?ἶ)
  ("i^'"  ?ἶ) ("i\\'" ?ἶ) ("i^v"  ?ἶ) ("i\\v" ?ἶ)
  ("i\"" ?ϊ)
- ("i/\"" ?ΐ) ("i\"/" ?ΐ)
+ ("i/\"" ?ΐ) ("i\"/" ?ΐ)
  ("i?\"" ?ῒ) ("i\"?" ?ῒ)
 
  ("^`"  ?῟) ("^V"  ?῟) ("\\`" ?῟) ("\\V" ?῟)
@@ -292,7 +292,7 @@ "greek-mizuochi"
 
  ("e`" ?ἑ) ("eV" ?ἑ)
  ("e'" ?ἐ) ("ev" ?ἐ)
- ("e/" ?έ)
+ ("e/" ?έ)
  ("e/`" ?ἕ) ("e/V" ?ἕ) ("e`/" ?ἕ) ("eV/" ?ἕ)
  ("e/'" ?ἔ) ("e/v" ?ἔ) ("e'/" ?ἔ) ("ev/" ?ἔ)
  ("e?" ?ὲ)
@@ -301,7 +301,7 @@ "greek-mizuochi"
 
  ("a`" ?ἁ) ("aV" ?ἁ)
  ("a'" ?ἀ) ("av" ?ἀ)
- ("a/" ?ά)
+ ("a/" ?ά)
  ("a/`" ?ἅ) ("a/V" ?ἅ) ("a`/" ?ἅ) ("aV/" ?ἅ)
  ("a/'" ?ἄ) ("a/v" ?ἄ) ("a'/" ?ἄ) ("av/" ?ἄ)
  ("a?" ?ὰ)
@@ -332,7 +332,7 @@ "greek-mizuochi"
 
  ("h`" ?ἡ) ("hV" ?ἡ)
  ("h'" ?ἠ) ("hv" ?ἠ)
- ("h/" ?ή)
+ ("h/" ?ή)
  ("h/`" ?ἥ) ("h/V" ?ἥ) ("h`/" ?ἥ) ("hV/" ?ἥ)
  ("h/'" ?ἤ) ("h/v" ?ἤ) ("h'/" ?ἤ) ("hv/" ?ἤ)
  ("h?" ?ὴ)
@@ -362,7 +362,7 @@ "greek-mizuochi"
 
  ("o`" ?ὁ) ("oV" ?ὁ)
  ("o'" ?ὀ) ("ov" ?ὀ)
- ("o/" ?ό)
+ ("o/" ?ό)
  ("o/`" ?ὅ) ("o/V" ?ὅ) ("o`/" ?ὅ) ("oV/" ?ὅ)
  ("o/'" ?ὄ) ("o/v" ?ὄ) ("o'/" ?ὄ) ("ov/" ?ὄ)
  ("o?" ?ὸ)
@@ -371,7 +371,7 @@ "greek-mizuochi"
 
  ("u`" ?ὑ) ("uV" ?ὑ)
  ("u'" ?ὐ) ("uv" ?ὐ)
- ("u/" ?ύ)
+ ("u/" ?ύ)
  ("u/`" ?ὕ) ("u/V" ?ὕ) ("u`/" ?ὕ) ("uV/" ?ὕ)
  ("u/'" ?ὔ) ("u/v" ?ὔ) ("u'/" ?ὔ) ("uv/" ?ὔ)
  ("u?" ?ὺ)
@@ -383,12 +383,12 @@ "greek-mizuochi"
  ("u^'"  ?ὖ) ("u^v"  ?ὖ) ("u\\'" ?ὖ) ("u\\v" ?ὖ)
  ("u'^"  ?ὖ) ("uv^"  ?ὖ) ("u'\\" ?ὖ) ("uv\\" ?ὖ)
  ("u\"" ?ϋ)
- ("u\"/" ?ΰ) ("u/\"" ?ΰ)
+ ("u\"/" ?ΰ) ("u/\"" ?ΰ)
  ("u\"?" ?ῢ) ("u?\"" ?ῢ)
 
  ("w`" ?ὡ) ("wV" ?ὡ)
  ("w'" ?ὠ) ("wv" ?ὠ)
- ("w/" ?ώ)
+ ("w/" ?ώ)
  ("w/`" ?ὥ) ("w/V" ?ὥ) ("w`/" ?ὥ) ("wV/" ?ὥ)
  ("w/'" ?ὤ) ("w/v" ?ὤ) ("w'/" ?ὤ) ("wv/" ?ὤ)
  ("w?" ?ὼ)
@@ -551,7 +551,7 @@ "greek-babel"
 
  ("<i" ?ἱ)
  (">i" ?ἰ)
- ("'i" ?ί)
+ ("'i" ?ί)
  ("<'i" ?ἵ)
  (">'i" ?ἴ)
  ("`i" ?ὶ)
@@ -561,12 +561,12 @@ "greek-babel"
  ("<~i"  ?ἷ)
  (">~i"  ?ἶ)
  ("\"i" ?ϊ)
- ("\"'i" ?ΐ)
+ ("\"'i" ?ΐ)
  ("\"`i" ?ῒ)
 
  ("<I" ?Ἱ)
  (">I" ?Ἰ)
- ("'I" ?Ί)
+ ("'I" ?Ί)
  ("<'I" ?Ἵ)
  (">'I" ?Ἴ)
  ("`I" ?Ὶ)
@@ -587,7 +587,7 @@ "greek-babel"
 
  ("<e" ?ἑ)
  (">e" ?ἐ)
- ("'e" ?έ)
+ ("'e" ?έ)
  ("<'e" ?ἕ)
  (">'e" ?ἔ)
  ("`e" ?ὲ)
@@ -596,7 +596,7 @@ "greek-babel"
 
  ("<E" ?Ἑ)
  (">E" ?Ἐ)
- ("'E" ?Έ)
+ ("'E" ?Έ)
  ("<'E" ?Ἕ)
  (">'E" ?Ἔ)
  ("`E" ?Ὲ)
@@ -605,7 +605,7 @@ "greek-babel"
 
  ("<a" ?ἁ)
  (">a" ?ἀ)
- ("'a" ?ά)
+ ("'a" ?ά)
  ("<'a" ?ἅ)
  (">'a" ?ἄ)
  ("`a" ?ὰ)
@@ -617,7 +617,7 @@ "greek-babel"
 
  ("<A" ?Ἁ)
  (">A" ?Ἀ)
- ("'A" ?Ά)
+ ("'A" ?Ά)
  ("<'A" ?Ἅ)
  (">'A" ?Ἄ)
  ("`A" ?Ὰ)
@@ -654,7 +654,7 @@ "greek-babel"
 
  ("<h" ?ἡ)
  (">h" ?ἠ)
- ("'h" ?ή)
+ ("'h" ?ή)
  ("<'h" ?ἥ)
  (">'h" ?ἤ)
  ("`h" ?ὴ)
@@ -666,7 +666,7 @@ "greek-babel"
 
  ("<H" ?Ἡ)
  (">H" ?Ἠ)
- ("'H" ?Ή)
+ ("'H" ?Ή)
  ("<'H" ?Ἥ)
  (">'H" ?Ἤ)
  ("`H" ?Ὴ)
@@ -700,7 +700,7 @@ "greek-babel"
 
  ("<o" ?ὁ)
  (">o" ?ὀ)
- ("'o" ?ό)
+ ("'o" ?ό)
  ("<'o" ?ὅ)
  (">'o" ?ὄ)
  ("`o" ?ὸ)
@@ -709,7 +709,7 @@ "greek-babel"
 
  ("<O" ?Ὁ)
  (">O" ?Ὀ)
- ("'O" ?Ό)
+ ("'O" ?Ό)
  ("<'O" ?Ὅ)
  (">'O" ?Ὄ)
  ("`O" ?Ὸ)
@@ -718,7 +718,7 @@ "greek-babel"
 
  ("<u"   ?ὑ)
  (">u"   ?ὐ)
- ("'u"   ?ύ)
+ ("'u"   ?ύ)
  ("<'u"  ?ὕ)
  (">'u"  ?ὔ)
  ("`u"   ?ὺ)
@@ -728,11 +728,11 @@ "greek-babel"
  ("<~u"  ?ὗ)
  (">~u"  ?ὖ)
  ("\"u"  ?ϋ)
- ("\"'u" ?ΰ)
+ ("\"'u" ?ΰ)
  ("`\"u" ?ῢ)
 
  ("<U"   ?Ὑ)
- ("'U"   ?Ύ)
+ ("'U"   ?Ύ)
  ("<'U"  ?Ὕ)
  ("`U"   ?Ὺ)
  ("<`U"  ?Ὓ)
@@ -741,7 +741,7 @@ "greek-babel"
 
  ("<w"  ?ὡ)
  (">w"  ?ὠ)
- ("'w"  ?ώ)
+ ("'w"  ?ώ)
  ("<'w" ?ὥ)
  (">'w" ?ὤ)
  ("`w"  ?ὼ)
@@ -753,7 +753,7 @@ "greek-babel"
 
  ("<W"  ?Ὡ)
  (">W"  ?Ὠ)
- ("'W"  ?Ώ)
+ ("'W"  ?Ώ)
  ("<'W" ?Ὥ)
  (">'W" ?Ὤ)
  ("`W"  ?Ὼ)
@@ -992,19 +992,19 @@ "greek-ibycus4"
  ("(=W" ?Ὧ)
 
  ("a`" ?ὰ)
- ("a'" ?ά)
+ ("a'" ?ά)
  ("e`" ?ὲ)
- ("e'" ?έ)
+ ("e'" ?έ)
  ("h`" ?ὴ)
- ("h'" ?ή)
+ ("h'" ?ή)
  ("i`" ?ὶ)
- ("i'" ?ί)
+ ("i'" ?ί)
  ("o`" ?ὸ)
- ("o'" ?ό)
+ ("o'" ?ό)
  ("u`" ?ὺ)
- ("u'" ?ύ)
+ ("u'" ?ύ)
  ("w`" ?ὼ)
- ("w'" ?ώ)
+ ("w'" ?ώ)
 
  ("a)|" ?ᾀ)
  ("a(|" ?ᾁ)
@@ -1067,7 +1067,7 @@ "greek-ibycus4"
  ("a=|" ?ᾷ)
 
  ("`A" ?Ὰ)
- ("'A" ?Ά)
+ ("'A" ?Ά)
  ("A|" ?ᾼ)
 
  (")" ?᾿) ; #x1fbf ; psili
@@ -1081,10 +1081,10 @@ "greek-ibycus4"
  ("h=|" ?ῇ)
 
  ("`E" ?Ὲ)
- ("'E" ?Έ)
+ ("'E" ?Έ)
 
  ("`H" ?Ὴ)
- ("'H" ?Ή)
+ ("'H" ?Ή)
  ("H|" ?ῌ)
 
  (")`" ?῍) ; #x1fcd
@@ -1092,19 +1092,19 @@ "greek-ibycus4"
  (")=" ?῏) ; #x1fcf
 
  ("i+`" ?ῒ)
- ("i+'" ?ΐ)
+ ("i+'" ?ΐ)
  ("i=" ?ῖ)
  ("i+=" ?ῗ)
 
  ("`I" ?Ὶ)
- ("'I" ?Ί)
+ ("'I" ?Ί)
 
  ("(`" ?῝) ; #x1fdd
  ("('" ?῞) ; #x1fde
  ("(=" ?῟) ; #x1fdf
 
  ("u+`" ?ῢ)
- ("u+'" ?ΰ)
+ ("u+'" ?ΰ)
 
  ("r)" ?ῤ)
  ("r(" ?ῥ)
@@ -1113,7 +1113,7 @@ "greek-ibycus4"
  ("u+=" ?ῧ)
 
  ("`U" ?Ὺ)
- ("'U" ?Ύ)
+ ("'U" ?Ύ)
 
  ("`R" ?Ῥ)
 
@@ -1128,10 +1128,10 @@ "greek-ibycus4"
  ("w=|" ?ῷ)
 
  ("`O" ?Ὸ)
- ("'O" ?Ό)
+ ("'O" ?Ό)
 
  ("`W" ?Ὼ)
- ("'W" ?Ώ)
+ ("'W" ?Ώ)
  ("W|" ?ῼ)
 
  ("'" ?´) ; #x1ffd ; oxia
diff --git a/lisp/leim/quail/rfc1345.el b/lisp/leim/quail/rfc1345.el
index da1a453a9c..c08fa398c0 100644
--- a/lisp/leim/quail/rfc1345.el
+++ b/lisp/leim/quail/rfc1345.el
@@ -35,7 +35,7 @@
  nil t nil nil nil nil nil nil nil nil t)
 
 (quail-define-rules
-;; There doesn't seem to be any point in including ASCII.
+ ;; There doesn't seem to be any point in including ASCII.
  ("&PA" ?\200)
  ("&HO" ?\201)
  ("&BH" ?\202)
@@ -928,19 +928,19 @@
  ("&W*," ?Ὠ)
  ("&W*;" ?Ὡ)
  ("&a*!" ?ὰ)
- ("&a*'" ?ά)
+ ("&a*'" ?ά)
  ("&e*!" ?ὲ)
- ("&e*'" ?έ)
+ ("&e*'" ?έ)
  ("&y*!" ?ὴ)
- ("&y*'" ?ή)
+ ("&y*'" ?ή)
  ("&i*!" ?ὶ)
- ("&i*'" ?ί)
+ ("&i*'" ?ί)
  ("&o*!" ?ὸ)
- ("&o*'" ?ό)
+ ("&o*'" ?ό)
  ("&u*!" ?ὺ)
- ("&u*'" ?ύ)
+ ("&u*'" ?ύ)
  ("&w*!" ?ὼ)
- ("&w*'" ?ώ)
+ ("&w*'" ?ώ)
  ("&a*(" ?ᾰ)
  ("&a*-" ?ᾱ)
  ("&a*j" ?ᾳ)
@@ -948,7 +948,7 @@
  ("&A*(" ?Ᾰ)
  ("&A*-" ?Ᾱ)
  ("&A*!" ?Ὰ)
- ("&A*'" ?Ά)
+ ("&A*'" ?Ά)
  ("&A*J" ?ᾼ)
  ("&)*" ?᾽)
  ("&J3" ?ι)
@@ -957,9 +957,9 @@
  ("&?:" ?῁)
  ("&y*j" ?ῃ)
  ("&y*?" ?ῆ)
- ("&E*'" ?Έ)
+ ("&E*'" ?Έ)
  ("&Y*!" ?Ὴ)
- ("&Y*'" ?Ή)
+ ("&Y*'" ?Ή)
  ("&Y*J" ?ῌ)
  ("&,!" ?῍)
  ("&,'" ?῎)
@@ -970,7 +970,7 @@
  ("&I*(" ?Ῐ)
  ("&I*-" ?Ῑ)
  ("&I*!" ?Ὶ)
- ("&I*'" ?Ί)
+ ("&I*'" ?Ί)
  ("&;!" ?῝)
  ("&;'" ?῞)
  ("&?;" ?῟)
@@ -982,7 +982,7 @@
  ("&U*(" ?Ῠ)
  ("&U*-" ?Ῡ)
  ("&U*!" ?Ὺ)
- ("&U*'" ?Ύ)
+ ("&U*'" ?Ύ)
  ("&R*;" ?Ῥ)
  ("&!:" ?῭)
  ("&:'" ?΅)
@@ -990,9 +990,9 @@
  ("&w*j" ?ῳ)
  ("&w*?" ?ῶ)
  ("&O*!" ?Ὸ)
- ("&O*'" ?Ό)
+ ("&O*'" ?Ό)
  ("&W*!" ?Ὼ)
- ("&W*'" ?Ώ)
+ ("&W*'" ?Ώ)
  ("&W*J" ?ῼ)
  ("&/*" ?´)
  ("&;;" ?῾)
-- 
2.20.1


[-- Attachment #3: Type: text/plain, Size: 395 bytes --]


> What about GREEK OXIA vs GREEK TONOS as standalone characters? Should
> we replace the former with the latter?

I'm not sure; see my other message.  AFAIK vowels composed with oxia
decompose to their tonos counterparts, but not so oxia itself.  I'm
still confused as to what Quail should do with these equivalences.
Should it always use the simplest possible composition?

Thanks,

-- 
Basil

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Thu, 18 Jul 2019 19:16:39 +0100, "Basil L. Contovounesios" <contovob@tcd.ie> said:
    >> What about GREEK OXIA vs GREEK TONOS as standalone characters? Should
    >> we replace the former with the latter?

    Basil> I'm not sure; see my other message.  AFAIK vowels composed with oxia
    Basil> decompose to their tonos counterparts, but not so oxia itself.  I'm
    Basil> still confused as to what Quail should do with these equivalences.
    Basil> Should it always use the simplest possible composition?

Iʼm not sure either. Neither are combining characters, so I doubt it
matters, but consistency probably calls for tonos rather than oxia
(although we can also arrange for both to be available).

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Thu, Jul 18, 2019 at 08:47:41PM +0200, Robert Pluim wrote:
> >>>>> On Thu, 18 Jul 2019 19:16:39 +0100, "Basil L. Contovounesios" <contovob@tcd.ie> said:
>     >> What about GREEK OXIA vs GREEK TONOS as standalone characters? Should
>     >> we replace the former with the latter?
> 
>     Basil> I'm not sure; see my other message.  AFAIK vowels composed with oxia
>     Basil> decompose to their tonos counterparts, but not so oxia itself.  I'm
>     Basil> still confused as to what Quail should do with these equivalences.
>     Basil> Should it always use the simplest possible composition?
> 
> Iʼm not sure either. Neither are combining characters, so I doubt it
> matters, but consistency probably calls for tonos rather than oxia
> (although we can also arrange for both to be available).

Surely, arranging for both to be available would get out of trouble
those who, like me, try to get substitution features to work as
expected.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

On Thu, Jul 18, 2019 at 07:16:39PM +0100, Basil L. Contovounesios wrote:
> >
> > I took a look at greek.el, that shouldn't be difficult.
> 
> Indeed, it's just a simple search+replace:
> 

Yes, but let us hold on until we have something solid to lean on.

Robert

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Pluim]

>>>>> On Thu, 18 Jul 2019 19:16:39 +0100, "Basil L. Contovounesios" <contovob@tcd.ie> said:

    Basil> diff --git a/lisp/leim/quail/rfc1345.el b/lisp/leim/quail/rfc1345.el

Note that rfc1345.el has entries for both vowel + oxia and vowel +
tonos variants, so we should probably not change it.

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Basil L. Contovounesios]

Robert Alessi <alessi@robertalessi.net> writes:

> As of 2016, the latest versions of Unicode (as of 2016) have now
> formally deprecated and removed the vowel+oxia combinations from the
> Greek extended range, leaving only the vowel+tonos from the basic Greek
> and Coptic range.

Where is the deprecation documented?  What do you mean by "removed"?
AFAIK all of the "deprecated" codepoints are still part of the latest
Unicode standard[1].

> As a result of this deprecation, the sixteen characters found in
> greek.el (Quail package for inputting Greek) that use extended
> codepoints should be replaced with those that use basic codepoints.

I'm not opposed to such a simple search+replace[2], but I'm no expert on
these matters (so please bear with me), and I wonder what effects, if
any, such a change may have.

AFAICT all occurrences of the "deprecated" codepoints in greek.el appear
in classical Greek input methods, not the modern Greek input methods
greek or greek-postfix.  Would users of the classical input methods ever
want to explicitly use the oxia, not tonos, variants?

What confuses me is that, AIUI, the "deprecated" codepoints should
decompose to their Greek and Coptic counterparts[3].  How does Quail
interplay with Unicode normalisation?

[1]: https://www.unicode.org/charts/PDF/U1F00.pdf
[2]: Indeed, I've seen people trip over this discrepancy, but I forgot
     to follow up on this: https://emacs.stackexchange.com/a/43927/15748
[3]: http://www.unicode.org/charts/normalization/

> All affected characters can be found here: -->
> https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels#Affected_characters
>
> Although most Unicode Greek fonts display both versions identically, in
> some cases, not using basic codepoints can break advanced features such
> as alternate forms in Greek script.  To take an example, if some feature
> is supposed to distinguish between regular and `curly' *beta* (β/ϐ) so
> as to print the `curly' shape if the *beta* is found in medial position,
> the substitution will succeed in βάρβαρος, but fail in λάβρος just
> because of the extended codepoint of ά that is used by `greek.el`.

How does the use of oxia instead of tonos on the alpha affect the
substitution of the beta?

Thanks,

-- 
Basil

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Robert Alessi]

Thank you for these valuable items of information, Basil.  I will do
some research and report back no later than tomorrow.

Robert

On Thu, Jul 18, 2019 at 07:16:34PM +0100, Basil L. Contovounesios wrote:
> Robert Alessi <alessi@robertalessi.net> writes:
> 
> > As of 2016, the latest versions of Unicode (as of 2016) have now
> > formally deprecated and removed the vowel+oxia combinations from the
> > Greek extended range, leaving only the vowel+tonos from the basic Greek
> > and Coptic range.
> 
> Where is the deprecation documented?  What do you mean by "removed"?
> AFAIK all of the "deprecated" codepoints are still part of the latest
> Unicode standard[1].
> 
> > As a result of this deprecation, the sixteen characters found in
> > greek.el (Quail package for inputting Greek) that use extended
> > codepoints should be replaced with those that use basic codepoints.
> 
> I'm not opposed to such a simple search+replace[2], but I'm no expert on
> these matters (so please bear with me), and I wonder what effects, if
> any, such a change may have.
> 
> AFAICT all occurrences of the "deprecated" codepoints in greek.el appear
> in classical Greek input methods, not the modern Greek input methods
> greek or greek-postfix.  Would users of the classical input methods ever
> want to explicitly use the oxia, not tonos, variants?
> 
> What confuses me is that, AIUI, the "deprecated" codepoints should
> decompose to their Greek and Coptic counterparts[3].  How does Quail
> interplay with Unicode normalisation?
> 
> [1]: https://www.unicode.org/charts/PDF/U1F00.pdf
> [2]: Indeed, I've seen people trip over this discrepancy, but I forgot
>      to follow up on this: https://emacs.stackexchange.com/a/43927/15748
> [3]: http://www.unicode.org/charts/normalization/
> 
> > All affected characters can be found here: -->
> > https://wiki.digitalclassicist.org/Greek_Unicode_duplicated_vowels#Affected_characters
> >
> > Although most Unicode Greek fonts display both versions identically, in
> > some cases, not using basic codepoints can break advanced features such
> > as alternate forms in Greek script.  To take an example, if some feature
> > is supposed to distinguish between regular and `curly' *beta* (β/ϐ) so
> > as to print the `curly' shape if the *beta* is found in medial position,
> > the substitution will succeed in βάρβαρος, but fail in λάβρος just
> > because of the extended codepoint of ά that is used by `greek.el`.
> 
> How does the use of oxia instead of tonos on the alpha affect the
> substitution of the beta?
> 
> Thanks,
> 
> -- 
> Basil

bug#36717: 25.3; greek.el: deprecated vowel+oxia combinations should be replaced with vowel+tonos counterparts

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Thu, 18 Jul 2019 19:16:34 +0100
> Cc: 36717@debbugs.gnu.org
> 
> What confuses me is that, AIUI, the "deprecated" codepoints should
> decompose to their Greek and Coptic counterparts[3].  How does Quail
> interplay with Unicode normalisation?

It doesn't.  Unicode normalization is implemented in core Emacs, and
it follows the Unicode rules on the one hand and the "decomposition"
property we read from UnicodeData.txt OTOH.