not quite understanding input methods

not quite understanding input methods

[Perry E. Metzger]

So I've built a small input method to test out my understanding, and 
I've hit a bit of a wall.

I would like to prefix my translations with a special character. I've 
mapped my caps lock key to send F19 (I don't use the caps lock key ever 
and using it as a compose key seems reasonable) and I've set Emacs to 
insert a special character for me when I hit that key, like so:

(global-set-key (kbd "<f19>")
                 (lambda (n)
                   (interactive "p")
                   (self-insert-command n ?⎄)))

(That special character happens to be the unicode "COMPOSE SYMBOL", 
which seemed intuitively appropriate.)

I've also created a small input method, and which has the following rules:

(quail-define-rules
  ("⎄gl" ?λ)
  ("⎄gL" ?Λ)
  ("⎄iA" ?∀)
  ("⎄iE" ?∃)
   ("xx" ?Π)
)

Now, if I hit "xx", Π is inserted as expected, but if I hit "<F19> g l", 
the buffer shows me "⎄gl" and not "λ" as I would expect.

My guess is that something in the belly of Quail is reading events and 
not the characters in the buffer, but as there's no documentation I'm 
not really clear on what is going on.


Perry

Re: not quite understanding input methods

[Stefan Monnier]

> My guess is that something in the belly of Quail is reading events and not
> the characters in the buffer, but as there's no documentation I'm not really
> clear on what is going on.

This is a hard-coded limit in the C code side of input-methods, as seen
in src/keyboard.c (line 3117):

    /* Pass this to the input method, if appropriate.  */
    if (FIXNUMP (c)
        && ! NILP (Vinput_method_function)
        /* Don't run the input method within a key sequence,
           after the first event of the key sequence.  */
        && NILP (prev_event)
        && ' ' <= XFIXNUM (c) && XFIXNUM (c) < 256 && XFIXNUM (c) != 127)
      {

It's already been requested to lift this restriction in the (fairly
distant) past.

IIRC there's no technical reason behind this limit, it reflects an
assumption that non-ASCII chars have presumably already gone through
some (other) form of input method.

I think it would be good to lift this restriction, tho we probably want
to do it progressively, e.g. by first introducing a var that controls it
(so you could lift the restriction without potentially impact everyone
else).


        Stefan

Re: not quite understanding input methods

[Perry E. Metzger]

On 8/30/21 13:37, Stefan Monnier wrote:
>> My guess is that something in the belly of Quail is reading events and not
>> the characters in the buffer, but as there's no documentation I'm not really
>> clear on what is going on.
> This is a hard-coded limit in the C code side of input-methods, as seen
> in src/keyboard.c (line 3117):

Seems to be at 3065 in master.

>
>      /* Pass this to the input method, if appropriate.  */
>      if (FIXNUMP (c)
>          && ! NILP (Vinput_method_function)
>          /* Don't run the input method within a key sequence,
>             after the first event of the key sequence.  */
>          && NILP (prev_event)
>          && ' ' <= XFIXNUM (c) && XFIXNUM (c) < 256 && XFIXNUM (c) != 127)
>        {
>
> It's already been requested to lift this restriction in the (fairly
> distant) past.
>
> IIRC there's no technical reason behind this limit, it reflects an
> assumption that non-ASCII chars have presumably already gone through
> some (other) form of input method.
>
> I think it would be good to lift this restriction, tho we probably want
> to do it progressively, e.g. by first introducing a var that controls it
> (so you could lift the restriction without potentially impact everyone
> else).
>
This sounds like a reasonable idea. If such a variable gets added to the 
master branch I will happily test it out. The one issue here is that I'm 
unsure that we would successfully learn about problems with the change 
if only a very few people were using it.

Perry

Re: not quite understanding input methods

[Stefan Monnier]

>> This is a hard-coded limit in the C code side of input-methods, as seen
>> in src/keyboard.c (line 3117):
> Seems to be at 3065 in master.

Looks like I wasn't on the right branch, sorry.

> This sounds like a reasonable idea. If such a variable gets added to the
> master branch I will happily test it out.

Can you cook up a patch for it?

> The one issue here is that I'm unsure that we would successfully learn
> about problems with the change if only a very few people were using it.

But we have to start somewhere.


        Stefan

Re: not quite understanding input methods

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 30 Aug 2021 14:00:18 -0400
> 
> > The one issue here is that I'm unsure that we would successfully learn
> > about problems with the change if only a very few people were using it.
> 
> But we have to start somewhere.

Please postpone this until after the emacs-28 branch is cut.

Re: not quite understanding input methods

[Perry E. Metzger]

On 8/30/21 14:22, Eli Zaretskii wrote:
>> The one issue here is that I'm unsure that we would successfully learn
>>> about problems with the change if only a very few people were using it.
>> But we have to start somewhere.
> Please postpone this until after the emacs-28 branch is cut.
>
If it isn't committed to master, it doesn't cause any harm.

Perry

Re: not quite understanding input methods

[Eli Zaretskii]

> Date: Mon, 30 Aug 2021 14:34:00 -0400
> Cc: emacs-devel@gnu.org
> From: "Perry E. Metzger" <perry@piermont.com>
> 
> On 8/30/21 14:22, Eli Zaretskii wrote:
> >> The one issue here is that I'm unsure that we would successfully learn
> >>> about problems with the change if only a very few people were using it.
> >> But we have to start somewhere.
> > Please postpone this until after the emacs-28 branch is cut.
> >
> If it isn't committed to master, it doesn't cause any harm.

That's true, but that's not what I understood Stefan was proposing.

Re: not quite understanding input methods

[Perry E. Metzger]

On 8/30/21 14:39, Eli Zaretskii wrote:
>>> Please postpone this until after the emacs-28 branch is cut.
>>>
>> If it isn't committed to master, it doesn't cause any harm.
> That's true, but that's not what I understood Stefan was proposing.
>
Perhaps I misunderstood as well, but in any case, I cooked up the patch 
and it didn't work, so it seems to have been worthwhile to try it even 
without committing it anywhere.

Perry

Re: not quite understanding input methods

[Perry E. Metzger]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]


On 8/30/21 14:00, Stefan Monnier wrote:
>
>> This sounds like a reasonable idea. If such a variable gets added to the
>> master branch I will happily test it out.
> Can you cook up a patch for it?

Cooked up (see attached), but it doesn't actually seem to work. My 
assumption is that `prev_event` is not set at the point where this 
executes; I have to confess I don't quite understand what will and won't 
set `prev_event`.

Perry


[-- Attachment #2: test.diff --]
[-- Type: text/plain, Size: 1201 bytes --]

diff --git a/src/keyboard.c b/src/keyboard.c
index 2e4c4e6aab..34322be6fe 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -3067,7 +3067,9 @@ read_char (int commandflag, Lisp_Object map,
       /* Don't run the input method within a key sequence,
 	 after the first event of the key sequence.  */
       && NILP (prev_event)
-      && ' ' <= XFIXNUM (c) && XFIXNUM (c) < 256 && XFIXNUM (c) != 127)
+      && ' ' <= XFIXNUM (c)
+      && (! NILP (Vinput_method_translate_unicode) || XFIXNUM (c) < 256)
+      && XFIXNUM (c) != 127)
     {
       Lisp_Object keys;
       ptrdiff_t key_count;
@@ -12325,6 +12327,11 @@ syms_of_keyboard (void)
 before running the input method.  It is nil if there was no message.  */);
   Vinput_method_previous_message = Qnil;
 
+  DEFVAR_LISP ("input-method-translate-unicode",
+	       Vinput_method_translate_unicode,
+	       doc: /* If non-nil, allows `read-char` to use charcodes over 255 in input methods. */);
+  Vinput_method_translate_unicode = Qnil;
+
   DEFVAR_LISP ("show-help-function", Vshow_help_function,
 	       doc: /* If non-nil, the function that implements the display of help.
 It's called with one argument, the help string to display.  */);

Re: not quite understanding input methods

[Perry E. Metzger]

On 8/30/21 14:37, Perry E. Metzger wrote:
>
> On 8/30/21 14:00, Stefan Monnier wrote:
>>
>>> This sounds like a reasonable idea. If such a variable gets added to 
>>> the
>>> master branch I will happily test it out.
>> Can you cook up a patch for it?
>
> Cooked up (see attached), but it doesn't actually seem to work. My 
> assumption is that `prev_event` is not set at the point where this 
> executes; I have to confess I don't quite understand what will and 
> won't set `prev_event`.
>
Having thought about this, is `read_char` actually going to be called 
under these circumstances? Remember that I've bound "<f19>" to a 
function that inserts a character into the buffer, and it is that 
character in the buffer that is in the quail rule. I may be confused 
about how all of this works, of course.

Perry

Re: not quite understanding input methods

[Eli Zaretskii]

> Date: Mon, 30 Aug 2021 15:00:21 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> 
> Having thought about this, is `read_char` actually going to be called 
> under these circumstances? Remember that I've bound "<f19>" to a 
> function that inserts a character into the buffer, and it is that 
> character in the buffer that is in the quail rule. I may be confused 
> about how all of this works, of course.

Before a key sequence bound to a command causes that command to be
invoked, it (the key sequence) must be read, and that's the job of
read_char.  Only after F19 is read, Emacs calls the command to which
you bound it, and that command inserts a character into the buffer.

Re: not quite understanding input methods

[Perry E. Metzger]

On 8/30/21 15:08, Eli Zaretskii wrote:
>> Date: Mon, 30 Aug 2021 15:00:21 -0400
>> From: "Perry E. Metzger" <perry@piermont.com>
>>
>> Having thought about this, is `read_char` actually going to be called
>> under these circumstances? Remember that I've bound "<f19>" to a
>> function that inserts a character into the buffer, and it is that
>> character in the buffer that is in the quail rule. I may be confused
>> about how all of this works, of course.
> Before a key sequence bound to a command causes that command to be
> invoked, it (the key sequence) must be read, and that's the job of
> read_char.  Only after F19 is read, Emacs calls the command to which
> you bound it, and that command inserts a character into the buffer.
>
Yes, but the point at which the character that's inserted into the 
buffer is present is after the Quail input method was invoked by 
read_char. Again, I'm finding the stack of things involved in the input 
method getting invoked rather difficult to follow, but it feels like 
Quail doesn't actually look backwards in the buffer and thus could care 
less that I inserted a particular character there. That said, I don't 
truly get how all this works yet, there's no documentation and a heap of 
twisty code involved.


Perry

Re: not quite understanding input methods

[Stefan Monnier]

> Yes, but the point at which the character that's inserted into the buffer is
> present is after the Quail input method was invoked by read_char. Again, I'm
> finding the stack of things involved in the input method getting invoked
> rather difficult to follow, but it feels like Quail doesn't actually look
> backwards in the buffer and thus could care less that I inserted
> a particular character there. That said, I don't truly get how all this
> works yet, there's no documentation and a heap of twisty code involved.

There might be multiple obstacles involved (e.g. multiple places that
assumes the chars are all <127), so it'll be important to test things
bit by bit.  E.g. check that with no patch the `input-method-function`
is not called at all when you press F19, then check that with
your patch `input-method-function` *is* called when you press F19.

Once that is done, the rest of the hacking should all be doable at the
Lisp level, but it may involve tracing through the quail code, indeed.


        Stefan

Re: not quite understanding input methods

[Perry E. Metzger]

On 8/30/21 15:19, Stefan Monnier wrote:
>> Yes, but the point at which the character that's inserted into the buffer is
>> present is after the Quail input method was invoked by read_char. Again, I'm
>> finding the stack of things involved in the input method getting invoked
>> rather difficult to follow, but it feels like Quail doesn't actually look
>> backwards in the buffer and thus could care less that I inserted
>> a particular character there. That said, I don't truly get how all this
>> works yet, there's no documentation and a heap of twisty code involved.
> There might be multiple obstacles involved (e.g. multiple places that
> assumes the chars are all <127), so it'll be important to test things
> bit by bit.  E.g. check that with no patch the `input-method-function`
> is not called at all when you press F19, then check that with
> your patch `input-method-function` *is* called when you press F19.
>
> Once that is done, the rest of the hacking should all be doable at the
> Lisp level, but it may involve tracing through the quail code, indeed.

The fact that `read_char` is a 1300 line function all on its own doesn't 
make it particularly easy to tease apart the logic. :( :( :(


Perry

Re: not quite understanding input methods

[Eli Zaretskii]

> Date: Mon, 30 Aug 2021 15:26:56 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> From: "Perry E. Metzger" <perry@piermont.com>
> 
> The fact that `read_char` is a 1300 line function all on its own doesn't 
> make it particularly easy to tease apart the logic. :( :( :(

Welcome to the Emacs input code.  You are in good company: when we
discuss how it works, we only pretend that we know what we are talking
about ;-)

Re: not quite understanding input methods

[Stefan Monnier]

> Welcome to the Emacs input code.  You are in good company: when we
> discuss how it works, we only pretend that we know what we are talking
> about ;-)

Indeed, I'm sure src/keyboard.c knows what I do much better than I know
what it does.


        Stefan

Re: not quite understanding input methods

[Eli Zaretskii]

> Date: Mon, 30 Aug 2021 15:13:25 -0400
> Cc: emacs-devel@gnu.org
> From: "Perry E. Metzger" <perry@piermont.com>
> 
> > Before a key sequence bound to a command causes that command to be
> > invoked, it (the key sequence) must be read, and that's the job of
> > read_char.  Only after F19 is read, Emacs calls the command to which
> > you bound it, and that command inserts a character into the buffer.
> >
> Yes, but the point at which the character that's inserted into the 
> buffer is present is after the Quail input method was invoked by 
> read_char.

Yes.  And the fact that the character is inserted means AFAIU that the
input method machinery didn't realize it was part of a sequence that
needs to be translated.

> Again, I'm finding the stack of things involved in the input 
> method getting invoked rather difficult to follow, but it feels like 
> Quail doesn't actually look backwards in the buffer and thus could care 
> less that I inserted a particular character there.

Correct.  Input method works on read_char level, it processes
characters before they get assembled into key sequences, thus before
the key sequences are recognized as being bound to commands.

Re: not quite understanding input methods

[André A. Gomes]

"Perry E. Metzger" <perry@piermont.com> writes:

> Yes, but the point at which the character that's inserted into the
> buffer is present is after the Quail input method was invoked by
> read_char. Again, I'm finding the stack of things involved in the
> input method getting invoked rather difficult to follow, but it feels
> like Quail doesn't actually look backwards in the buffer and thus
> could care less that I inserted a particular character there. That
> said, I don't truly get how all this works yet, there's no
> documentation and a heap of twisty code involved.

I feel your pain.  I tried to understand quail at some point, but it's
certainly not trivial.  My feeling was also that hardly anyone actually
understands it.


--
André A. Gomes
"Free Thought, Free World"

Re: not quite understanding input methods

[Yuri Khan]

On Tue, 31 Aug 2021 at 00:24, Perry E. Metzger <perry@piermont.com> wrote:

> I would like to prefix my translations with a special character. I've
> mapped my caps lock key to send F19 (I don't use the caps lock key ever
> and using it as a compose key seems reasonable) and I've set Emacs to
> insert a special character for me when I hit that key, like so:
>
> (global-set-key (kbd "<f19>")
>                  (lambda (n)
>                    (interactive "p")
>                    (self-insert-command n ?⎄)))
>
> (quail-define-rules
>   ("⎄gl" ?λ)
> )

Blunt question: Are you trying to solve this to get a better
understanding of the Emacs input subsystem (in which case please
ignore this), or do you just want something done? Because, if you want
Compose, it’s right there almost out-of-the-box in XKB.

Re: not quite understanding input methods

[Juri Linkov]

>> (quail-define-rules
>>   ("⎄gl" ?λ)
>> )
>
> Blunt question: Are you trying to solve this to get a better
> understanding of the Emacs input subsystem (in which case please
> ignore this), or do you just want something done? Because, if you want
> Compose, it’s right there almost out-of-the-box in XKB.

Emacs 28 has the input method "Compose" corresponding to XKB's Compose
in leim/quail/compose.el, but many useful key sequences in this
input method don't work because of the artificial limitation
to ASCII-only chars in read_char.

Re: not quite understanding input methods

[Yuri Khan]

On Wed, 1 Sept 2021 at 14:25, Juri Linkov <juri@linkov.net> wrote:

> > Blunt question: Are you trying to solve this to get a better
> > understanding of the Emacs input subsystem (in which case please
> > ignore this), or do you just want something done? Because, if you want
> > Compose, it’s right there almost out-of-the-box in XKB.
>
> Emacs 28 has the input method "Compose" corresponding to XKB's Compose
> in leim/quail/compose.el, but many useful key sequences in this
> input method don't work because of the artificial limitation
> to ASCII-only chars in read_char.

My point (which some will find offensive) is that maybe one doesn’t
need to implement input methods in Emacs.

If you have Compose in Emacs, it works in Emacs. If you have Compose
in XKB, it works across your whole desktop.

Re: not quite understanding input methods

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]

On Wed, Sep 01, 2021 at 02:43:37PM +0700, Yuri Khan wrote:

[...]

> My point (which some will find offensive) is that maybe one doesn’t
> need to implement input methods in Emacs.
> 
> If you have Compose in Emacs, it works in Emacs. If you have Compose
> in XKB, it works across your whole desktop.

As most extreme assertions, this is true and false at the same time.

In general, and for those input methods I use most, I do try to
get them functioning at the X level (Emacs, after all, isn't my
only text entry application: this might vary from person to person,
too). Personally I have a compose key I regularly add sequences to,
and two keyboard layouts (Latin, Greek) via group toggle.

That said, Emacs input machinery is (finitely, but still enormously)
more flexible and configurable than what X has to offer. For example,
I can only have alternate keyboard layouts I'm (somewhat) willing
to memorise. If I want to input, e.g. Cyrillic with mnemonic sequences
based on phonetics (because I can't be bothered to learn yet another
keyboard layout), cyrillic-translit is probably an input method
difficult to achieve with X.

On top of that, quail, albeit not easy, is probably still an order
of magnitude more hackable than the X11 methods.

In a nutshell: I enjoy having both and thank all the hackers of
the world for having both.

In a smaller nutshell: all generalisations suck :)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: not quite understanding input methods

[Joost Kremers]

On Wed, Sep 01 2021, Yuri Khan wrote:
> My point (which some will find offensive) is that maybe one doesn’t
> need to implement input methods in Emacs.
>
> If you have Compose in Emacs, it works in Emacs. If you have Compose
> in XKB, it works across your whole desktop.

Except that for some reason, XKB Compose doesn't work for me in Emacs...

Plus, hitting <AltGr " a> is still one key press more than <" a>. Of course I
could use an X keyboard layout that has dead keys, but that has its own
problems.

And, as tomas says, Emacs' input methods are more powerful and much easier to
adapt.

-- 
Joost Kremers
Life has its moments

Re: not quite understanding input methods

[Perry E. Metzger]

On 9/1/21 05:14, Joost Kremers wrote:
> On Wed, Sep 01 2021, Yuri Khan wrote:
>> My point (which some will find offensive) is that maybe one doesn’t
>> need to implement input methods in Emacs.
>>
>> If you have Compose in Emacs, it works in Emacs. If you have Compose
>> in XKB, it works across your whole desktop.
> Except that for some reason, XKB Compose doesn't work for me in Emacs...
>
> Plus, hitting <AltGr " a> is still one key press more than <" a>. Of course I
> could use an X keyboard layout that has dead keys, but that has its own
> problems.
>
> And, as tomas says, Emacs' input methods are more powerful and much easier to
> adapt.
>
And again, I find myself on a Mac, where I don't have XKB, and don't 
have AltGr, and it's overall a lot easier to experiment with this stuff 
in Emacs. Emacs also gives me the ability to have the input method give 
me help, which the native Mac input methods I might try (painfully) 
modifying do not.

One of the joys of open source is that people get to pick the mechanism 
for accomplishing something they find most congenial.


Perry

Re: not quite understanding input methods

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Wed, 1 Sep 2021 14:43:37 +0700
> Cc: Emacs developers <emacs-devel@gnu.org>,
>  "Perry E. Metzger" <perry@piermont.com>
> 
> My point (which some will find offensive) is that maybe one doesn’t
> need to implement input methods in Emacs.
> 
> If you have Compose in Emacs, it works in Emacs. If you have Compose
> in XKB, it works across your whole desktop.

IME, Emacs comes with many input methods that are either seldom
installed on garden-variety systems, or even aren't available.

Another problem that Emacs input methods solve is the lack of suitable
labels on the keyboard keys.

Of course, if your system has an input method that is also provided by
Emacs, and you can easily find the keys, it might be more convenient
to use the system-provided one.

Re: not quite understanding input methods

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

On Wed, Sep 1, 2021, 13:12 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Yuri Khan <yuri.v.khan@gmail.com>
> > Date: Wed, 1 Sep 2021 14:43:37 +0700
> > Cc: Emacs developers <emacs-devel@gnu.org>,
> >  "Perry E. Metzger" <perry@piermont.com>
> >
> > My point (which some will find offensive) is that maybe one doesn’t
> > need to implement input methods in Emacs.
> >
> > If you have Compose in Emacs, it works in Emacs. If you have Compose
> > in XKB, it works across your whole desktop.
>
> IME, Emacs comes with many input methods that are either seldom
> installed on garden-variety systems, or even aren't available.
>

Even when they are, they don't work the same way across operating systems,
whereas Emacs' portuguese-prefix does, perfectly predictably.

João

>

[-- Attachment #2: Type: text/html, Size: 1636 bytes --]

Re: not quite understanding input methods

[André A. Gomes]

João Távora <joaotavora@gmail.com> writes:

> Even when they are, they don't work the same way across operating
> systems, whereas Emacs' portuguese-prefix does, perfectly predictably.

Exactly!

This is a tangent topic, but let me share my experience.  I don't use
any portuguese IM for a long time.  To type in any latin-based language
I use the "us altgr-intl" IM and the compose key when needed (seldom).
Of course that such a solution makes sense when one is willing to
dismiss old muscle memory (of using the pt keyboard layout), and acquire
a new habit.

Now for the crucial point.  Do you go back forth the US and PT IM a lot?
In other words, how often do you type "cora;ão"?

Or do you use the "portuguese-prefix exclusively inside of Emacs?

I'm trying to gather some user-experience when it comes to my toy
project regarding input methods in Emacs.  Thanks.


--
André A. Gomes
"Free Thought, Free World"

Re: not quite understanding input methods

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

On Wed, Sep 1, 2021, 14:19 André A. Gomes <andremegafone@gmail.com> wrote:

> .
>
> Now for the crucial point.  Do you go back forth the US and PT IM a lot?
> In other words, how often do you type "cora;ão"?


Yes, I switch back and forth very often. Typing my name requires me to and
typing i.e. ~a for common lisp also requires me to switch back. I code with
the us layout, since most programming languages are friendly to it. Or did
you mean try to type cedilla directly? That, almost never. I still have
that muscle memory of the traditional pt layout tucked away, but I can
access it on demand and only if strictly needed.

Or do you use the "portuguese-prefix exclusively inside of Emacs?
>

No, I have alternatives outside Emacs. It took me a long while, but I have
a autohotkey script for windows, a karabiner script for Mac os, and a
thingy I forgot the name of for Linux. All painstakingly modeled after
Emacs' portuguese-prefix, all decently close to it, but sometimes very
subtly annoyingly off.

João

[-- Attachment #2: Type: text/html, Size: 1651 bytes --]

Re: not quite understanding input methods

[André A. Gomes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yuri Khan <yuri.v.khan@gmail.com>
>> Date: Wed, 1 Sep 2021 14:43:37 +0700
>> Cc: Emacs developers <emacs-devel@gnu.org>,
>>  "Perry E. Metzger" <perry@piermont.com>
>>
>> My point (which some will find offensive) is that maybe one doesn’t
>> need to implement input methods in Emacs.
>>
>> If you have Compose in Emacs, it works in Emacs. If you have Compose
>> in XKB, it works across your whole desktop.
>
> IME, Emacs comes with many input methods that are either seldom
> installed on garden-variety systems, or even aren't available.
>
> Another problem that Emacs input methods solve is the lack of suitable
> labels on the keyboard keys.
>
> Of course, if your system has an input method that is also provided by
> Emacs, and you can easily find the keys, it might be more convenient
> to use the system-provided one.

Emacs NEEDS to implement input methods.  Emacs was thought to be used
with the usual US (QWERTY) keyboard layout, and it needs to be used with
any latin based keyboard layout on the OS level.

Emacs lets you have a per-buffer IM, and this is extremely useful!  I
leverage that all the time.  Could the OS manage IMs across multiple
Emacs buffers?  Maybe.  But it doesn't matter, since it's Emacs'
responsibility anyway.

Speaking about OS level IM managers, let me draw your attention to IBus
(Intelligent Input Bus).  Unlike XIM, afaik, it let's you select an IM
per-app (analogous to the Emacs buffer-based selection).  This is
helpful since, as mentioned above, non latin-based IM methods should
never be activated externally otherwise they will make Emacs unusable.

What if I run Emacs on the Linux console (tty)?  I want to use the IM
that I'm used to.  Emacs is a portable program and that's good :)

Also, don't forget that free software aims to bring internationalized
software to the whole world.


--
André A. Gomes
"Free Thought, Free World"

Re: not quite understanding input methods

[Perry E. Metzger]

On 9/1/21 03:43, Yuri Khan wrote:
> My point (which some will find offensive) is that maybe one doesn’t
> need to implement input methods in Emacs.
>
> If you have Compose in Emacs, it works in Emacs. If you have Compose
> in XKB, it works across your whole desktop.
>
Not everyone is using X.

I do tend to agree that it would often be better to implement input 
methods at the OS level, but have you (for example) seen .keymap files 
on MacOS? They're not even officially documented any more (the page in 
the old documentation is marked so it won't be web indexed!) Apple 
prefers that people not use them, and there are significant limitations 
to the functionality. Sometimes doing this sort of thing in Emacs has 
advantages.

Perry

Re: not quite understanding input methods

[Maxim Nikulin]

On 01/09/2021 14:43, Yuri Khan wrote:
> 
> My point (which some will find offensive) is that maybe one doesn’t
> need to implement input methods in Emacs.

I like the idea to avoid input methods in Emacs and to switch keyboard 
layouts using XKB as in other applications when keycodes may be directly 
mapped to symbols. Unfortunately Emacs keymaps use current keyboard 
layout, not base one. E.g. when Russian layout active, key symbols for 
US locale should be checked. It is terribly inconvenient to switch 
layout before each shortcut. Control + Cyrillic S does not act as C-c 
despite it is the same key. That is why switching layouts outside of 
Emacs still leads to more inconvenience than input methods in Emacs.

It was discussed earlier: Keybindings in non-Latin layout
https://lists.gnu.org/archive/html/emacs-devel/2009-05/msg00031.html

Side note: some iBus input methods have quick way to disable input 
method, e.g. pressing and releasing Shift.

~15 years ago more Gtk applications suffered from a similar problem that 
Ctrl+C required namely Latin C, not Cyrillic S. Nowadays I can not name 
application where Ctrl is broken. For Alt+something the issue is that 
"something" is locale-dependent and may require switching to RU layout.

As to emacs, minor inconvenience: default layout switch Win+Space aka 
s-SPC aborts incremental search, so customization of key bindings is 
required. Other issues are significantly harder to overcome.

There is reverse-im package that can translate RU symbols back to US 
ones. Unfortunately it does not work with keystrokes like M-$ or C-c C-, 
(`org-insert-structure-template')

(quail-define-package "russian-computer" ; ...
(quail-define-rules ; ...
  ("," ?б) ; ...
  ("$" ?\;) ; ...
  ("?" ?,))

For shortcuts "," should be associated for particular physical key 
independently of active layout.

So it is necessary to ensure at the desktop level that keyboard layout 
is (almost) never switched for Emacs windows. Preferably the same 
shortcuts for layout switching should work for all applications, Emacs 
should not be an exception.

I have managed to achieve setup that *mostly* works. CapsLock and 
Shift+CapsLock select assigned layouts in Emacs and other applications, 
US layout is forced for Emacs windows. On the other hand I consider it 
too complex and hardly can recommend it since it is not really robust 
and has some drawbacks.

Re: not quite understanding input methods

[Filipp Gunbin]

On 01/09/2021 14:43 +0700, Yuri Khan wrote:

> On Wed, 1 Sept 2021 at 14:25, Juri Linkov <juri@linkov.net> wrote:
>
>> > Blunt question: Are you trying to solve this to get a better
>> > understanding of the Emacs input subsystem (in which case please
>> > ignore this), or do you just want something done? Because, if you want
>> > Compose, it’s right there almost out-of-the-box in XKB.
>>
>> Emacs 28 has the input method "Compose" corresponding to XKB's Compose
>> in leim/quail/compose.el, but many useful key sequences in this
>> input method don't work because of the artificial limitation
>> to ASCII-only chars in read_char.
>
> My point (which some will find offensive) is that maybe one doesn’t
> need to implement input methods in Emacs.
>
> If you have Compose in Emacs, it works in Emacs. If you have Compose
> in XKB, it works across your whole desktop.

I try to use Emacs input wherever possible, because it's cross-platform.
I used Emacs first on Windows/Cygwin, now on macOS, always in tty.  Of
course there's also remote usage via ssh, I don't do that much, but know
that it's possible if I need.  That is, xkb is not an option for me.

I even disable normal layout switching key combo in macOS, so that I
doesn't press it accidentally while in Emacs, and switch layouts by
clicking in dock (not very convenient so it forces me to learn how to
depend on OS less).

Filipp

Re: not quite understanding input methods

[Perry E. Metzger]

On 9/1/21 03:23, Juri Linkov wrote:

> Emacs 28 has the input method "Compose" corresponding to XKB's Compose
> in leim/quail/compose.el, but many useful key sequences in this
> input method don't work because of the artificial limitation
> to ASCII-only chars in read_char.
>
I've been looking at this a bit. If you simply lift the limitation in 
read_char everything downstream fails. It appears that the limitation is 
partially caused by the fact that quail.el expects to be able to create 
dense keymaps of size limited to 255, though that might not be the only 
problem.  Note that fixing this won't be overly difficult, but it would 
need to be fixed in anything that can supply a `input-method-function` 
that is hooked by read_char.

(BTW, as a pure aside: why aren't keymaps hash tables? Is that purely 
historical?)

(FYI, I fear that after the last few days of reading I may be the expert 
on how all of this works, which is terrifying because I feel completely 
lost in the code. It's a maze. It's pretty rare that I see a part of 
Emacs that I feel should be nuked from orbit, but the input subsystem 
feels that way; a rewrite would be required to make it actually 
comprehensible, but a rewrite would also break enormous amounts of stuff 
that people depend on.)


Perry

Re: not quite understanding input methods

[Perry E. Metzger]

On 9/1/21 01:44, Yuri Khan wrote:
>
> Blunt question: Are you trying to solve this to get a better
> understanding of the Emacs input subsystem (in which case please
> ignore this), or do you just want something done? Because, if you want
> Compose, it’s right there almost out-of-the-box in XKB.
>
I'm on a Mac, so no, it's not there out of the box in XKB for me. Also 
I'd prefer to experiment with things inside Emacs.

Perry

Partially answering my own question (was Re: not quite understanding input methods)

[Perry E. Metzger]

Answering my own question a bit, so that others might benefit from what 
I've learned.

On 8/30/21 13:24, Perry E. Metzger wrote:
> So I've built a small input method to test out my understanding, and 
> I've hit a bit of a wall.
>
> I would like to prefix my translations with a special character. I've 
> mapped my caps lock key to send F19 (I don't use the caps lock key 
> ever and using it as a compose key seems reasonable) and I've set 
> Emacs to insert a special character for me when I hit that key, like so:
>
> (global-set-key (kbd "<f19>")
>                 (lambda (n)
>                   (interactive "p")
>                   (self-insert-command n ?⎄)))
>
> (That special character happens to be the unicode "COMPOSE SYMBOL", 
> which seemed intuitively appropriate.)
>
> I've also created a small input method, and which has the following 
> rules:
>
> (quail-define-rules
>  ("⎄gl" ?λ)
>  ("⎄gL" ?Λ)
>  ("⎄iA" ?∀)
>  ("⎄iE" ?∃)
>   ("xx" ?Π)
> )
>
> Now, if I hit "xx", Π is inserted as expected, but if I hit "<F19> g 
> l", the buffer shows me "⎄gl" and not "λ" as I would expect.
>
> My guess is that something in the belly of Quail is reading events and 
> not the characters in the buffer, but as there's no documentation I'm 
> not really clear on what is going on.

For the benefit of all: indeed, Quail ends up hooking events, and not 
characters. It's fed those in read_char, which hands off all events 
optionally to a function stored in input-method-function, which is where 
Quail gets its hook into the whole thing. Quail functions by taking over 
reading the input events if it decides that an event might be the prefix 
of a thing it wants to translate, and afterwards returns the replacement 
event (that is, the translated character.)

The system is currently incapable (without a bunch of hacking that I 
haven't done yet and might not do) of handling characters to be 
translated that are above code 255 -- this limitation exists not only in 
read_char but also in quail.el itself. (If you just fix this in 
read_char you can easily end up getting emacs to hang, though I haven't 
yet tracked down exactly where in quail the hang happens.)

The right way for now to do something demented like what I'm trying 
above is first

1. Use a character like "¤" which is below 255 in Unicode to represent 
hitting your function key.

2. Insert a synthetic keystroke into the event stream by appending it to 
`unread-input-method-events`; you may do this approximately like so:

(global-set-key (kbd "<f19>")
                 (lambda (n)
                   (interactive "p")
                   (setq unread-input-method-events
                         (append unread-input-method-events '(?¤)))))


I still don't quite have everything right here; in particular, things 
like isearch don't work right. My suspicion is that I need to put this 
into an early input processing keymap and not the global keymap, but I 
thought I'd explain what I have figured out so far.


Perry