* Improve input handling documentation (was Re: Best way to intercept terminal escape sequences?) [not found] <20100904160740.988082D5@osgood.ece.cmu.edu> @ 2010-09-04 19:05 ` Ryan Johnson 2010-09-04 19:20 ` Improve input handling documentation Chong Yidong 2010-09-08 9:11 ` Stefan Monnier 0 siblings, 2 replies; 5+ messages in thread From: Ryan Johnson @ 2010-09-04 19:05 UTC (permalink / raw) To: emacs-devel On Sat, 04 Sep 2010 15:01:28 +0200 stepnem wrote: >> > It would be really nice to have, somewhere in the emacs docs, a diagram >> > showing what processing happens to keyboard input, starting from raw bytes and >> > UI events, and tracing them (or their translations) through coding systems, >> > input methods, command loop, various keymaps, etc. and showing where in that >> > process the different read-* functions intercept that data (and where the >> > various unread-*-events reinsert things). A similar diagram for reading and >> > writing files would probably also be useful. > Sounds great indeed! > > [...] >> > Unfortunately, even after spending so long on this problem I don't think I >> > know enough to generate that diagram... > But perhaps you could come up with some initial version that others > could continue improving upon? What format should the diagram take? I don't think I've ever seen ascii art, let alone an image, in the emacs docs... Meanwhile, here's what I have inferred, roughly, about input handling. Hopefully those familiar with emacs' guts can correct and improve it: <-------- keyboard input keyboard-coding-system <-------- unread-input-method-events [Input Method (optional, printable ascii chars only)] <-------- unread-post-input-method-events ??? <-------- non-keyboard (e.g. mouse) events <-------- unread-command-events (and unread-command-char) --------> read-char, read-event input-decode-map key-translation-map [local-]function-key-map --------> read-key assorted keymaps (global, local, minor mode, emulation, char/text properties, overriding) --------> read-key-sequence ??? read-number, read-string, etc. (???) (I have no idea where read-passwd fits in this hierarchy, nor how the command loop lines up with things). NOTE: the description of read-key and input-decode-map makes it sound like read-event comes before coding systems, but empirically this is not true. Regards, Ryan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Improve input handling documentation 2010-09-04 19:05 ` Improve input handling documentation (was Re: Best way to intercept terminal escape sequences?) Ryan Johnson @ 2010-09-04 19:20 ` Chong Yidong 2010-09-06 5:27 ` Ryan Johnson 2010-09-08 9:11 ` Stefan Monnier 1 sibling, 1 reply; 5+ messages in thread From: Chong Yidong @ 2010-09-04 19:20 UTC (permalink / raw) To: Ryan Johnson; +Cc: emacs-devel Ryan Johnson <ryanjohn@ece.cmu.edu> writes: > What format should the diagram take? I don't think I've ever seen > ascii art, let alone an image, in the emacs docs... Info can handle images nowadays, even though we don't currently use it in the Emacs manuals (see the "Box Diagrams" node in the Lisp manual for an example of ascii art). So if you want to take a stab at making a diagram, png is fine. It would be good to provide ascii art as a fallback, though. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Improve input handling documentation 2010-09-04 19:20 ` Improve input handling documentation Chong Yidong @ 2010-09-06 5:27 ` Ryan Johnson 0 siblings, 0 replies; 5+ messages in thread From: Ryan Johnson @ 2010-09-06 5:27 UTC (permalink / raw) To: Chong Yidong; +Cc: emacs-devel Chong Yidong writes: > Ryan Johnson writes: > >> What format should the diagram take? I don't think I've ever seen >> ascii art, let alone an image, in the emacs docs... > Info can handle images nowadays, even though we don't currently use it > in the Emacs manuals (see the "Box Diagrams" node in the Lisp manual for > an example of ascii art). So if you want to take a stab at making a > diagram, png is fine. It would be good to provide ascii art as a > fallback, though. OK, but let's get the information correct and complete first... Ryan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Improve input handling documentation 2010-09-04 19:05 ` Improve input handling documentation (was Re: Best way to intercept terminal escape sequences?) Ryan Johnson 2010-09-04 19:20 ` Improve input handling documentation Chong Yidong @ 2010-09-08 9:11 ` Stefan Monnier 1 sibling, 0 replies; 5+ messages in thread From: Stefan Monnier @ 2010-09-08 9:11 UTC (permalink / raw) To: Ryan Johnson; +Cc: emacs-devel > Meanwhile, here's what I have inferred, roughly, about input handling. > Hopefully those familiar with emacs' guts can correct and improve it: > <-------- keyboard input > keyboard-coding-system > <-------- unread-input-method-events > [Input Method (optional, printable ascii chars only)] > <-------- unread-post-input-method-events > ??? > <-------- non-keyboard (e.g. mouse) events > <-------- unread-command-events (and unread-command-char) --------> read-char, read-event > input-decode-map > key-translation-map > [local-]function-key-map --------> read-key > assorted keymaps (global, local, minor mode, emulation, char/text > properties, overriding) --------> read-key-sequence > ??? > read-number, read-string, etc. (???) That looks pretty good and would be a good addition. One error: key-translation-map comes after [local-]function-key-map. BTW: is there a good reason why C-x 8 is on key-translation-map rather than in input-decode-map, function-key-map, or even global-map? > NOTE: the description of read-key and input-decode-map makes it sound like > read-event comes before coding systems, but empirically this is not true. input-decode-map doesn't decode coding-systems but things like escape sequences. Stefan "who wants to obsolete key-translation-map" ^ permalink raw reply [flat|nested] 5+ messages in thread
[parent not found: <20100827142724.E1DD712F@hazard.ece.cmu.edu>]
* Re: Best way to intercept terminal escape sequences? [not found] <20100827142724.E1DD712F@hazard.ece.cmu.edu> @ 2010-08-27 14:44 ` Ryan Johnson 2010-08-27 15:40 ` Eli Zaretskii 0 siblings, 1 reply; 5+ messages in thread From: Ryan Johnson @ 2010-08-27 14:44 UTC (permalink / raw) To: emacs-devel On Fri, 27 Aug 2010 16:17:14 +0200, David Kastrup wrote: >> On Fri, 27 Aug 2010 12:36:52 +0200, David Kastrup wrote: >>> Ryan Johnson<ryanjohn@ece.cmu.edu> writes >>>> I tried setting the keyboard-coding-system to iso-latin-1, but no >>>> luck. >>> You should set it to raw-text, do your mouse code preprocessing, and >>> afterwards decode the remainder using the intended coding system. >> Like this? >> (defun xterm-mouse-pos-read () >> (let ((old-coding (keyboard-coding-system))) >> (set-keyboard-coding-system 'raw-text) >> (unwind-protect >> (cons (xterm-mouse-event-read) (xterm-mouse-event-read)) >> (set-keyboard-coding-system old-coding)))) > You can't go setting the keyboard reading system back and forth. It > operates into a buffer even before calling read-char. You have to put > it to raw and stick with it. Makes sense. That's why I was hoping there was a way intercept input before coding systems get their claws on it. Otherwise xt-mouse finds itself worrying about what coding system the user has requested, whether it changed recently, etc. If coding systems stacked (and if user-defined ones worked properly) it might be easier, but AFAIK neither is true. Is there really no other way to do this? Ryan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Best way to intercept terminal escape sequences? 2010-08-27 14:44 ` Best way to intercept terminal escape sequences? Ryan Johnson @ 2010-08-27 15:40 ` Eli Zaretskii 2010-08-27 23:54 ` Stefan Monnier 0 siblings, 1 reply; 5+ messages in thread From: Eli Zaretskii @ 2010-08-27 15:40 UTC (permalink / raw) To: Ryan Johnson; +Cc: emacs-devel > Date: Fri, 27 Aug 2010 16:44:12 +0200 > From: Ryan Johnson <ryanjohn@ece.cmu.edu> > > Is there really no other way to do this? You could use the :post-read-conversion attribute of a coding-system. That is, define a new coding-system that has this attribute specifying a function you will write. That function will first decode the mouse stuff, and then decode the rest by the terminal-coding-system set by the user. You can see an example of this in ctext-with-extensions. It is defined on mule-conf.el and its post-read-conversion function is defined on mule.el. Other than that, it's no surprise that this is not easy: we ask Emacs to read keyboard input that is encoded in two different encodings, which is not how keyboard input was designed. I think the only way that is easier and cleaner would be if Emacs could read the mouse input from a separate file descriptor. We could then set that file descriptor to use a different encoding. Of course, that would need low-level changes in Emacs, even if it is possible in xterm. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Best way to intercept terminal escape sequences? 2010-08-27 15:40 ` Eli Zaretskii @ 2010-08-27 23:54 ` Stefan Monnier 2010-08-28 7:54 ` Ryan Johnson 0 siblings, 1 reply; 5+ messages in thread From: Stefan Monnier @ 2010-08-27 23:54 UTC (permalink / raw) To: Eli Zaretskii; +Cc: Ryan Johnson, emacs-devel > I think the only way that is easier and cleaner would be if Emacs > could read the mouse input from a separate file descriptor. We could Note that under older Emacsen, read-event did not obey the keyboard-coding-system at all: it only applied to read-key-sequence. So maybe we should simply change read-event not to try and decoding keyboard input. Stefan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Best way to intercept terminal escape sequences? 2010-08-27 23:54 ` Stefan Monnier @ 2010-08-28 7:54 ` Ryan Johnson 2010-08-28 14:47 ` Stefan Monnier 0 siblings, 1 reply; 5+ messages in thread From: Ryan Johnson @ 2010-08-28 7:54 UTC (permalink / raw) To: Stefan Monnier; +Cc: Eli Zaretskii, emacs-devel On 8/28/2010 1:54 AM, Stefan Monnier wrote: >> I think the only way that is easier and cleaner would be if Emacs >> could read the mouse input from a separate file descriptor. We could > Note that under older Emacsen, read-event did not obey the > keyboard-coding-system at all: it only applied to read-key-sequence. > So maybe we should simply change read-event not to try and decoding > keyboard input. That might make sense... the caller could always apply the coding system manually before dumping things back in the unread-command-events queue. Who currently uses read-* that might be affected? xt-mouse.el would love it, mouse.el certainly won't care, and other xterm processing will be indifferent. BTW, I've been playing with read-key and it's perfect for making mouse.el and xt-mouse.el play nice together! I'm a tad unclear on the difference between read-key and read-key-sequence, though, other than the latter letting you supply a minibuffer prompt. Ryan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Best way to intercept terminal escape sequences? 2010-08-28 7:54 ` Ryan Johnson @ 2010-08-28 14:47 ` Stefan Monnier 2010-08-31 23:12 ` Ryan Johnson 0 siblings, 1 reply; 5+ messages in thread From: Stefan Monnier @ 2010-08-28 14:47 UTC (permalink / raw) To: Ryan Johnson; +Cc: Eli Zaretskii, emacs-devel > That might make sense... the caller could always apply the coding system > manually before dumping things back in the unread-command-events queue. Or coding-system decoding should be applied to events from unread-command-events. > Who currently uses read-* that might be affected? xt-mouse.el would love it, > mouse.el certainly won't care, and other xterm processing will > be indifferent. As mentioned, read-event did not do obey keyboard-coding-system in earlier Emacsen, so any affected package is more likely to be fixed than broken by making a change that reverts to this previous behavior. > BTW, I've been playing with read-key and it's perfect for making mouse.el > and xt-mouse.el play nice together! I'm a tad unclear on the difference > between read-key and read-key-sequence, though, other than the latter > letting you supply a minibuffer prompt. read-key only reads a single event. I.e. only C-x not C-x C-a. This event might be the result of processing several raw events (e.g. via keyboard-coding-system, input-decode-map, ...). Stefan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Best way to intercept terminal escape sequences? 2010-08-28 14:47 ` Stefan Monnier @ 2010-08-31 23:12 ` Ryan Johnson 2010-09-02 10:53 ` Stefan Monnier 0 siblings, 1 reply; 5+ messages in thread From: Ryan Johnson @ 2010-08-31 23:12 UTC (permalink / raw) To: Stefan Monnier; +Cc: Eli Zaretskii, emacs-devel On 8/28/2010 4:47 PM, Stefan Monnier wrote: >> Who currently uses read-* that might be affected? xt-mouse.el would love it, >> mouse.el certainly won't care, and other xterm processing will >> be indifferent. > As mentioned, read-event did not do obey keyboard-coding-system in > earlier Emacsen, so any affected package is more likely to be fixed than > broken by making a change that reverts to this previous behavior. Hmm... here's a twist: The elisp docs under keymaps -> translation keymaps explain that: If you have enabled keyboard character set decoding using `set-keyboard-coding-system', decoding is done after the translations listed above. See Terminal I/O Encoding. However, in future Emacs versions, character set decoding may be done at an earlier stage. However, the same info node admits that translation keymaps may want to read input (which does *not* escape I/O coding). So, suppose we do the following: (define-key input-decode-map "\M-[M"; CSI M '(keymap; pb (t keymap; px (t keymap; py (t . xterm-mouse-translate))))) In theory, the above matches any three characters following the start of the mouse escape sequence. Then inside xterm-mouse-translate (this-command-keys) comes close to being raw bytes. It now works great for any px I can throw at it, but still something goes wrong for py > #x7f. If I print (this-command-keys-vector) after a mouse click at (0 . 95), I get: [27 91 77 32 33 4194176] -- mouse-down -- and then emacs hangs waiting for more input; the next key I type ends up prefixed by \200. The lossage buffer shows ESC [ M SPC ! \300\200 ESC [ M # ! \300\200, but I don't know where the 'ESC [ M # ! \300' part disappeared to -- it doesn't get inserted into any buffer and yet xterm-mouse-translate never gets called, either. The docs seem out of date about where coding systems kick in... apparently it still tries to decode utf-8 somehow, even though I never call read-*. Even if I (set-keyboard-coding-system 'no-conversion), the bytes turn into all kinds of strange M- and C- versions of characters, which is no fun to disentangle. 'raw-text and 'binary give different but equally not-fun sets of weirdness. I'm beginning to think there's actually no way to get raw bytes from the terminal... Ideas? Ryan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Best way to intercept terminal escape sequences? 2010-08-31 23:12 ` Ryan Johnson @ 2010-09-02 10:53 ` Stefan Monnier 2010-09-02 12:33 ` Ryan Johnson 0 siblings, 1 reply; 5+ messages in thread From: Stefan Monnier @ 2010-09-02 10:53 UTC (permalink / raw) To: Kenichi Handa; +Cc: Eli Zaretskii, Ryan Johnson, emacs-devel > As mentioned, read-event did not do obey keyboard-coding-system in > earlier Emacsen, so any affected package is more likely to be fixed than > broken by making a change that reverts to this previous behavior. Handa, could you take a look at the feasibility of moving the decode_keyboard_code to a later stage such that read-event still returns raw bytes for ttys? There is a tension here, because raw events in GUIs are already decoded, whereas raw events in ttys are just bytes. You "fixed it" by decoding tty input in directly in tty_read_avail_input, so that read-event now always returns decoded input, but that in turns means that read-event doesn't return raw events any more. The decoding is desirable for read-key-sequence (and maybe also for read-char, tho I don't care much about this case since read-key is generally a better replacement) but not for read-event, since access to raw events is important for things like xt-mouse.el. > Hmm... here's a twist: The elisp docs under keymaps -> translation keymaps > explain that: > If you have enabled keyboard character set decoding using > `set-keyboard-coding-system', decoding is done after the translations > listed above. See Terminal I/O Encoding. However, in future Emacs > versions, character set decoding may be done at an earlier stage. This doc is out of date, indeed. Stefan ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Best way to intercept terminal escape sequences? 2010-09-02 10:53 ` Stefan Monnier @ 2010-09-02 12:33 ` Ryan Johnson 2010-09-04 13:01 ` Improve input handling documentation (was Re: Best way to intercept terminal escape sequences?) Štěpán Němec 0 siblings, 1 reply; 5+ messages in thread From: Ryan Johnson @ 2010-09-02 12:33 UTC (permalink / raw) To: Stefan Monnier; +Cc: Eli Zaretskii, emacs-devel, Kenichi Handa On 9/2/2010 12:53 PM, Stefan Monnier wrote: >> As mentioned, read-event did not do obey keyboard-coding-system in >> earlier Emacsen, so any affected package is more likely to be fixed than >> broken by making a change that reverts to this previous behavior. > Handa, could you take a look at the feasibility of moving the > decode_keyboard_code to a later stage such that read-event still returns > raw bytes for ttys? > > There is a tension here, because raw events in GUIs are already decoded, > whereas raw events in ttys are just bytes. You "fixed it" by decoding > tty input in directly in tty_read_avail_input, so that read-event now > always returns decoded input, but that in turns means that read-event > doesn't return raw events any more. The decoding is desirable for > read-key-sequence (and maybe also for read-char, tho I don't care much > about this case since read-key is generally a better replacement) but > not for read-event, since access to raw events is important for things > like xt-mouse.el. By the way, I noticed while working on something else [1] that read-key cannot actually replace all uses of read-event because the latter supports timeouts while the former does not. Would it be possible to add a timeout to read-key as a third (optional) parameter? I don't know whether read-key-delay would provide workarounds to some timeout uses, but it seems brittle. >> If you have enabled keyboard character set decoding using >> `set-keyboard-coding-system', decoding is done after the translations >> listed above. See Terminal I/O Encoding. However, in future Emacs >> versions, character set decoding may be done at an earlier stage. > This doc is out of date, indeed. It would be really nice to have, somewhere in the emacs docs, a diagram showing what processing happens to keyboard input, starting from raw bytes and UI events, and tracing them (or their translations) through coding systems, input methods, command loop, various keymaps, etc. and showing where in that process the different read-* functions intercept that data (and where the various unread-*-events reinsert things). A similar diagram for reading and writing files would probably also be useful. This would not only make it easier to figure out how to interface new code with emacs, it would probably expose gaps in the API which make existing code unnecessarily complex and certain features impossible (mouse.el and xt-mouse.el suffer from both of those latter problems). Unfortunately, even after spending so long on this problem I don't think I know enough to generate that diagram... Ryan [1] http://www.ece.cmu.edu/~ryanjohn/sticky-control.el ^ permalink raw reply [flat|nested] 5+ messages in thread
* Improve input handling documentation (was Re: Best way to intercept terminal escape sequences?) 2010-09-02 12:33 ` Ryan Johnson @ 2010-09-04 13:01 ` Štěpán Němec 0 siblings, 0 replies; 5+ messages in thread From: Štěpán Němec @ 2010-09-04 13:01 UTC (permalink / raw) To: Ryan Johnson; +Cc: Eli Zaretskii, Kenichi Handa, Stefan Monnier, emacs-devel Ryan Johnson <ryanjohn@ece.cmu.edu> writes: [...] > It would be really nice to have, somewhere in the emacs docs, a diagram > showing what processing happens to keyboard input, starting from raw bytes and > UI events, and tracing them (or their translations) through coding systems, > input methods, command loop, various keymaps, etc. and showing where in that > process the different read-* functions intercept that data (and where the > various unread-*-events reinsert things). A similar diagram for reading and > writing files would probably also be useful. Sounds great indeed! [...] > Unfortunately, even after spending so long on this problem I don't think I > know enough to generate that diagram... But perhaps you could come up with some initial version that others could continue improving upon? Štěpán ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2010-09-08 9:11 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- [not found] <20100904160740.988082D5@osgood.ece.cmu.edu> 2010-09-04 19:05 ` Improve input handling documentation (was Re: Best way to intercept terminal escape sequences?) Ryan Johnson 2010-09-04 19:20 ` Improve input handling documentation Chong Yidong 2010-09-06 5:27 ` Ryan Johnson 2010-09-08 9:11 ` Stefan Monnier [not found] <20100827142724.E1DD712F@hazard.ece.cmu.edu> 2010-08-27 14:44 ` Best way to intercept terminal escape sequences? Ryan Johnson 2010-08-27 15:40 ` Eli Zaretskii 2010-08-27 23:54 ` Stefan Monnier 2010-08-28 7:54 ` Ryan Johnson 2010-08-28 14:47 ` Stefan Monnier 2010-08-31 23:12 ` Ryan Johnson 2010-09-02 10:53 ` Stefan Monnier 2010-09-02 12:33 ` Ryan Johnson 2010-09-04 13:01 ` Improve input handling documentation (was Re: Best way to intercept terminal escape sequences?) Štěpán Němec
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.