Comint: handle raw tab

Comint: handle raw tab

[Fabian Ezequiel Gallina]

[-- Attachment #1: Type: text/plain, Size: 2172 bytes --]

Hello Emacs Devs,

I'm trying to send a raw tab to a comint process and handle it like it's
supposed.

The particular use case is the ipython shell: When you have typed the first
part of a python variable and hit TAB the name gets completed or a list of
possible completions is shown.

I tried several things in order to send the TAB to the comint process and
get the output to be handled but with no luck.


First case scenario, unique completion:

Consider the only possible completion being "True".

In [1]: Tr

Then hit C-q TAB so a TAB gets inserted after it and evaled:
(comint-send-input t)

In a normal ipython shell the result of this causes the input to be expanded
to "True" which is the unique completion. On the comint buffer this causes
the input to remain frozen. Internally, the input *does* get updated since
when I hit RET after evaling the code above the out shows "True" but I
didn't find a way to update the current input accordingly. Is there any way
to achieve that?


Second scenario, multiple completions available:

Consider now I have typed just T:

In [1]: T

Then hit C-q TAB so a TAB gets inserted after it and evaled:
(comint-send-input t)

Now interesting things happens, since ipython outputs the list of possible
completions I can get them with comint-output-filter-functions, the thing is
the buffer now looks like this:

In [1]: T
TabError       True       TypeError

And the only way I found to show the prompt again without sending "T" to the
process was sending a BREAK signal because comint-delete-input does not work
in that instance. Is there a better way to handle that?

I noticed that running ipython with M-x ansi-term works flawlessly so I
started looking into its code for some ideas.

Would it be wrong to use term-mode instead of comint for an inferior-python
shell? Does it have any drawbacks?

The only thing I can think of is having to rewrite the shell interactions
(pydoc, pdbtrack, etc) I already have in python.el. However all the inferior
shells implementations I know use comint so that makes me feel unsure about
it.



Regards,
Fabián E. Gallina

[-- Attachment #2: Type: text/html, Size: 2336 bytes --]

Re: Comint: handle raw tab

[Stefan Monnier]

> First case scenario, unique completion:
> Consider the only possible completion being "True".
> In [1]: Tr
> Then hit C-q TAB so a TAB gets inserted after it and evaled:
> (comint-send-input t)
> In a normal ipython shell the result of this causes the input to be expanded
> to "True" which is the unique completion. On the comint buffer this causes
> the input to remain frozen. Internally, the input *does* get updated since
> when I hit RET after evaling the code above the out shows "True" but I
> didn't find a way to update the current input accordingly. Is there any way
> to achieve that?

You're going to have to redirect the process's output to read the
shell's output and then use it to fill the user's current input.

> Second scenario, multiple completions available:
> Consider now I have typed just T:
> In [1]: T
> Then hit C-q TAB so a TAB gets inserted after it and evaled:
> (comint-send-input t)
> Now interesting things happens, since ipython outputs the list of possible
> completions I can get them with comint-output-filter-functions, the thing is
> the buffer now looks like this:
> In [1]: T
> TabError       True       TypeError

> And the only way I found to show the prompt again without sending "T" to the
> process was sending a BREAK signal because comint-delete-input does not work
> in that instance. Is there a better way to handle that?

The better way (IMNSHO) is to catch the process output so it doesn't get
inserted in the buffer, build a completion table from it, and then
call the normal in-buffer completion code with it so it gets displayed
in *Completions*.

> The only thing I can think of is having to rewrite the shell
> interactions (pydoc, pdbtrack, etc) I already have in python.el.
> However all the inferior shells implementations I know use comint so
> that makes me feel unsure about it.

I don't know if someone ever tried to use a term rather than a comint
inferior process, but I'd be interested to hear your experience with it.
I suspect it's going to be harder to interface it with compile.el.


        Stefan

Re: Comint: handle raw tab

[Štěpán Němec]

On Tue, 13 Sep 2011 14:43:10 +0200
Stefan Monnier wrote:

>> First case scenario, unique completion:
>> Consider the only possible completion being "True".
>> In [1]: Tr
>> Then hit C-q TAB so a TAB gets inserted after it and evaled:
>> (comint-send-input t)
>> In a normal ipython shell the result of this causes the input to be expanded
>> to "True" which is the unique completion. On the comint buffer this causes
>> the input to remain frozen. Internally, the input *does* get updated since
>> when I hit RET after evaling the code above the out shows "True" but I
>> didn't find a way to update the current input accordingly. Is there any way
>> to achieve that?
>
> You're going to have to redirect the process's output to read the
> shell's output and then use it to fill the user's current input.
>
>> Second scenario, multiple completions available:
>> Consider now I have typed just T:
>> In [1]: T
>> Then hit C-q TAB so a TAB gets inserted after it and evaled:
>> (comint-send-input t)
>> Now interesting things happens, since ipython outputs the list of possible
>> completions I can get them with comint-output-filter-functions, the thing is
>> the buffer now looks like this:
>> In [1]: T
>> TabError       True       TypeError
>
>> And the only way I found to show the prompt again without sending "T" to the
>> process was sending a BREAK signal because comint-delete-input does not work
>> in that instance. Is there a better way to handle that?
>
> The better way (IMNSHO) is to catch the process output so it doesn't get
> inserted in the buffer, build a completion table from it, and then
> call the normal in-buffer completion code with it so it gets displayed
> in *Completions*.

IMO native tab completion for subprocess REPLs is a common enough need
that the basic machinery should be handled by Emacs itself, i.e.
probably the comint library. I'm somewhat surprised there is no such
code yet (or is there, outside Emacs perhaps?).

-- 
Štěpán

Re: Comint: handle raw tab

[Fabian Ezequiel Gallina]

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

2011/9/13 Stefan Monnier <monnier@iro.umontreal.ca>

> > First case scenario, unique completion:
> > Consider the only possible completion being "True".
> > In [1]: Tr
> > Then hit C-q TAB so a TAB gets inserted after it and evaled:
> > (comint-send-input t)
> > In a normal ipython shell the result of this causes the input to be
> expanded
> > to "True" which is the unique completion. On the comint buffer this
> causes
> > the input to remain frozen. Internally, the input *does* get updated
> since
> > when I hit RET after evaling the code above the out shows "True" but I
> > didn't find a way to update the current input accordingly. Is there any
> way
> > to achieve that?
>
> You're going to have to redirect the process's output to read the
> shell's output and then use it to fill the user's current input.
>
>
I'll appreciate any pointers (perhaps to an existing code in Emacs) on how
to do that. I'm looking at the documentation and existing code with no luck
yet.


> > Second scenario, multiple completions available:
> > Consider now I have typed just T:
> > In [1]: T
> > Then hit C-q TAB so a TAB gets inserted after it and evaled:
> > (comint-send-input t)
> > Now interesting things happens, since ipython outputs the list of
> possible
> > completions I can get them with comint-output-filter-functions, the thing
> is
> > the buffer now looks like this:
> > In [1]: T
> > TabError       True       TypeError
>
> > And the only way I found to show the prompt again without sending "T" to
> the
> > process was sending a BREAK signal because comint-delete-input does not
> work
> > in that instance. Is there a better way to handle that?
>
> The better way (IMNSHO) is to catch the process output so it doesn't get
> inserted in the buffer, build a completion table from it, and then
> call the normal in-buffer completion code with it so it gets displayed
> in *Completions*.
>
>
That's simple enough, changing the process filter for that particular case
would do the trick. My guess is that once we can solve the first case they
will be easy to distinguish.


> > The only thing I can think of is having to rewrite the shell
> > interactions (pydoc, pdbtrack, etc) I already have in python.el.
> > However all the inferior shells implementations I know use comint so
> > that makes me feel unsure about it.
>
> I don't know if someone ever tried to use a term rather than a comint
> inferior process, but I'd be interested to hear your experience with it.
> I suspect it's going to be harder to interface it with compile.el.
>
>
I really would like to continue with the comint based one. Once TAB does
what I mean I don't see why change it ;)


Regards,
-- 
Fabián E. Gallina
http://www.from-the-cloud.com

[-- Attachment #2: Type: text/html, Size: 3774 bytes --]

Re: Comint: handle raw tab

[Fabian Ezequiel Gallina]

[-- Attachment #1: Type: text/plain, Size: 543 bytes --]

2011/9/13 Štěpán Němec <stepnem@gmail.com>

>
> IMO native tab completion for subprocess REPLs is a common enough need
> that the basic machinery should be handled by Emacs itself, i.e.
> probably the comint library. I'm somewhat surprised there is no such
> code yet (or is there, outside Emacs perhaps?).
>
>
I agree with the exposed above, many modern REPLs comint can interact with
have TAB completion out of the box. We should use that when it's available
instead of duplicating efforts.


Regards,
Fabián E. Gallina

[-- Attachment #2: Type: text/html, Size: 862 bytes --]

Re: Comint: handle raw tab

[Stefan Monnier]

>> IMO native tab completion for subprocess REPLs is a common enough need
>> that the basic machinery should be handled by Emacs itself, i.e.
>> probably the comint library. I'm somewhat surprised there is no such
>> code yet (or is there, outside Emacs perhaps?).
> I agree with the exposed above, many modern REPLs comint can interact with
> have TAB completion out of the box.  We should use that when it's available
> instead of duplicating efforts.

I agree that would be good.
You can check the gdb completion code in gud.el for an example.


        Stefan


(defun gud-gdb-completions (context command)
  "Completion table for GDB commands.
COMMAND is the prefix for which we seek completion.
CONTEXT is the text before COMMAND on the line."
  (let* ((start (- (point) (field-beginning)))
         (complete-list
	  (gud-gdb-run-command-fetch-lines (concat "complete " context command)
					   (current-buffer)
					   ;; From string-match above.
					   (length context))))
    ;; `gud-gdb-run-command-fetch-lines' has some nasty side-effects on the
    ;; buffer (via `gud-delete-prompt-marker'): it removes the prompt and then
    ;; re-adds it later, thus messing up markers and overlays along the way.
    ;; This is a problem for completion-in-region which uses an overlay to
    ;; create a field.
    ;; So we restore completion-in-region's field if needed.
    ;; FIXME: change gud-gdb-run-command-fetch-lines so it doesn't modify the
    ;; buffer at all.
    (when (/= start (- (point) (field-beginning)))
      (dolist (ol (overlays-at (1- (point))))
        (when (eq (overlay-get ol 'field) 'completion)
          (move-overlay ol (- (point) start) (overlay-end ol)))))
    ;; Protect against old versions of GDB.
    (and complete-list
	 (string-match "^Undefined command: \"complete\"" (car complete-list))
	 (error "This version of GDB doesn't support the `complete' command"))
    ;; Sort the list like readline.
    (setq complete-list (sort complete-list (function string-lessp)))
    ;; Remove duplicates.
    (let ((first complete-list)
	  (second (cdr complete-list)))
      (while second
	(if (string-equal (car first) (car second))
	    (setcdr first (setq second (cdr second)))
	  (setq first second
		second (cdr second)))))
    ;; Add a trailing single quote if there is a unique completion
    ;; and it contains an odd number of unquoted single quotes.
    (and (= (length complete-list) 1)
	 (let ((str (car complete-list))
	       (pos 0)
	       (count 0))
	   (while (string-match "\\([^'\\]\\|\\\\'\\)*'" str pos)
	     (setq count (1+ count)
		   pos (match-end 0)))
	   (and (= (mod count 2) 1)
		(setq complete-list (list (concat str "'"))))))
    complete-list))

(defun gud-gdb-completion-at-point ()
  "Return the data to complete the GDB command before point."
  (let ((end (point))
        (start
         (save-excursion
           (skip-chars-backward "^ " (comint-line-beginning-position))
           (point))))
    (list start end
          (completion-table-dynamic
           (apply-partially #'gud-gdb-completions
                            (buffer-substring (comint-line-beginning-position)
                                              start))))))

Re: Comint: handle raw tab

[Fabian Ezequiel Gallina]

[-- Attachment #1: Type: text/plain, Size: 4161 bytes --]

Hi all,

2011/9/13 Stefan Monnier <monnier@iro.umontreal.ca>

> >> IMO native tab completion for subprocess REPLs is a common enough need
> >> that the basic machinery should be handled by Emacs itself, i.e.
> >> probably the comint library. I'm somewhat surprised there is no such
> >> code yet (or is there, outside Emacs perhaps?).
> > I agree with the exposed above, many modern REPLs comint can interact
> with
> > have TAB completion out of the box.  We should use that when it's
> available
> > instead of duplicating efforts.
>
> I agree that would be good.
> You can check the gdb completion code in gud.el for an example.
>
>
The current shell completion machinery of my python.el works pretty much
like the gdb does, the thing is that on the python shell this approach is
likely to break on multi-line statements.

Good news is that this weekend I got back to try stuff again to get native
TAB support on the comint process, and while travelling back to my hometown
from the PyCon Argentina (the Bus ride was getting boring) I found something
that made the shell react pretty much like the real thing when sending a raw
tab: I started the comint process in the same way ansi-term would, using sh
+ stty.

Here is the quick snippet:

    (defun python-shell-make-comint (cmd proc-name &optional pop)
      "Create a python shell comint buffer.
    CMD is the python command to be executed and PROC-NAME is the
    process name the comint buffer will get.  After the comint buffer
    is created the `inferior-python-mode' is activated.  If POP is
    non-nil the buffer is shown."
      (save-excursion
        (let* ((proc-buffer-name (format "*%s*" proc-name))
               (process-environment
(python-shell-calculate-process-environment))
               (exec-path (python-shell-calculate-exec-path)))
          (when (not (comint-check-proc proc-buffer-name))
            (let* ((cmdlist (split-string-and-unquote cmd))
                   (sh-executable (executable-find "sh"))
                   (stty-executable (executable-find "stty"))
                   (buffer (if (and sh-executable stty-executable)
                               (apply 'make-comint proc-name
                                      sh-executable nil "-c"
                                      (format "%s -nl echo rows %d columns
%d sane 2>/dev/null;\
    if [ $1 = .. ]; then shift; fi; exec \"$@\""
                                              stty-executable
                                              (window-height) (1-
(window-width)))
                                      ".."
                                      (car cmdlist) (cdr cmdlist))
                             (apply 'make-comint proc-name (car cmdlist) nil
                                    (cdr cmdlist))))
                   (current-buffer (current-buffer)))
              (with-current-buffer buffer
                (inferior-python-mode)
                (python-util-clone-local-variables current-buffer))))
          (when pop
            (pop-to-buffer proc-buffer-name))
          proc-buffer-name)))

After that, when I evaled:

(process-send-string (get-buffer-process (current-buffer)) "Tru    ")

The shell got updated and said True.

If I evaled:

(process-send-string (get-buffer-process (current-buffer)) "T    ")

TabError   True       TypeError

Got returned and the shell displayed correctly as it does on a full featured
terminal.

I need to work a little more on the extra bookkeeping involved into getting
comint know about the stuff that got inserted in the process but this got me
really close to have it working as intended.

What worries me is the Window support for this, is there something similar
like stty there? What I'm thinking is a good approach would be that when sh
and stty are not found via `executable-find', fallback to the old style
completion.

I'll continue working on this in the following days (hopefully I'll get
lucky). After that we can see if there is some kind of improvement we can do
in the comint library itself.


Regards,
Fabián E. Gallina

[-- Attachment #2: Type: text/html, Size: 4754 bytes --]