Inner workings of `pcomplete'

Inner workings of `pcomplete'

[Oleksandr Manzyuk]

Dear all,

I'm working on a small package aiming at providing basic completion for
GHCi commands in `inferior-haskell-mode':

https://github.com/manzyuk/ghci-completion

The package uses a combination of the standard completion UI (for GHCi
commands) with `pcomplete' (for arguments to particular commands).

I've been experimenting recently with identifier completion and I've
stumbled upon the following issue.  I'm not sure whether it is a bug in
the code or in my understanding of it.  Everything below refers to the
latest Emacs 24 (24.0.93.1).

To enable `pcomplete' in `inferior-haskell-mode', I add the function
`pcomplete-completions-at-point' to `comint-dynamic-complete-functions'.
Experimenting with identifier completion, I started to receive the
errors "Wrong type argument: stringp, nil", which I traced back to the
call

(pcomplete--common-quoted-suffix pcomplete-stub buftext)

in the body of `pcomplete-completions-at-point'.  Now, `pcomplete-stub'
is bound to nil at the beginning of the function using let* and it is
expected that the function `pcomplete-completions', which is called
later, will change the value of `pcomplete-stub' by side-effects.
Apparently, this doesn't happen and `pcomplete--common-quoted-suffix'
receives nil as its first argument, which triggers the error.

`pcomplete-stub' is declared using defvar, and the top comment of the
file contains -*- lexical-binding: t -*-, so `pcomplete-stub' should be
a special variable.  This is not the case:

1. emacs -Q
2. M-: (boundp 'pcomplete-stub) RET ==> nil
3. M-x shell
4. M-: (boundp 'pcomplete-stub) RET ==> t
5. M-: (special-variable-p 'pcomplete-stub) RET ==> nil
6. Open the file `pcomplete.el' (for example, by looking up the variable
`pcomplete-stub') and M-x eval-buffer.  Now (special-variable-p
'pcomplete-stub) evaluates to t.

What is going on here?

To confirm my guess that `pcomplete-stub' was not a special variable, I
tried the following workaround: I removed `pcomplete-stub' from the let*
bindings, so that no lexical binding was created for it, and hence its
global value was used in the body of let*.  The errors were gone and
completion started to work as I wanted it to.

Actually, to make identifier completion really work the way I wanted, I
had to make another change in `pcomplete-completions-at-point', namely
to replace

(when completions
  ...)

with 

(unless (functionp completions)
  ...)

The thing is, `completions' (the value returned by the function
`pcomplete-completions') can be a function.  In fact, it is a function
in the case of "cd" command in shell (try to type "cd " at the shell
prompt and M-: (pcomplete-completions)).  However, it is also passed as
a table argument to the function `complete-with-action', whose
documentation says that this argument should not be a function.  What
happens if it is a function?  I don't know, but somehow, with the old
code allowing `completions' to be a function, I get the behavior that
`pcomplete-completions-at-point' always succeeds (returns a suitably
formatted plist, even though its components don't always produce
completions!), leaving no opportunity to the next function in
`comint-dynamic-complete-functions' (identifier completion) to fire.

I should mention that by this change I've lost some of completion
functionality (e.g., "cd" in shell doesn't offer the list of
directories), so this is clearly not the right way to do it.  What is
the right way then?

I'm confused.  Any comments and suggestions would be greatly
appreciated.

Regards,
Sasha

Re: Inner workings of `pcomplete'

[Stefan Monnier]

> 1. emacs -Q
> 2. M-: (boundp 'pcomplete-stub) RET ==> nil
> 3. M-x shell
> 4. M-: (boundp 'pcomplete-stub) RET ==> t
> 5. M-: (special-variable-p 'pcomplete-stub) RET ==> nil
> 6. Open the file `pcomplete.el' (for example, by looking up the variable
> `pcomplete-stub') and M-x eval-buffer.  Now (special-variable-p
> 'pcomplete-stub) evaluates to t.

> What is going on here?

Duh!  Thanks for catching this.  Please M-x report-emacs-bug!

> prompt and M-: (pcomplete-completions)).  However, it is also passed as
> a table argument to the function `complete-with-action', whose
> documentation says that this argument should not be a function.

The docstring is wrong, as the first line of the code indicates ;-)


        Stefan

Re: Inner workings of `pcomplete'

[Oleksandr Manzyuk]

On Wed, Feb 8, 2012 at 3:33 AM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 1. emacs -Q
>> 2. M-: (boundp 'pcomplete-stub) RET ==> nil
>> 3. M-x shell
>> 4. M-: (boundp 'pcomplete-stub) RET ==> t
>> 5. M-: (special-variable-p 'pcomplete-stub) RET ==> nil
>> 6. Open the file `pcomplete.el' (for example, by looking up the variable
>> `pcomplete-stub') and M-x eval-buffer.  Now (special-variable-p
>> 'pcomplete-stub) evaluates to t.
>
>> What is going on here?
>
> Duh!  Thanks for catching this.  Please M-x report-emacs-bug!

Done.

>> prompt and M-: (pcomplete-completions)).  However, it is also passed as
>> a table argument to the function `complete-with-action', whose
>> documentation says that this argument should not be a function.
>
> The docstring is wrong, as the first line of the code indicates ;-)

Huh, indeed.  Then I rephrase my question:
`pcomplete-completions-at-point' can return non-nil even when there
are no completions, which is easy to test: in a shell buffer, I type,
say, 5, and hit TAB.  Assuming that there are no files in the current
directory starting with 5, there will be no matches.  However, if I
evaluate (pcomplete-completions-at-point), I get some hairy data
structure.  This means, in particular, that the functions in
`comint-dynamic-complete-functions' that appear after the function
`pcomplete-completions-at-point' are not tried.  Is there a way around
this?  The identifier completion that I'm experimenting with logically
happens after completions for arguments of the interpreter commands
have been tried (if I put the function responsible for the identifier
completion, I get the identifier completion, but lose completion for
arguments of commands, and I'd love to have both).

Best,
Sasha
-- 
Oleksandr Manzyuk
http://oleksandrmanzyuk.wordpress.com

Re: Inner workings of `pcomplete'

[Stefan Monnier]

> Huh, indeed.  Then I rephrase my question:
> `pcomplete-completions-at-point' can return non-nil even when there
> are no completions,

Indeed.  What it means is that pcomplete takes responsibility for the
completion at point, including saying "no completions".

> which is easy to test: in a shell buffer, I type,
> say, 5, and hit TAB.  Assuming that there are no files in the current
> directory starting with 5, there will be no matches.  However, if I
> evaluate (pcomplete-completions-at-point), I get some hairy data
> structure.  This means, in particular, that the functions in
> `comint-dynamic-complete-functions' that appear after the function
> `pcomplete-completions-at-point' are not tried.

The completion code performs completion in several steps.
The first is "what field am I on", where the answer is "a field from
BEG..END that contains elements of type FOO".  If you want to say "I'm
not sure which type it is, could be FOO or BAR", you'll have to do that
explicitly: there's no automatic fallback along the lines of "he said
it's a field of FOOs, but there's no matching FOO so we'll see if
someone else has another opinion".

Maybe your problem is pcomplete-default-completion-function, which by
default always assumes responsibility for completion.  E.g. in shell.el
we set it via

   (set (make-local-variable 'pcomplete-default-completion-function) #'ignore)


-- Stefan