bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Drew Adams]

[The build noted below is the one this bug report is for.  But I'm
sending this using an older build than the one noted below, because of
another new bug, which prevents using `report-emacs-bug']


Just a preliminary heads up for now.  Hope to add more info later, when
I get some time.

I downloaded this build, and when `icomplete-mode' is on, with my setup
it takes several seconds to gather the list of file candidates in my
usual directory.  With a build from two days ago, this does not happen.
And if I turn off icomplete mode it also does not happen.

It seems that something was changed in the icomplete mode code recently
that breaks at least my file-finding code.

With emacs -Q I do not notice the problem (well, maybe a slight delay).
I see that C-x C-f now shows completions immediately, without my typing
any prefix.  That is not true in the build from 2 days ago.

In GNU Emacs 24.3.50.1 (i686-pc-mingw32)
 of 2013-12-24 on ODIEONE
Bzr revision: 115738 cyd@gnu.org-20131225030511-ru56hhc243pxja04
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --prefix=/c/Devel/emacs/binary --enable-checking=yes,glyphs
 'CFLAGS=-O0 -g3' LDFLAGS=-Lc:/Devel/emacs/lib
 CPPFLAGS=-Ic:/Devel/emacs/include'

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Jambunathan K]

Drew Adams <drew.adams@oracle.com> writes:

> It seems that something was changed in the icomplete mode code recently
> that breaks at least my file-finding code.

What happens if you turn this off:

    icomplete-show-matches-on-no-input

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Daniel Colascione]

On 12/24/2013 09:58 PM, Drew Adams wrote:
> [The build noted below is the one this bug report is for.  But I'm
> sending this using an older build than the one noted below, because of
> another new bug, which prevents using `report-emacs-bug']
>
>
> Just a preliminary heads up for now.  Hope to add more info later, when
> I get some time.
>
> I downloaded this build, and when `icomplete-mode' is on, with my setup
> it takes several seconds to gather the list of file candidates in my
> usual directory.  With a build from two days ago, this does not happen.
> And if I turn off icomplete mode it also does not happen.
>
> It seems that something was changed in the icomplete mode code recently
> that breaks at least my file-finding code.
>
> With emacs -Q I do not notice the problem (well, maybe a slight delay).
> I see that C-x C-f now shows completions immediately, without my typing
> any prefix.  That is not true in the build from 2 days ago.

Mea culpa: I committed a change that caused icomplete to try to show 
completions right away by default. As Jambunathan mentions, setting 
icomplete-show-matches-on-no-input to nil should restore the old 
behavior. The old behavior isn't optimal, though, and icomplete isn't a 
replacement for iswitchb without the feature I added.

Maybe we can change icomplete-show-matches-on-no-input to a command list 
--- this way, we could show completions early for buffer switching, but 
not for finding files.

Why is finding the list of files so slow for you? Don't you experience 
the same performance problem after typing a character and forcing 
completions to show up? We call the completion function inside 
while-no-input, so we should abort the "several seconds" of work as soon 
as you start typing.

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Drew Adams]

> > It seems that something was changed in the icomplete mode code recently
> > that breaks at least my file-finding code.
> 
> What happens if you turn this off: icomplete-show-matches-on-no-input

Yes, thank you!  That removes the problem.

Any idea (without knowing my code) why this makes a huge difference
with my code but not a huge difference with emacs -Q?  IOW, is there
something particular in the new icomplete code to be aware of here?

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Drew Adams]

> Why is finding the list of files so slow for you?

I don't know.  And I don't have time right now to track it down.
But I'll try to take a look later some time.

> Don't you experience the same performance problem after typing
> a character and forcing completions to show up?

No, absolutely not.

> We call the completion function inside while-no-input, so we
> should abort the "several seconds" of work as soon as you start
> typing.

I didn't start typing.  In Icicles it often makes sense to show
all initial completions from the outset, either automatically or
on demand.

While waiting for better understanding of the problem, and
possibly a fix if there is in fact a bug and solution, I will
automatically turn off `icomplete-show-matches-on-no-input'
when Icicle mode is turned on.

BTW, there is no such problem with using IswitchB with Icicles,
AFAIK.

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Eli Zaretskii]

> Date: Wed, 25 Dec 2013 00:30:07 -0800
> From: Daniel Colascione <dancol@dancol.org>
> 
> Why is finding the list of files so slow for you? Don't you experience 
> the same performance problem after typing a character and forcing 
> completions to show up? We call the completion function inside 
> while-no-input, so we should abort the "several seconds" of work as soon 
> as you start typing.

Maybe because input on Windows is not signal-driven, and therefore
while-no-input relies on the Lisp code paying frequent attention to
QUIT.

I tried just now enabling icomplete-mode in "emacs -Q", and can
confirm that "C-x C-f" becomes painfully slow to react to typing in a
large directory, especially with a cold cache.

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Stefan Monnier]

> I downloaded this build, and when `icomplete-mode' is on, with my setup
> it takes several seconds to gather the list of file candidates in my
> usual directory.

I just reverted icomplete-show-matches-on-no-input to nil, which I think
is the right default.  That it can take a long time to get the
completions is not in itself a bug.  There are 2 potential bugs left, tho:
- hitting a key should interrupt the completions processing (so that
  the long wait should not prevent you from getting work done).
  If it doesn't, then we have a bug.  Please report it separately, with
  as much details as possible to reproduce it (it's probably a problem
  in the C code).
- ideally completion should never take that long, so we probably have
  a performance bug somewhere.  Of course, that might also be in your
  local code (.emacs, icicles, ...).


-- Stefan

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Drew Adams]

> > I downloaded this build, and when `icomplete-mode' is on, with my
> > setup it takes several seconds to gather the list of file candidates
> > in my usual directory.

<embarrassment>
This was my bad.  I was passing this to `cl-delete-if':
(regexp-opt completion-ignored-extensions).  Taking that calculation
out of the loop saved 13 seconds! (99% of the total time).
</embarrassment>

FWIW, here are some times (now) of various parts (with my code, with
my find-file replacement, in my typical startup directory, about 2400
files):

 completion-all-completions: 680 ms
 deleting duplicates:         70 ms
 sorting:                    470 ms

Of course, sorting depends on the sort predicate.  This was with a
directories-first-then-alphabetical sort.

> I just reverted icomplete-show-matches-on-no-input to nil, which
> I think is the right default.

Thx.

> That it can take a long time to get the completions is not in itself
> a bug.  There are 2 potential bugs left, tho:
>
> - hitting a key should interrupt the completions processing (so that
>   the long wait should not prevent you from getting work done).
>   If it doesn't, then we have a bug.  Please report it separately, with
>   as much details as possible to reproduce it (it's probably a problem
>   in the C code).

That seems to work OK (as before).

> - ideally completion should never take that long, so we probably have
>   a performance bug somewhere.  Of course, that might also be in your
>   local code (.emacs, icicles, ...).

See <embarrassment>, above.

A problem I do notice now (not sure why now) is that sometimes keys
that I hit are "lost" instead of appearing in the input.  AFAICT, this
happens only when Icomplete is on.  It can make completing input
painful, to say the least.  I don't have a handle yet on just what
the behavior is or what causes it.  Just mentioning it now in case
someone happens to notice it also for vanilla Emacs.

---

FWIW - One other thing is somewhat unfortunate in my context, so far:
When you cycle among completion candidates (Icicles cycling, not
Icomplete cycling), or when there is a sole matching candidate, by
default Icicles shows info about the current (or the sole) candidate
in the (*Completions*) mode line, for N sec.  Hitting a key interrupts
this, of course - it is done using `sit-for'.

But of course `post-command-hook' actions do not take place until
that delay is over.  This means that icompletions do not show up
until the mode-line display is finished.  I guess I should instead
show the info until some timer gets rid of it, so that it stays
visible even when `post-command-hook' is run.

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Eli Zaretskii]

> Date: Fri, 27 Dec 2013 16:55:11 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 16251-done@debbugs.gnu.org
> 
>  completion-all-completions: 680 ms
>  deleting duplicates:         70 ms
>  sorting:                    470 ms
> 
> Of course, sorting depends on the sort predicate.  This was with a
> directories-first-then-alphabetical sort.

How much time does sorting take if you request just alphabetical,
without directories-first?  It should take zero on Windows; if it
doesn't, our sorting algorithm should be improved.

bug#16251: 24.3.50; `icomplete-mode' breaks my file opening now

[Drew Adams]

> >  completion-all-completions: 680 ms
> >  deleting duplicates:         70 ms
> >  sorting:                    470 ms
> >
> > Of course, sorting depends on the sort predicate.  This was with
> > a directories-first-then-alphabetical sort.
> 
> How much time does sorting take if you request just alphabetical,
> without directories-first?  It should take zero on Windows; if it
> doesn't, our sorting algorithm should be improved.

I no longer have the debugging set up for this (calls to `message'
etc., since it is impossible to use the debugger with Icomplete).  

But IIRC it was not very different from that.  But this the sorting
I use here is the Icicles sorting, so the efficiency of its
implementation should be irrelevant to emacs -Q.  That is, if
Icicle mode is on, I use the current Icicles sort function.