CVS directories in completion-ignored-extensions

CVS directories in completion-ignored-extensions

[Eli Zaretskii]

The default value of completion-ignored-extensions includes "CVS/".
This is supposed to ignore the (usually uninteresting) `CVS'
directories, but it also ignores directories like `foo-CVS', which
seems to be an unintended consequence.

Should we do something about this?

Re: CVS directories in completion-ignored-extensions

[Eric Hanchrow]

>>>>> "Eli" == Eli Zaretskii <eliz@elta.co.il> writes:

    Eli> The default value of completion-ignored-extensions includes
    Eli> "CVS/".  This is supposed to ignore the (usually
    Eli> uninteresting) `CVS' directories, but it also ignores
    Eli> directories like `foo-CVS', which seems to be an unintended
    Eli> consequence.

    Eli> Should we do something about this?

Funny -- I *just* ran into this problem now.  Yes, I wish this were
fixed.  I don't know how to fix it, unless there's another mechanism
that uses full-blown regular expressions.
 
-- 
The whole point of loud music is to make it possible
to date without talking.
        -- Roger Ebert

Re: CVS directories in completion-ignored-extensions

[Luc Teirlinck]

Eric Hanchrow wrote:

   >>>>> "Eli" == Eli Zaretskii <eliz@elta.co.il> writes:

       Eli> The default value of completion-ignored-extensions includes
       Eli> "CVS/".  This is supposed to ignore the (usually
       Eli> uninteresting) `CVS' directories, but it also ignores
       Eli> directories like `foo-CVS', which seems to be an unintended
       Eli> consequence.

       Eli> Should we do something about this?

   Funny -- I *just* ran into this problem now.  Yes, I wish this were
   fixed.  I don't know how to fix it, unless there's another mechanism
   that uses full-blown regular expressions.

If I understand correctly, one does not want a directory name to
complete to "CVS", but one _does_ want directory names to complete to
strings ending in CVS.  In that case, why not keep it simple and just
remove "CVS/" from `completion-ignored-extensions'?  Trying to exclude
a three letter name from completion does not seem to be worth any trouble.

Sincerely,

Luc.

Re: CVS directories in completion-ignored-extensions

[Miles Bader]

On Tue, Dec 30, 2003 at 07:08:33PM -0600, Luc Teirlinck wrote:
>  In that case, why not keep it simple and just
> remove "CVS/" from `completion-ignored-extensions'?  Trying to exclude
> a three letter name from completion does not seem to be worth any trouble.

Because then it would Very Annoying -- CVS subdirs are _very common_, and
interfere with some comon filenames (notably, ChangeLog).
Directory/filenames _ending_ in CVS, on the other hand, seem quite rare.

So sure, without a better mechanism, there will be some annoying side-effects
in completion, but let's program for the common case, and _keep_ CVS in
completion-ignored-extensions.

-Miles
-- 
Somebody has to do something, and it's just incredibly pathetic that it
has to be us.  -- Jerry Garcia

Re: CVS directories in completion-ignored-extensions

[Eli Zaretskii]

> Date: Tue, 30 Dec 2003 20:34:07 -0500
> From: Miles Bader <miles@gnu.org>
> 
> So sure, without a better mechanism, there will be some annoying side-effects
> in completion, but let's program for the common case, and _keep_ CVS in
> completion-ignored-extensions.

I agree--I never meant to suggest that we remove CVS from the list.

Suggestions for improving the mechanism of ignoring are welcome.

Re: CVS directories in completion-ignored-extensions

[Eli Zaretskii]

> Date: Tue, 30 Dec 2003 20:34:07 -0500
> From: Miles Bader <miles@gnu.org>
> 
> So sure, without a better mechanism, there will be some annoying side-effects
> in completion, but let's program for the common case, and _keep_ CVS in
> completion-ignored-extensions.

How about if we change "CVS/" to "/CVS/" and modify the code to match
the whole name if the member of the list begins with a slash?

The only problem with that approach, AFAICT, is when a user wants to
ignore files under the root directory.  If that sounds like a serious
problem, we could have something like "//foo/" to DTRT in such cases.

What do you-all think?

Re: CVS directories in completion-ignored-extensions

[Richard Stallman]

    How about if we change "CVS/" to "/CVS/" and modify the code to match
    the whole name if the member of the list begins with a slash?

I'd rather have a list of directory file names to be matched exactly
(not as substrings), and put CVS into that.

Re: CVS directories in completion-ignored-extensions

[Kim F. Storm]

Eli Zaretskii <eliz@elta.co.il> writes:

> > Date: Tue, 30 Dec 2003 20:34:07 -0500
> > From: Miles Bader <miles@gnu.org>
> > 
> > So sure, without a better mechanism, there will be some annoying side-effects
> > in completion, but let's program for the common case, and _keep_ CVS in
> > completion-ignored-extensions.
> 
> How about if we change "CVS/" to "/CVS/" and modify the code to match
> the whole name if the member of the list begins with a slash?

That's a good idea!

> 
> The only problem with that approach, AFAICT, is when a user wants to
> ignore files under the root directory.  If that sounds like a serious
> problem, we could have something like "//foo/" to DTRT in such cases.

IIUC, the completion-ignored-extensions doesn't have any notion of
things being relative to a certain directory, so why is the root
directory a problem?

> 
> What do you-all think?

I like it, but I think /CVS/ is sufficient.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: CVS directories in completion-ignored-extensions

[Miles Bader]

Richard Stallman <rms@gnu.org> writes:
>     How about if we change "CVS/" to "/CVS/" and modify the code to match
>     the whole name if the member of the list begins with a slash?
> 
> I'd rather have a list of directory file names to be matched exactly
> (not as substrings), and put CVS into that.

How about a list of regexps (and add `^CVS/$' -- using the same
`directories have slashes' rule as completion-ignored-extensions)?

That'd be more useful for future weird cases, and most people will use
completion-ignored-extensions anyway, so I don't think `regexps are
confusing to novices' would be an objection to it.

-Miles
-- 
We live, as we dream -- alone....

Re: CVS directories in completion-ignored-extensions

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

>     How about if we change "CVS/" to "/CVS/" and modify the code to match
>     the whole name if the member of the list begins with a slash?
> 
> I'd rather have a list of directory file names to be matched exactly
> (not as substrings), and put CVS into that.


Actually, I don't really see why we need to worry about matching
directory names as an extension (it may easily lead to bogus results
as the example with xxx-CVS/ shows).

Couldn't we simply say that if a directory name (an element ending in
a /) is present in completion-ignored-extensions, it must match the
directory name exactly?

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: CVS directories in completion-ignored-extensions

[Eli Zaretskii]

> From: storm@cua.dk (Kim F. Storm)
> Date: 02 Jan 2004 02:06:45 +0100
> 
> Couldn't we simply say that if a directory name (an element ending in
> a /) is present in completion-ignored-extensions, it must match the
> directory name exactly?

Fine with me.

Re: CVS directories in completion-ignored-extensions

[Eli Zaretskii]

> From: storm@cua.dk (Kim F. Storm)
> Date: 02 Jan 2004 02:06:45 +0100
> 
> Couldn't we simply say that if a directory name (an element ending in
> a /) is present in completion-ignored-extensions, it must match the
> directory name exactly?

Richard, I tend to think that this is the best proposition for a
solution in this thread.  In addition to being the simplest, it is
also 100% backwards-compatible, at least as far as the user interface
and the semantics of the default value of the option
completion-ignored-extensions are concerned.

Do you see any disadvantages of what Kim suggested?

Re: CVS directories in completion-ignored-extensions

[Richard Stallman]

    > Couldn't we simply say that if a directory name (an element ending in
    > a /) is present in completion-ignored-extensions, it must match the
    > directory name exactly?

    Richard, I tend to think that this is the best proposition for a
    solution in this thread.  In addition to being the simplest, it is
    also 100% backwards-compatible, at least as far as the user interface
    and the semantics of the default value of the option
    completion-ignored-extensions are concerned.

This is undesirable because it contradicts the name
`completion-ignored-extensions'.  That's why I suggested adding a new
variable `completion-ignored-names'.

Re: CVS directories in completion-ignored-extensions

[Stefan Monnier]

> This is undesirable because it contradicts the name
> `completion-ignored-extensions'.  That's why I suggested adding a new
> variable `completion-ignored-names'.

I think that Kim's suggestion is the best as long as we only have
completion-ignored-extensions (and it's indeed 100% compatible since names
with slashes were completely ignored in released versions of Emacs).

But I for one would like to move on from completion-ignored-extensions to
something more powerful that can ignore .#* as well (for example).

So instead of completion-ignored-names I'd suggest completion-ignored-regexp
(or completion-ignored-regexps if people feel like having a list of regexps
instead of a single one is worth the trouble).


        Stefan

Re: CVS directories in completion-ignored-extensions

[Miles Bader]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> So instead of completion-ignored-names I'd suggest completion-ignored-regexp
> (or completion-ignored-regexps if people feel like having a list of regexps
> instead of a single one is worth the trouble).

I've always thought it would be nice, instead of having multiple variables
each of which contain a different types of entry, to have something like
`completion-ignored-names', where each element could be either: a string
(which is matched using just `equal'), or a list like (regexp REGEXP),
which the matcher would treat accordingly.

So for CVS, we'd just add "CVS/" to the list, but you could also add
(regexp "#.*") to ignore backup files.

This sort of situation happens often in emacs, so we've ended up with many
sets of related variables, FOO-ignored-buffers, FOO-ignored-regexps,
etc.).  I think having separate variables which do almost the same thing
is more confusing that having a single variable in which the elements of
the list can have various meanings, as long as the notation used for
indicating the meaning of each element is sufficiently clear -- and I
think "STRING" => exact match, (regexp "REGEXP") => regexp match, etc.,
meets that criterion.  The actual mechanics of the matching could probably
be done by a single common function, so it need not be an implementation
burden.

[BTW, even in the case of a regexp-only variable, it's a lot nicer for the
user to be able to add elements to a list than to add to a regexp...]

-Miles
-- 
Somebody has to do something, and it's just incredibly pathetic that it
has to be us.  -- Jerry Garcia

Re: CVS directories in completion-ignored-extensions

[Richard Stallman]

    I've always thought it would be nice, instead of having multiple variables
    each of which contain a different types of entry, to have something like
    `completion-ignored-names', where each element could be either: a string
    (which is matched using just `equal'), or a list like (regexp REGEXP),
    which the matcher would treat accordingly.

That is a good idea.

Re: CVS directories in completion-ignored-extensions

[Miles Bader]

Richard Stallman <rms@gnu.org> writes:
> > I've always thought it would be nice, instead of having multiple
> > variables each of which contain a different types of entry, to have
> > something like `completion-ignored-names', where each element could
> > be either: a string (which is matched using just `equal'), or a list
> > like (regexp REGEXP), which the matcher would treat accordingly.
> 
> That is a good idea.

Ok, how about something like:

   (defun string-matcher-member (string matcher)
     "Return non-nil if STRING is matched by the `string matcher' MATCHER.
   MATCHER should be a list, where each element is either a string,
   which will be compared with STRING using `equal', or a cons of the
   form (regexp REGEXP), in which case REGEXP will be compared with
   STRING using `string-match'.  The actual return value is the first
   element of MATCHER that matches."
     (let (test)
       (while matcher
         (setq test (pop matcher))
         (if (cond ((stringp test) 
                    (equal string test))
                   ((eq (car-safe test) 'regexp)
                    (if (consp (cdr test))
                        (string-match (car-safe (cdr-safe test)) string)
                      (string-match (cdr-safe test) string))))
             (setq matcher nil)
           (setq test nil)))
       test))

The name `string-matcher-member' is kind of ugly, but at least it
seems fairly clear; maybe someone can think of a better name.

The nice thing is that this is basically backward-compatible with existing
lists of strings that are matched exactly, so perhaps some of them could
be retro-fitted accordingly (e.g.  `dabbrev-ignored-buffer-names').

[`special-display-buffer-names' is similar (it has a corresponding
`special-display-regexps'), but because it itself gives special treatment
to list members that are cons-cells, doesn't fit exactly into such a
scheme (maybe it could be finessed, I don't know).]

-Miles
-- 
`Suppose Korea goes to the World Cup final against Japan and wins,' Moon said.
`All the past could be forgiven.'   [NYT]

Re: CVS directories in completion-ignored-extensions

[Kenichi Handa]

FCC: ~/mailbox/MINE
In article <87r7yeq6py.fsf@tc-1-100.kawasaki.gol.ne.jp>, Miles Bader <miles@gnu.org> writes:
> Richard Stallman <rms@gnu.org> writes:
>>  > I've always thought it would be nice, instead of having multiple
>>  > variables each of which contain a different types of entry, to have
>>  > something like `completion-ignored-names', where each element could
>>  > be either: a string (which is matched using just `equal'), or a list
>>  > like (regexp REGEXP), which the matcher would treat accordingly.
>>  
>>  That is a good idea.

> Ok, how about something like:

>    (defun string-matcher-member (string matcher)

Thank you.  As this function is quite handy, I want it
implemented in C so that C code can use it.  There are many
places in C code that can use it.

And, while reading this discussion, I remember one feature
implemented only in the old mule.  That is `regexp-compile'.
It compiles a regular expression and returns a special
object that represent the result of compilation, and the
object can be given to any function that accept a regular
expression.  What do you think about this feature?  If we
implement this feature, we can directry use that object in
MATCHER, which I think is better than the form `(regexp
REGEXP)'.

---
Ken'ichi HANDA
handa@m17n.org

Re: CVS directories in completion-ignored-extensions

[Miles Bader]

Kenichi Handa <handa@m17n.org> writes:
> I remember one feature implemented only in the old mule.  That is
> `regexp-compile'.  It compiles a regular expression and returns a
> special object that represent the result of compilation, and the
> object can be given to any function that accept a regular expression.

Does it really help?  I think the current emacs caches compiled regexps,
which I expect should achieve a similar speedup without any interface
change (if typical regexp usage is as I imagine it is, with a fairly
small number of regexps accounting for most of the usage).

-Miles
-- 
`To alcohol!  The cause of, and solution to,
 all of life's problems' --Homer J. Simpson

Re: CVS directories in completion-ignored-extensions

[Kenichi Handa]

In article <buo3cat4zak.fsf@mcspd15.ucom.lsi.nec.co.jp>, Miles Bader <miles@lsi.nec.co.jp> writes:
> Kenichi Handa <handa@m17n.org> writes:
>>  I remember one feature implemented only in the old mule.  That is
>>  `regexp-compile'.  It compiles a regular expression and returns a
>>  special object that represent the result of compilation, and the
>>  object can be given to any function that accept a regular expression.

> Does it really help?  I think the current emacs caches compiled regexps,
> which I expect should achieve a similar speedup without any interface
> change (if typical regexp usage is as I imagine it is, with a fairly
> small number of regexps accounting for most of the usage).

I agree that, for the moment, the current size of
REGEXP_CACHE_SIZE (== 20) is good enough.  But, for
instance, regular expression is heavily used for displaying
such complicated script as Indic to genarate composition.
Gnus also uses regexp frequently.  If they are mixed
(i.e. for people reading Devanagari mails in Gnus), I'm sure
the size 20 is not enough.

But, hmmm, perhaps 40 is enough.  I have to estimate it.

---
Ken'ichi HANDA
handa@m17n.org

Re: CVS directories in completion-ignored-extensions

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Date: 06 Jan 2004 09:08:25 +0900
> 
> Ok, how about something like:
> 
>    (defun string-matcher-member (string matcher)
>      "Return non-nil if STRING is matched by the `string matcher' MATCHER.
>    MATCHER should be a list, where each element is either a string,
>    which will be compared with STRING using `equal', or a cons of the
>    form (regexp REGEXP), in which case REGEXP will be compared with
>    STRING using `string-match'.  The actual return value is the first
>    element of MATCHER that matches."

The current completion-ignored-extensions machinery is implemented in
C (see dired.c).  I'm worried about the complexity and performance of
walking a non-trivial list and trying to match a regexp for every
file in a directory returned by `readdir'.

Do these concerns have merit, and if so, should we perhaps rewrite
file_name_completion and/or maybe slightly redesign this feature so it
would be easier to implement efficiently?

Possible alternative designs:

 1) Use shell wildcards instead of regexps, and make
    `file_name_completion' use `fnmatch' instead of regexp matching.

 2) Restructure `file_name_completion', so that it doesn't need to
    compile the regexps for each file name it examines.

Thoughts?

Re: CVS directories in completion-ignored-extensions

[Miles Bader]

On Tue, Jan 06, 2004 at 07:55:52AM +0200, Eli Zaretskii wrote:
> The current completion-ignored-extensions machinery is implemented in
> C (see dired.c).  I'm worried about the complexity and performance of
> walking a non-trivial list and trying to match a regexp for every
> file in a directory returned by `readdir'.

I think that's an orthogonal issue -- obviously walking the list isn't too
big a problem (the function example I gave is obviously simple enough to
implement in C if that would be desirable), and if regexps prove to be a
problem, we don't have to use them by default (though my impression was that
as long as they're in the compiled-regexp cache, regexps are actually pretty
darn fast).

I was more interested in the general concept of `string matcher' lists like
this, as I believe they are a very good interface for this sort of thing
(especially for the user -- easy to customize, very clear in raw-lisp, avoids
having dones of `almost the same' variables, etc).

> Possible alternative designs:
> 
>  1) Use shell wildcards instead of regexps, and make
>     `file_name_completion' use `fnmatch' instead of regexp matching.

Note that this is the beauty of the approach I'm advocating -- you can have
multiple match types without adding tons of new variables.

So for instance, we could say that a cons like (shell-regexp SHELL_REGEXP)
used fnmatch or whatever; then the user can use the interface he likes
(which I suppose might often be the case).

>  2) Restructure `file_name_completion', so that it doesn't need to
>     compile the regexps for each file name it examines.

AFAIK, the compiled-regexp cache should make this not much of a problem
anyway (though I'm not really up-to-date on the details).

-Miles
-- 
Freedom's just another word, for nothing left to lose   --Janis Joplin

Re: CVS directories in completion-ignored-extensions

[Richard Stallman]

The idea is ok, but that function name is not very good.

Re: CVS directories in completion-ignored-extensions

[Kim F. Storm]

Kenichi Handa <handa@m17n.org> writes:

> I agree that, for the moment, the current size of
> REGEXP_CACHE_SIZE (== 20) is good enough.  But, for
> instance, regular expression is heavily used for displaying
> such complicated script as Indic to genarate composition.
> Gnus also uses regexp frequently.  If they are mixed
> (i.e. for people reading Devanagari mails in Gnus), I'm sure
> the size 20 is not enough.
>
> But, hmmm, perhaps 40 is enough.  I have to estimate it.

Did you make some conclusion here?

To me it seems like the regexp cache could be improved by
keeping a usage counter and a lru timestamp so frequently
used regexps could be kept in the cache -- and the cache
could grow dynamically to always hold the most frequently
used regexps [with some hard upper limit of course].

E.g. if I have a large number of regexps to split mail in gnus,
it would be nice to keep all of those in the cache.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: CVS directories in completion-ignored-extensions

[Stefan Monnier]

> To me it seems like the regexp cache could be improved by
> keeping a usage counter and a lru timestamp so frequently
> used regexps could be kept in the cache -- and the cache
> could grow dynamically to always hold the most frequently
> used regexps [with some hard upper limit of course].

A cheap way to approximate something like that is to keep a `hit' flag
indicating whether this cache entry has ever been re-used.  Upon GC remove
all entries for which `hit' is false and reset all `hit' flags to false.

I.e. only those entries that have been reused since the last GC survive.

> E.g. if I have a large number of regexps to split mail in gnus,
> it would be nice to keep all of those in the cache.

But of course a larger cache means slower lookup.


        Stefan "who likes the idea of explicit regexp-compilation to
                regexp-objects"

Re: CVS directories in completion-ignored-extensions

[Kim F. Storm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> To me it seems like the regexp cache could be improved by
>> keeping a usage counter and a lru timestamp so frequently
>> used regexps could be kept in the cache -- and the cache
>> could grow dynamically to always hold the most frequently
>> used regexps [with some hard upper limit of course].
>
> A cheap way to approximate something like that is to keep a `hit' flag
> indicating whether this cache entry has ever been re-used.  Upon GC remove
> all entries for which `hit' is false and reset all `hit' flags to false.

And just reuse entries in the current LRC (least recently compiled)
way if we reach the hard limit.

>
> I.e. only those entries that have been reused since the last GC survive.
>
>> E.g. if I have a large number of regexps to split mail in gnus,
>> it would be nice to keep all of those in the cache.
>
> But of course a larger cache means slower lookup.

We can optimize that quite a bit, e.g. by hashing on SCHARS (pattern).

We could also optimize this test:

      if (SCHARS (cp->regexp) == SCHARS (pattern)
	  && STRING_MULTIBYTE (cp->regexp) == STRING_MULTIBYTE (pattern)
	  && !NILP (Fstring_equal (cp->regexp, pattern))
          ...

with this test:

      if ((EQ (cp->regexp, pattern)
           || (SCHARS (cp->regexp) == SCHARS (pattern)
               && STRING_MULTIBYTE (cp->regexp) == STRING_MULTIBYTE (pattern)
	       && !NILP (Fstring_equal (cp->regexp, pattern))))
          ...

--
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: CVS directories in completion-ignored-extensions

[Stefan Monnier]

>> A cheap way to approximate something like that is to keep a `hit' flag
>> indicating whether this cache entry has ever been re-used.  Upon GC remove
>> all entries for which `hit' is false and reset all `hit' flags to false.

> And just reuse entries in the current LRC (least recently compiled)
> way if we reach the hard limit.

Right, and the hard limit can be adjusted after each GC to be N times the
number of entries that the GC preserved.

> We could also optimize this test:

>       if (SCHARS (cp->regexp) == SCHARS (pattern)
> 	  && STRING_MULTIBYTE (cp->regexp) == STRING_MULTIBYTE (pattern)
> 	  && !NILP (Fstring_equal (cp->regexp, pattern))
>           ...

> with this test:

>       if ((EQ (cp->regexp, pattern)
>            || (SCHARS (cp->regexp) == SCHARS (pattern)
>                && STRING_MULTIBYTE (cp->regexp) == STRING_MULTIBYTE (pattern)
> 	       && !NILP (Fstring_equal (cp->regexp, pattern))))
>           ...

I doubt it'll be very beneficial:
If the entry we're looking for is at position P in the list, the EQ test
will fail at least P-1 times and succeed at most once.

If we want to speed up the case of the reuse of the exact same string
object, we could add a text-property to the string holding the index of the
compiled regexp in searchbufs.


        Stefan