occur API change proposal

occur API change proposal

[Ted Zlatanov]

This can wait until after the next release.

I propose changing the occur API to allow an optional predicate
function.  For example, this is useful to invert a regular expression so
you don't have to run occur on the whole buffer matching everything,
then limit the results.

Ted

Re: occur API change proposal

[Juri Linkov]

> I propose changing the occur API to allow an optional predicate
> function.  For example, this is useful to invert a regular expression so
> you don't have to run occur on the whole buffer matching everything,
> then limit the results.

The occur engine currently hard-codes the function call of
`re-search-forward'.  Instead of this, we could do the same
as is done already in isearch for customization of the search
function using a variable `isearch-search-fun-function', and
add a new analogous variable `occur-fun-function'.

Even though such a variable will be useful for other purposes,
no simple function will allow inverting a regular expression.
Implementing this feature requires significant modification in
the logic of the occur engine: either changing the algorithm to
call `looking-at' at every line (very inefficient), or recording
a position of the previous match and outputing lines between it
and the current position (not easy to customize).

-- 
Juri Linkov
http://www.jurta.org/emacs/

add INVERT to re-search-{forward, backward} (was: occur API change proposal)

[Ted Zlatanov]

On Sat, 01 Dec 2007 02:34:35 +0200 Juri Linkov <juri@jurta.org> wrote: 

>> I propose changing the occur API to allow an optional predicate
>> function.  For example, this is useful to invert a regular expression so
>> you don't have to run occur on the whole buffer matching everything,
>> then limit the results.

JL> The occur engine currently hard-codes the function call of
JL> `re-search-forward'.  Instead of this, we could do the same
JL> as is done already in isearch for customization of the search
JL> function using a variable `isearch-search-fun-function', and
JL> add a new analogous variable `occur-fun-function'.

JL> Even though such a variable will be useful for other purposes,
JL> no simple function will allow inverting a regular expression.
JL> Implementing this feature requires significant modification in
JL> the logic of the occur engine: either changing the algorithm to
JL> call `looking-at' at every line (very inefficient), or recording
JL> a position of the previous match and outputing lines between it
JL> and the current position (not easy to customize).

After thinking about this (yes, it sat in my TODO queue for a while) I
realize the problem is fundamental: Emacs doesn't have a way to invert
at the re-search-{forward,backward} API level.  There's no way to
automatically invert a regular expression into another regular
expression, so this change must happen at the API level to avoid
unpleasant workarounds as Juri describes.

I propose simply adding an optional INVERT parameter to those two
functions.  The API cost is small with an optional parameter, though I
don't know if performance will be affected by the extra check on every
search cycle.

Once this API is in place, making the change in occur-mode and other
places to use the new API is very easy.

Ted

Re: add INVERT to re-search-{forward, backward} (was: occur API change proposal)

[Juri Linkov]

>>> I propose changing the occur API to allow an optional predicate
>>> function.  For example, this is useful to invert a regular expression so
>>> you don't have to run occur on the whole buffer matching everything,
>>> then limit the results.
>
> JL> The occur engine currently hard-codes the function call of
> JL> `re-search-forward'.  Instead of this, we could do the same
> JL> as is done already in isearch for customization of the search
> JL> function using a variable `isearch-search-fun-function', and
> JL> add a new analogous variable `occur-fun-function'.
>
> JL> Even though such a variable will be useful for other purposes,
> JL> no simple function will allow inverting a regular expression.
> JL> Implementing this feature requires significant modification in
> JL> the logic of the occur engine: either changing the algorithm to
> JL> call `looking-at' at every line (very inefficient), or recording
> JL> a position of the previous match and outputing lines between it
> JL> and the current position (not easy to customize).
>
> After thinking about this (yes, it sat in my TODO queue for a while) I
> realize the problem is fundamental: Emacs doesn't have a way to invert
> at the re-search-{forward,backward} API level.  There's no way to
> automatically invert a regular expression into another regular
> expression, so this change must happen at the API level to avoid
> unpleasant workarounds as Juri describes.
>
> I propose simply adding an optional INVERT parameter to those two
> functions.  The API cost is small with an optional parameter, though I
> don't know if performance will be affected by the extra check on every
> search cycle.
>
> Once this API is in place, making the change in occur-mode and other
> places to use the new API is very easy.

It seems such an API with a new INVERT parameter will be useless
because these two character-based search functions will find too much
matches. For example, calling it with the parameter "def" on the string
"abcdefghi" will match in it 6 places, i.e. everywhere except the
position under the character "d".  Thus this will provide no useful
information to occur.

Instead of this, I think the change should be done at the occur level,
since occur's algorithm is line-based and can skip lines that contain
a given regexp.  This could be similar to the algorithm used by the
command `flush-lines' in the same file.  `occur' currently works like the
command `keep-lines' that keeps lines containing the specified regexp.
But with a new INVERT argument, `occur' could work like `flush-lines' that
keep everything except lines containing the specified regexp.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: add INVERT to re-search-{forward, backward}

[Lennart Borgman (gmail)]

Juri Linkov wrote:
> Instead of this, I think the change should be done at the occur level,
> since occur's algorithm is line-based and can skip lines that contain
> a given regexp.  This could be similar to the algorithm used by the
> command `flush-lines' in the same file.  `occur' currently works like the
> command `keep-lines' that keeps lines containing the specified regexp.
> But with a new INVERT argument, `occur' could work like `flush-lines' that
> keep everything except lines containing the specified regexp.

This is also similar to grep etc.

Re: add INVERT to re-search-{forward, backward}

[Juri Linkov]

>> Instead of this, I think the change should be done at the occur level,
>> since occur's algorithm is line-based and can skip lines that contain
>> a given regexp.  This could be similar to the algorithm used by the
>> command `flush-lines' in the same file.  `occur' currently works like the
>> command `keep-lines' that keeps lines containing the specified regexp.
>> But with a new INVERT argument, `occur' could work like `flush-lines' that
>> keep everything except lines containing the specified regexp.
>
> This is also similar to grep etc.

Yes, as a line-oriented tool, grep's -v --invert-match option is what
could be modelled in occur.

Then a negative prefix argument of `occur' could mean inverting the
regexp and displaying non-matching lines.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: add INVERT to re-search-{forward, backward}

[Lennart Borgman (gmail)]

Juri Linkov wrote:
>>> Instead of this, I think the change should be done at the occur level,
>>> since occur's algorithm is line-based and can skip lines that contain
>>> a given regexp.  This could be similar to the algorithm used by the
>>> command `flush-lines' in the same file.  `occur' currently works like the
>>> command `keep-lines' that keeps lines containing the specified regexp.
>>> But with a new INVERT argument, `occur' could work like `flush-lines' that
>>> keep everything except lines containing the specified regexp.
>> This is also similar to grep etc.
> 
> Yes, as a line-oriented tool, grep's -v --invert-match option is what
> could be modelled in occur.
> 
> Then a negative prefix argument of `occur' could mean inverting the
> regexp and displaying non-matching lines.

Maybe it also would be practical to be able to access that function from
within the occur window too?