bug#63272: 29.0.90; xref fails on long lines

bug#63272: 29.0.90; xref fails on long lines

[Juri Linkov]

1. Create a file with a long line, e.g. type

   a C-u 500000 b c

Save the file and commit to git.
(long-line-optimizations-p returns t)

2. Try to search a regexp that matches the whole long line, e.g.

   C-x p g a.*c RET

Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
  xref--collect-matches-1("a.*c" "/tmp/file" 1 1 500003 nil)
  xref--collect-matches((1 "/tmp/file" "abbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb...") "a.*c" #<buffer  *xref-temp*> nil)
  #f(compiled-function (hit) #<bytecode 0x122b1e80d7055e69>)((1 "/tmp/file" "abbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb..."))
  xref--convert-hits(((1 "/tmp/file" "abbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb...")) "a.*c")
  xref-matches-in-files("a.*c" ("/tmp/file"))
  project--find-regexp-in-files("a.*c" ("/tmp/file"))
  apply(project--find-regexp-in-files ("a.*c" ("/tmp/file")))
  #f(compiled-function (&rest args2) #<bytecode -0xae28f07f9498cbf>)()
  xref--show-xref-buffer(#f(compiled-function (&rest args2) #<bytecode -0xae28f07f9498cbf>) ((window . #<window 3 on tmp>) (display-action) (auto-jump)))
  xref--show-xrefs(#f(compiled-function (&rest args2) #<bytecode -0xae28f07f9498cbf>) nil)
  xref-show-xrefs(#f(compiled-function (&rest args2) #<bytecode -0xae28f07f9498cbf>) nil)
  project-find-regexp("a.*c")
  funcall-interactively(project-find-regexp "a.*c")
  call-interactively(project-find-regexp nil nil)
  command-execute(project-find-regexp)

bug#63272: 29.0.90; xref fails on long lines

[Dmitry Gutov]

On 04/05/2023 18:07, Juri Linkov wrote:
> 1. Create a file with a long line, e.g. type
> 
>     a C-u 500000 b c
> 
> Save the file and commit to git.
> (long-line-optimizations-p returns t)
> 
> 2. Try to search a regexp that matches the whole long line, e.g.
> 
>     C-x p g a.*c RET
> 
> Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")

Isn't that more like a problem with the regexp you entered, or with our 
regexp engine?

E.g. try this:

   C-x p g a[^c]*c RET

bug#63272: 29.0.90; xref fails on long lines

[Gregory Heytings]

>
> 1. Create a file with a long line, e.g. type
>
> a C-u 500000 b c
>
> Save the file and commit to git.
>
> (long-line-optimizations-p returns t)
>
> 2. Try to search a regexp that matches the whole long line, e.g.
>
>   C-x p g a.*c RET
>
> Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
>

That seems to be a problem with/limitation of the regexp engine that is 
not immediately related to (displaying) long lines.  After (setq 
long-line-threshold nil) you will get the same error.

bug#63272: 29.0.90; xref fails on long lines

[Gregory Heytings]

>> 2. Try to search a regexp that matches the whole long line, e.g.
>>
>> C-x p g a.*c RET
>>
>> Debugger entered--Lisp error: (error "Stack overflow in regexp 
>> matcher")
>
> That seems to be a problem with/limitation of the regexp engine that is 
> not immediately related to (displaying) long lines.  After (setq 
> long-line-threshold nil) you will get the same error.
>

By the way, you may want to have a look at bug#61514, in which such 
problematic regexps are discussed.

bug#63272: 29.0.90; xref fails on long lines

[Juri Linkov]

>>> 2. Try to search a regexp that matches the whole long line, e.g.
>>>
>>> C-x p g a.*c RET
>>>
>>> Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
>>
>> That seems to be a problem with/limitation of the regexp engine that is
>> not immediately related to (displaying) long lines.  After (setq
>> long-line-threshold nil) you will get the same error.
>
> By the way, you may want to have a look at bug#61514, in which such
> problematic regexps are discussed.

Thanks for the reference, I see it's fixed, and I can't find more problems.

bug#63272: 29.0.90; xref fails on long lines

[Juri Linkov]

tags 63272 notabug
close 63272 29.0.60
thanks

>> 1. Create a file with a long line, e.g. type
>>     a C-u 500000 b c
>> Save the file and commit to git.
>> (long-line-optimizations-p returns t)
>> 2. Try to search a regexp that matches the whole long line, e.g.
>>     C-x p g a.*c RET
>> Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
>
> Isn't that more like a problem with the regexp you entered, or with our
> regexp engine?
>
> E.g. try this:
>
>   C-x p g a[^c]*c RET

In the real case the prefix and suffix were unique, so I didn't expect
that in a generated file there was a very long distance between prefix
and suffix.  To limit the distance between prefix and suffix I tried:

  C-x p g prefix.{0,100}suffix

but xref that uses ripgrep fails to find matches.
So needed to fall back to ripgrep-based rgrep in this case:

  M-x rgrep prefix.{0,100}suffix

that works successfully.

It seems the problem is because of different regexp syntax
used by ripgrep and re-search-forward in xref--collect-matches-1.

Since this is a separate problem, I'm closing this one,
then a new feature request could be opened if you want.

bug#63272: 29.0.90; xref fails on long lines

[Dmitry Gutov]

On 05/05/2023 20:50, Juri Linkov wrote:
> In the real case the prefix and suffix were unique, so I didn't expect
> that in a generated file there was a very long distance between prefix
> and suffix.  To limit the distance between prefix and suffix I tried:
> 
>    C-x p g prefix.{0,100}suffix

Try escaping { and }:

   C-x p g prefix.\{0,100\}suffix

The regexp needs to use the syntax that Emacs can understand.

Just the subset of it that can be translated to command like, so that 
Grep and Ripgrep can work with it too.

> but xref that uses ripgrep fails to find matches.
> So needed to fall back to ripgrep-based rgrep in this case:
> 
>    M-x rgrep prefix.{0,100}suffix
> 
> that works successfully.
> 
> It seems the problem is because of different regexp syntax
> used by ripgrep and re-search-forward in xref--collect-matches-1.

The conversion is performed by xref--regexp-to-extended.

bug#63272: 29.0.90; xref fails on long lines

[Juri Linkov]

>> In the real case the prefix and suffix were unique, so I didn't expect
>> that in a generated file there was a very long distance between prefix
>> and suffix.  To limit the distance between prefix and suffix I tried:
>>    C-x p g prefix.{0,100}suffix
>
> Try escaping { and }:
>
>   C-x p g prefix.\{0,100\}suffix

Thanks, this works.

> The regexp needs to use the syntax that Emacs can understand.
>
> Just the subset of it that can be translated to command like, so that Grep
> and Ripgrep can work with it too.
>
>> but xref that uses ripgrep fails to find matches.
>> So needed to fall back to ripgrep-based rgrep in this case:
>>    M-x rgrep prefix.{0,100}suffix
>> that works successfully.
>> It seems the problem is because of different regexp syntax
>> used by ripgrep and re-search-forward in xref--collect-matches-1.
>
> The conversion is performed by xref--regexp-to-extended.

Shouldn't the conversion also escape { and } ?

bug#63272: 29.0.90; xref fails on long lines

[Dmitry Gutov]

On 08/05/2023 18:41, Juri Linkov wrote:
>>> In the real case the prefix and suffix were unique, so I didn't expect
>>> that in a generated file there was a very long distance between prefix
>>> and suffix.  To limit the distance between prefix and suffix I tried:
>>>     C-x p g prefix.{0,100}suffix
>>
>> Try escaping { and }:
>>
>>    C-x p g prefix.\{0,100\}suffix
> 
> Thanks, this works.
> 
>> The regexp needs to use the syntax that Emacs can understand.
>>
>> Just the subset of it that can be translated to command like, so that Grep
>> and Ripgrep can work with it too.
>>
>>> but xref that uses ripgrep fails to find matches.
>>> So needed to fall back to ripgrep-based rgrep in this case:
>>>     M-x rgrep prefix.{0,100}suffix
>>> that works successfully.
>>> It seems the problem is because of different regexp syntax
>>> used by ripgrep and re-search-forward in xref--collect-matches-1.
>>
>> The conversion is performed by xref--regexp-to-extended.
> 
> Shouldn't the conversion also escape { and } ?

It _un_escapes them in this case: { and } have to be escaped in Emacs 
regexps, but they don't need to be escaped in "extended regular extensions".

So the conversion toggles escaping.