Regex to match lines with a specific number of words

Regex to match lines with a specific number of words

[Joost Kremers]

Hi all,

I've been trying to come up with a regex that will match any line containing at
least 30 words in order to kill them from the buffer (preferably with
`kill-matching-lines`, because I need to move the lines to another buffer.)

Frustratingly enough, I have not been successful. Since "word" here can be
interpreted very broadly, I thought this would be easy. Any sequence of
non-whitespace characters surrounded by whitespace can be considered a "word"
(even if it's a number of some special character such as & or #.) So I did this:

\([^[:space:]]+[[:space:]]+\)

This seems to capture a word (in the above sense) plus any following white space
well enough.

But when I try to modify the regex to only match those lines that repeat this
pattern at least 30 times, it fails:

\([^[:space:]]+[[:space:]]+\)\{30,\}

Passing this to `flush-lines` simply deletes everything in the buffer starting
at point, telling me it "[d]eleted 1 matching line", even though (many) more
lines were deleted. Adding ^ and $ around the regex didn't have any effect.

So what am I doing wrong here? 


-- 
Joost Kremers
Life has its moments

Re: Regex to match lines with a specific number of words

[Thibaut Verron]

Hi,

The group [:space:] also matches newline characters. So your search has
exactly one match, spanning many lines.
You can use [:blank:] instead to match spaces and tabs only, for the
separator.

It's probably better to keep [^[:space:]] for the first group, you wouldn't
want to start matching newlines there.

Best wishes,
Thibaut

Le sam. 23 avr. 2022 à 22:39, Joost Kremers <joostkremers@fastmail.fm> a
écrit :

> Hi all,
>
> I've been trying to come up with a regex that will match any line
> containing at
> least 30 words in order to kill them from the buffer (preferably with
> `kill-matching-lines`, because I need to move the lines to another buffer.)
>
> Frustratingly enough, I have not been successful. Since "word" here can be
> interpreted very broadly, I thought this would be easy. Any sequence of
> non-whitespace characters surrounded by whitespace can be considered a
> "word"
> (even if it's a number of some special character such as & or #.) So I did
> this:
>
> \([^[:space:]]+[[:space:]]+\)
>
> This seems to capture a word (in the above sense) plus any following white
> space
> well enough.
>
> But when I try to modify the regex to only match those lines that repeat
> this
> pattern at least 30 times, it fails:
>
> \([^[:space:]]+[[:space:]]+\)\{30,\}
>
> Passing this to `flush-lines` simply deletes everything in the buffer
> starting
> at point, telling me it "[d]eleted 1 matching line", even though (many)
> more
> lines were deleted. Adding ^ and $ around the regex didn't have any effect.
>
> So what am I doing wrong here?
>
>
> --
> Joost Kremers
> Life has its moments
>
>

Re: Regex to match lines with a specific number of words

[Joost Kremers]

On Sat, Apr 23 2022, Thibaut Verron wrote:
> The group [:space:] also matches newline characters. So your search has
> exactly one match, spanning many lines.
> You can use [:blank:] instead to match spaces and tabs only, for the
> separator.

Thanks! I never would have thought of that. (Why isn't this mentioned explicitly
in the manual?)

Unfortunately, passing this regexp to `flush-lines` or `kill-matching-lines` in
a file of close to 65000 lines completely cripples Emacs... One CPU core runs up
to 100% and Emacs becomes unresponsive.

Lemme see if a function that goes through the buffer, splits every line on white
space and deletes those that are too long works better.

Thanks, though, for the quick reply!



-- 
Joost Kremers
Life has its moments

Re: Regex to match lines with a specific number of words

[Thibaut Verron]

No problem! The information is in the manual, but hidden behind several
layers of redirection.
I find the emacswiki page on regular expressions both more synthetic and
more informative.

Regarding performances, that's a bit strange.
Is it better if you add ^ and $ around the expression? Or if you add only ^
and search for exactly 30 repetitions (not 30 or more)?

Best wishes,
Thibaut

Le sam. 23 avr. 2022 à 23:34, Joost Kremers <joostkremers@fastmail.fm> a
écrit :

>
> On Sat, Apr 23 2022, Thibaut Verron wrote:
> > The group [:space:] also matches newline characters. So your search has
> > exactly one match, spanning many lines.
> > You can use [:blank:] instead to match spaces and tabs only, for the
> > separator.
>
> Thanks! I never would have thought of that. (Why isn't this mentioned
> explicitly
> in the manual?)
>
> Unfortunately, passing this regexp to `flush-lines` or
> `kill-matching-lines` in
> a file of close to 65000 lines completely cripples Emacs... One CPU core
> runs up
> to 100% and Emacs becomes unresponsive.
>
> Lemme see if a function that goes through the buffer, splits every line on
> white
> space and deletes those that are too long works better.
>
> Thanks, though, for the quick reply!
>
>
>
> --
> Joost Kremers
> Life has its moments
>

RE: [External] : Re: Regex to match lines with a specific number of words

[Drew Adams]

> The information is in the manual, but hidden
> behind several layers of redirection.
> I find the emacswiki page on regular expressions 
> both more synthetic and more informative.

https://www.emacswiki.org/emacs/CategoryRegexp

Re: Regex to match lines with a specific number of words

[Joost Kremers]

On Sat, Apr 23 2022, Thibaut Verron wrote:
> No problem! The information is in the manual, but hidden behind several
> layers of redirection.
> I find the emacswiki page on regular expressions both more synthetic and
> more informative.

Thanks, I'll check it out.

> Regarding performances, that's a bit strange.
> Is it better if you add ^ and $ around the expression? Or if you add only ^
> and search for exactly 30 repetitions (not 30 or more)?

Well, either version still runs up one CPU core to 100%. The only difference
seems to be that they are more easily interruptable with C-g: Emacs responds
immediately, whereas before it would take seconds to respond to C-g and in one
case it did not respond at all. (I ended up killing Emacs when GNOME popped up a
a suggestion to do so...)

> Le sam. 23 avr. 2022 à 23:34, Joost Kremers <joostkremers@fastmail.fm> a
> écrit :

>> Lemme see if a function that goes through the buffer, splits every line on
>> white space and deletes those that are too long works better.

That actually worked well. It still takes a few seconds to run, but really just
a few seconds. And that's including dumping all the extracted lines into a
separate buffer.



-- 
Joost Kremers
Life has its moments

Re: [External] : Re: Regex to match lines with a specific number of words

[Thibaut Verron]

Right, I meant this one: https://www.emacswiki.org/emacs/RegularExpression

Le dim. 24 avr. 2022 à 00:12, Drew Adams <drew.adams@oracle.com> a écrit :

> > The information is in the manual, but hidden
> > behind several layers of redirection.
> > I find the emacswiki page on regular expressions
> > both more synthetic and more informative.
>
> https://www.emacswiki.org/emacs/CategoryRegexp
>

Re: Regex to match lines with a specific number of words

[Stefan Monnier via Users list for the GNU Emacs text editor]

>> \([^[:space:]]+[[:space:]]+\)\{30,\}

I don't think yo want to match more than 30 times: as soon as you have
30 repetitions you know what to do.
And indeed you should likely anchor your regexp at beg of line (but not
end of line, since you want to ignore the rest of the line after matching
30 repetitions).

Note also that regardless if [:space:] matches line-feed or not, one of
[[:space:]] or [^[:space:]] will match line-feed.

Personally, I'd go with something like:

    ^\([^\n\t ]+[\t ]+\)\{30\}

[ Tho, IIUC `flush-lines` doesn't know \t and \n, so you'll probably
  need C-q C-j and C-q TAB.  ]


        Stefan

Re: Regex to match lines with a specific number of words

[Joost Kremers]

On Sat, Apr 23 2022, Stefan Monnier via Users list for the GNU Emacs text editor wrote:
>>> \([^[:space:]]+[[:space:]]+\)\{30,\}
>
> I don't think yo want to match more than 30 times: as soon as you have
> 30 repetitions you know what to do.

True. I modelled it on a regex for SublimeText, which, however, was used in a
search and replace operation (where the replacement string was empty), so
matching more than 30 times was necessary. But for flush-lines it's not.

[...]
> Personally, I'd go with something like:
>
>     ^\([^\n\t ]+[\t ]+\)\{30\}

Thanks, I'll try that and see if it gives any speed advantages.


-- 
Joost Kremers
Life has its moments

Re: Regex to match lines with a specific number of words

[Nick Dokos]

Joost Kremers <joostkremers@fastmail.fm> writes:


> I've been trying to come up with a regex that will match any line containing at
> least 30 words in order to kill them from the buffer (preferably with
> `kill-matching-lines`, because I need to move the lines to another buffer.)
>
> ...
>
> Passing this to `flush-lines` simply deletes everything in the buffer starting
> at point, telling me it "[d]eleted 1 matching line", even though (many) more
> lines were deleted. Adding ^ and $ around the regex didn't have any effect.
>

Do you have to do it in emacs? Why not use `awk'? It could be as simple as

awk 'NF < 30  {print;}'

and saving the other lines to a different file is not much more difficult:

awk 'NF < 5 {print;}
NF >= 5 { print > "long-stuff.txt"}'

-- 
Nick

"There are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors." -Martin Fowler

Re: Regex to match lines with a specific number of words

[Jean Louis]

I would solve that problem first by making a list of lines with more
than 30 words, each line being designated, and then searching within
those lines.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/