replace regex takes long

replace regex takes long

[Gary Wessle]

Hi

I have a file with data in the form
date<spac>name<space>age<space>higth<newline>
the <space> are 5 to 20 spaces, so I want to replace them with just
one <space> between fields. so I did
M-x query-replace-regex <RET> +<RET><ONE SPACE>
the file size is 4162111 with 77000 lines on PentiumIII 500, it has 
been 10 minutes so I opened another emacs processor to write and post this.
is there a problem with my command and why it is taking that long? and
for how much longer?

thanks

Re: replace regex takes long

[Peter Dyballa]

Am 15.04.2006 um 23:29 schrieb Gary Wessle:

> M-x query-replace-regex <RET> +<RET><ONE SPACE>
> the file size is 4162111 with 77000 lines on PentiumIII 500, it has
> been 10 minutes so I opened another emacs processor to write and  
> post this.
> is there a problem with my command and why it is taking that long?

Yes, because the command needs your co-operation. And if you don't  
press "yes" no change will be made.

> and for how much longer?

That's really hard to tell! You might fail to finish because of death  
or your fingers failing to type or you might fall asleep. Your  
keyboard might break. Power might fail.

I think replace-regexp can be much faster.

--
Greetings

   Pete

Bake Pizza not war!

Re: replace regex takes long

[Gary Wessle]

Peter Dyballa <Peter_Dyballa@Web.DE> writes:

> Am 15.04.2006 um 23:29 schrieb Gary Wessle:
> 
> > M-x query-replace-regex <RET> +<RET><ONE SPACE>
> > the file size is 4162111 with 77000 lines on PentiumIII 500, it has
> > been 10 minutes so I opened another emacs processor to write and
> > post this.
> > is there a problem with my command and why it is taking that long?
> 
> Yes, because the command needs your co-operation. And if you don't
> press "yes" no change will be made.

if fact I forget to say that I did hit ! which means "replace all"

Re: replace regex takes long

[Gary Wessle]

Gary Wessle <phddas@yahoo.com> writes:

> Peter Dyballa <Peter_Dyballa@Web.DE> writes:
> 
> > Am 15.04.2006 um 23:29 schrieb Gary Wessle:
> > 
> > > M-x query-replace-regex <RET> +<RET><ONE SPACE>
> > > the file size is 4162111 with 77000 lines on PentiumIII 500, it has
> > > been 10 minutes so I opened another emacs processor to write and
> > > post this.
> > > is there a problem with my command and why it is taking that long?
> > 
> > Yes, because the command needs your co-operation. And if you don't
> > press "yes" no change will be made.
> 
> if fact I forget to say that I did hit ! which means "replace all"
with M-x regex-replace it shows (replaced 304172 occurrences) and it
is taking more than 5 minutes now and still have not finished the
task.

Re: replace regex takes long

[Eli Zaretskii]

> From: Gary Wessle <phddas@yahoo.com>
> Date: 16 Apr 2006 09:35:00 +1000
> 
> Gary Wessle <phddas@yahoo.com> writes:
> 
> > Peter Dyballa <Peter_Dyballa@Web.DE> writes:
> > 
> > > Am 15.04.2006 um 23:29 schrieb Gary Wessle:
> > > 
> > > > M-x query-replace-regex <RET> +<RET><ONE SPACE>
> > > > the file size is 4162111 with 77000 lines on PentiumIII 500, it has
> > > > been 10 minutes so I opened another emacs processor to write and
> > > > post this.
> > > > is there a problem with my command and why it is taking that long?
> > > 
> > > Yes, because the command needs your co-operation. And if you don't
> > > press "yes" no change will be made.
> > 
> > if fact I forget to say that I did hit ! which means "replace all"
> with M-x regex-replace it shows (replaced 304172 occurrences) and it
> is taking more than 5 minutes now and still have not finished the
> task.

I just tried query-replace-regex on a file with the structure you
described, and the command ran to completion in just 40 seconds,
including 3 questions whether to discard undo info.  That was on a
3-GHz Pentium 4 in a buffer that is in Fundamental mode.  So your
timing seems very unusual, to say the least.

Do you have some specialized major mode in that buffer?  If so,
perhaps it slows down the replacement; try in Fundamental mode.  I'd
also suggest to try the command in "emacs -q", perhaps something in
your customizations slows down the replacements.

In any case, your command is wasteful because it forces Emacs to
replace even a single space.  I suggest to try this instead:

    M-x query-replace-regex <RET>  +<RET> <RET>
                                 ^^
                             two spaces here

This command takes just 20 seconds on my machine.

Btw, what version of Emacs do you have, and on which OS?

Re: replace regex takes long

[Peter Dyballa]

Am 16.04.2006 um 01:35 schrieb Gary Wessle:

> with M-x regex-replace it shows (replaced 304172 occurrences) and it
> is taking more than 5 minutes now and still have not finished the
> task.

When the message appears in the mini-buffer the task is completed.

Could be your operating system is recovering from the task. 4 MB of  
data are a lot, 304,172 replacements (four in every line) are  
"buffered" for undo's *and* 304,172 times marks where set in the  
buffer *and* areas highlighted and in the end de-highlighted. GNU  
Emacs itself is clearing its programme stack to be prepared for the  
next command.

replace.el explains:

	This function is usually the wrong thing to use in a Lisp program.
	What you probably want is a loop like this:
	  (while (re-search-forward REGEXP nil t)
	    (replace-match TO-STRING nil nil))
	which will run faster and will not set the mark or print anything."

--
Greetings

   Pete     === -Q
              ==<__/% >>
_____________(_)____@_____________________________

Re: replace regex takes long

[Eli Zaretskii]

> From: Peter Dyballa <Peter_Dyballa@Web.DE>
> Date: Sun, 16 Apr 2006 12:17:26 +0200
> Cc: help-gnu-emacs@gnu.org
> 
> replace.el explains:
> 
> 	This function is usually the wrong thing to use in a Lisp program.
> 	What you probably want is a loop like this:
> 	  (while (re-search-forward REGEXP nil t)
> 	    (replace-match TO-STRING nil nil))
> 	which will run faster and will not set the mark or print anything."

This explanation is not really relevant in this case, since the OP
did, in fact, invoke query-replace-regexp interactively, not from a
Lisp program.

Re: replace regex takes long

[Gary Wessle]

Eli Zaretskii <eliz@gnu.org> writes:

> > From: Gary Wessle <phddas@yahoo.com>
> > Date: 16 Apr 2006 09:35:00 +1000
> > 
> > Gary Wessle <phddas@yahoo.com> writes:
> > 
> > > Peter Dyballa <Peter_Dyballa@Web.DE> writes:
> > > 
> > > > Am 15.04.2006 um 23:29 schrieb Gary Wessle:
> > > > 
> > > > > M-x query-replace-regex <RET> +<RET><ONE SPACE>
> > > > > the file size is 4162111 with 77000 lines on PentiumIII 500, it has
> > > > > been 10 minutes so I opened another emacs processor to write and
> > > > > post this.
> > > > > is there a problem with my command and why it is taking that long?
> > > > 
> > > > Yes, because the command needs your co-operation. And if you don't
> > > > press "yes" no change will be made.
> > > 
> > > if fact I forget to say that I did hit ! which means "replace all"
> > with M-x regex-replace it shows (replaced 304172 occurrences) and it
> > is taking more than 5 minutes now and still have not finished the
> > task.
> 
> I just tried query-replace-regex on a file with the structure you
> described, and the command ran to completion in just 40 seconds,
> including 3 questions whether to discard undo info.  That was on a
> 3-GHz Pentium 4 in a buffer that is in Fundamental mode.  So your
> timing seems very unusual, to say the least.
> 
> Do you have some specialized major mode in that buffer?  If so,
> perhaps it slows down the replacement; try in Fundamental mode.  I'd
> also suggest to try the command in "emacs -q", perhaps something in
> your customizations slows down the replacements.
> 
> In any case, your command is wasteful because it forces Emacs to
> replace even a single space.  I suggest to try this instead:
> 
>     M-x query-replace-regex <RET>  +<RET> <RET>
>                                  ^^
>                              two spaces here
> 
> This command takes just 20 seconds on my machine.
> 
> Btw, what version of Emacs do you have, and on which OS?

I did it as you described, 
#emacs -q
M-x query-replace-regex<RET>  +<RET> <RET>
it took 24 sec. my OS is debian testing and
emacs version 21.4.1

thanks

Re: replace regex takes long

[Stefan Monnier]

>> in fact I forget to say that I did hit ! which means "replace all"
> with M-x regex-replace it shows (replaced 304172 occurrences) and it is
> taking more than 5 minutes now and still have not finished the task.

After (replaced 304172 occurrences) is shown, the replacement is
completely done.  So if Emacs is still busy afterwards, it's busy doing
something else.  Could be something related to font-lock, or could be many
other things, but if it stays that way for 5minutes, it looks like a bug, so
please try to investigate to see what's using up so much time and report
your findings via M-x report-emacs-bug.


        Stefan