Performance problems (CPU 100%) with NULs in files

Performance problems (CPU 100%) with NULs in files

[Ludwig, Mark]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

I frequently encounter data files that are supposed to be 100% ASCII but contain long sequences of NUL characters (ASCII zero).  (The reasons are out of my control.)

When I first started using EMACS [sic] in 1980, I was delighted to find it served as a very fine binary file editor.  It seems that modern Emacs no longer is a fine binary file - at least not by default.

What happens is that as I scroll through the file, when the NULs are visible, Emacs gets into some intensive processing for a long time (minutes, sometimes!).  It eventually unwinds and repaints the display, but any movement of point sends it into this loop again.  I have found that M-< or M-> will quickly reposition away from the problem (assuming the beginning and/or end of the file do not contain NULs).  Most other movement operations send it into the loop.

I understand about encodings, and have messed around with forcing it into us-ascii, but it appears not to be related to this CPU consumption problem.  Does anyone know how to solve this?  I'll file a bug report if this is a legitimate bug.  I'm just concerned that it's a "feature" of some sort, though I hope not.

Thanks!

Mark


[-- Attachment #2: Type: text/html, Size: 3411 bytes --]

RE: Performance problems (CPU 100%) with NULs in files

[Drew Adams]

> What happens is that as I scroll through the file, when
> the NULs are visible, Emacs gets into some intensive
> processing for a long time (minutes, sometimes!).  It
> eventually unwinds and repaints the display, but any
> movement of point sends it into this loop again.  I have
> found that M-< or M-> will quickly reposition away from
> the problem (assuming the beginning and/or end of the file
> do not contain NULs).  Most other movement operations send
> it into the loop.
>
> I understand about encodings, and have messed around with
> forcing it into us-ascii, but it appears not to be related
> to this CPU consumption problem.  Does anyone know how to
> solve this?  I'll file a bug report if this is a legitimate bug.
> I'm just concerned that it's a "feature" of some sort, though
> I hope not.

Are you using the development version of Emacs (aka Emacs 24)?

If so, I have no idea whether this will help or help you find the problem, but
you might try doing this to see if it makes a difference:

(setq-default bidi-display-reordering  nil)

Others might have helpful suggestions.  In any case, especially if you have a
reproducible case starting from `emacs -Q' (no init file), I'd suggest reporting
a bug.  Emacs developers will be able to tell you whether there is really a bug.

Whatever Emacs version you are using, this sounds like a regression wrt the
behavior you report with an older version.

RE: Performance problems (CPU 100%) with NULs in files

[Ludwig, Mark]

> From: Drew Adams [mailto:drew.adams@oracle.com]
> Subject: RE: Performance problems (CPU 100%) with NULs in files
> 
> > What happens is that as I scroll through the file, when
> > the NULs are visible, Emacs gets into some intensive
> > processing for a long time (minutes, sometimes!).  It
> > eventually unwinds and repaints the display, but any
> > movement of point sends it into this loop again.  I have
> > found that M-< or M-> will quickly reposition away from
> > the problem (assuming the beginning and/or end of the file
> > do not contain NULs).  Most other movement operations send
> > it into the loop.
> >
> > I understand about encodings, and have messed around with
> > forcing it into us-ascii, but it appears not to be related
> > to this CPU consumption problem.  Does anyone know how to
> > solve this?  I'll file a bug report if this is a legitimate bug.
> > I'm just concerned that it's a "feature" of some sort, though
> > I hope not.
> 
> Are you using the development version of Emacs (aka Emacs 24)?

No, I'm still on Emacs 23, but this behavior has been happening since at least Emacs 21, and perhaps longer than that.  (I've just never bothered to report it, because it wasn't affecting me very often.)

> If so, I have no idea whether this will help or help you find the
> problem, but
> you might try doing this to see if it makes a difference:
> 
> (setq-default bidi-display-reordering  nil)

This variable doesn't seem to exist in Emacs 23, which you probably were implying.  I tried setting it anyway, and it had no effect.  (Is it possible to have undefined/invisible variables that the code uses if they are brought into existence?)

> Others might have helpful suggestions.  In any case, especially if you
> have a
> reproducible case starting from `emacs -Q' (no init file), I'd suggest
> reporting
> a bug.  Emacs developers will be able to tell you whether there is
> really a bug.

The problem does reproduce with -Q.  It only happens where there are a significant number of NULs, though.  I am narrowing down the range to get to a minimal test case that shows the problem.

> Whatever Emacs version you are using, this sounds like a regression wrt
> the
> behavior you report with an older version.

Hehe.  I have no idea what Emacs version did NOT do this, other than being certain that it did not happen in the TECO-based EMACS in the 1980's....

Cheers,

Mark

Re: Performance problems (CPU 100%) with NULs in files

[Eli Zaretskii]

> From: "Ludwig, Mark" <ludwig.mark@siemens.com>
> Thread-Topic: Performance problems (CPU 100%) with NULs in files
> Date: Wed, 21 Sep 2011 21:08:42 +0000
> 
> What happens is that as I scroll through the file, when the NULs are visible, Emacs gets into some intensive processing for a long time (minutes, sometimes!).  It eventually unwinds and repaints the display, but any movement of point sends it into this loop again.  I have found that M-< or M-> will quickly reposition away from the problem (assuming the beginning and/or end of the file do not contain NULs).  Most other movement operations send it into the loop.

Does it help to visit such files without code conversions, i.e.

  M-x find-file-literally RET FILENAME RET

?

If not, please file a bug report and attach to it an example file that
causes this slowdown.

RE: Performance problems (CPU 100%) with NULs in files

[Ludwig, Mark]

> From: Eli Zaretskii
> Subject: Re: Performance problems (CPU 100%) with NULs in files
> 
> > From: "Ludwig, Mark" <ludwig.mark@siemens.com>
> > Thread-Topic: Performance problems (CPU 100%) with NULs in files
> > Date: Wed, 21 Sep 2011 21:08:42 +0000
> >
> > What happens is that as I scroll through the file, when the NULs are
> visible, Emacs gets into some intensive processing for a long time
> (minutes, sometimes!).  It eventually unwinds and repaints the display,
> but any movement of point sends it into this loop again.  I have found
> that M-< or M-> will quickly reposition away from the problem (assuming
> the beginning and/or end of the file do not contain NULs).  Most other
> movement operations send it into the loop.
> 
> Does it help to visit such files without code conversions, i.e.
> 
>   M-x find-file-literally RET FILENAME RET
> 
> ?

No, it doesn't, but when doing this, I noticed that something else that happens is that the displayed line number changes to "??" when point gets into the NULs.  I turned off line-number-mode, but that had no effect either.  (I could imagine the code would waste a lot of time looking for end-of-line characters in those NULs....)

> If not, please file a bug report and attach to it an example file that
> causes this slowdown.

As soon as I get a minimal test case that shows the problem, I will.

Thanks,

Mark

RE: Performance problems (CPU 100%) with NULs in files

[Ludwig, Mark]

> From: Eli Zaretskii
> Subject: Re: Performance problems (CPU 100%) with NULs in files
> 
> > From: "Ludwig, Mark" <ludwig.mark@siemens.com>
> > Thread-Topic: Performance problems (CPU 100%) with NULs in files
> > Date: Wed, 21 Sep 2011 21:08:42 +0000
> >
> > What happens is that as I scroll through the file, when the NULs are
> visible, Emacs gets into some intensive processing for a long time
> (minutes, sometimes!).  It eventually unwinds and repaints the display,
> but any movement of point sends it into this loop again.  I have found
> that M-< or M-> will quickly reposition away from the problem (assuming
> the beginning and/or end of the file do not contain NULs).  Most other
> movement operations send it into the loop.
> 
> Does it help to visit such files without code conversions, i.e.
> 
>   M-x find-file-literally RET FILENAME RET
> 
> ?
> 
> If not, please file a bug report and attach to it an example file that
> causes this slowdown.

Thanks for the advice, but I have investigated and decided this is probably too unusual to expect any "fix" for it.

What I have found is that the "problem" is due to a "line" of text being extremely long.  In the test file I have, it is ~800,000 characters (bytes).  (It came to me with NULs, but I can replace those with any other printable character and get the same result.)

What I find is that some movement actions are rather slow -- take 4-7 seconds -- while others are extremely quick.  Specifically, the "forward" movement actions (C-e, M-f) are slow, while the "backward" movement actions (C-a, M-b) are instantaneous.  Reposition (C-l) is also slow, as are the line-oriented commands (C-p, C-n).

Thinking through the magnitude of the oddity, I don't think it would be reasonable to expect Emacs to handle this any better than it does.  It's just gratifying that C-g works, so I can interrupt it when I stumble into some junk, and now that I know which actions are fast and slow, I can work around it.

OTOH, if you guys really think this is worth asking any developer to fix, I'll file a bug report.  I don't need to send any data, because it's easy to reproduce this behavior starting with an empty buffer.

Thanks!

Mark

Re: Performance problems (CPU 100%) with NULs in files

[Le Wang]

[-- Attachment #1: Type: text/plain, Size: 2600 bytes --]

File a bug.  Let the powers that be decide whether is fix worthy or not.

This way even if it's closed as wont fix, at least when someone does a bug
search in the future they'll see that a similar issue has come up.

On Sat, Sep 24, 2011 at 8:20 AM, Ludwig, Mark <ludwig.mark@siemens.com>wrote:

> > From: Eli Zaretskii
> > Subject: Re: Performance problems (CPU 100%) with NULs in files
> >
> > > From: "Ludwig, Mark" <ludwig.mark@siemens.com>
> > > Thread-Topic: Performance problems (CPU 100%) with NULs in files
> > > Date: Wed, 21 Sep 2011 21:08:42 +0000
> > >
> > > What happens is that as I scroll through the file, when the NULs are
> > visible, Emacs gets into some intensive processing for a long time
> > (minutes, sometimes!).  It eventually unwinds and repaints the display,
> > but any movement of point sends it into this loop again.  I have found
> > that M-< or M-> will quickly reposition away from the problem (assuming
> > the beginning and/or end of the file do not contain NULs).  Most other
> > movement operations send it into the loop.
> >
> > Does it help to visit such files without code conversions, i.e.
> >
> >   M-x find-file-literally RET FILENAME RET
> >
> > ?
> >
> > If not, please file a bug report and attach to it an example file that
> > causes this slowdown.
>
> Thanks for the advice, but I have investigated and decided this is probably
> too unusual to expect any "fix" for it.
>
> What I have found is that the "problem" is due to a "line" of text being
> extremely long.  In the test file I have, it is ~800,000 characters (bytes).
>  (It came to me with NULs, but I can replace those with any other printable
> character and get the same result.)
>
> What I find is that some movement actions are rather slow -- take 4-7
> seconds -- while others are extremely quick.  Specifically, the "forward"
> movement actions (C-e, M-f) are slow, while the "backward" movement actions
> (C-a, M-b) are instantaneous.  Reposition (C-l) is also slow, as are the
> line-oriented commands (C-p, C-n).
>
> Thinking through the magnitude of the oddity, I don't think it would be
> reasonable to expect Emacs to handle this any better than it does.  It's
> just gratifying that C-g works, so I can interrupt it when I stumble into
> some junk, and now that I know which actions are fast and slow, I can work
> around it.
>
> OTOH, if you guys really think this is worth asking any developer to fix,
> I'll file a bug report.  I don't need to send any data, because it's easy to
> reproduce this behavior starting with an empty buffer.
>
> Thanks!
>
> Mark
>
>
>


-- 
Le

[-- Attachment #2: Type: text/html, Size: 3276 bytes --]

Re: Performance problems (CPU 100%) with NULs in files

[Eli Zaretskii]

> From: "Ludwig, Mark" <ludwig.mark@siemens.com>
> Date: Sat, 24 Sep 2011 00:20:24 +0000
> 
> What I have found is that the "problem" is due to a "line" of text being extremely long.  In the test file I have, it is ~800,000 characters (bytes).  (It came to me with NULs, but I can replace those with any other printable character and get the same result.)

Yes, this is a well-known case that causes the current display engine
become very slow.  IIRC, it has been like that since Emacs 21.1.

> OTOH, if you guys really think this is worth asking any developer to fix, I'll file a bug report.  I don't need to send any data, because it's easy to reproduce this behavior starting with an empty buffer.

Please file a bug report.  It is not unreasonable to expect the code
to be optimized in some way to cater to such use cases.

And thanks for taking your time to investigate this.

RE: Performance problems (CPU 100%) with NULs in files

[Ludwig, Mark]

I filed bug#9589 before your message.  Since you know about this sort of problem generally, I assume the description will make sense.

Cheers,

Mark

-----Original Message-----
From: help-gnu-emacs-bounces+ludwig.mark=siemens.com@gnu.org [mailto:help-gnu-emacs-bounces+ludwig.mark=siemens.com@gnu.org] On Behalf Of Eli Zaretskii
Sent: Saturday, September 24, 2011 1:04 AM
To: help-gnu-emacs@gnu.org
Subject: Re: Performance problems (CPU 100%) with NULs in files

> From: "Ludwig, Mark" <ludwig.mark@siemens.com>
> Date: Sat, 24 Sep 2011 00:20:24 +0000
> 
> What I have found is that the "problem" is due to a "line" of text being extremely long.  In the test file I have, it is ~800,000 characters (bytes).  (It came to me with NULs, but I can replace those with any other printable character and get the same result.)

Yes, this is a well-known case that causes the current display engine
become very slow.  IIRC, it has been like that since Emacs 21.1.

> OTOH, if you guys really think this is worth asking any developer to fix, I'll file a bug report.  I don't need to send any data, because it's easy to reproduce this behavior starting with an empty buffer.

Please file a bug report.  It is not unreasonable to expect the code
to be optimized in some way to cater to such use cases.

And thanks for taking your time to investigate this.

Re: Performance problems (CPU 100%) with NULs in files

[Eli Zaretskii]

> From: "Ludwig, Mark" <ludwig.mark@siemens.com>
> Date: Sat, 24 Sep 2011 13:32:10 +0000
> 
> I filed bug#9589 before your message.  Since you know about this sort of problem generally, I assume the description will make sense.

Yes, thanks.

Re: Performance problems (CPU 100%) with NULs in files

[XeCycle]

[-- Attachment #1: Type: text/plain, Size: 1520 bytes --]

"Ludwig, Mark" <ludwig.mark@siemens.com> writes:

> I frequently encounter data files that are supposed to be 100%
> ASCII but contain long sequences of NUL characters (ASCII zero).
> (The reasons are out of my control.)
>
> When I first started using EMACS [sic] in 1980, I was delighted
> to find it served as a very fine binary file editor.  It seems
> that modern Emacs no longer is a fine binary file – at least not
> by default.
>
> What happens is that as I scroll through the file, when the NULs
> are visible, Emacs gets into some intensive processing for a long
> time (minutes, sometimes!).  It eventually unwinds and repaints
> the display, but any movement of point sends it into this loop
> again.  I have found that M-< or M-> will quickly reposition away
> from the problem (assuming the beginning and/or end of the file
> do not contain NULs).  Most other movement operations send it
> into the loop.
>
> I understand about encodings, and have messed around with forcing
> it into us-ascii, but it appears not to be related to this CPU
> consumption problem.  Does anyone know how to solve this?  I’ll
> file a bug report if this is a legitimate bug.  I’m just
> concerned that it’s a “feature” of some sort, though I hope not.
>
> Thanks!
> Mark

What about trying `hexl-mode'?

-- 
Carl Lei (XeCycle)
Department of Physics, Shanghai Jiao Tong University
OpenPGP public key: 7795E591
Fingerprint: 1FB6 7F1F D45D F681 C845 27F7 8D71 8EC4 7795 E591

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

RE: Performance problems (CPU 100%) with NULs in files

[Ludwig, Mark]

> From: Carl Lei (XeCycle)
> Sent: Thursday, September 22, 2011 1:10 AM
> Subject: Re: Performance problems (CPU 100%) with NULs in files
> 
> "Ludwig, Mark" <ludwig.mark@siemens.com> writes:
> 
> > I frequently encounter data files that are supposed to be 100%
> > ASCII but contain long sequences of NUL characters (ASCII zero).
> > (The reasons are out of my control.)
> >
> > When I first started using EMACS [sic] in 1980, I was delighted
> > to find it served as a very fine binary file editor.  It seems
> > that modern Emacs no longer is a fine binary file – at least not
> > by default.
> >
> > What happens is that as I scroll through the file, when the NULs
> > are visible, Emacs gets into some intensive processing for a long
> > time (minutes, sometimes!).  It eventually unwinds and repaints
> > the display, but any movement of point sends it into this loop
> > again.  I have found that M-< or M-> will quickly reposition away
> > from the problem (assuming the beginning and/or end of the file
> > do not contain NULs).  Most other movement operations send it
> > into the loop.
> >
> > I understand about encodings, and have messed around with forcing
> > it into us-ascii, but it appears not to be related to this CPU
> > consumption problem.  Does anyone know how to solve this?  I’ll
> > file a bug report if this is a legitimate bug.  I’m just
> > concerned that it’s a “feature” of some sort, though I hope not.
> >
> > Thanks!
> > Mark
> 
> What about trying `hexl-mode'?

Thanks for the pointer.  I was not aware of this mode.  This would be very helpful if I were actually intending to edit binary data.

In this thread, I am discussing simply visiting a file with the intention/belief that it's just ASCII.  I am not trying to change the text at all, just view it, scroll through it, etc.  (These are log files that should just be ASCII.)

Cheers,

Mark