Emacs opens only first 16384 bytes of file?!

Emacs opens only first 16384 bytes of file?!

[Christoph Groth]

Hello,

I just noticed something that made me doubt my own sanity:  With Emacs
27.1 (from Debian stable running Debian’s Linux kernel
6.0.0-0.deb11.6-amd64) the command

  emacs -q -nw /proc/cpuinfo

reproducibly opens cpuinfo only up to a portion of line 272, while the
entire file is 560 lines long on that machine.  I checked on a different
machine (with same Emacs), and I see the same behavior.

C-u C-x = tells that the file is read up to position 16384 (= 2^14).

Running M-x revert-buffer loads the whole file...

Is/was this a known issue?  I could not find anything on the web.

----------------------------------------------------------------

I’m willing to investigate this further, but it would seem very strange
if Emacs indeed had a bug that prevented it from loading some files
entirely.  Perhaps someone here knows the answer.

Thanks
Christoph

Re: Emacs opens only first 16384 bytes of file?!

[Ruijie Yu via Users list for the GNU Emacs text editor]

Christoph Groth <christoph@grothesque.org> writes:

> Hello,
>
> I just noticed something that made me doubt my own sanity:  With Emacs
> 27.1 (from Debian stable running Debian’s Linux kernel
> 6.0.0-0.deb11.6-amd64) the command
>
>   emacs -q -nw /proc/cpuinfo
>
> reproducibly opens cpuinfo only up to a portion of line 272, while the
> entire file is 560 lines long on that machine.  I checked on a different
> machine (with same Emacs), and I see the same behavior.
>
> C-u C-x = tells that the file is read up to position 16384 (= 2^14).

Can't reproduce on 28.1 nor 29 (8a18369afdc3).  Can you compile 28.1 and
see if the problem goes away?

Since my /proc/cpuinfo is smaller than 16384 bytes, I used the following
steps to try to reproduce the issue:

$ zcat /proc/config.gz > ~/c
$ /path/to/emacs-28.1/src/emacs -Q -nw ~/c
`C-u C-x =' says: "position: 1 of 252720 (0%), column: 0"

> Running M-x revert-buffer loads the whole file...
>
> Is/was this a known issue?  I could not find anything on the web.

In the future you might get more responses if you mail to
bug-gnu-emacs@gnu.org.

> ----------------------------------------------------------------
>
> I’m willing to investigate this further, but it would seem very strange
> if Emacs indeed had a bug that prevented it from loading some files
> entirely.  Perhaps someone here knows the answer.
>
> Thanks
> Christoph

--
Best,


RY

Re: Emacs opens only first 16384 bytes of file?!

[Óscar Fuentes]

Christoph Groth <christoph@grothesque.org> writes:

> Hello,
>
> I just noticed something that made me doubt my own sanity:  With Emacs
> 27.1 (from Debian stable running Debian’s Linux kernel
> 6.0.0-0.deb11.6-amd64) the command
>
>   emacs -q -nw /proc/cpuinfo
>
> reproducibly opens cpuinfo only up to a portion of line 272, while the
> entire file is 560 lines long on that machine.  I checked on a different
> machine (with same Emacs), and I see the same behavior.
>
> C-u C-x = tells that the file is read up to position 16384 (= 2^14).
>
> Running M-x revert-buffer loads the whole file...
>
> Is/was this a known issue?  I could not find anything on the web.
>
> ----------------------------------------------------------------
>
> I’m willing to investigate this further, but it would seem very strange
> if Emacs indeed had a bug that prevented it from loading some files
> entirely.  Perhaps someone here knows the answer.

As of today (Emacs 30, current development branch) the same is true.

/proc/cpuinfo is not a regular file, in the sense that its content is
not stored in a device. AFAIK it is generated on-the-fly when it is
read.

You can do some simple observations:

$ ls -l /proc/cpuinfo 
-r--r--r-- 1 root root 0 feb  7 03:34 /proc/cpuinfo
$ du /proc/cpuinfo 
0       /proc/cpuinfo

Here those tools are saying that the file's size is 0.

I guess that Emacs detects that the file is special and reads its
contents following some heuristics. What surprises me is that M-x
revert-file actually reads all the content.

Of course, looking at the sources would be enlightening, but why do the
effort of actually clearing the matter when it is so cheap to throw
speculation? ;-)

(I looked at insert-file-contents, but bailed out after the fifth
screenful of code. A 1000+ lines function, no kidding.)

Re: Emacs opens only first 16384 bytes of file?!

[Ruijie Yu via Users list for the GNU Emacs text editor]

Óscar Fuentes <ofv@wanadoo.es> writes:

> [...]
> As of today (Emacs 30, current development branch) the same is true.
>
> /proc/cpuinfo is not a regular file, in the sense that its content is
> not stored in a device. AFAIK it is generated on-the-fly when it is
> read.
>
> You can do some simple observations:
>
> $ ls -l /proc/cpuinfo
> -r--r--r-- 1 root root 0 feb  7 03:34 /proc/cpuinfo
> $ du /proc/cpuinfo
> 0       /proc/cpuinfo
>
> Here those tools are saying that the file's size is 0.

Thanks for elaborating on the part that I missed.  So, to reproduce the
issue, we need to find a file which does not show size, and whose
contents are generated on-the-fly when read -- in the original report,
it is /proc/cpuinfo, and I have also found /proc/kallsyms which contains
around 12M of data.  In this case, I have reproduced the issue on my
Emacs 29 build.

> I guess that Emacs detects that the file is special and reads its
> contents following some heuristics. What surprises me is that M-x
> revert-file actually reads all the content.
>
> Of course, looking at the sources would be enlightening, but why do the
> effort of actually clearing the matter when it is so cheap to throw
> speculation? ;-)

I think the limit of 16384 is probably caused by fileio.c:3919
(907fd1f7ff4 somewhere on master, inside DEFUN "insert-file-contents")
which declares a read buffer.  The constant READ_BUF_SIZE is indirectly
set as 16 * 1024 = 16384.  Didn't read further there, nor inside
`revert-buffer'.

> (I looked at insert-file-contents, but bailed out after the fifth
> screenful of code. A 1000+ lines function, no kidding.)
Yeah, that's exactly why I didn't read further there.

--
Best,


RY

Re: Emacs opens only first 16384 bytes of file?!

[Christoph Groth]

Ruijie Yu wrote:
> Óscar Fuentes <ofv@wanadoo.es> writes:
>
> > [...]
> > As of today (Emacs 30, current development branch) the same is true.
> >
> > /proc/cpuinfo is not a regular file, in the sense that its content
> > is not stored in a device. AFAIK it is generated on-the-fly when it
> > is read.
> >
> > You can do some simple observations:
> >
> > $ ls -l /proc/cpuinfo
> > -r--r--r-- 1 root root 0 feb  7 03:34 /proc/cpuinfo
> > $ du /proc/cpuinfo
> > 0       /proc/cpuinfo
> >
> > Here those tools are saying that the file's size is 0.
>
> Thanks for elaborating on the part that I missed.  So, to reproduce
> the issue, we need to find a file which does not show size, and whose
> contents are generated on-the-fly when read -- in the original report,
> it is /proc/cpuinfo, and I have also found /proc/kallsyms which
> contains around 12M of data.  In this case, I have reproduced the
> issue on my Emacs 29 build.

Yes, that’s it, thanks for examining this further!

> > I guess that Emacs detects that the file is special and reads its
> > contents following some heuristics. What surprises me is that M-x
> > revert-file actually reads all the content.
> >
> > Of course, looking at the sources would be enlightening, but why do the
> > effort of actually clearing the matter when it is so cheap to throw
> > speculation? ;-)
>
> I think the limit of 16384 is probably caused by fileio.c:3919
> (907fd1f7ff4 somewhere on master, inside DEFUN "insert-file-contents")
> which declares a read buffer.  The constant READ_BUF_SIZE is
> indirectly set as 16 * 1024 = 16384.  Didn't read further there, nor
> inside `revert-buffer'.

Indeed this seems to be related to the advertised file size being zero.

However, no other programs (including editors) seem to have a problem
with this.  So it really must be Emacs trying to be clever.

Finding files under /proc/ is, of course, not a very relevant
application of Emacs, but Elisp code gets posted that relies on this:
https://nullprogram.com/blog/2015/10/14/

I had something similar in my init.el to determine the value to use with
"make -j" as a default compile command - this is how I noticed this
issue in the first place.  (Now I changed that to executing the command
nproc.)

What worries me more is that there may be other, more sneaky, cases
where this backfires.  For example, perhaps opening a file that is being
written (so that its size increases), is broken as well in some cases?

Should I report this to bug-gnu-emacs@gnu.org, or is posting on this
list enough?  Or should I open a ticket with debbugs.gnu.org?

Cheers
Christoph

Re: Emacs opens only first 16384 bytes of file?!

[Eli Zaretskii]

> From: Christoph Groth <christoph@grothesque.org>
> Cc: Óscar Fuentes <ofv@wanadoo.es>,  help-gnu-emacs@gnu.org
> Date: Sun, 12 Feb 2023 10:58:14 +0100
> 
> However, no other programs (including editors) seem to have a problem
> with this.  So it really must be Emacs trying to be clever.

Not clever, but flexible, powerful, and performant.  Which other
program supports on-the-fly decoding of non-ASCII text in so many
encodings?

> Should I report this to bug-gnu-emacs@gnu.org, or is posting on this
> list enough?  Or should I open a ticket with debbugs.gnu.org?

A bug for this already exists: bug#9800.  A possible solution was also
pointed out there.

Re: Emacs opens only first 16384 bytes of file?!

[Eli Zaretskii]

> From: Christoph Groth <christoph@grothesque.org>
> Date: Sat, 11 Feb 2023 22:16:52 +0100
> 
> I just noticed something that made me doubt my own sanity:  With Emacs
> 27.1 (from Debian stable running Debian’s Linux kernel
> 6.0.0-0.deb11.6-amd64) the command
> 
>   emacs -q -nw /proc/cpuinfo
> 
> reproducibly opens cpuinfo only up to a portion of line 272, while the
> entire file is 560 lines long on that machine.  I checked on a different
> machine (with same Emacs), and I see the same behavior.
> 
> C-u C-x = tells that the file is read up to position 16384 (= 2^14).
> 
> Running M-x revert-buffer loads the whole file...
> 
> Is/was this a known issue?  I could not find anything on the web.

Please report this as a bug.

Re: Emacs opens only first 16384 bytes of file?!

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1004 bytes --]

On Sun, Feb 12, 2023 at 08:02:37AM +0200, Eli Zaretskii wrote:
> > From: Christoph Groth <christoph@grothesque.org>
> > Date: Sat, 11 Feb 2023 22:16:52 +0100
> > 
> > I just noticed something that made me doubt my own sanity:  With Emacs
> > 27.1 (from Debian stable running Debian’s Linux kernel
> > 6.0.0-0.deb11.6-amd64) the command
> > 
> >   emacs -q -nw /proc/cpuinfo
> > 
> > reproducibly opens cpuinfo only up to a portion of line 272, while the
> > entire file is 560 lines long on that machine.  I checked on a different
> > machine (with same Emacs), and I see the same behavior.
> > 
> > C-u C-x = tells that the file is read up to position 16384 (= 2^14).
> > 
> > Running M-x revert-buffer loads the whole file...
> > 
> > Is/was this a known issue?  I could not find anything on the web.
> 
> Please report this as a bug.

I think the bug is already there (#9800) [1]

[1] https://lists.gnu.org/archive/html/bug-gnu-emacs/2022-02/msg00459.html

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Emacs opens only first 16384 bytes of file?!

[Eli Zaretskii]

> Date: Sun, 12 Feb 2023 08:11:04 +0100
> From: <tomas@tuxteam.de>
> 
> On Sun, Feb 12, 2023 at 08:02:37AM +0200, Eli Zaretskii wrote:
> > Please report this as a bug.
> 
> I think the bug is already there (#9800) [1]

Right.

Re: Emacs opens only first 16384 bytes of file?!

[Christoph Groth]

Eli Zaretskii wrote:

> > From: Christoph Groth <christoph@grothesque.org>
> > 
> > However, no other programs (including editors) seem to have
> > a problem with this.  So it really must be Emacs trying to be
> > clever.
> 
> Not clever, but flexible, powerful, and performant.  Which other
> program supports on-the-fly decoding of non-ASCII text in so many
> encodings?

No need to convince me.  I have been living in Emacs since 1998.

That said, the finding that Emacs does not read files until EOF
surprised me quite a lot.  I thought that it had solid underpinnings.

> > Should I report this to bug-gnu-emacs@gnu.org, or is posting on this
> > list enough?  Or should I open a ticket with debbugs.gnu.org?
> 
> A bug for this already exists: bug#9800.  A possible solution was also
> pointed out there.

Thanks, I did not manage to find this one.

Re: Emacs opens only first 16384 bytes of file?!

[Eli Zaretskii]

> From: Christoph Groth <christoph@grothesque.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Mon, 13 Feb 2023 08:59:01 +0100
> 
> > Not clever, but flexible, powerful, and performant.  Which other
> > program supports on-the-fly decoding of non-ASCII text in so many
> > encodings?
> 
> No need to convince me.  I have been living in Emacs since 1998.
> 
> That said, the finding that Emacs does not read files until EOF
> surprised me quite a lot.  I thought that it had solid underpinnings.

It does.  There are several valid reasons why we don't read
everything: insert-file-contents has more than a single operation
mode.