bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Mike Coleman]

I'm trying to open a large (between 4GB and 5GB) file on a x86-64
Linux box that has 64GB of RAM.  I'm getting the error message
"Maximum buffer size exceeded".  I'm able to create 32GB strings in
Python on this box, so I don't believe it's a ulimit problem or
anything like that.

The contents of /proc/cpuinfo say that the CPU has 48 virtual bits.
I'm guessing that emacs uses maybe six bits or so for overhead, but
that doesn't seem like enough to be causing my problem.

Given the error message, it seems like emacs probably has an internal
idea of a maximum buffer size, but apropos doesn't turn up anything.

I'm sure this has been discussed and I've tried to find these
discussions, but I don't seem to be able to track down the current
thinking on this subject.

Is this a know limitation or a bug?  Can someone who knows provide a
quick summary?

Thanks,
Mike

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Eli Zaretskii]

> Date: Thu, 26 Mar 2009 10:50:35 -0500
> From: Mike Coleman <tutufan@gmail.com>
> Cc: 
> 
> I'm trying to open a large (between 4GB and 5GB) file on a x86-64
> Linux box that has 64GB of RAM.  I'm getting the error message
> "Maximum buffer size exceeded".

Thank you for your report.  However, it lacks crucial info, because
you didn't use "M-x report-emacs-bug RET" to submit it.  Please do
that (and do it from the same version of Emacs where you get the error
message), so that important aspects of your system and Emacs
configuration will be reported and considered in resolving this
problem.

Next, please type "file `which emacs`" at the shell prompt on that
machine, and tell what this command displays.

Also, please type "M-: most-positive-fixnum RET" inside Emacs, and
tell what it displays.

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Mike Coleman]

Hi,

This machine isn't really connected to the Internet, but here's the
contents of that buffer, after trying to load the large file in
question.

'which emacs' is /usr/bin/emacs.

most-positive-fixnum gives

    1152921504606846975 (#o37777777777, #xffffffff)

Mike




Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:



If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
/usr/share/emacs/22.1/etc/DEBUG for instructions.


In GNU Emacs 22.1.1 (x86_64-redhat-linux-gnu, GTK+ Version 2.10.4)
 of 2007-08-23 on monk.karan.org
Windowing system distributor `The X.Org Foundation', version 11.0.10600000
configured using `configure  '--build=x86_64-redhat-linux-gnu'
'--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include'
'--libdir=/usr/lib64' '--libexecdir=/usr/libexec'
'--localstatedir=/var' '--sharedstatedir=/usr/com'
'--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-pop'
'--with-sound' '--with-gtk' 'build_alias=x86_64-redhat-linux-gnu'
'host_alias=x86_64-redhat-linux-gnu'
'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-DMAIL_USE_LOCKF
-DSYSTEM_PURESIZE_EXTRA=16777216 -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: nil
  locale-coding-system: nil
  default-enable-multibyte-characters: t

Major mode: Mail

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent input:
<down-mouse-1> <mouse-1> M-x r e p o r t - e m a <tab>
<return> x x x <return> C-x 1 C-v C-v C-p C-x C-f /
n / d a t a 1 / b l a s t / d b / n r <return> y C-x
<escape> <escape> M-p M-p M-n C-a C-k C-g C-g M-x r
e p r t <backspace> <backspace> o r t - e m a <tab>
<return>

Recent messages:
Loading /usr/share/emacs/site-lisp/site-start.d/php-mode-init.el (source)...done
Loading /usr/share/emacs/site-lisp/site-start.d/po-mode-init.el (source)...done
Loading /usr/share/emacs/site-lisp/site-start.d/psgml-init.el (source)...done
Loading /usr/share/emacs/site-lisp/site-start.d/python-mode-init.el
(source)...done
Loading /usr/share/emacs/site-lisp/site-start.d/rpm-spec-mode-init.el
(source)...done
Loading emacsbug...done
Loading help-mode...done
File nr is large (4187MB), really open? (y or n)
byte-code: Maximum buffer size exceeded
next-history-element: Beginning of history; no preceding item
Quit [2 times]




On Thu, Mar 26, 2009 at 2:58 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 26 Mar 2009 10:50:35 -0500
>> From: Mike Coleman <tutufan@gmail.com>
>> Cc:
>>
>> I'm trying to open a large (between 4GB and 5GB) file on a x86-64
>> Linux box that has 64GB of RAM.  I'm getting the error message
>> "Maximum buffer size exceeded".
>
> Thank you for your report.  However, it lacks crucial info, because
> you didn't use "M-x report-emacs-bug RET" to submit it.  Please do
> that (and do it from the same version of Emacs where you get the error
> message), so that important aspects of your system and Emacs
> configuration will be reported and considered in resolving this
> problem.
>
> Next, please type "file `which emacs`" at the shell prompt on that
> machine, and tell what this command displays.
>
> Also, please type "M-: most-positive-fixnum RET" inside Emacs, and
> tell what it displays.
>

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Eli Zaretskii]

> Date: Thu, 26 Mar 2009 16:01:51 -0500
> From: Mike Coleman <tutufan@gmail.com>
> Cc: 2790@emacsbugs.donarmstrong.com
> 
> 'which emacs' is /usr/bin/emacs.

I asked to type "file `which emacs`".  The `file' part is important.

> most-positive-fixnum gives
> 
>     1152921504606846975 (#o37777777777, #xffffffff)

That sounds like a bug: the decimal representation is like I'd expect
on a 64-bit machine, but the octal and hex representations are like on
a 32-bit machine.  Strange...

Perhaps this is some bug that was in Emacs 22.1.  Can you upgrade to
22.3?

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Mike Coleman]

On Thu, Mar 26, 2009 at 4:14 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 26 Mar 2009 16:01:51 -0500
>> From: Mike Coleman <tutufan@gmail.com>
>> Cc: 2790@emacsbugs.donarmstrong.com
>>
>> 'which emacs' is /usr/bin/emacs.
>
> I asked to type "file `which emacs`".  The `file' part is important.

Ah, missed it.  Here you go:

$ file `which emacs`
/usr/bin/emacs: symbolic link to `/etc/alternatives/emacs'

Also

$ file -L `which emacs`
/usr/bin/emacs: sticky ELF 64-bit LSB executable, AMD x86-64, version
1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs),
for GNU/Linux 2.6.9, stripped

in case that's where we were headed.  :-)

And

$ lsb_release -rd
Description:	CentOS release 5.2 (Final)
Release:	5.2


>> most-positive-fixnum gives
>>
>>     1152921504606846975 (#o37777777777, #xffffffff)
>
> That sounds like a bug: the decimal representation is like I'd expect
> on a 64-bit machine, but the octal and hex representations are like on
> a 32-bit machine.  Strange...

Yeah, that looked a little odd to me, too.

I gather from your questions that this (loading the large file) was
more or less supposed to work?

> Perhaps this is some bug that was in Emacs 22.1.  Can you upgrade to
> 22.3?

This distribution version has no later package available.  I can just
go ahead and build from source, though.  Are there any particular
flags I should use, etc., to support this debugging process?

Mike

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Stefan Monnier]

> I gather from your questions that this (loading the large file) was
> more or less supposed to work?

Well, on 32bit machines, the largest file you can open is 256MB or so.
In a 64bit system, the limit is much larger (so you really shouldn't see
the "Maximum buffer size exceeded" message you're seeing), but then
again, there are various places in the C code where we turn
buffer-positions into `int', so things larger than 2GB will hit bugs
sooner or later (which will manifest in various funny ways).  These bugs
shouldn't be too hard to fix, but even with my 4GB machine, opening such
large files is really painful, which makes it difficult to debug.


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Mike Coleman]

On Thu, Mar 26, 2009 at 7:32 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> These bugs
> shouldn't be too hard to fix, but even with my 4GB machine, opening such
> large files is really painful, which makes it difficult to debug.

I understand.  Part of the reason I'm reporting this is that have easy
access to a somewhat unusually large machine.  I don't know much of
anything about debugging emacs per se (as opposed to any random GNU C
program), but if I can get some hints, I'll try to do what I can.

Mike

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux

[Stefan Monnier]

>>  These bugs shouldn't be too hard to fix, but even with my 4GB
>> machine, opening such large files is really painful, which makes it
>> difficult to debug.
> I understand.  Part of the reason I'm reporting this is that have easy
> access to a somewhat unusually large machine.  I don't know much of
> anything about debugging emacs per se (as opposed to any random GNU C
> program), but if I can get some hints, I'll try to do what I can.

Most of those bugs get fixed over time as we work on other bugs in
nearby areas.  So there's no point debugging an old version of the code,
because there's a good chance that several of the bugs you're going to
hit have been fixed already.

I.e. first try it with the head the trunk of the CVS repository.
Then report here with the problems you encounter.


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Richard M Stallman]

Please try the pretest of Emacs 23 and see if it works.
See ftp://alpha.gnu.org/gnu/emacs/pretest/.

And please don't call it a "Linux box".  The system running in it
is more GNU than Linux, so please call it a GNU/Linux box.
(See http://www.gnu.org/gnu/gnu-linux-faq.html.)

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Eli Zaretskii]

> Date: Thu, 26 Mar 2009 16:33:29 -0500
> From: Mike Coleman <tutufan@gmail.com>
> Cc: 2790@emacsbugs.donarmstrong.com
> 
> I gather from your questions that this (loading the large file) was
> more or less supposed to work?

Yes, and rather more than less ;-)

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 26 Mar 2009 16:01:51 -0500
>> From: Mike Coleman <tutufan@gmail.com>
>> Cc: 2790@emacsbugs.donarmstrong.com
>> 
>> most-positive-fixnum gives
>> 
>>     1152921504606846975 (#o37777777777, #xffffffff)
>
> That sounds like a bug: the decimal representation is like I'd expect
> on a 64-bit machine, but the octal and hex representations are like on
> a 32-bit machine.  Strange...

22.1 had a buggy format that truncated numbers to the range of int.  I
fixed some bugs here for 22.2.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Mike Coleman]

On Thu, Mar 26, 2009 at 11:59 PM, Richard M Stallman <rms@gnu.org> wrote:
> Please try the pretest of Emacs 23 and see if it works.
> See ftp://alpha.gnu.org/gnu/emacs/pretest/.
>
> And please don't call it a "Linux box".  The system running in it
> is more GNU than Linux, so please call it a GNU/Linux box.
> (See http://www.gnu.org/gnu/gnu-linux-faq.html.)
>

Okay, tried it (emacs-23.0.91), but no luck.  Looks very nice, but
finding that large file produced the same error.  The value of
'most-positive-fixnum' prints correctly, though (which is different).

The smallest file that produces the error has 536870912 bytes, exactly
512MB.  (This is via binary search, so assumes that there is just one
transition from "will" to "won't" as filesize increases.)


On an unrelated note, I did see the following message in the 'make
install' output, for what it's worth:

    gzip: ./quail/hangul3.el: No such file or directory

It didn't stop the install, though.

Mike

P.S.  Re GNU/Linux, yes, preaching to the choir.  Thanks for the reminder.

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux

[Mike Coleman]

On Thu, Mar 26, 2009 at 10:08 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> I.e. first try it with the head the trunk of the CVS repository.
> Then report here with the problems you encounter.

Is there anything newer than 23.0.91 worth trying?

Or, alternatively, since the trigger file is 512MB, can someone with
the head built verify that this bombs there as well?

Mike

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Eli Zaretskii]

> Date: Fri, 27 Mar 2009 11:27:23 -0500
> From: Mike Coleman <tutufan@gmail.com>
> Cc: 2790@emacsbugs.donarmstrong.com, bug-gnu-emacs@gnu.org
> 
> On Thu, Mar 26, 2009 at 11:59 PM, Richard M Stallman <rms@gnu.org> wrote:
> > Please try the pretest of Emacs 23 and see if it works.
> > See ftp://alpha.gnu.org/gnu/emacs/pretest/.
> >
> > And please don't call it a "Linux box".  The system running in it
> > is more GNU than Linux, so please call it a GNU/Linux box.
> > (See http://www.gnu.org/gnu/gnu-linux-faq.html.)
> >
> 
> Okay, tried it (emacs-23.0.91), but no luck.  Looks very nice, but
> finding that large file produced the same error.  The value of
> 'most-positive-fixnum' prints correctly, though (which is different).
> 
> The smallest file that produces the error has 536870912 bytes, exactly
> 512MB.

This means that the bug is still there in the current code base.
Thanks for testing.

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Stefan Monnier]

> Okay, tried it (emacs-23.0.91), but no luck.  Looks very nice, but
> finding that large file produced the same error.  The value of
> 'most-positive-fixnum' prints correctly, though (which is different).

There was an incorrect check that limited the size to INT_MAX/4
(i.e. 512MB for systems where ints are 32bit).  I've removed this check
in the CVS code (see patch below).  I am now able to open an 800MB file
with this check removed.  OTOH with a 2GB file, I got some unjustified
"memory exhausted" error, but I haven't tracked it down yet.  Note also
that when you open large files, it's worthwhile to use
find-file-literally to be sure it's opened in unibyte mode; otherwise
it gets decoded which takes ages.  Also if the file has many lines (my
800MB file was made up by copying a C file many times, so it had
millions of lines), turning off line-number-mode is is needed to recover
responsiveness when navigating near the end of the buffer.


        Stefan


Index: src/fileio.c
===================================================================
RCS file: /sources/emacs/emacs/src/fileio.c,v
retrieving revision 1.651
retrieving revision 1.652
diff -u -r1.651 -r1.652
--- src/fileio.c	24 Mar 2009 14:14:54 -0000	1.651
+++ src/fileio.c	28 Mar 2009 02:06:08 -0000	1.652
@@ -3300,7 +3300,11 @@
 	     overflow.  The calculations below double the file size
 	     twice, so check that it can be multiplied by 4 safely.  */
 	  if (XINT (end) != st.st_size
-	      || st.st_size > INT_MAX / 4)
+	      /* Actually, it should test either INT_MAX or LONG_MAX
+		 depending on which one is used for EMACS_INT.  But in
+		 any case, in practice, this test is redundant with the
+		 one above.
+		 || st.st_size > INT_MAX / 4 */)
 	    error ("Maximum buffer size exceeded");
 
 	  /* The file size returned from stat may be zero, but data

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Mike Coleman]

I'm not quite sure I'm following this.  Are you running your
experiments with a 64-bit emacs, or a 32-bit one?



On Fri, Mar 27, 2009 at 9:12 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Okay, tried it (emacs-23.0.91), but no luck.  Looks very nice, but
>> finding that large file produced the same error.  The value of
>> 'most-positive-fixnum' prints correctly, though (which is different).
>
> There was an incorrect check that limited the size to INT_MAX/4
> (i.e. 512MB for systems where ints are 32bit).  I've removed this check
> in the CVS code (see patch below).  I am now able to open an 800MB file
> with this check removed.  OTOH with a 2GB file, I got some unjustified
> "memory exhausted" error, but I haven't tracked it down yet.  Note also
> that when you open large files, it's worthwhile to use
> find-file-literally to be sure it's opened in unibyte mode; otherwise
> it gets decoded which takes ages.  Also if the file has many lines (my
> 800MB file was made up by copying a C file many times, so it had
> millions of lines), turning off line-number-mode is is needed to recover
> responsiveness when navigating near the end of the buffer.
>
>
>        Stefan
>
>
> Index: src/fileio.c
> ===================================================================
> RCS file: /sources/emacs/emacs/src/fileio.c,v
> retrieving revision 1.651
> retrieving revision 1.652
> diff -u -r1.651 -r1.652
> --- src/fileio.c        24 Mar 2009 14:14:54 -0000      1.651
> +++ src/fileio.c        28 Mar 2009 02:06:08 -0000      1.652
> @@ -3300,7 +3300,11 @@
>             overflow.  The calculations below double the file size
>             twice, so check that it can be multiplied by 4 safely.  */
>          if (XINT (end) != st.st_size
> -             || st.st_size > INT_MAX / 4)
> +             /* Actually, it should test either INT_MAX or LONG_MAX
> +                depending on which one is used for EMACS_INT.  But in
> +                any case, in practice, this test is redundant with the
> +                one above.
> +                || st.st_size > INT_MAX / 4 */)
>            error ("Maximum buffer size exceeded");
>
>          /* The file size returned from stat may be zero, but data
>

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 27 Mar 2009 22:12:54 -0400
> Cc: 2790@emacsbugs.donarmstrong.com
> 
> > Okay, tried it (emacs-23.0.91), but no luck.  Looks very nice, but
> > finding that large file produced the same error.  The value of
> > 'most-positive-fixnum' prints correctly, though (which is different).
> 
> There was an incorrect check that limited the size to INT_MAX/4
> (i.e. 512MB for systems where ints are 32bit).  I've removed this check
> in the CVS code (see patch below).

The patch below does this:

> -	      || st.st_size > INT_MAX / 4)
> +	      /* Actually, it should test either INT_MAX or LONG_MAX
> +		 depending on which one is used for EMACS_INT.  But in
> +		 any case, in practice, this test is redundant with the
> +		 one above.
> +		 || st.st_size > INT_MAX / 4 */)
>  	    error ("Maximum buffer size exceeded");

But what about the commentary immediately preceding the modified code:

  The calculations below double the file size twice, so check that it
  can be multiplied by 4 safely.

I'm not sure to which calculations it alludes, but if you think they
are no longer relevant, please remove that part of the comment,
otherwise we will wonder in a couple of years why the code does not do
what the comment says it should.

Personally, I would change INT_MAX/4 to LONG_MAX/4, because that does
TRT on all supported platforms, 32-bit and 64-bit alike (long and int
are both 32-bit wide on 32-bit machines).  That would avoid too
radical changes during a pretest, which is a Good Thing, IMO.

> Note also that when you open large files, it's worthwhile to use
> find-file-literally to be sure it's opened in unibyte mode;
> otherwise it gets decoded which takes ages.

Perhaps the prompt we pop for large file should suggest visiting
literally as an option.

> Also if the file has many lines (my
> 800MB file was made up by copying a C file many times, so it had
> millions of lines), turning off line-number-mode is is needed to recover
> responsiveness when navigating near the end of the buffer.

Perhaps we should make the default value of line-number-display-limit
non-nil, at least in 64-bit builds.

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Personally, I would change INT_MAX/4 to LONG_MAX/4,

This is wrong, see also make_gap_larger.  The range for buffer positions
is still limited by the range of int.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: 2790@emacsbugs.donarmstrong.com,  Stefan Monnier <monnier@iro.umontreal.ca>,  tutufan@gmail.com
> Date: Sat, 28 Mar 2009 12:12:04 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Personally, I would change INT_MAX/4 to LONG_MAX/4,
> 
> This is wrong, see also make_gap_larger.  The range for buffer positions
> is still limited by the range of int.

Then that's another bug waiting to happen, isn't it?  Buffer positions
should be limited by EMACS_INT, not by int, right?

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Then that's another bug waiting to happen, isn't it?  Buffer positions
> should be limited by EMACS_INT, not by int, right?

If you fix _every_ use of int for buffer positions.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: 2790@emacsbugs.donarmstrong.com,  monnier@iro.umontreal.ca,  tutufan@gmail.com
> Date: Sat, 28 Mar 2009 13:40:24 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then that's another bug waiting to happen, isn't it?  Buffer positions
> > should be limited by EMACS_INT, not by int, right?
> 
> If you fix _every_ use of int for buffer positions.

Right.  And in the meantime, we should probably tell Emacs on 64-bit
platforms is capable of visiting files up to 2GB large.

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Richard M Stallman]

    The smallest file that produces the error has 536870912 bytes, exactly
    512MB.  (This is via binary search, so assumes that there is just one
    transition from "will" to "won't" as filesize increases.)

It looks like you're simply hitting the limit on buffer size.

But I suspect there is a bug here, because it sounds like
you can visit a file whose size is bigger than most-positive-fixnum.
If you do that, what value do you get for (point-max)?
Is it negative?  I think it will be, and that we need
to restrict the buffer size to be no more than most-positive-fixnum.

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Mike Coleman]

On Sat, Mar 28, 2009 at 10:17 AM, Richard M Stallman <rms@gnu.org> wrote:
>    The smallest file that produces the error has 536870912 bytes, exactly
>    512MB.  (This is via binary search, so assumes that there is just one
>    transition from "will" to "won't" as filesize increases.)
>
> It looks like you're simply hitting the limit on buffer size.
>
> But I suspect there is a bug here, because it sounds like
> you can visit a file whose size is bigger than most-positive-fixnum.
> If you do that, what value do you get for (point-max)?
> Is it negative?  I think it will be, and that we need
> to restrict the buffer size to be no more than most-positive-fixnum.

It appears to me that visiting a file larger than 512MB is simply
failing, with the given error message about the max buffer size.
(Note that 512MB is much, much smaller than most-positive-fixnum (==
2^64).)

So, to me, it seems like the behavior is "correct", if emacs only
promises to handle files as large as 512MB (on 64-bit platforms).

Nonetheless, I'd like it to handle much larger files.  Is there any
real reason not to just turn every (32-bit) "int" into a (64-bit)
"long" on 64-bit platforms?

Mike

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Richard M Stallman]

    There was an incorrect check that limited the size to INT_MAX/4
    (i.e. 512MB for systems where ints are 32bit).  I've removed this check
    in the CVS code (see patch below).  I am now able to open an 800MB file
    with this check removed.

After you visit such a file, what does (point-max) return?

Perhaps you're talking only about 64-bit systems?

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit

[Stefan Monnier]

> I'm not quite sure I'm following this.  Are you running your
> experiments with a 64-bit emacs, or a 32-bit one?

64bit, of course.  Otherwise the 256MB limit kicks in.


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Stefan Monnier]

> The patch below does this:

>> -	      || st.st_size > INT_MAX / 4)
>> +	      /* Actually, it should test either INT_MAX or LONG_MAX
>> +		 depending on which one is used for EMACS_INT.  But in
>> +		 any case, in practice, this test is redundant with the
>> +		 one above.
>> +		 || st.st_size > INT_MAX / 4 */)
>> error ("Maximum buffer size exceeded");

> But what about the commentary immediately preceding the modified code:
>   The calculations below double the file size twice, so check that it
>   can be multiplied by 4 safely.

The patch also adds a comment explaining that this test is actually
redundant in practice (and it will stay redundant as long as our Lisp
integers have at least 2bits of tag).

> I'm not sure to which calculations it alludes, but if you think they
> are no longer relevant, please remove that part of the comment,
> otherwise we will wonder in a couple of years why the code does not do
> what the comment says it should.

Since I'm not sure either, I kept the comment and added another one
explaining why I removed the check anyway.

> Personally, I would change INT_MAX/4 to LONG_MAX/4, because that does
> TRT on all supported platforms, 32-bit and 64-bit alike (long and int
> are both 32-bit wide on 32-bit machines).  That would avoid too
> radical changes during a pretest, which is a Good Thing, IMO.

In that case I'd rather do the check more directly, e.g.:

    (((EMACS_INT)st.st_size)*4)/4 == st.st_size

But as explained, I'm not convinced the check is needed/useful.

>> Note also that when you open large files, it's worthwhile to use
>> find-file-literally to be sure it's opened in unibyte mode;
>> otherwise it gets decoded which takes ages.
> Perhaps the prompt we pop for large file should suggest visiting
> literally as an option.

Yes, that's also what I was thinking.  Together with having different
"large-threshold" values for unibyte and multibyte.

>> Also if the file has many lines (my
>> 800MB file was made up by copying a C file many times, so it had
>> millions of lines), turning off line-number-mode is is needed to recover
>> responsiveness when navigating near the end of the buffer.

> Perhaps we should make the default value of line-number-display-limit
> non-nil, at least in 64-bit builds.

Agreed.  We could even do something better:
- do it more efficiently (once computed for a page, it should be able
  to update the count instantly when paging up/down, whereas it seems
  not to always be able to do that).
- when computing really would take a lot of time (e.g. we're far from
  the closest known line position), display ??? and postpone the actual
  computation to some future idle time.

In any case, large file introduce lots of problem.


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Stefan Monnier]

>> Personally, I would change INT_MAX/4 to LONG_MAX/4,
> This is wrong, see also make_gap_larger.  The range for buffer positions
> is still limited by the range of int.

Every place where that is the case is a bug (and yes, there are many
still).  We should remove the check in make_gap_larger, BTW.


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Stefan Monnier]

>     There was an incorrect check that limited the size to INT_MAX/4
>     (i.e. 512MB for systems where ints are 32bit).  I've removed this check
>     in the CVS code (see patch below).  I am now able to open an 800MB file
>     with this check removed.

> After you visit such a file, what does (point-max) return?

It returns the correct value.

> Perhaps you're talking only about 64-bit systems?

Yes, all this discussion is in this context.  For 32bit systems, the
256MB limit kicks in long before we hit any of those problems.


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> In that case I'd rather do the check more directly, e.g.:
>
>     (((EMACS_INT)st.st_size)*4)/4 == st.st_size

That will reintroduce the buggy reliance on undefined integer overflow.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Stefan Monnier]

>> In that case I'd rather do the check more directly, e.g.:
>> (((EMACS_INT)st.st_size)*4)/4 == st.st_size
> That will reintroduce the buggy reliance on undefined integer overflow.

I do not know what you're referring to.  If you're referring to the fact
that (((EMACS_INT)st.st_size)*4) may overflow, then I'm not sure what
problem you're envisaging: do you think that in some cases where it
overflows, (((EMACS_INT)st.st_size)*4)/4 may still be equal to
st.st_size (I can't think of any meningful semantics for overflow that
would give me that property)?  Or are you concerned about trapping
overflow semantics (I've never seen it in use and doubt Emacs would
work in such a system)?


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I do not know what you're referring to.  If you're referring to the fact
> that (((EMACS_INT)st.st_size)*4) may overflow, then I'm not sure what
> problem you're envisaging:

The compiler can (and does) optimize on the fact that overflow does not
occur, thus the operation x*4/4 becomes a no-op.  Never ever do overflow
checking after the fact.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Stefan Monnier]

> The compiler can (and does) optimize on the fact that overflow does not
> occur, thus the operation x*4/4 becomes a no-op.

No way!?  Hmm... yes, I guess it's within the allowed semantics, indeed.
We could prevent optimization by going through a volatile var,
of course.

So in the end, I still prefer the current solution of removing the
check altogether.


        Stefan

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> We could prevent optimization by going through a volatile var,
> of course.

That would mean it would still depend on an undefined operation.  The
right way to check for overflow is to check against the limit
beforehand.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit GNU/Linux box

[Ted Zlatanov]

On Sat, 28 Mar 2009 11:27:57 -0500 Mike Coleman <tutufan@gmail.com> wrote: 

MC> Nonetheless, I'd like it to handle much larger files.  Is there any
MC> real reason not to just turn every (32-bit) "int" into a (64-bit)
MC> "long" on 64-bit platforms?

There are many other issues, because almost every Emacs package treats
the buffer as a small file (scanning and saving are presumed to be
cheap).  I proposed a large file mode in emacs-devel recently, but any
code additions or changes will have to wait until the current pretest is
done.

Ted

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Lars Magne Ingebrigtsen]

Mike Coleman <tutufan@gmail.com> writes:

> I'm trying to open a large (between 4GB and 5GB) file on a x86-64
> Linux box that has 64GB of RAM.  I'm getting the error message
> "Maximum buffer size exceeded".

A lot of work has gone into making Emacs 24 usable for visiting buffers
that are larger than 2GB.  But I'm not sure if Emacs 24 is quite there
yet.

Has anybody tried this lately?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#2790: emacs 22.1.1 cannot open 5GB file on 64GB 64-bit Linux box

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 12 Sep 2011 00:10:05 +0200
> Cc: 2790@debbugs.gnu.org
> 
> Mike Coleman <tutufan@gmail.com> writes:
> 
> > I'm trying to open a large (between 4GB and 5GB) file on a x86-64
> > Linux box that has 64GB of RAM.  I'm getting the error message
> > "Maximum buffer size exceeded".
> 
> A lot of work has gone into making Emacs 24 usable for visiting buffers
> that are larger than 2GB.  But I'm not sure if Emacs 24 is quite there
> yet.
> 
> Has anybody tried this lately?

We are there.  Before I removed the 2GB limit hat was artificially put
there to avoid crashes, I actually visited a file larger than 2GB (I
think it was 3GB and change) and made sure I can move in it, edit it,
and save it.

So I'm closing this bug.