Re: opening large files (few hundred meg)

[Stefan Monnier <monnier@iro.umontreal.ca>]

>> Not really, since visiting a file reads all of it into an Emacs buffer.

> The problem is not with the buffer size per se, it's with the fact
> that Emacs needs to be able to address each byte of the file's text
> with an Emacs integer data type, which is 29 bit wide on 32-bit
> machines.

Well, that's true but even if we lift this restriction (might take some
tedious work and have other downsides, but nothing really difficult),
but it won't help that much:

On a 32bit system, the maximum memory available to a single process is
limited to anywhere between 1GB and 4GB depending on the OS.  If you
consider that the large file will not be the only thing in Emacs's
memory, the best we can hope for is to handle a single 2GB file (and
there can be all kinds of reasons why even this much will fail, e.g. if
the file's content is not put into a unibyte but a multibyte buffer.
And of course, if we ever need to edit the buffer, we may need to grow
the gap, which implies reallocating extra memory which may fail in such
limit cases).  More realistic is 1GB as the upper bound.

Currently, the largest integer is 128MB.  This can be easily bumped up
to 256MB (I'm using such a hack on my local Emacs).  If we're willing to
work a bit more at it (at some small cost in other areas) we can push
this to 512MB.  XEmacs has pushed this even further and has a 1GB limit
(IIUC).  So the integer-size limit and the absolute theoretical maximum
imposed by the OS are about the same.


        Stefan