unofficial mirror of help-gnu-emacs@gnu.org
 help / color / mirror / Atom feed
From: Stefan Monnier <monnier@iro.umontreal.ca>
To: help-gnu-emacs@gnu.org
Subject: Re: opening large files (few hundred meg)
Date: Wed, 30 Jan 2008 10:12:24 -0500	[thread overview]
Message-ID: <jwvir1b745p.fsf-monnier+gnu.emacs.help@gnu.org> (raw)
In-Reply-To: mailman.6646.1201548710.18990.help-gnu-emacs@gnu.org

>> Not really, since visiting a file reads all of it into an Emacs buffer.

> The problem is not with the buffer size per se, it's with the fact
> that Emacs needs to be able to address each byte of the file's text
> with an Emacs integer data type, which is 29 bit wide on 32-bit
> machines.

Well, that's true but even if we lift this restriction (might take some
tedious work and have other downsides, but nothing really difficult),
but it won't help that much:

On a 32bit system, the maximum memory available to a single process is
limited to anywhere between 1GB and 4GB depending on the OS.  If you
consider that the large file will not be the only thing in Emacs's
memory, the best we can hope for is to handle a single 2GB file (and
there can be all kinds of reasons why even this much will fail, e.g. if
the file's content is not put into a unibyte but a multibyte buffer.
And of course, if we ever need to edit the buffer, we may need to grow
the gap, which implies reallocating extra memory which may fail in such
limit cases).  More realistic is 1GB as the upper bound.

Currently, the largest integer is 128MB.  This can be easily bumped up
to 256MB (I'm using such a hack on my local Emacs).  If we're willing to
work a bit more at it (at some small cost in other areas) we can push
this to 512MB.  XEmacs has pushed this even further and has a 1GB limit
(IIUC).  So the integer-size limit and the absolute theoretical maximum
imposed by the OS are about the same.


        Stefan

  parent reply	other threads:[~2008-01-30 15:12 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-01-28 17:35 opening large files (few hundred meg) Xah Lee
2008-01-28 18:05 ` Sven Joachim
2008-01-28 19:31   ` Eli Zaretskii
2008-01-28 20:36     ` Andreas Röhler
     [not found]     ` <mailman.6652.1201552566.18990.help-gnu-emacs@gnu.org>
2008-01-28 21:50       ` Jason Rumney
2008-01-29  7:07         ` Andreas Röhler
2008-01-29  7:20         ` Thierry Volpiatto
     [not found]         ` <mailman.6666.1201591238.18990.help-gnu-emacs@gnu.org>
2008-01-29  9:08           ` Tim X
2008-01-29 16:34             ` Xah Lee
2008-01-29 19:06               ` Tom Tromey
2008-01-29 20:44                 ` Eli Zaretskii
     [not found]                 ` <mailman.6705.1201639469.18990.help-gnu-emacs@gnu.org>
2008-01-30 20:01                   ` Stefan Monnier
2008-01-30 22:04                     ` Eli Zaretskii
2008-01-29 22:10               ` Jason Rumney
2008-01-30 17:08                 ` Joel J. Adamson
2008-01-31  5:57               ` Tim X
2008-01-31 15:35                 ` Stefan Monnier
2008-02-08 11:25               ` Giacomo Boffi
2008-02-06  1:47             ` Samuel Karl Peterson
2008-01-29 14:52           ` Joel J. Adamson
2008-01-30 14:55         ` Stefan Monnier
2008-02-06 16:42         ` Mathias Dahl
2008-02-06 16:55           ` Mathias Dahl
2008-01-29 10:43       ` Johan Bockgård
2008-01-29 15:35         ` Andreas Röhler
2008-02-06  1:25         ` Samuel Karl Peterson
2008-02-17 16:01           ` Kevin Rodgers
2008-01-29 16:33       ` Ted Zlatanov
     [not found]   ` <mailman.6646.1201548710.18990.help-gnu-emacs@gnu.org>
2008-01-30 15:12     ` Stefan Monnier [this message]
2008-01-30 16:55       ` Sven Joachim
2008-01-30 21:53         ` Stefan Monnier
2008-01-31 22:55     ` Ilya Zakharevich
     [not found]     ` <200801312255.m0VMt701019096@powdermilk.math.berkeley.edu>
2008-02-01 11:04       ` Eli Zaretskii
     [not found]       ` <mailman.6836.1201863892.18990.help-gnu-emacs@gnu.org>
2008-02-01 22:26         ` Ilya Zakharevich

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/emacs/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=jwvir1b745p.fsf-monnier+gnu.emacs.help@gnu.org \
    --to=monnier@iro.umontreal.ca \
    --cc=help-gnu-emacs@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).