Maximum buffer size exceeded

Maximum buffer size exceeded

[Harish Narayanan]

Hello all.

I've been trying to open a large file (208 MB) but I receive the error 
"Maximum buffer size exceeded" and Emacs fails to open the file. This is 
Emacs 21.2.1 on a Redhat GNU/Linux 9 box.

I tried searching in the archives, and noticed this has been asked 
before, but the responses don't seem encouraging. Is there a way of 
working around this that's arisen in the recent past?

Thanks.
Harish

Re: Maximum buffer size exceeded

[Kin Cho]

Harish Narayanan <harish@gamebox.net> writes:

> I've been trying to open a large file (208 MB) but I receive the
> error "Maximum buffer size exceeded" and Emacs fails to open the
> file. This is Emacs 21.2.1 on a Redhat GNU/Linux 9 box.
> 
> I tried searching in the archives, and noticed this has been
> asked before, but the responses don't seem encouraging. Is there
> a way of working around this that's arisen in the recent past?

Use head/tail/dd to extract the small part of the file you want
to edit, edit that small file, then use cat/head/tail/dd to put
it back into your original file.

Let's say you want to edit the last line of the file 208MB, which
is 1 million lines long:

tail -1 208MB > foo
emacs foo
(head -999999 208MB; cat foo) > new-208MB

-kin

Re: Maximum buffer size exceeded

[Harish Narayanan]

Thanks Kin.

I do things like that when I need to edit the last or first few lines. 
What if the portions I need edited are at some arbitrary line numbers in 
the middle? I don't always know exactly where they will be.

Harish

Kin Cho wrote:

>Harish Narayanan <harish@gamebox.net> writes:
>
>  
>
>>I've been trying to open a large file (208 MB) but I receive the
>>error "Maximum buffer size exceeded" and Emacs fails to open the
>>file. This is Emacs 21.2.1 on a Redhat GNU/Linux 9 box.
>>
>>I tried searching in the archives, and noticed this has been
>>asked before, but the responses don't seem encouraging. Is there
>>a way of working around this that's arisen in the recent past?
>>    
>>
>
>Use head/tail/dd to extract the small part of the file you want
>to edit, edit that small file, then use cat/head/tail/dd to put
>it back into your original file.
>
>Let's say you want to edit the last line of the file 208MB, which
>is 1 million lines long:
>
>tail -1 208MB > foo
>emacs foo
>(head -999999 208MB; cat foo) > new-208MB
>
>-kin
>
>_______________________________________________
>Help-gnu-emacs mailing list
>Help-gnu-emacs@gnu.org
>http://mail.gnu.org/mailman/listinfo/help-gnu-emacs
>
>  
>

Re: Maximum buffer size exceeded

[Kin Cho]

Harish Narayanan <harish@gamebox.net> writes:

> I do things like that when I need to edit the last or first few
> lines. What if the portions I need edited are at some arbitrary
> line numbers in the middle? I don't always know exactly where
> they will be.

If you're going to do this a lot in the future, learn sed.

Otherwise, use grep -n to locate the line number(s) so you can
use a combination of head and tail to extract the line(s).

-kin

Re: Maximum buffer size exceeded

[Thien-Thi Nguyen]

Kin Cho <ignore-this-prefixkin@techie.com> writes:

> Otherwise, use grep -n to locate the line number(s) so you can
> use a combination of head and tail to extract the line(s).

most sed implementations can also select lines:

  sed 5,10!d FILE

should be equivalent to:

  tail +5 FILE | head -5

this saves a fork, i/o overhead, and the need to
compute the arg to `head'.  sed is your friend.

thi

Re: Maximum buffer size exceeded

[Marco Gidde]

Harish Narayanan <harish@gamebox.net> writes:

> I do things like that when I need to edit the last or first few
> lines. What if the portions I need edited are at some arbitrary line
> numbers in the middle? I don't always know exactly where they will be.

I was in the same situation a few days ago, a large file with more
than 200MB. It's sad, but vim did the job without complaining. It took
a while to open the file, but after that everything was
fine. Searching the specific strings, editing them, saving the file
and done.


Marco

Re: Maximum buffer size exceeded

[Eli Zaretskii]

> From: Marco Gidde <marco.gidde@tiscali.de>
> Newsgroups: gnu.emacs.help
> Date: 05 May 2004 07:53:04 +0200
> 
> I was in the same situation a few days ago, a large file with more
> than 200MB. It's sad, but vim did the job without complaining.

Why ``sad''?  Every editor has its breaking point wrt file sizes; I
imagine vim would break somewhere near 2GB, if not before that.

OTOH, why would someone need to load such a huge file into an
interactive editor like Emacs or vim?  Sed sounds like a perfect tool
for the job (faster, too).

In other words, there's the best tool for every job, and Emacs (nor
vim) is not the one for this job.

Re: Maximum buffer size exceeded

[Marco Gidde]

Eli Zaretskii <eliz@gnu.org> writes:

> > From: Marco Gidde <marco.gidde@tiscali.de>
> > Newsgroups: gnu.emacs.help
> > Date: 05 May 2004 07:53:04 +0200
> > 
> > I was in the same situation a few days ago, a large file with more
> > than 200MB. It's sad, but vim did the job without complaining.
> 
> Why ``sad''?  

Simply because I couldn't use Emacs for that.

> OTOH, why would someone need to load such a huge file into an
> interactive editor like Emacs or vim?  Sed sounds like a perfect tool
> for the job (faster, too).

In my special case it was a log file, that I had to modify in a
special way and defining a kbd macro in Emacs was much faster than
writing a perl/awk/sed script for the same purpose. After that I had
to compare the file with another log file and this was just about the
time when I first tried ediff and found that it is great.

And what if the modifications you want to make are not specific to
lines but to paragraphs? I think that would be a problem with sed. And
although I like hacking I don't like to write parsers that must (since
they work without any help from my side) handle all edge cases just to
throw it away after one use. 

So I think that interactive editing is not constrained to the case of
files with a few thousand lines of text or code. And interactive
editing requires an interactive editor.

> In other words, there's the best tool for every job, ...

Sure.

> ... and Emacs (nor vim) is not the one for this job.

What I doubt.

Re: Maximum buffer size exceeded

[Eli Zaretskii]

> From: Marco Gidde <marco.gidde@tiscali.de>
> Newsgroups: gnu.emacs.help
> Date: 05 May 2004 13:45:22 +0200
> 
> And what if the modifications you want to make are not specific to
> lines but to paragraphs? I think that would be a problem with sed.

No, it isn't.  Sed has a block construct, something like this:

/FOO/,/BAR/ {
  s,FOOBAR,WHATEVER,
}

This replaces FOOBAR with WHATEVER in the range of lines between the
line that matches FOO and the line that matches BAR.

Re: Maximum buffer size exceeded

[Marco Gidde]

Eli Zaretskii <eliz@gnu.org> writes:

> > From: Marco Gidde <marco.gidde@tiscali.de>
> > Newsgroups: gnu.emacs.help
> > Date: 05 May 2004 13:45:22 +0200
> > 
> > And what if the modifications you want to make are not specific to
> > lines but to paragraphs? I think that would be a problem with sed.
> 
> No, it isn't.  Sed has a block construct, something like this:
> 
> /FOO/,/BAR/ {
>   s,FOOBAR,WHATEVER,
> }
> 
> This replaces FOOBAR with WHATEVER in the range of lines between the
> line that matches FOO and the line that matches BAR.

I did not know that. Thanks for the hint.

Re: Maximum buffer size exceeded

[Thien-Thi Nguyen]

Marco Gidde <marco.gidde@tiscali.de> writes:

> I did not know that. Thanks for the hint.

many interesting sed recipes can be found in the sedfaq:

  http://sed.sourceforge.net/sedfaq.html

if i had to choose only two tools, they would be emacs and sed.
an interesting project would be to translate elisp programs into
sed scripts (or vice versa).

thi

Re: Maximum buffer size exceeded

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
> In other words, there's the best tool for every job, and Emacs (nor
> vim) is not the one for this job.

Especially for occasional tasks, people usually don't _want_ to use the
`best' tool, they want to use something that's `good enough' and
familiar.  E.g., I very often used emacs to edit executable files even
though I had to waste time being very careful not to screw anything up
(this was before hexl of course).

For many of us, emacs is what's familiar, and we'd like to use it even
when it's not optimal... :-/

-Miles
-- 
Would you like fries with that?

Re: Maximum buffer size exceeded

[Stefan Monnier]

> For many of us, emacs is what's familiar, and we'd like to use it even
> when it's not optimal... :-/

Agreed 99%.


        Stefan


PS: The last 1% has to do with capitalization of "Emacs".

Re: Maximum buffer size exceeded

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Newsgroups: gnu.emacs.help
> Date: 06 May 2004 00:56:50 +0900
> 
> For many of us, emacs is what's familiar, and we'd like to use it even
> when it's not optimal... :-/

I doubt that ``many of us'' are familiar only with one tool.  Or
should be, anyway.

I'm not saying I'm against pushing the limit up, but I don't expect
from an interactive editor to be able to edit 200MB files without
problems.  So I simply don't think about it as the tool of choice in
such cases.

Re: Maximum buffer size exceeded

[Michael Welsh Duggan]

Harish Narayanan <harish@gamebox.net> writes:

> Thanks Kin.
>
> I do things like that when I need to edit the last or first few
> lines. What if the portions I need edited are at some arbitrary line
> numbers in the middle? I don't always know exactly where they will be.

I had to deal with this recently for some files, as I ended up using
something like "split -C 100m".  You can use cat to rejoin them
afterwards.

> Kin Cho wrote:
>
>>Harish Narayanan <harish@gamebox.net> writes:
>>
>>  
>>
>>>I've been trying to open a large file (208 MB) but I receive the
>>>error "Maximum buffer size exceeded" and Emacs fails to open the
>>>file. This is Emacs 21.2.1 on a Redhat GNU/Linux 9 box.
>>>
>>>I tried searching in the archives, and noticed this has been
>>>asked before, but the responses don't seem encouraging. Is there
>>>a way of working around this that's arisen in the recent past?
>>>    
>>>
>>
>>Use head/tail/dd to extract the small part of the file you want
>>to edit, edit that small file, then use cat/head/tail/dd to put
>>it back into your original file.
>>
>>Let's say you want to edit the last line of the file 208MB, which
>>is 1 million lines long:
>>
>>tail -1 208MB > foo
>>emacs foo
>>(head -999999 208MB; cat foo) > new-208MB
>>
>>-kin
>>
>>_______________________________________________
>>Help-gnu-emacs mailing list
>>Help-gnu-emacs@gnu.org
>>http://mail.gnu.org/mailman/listinfo/help-gnu-emacs
>>
>>  
>>
>
>
>

-- 
Michael Welsh Duggan
(md5i@cs.cmu.edu)

Re: Maximum buffer size exceeded

[upro]

Michael Welsh Duggan <md5i@cs.cmu.edu> writes:

> Harish Narayanan <harish@gamebox.net> writes:
>
>> Thanks Kin.
>>
>> I do things like that when I need to edit the last or first few
>> lines. What if the portions I need edited are at some arbitrary line
>> numbers in the middle? I don't always know exactly where they will be.
>
> I had to deal with this recently for some files, as I ended up using
> something like "split -C 100m".  You can use cat to rejoin them
> afterwards.

I'm sorry if what I say is wrong but I think to remember that there is
a compile-time option which, when set, enables emacs to open larger
files than the usual compiled version. Not sure how big though...

Greetings, Michael

>
>> Kin Cho wrote:
>>
>>>Harish Narayanan <harish@gamebox.net> writes:
>>>
>>>  
>>>
>>>>I've been trying to open a large file (208 MB) but I receive the
>>>>error "Maximum buffer size exceeded" and Emacs fails to open the
>>>>file. This is Emacs 21.2.1 on a Redhat GNU/Linux 9 box.
>>>>
>>>>I tried searching in the archives, and noticed this has been
>>>>asked before, but the responses don't seem encouraging. Is there
>>>>a way of working around this that's arisen in the recent past?
>>>>    
>>>>
>>>
>>>Use head/tail/dd to extract the small part of the file you want
>>>to edit, edit that small file, then use cat/head/tail/dd to put
>>>it back into your original file.
>>>
>>>Let's say you want to edit the last line of the file 208MB, which
>>>is 1 million lines long:
>>>
>>>tail -1 208MB > foo
>>>emacs foo
>>>(head -999999 208MB; cat foo) > new-208MB
>>>
>>>-kin
>>>
>>>_______________________________________________
>>>Help-gnu-emacs mailing list
>>>Help-gnu-emacs@gnu.org
>>>http://mail.gnu.org/mailman/listinfo/help-gnu-emacs
>>>
>>>  
>>>
>>
>>
>>

-- 
Michael

r-znvy: zvpunry.wryqra  jro.qr (chg gur "@" jurer vg svgf...)
ab fcnz cyrnfr

Re: Maximum buffer size exceeded

[Matthieu Moy]

Harish Narayanan <harish@gamebox.net> writes:

> Hello all.

Hello,

> I've been trying to open a large file (208 MB) but I receive the error
> "Maximum buffer size exceeded" and Emacs fails to open the file.

This is because cursor position are stored in a lisp pointer, which is
coded on a bit less than 32 bits. 

Anyway, a 208 MB file in  Emacs would require a *really* big amount of
RAM. 

So,  you  have  two options  in  addition  to  what has  already  been
suggested: Use  a 64bits  machine with  a lot of  RAM, or  use another
editor for this specific task.

-- 
Matthieu

Re: Maximum buffer size exceeded

[Stefan Monnier]

> I've been trying to open a large file (208 MB) but I receive the error
> "Maximum buffer size exceeded" and Emacs fails to open the file. This is
> Emacs 21.2.1 on a Redhat GNU/Linux 9 box.

Try Emacs-CVS whose upper limit was bumped from 128MB to 256MB.


        Stefan

maximum buffer size exceeded

[Daniel C. Bastos]

I'm dealing with an INBOX of size 252 MiB and I expect it to double on
my next mail downloads. I use VM under the GNU emacs. How can I increase
the buffer?

Ideally, I would finish my downloading; my mailbox will be about 530 MiB
and then I will compress it and save it as a backup. And then I can
delete mail. Any help?

On 

http://www.gnu.org/software/emacs/manual/html_node/emacs/Buffers.html#index-buffer-size_002c-maximum-1081

I read bad news: ``buffer's size cannot be larger than some maximum,
which is defined by the largest buffer position representable by the
Emacs integer data type. This is because Emacs tracks buffer positions
using that data type. For 32-bit machines, the largest buffer size is
256 megabytes.''

That's pretty much how large my INBOX is. Just thought of asking some
opinions. I suppose I will just have save my mail in three INBOXes. 

Re: maximum buffer size exceeded

[Emilio Lopes]

Daniel C Bastos writes:

> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
> my next mail downloads. I use VM under the GNU emacs. How can I increase
> the buffer?

Does VM support other formats for mail boxes?  You could try using the
MH format for your mail boxes, if VM supports it.  It stores one
message per file.

Gnus supports MH mail boxes and the nnml format, a Gnus specific
one-file-per-message format.

-- 
Emílio C. Lopes
Munich, Germany

Re: maximum buffer size exceeded

[Daniel C. Bastos]

Emilio Lopes <eclig@gmx.net> writes:

> Daniel C Bastos writes:
>
>> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
>> my next mail downloads. I use VM under the GNU emacs. How can I increase
>> the buffer?
>
> Does VM support other formats for mail boxes?  You could try using the
> MH format for your mail boxes, if VM supports it.  It stores one
> message per file.

I think does; and that's a good idea. Thanks.

Re: maximum buffer size exceeded

[Tim X]

dbast0s@yahoo.com.br (Daniel C. Bastos) writes:

> Emilio Lopes <eclig@gmx.net> writes:
>
>> Daniel C Bastos writes:
>>
>>> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
>>> my next mail downloads. I use VM under the GNU emacs. How can I increase
>>> the buffer?
>>
>> Does VM support other formats for mail boxes?  You could try using the
>> MH format for your mail boxes, if VM supports it.  It stores one
>> message per file.
>
> I think does; and that's a good idea. Thanks.

I'm pretty sure your out of luck - VM doesn't support mh or maildir folders
as far as I know.

If your  running on a 32 bit machine, I think your out of luck. You cannot
increase the maximum buffer size. Under Emacs, I *think* you are limited to
a maximum integer you can represent with 28 bits. If your on a 64 bit
system, its the maximum you can represent with 62 bits. 

Note that there is a difference between emacs and XEmacs. I believe XEmacs
has the ability to represent 31 bit integers using some trickery with tag
bits.  

The best suggestion I have is to use some other editor to open your mail
file and split it into two files at a sensible position (i.e. between
messages. You should then be able to open these files with VM and sort them
into smaller folders. I'd suggest using something like procmail to split
your download - maybe use fetchmail/getmail to retrieve the mail and split
it  into different folders. 

Of course, if you have access to a 64 bit machine, you could move your
folder there, run a 64 bit verion of emacs and split things how you want,
move it back and your done - a hassle, but a solution none th eless.

Tim

-- 
tcross (at) rapttech dot com dot au

Re: maximum buffer size exceeded

[Daniel C. Bastos]

Tim X <timx@nospam.dev.null> writes:

> dbast0s@yahoo.com.br (Daniel C. Bastos) writes:
>
>> Emilio Lopes <eclig@gmx.net> writes:
>>
>>> Daniel C Bastos writes:
>>>
>>>> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
>>>> my next mail downloads. I use VM under the GNU emacs. How can I increase
>>>> the buffer?
>>>
>>> Does VM support other formats for mail boxes?  You could try using the
>>> MH format for your mail boxes, if VM supports it.  It stores one
>>> message per file.
>>
>> I think does; and that's a good idea. Thanks.
>
> I'm pretty sure your out of luck - VM doesn't support mh or maildir folders
> as far as I know.

You're right. If your mail is stored in maildir format, then it seems
that you can get that to co-exist with VM as Arnaldo Mandel suggests at

           http://www.wonderworks.com/vm/vm-faq-4.html#ss4.5

but VM doesn't support maildir, nevertheless.

> If your  running on a 32 bit machine, I think your out of luck. You cannot
> increase the maximum buffer size. Under Emacs, I *think* you are limited to
> a maximum integer you can represent with 28 bits. If your on a 64 bit
> system, its the maximum you can represent with 62 bits. 
>
> Note that there is a difference between emacs and XEmacs. I believe XEmacs
> has the ability to represent 31 bit integers using some trickery with tag
> bits.  
>
> The best suggestion I have is to use some other editor to open your mail
> file and split it into two files at a sensible position (i.e. between
> messages. You should then be able to open these files with VM and sort them
> into smaller folders. I'd suggest using something like procmail to split
> your download - maybe use fetchmail/getmail to retrieve the mail and split
> it  into different folders. 

Actually VM won't let your mail file grow past of what it can handle, so
I didn't run into bigger problems. After downloading the chunk that
exceeds (that happens to be INBOX.CRASH), VM or emacs tells you that it
won't handle that much, so you're forced to give up on integrating the
new mail into your mailbox. So I moved my INBOX to something else and
integrated INBOX.CRASH into a new INBOX. I ended up with three chunks
which is now backed up.

> Of course, if you have access to a 64 bit machine, you could move your
> folder there, run a 64 bit verion of emacs and split things how you want,
> move it back and your done - a hassle, but a solution none th eless.

A hassle indeed, and I don't have a 64 bit machine.

Re: maximum buffer size exceeded

[Eli Zaretskii]

> From: Tim X <timx@nospam.dev.null>
> Date: Mon, 03 Sep 2007 18:18:03 +1000
> 
> If your  running on a 32 bit machine, I think your out of luck. You cannot
> increase the maximum buffer size. Under Emacs, I *think* you are limited to
> a maximum integer you can represent with 28 bits.

29

> If your on a 64 bit system, its the maximum you can represent with
> 62 bits.

61

(It's the size of an int minus 3.)

Re: maximum buffer size exceeded

[Johan Bockgård]

Tim X <timx@nospam.dev.null> writes:

> Note that there is a difference between emacs and XEmacs. I believe XEmacs
> has the ability to represent 31 bit integers using some trickery with tag
> bits.  

Also, beta versions of XEmacs can be built with bignum support
(arbitrary precision integers).

-- 
Johan Bockgård

Re: maximum buffer size exceeded

[Giorgos Keramidas]

On Mon, 03 Sep 2007 22:24:46 +0300, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tim X <timx@nospam.dev.null>
>> Date: Mon, 03 Sep 2007 18:18:03 +1000
>>
>> If your  running on a 32 bit machine, I think your out of luck. You cannot
>> increase the maximum buffer size. Under Emacs, I *think* you are limited to
>> a maximum integer you can represent with 28 bits.
>
> 29
>
>> If your on a 64 bit system, its the maximum you can represent with
>> 62 bits.
>
> 61
>
> (It's the size of an int minus 3.)

Hi Eli,

this is probably a silly question to ask, because I don't know the
internals of Emacs so well; it's just a thought which I had though.

Are there plans to make the buffer size use one of the types with "at
least X bits", like uint32_t or uint64_t?  Do you think there is a good
reason *not* to do this (i.e. an absurd performance penalty on some of
the supported platforms)?

- Giorgos

Re: maximum buffer size exceeded

[Peter Dyballa]

Am 04.09.2007 um 14:13 schrieb Giorgos Keramidas:

> Are there plans to make the buffer size use one of the types with "at
> least X bits", like uint32_t or uint64_t?  Do you think there is a  
> good
> reason *not* to do this (i.e. an absurd performance penalty on some of
> the supported platforms)?

Because of performance issues exactly these aligned data types are  
used – and a small part, those mentioned three bits, is reserved for  
Lisp, to record the Lisp type of the variable.

--
Greetings

   Pete

The most exciting phrase to hear in science, the one that heralds new  
discoveries, is not "Eureka!" (I found it!) but "That's funny..."
                                       Isaac Asimov

Re: maximum buffer size exceeded

[Emilio Lopes]

Peter Dyballa writes:

> Because of performance issues exactly these aligned data types are
> used – and a small part, those mentioned three bits, is reserved for
> Lisp, to record the Lisp type of the variable.

A small nitpick: the Lisp type of the *value*.  In Lisp the type goes
with the values not with the addresses pointing to them (variables).

-- 
Emílio C. Lopes
Munich, Germany

Re: maximum buffer size exceeded

[Giorgos Keramidas]

On Tue, 4 Sep 2007 16:48:46 +0200, Peter Dyballa <Peter_Dyballa@Web.DE> wrote:
>Am 04.09.2007 um 14:13 schrieb Giorgos Keramidas:
>> Are there plans to make the buffer size use one of the types
>> with "at least X bits", like uint32_t or uint64_t?  Do you
>> think there is a good reason *not* to do this (i.e. an absurd
>> performance penalty on some of the supported platforms)?
>
> Because of performance issues exactly these aligned data types
> are used – and a small part, those mentioned three bits, is
> reserved for Lisp, to record the Lisp type of the variable.

I got the reason why 3 bits are reserved, thanks :)

My question (which was probably phrased badly) was ``is there any
reason not to use uint64_t on all platforms?''.  The answer to
this may be ``Yes, there is.  Using uint64_t will unnecessarily
pessimize performance on 32-bit only systems'', or even ``Yes
there is.  Using uint64_t will make it impossible to build Emacs
with Foo C Compiler on Bar platform''.

I'm not acquainted with all the possible platforms or compiler
suites Emacs should normally build fine with, so that's why I
wondered :-)

- Giorgos

Re: maximum buffer size exceeded

[Eli Zaretskii]

> From: Giorgos Keramidas <keramida@ceid.upatras.gr>
> Date: Tue, 04 Sep 2007 15:13:45 +0300
> 
> Are there plans to make the buffer size use one of the types with "at
> least X bits", like uint32_t or uint64_t?

If there are such plans, I'm not aware of them.

> Do you think there is a good reason *not* to do this (i.e. an absurd
> performance penalty on some of the supported platforms)?

Performance is one reason; less than universal support for those C9x
features is another.

I think the current view in Emacs development is that 64-bit platforms
solve this problem so easily that its solution for 32-bit machines is
much less important than working on other Emacs features.

Re: maximum buffer size exceeded

[Kim F. Storm]

Eli Zaretskii <eliz@gnu.org> writes:

> I think the current view in Emacs development is that 64-bit platforms
> solve this problem so easily that its solution for 32-bit machines is
> much less important than working on other Emacs features.

Actually, I think a small trick could increase the buffer size to 1 GB
on 32 bit machines at the cost of a little(?) wasted memory.

[Note: Assuming USE_LSB_TAG is defined]

Currently, we have the lowest 3 bits reserved for the Lisp Type,
meaning that the largest positive Emacs integer is 2^28-1 (256MB).

Now, consider if we reserve 4 bits for the Lisp Type, but
in such a way the Lisp_Int == 0, while the other Lisp types
are odd numbers 1,3,5,7,...

In this setup, an integer can be recognized by looking at the lowest
bit alone (== 0), while the other Lisp types are recognized using the
current methods (looking at all 4 type bits).

The only drawback I can see is that Lisp_Objects have to be allocated
on 16 byte boundaries rather than the current 8 byte boundary, so a
little space may be wasted (and maybe not...).

I haven't tried this, but given that Lisp_Objects are usually accessed
via suitable macros, it looks quite doable.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: maximum buffer size exceeded

[Stefan Monnier]

>> I think the current view in Emacs development is that 64-bit platforms
>> solve this problem so easily that its solution for 32-bit machines is
>> much less important than working on other Emacs features.

> Actually, I think a small trick could increase the buffer size to 1 GB
> on 32 bit machines at the cost of a little(?) wasted memory.

> [Note: Assuming USE_LSB_TAG is defined]

> Currently, we have the lowest 3 bits reserved for the Lisp Type,
> meaning that the largest positive Emacs integer is 2^28-1 (256MB).

> Now, consider if we reserve 4 bits for the Lisp Type, but
> in such a way the Lisp_Int == 0, while the other Lisp types
> are odd numbers 1,3,5,7,...

> In this setup, an integer can be recognized by looking at the lowest
> bit alone (== 0), while the other Lisp types are recognized using the
> current methods (looking at all 4 type bits).

> The only drawback I can see is that Lisp_Objects have to be allocated
> on 16 byte boundaries rather than the current 8 byte boundary, so a
> little space may be wasted (and maybe not...).

> I haven't tried this, but given that Lisp_Objects are usually accessed
> via suitable macros, it looks quite doable.

Increasing from 8 to 16 bytes alignment may be a non-trivial problem:
1 - cons cells use 8 bytes right now, so you'd waste a lot of space for them.
2 - same for floats.
3 - in many places, we rely on malloc to align objects on multiple of 8, so
    we'd have to use some other approach.

Numbers 1 and 2 can be solved by giving two tags to cons and floats, so they
only need alignment on multiple of 8.

Number 3 is more work.  But this work may be the same as the one needed to
allow us to use USE_LSB_TAG everywhere (even on machines where malloc and
static-vars do not guarantee mult-of-8 alignment).

We currently have 7 different types (of the 8 possible tag we only use 7).

My own local Emacs build uses the trick you suggest but on the 3bits of
tags, so I gave 2 tags to integers to allow them to grow up the 2^29
(i.e. max buffer size = 512MB).  That's a very simple change.

What you suggest would be to use 4 bits i.e. 16 possible tags:
- 8 tags for integers (i.e. 8 tags left for the 6 other types)
- 2 tags for cons cells (6 tags left for the 5 other types)
- 2 tags for floats
- one tag each for the remaining 4 types (arrays, symbols, strings, misc).

One other problem: currently `misc' objects need 5 32bit words which
USE_LSB_TAG forced to round up to 6 32bit words and symbols use 6 32bit
words.  So rounding up to mult-of-16 would round them both up to
8 32bit words.

The two subtypes of misc which use up 5 words are markers and overlays.
So with your rounding up, an overlay would use up 3*8=24 words (3 because
there's the overlay object plus the two associated marker objects) instead
of 15 (without USE_LSB_TAG) or 18.

I had plans to try and squeeze `misc' objects down to 4 words (and hence
overlays down to 12 words), but this is a non-trivial change.  One possible
approach is to replace the linked lists of overlays and markers by arrays
(managed just like buffer text: with a gap).

Another option is to remove the `symbol' and `string' tags and make symbols
and strings subtype of `misc'.  Then we could keep 3 tag bits and give 4 of
the 8 tags to integers.  This would simplify the alloc.c code but would also
waste more memory (6 words for string objects) and slow down SYMBOLP and
STRINGP slightly.

Still, the fundamental problem remains the same: files larger than 256MB
are most likely not generated manually.  So they may very likely grow to
more than 4GB tomorrow.  Bumping the limit to 512MB or 1GB (or even 4GB for
that matter) is only going to help in some fraction of the cases.

I think a better approach to handle this problem is to create a special
package to visit arbitrarily large files which would work by loading only
parts of the file at a time and do manual "swapping".  This would not work
as smoothly, but then again manipulating 256MB files in Emacs is currently
not that smooth either.


        Stefan


PS: You can supposedly open >4GB files in Emacs with 64bit systems, but
looking at the C code, it's clear that you'll bump into bugs where we cast
EMACS_INT values to and from `int' (which on many 64bit systems are only
32bit).  I tend to fix those bugs when I bump into them, but they're
everywhere and I've fixed only a tiny fraction of them.

Re: maximum buffer size exceeded

[Jason Rumney]

Stefan Monnier wrote:
> PS: You can supposedly open >4GB files in Emacs with 64bit systems, but
> looking at the C code, it's clear that you'll bump into bugs where we cast
> EMACS_INT values to and from `int' (which on many 64bit systems are only
> 32bit).  I tend to fix those bugs when I bump into them, but they're
> everywhere and I've fixed only a tiny fraction of them.
>   
long is also 32 bits on 64bit versions of Windows, so avoid simply
replacing int with long.

Re: maximum buffer size exceeded

[Stefan Monnier]

>> PS: You can supposedly open >4GB files in Emacs with 64bit systems, but
>> looking at the C code, it's clear that you'll bump into bugs where we cast
>> EMACS_INT values to and from `int' (which on many 64bit systems are only
>> 32bit).  I tend to fix those bugs when I bump into them, but they're
>> everywhere and I've fixed only a tiny fraction of them.
>> 
> long is also 32 bits on 64bit versions of Windows, so avoid simply
> replacing int with long.

Of course, it should be replaced with EMACS_INT or EMACS_UINT.


        Stefan

Re: maximum buffer size exceeded

[Richard Stallman]

    The only drawback I can see is that Lisp_Objects have to be allocated
    on 16 byte boundaries rather than the current 8 byte boundary, so a
    little space may be wasted (and maybe not...).

For cons cells and floats, it would mean half the space is wasted.
Markers and symbols and miscs will also waste space, but a smaller
fraction.

It would be useful to calculate the expected amount of waste in some
real Emacs jobs, and compare that with the total memory usage.

Re: maximum buffer size exceeded

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     The only drawback I can see is that Lisp_Objects have to be allocated
>     on 16 byte boundaries rather than the current 8 byte boundary, so a
>     little space may be wasted (and maybe not...).
>
> For cons cells and floats, it would mean half the space is wasted.

Maybe we'll need one higher order bit after all.  Or we let every cons
cell have a siamese twin float in its second half.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: maximum buffer size exceeded

[Eli Zaretskii]

> From: dbast0s@yahoo.com.br (Daniel C. Bastos)
> Date: Sun, 02 Sep 2007 02:44:44 -0400
> 
> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
> my next mail downloads. I use VM under the GNU emacs. How can I increase
> the buffer?

The easiest way is to run Emacs on a 64-bit machine (and compile it as
a 64-bit executable).

Re: maximum buffer size exceeded

[Glenn Morris]

Daniel C. Bastos wrote:

> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
> my next mail downloads. I use VM under the GNU emacs.

Out of interest, do you find this usable? I find VM intolerably slow
(on fast machines) with mail folders larger than a few MB. Perhaps
it's my font-locking which is to blame...

Re: maximum buffer size exceeded

[Tim X]

Glenn Morris <rgm+news@stanford.edu> writes:

> Daniel C. Bastos wrote:
>
>> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
>> my next mail downloads. I use VM under the GNU emacs.
>
> Out of interest, do you find this usable? I find VM intolerably slow
> (on fast machines) with mail folders larger than a few MB. Perhaps
> it's my font-locking which is to blame...

I have no problems with VM speed and I've got some fairly large
mailboxes. I have got the indexing of mail folders enabled and I use the
perl modules rather than the included VM binaries for doing
encoding/decoding etc. I also use quite a few extensions, including pcrisis
and add on packages for font-locking the summary buffer and messages window
etc. 

I don't use VMs ipop/imap support, preferring to use other programs to
fetch my mail from various places, run it through spamassassin and use
procmail.to sort it into various folders. I've also been trying out vm 8.x
and the latest bleeding edge dev tree. 

Tim



-- 
tcross (at) rapttech dot com dot au

Re: maximum buffer size exceeded

[Glenn Morris]

Tim X wrote:

> I have no problems with VM speed and I've got some fairly large
> mailboxes. I have got the indexing of mail folders enabled

What is "the indexing"? Is it that XEmacs-only feature that hooks it
into some database backend?

Re: maximum buffer size exceeded

[Tim X]

Glenn Morris <rgm+news@stanford.edu> writes:

> Tim X wrote:
>
>> I have no problems with VM speed and I've got some fairly large
>> mailboxes. I have got the indexing of mail folders enabled
>
> What is "the indexing"? Is it that XEmacs-only feature that hooks it
> into some database backend?

No, I don't run XEmacs. You can configure folder indexing - it creates .idx
files for your folder. To what extent it improves performance, I don't
know.

Tim

-- 
tcross (at) rapttech dot com dot au

Re: maximum buffer size exceeded

[Daniel C. Bastos]

Glenn Morris <rgm+news@stanford.edu> writes:

> Daniel C. Bastos wrote:
>
>> I'm dealing with an INBOX of size 252 MiB and I expect it to double on
>> my next mail downloads. I use VM under the GNU emacs.
>
> Out of interest, do you find this usable? I find VM intolerably slow
> (on fast machines) with mail folders larger than a few MB. Perhaps
> it's my font-locking which is to blame...

I have only days of experience with VM, and only hours with a mailbox
that big; I disabled backups to avoid emacs saving buffers all the time
which was a slow task and has nothing to do with VM. Sorting and summary
generation was quick enough.