unofficial mirror of help-gnu-emacs@gnu.org
 help / color / mirror / Atom feed
* gnus, Maximum buffer size exceeded
@ 2015-07-23 21:30 Jason Hunter
  2015-07-23 21:31 ` Jason Hunter
  2015-07-28 23:33 ` Jason Hunter
  0 siblings, 2 replies; 19+ messages in thread
From: Jason Hunter @ 2015-07-23 21:30 UTC (permalink / raw)
  To: help-gnu-emacs

I got a problem with gnus; I downloaded quite a big pop3 account today,
cause I've been on vacation
                                \



Processing mail from ~/.emacs-mail-crash-box... Mail source (pop:server
mail.xxxx :user foo@bar :password xxxx) failed: (error Maximum buffer size
exceeded)



..so, basically, I can't start gnus;)





Is there some way to increase this limit?





I'm on 32-bit GNU/Linux


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-23 21:30 gnus, Maximum buffer size exceeded Jason Hunter
@ 2015-07-23 21:31 ` Jason Hunter
  2015-07-24  0:53   ` John Mastro
  2015-07-28 23:33 ` Jason Hunter
  1 sibling, 1 reply; 19+ messages in thread
From: Jason Hunter @ 2015-07-23 21:31 UTC (permalink / raw)
  To: help-gnu-emacs

Also, the size of the file is about 400Mio

2015-07-23 23:30 GMT+02:00 Jason Hunter <jhunterwu@gmail.com>:

> I got a problem with gnus; I downloaded quite a big pop3 account today,
> cause I've been on vacation
>                                 \
>
>
>
> Processing mail from ~/.emacs-mail-crash-box... Mail source (pop:server
> mail.xxxx :user foo@bar :password xxxx) failed: (error Maximum buffer
> size exceeded)
>
>
>
> ..so, basically, I can't start gnus;)
>
>
>
>
>
> Is there some way to increase this limit?
>
>
>
>
>
> I'm on 32-bit GNU/Linux
>
>
>
>
>
>
>
>
>
>
>


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-23 21:31 ` Jason Hunter
@ 2015-07-24  0:53   ` John Mastro
  2015-07-24  6:24     ` Eli Zaretskii
  0 siblings, 1 reply; 19+ messages in thread
From: John Mastro @ 2015-07-24  0:53 UTC (permalink / raw)
  To: help-gnu-emacs@gnu.org; +Cc: Jason Hunter


> Also, the size of the file is about 400Mio
>> I'm on 32-bit GNU/Linux

If I understand correctly, there's a buffer size limit of 128mb on 32
bit machines. 

http://www.emacswiki.org/emacs/EmacsFileSizeLimit

Perhaps you could use a 64 bit machine to get caught up?

-- 
john



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-24  0:53   ` John Mastro
@ 2015-07-24  6:24     ` Eli Zaretskii
  2015-07-24  7:00       ` Eli Zaretskii
  0 siblings, 1 reply; 19+ messages in thread
From: Eli Zaretskii @ 2015-07-24  6:24 UTC (permalink / raw)
  To: help-gnu-emacs

> From: John Mastro <john.b.mastro@gmail.com>
> Date: Thu, 23 Jul 2015 17:53:44 -0700
> Cc: Jason Hunter <jhunterwu@gmail.com>
> 
> > Also, the size of the file is about 400Mio
> >> I'm on 32-bit GNU/Linux
> 
> If I understand correctly, there's a buffer size limit of 128mb on 32
> bit machines. 

No, it's 512 MB since Emacs 23.2.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-24  6:24     ` Eli Zaretskii
@ 2015-07-24  7:00       ` Eli Zaretskii
       [not found]         ` <CAChpLT=0+4tsPiNyAQXucyNFEPOfJ0qMFodmi3eaZL1K=6AQEA@mail.gmail.com>
  0 siblings, 1 reply; 19+ messages in thread
From: Eli Zaretskii @ 2015-07-24  7:00 UTC (permalink / raw)
  To: help-gnu-emacs

> Date: Fri, 24 Jul 2015 09:24:01 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: John Mastro <john.b.mastro@gmail.com>
> > Date: Thu, 23 Jul 2015 17:53:44 -0700
> > Cc: Jason Hunter <jhunterwu@gmail.com>
> > 
> > > Also, the size of the file is about 400Mio
> > >> I'm on 32-bit GNU/Linux
> > 
> > If I understand correctly, there's a buffer size limit of 128mb on 32
> > bit machines. 
> 
> No, it's 512 MB since Emacs 23.2.

But of course, a 400MB file could exceed 512MB when it's decoded
(assuming there were non-ASCII characters there).



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
       [not found]         ` <CAChpLT=0+4tsPiNyAQXucyNFEPOfJ0qMFodmi3eaZL1K=6AQEA@mail.gmail.com>
@ 2015-07-25  7:21           ` Eli Zaretskii
  2015-07-29  0:11             ` Glenn Morris
  2015-07-30  6:52             ` Stefan Monnier
  0 siblings, 2 replies; 19+ messages in thread
From: Eli Zaretskii @ 2015-07-25  7:21 UTC (permalink / raw)
  To: help-gnu-emacs

> Date: Sat, 25 Jul 2015 01:32:00 +0200
> From: Jason Hunter <jhunterwu@gmail.com>
> 
> So, there's no way I could increase this limit on my 32-bit system?

Emacs 25 will support a configure time option to extend this to 2GB
(or maybe even 4GB, depending on the OS).  Otherwise, no, there's no
way.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-23 21:30 gnus, Maximum buffer size exceeded Jason Hunter
  2015-07-23 21:31 ` Jason Hunter
@ 2015-07-28 23:33 ` Jason Hunter
  2015-08-03  1:24   ` Eric Brown
  1 sibling, 1 reply; 19+ messages in thread
From: Jason Hunter @ 2015-07-28 23:33 UTC (permalink / raw)
  To: help-gnu-emacs

Yeah, sorry, Eli Zaretskii; I'm trying to figure out how to use gmail,
which I don't really

> Emacs 25 will support a configure time option to extend this to 2GB
> (or maybe even 4GB, depending on the OS).  Otherwise, no, there's no
> way.

I find this so difficult to believe. I'm in deep trouble here. I can't
start my gnus.

Is there any way to cut this file in three or four and process it one by one?

Can I do it manually and then just restart gnus several times with a
smaller file?

I mean, this can't bring down the whole nuclear powerplant?



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-25  7:21           ` Eli Zaretskii
@ 2015-07-29  0:11             ` Glenn Morris
  2015-07-29  2:47               ` Eli Zaretskii
  2015-07-30  6:52             ` Stefan Monnier
  1 sibling, 1 reply; 19+ messages in thread
From: Glenn Morris @ 2015-07-29  0:11 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: help-gnu-emacs

Eli Zaretskii wrote:

> Emacs 25 will support a configure time option to extend this to 2GB
> (or maybe even 4GB, depending on the OS).

Do you mean --with-wide-int? That was added in 24.1.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-29  0:11             ` Glenn Morris
@ 2015-07-29  2:47               ` Eli Zaretskii
  0 siblings, 0 replies; 19+ messages in thread
From: Eli Zaretskii @ 2015-07-29  2:47 UTC (permalink / raw)
  To: help-gnu-emacs

> From: Glenn Morris <rgm@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Tue, 28 Jul 2015 20:11:07 -0400
> 
> Eli Zaretskii wrote:
> 
> > Emacs 25 will support a configure time option to extend this to 2GB
> > (or maybe even 4GB, depending on the OS).
> 
> Do you mean --with-wide-int? That was added in 24.1.

Yes, sorry.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-25  7:21           ` Eli Zaretskii
  2015-07-29  0:11             ` Glenn Morris
@ 2015-07-30  6:52             ` Stefan Monnier
  2015-07-30 23:36               ` Jason Hunter
  1 sibling, 1 reply; 19+ messages in thread
From: Stefan Monnier @ 2015-07-30  6:52 UTC (permalink / raw)
  To: help-gnu-emacs

>> So, there's no way I could increase this limit on my 32-bit system?
> Emacs 25 will support a configure time option to extend this to 2GB
> (or maybe even 4GB, depending on the OS).  Otherwise, no, there's no way.

As Glenn points out, this option was actually added to
Emacs-24 already.  And in practice, the real practical limit can end up
lower than "2GB or 4GB" (because Emacs additionally needs space to keep
other data, and/or to move part of the buffer around).


        Stefan




^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-30  6:52             ` Stefan Monnier
@ 2015-07-30 23:36               ` Jason Hunter
  2015-07-31  6:50                 ` Glenn Morris
  2015-07-31  7:46                 ` Stefan Monnier
  0 siblings, 2 replies; 19+ messages in thread
From: Jason Hunter @ 2015-07-30 23:36 UTC (permalink / raw)
  To: help-gnu-emacs

2015-07-30 8:52 GMT+02:00 Stefan Monnier <monnier@iro.umontreal.ca>:
>>> So, there's no way I could increase this limit on my 32-bit system?
>> Emacs 25 will support a configure time option to extend this to 2GB
>> (or maybe even 4GB, depending on the OS).  Otherwise, no, there's no way.
>
> As Glenn points out, this option was actually added to
> Emacs-24 already.  And in practice, the real practical limit can end up
> lower than "2GB or 4GB" (because Emacs additionally needs space to keep
> other data, and/or to move part of the buffer around).
>

Thanks for following up on this, cause I really need help;)

Ok, so, I'm using emacs on ubuntu-14.10 and this compile time flag has
not been added then?

My file is only like 400Mio.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-30 23:36               ` Jason Hunter
@ 2015-07-31  6:50                 ` Glenn Morris
  2015-07-31  7:46                 ` Stefan Monnier
  1 sibling, 0 replies; 19+ messages in thread
From: Glenn Morris @ 2015-07-31  6:50 UTC (permalink / raw)
  To: Jason Hunter; +Cc: help-gnu-emacs

Jason Hunter wrote:

> Ok, so, I'm using emacs on ubuntu-14.10 and this compile time flag has
> not been added then?

I doubt there is a pre-built binary with this option on, no.
You could build your own with:


sudo apt-get build-dep emacs24
wget http://ftpmirror.gnu.org/emacs/emacs-24.5.tar.xz
tar -avxf emacs-24.5.tar.xz
cd emacs-24.5
./configure --with-wide-int
make
sudo make install



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-30 23:36               ` Jason Hunter
  2015-07-31  6:50                 ` Glenn Morris
@ 2015-07-31  7:46                 ` Stefan Monnier
  2015-07-31  8:11                   ` Eli Zaretskii
  1 sibling, 1 reply; 19+ messages in thread
From: Stefan Monnier @ 2015-07-31  7:46 UTC (permalink / raw)
  To: help-gnu-emacs

> Ok, so, I'm using emacs on ubuntu-14.10 and this compile time flag has
> not been added then?

Probably not.  I wouldn't encourage people to use it: it increases the
memory usage of Emacs (and hence slows it down by eating up your caches
more quickly), but only increases the maximum buffer size by a fairly
small amount.

If you bump into the 512MB limit, you're better off either using vlf.el
or using a 64bit build of Emacs (e.g. the one that comes with Ubuntu in
the amd64 version of Ubuntu).


        Stefan




^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-31  7:46                 ` Stefan Monnier
@ 2015-07-31  8:11                   ` Eli Zaretskii
  2015-07-31 21:25                     ` Stefan Monnier
  0 siblings, 1 reply; 19+ messages in thread
From: Eli Zaretskii @ 2015-07-31  8:11 UTC (permalink / raw)
  To: help-gnu-emacs

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 31 Jul 2015 03:46:33 -0400
> 
> I wouldn't encourage people to use it: it increases the memory usage
> of Emacs (and hence slows it down by eating up your caches more
> quickly), but only increases the maximum buffer size by a fairly
> small amount.

2GB vs 512MB doesn't sound like "fairly small amount" to me, more like
a factor of 4 and 1.5GB of absolute increment, not a small deal.

I agree that a 64-bit build is better (and spent some non-trivial
effort myself making that work at the time), but I don't agree with
discouraging users of 32-bit systems from using the wide-int build,
especially if switching to a different system is not an easy
alternative.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-31  8:11                   ` Eli Zaretskii
@ 2015-07-31 21:25                     ` Stefan Monnier
  2015-08-01  7:15                       ` Eli Zaretskii
  0 siblings, 1 reply; 19+ messages in thread
From: Stefan Monnier @ 2015-07-31 21:25 UTC (permalink / raw)
  To: help-gnu-emacs

>> I wouldn't encourage people to use it: it increases the memory usage
>> of Emacs (and hence slows it down by eating up your caches more
>> quickly), but only increases the maximum buffer size by a fairly
>> small amount.
> 2GB vs 512MB doesn't sound like "fairly small amount" to me, more like
> a factor of 4 and 1.5GB of absolute increment, not a small deal.

If you don't have a separate session just for this one file, or if you
want to edit the file, or if the file will be font-locked, or in several
other circumstances, the limit will be significantly smaller than 2GB.

And since manually-created 500MB files are extremely rare, your 600MB
might very well grow to 2.3GB tomorrow, so yes I consider the difference
between 512MB and 2GB to be pretty small in this context: it's unlikely
that --with-wide-int will satisfy all your needs if a standard build
doesn't already satisfy them.  I'm not saying it's never useful, just
that those cases where it's useful are rare.


        Stefan




^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-31 21:25                     ` Stefan Monnier
@ 2015-08-01  7:15                       ` Eli Zaretskii
  2015-08-03 10:07                         ` Stefan Monnier
  0 siblings, 1 reply; 19+ messages in thread
From: Eli Zaretskii @ 2015-08-01  7:15 UTC (permalink / raw)
  To: help-gnu-emacs

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 31 Jul 2015 17:25:40 -0400
> 
> >> I wouldn't encourage people to use it: it increases the memory usage
> >> of Emacs (and hence slows it down by eating up your caches more
> >> quickly), but only increases the maximum buffer size by a fairly
> >> small amount.
> > 2GB vs 512MB doesn't sound like "fairly small amount" to me, more like
> > a factor of 4 and 1.5GB of absolute increment, not a small deal.
> 
> If you don't have a separate session just for this one file, or if you
> want to edit the file, or if the file will be font-locked, or in several
> other circumstances, the limit will be significantly smaller than 2GB.

Please explain how each one of those makes the limit significantly
smaller, because I don't see it.  The gap is very small compared to
2GB, so editing the file incurs a small penalty.  The other 2 issues I
don't see how they are involved at all.  Maybe we have bugs that
should be fixed, as we had in the 64-bit builds not so long ago.

> And since manually-created 500MB files are extremely rare, your 600MB
> might very well grow to 2.3GB tomorrow

It may well grow also beyond what the total VM on a typical 64-bit
machine is, so this problem is always present.  But as long as it
didn't grow, the user still has the advantage of being able to visit
files she couldn't before.  IOW, the risk of exceeding the limit
becomes lower as the limit goes up.

> so yes I consider the difference between 512MB and 2GB to be pretty
> small in this context: it's unlikely that --with-wide-int will
> satisfy all your needs if a standard build doesn't already satisfy
> them.

It doesn't need to satisfy all of them, just some.  You have limits on
a 64-bit machine as well, and they are not so much farther.  I don't
see how the situation there is qualitatively different.

> I'm not saying it's never useful, just that those cases where it's
> useful are rare.

Not for me, they aren't.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-07-28 23:33 ` Jason Hunter
@ 2015-08-03  1:24   ` Eric Brown
  2015-08-03  1:29     ` Jason Hunter
  0 siblings, 1 reply; 19+ messages in thread
From: Eric Brown @ 2015-08-03  1:24 UTC (permalink / raw)
  To: Jason Hunter; +Cc: help-gnu-emacs


Jason Hunter <jhunterwu@gmail.com> writes:

>
> I find this so difficult to believe. I'm in deep trouble here. I 
> can't
> start my gnus.
>
> Is there any way to cut this file in three or four and process 
> it one by one?
>
> Can I do it manually and then just restart gnus several times 
> with a
> smaller file?
>
> I mean, this can't bring down the whole nuclear powerplant?

Being caught in this situation should be sufficient motivation to 
switch
to IMAP, Maildir, and/or a 64-bit operating system.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-08-03  1:24   ` Eric Brown
@ 2015-08-03  1:29     ` Jason Hunter
  0 siblings, 0 replies; 19+ messages in thread
From: Jason Hunter @ 2015-08-03  1:29 UTC (permalink / raw)
  To: help-gnu-emacs

On Aug 3, 2015 3:24 AM, "Eric Brown" <brown@fastmail.com> wrote:
>
>
> Jason Hunter <jhunterwu@gmail.com> writes:
>
>>
>> I find this so difficult to believe. I'm in deep trouble here. I can't
>> start my gnus.
>>
>> Is there any way to cut this file in three or four and process it one by
one?
>>
>> Can I do it manually and then just restart gnus several times with a
>> smaller file?
>>
>> I mean, this can't bring down the whole nuclear powerplant?
>
>
> Being caught in this situation should be sufficient motivation to switch
> to IMAP, Maildir, and/or a 64-bit operating system.

Actually, that is what I am doing.

I'm moving everything to imap on tuffmail which seems like the best idea.


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: gnus, Maximum buffer size exceeded
  2015-08-01  7:15                       ` Eli Zaretskii
@ 2015-08-03 10:07                         ` Stefan Monnier
  0 siblings, 0 replies; 19+ messages in thread
From: Stefan Monnier @ 2015-08-03 10:07 UTC (permalink / raw)
  To: help-gnu-emacs

> The gap is very small compared to 2GB, so editing the file incurs
> a small penalty.  The other 2 issues I don't see how they are involved
> at all.  Maybe we have bugs that should be fixed, as we had in the
> 64-bit builds not so long ago.

I don't think any of the issues can be considered bugs.  They're just
the kinds of problems you can start seeing when you get close to the
limit.

>> so yes I consider the difference between 512MB and 2GB to be pretty
>> small in this context: it's unlikely that --with-wide-int will
>> satisfy all your needs if a standard build doesn't already satisfy
>> them.
> It doesn't need to satisfy all of them, just some.  You have limits on
> a 64-bit machine as well, and they are not so much farther.  I don't
> see how the situation there is qualitatively different.

Qualitatively maybe not, but quantitatively the difference is very
significant in my experience.


        Stefan




^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2015-08-03 10:07 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-07-23 21:30 gnus, Maximum buffer size exceeded Jason Hunter
2015-07-23 21:31 ` Jason Hunter
2015-07-24  0:53   ` John Mastro
2015-07-24  6:24     ` Eli Zaretskii
2015-07-24  7:00       ` Eli Zaretskii
     [not found]         ` <CAChpLT=0+4tsPiNyAQXucyNFEPOfJ0qMFodmi3eaZL1K=6AQEA@mail.gmail.com>
2015-07-25  7:21           ` Eli Zaretskii
2015-07-29  0:11             ` Glenn Morris
2015-07-29  2:47               ` Eli Zaretskii
2015-07-30  6:52             ` Stefan Monnier
2015-07-30 23:36               ` Jason Hunter
2015-07-31  6:50                 ` Glenn Morris
2015-07-31  7:46                 ` Stefan Monnier
2015-07-31  8:11                   ` Eli Zaretskii
2015-07-31 21:25                     ` Stefan Monnier
2015-08-01  7:15                       ` Eli Zaretskii
2015-08-03 10:07                         ` Stefan Monnier
2015-07-28 23:33 ` Jason Hunter
2015-08-03  1:24   ` Eric Brown
2015-08-03  1:29     ` Jason Hunter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).