gnus, Maximum buffer size exceeded

gnus, Maximum buffer size exceeded

[Jason Hunter]

I got a problem with gnus; I downloaded quite a big pop3 account today,
cause I've been on vacation
                                \



Processing mail from ~/.emacs-mail-crash-box... Mail source (pop:server
mail.xxxx :user foo@bar :password xxxx) failed: (error Maximum buffer size
exceeded)



..so, basically, I can't start gnus;)





Is there some way to increase this limit?





I'm on 32-bit GNU/Linux

Re: gnus, Maximum buffer size exceeded

[Jason Hunter]

Also, the size of the file is about 400Mio

2015-07-23 23:30 GMT+02:00 Jason Hunter <jhunterwu@gmail.com>:

> I got a problem with gnus; I downloaded quite a big pop3 account today,
> cause I've been on vacation
>                                 \
>
>
>
> Processing mail from ~/.emacs-mail-crash-box... Mail source (pop:server
> mail.xxxx :user foo@bar :password xxxx) failed: (error Maximum buffer
> size exceeded)
>
>
>
> ..so, basically, I can't start gnus;)
>
>
>
>
>
> Is there some way to increase this limit?
>
>
>
>
>
> I'm on 32-bit GNU/Linux
>
>
>
>
>
>
>
>
>
>
>

Re: gnus, Maximum buffer size exceeded

[John Mastro]

> Also, the size of the file is about 400Mio
>> I'm on 32-bit GNU/Linux

If I understand correctly, there's a buffer size limit of 128mb on 32
bit machines. 

http://www.emacswiki.org/emacs/EmacsFileSizeLimit

Perhaps you could use a 64 bit machine to get caught up?

-- 
john

Re: gnus, Maximum buffer size exceeded

[Eli Zaretskii]

> From: John Mastro <john.b.mastro@gmail.com>
> Date: Thu, 23 Jul 2015 17:53:44 -0700
> Cc: Jason Hunter <jhunterwu@gmail.com>
> 
> > Also, the size of the file is about 400Mio
> >> I'm on 32-bit GNU/Linux
> 
> If I understand correctly, there's a buffer size limit of 128mb on 32
> bit machines. 

No, it's 512 MB since Emacs 23.2.

Re: gnus, Maximum buffer size exceeded

[Eli Zaretskii]

> Date: Fri, 24 Jul 2015 09:24:01 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: John Mastro <john.b.mastro@gmail.com>
> > Date: Thu, 23 Jul 2015 17:53:44 -0700
> > Cc: Jason Hunter <jhunterwu@gmail.com>
> > 
> > > Also, the size of the file is about 400Mio
> > >> I'm on 32-bit GNU/Linux
> > 
> > If I understand correctly, there's a buffer size limit of 128mb on 32
> > bit machines. 
> 
> No, it's 512 MB since Emacs 23.2.

But of course, a 400MB file could exceed 512MB when it's decoded
(assuming there were non-ASCII characters there).

Re: gnus, Maximum buffer size exceeded

[Eli Zaretskii]

> Date: Sat, 25 Jul 2015 01:32:00 +0200
> From: Jason Hunter <jhunterwu@gmail.com>
> 
> So, there's no way I could increase this limit on my 32-bit system?

Emacs 25 will support a configure time option to extend this to 2GB
(or maybe even 4GB, depending on the OS).  Otherwise, no, there's no
way.

Re: gnus, Maximum buffer size exceeded

[Jason Hunter]

Yeah, sorry, Eli Zaretskii; I'm trying to figure out how to use gmail,
which I don't really

> Emacs 25 will support a configure time option to extend this to 2GB
> (or maybe even 4GB, depending on the OS).  Otherwise, no, there's no
> way.

I find this so difficult to believe. I'm in deep trouble here. I can't
start my gnus.

Is there any way to cut this file in three or four and process it one by one?

Can I do it manually and then just restart gnus several times with a
smaller file?

I mean, this can't bring down the whole nuclear powerplant?

Re: gnus, Maximum buffer size exceeded

[Glenn Morris]

Eli Zaretskii wrote:

> Emacs 25 will support a configure time option to extend this to 2GB
> (or maybe even 4GB, depending on the OS).

Do you mean --with-wide-int? That was added in 24.1.

Re: gnus, Maximum buffer size exceeded

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: help-gnu-emacs@gnu.org
> Date: Tue, 28 Jul 2015 20:11:07 -0400
> 
> Eli Zaretskii wrote:
> 
> > Emacs 25 will support a configure time option to extend this to 2GB
> > (or maybe even 4GB, depending on the OS).
> 
> Do you mean --with-wide-int? That was added in 24.1.

Yes, sorry.

Re: gnus, Maximum buffer size exceeded

[Stefan Monnier]

>> So, there's no way I could increase this limit on my 32-bit system?
> Emacs 25 will support a configure time option to extend this to 2GB
> (or maybe even 4GB, depending on the OS).  Otherwise, no, there's no way.

As Glenn points out, this option was actually added to
Emacs-24 already.  And in practice, the real practical limit can end up
lower than "2GB or 4GB" (because Emacs additionally needs space to keep
other data, and/or to move part of the buffer around).


        Stefan

Re: gnus, Maximum buffer size exceeded

[Jason Hunter]

2015-07-30 8:52 GMT+02:00 Stefan Monnier <monnier@iro.umontreal.ca>:
>>> So, there's no way I could increase this limit on my 32-bit system?
>> Emacs 25 will support a configure time option to extend this to 2GB
>> (or maybe even 4GB, depending on the OS).  Otherwise, no, there's no way.
>
> As Glenn points out, this option was actually added to
> Emacs-24 already.  And in practice, the real practical limit can end up
> lower than "2GB or 4GB" (because Emacs additionally needs space to keep
> other data, and/or to move part of the buffer around).
>

Thanks for following up on this, cause I really need help;)

Ok, so, I'm using emacs on ubuntu-14.10 and this compile time flag has
not been added then?

My file is only like 400Mio.

Re: gnus, Maximum buffer size exceeded

[Glenn Morris]

Jason Hunter wrote:

> Ok, so, I'm using emacs on ubuntu-14.10 and this compile time flag has
> not been added then?

I doubt there is a pre-built binary with this option on, no.
You could build your own with:


sudo apt-get build-dep emacs24
wget http://ftpmirror.gnu.org/emacs/emacs-24.5.tar.xz
tar -avxf emacs-24.5.tar.xz
cd emacs-24.5
./configure --with-wide-int
make
sudo make install

Re: gnus, Maximum buffer size exceeded

[Stefan Monnier]

> Ok, so, I'm using emacs on ubuntu-14.10 and this compile time flag has
> not been added then?

Probably not.  I wouldn't encourage people to use it: it increases the
memory usage of Emacs (and hence slows it down by eating up your caches
more quickly), but only increases the maximum buffer size by a fairly
small amount.

If you bump into the 512MB limit, you're better off either using vlf.el
or using a 64bit build of Emacs (e.g. the one that comes with Ubuntu in
the amd64 version of Ubuntu).


        Stefan

Re: gnus, Maximum buffer size exceeded

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 31 Jul 2015 03:46:33 -0400
> 
> I wouldn't encourage people to use it: it increases the memory usage
> of Emacs (and hence slows it down by eating up your caches more
> quickly), but only increases the maximum buffer size by a fairly
> small amount.

2GB vs 512MB doesn't sound like "fairly small amount" to me, more like
a factor of 4 and 1.5GB of absolute increment, not a small deal.

I agree that a 64-bit build is better (and spent some non-trivial
effort myself making that work at the time), but I don't agree with
discouraging users of 32-bit systems from using the wide-int build,
especially if switching to a different system is not an easy
alternative.

Re: gnus, Maximum buffer size exceeded

[Stefan Monnier]

>> I wouldn't encourage people to use it: it increases the memory usage
>> of Emacs (and hence slows it down by eating up your caches more
>> quickly), but only increases the maximum buffer size by a fairly
>> small amount.
> 2GB vs 512MB doesn't sound like "fairly small amount" to me, more like
> a factor of 4 and 1.5GB of absolute increment, not a small deal.

If you don't have a separate session just for this one file, or if you
want to edit the file, or if the file will be font-locked, or in several
other circumstances, the limit will be significantly smaller than 2GB.

And since manually-created 500MB files are extremely rare, your 600MB
might very well grow to 2.3GB tomorrow, so yes I consider the difference
between 512MB and 2GB to be pretty small in this context: it's unlikely
that --with-wide-int will satisfy all your needs if a standard build
doesn't already satisfy them.  I'm not saying it's never useful, just
that those cases where it's useful are rare.


        Stefan

Re: gnus, Maximum buffer size exceeded

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 31 Jul 2015 17:25:40 -0400
> 
> >> I wouldn't encourage people to use it: it increases the memory usage
> >> of Emacs (and hence slows it down by eating up your caches more
> >> quickly), but only increases the maximum buffer size by a fairly
> >> small amount.
> > 2GB vs 512MB doesn't sound like "fairly small amount" to me, more like
> > a factor of 4 and 1.5GB of absolute increment, not a small deal.
> 
> If you don't have a separate session just for this one file, or if you
> want to edit the file, or if the file will be font-locked, or in several
> other circumstances, the limit will be significantly smaller than 2GB.

Please explain how each one of those makes the limit significantly
smaller, because I don't see it.  The gap is very small compared to
2GB, so editing the file incurs a small penalty.  The other 2 issues I
don't see how they are involved at all.  Maybe we have bugs that
should be fixed, as we had in the 64-bit builds not so long ago.

> And since manually-created 500MB files are extremely rare, your 600MB
> might very well grow to 2.3GB tomorrow

It may well grow also beyond what the total VM on a typical 64-bit
machine is, so this problem is always present.  But as long as it
didn't grow, the user still has the advantage of being able to visit
files she couldn't before.  IOW, the risk of exceeding the limit
becomes lower as the limit goes up.

> so yes I consider the difference between 512MB and 2GB to be pretty
> small in this context: it's unlikely that --with-wide-int will
> satisfy all your needs if a standard build doesn't already satisfy
> them.

It doesn't need to satisfy all of them, just some.  You have limits on
a 64-bit machine as well, and they are not so much farther.  I don't
see how the situation there is qualitatively different.

> I'm not saying it's never useful, just that those cases where it's
> useful are rare.

Not for me, they aren't.

Re: gnus, Maximum buffer size exceeded

[Eric Brown]

Jason Hunter <jhunterwu@gmail.com> writes:

>
> I find this so difficult to believe. I'm in deep trouble here. I 
> can't
> start my gnus.
>
> Is there any way to cut this file in three or four and process 
> it one by one?
>
> Can I do it manually and then just restart gnus several times 
> with a
> smaller file?
>
> I mean, this can't bring down the whole nuclear powerplant?

Being caught in this situation should be sufficient motivation to 
switch
to IMAP, Maildir, and/or a 64-bit operating system.

Re: gnus, Maximum buffer size exceeded

[Jason Hunter]

On Aug 3, 2015 3:24 AM, "Eric Brown" <brown@fastmail.com> wrote:
>
>
> Jason Hunter <jhunterwu@gmail.com> writes:
>
>>
>> I find this so difficult to believe. I'm in deep trouble here. I can't
>> start my gnus.
>>
>> Is there any way to cut this file in three or four and process it one by
one?
>>
>> Can I do it manually and then just restart gnus several times with a
>> smaller file?
>>
>> I mean, this can't bring down the whole nuclear powerplant?
>
>
> Being caught in this situation should be sufficient motivation to switch
> to IMAP, Maildir, and/or a 64-bit operating system.

Actually, that is what I am doing.

I'm moving everything to imap on tuffmail which seems like the best idea.

Re: gnus, Maximum buffer size exceeded

[Stefan Monnier]

> The gap is very small compared to 2GB, so editing the file incurs
> a small penalty.  The other 2 issues I don't see how they are involved
> at all.  Maybe we have bugs that should be fixed, as we had in the
> 64-bit builds not so long ago.

I don't think any of the issues can be considered bugs.  They're just
the kinds of problems you can start seeing when you get close to the
limit.

>> so yes I consider the difference between 512MB and 2GB to be pretty
>> small in this context: it's unlikely that --with-wide-int will
>> satisfy all your needs if a standard build doesn't already satisfy
>> them.
> It doesn't need to satisfy all of them, just some.  You have limits on
> a 64-bit machine as well, and they are not so much farther.  I don't
> see how the situation there is qualitatively different.

Qualitatively maybe not, but quantitatively the difference is very
significant in my experience.


        Stefan