Gnus: How to catch up huge newsgroups?

Gnus: How to catch up huge newsgroups?

[Joe Fineman]

I run Gnus v5.10.6 under Emacs 21.3.1 under Windows XP (alas).
Because Verizon has truncated its service, I recently signed on to a
latitudinarian news server (Motzarella).  I find that four of the
newsgroups I want to read come up with backlogs on the order of 30
million postings.  When I try to do catchup (C) on one of them, it
stalls & eventually gives an out-of-memory message.  (The first time,
when I let it try for a longish time, it ended up giving interminable
beeps & hanging both Emacs and Windows; I had to do a hardware
shutdown.)

What does one do in such a situation?  I see nothing about it in info
or Google or the Gnus FAQ.

I have tried twice to post this query on gnu.emacs.gnus (a moderated
group), but it has not appeared, nor have I received any reason for
its rejection.  I have also sent email to the proprietor of
Motzarella, who was courteous & patient in helping me to set up, but I
have received no reply.  If, likewise, this query does not belong on
gnu.emacs.help, I would be grateful for advice on where it would be on
topic.
-- 
---  Joe Fineman    joe_f@verizon.net

||:  If you can keep your head while all about you are losing    :||
||:  theirs, you don't appreciate the gravity of the situation.  :||

Re: Gnus: How to catch up huge newsgroups?

[Sivaram Neelakantan]

Joe Fineman <joe_f@verizon.net> writes:

> I run Gnus v5.10.6 under Emacs 21.3.1 under Windows XP (alas).
> Because Verizon has truncated its service, I recently signed on to a
> latitudinarian news server (Motzarella).  I find that four of the
> newsgroups I want to read come up with backlogs on the order of 30
> million postings.  When I try to do catchup (C) on one of them, it
> stalls & eventually gives an out-of-memory message.  (The first time,
> when I let it try for a longish time, it ended up giving interminable
> beeps & hanging both Emacs and Windows; I had to do a hardware
> shutdown.)
>
> What does one do in such a situation?  I see nothing about it in info
> or Google or the Gnus FAQ.

[...]

I think I had the same problem with the same free news server
account.  I don't remember exactly what I did to solve it. :(
I believe, after subscribing to comp.text.tex and comp.shell, in the
Group Buffer, I entered the group as C-u 50 RET and did a catch up
within it with a 'c'.  then quit  it and configure the Group Parameter
for that the NG.

And I'm not sure whether I set the Group Parameters (G P) to the
following BEFORE I entered the group or not.

((display . 150))

And I'm not even sure whether the above has any effect for the first
time entry.

Damn it.  This whole post of mine doesn't seem useful at all with all
the caveats.

 sivaram
 -- 

Re: Gnus: How to catch up huge newsgroups?

[Ted Zlatanov]

On Mon, 03 Nov 2008 20:10:17 -0500 Joe Fineman <joe_f@verizon.net> wrote: 

JF> I run Gnus v5.10.6 under Emacs 21.3.1 under Windows XP (alas).
JF> Because Verizon has truncated its service, I recently signed on to a
JF> latitudinarian news server (Motzarella).  I find that four of the
JF> newsgroups I want to read come up with backlogs on the order of 30
JF> million postings.  When I try to do catchup (C) on one of them, it
JF> stalls & eventually gives an out-of-memory message.  (The first time,
JF> when I let it try for a longish time, it ended up giving interminable
JF> beeps & hanging both Emacs and Windows; I had to do a hardware
JF> shutdown.)

JF> What does one do in such a situation?  I see nothing about it in info
JF> or Google or the Gnus FAQ.

I've never seen this, but it should be fairly easy to edit the
newsrc.eld file and catch up from there.  Make sure you're not running
Gnus.  For example I have:

("nntp+news.albasani.net:comp.risks" 3 ((1 . 148)) ((seen (118 . 148))) (nntp "news.albasani.net"))

Which says that 118-148 have been seen.

You may run into problems with very large numbers as far as Emacs is
concerned.  There's no easy fix there but you should report it to the
Gnus mailing list nevertheless so you can get some answers and maybe
someone will work on it.

JF> I have tried twice to post this query on gnu.emacs.gnus (a moderated
JF> group), but it has not appeared, nor have I received any reason for
JF> its rejection.

I'm not sure what's up with your news server--that group is not
moderated on mine, but you can try the ding mailing list <ding@gnus.org>
instead.  That's the usual place to talk about Gnus development, and
it's accesible through gmane.org as well if you prefer NNTP.

Ted

Re: Gnus: How to catch up huge newsgroups?

[Joe Fineman]

Thank you, Ted Zlatanov and Sivaram Neelakantan, for trying to help.
I am rather nervous about changing things that I do not understand,
but I have at last steeled myself to try to follow your advice.

Sivaram Neelakantan <nsivaram.net@gmail.com> wrote:

> I think I had the same problem with the same free news server
> account.  I don't remember exactly what I did to solve it. :(
> I believe, after subscribing to comp.text.tex and comp.shell, in the
> Group Buffer, I entered the group as C-u 50 RET and did a catch up
> within it with a 'c'.

I did that, but there was no apparent difference:  Nothing happened
within a reasonable time.

> then quit it and configure the Group Parameter for that the NG.

> And I'm not sure whether I set the Group Parameters (G P)

I presume you mean G p.  G P does nothing.

> to the following BEFORE I entered the group or not.

> ((display . 150))

I tried that, though I am not sure I succeeded -- when I did C-c C-c
to finish, nothing happened, and I had to do C-g to get the cursor
blinking again.  It had no effect; C-u 50 RET did not succeed in
entering the group.

Ted Zlatanov <tzz@lifelogs.com> wrote

> On Mon, 03 Nov 2008 20:10:17 -0500 Joe Fineman <joe_f@verizon.net> wrote: 

> JF> I run Gnus v5.10.6 under Emacs 21.3.1 under Windows XP (alas).
> JF> Because Verizon has truncated its service, I recently signed on to a
> JF> latitudinarian news server (Motzarella).  I find that four of the
> JF> newsgroups I want to read come up with backlogs on the order of 30
> JF> million postings.  When I try to do catchup (C) on one of them, it
> JF> stalls & eventually gives an out-of-memory message.  (The first time,
> JF> when I let it try for a longish time, it ended up giving interminable
> JF> beeps & hanging both Emacs and Windows; I had to do a hardware
> JF> shutdown.)

> JF> What does one do in such a situation?  I see nothing about it in info
> JF> or Google or the Gnus FAQ.

> I've never seen this, but it should be fairly easy to edit the
> newsrc.eld file and catch up from there.  Make sure you're not running
> Gnus.  For example I have:

> ("nntp+news.albasani.net:comp.risks" 3 ((1 . 148)) ((seen (118 . 148))) (nntp "news.albasani.net"))

> Which says that 118-148 have been seen.

I found the apposite section for comp.text.tex and changed it to

("comp.text.tex" 3 ((1 . 230036)) ((seen 229707 . 230036)) (reply
221367 228504 228579))

There was nothing corresponding to the (nntp "...").  When I tried to
use Gnus, I got an error message.

> You may run into problems with very large numbers as far as Emacs is
> concerned.  There's no easy fix there but you should report it to
> the Gnus mailing list nevertheless so you can get some answers and
> maybe someone will work on it.

I think, first, I will try again to get the attention of Motzarella.
Perhaps my first attempt got classified as spam for some reason.

Thank you again for your patience.
-- 
---  Joe Fineman    joe_f@verizon.net

||:  You may ask a scientist "Why?" until you get to "I don't  :||
||:  know" or "For that you need some math".                   :||

Re: Gnus: How to catch up huge newsgroups?

[Reiner Steib]

On Tue, Nov 04 2008, Joe Fineman wrote:

> I run Gnus v5.10.6 under Emacs 21.3.1 under Windows XP (alas).
> Because Verizon has truncated its service, I recently signed on to a
> latitudinarian news server (Motzarella).  I find that four of the
> newsgroups I want to read come up with backlogs on the order of 30
> million postings.  

Unless it is some binary newsgroup, this seem unlikely.  Please verify
the numbers using the following command in a shell and show us the
output.

  telnet news.motzarella.org nntp                          <== You type this!
  Trying 85.214.105.209...
  Connected to news.motzarella.org.
  Escape character is '^]'.
  200 news.[...] INN 2.4.5 ready (posting ok).
  MODE READER                                              <== You type this!
  GROUP name.of.the.group                                  <== You type this!

The maximum number Gnus can handle depends on your Emacs version and
architecture.

What is the output of <f1> v most-positive-fixnum RET?
You may need to load cl before: M-x load-library RET cl RET

> I have tried twice to post this query on gnu.emacs.gnus (a moderated
> group), 

gnu.emacs.gnus isn't moderated.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus: How to catch up huge newsgroups?

[Joe Fineman]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Tue, Nov 04 2008, Joe Fineman wrote:
>
>> I run Gnus v5.10.6 under Emacs 21.3.1 under Windows XP (alas).
>> Because Verizon has truncated its service, I recently signed on to
>> a latitudinarian news server (Motzarella).  I find that four of the
>> newsgroups I want to read come up with backlogs on the order of 30
>> million postings.
>
> Unless it is some binary newsgroup, this seem unlikely.

None of the four is a binary newsgroup.

> Please verify the numbers using the following command in a shell and
> show us the output.

I'll give this a try.  Thank you.

> What is the output of <f1> v most-positive-fixnum RET?
> You may need to load cl before: M-x load-library RET cl RET

Ditto.

>> I have tried twice to post this query on gnu.emacs.gnus (a
>> moderated group),
>
> gnu.emacs.gnus isn't moderated.

I so interpreted the (m) in the Group buffer.  If it is not moderated,
then it is all the more mysterious that I cannot succeed in posting to
it.

Thank you for your attention.
-- 
---  Joe Fineman    joe_f@verizon.net

||:  The idea of appearing modest appeals to my vanity.  :||

Re: Gnus: How to catch up huge newsgroups?

[Reiner Steib]

On Mon, Nov 10 2008, Joe Fineman wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
>> On Tue, Nov 04 2008, Joe Fineman wrote:
>>
>>> I run Gnus v5.10.6 under Emacs 21.3.1 under Windows XP (alas).
>>> Because Verizon has truncated its service, I recently signed on to
>>> a latitudinarian news server (Motzarella).  I find that four of the
>>> newsgroups I want to read come up with backlogs on the order of 30
>>> million postings.
>>
>> Unless it is some binary newsgroup, this seem unlikely.
>
> None of the four is a binary newsgroup.

[ Please reply to the list! This is from the mail you sent to me: ]
> ----------------------------------------------------------------------
> /cygdrive/c/usr/own/f/p/networks/motzarella> telnet news.motzarella.org nntp
> Trying 85.214.105.209...
> Connected to news.motzarella.org.
> Escape character is '^]'.
> 200 news.eternal-september.org InterNetNews NNRP server INN 2.4.5 ready (posting ok).
> MODE READER
> GROUP comp.text.tex
> ----------------------------------------------------------------------

> There was no response in 20 min.  Indeed, the shell (an Emacs shell)
> seems to have died.

Does it work in cmd.exe?  Or can anyone else with access to motzarella
show the result?

On one of the biggest news provides in Germany, I get:

,----
| GROUP comp.text.tex
| 211 3562 328642 332246 comp.text.tex
`----

I doubt that motzarella has an older archive (higher article count)
unless there some bug.

> > The maximum number Gnus can handle depends on your Emacs version and
> > architecture.
> >
> > What is the output of <f1> v most-positive-fixnum RET?
> > You may need to load cl before: M-x load-library RET cl RET

> ----------------------------------------------------------------------
> most-positive-fixnum's value is 134217727
> ----------------------------------------------------------------------

So groups with 100 million postings should work.

>>> I have tried twice to post this query on gnu.emacs.gnus (a
>>> moderated group),
>>
>> gnu.emacs.gnus isn't moderated.
>
> I so interpreted the (m) in the Group buffer.  If it is not moderated,
> then it is all the more mysterious that I cannot succeed in posting to
> it.

All newsservers I have, don't list it as moderated:

,----
| # news.[...].de
| 201 news.[...].de InterNetNews NNRP server INN 2.4.4 (20060818 snapshot) ready (no posting).
| 215 Descriptions in form "group description".
| gnu.emacs.gnus		News reading under GNU Emacs using GNUS (in English).
| de.admin.news.announce	RFDs, CFVs, Zwischen- und Endergebnisse. <moderator@dana.de> (Moderated)
| 205 .
| # news.[...].de
| 200 news.[...].de DNEWS Version  5.6f3,, S0, posting OK 
| 215 list of newsgroup names follow
| de.admin.news.announce	RFDs, CFVs, Zwischen- und Endergebnisse. (Moderated)
| gnu.emacs.gnus	News reading under GNU Emacs using GNUS (in English).
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus: How to catch up huge newsgroups?

[Joe Fineman]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Mon, Nov 10 2008, Joe Fineman wrote:

>> None of the four is a binary newsgroup.
>
> [ Please reply to the list! This is from the mail you sent to me: ]

Sorry!  I meant to reply to the newsgroup, and was puzzled when it
didn't show up.  I must have hit R instead of F.

>> There was no response in 20 min.  Indeed, the shell (an Emacs
>> shell) seems to have died.

Subsequently I got the same result with one of the newsgroups I *can*
read.

I tried it in the Cygwin shell.  The shell survived, but I got a
message "502 Permission denied".

> Does it work in cmd.exe?

I don't know what cmd.exe is.  How do I get to it?

>> ----------------------------------------------------------------------
>> most-positive-fixnum's value is 134217727
>> ----------------------------------------------------------------------
>
> So groups with 100 million postings should work.

Well, the number is (or purports to be) only 30 million.  The current
state of my *Group* buffer is

       0: nnml:mail.misc 
      88: alt.politics.libertarian 
   30221113: alt.usage.english 
       1: gnu.announce 
      25: gnu.emacs.bug 
       6: gnu.emacs.gnus 
       6:*gnu.emacs.help 
       2: ne.transportation 
   29993938: sci.lang 
      16: soc.motss 
   29988010: alt.suicide.methods 
      30: alt.quotations 
   30058577: comp.text.tex 

>>> gnu.emacs.gnus isn't moderated.
>>
>> I so interpreted the (m) in the Group buffer.

Which is now gone!  Perhaps I hallucinated it.

I also tried another email (yesterday noon) to Wolfgang M. Weyand at
Motzarella.

Thank you for your patience.
-- 
---  Joe Fineman    joe_f@verizon.net

||:  The first half of our lives is ruined by our parents, and  :||
||:  the second half by our children.                           :||

Re: Gnus: How to catch up huge newsgroups?

[Reiner Steib]

On Mon, Nov 10 2008, Joe Fineman wrote:

> I tried it in the Cygwin shell.  The shell survived, but I got a
> message "502 Permission denied".

Ah, it needs authentication (The password is shown on the screen, so
be careful!):

$ telnet news.motzarella.org nntp
MODE READER
AUTHINFO USER yourname
AUTHINFO PASS yourpassword
GROUP comp.text.tex

>> Does it work in cmd.exe?
>
> I don't know what cmd.exe is.  How do I get to it?

The command prompt window on windows (aka "DOS prompt")
Start -> Run -> (Enter) cmd.exe

(But you can also use the cygwin shell.)

>       30: alt.quotations 
>    30058577: comp.text.tex 

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Gnus: How to catch up huge newsgroups?

[Joe Fineman]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> On Mon, Nov 10 2008, Joe Fineman wrote:
>
>> I tried it in the Cygwin shell.  The shell survived, but I got a
>> message "502 Permission denied".
>
> Ah, it needs authentication (The password is shown on the screen, so
> be careful!):
>
> $ telnet news.motzarella.org nntp
> MODE READER
> AUTHINFO USER yourname
> AUTHINFO PASS yourpassword
> GROUP comp.text.tex

That did the job.  The result for comp.text.tex was indeed 30288629.
So evidently it's for real.

Oddly, however, the result for alt.politics.libertarian, which I have
been reading without difficulty, is also huge: 30645382.

I repeated the experiment of doing c on one of the problem groups in
the Groups buffer.  After a few minutes, it beeped and gave an Emacs
low-memory warning.  This time, however, it did not crash anything.
-- 
---  Joe Fineman    joe_f@verizon.net

||:  One learns a lot from a standoff, a little from a defeat,  :||
||:  and nothing from a victory.                                :||