23.0.60; File corruption when Gnus reads .overview

23.0.60; File corruption when Gnus reads .overview

[Stephen Berman]

[-- Attachment #1: Type: text/plain, Size: 2705 bytes --]

Some change between 2008-02-21 and 2008-03-03 has the effect that, when
Gnus reads an .overview file containing certain non-ascii characters,
that file immediately gets corrupted.  With my current Emacs build
(which is really from 2008-03-03; the date of 2008-03-06 given below is
because I ran `make TAGS' then) I can visit the file outside of Gnus
without problem, and in my build from 2008-02-21 reading it with Gnus
also does not cause corruption.  The file in question is my Gnus
.overview file of the gmane.emacs.help newsgroup.  I have attached an
exerpt of this file (the whole file is 14 MB large) containing the first
1000 entries, which is enough to induce the corruption.  (When Gnus
reads the whole file, the serious corruption begins at entry 586, but
when I substituted the attached excerpt, I found the serious corruption
starting at entry 292.  By "serious" I mean a huge sequence of
non-printing characters; some earlier entries show brief corrupt
sequences.)  To reproduce, I copy the attached file as .overview to the
following location (the News directory is by default in ~, but I keep it
in ~/.emacs.d): News/agent/nntp/news.gmane.org/gmane/emacs/help.  Then I
start Gnus with gnus-unplugged (I use the Gnus Agent), go online with `J
j', and open the gmane.emacs.help group.  At this point the corruption
is already there, even if I quit the group and Gnus without saving.  I
don't know enough about Gnus to know whether these instructions plus the
attached file are enough for others to reproduce the corruption.  If
more information or additional Gnus files are needed, I will gladly
provide them.

Steve Berman

In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.12.0)
 of 2008-03-06 on escher
Windowing system distributor `The X.Org Foundation', version 11.0.70200000
configured using `configure  '--enable-font-backend''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=local
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: Change Log

Minor modes in effect:
  isearch-buffers-minor-mode: t
  tabbar-mwheel-mode: t
  tabbar-mode: t
  recentf-mode: t
  display-time-mode: t
  show-paren-mode: t
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  column-number-mode: t
  line-number-mode: t


[-- Attachment #2: Gnus .overview --]
[-- Type: application/octet-stream, Size: 233545 bytes --]

Re: 23.0.60; File corruption when Gnus reads .overview

[Stephen Berman]

On Fri, 07 Mar 2008 11:54:48 +0100 Stephen Berman <Stephen.Berman@gmx.net> wrote:

> Some change between 2008-02-21 and 2008-03-03 has the effect that, when
> Gnus reads an .overview file containing certain non-ascii characters,
> that file immediately gets corrupted.  With my current Emacs build
> (which is really from 2008-03-03; the date of 2008-03-06 given below is
> because I ran `make TAGS' then) I can visit the file outside of Gnus
> without problem, and in my build from 2008-02-21 reading it with Gnus
> also does not cause corruption.  The file in question is my Gnus
> .overview file of the gmane.emacs.help newsgroup.  I have attached an
> exerpt of this file (the whole file is 14 MB large) containing the first
> 1000 entries, which is enough to induce the corruption.  (When Gnus
> reads the whole file, the serious corruption begins at entry 586, but
> when I substituted the attached excerpt, I found the serious corruption
> starting at entry 292.  By "serious" I mean a huge sequence of
> non-printing characters; some earlier entries show brief corrupt
> sequences.)  To reproduce, I copy the attached file as .overview to the
> following location (the News directory is by default in ~, but I keep it
> in ~/.emacs.d): News/agent/nntp/news.gmane.org/gmane/emacs/help.  Then I
> start Gnus with gnus-unplugged (I use the Gnus Agent), go online with `J
> j', and open the gmane.emacs.help group.  At this point the corruption
> is already there, even if I quit the group and Gnus without saving.  I
> don't know enough about Gnus to know whether these instructions plus the
> attached file are enough for others to reproduce the corruption.  If
> more information or additional Gnus files are needed, I will gladly
> provide them.
>
> Steve Berman
>
> In GNU Emacs 23.0.60.2 (i686-pc-linux-gnu, GTK+ Version 2.12.0)
>  of 2008-03-06 on escher

With my current build (GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+
Version 2.12.0) of 2008-03-13 on escher), the file corruption has
disappeared, but Gnus still has a problem with the .overview file: when
I open the gmane.emacs.help group, all my marks are gone except the one
corresponding to the entry in the .overview file that occurs earlier
than the first entry containing the problematic non-ascii characters.
In other words, it appears that Gnus is not processing the .overview
file after it reads the problematic non-ascii characters.  I would be
grateful if a Gnus hacker could look into this, as I cannot usefully
read gmane.emacs.help with Gnus from the current trunk.  (The problem
with the marks was also present in the build for which I posted my OP,
but I refrained from mentioning it because I thought the file corruption
was the real problem.  But now the corruption is gone, but the marks
problem remains.)

Steve Berman

Re: 23.0.60; File corruption when Gnus reads .overview

[Reiner Steib]

On Fri, Mar 14 2008, Stephen Berman wrote:

> With my current build (GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+
> Version 2.12.0) of 2008-03-13 on escher), the file corruption has
> disappeared, but Gnus still has a problem with the .overview file: when
> I open the gmane.emacs.help group, all my marks are gone except the one
> corresponding to the entry in the .overview file that occurs earlier
> than the first entry containing the problematic non-ascii characters.
> In other words, it appears that Gnus is not processing the .overview
> file after it reads the problematic non-ascii characters.  I would be
> grateful if a Gnus hacker could look into this, as I cannot usefully
> read gmane.emacs.help with Gnus from the current trunk.

It's the agent cache overview, right?  I don't use the agent, so I'm
not familiar with it, but IIRC it's no problem to delete the corrupt
overview file.  Does this help?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: 23.0.60; File corruption when Gnus reads .overview

[Leo]

On 2008-03-27 20:57 +0000, Reiner Steib wrote:
> It's the agent cache overview, right?  I don't use the agent, so I'm
> not familiar with it, but IIRC it's no problem to delete the corrupt
> overview file.  Does this help?

How to set up Gnus not to use agent? I had some .overview files that was
bigger than 10M and they slowed down Gnus severely.

-- 
.:  Leo  :.  [ sdl.web AT gmail.com ]  .:  [ GPG Key: 9283AA3F ]  :.

          Use the best OS -- http://www.fedoraproject.org/

Re: 23.0.60; File corruption when Gnus reads .overview

[Tassilo Horn]

Leo <sdl.web@gmail.com> writes:

Hi Leo,

> On 2008-03-27 20:57 +0000, Reiner Steib wrote:
>> It's the agent cache overview, right?  I don't use the agent, so I'm
>> not familiar with it, but IIRC it's no problem to delete the corrupt
>> overview file.  Does this help?
>
> How to set up Gnus not to use agent?

Hit ^ in the *Group* buffed to get to the *Server* buffer.  Each
agentized server has a (agent) indicator in its line.  You can
enable/disable the agent for a server with `J a'/`J r'.

> I had some .overview files that was bigger than 10M and they slowed
> down Gnus severely.

Not using the agent for non-local servers will probably slow down Gnus
much more, because then it'll download an article from the server each
time it's requested.  With the agent it's downloaded only once, and then
it's cached.

Bye,
Tassilo

Re: 23.0.60; File corruption when Gnus reads .overview

[Reiner Steib]

On Fri, Mar 28 2008, Tassilo Horn wrote:

> Leo <sdl.web@gmail.com> writes:
>
> Hi Leo,
>
>> On 2008-03-27 20:57 +0000, Reiner Steib wrote:
>>> It's the agent cache overview, right?  I don't use the agent, so I'm
>>> not familiar with it, but IIRC it's no problem to delete the corrupt
>>> overview file.  Does this help?
>>
>> How to set up Gnus not to use agent?

(Why) Is it difficult to find this in the manual?

,----[ (info "(gnus)Agent Variables") ]
| `gnus-agent'
|      Is the agent enabled?  The default is `t'.  When first enabled,
|      the agent will use `gnus-agent-auto-agentize-methods' to
|      automatically mark some back ends as agentized.  You may change
|      which back ends are agentized using the agent commands in the
|      server buffer.
`----

> Hit ^ in the *Group* buffed to get to the *Server* buffer.  Each
> agentized server has a (agent) indicator in its line.  You can
> enable/disable the agent for a server with `J a'/`J r'.

My setup disables the agent globally because I run a local NNTP server
(leafnode2) and don't use IMAP much, so I have no need for it.

> Not using the agent for non-local servers will probably slow down Gnus
> much more, because then it'll download an article from the server each
> time it's requested.  With the agent it's downloaded only once, and then
> it's cached.

Someday I should try how it affects IMAP performance for me.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: 23.0.60; File corruption when Gnus reads .overview

[Tassilo Horn]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

>> Not using the agent for non-local servers will probably slow down
>> Gnus much more, because then it'll download an article from the
>> server each time it's requested.  With the agent it's downloaded only
>> once, and then it's cached.
>
> Someday I should try how it affects IMAP performance for me.

I used the agent for a remote IMAP server once, but that wasn't a nice
experience.  The agent got confused when I moved, copied or deleted
messages on the server not using Gnus.  It started duplicating messages
and other strange things I don't remember.

Bye,
Tassilo

Re: 23.0.60; File corruption when Gnus reads .overview

[Stephen Berman]

On Thu, 27 Mar 2008 21:57:07 +0100 Reiner Steib <reinersteib+gmane@imap.cc> wrote:

> On Fri, Mar 14 2008, Stephen Berman wrote:
>
>> With my current build (GNU Emacs 23.0.60.1 (i686-pc-linux-gnu, GTK+
>> Version 2.12.0) of 2008-03-13 on escher), the file corruption has
>> disappeared, but Gnus still has a problem with the .overview file: when
>> I open the gmane.emacs.help group, all my marks are gone except the one
>> corresponding to the entry in the .overview file that occurs earlier
>> than the first entry containing the problematic non-ascii characters.
>> In other words, it appears that Gnus is not processing the .overview
>> file after it reads the problematic non-ascii characters.  I would be
>> grateful if a Gnus hacker could look into this, as I cannot usefully
>> read gmane.emacs.help with Gnus from the current trunk.
>
> It's the agent cache overview, right?  I don't use the agent, so I'm
> not familiar with it, but IIRC it's no problem to delete the corrupt
> overview file.  Does this help?

This does not help: after deleting the .overview file from
News/agent/nntp/news.gmane.org/gmane/emacs/help (but retaining the
corresponding .agentview file), all marked (ticked) articles, as well as
all seen but unread articles, are missing from the summary buffer.
However, the good news is that bug has apparently been fixed in the mean
time: I updated from the trunk today and after opening gmane.emacs.help
with Gnus all the marks were intact, and remained so after closing and
reopening.  This is with the previous 14 MB .overview file reinstated.
(I was sent an email on March 19 from the testbed Emacs bugtracker
saying that Stefan Monnier had closed the bug, but it didn't give
details and I cannot tell from the change logs which change contains the
fix (I was away when the email was sent and wasn't able to update until
today).)

Steve Berman