RMAIL file locking problem?

RMAIL file locking problem?

[Evil Boris]

I use rmail to read mail.  I noticed that starting sometime after the
summer, rmail would lock the mail file (i.e., ~/RMAIL) even if it did
not _seem_ modified (I have no way of knowing if it actually is
modified).

More precisely, here's the behavior I observe: I type "s" in rmail.
It saves the file.  I do it again.  It says "no changes need to be
saved".  I check that ~/RMAIL is not locked.  Some time later, __with
no actions on my part__, I notice that ~/RMAIL is locked (i.e., the
~/.#RMAIL symbolic link appears).  If I go back to Emacs and type "s"
in rmail, it says "no changes need to be saved"!  I can __force__ a
save by typing "t" and "t".  But that's silly, of course...

Versions: current CVS, updated reasonably frequently.  Sun/Solaris
2.7.  Noticed this sometime after last summer.

Why it bothers me: I often have several copies of Emacs running
simultaneously, one on the X display at work, one on ssh/terminal from
home or in screen/terminal.  I switch between the two and would like
to be able to read mail in both (not concurrently, of course).  Rmail
locking the files makes me have to jump through hoops to get one copy
to unlock the file so I can use it from the other.

Any and all ideas would be appreciated.  I tried this with "emacs -Q"
and seemed to do the same, btw.

    --Boris

Re: RMAIL file locking problem?

[Richard Stallman]

I don't know how to debug this remotely, but if you run
under GDB, you can put a breakpoint at lock_file and see
when and where it locks the file.  Using the xbacktrace
GDB command at that point would give us a lot of info
about the reason, probably enough to figure out what's
happening.

Re: RMAIL file locking problem?

[Evil Boris]

Richard Stallman <rms@gnu.org> writes:

> I don't know how to debug this remotely, but if you run
> under GDB, you can put a breakpoint at lock_file and see
> when and where it locks the file.  Using the xbacktrace
> GDB command at that point would give us a lot of info
> about the reason, probably enough to figure out what's
> happening.

I played with it a bit.  Emacs locks many files, so this gets quite
tedious.  As far as I can tell, this always happens here:
==========
(gdb) xbacktrace
"insert-file-contents"
"rmail-insert-inbox-text"
"rmail-get-new-mail"
"call-interactively"
(gdb) cont
==========

As far as I can tell, it is the following code:

		  (if file-name
		      (rmail-insert-inbox-text files nil)
		    (setq delete-files (rmail-insert-inbox-text files t)))

On my machine the locking trouble seems to happen when there is no new
mail.  My understanding is that RMAIL reads in the contents of the
(empty!) mailbox (by the way, would it make sense to check for 0
length and just skip the rest of jumping through hoops in this
case---no locking, no inserting of zero bytes, no testing for having
inserted zero bytes?)  

Any ideas of what else I can check?  It seems like one way the
lockfile appears (but not consistently!?!?!?) when I type "g" to get
new mail, while there isn't any in the mail box.

Any and all ideas welcome.
    --Boris

Re: RMAIL file locking problem?

[Richard Stallman]

    As far as I can tell, it is the following code:

		      (if file-name
			  (rmail-insert-inbox-text files nil)
			(setq delete-files (rmail-insert-inbox-text files t)))

Ok, it seems natural that this would lock the current buffer.
It is trying to insert something.

    On my machine the locking trouble seems to happen when there is no new
    mail.  My understanding is that RMAIL reads in the contents of the
    (empty!) mailbox (by the way, would it make sense to check for 0
    length and just skip the rest of jumping through hoops in this
    case---no locking, no inserting of zero bytes, no testing for having
    inserted zero bytes?)  

It seems strange that insert-file-contents would leave the buffer
locked if it does not really change the buffer.  That could be a bug
in insert-file-contents: that it handles the zero-size case wrong.

Re: RMAIL file locking problem?

[Evil Boris]

Richard Stallman <rms@gnu.org> writes:

>     As far as I can tell, it is the following code:
>
> 		      (if file-name
> 			  (rmail-insert-inbox-text files nil)
> 			(setq delete-files (rmail-insert-inbox-text files t)))
>
> Ok, it seems natural that this would lock the current buffer.
> It is trying to insert something.
>
>     On my machine the locking trouble seems to happen when there is no new
>     mail.  My understanding is that RMAIL reads in the contents of the
>     (empty!) mailbox (by the way, would it make sense to check for 0
>     length and just skip the rest of jumping through hoops in this
>     case---no locking, no inserting of zero bytes, no testing for having
>     inserted zero bytes?)  
>
> It seems strange that insert-file-contents would leave the buffer
> locked if it does not really change the buffer.  That could be a bug
> in insert-file-contents: that it handles the zero-size case wrong.

I may have found the problem.  I leave the rest of my blabberings (the
note I was editing before I discovered this) at the end, just in case
it gives any further clues...

As Richard has suggested, the problem may be in insert-file-contents.
I was very reluctant to believe it, but now I am almost certain it is
true, as I have repeatedly observed insert-file-contents called from
rmail-insert-inbox-text, returning 0 as the number of bytes read, with
the effect of leaving the underlying buffer (visiting ~/RMAIL in this
case) locked.

Can anyone confirm this, or give me some hints of how to indentify the
circumstances when it happens?

Thanks in advance,
       --Boris
=========================
I am not very good at C-level debugging Emacs (nor Lisp-level, for
that matter).  I have been able somewhat reliably (but no very
predictably, as in "I cannot tell how we get there") get Emacs into a
state where there is no new mail, ~/RMAIL is unlocked, I type "g", and
after checking the mail ~/RMAIL is locked.  No message is "unseen".
(I can force a removal of the lock by making ~/RMAIL explicitly
modified by "t" "t" and the "s"aving.  After that, the lock
disappears.  It reappears in the next "g".)

Any hints at what to look for?  I tried stepping through most of
rmail-get-new-mail, but it does an excruciating number of
file-name-rewriting and other odd operations.  

Or could one check the C code for "insert-file-contents"?  Does it
mark the buffer modified if 0 bytes were read.  (BTW I looked at the
C code there and it seems to say that it is not safe to ask the file
descriptor for file size [eg if /proc file system] so my earlier
suggestion of checking the file size of the mailbox before trying to
insert its contents might not make as much sense I thought...)

In short, I seem to be able to get Emacs somewhat predictable into a
strange state.  What data do I need to collect to try and figure out
what's going on?

       --Boris

PS. BTW, I did notice that on non-empty mboxes for some reason (before
saving modified ~/RMAIL after the mailbox has been read in), the
mailbox is locked twice.  I first thought it very odd.  Then I realize
the first lock was in rmail-insert-inbox-text (when new stuff is
inserted) and the second was in save-buffer, after Babyl-ifying the
new data.  The first seems to happen for empty mailbox too.  Or does it?

Re: RMAIL file locking problem?

[Richard Stallman]

    Or could one check the C code for "insert-file-contents"?  Does it
    mark the buffer modified if 0 bytes were read.

The code appears specifically to avoid marking the buffer modified
if the file has zero size.  I just did a test, inserting an empty
file into a buffer this way.  It did not mark the buffer modified
and did not lock the file.

Something we don't understand must be happening, and only you
are in a position to debug it.

Re: RMAIL file locking problem?

[Evil Boris]

Richard Stallman <rms@gnu.org> writes:

>     Or could one check the C code for "insert-file-contents"?  Does it
>     mark the buffer modified if 0 bytes were read.
>
> The code appears specifically to avoid marking the buffer modified
> if the file has zero size.  I just did a test, inserting an empty
> file into a buffer this way.  It did not mark the buffer modified
> and did not lock the file.
>
> Something we don't understand must be happening, and only you
> are in a position to debug it.

I have spent some time tracing the code of insert-file-contents.  I do
not understand what it is trying to do, to be honest, but this is what
it **appears** to be doing:  If the file it is trying to read has ZERO
bytes according to what stat() returns, it does the following:

----lines 176-181 of fileio.c
          /* The file size returned from stat may be zero, but data
             may be readable nonetheless, for example when this is a
             file in the /proc filesystem.  */
          if (st.st_size == 0)
            XSETINT (end, READ_BUF_SIZE);
---------

This has the effect of setting the amount to be read to non-zero.
Around line 707 it does

      total = XINT (end) - XINT (beg);

setting total to something very much other than zero (some buffer
size), and in the next if statement (line 718)

  if (NILP (visit) && total > 0)
    prepare_to_modify_buffer (PT, PT, NULL);

which as far as I can tell calls lock_file on the underlying file.
Any other suggestions/ideas?  

--Boris

PS. I can see the comment in fileio.c regarding /proc and can guess
that this sometimes might come in handy, but I do not see why reading
a regular zero-byte file should force a modification and/or locking of
the current buffer.  But if I am right, I do not see how reading a
zero-byte file can EVER work right.  Richard said he tried some casual
tests and it seems to work.  I just tried creating a file qqqq in /tmp,
visiting it, and doing (insert-file-contents "/var/mail/whatever") and
lo and behold, a lock file for /tmp/qqqq showed up!  Just tried it
again with /tmp/empty in place of /var/mail/whatever---same effect.
Does it mean that 0-size insertion is just broken on my machine
(Solaris 5.7, gcc 3.1)?

Re: RMAIL file locking problem?

[Richard Stallman]

I think I have fixed this; does this give good results?


*** fileio.c	09 May 2005 16:51:43 -0400	1.540
--- fileio.c	11 May 2005 03:26:34 -0400	
*** 3728,3733 ****
--- 3732,3739 ----
    int set_coding_system = 0;
    int coding_system_decided = 0;
    int read_quit = 0;
+   int old_Vdeactivate_mark = Vdeactivate_mark;
+   int we_locked_file = 0;
  
    if (current_buffer->base_buffer && ! NILP (visit))
      error ("Cannot do file visiting in an indirect buffer");
***************
*** 4400,4407 ****
      /* For a special file, all we can do is guess.  */
      total = READ_BUF_SIZE;
  
!   if (NILP (visit) && total > 0)
!     prepare_to_modify_buffer (PT, PT, NULL);
  
    move_gap (PT);
    if (GAP_SIZE < total)
--- 4406,4422 ----
      /* For a special file, all we can do is guess.  */
      total = READ_BUF_SIZE;
  
!   if (NILP (visit) && inserted > 0)
!     {
! #ifdef CLASH_DETECTION
!       if (!NILP (current_buffer->file_truename)
! 	  /* Make binding buffer-file-name to nil effective.  */
! 	  && !NILP (current_buffer->filename)
! 	  && SAVE_MODIFF >= MODIFF)
! 	we_locked_file = 1;
! #endif /* CLASH_DETECTION */
!       prepare_to_modify_buffer (GPT, GPT, NULL);
!     }
  
    move_gap (PT);
    if (GAP_SIZE < total)
***************
*** 4491,4496 ****
--- 4506,4521 ----
        }
    }
  
+   /* Now we have read all the file data into the gap.
+      If it was empty, undo marking the buffer modified.  */
+ 
+   if (inserted == 0)
+     {
+       if (we_locked_file)
+ 	unlock_file (current_buffer->file_truename);
+       Vdeactivate_mark = old_Vdeactivate_mark;
+     }
+ 
    /* Make the text read part of the buffer.  */
    GAP_SIZE -= inserted;
    GPT      += inserted;

Re: RMAIL file locking problem?

[Evil Boris]

Richard Stallman <rms@gnu.org> writes:

> I think I have fixed this; does this give good results?
>
> [...]
>   
> +   /* Now we have read all the file data into the gap.
> +      If it was empty, undo marking the buffer modified.  */
> + 
> +   if (inserted == 0)
> +     {
> +       if (we_locked_file)
> + 	unlock_file (current_buffer->file_truename);
> +       Vdeactivate_mark = old_Vdeactivate_mark;
> +     }
> + 
>     /* Make the text read part of the buffer.  */

I will try it shortly.  I have one concern though---does this code
take care of the situation when the buffer was modified even before
insert-file-contents is called?  I do not understand the details of
the code, but I am worried that if you read in 0 bytes into an
already-modified buffer, you might remove a lock that was there
initially for reasons unrelated to the insert-file-contents.  

I wonder if one should use the functionality equivalent to 
file-locked-p to test for this.  Perhaps you are already doing it, as
I do not understand the "#ifdef CLASH_DETECTION" surrounded code.

--Boris

Re: RMAIL file locking problem?

[Richard Stallman]

    I will try it shortly.  I have one concern though---does this code
    take care of the situation when the buffer was modified even before
    insert-file-contents is called?

Yes.

Re: RMAIL file locking problem?

[Evil Boris]

Richard,

Your patch seems to work.  Are there any obvious things to check in
the spirit of regression testing?  Rmail locking seems to be back to
sanity and nothing else obvious seems broken.  I did use if for only a
few minites today, though...

Thanks,
        --Boris

Re: RMAIL file locking problem?

[Richard Stallman]

I installed it, so it will be tested now.

Re: RMAIL file locking problem?

[Evil Boris]

As someone pointed out in a different msg[1], your patch is missing some
#ifdef's around unlock_file, as the function is only defined when
CLASH_DETECTION is defined, so it breaks, say, on win32. 

--Boris

[1] gmane.emacs.devel:37082