modifying movemail

modifying movemail

[Florian von Savigny]

Rmail mode uses a small C program, movemail (movemail.exe on Win32),
to move mail from a spool file in Unix mbox format to an rmail file.

The problem I've got with it is actually a feature: it uses some kind
of file locking mechanism to prevent the spool file being accessed by
the mail fetching program at the same time. As strace has told me,
this seems to use some kind of link, but I don't even know if symbolic
or hard. In any case, this seems to be the reason why the spool file
cannot be on a FAT32 partition: movemail will get into an endless
cycle, trying over and over again to create some link, which the
kernel refuses with never-ending patience.

The reason why this is a problem is that I use both Emacs versions on
a dual-boot system (GNU/Linux and Win32), making it desirable to have
the spool files accessible under both OSes, and this is only possible
on a FAT32 partition.

>From what I said above, it seems evident that movemail.exe cannot
possibly use the same kind of file locking system as movemail (it
might be Win32 never permits two processes to modify the same file at
the same time, making any further measures superfluous, but who
knows). Since both appear to be compiled from the same source, I have
had a look at them, hoping to "disable" the file locking mechanism in
some way.

This seems, however, no trivial task, as so much in the emacs sources
understandably has multiple dependencies, and I haven't been able to
sort out the crucial difference between compiling movemail.c for Win32
and for Linux.

Could anybody provide guidance for me here? Or is the very idea flawed
in some way? I am well aware disabling the file locking mechanism
means in fact creating an unsafe program (or crippling it, as some
would probably put it), but this would pose no danger to my system as
it is set up. But it would spare me numerous reboots.

-- 


Florian v. Savigny

If you are going to reply in private, please be patient, as I only
check for mail something like once a week. - Si vous allez répondre
personellement, patientez s.v.p., car je ne lis les courriels
qu'environ une fois par semaine.

Re: modifying movemail

[Pascal Bourguignon]

Florian von Savigny <florian265@uboot.com> writes:

> Rmail mode uses a small C program, movemail (movemail.exe on Win32),
> to move mail from a spool file in Unix mbox format to an rmail file.
> 
> The problem I've got with it is actually a feature: it uses some kind
> of file locking mechanism to prevent the spool file being accessed by
> the mail fetching program at the same time. As strace has told me,
> this seems to use some kind of link, but I don't even know if symbolic
> or hard. In any case, this seems to be the reason why the spool file
> cannot be on a FAT32 partition: movemail will get into an endless
> cycle, trying over and over again to create some link, which the
> kernel refuses with never-ending patience.
> 
> The reason why this is a problem is that I use both Emacs versions on
> a dual-boot system (GNU/Linux and Win32), making it desirable to have
> the spool files accessible under both OSes, and this is only possible
> on a FAT32 partition.
> 
> From what I said above, it seems evident that movemail.exe cannot
> possibly use the same kind of file locking system as movemail (it
> might be Win32 never permits two processes to modify the same file at
> the same time, making any further measures superfluous, but who
> knows). Since both appear to be compiled from the same source, I have
> had a look at them, hoping to "disable" the file locking mechanism in
> some way.
> 
> This seems, however, no trivial task, as so much in the emacs sources
> understandably has multiple dependencies, and I haven't been able to
> sort out the crucial difference between compiling movemail.c for Win32
> and for Linux.
> 
> Could anybody provide guidance for me here? Or is the very idea flawed
> in some way? I am well aware disabling the file locking mechanism
> means in fact creating an unsafe program (or crippling it, as some
> would probably put it), but this would pose no danger to my system as
> it is set up. But it would spare me numerous reboots.

Here's a poor man's movemail. If you don't have bash on MS-Windows,
you can always convert it to bat.

#!/bin/bash
echo $$ > "$1".lock
while [ `cat "$1".lock` -ne "$$" ] ; do
    while [ -x "$1".lock ] ; do
        sleep 5
    done
    echo $$ > "$1".lock
done
mv "$1" "$2"
rm "$1".lock
#### movemail                         -- 2003-10-27 01:10:45 -- pascal   ####

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

There is no worse tyranny than to force a man to pay for what he does not
want merely because you think it would be good for him. -- Robert Heinlein

Re: modifying movemail

[Florian von Savigny]

Pascal Bourguignon <spam@thalassa.informatimago.com> writes:

> #!/bin/bash
> echo $$ > "$1".lock
> while [ `cat "$1".lock` -ne "$$" ] ; do
>     while [ -x "$1".lock ] ; do
>         sleep 5
>     done
>     echo $$ > "$1".lock
> done
.
.
.
> rm "$1".lock
> #### movemail                         -- 2003-10-27 01:10:45 -- pascal   ####


Thanks for this quick and simple poor man's file-locking mechanism,
Pascal, which I find quite charming (and couldn't have thought of) and
hadn't hoped for at all.

I fear, however, that the following is a bit less than what movemail
is expected to do:

> mv "$1" "$2"

as it does not overwrite the target Rmail file, but append to it while
emptying the spool file. So I suppose s.th like

cat "$1" >> "$2"
echo "" > "$1"

should do it. But movemail also converts Unix mbox format to Babyl
format on the fly (or have they switched to mbox as internal Rmail
format in the meantime? I read they were planning it.). How to achieve
that?

[BTW, I seem to have been less than clear about one thing: the movemail
I need to modify is the one to run under Linux, not the one that runs
under Windows. That has to deal with FAT32 anyway, thus should be able
to do the trick (though I haven't yet tested it--brrr...). But this
does not interfere with the usefulness of your solution.]


So thanks a lot - but is anybody able to help with the conversion (or
supply a pointer to the right information)?


-- 


Florian v. Savigny

If you are going to reply in private, please be patient, as I only
check for mail something like once a week. - Si vous allez répondre
personellement, patientez s.v.p., car je ne lis les courriels
qu'environ une fois par semaine.

Re: modifying movemail

[Pascal Bourguignon]

Florian von Savigny <florian265@uboot.com> writes:
> Thanks for this quick and simple poor man's file-locking mechanism,
> Pascal, which I find quite charming (and couldn't have thought of) and
> hadn't hoped for at all.
> 
> I fear, however, that the following is a bit less than what movemail
> is expected to do:
> 
> > mv "$1" "$2"
> 
> as it does not overwrite the target Rmail file, but append to it while
> emptying the spool file. So I suppose s.th like
> 
> cat "$1" >> "$2"
> echo "" > "$1"
> 
> should do it. But movemail also converts Unix mbox format to Babyl
> format on the fly (or have they switched to mbox as internal Rmail
> format in the meantime? I read they were planning it.). How to achieve
> that?
> 
> [BTW, I seem to have been less than clear about one thing: the movemail
> I need to modify is the one to run under Linux, not the one that runs
> under Windows. That has to deal with FAT32 anyway, thus should be able
> to do the trick (though I haven't yet tested it--brrr...). But this
> does not interfere with the usefulness of your solution.]
> 
> 
> So thanks a lot - but is anybody able to help with the conversion (or
> supply a pointer to the right information)?


In emacs:  C-h i m emacs RET m rmail RET m movemail RET



-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

There is no worse tyranny than to force a man to pay for what he does not
want merely because you think it would be good for him. -- Robert Heinlein

Re: modifying movemail

[Florian von Savigny]

We both seem to be online right now, the way you were faster than my
immediate addendum...

Pascal Bourguignon <spam@thalassa.informatimago.com> writes:

> > So thanks a lot - but is anybody able to help with the conversion (or
> > supply a pointer to the right information)?
> 
> 
> In emacs:  C-h i m emacs RET m rmail RET m movemail RET

Sorry, I thought I had already read this information and that it
contained nothing helpful. That wasn't quite right:

--snip--
   When getting new mail, Rmail first copies the new mail from the inbox
file to the Rmail file; then it saves the Rmail file; then it truncates
the inbox file.  This way, a system crash may cause duplication of mail
between the inbox and the Rmail file, but cannot lose mail.
--snap--

Thus, normally, movemail isn't even used. That had escaped me.

--snip--
   In some cases, Rmail copies the new mail from the inbox file
indirectly.  First it runs the `movemail' program to move the mail from
the inbox to an intermediate file called `~/.newmail-INBOXNAME'.  Then
Rmail merges the new mail from that file, saves the Rmail file, and
only then deletes the intermediate file.
--snap--

Sounds like this intermediate file is the second argument to movemail.

What makes me wonder now is: is the fact alone that emacs under Linux
hangs when trying to rmail-get-new-mail from a spool file on a FAT32
partition enough proof that it is using movemail? After all, it might
hang when executing its lisp code, and modifying movemail would not
change a bit about that...


[I'll spare the theoretical question: under what circumstances and why
Emacs (does not) use(s) movemail.]


-- 


Florian v. Savigny

If you are going to reply in private, please be patient, as I only
check for mail something like once a week. - Si vous allez répondre
personellement, patientez s.v.p., car je ne lis les courriels
qu'environ une fois par semaine.

Re: modifying movemail

[Pascal Bourguignon]

Florian von Savigny <florian265@uboot.com> writes:
> Sounds like this intermediate file is the second argument to movemail.
Indeed.
 
> What makes me wonder now is: is the fact alone that emacs under Linux
> hangs when trying to rmail-get-new-mail from a spool file on a FAT32
> partition enough proof that it is using movemail? After all, it might
> hang when executing its lisp code, and modifying movemail would not
> change a bit about that...

It's possible.  It would be trying to lock it itself.

> [I'll spare the theoretical question: under what circumstances and why
> Emacs (does not) use(s) movemail.]

M-x apropos RET movemail RET 

may give some clues...
I guess that if all the variables are nil, it won't be able to use an
external program.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

There is no worse tyranny than to force a man to pay for what he does not
want merely because you think it would be good for him. -- Robert Heinlein

Re: modifying movemail

[Florian von Savigny]

Oops!

I <florian265@uboot.com> just wrote:


> I fear, however, that the following is a bit less than what movemail
> is expected to do:
> 
> > mv "$1" "$2"
> 
> as it does not overwrite the target Rmail file, but append to it while
> emptying the spool file. So I suppose s.th like
> 
> cat "$1" >> "$2"
> echo "" > "$1"
> 
> should do it. 

Or could this be a misconception? I just tried to read the code of
rmail-get-new-mail and rmail-insert-inbox-text, which is so
astonishingly complex that I lost count of what is happening
where. I'm no longer sure what movemail does with its arguments, but a
short test I've run suggests it does something roughly equivalent to
the following:

if [ -x "$2" ]
        echo "File $2 exists"
else
        do
        cat "$1" > "$2"
        echo "" > "$1"
done

So it seems the second argument might not be the rmail file but some
temporary file rmail mode creates. Correct?

> But movemail also converts Unix mbox format to Babyl format on the
> fly

This seems in fact to be wrong. I've just run a little test where
movemail left the contents alone (apart from moving them, that is),
and the conversion seemed to be done by rmail mode later.

So what to do? Is the little script Pascal has supplied in fact a
functional replacement for movemail, which only has to be put in its
place to work (or do I only have to apply the modification I have
explained)?

I am wondering why this is all so complex.

-- 


Florian v. Savigny

If you are going to reply in private, please be patient, as I only
check for mail something like once a week. - Si vous allez répondre
personellement, patientez s.v.p., car je ne lis les courriels
qu'environ une fois par semaine.

Re: modifying movemail

[Pascal Bourguignon]

Florian von Savigny <florian265@uboot.com> writes:
> So what to do? Is the little script Pascal has supplied in fact a
> functional replacement for movemail, which only has to be put in its
> place to work (or do I only have to apply the modification I have
> explained)?
> 
> I am wondering why this is all so complex.

movemail can be complex because the locking mechanisms are not very
standardized, there are several of them, and they don't work on all
the file systems. See for example the problem you have with FAT!  My
script won't work 100% of the time.  It may work correctly 80% or
never or anything betweek 0% and 100%.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

There is no worse tyranny than to force a man to pay for what he does not
want merely because you think it would be good for him. -- Robert Heinlein

Re: modifying movemail

[Florian von Savigny]

Pascal Bourguignon <spam@thalassa.informatimago.com> writes:

> My script won't work 100% of the time.  It may work correctly 80% or
> never or anything betweek 0% and 100%.

I'll give it a try, hoping it works with my (simple)
setup. Regrettably, the news server seems to have discarded the
beginnings of this thread, and I have failed to save away your
script. Trying to re-write it from memory, I arrived at:

#!/bin/sh

echo $$ > "$1".lock

while [ -ne `cat "$1".lock` $$ ]
  do
  while [ -x "$1".lock ]
    do
    sleep 5
    done
  done

mv "$1" "$2"
rm "$1".lock


Was that correct? I am not sure, because that way I don't see how the
condition in the first "while" should ever become false (perhaps when
the echo in the first line fails?).



-- 


Florian v. Savigny

If you are going to reply in private, please be patient, as I only
check for mail something like once a week. - Si vous allez répondre
personellement, patientez s.v.p., car je ne lis les courriels
qu'environ une fois par semaine.

Re: poor man's movemail

[Florian von Savigny]

Hmm ... second thoughts: how to handle any flags emacs passes to
movemail? I understand there is a -p flag and an -r flag. (I mean in a
poor man's movemail, it's OK to ignore any flags, but won't they get
in the way of $1 and $2?)

However, I've just tested a poor man's movemail without even a poor
man's locking mechanism, and it worked fine. I suppose I just don't
pass any flags to it on my setup.

-- 


Florian v. Savigny

If you are going to reply in private, please be patient, as I only
check for mail something like once a week. - Si vous allez répondre
personellement, patientez s.v.p., car je ne lis les courriels
qu'environ une fois par semaine.

Re: modifying movemail

[Thien-Thi Nguyen]

Florian von Savigny <florian265@uboot.com> writes:

> echo "" > "$1"

this may result in a file of one character, a newline, which
is probably not what you want.  you can try, instead:

  > "$1"

that is, no echo command, only redirection.  if that fails,
you can try:

  cat /dev/null > "$1"

if that fails, it means /dev/null is broken (weird!).

thi

Re: modifying movemail

[Kai Grossjohann]

Florian von Savigny <florian265@uboot.com> writes:

> But movemail also converts Unix mbox format to Babyl format on the
> fly (or have they switched to mbox as internal Rmail format in the
> meantime? I read they were planning it.).

IIUC, movemail does not convert to Babyl format.  (It does write
Babyl format when reading from POP, though.)  Instead, Rmail knows to
handle Babyl files with some mbox-style messages appended at the end.

Kai