`notmuch setup` replaces `~/.notmuch-config` instead of truncating it

`notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Ciprian Dorin, Craciun]

    Hello all!

    First congratulations for the nice software! I hardly wait for a
notmuch native (i.e. libnotmuch) and curses client (like `ner`) to
become more stable, and thus I'll be able to ditch GMail. :) But until
then a small glitch...

    While upgrading from notmuch 0.4 to 0.5, I've re-runned `notmuch
config` as suggested in the release email.

    But in my particular case `~/.notmuch-config` is symlinked to an
applications configuration directory which is versioned. Thus I've
expected than when notmuch updates the config, it opens it for
read-write, but with the truncation flag (which as a consequence would
have modified the symlinked file). But instead it deleted the symlink,
and replaced it with a newly created file (thus breaking my custom
configuration backup system.)

    So my question is: is this behaviour (of deleting the file and
creating a new one) deliberate? If not, could it be fixed (I could
provide a patch) to just update the file in place?

    Thanks,
    Ciprian.

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

On Tue, 16 Nov 2010 15:33:30 +0200, "Ciprian Dorin, Craciun" <ciprian.craciun@gmail.com> wrote:
>     But in my particular case `~/.notmuch-config` is symlinked to an
> applications configuration directory which is versioned. Thus I've
> expected than when notmuch updates the config, it opens it for
> read-write, but with the truncation flag (which as a consequence would
> have modified the symlinked file). But instead it deleted the symlink,
> and replaced it with a newly created file (thus breaking my custom
> configuration backup system.)
> 
>     So my question is: is this behaviour (of deleting the file and
> creating a new one) deliberate? If not, could it be fixed (I could
> provide a patch) to just update the file in place?

Hi, Ciprian.  I had not noticed this, but now that you mention it, I see
that the same thing happened to me.  This behavior is surely not
deliberate, and is definitely undesirable.  A patch would be welcome.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

On 11/16/2010 08:38 AM, Jameson Rollins wrote:
> On Tue, 16 Nov 2010 15:33:30 +0200, "Ciprian Dorin, Craciun" <ciprian.craciun@gmail.com> wrote:
>>     So my question is: is this behaviour (of deleting the file and
>> creating a new one) deliberate? If not, could it be fixed (I could
>> provide a patch) to just update the file in place?
> 
> Hi, Ciprian.  I had not noticed this, but now that you mention it, I see
> that the same thing happened to me.  This behavior is surely not
> deliberate, and is definitely undesirable.  A patch would be welcome.

It often is desirable to do a replacement instead of truncation or
appending -- atomic replacement guarantees that a legitimate copy is
always available.  It eliminates a window of time when the config file
would be in an intermediate state.  Readers of the file always get
either the old state or the new state -- nothing in between.

However, you can address both concerns by detecting whether
~/.notmuch-config is a symlink and readlink()ing until you get to the
underlying file.  Then atomically replace the target instead of the
symlink itself.

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

On Tue, 16 Nov 2010 15:33:30 +0200, "Ciprian Dorin, Craciun" <ciprian.craciun@gmail.com> wrote:
>     First congratulations for the nice software!

Thanks so much! Welcome to notmuch.

>     So my question is: is this behaviour (of deleting the file and
> creating a new one) deliberate? If not, could it be fixed (I could
> provide a patch) to just update the file in place?

Daniel gave the perfect answer later in the thread. It is intentional to
replace the file with a new, complete version, (to avoid loss/corruption
of the file if "notmuch setup" is interrupted). But we should fix this
to replace the target of any symlinks.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Ciprian Dorin, Craciun]

On Tue, Nov 16, 2010 at 21:09, Carl Worth <cworth@cworth.org> wrote:
> On Tue, 16 Nov 2010 15:33:30 +0200, "Ciprian Dorin, Craciun" <ciprian.craciun@gmail.com> wrote:
>>     So my question is: is this behaviour (of deleting the file and
>> creating a new one) deliberate? If not, could it be fixed (I could
>> provide a patch) to just update the file in place?
>
> Daniel gave the perfect answer later in the thread. It is intentional to
> replace the file with a new, complete version, (to avoid loss/corruption
> of the file if "notmuch setup" is interrupted). But we should fix this
> to replace the target of any symlinks.
>
> -Carl


    I understand now the reason for your file replacement choice.
(I'll look over tomorrow to see if I can provide a patch on the line
Daniel has described).

    But -- in general, and totally overlooking the "pseudo" atomic
effect obtained from of POSIX file semantics -- doesn't this practice
mislead some software (like backup systems, etc.) that would rely
maybe on the inode number as part of the identity of the file?
Moreover what if the user has set any ACL's or extended attributes on
the file, wouldn't these be lost? (Wouldn't also SELinux be bothered?)

    So after browsing through the source code, I've found inside
`notmuch-config.c`, inside the function `notmuch_config_save`, the
call which actually overrides the file: `g_file_set_contents
(config->filename, data, length, &error)`. Now searching for the
documentation of this function, I've stumbled upon, from which I cite
(I've never used glib before, so maybe the link is not the best one):
        http://library.gnome.org/devel/glib/unstable/glib-File-Utilities.html
~~~~
On Unix, if filename already exists hard links to filename will break.
Also since the file is recreated, existing permissions, access control
lists, metadata etc. may be lost. If filename is a symbolic link, the
link itself will be replaced, not the linked file.
~~~~

    So in the light of the above quoted "glitches", my question is:
due to the small chance of a power loss happening right when we write
such a small file, doesn't the inconvenience weight more than the
(fairly remote probable) file loss? (I must admit I've lost once the
`/etc/network/interfaces` file after an edit and immediately after a
quick cold reboot, but it was my fault as I've not sync-ed the file
system.)

    Ciprian.

    P.S.: I say "pseudo" atomic because only the rename is atomic,
thus in order to override file `a` for the target file `b` which
exists, we must execute two **non-atomic** operations as a whole, but
each atomic in part, rename operations: make `b` -> `c`, and then
rename `a` -> `b`. So there is actually a small time-frame when I can
be left with two files (`a` and `c`), none of which is my config file
`b`. (This can be solved when opening the config file by checking if
there isn't any leftover `c` or `a` file, in which case I take the `a`
file and complete the rename.)

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1508 bytes --]

On 11/16/2010 03:26 PM, Ciprian Dorin, Craciun wrote:
>     So in the light of the above quoted "glitches", my question is:
> due to the small chance of a power loss happening right when we write
> such a small file, doesn't the inconvenience weight more than the
> (fairly remote probable) file loss?

What inconvenience?  The inconvenience of writing the code correctly?
that's a small one-time cost compared to *any* risk of a user ending up
with a damaged .notmuch-config (not to mention possible concurrent uses
of notmuch seeing an intermediate version of the file)

>     P.S.: I say "pseudo" atomic because only the rename is atomic,
> thus in order to override file `a` for the target file `b` which
> exists, we must execute two **non-atomic** operations as a whole, but
> each atomic in part, rename operations: make `b` -> `c`, and then
> rename `a` -> `b`. So there is actually a small time-frame when I can
> be left with two files (`a` and `c`), none of which is my config file
> `b`. (This can be solved when opening the config file by checking if
> there isn't any leftover `c` or `a` file, in which case I take the `a`
> file and complete the rename.)

There is only one ".notmuch-config" entry in the inode directory that is
your homedir.  it points either to the old file, or the new file.  it
cannot point to both, and it will not point to anything but those two
possibilities.  This is what the atomicity of the operation is expected
to guarantee.

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On 11/16/2010 03:37 PM, Daniel Kahn Gillmor wrote:
> On 11/16/2010 03:26 PM, Ciprian Dorin, Craciun wrote:
>>     So in the light of the above quoted "glitches", my question is:
>> due to the small chance of a power loss happening right when we write
>> such a small file, doesn't the inconvenience weight more than the
>> (fairly remote probable) file loss?
> 
> What inconvenience?  The inconvenience of writing the code correctly?

Ah sorry -- on re-reading, i see you're probably referring to the
inconvenience of breaking hardlinks, dropping permissions, ACLs and
other metadata.

That's an open question, as far as i'm concerned.  it'd be nice if there
was a way to "clone" a file's permissions and metadata to get the best
of both worlds.  maybe someone knows some tricks to do that?

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Ciprian Dorin, Craciun]

On Tue, Nov 16, 2010 at 22:42, Daniel Kahn Gillmor
<dkg@fifthhorseman.net> wrote:
> On 11/16/2010 03:37 PM, Daniel Kahn Gillmor wrote:
>> On 11/16/2010 03:26 PM, Ciprian Dorin, Craciun wrote:
>>>     So in the light of the above quoted "glitches", my question is:
>>> due to the small chance of a power loss happening right when we write
>>> such a small file, doesn't the inconvenience weight more than the
>>> (fairly remote probable) file loss?
>>
>> What inconvenience?  The inconvenience of writing the code correctly?
>
> Ah sorry -- on re-reading, i see you're probably referring to the
> inconvenience of breaking hardlinks, dropping permissions, ACLs and
> other metadata.

    Yes, exactly this is what I'm referring to: the meta-data that is
not copied when a new file is created.


> That's an open question, as far as i'm concerned.  it'd be nice if there
> was a way to "clone" a file's permissions and metadata to get the best
> of both worlds.  maybe someone knows some tricks to do that?
>
>        --dkg

Re: `notmuch setup` replaces `~/.notmuch-config` instead of truncating it

[Ciprian Dorin, Craciun]

On Tue, Nov 16, 2010 at 22:37, Daniel Kahn Gillmor
<dkg@fifthhorseman.net> wrote:
> On 11/16/2010 03:26 PM, Ciprian Dorin, Craciun wrote:
>>     P.S.: I say "pseudo" atomic because only the rename is atomic,
>> thus in order to override file `a` for the target file `b` which
>> exists, we must execute two **non-atomic** operations as a whole, but
>> each atomic in part, rename operations: make `b` -> `c`, and then
>> rename `a` -> `b`. So there is actually a small time-frame when I can
>> be left with two files (`a` and `c`), none of which is my config file
>> `b`. (This can be solved when opening the config file by checking if
>> there isn't any leftover `c` or `a` file, in which case I take the `a`
>> file and complete the rename.)
>
> There is only one ".notmuch-config" entry in the inode directory that is
> your homedir.  it points either to the old file, or the new file.  it
> cannot point to both, and it will not point to anything but those two
> possibilities.  This is what the atomicity of the operation is expected
> to guarantee.
>
>        --dkg


    Actually I've been wrong about this... I've thought that the way
the file is "overwritten" is actually done by either two `rename`
calls or by `link`, followed by another `link`, and then finally an
`unlink` (this is what I've tried to explain in my previous email).

    In fact I've thought that the `rename` OS call can't overwrite a
file if it exists.

    But after reading the man page of `rename(2)` -- quoted below -- I
was indeed wrong to call the atomicity as being "pseudo".

~~~~
       int rename(const char *oldpath, const char *newpath);
...
       If newpath already exists it will be atomically replaced
(subject to a few conditions; see ERRORS below),  so  that  there  is
no
       point at which another process attempting to access newpath
will find it missing.
...
       If newpath exists but the operation fails for some reason
rename() guarantees to leave an instance of newpath in place.
~~~~

    So indeed the behavior is completely atomic.

    Ciprian.