[OT] Git plus Syncthing: breaking hard links

[OT] Git plus Syncthing: breaking hard links

[Eric Abrahamsen]

This isn't really a Org question at all, but you all are smart and
friendly people, and are likely to have run into this situation before.
So I'm trying here before I turn to StackOverflow!

I have Org files in a git repo, synced across two Linux machines. At the
same time, I have a few directories on these computers that I sync with
colleagues, via Syncthing.

One of these colleagues also uses Org, so instead of giving her access
to my Org git repo, which would be a bit too much exposure, I hardlink
some of the Org files into the Syncthing directories, and she accesses
them there. With auto-revert-mode turned on, it's not too bad.

Except that the hardlinks keep breaking! There are so many different
systems interacting here that I have no idea where to even start
looking. She's on a Mac, not Linux, and uses a fairly recent version of
Emacs, plus Org from the ELPA package. She is *not* using git on her
machine, just Syncthing and Emacs.

I'm inclined to blame Syncthing, but I really don't know where to start
debugging. If any of you have any relevant experience, or any advice
about where to look, that would be much appreciated. Otherwise, please
excuse the off-topic post...

Thanks,
Eric

Re: [OT] Git plus Syncthing: breaking hard links

[Eric S Fraga]

Hi Eric,

I don't know syncthing at all and how MacOS works even less [1] but I do
know emacs... a little.  Could the problem be related to how emacs does
backups?  cf. backup-by-copying and make-backup-files

Just throwing things into the mix in case something helps!

Footnotes: 
[1]  the last Apple computer I used was the Lisa... ;-)

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-668-g809a83

Re: [OT] Git plus Syncthing: breaking hard links

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Hi Eric,
>
> I don't know syncthing at all and how MacOS works even less [1] but I do
> know emacs... a little.  Could the problem be related to how emacs does
> backups?  cf. backup-by-copying and make-backup-files

Hmm, I have my Emacs set up to not make backups, but maybe my
colleague's Emacs isn't... I will take a look.

> Just throwing things into the mix in case something helps!

That's basically all I was hoping for from this list! Thanks for
throwing things in :)

E

Re: [OT] Git plus Syncthing: breaking hard links

[Nick Dokos]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> This isn't really a Org question at all, but you all are smart and
> friendly people, and are likely to have run into this situation before.
> So I'm trying here before I turn to StackOverflow!
>
> I have Org files in a git repo, synced across two Linux machines. At the
> same time, I have a few directories on these computers that I sync with
> colleagues, via Syncthing.
>
> One of these colleagues also uses Org, so instead of giving her access
> to my Org git repo, which would be a bit too much exposure, I hardlink
> some of the Org files into the Syncthing directories, and she accesses
> them there. With auto-revert-mode turned on, it's not too bad.
>
> Except that the hardlinks keep breaking! There are so many different
> systems interacting here that I have no idea where to even start
> looking. She's on a Mac, not Linux, and uses a fairly recent version of
> Emacs, plus Org from the ELPA package. She is *not* using git on her
> machine, just Syncthing and Emacs.
>

How do the hardlinks break exactly?

Hardlinks to a file are limited to the same filesystem: a file with
inode number N has essentially two or more directory entries, with
different names associated to the same inode number. Syncthing certainly
cannot maintain that as it copies the file to the cloud and back, so
there seem to be plenty of opportunities for breakage, but it's not
clear which one(s) obtain.

> I'm inclined to blame Syncthing, but I really don't know where to start
> debugging. If any of you have any relevant experience, or any advice
> about where to look, that would be much appreciated. Otherwise, please
> excuse the off-topic post...
>

-- 
Nick

Re: [OT] Git plus Syncthing: breaking hard links

[Tim Howes]

To avoid data loss, Syncthing creates a temporary file during transfer
with the name ~syncthing~.{filename}.tmp

If the transfer completes successfully, then it moves that file in place
of the previous version. It's a new inode, not an update of the previous
inode, so the hard link will be lost.

Instead of using hard links, you could share your git repo folder
directly using Syncthing, but use an .stignore file to exclude the files
that you want to keep private (such as the .git directory).

https://docs.syncthing.net/users/ignoring.html

-Tim

Re: [OT] Git plus Syncthing: breaking hard links

[briangpowell .]

[-- Attachment #1: Type: text/plain, Size: 3188 bytes --]

* Strongly recommend you pay close attention to Nick Dokos--he's
brilliant--agree totally with his suggestions; "hardlinks cannot span
filesystems" etc.

** "hardlinks breaking"--in rsync I throw -H to include/follow hard links
across filesystems when syncing.

*** I'm with Nick though again--not sure what you mean by "hardlinks
breaking"

* Thanks for the "syncthing" link and comments--its open source:

"Syncthing
https://syncthing.net/
Syncthing replaces proprietary sync and cloud services with something open,
trustworthy and decentralized. Your data is your data alone and you deserve
to ..."

** Sounds great, hope its easy to implement, thanks.

*** I prefer rsync for such problems at the moment.

* Suggest you use FuseFS and sshfs---and on her MacOS she can use BREW to
install the FUSE .DMG package and use OSX shell to get to the filesystems
that you're willing to share (you mentioned you were worried about
providing too much).

** Not recommending SAMBA+CIFS protocol--especially right now, since there
is a vulnerability to be revealed on 12APR16

** Or you alone could use FuseFS and sshfs the dirs you want her to have
access to on the machine you both share/access (you seem to indicate she
has full and comfortable access to those dirs already--so installing the
FUSE .DMG package is probably overkill).



On Tue, Apr 5, 2016 at 8:45 AM, Nick Dokos <ndokos@gmail.com> wrote:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> > This isn't really a Org question at all, but you all are smart and
> > friendly people, and are likely to have run into this situation before.
> > So I'm trying here before I turn to StackOverflow!
> >
> > I have Org files in a git repo, synced across two Linux machines. At the
> > same time, I have a few directories on these computers that I sync with
> > colleagues, via Syncthing.
> >
> > One of these colleagues also uses Org, so instead of giving her access
> > to my Org git repo, which would be a bit too much exposure, I hardlink
> > some of the Org files into the Syncthing directories, and she accesses
> > them there. With auto-revert-mode turned on, it's not too bad.
> >
> > Except that the hardlinks keep breaking! There are so many different
> > systems interacting here that I have no idea where to even start
> > looking. She's on a Mac, not Linux, and uses a fairly recent version of
> > Emacs, plus Org from the ELPA package. She is *not* using git on her
> > machine, just Syncthing and Emacs.
> >
>
> How do the hardlinks break exactly?
>
> Hardlinks to a file are limited to the same filesystem: a file with
> inode number N has essentially two or more directory entries, with
> different names associated to the same inode number. Syncthing certainly
> cannot maintain that as it copies the file to the cloud and back, so
> there seem to be plenty of opportunities for breakage, but it's not
> clear which one(s) obtain.
>
> > I'm inclined to blame Syncthing, but I really don't know where to start
> > debugging. If any of you have any relevant experience, or any advice
> > about where to look, that would be much appreciated. Otherwise, please
> > excuse the off-topic post...
> >
>
> --
> Nick
>
>
>

[-- Attachment #2: Type: text/html, Size: 4271 bytes --]

Re: [OT] Git plus Syncthing: breaking hard links

[Eric Abrahamsen]

Tim Howes <timhowes@berkeley.edu> writes:

> To avoid data loss, Syncthing creates a temporary file during transfer
> with the name ~syncthing~.{filename}.tmp
>
> If the transfer completes successfully, then it moves that file in place
> of the previous version. It's a new inode, not an update of the previous
> inode, so the hard link will be lost.

Ugh, that's what I was starting to suspect.

> Instead of using hard links, you could share your git repo folder
> directly using Syncthing, but use an .stignore file to exclude the files
> that you want to keep private (such as the .git directory).

Luckily you're right -- it isn't all that hard to work around. I'm
currently using one git repo to manage both my work-related *and*
personal Org files, and there's no particular reason to be doing that.
I'll split the repos, keep the work one in the Syncthing folder, and
sidestep the problem altogether.

Thanks to all who responded!

Eric

Re: [OT] Git plus Syncthing: breaking hard links

[Adam Porter]

Hi Eric,

Sorry I'm late, just thought I'd share an idea: did you try symlinks?  A
symlink points to a path, not an inode, so it shouldn't get broken like a
hard link.

Re: [OT] Git plus Syncthing: breaking hard links

[Adam Porter]

Hi again Eric,

On second thought, while symlinks may be worth trying, they might not work.
 I've noticed similar issues using Dropbox.

If you put the file in your git repo and the symlink in Syncthing's repo,
I'm guessing Syncthing will not follow the symlink, in which case it
wouldn't sync the contents of the file.  (I don't use Syncthing, so I could
be wrong.  If I am wrong, then this would probably solve the problem for you.)

If you put the file in Syncthing and the symlink in the git repo, git will
definitely not follow the symlink, so it will only store the symlink itself,
meaning the file contents would no longer be stored in git.  (This could
work, but I doubt it's what you want.)

It sounds like your workaround will solve the problem for you, so that's
great.  One suggestion though: I recommend excluding the .git directory from
Syncthing.  If anything happened on the other end and the git repo were
accidentally corrupted, or if it was committed to on both ends before the
changes were synced, your git repo could end up corrupted, and fixing it
could be a lot of trouble.

I've had a few problems like this before, so I no longer store VCS repos in
Dropbox.  I keep all my Org files in git repos, and I sync the Org files
themselves, but the git repos are local to each system.

Hope this helps!

Re: [OT] Git plus Syncthing: breaking hard links

[Eric Abrahamsen]

Adam Porter <adam@alphapapa.net> writes:

> Hi again Eric,
>
> On second thought, while symlinks may be worth trying, they might not work.
>  I've noticed similar issues using Dropbox.
>
> If you put the file in your git repo and the symlink in Syncthing's repo,
> I'm guessing Syncthing will not follow the symlink, in which case it
> wouldn't sync the contents of the file.  (I don't use Syncthing, so I could
> be wrong.  If I am wrong, then this would probably solve the problem for you.)
>
> If you put the file in Syncthing and the symlink in the git repo, git will
> definitely not follow the symlink, so it will only store the symlink itself,
> meaning the file contents would no longer be stored in git.  (This could
> work, but I doubt it's what you want.)

Yeah, I did think about symlinks, but think you're right -- they
probably won't function correctly no matter which "end" you put them in.

> It sounds like your workaround will solve the problem for you, so that's
> great.  One suggestion though: I recommend excluding the .git directory from
> Syncthing.  If anything happened on the other end and the git repo were
> accidentally corrupted, or if it was committed to on both ends before the
> changes were synced, your git repo could end up corrupted, and fixing it
> could be a lot of trouble.
>
> I've had a few problems like this before, so I no longer store VCS repos in
> Dropbox.  I keep all my Org files in git repos, and I sync the Org files
> themselves, but the git repos are local to each system.

That's good advice, I think I'll follow it! Thanks for sharing your
experience.

Eric