Re: Git-LFS or Git Annex?

[Nicolas Graves via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org>]

On 2024-01-28 18:32, Giovanni Biscuolo wrote:

> Hi Nicolas,
>
> Nicolas Graves <ngraves@ngraves.fr> writes:
>
> [...]
>
>> This is not always true. Git-LFS also has the concept of Custom Transfer
>> Agents, which in some cases do not need a running server. One example is
>> lfs-folderstore, which can simply use a remote directory as a LFS
>> remote.
>
> thanks, i didn't know about custom transfer agents, the use withous an
> API server is documented here:
>
> --8<---------------cut here---------------start------------->8---
>
> In some cases the transfer agent can figure out by itself how and where
> the transfers should be made, without having to query the API server. In
> this case it's possible to use the custom transfer agent directly,
> without querying the server, by using the following config option:
>
>  lfs.standalonetransferagent, lfs.<url>.standalonetransferagent
>
> Specifies a custom transfer agent to be used if the API server URL
> matches as in "git config --get-urlmatch lfs.standalonetransferagent
> <apiurl>". git-lfs will not contact the API server. It instead sets
> stage 2 transfer actions to null. "lfs.<url>.standalonetransferagent"
> can be used to configure a custom transfer agent for individual
> remotes. "lfs.standalonetransferagent" unconditionally configures a
> custom transfer agent for all remotes. The custom transfer agent must be
> specified in a "lfs.customtransfer.<name>" settings group.
>
> --8<---------------cut here---------------end--------------->8---
> (https://github.com/git-lfs/git-lfs/blob/main/docs/custom-transfers.md#using-a-custom-transfer-type-without-the-api-server)
>
> some examples:
>
> 1. git-lfs-agent-scp: A custom transfer agent for git-lfs that uses scp
>    to transfer files. This transfer agent makes it possible to use
>    git-lfs in situations where the remote only speaks ssh. This is
>    useful if you do not want to install a git-lfs server. (MIT license,
>    written in C, URL: https://github.com/tdons/git-lfs-agent-scp)
>
> 2. git-lfs-rsync-agent: The rsync git-lfs custom transfer agent allows
>    transferring the data through rsync, for example using SSH
>    authentication. (MIT license, written in Go, URL:
>    https://github.com/excavador/git-lfs-rsync-agent)
>
> 3. git-lfs-agent-scp-bash: A custom transfer agent for git-lfs that uses
>    scp to transfer files. This is a self-contained bash script designed
>    for seamless installation, requiring no prerequisites with the
>    exception of the external command scp. It enables to use git-lfs even
>    if you can not use http/https but ssh only. (MIT License, written in
>    bash, URL: https://github.com/yoshimoto/git-lfs-agent-scp-bash)
>
> So yes: we could use git-lfs without a git-lfs server and set an rsync
> or scp transfer agent for each remote (documenting it for users, since
> this must be done client-side)
>
> It's not at all as powerful as the location tracking features of
> git-annex but... doable :-)

One downside however : For some reason, Custom Transfer Agents are
rarely well written, well supported projects. I had to write one myself
for my use-case, I now have a good understanding of the protocol which
is definitely simple, but it's a necessary thing to note that despite
this rather easy extensibility, few CTAs are good/reliable in the long
run.

>
> [...]
>
>>> Another important limitation of Git-LFS is that you cannot delete
>>> (remotely stored) objects [1], with git-annex is very easy.
>>
>> Probably true, haven't encountered the use-case yet.
>
> IMHO this is a very important feature when you have to manage media
> archives.

Depends on the use-case! If you're just looking for an archival tool of
a media with one unmutable version which you want to support for an
indefinite amount of time, doesn't matter that much.

Gitlab and Github documentation say it's possible with the
git-filter-repo extension, but that is indeed not easy.
https://docs.gitlab.com/ee/topics/git/lfs/#removing-objects-from-lfs

>
> [...]
>
>> Just a note on upsides of Git-LFS :
>> - integration with git is better. A special magit extension to use
>> git-lfs is not needed, whereas it is with git-annex.
>
> true :-D
>
>> - less operations: once I know which files will be my media files, I
>> have less headaches (basically the exact git experience, you don't have
>> to think about where I should `git add` or `git annex add` a file).
>
> it's the same with git-annex, you just have to configure/distribute a
> .gitattributes file, i.e.:
>
> --8<---------------cut here---------------start------------->8---
>
> * annex.largefiles=(largerthan=5Mb)
> * annex.largefiles=(not(mimetype=text/*))
>
> --8<---------------cut here---------------end--------------->8---
>
> see https://git-annex.branchable.com/tips/largefiles/ for a description
> of this feature

Nice! I haven't experimented with this. Funny how it's one extension of
a git extension that provides what LFS does natively. 

>
>> It's indeed less copyleft though. Simpler, but also maybe less adapted
>> to this use-case.
>
> With git-annex everyone can set up a "git-annex enabled" server
> (although haskel dependency is a limitation since it's unsupported in
> many architectures)... or use one of the available special remotes.
>
> Thanks! Gio'

One upside I've forgotten that git-annex may also provide (does it ?) :
if the CTA is well-written, progress update during upload/download is
quite reassuring when sending heavy files.

-- 
Best regards,
Nicolas Graves