Org + git branches for derived files

Org + git branches for derived files

[Ken Mankoff]

Hello,

I think this might be more of a git question than an Org question, but I imagine I might find the answer here and that it might be useful to others, so I ask here.

I'd like to keep derivative products (the LaTeX output, the final PDF, etc.) available in Git, but not commit those changes each time I change the Org file. Perhaps git-annex as appropriate for this, but seems over-kill.

Is there some way to mostly-seamlessly commit the LaTeX and/or PDF and/or other files to their own git branches but in a way that overwrites the history of that branch, so that a small Org file that generates a big binary PDF does not pollute the git tree?

Thanks,

  -k.

Re: Org + git branches for derived files

[Juan Manuel Macías]

Hi Ken,

Ken Mankoff writes:

> I'd like to keep derivative products (the LaTeX output, the final PDF,
> etc.) available in Git, but not commit those changes each time I
> change the Org file. Perhaps git-annex as appropriate for this, but
> seems over-kill.
>
> Is there some way to mostly-seamlessly commit the LaTeX and/or PDF
> and/or other files to their own git branches but in a way that
> overwrites the history of that branch, so that a small Org file that
> generates a big binary PDF does not pollute the git tree?

There are probably better solutions, but maybe this could help you:
use org-publish (which also works when the output is LaTeX/PDF, and
not only with HTML). See https://orgmode.org/manual/Publishing.html

You should have to define a new publishing project and declare a
directory for the PDFs derivatives, applying a value to
:publishing-directory, and configure in this directory a second git
repository, only for PDFs. The value of :publishing-function keyword
should be `org-latex-publish-to-pdf'.

Best regards,

Juan Manuel 

Re: Org + git branches for derived files

[Tim Cross]

Ken Mankoff <mankoff@gmail.com> writes:

> Hello,
>
> I think this might be more of a git question than an Org question, but I imagine
> I might find the answer here and that it might be useful to others, so I ask
> here.
>
> I'd like to keep derivative products (the LaTeX output, the final PDF, etc.)
> available in Git, but not commit those changes each time I change the Org file.
> Perhaps git-annex as appropriate for this, but seems over-kill.
>
> Is there some way to mostly-seamlessly commit the LaTeX and/or PDF and/or other
> files to their own git branches but in a way that overwrites the history of that
> branch, so that a small Org file that generates a big binary PDF does not
> pollute the git tree?
>
> Thanks,
>
>   -k.

I think your definitely in salmon mode here :-) and swimming against the
flow. Typically, you would not add derived artefacts to git at all. You
have a full history of your original org file versions and therefore can
extract any version and use it to re-generate any version of a derived
artefact, so you would typically just ignore them and just re-generate
as needed.

Of course the 'easy' solution is simply not to do 'git add' on the
artefact until you want it committed. Provided you don't use 'git commit
-a' and only use 'git add' then the modified file will not be added to
git. When your ready to add the updates to git, then do 'git add' and
then commit. 

If you really want this, I can think of 2 possible solutions.
The first would be to use git submodules - basically, turn a directory
in your main git repository into a separate git repository that is a
child of your main git repository. When you have generated a new
artefact and want to keep it in git, copy it to the sub-module
repository and commit it there. In your main repository, you would add
the files to .gitignore. Now, unless you copy a new version into the
sub-module directory, it won't be added. It also won't show up in
diffs/patches generated from your main parent repository.  

The second solution would be to use the command

git update-index --assume-unchanged FILE_NAME

to prevent git from committing that file and when your ready to commit
it, do

git update-index --no-assume-unchanged FILE_NAME

which will make the file show up in git status etc again.

There is no way I know of to stop git from tracking/recording the
changes - this is how it works and of course, it doesn't work well with
binary or semi-binary files. You will get large diffs when dealing with
PDFs etc.

Personally, if I had to do something like this, I would go with the
sub-module approach rather than 'update-index'. This would at least keep
the 'mess' in a separate git repo so that you can do diffs and patches
on your main repo without lots of 'noise' and it would be less prone to
forgetting to change the file index settings etc.

In the past, when I have had a repository where I wanted a PDF to be
available to users of the repository, rather than adding that PDF to the
repository, I added a command (script, makefile entry etc) that would
generate the derived artefact on demand rather than having it in the
repository. 

Re: Org + git branches for derived files

[Mark Barton]

Ken,

You could consider using git-lfs, Large File Support. There is some setup and then you can say track *.pdf and that will tell git to track the binary file in a more efficient way. I use this mailing for csv files that I want to have a snapshot version of with the Jupyter notebook that used them. Once you are tracking the files with git-lfs, they will be tracked with the normal git commits.

I agree that the best practice is not to commit these types of files, but sometimes it is handy to. By committing the PDF files to the repo, I can use Working Copy, a git client, on my iPad to quickly reference a document. Since the iPad cannot run Emacs, I am unable to regenerate the PDF from there.

Mark

> On Aug 13, 2021, at 11:40 AM, Ken Mankoff <mankoff@gmail.com> wrote:
> 
> Hello,
> 
> I think this might be more of a git question than an Org question, but I imagine I might find the answer here and that it might be useful to others, so I ask here.
> 
> I'd like to keep derivative products (the LaTeX output, the final PDF, etc.) available in Git, but not commit those changes each time I change the Org file. Perhaps git-annex as appropriate for this, but seems over-kill.
> 
> Is there some way to mostly-seamlessly commit the LaTeX and/or PDF and/or other files to their own git branches but in a way that overwrites the history of that branch, so that a small Org file that generates a big binary PDF does not pollute the git tree?
> 
> Thanks,
> 
>  -k.
> 

Re: Org + git branches for derived files

[Rob Sargent]

> On Aug 13, 2021, at 6:54 PM, Mark Barton <mbarton98@gmail.com> wrote:
> 
> Ken,
> 
> You could consider using git-lfs, Large File Support. There is some setup and then you can say track *.pdf and that will tell git to track the binary file in a more efficient way. I use this mailing for csv files that I want to have a snapshot version of with the Jupyter notebook that used them. Once you are tracking the files with git-lfs, they will be tracked with the normal git commits.
> 
> I agree that the best practice is not to commit these types of files, but sometimes it is handy to. By committing the PDF files to the repo, I can use Working Copy, a git client, on my iPad to quickly reference a document. Since the iPad cannot run Emacs, I am unable to regenerate the PDF from there.
> 
> Mark

If you’re using GitHub or gitlab you can place artifacts along side your code repo. One often sees executables and jars there. Typically built and updated by mechanisms on the holster on a  “release” action or similar event

Re: Org + git branches for derived files

[Rob Sargent]

> On Aug 13, 2021, at 7:53 PM, Rob Sargent <robjsargent@gmail.com> wrote:
> 
> 
> 
>> On Aug 13, 2021, at 6:54 PM, Mark Barton <mbarton98@gmail.com> wrote:
>> 
>> Ken,
>> 
>> You could consider using git-lfs, Large File Support. There is some setup and then you can say track *.pdf and that will tell git to track the binary file in a more efficient way. I use this mailing for csv files that I want to have a snapshot version of with the Jupyter notebook that used them. Once you are tracking the files with git-lfs, they will be tracked with the normal git commits.
>> 
>> I agree that the best practice is not to commit these types of files, but sometimes it is handy to. By committing the PDF files to the repo, I can use Working Copy, a git client, on my iPad to quickly reference a document. Since the iPad cannot run Emacs, I am unable to regenerate the PDF from there.
>> 
>> Mark
> 
> If you’re using GitHub or gitlab you can place artifacts along side your code repo. One often sees executables and jars there. Typically built and updated by mechanisms on the holster on a  “release” action or similar event

I see autocorrect preferred “holster” to “hoster”

Re: Org + git branches for derived files

[Samuel Wales]

it is an interesting question as many things go into the product:

- org file
- your .el files
- org version
- emacs version
- os
- hw
- export commands run

so the desire to keep the product is understandable.  not only is it
effort to find the export commands run, but other stuff might change
out from under you.

could you perhaps export to something more compressible like ascii or
latex only?  and then use whatever org uses to create pdf?  git isn't
supposed to be for recreatable products, but ... is it TRULY trivially
recreatable?  :)

a colophon in the result can be useful maybe.


On 8/13/21, Rob Sargent <robjsargent@gmail.com> wrote:
>
>
>> On Aug 13, 2021, at 7:53 PM, Rob Sargent <robjsargent@gmail.com> wrote:
>>
>>
>>
>>> On Aug 13, 2021, at 6:54 PM, Mark Barton <mbarton98@gmail.com> wrote:
>>>
>>> Ken,
>>>
>>> You could consider using git-lfs, Large File Support. There is some setup
>>> and then you can say track *.pdf and that will tell git to track the
>>> binary file in a more efficient way. I use this mailing for csv files
>>> that I want to have a snapshot version of with the Jupyter notebook that
>>> used them. Once you are tracking the files with git-lfs, they will be
>>> tracked with the normal git commits.
>>>
>>> I agree that the best practice is not to commit these types of files, but
>>> sometimes it is handy to. By committing the PDF files to the repo, I can
>>> use Working Copy, a git client, on my iPad to quickly reference a
>>> document. Since the iPad cannot run Emacs, I am unable to regenerate the
>>> PDF from there.
>>>
>>> Mark
>>
>> If you’re using GitHub or gitlab you can place artifacts along side your
>> code repo. One often sees executables and jars there. Typically built and
>> updated by mechanisms on the holster on a  “release” action or similar
>> event
>
> I see autocorrect preferred “holster” to “hoster”
>
>
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Org + git branches for derived files

[Ken Mankoff]

Hi All,

Thank you for the suggestions.

I think the most elegant solution is to have a hook on GitHub that compiles the PDF on a remote server. But it takes a lot more work, because I don't necessarily have *everything* in Git - my local 'library.bib' usually isn't included, nor my custom emacs config, latexmkrc, etc. 

I'd just like the compiled PDF easily readable by anyone, but I don't want 100s of historical copies.

There are a few solutions.

1) Maintain a branch with the 'no-history' files. When they need to be updated, commit and amend, then force-push. See https://stackoverflow.com/questions/22824922/git-commit-and-push-a-binary-file-but-dont-keep-history

2) Add the 'no-history' files to their own commit in the main branch. When they need to be updated, make a new commit and rebase/fixup from the previous commit. See https://stackoverflow.com/questions/12964145/how-to-config-git-to-overwrite-non-text-file-instead-of-version-controlled-it

  -k.

Re: Org + git branches for derived files

[Rob Sargent]

> On Aug 15, 2021, at 7:02 PM, Ken Mankoff <mankoff@gmail.com> wrote:
> 
> Hi All,
> 
> Thank you for the suggestions.
> 
> I think the most elegant solution is to have a hook on GitHub that compiles the PDF on a remote server. But it takes a lot more work, because I don't necessarily have *everything* in Git - my local 'library.bib' usually isn't included, nor my custom emacs config, latexmkrc, etc. 
> 
> I'd just like the compiled PDF easily readable by anyone, but I don't want 100s of historical copies.
> 
> There are a few solutions.
> 
> 1) Maintain a branch with the 'no-history' files. When they need to be updated, commit and amend, then force-push. See https://stackoverflow.com/questions/22824922/git-commit-and-push-a-binary-file-but-dont-keep-history
> 
> 2) Add the 'no-history' files to their own commit in the main branch. When they need to be updated, make a new commit and rebase/fixup from the previous commit. See https://stackoverflow.com/questions/12964145/how-to-config-git-to-overwrite-non-text-file-instead-of-version-controlled-it
> 
>  -k.
Just to be clear, one is not required to use automated tools to generate the artifacts. They may be added manually. However, they are not typically retained forever so your own separate report may better suit your needs. 
> 

Re: Org + git branches for derived files

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

Hi Ken,

It’s been a while, but just to chime in: if you use GitHub you may find
<https://github.com/marketplace/actions/org-knit> to be of use.

Let me know if you have any questions, I wrote it a few months ago.

Ken Mankoff <mankoff@gmail.com> writes:

> Hello,
>
> I think this might be more of a git question than an Org question, but I imagine I might find the answer here and that it might be useful to others, so I ask here.
>
> I’d like to keep derivative products (the LaTeX output, the final PDF, etc.)
> available in Git, but not commit those changes each time I change the Org file.
> Perhaps git-annex as appropriate for this, but seems over-kill.
>
> Is there some way to mostly-seamlessly commit the LaTeX and/or PDF and/or other
> files to their own git branches but in a way that overwrites the history of that
> branch, so that a small Org file that generates a big binary PDF does not
> pollute the git tree?

All the best,
Timothy