org files and projects nested git repositories

org files and projects nested git repositories

[Tyler Smith]

Hello list,

I'm struggling to incorporate project-specific org files into my global
org system. My file system looks something like this:

~
├── org
│   ├── todo.org
│   ├── reading.org
│   └── personal.org
└── research
    ├── project1
    │   ├── project1.org
    │   └── project1_reference.pdf
    └── project2
        ├── project2.org
        ├── data2
        │   ├── data2A.csv
        │   └── data2B.csv
        └── project2_reference.pdf

When I open a new project, it gets its own folder, containing supporting
documents, code, data, as well as an org file to manage todos and
timelines. This doesn't seem to work well with org: most tutorials
assume that all your planning files are found in one directory (e.g.,
~/org).

I wonder if it would be possible to maintain a global git repository
containing my entire directory tree, but only the org files; and have
all of these org files incorporated into (and new ones automatically
detected by) the agenda. At the same time, I need to maintain individual
git repositories for each project directory.

Does anyone here have experience with such a system? Or other advice on
how to keep many scattered org files in a single git repo and also
captured in the org agenda?

Thanks for your patience,

Tyler

-- 
http://plantarum.ca

Re: org files and projects nested git repositories

[Ken Mankoff]

Hi Tyler,

I have a similar setup.

On 2015-12-31 at 12:38, Tyler Smith <tyler@plantarum.ca> wrote:
> ├── org
> │   ├── todo.org
> │   ├── reading.org
> │   └── personal.org

On my system this is a git repository and a cron (or actually LaunchAgent since I'm on OS X) does a =git commit -a <date-time-stamp>= every night. I also have these variables set:

(setq org-directory "~/Documents/Org")
(setq org-agenda-files (quote ("~/Documents/Org/" "~/Dropbox/Org/")))

> └── research
>     ├── project1
>     │   ├── project1.org
>     │   └── project1_reference.pdf
>     └── project2
>         ├── project2.org
>         ├── data2
>         │   ├── data2A.csv
>         │   └── data2B.csv
>         └── project2_reference.pdf

In my version of this, I have each project as its own git repository, where git contains the Org file, other code, the manuscript, etc. I add any Org files to my global agenda with =M-x org-agenda-file-to-front= and remove with =   M-x org-remove-file=.

> When I open a new project, it gets its own folder, containing supporting
> documents, code, data, as well as an org file to manage todos and
> timelines. This doesn't seem to work well with org: most tutorials
> assume that all your planning files are found in one directory (e.g.,
> ~/org).

That is only because it is easier to show that setup in the tutorial. You can have Org files all over the place, and each can be added to the Agenda or not, as you wish.

  -k.

Re: org files and projects nested git repositories

[Kyle Meyer]

Hi Tyler,

Tyler Smith <tyler@plantarum.ca> writes:

> Hello list,
>
> I'm struggling to incorporate project-specific org files into my global
> org system. My file system looks something like this:
>
> ~
> ├── org
> │   ├── todo.org
> │   ├── reading.org
> │   └── personal.org
> └── research
>     ├── project1
>     │   ├── project1.org
>     │   └── project1_reference.pdf
>     └── project2
>         ├── project2.org
>         ├── data2
>         │   ├── data2A.csv
>         │   └── data2B.csv
>         └── project2_reference.pdf

[...]

> I wonder if it would be possible to maintain a global git repository
> containing my entire directory tree, but only the org files; and have
> all of these org files incorporated into (and new ones automatically
> detected by) the agenda. At the same time, I need to maintain individual
> git repositories for each project directory.
>
> Does anyone here have experience with such a system? Or other advice on
> how to keep many scattered org files in a single git repo and also
> captured in the org agenda?

In addition to what Ken suggested, another option is to use links to
populate the agenda directory.  Have a git repo for your Org notes and
agenda file (your "org" directory above), and add this directory (or one
of its subdirectories) to org-agenda-files so that all the Org files in
that directory are considered agenda files.

Keep project-related Org files in the given project directory (as you do
above), and create a symbolic link for the Org file in the agenda
directory.  The project directory will then track the content of the Org
file, and the agenda repo will just track the presence of the link.

Your example from above would look like this:

     ~
     ├── org
     │   ├── todo.org
     │   ├── reading.org
     │   ├── personal.org
     │   ├── link to ~/research/project1/project1.org
     │   └── link to ~/research/project2/project2.org
     └── research
         ├── project1
         │   ├── project1.org
         │   └── project1_reference.pdf
         └── project2
             ├── project2.org
             ├── data2
             │   ├── data2A.csv
             │   └── data2B.csv
             └── project2_reference.pdf

I use the command at the link below to add Org files to my agenda
directory.

  https://github.com/kyleam/emacs.d/blob/3eda2253d80322640be4ec0381194d4f387e6cc9/lisp/init-org.el#L437

Also, if the project Org file isn't something you want to always show up
in your global TODO list, you can just restrict the agenda to that file
(i.e., press "<") instead of adding it to the agenda directory.

--
Kyle

Re: org files and projects nested git repositories

[Tyler Smith]

Ken Mankoff <mankoff@gmail.com> writes:
>
> In my version of this, I have each project as its own git repository,
> where git contains the Org file, other code, the manuscript, etc. I
> add any Org files to my global agenda with =M-x
> org-agenda-file-to-front= and remove with = M-x org-remove-file=.
>
>>  most tutorials assume that all your planning files are found in one
>>  directory (e.g., ~/org).
>
> That is only because it is easier to show that setup in the tutorial. You
> can have Org files all over the place, and each can be added to the Agenda
> or not, as you wish.

Thanks Ken. Perhaps I've been trying to hard to force everything into
one directory. I did find a post on Reddit that describes a convenient
way to create org files for projects anywhere in the directory tree in a
shared global location:

https://www.reddit.com/r/emacs/comments/2qwh8q/org_mode_one_massive_file_or_tons_of_small_ones/cnacdpl

This has the benefit of keeping all my planning (.org) files in one
repo, but the disadvantage of separating the planning files from the
project they relate to. Something your approach deals with nicely. I'll
try that out.

Best,

Tyler

-- 
http://plantarum.ca

Re: org files and projects nested git repositories

[Tyler Smith]

Kyle Meyer <kyle@kyleam.com> writes:
>
> In addition to what Ken suggested, another option is to use links to
> populate the agenda directory.  Have a git repo for your Org notes and
> agenda file (your "org" directory above), and add this directory (or one
> of its subdirectories) to org-agenda-files so that all the Org files in
> that directory are considered agenda files.
>
> Keep project-related Org files in the given project directory (as you do
> above), and create a symbolic link for the Org file in the agenda
> directory.  The project directory will then track the content of the Org
> file, and the agenda repo will just track the presence of the link.
>

Thanks Kyle.

I have tried links, but in all my attempts the links break whenever the
source or the target is committed to version control. My understanding
that each new commit to a file effectively changes the inode, breaking
the link. In your use, are the targets and their links both under
version control? If so, obviously they aren't breaking, so I must be
doing something wrong when I try the same thing.

Best,

Tyler

-- 
http://plantarum.ca

Re: org files and projects nested git repositories

[Mike McLean]

[-- Attachment #1: Type: text/plain, Size: 1738 bytes --]

On Thu, Dec 31, 2015 at 1:33 PM, Tyler Smith <tyler@plantarum.ca> wrote:

> Ken Mankoff <mankoff@gmail.com> writes:
> >
> > In my version of this, I have each project as its own git repository,
> > where git contains the Org file, other code, the manuscript, etc. I
> > add any Org files to my global agenda with =M-x
> > org-agenda-file-to-front= and remove with = M-x org-remove-file=.
> >
> >>  most tutorials assume that all your planning files are found in one
> >>  directory (e.g., ~/org).
> >
> > That is only because it is easier to show that setup in the tutorial. You
> > can have Org files all over the place, and each can be added to the
> Agenda
> > or not, as you wish.
>
> Thanks Ken. Perhaps I've been trying to hard to force everything into
> one directory. I did find a post on Reddit that describes a convenient
> way to create org files for projects anywhere in the directory tree in a
> shared global location:
>
>
> https://www.reddit.com/r/emacs/comments/2qwh8q/org_mode_one_massive_file_or_tons_of_small_ones/cnacdpl
>
> This has the benefit of keeping all my planning (.org) files in one
> repo, but the disadvantage of separating the planning files from the
> project they relate to. Something your approach deals with nicely. I'll
> try that out.
>

Another option: I have a function that recursively searches a parent
directory (using the "find-lisp" library) for files that are named *.org. I
then add that function to org-agenda-mode-hook so that every time I open
and agenda view, it recalculates the org-agenda-files variable. On a local
filesystem (laptop, SSD) i don't notice a thing, slow-down wise. YMMV if
your Org files are on a network share.


>
> Best,
>
> Tyler
>
> --
> http://plantarum.ca
>
>

[-- Attachment #2: Type: text/html, Size: 2710 bytes --]

Re: org files and projects nested git repositories

[Brenda Butler]

[-- Attachment #1: Type: text/plain, Size: 488 bytes --]

On Thu, Dec 31, 2015 at 1:38 PM, Tyler Smith <tyler@plantarum.ca> wrote:

> I have tried links, but in all my attempts the links break whenever the
> source or the target is committed to version control. My understanding
> that each new commit to a file effectively changes the inode, breaking
> the link


Sounds like you might be using hard links instead of symbolic links ...
but first, are you using a unix-like system or another system?  The
symbolic link is a unix-like thing.

bjb

[-- Attachment #2: Type: text/html, Size: 866 bytes --]

Re: org files and projects nested git repositories

[Kyle Meyer]

Tyler Smith <tyler@plantarum.ca> writes:

[...]

> I have tried links, but in all my attempts the links break whenever the
> source or the target is committed to version control. My understanding
> that each new commit to a file effectively changes the inode, breaking
> the link.

Are you thinking of hard links rather than symbolic links?  Symbolic
links should play well with Git (ignoring Windows).  The popular tool
git-annex is even built around symbolic links.

> In your use, are the targets and their links both under
> version control? If so, obviously they aren't breaking, so I must be
> doing something wrong when I try the same thing.

Yes, the target is under version control in the project repo, and the
agenda repo tracks a symbolic link to the target, so the content of that
file is just a path to the target.

--
Kyle

Re: org files and projects nested git repositories

[Tyler Smith]

Brenda Butler <bjb@mojatatu.com> writes:

>  I have tried links, but in all my attempts the links break whenever the
>  source or the target is committed to version control. My understanding
>  that each new commit to a file effectively changes the inode, breaking
>  the link
>
> Sounds like you might be using hard links instead of symbolic links ... 
> but first, are you using a unix-like system or another system? The symbolic link is a unix-like thing.
>

I'm on Debian, so links are available. I recall having to fix an issue
with Emacs' backup-files taking over the inodes. That was easily fixed
once I knew it was an issue. I don't recall the details, but it seemed
that the links still got garbled on commit. I'll take another look,
now that I know it should work.

Tyler

-- 
http://plantarum.ca

Re: org files and projects nested git repositories

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

Hello Ken,

On 2015-12-31 13:01, Ken Mankoff <mankoff@gmail.com> writes:

> On 2015-12-31 at 12:38, Tyler Smith <tyler@plantarum.ca> wrote:
>> ├── org
>> │   ├── todo.org
>> │   ├── reading.org
>> │   └── personal.org
>
> On my system this is a git repository and a cron (or actually
> LaunchAgent since I'm on OS X) does a =git commit -a
> <date-time-stamp>= every night.

I'm very interested about this. Could you please share your LaunchAgent
configuration file?

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Athmospheric CO₂ (Updated December 28, 2015, Mauna Loa Obs.): 402.60 ppm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: org files and projects nested git repositories

[Ken Mankoff]

On 2016-01-01 at 05:02, Alan Schmitt <alan.schmitt@polytechnique.org> wrote:
> Hello Ken,
>>
>> On my system this is a git repository and a cron (or actually
>> LaunchAgent since I'm on OS X) does a =git commit -a
>> <date-time-stamp>= every night.
>
> I'm very interested about this. Could you please share your LaunchAgent
> configuration file?

I miss the simpler days of cron, but when in Rome or on OS X...

The following is ~/Library/LaunchAgents/com.kenmankoff.orggit.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>com.kenmankoff.orggit</string>
	<key>ProgramArguments</key>
	<array>
		<string>/Users/mankoff/bin/orggitcommit.sh</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>StartInterval</key>
	<integer>86400</integer>
</dict>
</plist>

Which runs the following shell script:

#!/usr/bin/env bash
cd ~/Documents/Org/
/usr/local/bin/git commit -a -m "`/bin/date +%Y-%m-%d\ %T`"

Run the following 1x:

cd ~/Library/LaunchAgents/
launchctl load com.kenmankoff.orggit

Then in my Org folder, =git lg= (aliased to git log --graph --pretty=format:'%C(red)%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%ad)' --abbrev-commit --date=short) shows:

* 926cfa9 - 2015-12-30 12:51:55 (2015-12-30)
* cac0191 - 2015-12-22 08:53:39 (2015-12-22)
* be839e2 - 2015-12-20 01:32:03 (2015-12-20)
* 9a68231 - 2015-12-17 09:26:40 (2015-12-17)
* 9782112 - 2015-12-16 09:26:36 (2015-12-16)
* d7df50c - 2015-12-15 21:27:53 (2015-12-15)
* 635a0fa - 2015-12-15 17:41:55 (2015-12-15)
* 72cafc1 - 2015-12-14 11:08:48 (2015-12-14)

So you can see things get checked in to git about once every few days, unless my laptop is closed a lot like it just was over the holidays. If you wanted more frequent commits, LaunchAgents can run every time a file in a folder is saved. If that is too often, a middle ground could be to run the bash script every time a file is saved, but have that script only do the commit if it has been >12 or >24 hours, using the "find -newer" command, for example. Or only if >n lines have changed (from git diff).

  -k.

Re: org files and projects nested git repositories

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

On 2016-01-01 07:33, Ken Mankoff <mankoff@gmail.com> writes:

> On 2016-01-01 at 05:02, Alan Schmitt <alan.schmitt@polytechnique.org> wrote:
>> Hello Ken,
>>>
>>> On my system this is a git repository and a cron (or actually
>>> LaunchAgent since I'm on OS X) does a =git commit -a
>>> <date-time-stamp>= every night.
>>
>> I'm very interested about this. Could you please share your LaunchAgent
>> configuration file?
>
> I miss the simpler days of cron, but when in Rome or on OS X...

This is most useful, thanks a lot!

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Athmospheric CO₂ (Updated December 28, 2015, Mauna Loa Obs.): 402.60 ppm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]