Re: Managing articles in orgmode and collaboration

[Christoph Groth <christoph@grothesque.org>]

Jorge A. Alfaro-Murillo wrote:

> Perhaps I am biased because I learned LaTeX and BibTeX before 
> Org, but I think that for references BibTeX (plus a little bit 
> of emacs configuration) has everything I could need. I guess if 
> you are more used to Org, it might be worth to invest time and 
> come up with a org-based solution. Please keep us posted, I find 
> this a very interesting thread.

This is an update about my experiments so far.  Please comment! 
This is my current setup:

All the articles (mostly PDF files) are kept in one directory. 
There’s also an index.org file in there with headlines like this

--8<---------------cut here---------------start------------->8---
** 
   [[file:vazifeh13-electromagnetic_response_weyl.pdf][Electromagnetic 
   Response of Weyl Semimetals]]
:PROPERTIES:
:TITLE:    Electromagnetic Response of Weyl Semimetals
:BTYPE:    article
:ID:       vazifeh13-electromagnetic_response_weyl
:AUTHOR:   Vazifeh, M. M. and Franz, M.
:JOURNAL:  Phys. Rev. Lett.
(…)
:END:
Some comments/notes with links and in-line formulas.
--8<---------------cut here---------------end--------------->8---

These entries are created by org-bibtex-yank, Emacs’ bibtex is 
configured to generate the unique IDs (authorYY-three_first_words 
should be sufficiently unique in practice).

This setup works pretty well:

- Adding new articles is easy (there's still ample room for 
  improvement).
- Opening associated the associated article files and other links 
  works well.
- It’s easy to add notes (with in-line maths!) and do simple 
  searches.
- Linking from and to other org files (agenda, project-specific 
  notes) is possible.
- Additional attachment files (e.g. whiteboard photos from a 
  discussion) can added with org-attach.
- org-bibtex provides support to put entries into project-specific 
  bibtex files (org-bibtex-export-to-kill-ring).
- Thanks to the non-random unique IDs, it’s possible to jump to 
  articles in the library from references in latex files, even if 
  the key scheme in the latex file differs from the one used in 
  index.org.

But not all is good:

- Scaling: Some simple tests seem to indicate that org mode 
  becomes too sluggish with files of about 50k lines.  This is a 
  dimension that could be easily reached over 10 years if the file 
  grows by 20 lines per day on average.  This is not a problem in 
  the beginning, but if the scheme does not scale to a few 
  thousand entries, this renders the whole idea way less 
  interesting.

- More advanced searching is lacking: AFAIK org mode currently 
  does not support searching for articles of a given author that 
  also contain a given keyword in the notes.

Any insights about these two problems?  Perhaps the scaling could 
be managed by splitting index.org into several files (by year for 
example).  But how to search then?  It's probably not a good idea 
to add all the bibliography org-files into the agenda.  (Is there 
a way to have a secondary list of agenda files?)  Perhaps the 
solution for both problems would be to write a fast commandline 
query tool for such org-databases?  The tool could even use a fast 
cache if necessary.