Orgmode for research information management

Orgmode for research information management

[Graham Smith]

I have used several information management tools over the years, but I
am trying to use Emacs/orgmode more. There are many examples of people
using Orgmode for time management type tasks, but not a lot on
information management.

I would be interested in any insights into  how people use orgmode for
information management eg. gathering information for scientific paper.

Many thanks,

Graahm

Re: Orgmode for research information management

[Chris Gray]

Graham Smith wrote:

> I have used several information management tools over the years, but I
> am trying to use Emacs/orgmode more. There are many examples of people
> using Orgmode for time management type tasks, but not a lot on
> information management.

> I would be interested in any insights into  how people use orgmode for
> information management eg. gathering information for scientific paper.

For me, it works great because it is so easy to use as an outliner.  I
have also been using it to write papers.  When I do this, I use it like
a glorified LaTeX.  The real advantage there is the ability to show and
hide sections.  The only disadvantage that I have found with the LaTeX
export is that it is best to use \( and \) rather than $ to enter and
exit math mode because org has some trouble parsing the $ across line
breaks.  But that is minor, and your fingers will get used to it if you
use a lot of math.

However, I still keep all my references in a separate
database (I use the Zotero plugin for Firefox, which I export to
bibtex).

Hope that helps.

Cheers,
Chris

Re: Re: Orgmode for research information management

[Graham Smith]

Chris,

Thanks, but I have a couple of more questions if that's OK

> For me, it works great because it is so easy to use as an outliner.

Do you use tags for particular linked topics or do you just rely on
the outline and drop related information into the same level of the
outline

> However, I still keep all my references in a separate
> database (I use the Zotero plugin for Firefox, which I export to
> bibtex).

Do you keep notes specific to a paper in Zotero, or keep them in Orgmode.

Graham

Re: Orgmode for research information management

[Chris Gray]

Graham Smith wrote:

> Chris,

> Thanks, but I have a couple of more questions if that's OK

>> For me, it works great because it is so easy to use as an outliner.

> Do you use tags for particular linked topics or do you just rely on
> the outline and drop related information into the same level of the
> outline

So far I have been using just the outlining capabilities, but tags
sounds like a great idea -- I think I'll start doing that too. :)  Until
now, though, I mostly find the information that I am looking for with
isearch.  

>> However, I still keep all my references in a separate
>> database (I use the Zotero plugin for Firefox, which I export to
>> bibtex).

> Do you keep notes specific to a paper in Zotero, or keep them in
> Orgmode.

I keep it all in orgmode.  I am addicted to the emacs key layout, so it
is easier to do the note-taking in emacs than in Firefox.  Plus, Zotero
doesn't seem so easy to search for me.

Cheers,
Chris

Re: Re: Orgmode for research information management

[Graham Smith]

Chris

> So far I have been using just the outlining capabilities, but tags
> sounds like a great idea -- I think I'll start doing that too. :)  Until
> now, though, I mostly find the information that I am looking for with
> isearch.
>
>>> However, I still keep all my references in a separate
>>> database (I use the Zotero plugin for Firefox, which I export to
>>> bibtex).
>
>> Do you keep notes specific to a paper in Zotero, or keep them in
>> Orgmode.
>
> I keep it all in orgmode.  I am addicted to the emacs key layout, so it
> is easier to do the note-taking in emacs than in Firefox.  Plus, Zotero
> doesn't seem so easy to search for me.

Thanks for the additional information. I need to think a bit more
about how to structure this.

Graham

Re: Orgmode for research information management

[Matthew Lundin]

Hi Graham,

Graham Smith <myotisone@gmail.com> writes:

> I would be interested in any insights into  how people use orgmode for
> information management eg. gathering information for scientific paper.

Org-mode is very well suited to this purpose. I like to think of
org-mode as an outliner with the functionality of a database. Each
outline heading/subheading is a node to which all sorts of metadata can
be attached. Thus, notes can be as free-form or as structured as the
project requires. The structure can emerge as the project progresses.

When beginning a research project, I dump a lot of material into a new
org file, making heavy use of org-remember and org-protocol. I tag items
as they come in, so I know that I'll be able to find them easily later.
I also create links to relevant files (pdfs, images, etc.). And any time
I need to follow up on something, I create a todo. Indeed, todo state
changes are immensely helpful for keeping track of what I've done so
far.

My citations are located in a bibtex database; org-mode is able to
create links to individual items via org-store-link or org-remember. I
make sure to add a link to every outline heading, whether it be to a
website, a bibtex item, a file, or to some other node in the outline
(via <<internal-links>> or ID properties).

The beauty of org-mode is that I can organize my outline only when the
need arises. Thanks to tags, searches, and sparse trees, I can easily
find individual notes. And thanks to hyperlinks I can travel quickly to
the source of the information.

Anytime I need a more structured database, I create one within the
outline by using properties and column views. Let's say I need to keep
track of sources I've looked at an archive. I can easily create a
mini-database for the archive.

--8<---------------cut here---------------start------------->8---
* Some Archive
  :PROPERTIES:
  :COLUMNS:  %25ITEM %10TIMESTAMP_IA %17collection %6doc-number %18author %25title %10origdate %TAGS
  :END:
** An old letter 				  :industrialization:letters:
   :PROPERTIES:
   :collection: Personal Letters
   :doc-number:   768b
   :author:   Michael Smithson
   :origdate: 1868
   :title:    Letter to brother
   :END:

Some notes about the letter here...

[2009-06-23 Tue 09:37]
--8<---------------cut here---------------end--------------->8---

So an individual org outline can contain numerous custom databases! The
possibilities are endless.

And, of course, at any point I can export any part of the outline to
pdf, html, ascii, etc.

Hope this helps.

- Matt

Re: Orgmode for research information management

[Graham Smith]

Matt


> Org-mode is very well suited to this purpose. I like to think of
> org-mode as an outliner with the functionality of a database. Each
> outline heading/subheading is a node to which all sorts of metadata can
> be attached. Thus, notes can be as free-form or as structured as the
> project requires. The structure can emerge as the project progresses.

That is all really useful and given me several things to think about,
and some thing to check out how to do. I'm just trying as far as
possible to get the structure right at the beginning.

Graham

Re: Orgmode for research information management

[Eric S Fraga, Eric S Fraga]

Graham Smith writes:
> [...]
> That is all really useful and given me several things to think about,
> and some thing to check out how to do. I'm just trying as far as
> possible to get the structure right at the beginning.

Actually, if nothing else, I think a very appealing aspect of org-mode
is its flexibility.  Even if you have the wrong structure at the
beginning, it's really easy to fix later.  Re-organisation (moving
headlines, for instance) is trivial *and* as org-mode has the full
power of emacs and is text based, making global changes are just as
trivial.

It's this flexibility that encourages one to simply jump in and use it
and the benefits then quickly make you wonder how you got along
without it!

Re: Re: Orgmode for research information management

[Graham Smith]

Eric

> It's this flexibility that encourages one to simply jump in and use it
> and the benefits then quickly make you wonder how you got along
> without it!

Strangely, when you are used to a more structured approach this makes
it difficult to start.

Graham

Re: Re: Orgmode for research information management

[Sebastian Rose]

Graham Smith <myotisone@gmail.com> writes:
> Strangely, when you are used to a more structured approach this makes
> it difficult to start.


I'd propose to start by partitioning _what_ you actually need to store
on your disk.

This is what I do:


* The `org-directory'

  The default is ~/org/ and you could simply live with that default for
  a while. Changing that later on is easy, as long as most of the
  targets for your remember templates live there (the org-directory is
  search if just a filename is supplied for a certain template). Just
  `mv' the directory and adjust `org-directory'. So no need to worry
  here.

  The main purpose of the org-directory is to find files without hassle:

    * agenda files (I use an extra sub directory for those)
    * Remember targets

  
  I use pretty much the Org-modes defaults for fast note taking
  (remember) and agenda stuff. Also, it makes sense to store all agenda
  stuff and notes you want to keep below one directory for easy backup
  and restore, or synching with your notebook(s).



*** Agenda files

    My org-directory has a subdirectory where I store all my agenda
    files, regardless of project. I have a file for each customer I work
    for over a longer period and a private `organizer.org'. Some of
    those files grew very big during the last two years, but that's
    absolutely no problem, neither for Org-mode, nor for me. But it
    makes it soo easy to find stuff again. I just press `C-c ,'
    repeatedly until the correct file is on screen.

    For each customer I created a remember template to record new tasks
    on the fly (e.g. phone calls).

    Some customers have a somewhat limited `lifetime' in my agenda (one
    project and that's it). Those agenda files are simply removed from
    the list of agenda files: `C-c ]'

    Short projects were the reason to leave all agenda files in the
    org-directory. I remove old projects from my file system from time
    to time (their all in my repo anyway). But I still can refer to the
    project plan if neccessary.



*** General note taking

    As quick notes go to files in my org-directory, it's natural to have
    a file tree beneeth it, where I sort stuff away that is of more
    general nature - i.e. not related to special project or
    customer. Refiling is very easy, so it turns out that two or three
    files for such notes are sufficient in my case.

    After a while I file my quick notes away. For that purpos my
    org-directory has a sub-folder (a tree actually). That tree is for
    notes I want to keep locally. I publish those notes locally to HTML
    for browsing. The published content is awailable in our local
    network here. That's why I have that subtree. Particular private
    notes may still live in my org-directory.






* Projects

  Here it comes: I use no special concept for Projects. That's the
  beauty of using emacs and Org-mode:

  They adjust to what you wanna do. Just keep the structure you're
  already working with.

  If you need to create a LaTeX file for your project, just start from
  an Org-file. Org-mode is with you if your an emacs user, so why worry?

  I found, that Org-mode does not really add any extras to my project
  directories. All that it changes is the way that stuff is added. No
  extra software that eats up my memory, makes me wait to start up, or
  drives my crazy.

  Links to source files I work on are sometimes stored next to the TODO
  entries in the agenda files. That way I can go the source file in
  question at a later date just by clicking a link.





May the Emacs Org-mode be with you!






Best wishes

   Sebastian

Re: Re: Orgmode for research information management

[Graham Smith]

Sebastian,

Thanks for this.


> I'd propose to start by partitioning _what_ you actually need to store
> on your disk.
>
> This is what I do:

Actually, I already do something similar to you now as a general PIM,
I was really looking for some tips specifically related to management
of information for a research, but it seems that the point that you
and others are making is that the flexibility of Emacs/orgmode means
you don't particularly need a strong plan.

Graham

Orgmode for research information management

[Shrutarshi Basu]

[-- Attachment #1.1: Type: text/plain, Size: 1086 bytes --]

I just started using orgmode today to organize the papers that I'm
collecting for my own research. The way I work is a bit complicated, but I
like it. I have all my PDFs in a single directory and my notes in a separate
directory. These notes are extensive outlines and notes for each paper.
Corresponding files are named the same in each directory to make it easy to
relate the paper to the notes. I also have a bibtex file with citation info
for each file. I then run a Python script over the two directories and the
bibtex file which generates an org file containing links to the PDF and
notes and some of the bibtex data (authors and publication). This file is my
main interface to all my research info. I use Org-mode tags to do some
simple categorization and also store short notes. If you like, I could give
you my Python script. Some of the file paths are hard coded in, but they're
easy to change. You can also view PDFs directly in Emacs.

Basu

--
Shrutarshi Basu
Computer Science,
Electrical and Computer Engineering,
Lafayette College,
The ByteBaker -- http://bytebaker.com

[-- Attachment #1.2: Type: text/html, Size: 1178 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Orgmode for research information management

[Greg Newman]

[-- Attachment #1.1: Type: text/plain, Size: 1747 bytes --]

Hey Basu.  I'd love to see this python script.  I'm a complete n00b with
latex so i'm trying to get up to speed on it fast; however all my research
on this has led me to believe this is the perfect setup and your description
of your flow is confirming it more.
Thanks,


*Greg Newman*

http://20seven.org

twitter: 20seven

On Wed, Jun 24, 2009 at 5:27 PM, Shrutarshi Basu <shr@basushr.net> wrote:

> I just started using orgmode today to organize the papers that I'm
> collecting for my own research. The way I work is a bit complicated, but I
> like it. I have all my PDFs in a single directory and my notes in a separate
> directory. These notes are extensive outlines and notes for each paper.
> Corresponding files are named the same in each directory to make it easy to
> relate the paper to the notes. I also have a bibtex file with citation info
> for each file. I then run a Python script over the two directories and the
> bibtex file which generates an org file containing links to the PDF and
> notes and some of the bibtex data (authors and publication). This file is my
> main interface to all my research info. I use Org-mode tags to do some
> simple categorization and also store short notes. If you like, I could give
> you my Python script. Some of the file paths are hard coded in, but they're
> easy to change. You can also view PDFs directly in Emacs.
>
> Basu
>
> --
> Shrutarshi Basu
> Computer Science,
> Electrical and Computer Engineering,
> Lafayette College,
> The ByteBaker -- http://bytebaker.com
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
>

[-- Attachment #1.2: Type: text/html, Size: 2653 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Orgmode for research information management

[Alexander]

Hi,

I've recently started using org-mode to organize reading notes and to
write papers, and I'd be interested in this script too. One thing that
I have a little trouble with is connecting these three pieces of
information for each reference: bibtex key, pdf file with actual
article, and org file with notes.

Best,

Alexander

On 6/25/09, Greg Newman <greg@20seven.org> wrote:
> Hey Basu.  I'd love to see this python script.  I'm a complete n00b with
> latex so i'm trying to get up to speed on it fast; however all my research
> on this has led me to believe this is the perfect setup and your description
> of your flow is confirming it more.
> Thanks,
>
>
> *Greg Newman*
>
> http://20seven.org
>
> twitter: 20seven
>
> On Wed, Jun 24, 2009 at 5:27 PM, Shrutarshi Basu <shr@basushr.net> wrote:
>
>> I just started using orgmode today to organize the papers that I'm
>> collecting for my own research. The way I work is a bit complicated, but I
>> like it. I have all my PDFs in a single directory and my notes in a
>> separate
>> directory. These notes are extensive outlines and notes for each paper.
>> Corresponding files are named the same in each directory to make it easy
>> to
>> relate the paper to the notes. I also have a bibtex file with citation
>> info
>> for each file. I then run a Python script over the two directories and the
>> bibtex file which generates an org file containing links to the PDF and
>> notes and some of the bibtex data (authors and publication). This file is
>> my
>> main interface to all my research info. I use Org-mode tags to do some
>> simple categorization and also store short notes. If you like, I could
>> give
>> you my Python script. Some of the file paths are hard coded in, but
>> they're
>> easy to change. You can also view PDFs directly in Emacs.
>>
>> Basu
>>
>> --
>> Shrutarshi Basu
>> Computer Science,
>> Electrical and Computer Engineering,
>> Lafayette College,
>> The ByteBaker -- http://bytebaker.com
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Remember: use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>>
>>
>

Re: Orgmode for research information management

[Shrutarshi Basu]

[-- Attachment #1.1: Type: text/plain, Size: 1752 bytes --]

I've uploaded the script to http://ww2.lafayette.edu/~basus/paperorg
No comments, but it should be fairly self-explanatory. Keeping pdfs, bibtex
and org notes files connected was a problem that I solved with some low-tech
organization. My pdfs, orgs and bibtex keys all have the same format:

<publication><year>-<first author>

That way I can have the script easily pull together the different
information sources because they have the same unique identifier (without
the extension). This approach means that I don't see the paper title or any
other information in the file manager. I basically use Emacs as the
paper-organization interface, which is really nice since I can use tags and
the like to separate out the papers while keeping them in one place if I
need to find them.

I have one such setup for each research project, though in theory I could
just use one large org-file as a complete paper database. But I feel that
would be putting all my eggs in one basket, so that's not something I'd be
comfortable doing.

I would like to get org mode to open the PDF links in an external viewer as
opposed to in Emacs with doc-view as it does now. Any suggestions on how to
do this?

Thanks,
Basu

--
Shrutarshi Basu
Computer Science,
Electrical and Computer Engineering,
Lafayette College,
The ByteBaker -- http://bytebaker.com


On Thu, Jun 25, 2009 at 11:27 AM, Alexander <alejck@gmail.com> wrote:

> Hi,
>
> I've recently started using org-mode to organize reading notes and to
> write papers, and I'd be interested in this script too. One thing that
> I have a little trouble with is connecting these three pieces of
> information for each reference: bibtex key, pdf file with actual
> article, and org file with notes.
>
> Best,
>
> Alexander

[-- Attachment #1.2: Type: text/html, Size: 2295 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Orgmode for research information management

[Bastien]

Shrutarshi Basu <shr@basushr.net> writes:

> I would like to get org mode to open the PDF links in an external viewer as
> opposed to in Emacs with doc-view as it does now. Any suggestions on how to do
> this?

You can customize `org-file-apps' to your needs.

-- 
 Bastien

Re: Orgmode for research information management

[Graham Smith]

Shrutarshi

2009/6/24 Shrutarshi Basu <shr@basushr.net>:
> I just started using orgmode today to organize the papers that I'm
> collecting for my own research. The way I work is a bit complicated, but I
> like it. I have all my PDFs in a single directory and my notes in a separate
> directory.

That sounds really interesting, thanks. And yes I would like to have a
look at the python script, but I may need to come back and ask for
help in using it.

None the less, this is the sort of set up I would like to try and get working.

Graham