Re: Using org-mode for laboratory notes.

[John Hendy <jw.hendy@gmail.com>]

On Wed, Sep 19, 2012 at 10:18 PM, Eric Lubeck <eric.lubeck@gmail.com> wrote:
> Hey,
>
> Thanks for the good idea.  I'll have to look into figuring that system out.
>
> On the broader point of organizing the notebook, I am still having a bit of
> a dilemma coming up with an effective system.  My first thought was to just
> place all my work in a dated hierarchy, such as with org-datetree.  This
> would be simple and mirror a conventional notebook, but would loose a lot of
> the logical hierarchy possible with digital tools.

First off, this is great stuff. This type of discussion happens all
the time, at least in part due to me :) At least I think these
partially get at some of the struggles you're having:

- http://lists.gnu.org/archive/html/emacs-orgmode/2011-07/msg01173.html
- http://osdir.com/ml/emacs-orgmode-gnu/2012-01/msg00681.html

>
> On the other hand, organizing all my experiments as a non-linear outline is
> getting a bit messy.  I found myself navigating around headlines everyday
> searching for items I need to schedule for the next day.  As my notebook
> gets bigger, this system will probably get very inefficient.  If i properly
> tag and schedule my tasks for the day this should be less of a problem, but
> i still foresee potential chaos if I get lazy.  In addition to tasks I
> intend to record other observations in the notebook that may not be
> associated with a recent task, yet are important for me planning future
> experiments.  Without proper timestamps I could loose these observations
> over time.

Yup. Same with me. I'm in R&D for a large-ish (80k employees
globally), worldwide science and technology place. Technical notebooks
are a big deal. I do everything in Org, then print, chop the margins,
and permanently double tape in bound books. I witness them across the
paper edge and have them witnessed by my lab mate. When the book is
full, it gets sent to the vault in who knows what building.

I have current projects. Then I have odds and end ideas that might or
might not be relevant. But my project specific work is often much more
broadly applicable to something else. For example, I might be trying
to create a particular feature because I'm assigned to do so on a
project... but as I'm finding that solution, I think it would be
really cool down the road for something else.

I currently organize by project and then archive when it's done... but
it'd be nice to easily call up these broader ideas later. I know I can
search with agenda and then `v a` to add in archives, but thing still
feel a bit weird and the whole heirarchical or non-linear thing really
resonates with me.

Prior to Org-mode, I was using TiddlyWiki, which is actually quite
neat. I was able to do everything in daily entries, but a plugin
allowed me to mark each section as pertaining to a particular project.
I could view things completely along a timeline by date or filter down
to just a project. And I could add whatever other tags so that I could
call up any entries with that tag.

>
> This is where I came up with the idea of tagging all of my headlines with
> their entry date and timestamp.  Potentially such a system would enable me
> to view the logical hierarchy of an experiment, but also view my work in the
> conventional linear order.
>
> Anyone have any other ideas for reasonable systems?

Per my request, Bastien awesomely added the ability to change sparse
tree searching timestamp types. Thus, inactive stamps can be filtered
out, which is great. Search by a tag and timestamps (I think combining
is possible) and you kind of have your solution.

I would think out your tags ahead of time. The problem, is that even
these get tough to manage. I can't forecast ahead of time what to call
a tag. If it has to do with laser processing of a material, do I tag
it :laser: or if the laser processing is for a particular product
idea, do I tag it with the product idea? How many product ideas are
reasonable? And would the tag be :cool-thing-some-could-use-to-x:?
Sometimes, anything much less specific really won't serve me long
term!

>
> I'm also a bit confused about the proper way to implement such a system.  I
> imagine I could hack together some auto-timestamp property, but than it
> would only apply to headlines, not to my nested observations in list form.
> For this reason I have little used lists at all in org-mode, as it seems
> that any data that could potentially be nested, such as with different tasks
> or properties, must be converted back to headline form before it can be
> annotated.  Am I missing something?
>

Nope, you're not missing anything. Headlines are where it's at. You
might check out this thread where some nifty capture templates were
presented. Completely changed how I use Org:
- http://osdir.com/ml/emacs-orgmode-gnu/2012-08/msg00396.html

Instead of entering things manually, this makes it *far* easier to
just use a quick capture template. As with some other "power users"
here, I've found myself shifting more toward using capture to enter
and agenda/sparse trees to view/access. This way, you don't really
even have to deal much with the org files themselves, which can get a
bit unwieldy when you're filing all kinds of bits and pieces.

I used to worry about hierarchies quite a bit. Obsess even. Some of
these advances have really helped that. Just cram it in there
somewhere with the date stamp and some reasonably helpful tags and I
know I can find it with enough `C-a s` agenda searches for various
words. I just have to put in words I think my future self will recall
:)

> Also, I have a general question about nesting headings demonstrated by the
> below example.
>
> * Today's Experiment       :EXPERIMENT:
> ** Do today's Experiment :RATIONALE:
> ** Data link :DATA:
> ** Experiments are sad.  :DISCUSSION:
> ** Repeat, but change X <Tomorrow>  :FOLLOW-UP:
> * Tomorrow's Experiment  :EXPERIMENT:
> ** <Yesterday's Experiment> Failed :RATIONALE:
> *** Determine if X was the cause
>

Not a bad idea. Perhaps consider using properties for some things
instead? I know there's some neat properties features, though I have
to admit I'm not using properties really at all. Only to log todo
state change notes and logging time.


> In the above case I have two options, to either continually nest all
> follow-up experiments, or rely on a network of links to get me back to the
> data that led to the follow-up experiments.  Anybody have any advice on
> pursing either option?
>

My ending comment is that I've not found it that great to use links. I
use the `C-c l` shortcut to store a link and then `C-c C-l` to insert
it, but I don't like it that much. I also would find it awesome to
extract all linked headlines somehow. In my case, I might try some
experiment and then not be able to follow up and try again for days or
weeks. I'm always torn on whether to do something like:
-----
* Idea
** Experiment 1
[timestamp]

** Experiment 2
[timestamp]
-----

Or use my typical "logbook" structure:
-----
* 2012 September
** Experiment 1
[timestamp]

** Experiment 2
[timestamp]

Continuation from [[Experiment 1]]
-----

The second seems more natural (just write it down when it happened),
but then I have data that's separated (the above assumes it happened
the same month... but what if I remember something from months or
years ago and want to make sure my continuation refers back to the
original experiment/idea?

Quick and easy ways to find a headline and link to it like a wiki
would be *fantastic*. I could probably get into this with the
link-to-unique-ids thingy, but just haven't due into it.


I love these sorts of dialogs. They fill some sort of deep need to
continually optimize and go through drastic re-arrangements of my
files instead of doing real work.


Best regards,
John