Using org-mode for laboratory notes.

Using org-mode for laboratory notes.

[Eric Lubeck]

[-- Attachment #1: Type: text/plain, Size: 1710 bytes --]

Hello Everybody, 

I had a look around the web for awhile, but couldn't find any information pertaining to my particular needs.  I hope somebody here will be able to help me out.

Anyway, I've been looking around for quite a while for the proper system to set-up an electronic laboratory notebook in.  I will be using org-mode to document wet-lab experiments in addition to computational work. 

One of my particular concerns is this:  I'm accustomed to using a chronological laboratory notebook for recording all of my data.  The agenda views in org-mode seem to provide a means to retrieve chronological information out of my outlines, but I would than need to timestamp every single entry in my outline.  Is there a means for doing this?  Currently I am manually typing C-u C-c ! , but it would be helpful to have something automatically configured to timestamp and place the time in a drawer for any entry in a particular file.  

My other question pertains to efficiently representing linked or nested data.  I'd like to record my detailed laboratory protocols in another outline.  As most of my day-to-day work is using these protocols with minimal modifications, I'd like to record in my primary outline a property or hyperlink that points to the primary protocol and suggests that this days experiment "inherits" from the main protocol with given modifications.  It would be really awesome if the protocol tree could than pick up on these distant inheriting protocols and transclude in the dates I have performed this protocol and subsequent modifications from the "lab notebook" section.  Is such a task possible with org-mode, or must I look towards a more traditional database?

Thanks,
Eric Lubeck

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: Using org-mode for laboratory notes.

[Russell Adams]

On Wed, Sep 19, 2012 at 09:15:35AM -0700, Eric Lubeck wrote:
> Hello Everybody,
>
> I had a look around the web for awhile, but couldn't find any
> information pertaining to my particular needs.  I hope somebody here
> will be able to help me out.
>
> Anyway, I've been looking around for quite a while for the proper
> system to set-up an electronic laboratory notebook in.  I will be
> using org-mode to document wet-lab experiments in addition to
> computational work.
>
> One of my particular concerns is this: I'm accustomed to using a
> chronological laboratory notebook for recording all of my data.  The
> agenda views in org-mode seem to provide a means to retrieve
> chronological information out of my outlines, but I would than need
> to timestamp every single entry in my outline.  Is there a means for
> doing this?  Currently I am manually typing C-u C-c ! , but it would
> be helpful to have something automatically configured to timestamp
> and place the time in a drawer for any entry in a particular file.

There's an emacs customization I use which binds F9 to insert NOW as
an inactive date & time stamp.

 ;; Insert immediate timestamp
 (define-key global-map (kbd "<f9>")
   '(lambda () (interactive)
      (when (eq major-mode 'org-mode)
        (org-insert-time-stamp nil t t)
        (insert "\n"))))

I use this in my org files. I outline my projects by major task or
product. Every time I change tasks or return to taking notes I just
hit F9 before I start typing.

Then at any time I can use the agenda in log mode and enable inactive
time stamps to show me a hyperlinked timeline.

> My other question pertains to efficiently representing linked or
> nested data.  I'd like to record my detailed laboratory protocols in
> another outline.  As most of my day-to-day work is using these
> protocols with minimal modifications, I'd like to record in my
> primary outline a property or hyperlink that points to the primary
> protocol and suggests that this days experiment "inherits" from the
> main protocol with given modifications.  It would be really awesome
> if the protocol tree could than pick up on these distant inheriting
> protocols and transclude in the dates I have performed this protocol
> and subsequent modifications from the "lab notebook" section.  Is
> such a task possible with org-mode, or must I look towards a more
> traditional database?

The only word I got here was "database". There is some support for
using property drawers for storing data and summarizing. If you had a
headline which you kept notes under and stored values in a property
drawer you may be able to manipulate it.

Good luck!

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Using org-mode for laboratory notes.

[Tim]

At Wed, 19 Sep 2012 12:08:21 -0500,
Russell Adams wrote:

On Wed, Sep 19, 2012 at 09:15:35AM -0700, Eric Lubeck wrote:
> One of my particular concerns is this: I'm accustomed to using a
> chronological laboratory notebook for recording all of my data.  The
> agenda views in org-mode seem to provide a means to retrieve
> chronological information out of my outlines, but I would than need
> to timestamp every single entry in my outline.  Is there a means for
> doing this?  Currently I am manually typing C-u C-c ! , but it would
> be helpful to have something automatically configured to timestamp
> and place the time in a drawer for any entry in a particular file.

Hello, 

   I have the following setup in order to keep cronological notes.

* In my capture templates I have the following :

  #+begin_src elisp
	("j" "Journal entry" entry
	 (file make-journal-file-name )
	 "* %? :journal:REVIEW: \n %u")
  #+end_src
   and I've defined the function like so :

  #+begin_src elisp 
   (defun make-journal-file-name ()
     (concat "~/Planner/Journal/" (format-time-string "%Y.%m.%d.%a" nil) ".org" )
    )
  #+end_src
  That way, to create a new entry I just hit 'C-c c j'

  which gives me the following :
  *  :journal:REVIEW: 
    [2012-09-20 Thu]

  in the file ~/Planner/Journal/2012.09.20.Thu.org

  Please let me know if any of that is unclear.


  Hope that helps, 

  -T

Re: Using org-mode for laboratory notes.

[Torsten Wagner]

Hey,

this is a nice idea and makes me wonder if it would be possible to use
templates to automatically commit all recent changes in a git repro,
and start a new day by a new fresh working space in git.
git links are already possible. The template could create a link to
the just checked in status in a extra org-file. You would end-up with
a link for each date and file that changed and following this links,
your files would magically be set-back to that particular timestamp.
Furthermore, having the capabilities to see the diff and to see the
history of not only complete files but of single headlines in org-mode
would be then very helpful.

I believe having a more tied integration of git capabilities within
org-mode could really come up with some nice solutions.

Torsten

On 21 September 2012 11:09, Tim <wiskey5alpha@gmail.com> wrote:
> At Wed, 19 Sep 2012 12:08:21 -0500,
> Russell Adams wrote:
>
> On Wed, Sep 19, 2012 at 09:15:35AM -0700, Eric Lubeck wrote:
>> One of my particular concerns is this: I'm accustomed to using a
>> chronological laboratory notebook for recording all of my data.  The
>> agenda views in org-mode seem to provide a means to retrieve
>> chronological information out of my outlines, but I would than need
>> to timestamp every single entry in my outline.  Is there a means for
>> doing this?  Currently I am manually typing C-u C-c ! , but it would
>> be helpful to have something automatically configured to timestamp
>> and place the time in a drawer for any entry in a particular file.
>
> Hello,
>
>    I have the following setup in order to keep cronological notes.
>
> * In my capture templates I have the following :
>
>   #+begin_src elisp
>         ("j" "Journal entry" entry
>          (file make-journal-file-name )
>          "* %? :journal:REVIEW: \n %u")
>   #+end_src
>    and I've defined the function like so :
>
>   #+begin_src elisp
>    (defun make-journal-file-name ()
>      (concat "~/Planner/Journal/" (format-time-string "%Y.%m.%d.%a" nil) ".org" )
>     )
>   #+end_src
>   That way, to create a new entry I just hit 'C-c c j'
>
>   which gives me the following :
>   *  :journal:REVIEW:
>     [2012-09-20 Thu]
>
>   in the file ~/Planner/Journal/2012.09.20.Thu.org
>
>   Please let me know if any of that is unclear.
>
>
>   Hope that helps,
>
>   -T
>

Re: Using org-mode for laboratory notes.

[Tim]

At Fri, 21 Sep 2012 11:20:41 +0900,
Torsten Wagner wrote:
> 
> Hey,
> 
> this is a nice idea and makes me wonder if it would be possible to use
> templates to automatically commit all recent changes in a git repro,
> and start a new day by a new fresh working space in git.
> git links are already possible. The template could create a link to
> the just checked in status in a extra org-file. You would end-up with
> a link for each date and file that changed and following this links,
> your files would magically be set-back to that particular timestamp.
> Furthermore, having the capabilities to see the diff and to see the
> history of not only complete files but of single headlines in org-mode
> would be then very helpful.

I suppose we could use call-process in the function to run the git commands ?


--
-T

Re: Using org-mode for laboratory notes.

[Eric Schulte]

Eric Lubeck <eric.lubeck@gmail.com> writes:

> Hello Everybody, 
>
> I had a look around the web for awhile, but couldn't find any
> information pertaining to my particular needs.  I hope somebody here
> will be able to help me out.
>
> Anyway, I've been looking around for quite a while for the proper
> system to set-up an electronic laboratory notebook in.  I will be
> using org-mode to document wet-lab experiments in addition to
> computational work.
>
> One of my particular concerns is this: I'm accustomed to using a
> chronological laboratory notebook for recording all of my data.  The
> agenda views in org-mode seem to provide a means to retrieve
> chronological information out of my outlines, but I would than need to
> timestamp every single entry in my outline.  Is there a means for
> doing this?  Currently I am manually typing C-u C-c ! , but it would
> be helpful to have something automatically configured to timestamp and
> place the time in a drawer for any entry in a particular file.
>

I believe such automated functionality may exist (although I don't use
it personally).  Take a look at this portion of the manual [1].

>
> My other question pertains to efficiently representing linked or
> nested data.  I'd like to record my detailed laboratory protocols in
> another outline.  As most of my day-to-day work is using these
> protocols with minimal modifications, I'd like to record in my primary
> outline a property or hyperlink that points to the primary protocol
> and suggests that this days experiment "inherits" from the main
> protocol with given modifications.  It would be really awesome if the
> protocol tree could than pick up on these distant inheriting protocols
> and transclude in the dates I have performed this protocol and
> subsequent modifications from the "lab notebook" section.  Is such a
> task possible with org-mode, or must I look towards a more traditional
> database?
>

If I read this paragraph correctly, you want to be able to collect the
dates from every entry in which a particular protocol was used.  If you
assign each protocol a short tag name, then you could tag each daily
experiment with the date and the protocol used.  Armed with your tagged
headlines Org-mode should provide everything required to search for and
collect your experiments based on tags, and to do things like collect
the dates on which each tag was used.  Look at the tag and searching
sections of the Org-mode manual [2], and maybe look at the org-collector
in the contrib portion of Worg [3].

>
> Thanks,
> Eric Lubeck


Footnotes: 
[1]  http://orgmode.org/manual/Clocking-work-time.html

[2]  http://orgmode.org/manual/Tags.html

[3]  http://orgmode.org/worg/org-contrib/org-collector.html

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: Using org-mode for laboratory notes.

[Torsten Wagner]

Hi,

one way which works rather differently is the combination of git and org-mode.
You could write your protocols in separate org-files and link to them
in your records.
org allows (at least it did a while ago) to link not only to a file
but also to a specific version of a file.
You could do small modification in the protocol-files as you need them
and check them into the git system.
Link to them and you will see the version you used for exactly this experiments.

Actually using something like git and a git sensitive link is
important if you might plan link to a lot of external files. Imaging
you overwrite a file by accident or because you can't remember you
referred to the original file already. A normal link would quietly
point to the new file and would not be in-sync with anything you
mentioned in your org-file.

Other benefits are gits diff, merge and change-recording capabilities.
If you set-up the git repro with entire lab-book on a server (a PC
reachable from all your other devices) you could easily add data from
within the lab, go to your office to add more data and at a certain
point merge all this together. Both PCs could work offline and only
need to be online for check-in and check-out new data.
Another benefit of combining org-mode and git... you can tag certain
versions of your lab book. E.g. tag them whenever you write a paper
and make a notice in org-mode. This enables you to get back to all the
measurement and reps. data evaluation results as you found them during
writing your paper, even years and many many changes later (e.g. you
might improved your data analysis method over time but for the paper
you still want to see the old stupid way how you dealt with the data).

Recently we got an org-file sensitive git-module, which makes merging
org-files much more nice.
Check here:

http://orgmode.org/worg/org-contrib/org-git-link.html
http://orgmode.org/worg/org-contrib/gsoc2012/student-projects/git-merge-tool/index.html

Albeit I have to say I like to do all kind of stuff in org-mode, I
faced problems using only org-mode as lab-notebook. Sometimes things
in a lab are to numerous and to verbose to type them all in as they
happen. Sometimes a little sketch, some quick scribbled note, etc.
contains the real important data.... sentences like "Damn, Joe slammed
the door AGAIN", during an AFM measurement. Or in your case "Uhh..
what are the funny little flakes in my buffer solutions".... I think
sometimes a keyboard still filters to much and hence org-mode might
not really contain all the necessary info.
As a summary:
org-mode as a lab-book will work fine if you are strict in using it
and force yourself to be verbose enough.

Hope thats helps

Totti

CC. There might be some legal issue with real lab books and electronic
once. Back in the good old time where scientists didn't publish each
and every result and where it could take many many months until some
discovery reached the other side of the planet, the laboratory books
where the legal evidence of the original work. If someone made a wrong
claim or someone accused someone else of falsify reporting, the
laboratory books where used to proof those claims. That is the reason,
you find e.g. nicely archived laboratory books of all the great
scientists of the Bell laboratories. Not sure how much this is still
relevant today.


On 20 September 2012 03:49, Eric Schulte <eric.schulte@gmx.com> wrote:
> Eric Lubeck <eric.lubeck@gmail.com> writes:
>
>> Hello Everybody,
>>
>> I had a look around the web for awhile, but couldn't find any
>> information pertaining to my particular needs.  I hope somebody here
>> will be able to help me out.
>>
>> Anyway, I've been looking around for quite a while for the proper
>> system to set-up an electronic laboratory notebook in.  I will be
>> using org-mode to document wet-lab experiments in addition to
>> computational work.
>>
>> One of my particular concerns is this: I'm accustomed to using a
>> chronological laboratory notebook for recording all of my data.  The
>> agenda views in org-mode seem to provide a means to retrieve
>> chronological information out of my outlines, but I would than need to
>> timestamp every single entry in my outline.  Is there a means for
>> doing this?  Currently I am manually typing C-u C-c ! , but it would
>> be helpful to have something automatically configured to timestamp and
>> place the time in a drawer for any entry in a particular file.
>>
>
> I believe such automated functionality may exist (although I don't use
> it personally).  Take a look at this portion of the manual [1].
>
>>
>> My other question pertains to efficiently representing linked or
>> nested data.  I'd like to record my detailed laboratory protocols in
>> another outline.  As most of my day-to-day work is using these
>> protocols with minimal modifications, I'd like to record in my primary
>> outline a property or hyperlink that points to the primary protocol
>> and suggests that this days experiment "inherits" from the main
>> protocol with given modifications.  It would be really awesome if the
>> protocol tree could than pick up on these distant inheriting protocols
>> and transclude in the dates I have performed this protocol and
>> subsequent modifications from the "lab notebook" section.  Is such a
>> task possible with org-mode, or must I look towards a more traditional
>> database?
>>
>
> If I read this paragraph correctly, you want to be able to collect the
> dates from every entry in which a particular protocol was used.  If you
> assign each protocol a short tag name, then you could tag each daily
> experiment with the date and the protocol used.  Armed with your tagged
> headlines Org-mode should provide everything required to search for and
> collect your experiments based on tags, and to do things like collect
> the dates on which each tag was used.  Look at the tag and searching
> sections of the Org-mode manual [2], and maybe look at the org-collector
> in the contrib portion of Worg [3].
>
>>
>> Thanks,
>> Eric Lubeck
>
>
> Footnotes:
> [1]  http://orgmode.org/manual/Clocking-work-time.html
>
> [2]  http://orgmode.org/manual/Tags.html
>
> [3]  http://orgmode.org/worg/org-contrib/org-collector.html
>
> --
> Eric Schulte
> http://cs.unm.edu/~eschulte
>

Re: Using org-mode for laboratory notes.

[Eric Lubeck]

[-- Attachment #1: Type: text/plain, Size: 9321 bytes --]

Hey, 

Thanks for the good idea.  I'll have to look into figuring that system out.

On the broader point of organizing the notebook, I am still having a bit of a dilemma coming up with an effective system.  My first thought was to just place all my work in a dated hierarchy, such as with org-datetree.  This would be simple and mirror a conventional notebook, but would loose a lot of the logical hierarchy possible with digital tools.  

On the other hand, organizing all my experiments as a non-linear outline is getting a bit messy.  I found myself navigating around headlines everyday searching for items I need to schedule for the next day.  As my notebook gets bigger, this system will probably get very inefficient.  If i properly tag and schedule my tasks for the day this should be less of a problem, but i still foresee potential chaos if I get lazy.  In addition to tasks I intend to record other observations in the notebook that may not be associated with a recent task, yet are important for me planning future experiments.  Without proper timestamps I could loose these observations over time.

This is where I came up with the idea of tagging all of my headlines with their entry date and timestamp.  Potentially such a system would enable me to view the logical hierarchy of an experiment, but also view my work in the conventional linear order. 

Anyone have any other ideas for reasonable systems?

I'm also a bit confused about the proper way to implement such a system.  I imagine I could hack together some auto-timestamp property, but than it would only apply to headlines, not to my nested observations in list form.  For this reason I have little used lists at all in org-mode, as it seems that any data that could potentially be nested, such as with different tasks or properties, must be converted back to headline form before it can be annotated.  Am I missing something?

Also, I have a general question about nesting headings demonstrated by the below example.

* Today's Experiment       :EXPERIMENT:
** Do today's Experiment :RATIONALE:
** Data link :DATA:
** Experiments are sad.  :DISCUSSION:
** Repeat, but change X <Tomorrow>  :FOLLOW-UP:
* Tomorrow's Experiment  :EXPERIMENT:
** <Yesterday's Experiment> Failed :RATIONALE:
*** Determine if X was the cause

In the above case I have two options, to either continually nest all follow-up experiments, or rely on a network of links to get me back to the data that led to the follow-up experiments.  Anybody have any advice on pursing either option?

Thanks for the help,
Eric Lubeck


On Wednesday, September 19, 2012 at 6:44 PM, Torsten Wagner wrote:

> Hi,
> 
> one way which works rather differently is the combination of git and org-mode.
> You could write your protocols in separate org-files and link to them
> in your records.
> org allows (at least it did a while ago) to link not only to a file
> but also to a specific version of a file.
> You could do small modification in the protocol-files as you need them
> and check them into the git system.
> Link to them and you will see the version you used for exactly this experiments.
> 
> Actually using something like git and a git sensitive link is
> important if you might plan link to a lot of external files. Imaging
> you overwrite a file by accident or because you can't remember you
> referred to the original file already. A normal link would quietly
> point to the new file and would not be in-sync with anything you
> mentioned in your org-file.
> 
> Other benefits are gits diff, merge and change-recording capabilities.
> If you set-up the git repro with entire lab-book on a server (a PC
> reachable from all your other devices) you could easily add data from
> within the lab, go to your office to add more data and at a certain
> point merge all this together. Both PCs could work offline and only
> need to be online for check-in and check-out new data.
> Another benefit of combining org-mode and git... you can tag certain
> versions of your lab book. E.g. tag them whenever you write a paper
> and make a notice in org-mode. This enables you to get back to all the
> measurement and reps. data evaluation results as you found them during
> writing your paper, even years and many many changes later (e.g. you
> might improved your data analysis method over time but for the paper
> you still want to see the old stupid way how you dealt with the data).
> 
> Recently we got an org-file sensitive git-module, which makes merging
> org-files much more nice.
> Check here:
> 
> http://orgmode.org/worg/org-contrib/org-git-link.html
> http://orgmode.org/worg/org-contrib/gsoc2012/student-projects/git-merge-tool/index.html
> 
> Albeit I have to say I like to do all kind of stuff in org-mode, I
> faced problems using only org-mode as lab-notebook. Sometimes things
> in a lab are to numerous and to verbose to type them all in as they
> happen. Sometimes a little sketch, some quick scribbled note, etc.
> contains the real important data.... sentences like "Damn, Joe slammed
> the door AGAIN", during an AFM measurement. Or in your case "Uhh..
> what are the funny little flakes in my buffer solutions".... I think
> sometimes a keyboard still filters to much and hence org-mode might
> not really contain all the necessary info.
> As a summary:
> org-mode as a lab-book will work fine if you are strict in using it
> and force yourself to be verbose enough.
> 
> Hope thats helps
> 
> Totti
> 
> CC. There might be some legal issue with real lab books and electronic
> once. Back in the good old time where scientists didn't publish each
> and every result and where it could take many many months until some
> discovery reached the other side of the planet, the laboratory books
> where the legal evidence of the original work. If someone made a wrong
> claim or someone accused someone else of falsify reporting, the
> laboratory books where used to proof those claims. That is the reason,
> you find e.g. nicely archived laboratory books of all the great
> scientists of the Bell laboratories. Not sure how much this is still
> relevant today.
> 
> 
> On 20 September 2012 03:49, Eric Schulte <eric.schulte@gmx.com (mailto:eric.schulte@gmx.com)> wrote:
> > Eric Lubeck <eric.lubeck@gmail.com (mailto:eric.lubeck@gmail.com)> writes:
> > 
> > > Hello Everybody,
> > > 
> > > I had a look around the web for awhile, but couldn't find any
> > > information pertaining to my particular needs. I hope somebody here
> > > will be able to help me out.
> > > 
> > > Anyway, I've been looking around for quite a while for the proper
> > > system to set-up an electronic laboratory notebook in. I will be
> > > using org-mode to document wet-lab experiments in addition to
> > > computational work.
> > > 
> > > One of my particular concerns is this: I'm accustomed to using a
> > > chronological laboratory notebook for recording all of my data. The
> > > agenda views in org-mode seem to provide a means to retrieve
> > > chronological information out of my outlines, but I would than need to
> > > timestamp every single entry in my outline. Is there a means for
> > > doing this? Currently I am manually typing C-u C-c ! , but it would
> > > be helpful to have something automatically configured to timestamp and
> > > place the time in a drawer for any entry in a particular file.
> > > 
> > 
> > 
> > I believe such automated functionality may exist (although I don't use
> > it personally). Take a look at this portion of the manual [1].
> > 
> > > 
> > > My other question pertains to efficiently representing linked or
> > > nested data. I'd like to record my detailed laboratory protocols in
> > > another outline. As most of my day-to-day work is using these
> > > protocols with minimal modifications, I'd like to record in my primary
> > > outline a property or hyperlink that points to the primary protocol
> > > and suggests that this days experiment "inherits" from the main
> > > protocol with given modifications. It would be really awesome if the
> > > protocol tree could than pick up on these distant inheriting protocols
> > > and transclude in the dates I have performed this protocol and
> > > subsequent modifications from the "lab notebook" section. Is such a
> > > task possible with org-mode, or must I look towards a more traditional
> > > database?
> > > 
> > 
> > 
> > If I read this paragraph correctly, you want to be able to collect the
> > dates from every entry in which a particular protocol was used. If you
> > assign each protocol a short tag name, then you could tag each daily
> > experiment with the date and the protocol used. Armed with your tagged
> > headlines Org-mode should provide everything required to search for and
> > collect your experiments based on tags, and to do things like collect
> > the dates on which each tag was used. Look at the tag and searching
> > sections of the Org-mode manual [2], and maybe look at the org-collector
> > in the contrib portion of Worg [3].
> > 
> > > 
> > > Thanks,
> > > Eric Lubeck
> > > 
> > 
> > 
> > 
> > Footnotes:
> > [1] http://orgmode.org/manual/Clocking-work-time.html
> > 
> > [2] http://orgmode.org/manual/Tags.html
> > 
> > [3] http://orgmode.org/worg/org-contrib/org-collector.html
> > 
> > --
> > Eric Schulte
> > http://cs.unm.edu/~eschulte
> > 
> 
> 
> 



[-- Attachment #2: Type: text/html, Size: 11868 bytes --]

Re: Using org-mode for laboratory notes.

[John Hendy]

On Wed, Sep 19, 2012 at 10:18 PM, Eric Lubeck <eric.lubeck@gmail.com> wrote:
> Hey,
>
> Thanks for the good idea.  I'll have to look into figuring that system out.
>
> On the broader point of organizing the notebook, I am still having a bit of
> a dilemma coming up with an effective system.  My first thought was to just
> place all my work in a dated hierarchy, such as with org-datetree.  This
> would be simple and mirror a conventional notebook, but would loose a lot of
> the logical hierarchy possible with digital tools.

First off, this is great stuff. This type of discussion happens all
the time, at least in part due to me :) At least I think these
partially get at some of the struggles you're having:

- http://lists.gnu.org/archive/html/emacs-orgmode/2011-07/msg01173.html
- http://osdir.com/ml/emacs-orgmode-gnu/2012-01/msg00681.html

>
> On the other hand, organizing all my experiments as a non-linear outline is
> getting a bit messy.  I found myself navigating around headlines everyday
> searching for items I need to schedule for the next day.  As my notebook
> gets bigger, this system will probably get very inefficient.  If i properly
> tag and schedule my tasks for the day this should be less of a problem, but
> i still foresee potential chaos if I get lazy.  In addition to tasks I
> intend to record other observations in the notebook that may not be
> associated with a recent task, yet are important for me planning future
> experiments.  Without proper timestamps I could loose these observations
> over time.

Yup. Same with me. I'm in R&D for a large-ish (80k employees
globally), worldwide science and technology place. Technical notebooks
are a big deal. I do everything in Org, then print, chop the margins,
and permanently double tape in bound books. I witness them across the
paper edge and have them witnessed by my lab mate. When the book is
full, it gets sent to the vault in who knows what building.

I have current projects. Then I have odds and end ideas that might or
might not be relevant. But my project specific work is often much more
broadly applicable to something else. For example, I might be trying
to create a particular feature because I'm assigned to do so on a
project... but as I'm finding that solution, I think it would be
really cool down the road for something else.

I currently organize by project and then archive when it's done... but
it'd be nice to easily call up these broader ideas later. I know I can
search with agenda and then `v a` to add in archives, but thing still
feel a bit weird and the whole heirarchical or non-linear thing really
resonates with me.

Prior to Org-mode, I was using TiddlyWiki, which is actually quite
neat. I was able to do everything in daily entries, but a plugin
allowed me to mark each section as pertaining to a particular project.
I could view things completely along a timeline by date or filter down
to just a project. And I could add whatever other tags so that I could
call up any entries with that tag.

>
> This is where I came up with the idea of tagging all of my headlines with
> their entry date and timestamp.  Potentially such a system would enable me
> to view the logical hierarchy of an experiment, but also view my work in the
> conventional linear order.
>
> Anyone have any other ideas for reasonable systems?

Per my request, Bastien awesomely added the ability to change sparse
tree searching timestamp types. Thus, inactive stamps can be filtered
out, which is great. Search by a tag and timestamps (I think combining
is possible) and you kind of have your solution.

I would think out your tags ahead of time. The problem, is that even
these get tough to manage. I can't forecast ahead of time what to call
a tag. If it has to do with laser processing of a material, do I tag
it :laser: or if the laser processing is for a particular product
idea, do I tag it with the product idea? How many product ideas are
reasonable? And would the tag be :cool-thing-some-could-use-to-x:?
Sometimes, anything much less specific really won't serve me long
term!

>
> I'm also a bit confused about the proper way to implement such a system.  I
> imagine I could hack together some auto-timestamp property, but than it
> would only apply to headlines, not to my nested observations in list form.
> For this reason I have little used lists at all in org-mode, as it seems
> that any data that could potentially be nested, such as with different tasks
> or properties, must be converted back to headline form before it can be
> annotated.  Am I missing something?
>

Nope, you're not missing anything. Headlines are where it's at. You
might check out this thread where some nifty capture templates were
presented. Completely changed how I use Org:
- http://osdir.com/ml/emacs-orgmode-gnu/2012-08/msg00396.html

Instead of entering things manually, this makes it *far* easier to
just use a quick capture template. As with some other "power users"
here, I've found myself shifting more toward using capture to enter
and agenda/sparse trees to view/access. This way, you don't really
even have to deal much with the org files themselves, which can get a
bit unwieldy when you're filing all kinds of bits and pieces.

I used to worry about hierarchies quite a bit. Obsess even. Some of
these advances have really helped that. Just cram it in there
somewhere with the date stamp and some reasonably helpful tags and I
know I can find it with enough `C-a s` agenda searches for various
words. I just have to put in words I think my future self will recall
:)

> Also, I have a general question about nesting headings demonstrated by the
> below example.
>
> * Today's Experiment       :EXPERIMENT:
> ** Do today's Experiment :RATIONALE:
> ** Data link :DATA:
> ** Experiments are sad.  :DISCUSSION:
> ** Repeat, but change X <Tomorrow>  :FOLLOW-UP:
> * Tomorrow's Experiment  :EXPERIMENT:
> ** <Yesterday's Experiment> Failed :RATIONALE:
> *** Determine if X was the cause
>

Not a bad idea. Perhaps consider using properties for some things
instead? I know there's some neat properties features, though I have
to admit I'm not using properties really at all. Only to log todo
state change notes and logging time.


> In the above case I have two options, to either continually nest all
> follow-up experiments, or rely on a network of links to get me back to the
> data that led to the follow-up experiments.  Anybody have any advice on
> pursing either option?
>

My ending comment is that I've not found it that great to use links. I
use the `C-c l` shortcut to store a link and then `C-c C-l` to insert
it, but I don't like it that much. I also would find it awesome to
extract all linked headlines somehow. In my case, I might try some
experiment and then not be able to follow up and try again for days or
weeks. I'm always torn on whether to do something like:
-----
* Idea
** Experiment 1
[timestamp]

** Experiment 2
[timestamp]
-----

Or use my typical "logbook" structure:
-----
* 2012 September
** Experiment 1
[timestamp]

** Experiment 2
[timestamp]

Continuation from [[Experiment 1]]
-----

The second seems more natural (just write it down when it happened),
but then I have data that's separated (the above assumes it happened
the same month... but what if I remember something from months or
years ago and want to make sure my continuation refers back to the
original experiment/idea?

Quick and easy ways to find a headline and link to it like a wiki
would be *fantastic*. I could probably get into this with the
link-to-unique-ids thingy, but just haven't due into it.


I love these sorts of dialogs. They fill some sort of deep need to
continually optimize and go through drastic re-arrangements of my
files instead of doing real work.


Best regards,
John

Re: Using org-mode for laboratory notes.

[Nick Dokos]

Eric Lubeck <eric.lubeck@gmail.com> wrote:


> On the broader point of organizing the notebook, I am still having a
> bit of a dilemma coming up with an effective system.  My first thought
> was to just place all my work in a dated hierarchy, such as with
> org-datetree.  This would be simple and mirror a conventional
> notebook, but would loose a lot of the logical hierarchy possible with
> digital tools.

This is off-topic for the group but it may be of some interest to the
participants of this thread, if only as a comparison point:

   http://packages.python.org/Sumatra/

Nick

Re: Using org-mode for laboratory notes.

[John Hendy]

On Wed, Sep 19, 2012 at 8:44 PM, Torsten Wagner
<torsten.wagner@gmail.com> wrote:
> Hi,
>
> one way which works rather differently is the combination of git and org-mode.
> You could write your protocols in separate org-files and link to them
> in your records.
> org allows (at least it did a while ago) to link not only to a file
> but also to a specific version of a file.
> You could do small modification in the protocol-files as you need them
> and check them into the git system.
> Link to them and you will see the version you used for exactly this experiments.
>
> Actually using something like git and a git sensitive link is
> important if you might plan link to a lot of external files. Imaging
> you overwrite a file by accident or because you can't remember you
> referred to the original file already. A normal link would quietly
> point to the new file and would not be in-sync with anything you
> mentioned in your org-file.
>
> Other benefits are gits diff, merge and change-recording capabilities.
> If you set-up the git repro with entire lab-book on a server (a PC
> reachable from all your other devices) you could easily add data from
> within the lab, go to your office to add more data and at a certain
> point merge all this together. Both PCs could work offline and only
> need to be online for check-in and check-out new data.
> Another benefit of combining org-mode and git... you can tag certain
> versions of your lab book. E.g. tag them whenever you write a paper
> and make a notice in org-mode. This enables you to get back to all the
> measurement and reps. data evaluation results as you found them during
> writing your paper, even years and many many changes later (e.g. you
> might improved your data analysis method over time but for the paper
> you still want to see the old stupid way how you dealt with the data).
>
> Recently we got an org-file sensitive git-module, which makes merging
> org-files much more nice.
> Check here:
>
> http://orgmode.org/worg/org-contrib/org-git-link.html
> http://orgmode.org/worg/org-contrib/gsoc2012/student-projects/git-merge-tool/index.html

Thanks for re-bringing this up. The video demo of org-merge was
*amazing.* I should really try that. Git scares me a bit, but it seems
amazingly powerful. I really like the idea of being able to have tons
of versions to look back to, especially if I start rearranging and
wonder where I put some tidbit. I've done a few org file overhauls
which have both helped and hurt for this reason. I know something
existed, but then can't find it.

>
> Albeit I have to say I like to do all kind of stuff in org-mode, I
> faced problems using only org-mode as lab-notebook. Sometimes things
> in a lab are to numerous and to verbose to type them all in as they
> happen. Sometimes a little sketch, some quick scribbled note, etc.
> contains the real important data.... sentences like "Damn, Joe slammed
> the door AGAIN", during an AFM measurement. Or in your case "Uhh..
> what are the funny little flakes in my buffer solutions".... I think
> sometimes a keyboard still filters to much and hence org-mode might
> not really contain all the necessary info.

True, though one could document via voice recorder and link to the audio file.

I also sketch on a whiteboard, take a pic with my tablet, and then put
the file in a folder called "ip-pics" which serves to hold a huge
repository of any experiment, equipment, or sketch pics used for my
intellectual property notebook. I'll trade having to do a bit of file
and device maneuvering (taking pictures/sketch on x, getting it into
the right directory for Org to find) for all of the other stuff Org
provides.

In other words, imagine gaining the ability to sketch on pen and paper
and losing the ability to
- easily rearrange
- tag
- log time
- create awesome exports with minor modifications to notes vs. typing
from handwriting
- the list goes literally on, and on, and on

> As a summary:
> org-mode as a lab-book will work fine if you are strict in using it
> and force yourself to be verbose enough.

Yup. I think writing more than you think you do and trying to
anticipate one's future self is huge. When I file contacts, for
example, I try to think of any possible words I might use. For a
machining shop, things like "machining, machin, mill, lathe, molding."

>
> Hope thats helps
>
> Totti
>
> CC. There might be some legal issue with real lab books and electronic
> once. Back in the good old time where scientists didn't publish each
> and every result and where it could take many many months until some
> discovery reached the other side of the planet, the laboratory books
> where the legal evidence of the original work. If someone made a wrong
> claim or someone accused someone else of falsify reporting, the
> laboratory books where used to proof those claims. That is the reason,
> you find e.g. nicely archived laboratory books of all the great
> scientists of the Bell laboratories. Not sure how much this is still
> relevant today.
>

The large R&D company I work at is currently experimenting with a few
e-notebook vendors. They still serve much of the same purpose.
"Filing" an entry electronically timestamps it. I'm sure there could
be issues with fraud, but paper notes can have the same issues. After
all, I write my own dates on my own notes. This is the point of a
witness, and the witness signing date is what really matters, not when
I enter it. We're encouraged to get things witnessed asap.

The US is undergoing the move from first to invent to first to file,
so most of this will be irrelevant within the next 1-2 years. The old
practice was that one needed to prove first inventorship. Now it wil
just matter who gets to the patent office first, so long as they've
reduced the invention to practice and have actually been using it.


>