Merging .org files

Merging .org files

[Pere Quintana Seguí]

[-- Attachment #1.1: Type: text/plain, Size: 831 bytes --]

Hello,

In my current system, I have about 200 .org files. With this number of
files, building the agenda is very slow in my home computer and, also,
makes the performance of MobileOrg quite poor.

So, I would like to join files, to increase the system's performance.

Is there any script to cleanly join files, by transforming the title of
the file in a first level heading (*) and adding an star to all other
headings of the file?

I think this is the cleaner way to join files.

Thank You,

Pere

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Merging .org files

[Juan Pechiar]

Hola Pere,

I'd suggest creating a master document with #includes of all your
Orgmode files, and then export this master document as org-mode.

http://orgmode.org/manual/Include-files.html#Include-files

There is a :minlevel attribute to automatically demote included
org-mode files.

Saludos,
.j.

On Sun, Jan 30, 2011 at 12:39:35PM +0100, Pere Quintana Seguí wrote:
> In my current system, I have about 200 .org files. With this number of
> files, building the agenda is very slow in my home computer and, also,
> makes the performance of MobileOrg quite poor.
>
> So, I would like to join files, to increase the system's performance.
>
> Is there any script to cleanly join files, by transforming the title of
> the file in a first level heading (*) and adding an star to all other
> headings of the file?
>
> I think this is the cleaner way to join files.

Re: Merging .org files

[Michael Welle]

Hello,

Pere Quintana Seguí <pquintana@obsebre.es> writes:

> Hello,
>
> In my current system, I have about 200 .org files. With this number of
> files, building the agenda is very slow in my home computer and, also,
> makes the performance of MobileOrg quite poor.
>
> So, I would like to join files, to increase the system's performance.
don't the org files reflect another structure, for instance projects
you are working on, customers you are dealing with etc? 


> Is there any script to cleanly join files, by transforming the title of
> the file in a first level heading (*) and adding an star to all other
> headings of the file?
If you don't follow the advice of Juan, maybe you want only a few
files merged into one file each, you can hack a Perl script or something
similar. Finding regexp that match the headings isn't that complicated.

VG
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

Re: Merging .org files

[Matt Lundin]

Pere Quintana Seguí <pquintana@obsebre.es> writes:

> In my current system, I have about 200 .org files. With this number of
> files, building the agenda is very slow in my home computer and, also,
> makes the performance of MobileOrg quite poor.
>
> So, I would like to join files, to increase the system's performance.

FWIW, I've found the opposite to be true. Splitting up larger files and
reducing the depth of the hierarchy marginally improves agenda
performance (~ 0.2 seconds acc. to elp). But in my case, I split 10
files into 40, so I have no idea what 200 files would do. Perhaps
there's a sweet spot somewhere? I also haven't used mobile org.

In my experience, archiving old items and reducing the number of active
todos is the best way to keep the agenda snappy.

> Is there any script to cleanly join files, by transforming the title of
> the file in a first level heading (*) and adding an star to all other
> headings of the file?

Perl is always handy for this type of thing:

--8<---------------cut here---------------start------------->8---
#!/usr/bin/perl
use strict;
use warnings;

my $mergefile = "/tmp/merged.org";

open NEWFILE, ">", $mergefile 
  or die "Can't open $mergefile: $!";

while (<>) {
  s/^(\*+)/*$1/;
  s/^#\+TITLE:\s+(.*)$/* $1/;
  s/^(#\+\w+:.*)$/: $1/;
  print NEWFILE "$_";
}

close NEWFILE;
--8<---------------cut here---------------end--------------->8---

Note: This is a quick proof of concept. Use at your own risk. ;)

Best,
Matt

Re: Merging .org files

[Pere Quintana Seguí]

[-- Attachment #1.1: Type: text/plain, Size: 779 bytes --]

Al 30/01/11 18:39, En/na Juan Pechiar ha escrit:
> Hola Pere,
> 
> I'd suggest creating a master document with #includes of all your
> Orgmode files, and then export this master document as org-mode.
> 
> http://orgmode.org/manual/Include-files.html#Include-files
> 
> There is a :minlevel attribute to automatically demote included
> org-mode files.
> 
> Saludos,
> .j.

I'll try this. It looks like a very clean way to do this.

Gracias,

Pere

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Merging .org files

[Pere Quintana Seguí]

[-- Attachment #1.1: Type: text/plain, Size: 1846 bytes --]

Al 30/01/11 20:22, En/na Michael Welle ha escrit:
>> In my current system, I have about 200 .org files. With this number of
>> files, building the agenda is very slow in my home computer and, also,
>> makes the performance of MobileOrg quite poor.
>>
>> So, I would like to join files, to increase the system's performance.
> don't the org files reflect another structure, for instance projects
> you are working on, customers you are dealing with etc? 

Yes, they are organised in subjects. But, they could be organised in
another way. There is more than one valid way to efficiently organise my
information and I should choose the one that performs better. My current
system is not good enough.

> 
>> Is there any script to cleanly join files, by transforming the title of
>> the file in a first level heading (*) and adding an star to all other
>> headings of the file?
> If you don't follow the advice of Juan, maybe you want only a few
> files merged into one file each, you can hack a Perl script or something
> similar. Finding regexp that match the headings isn't that complicated.

I'll probably follow Juan's advice. Which is very clean.

I know that hacking a script is easy, but it is even more efficient to
use an already written script if it was already written by someone. My
belief is that one of the points of free software is to reuse code as
much as possible. And I think that there is much more wisdom in this
list than in my humble brain, as Juan has proved.

Thank you,

Pere

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Merging .org files

[Pere Quintana Seguí]

[-- Attachment #1.1: Type: text/plain, Size: 2151 bytes --]

Al 30/01/11 22:51, En/na Matt Lundin ha escrit:
> Pere Quintana Seguí <pquintana@obsebre.es> writes:
> 
>> In my current system, I have about 200 .org files. With this number of
>> files, building the agenda is very slow in my home computer and, also,
>> makes the performance of MobileOrg quite poor.
>>
>> So, I would like to join files, to increase the system's performance.
> 
> FWIW, I've found the opposite to be true. Splitting up larger files and
> reducing the depth of the hierarchy marginally improves agenda
> performance (~ 0.2 seconds acc. to elp). But in my case, I split 10
> files into 40, so I have no idea what 200 files would do. Perhaps
> there's a sweet spot somewhere? I also haven't used mobile org.

So, maybe my initial assumption is wrong.

Has anybody run some experiments to check if it is better to have a
small number of big files or many small files?


> In my experience, archiving old items and reducing the number of active
> todos is the best way to keep the agenda snappy.
> 
>> Is there any script to cleanly join files, by transforming the title of
>> the file in a first level heading (*) and adding an star to all other
>> headings of the file?
> 
> Perl is always handy for this type of thing:
> 
> --8<---------------cut here---------------start------------->8---
> #!/usr/bin/perl
> use strict;
> use warnings;
> 
> my $mergefile = "/tmp/merged.org";
> 
> open NEWFILE, ">", $mergefile 
>   or die "Can't open $mergefile: $!";
> 
> while (<>) {
>   s/^(\*+)/*$1/;
>   s/^#\+TITLE:\s+(.*)$/* $1/;
>   s/^(#\+\w+:.*)$/: $1/;
>   print NEWFILE "$_";
> }
> 
> close NEWFILE;
> --8<---------------cut here---------------end--------------->8---
> 
> Note: This is a quick proof of concept. Use at your own risk. ;)

Great, thank you!

Pere


-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Merging .org files

[Michael Welle]

Hello,

Pere Quintana Seguí <pquintana@obsebre.es> writes:
[...]
> Has anybody run some experiments to check if it is better to have a
> small number of big files or many small files?
it depends on so many factors, disk speed vs. cpu speed for
instance. Maybe you could experiment with your data set and post some
numbers ;).

Regards
hmw

-- 
biff4emacsen - A biff-like tool for (X)Emacs
http://www.c0t0d0s0.de/biff4emacsen/biff4emacsen.html
Flood - Your friendly network packet generator
http://www.c0t0d0s0.de/flood/flood.html

Re: Re: Merging .org files

[Pere Quintana Seguí]

[-- Attachment #1.1: Type: text/plain, Size: 1441 bytes --]

Al 31/01/11 21:45, En/na Michael Welle ha escrit:
> it depends on so many factors, disk speed vs. cpu speed for
> instance. Maybe you could experiment with your data set and post some
> numbers ;).

At the end, I did my experiment, but not in a scientific manner. I did
no measurements. Nevertheless, qualitatively, I can now say that I
reduced the number of org files from ~200 to 8 and the agenda builds
much faster, specially on my netbook which was very slow and now it is
just slow. On my Desktop computer, building the agenda was fast and now
it is very fast. But, what is better, now MobileOrg works fine on my
iPhone, before it was useless. Sorry for not being able to provide
quantitative details.

At the end, I did the merging of the files manually. Inspecting the
files, one by one, was very useful, as it allowed me to delete some
useless information and better organise the rest.

Now I have to learn to better navigate within my much longer org files.
Before, I used ido-mode to jump from buffer to buffer, now I guess I
have to practise more sparse trees to jump from headline to headline.

-- 
Dr. Pere Quintana Seguí
Observatori de l'Ebre (Universitat Ramon Llull - CSIC)
Horta Alta, 38. 43520 Roquetes (Tarragona), Spain.
T. (+34) 977 500 511 - F. (+34) 977 504 660
http://www.obsebre.es - http://pere.quintanasegui.com
GnuPG:2CAB4330
http://pere.quintanasegui.com/coses/key-transition.txt


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Merging .org files

[Matt Lundin]

Pere Quintana Seguí <pquintana@obsebre.es> writes:

> Now I have to learn to better navigate within my much longer org files.
> Before, I used ido-mode to jump from buffer to buffer, now I guess I
> have to practise more sparse trees to jump from headline to headline.

I use this function to jump quickly (via ido) to a first level headline
in my org files:

--8<---------------cut here---------------start------------->8---
(defun my-org-global-goto ()
  (interactive)
  (let ((org-completion-use-ido t)
	(org-outline-path-complete-in-steps nil)
	(org-refile-use-outline-path nil)
	(org-refile-targets '((my-org-refile-targets :maxlevel . 1))))
    (org-refile t)))
--8<---------------cut here---------------end--------------->8---

Best,
Matt

Re: Re: Merging .org files

[Aankhen]

On Sat, Mar 19, 2011 at 02:08, Matt Lundin <mdl@imapmail.org> wrote:
> Pere Quintana Seguí <pquintana@obsebre.es> writes:
>
>> Now I have to learn to better navigate within my much longer org files.
>> Before, I used ido-mode to jump from buffer to buffer, now I guess I
>> have to practise more sparse trees to jump from headline to headline.
>
> I use this function to jump quickly (via ido) to a first level headline
> in my org files:
>
> [snip]

Do you normally have ‘org-completion-use-ido’ turned off or something?
(Just wondering why you couldn’t use ‘org-refile’ directly.)

Aankhen

Re: Merging .org files

[Matt Lundin]

Aankhen <aankhen@gmail.com> writes:

> On Sat, Mar 19, 2011 at 02:08, Matt Lundin <mdl@imapmail.org> wrote:
>> Pere Quintana Seguí <pquintana@obsebre.es> writes:
>>
>>> Now I have to learn to better navigate within my much longer org files.
>>> Before, I used ido-mode to jump from buffer to buffer, now I guess I
>>> have to practise more sparse trees to jump from headline to headline.
>>
>> I use this function to jump quickly (via ido) to a first level headline
>> in my org files:
>>
>> [snip]
>
> Do you normally have ‘org-completion-use-ido’ turned off or something?
> (Just wondering why you couldn’t use ‘org-refile’ directly.)

Yes, that is correct. I normally have org-completion-use-ido turned off.

You could easily call org-refile with a prefix argument directly from
within an org-buffer. However, I find it more convenient to bind
"(org-refile t)" to one of the function keys than to type C-u C-c C-w.
The latter works only on org buffers, while the former is global.
Moreover, when navigating org files in this way, I only want to see
first level headlines, whereas my default refile binding uses deeper
levels.

Best,
Matt

Re: Merging .org files

[Aankhen]

On Sat, Mar 19, 2011 at 18:23, Matt Lundin <mdl@imapmail.org> wrote:
> Aankhen <aankhen@gmail.com> writes:
>> [snip]
>>
>> Do you normally have ‘org-completion-use-ido’ turned off or something?
>> (Just wondering why you couldn’t use ‘org-refile’ directly.)
>
> Yes, that is correct. I normally have org-completion-use-ido turned off.
>
> You could easily call org-refile with a prefix argument directly from
> within an org-buffer. However, I find it more convenient to bind
> "(org-refile t)" to one of the function keys than to type C-u C-c C-w.
> The latter works only on org buffers, while the former is global.
> Moreover, when navigating org files in this way, I only want to see
> first level headlines, whereas my default refile binding uses deeper
> levels.

A’right, makes sense.  I appreciate the detailed explanation.

Aankhen