observations on updating to recent org

observations on updating to recent org

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]


I use org for the usual notes-to-self and TODO - nothing super fancy.  I
had been running from master of the git repo, but in 2013-01 stopped
updating, probably because I had some issue.  The recent release
provoked me to try again, and this note reports some issues.

I had been running (for no really good reason)
  commit 3a1c27060792fc095435efcaf270a6207d5d8d27
  Author: Bastien Guerry <bzg@altern.org>
  Date:   Mon Jan 7 00:25:58 2013 +0100
and just updated to the head of maint:
  commit 1fa6ccab10c9f97dc9317f9c3eedea82c4cdc357
  Author: Bastien Guerry <bzg@altern.org>
  Date:   Tue Apr 22 15:24:14 2014 +0200
which is just a docstring fix past release_8.2.6.

I then did an org-mobile-push, and that was as fast as I remembered.

I noticed two things:

  Exporting to ical as a single file took a really long time, perhaps a
  whole minute, whereas it used to take a second to a few seconds.  The
  resulting export did seem ok.

  I used to get an ID PROPERTIES entries for nodes that were exported to the
  calendar, which was basically nodes that had an active timestamp.
  But now I had a huge number of changes, adding ID to every node, even
  those with no real content and just children.  Is this a feature?


[-- Attachment #2: Type: application/pgp-signature, Size: 180 bytes --]

Re: observations on updating to recent org

[Bastien]

Hi Greg,

Greg Troxel <gdt@ir.bbn.com> writes:

>   I used to get an ID PROPERTIES entries for nodes that were exported to the
>   calendar, which was basically nodes that had an active timestamp.
>   But now I had a huge number of changes, adding ID to every node, even
>   those with no real content and just children.  Is this a feature?

Indeed.

Nicolas, do you remember why we added IDs to all headlines and not
just the one that we be exported in the .ics file?

-- 
 Bastien

Re: observations on updating to recent org

[Nicolas Goaziou]

Hello,

Bastien <bzg@gnu.org> writes:

> Nicolas, do you remember why we added IDs to all headlines and not
> just the one that we be exported in the .ics file?

It is a bit difficult to do otherwise.

We don't know beforehand what headlines are going to be exported. We get
this information when a headline is encountered during export, which
happens in a different buffer, and possibly in a different process.

I think it is a minor inconvenience and we can live with it.


Regards,

-- 
Nicolas Goaziou

Re: observations on updating to recent org

[Bastien]

Hi Nicolas,

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Bastien <bzg@gnu.org> writes:
>
>> Nicolas, do you remember why we added IDs to all headlines and not
>> just the one that we be exported in the .ics file?
>
> It is a bit difficult to do otherwise.

Assuming this is just difficult, not impossible, what would be the way
to do it?

> We don't know beforehand what headlines are going to be exported.

Would it be possible to emulate export first just for the sake of
adding IDs where it's necessary?

> We get this information when a headline is encountered during
> export, which happens in a different buffer, and possibly in a
> different process.
>
> I think it is a minor inconvenience and we can live with it.

`org-icalendar-store-UID' is `nil' by default because a value of
`t' might be very inconvenient in some circumstances -- quoting the
docstring:

  "This variable is not turned on by default because we want to avoid
  creating a property drawer in every entry if people are only playing
  with this feature, or if they are only using it locally."

One way to adapt to the current behavior is to have a file dedicated
to being exported as an ics file -- but for people who want to export
entries from their regular files, having all those drawers created
everywhere just for the sake of a few entries being picked up for
the .ics output certainly feels wrong.

I'm interested in enhancing the current behavior if you can give me
a few directions.

Thanks,

-- 
 Bastien

Re: observations on updating to recent org

[Nicolas Goaziou]

Bastien <bzg@gnu.org> writes:

> Assuming this is just difficult, not impossible, what would be the way
> to do it?

The major difficulty is to keep an association table between headlines
in the pristine original buffer, and headlines in the copy being
exported. Note that hooks and Babel code may have deleted or added some,
or altered their contents, which means that it is theoretically
impossible to get it right.

You may want to go for an approximation, i.e., add an ID property only
for headlines or inlinetasks with either

  - a TODO-like keyword,
  - a SCHEDULED value,
  - a DEADLINE value,
  - a timestamp in their contents,
  - a diary-sexp in their contents.

AFAIK, ox-icalendar only considers these for export, so you will often
end up marking a super-set of actually exported entries.

Unfortunately, this will fail if a user decides to add TODO keywords or
timestamps through hooks or Babel (e.g., in order to mark current
headline with a "today" mark).

I don't think you can limit the numbers of properties drawers created
without inserting some limitations.

> Would it be possible to emulate export first just for the sake of
> adding IDs where it's necessary?

I don't think so.

> `org-icalendar-store-UID' is `nil' by default because a value of
> `t' might be very inconvenient in some circumstances -- quoting the
> docstring:
>
>   "This variable is not turned on by default because we want to avoid
>   creating a property drawer in every entry if people are only playing
>   with this feature, or if they are only using it locally."

I know. Though, I don't find it very inconvenient to create ID
properties everywhere once per file.


Regards,

-- 
Nicolas Goaziou

Re: observations on updating to recent org

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]


Greg Troxel <gdt@ir.bbn.com> writes:

>   Exporting to ical as a single file took a really long time, perhaps a
>   whole minute, whereas it used to take a second to a few seconds.  The
>   resulting export did seem ok.

I timed this.  With 6161 lines in 14 org-mode files (about 2175 of which
are due to PROPERTIES/ID/END), doing a combined export took 88s of cpu
time.   emacs-23.4.1, NetBSD 6, i386, plenty of RAM, Core i5 2.9 GHz.
In contrast, starting up emacs and generating the agenda took 0.97s.

I noticed that TODO entries got exported multiple times, apparently once
for each inactive timestamp.  (I realize I need to prepare a minimal
example.)

>   I used to get an ID PROPERTIES entries for nodes that were exported to the
>   calendar, which was basically nodes that had an active timestamp.
>   But now I had a huge number of changes, adding ID to every node, even
>   those with no real content and just children.  Is this a feature?

Thanks for the comments; I see the point that this is hard..  For now,
I've just turned off uid storing, because I don't sync the exported
calendar, and I'll nuke the ID entries and properties drawers at some
point; I find them distracting when editing.

I wonder if there's some way to go back and store the UID when it
actually needs to be generated, so that UIDs are only stored for entries
that actually have been exported.  I only want to export appointments,
by which I mean entries with active timestamps, which are pretty few in
number compared to nodes.

Greg

[-- Attachment #2: Type: application/pgp-signature, Size: 180 bytes --]

Re: observations on updating to recent org

[Nicolas Goaziou]

Hello,

Greg Troxel <gdt@ir.bbn.com> writes:

> I timed this.  With 6161 lines in 14 org-mode files (about 2175 of which
> are due to PROPERTIES/ID/END), doing a combined export took 88s of cpu
> time.   emacs-23.4.1, NetBSD 6, i386, plenty of RAM, Core i5 2.9 GHz.
> In contrast, starting up emacs and generating the agenda took 0.97s.

You may want to use ELP to instrument Org and report where most time is
spent.

> I noticed that TODO entries got exported multiple times, apparently once
> for each inactive timestamp.  (I realize I need to prepare a minimal
> example.)

This is to be expected. Each plain timestamp defines a new VEVENT block.

By default, it shouldn't do this for inactive timestamps, though. See
`org-icalendar-with-timestamps'.

> Thanks for the comments; I see the point that this is hard..  For now,
> I've just turned off uid storing, because I don't sync the exported
> calendar, and I'll nuke the ID entries and properties drawers at some
> point; I find them distracting when editing.

Note that UID storing is off by default.

> I wonder if there's some way to go back and store the UID when it
> actually needs to be generated, so that UIDs are only stored for entries
> that actually have been exported.

This is not really possible without some sacrifices (which Bastien may
or may not want to do).  See other messages in this thread.

Also, since you don't need UID anyway, why is that a problem anymore?
I'm a bit confused here.


Regards,

-- 
Nicolas Goaziou

Re: observations on updating to recent org

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 3727 bytes --]


Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Greg Troxel <gdt@ir.bbn.com> writes:
>
>> I timed this.  With 6161 lines in 14 org-mode files (about 2175 of which
>> are due to PROPERTIES/ID/END), doing a combined export took 88s of cpu
>> time.   emacs-23.4.1, NetBSD 6, i386, plenty of RAM, Core i5 2.9 GHz.
>> In contrast, starting up emacs and generating the agenda took 0.97s.
>
> You may want to use ELP to instrument Org and report where most time is
> spent.

OK - will try to do that.

>> I noticed that TODO entries got exported multiple times, apparently once
>> for each inactive timestamp.  (I realize I need to prepare a minimal
>> example.)
>
> This is to be expected. Each plain timestamp defines a new VEVENT block.

I guess that's an interesting question about what makes sense.   Here's
the actual todo entry, with just a few words redacted.  I don't see why
someone would want VEVENTS for this kind of history, but I suppose maybe
that's what you get when you turn on events from inactive timestamps.

  ***** TODO [#C] order more [redacted]
        SCHEDULED: <2014-04-28 Mon .+4w>
        - State "DONE"       from "TODO"       [2013-11-07 Thu 10:30]
        - State "DONE"       from "TODO"       [2013-08-05 Mon 16:27]
        - State "DONE"       from "WAITING"    [2013-06-04 Tue 10:27]
        - State "WAITING"    from "TODO"       [2013-05-28 Tue 11:19] \\
          ordered via [redacted]
        - State "DONE"       from "TODO"       [2013-03-25 Mon 23:16]
          laura did
        - State "DONE"       from "WAITING"    [2013-01-08 Tue 11:49]
        - State "WAITING"    from "TODO"       [2012-12-18 Tue 20:41] \\
          ordered
        - State "TODO"       from "WAITING"    [2012-09-20 Thu 13:05]
        - State "WAITING"    from "TODO"       [2012-09-13 Thu 10:01] \\
          ordered
        - State "DONE"       from "TODO"       [2012-07-12 Thu 10:39]
        - State "DONE"       from "TODO"       [2012-06-04 Mon 14:22]
        - State "DONE"       from "TODO"       [2012-03-27 Tue 08:54]
        - State "DONE"       from "TODO"       [2011-10-01 Sat 08:10]
        :PROPERTIES:
        :ID:       b617c8e4-c8f2-11e0-8735-000476353fb4
        :LAST_REPEAT: [2013-11-07 Thu 10:30]
        :END:

> By default, it shouldn't do this for inactive timestamps, though. See
> `org-icalendar-with-timestamps'.

I didn't try to turn this on.  My icalendar-relevant settings are

      (setq org-icalendar-alarm-time 10)
      (setq org-icalendar-use-scheduled nil)
      (setq org-icalendar-use-deadline nil)

I am trying to get ics entries only for headlines with active timestamps.

>> Thanks for the comments; I see the point that this is hard..  For now,
>> I've just turned off uid storing, because I don't sync the exported
>> calendar, and I'll nuke the ID entries and properties drawers at some
>> point; I find them distracting when editing.
>
> Note that UID storing is off by default.

Yes - I had it turned on intentionally from long ago.

>> I wonder if there's some way to go back and store the UID when it
>> actually needs to be generated, so that UIDs are only stored for entries
>> that actually have been exported.
>
> This is not really possible without some sacrifices (which Bastien may
> or may not want to do).  See other messages in this thread.

I did see them, but I guess I just don't understand enough for this to
make sense.  That's ok - please don't try harder to explain to me :-)

> Also, since you don't need UID anyway, why is that a problem anymore?
> I'm a bit confused here.

It's not a problem for me any more.  It just seems really unfortunate
for others in the general case to have extra content in every headline
when it isn't necessary.


[-- Attachment #2: Type: application/pgp-signature, Size: 180 bytes --]

Re: observations on updating to recent org

[Nicolas Goaziou]

Greg Troxel <gdt@ir.bbn.com> writes:

> I guess that's an interesting question about what makes sense.   Here's
> the actual todo entry, with just a few words redacted.  I don't see why
> someone would want VEVENTS for this kind of history, but I suppose maybe
> that's what you get when you turn on events from inactive timestamps.
>
>   ***** TODO [#C] order more [redacted]
>         SCHEDULED: <2014-04-28 Mon .+4w>
>         - State "DONE"       from "TODO"       [2013-11-07 Thu 10:30]
>         - State "DONE"       from "TODO"       [2013-08-05 Mon 16:27]
>         - State "DONE"       from "WAITING"    [2013-06-04 Tue 10:27]
>         - State "WAITING"    from "TODO"       [2013-05-28 Tue 11:19] \\
>           ordered via [redacted]
>         - State "DONE"       from "TODO"       [2013-03-25 Mon 23:16]
>           laura did
>         - State "DONE"       from "WAITING"    [2013-01-08 Tue 11:49]
>         - State "WAITING"    from "TODO"       [2012-12-18 Tue 20:41] \\
>           ordered
>         - State "TODO"       from "WAITING"    [2012-09-20 Thu 13:05]
>         - State "WAITING"    from "TODO"       [2012-09-13 Thu 10:01] \\
>           ordered
>         - State "DONE"       from "TODO"       [2012-07-12 Thu 10:39]
>         - State "DONE"       from "TODO"       [2012-06-04 Mon 14:22]
>         - State "DONE"       from "TODO"       [2012-03-27 Tue 08:54]
>         - State "DONE"       from "TODO"       [2011-10-01 Sat 08:10]
>         :PROPERTIES:
>         :ID:       b617c8e4-c8f2-11e0-8735-000476353fb4
>         :LAST_REPEAT: [2013-11-07 Thu 10:30]
>         :END:

By default, no VEVENT should be created from any timestamp in your
example.

>> By default, it shouldn't do this for inactive timestamps, though. See
>> `org-icalendar-with-timestamps'.
>
> I didn't try to turn this on.  My icalendar-relevant settings are
>
>       (setq org-icalendar-alarm-time 10)
>       (setq org-icalendar-use-scheduled nil)
>       (setq org-icalendar-use-deadline nil)
>
> I am trying to get ics entries only for headlines with active
> timestamps.

It is a bug then. I'll look into it in a few hours.

> It's not a problem for me any more.  It just seems really unfortunate
> for others in the general case to have extra content in every headline
> when it isn't necessary.

If there are few exported headlines in the document, it is also possible
to keep `org-icalendar-store-UID' to nil and add ID value manually for
each of them.


Regards,

-- 
Nicolas Goaziou

Re: observations on updating to recent org

[Nicolas Goaziou]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Greg Troxel <gdt@ir.bbn.com> writes:

>> I didn't try to turn this on.  My icalendar-relevant settings are
>>
>>       (setq org-icalendar-alarm-time 10)
>>       (setq org-icalendar-use-scheduled nil)
>>       (setq org-icalendar-use-deadline nil)
>>
>> I am trying to get ics entries only for headlines with active
>> timestamps.
>
> It is a bug then. I'll look into it in a few hours.

This should be fixed. Thank you for the report.


Regards,

-- 
Nicolas Goaziou

Re: observations on updating to recent org

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]


Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> Greg Troxel <gdt@ir.bbn.com> writes:
>
>>> I didn't try to turn this on.  My icalendar-relevant settings are
>>>
>>>       (setq org-icalendar-alarm-time 10)
>>>       (setq org-icalendar-use-scheduled nil)
>>>       (setq org-icalendar-use-deadline nil)
>>>
>>> I am trying to get ics entries only for headlines with active
>>> timestamps.
>>
>> It is a bug then. I'll look into it in a few hours.
>
> This should be fixed. Thank you for the report.

Thanks for the quick fix.  export now behaves correctly, and it takes
only 5s, which is way faster than 90!


[-- Attachment #2: Type: application/pgp-signature, Size: 180 bytes --]

Re: observations on updating to recent org

[Bastien]

Hi Nicolas,

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Bastien <bzg@gnu.org> writes:
>
>> Assuming this is just difficult, not impossible, what would be the way
>> to do it?
>
> The major difficulty is to keep an association table between headlines
> in the pristine original buffer, and headlines in the copy being
> exported.

Copy that.  It seems the biggest annoyances are now gone, so let's not
digg in this direction.

-- 
 Bastien