More clocktable breakage

More clocktable breakage

[Achim Gratz]

I've just noticed that in my the clocktables at work I can't adjust the
start and end ranges anymore with up/down.  Apparently the timestamps
are not recognized anymore and the it falls through to some code that
tries to adjust the :block argument (which is not present in that table
since it's using :tstart / :tend) and fails with an error message.  I've
bisected that behaviour to commit 2a59d2f76f.  Unfortunately I can't
reproduce that here at home, maybe due to the fact that the language
settings are different.  At home the timestamps are recognized and the
clocktable is re-calculated accordingly.  However, I cannot refresh the
clocktable via C-c C-c, which tells me it cannot do anything useful at
that line.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: More clocktable breakage

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> I've just noticed that in my the clocktables at work I can't adjust the
> start and end ranges anymore with up/down.  Apparently the timestamps
> are not recognized anymore and the it falls through to some code that
> tries to adjust the :block argument (which is not present in that table
> since it's using :tstart / :tend) and fails with an error message.  I've
> bisected that behaviour to commit 2a59d2f76f.  Unfortunately I can't
> reproduce that here at home, maybe due to the fact that the language
> settings are different.  At home the timestamps are recognized and the
> clocktable is re-calculated accordingly.  However, I cannot refresh the
> clocktable via C-c C-c, which tells me it cannot do anything useful at
> that line.

At the moment, I cannot reproduce it. I tried M-up in the following
document:

     #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>" :tend "<2006-08-10 Thu 12:00>"
     #+END: clocktable

I'm interested in an ECM, if you can provide one.

Regards,

-- 
Nicolas Goaziou

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
> At the moment, I cannot reproduce it. I tried M-up in the following
> document:
>
>      #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>" :tend "<2006-08-10 Thu 12:00>"
>      #+END: clocktable

The breakage happens in this clause in org-at-timestamp-p:

--8<---------------cut here---------------start------------->8---
	 (match
	  (let ((boundaries (org-in-regexp tsr)))
	    (save-match-data
	      (cond ((null boundaries) nil)
		    ((org-at-planning-p))
		    ((org-at-property-p))
		    ;; CLOCK lines only contain inactive time-stamps.
		    ((and inactive-ok (org-at-clock-log-p)))
		    (t
		     (eq 'timestamp
			 (save-excursion
			   (when (= pos (cdr boundaries)) (forward-char -1))
			   (org-element-type (org-element-context))))))))))
--8<---------------cut here---------------end--------------->8---

After matching the timestamp in the header argument correctly, the code
falls through to the default cond, where (org-element-type
(org-element-context)) returns 'dynamic-block, which isn't a 'timestamp.
The successful match gets discarded and the timestamp doesn't get
recognized.  An empty clause for (org-at-block-p) would fix it, but I'm
not sure that is the right thing to do.  I haven't looked at
org-element-context to see whether it might misinterpret something.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: More clocktable breakage

[Achim Gratz]

Achim Gratz writes:
> Nicolas Goaziou writes:
>> At the moment, I cannot reproduce it. I tried M-up in the following
>> document:
>>
>>      #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>" :tend "<2006-08-10 Thu 12:00>"
>>      #+END: clocktable
>
> The breakage happens in this clause in org-at-timestamp-p:
>
> 	 (match
> 	  (let ((boundaries (org-in-regexp tsr)))
> 	    (save-match-data
> 	      (cond ((null boundaries) nil)
> 		    ((org-at-planning-p))
> 		    ((org-at-property-p))
> 		    ;; CLOCK lines only contain inactive time-stamps.
> 		    ((and inactive-ok (org-at-clock-log-p)))
> 		    (t
> 		     (eq 'timestamp
> 			 (save-excursion
> 			   (when (= pos (cdr boundaries)) (forward-char -1))
> 			   (org-element-type (org-element-context))))))))))
>
> After matching the timestamp in the header argument correctly, the code
> falls through to the default cond, where (org-element-type
> (org-element-context)) returns 'dynamic-block, which isn't a 'timestamp.
> The successful match gets discarded and the timestamp doesn't get
> recognized.  An empty clause for (org-at-block-p) would fix it, but I'm
> not sure that is the right thing to do.  I haven't looked at
> org-element-context to see whether it might misinterpret something.

I did look at org-element-context and the code in org-at-timstamp-p
makes even less sense to me now.  The only time org-element-context
returns 'timestamp is when it is on a planning line, but
org-at-timstamp-p has already left the cond in that case in the second
clause.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: More clocktable breakage

[Nicolas Goaziou]

Achim Gratz <Stromeko@nexgo.de> writes:

> Achim Gratz writes:
>> Nicolas Goaziou writes:
>>> At the moment, I cannot reproduce it. I tried M-up in the following
>>> document:
>>>
>>>      #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>" :tend "<2006-08-10 Thu 12:00>"
>>>      #+END: clocktable

These are not timestamps. Even though they look like timestamps, you
wouldn't want them to appear in the agenda. So `org-at-timestamp-p' is
correct, IMO. Moreover, its docstring says

  This function checks context and only return non-nil for valid
  time stamps.  If you need to match anything looking like a time
  stamp, or if you are sure about the context, consider using
  ‘org-in-regexp’, e.g.,

    (org-in-regexp org-ts-regexp)

so ISTM a correct fix would be to have the function you're calling (I
don't remember it) use this instead of `org-at-timestamp-p'.

WDYT?

-- 
Nicolas Goaziou

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
>>>>
>>>>      #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>" :tend "<2006-08-10 Thu 12:00>"
>>>>      #+END: clocktable
>
> These are not timestamps. Even though they look like timestamps, you
> wouldn't want them to appear in the agenda. So `org-at-timestamp-p' is
> correct, IMO. Moreover, its docstring says
[…]

Well, that's the change you made that I pointed at earlier.  The
functionality of org-shift* has unfortunately regressed due to
that change you introduced by "hardening" the recognition of timestamps.
And no, to the user they really are timestamps in the arguments of a
dynamic block, splitting syntactical hairs aside.

> so ISTM a correct fix would be to have the function you're calling (I
> don't remember it) use this instead of `org-at-timestamp-p'.

I already said that the fix might not be the right one, although it
would be in the same spirit as the exceptions already present for
properties drawers, planning lines and clocks and org-at-block-p only
matches in the first line of a dynamic block, so I'd say it's still
sufficiently constrained.

N.B.: The regex used in org-at-block-p is overbroad since it matches the
whole block, I think it should use org-at-dblock-start-re instead.

Anyway, when you changed the scope of that function you'd need to check
all users of the API and fix them where necessary.  The main users of
that predicate were and still are the org-shift* family of commands and
they do expect a looser recognition than what you implemented based on
the docstring.  Maybe that's true in other places also, I haven't
checked.  It's also obvious that the test coverage of this is bad.  So
that looser recognition would need to be factored out again or you
revert this predicate and apply the stricter check where it matters (the
agenda functions, most likely).

Also, again, I think that function is buggy even without these issues as
the only code I can find in org-element-context that deals with
timestamps is conditional on being on a planning line and if that's true
we should never get there.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: More clocktable breakage

[Nicolas Goaziou]

Achim Gratz <Stromeko@nexgo.de> writes:

> and org-at-block-p only matches in the first line of a dynamic block,

[...]

> N.B.: The regex used in org-at-block-p is overbroad since it matches the
> whole block, I think it should use org-at-dblock-start-re instead.

This old and buggy function has nothing to do with the problem at hand.
Also, even if it used org-at-dblock-start-re, it would still be buggy
(is it closed? is it in an example block?).

> Anyway, when you changed the scope of that function you'd need to check
> all users of the API and fix them where necessary.

Really? I thought it would magically happen... somehow. Silly me.

> The main users of that predicate were and still are the org-shift*
> family of commands and they do expect a looser recognition than what
> you implemented based on the docstring. Maybe that's true in other
> places also, I haven't checked.

The main user of that predicate is the agenda, now.

> It's also obvious that the test coverage of this is bad.

Patch welcome.

> So that looser recognition would need to be factored out again or you
> revert this predicate and apply the stricter check where it matters
> (the agenda functions, most likely).

Another idea is to add an optional argument to `org-at-timestamp-p' to
allow "sloppy" matching (or strict matching, it doesn't really matter)
and skip checks when it is required. So all functions requiring this
predicate can make use of it, as long as they call it properly.

WDYT?

> Also, again, I think that function is buggy even without these issues as
> the only code I can find in org-element-context that deals with
> timestamps is conditional on being on a planning line and if that's true
> we should never get there.

I don't think there is a bug there. Do you have an ECM?

Regards,

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
> Another idea is to add an optional argument to `org-at-timestamp-p' to
> allow "sloppy" matching (or strict matching, it doesn't really matter)
> and skip checks when it is required. So all functions requiring this
> predicate can make use of it, as long as they call it properly.
>
> WDYT?

That's what I've been asking the whole time: when you changed that
predicate, you've basically cut off some of its consumers.  It looks
like bad factoring to me to then splitting the same predicate based on
some argument.  I'd rather pull out the old sloppy matching into a new
predicate for instance.

>> Also, again, I think that function is buggy even without these issues as
>> the only code I can find in org-element-context that deals with
>> timestamps is conditional on being on a planning line and if that's true
>> we should never get there.
>
> I don't think there is a bug there. Do you have an ECM?

I don't use planning, so no.  But it seems to me that the only way for
org-element-context to deliver a 'timestamp is when pos is on a planning
line (that may be wrong, I just don't see where else a 'timestamp would
be returned).  In that case we've already left the cond, so testing for it
doesn't do anything useful.  I'm also not really sure why the existing
exceptions from the "true" timestamp (planning, property and clock-log)
are any different than "dynamic block" would be (with the appropriate
change of the doc string of course).y


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: More clocktable breakage

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> That's what I've been asking the whole time: when you changed that
> predicate, you've basically cut off some of its consumers.  It looks
> like bad factoring to me to then splitting the same predicate based on
> some argument.  I'd rather pull out the old sloppy matching into a new
> predicate for instance.

There are drawbacks in both choices. More on this below.

> I don't use planning, so no.  But it seems to me that the only way for
> org-element-context to deliver a 'timestamp is when pos is on a planning
> line (that may be wrong, I just don't see where else a 'timestamp would
> be returned).  In that case we've already left the cond, so testing for it
> doesn't do anything useful.  I'm also not really sure why the existing
> exceptions from the "true" timestamp (planning, property and clock-log)
> are any different than "dynamic block" would be (with the appropriate
> change of the doc string of course).y

I start to see where the confusion comes from. 

Sadly, what a "timestamp" is depends on what function is asking. AFAICT,
there are three categories of "timestamps".

Let's consider the following examples:

  (1)
  SCHEDULED: <2017-04-30 dim.>

  (2)
  CLOCK: [2017-04-30 dim. 08:10]--[2017-04-30 dim. 08:11] =>  0:01

  (3)
  : <2017-04-30 dim.>

  (4)
  :PROPERTIES:
  :DATE: <2017-04-30 dim.>
  :END:

  (5)
  # <2017-04-30 dim.>

  (6)
  #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>"

The first category is the "strict" one, which follows Org syntax
definition. None of the examples above belong to that category.
`org-element-context' should be the relative predicate, which means that
it probably shouldn't return a timestamp object on planning lines, as it
does at the moment.

The second category, which is a super-set of the previous one, is
"agenda" one. Historically, Org allowed to insert timestamps in property
drawers, e.g., as in (4), and use them in the agenda. So basically, this
category contains "every timestamp that would appear as a plain
timestamp in the agenda if it were active". `org-at-timestamp-p' is
currently the relative predicate for that category. Note that the
category also contains (2) if inactive timestamps are meant to be
displayed in the agenda.

The last category, a super-set of the "agenda" one, is the "convenience"
category. Basically, it contains every occurrence of what looks like
a timestamp, but isn't necessarily one. Obviously, every example given
above fits in there, as in every case, you can ignore that Org has
a context-dependant grammar and pretend you are on a timestamp. There is
no predicate relative to that category, but `org-at-timestamp-p'
docstring suggests to use (org-in-regexp org-ts-regexp).

So, about the predicates, I guess we could move the second one into
"org-agenda.el" and rename it `org-agenda-at-timestamp-p'. However,
using `org-at-timestamp-p' for the last category seems a bit
far-stretched to me, since it doesn't always match timestamps. In
particular, (3) and (5) sound counter-intuitive to me and I wouldn't
like it from a developer standpoint. `org-at-timestamp' is also not
really needed for the first category, since there is already
`org-element-context'.

Yet, `org-at-timestamp-p' is something users are going to look after.
Different names may also introduce confusion (remember `org-at-regexp-p'
and `org-in-regexp-p'?). So, even if it looks like bad factoring to you,
a single predicate, or even two if we set "agenda" apart, with
a docstring explaining the different categories and how to select them
may also be a good natural choice. Hence my suggestion.

WDYT? Do you have any other concrete proposal?

Regards,

-- 
Nicolas Goaziou

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
> Sadly, what a "timestamp" is depends on what function is asking. AFAICT,
> there are three categories of "timestamps".

Well, taking a further setp back, before Org started to have a formal
syntax anything that looked like a timestamp was treated as one.  The
different categories of timestamps really arise from the fact that we
now have syntactical timestamps and non-syntactical ones.  This in turn
requires that each function dealing with timestamps needs to specify
what exactly it wants to deal with, but as this discussion amply shows,
that isn't quite as straightforward as one would think.

> Let's consider the following examples:
>
>   (1)
>   SCHEDULED: <2017-04-30 dim.>
>
>   (2)
>   CLOCK: [2017-04-30 dim. 08:10]--[2017-04-30 dim. 08:11] =>  0:01
>
>   (3)
>   : <2017-04-30 dim.>
>
>   (4)
>   :PROPERTIES:
>   :DATE: <2017-04-30 dim.>
>   :END:
>
>   (5)
>   # <2017-04-30 dim.>
>
>   (6)
>
>   #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>"
>
> The first category is the "strict" one, which follows Org syntax
> definition. None of the examples above belong to that category.
> `org-element-context' should be the relative predicate, which means that
> it probably shouldn't return a timestamp object on planning lines, as it
> does at the moment.

I'd say anything org-element-* should exclusively return syntactical
things.  Context dependence needs to be dealt with elsewhere.

> The second category, which is a super-set of the previous one, is
> "agenda" one. Historically, Org allowed to insert timestamps in property
> drawers, e.g., as in (4), and use them in the agenda. So basically, this
> category contains "every timestamp that would appear as a plain
> timestamp in the agenda if it were active". `org-at-timestamp-p' is
> currently the relative predicate for that category. Note that the
> category also contains (2) if inactive timestamps are meant to be
> displayed in the agenda.

I would call that meta-syntax or maybe quoted syntax: you are looking at
a proper timestamp, to be used somewhere else where a timestamp is
needed, but in a context that doesn't specify a timestamp itself.  There
are many analogous cases elsewhere in Org, so it may be a fruitful
endeavour to consider this in a more general fashion.  Providing
timestamps as arguments to any processing functions (built into Org or
otherwise) carries the requirement that they should syntactically be
correct as a timestamp (so they can be converted into a timestamp object
at the place of use), so all functions to insert and edit a timestamp
need to work at those places.

> The last category, a super-set of the "agenda" one, is the "convenience"
> category. Basically, it contains every occurrence of what looks like
> a timestamp, but isn't necessarily one. Obviously, every example given
> above fits in there, as in every case, you can ignore that Org has
> a context-dependant grammar and pretend you are on a timestamp. There is
> no predicate relative to that category, but `org-at-timestamp-p'
> docstring suggests to use (org-in-regexp org-ts-regexp).

If I'd follow that road, I could edit what looks like a timestamp
everywhere, regardless of context.  I don't know if that's the right
thing to do and I don't even expect consensus among the Org user base.
I personally see no need to do that.

> So, about the predicates, I guess we could move the second one into
> "org-agenda.el" and rename it `org-agenda-at-timestamp-p'. However,
> using `org-at-timestamp-p' for the last category seems a bit
> far-stretched to me, since it doesn't always match timestamps. In
> particular, (3) and (5) sound counter-intuitive to me and I wouldn't
> like it from a developer standpoint. `org-at-timestamp' is also not
> really needed for the first category, since there is already
> `org-element-context'.

Agreed.  That's why I'm hesitant to change the org-at-timestamp-p to
(org-in-regexp org-ts-regexp) in the edit functions.  What I don't agree
with (if I've read you correctly) is the implied assertion that the
clocktable example is in the last category.  Or maybe it is at the
moment, but it ought to be in the second one.  I consider only examples
(3) and (5) to be in the "last" category.

> Yet, `org-at-timestamp-p' is something users are going to look after.
> Different names may also introduce confusion (remember `org-at-regexp-p'
> and `org-in-regexp-p'?). So, even if it looks like bad factoring to you,
> a single predicate, or even two if we set "agenda" apart, with
> a docstring explaining the different categories and how to select them
> may also be a good natural choice. Hence my suggestion.

Since org-at-timestamp-p already has an argument, adding a second one
doesn't look appealing to me, especially when the first one would often
be nil.  Maybe it's possible to change the definition of that argument
(which would need the equivalence between t meaning 'inactive).

> WDYT? Do you have any other concrete proposal?

I really only need the clocktables to work again, which they'd be if
they were in the second category, I gather.  I've monkeypatched
org-at-timestamp-p with an ((org-at-block-p)) clause so I'm good for
now, but per our discussion this wouldn't be an acceptable solution.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: More clocktable breakage

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> Well, taking a further setp back, before Org started to have a formal
> syntax anything that looked like a timestamp was treated as one.  The
> different categories of timestamps really arise from the fact that we
> now have syntactical timestamps and non-syntactical ones.

I don't pretend the contrary. I'm just talking about the current state
of timestamps.

Oh, and it is a really good thing Org now has a formal syntax.

> This in turn requires that each function dealing with timestamps needs
> to specify what exactly it wants to deal with, but as this discussion
> amply shows, that isn't quite as straightforward as one would think.

Indeed.

>> The first category is the "strict" one, which follows Org syntax
>> definition. None of the examples above belong to that category.
>> `org-element-context' should be the relative predicate, which means that
>> it probably shouldn't return a timestamp object on planning lines, as it
>> does at the moment.
>
> I'd say anything org-element-* should exclusively return syntactical
> things.  Context dependence needs to be dealt with elsewhere.

I'm not sure to understand this. Syntactical things are all about
context dependence in Org. Do you mean /context independance/ needs to
be dealt with elsewhere?

> I would call that meta-syntax or maybe quoted syntax: you are looking at
> a proper timestamp, to be used somewhere else where a timestamp is
> needed, but in a context that doesn't specify a timestamp itself.  There
> are many analogous cases elsewhere in Org, so it may be a fruitful
> endeavour to consider this in a more general fashion.  Providing
> timestamps as arguments to any processing functions (built into Org or
> otherwise) carries the requirement that they should syntactically be
> correct as a timestamp (so they can be converted into a timestamp object
> at the place of use), so all functions to insert and edit a timestamp
> need to work at those places.

Could you provide examples about that? In particular "providing
timestamps as arguments to any processing functions" sounds odd, in
particular from someone who cringes when I suggest to add a second
optional argument to a single function.

> If I'd follow that road, I could edit what looks like a timestamp
> everywhere, regardless of context.  I don't know if that's the right
> thing to do and I don't even expect consensus among the Org user base.
> I personally see no need to do that.

I do see it, tho. This is analogous to links in comments. If you see
something looking like a timestamp (or a link), you can expect some
commands to operate on it. This is exactly what is biting us at the
moment.

The real problem is that, deep into the functions calls triggered by
(org-shift*), one of them expects to operate on timestamps at least from
category 2 (i.e., it uses `org-at-timestamp-p') while we're in
category 3. The solution is not to promote current timestamp to category
2 but to relax requirements from the guilty function, so that it also
operates on category 3. Fixing it is easy: we just need to replace
`org-at-timestamp-p' with (org-in-regexp org-ts-regexp) at the
appropriate place.

IIUC, however, we are discussing higher level details, i.e., about
predicates for each category, and the developer interface we want to
provide.

> Agreed.  That's why I'm hesitant to change the org-at-timestamp-p to
> (org-in-regexp org-ts-regexp) in the edit functions.  What I don't agree
> with (if I've read you correctly) is the implied assertion that the
> clocktable example is in the last category.  Or maybe it is at the
> moment, but it ought to be in the second one.  I consider only examples
> (3) and (5) to be in the "last" category.

The second category, according to my previous message, is about agenda.
Writing

  * Entry
  #+BEGIN: clocktable :tstart "<2006-08-10 Thu 10:00>"
  ...
  #+END:

doesn't mean you want to display "Entry" in the agenda for today. It is
but a parameter to the clocktable that happens to use timestamp syntax.

It belongs neither to the first category nor the second one.

>> Yet, `org-at-timestamp-p' is something users are going to look after.
>> Different names may also introduce confusion (remember `org-at-regexp-p'
>> and `org-in-regexp-p'?). So, even if it looks like bad factoring to you,
>> a single predicate, or even two if we set "agenda" apart, with
>> a docstring explaining the different categories and how to select them
>> may also be a good natural choice. Hence my suggestion.
>
> Since org-at-timestamp-p already has an argument, adding a second one
> doesn't look appealing to me, especially when the first one would often
> be nil.  Maybe it's possible to change the definition of that argument
> (which would need the equivalence between t meaning 'inactive).
>
>> WDYT? Do you have any other concrete proposal?
>
> I really only need the clocktables to work again, which they'd be if
> they were in the second category, I gather.  I've monkeypatched
> org-at-timestamp-p with an ((org-at-block-p)) clause so I'm good for
> now, but per our discussion this wouldn't be an acceptable solution.

Indeed. So that doesn't qualify as a concrete proposal.

Regards,

-- 
Nicolas Goaziou

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
>> I'd say anything org-element-* should exclusively return syntactical
>> things.  Context dependence needs to be dealt with elsewhere.
>
> I'm not sure to understand this. Syntactical things are all about
> context dependence in Org. Do you mean /context independance/ needs to
> be dealt with elsewhere?

No, I meant context of application, rather than context in the
syntactical sense.  Org-element-* deals with syntax, nothing else.
Whether you need strict syntactical interpretation or something else
gets decided someplace else.

> Could you provide examples about that? In particular "providing
> timestamps as arguments to any processing functions" sounds odd, in
> particular from someone who cringes when I suggest to add a second
> optional argument to a single function.

Whatever it sounds like, what you want in the clocktables example and
the properties example (and elsewhere) is something that looks, walks
and talks like a timestamp if you'd put it into the proper context.  In
each of these places the text that looks like timestamp but isn't
(org-element says so) will be fed into some machinery that emerges with
a result that is indistinguishable from what you'd get if that text
would have been a proper timestamp and then uses that result to do
whatever it wants to do with it (i.e. most certainly not build up an
agenda, although it could do that as well).  It uses a bit of Org syntax
in the improper context to achieve this (and this requires precisely to
ignore that context or at least check with something more loose than
org-element).

>> If I'd follow that road, I could edit what looks like a timestamp
>> everywhere, regardless of context.  I don't know if that's the right
>> thing to do and I don't even expect consensus among the Org user base.
>> I personally see no need to do that.
>
> I do see it, tho. This is analogous to links in comments. If you see
> something looking like a timestamp (or a link), you can expect some
> commands to operate on it. This is exactly what is biting us at the
> moment.

In a comment that timestamp-looking text doesn't have any function, so
it's in a different category, I must insist.  As I said, I can see
somebody wanting to have this text be editable like any other timestamp
also, but it's really the other uses where it's used meta-syntactically
that I'd like to focus on.  ONe of the differences to text in comments
(or generally quoted material) is that there is an expectation that this
sort of timestamp is correct, since they are intended to be input to
further processing.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: More clocktable breakage

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> No, I meant context of application, rather than context in the
> syntactical sense.  Org-element-* deals with syntax, nothing else.
> Whether you need strict syntactical interpretation or something else
> gets decided someplace else.

OK. Then we agree here.

> Whatever it sounds like, what you want in the clocktables example and
> the properties example (and elsewhere) is something that looks, walks
> and talks like a timestamp if you'd put it into the proper context.  In
> each of these places the text that looks like timestamp but isn't
> (org-element says so) will be fed into some machinery that emerges with
> a result that is indistinguishable from what you'd get if that text
> would have been a proper timestamp and then uses that result to do
> whatever it wants to do with it (i.e. most certainly not build up an
> agenda, although it could do that as well).  It uses a bit of Org syntax
> in the improper context to achieve this (and this requires precisely to
> ignore that context or at least check with something more loose than
> org-element).

I also agree, but it seems to contradict what you write below.

> In a comment that timestamp-looking text doesn't have any function, so
> it's in a different category, I must insist.  As I said, I can see
> somebody wanting to have this text be editable like any other timestamp
> also, but it's really the other uses where it's used meta-syntactically
> that I'd like to focus on.

Here, I don't follow you anymore. A timestamp in a comment is "something
that looks, walks and talks like a timestamp if you'd put it into the
proper context", too. So there's no difference with properties or the
clock table.

> One of the differences to text in comments (or generally quoted
> material) is that there is an expectation that this sort of timestamp
> is correct, since they are intended to be input to further processing.

True, but if that timestamps isn't correct, it doesn't "look, walk and
talk" like a timestamp anymore, so this doesn't apply to the above.

Anyway, I think we're digressing. We're talking about design, yet, to
tell the truth, I don't even know anymore what the original, concrete,
problem is really about.

As I asked 5 weeks ago (!), could you provide an ECM demonstrating the
issue so that I can fix it, in the light of our discussion?


Regards,

-- 
Nicolas Goaziou

Re: More clocktable breakage

[Achim Gratz]

> As I asked 5 weeks ago (!), could you provide an ECM demonstrating the
> issue so that I can fix it, in the light of our discussion?

I've told you from the beginning that it was a file at work and that it
would take some time to dig down to the problem since it did work at
home when I tried to create said ECM.  After a few false starts
(thinking that it was Windows vs. Linux and the local configurations on
each side) I've finally converged to this (just insert a new clocktable,
then replace the default parameters with tstart/tend):

--8<---------------cut here---------------start------------->8---
  #+BEGIN: clocktable :tstart "<2017-03-24 Fri 00:00>" :tend "<2017-04-23 Sat 23:45>"
  #+CAPTION: Clock summary at [2017-05-06 Sat 11:09]
  | L | Headline     | Time   |   |
  |---+--------------+--------+---|
  |   | *Total time* | *0:00* |   |
  #+END:
--8<---------------cut here---------------end--------------->8---

Sometimes org-element-context recognizes the clocktable as a paragraph
instead of dynamic-block (I've not yet figured out why and it isn't vary
reproducible, but it must have something to do with the cache since it
goes away when I reload the file) and then 'timestamp gets returned when
the cursor is on one of the timestamps.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: More clocktable breakage

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> I've told you from the beginning that it was a file at work and that it
> would take some time to dig down to the problem since it did work at
> home when I tried to create said ECM.

I know, but I was hoping a few weeks would be enough, since the
resolution of the issue was stalled.

> After a few false starts (thinking that it was Windows vs. Linux and
> the local configurations on each side) I've finally converged to this
> (just insert a new clocktable, then replace the default parameters
> with tstart/tend):
>
>   #+BEGIN: clocktable :tstart "<2017-03-24 Fri 00:00>" :tend "<2017-04-23 Sat 23:45>"
>   #+CAPTION: Clock summary at [2017-05-06 Sat 11:09]
>   | L | Headline     | Time   |   |
>   |---+--------------+--------+---|
>   |   | *Total time* | *0:00* |   |
>   #+END:

OK. I inserted it in a fresh Org buffer. Is there any command to call on
it now?

> Sometimes org-element-context recognizes the clocktable as a paragraph
> instead of dynamic-block

FWIW, I get `dynamic-block'.

> (I've not yet figured out why and it isn't vary reproducible, but it
> must have something to do with the cache since it goes away when
> I reload the file)

It is possible, indeed. You can also use M-x org-element-cache-reload to
check this. However, cache is disabled by default, so the problem
shouldn't appear in normal usage.

I start to think that there is no bug in clock tables (but certainly in
the cache mechanism, probably related to some `before-change-functions'
and `after-change-functions' misuse there).

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
> OK. I inserted it in a fresh Org buffer. Is there any command to call on
> it now?

Yes, put the cursor on the date or time of one of the timestamps and
press S-Up or S-Down.  It should increase or decrease the corresponding
element of the timestamp, but instead you'll get an error message:

org-clocktable-shift: Line needs a :block definition before this command works

which appears because the timestamp wasn't recognized and the
fallthrough of org-shift* then tries to apply another function that
deals with the :block argument (which isn't present here and shouldn't
be).

>> Sometimes org-element-context recognizes the clocktable as a paragraph
>> instead of dynamic-block
>
> FWIW, I get `dynamic-block'.

OK, then that should get you the same error.

>> (I've not yet figured out why and it isn't vary reproducible, but it
>> must have something to do with the cache since it goes away when
>> I reload the file)
>
> It is possible, indeed. You can also use M-x org-element-cache-reload to
> check this. However, cache is disabled by default, so the problem
> shouldn't appear in normal usage.

I have not enabled any cache that I know of.  All I can say is that
sometimes the clocktable doesn't get recognized as dynamic-block but a
paragraph instead.  That re-enables the recognition of the timestamp
incidentally (why exactly I don't really understand), which was why I
couldn't reproduce the error at home for some time.

> I start to think that there is no bug in clock tables (but certainly in
> the cache mechanism, probably related to some `before-change-functions'
> and `after-change-functions' misuse there).

I'm not using any of those unless they already come with Emacs or Org.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: More clocktable breakage

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> Yes, put the cursor on the date or time of one of the timestamps and
> press S-Up or S-Down.  It should increase or decrease the corresponding
> element of the timestamp, but instead you'll get an error message:
>
> org-clocktable-shift: Line needs a :block definition before this command works
>
> which appears because the timestamp wasn't recognized and the
> fallthrough of org-shift* then tries to apply another function that
> deals with the :block argument (which isn't present here and shouldn't
> be).

OK, reproduced.

I fixed the issue by extending `org-at-timestamp-p' optional argument
while preserving backward-compatibility.

The new docstring is:

    Non-nil if point is inside a timestamp.

    By default, the function only consider syntactically valid active
    timestamps.  However, the caller may have a broader definition
    for timestamps.  As a consequence, optional argument EXTENDED can
    be set to the following values

      `inactive'

        Include also syntactically valid inactive timestamps.

      `agenda'

        Include timestamps allowed in Agenda, i.e., those in
        properties drawers, planning lines and clock lines.

      `lax'

        Ignore context.  The function matches any part of the
        document looking like a timestamp.  This includes comments,
        example blocks...

    For backward-compatibility with Org 9.0, every other non-nil
    value is equivalent to `inactive'.

    When at a timestamp, return the position of the point as a symbol
    among `bracket', `after', `year', `month', `hour', `minute',
    `day' or a number of character from the last know part of the
    time stamp.

    When matching, the match groups are the following:
      group 1: year
      group 2: month
      group 3: day number
      group 4: day name
      group 5: hours, if any
      group 6: minutes, if any

I also updated the callers throughout the code base.

>> I start to think that there is no bug in clock tables (but certainly in
>> the cache mechanism, probably related to some `before-change-functions'
>> and `after-change-functions' misuse there).
>
> I'm not using any of those unless they already come with Emacs or Org.

What I meant is the use of `before-change-functions' and
`after-change-functions' is wrong in the caching mechanism, not in your
configuration.

Anyway, it doesn't matter for the problem at hand.

Is your issue solved?

Regards,

-- 
Nicolas Goaziou

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
> I fixed the issue by extending `org-at-timestamp-p' optional argument
> while preserving backward-compatibility.

Thanks.  That seems like a much better soultion than what we've had
before and some of what we've discussed before getting there.

> Is your issue solved?

I'll have final confirmation sometime next week, but I assume yes.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: More clocktable breakage

[Achim Gratz]

Nicolas Goaziou writes:
> Is your issue solved?

I can confirm the issue is solved.  Thanks again.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada