Plotting a table of date+time pairs

Plotting a table of date+time pairs

[Michael Heerdegen]

Hi,

I want to plot two columns of an org table: the first column (the "x"
coordinate) consists of dates, the second column contains a time value
associated with each date.

It seems this is not as easy to do as I had hoped: AFAIU, for making
gnuplot interpret the "y"-value (!) as a time value I need to add
something like

  using "1:(timecolumn(2,\"%H:%M\"))"

to the gnuplot command line (it took me quite a while to find out how to
do it at all...)

And AFAIK this is only possible by tweaking
`org-plot/preset-plot-types'.  I need to add a complete new type of
plot!

Is there maybe a simpler way (I know I could add redundant columns to
the table to help org - that's not really what I want, though)?


TIA,

Michael.

Re: Plotting a table of date+time pairs

[Fraga, Eric]

Response below/inline for email Michael Heerdegen wrote:
> (original email sent 28 Nov 2024 at 02:21)
> 
> [...]
> And AFAIK this is only possible by tweaking
> `org-plot/preset-plot-types'.  I need to add a complete new type of
> plot!

You could use a gnuplot src block instead of org-plot?  It's what I do
usually as org-plot is not powerful enough for most of my plots
(although very nice for simple x-y graphs etc.).

-- 
: Eric S Fraga, with org 9.7.15-a1df10 in Emacs 31.0.50

Re: Plotting a table of date+time pairs

[Michael Heerdegen]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> You could use a gnuplot src block instead of org-plot?  It's what I do
> usually as org-plot is not powerful enough for most of my plots
> (although very nice for simple x-y graphs etc.).

Ah ok, thank you.  Sounds reasonable.

I'm not a sophisticated org user.  May I ask for a simple example,
and how I would have to invoke gnuplot instead?


Thank you very much,

Michael.

Re: Plotting a table of date+time pairs

[Fraga, Eric]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Response below/inline for email Michael Heerdegen wrote:
> (original email sent 29 Nov 2024 at 00:17)
> 
> I'm not a sophisticated org user.  May I ask for a simple example,
> and how I would have to invoke gnuplot instead?

Sure, see attached.  This example has a table of data to plot (simple
example with two columns) and the gnuplot src block to create a PDF
plot.  You invoke gnuplot by typing C-c C-c in the src block.  You can
then view the result with C-c C-o.

Check the info manual for org, specifically
(org) Working with Source Code

HTH,
eric
-- 
: Eric S Fraga, with org 9.7.15-a1df10 in Emacs 31.0.50

[-- Attachment #2: test.org --]
[-- Type: application/vnd.lotus-organizer, Size: 550 bytes --]

Re: Plotting a table of date+time pairs

[Max Nikulin]

On 29/11/2024 06:17, Michael Heerdegen wrote:
> I'm not a sophisticated org user.  May I ask for a simple example,
> and how I would have to invoke gnuplot instead?

<https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-gnuplot.html>

Link to its source is incorrect, it should be
<https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-gnuplot.org>

Re: Plotting a table of date+time pairs

[Michael Heerdegen]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> Sure, see attached.  This example has a table of data to plot (simple
> example with two columns) and the gnuplot src block to create a PDF
> plot.  You invoke gnuplot by typing C-c C-c in the src block.  You can
> then view the result with C-c C-o.
> [...]

Ok... I had to (require 'ob-gnuplot), but then the example worked fine.
It took a while but I got my own table work as well, more or less (see
below).

Thank you very much!


"Max Nikulin" <manikulin@gmail.com> writes:

> <https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-gnuplot.html>

Very nice overview and reference, thank you very much as well.

I now have one remaining problem: my time values in the second column
contain the name of the time zone - the purpose is to flag the time as
summer or winter time.  Currently I need to remove these indicators from
the table, else plotting fail, gnuplot doesn't understand the data.

Can I tell "ob-gnuplot" to export the table in a different way - how?

The time zone makes a difference when exporting since the generated
data file has a different format: the time fields including the timezone
get wrapped in quotation marks, without time zone they are exported
literally.


Thanks again,

Michael.

Re: Plotting a table of date+time pairs

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> I now have one remaining problem: my time values in the second column
> contain the name of the time zone - the purpose is to flag the time as
> summer or winter time.  Currently I need to remove these indicators from
> the table, else plotting fail, gnuplot doesn't understand the data.

I could adjust `org-table-number-regexp' - this makes plotting succeed.
Unfortunately does "ob-gnuplot" not respect a buffer local binding of
that variable (in the org buffer containing the table).  I need to
change the global binding of that variable.


Michael.

Re: Plotting a table of date+time pairs

[Fraga, Eric]

Response below/inline for email Michael Heerdegen wrote:
> (original email sent 30 Nov 2024 at 07:17)
>
> I now have one remaining problem: my time values in the second column
> contain the name of the time zone - the purpose is to flag the time as
> summer or winter time.  Currently I need to remove these indicators from
> the table, else plotting fail, gnuplot doesn't understand the data.

You can specify the format of dates/times in gnuplot and then manipulate
them.  See "time/date specifiers" in the gnuplot info manual and also
read about the strptime() function.  I don't have any examples of these
at hand unfortunately.

-- 
: Eric S Fraga, with org 9.7.15-a1df10 in Emacs 31.0.50

Re: Plotting a table of date+time pairs

[Michael Heerdegen]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> > I now have one remaining problem: my time values in the second column
> > contain the name of the time zone - the purpose is to flag the time as
> > summer or winter time.  Currently I need to remove these indicators from
> > the table, else plotting fail, gnuplot doesn't understand the data.
>
> You can specify the format of dates/times in gnuplot and then manipulate
> them.  See "time/date specifiers" in the gnuplot info manual and also
> read about the strptime() function.  I don't have any examples of these
> at hand unfortunately.

Yes thanks - but I really think Org gets it wrong.  Look:
With

| plot data using 1:(timecolumn(2,"%H:%M") ...

gnuplot happily accepts a data file looking like

| 2024-09-18      11:40 CEST
| 2024-09-19      11:27 CEST
|    ...             ...

It just discards the timezone which is good enough.  No problem with the
kind of data for gnuplot (AFAIR I read in the gnuplot manual that time
zones are not supported ad-hoc - anyway, I don't need this for
plotting.)

But the real problem is that the org table export mechanism thinks that
the time field doesn't look like a numerical field, so it wraps every
such field in quotes, and the exported data file will look like

| 2024-09-18      "11:40 CEST"
| 2024-09-19      "11:27 CEST"
|    ...             ...

instead.  Then gnuplot doesn't find data matching "%H:%M in the second
column of the data file and interprets it as zero or 00:00am or
whatever.

Can I make gnuplot disregard the quotation marks?  Else I must tell org
that these fields should not be quoted.  But ironically the only way I
know to do this is again a hack: I must adjust
`org-table-number-regexp' which... sounds like a good solution but since
org disregards a file local binding I must change the global binding of
the variable!  Ugh.  Better solutions welcome.


Thanks,

Michael.

Re: Plotting a table of date+time pairs

[Fraga, Eric]

Response below/inline for email Michael Heerdegen wrote:
> (original email sent  4 Dec 2024 at 11:50)
> 
> But the real problem is that the org table export mechanism thinks that
> the time field doesn't look like a numerical field, so it wraps every
> such field in quotes, and the exported data file will look like
>
> | 2024-09-18      "11:40 CEST"
> | 2024-09-19      "11:27 CEST"

Is it possible to specify a format to timecolumn that includes both the
quotes and the time zone?  I'm thinking something along the lines of

 ... timecolumn(2, "\"%H:%M %Z\"")

(untested)

-- 
: Eric S Fraga, with org 9.7.15-a1df10 in Emacs 31.0.50

Re: Plotting a table of date+time pairs

[Michael Heerdegen]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> > But the real problem is that the org table export mechanism thinks that
> > the time field doesn't look like a numerical field, so it wraps every
> > such field in quotes, and the exported data file will look like
> >
> > | 2024-09-18      "11:40 CEST"
> > | 2024-09-19      "11:27 CEST"
>
> Is it possible to specify a format to timecolumn that includes both the
> quotes and the time zone?  I'm thinking something along the lines of
>
>  ... timecolumn(2, "\"%H:%M %Z\"")
>
> (untested)

Tested!

The escaped quotes seem to be recognized literally as expected.  This
saves me from my org hack!

Unfortunately %Z is ignored - I think it's only significant for output:

|  [...] However this default can be replaced when reading any
|  particular file or column of input using the timecolumn function in
|  the corresponding using specifier.
|
|  The conversion to and from seconds assumes Universal Time (which is
|  the same as Greenwich Standard Time). There is no provision for
|  changing the time zone or for daylight savings. If all your data
|  refer to the same time zone (and are all either daylight or standard)
|  you don’t need to worry about these things. But if the absolute time
|  is crucial for your application, you’ll need to convert to UT
|  yourself.

I thought maybe %z, which is meant to support a time zone in [+/-]mm:hh
format, could work, but parsing completely fails when using it.  Maybe
it's not implemented for input, not even as "ignore that part".

But that's not a big deal, I'm happy with the version that works.  You
were a great help, Eric.


Thanks again,

Michael.

Re: Plotting a table of date+time pairs

[Fraga, Eric]

Response below/inline for email Michael Heerdegen wrote:
> (original email sent  4 Dec 2024 at 14:47)
> 
> Tested!
>
> The escaped quotes seem to be recognized literally as expected.  This
> saves me from my org hack!

Excellent!

> Unfortunately %Z is ignored - I think it's only significant for output:

Yeah, I had a feeling that would be the case.  The documentation implied
this by omission.

> But that's not a big deal, I'm happy with the version that works.  You
> were a great help, Eric.

Happy to have been able to help.  gnuplot is a brilliant tool but
sometimes difficult to get exactly what you want due to its rather
idiosyncratic syntax (that comes from the organic growth of features).

But ever so powerful (like org) once you get comfortable with the
various features.

-- 
: Eric S Fraga, with org 9.7.15-a1df10 in Emacs 31.0.50

Re: Plotting a table of date+time pairs

[Michael Heerdegen]

"Fraga, Eric" <e.fraga@ucl.ac.uk> writes:

> > The escaped quotes seem to be recognized literally as expected.  This
> > saves me from my org hack!
>
> Excellent!

Indeed.

> Happy to have been able to help.  gnuplot is a brilliant tool but
> sometimes difficult to get exactly what you want due to its rather
> idiosyncratic syntax (that comes from the organic growth of features).
>
> But ever so powerful (like org) once you get comfortable with the
> various features.

I'll try to use it a bit more, also for math.  It's a bit confusing that
it's not "Gnu" software despite its name.

And one of the versions that ships with Debian stable (there a multiple
- AFAIR it was the "gnuplot-x11" named one) was terribly broken - when I
started gnuplot it ate all of my free RAM memory, lots of GiB, in five
seconds.  Had to restart my computer.


Michael.

Re: Plotting a table of date+time pairs

[Ihor Radchenko]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> But the real problem is that the org table export mechanism thinks that
> the time field doesn't look like a numerical field, so it wraps every
> such field in quotes, and the exported data file will look like
>
> | 2024-09-18      "11:40 CEST"
> | 2024-09-19      "11:27 CEST"
> |    ...             ...
> ...

Did you try using ob-gnuplot?

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Plotting a table of date+time pairs

[Michael Heerdegen]

Ihor Radchenko <yantar92@posteo.net> writes:

> Michael Heerdegen <michael_heerdegen@web.de> writes:
>
> > But the real problem is that the org table export mechanism thinks that
> > the time field doesn't look like a numerical field, so it wraps every
> > such field in quotes, and the exported data file will look like
> >
> > | 2024-09-18      "11:40 CEST"
> > | 2024-09-19      "11:27 CEST"
> > |    ...             ...
> > ...
>
> Did you try using ob-gnuplot?

Yes - I was using ob-gnuplot approximately in the complete second half
of this thread.  And it is indeed the better choice for my case.

The quoting problem you quoted above originates from org's table export,
though.  This quoting is hard to avoid, not without a hack, and
independent from the used "export tool".

My final solution for this little remaining problem is to tell gnuplot
to expect a time value wrapped in quotes:

| plot data using 1:(timecolumn(2,"\"%H:%M\"")) with ...
                                   ^^     ^^

Thx,

Michael.