BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Robert Klein]

[-- Attachment #1: Type: text/plain, Size: 1840 bytes --]

Hi everyone,
 a bug report:


Emacs  : GNU Emacs 27.2 (build 1, x86_64-unknown-openbsd, GTK+ Version 2.24.33)
 of 2021-04-20
Package: Org mode version 9.4.6 (release_9.4.6-4-g093c94 @ /home/roklein/.emacs.d/org-mode/lisp/)


When trying to run a table with missing values with ob-gnuplot.el
there's a “:missing value” option.  ob-gnuplot is supposed to put the
“value” into the place of empty cell values before calling on gnuplot.
It doesn't do this currently.  Bi-secting suggests,
“9209aa3c9d456bcfdd9d8cd223367f7cfb6b3231 is the first bad commit”:

--  commit 9209aa3c9d456bcfdd9d8cd223367f7cfb6b3231
--  Author: Nicolas Goaziou <mail@nicolasgoaziou.fr>
--  --  Sun Aug 24 01:31:56 2014 +0200
--
--  org-table: Use "ox.el" internally for radio tables


By my reading:
- in ob-gnuplot.el:org-babel-gnuplot-process-vars :missing variable is
  read, used in
- ob-gnuplot.el:org-babel-gnuplot-quote-tsv-field, called from
- ob-gnuplot.el:org-babel-gnuplot-table-to-data
- used in org-table.el:orgtbl-to-generic (as (part of the) :fmt param)
- which calls org-table--to-generic-cell
- which calle org-table.el:org-table--generic-apply
- which I don't understand

my dotemacs and a sample org file are attached.

When running the code (e.g. C-c C-c on the #+Call: line), temporary
files are created in /tmp/babel-<random>/gnuplot<other-random>.

These files should have "?" (as choosen :missing).  In the current
version there is nothing, so gnuplot uses up the values in the following
columns and the resulting graph is wrong.

Also, I couldn't find this in the manual, but there's a thread “using
gnuplot's "splot" and "every" comma nds on org-mode table” on the
mailing list (2013/09/23 - 3013/09/25) about the :missing feature.

Thanks for any help.

Best regards
Robert


[-- Attachment #2: gp.org --]
[-- Type: application/octet-stream, Size: 8311 bytes --]

# -*- fill-column: 72; coding: utf-8-unix;  -*-

#+call: create_health_monitor_plot2()

#+RESULTS:
: nil

* blood pressure 2021

#+Name: bdq2_21
| Datum | sys. | dia. | pul. | sys.2 | dia.2 | pul.2 |    5ds |   5dd |   5ds2 |  5dd2 |
|-------+------+------+------+-------+-------+-------+--------+-------+--------+-------|
| 23.3. |      |      |      |   121 |    79 |    80 | 120.50 | 79.50 | 120.00 | 80.00 |
| 24.3. |  120 |   80 |   60 |   120 |    80 |    80 | 121.00 | 79.00 | 119.50 | 80.50 |
| 25.3. |  121 |   79 |   60 |   119 |    81 |    80 | 121.50 | 78.50 | 119.00 | 81.00 |
| 26.3. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
| 27.3. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
| 28.3. |  124 |   76 |   60 |   116 |    84 |    80 | 122.80 | 77.20 | 117.20 | 82.80 |
| 29.3. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
| 30.3. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
| 31.3. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.60 | 81.40 |
|  1.4. |  120 |   80 |   60 |   120 |    80 |    80 | 121.20 | 78.80 | 118.80 | 81.20 |
|  2.4. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.60 | 81.40 |
|  3.4. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
|  4.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
|  5.4. |  124 |   76 |   60 |   116 |    84 |    80 | 122.80 | 77.20 | 117.20 | 82.80 |
|  6.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.40 | 82.60 |
|  7.4. |  122 |   78 |   60 |   118 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.00 |
|  8.4. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.60 | 81.40 |
|  9.4. |  120 |   80 |   60 |   120 |    80 |    80 | 121.20 | 78.80 | 119.00 | 81.00 |
| 10.4. |  121 |   79 |   60 |   119 |    81 |    80 | 121.40 | 78.60 | 118.75 | 81.25 |
| 11.4. |  122 |   78 |   60 |       |       |       | 122.00 | 78.00 | 118.00 | 82.00 |
| 12.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.25 | 82.75 |
| 13.4. |  124 |   76 |   60 |   116 |    84 |    80 | 122.80 | 77.20 | 117.25 | 83.00 |
| 14.4. |  123 |   77 |   60 |   117 |    83 |    80 | 122.60 | 77.40 | 117.25 | 83.00 |
| 15.4. |  122 |   78 |   60 |   119 |    82 |    80 | 122.00 | 78.00 | 118.00 | 82.25 |
| 16.4. |  121 |   79 |   60 |       |       |       | 121.50 | 78.50 | 118.67 | 81.67 |
| 17.4. |  120 |   80 |   60 |   120 |    80 |    80 | 121.00 | 79.00 | 119.50 | 81.00 |
#+TBLFM: @>$9=if("@-2$2..@>$3" == "[]", string(""), vmean(@-2$3..@>$3));%.2f::@>>$9=if("@-2$3..@+1$3" == "[]", string(""), vmean(@-2$3..@+1$3));%.2f::@>$8=if("@-2$2..@>$2" == "[]", string(""), vmean(@-2$2..@>$2));%.2f::@>>$8=if("@-2$2..@+1$2" == "[]", string(""), vmean(@-2$2..@+1$2));%.2f::@>$10=if("@-2$5..@>$5" == "[]", string(""), vmean(@-2$5..@>$5));%.2f::@>>$10=if("@-2$5..@+1$5" == "[]", string(""), vmean(@-2$5..@+1$5));%.2f::@>$11=if("@-2$6..@>$6" == "[]", string(""), vmean(@-2$6..@>$6));%.2f::@>>$11=if("@-2$6..@+1$6" == "[]", string(""), vmean(@-2$6..@+1$6));%.2f::@2$8=if("@2$2..@+2$2" == "[]", string(""), vmean(@2$2..@+2$2));%.2f::@2$9=if("@2$3..@+2$3" == "[]", string(""), vmean(@2$3..@+2$3));%.2f::@2$10=if("@2$5..@+2$5" == "[]", string(""), vmean(@2$5..@+2$5));%.2f::@2$11=if("@2$6..@+2$6" == "[]", string(""), vmean(@2$6..@+2$6));%.2f::@3$8=if("@-1$2..@+2$2" == "[]", string(""), vmean(@-1$2..@+2$2));%.2f::@3$9=if("@-1$3..@+2$3" == "[]", string(""), vmean(@-1$3..@+2$3));%.2f::@3$10=if("@-1$5..@+2$5" == "[]", string(""), vmean(@-1$5..@+2$5));%.2f::@3$11=if("@-1$6..@+2$6" == "[]", string(""), vmean(@-1$6..@+2$6));%.2f::@4$8..@>>>$8=if("@-2$2..@+2$2" == "[]", string(""), vmean(@-2$2..@+2$2));%.2f::@4$9..@>>>$9=if("@-2$3..@+2$3" == "[]", string(""), vmean(@-2$3..@+2$3));%.2f::@4$10..@>>>$10=if("@-2$5..@+2$5" == "[]", string(""), vmean(@-2$5..@+2$5));%.2f::@4$11..@>>>$11=if("@-2$6..@+2$6" == "[]", string(""), vmean(@-2$6..@+2$6));%.2f




#+name: create_health_monitor_plot2
#+BEGIN_SRC gnuplot "," :missing "?" :var data=bdq2_21 :results silent
  set terminal png

  # get stats data
  stats data
  set terminal png enhanced giant size 1280,768

  set output 'q2-1.png'

  ## Define X Axis
  # globally set the time format for input data
  set timefmt "%d.%m."
  # X coordinates represent time values (here: date)
  set xdata time
  # set format of X axis labels to "%d."
  set xtics timedate format "%d.%m."
  # no minor tics on X Axis
  #unset mxtics
  set mxtics 14
  # don't extend x-axis beyond x values in data
  set autoscale noextend

  ## Define Y axis
  # Tics and microtics, so each integer y value gets a (micro)tic
  set ytics 10 out nomirror
  set mytics 10
  set yrange [60:200]
  set ylabel "blood pressure (mmHg)"
  # labeled tics also on the  right
  set y2tics 10 out
  set mytics 10

  ## Tics on both Axis outward
  set grid xtics ytics front

  # missing data
  set datafile missing "?"

  # set style fill  transparent solid 0.50 noborder
  # plot "~/mm.txt" u 1:3 axes x1y1 title "mg/dL"

  ## colors
  # darkred: 8b0000
  # firebrick: b22222
  # crimson: dc143c
  # coral: ff7f50
  # peachpuff: ffdab9
  # palegoldenrod: eee8aa
  # palegreen: 98fb98

  plot \
       400 axes x1y1 with filledcurves y2=180 fc "#b22222" notitle, \
       180 axes x1y1 with filledcurves y2=160 fc "#dc143c" notitle, \
       160 axes x1y1 with filledcurves y2=140 fc "#ff7f50" notitle, \
       140 axes x1y1 with filledcurves y2=130 fc "#ffdab9" notitle, \
       130 axes x1y1 with filledcurves y2=120 fc "#eee8aa" notitle, \
       120 axes x1y1 with filledcurves y2=105 fc "#98fb98" notitle, \
       105 axes x1y1 with filledcurves y2=100 fc "#dc143c" notitle, \
       \
       100 axes x1y1 with filledcurves y2=90  fc "#ff7f50" notitle, \
       90  axes x1y1 with filledcurves y2=85  fc "#ffdab9" notitle, \
       85  axes x1y1 with filledcurves y2=80  fc "#eee8aa" notitle, \
       80  axes x1y1 with filledcurves y2=65  fc "#98fb98" notitle, \
       60  axes x1y1 with filledcurves y2=65  fc "#dc143c" notitle, \
       data u 1:2 axes x1y1 with lines lc "dark-green" lw 2 title "syst. blood pressure (am)", \
       data every ::0::1                             u 1:8 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
       data every ::1::2                             u 1:8 axes x1y1  \
            with lines lc "gray60" lw 1 notitle, \
       data every ::2::STATS_records-3               u 1:8 axes x1y1 \
            with lines lc "gray30" lw 1 title "5-day moving syst.", \
       data every ::STATS_records-3::STATS_records-2 u 1:8 axes x1y1 \
            with lines lc "gray60" lw 1 notitle, \
       data every ::STATS_records-2::STATS_records-1 u 1:8 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
     \
       data u 1:3 axes x1y1 with lines lc "dark-blue" lw 2 title "diast. blood pressure (am)", \
       data every ::0::1                             u 1:9 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
       data every ::1::2                             u 1:9 axes x1y1  \
            with lines lc "gray60" lw 1 notitle, \
       data every ::2::STATS_records-3               u 1:9 axes x1y1 \
            with lines lc "gray30" lw 1 title "5-day moving diast.", \
       data every ::STATS_records-3::STATS_records-2 u 1:9 axes x1y1 \
            with lines lc "gray60" lw 1 notitle, \
       data every ::STATS_records-2::STATS_records-1 u 1:9 axes x1y1 \
            with lines lc "gray" lw 1 notitle, \
     \
       data u 1:($5) axes x1y1 with lines lc "green" lw 2 title "syst. blood pressure (pm)", \
     \
       data u 1:($6) axes x1y1 with lines lc "blue" lw 2 title "diast. blood pressure (pm)", \
     \

  #     data every ::0::1                             u 1:2 axes x1y1 \
  #          with lines lc "gray" lw 1 notitle, \
  #     data every ::1::2                             u 1:2 axes x1y1  \
  #          with lines lc "gray60" lw 1 notitle, \
  #     data every ::2::STATS_records-3               u 1:2 axes x1y1 \
  #          with lines lc "gray30" lw 1 title "5-day moving syst.", \

  quit
#+END_SRC

#+Results: create_health_monitor_plot2

[-- Attachment #3: dotemacs --]
[-- Type: application/octet-stream, Size: 1180 bytes --]

;; -*- lisp-mode:t; -*-
;; get rid of C-z
;;(unbind-key "C-z")
(global-unset-key (kbd "C-z"))
;; no startup message
(setq inhibit-startup-message t)

;; no toolbar
(if (not (equal system-configuration "aarch64-unknown-linux-android"))
    (tool-bar-mode -1))

;; I'm feeling religious about this:
(setq-default indent-tabs-mode nil)
;; changed my mind back.
;; (setq-default indent-tabs-mode t)

;;  ask y or n instead of yes or no
(fset 'yes-or-no-p 'y-or-n-p)

;; my ears hurt. Especially when wearing headphones
(setq visible-bell t)

;; color-theme
(setq-default custom-enabled-themes (quote (adwaita)))
(load-theme 'adwaita)

(setq load-path (cons "~/gp/org-mode/lisp" load-path))
(add-to-list 'auto-mode-alist '("\\.\\(org\\|org_archive\\|txt\\)$" . org-mode))
(add-hook 'org-mode-hook (lambda () (electric-indent-local-mode -1)))

(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (gnuplot . t)))

(add-to-list 'load-path
             (expand-file-name "~/gp/gnuplot-0.8.0"))
  ;; autoload gnuplot-mode
  ;(autoload 'gnuplot-mode "gnuplot" "gnuplot major mode" t)
  ;(autoload 'gnuplot-make-buffer "gnuplot" "open a buffer in gnuplot-mode" t)

Re: BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Robert Klein <roklein@roklein.de> writes:

> When trying to run a table with missing values with ob-gnuplot.el
> there's a “:missing value” option.  ob-gnuplot is supposed to put the
> “value” into the place of empty cell values before calling on gnuplot.
> It doesn't do this currently.

Confirmed

The problem is partially because of the way `org-table--to-generic-cell'
works. It disallows custom processing of empty table cells unless both
:raw and :backend arguments are supplied to `orgtbl-to-generic'.

The attached patch should fix the issue, though it may be more
reasonable to fix orgtbl-to-generic logic.

Nicolas,

Is there a rationale behind bypassing :fmt and :hfmt for empty table
cells in `org-table--to-generic-cell'? In this patch, I have to work
around this using :raw parameter in `orgtbl-to-generic' call. I do not
feel like such workaround should be needed.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-gnuplot-Honour-missing-argument.patch --]
[-- Type: text/x-diff, Size: 1268 bytes --]

From 7033289d7e717eb841d550fbdbc0cc878b96cd32 Mon Sep 17 00:00:00 2001
Message-Id: <7033289d7e717eb841d550fbdbc0cc878b96cd32.1633005978.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Thu, 30 Sep 2021 20:40:37 +0800
Subject: [PATCH] ob-gnuplot: Honour :missing argument

* lisp/ob-gnuplot.el (org-babel-gnuplot-table-to-data): Force applying
`org-babel-gnuplot-quote-tsv-field' to empty table cells when
converting org tables to gnuplot input.  By default,
`orgtbl-to-generic' ignores :fmt for empty table cells.  We need to
set :raw to non-nil AND :backend to non-nil to force custom formatting
of empty table cells.

Fixes https://orgmode.org/list/20210617211347.59e4de56@happy.intern.roklein.de
---
 lisp/ob-gnuplot.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ob-gnuplot.el b/lisp/ob-gnuplot.el
index 6489c23f5..120aaa18e 100644
--- a/lisp/ob-gnuplot.el
+++ b/lisp/ob-gnuplot.el
@@ -272,7 +272,7 @@ (defun org-babel-gnuplot-table-to-data (table data-file params)
 	      (orgtbl-to-generic
 	       table
 	       (org-combine-plists
-		'(:sep "\t" :fmt org-babel-gnuplot-quote-tsv-field)
+		'(:sep "\t" :fmt org-babel-gnuplot-quote-tsv-field :raw t :backend ascii)
 		params)))))
   data-file)
 
-- 
2.32.0

Re: BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> Nicolas,
>
> Is there a rationale behind bypassing :fmt and :hfmt for empty table
> cells in `org-table--to-generic-cell'? In this patch, I have to work
> around this using :raw parameter in `orgtbl-to-generic' call. I do not
> feel like such workaround should be needed.

I'm not sure. I wrote this some years ago. I guess the rationale at that
time was that it didn't feel useful to apply a format to an empty cell.
For example, if you use `:fmt "$%s$"', I assume you wouldn't want to get
"$$" in otherwise empty cells.

Regards,
-- 
Nicolas Goaziou

Re: BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> The attached patch should fix the issue, though it may be more
> reasonable to fix orgtbl-to-generic logic.

Given Nicholas' reply, changing orgtbl-to-generic may not be a good idea
and break other code using orgtbl-to-generic.

Applied as ddee7b617 on bugfix

Best,
Ihor

----

Fixed

Re: BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 589 bytes --]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> I'm not sure. I wrote this some years ago. I guess the rationale at that
> time was that it didn't feel useful to apply a format to an empty cell.
> For example, if you use `:fmt "$%s$"', I assume you wouldn't want to get
> "$$" in otherwise empty cells.

Also makes sense.

At this point, third-party code is likely to rely on the existing logic,
so I do not see any reason to insist on changing org-table.

To avoid confusion, can we change the docstring explicitly saying that
empty cells are ignored? (see the attached)

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-orgtbl-to-generic-Mention-that-fmt-is-ignored-for-em.patch --]
[-- Type: text/x-diff, Size: 1018 bytes --]

From 60365a164147dfc71f0047661c143b54064a459b Mon Sep 17 00:00:00 2001
Message-Id: <60365a164147dfc71f0047661c143b54064a459b.1633060596.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Fri, 1 Oct 2021 11:40:54 +0800
Subject: [PATCH] orgtbl-to-generic: Mention that :fmt is ignored for empty
 cells

* lisp/org-table.el (orgtbl-to-generic): Amend docstring mentioning
that :fmt is ignored for empty table cells.
---
 lisp/org-table.el | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 89c57fb06..3fed15054 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -5649,6 +5649,9 @@ (defun orgtbl-to-generic (table params)
 
     (:fmt (2 \"$%s$\" 4 (lambda (c) (format \"$%s$\" c))))
 
+  The format is ignored for empty fields.  Use :raw t with non-nil
+  :backend option to force formatting empty fields.
+
 :hlstart :hllstart :hlend :hllend :hsep :hlfmt :hllfmt :hfmt
 
  Same as above, specific for the header lines in the table.
-- 
2.32.0

Re: BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Nicolas Goaziou]

Hello,

Ihor Radchenko <yantar92@gmail.com> writes:

> At this point, third-party code is likely to rely on the existing logic,
> so I do not see any reason to insist on changing org-table.

If you think it is overall an improvement, feel free to discuss it. I'm
merely pointing out what was the initial intent, not necessarily how it
should be.

> To avoid confusion, can we change the docstring explicitly saying that
> empty cells are ignored? (see the attached)

Sure. Thanks.

Regards,
-- 
Nicolas Goaziou

Re: BUG: org-table: table option “:missing” not working (in ob-gnuplot.el)

[Ihor Radchenko]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> At this point, third-party code is likely to rely on the existing logic,
>> so I do not see any reason to insist on changing org-table.
>
> If you think it is overall an improvement, feel free to discuss it. I'm
> merely pointing out what was the initial intent, not necessarily how it
> should be.

I feel that it would be more consistent, though not necessary an
improvement.  Special formatting for empty cells is probably a rare
need---much more rare compared to skipping them.  I am fine with current
behaviour if it is documented.  It's just that it caught me by surprise
while fixing the current bug report.

>> To avoid confusion, can we change the docstring explicitly saying that
>> empty cells are ignored? (see the attached)

> Sure. Thanks.

Applied as 60365a164 on main.

Best,
Ihor