working with tables can be quite painful...

working with tables can be quite painful...

[Eric S Fraga]

Hello,

I am working with a table.  It has approximately 130 rows and 20 columns
so it's not huge but also not small.  Three columns are text but the
rest are all numbers with some degree of sparsity.  Instrumenting org
while working on this table, manipulating the entries in just one row, I
get the following:

| Function                 | calls | elapsed time | average time |
|--------------------------+-------+--------------+--------------|
| org-cycle                |    20 | 41.550963140 | 2.0775481570 |
| org-table-next-field     |    20 | 41.544266727 | 2.0772133363 |
| org-table-align          |     5 | 41.470595702 | 8.2941191404 |
| org-mode-flyspell-verify |    52 | 1.0647362189 | 0.0204756965 |
| org-do-latex-and-related |    21 | 0.6656267140 | 0.0316965101 |
| org-element-at-point     |   125 | 0.6356939890 | 0.0050855519 |
| org-element--parse-to    |   125 | 0.6086256940 | 0.0048690055 |
| org-element--cache-put   |  1399 | 0.4963533770 | 0.0003547915 |

From this, it would seem that the table align function is killing the
performance.  8 seconds per call?  On an 8 core Intel(R) Core(TM)
i7-2760QM CPU @ 2.40GHz... so not my wee Pandora where I expect
slowness!

This is with a not quite up to date org.  I'm avoiding upgrading as I am
preparing material for teaching which starts soon and I don't want to
run into issues due to changes in org... so I apologise if things have
changed recently. This performance issue has existed for quite some time
now, however.

Any suggestions on speeding things up?

Thanks,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.94.1, Org release_8.3.5-1070-g190476

Re: working with tables can be quite painful...

[Nicolas Goaziou]

Hello,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> I am working with a table.  It has approximately 130 rows and 20 columns
> so it's not huge but also not small.  Three columns are text but the
> rest are all numbers with some degree of sparsity.  Instrumenting org
> while working on this table, manipulating the entries in just one row, I
> get the following:
>
> | Function                 | calls | elapsed time | average time |
> |--------------------------+-------+--------------+--------------|
> | org-cycle                |    20 | 41.550963140 | 2.0775481570 |
> | org-table-next-field     |    20 | 41.544266727 | 2.0772133363 |
> | org-table-align          |     5 | 41.470595702 | 8.2941191404 |
> | org-mode-flyspell-verify |    52 | 1.0647362189 | 0.0204756965 |
> | org-do-latex-and-related |    21 | 0.6656267140 | 0.0316965101 |
> | org-element-at-point     |   125 | 0.6356939890 | 0.0050855519 |
> | org-element--parse-to    |   125 | 0.6086256940 | 0.0048690055 |
> | org-element--cache-put   |  1399 | 0.4963533770 | 0.0003547915 |
>
> From this, it would seem that the table align function is killing the
> performance.  8 seconds per call?  On an 8 core Intel(R) Core(TM)
> i7-2760QM CPU @ 2.40GHz... so not my wee Pandora where I expect
> slowness!
>
> This is with a not quite up to date org.  I'm avoiding upgrading as I am
> preparing material for teaching which starts soon and I don't want to
> run into issues due to changes in org... so I apologise if things have
> changed recently. This performance issue has existed for quite some time
> now, however.
>
> Any suggestions on speeding things up?

Could you send a profiler report so that I can get a better glimpse on
what part of `org-table-align' is lagging?

An ECM would help too.

Regards,

-- 
Nicolas Goaziou

Re: working with tables can be quite painful...

[Adam Porter]

In case this helps anyone, I've found this code makes profiling a lot
easier.  It automatically instruments the desired functions, runs the
code you want to test, removes the instrumentation, and presents the
results.

#+BEGIN_SRC elisp
  (defmacro profile-org (times &rest body)
    `(let (output)
       (dolist (p '("org-"))  ; symbol prefixes to instrument
         (elp-instrument-package p))
       (dotimes (x ,times)
         ,@body)
       (elp-results)
       (elp-restore-all)
       (point-min)
       (forward-line 20)
       (delete-region (point) (point-max))
       (setq output (buffer-substring-no-properties (point-min) (point-max)))
       (kill-buffer)
       (delete-window)
       output))

  ;; Used like this:
  (profile-org 10
    (org-table-next-field)
    (org-table-align))
#+END_SRC

Re: working with tables can be quite painful...

[Michael Brand]

Hi Eric

Question, out of curiosity: Is there a difference when you delete all lines
above and below the table, with and without adding a headline above?

One of my tables fluctuates around 150 rows and around 20 to 40
columns, overall a few hundred characters wide (columns with some
history window for each row). Almost all fields are just text although
I think this doesn't matter at all. The table starts at about line 50
and is followed by about 7600 lines. Since years I manipulate it at
least every few days without any slowness, often on much weaker
hardware than yours.

Maybe you can find a temporary workaround.

Michael

Re: working with tables can be quite painful...

[Eric S Fraga]

On Saturday, 17 Sep 2016 at 08:48, Nicolas Goaziou wrote:

[...]

> Could you send a profiler report so that I can get a better glimpse on
> what part of `org-table-align' is lagging?

Hi Nicolas,

this morning, working on the same table is much less painful.  I've run
the profile and did some movements and changes to the table and here is
the output from the profiler (I can send you the full report if you
wish).

I know that if I narrow the buffer to just the section (headline and
contents) that contains the table, the performance is
better. Anecdotally, it would seem that the performance degrades over
time so I wonder if there is a cache issue?

My gut feeling is that it is related to fontification as well.

--8<---------------cut here---------------start------------->8---
- command-execute                                                2420  53%
 - call-interactively                                            2420  53%
  - funcall-interactively                                        2325  51%
   - pabbrev-expand-maybe                                        1244  27%
    - if                                                         1244  27%
     - pabbrev-expand-maybe-full                                 1244  27%
      - cond                                                     1244  27%
       - pabbrev-call-previous-tab-binding                       1205  26%
        - let                                                    1205  26%
         - if                                                    1205  26%
          - let                                                  1205  26%
           - if                                                  1205  26%
            - funcall                                            1205  26%
             - org-cycle                                         1205  26%
              - call-interactively                               1198  26%
               - funcall-interactively                           1198  26%
                - org-table-next-field                           1198  26%
                 - org-table-align                               1122  24%
                  + font-lock-fontify-region                      258   5%
                  + mapcar                                        219   4%
                  + org-indent-refresh-maybe                      152   3%
                    apply                                          45   0%
                    org-element--cache-before-change                 32   0%
                    org-element--cache-after-change                 32   0%
                  + jit-lock-after-change                          16   0%
                    org-table-begin                                 9   0%
                 + org-table-maybe-eval-formula                     4   0%
                   org-table-end                                    4   0%
--8<---------------cut here---------------end--------------->8---

> An ECM would help too.

I cannot send the actual table I am working on as it relates to grades
for students.  I will try to create a similar table with random entries.

thanks,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.94.1, Org release_8.3.5-1070-g190476

Re: working with tables can be quite painful...

[Eric S Fraga]

On Saturday, 17 Sep 2016 at 15:37, Michael Brand wrote:
> Hi Eric
>
> Question, out of curiosity: Is there a difference when you delete all lines
> above and below the table, with and without adding a headline above?

Hi Michael,

Maybe.  I know that org's performance improves when I narrow the buffer
to just the section I am working on, when I work on long documents so
the lines above and below the table may affect things.  

> One of my tables fluctuates around 150 rows and around 20 to 40
> columns, overall a few hundred characters wide (columns with some
> history window for each row). Almost all fields are just text although
> I think this doesn't matter at all. The table starts at about line 50
> and is followed by about 7600 lines. Since years I manipulate it at
> least every few days without any slowness, often on much weaker
> hardware than yours.

How long do you spend manipulating the table at any one time?  I have an
impression that performance degrades over time in a single session.  But
I have no evidence for this.  It's just an impression.

thanks,
eric


-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.94.1, Org release_8.3.5-1070-g190476

Re: working with tables can be quite painful...

[Michael Welle]

Hello,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Saturday, 17 Sep 2016 at 08:48, Nicolas Goaziou wrote:
>
> [...]
>
>> Could you send a profiler report so that I can get a better glimpse on
>> what part of `org-table-align' is lagging?
>
> Hi Nicolas,
>
> this morning, working on the same table is much less painful.  I've run

> the profile and did some movements and changes to the table and here is
> the output from the profiler (I can send you the full report if you
> wish).
>
> I know that if I narrow the buffer to just the section (headline and
> contents) that contains the table, the performance is
> better. Anecdotally, it would seem that the performance degrades over
> time so I wonder if there is a cache issue?
the output of your profiler run doesn't ring a bell, but a few years ago
I had the problem that after some uptime (my Emacs uptime is typical
several days, up to a few weeks) Org operations like building the agenda
became sloooow. Restarting Emacs fixed that, until the problem occurred
again after some time.

If I remember correctly I asked the list for advise. But it looked like
one of these problems only I have ;). Maybe some other lisp package had 
its hands in the game, I don't know. Eventually, after some Emacs, Org
and other updates the problem was gone.

As I said, I don't think, you suffer from the same problem. But you said
you don't use the newest software versions, so just in case.

Regards
hmw

Re: working with tables can be quite painful...

[Eric S Fraga]

On Monday, 19 Sep 2016 at 12:58, Michael Welle wrote:

[...]

> the output of your profiler run doesn't ring a bell, but a few years ago
> I had the problem that after some uptime (my Emacs uptime is typical
> several days, up to a few weeks) Org operations like building the agenda
> became sloooow. Restarting Emacs fixed that, until the problem occurred
> again after some time.

Interesting.  The performance I was referring to last week was indeed in
an emacs that had been running for quite some time.  I did close down
that emacs instance on Friday for unrelated reasons and the instance
this morning had only been running for minutes.  Maybe the problem is
not org alone but relates to emacs as well.

> If I remember correctly I asked the list for advise. But it looked like
> one of these problems only I have ;). Maybe some other lisp package had 
> its hands in the game, I don't know. Eventually, after some Emacs, Org
> and other updates the problem was gone.

Well, maybe it's a problem that others do have!

> As I said, I don't think, you suffer from the same problem. But you said
> you don't use the newest software versions, so just in case.

To clarify, when I said I wasn't using the latest version of org, it's
that I hadn't updated from git in a few weeks but I am using a snapshot
of emacs tracking the development version (and now I have org up to date).

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.1.50.1, Org release_8.3.6-1149-g582233

Re: working with tables can be quite painful...

[Michael Welle]

Hello,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Monday, 19 Sep 2016 at 12:58, Michael Welle wrote:
>
> [...]
>
>> the output of your profiler run doesn't ring a bell, but a few years ago
>> I had the problem that after some uptime (my Emacs uptime is typical
>> several days, up to a few weeks) Org operations like building the agenda
>> became sloooow. Restarting Emacs fixed that, until the problem occurred
>> again after some time.
>
> Interesting.  The performance I was referring to last week was indeed in
> an emacs that had been running for quite some time.  I did close down
> that emacs instance on Friday for unrelated reasons and the instance
> this morning had only been running for minutes.  Maybe the problem is
> not org alone but relates to emacs as well.
yepp, that's what I thought. Maybe a minor mode or some function that
got evaluated only once in a while. Or something, that depends on the
input data of that something.


>> If I remember correctly I asked the list for advise. But it looked like
>> one of these problems only I have ;). Maybe some other lisp package had 
>> its hands in the game, I don't know. Eventually, after some Emacs, Org
>> and other updates the problem was gone.
>
> Well, maybe it's a problem that others do have!
;)


>> As I said, I don't think, you suffer from the same problem. But you said
>> you don't use the newest software versions, so just in case.
>
> To clarify, when I said I wasn't using the latest version of org, it's
> that I hadn't updated from git in a few weeks but I am using a snapshot
> of emacs tracking the development version (and now I have org up to date).
Hm, my problem was definitely way more in the past. The only performance
related problem I have from time to time is when the vc-mode hooks get
in the way. But that's nothing Org can change. But on the other hand, my
tables are much smaller than yours and contain mostly static data.

Regards
hmw

Re: working with tables can be quite painful...

[Jacob Nielsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

Hi, I've had these lines in my org files for a long time. Perhaps they
help ?

# -*- cache-long-scans: nil; -*-
# This makes forward-line much faster and thus org-goto-line
# and thus org-table-sum (C-c +)

Best regards,
Jacob
> Hello,
>
> I am working with a table.  It has approximately 130 rows and 20 columns
> so it's not huge but also not small.  Three columns are text but the
> rest are all numbers with some degree of sparsity.  Instrumenting org
> while working on this table, manipulating the entries in just one row, I
> get the following:
>
> | Function                 | calls | elapsed time | average time |
> |--------------------------+-------+--------------+--------------|
> | org-cycle                |    20 | 41.550963140 | 2.0775481570 |
> | org-table-next-field     |    20 | 41.544266727 | 2.0772133363 |
> | org-table-align          |     5 | 41.470595702 | 8.2941191404 |
> | org-mode-flyspell-verify |    52 | 1.0647362189 | 0.0204756965 |
> | org-do-latex-and-related |    21 | 0.6656267140 | 0.0316965101 |
> | org-element-at-point     |   125 | 0.6356939890 | 0.0050855519 |
> | org-element--parse-to    |   125 | 0.6086256940 | 0.0048690055 |
> | org-element--cache-put   |  1399 | 0.4963533770 | 0.0003547915 |
>
> From this, it would seem that the table align function is killing the
> performance.  8 seconds per call?  On an 8 core Intel(R) Core(TM)
> i7-2760QM CPU @ 2.40GHz... so not my wee Pandora where I expect
> slowness!
>
> This is with a not quite up to date org.  I'm avoiding upgrading as I am
> preparing material for teaching which starts soon and I don't want to
> run into issues due to changes in org... so I apologise if things have
> changed recently. This performance issue has existed for quite some time
> now, however.
>
> Any suggestions on speeding things up?
>
> Thanks,
> eric

Re: working with tables can be quite painful...

[Eric S Fraga]

On Tuesday, 20 Sep 2016 at 08:54, Jacob Nielsen wrote:
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
> Hi, I've had these lines in my org files for a long time. Perhaps they
> help ?
>
> # -*- cache-long-scans: nil; -*-
> # This makes forward-line much faster and thus org-goto-line
> # and thus org-table-sum (C-c +)

Interesting.  Reading the documentation for this variable notes that
having this set to t is most useful for buffers with very long
lines.  As I use visual-line-mode, most of my lines are whole paragraphs
so frequently longer than the 500 characters that the documentation
mentions as "long lines".  So it would seem that I should keep the
default setting of t...  but it may be worthwhile playing around with
this variable.

There are obviously quite a few interacting elements which can affect
the performance of emacs.

Thanks!
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.1.50.1, Org release_8.3.6-1149-g582233

Re: working with tables can be quite painful...

[Nicolas Goaziou]

Hello,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> I cannot send the actual table I am working on as it relates to grades
> for students.  I will try to create a similar table with random
> entries.

You could also try the imperfect (make sure to eyeball the result):

  (defun scramble-contents ()
    (interactive)
    (let ((tree (org-element-parse-buffer)))
      (org-element-map tree '(code comment comment-block example-block fixed-width
                                   keyword link node-property plain-text verbatim)
        (lambda (obj)
          (cl-case (org-element-type obj)
            ((code comment comment-block example-block fixed-width keyword
                   node-property verbatim)
             (let ((value (org-element-property :value obj)))
               (org-element-put-property
                obj :value (replace-regexp-in-string "[[:alnum:]]" "x" value))))
            (link
             (unless (string= (org-element-property :type obj) "radio")
               (org-element-put-property obj :raw-link "http://orgmode.org")))
            (plain-text
             (org-element-set-element
              obj (replace-regexp-in-string "[[:alnum:]]" "x" obj)))))
        nil nil nil t)
      (let ((buffer (get-buffer-create "*Scrambled text*")))
        (with-current-buffer buffer
          (insert (org-element-interpret-data tree))
          (goto-char (point-min)))
        (switch-to-buffer buffer))))

Regards,

-- 
Nicolas Goaziou