#4 Org mode profiling meetup on Sat, May 7

#4 Org mode profiling meetup on Sat, May 7

[Ihor Radchenko]

Dear All,

I am continuing my experiment with Org mode meetups and online
debugging.

Anyone who wants to discuss and/or debug Org-related issues is welcome.

If you want to provide improvements to Org, but feel overwhelmed about
the contribution process or the vast Org codebase, we can touch on that
as well.

This time, I plan to talk about some Org mode internals. Specifically,
the newly added folding engine org-fold.el

Note that using microphone and/or camera should not be required. Jitsi
does have chat.

The time will be the same: 9pm SG time (4pm Kyiv; 2pm London; 9am New
York). Sat, May 7

I will post the link to the meeting one hour before the meeting start.

Best,
Ihor

Re: #4 Org mode profiling meetup on Sat, May 7

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> The time will be the same: 9pm SG time (4pm Kyiv; 2pm London; 9am New
> York). Sat, May 7
>
> I will post the link to the meeting one hour before the meeting start.

Meeting link: https://teamjoin.de/Org-dev-profiling-20220507-d708k

Best,
Ihor

Re: #4 Org mode profiling meetup on Sat, May 7

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 378 bytes --]

Ihor Radchenko <yantar92@gmail.com> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> The time will be the same: 9pm SG time (4pm Kyiv; 2pm London; 9am New
>> York). Sat, May 7
>>
>> I will post the link to the meeting one hour before the meeting start.
>
> Meeting link: https://teamjoin.de/Org-dev-profiling-20220507-d708k

Attaching the meeting notes.

Best,
Ihor


[-- Attachment #2: 2022-05-07-summary.org --]
[-- Type: application/vnd.lotus-organizer, Size: 5058 bytes --]

# Created 2022-05-08 Sun 14:18
#+title: Ihor Radchenko [ML:Org mode] (2022) #4 Org mode profiling meetup on Sat, May 7
#+date: May 7, 2022
#+author: Ihor Radchenko

* Discussion summary

- org-ql (https://github.com/alphapapa/org-ql) contributions
  - I maintain a fork (https://github.com/yantar92/org-ql/) making use of recent org-element-cache improvements
  - A function to use org-ql as ~org-agenda-skip-function~
    See ~yant/org-agenda-skip-org-ql~ in https://github.com/yantar92/emacs-config/blob/master/config.org#archiving
- Max mentioned about =yank-handled-properties:= [[info:elisp#Yanking][elisp#Yanking]] to be used instead of =filter-buffer-substring-function= to handle copying invisible parts of Org buffers into non-Org buffers.
- Discussed weird behaviour of M-<RET> and C-<RET> when point is inside folded headlines in Org (reasons unclear)
- Discussed possible improvements to ~org-babel-tangle~ performance
  1. Not overwriting existing files when they are not going to be changed
     https://list.orgmode.org/878rrcws6q.fsf@localhost/T/#u
  2. Caching hash of the src blocks in .org buffer and not tangling them if the code inside remains unchanged
     - This will not work in cases when src blocks contain code with side effects:
       #+begin_src emacs-lisp
              # this code may yield different results depending on contents
              # of /tmp folder even when this code remains unchanged
              ls /tmp/
       #+end_src

* =org-fold-core=: Universal folding library for Emacs


Yup, it is not just for Org mode. It can be used by third-party code and has minimum dependencies from Org.

1. Can use text properties or overlays to hide text
   - Text properties are significantly faster than overlays in Emacs display engine
   - Attempts to optimise overlays are ongoing...
     https://lists.gnu.org/archive/html/emacs-devel/2016-09/msg00582.html
     ... and ongoing
     https://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00565.html
     ... and still ongoing
     https://lists.gnu.org/archive/html/emacs-devel/2019-12/msg00115.html
     ... and it is already [2022-05-07 Sat]

!! *Read the commentary of org-fold-core.el*

1. Features
   - Folding/unfolding regions of text
   - Searching and examining boundaries of folded text (high-level)
   - Interactive searching in folded text (via isearch)
   - Handling edits in folded text
     - When using text properties, default functions like ~insert~ do not inherit text properties
       We need special handling of buffer edits to keep text inserted in the middle of the fold invisible
   - Killing/yanking (copying/pasting) of the folded text
     - We do not want folded text to carry over the hidden state when copying to other buffers
     - Or sometimes we do
   - Fontification of the folded text
     - Large chunks of hidden text are extremely slow to fontify

   *All the above features are configurable using different fold types*: see ~org-fold-core--specs~ docstring.

2. API symbols to check out
   - Initialisation
     ~org-fold-core-style~; ~org-fold-core--specs~; ~org-fold-core-initialize~
   - Folding/unfolding
     ~org-fold-core-region~
   - Working with folds
     ~org-fold-core-folded-p~; ~org-fold-core-region-folded-p~;
     ~org-fold-core-get-folding-spec~; ~org-fold-core-get-folding-specs-in-region~;
     ~org-fold-core-get-region-at-point~
     ~org-fold-core-next-visibility-change~; ~org-fold-core-next-folding-state-change~;
     ~org-fold-core-search-forward~;
   - Editing folds
     When we hide text using text properties, extra care should be taken when inserting text inside
     - org-fold takes care about this, but at the cost of slowdown
       - See =:front-sticky=, =:rear-sticky= in ~org-fold-core--specs~
     - Ideally, use ~insert-and-inherit~ + ~org-fold-core-ignore-modifications~ to preserve folding state around
     - Sometimes, we want to attract user attention when edits cause major change related to folded text
       - =:fragile= in ~org-fold-core--specs~
   - Performance optimisations
     ~org-fold-core--optimise-for-huge-buffers~
     - What can be done to reduce load on Emacs redisplay engine in huge (10Mb+) buffers
     ~org-fold-core-ignore-fragility-checks~
     ~org-fold-core-ignore-modifications~
     - Macros to know about for developers
   - Isearch integration (isearch does not search inside 'invisible text properties by default)
     - See ~org-fold-core--isearch-setup~
     - =:isearch-ignore= and =:isearch-open= in ~org-fold-core--specs~
   - Clearing folded state when text is copied interactively
     When using text properties, hidden text will remain hidden when inserted in other buffers
     Implementation: ~org-fold-core--buffer-substring-filter~
     Use ~filter-buffer-substring~ instead of usual ~buffer-substring~
     - Also, see =:global= in ~org-fold-core--specs~
   - Font-lock interaction
     It does not make sense to fontify folded text
     - ~org-fold-core-fontify-region~
     - Also see =:font-lock-skip= in ~org-fold-core--specs~