[SUMMARY] #9 [[bbb:OrgMeetup]] on Wed, July 10, 19:00 UTC+3

[Ihor Radchenko <yantar92@posteo.net>]

This meetup started a bit late (~20 min). I messed up the time zone I
am in and thought that the start time is an hour later than it
actually was. Luckily, we did not start full hour late, thanks to
Dave Marquardt, Jeff Trull, and William Denton who followed up on
Mastodon and the mailing list.

- As usual, at beginning it was Sacha's News:
  https://sachachua.com/blog/2024/07/2024-07-08-emacs-news/

- =Demo= reported about Emacs slowing down on large (~100k lines) Org
  file. The slower, the longer Emacs session is. He was not able to
  showcase the problem on screen as the problematic file contains
  sensitive info, but we still troubleshooted by voice.

  - He also briefly talked about his workflow and file structure,
    which includes
    1. Heavy usage of sparse trees to see what to do next (no agenda, AFAIU)
    2. Up to 6 levels nesting with subtrees representing individual
       projects/subprojects

  - My immediate reaction for "slower the longer Emacs session is" is
    problems with Emacs redisplay.
    - Emacs redisplay works by examining the buffer text between first
      char at the beginning of the current window and the last char
      - Usually, it means a very small portion of the buffer (Emacs
        window cannot normally fit too much text)
      - ... but not for Org mode
      - In Org mode, large chunks of text may be invisible, making
        Emacs redisplay engine go through the whole document
        (megabytes of text in Demo's example), which may be slow in
        some scenarios
	- More specifically, the redisplay scales with the number of
          /intervals/ in the displayed buffer range
	- Each /interval/ is a segment of the buffer where no single
          text property changes
	- The number of /intervals/ tends to grow over time because of
          how Emacs fontification works
	  - Each time Emacs needs to apply fonts/colors in buffer, it
            puts a number of text properties, but only onto the text
            currently visible (not onto folded, invisible text)
	  - The fontification keeps accumulating as more and more
            parts of the buffer gets displayed and this may cause
            slowing down over time if the buffer gets particularly
            large + fontification is particularly complex (many text
            properties)

  - If the above is what is really happening, it should be sufficient
    to disable fontification via M-x font-lock-mode, use a more
    radical M-x fundamental-mode, or an even more radical M-x
    clean-mode (clean-mode is only for Emacs 29+)

    - Demo tried this, and it turned out that the slowdown was not
      solved even with fundamental-mode (I forgot about clean-mode
      during the meetup)

    - So, something else was the culprit - not fontification

    - We then tried emacs -Q opening the same file and figured that
      the lags disappeared

    - So, as it usually goes, some pesky package / setting is creating
      the fuss.  We sent Demo to bisect the config :)

- Dave Marquardt shared his experience exporting from Org mode to docx
  - He used ox-pandoc: https://github.com/emacsorphanage/ox-pandoc
  - When exporting, he was able to use "reference document" to derive
    in-document styles from.  Such references are extremely useful
    when one needs to follow specific document format without manually
    editing styles in the Word/LibreOffice
  - We went on comparing ox-pandoc with built-in ox-odt
    - Some people are ok with ox-odt defaults (they are reasonably
      good), but I also heard from other people who strongly believe
      that Pandoc produces more beautiful documents out of the box
      (without tweaking)

  - Aside, built-in ox-odt has quite a history
    - Its original author is... (to say) no longer interested to deal
      with FSF copyright assignment (it was quite a drama at some point)
    - So, he maintains his own work of built-in ox-odt with a number
      of additional features: https://github.com/kjambunathan/org-mode-ox-odt
    - He even puts it into FAQ
      https://github.com/kjambunathan/org-mode-ox-odt?tab=readme-ov-file#faqs
      : Will you merge this repo to upstream Org mode or GNU Emacs?
      : Never
      : Will you put this repo on MELPA or GNU ELPA?
      : Never
    - It is a pity that the additional features in the fork are not
      portable upstream, but we never force anyone to contribute or
      sign the paperwork. So, it is what it is.
      (I am not 100% sure is the problem with MELPA (or non-GNU ELPA?)
      though - it does not enforce copyright or licensing)

  - Dave Marquardt pointed one missing feature in ox-odt - it cannot
    export inline code blocks:
    #+begin_src emacs-lisp
      (defun org-odt-inline-src-block (_inline-src-block _contents _info)
        "Transcode an INLINE-SRC-BLOCK element from Org to ODT.
      CONTENTS holds the contents of the item.  INFO is a plist holding
      contextual information."
        (error "FIXME"))
    #+end_src
    - Same for the fork
    - I am not very familiar wit Open Document format, but I suspect
      that it might be some kind of technical limitation with ODT
      spec. Will look into this in the future.
    - As a reminder, inline src blocks are blocks like
      src_emacs-lisp{(format "Hello %s" user-login-name)}
      {{{results(=Hello yantar92=)}}}, which can be right inside a
      paragraph, unlike src code blocks.

- We had a quick question about the difference between Org mode
  version on GNU ELPA (bugfix branch in git repo) and the development
  version (main branch)
  - GNU ELPA contains our official releases
    - Apart from major releases that are announced and introduce new
      features, we only put simple/critical bug fixes there. So new
      "bugfix" versions like Org 9.7.3 -> Org 9.7.4 never contain new
      features and should not introduce regressions (we are quite
      paranoid about not introducing regressions when doing bugfix
      releases, even with critical fixes)

  - Main branch (also available from GNU ELPA devel,
    https://protesilaos.com/codelog/2022-05-13-emacs-elpa-devel/) is different
    - It is a WIP branch where we work on the next release
    - We try our best, but we can break things there sometimes
    - We also introduce experimental features, seeking for early
      feedback from people tracking cutting edge development

  - Here is a more formal description of the branches in our internal docs:
    https://orgmode.org/worg/org-maintenance.html#release-types

- Jeff Trull shared his WIP on writing Org -> Keynote exporter
  - In this exporter, he wants to automatically represent Org tables
    as charts in the exported keynote documents
  - The idea is to utilize #+PLOT keyword to get hints on what to put onto the charts
  - His concern was that #+PLOT is designed to work with Gnuplot and
    Gnuplot may not have all the appropriate plot types
    - Then, #+PLOT semantics may not be good enough as it will not
      cover things not supported by Gnuplot
    - As an example, he stated that pie charts are not possible in Gnuplot
      - (technically, they are possible, but people often use hacks
        like https://stackoverflow.com/questions/31896718/generation-of-pie-chart-using-gnuplot)
  - Since Gnuplot 6 (the recent release), pie charts are, in fact, available
    - http://www.gnuplot.info/demo/sectors.html

  - In any case, #+PLOT keyword is extendable. I may contain
    non-standard entries as needed. See
    ~org-plot/preset-plot-types~. As long as general semantics is
    followed, having a #+PLOT value that is not recognized is ok.


- I shared the ongoing work on improving org-crypt library
  - Daniel Clemente recently reminded about the known problem with
    org-crypt sometimes leaking the unencrypted data onto the disk
    when saving file, saving backups, or during auto-save
    https://list.orgmode.org/CAJKAhPAjcy3WwbR_yiXCddS9xn6rsOeea7r94TsNQ7NNLofs2A@mail.gmail.com/
  - The problem is indeed quite annoying, especially because org-crypt
    does provide some half-baked workarounds to deal with it -
    ~org-crypt-use-before-save-magic~
  - Unfortunately, ~org-crypt-use-before-save-magic~ has problems:
    1. It has to be called manually
    2. It uses ~before-save-hook~ and Emacs removes hooks from
       ~before-save-hook~ if there is any error thrown while running
       the hook functions.  So, if there is any problem with
       encryption, the hook gets removed (silently!), and the data
       starts leaking
    3. ~before-save-hook~ does not get called during auto-save
       So, org-crypt employs ~org-crypt-disable-auto-save~ that (by
       default) keeps asking to disable auto-save completely (without
       mentioning the customization name) - fairly annoying query when
       one does not know that customization exists
    4. When ~before-save-hook~ is set to ~'encrypt~, it only actually
       encrypts things before auto-save in current buffer, even when
       auto-save saves multiple buffers
    5. The above two options for auto-encryption make the :crypt:
       headings disappear in Emacs buffers as well, leading to
       annoying scenario when pressing C-x C-s while editing :crypt:
       heading contents encrypts it in front of the user
  - I am now working on improving reliability of org-crypt to avoid
    unexpected data leaks and to make the UI less annoying
    The WIP branch is https://git.sr.ht/~yantar92/org-mode/log/feature/org-crypt-refactor
    
    Planned changes:
    1. Enable ~org-crypt-use-before-save-magic~ by default
    2. Avoid encryption inside Emacs buffers - even when the encrypted
       headings are auto-saved to disk, there is no reason to encrypt
       everything in buffer itself
    3. Provide interactive commands to disable/enable encryption to
       avoid blocking saving when encryption is broken for some reason
       (say, GnuPG does not work)
    4. Improve performance (org-crypt is quite slow on large buffers
       now because it scans through every single heading)
    5. Improve UI when encrypting multiple headings symmetrically.
       The current design is to encrypt each heading separately, using
       a separate password - it is hardly useful to anyone. If
       passwords are not same, how would one go around remembering
       them?

  - The last thing I was fighting with was (5)
    - On the surface, it is not too hard to hook into Emacs EPG and
      query password only once for all the encrypted headings
    - However, the problem comes when there are existing symmetrically
      encrypted headings in the buffer. What if they are encrypted
      using a different password?
      - =emacs-user= suggested trying decrypting the already encrypted
        headings with a given new encryption password and if it does
        not work, suggest to re-encrypt using the new password

  - Jeff Trull asked why not to display decrypted text as an overlay
    and avoid all the above trickery with data leaks and whatnot
    - Overlaying decrypted text is all fine until one needs to
      actually edit that text (using normal Org mode commands!)
    - So, we do need to put the decrypted text into actual buffer one
      way or another. And we go back to the same problem with leakage,
      albeit less frequent.
    - Since leakage should not happen at all, we should solve it even
      with overlay approach
    - So, overlay idea just complicates things

- =emacs-user= asked about calling function from Library of Babel from elisp
  - The general answer is ~org-sbe~
  - However, as =emacs-user= pointed, it triggers warning when used outside Org buffer
  - Oh well. ~org-sbe~ is a hack (on top of
    ~org-babel-execute-src-block~) in reality and not a properly
    designed API
  - One day we need to implement something better
  - Patches welcome! At least, it is a good idea to start a discussion
    on the mailing list. See https://orgmode.org/manual/Feedback.html#Feedback

:comments:
[18:15] Welcome to <b>[[bbb:OrgMeetup]]</b>!<br /><br />For help on using BigBlueButton see these (short) <a href="https://www.bigbluebutton.org/html5" target="_blank"><u>tutorial videos</u></a>.<br /><br />To join the audio bridge click the phone button.  Use a headset to avoid causing background noise for others.<br /><br />This server is running <a href="https://docs.bigbluebutton.org/" target="_blank"><u>BigBlueButton</u></a>.
[18:16] William Denton : Hello!
[18:17] Dave Marquardt : Hi
[18:17] cryptk : hello
[18:19] Ihor Radchenko : As usual, the latest edition of Emacs News: https://sachachua.com/blog/2024/07/2024-07-08-emacs-news/
[18:19] fnat : Hello! (Still sorting out audio on my end.)
[18:19] William Denton : No problem.
[18:23] Demo : Hihi! Listening only for the moment.
[18:23] fnat : Same here, listening mode. I was able to work around the audio issues using a second device.
[18:25] William Denton : Nothing new from me, and no problems ... I'm just lurking.
[18:26] Demo : I think my only problem is i'm still suffering from slowdowns in huge files.
[18:26] Demo : if you want to discuss that, i just can't show the file :P
[18:26] Dave Marquardt : I recently used the Pandoc exporter to create a nice Word document, mostly avoiding using Word or LibreOffice, except for fixing a reference document. Cool stuff!
[18:26] Demo : let me get mic
[18:30] Dave Marquardt : :)
[18:42] Ihor Radchenko : ox-pandoc?
[18:42] Dave Marquardt : ox-pandoc
[18:43] Ihor Radchenko : https://github.com/emacsorphanage/ox-pandoc
[18:43] Dave Marquardt : Yes, I used ox-pandoc
[18:44] Ihor Radchenko : https://github.com/kjambunathan/org-mode-ox-odt
[18:44] Ihor Radchenko : ox-odt fork
[18:45] Ihor Radchenko : from the original author
[18:46] Ihor Radchenko : https://github.com/kjambunathan/org-mode-ox-odt?tab=readme-ov-file#faqs
[18:46] Ihor Radchenko : Will you merge this repo to upstream Orgmode or GNU Emacs?

Never
[18:46] Ihor Radchenko : Will you put this repo on MELPA or GNU ELPA?

Never
[18:46] Dave Marquardt : Yes, I remember the drama!
[18:48] Dave Marquardt : IIRC, the ODT exporter doesn't support inline code snippets, which affects me
[18:48] Dave Marquardt : The original one
[18:48] Dave Marquardt : src_lang{foo}
[18:49] Dave Marquardt : Yeah, the FIXME error!
[18:49] Demo : its very clear!
[18:49] Demo : so glad to see you here too davemq ;]
[18:51] Dave Marquardt : Most exporters do nice font and color stuff with inline code
[18:51] Dave Marquardt : Thanks
[18:52] William Denton : I need to drop out for something at the top of the hour ... thanks for organizing, Ihor, and as always, thanks for all your Org work.
[18:58] fnat : Thanks Ihor and all. I also need to switch to another meeting in a few mins. This is very interesting and useful, thanks for organising it, I'll be back next time.
[19:00] Jeff Trull : I have a general discussion topic about plots if that's interesting
[19:02] Ihor Radchenko : https://stackoverflow.com/questions/31896718/generation-of-pie-chart-using-gnuplot
[19:03] Ihor Radchenko : http://www.gnuplot.info/demo/sectors.html
[19:03] Ihor Radchenko : with sectors
[19:04] Ihor Radchenko : Gnuplot 6
[19:11] Jeff Trull : Actually please do share
[19:11] Jeff Trull : I was reading about gnuplot 6 features and spaced out for a moment
[19:13] Jeff Trull : Nice
[19:23] emacs-user : why not do a (optional)decrypt-encrypt for all entries upon save? it may not be performant, but it would avoid this issue
[19:24] emacs-user : yes
[19:24] Jeff Trull : Could we *not* decrypt them for real, but just display an overlay with the decrypted data?
[19:24] Jeff Trull : Yeah, decrypt on editing
[19:25] emacs-user : doing a decrypt-encrypt for ALL entries would also allow org to prompt user when newly entered key differs from the key used in another block in the file
[19:29] emacs-user : if we don't want to prompt users for key when their changes are in unencrypted regions, then it would be necessary to track whether the modified changes include a new encrypted entry or not. at that point decryption of all would need to be attempted. the other place would be when an existed encrypted entry is being modified
[19:33] Jeff Trull : Not for me but as usual it's very motivating to attend thanks!
[19:34] emacs-user : i joined late, but any updates on the new latex preview work?
[19:34] Jeff Trull : it's very impressive work
[19:35] emacs-user : thanks. so maybe 9.8 but could be later
[19:35] emacs-user : heh
[19:36] emacs-user : completely random, but what is the recommended way to call a function from "library of babel" via elisp?
[19:38] emacs-user : org-sbe seems to provide a mechanism, however, it gives a warning unless invoked from an org-mode buffer
[19:38] emacs-user : org-element (if i recall correctly) gives a warning with org-sbe
[19:39] Ihor Radchenko : please post a feature request on the mailing list
[19:40] emacs-user : (with-temp-buffer (org-mode) ...) seems to work, but i wasn't sure if that was recommended
[19:40] emacs-user : understood, thanks
[19:42] Jeff Trull : Excellent thanks for holding this meeting
[19:43] emacs-user : thank you for organizing, as well as continuing to develop and maintain org-mode
[19:43] cryptk : thank you!
:end:

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>