* [SUMMARY] #9 [[bbb:OrgMeetup]] on Wed, July 10, 19:00 UTC+3
2024-06-26 15:48 #9 [[bbb:OrgMeetup]] on Wed, July 10, 19:00 UTC+3 Ihor Radchenko
2024-07-10 16:09 ` Dave Marquardt
2024-07-10 16:12 ` William Denton
@ 2024-07-16 15:42 ` Ihor Radchenko
2024-07-16 22:43 ` Suhail Singh
2 siblings, 1 reply; 10+ messages in thread
From: Ihor Radchenko @ 2024-07-16 15:42 UTC (permalink / raw)
To: emacs-orgmode
This meetup started a bit late (~20 min). I messed up the time zone I
am in and thought that the start time is an hour later than it
actually was. Luckily, we did not start full hour late, thanks to
Dave Marquardt, Jeff Trull, and William Denton who followed up on
Mastodon and the mailing list.
- As usual, at beginning it was Sacha's News:
https://sachachua.com/blog/2024/07/2024-07-08-emacs-news/
- =Demo= reported about Emacs slowing down on large (~100k lines) Org
file. The slower, the longer Emacs session is. He was not able to
showcase the problem on screen as the problematic file contains
sensitive info, but we still troubleshooted by voice.
- He also briefly talked about his workflow and file structure,
which includes
1. Heavy usage of sparse trees to see what to do next (no agenda, AFAIU)
2. Up to 6 levels nesting with subtrees representing individual
projects/subprojects
- My immediate reaction for "slower the longer Emacs session is" is
problems with Emacs redisplay.
- Emacs redisplay works by examining the buffer text between first
char at the beginning of the current window and the last char
- Usually, it means a very small portion of the buffer (Emacs
window cannot normally fit too much text)
- ... but not for Org mode
- In Org mode, large chunks of text may be invisible, making
Emacs redisplay engine go through the whole document
(megabytes of text in Demo's example), which may be slow in
some scenarios
- More specifically, the redisplay scales with the number of
/intervals/ in the displayed buffer range
- Each /interval/ is a segment of the buffer where no single
text property changes
- The number of /intervals/ tends to grow over time because of
how Emacs fontification works
- Each time Emacs needs to apply fonts/colors in buffer, it
puts a number of text properties, but only onto the text
currently visible (not onto folded, invisible text)
- The fontification keeps accumulating as more and more
parts of the buffer gets displayed and this may cause
slowing down over time if the buffer gets particularly
large + fontification is particularly complex (many text
properties)
- If the above is what is really happening, it should be sufficient
to disable fontification via M-x font-lock-mode, use a more
radical M-x fundamental-mode, or an even more radical M-x
clean-mode (clean-mode is only for Emacs 29+)
- Demo tried this, and it turned out that the slowdown was not
solved even with fundamental-mode (I forgot about clean-mode
during the meetup)
- So, something else was the culprit - not fontification
- We then tried emacs -Q opening the same file and figured that
the lags disappeared
- So, as it usually goes, some pesky package / setting is creating
the fuss. We sent Demo to bisect the config :)
- Dave Marquardt shared his experience exporting from Org mode to docx
- He used ox-pandoc: https://github.com/emacsorphanage/ox-pandoc
- When exporting, he was able to use "reference document" to derive
in-document styles from. Such references are extremely useful
when one needs to follow specific document format without manually
editing styles in the Word/LibreOffice
- We went on comparing ox-pandoc with built-in ox-odt
- Some people are ok with ox-odt defaults (they are reasonably
good), but I also heard from other people who strongly believe
that Pandoc produces more beautiful documents out of the box
(without tweaking)
- Aside, built-in ox-odt has quite a history
- Its original author is... (to say) no longer interested to deal
with FSF copyright assignment (it was quite a drama at some point)
- So, he maintains his own work of built-in ox-odt with a number
of additional features: https://github.com/kjambunathan/org-mode-ox-odt
- He even puts it into FAQ
https://github.com/kjambunathan/org-mode-ox-odt?tab=readme-ov-file#faqs
: Will you merge this repo to upstream Org mode or GNU Emacs?
: Never
: Will you put this repo on MELPA or GNU ELPA?
: Never
- It is a pity that the additional features in the fork are not
portable upstream, but we never force anyone to contribute or
sign the paperwork. So, it is what it is.
(I am not 100% sure is the problem with MELPA (or non-GNU ELPA?)
though - it does not enforce copyright or licensing)
- Dave Marquardt pointed one missing feature in ox-odt - it cannot
export inline code blocks:
#+begin_src emacs-lisp
(defun org-odt-inline-src-block (_inline-src-block _contents _info)
"Transcode an INLINE-SRC-BLOCK element from Org to ODT.
CONTENTS holds the contents of the item. INFO is a plist holding
contextual information."
(error "FIXME"))
#+end_src
- Same for the fork
- I am not very familiar wit Open Document format, but I suspect
that it might be some kind of technical limitation with ODT
spec. Will look into this in the future.
- As a reminder, inline src blocks are blocks like
src_emacs-lisp{(format "Hello %s" user-login-name)}
{{{results(=Hello yantar92=)}}}, which can be right inside a
paragraph, unlike src code blocks.
- We had a quick question about the difference between Org mode
version on GNU ELPA (bugfix branch in git repo) and the development
version (main branch)
- GNU ELPA contains our official releases
- Apart from major releases that are announced and introduce new
features, we only put simple/critical bug fixes there. So new
"bugfix" versions like Org 9.7.3 -> Org 9.7.4 never contain new
features and should not introduce regressions (we are quite
paranoid about not introducing regressions when doing bugfix
releases, even with critical fixes)
- Main branch (also available from GNU ELPA devel,
https://protesilaos.com/codelog/2022-05-13-emacs-elpa-devel/) is different
- It is a WIP branch where we work on the next release
- We try our best, but we can break things there sometimes
- We also introduce experimental features, seeking for early
feedback from people tracking cutting edge development
- Here is a more formal description of the branches in our internal docs:
https://orgmode.org/worg/org-maintenance.html#release-types
- Jeff Trull shared his WIP on writing Org -> Keynote exporter
- In this exporter, he wants to automatically represent Org tables
as charts in the exported keynote documents
- The idea is to utilize #+PLOT keyword to get hints on what to put onto the charts
- His concern was that #+PLOT is designed to work with Gnuplot and
Gnuplot may not have all the appropriate plot types
- Then, #+PLOT semantics may not be good enough as it will not
cover things not supported by Gnuplot
- As an example, he stated that pie charts are not possible in Gnuplot
- (technically, they are possible, but people often use hacks
like https://stackoverflow.com/questions/31896718/generation-of-pie-chart-using-gnuplot)
- Since Gnuplot 6 (the recent release), pie charts are, in fact, available
- http://www.gnuplot.info/demo/sectors.html
- In any case, #+PLOT keyword is extendable. I may contain
non-standard entries as needed. See
~org-plot/preset-plot-types~. As long as general semantics is
followed, having a #+PLOT value that is not recognized is ok.
- I shared the ongoing work on improving org-crypt library
- Daniel Clemente recently reminded about the known problem with
org-crypt sometimes leaking the unencrypted data onto the disk
when saving file, saving backups, or during auto-save
https://list.orgmode.org/CAJKAhPAjcy3WwbR_yiXCddS9xn6rsOeea7r94TsNQ7NNLofs2A@mail.gmail.com/
- The problem is indeed quite annoying, especially because org-crypt
does provide some half-baked workarounds to deal with it -
~org-crypt-use-before-save-magic~
- Unfortunately, ~org-crypt-use-before-save-magic~ has problems:
1. It has to be called manually
2. It uses ~before-save-hook~ and Emacs removes hooks from
~before-save-hook~ if there is any error thrown while running
the hook functions. So, if there is any problem with
encryption, the hook gets removed (silently!), and the data
starts leaking
3. ~before-save-hook~ does not get called during auto-save
So, org-crypt employs ~org-crypt-disable-auto-save~ that (by
default) keeps asking to disable auto-save completely (without
mentioning the customization name) - fairly annoying query when
one does not know that customization exists
4. When ~before-save-hook~ is set to ~'encrypt~, it only actually
encrypts things before auto-save in current buffer, even when
auto-save saves multiple buffers
5. The above two options for auto-encryption make the :crypt:
headings disappear in Emacs buffers as well, leading to
annoying scenario when pressing C-x C-s while editing :crypt:
heading contents encrypts it in front of the user
- I am now working on improving reliability of org-crypt to avoid
unexpected data leaks and to make the UI less annoying
The WIP branch is https://git.sr.ht/~yantar92/org-mode/log/feature/org-crypt-refactor
Planned changes:
1. Enable ~org-crypt-use-before-save-magic~ by default
2. Avoid encryption inside Emacs buffers - even when the encrypted
headings are auto-saved to disk, there is no reason to encrypt
everything in buffer itself
3. Provide interactive commands to disable/enable encryption to
avoid blocking saving when encryption is broken for some reason
(say, GnuPG does not work)
4. Improve performance (org-crypt is quite slow on large buffers
now because it scans through every single heading)
5. Improve UI when encrypting multiple headings symmetrically.
The current design is to encrypt each heading separately, using
a separate password - it is hardly useful to anyone. If
passwords are not same, how would one go around remembering
them?
- The last thing I was fighting with was (5)
- On the surface, it is not too hard to hook into Emacs EPG and
query password only once for all the encrypted headings
- However, the problem comes when there are existing symmetrically
encrypted headings in the buffer. What if they are encrypted
using a different password?
- =emacs-user= suggested trying decrypting the already encrypted
headings with a given new encryption password and if it does
not work, suggest to re-encrypt using the new password
- Jeff Trull asked why not to display decrypted text as an overlay
and avoid all the above trickery with data leaks and whatnot
- Overlaying decrypted text is all fine until one needs to
actually edit that text (using normal Org mode commands!)
- So, we do need to put the decrypted text into actual buffer one
way or another. And we go back to the same problem with leakage,
albeit less frequent.
- Since leakage should not happen at all, we should solve it even
with overlay approach
- So, overlay idea just complicates things
- =emacs-user= asked about calling function from Library of Babel from elisp
- The general answer is ~org-sbe~
- However, as =emacs-user= pointed, it triggers warning when used outside Org buffer
- Oh well. ~org-sbe~ is a hack (on top of
~org-babel-execute-src-block~) in reality and not a properly
designed API
- One day we need to implement something better
- Patches welcome! At least, it is a good idea to start a discussion
on the mailing list. See https://orgmode.org/manual/Feedback.html#Feedback
:comments:
[18:15] Welcome to <b>[[bbb:OrgMeetup]]</b>!<br /><br />For help on using BigBlueButton see these (short) <a href="https://www.bigbluebutton.org/html5" target="_blank"><u>tutorial videos</u></a>.<br /><br />To join the audio bridge click the phone button. Use a headset to avoid causing background noise for others.<br /><br />This server is running <a href="https://docs.bigbluebutton.org/" target="_blank"><u>BigBlueButton</u></a>.
[18:16] William Denton : Hello!
[18:17] Dave Marquardt : Hi
[18:17] cryptk : hello
[18:19] Ihor Radchenko : As usual, the latest edition of Emacs News: https://sachachua.com/blog/2024/07/2024-07-08-emacs-news/
[18:19] fnat : Hello! (Still sorting out audio on my end.)
[18:19] William Denton : No problem.
[18:23] Demo : Hihi! Listening only for the moment.
[18:23] fnat : Same here, listening mode. I was able to work around the audio issues using a second device.
[18:25] William Denton : Nothing new from me, and no problems ... I'm just lurking.
[18:26] Demo : I think my only problem is i'm still suffering from slowdowns in huge files.
[18:26] Demo : if you want to discuss that, i just can't show the file :P
[18:26] Dave Marquardt : I recently used the Pandoc exporter to create a nice Word document, mostly avoiding using Word or LibreOffice, except for fixing a reference document. Cool stuff!
[18:26] Demo : let me get mic
[18:30] Dave Marquardt : :)
[18:42] Ihor Radchenko : ox-pandoc?
[18:42] Dave Marquardt : ox-pandoc
[18:43] Ihor Radchenko : https://github.com/emacsorphanage/ox-pandoc
[18:43] Dave Marquardt : Yes, I used ox-pandoc
[18:44] Ihor Radchenko : https://github.com/kjambunathan/org-mode-ox-odt
[18:44] Ihor Radchenko : ox-odt fork
[18:45] Ihor Radchenko : from the original author
[18:46] Ihor Radchenko : https://github.com/kjambunathan/org-mode-ox-odt?tab=readme-ov-file#faqs
[18:46] Ihor Radchenko : Will you merge this repo to upstream Orgmode or GNU Emacs?
Never
[18:46] Ihor Radchenko : Will you put this repo on MELPA or GNU ELPA?
Never
[18:46] Dave Marquardt : Yes, I remember the drama!
[18:48] Dave Marquardt : IIRC, the ODT exporter doesn't support inline code snippets, which affects me
[18:48] Dave Marquardt : The original one
[18:48] Dave Marquardt : src_lang{foo}
[18:49] Dave Marquardt : Yeah, the FIXME error!
[18:49] Demo : its very clear!
[18:49] Demo : so glad to see you here too davemq ;]
[18:51] Dave Marquardt : Most exporters do nice font and color stuff with inline code
[18:51] Dave Marquardt : Thanks
[18:52] William Denton : I need to drop out for something at the top of the hour ... thanks for organizing, Ihor, and as always, thanks for all your Org work.
[18:58] fnat : Thanks Ihor and all. I also need to switch to another meeting in a few mins. This is very interesting and useful, thanks for organising it, I'll be back next time.
[19:00] Jeff Trull : I have a general discussion topic about plots if that's interesting
[19:02] Ihor Radchenko : https://stackoverflow.com/questions/31896718/generation-of-pie-chart-using-gnuplot
[19:03] Ihor Radchenko : http://www.gnuplot.info/demo/sectors.html
[19:03] Ihor Radchenko : with sectors
[19:04] Ihor Radchenko : Gnuplot 6
[19:11] Jeff Trull : Actually please do share
[19:11] Jeff Trull : I was reading about gnuplot 6 features and spaced out for a moment
[19:13] Jeff Trull : Nice
[19:23] emacs-user : why not do a (optional)decrypt-encrypt for all entries upon save? it may not be performant, but it would avoid this issue
[19:24] emacs-user : yes
[19:24] Jeff Trull : Could we *not* decrypt them for real, but just display an overlay with the decrypted data?
[19:24] Jeff Trull : Yeah, decrypt on editing
[19:25] emacs-user : doing a decrypt-encrypt for ALL entries would also allow org to prompt user when newly entered key differs from the key used in another block in the file
[19:29] emacs-user : if we don't want to prompt users for key when their changes are in unencrypted regions, then it would be necessary to track whether the modified changes include a new encrypted entry or not. at that point decryption of all would need to be attempted. the other place would be when an existed encrypted entry is being modified
[19:33] Jeff Trull : Not for me but as usual it's very motivating to attend thanks!
[19:34] emacs-user : i joined late, but any updates on the new latex preview work?
[19:34] Jeff Trull : it's very impressive work
[19:35] emacs-user : thanks. so maybe 9.8 but could be later
[19:35] emacs-user : heh
[19:36] emacs-user : completely random, but what is the recommended way to call a function from "library of babel" via elisp?
[19:38] emacs-user : org-sbe seems to provide a mechanism, however, it gives a warning unless invoked from an org-mode buffer
[19:38] emacs-user : org-element (if i recall correctly) gives a warning with org-sbe
[19:39] Ihor Radchenko : please post a feature request on the mailing list
[19:40] emacs-user : (with-temp-buffer (org-mode) ...) seems to work, but i wasn't sure if that was recommended
[19:40] emacs-user : understood, thanks
[19:42] Jeff Trull : Excellent thanks for holding this meeting
[19:43] emacs-user : thank you for organizing, as well as continuing to develop and maintain org-mode
[19:43] cryptk : thank you!
:end:
--
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>
^ permalink raw reply [flat|nested] 10+ messages in thread