Enriched/Org is a colorful Org

Enriched/Org is a colorful Org

[Jambunathan K]

See "Side note" towards the end of this message

        http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8

Re: Enriched/Org is a colorful Org

[Suvayu Ali]

On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
> 
> See "Side note" towards the end of this message
> 
>         http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
> 

This request is common enough; every time it comes up overlays are
proposed as a solution.  It would be good if this is available even as a
library outside of Org.

-- 
Suvayu

Open source is the future. It sets us free.

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:

> On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
>> 
>> See "Side note" towards the end of this message
>> 
>>        http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
>> 
> 
> This request is common enough; every time it comes up overlays are
> proposed as a solution.  It would be good if this is available even as a
> library outside of Org.

Yes, overlays are better.  However, maybe I am just no getting it, but what is even the purpose of facemenu?  AFAICS, the faces are non-permanent, so when I save the file and reopen it, all the faces are gone.  I really cannot see a useful application for this.

- Carsten

Re: Enriched/Org is a colorful Org

[Torsten Wagner]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Hmmm....
as application.... an ambient-light org-mode coloring

just joking

Torsten




On 10 April 2013 12:16, Carsten Dominik <carsten.dominik@gmail.com> wrote:

>
> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>
> > On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
> >>
> >> See "Side note" towards the end of this message
> >>
> >>        http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
> >>
> >
> > This request is common enough; every time it comes up overlays are
> > proposed as a solution.  It would be good if this is available even as a
> > library outside of Org.
>
> Yes, overlays are better.  However, maybe I am just no getting it, but
> what is even the purpose of facemenu?  AFAICS, the faces are non-permanent,
> so when I save the file and reopen it, all the faces are gone.  I really
> cannot see a useful application for this.
>
> - Carsten
>
>
>

[-- Attachment #2: Type: text/html, Size: 1673 bytes --]

Re: Enriched/Org is a colorful Org

[Suvayu Ali]

Hi Carsten,

On Wed, Apr 10, 2013 at 12:16:28PM +0200, Carsten Dominik wrote:
> 
> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
> 
> > On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
> >> 
> >> See "Side note" towards the end of this message
> >> 
> >>        http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
> >> 
> > 
> > This request is common enough; every time it comes up overlays are
> > proposed as a solution.  It would be good if this is available even as a
> > library outside of Org.
> 
> Yes, overlays are better.  However, maybe I am just no getting it, but
> what is even the purpose of facemenu?  AFAICS, the faces are
> non-permanent, so when I save the file and reopen it, all the faces
> are gone.  I really cannot see a useful application for this.

AFAIR, the use cases presented so far involved adding highlighting-like
information in Org files.  If the overlays are generated consistently
based on the user's setup, it doesn't matter if they are non-permanent
(as in not part of the Org file, but dependent on the Emacs setup
instead).  Of course this means the highlighting information will not be
portable; but I don't think people mind that.

I personally do not find any use for the feature as such; although it
might be interesting to be able to insert plain text cookies in Org
files and have them highlighted in some fashion.  I could then use a
list of ideas like this:

  Some topic ...
  1. Idea 1
  2. Idea 2 (?)

where I'm doubtful about idea (2); having (?) highlighted would remind
me of that.  Just an idea.

:)

-- 
Suvayu

Open source is the future. It sets us free.

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 10 apr. 2013, at 12:48, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:

> Hi Carsten,
> 
> On Wed, Apr 10, 2013 at 12:16:28PM +0200, Carsten Dominik wrote:
>> 
>> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>> 
>>> On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
>>>> 
>>>> See "Side note" towards the end of this message
>>>> 
>>>>       http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
>>>> 
>>> 
>>> This request is common enough; every time it comes up overlays are
>>> proposed as a solution.  It would be good if this is available even as a
>>> library outside of Org.
>> 
>> Yes, overlays are better.  However, maybe I am just no getting it, but
>> what is even the purpose of facemenu?  AFAICS, the faces are
>> non-permanent, so when I save the file and reopen it, all the faces
>> are gone.  I really cannot see a useful application for this.
> 
> AFAIR, the use cases presented so far involved adding highlighting-like
> information in Org files.  If the overlays are generated consistently
> based on the user's setup, it doesn't matter if they are non-permanent
> (as in not part of the Org file, but dependent on the Emacs setup
> instead).  Of course this means the highlighting information will not be
> portable; but I don't think people mind that.
> 
> I personally do not find any use for the feature as such; although it
> might be interesting to be able to insert plain text cookies in Org
> files and have them highlighted in some fashion.  I could then use a
> list of ideas like this:
> 
>  Some topic ...
>  1. Idea 1
>  2. Idea 2 (?)
> 
> where I'm doubtful about idea (2); having (?) highlighted would remind
> me of that.  Just an idea.

Yes, this would make sense if the highlighting would be reestablished
automatically next time you visit the file.  If not, it would be pretty
useless in my eyes.

- Carsten

Re: Enriched/Org is a colorful Org

[Sebastien Vauban]

Carsten Dominik wrote:
> On 10 apr. 2013, at 12:48, Suvayu Ali <fatkasuvayu+linux-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Wed, Apr 10, 2013 at 12:16:28PM +0200, Carsten Dominik wrote:
>>> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
>>>>> 
>>>>> See "Side note" towards the end of this message
>>>>> 
>>>>>       http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
>>>>> 
>>>> 
>>>> This request is common enough; every time it comes up overlays are
>>>> proposed as a solution.  It would be good if this is available even as a
>>>> library outside of Org.
>>> 
>>> Yes, overlays are better.  However, maybe I am just no getting it, but
>>> what is even the purpose of facemenu?  AFAICS, the faces are
>>> non-permanent, so when I save the file and reopen it, all the faces
>>> are gone.  I really cannot see a useful application for this.
>> 
>> AFAIR, the use cases presented so far involved adding highlighting-like
>> information in Org files.  If the overlays are generated consistently
>> based on the user's setup, it doesn't matter if they are non-permanent
>> (as in not part of the Org file, but dependent on the Emacs setup
>> instead).  Of course this means the highlighting information will not be
>> portable; but I don't think people mind that.
>> 
>> I personally do not find any use for the feature as such; although it
>> might be interesting to be able to insert plain text cookies in Org
>> files and have them highlighted in some fashion.  I could then use a
>> list of ideas like this:
>> 
>>  Some topic ...
>>  1. Idea 1
>>  2. Idea 2 (?)
>> 
>> where I'm doubtful about idea (2); having (?) highlighted would remind
>> me of that.  Just an idea.
>
> Yes, this would make sense if the highlighting would be reestablished
> automatically next time you visit the file.  If not, it would be pretty
> useless in my eyes.

IIUC, I do use something similar: automatic highlighting of some words, hooked
on the mode (so, permanent... for me).

--8<---------------cut here---------------start------------->8---
  (defface lvn/highlight-face
    '((t (:weight normal :slant normal :box '(:line-width 1 :color "#CC0000")
          :foreground "#CC0000" :background "#FFFF88")))
    "Face for making FIXME and other warnings stand out.")

  (defvar lvn/highlight-org-regexps
    "\\(FIXME\\|BUG\\|XXX\\|[Ee]rror\\|[Ww]arning\\|WARNING\\)"
    "Patterns to highlight (for Org mode only).")

  ;; set up highlighting of special patterns for Org mode only
  (dolist (mode '(org-mode))
    (font-lock-add-keywords mode
     `((,lvn/highlight-org-regexps 1 'lvn/highlight-face prepend))))
--8<---------------cut here---------------end--------------->8---

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 10 apr. 2013, at 13:20, "Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> wrote:

> Carsten Dominik wrote:
>> On 10 apr. 2013, at 12:48, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>>> On Wed, Apr 10, 2013 at 12:16:28PM +0200, Carsten Dominik wrote:
>>>> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>>>>> On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
>>>>>> 
>>>>>> See "Side note" towards the end of this message
>>>>>> 
>>>>>>      http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
>>>>>> 
>>>>> 
>>>>> This request is common enough; every time it comes up overlays are
>>>>> proposed as a solution.  It would be good if this is available even as a
>>>>> library outside of Org.
>>>> 
>>>> Yes, overlays are better.  However, maybe I am just no getting it, but
>>>> what is even the purpose of facemenu?  AFAICS, the faces are
>>>> non-permanent, so when I save the file and reopen it, all the faces
>>>> are gone.  I really cannot see a useful application for this.
>>> 
>>> AFAIR, the use cases presented so far involved adding highlighting-like
>>> information in Org files.  If the overlays are generated consistently
>>> based on the user's setup, it doesn't matter if they are non-permanent
>>> (as in not part of the Org file, but dependent on the Emacs setup
>>> instead).  Of course this means the highlighting information will not be
>>> portable; but I don't think people mind that.
>>> 
>>> I personally do not find any use for the feature as such; although it
>>> might be interesting to be able to insert plain text cookies in Org
>>> files and have them highlighted in some fashion.  I could then use a
>>> list of ideas like this:
>>> 
>>> Some topic ...
>>> 1. Idea 1
>>> 2. Idea 2 (?)
>>> 
>>> where I'm doubtful about idea (2); having (?) highlighted would remind
>>> me of that.  Just an idea.
>> 
>> Yes, this would make sense if the highlighting would be reestablished
>> automatically next time you visit the file.  If not, it would be pretty
>> useless in my eyes.
> 
> IIUC, I do use something similar: automatic highlighting of some words, hooked
> on the mode (so, permanent... for me).
> 
> --8<---------------cut here---------------start------------->8---
>  (defface lvn/highlight-face
>    '((t (:weight normal :slant normal :box '(:line-width 1 :color "#CC0000")
>          :foreground "#CC0000" :background "#FFFF88")))
>    "Face for making FIXME and other warnings stand out.")
> 
>  (defvar lvn/highlight-org-regexps
>    "\\(FIXME\\|BUG\\|XXX\\|[Ee]rror\\|[Ww]arning\\|WARNING\\)"
>    "Patterns to highlight (for Org mode only).")
> 
>  ;; set up highlighting of special patterns for Org mode only
>  (dolist (mode '(org-mode))
>    (font-lock-add-keywords mode
>     `((,lvn/highlight-org-regexps 1 'lvn/highlight-face prepend))))
> --8<---------------cut here---------------end--------------->8---

Yes, this is definitely very useful, I do similar stuff for FIXME etc.

- Carsten


> 
> Best regards,
>  Seb
> 
> -- 
> Sebastien Vauban
> 
> 

Re: Enriched/Org is a colorful Org

[Jambunathan K]

>   ;; set up highlighting of special patterns for Org mode only
>   (dolist (mode '(org-mode))
>     (font-lock-add-keywords mode
>      `((,lvn/highlight-org-regexps 1 'lvn/highlight-face prepend))))

hi-lock.el is for regexp based highlighting.

facemenu.el is for one-off, case-by-case highlighting.

If facemenu.el can be enhanced to use overlays and have it co-exist with
font lock based modes, one should be able to color code one's notes in a
text/org file and have it saved in a disk.

> Best regards,
>   Seb

Re: Enriched/Org is a colorful Org

[Jambunathan K]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>
>> On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
>>> 
>>> See "Side note" towards the end of this message
>>> 
>>>        http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
>>> 
>> 
>> This request is common enough; every time it comes up overlays are
>> proposed as a solution.  It would be good if this is available even as a
>> library outside of Org.
>
> Yes, overlays are better.  However, maybe I am just no getting it, but
> what is even the purpose of facemenu?  AFAICS, the faces are
> non-permanent, so when I save the file and reopen it, all the faces
> are gone.  I really cannot see a useful application for this.

Look up `enriched-mode' or just visit enriched.doc in `data-directory'
(/etc).

facemenu.el is internally used by `enriched-mode'.  Let the color markup
- encoding and decoding - be persisted by enriched mode.

>
> - Carsten

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> From: Carsten Dominik <carsten.dominik@gmail.com>
> Date: Wed, 10 Apr 2013 12:16:28 +0200
> Cc: emacs-orgmode@gnu.org
> 
> 
> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
> 
> > On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
> >> 
> >> See "Side note" towards the end of this message
> >> 
> >>        http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
> >> 
> > 
> > This request is common enough; every time it comes up overlays are
> > proposed as a solution.  It would be good if this is available even as a
> > library outside of Org.
> 
> Yes, overlays are better.

I beg the Org developers to please be very careful when introducing
expensive display features such as overlays into Org.  Org already
puts the Emacs display engine to its limits in many of its popular
features; adding overlays to this mess might be too much.

I don't know enough about Org to understand why overlays are being
considered instead of text properties, but feel free to describe the
issues (preferably on emacs-devel) and start a discussion about the
possible alternatives.

Re: Enriched/Org is a colorful Org

[Jambunathan K]

Eli Zaretskii <eliz@gnu.org> writes:

> I beg the Org developers to please be very careful when introducing
> expensive display features such as overlays into Org.  Org already
> puts the Emacs display engine to its limits in many of its popular
> features; adding overlays to this mess might be too much.

We shouldn't be afraid of building a prototype or release the feature
with a warning "you are on your own".

> I don't know enough about Org to understand why overlays are being
> considered instead of text properties, but feel free to describe the
> issues (preferably on emacs-devel) and start a discussion about the
> possible alternatives.

From Org side of things, it is the rigidity of Org syntax wrt emphasis
etc.  From user side of things, an ability to color code his notes (and
possibly create an exported document that retains those highlights.)

The intention is to really keep the discussion moving and see what comes
out of it.

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> From: Jambunathan K <kjambunathan@gmail.com>
> Cc: Carsten Dominik <carsten.dominik@gmail.com>,  emacs-orgmode@gnu.org
> Date: Wed, 10 Apr 2013 22:13:46 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I beg the Org developers to please be very careful when introducing
> > expensive display features such as overlays into Org.  Org already
> > puts the Emacs display engine to its limits in many of its popular
> > features; adding overlays to this mess might be too much.
> 
> We shouldn't be afraid of building a prototype or release the feature
> with a warning "you are on your own".

Fear has nothing to do with this.  Good software design takes into
consideration the particulars and the limitation of the
infrastructure, and tries to avoid limitations and weak spots in that
infrastructure.  Building a house on shaky ground is silly; at best it
is waste of effort.

> >From Org side of things, it is the rigidity of Org syntax wrt emphasis
> etc.  From user side of things, an ability to color code his notes (and
> possibly create an exported document that retains those highlights.)

I meant technical details.  The above is too high-level to be useful.

> The intention is to really keep the discussion moving and see what comes
> out of it.

My intent was to advise people not to go in directions that can hardly
be fruitful.  Of course, if they want to...

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 10.4.2013, at 18:21, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Carsten Dominik <carsten.dominik@gmail.com>
>> Date: Wed, 10 Apr 2013 12:16:28 +0200
>> Cc: emacs-orgmode@gnu.org
>> 
>> 
>> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>> 
>>> On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
>>>> 
>>>> See "Side note" towards the end of this message
>>>> 
>>>>       http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
>>>> 
>>> 
>>> This request is common enough; every time it comes up overlays are
>>> proposed as a solution.  It would be good if this is available even as a
>>> library outside of Org.
>> 
>> Yes, overlays are better.
> 
> I beg the Org developers to please be very careful when introducing
> expensive display features such as overlays into Org.  Org already
> puts the Emacs display engine to its limits in many of its popular
> features;

Hi Eli,

this is interesting input, I was not aware of this.  Has this been discussed before, can you point me to relevant threads, and what are the symptoms of the display engine being at its limits?

Regards

- Carsten

> adding overlays to this mess might be too much.
> 
> I don't know enough about Org to understand why overlays are being
> considered instead of text properties, but feel free to describe the
> issues (preferably on emacs-devel) and start a discussion about the
> possible alternatives.
> 

Re: Enriched/Org is a colorful Org

[Sebastien Vauban]

Hi Carsten,

Carsten Dominik wrote:
> On 10.4.2013, at 18:21, Eli Zaretskii <eliz-mXXj517/zsQ@public.gmane.org> wrote:
>>> From: Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>>> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> This request is common enough; every time it comes up overlays are
>>>> proposed as a solution.  It would be good if this is available even as a
>>>> library outside of Org.
>>> 
>>> Yes, overlays are better.
>> 
>> I beg the Org developers to please be very careful when introducing
>> expensive display features such as overlays into Org.  Org already
>> puts the Emacs display engine to its limits in many of its popular
>> features;
>
> this is interesting input, I was not aware of this. Has this been discussed
> before, can you point me to relevant threads, and what are the symptoms of the
> display engine being at its limits?
>
>> adding overlays to this mess might be too much.

I guess Eli simply means, in a general way, that overlays do negatively impact
display performance, as you said as well a couple of times:

  ╭────
  │ From: Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
  │ Subject: Re: performance problems with drawers
  │ Newsgroups: gmane.emacs.orgmode
  │ To: Al <gmane00-4hnwnHkw6MReoWH0uzbU5w@public.gmane.org>
  │ Cc: emacs-orgmode-mXXj517/zsQ@public.gmane.org
  │ Date: Wed, 8 Jul 2009 07:05:53 +0200 (3 years, 39 weeks, 3 days ago)
  │ 
  │ Hi Al,
  │ 
  │ first of all, I cannot reproduce the fact that drawers have such
  │ a major influence on time, wit a test file that I created to
  │ be similar to what you describe.
  │ 
  │ There is a way to speed up drawer handling, by using text properties
  │ instead of overlays.  How have some vague plans to do this, but nothing
  │ concrete or soon.
  │ 
  │ - Carsten
  ╰────

and

  ╭────
  │ From: Carsten Dominik <carsten.dominik-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
  │ Subject: Re: fontification and icon issues
  │ Newsgroups: gmane.emacs.orgmode
  │ To: David O'Toole <dto1138-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
  │ Cc: emacs-orgmode-mXXj517/zsQ@public.gmane.org
  │ Date: Thu, 24 Sep 2009 10:46:24 +0100 (3 years, 28 weeks, 2 days ago)
  │ 
  │ On Sep 22, 2009, at 4:11 PM, David O'Toole wrote:
  │ > [...]
  │ >
  │ > 2. using add-text-properties to specify a display property (or even just
  │ > a face) for any part of an org headline text kills the fontification of
  │ > the rest of the text (TODO keyword and leading stars unaffected.) I'm
  │ > trying to use font-lock-add-keywords to display the images.
  │ 
  │ Can you make an example file, and maybe a small function that does set these
  │ properties, so that I can see what you mean?
  │ 
  │ - Carsten
  │ 
  │ > Maybe I should use overlays instead?
  │ 
  │ This can be done, but if every line in a very large file gets
  │ an overlay, performance is severely degraded.
  │ 
  │ - Carsten
  ╰────

>> I don't know enough about Org to understand why overlays are being
>> considered instead of text properties, but feel free to describe the
>> issues (preferably on emacs-devel) and start a discussion about the
>> possible alternatives.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 10.4.2013, at 22:16, "Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> wrote:

> Hi Carsten,
> 
> Carsten Dominik wrote:
>> On 10.4.2013, at 18:21, Eli Zaretskii <eliz@gnu.org> wrote:
>>>> From: Carsten Dominik <carsten.dominik@gmail.com>
>>>> On 10 apr. 2013, at 11:54, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>>>>> This request is common enough; every time it comes up overlays are
>>>>> proposed as a solution.  It would be good if this is available even as a
>>>>> library outside of Org.
>>>> 
>>>> Yes, overlays are better.
>>> 
>>> I beg the Org developers to please be very careful when introducing
>>> expensive display features such as overlays into Org.  Org already
>>> puts the Emacs display engine to its limits in many of its popular
>>> features;
>> 
>> this is interesting input, I was not aware of this. Has this been discussed
>> before, can you point me to relevant threads, and what are the symptoms of the
>> display engine being at its limits?
>> 
>>> adding overlays to this mess might be too much.
> 
> I guess Eli simply means, in a general way, that overlays do negatively impact
> display performance, as you said as well a couple of times:

Yes, but Eli says that Org already severely tests the
display engine, and he uses the word "mess", even though
we mostly use text properties for faces and other
display-related things.  So I was wondering if there is
something we should put onto our todo list.

Of course, Org already uses overlays, for example for
folding (as does outline.el), and for temporary marking
of text like during src block editing.  But as your digging
shows, I ave avoided them in the past, and we are also not
using them for org-indent.el, for example.

The reason why I said "overlays would be better" is simply
that they would allow to add display properties in a
persistent way that would not interfere that our
font-lock-unfontify-region function removes face and
invisibility text properties.  So they are "better" for
implementing hand-made faces selection that should overrule
font-lock.

- Carsten


> 
>  ╭────
>  │ From: Carsten Dominik <carsten.dominik@gmail.com>
>  │ Subject: Re: performance problems with drawers
>  │ Newsgroups: gmane.emacs.orgmode
>  │ To: Al <gmane00@wilec.net>
>  │ Cc: emacs-orgmode@gnu.org
>  │ Date: Wed, 8 Jul 2009 07:05:53 +0200 (3 years, 39 weeks, 3 days ago)
>  │ 
>  │ Hi Al,
>  │ 
>  │ first of all, I cannot reproduce the fact that drawers have such
>  │ a major influence on time, wit a test file that I created to
>  │ be similar to what you describe.
>  │ 
>  │ There is a way to speed up drawer handling, by using text properties
>  │ instead of overlays.  How have some vague plans to do this, but nothing
>  │ concrete or soon.
>  │ 
>  │ - Carsten
>  ╰────
> 
> and
> 
>  ╭────
>  │ From: Carsten Dominik <carsten.dominik@gmail.com>
>  │ Subject: Re: fontification and icon issues
>  │ Newsgroups: gmane.emacs.orgmode
>  │ To: David O'Toole <dto1138@gmail.com>
>  │ Cc: emacs-orgmode@gnu.org
>  │ Date: Thu, 24 Sep 2009 10:46:24 +0100 (3 years, 28 weeks, 2 days ago)
>  │ 
>  │ On Sep 22, 2009, at 4:11 PM, David O'Toole wrote:
>  │ > [...]
>  │ >
>  │ > 2. using add-text-properties to specify a display property (or even just
>  │ > a face) for any part of an org headline text kills the fontification of
>  │ > the rest of the text (TODO keyword and leading stars unaffected.) I'm
>  │ > trying to use font-lock-add-keywords to display the images.
>  │ 
>  │ Can you make an example file, and maybe a small function that does set these
>  │ properties, so that I can see what you mean?
>  │ 
>  │ - Carsten
>  │ 
>  │ > Maybe I should use overlays instead?
>  │ 
>  │ This can be done, but if every line in a very large file gets
>  │ an overlay, performance is severely degraded.
>  │ 
>  │ - Carsten
>  ╰────
> 
>>> I don't know enough about Org to understand why overlays are being
>>> considered instead of text properties, but feel free to describe the
>>> issues (preferably on emacs-devel) and start a discussion about the
>>> possible alternatives.
> 
> Best regards,
>  Seb
> 
> -- 
> Sebastien Vauban
> 
> 

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> From: Carsten Dominik <carsten.dominik@gmail.com>
> Date: Thu, 11 Apr 2013 04:58:15 +0200
> Cc: "emacs-orgmode@gnu.org List" <emacs-orgmode@gnu.org>,
>  Eli Zaretskii <eliz@gnu.org>
> 
> > I guess Eli simply means, in a general way, that overlays do negatively impact
> > display performance, as you said as well a couple of times:
> 
> Yes, but Eli says that Org already severely tests the
> display engine, and he uses the word "mess", even though
> we mostly use text properties for faces and other
> display-related things.

Well, don't interpret "mess" too literally ;-)

> Of course, Org already uses overlays, for example for
> folding (as does outline.el), and for temporary marking
> of text like during src block editing.  But as your digging
> shows, I ave avoided them in the past, and we are also not
> using them for org-indent.el, for example.
> 
> The reason why I said "overlays would be better" is simply
> that they would allow to add display properties in a
> persistent way that would not interfere that our
> font-lock-unfontify-region function removes face and
> invisibility text properties.  So they are "better" for
> implementing hand-made faces selection that should overrule
> font-lock.

Overlays should be OK as long as they aren't too many, and as long as
you don't move them around too much, particularly in post-command-hook
or some such.

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 11.4.2013, at 19:30, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Carsten Dominik <carsten.dominik@gmail.com>
>> Date: Thu, 11 Apr 2013 04:58:15 +0200
>> Cc: "emacs-orgmode@gnu.org List" <emacs-orgmode@gnu.org>,
>> Eli Zaretskii <eliz@gnu.org>
>> 
>>> I guess Eli simply means, in a general way, that overlays do negatively impact
>>> display performance, as you said as well a couple of times:
>> 
>> Yes, but Eli says that Org already severely tests the
>> display engine, and he uses the word "mess", even though
>> we mostly use text properties for faces and other
>> display-related things.
> 
> Well, don't interpret "mess" too literally ;-)

I will add an overlay that displays "mess" as "mix" :D

> 
>> Of course, Org already uses overlays, for example for
>> folding (as does outline.el), and for temporary marking
>> of text like during src block editing.  But as your digging
>> shows, I ave avoided them in the past, and we are also not
>> using them for org-indent.el, for example.
>> 
>> The reason why I said "overlays would be better" is simply
>> that they would allow to add display properties in a
>> persistent way that would not interfere that our
>> font-lock-unfontify-region function removes face and
>> invisibility text properties.  So they are "better" for
>> implementing hand-made faces selection that should overrule
>> font-lock.
> 
> Overlays should be OK as long as they aren't too many, and as long as
> you don't move them around too much, particularly in post-command-hook
> or some such.

This explanation sounds to me as if the display engine is building
some kind of tree of overlays to find properties changes quickly.  Too bad I don't have time to understand this stuff in more detail, it sounds interesting.

- Carsten

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> From: Carsten Dominik <carsten.dominik@gmail.com>
> Date: Fri, 12 Apr 2013 00:49:32 +0200
> Cc: "emacs-orgmode@gnu.org List" <emacs-orgmode@gnu.org>
> 
> > Overlays should be OK as long as they aren't too many, and as long as
> > you don't move them around too much, particularly in post-command-hook
> > or some such.
> 
> This explanation sounds to me as if the display engine is building
> some kind of tree of overlays to find properties changes quickly.  Too bad I don't have time to understand this stuff in more detail, it sounds interesting.

Just search xdisp.c for "overlay", you will see the story quite
clearly, I think.

There are actually 2 aspects that make display engine's job harder
with overlays.  One is indeed that finding overlays that "cover" a
given buffer position is not a very efficient operation.  The display
engine tries to overcome this to some extent by "centering" the
overlay data base around the place in the buffer it is rendering.
This is done for every screen line that is being examined by the
display code.

The other problem is that there's no easy way of finding out which
parts of the buffer are being affected by changes in overlays.  The
consequence of this is that redisplay cannot easily determine whether
a given portion of what's on the glass needs to be redrawn when
overlays change.  The analysis of which part(s) of a window need to be
redrawn is at the heart of redisplay optimizations, whereby Emacs
tries very hard to reuse the portions of display that are already up
to date.  Changes in overlays defeat that.  Therefore, changing _any_
overlays in a buffer disables most, if not all, redisplay
optimizations, meaning that the entire portion of the buffer that is
displayed will be completely redrawn each time overlays _anywhere_ in
the buffer are changed.

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 12 apr. 2013, at 08:41, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Carsten Dominik <carsten.dominik@gmail.com>
>> Date: Fri, 12 Apr 2013 00:49:32 +0200
>> Cc: "emacs-orgmode@gnu.org List" <emacs-orgmode@gnu.org>
>> 
>>> Overlays should be OK as long as they aren't too many, and as long as
>>> you don't move them around too much, particularly in post-command-hook
>>> or some such.
>> 
>> This explanation sounds to me as if the display engine is building
>> some kind of tree of overlays to find properties changes quickly.  Too bad I don't have time to understand this stuff in more detail, it sounds interesting.
> 
> Just search xdisp.c for "overlay", you will see the story quite
> clearly, I think.

My Sunday pleasure reading project.

> 
> There are actually 2 aspects that make display engine's job harder
> with overlays.  One is indeed that finding overlays that "cover" a
> given buffer position is not a very efficient operation.  The display
> engine tries to overcome this to some extent by "centering" the
> overlay data base around the place in the buffer it is rendering.
> This is done for every screen line that is being examined by the
> display code.
> 
> The other problem is that there's no easy way of finding out which
> parts of the buffer are being affected by changes in overlays.  The
> consequence of this is that redisplay cannot easily determine whether
> a given portion of what's on the glass needs to be redrawn when
> overlays change.  The analysis of which part(s) of a window need to be
> redrawn is at the heart of redisplay optimizations, whereby Emacs
> tries very hard to reuse the portions of display that are already up
> to date.  Changes in overlays defeat that.  Therefore, changing _any_
> overlays in a buffer disables most, if not all, redisplay
> optimizations, meaning that the entire portion of the buffer that is
> displayed will be completely redrawn each time overlays _anywhere_ in
> the buffer are changed.

Wow.  OK.  I need to think to that, and I will try to take a
fresh look at overlay use in Org.

So the reason that the combination with hi-line is slow is because
hl-line is using post-command-hook to move its overlay, and redisplay
of a full window with org-mode is slow because so much stuff is
hidden and Emacs makes a full re-evaluation of what needs
to be displayed?

This makes sense.

Thanks for taking the time to get me this far.

- Carsten

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> From: Carsten Dominik <carsten.dominik@gmail.com>
> Date: Fri, 12 Apr 2013 09:13:47 +0200
> Cc: emacs-orgmode@gnu.org
> 
> > Just search xdisp.c for "overlay", you will see the story quite
> > clearly, I think.
> 
> My Sunday pleasure reading project.

Good luck, and let me know if you need something explained.  The
commentary at the beginning of the file might serve as an
introduction, although it doesn't really touch the issue at hand.

> So the reason that the combination with hi-line is slow is because
> hl-line is using post-command-hook to move its overlay, and redisplay
> of a full window with org-mode is slow because so much stuff is
> hidden and Emacs makes a full re-evaluation of what needs
> to be displayed?

Right.  If hi-line (or any similar mode) is off, then at least
horizontal cursor motion should be fast, because then Emacs knows that
nothing changed, and finding the place where to put the cursor on the
same line it was before is relatively easy.

But even C-n and C-p is quite another story in an Org buffer: Emacs
needs to determine where that puts point, and doing so generally means
traversing all of the hidden parts of the buffer between the line
which was current and the new current line.  In a complex Org buffer,
that could easily be many thousands of buffer positions.

Also, recall that, under line-move-visual, which is nowadays on by
default, Emacs moves by _screen_ lines, not by physical lines.  So a
simple C-n must internally emulate display to find the next line
visible on the screen by traversing the buffer one character at a time
and taking note of each and every text property and overlay in
between, until it finds the buffer position whose screen coordinates
are [X,Y+N], where [X,Y] are the coordinates of the previous cursor
position and N is the line height in pixels.  And this is just to find
where point will be; then the screen must actually be redisplayed,
which might mean more work, if the new position of point requires
scrolling, e.g. if cursor went off the scroll margins or whatever.

We only get reasonably fast performance with all this complexity
because our machines are incredibly fast.  But we are many times on
the edge, as the bug I cited and similar ones show.

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 12 apr. 2013, at 10:31, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Carsten Dominik <carsten.dominik@gmail.com>
>> Date: Fri, 12 Apr 2013 09:13:47 +0200
>> Cc: emacs-orgmode@gnu.org
>> 
>>> Just search xdisp.c for "overlay", you will see the story quite
>>> clearly, I think.
>> 
>> My Sunday pleasure reading project.
> 
> Good luck, and let me know if you need something explained.  The
> commentary at the beginning of the file might serve as an
> introduction, although it doesn't really touch the issue at hand.
> 
>> So the reason that the combination with hi-line is slow is because
>> hl-line is using post-command-hook to move its overlay, and redisplay
>> of a full window with org-mode is slow because so much stuff is
>> hidden and Emacs makes a full re-evaluation of what needs
>> to be displayed?
> 
> Right.  If hi-line (or any similar mode) is off, then at least
> horizontal cursor motion should be fast, because then Emacs knows that
> nothing changed, and finding the place where to put the cursor on the
> same line it was before is relatively easy.
> 
> But even C-n and C-p is quite another story in an Org buffer: Emacs
> needs to determine where that puts point, and doing so generally means
> traversing all of the hidden parts of the buffer between the line
> which was current and the new current line.  In a complex Org buffer,
> that could easily be many thousands of buffer positions.

I guess outline mode does have the exact same problem in this case, in
fact any mode with large amount of hidden text.

> 
> Also, recall that, under line-move-visual, which is nowadays on by
> default,

Not in my setup, but since it the default, yes, this causes more
issues.  Another important point to be aware of.


> Emacs moves by _screen_ lines, not by physical lines.  So a
> simple C-n must internally emulate display to find the next line
> visible on the screen by traversing the buffer one character at a time
> and taking note of each and every text property and overlay in
> between, until it finds the buffer position whose screen coordinates
> are [X,Y+N], where [X,Y] are the coordinates of the previous cursor
> position and N is the line height in pixels.  And this is just to find
> where point will be; then the screen must actually be redisplayed,
> which might mean more work, if the new position of point requires
> scrolling, e.g. if cursor went off the scroll margins or whatever.
> 
> We only get reasonably fast performance with all this complexity
> because our machines are incredibly fast.  But we are many times on
> the edge, as the bug I cited and similar ones show.

Thanks again.

- Carsten

Re: Enriched/Org is a colorful Org

[Torsten Wagner]

[-- Attachment #1: Type: text/plain, Size: 4085 bytes --]

Hi,

just want to add some observation. I guess it has nothing to do with the
display engine but it might be somehow related. I used to use line-mode to
display line-numbers as a left column on all my buffers.
I noticed a very painful slowdown up to a totally unusable state during
working on very large org-files. It consisted of coursework for a
programming class and contained single headers with the student-id numbers
and a babel-code block in the headers body (hence, easily goes into 1000th
of lines). I was happy with it since I could execute and proof each
submitted coursework within a single org-file and folding helped me to move
quickly from one to the other coursework.
However, as longer as the list get as more it slowed down.

After some fiddling and searching, I noticed that the line-mode was kind
of struggling with the org-mode text-collapse feature. Whenever, I closed a
header, it took large amount of times to recalculate the line-numbers. Not
sure where exactly line-mode did consume the time. But it might as well
be related to the redisplaying of the numbers. Switching off the line-mode
made the time delay disappear completely.

Just an observation which might or might not related to the later
discussion.

Torsten




On 12 April 2013 12:56, Carsten Dominik <carsten.dominik@gmail.com> wrote:

>
> On 12 apr. 2013, at 10:31, Eli Zaretskii <eliz@gnu.org> wrote:
>
> >> From: Carsten Dominik <carsten.dominik@gmail.com>
> >> Date: Fri, 12 Apr 2013 09:13:47 +0200
> >> Cc: emacs-orgmode@gnu.org
> >>
> >>> Just search xdisp.c for "overlay", you will see the story quite
> >>> clearly, I think.
> >>
> >> My Sunday pleasure reading project.
> >
> > Good luck, and let me know if you need something explained.  The
> > commentary at the beginning of the file might serve as an
> > introduction, although it doesn't really touch the issue at hand.
> >
> >> So the reason that the combination with hi-line is slow is because
> >> hl-line is using post-command-hook to move its overlay, and redisplay
> >> of a full window with org-mode is slow because so much stuff is
> >> hidden and Emacs makes a full re-evaluation of what needs
> >> to be displayed?
> >
> > Right.  If hi-line (or any similar mode) is off, then at least
> > horizontal cursor motion should be fast, because then Emacs knows that
> > nothing changed, and finding the place where to put the cursor on the
> > same line it was before is relatively easy.
> >
> > But even C-n and C-p is quite another story in an Org buffer: Emacs
> > needs to determine where that puts point, and doing so generally means
> > traversing all of the hidden parts of the buffer between the line
> > which was current and the new current line.  In a complex Org buffer,
> > that could easily be many thousands of buffer positions.
>
> I guess outline mode does have the exact same problem in this case, in
> fact any mode with large amount of hidden text.
>
> >
> > Also, recall that, under line-move-visual, which is nowadays on by
> > default,
>
> Not in my setup, but since it the default, yes, this causes more
> issues.  Another important point to be aware of.
>
>
> > Emacs moves by _screen_ lines, not by physical lines.  So a
> > simple C-n must internally emulate display to find the next line
> > visible on the screen by traversing the buffer one character at a time
> > and taking note of each and every text property and overlay in
> > between, until it finds the buffer position whose screen coordinates
> > are [X,Y+N], where [X,Y] are the coordinates of the previous cursor
> > position and N is the line height in pixels.  And this is just to find
> > where point will be; then the screen must actually be redisplayed,
> > which might mean more work, if the new position of point requires
> > scrolling, e.g. if cursor went off the scroll margins or whatever.
> >
> > We only get reasonably fast performance with all this complexity
> > because our machines are incredibly fast.  But we are many times on
> > the edge, as the bug I cited and similar ones show.
>
> Thanks again.
>
> - Carsten
>
>

[-- Attachment #2: Type: text/html, Size: 5264 bytes --]

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> Date: Fri, 12 Apr 2013 13:49:47 +0200
> From: Torsten Wagner <torsten.wagner@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Org Mode Mailing List <emacs-orgmode@gnu.org>
> just want to add some observation. I guess it has nothing to do with the
> display engine but it might be somehow related. I used to use line-mode to
> display line-numbers as a left column on all my buffers.
> I noticed a very painful slowdown up to a totally unusable state during
> working on very large org-files. It consisted of coursework for a
> programming class and contained single headers with the student-id numbers
> and a babel-code block in the headers body (hence, easily goes into 1000th
> of lines). I was happy with it since I could execute and proof each
> submitted coursework within a single org-file and folding helped me to move
> quickly from one to the other coursework.
> However, as longer as the list get as more it slowed down.
> 
> After some fiddling and searching, I noticed that the line-mode was kind
> of struggling with the org-mode text-collapse feature. Whenever, I closed a
> header, it took large amount of times to recalculate the line-numbers. Not
> sure where exactly line-mode did consume the time. But it might as well
> be related to the redisplaying of the numbers. Switching off the line-mode
> made the time delay disappear completely.

So this was an Org file with only 1 level of headers, but with large
blocks of text between the headers, is that right?

Can you show an example of such babel-code block?  I'd like to look
into the slowdown and find its reasons.

In general, linum does exactly what defeats redisplay optimizations:
it modifies overlays in a post-command-hook.  But that doesn't mean
this was the reason for the slow operation you saw, it could be
something else.

If you can easily produce a recipe for recreating the problem, it
might be worthwhile to file an Emacs bug report.

Thanks.

Re: Enriched/Org is a colorful Org

[Suvayu Ali]

Hi Eli,

I hope you don't mind me taking this opportunity to ask a tangential
question.

On Fri, Apr 12, 2013 at 04:03:10PM +0300, Eli Zaretskii wrote:
> In general, linum does exactly what defeats redisplay optimizations:
> it modifies overlays in a post-command-hook.  But that doesn't mean

If some package wants to keep something updated (line number displays in
this case), is using the post-command-hook the only option?

-- 
Suvayu

Open source is the future. It sets us free.

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> Date: Fri, 12 Apr 2013 20:00:56 +0200
> From: Suvayu Ali <fatkasuvayu+linux@gmail.com>
> Cc: Torsten Wagner <torsten.wagner@gmail.com>, emacs-orgmode@gnu.org,
> 	carsten.dominik@gmail.com
> 
> If some package wants to keep something updated (line number displays in
> this case), is using the post-command-hook the only option?

No.  The other one is piggy-back jit-lock.  See nlinum in ELPA for an
example, which actually is a drop-in replacement for linum, but
without many of its problems.

Re: Enriched/Org is a colorful Org

[Suvayu Ali]

Hi Eli,

On Fri, Apr 12, 2013 at 09:38:47PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 12 Apr 2013 20:00:56 +0200
> > From: Suvayu Ali <fatkasuvayu+linux@gmail.com>
> > Cc: Torsten Wagner <torsten.wagner@gmail.com>, emacs-orgmode@gnu.org,
> > 	carsten.dominik@gmail.com
> > 
> > If some package wants to keep something updated (line number displays in
> > this case), is using the post-command-hook the only option?
> 
> No.  The other one is piggy-back jit-lock.  See nlinum in ELPA for an
> example, which actually is a drop-in replacement for linum, but
> without many of its problems.

Thanks a lot :)

-- 
Suvayu

Open source is the future. It sets us free.

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> From: Carsten Dominik <carsten.dominik@gmail.com>
> Date: Fri, 12 Apr 2013 12:56:11 +0200
> Cc: emacs-orgmode@gnu.org
> 
> I guess outline mode does have the exact same problem in this case, in
> fact any mode with large amount of hidden text.

Of course.  The only difference is that outline is not as popular as
Org, and usually is used with relatively short chunks of text.  But
the problems are exactly the same.

Re: Enriched/Org is a colorful Org

[Sean O'Halpin]

In your opinion, would it be possible to reproduce the functionality
of outline-mode using text properties rather than overlays? And in the
case of org-mode, would this really make that much of a difference in
terms of performance?

Regards,
Sean

On Fri, Apr 12, 2013 at 1:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Carsten Dominik <carsten.dominik@gmail.com>
>> Date: Fri, 12 Apr 2013 12:56:11 +0200
>> Cc: emacs-orgmode@gnu.org
>>
>> I guess outline mode does have the exact same problem in this case, in
>> fact any mode with large amount of hidden text.
>
> Of course.  The only difference is that outline is not as popular as
> Org, and usually is used with relatively short chunks of text.  But
> the problems are exactly the same.
>

Re: Enriched/Org is a colorful Org

[Jambunathan K]

"Sean O'Halpin" <sean.ohalpin@gmail.com> writes:

> In your opinion, would it be possible to reproduce the functionality
> of outline-mode using text properties rather than overlays? And in the
> case of org-mode, would this really make that much of a difference in
> terms of performance?

Broadly speaking, answer seems to be a "Yes".  It is possible that some
details have to be hammered out wrt indirect buffers.

See the last item in this post

        http://lists.gnu.org/archive/html/emacs-devel/2010-10/msg00177.html

>
> Regards,
> Sean
>
> On Fri, Apr 12, 2013 at 1:36 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Carsten Dominik <carsten.dominik@gmail.com>
>>> Date: Fri, 12 Apr 2013 12:56:11 +0200
>>> Cc: emacs-orgmode@gnu.org
>>>
>>> I guess outline mode does have the exact same problem in this case, in
>>> fact any mode with large amount of hidden text.
>>
>> Of course.  The only difference is that outline is not as popular as
>> Org, and usually is used with relatively short chunks of text.  But
>> the problems are exactly the same.
>>

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

> Date: Sat, 13 Apr 2013 13:24:13 +0100
> From: "Sean O'Halpin" <sean.ohalpin@gmail.com>
> Cc: Carsten Dominik <carsten.dominik@gmail.com>, Org Mode <emacs-orgmode@gnu.org>
> 
> In your opinion, would it be possible to reproduce the functionality
> of outline-mode using text properties rather than overlays?

This needs to be analyzed.  Outline mode uses several features that
are specific to overlays, but not to text properties:
isearch-open-invisible and front-advance.  (It also uses 'evaporate',
but that happens automatically with text properties.)  I think
front-advance will happen automatically, but searching inside
invisible text is currently supported only for overlays (for reasons
that I consider irrelevant, but that's me).

> And in the case of org-mode, would this really make that much of a
> difference in terms of performance?

I don't know, because I didn't measure that, and neither did I see any
measurements published.  If there are many overlays in a buffer,
redisplay performance will certainly be better with text properties,
but I don't know whether this improvement will be significant: there
could be other factors at work that affect performance to a much
greater effect.  If we want to know for sure, there's no way around
profiling this.  Luckily, we now have profiler.el to help us.

Re: Enriched/Org is a colorful Org

[Bastien]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Thanks for taking the time to get me this far.

+1!

Thanks Eli, great to learn about the internals of Emacs display
engine.  The Emacs Lisp manual already contains some directions
and warnings, but not so detailed.

-- 
 Bastien

Re: Enriched/Org is a colorful Org

[François Pinard]

Bastien <bzg@gnu.org> writes:

> Thanks Eli, great to learn about the internals of Emacs display
> engine.

Eli is, and always has been, quite a resourceful man.  And along the
years, I got the pleasure of discovering him as a good friend too! :-)

François

Re: Enriched/Org is a colorful Org

[Samuel Wales]

I want to add to the thanks to everybody for making speed improvements.

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  ANYBODY
can get it.  There is NO hope without action.  This means YOU.

Re: Enriched/Org is a colorful Org

[Eli Zaretskii]

[Please CC me on responses, as I'm not subscribed to this list.]

> From: Carsten Dominik <carsten.dominik@gmail.com>
> Date: Wed, 10 Apr 2013 21:58:06 +0200
> Cc: emacs-orgmode@gnu.org
> 
> > I beg the Org developers to please be very careful when introducing
> > expensive display features such as overlays into Org.  Org already
> > puts the Emacs display engine to its limits in many of its popular
> > features;
> 
> this is interesting input, I was not aware of this.  Has this been discussed before, can you point me to relevant threads, and what are the symptoms of the display engine being at its limits?

You won't find explicit discussions of this, except maybe a random
comment from me here and there.  There aren't too many discussions
about the display engine in general; maybe it's my fault.

But you can find indirect evidence to what I say in quite a few
reports about slow redisplay.  Here's one example (it's just the first
one that popped up on Google):

  http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00276.html

Note how two display features: bidi and hl-line -- each one of them
cause significant slow-down in Org buffers, and almost nowhere else.

This is just an example.  I keep bumping into similar issues
frequently enough to lead me to the conclusion you see above.

In general, hiding from display large parts of a buffer, and using a
lot of display strings and overlays that add to buffer text or replace
buffer text with something else -- these all make redisplay much more
expensive.  In particular, moving overlays disables many redisplay
optimizations, so e.g. any mode that moves overlays as result of
post-command-hook will considerably slow down display and degrade user
experience.

After hacking the display code for a few years, it is painfully clear
to me that its basic design assumed that such use cases are rare.  Org
mode makes these assumptions more and more false, and it does that
faster than the CPU speed improves ;-)

For these reasons, and as long as we don't have any development going
on that aims at a complete redesign of the display engine, I think
every feature, especially one expected to be popular, that adversely
impacts redisplay efficiency, should be considered very carefully, and
the various alternatives for its implementation assessed also from
this aspect.

HTH

Re: Enriched/Org is a colorful Org

[Carsten Dominik]

On 11.4.2013, at 19:27, Eli Zaretskii <eliz@gnu.org> wrote:

> [Please CC me on responses, as I'm not subscribed to this list.]
> 
>> From: Carsten Dominik <carsten.dominik@gmail.com>
>> Date: Wed, 10 Apr 2013 21:58:06 +0200
>> Cc: emacs-orgmode@gnu.org
>> 
>>> I beg the Org developers to please be very careful when introducing
>>> expensive display features such as overlays into Org.  Org already
>>> puts the Emacs display engine to its limits in many of its popular
>>> features;
>> 
>> this is interesting input, I was not aware of this.  Has this been discussed before, can you point me to relevant threads, and what are the symptoms of the display engine being at its limits?
> 
> You won't find explicit discussions of this, except maybe a random
> comment from me here and there.  There aren't too many discussions
> about the display engine in general; maybe it's my fault.
> 
> But you can find indirect evidence to what I say in quite a few
> reports about slow redisplay.  Here's one example (it's just the first
> one that popped up on Google):
> 
>  http://lists.gnu.org/archive/html/emacs-devel/2011-09/msg00276.html
> 
> Note how two display features: bidi and hl-line -- each one of them
> cause significant slow-down in Org buffers, and almost nowhere else.

> This is just an example.  I keep bumping into similar issues
> frequently enough to lead me to the conclusion you see above.

Yes, OK, I also remember reports like this.  Funny, often it is not Org by itself, but in combination with something else that affects the display engine.

> 
> In general, hiding from display large parts of a buffer, and using a
> lot of display strings and overlays that add to buffer text or replace
> buffer text with something else -- these all make redisplay much more
> expensive.  In particular, moving overlays disables many redisplay
> optimizations, so e.g. any mode that moves overlays as result of
> post-command-hook will considerably slow down display and degrade user
> experience.

OK, this is a concrete thing we can be on the lookout for.  I don't think we do that, but I will take a look.

> 
> After hacking the display code for a few years, it is painfully clear
> to me that its basic design assumed that such use cases are rare.  Org
> mode makes these assumptions more and more false, and it does that
> faster than the CPU speed improves ;-)
> 
> For these reasons, and as long as we don't have any development going
> on that aims at a complete redesign of the display engine, I think
> every feature, especially one expected to be popular, that adversely
> impacts redisplay efficiency, should be considered very carefully, and
> the various alternatives for its implementation assessed also from
> this aspect.

This is clear enough.  I will try to keep this in mind and evaluate changes in Org in this way.  If you have other concrete things where you think Org could be improved in this direction, let us know.

> 
> HTH

Certainly, thank you.

- Carsten

Re: Enriched/Org is a colorful Org

[Jambunathan K]

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> On Wed, Apr 10, 2013 at 09:32:44AM +0530, Jambunathan K wrote:
>> 
>> See "Side note" towards the end of this message
>> 
>>         http://debbugs.gnu.org/cgi/bugreport.cgi?bug=14157#8
>> 
>
> This request is common enough; every time it comes up overlays are
> proposed as a solution.  It would be good if this is available even as a
> library outside of Org.

It is possible with little work of existing libraries.