Q on performance with 10000 faces

Q on performance with 10000 faces

[Drew Adams]

I have a color palette that is 100 chars by 100 chars. Each char has a
different face text-property value. Once the palette is displayed, it brings
Emacs to a crawl. That is, just having the palette displayed makes
everything in Emacs slow to a crawl from then on. If the palette is
iconified, then there is no problem with response time. It is the mere
display of it that clogs the wheels.

There was a thread in 2005/11 about slow frame creation when there are
numerous faces. That is not the problem here - I can live with slow frame
creation. The problem is that after the frame with the faces is created
Emacs becomes and remains unbearably slow.

Reducing the character size to, say, 3, improves responsiveness a little
bit, I think (and that smaller size is in fact what I want), but things are
still so slow that it makes the feature unusable.

Does this mean that Emacs cannot reasonably be expected to display 10000
face text-properties? If so, any advice on workarounds or other approaches?
Is the slowdown perhaps because of some kind of automatic updating or
refreshing, which I could turn off somehow (I hope)? The palette is
constant; there is no need to refresh it (except because the frame is moved
etc.). Its face text-property values will never change.

The palette is quite nice visually, and I would like to be able to use it
(e.g. for color selection), but this is obviously not the way to go. I'm
hoping this might be a bug and not a design limitation, but perhaps that
hope is unrealistic.

In GNU Emacs 22.0.50.1 (i386-mingw-nt5.1.2600)
 of 2006-03-20 on W2ONE
X server distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4) --cflags -Id:/g/include'

Re: Q on performance with 10000 faces

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sun, 21 May 2006 17:02:40 -0700
> 
> Does this mean that Emacs cannot reasonably be expected to display 10000
> face text-properties?

AFAIK, yes.  Doing what you did (a) disables all possible display
optimizations that the redisplay engine has up its sleeve to speed up
the common cases, and (b) forces Emacs to traverse the 10000 text
properties for each character it is about to display.

> If so, any advice on workarounds or other approaches?

I don't know of any, except ``don't do that'', but perhaps Kim or
Stefan will.

> Is the slowdown perhaps because of some kind of automatic updating or
> refreshing, which I could turn off somehow (I hope)?

It's because even cursor motion, that usually takes a fast shortcut
through the redisplay code, requires to search all the text properties
in your situation.

RE: Q on performance with 10000 faces

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 2806 bytes --]

    > Does this mean that Emacs cannot reasonably be expected to
    > display 10000 face text-properties?

    AFAIK, yes.  Doing what you did (a) disables all possible display
    optimizations that the redisplay engine has up its sleeve to speed up
    the common cases, and (b) forces Emacs to traverse the 10000 text
    properties for each character it is about to display.

What is it that you think I did that causes that? Perhaps I could do things
differently. Do you mean just the fact of using 10000 different face text
properties or something else also?

    > If so, any advice on workarounds or other approaches?

    I don't know of any, except ``don't do that'', but perhaps Kim or
    Stefan will.

Don't do what? Use that many faces or something else also?

    > Is the slowdown perhaps because of some kind of automatic updating or
    > refreshing, which I could turn off somehow (I hope)?

    It's because even cursor motion, that usually takes a fast shortcut
    through the redisplay code, requires to search all the text properties
    in your situation.

I'm a little surprised that a frame that is not selected and has no current
activity (no cursor motion etc.), would continue to slow Emacs down just by
being displayed on the computer screen (and listening for input). I can see
why activity within that frame (or moving the frame, or moving another frame
across it, etc.) might cause Emacs to slow down due to redisplay, but the
same slowness continues even when that frame is ignored (just sits there,
selected or unselected).

I'd appreciate any advice on achieving the same end, either in some other
way or by somehow tweaking this approach.

I guess one alternative would be to insert an image of a complete palette
(like the image attached) and implement something similar to an HTML image
map (unless that functionality already exists in Emacs), to pick up the
position of a mouse click. That is, monitor mouse-click positions, and work
backward to determine the picked color. But that would require associating
the original color info (e.g. #RRRRGGGGBBBB with image positions. The
advantage of the approach I chose is that each character's text property has
the color info within it. Of course, if the approach isn't realistic due to
display performance, it's back to the drawing board.

I'm not familiar with the Emacs treatment of images. Looking in the Elisp
manual for info on them, I didn't see anything similar to HTML image maps.
Is there such a functionality already?

Attached is a screen capture of my hue x saturation palette (the 100 char x
100 char frame that slows Emacs down), to give you an idea of the kind of
thing I'd like to do. (The frame could be made square if I could figure out
a way to get rid of inter-line spacing.)

Thanks for the info.

[-- Attachment #2: hue-x-saturation-palette.jpg --]
[-- Type: image/jpeg, Size: 31291 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Q on performance with 10000 faces

[Miles Bader]

"Drew Adams" <drew.adams@oracle.com> writes:
> (The frame could be made square if I could figure out
> a way to get rid of inter-line spacing.)

I think you can use the `line-height' text property (on newlines) or the
`line-spacing' frame parameter.

See:  (elisp)Line Height

-miles
-- 
`There are more things in heaven and earth, Horatio,
 Than are dreamt of in your philosophy.'

RE: Q on performance with 10000 faces

[Drew Adams]

    > (The frame could be made square if I could figure out
    > a way to get rid of inter-line spacing.)

    I think you can use the `line-height' text property (on newlines) or the
    `line-spacing' frame parameter.    See:  (elisp)Line Height

Doesn't work, AFAICT. See bug report from yesterday.

Re: Q on performance with 10000 faces

[Kim F. Storm]

"Drew Adams" <drew.adams@oracle.com> writes:

> I guess one alternative would be to insert an image of a complete palette
> (like the image attached) 

You can construct that image on the fly using eg. XPM format.

To overcome limitations in number of available colors, you could
construct 100 images covering one line each.

Then use the line-height property of the newline character to
make the display "tight":

      (insert (propertize "\n" 'line-height t)))))

Finally, you may use the :map property on images:

`:map MAP'
     This associates an image map of "hot spots" with this image.

     An image map is an alist where each element has the format `(AREA
     ID PLIST)'.  An AREA is specified as either a rectangle, a circle,
     or a polygon.

     A rectangle is a cons `(rect . ((X0 . Y0) . (X1 . Y1)))' which
     specifies the pixel coordinates of the upper left and bottom right
     corners of the rectangle area.

     A circle is a cons `(circle . ((X0 . Y0) . R))' which specifies
     the center and the radius of the circle; R may be a float or
     integer.

     A polygon is a cons `(poly . [X0 Y0 X1 Y1 ...])' where each pair
     in the vector describes one corner in the polygon.

     When the mouse pointer is above a hot-spot area of an image, the
     PLIST of that hot-spot is consulted; if it contains a `help-echo'
     property it defines a tool-tip for the hot-spot, and if it contains
     a `pointer' property, it defines the shape of the mouse cursor when
     it is over the hot-spot.  *Note Pointer Shape::, for available
     pointer shapes.

     When you click the mouse when the mouse pointer is over a
     hot-spot, an event is composed by combining the ID of the hot-spot
     with the mouse event; for instance, `[area4 mouse-1]' if the
     hot-spot's ID is `area4'.


Or simply use a common mouse click function and use the x,y coordinates
of the mouse event to calculate which color is clicked.  
See posn-x-y and posn-object-x-y.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

RE: Q on performance with 10000 faces

[Drew Adams]

    > I guess one alternative would be to insert an image of a
    > complete palette (like the image attached)

    You can construct that image on the fly using eg. XPM format.

Where can I find doc to tell me how to do that? Anything in Emacs about it,
or should I just google for "xpm"?

    To overcome limitations in number of available colors, you could
    construct 100 images covering one line each.

What are the limitations you refer to here? I'm not clear how 100 single-row
images would be better than one 110-row image. Is it because the number of
colors is somehow limited per line?

    Then use the line-height property of the newline character to
    make the display "tight":

          (insert (propertize "\n" 'line-height t)))))

BTW, why does that work for images but not, apparently, for text lines? At
least it didn't seem to work for me - see bug report yesterday.

    Finally, you may use the :map property on images:

    `:map MAP'
         This associates an image map of "hot spots" with this image.

That's what I couldn't find (what I meant by HTML image maps). Thx.

         An image map is an alist where each element has the format `(AREA
         ID PLIST)'.  An AREA is specified as either a rectangle, a circle,
         or a polygon.

         A rectangle is a cons `(rect . ((X0 . Y0) . (X1 . Y1)))' which
         specifies the pixel coordinates of the upper left and bottom right
         corners of the rectangle area.

         A circle is a cons `(circle . ((X0 . Y0) . R))' which specifies
         the center and the radius of the circle; R may be a float or
         integer.

         A polygon is a cons `(poly . [X0 Y0 X1 Y1 ...])' where each pair
         in the vector describes one corner in the polygon.

         When the mouse pointer is above a hot-spot area of an image, the
         PLIST of that hot-spot is consulted; if it contains a `help-echo'
         property it defines a tool-tip for the hot-spot, and if it contains
         a `pointer' property, it defines the shape of the mouse cursor when
         it is over the hot-spot.  *Note Pointer Shape::, for available
         pointer shapes.

         When you click the mouse when the mouse pointer is over a
         hot-spot, an event is composed by combining the ID of the hot-spot
         with the mouse event; for instance, `[area4 mouse-1]' if the
         hot-spot's ID is `area4'.

    Or simply use a common mouse click function and use the x,y coordinates
    of the mouse event to calculate which color is clicked.
    See posn-x-y and posn-object-x-y.

That's what I meant. That would be my next move: just use the image I sent,
knowing that it fits over the character grid I made originally and that I
can record the colors of.

Re: Q on performance with 10000 faces

[Stefan Monnier]

>> Does this mean that Emacs cannot reasonably be expected to
>> display 10000 face text-properties?

>     AFAIK, yes.  Doing what you did (a) disables all possible display
>     optimizations that the redisplay engine has up its sleeve to speed up
>     the common cases,

Indeed, changes in the face property take time.  I wouldn't say that what
he describes disables any particular optimization, but it makes
them ineffective.

>     and (b) forces Emacs to traverse the 10000 text
>     properties for each character it is about to display.

I don't see why the redisplay would have to traverse 10000 text properties
for each char it is about to display.  After all, there are 10000 chars each
with a different face property, but each one only has one text property (or
maybe a couple, but not 10000).

> Don't do what? Use that many faces or something else also?

It could be either don't use that many face-changes, or don't use that
many faces.  I suspect that the number of faces is what bogs you down which
is why it also slows down redisplay of other windows.  But I'm very
non-knowledgeable about the redisplay code.

>     It's because even cursor motion, that usually takes a fast shortcut
>     through the redisplay code, requires to search all the text properties
>     in your situation.

I don't see why that'd be the case.

> I'm a little surprised that a frame that is not selected and has no current
> activity (no cursor motion etc.), would continue to slow Emacs down just by
> being displayed on the computer screen (and listening for input).

Agreed.  I think there is a missing redisplay optimization here (and
I suspect that I sometimes suffer from a related missing optimization when
I have 100 frames open).


        Stefan

RE: Q on performance with 10000 faces

[Drew Adams]

    > I'm a little surprised that a frame that is not selected and
    > has no current
    > activity (no cursor motion etc.), would continue to slow
    > Emacs down just by
    > being displayed on the computer screen (and listening for input).

    Agreed.  I think there is a missing redisplay optimization here (and
    I suspect that I sometimes suffer from a related missing
    optimization when I have 100 frames open).

If this gets fixed (optimized), then the performance might be usable and I
might stick with this design - it has a few advantages. Any chance someone
will look at this problem (considered as a bug) before the release?

Re: Q on performance with 10000 faces

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Mon, 22 May 2006 06:49:18 -0700
> 
> If this gets fixed (optimized), then the performance might be usable and I
> might stick with this design - it has a few advantages. Any chance someone
> will look at this problem (considered as a bug) before the release?

Not without you posting some minimal code that creates this slowdown.

Re: Q on performance with 10000 faces

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 22 May 2006 08:59:13 -0400
> Cc: emacs-devel@gnu.org
> 
> >     It's because even cursor motion, that usually takes a fast shortcut
> >     through the redisplay code, requires to search all the text properties
> >     in your situation.
> 
> I don't see why that'd be the case.

Granted, I could only guess, since no actual code was posted, but the
description of the symptoms is quite revealing.

RE: Q on performance with 10000 faces

[Drew Adams]

            > > It's because even cursor motion, that usually takes a 
            > > fast shortcut through the redisplay code, requires to
            > > search all the text properties in your situation.
            > 
            > I don't see why that'd be the case.
    
            Granted, I could only guess, since no actual code was
            posted, but the description of the symptoms is quite
            revealing.

          > Any chance someone will look at this problem (considered
          > as a bug) before the release?
    
          Not without you posting some minimal code that creates this
          slowdown.
 
        If you are interested, please post the shortest Lisp code
        that can be used to reproduce and investigate this situation.

    In hopes that someone might be able to see why the palette code 
    slows things down (or suggest improvements to it), here is the
    basic code: <snip>

Any news on this?

RE: Q on performance with 10000 faces

[Drew Adams]

    Any news on this?

A more complete version of the code is here, in case that helps:
http://www.emacswiki.org/cgi-bin/wiki/palette.el.

Re: Q on performance with 10000 faces

[Kim F. Storm]

"Drew Adams" <drew.adams@oracle.com> writes:

>     Any news on this?
>
> A more complete version of the code is here, in case that helps:
> http://www.emacswiki.org/cgi-bin/wiki/palette.el.

Note: It depends on hexrgb.el (although is says no dependencies).

Anyway, I tried your code, and I don't see any apparent slowdown, once
the palette frame is shown (but that takes a few seconds).

Repeated cursor movement doesn't happen instantly with the default
settings, but with (setq redisplay-dont-pause t), key response is ok.

How significant is the slowdown you see?

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

RE: Q on performance with 10000 faces

[Drew Adams]

Hi Kim,

    > A more complete version of the code is here, in case that helps:
    > http://www.emacswiki.org/cgi-bin/wiki/palette.el.

    Note: It depends on hexrgb.el

Yes.

    (although is says no dependencies).

You mean the "Features that might be required by this library:" comment in
the file header? Yes, I forgot to byte-compile the file and load the
byte-compiled version before I saved it, so that automatically generated
comment didn't get updated correctly. (There is an explicit (require
'hexrgb), at least.)

FYI - The way the automatic file-dependency header works is that if a
byte-compiled (`*.elc') version of a library is available it is loaded, in
preference to the source library - this is the standard behavior of
`load-library'. The list of required features then reflects the dependencies
indicated in the byte-compiled file, not the source file. If the
byte-compiled file is out-of-date, so will be the file header list of
possible dependencies. The code for this is here, BTW:
http://www.emacswiki.org/cgi-bin/wiki/lib-requires.el (and
http://www.emacswiki.org/cgi-bin/wiki/header2.el). Suggestions welcome.

    Anyway, I tried your code, and I don't see any apparent slowdown, once
    the palette frame is shown (but that takes a few seconds).

Yes, it takes about 4 sec on my machine to create the palette. That's not a
real problem (for me).

    Repeated cursor movement doesn't happen instantly with the default
    settings, but with (setq redisplay-dont-pause t), key response is ok.

Use of that variable is not clear to me, even after reading the doc. I was
thinking that non-nil would slow things down when there is pending input, by
allowing redisplay instead of inhibiting it. How does non-nil improve
response here? Where would you suggest binding it to non-nil - in each of
the interactive functions (in the palette code) or just some in particular?
I'm confused about this variable. Usage guidelines welcome.

Anyway, I tried setting redisplay-dont-pause to non-nil globally, just to
see what you mean, and I don't notice any improvement. I don't doubt you,
but I didn't notice a change, myself.

    How significant is the slowdown you see?

As I mentioned before, with various hooks and such that I have running in my
setup, M-x takes a couple of seconds to give me the prompt (once the palette
has been displayed). However, response for cursor movement, mouse clicking
etc. is not a problem. It's mainly minibuffer response that is slow.

That's the fault of other libraries I use, but the point is that I don't see
any slowdown unless I run the palette code. IOW, it slows things down enough
to be quite annoying, because I already have a lot going on. The other stuff
going on I never notice, in terms of response, but in combination with the
palette code, it's quite noticeable. That suggests to me that the palette
code is slowing things down, and that slowdown is just not very noticeable
with emacs -Q + palette.

In any case, even without any special attempt at optimization, the palette
seems quite usable. I'm sure there is room for optimization of my code, even
apart from any possible improvement in Emacs' display of faces.

[The problem I'm having now with the library is finding a font to use - each
user's font selection is different. I try to pick up a small font (size 5),
using a priority list of fonts and ending with whatever size-5 font can be
found. But I'm not convinced that is a good solution. All I need is a small
font, preferably fixed-width - and I use only two characters (space and
`e').]

Thanks for having taken a look.

Re: Q on performance with 10000 faces

[Kim F. Storm]

"Drew Adams" <drew.adams@oracle.com> writes:

>     How significant is the slowdown you see?
>
> As I mentioned before, with various hooks and such that I have running in my
> setup, M-x takes a couple of seconds to give me the prompt (once the palette
> has been displayed). However, response for cursor movement, mouse clicking
> etc. is not a problem. It's mainly minibuffer response that is slow.

Well, redisplay-dont-pause probably wont have any effect on that.

> That's the fault of other libraries I use, but the point is that I don't see
> any slowdown unless I run the palette code. IOW, it slows things down enough
> to be quite annoying, because I already have a lot going on. The other stuff
> going on I never notice, in terms of response, but in combination with the
> palette code, it's quite noticeable. That suggests to me that the palette
> code is slowing things down, and that slowdown is just not very noticeable
> with emacs -Q + palette.

That's odd, but I don't think we will investigate this before the release.

>
> In any case, even without any special attempt at optimization, the palette
> seems quite usable. I'm sure there is room for optimization of my code, even
> apart from any possible improvement in Emacs' display of faces.

Did you consider using images as I suggested.  That must be quite
trivial, and besides running faster (I assume) you get over problems
with font selection that you have now.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

RE: Q on performance with 10000 faces

[Drew Adams]

    > In any case, even without any special attempt at
    > optimization, the palette seems quite usable. I'm sure
    > there is room for optimization of my code, even
    > apart from any possible improvement in Emacs' display of
    > faces.

    Did you consider using images as I suggested.

Yes. I wanted to do it with faces ;-). It's worth doing with images also.

I still think Emacs should not slow down appreciably just because an
application uses lots of faces - unless you are doing something with those
faces (e.g. updating them) dynamically. I think there might be something to
look into there (after the release, if someone knowledgeable is interested).
But I know nothing about the display code; I'm just being stubbornly
optimistic.

    That must be quite trivial, and besides running faster (I
    assume)

Yes, I'm sure that using an image would be both easy and fast. Sometimes I
like hard and slow ;-).

    you get over problems with font selection that you have now.

Not a problem. The code works fine with any font - you just get a larger or
smaller palette. The problem was that I was assuming that everyone had a
particular font - now the font is an option.

I'm very ignorant wrt fonts. I was mainly pointing out my ignorance and the
poor shape of the Emacs doc about fonts.

Re: Q on performance with 10000 faces

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sun, 21 May 2006 23:26:00 -0700
> 
>     > Does this mean that Emacs cannot reasonably be expected to
>     > display 10000 face text-properties?
> 
>     AFAIK, yes.  Doing what you did (a) disables all possible display
>     optimizations that the redisplay engine has up its sleeve to speed up
>     the common cases, and (b) forces Emacs to traverse the 10000 text
>     properties for each character it is about to display.
> 
> What is it that you think I did that causes that? Perhaps I could do things
> differently. Do you mean just the fact of using 10000 different face text
> properties or something else also?

Are you still interested in the answers, given that Kim suggested a
different approach?

If you are interested, please post the shortest Lisp code that can be
used to reproduce and investigate this situation.

RE: Q on performance with 10000 faces

[Drew Adams]

    >     > Does this mean that Emacs cannot reasonably be expected to
    >     > display 10000 face text-properties?
    >
    >     AFAIK, yes.  Doing what you did (a) disables all possible display
    >     optimizations that the redisplay engine has up its sleeve
    >     to speed up the common cases, and (b) forces Emacs to traverse
    >     the 10000 text properties for each character it is about to
display.
    >
    > What is it that you think I did that causes that? Perhaps I
    > could do things differently. Do you mean just the fact of using
    > 10000 different face text properties or something else also?

    Are you still interested in the answers, given that Kim suggested a
    different approach?

Yes. I'm interested in using faces, if possible, and I'm interested in using
images, if not.

    If you are interested, please post the shortest Lisp code that can be
    used to reproduce and investigate this situation.

OK. I pared it down a bit to send along, and then I tried it in emacs -q.
Result: I do not see the huge slowdown in emacs -q. So, there must be
somthing else that I'm doing in my own environment. Sorry for the false
alarm. We can be thankful for the good news!

Could someone please either describe how to profile or debug the problem
locally, to see where Emacs is spending its cycles, or point me to some doc
that describes that process? I'm hoping there is an easy way to do this with
Emacs Lisp, and without installing extra development tools. (The alternative
is to try removing libraries I use until I no longer see the problem, but
that risks taking a while.)

Thx.

RE: Q on performance with 10000 faces

[Drew Adams]

    Could someone please either describe how to profile or debug
    the problem locally, to see where Emacs is spending its
    cycles, or point me to some doc that describes that process?
    I'm hoping there is an easy way to do this with Emacs Lisp,
    and without installing extra development tools. (The
    alternative is to try removing libraries I use until I no
    longer see the problem, but that risks taking a while.)

I took a quick look at elp.el and benchmark.el, but it seems like you must
provide explicit functions or a package to be profiled. I'm wondering if
there is something that will let me profile the whole ball of string that I
have in my Emacs session.

Re: Q on performance with 10000 faces

[Kevin Rodgers]

Drew Adams wrote:
> I took a quick look at elp.el and benchmark.el, but it seems like you must
> provide explicit functions or a package to be profiled. I'm wondering if
> there is something that will let me profile the whole ball of string that I
> have in my Emacs session.

Looking at the source for elp-instrument-package leads me to
suggest:

   (elp-instrument-list
    (mapcar
     'intern
     (all-completions "." obarray 'elp-profilable-p)))

-- 
Kevin

RE: Q on performance with 10000 faces

[Drew Adams]

    Looking at the source for elp-instrument-package leads me to
    suggest:

       (elp-instrument-list
        (mapcar
         'intern
         (all-completions "." obarray 'elp-profilable-p)))

That would have to be "", not ".", I believe. In any case, "." doesn't seem
to work. elp-results shows nothing and elp-all-instrumented-list remains
nil.

I tried "", but I soon got this, so I bailed out:

  Emergency (alloc): Warning: past 75% of memory limit

Note too this message inside elp-instrument-package, which seems pertinent:
 "Instrumenting all Emacs functions would render Emacs unusable"

I guess one must have some idea of what to profile first. I was hoping to
get that idea from a profiler ;-).

Re: Q on performance with 10000 faces

[Luc Teirlinck]

Drew Adams wrote:

   OK. I pared it down a bit to send along, and then I tried it in emacs -q.
   Result: I do not see the huge slowdown in emacs -q. So, there must be
   somthing else that I'm doing in my own environment.

Anything suspicious on {pre,post}-command-hook?

Sincerely,

Luc.

RE: Q on performance with 10000 faces

[Drew Adams]

       OK. I pared it down a bit to send along, and then I tried it
       in emacs -q.
       Result: I do not see the huge slowdown in emacs -q. So, there must be
       somthing else that I'm doing in my own environment.

    Anything suspicious on {pre,post}-command-hook?

Bingo - hooks! Actually, I don't know if the stuff is suspicious, but it's
at least worth looking into.

I had some stuff on those two hooks, but even after removing it I have the
slowdown (although seemingly less).

I also have a bunch of stuff on minibuffer-setup-hook and
minibuffer-exit-hook. I'll take a look at that stuff by profiling it when I
get a chance.

Thanks, Luc!

When I do track this down a bit more, I suspect it will be a combination of
things - I've already seen an improvement by getting rid of some stuff on
pre/post-command-hook.

If that's the case, and given that without the 10000-face palette I do get
good response time, then I will perhaps post the palette code in hopes that
someone can check it and see why it slows things down so much (i.e. see if
the re-display can be optimized a bit more).

IOW, even if the palette code by itself (emacs -q) is not noticeably slow,
and the hook stuff by itself is not noticeably slow, perhaps the 10000 faces
is just too much when added to the hook code. First I need to see if the
hook code is the culprit when combined with palette.

Thx.

RE: Q on performance with 10000 faces

[Drew Adams]

           OK. I pared it down a bit to send along, and then I tried it
           in emacs -q. Result: I do not see the huge slowdown in emacs
           -q. So, there must be somthing else that I'm doing in my own
           environment.

    When I do track this down a bit more, I suspect it will be a
    combination of things - I've already seen an improvement by getting
    rid of some stuff on pre/post-command-hook.

    If that's the case, and given that without the 10000-face
    palette I do get good response time, then I will perhaps post the
    palette code in hopes that someone can check it and see why it slows
    things down so much (i.e. see if the re-display can be optimized a
    bit more).

I finally found some time to play with elp a bit, after getting rid of
various hooks (pre/post-command, minibuffer-setup/exit). As I suspected, I
didn't notice anything special. I suspect the overall slowness is a
combination of less-noticeable slowdowns due to some of the hooks and the
palette code.

In hopes that someone might be able to see why the palette code slows things
down (or suggest improvements to it), here is the basic code:
_________________________________________________________

;; from library hexrgb.el
(eval-when-compile (require 'cl)) ;; case

(defun approx-equal (x y)
  "Return non-nil if numbers X and Y are approximately equal.
Taken from Elisp Info manual, node \"Comparison of Numbers\", with
\"fuzz-factor\" = 1.0e-8."
  (or (and (= x 0.0) (= y 0.0))
      (< (/ (abs (- x y))
            (max (abs x) (abs y)))
         1.0e-8)))

(defun hexrgb-hsv-to-hex (hue saturation value)
  "Return the hex RBG color string for inputs HUE, SATURATION, VALUE.
The inputs are each in the range 0 to 1."
  (hexrgb-color-values-to-hex
   (mapcar (lambda (x) (floor (* x 65535.0)))
           (hexrgb-hsv-to-rgb hue saturation value))))

(defun hexrgb-color-values-to-hex (values)
  "Convert list of rgb color VALUES to a hex string, #XXXXXXXXXXXX.
Each X in the string is a hexadecimal digit.
Input VALUES is as for the output of `x-color-values'."
  (concat "#"
          (hexrgb-int-to-hex (nth 0 values) 4) ; red
          (hexrgb-int-to-hex (nth 1 values) 4) ; green
          (hexrgb-int-to-hex (nth 2 values) 4))) ; blue

(defun hexrgb-int-to-hex (int &optional nb-digits)
  "Convert integer argument INT to a #XXXXXXXXXXXX format hex string.
Each X in the output string is a hexadecimal digit.
NB-DIGITS is the number of hex digits.  If INT is too large to be
represented with NB-DIGITS, then the result is truncated from the
left.  So, for example, INT=256 and NB-DIGITS=2 returns \"00\", since
the hex equivalent of 256 decimal is 100, which is more than 2 digits."
  (setq nb-digits (or nb-digits 4))
  (substring (format (concat "%0" (int-to-string nb-digits) "X") int)
             (- nb-digits)))

(defun hexrgb-hsv-to-rgb (hue saturation value)
  "Convert HUE, SATURATION, VALUE components to RGB (red, green, blue).
Each input component is 0.0 to 1.0, inclusive.
Returns a list of RGB components of value 0.0 to 1.0, inclusive."
  (let (red green blue int-hue fract pp qq tt ww)
    (if (approx-equal 0.0 saturation)
        (setq red value green value blue value) ; Gray
      (setq hue (* hue 6.0)             ; Sectors: 0 to 5
            int-hue (floor hue)
            fract (- hue int-hue)
            pp (* value (- 1 saturation))
            qq (* value (- 1 (* saturation fract)))
            ww (* value (- 1 (* saturation (- 1 (- hue int-hue))))))
      (case int-hue
        (0 (setq red value green ww blue pp))
        (1 (setq red qq green value blue pp))
        (2 (setq red pp green value blue ww))
        (3 (setq red pp green qq blue value))
        (4 (setq red ww green pp blue value))
        (otherwise (setq red value green pp blue qq))))
    (list red green blue)))

;; from library doremi-color.el
(defun palette ()
  "Display a hue x saturation color palette."
  (interactive)
  (let ((pop-up-frames t))
    (with-output-to-temp-buffer "Palette"
      (let* ((cells (make-string 10000 ?\s- ))
             (hue 0.999999) (sat 1.0) (index 0)
             (color "#000000000000") (hhh 0) (sss 0))
        (while (< index 10000)
          (setq sss 0)
          (while (< sss 100)
            (setq hue 0.999999 hhh 0)
            (while (< hhh 100)
              (put-text-property
               index (1+ index) 'face
               (cons 'background-color
                     (setq color (hexrgb-hsv-to-hex hue sat 1.0)))
               cells)
              (setq hue (* (- hue 0.01) 0.999) hhh (1+ hhh)
                    index (1+ index)))
            (setq sat (* sat 0.97) sss (1+ sss))))
        (set-buffer "Palette")
        (setq hhh 0 index 0)
        (while (< hhh 100)
          (insert (substring cells index (+ index 100)) ?\n)
          (setq hhh (1+ hhh) index (+ index 100)))))
    (select-window (get-buffer-window "Palette" 'visible))
    (modify-frame-parameters
     (selected-frame)
     `((font . "-*-Courier-*-*-*-*-3-*-*-*-*-*-iso8859-1")
       (menu-bar-lines . 0) (tool-bar-lines . 0)
       (left-fringe . 0) (right-fringe . 0)
       (minibuffer) (vertical-scroll-bars)
       (background-color . "Black") (mouse-color . "Black")
       (height . 101) (width . 101)
       (cursor-type . box) (cursor-color . "Black")))))

Re: Q on performance with 10000 faces

[Stefan Monnier]

> Could someone please either describe how to profile or debug the problem
> locally, to see where Emacs is spending its cycles, or point me to some doc
> that describes that process? I'm hoping there is an easy way to do this with
> Emacs Lisp, and without installing extra development tools. (The alternative
> is to try removing libraries I use until I no longer see the problem, but
> that risks taking a while.)

Most likely this can't be seen on an elisp profile.

To get a C profile, add "-pg -DPROFILING=1" to your CFLAGS.

I use the patch below (under GNU/Linux) to provide a `moncontrol' elisp file
that allows me to start/stop C profiling so I can start it once I get into
a "strangely slow" state.


        Stefan


--- orig/src/emacs.c
+++ mod/src/emacs.c
@@ -1769,7 +1777,7 @@
 	 defined on all systems now.  */
       monstartup (safe_bcopy, &etext);
     }
-  else
+  /* else */
     moncontrol (0);
 #endif
 #endif
@@ -1791,6 +1799,14 @@
   return 0;
 }
 \f
+DEFUN ("moncontrol", Fmoncontrol, Smoncontrol, 1, 1, 0,
+				/* doc: toggle profiling.  */)
+     (arg)
+     Lisp_Object arg;
+{
+  return moncontrol (!NILP (arg)) ? Qt : Qnil;
+}
+
 /* Sort the args so we can find the most important ones
    at the beginning of argv.  */
 
@@ -2450,6 +2466,7 @@
 
   defsubr (&Sinvocation_name);
   defsubr (&Sinvocation_directory);
+  defsubr (&Smoncontrol);
 
   DEFVAR_LISP ("command-line-args", &Vcommand_line_args,
 	       doc: /* Args passed by shell to Emacs, as a list of strings.

RE: Q on performance with 10000 faces

[Drew Adams]

    Most likely this can't be seen on an elisp profile.
    To get a C profile, add "-pg -DPROFILING=1" to your CFLAGS.

Sorry, I don't know how to do that. I'm not using C (or building Emacs).    

Re: Q on performance with 10000 faces

[Luc Teirlinck]

Drew Adama wrote:

       Most likely this can't be seen on an elisp profile.
       To get a C profile, add "-pg -DPROFILING=1" to your CFLAGS.

   Sorry, I don't know how to do that. I'm not using C (or building Emacs).    

You could try to see whether anything in the code you load in your
.emacs calls one of the text property search functions in a loop
(10000 times in your case).  For instance

\(next\|previous\)-\(char-\)?property-change.

If this happens from a hook or timer, it definitely could give you trouble,

You could grep the code loaded by your .emacs for these and related
text property search functions.

I can definitely see why things might get slow for you if you would
make parts of your buffer invisible (would probably make line-move
slow) or when you would try to copy a large part of your buffer by
yanking (remove-yank-excluded-properties would be doing a lot of work).

Sincerely,

Luc.

RE: Q on performance with 10000 faces

[Drew Adams]

    You could try to see whether anything in the code you load in your
    .emacs calls one of the text property search functions in a loop
    (10000 times in your case).  For instance

    \(next\|previous\)-\(char-\)?property-change.

    If this happens from a hook or timer, it definitely could give
    you trouble,

    You could grep the code loaded by your .emacs for these and related
    text property search functions.

    I can definitely see why things might get slow for you if you would
    make parts of your buffer invisible (would probably make line-move
    slow) or when you would try to copy a large part of your buffer by
    yanking (remove-yank-excluded-properties would be doing a lot of work).

Thanks, Luc. I'll take a look when I get some time. I don't make any text
invisible, and I don't yank text by program. I will look for property-change
searches.

Re: Q on performance with 10000 faces

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 23 May 2006 07:08:47 -0400
> Cc: emacs-devel@gnu.org
> 
> I use the patch below (under GNU/Linux) to provide a `moncontrol' elisp file
> that allows me to start/stop C profiling so I can start it once I get into
> a "strangely slow" state.

I think this is a good idea, and it will work on MS-Windows as well,
at least in the MinGW build.

Richard, is it okay for Stefan to install the patch below?  I don't
see how can it possibly do any harm.

> --- orig/src/emacs.c
> +++ mod/src/emacs.c
> @@ -1769,7 +1777,7 @@
>  	 defined on all systems now.  */
>        monstartup (safe_bcopy, &etext);
>      }
> -  else
> +  /* else */
>      moncontrol (0);
>  #endif
>  #endif
> @@ -1791,6 +1799,14 @@
>    return 0;
>  }
>  \f
> +DEFUN ("moncontrol", Fmoncontrol, Smoncontrol, 1, 1, 0,
> +				/* doc: toggle profiling.  */)
> +     (arg)
> +     Lisp_Object arg;
> +{
> +  return moncontrol (!NILP (arg)) ? Qt : Qnil;
> +}
> +
>  /* Sort the args so we can find the most important ones
>     at the beginning of argv.  */
>  
> @@ -2450,6 +2466,7 @@
>  
>    defsubr (&Sinvocation_name);
>    defsubr (&Sinvocation_directory);
> +  defsubr (&Smoncontrol);
>  
>    DEFVAR_LISP ("command-line-args", &Vcommand_line_args,
>  	       doc: /* Args passed by shell to Emacs, as a list of strings.

Re: Q on performance with 10000 faces

[Stefan Monnier]

>> I use the patch below (under GNU/Linux) to provide a `moncontrol' elisp file
>> that allows me to start/stop C profiling so I can start it once I get into
>> a "strangely slow" state.

> I think this is a good idea, and it will work on MS-Windows as well,
> at least in the MinGW build.

> Richard, is it okay for Stefan to install the patch below?  I don't
> see how can it possibly do any harm.

I'd be a bit nervous to add a reference to a function `moncontrol' just
like that.  I did it as a quick hack at a time I needed it.  But if it's
conditionalized on PROFILING, maybe it's OK.


        Stefan


>> --- orig/src/emacs.c
>> +++ mod/src/emacs.c
>> @@ -1769,7 +1777,7 @@
>> defined on all systems now.  */
>> monstartup (safe_bcopy, &etext);
>> }
>> -  else
>> +  /* else */
>> moncontrol (0);
>> #endif
>> #endif
>> @@ -1791,6 +1799,14 @@
>> return 0;
>> }
>> \f
>> +DEFUN ("moncontrol", Fmoncontrol, Smoncontrol, 1, 1, 0,
>> +				/* doc: toggle profiling.  */)
>> +     (arg)
>> +     Lisp_Object arg;
>> +{
>> +  return moncontrol (!NILP (arg)) ? Qt : Qnil;
>> +}
>> +
>> /* Sort the args so we can find the most important ones
>> at the beginning of argv.  */
>> 
>> @@ -2450,6 +2466,7 @@
>> 
>> defsubr (&Sinvocation_name);
>> defsubr (&Sinvocation_directory);
>> +  defsubr (&Smoncontrol);
>> 
>> DEFVAR_LISP ("command-line-args", &Vcommand_line_args,
>> doc: /* Args passed by shell to Emacs, as a list of strings.

Re: Q on performance with 10000 faces

[Richard Stallman]

What is this function `moncontrol'?

Re: Q on performance with 10000 faces

[Stefan Monnier]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:
> What is this function `moncontrol'?

The FreeBSD manpage says:

     moncontrol() selectively controls profiling within a program.  When the
     program starts, profiling begins.	To stop the collection of histogram
     ticks and call counts use moncontrol(0); to resume the collection of his-
     togram ticks and call counts use moncontrol(1).  This feature allows the
     cost of particular operations to be measured.  Note that an output file
     will be produced on program exit regardless of the state of moncontrol().

I can't seem to find the corresponding doc on my GNU/Linux system,


        Stefan

Re: Q on performance with 10000 faces

[Richard Stallman]

	 moncontrol() selectively controls profiling within a program.  When the
	 program starts, profiling begins.	To stop the collection of histogram
	 ticks and call counts use moncontrol(0); to resume the collection of his-
	 togram ticks and call counts use moncontrol(1).  This feature allows the
	 cost of particular operations to be measured.  Note that an output file
	 will be produced on program exit regardless of the state of moncontrol().

    I can't seem to find the corresponding doc on my GNU/Linux system,

Does moncontrol exist on GNU/Linux?

The facility sounds useful, but I think we need to make configure
check whether it is available.

Re: Q on performance with 10000 faces

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org
> Date: Wed, 24 May 2006 20:36:28 -0400
> 
> Does moncontrol exist on GNU/Linux?

Yes, definitely.

> The facility sounds useful, but I think we need to make configure
> check whether it is available.

We already call moncontrol in emacs.c on two operating systems.
Stefan's patch doesn't change the situation wrt this, it just makes a
better use of this facility.

Re: Q on performance with 10000 faces

[Richard Stallman]

    We already call moncontrol in emacs.c on two operating systems.
    Stefan's patch doesn't change the situation wrt this, it just makes a
    better use of this facility.

It is ok with me, as long as the #ifdefs test the same conditions
as are tested in emacs.c.

(Except that it shouldn't use __linux as a condition.  It should use
GNU_LINUX.)

Re: Q on performance with 10000 faces

[Richard Stallman]

It could be that this slowness is due to some foolish mistake we could
easily fix.  If not, if it requires a redesign, we surely can't do
that now.  So the crucial question is, where does it spend its time?

You can find that out by running under a debugger, and stopping Emacs
with the debugger about 10 times while the slowness is happening.
When you notice one place appearing repeatedly, you have found it.

RE: Q on performance with 10000 faces

[Drew Adams]

    It could be that this slowness is due to some foolish mistake we could
    easily fix.  If not, if it requires a redesign, we surely can't do
    that now.  So the crucial question is, where does it spend its time?

    You can find that out by running under a debugger, and stopping Emacs
    with the debugger about 10 times while the slowness is happening.
    When you notice one place appearing repeatedly, you have found it.

I'm sorry, but I don't have either the knowledge or the time to investigate
that. Perhaps someone acquainted with the Emacs redisplay code will take up
the challenge.