Split `simple.el'?

Split `simple.el'?

[Drew Adams]

Could we maybe split `simple.el'?

Because there are a few parts of it that contain odd characters,
when I visit it it takes almost 2 minutes (!) for it to finally
get displayed.  I rarely need to access those parts of the file
that are problematic - I typically just want to see the code for
some relatively common function.

The file is also quite large.

How about splitting it up, in particular, moving the parts that
use oddball chars to a different file?

Re: Split `simple.el'?

[Stefan Monnier]

> Because there are a few parts of it that contain odd characters,
> when I visit it it takes almost 2 minutes (!) for it to finally
> get displayed.

Sounds like a bug.
Splitting a file to circumvent such a bug sounds like a bad idea (at
least until we understand enough of the problem to decide that it's too
hard to fix or something).

I have never noticed such a performance problem when opening simple.el,
so it may depend on your particular config.


        Stefan

Re: Split `simple.el'?

[Clément Pit-Claudel]

On 2018-04-03 17:41, Drew Adams wrote:
> Could we maybe split `simple.el'?
> 
> Because there are a few parts of it that contain odd characters,
> when I visit it it takes almost 2 minutes (!) for it to finally
> get displayed.  I rarely need to access those parts of the file
> that are problematic - I typically just want to see the code for
> some relatively common function.
> 
> The file is also quite large.
> 
> How about splitting it up, in particular, moving the parts that
> use oddball chars to a different file?

Shouldn't we rather fix your display issue?
Which version of Emacs are you on?

Re: Split `simple.el'?

[Paul Eggert]

On 04/03/2018 03:25 PM, Stefan Monnier wrote:
> Splitting a file to circumvent such a bug sounds like a bad idea (at
> least until we understand enough of the problem to decide that it's too
> hard to fix or something).

Clearly we should fix his display bug.

That being said, the problem he's having is due to the value of 
password-word-equivalents, a defcustom that doesn't really belong in 
simple.el. A better home would be lisp/international/mule-conf.el, say.

RE: Split `simple.el'?

[Drew Adams]

> > Could we maybe split `simple.el'?
> >
> > Because there are a few parts of it that contain odd characters,
> > when I visit it it takes almost 2 minutes (!) for it to finally
> > get displayed.  I rarely need to access those parts of the file
> > that are problematic - I typically just want to see the code for
> > some relatively common function.
> >
> > The file is also quite large.
> >
> > How about splitting it up, in particular, moving the parts that
> > use oddball chars to a different file?
> 
> Shouldn't we rather fix your display issue?
> Which version of Emacs are you on?

Any version after 24.5 (e.g. 25-27).

In the past I was getting a lot of crashes in the same context,
but it seems that some bugs related to fonts got fixed or something.

The problem is apparently just that I have a lot of installed
fonts etc. and simple.el now contains chars that make the
font-search take a long time with a lot of fonts (or perhaps
with particular fonts, as for `char-displayable-p').

I (and others, AFAIK) see such slowdowns due to font stuff.
We've been through this many times, with Eli generally pointing
out what the problem is (fonts).  AFAIK, there's nothing to be
done here, apart from uninstalling fonts.

But, since you asked, there still is at least bug #30539,
which I would like to see fixed if possible.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30539

I can work around that bug by setting variable
`inhibit-compacting-font-caches' to t.  But even if it is t
it takes 2.5 minutes or so for simple.el to show its text.

Anyway, besides my font-related slowdown, which I can live
with, `simple.el' is pretty large.  I thought perhaps it was
time anyway to split some of it off.  If not, so be it -
just a suggestion.  Does everything there belong there?

RE: Split `simple.el'?

[Drew Adams]

> > Splitting a file to circumvent such a bug sounds like a bad idea (at
> > least until we understand enough of the problem to decide that it's too
> > hard to fix or something).
> 
> Clearly we should fix his display bug.
> 
> That being said, the problem he's having is due to the value of
> password-word-equivalents,

Bingo!  At least it seems you're right.
I tried `C-h v password-word-equivalents', without visiting
simple.el, and I suffered an even worse delay (_many_ minutes).

How did you know?

> a defcustom that doesn't really belong in
> simple.el. A better home would be lisp/international/mule-conf.el, say.

Thank you!

As I said, I go to simple.el quite often, to see some code.
And I doubt that I will ever need to go there to see code
that involves non-ASCII chars.

For me, at least, splitting the file into SIMPLE simple and
FANCY "simple", would be great.  (Yes, I know that one person's
fancy is another person's simple.)

Re: Split `simple.el'?

[Davis Herring]

> Bingo!  At least it seems you're right.
> I tried `C-h v password-word-equivalents', without visiting
> simple.el, and I suffered an even worse delay (_many_ minutes).
> 
> How did you know?

Because that variable's definition contains the only non-ASCII 
characters in the file, surely:

   C-M-s [ ^ C-q C-SPC - ~ ]

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

RE: Split `simple.el'?

[Drew Adams]

> > the problem he's having is due to the value of
> > password-word-equivalents,
> 
> Bingo!  At least it seems you're right.
> I tried `C-h v password-word-equivalents', without visiting
> simple.el, and I suffered an even worse delay (_many_ minutes).
> 
> How did you know?
> 
> > a defcustom that doesn't really belong in
> > simple.el. A better home would be lisp/international/mule-conf.el, say.
> 
> Thank you!
> 
> As I said, I go to simple.el quite often, to see some code.
> And I doubt that I will ever need to go there to see code
> that involves non-ASCII chars.
> 
> For me, at least, splitting the file into SIMPLE simple and
> FANCY "simple", would be great.  (Yes, I know that one person's
> fancy is another person's simple.)

And once I've shown the doc string of `password-word-equivalents',
when I then visit simple.el for the first time it is much quicker
(30 sec instead of over 2 minutes).

RE: Split `simple.el'?

[Drew Adams]

> > Bingo!  At least it seems you're right.
> > I tried `C-h v password-word-equivalents', without visiting
> > simple.el, and I suffered an even worse delay (_many_ minutes).
> >
> > How did you know?
> 
> Because that variable's definition contains the only non-ASCII
> characters in the file, surely:
> 
>    C-M-s [ ^ C-q C-SPC - ~ ]

OK.  But that var is present also in Emacs 24.5, and it has
the same number of elements, and in 24.5 it takes only (!)
about 30 sec to visit simple.el.

(Anyway, just having simple.el or *Help* with that var's doc
string open brings Emacs to a crawl - essentially hangs it up,
in Emacs 25-27.  In Emacs 24.5 those just slow it down a bit.)

Re: Split `simple.el'?

[Stefan Monnier]

> That being said, the problem he's having is due to the value of
> password-word-equivalents

Do we know that for a fact?
Usually non-displayed characters do not affect the time to
open&display it.


        Stefan

Re: Split `simple.el'?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

On 04/03/2018 05:21 PM, Stefan Monnier wrote:
>> That being said, the problem he's having is due to the value of
>> password-word-equivalents
> Do we know that for a fact?

Although it seems obvious to me, Drew can easily verify this by temporarily 
removing the non-ASCII characters from simple.el, restarting Emacs, and visiting 
the file.

While we're on the subject, free_realized_fontsets uses an inefficient algorithm 
and is likely related to the problem. So he might also try commenting out the 
body of free_realized_fontsets and seeing what happens.

Or better yet he could profile Emacs, either at a high level or via 
--enable-profiling.


[-- Attachment #2: Type: text/html, Size: 1163 bytes --]

RE: Split `simple.el'?

[Drew Adams]

>>> That being said, the problem he's having is due to the value of
>>> password-word-equivalents
>>
>> Do we know that for a fact?
>
> Although it seems obvious to me, Drew can easily verify this by
> temporarily removing the non-ASCII characters from simple.el,
> restarting Emacs, and visiting the file.

I copied simple.el to file foo.el, removed the password
defcustom, saved the file, restarted Emacs and visited
the file.  There was no slowdown at all.

Actually, I also saved the defcustom as a separate file,
and I tried to visit that file and got the same kind of
slowdown.

So yes, it seems to be that defcustom that is the
problem, for me.

Re: Split `simple.el'?

[Stefan Monnier]

> So yes, it seems to be that defcustom that is the
> problem, for me.

Could you report this as a bug (where it belongs)?


        Stefan

Re: Split `simple.el'?

[Eli Zaretskii]

> Date: Tue, 3 Apr 2018 14:41:11 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> Could we maybe split `simple.el'?
> 
> Because there are a few parts of it that contain odd characters,
> when I visit it it takes almost 2 minutes (!) for it to finally
> get displayed.  I rarely need to access those parts of the file
> that are problematic - I typically just want to see the code for
> some relatively common function.
> 
> The file is also quite large.
> 
> How about splitting it up, in particular, moving the parts that
> use oddball chars to a different file?

As long as the VCS we use doesn't support well moving stuff from one
file to another, I submit that we should resist the temptation of
splitting files or moving stuff between files as much as we can.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 3 Apr 2018 15:43:26 -0700
> 
> On 04/03/2018 03:25 PM, Stefan Monnier wrote:
> > Splitting a file to circumvent such a bug sounds like a bad idea (at
> > least until we understand enough of the problem to decide that it's too
> > hard to fix or something).
> 
> Clearly we should fix his display bug.

I don't think there's a display bug.  Drew has a very large number of
fonts installed, so searching for a font that supports some character
can take a long time, if none of them support that character.

IOW, it's a system configuration bug: with such a large number of
fonts, one should have at least one font that covers each Unicode
codepoint, and one might need to customize their fontsets to find the
fonts more quickly.

One possibility is to install the GNU Unifont, which AFAIK supports
the entire Unicode range, albeit with glyphs that are in many areas
simply ugly.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Tue, 03 Apr 2018 20:21:28 -0400
> Cc: emacs-devel@gnu.org
> 
> Usually non-displayed characters do not affect the time to
> open&display it.

They do, because Emacs searches all the available fonts to try to find
one that can display such characters.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 3 Apr 2018 19:02:20 -0700
> Cc: emacs-devel@gnu.org
> 
> While we're on the subject, free_realized_fontsets uses an inefficient algorithm and is likely related to the
> problem. So he might also try commenting out the body of free_realized_fontsets and seeing what happens.
> 
> Or better yet he could profile Emacs, either at a high level or via --enable-profiling.

I indeed think we should consider the code in free_realized_fontsets a
possible culprit only if the profile shows it uses some significant
portion of CPU time in this case (or some other real-life case).
Based on where that function is called from, I don't immediately
understand why would it be a hot spot when looking for fonts.

Re: Split `simple.el'?

[Andreas Schwab]

On Apr 04 2018, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
>> Date: Tue, 03 Apr 2018 20:21:28 -0400
>> Cc: emacs-devel@gnu.org
>> 
>> Usually non-displayed characters do not affect the time to
>> open&display it.
>
> They do, because Emacs searches all the available fonts to try to find
> one that can display such characters.

But not until that part of the file is actually rendered.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Stefan Monnier <monnier@IRO.UMontreal.CA>,  eggert@cs.ucla.edu,  emacs-devel@gnu.org
> Date: Wed, 04 Apr 2018 10:19:32 +0200
> 
> >> Usually non-displayed characters do not affect the time to
> >> open&display it.
> >
> > They do, because Emacs searches all the available fonts to try to find
> > one that can display such characters.
> 
> But not until that part of the file is actually rendered.

Yes, of course (though redisplay many times looks also at characters a
short ways above or below what will be visible).  Do we have any
evidence that in this case the slowdown happens also when the
problematic part of the file is nowhere near the visible part of the
buffer?

Re: Split `simple.el'?

[Stefan Monnier]

> I don't think there's a display bug.  Drew has a very large number of
> fonts installed, so searching for a font that supports some character
> can take a long time, if none of them support that character.
>
> IOW, it's a system configuration bug: with such a large number of
> fonts, one should have at least one font that covers each Unicode
> codepoint, and one might need to customize their fontsets to find the
> fonts more quickly.

Maybe we should setup some cache to avoid always recomputing this.


        Stefan

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel@gnu.org
> Date: Wed, 04 Apr 2018 09:14:43 -0400
> 
> > IOW, it's a system configuration bug: with such a large number of
> > fonts, one should have at least one font that covers each Unicode
> > codepoint, and one might need to customize their fontsets to find the
> > fonts more quickly.
> 
> Maybe we should setup some cache to avoid always recomputing this.

I think we already do: the realized fontset is implemented as a
char-table, where we record that a character has no fonts in the
fontset.

The underlying problem in this case happens when such characters need
to be displayed for the first time.

Maybe you mean a persistent cache on disk?  If so, Unix back-ends
already have that, AFAIK.  But Drew is on Windows.  Maybe Emacs on
Windows could also benefit from some system-wide caching, but I don't
know if something like that exists, or how to change Emacs to use it
if it does.  Or maybe the methods we use already employ what caching
is there, and this is the best we can do.

In any case, setting up a cache would mean it needs to be updated each
time a font is installed or removed.

Re: Split `simple.el'?

[Paul Eggert]

On 04/03/2018 11:50 PM, Eli Zaretskii wrote:
> I don't immediately
> understand why would it be a hot spot when looking for fonts.

I don't either. However, when I profiled it on Fedora 27 (a low-level 
profile, with --enable-profiling) it was a hotspot. Although I am not 
observing the same horrible slowdown that Drew is (that is, first 
display of password-word-equivalents help is noticeably slower than for 
typical variables but it is not *horribly* slow), I'm suspicious of 
free_realized_fontsets, which is why I suggested that he comment it out.

We should move password-word-equivalents to an i18n-related file anyway 
(that is, regardless of Drew's problem), as password-word-equivalents 
doesn't belong in simple.el.

Re: Split `simple.el'?

[Eli Zaretskii]

> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 4 Apr 2018 10:20:13 -0700
> 
> On 04/03/2018 11:50 PM, Eli Zaretskii wrote:
> > I don't immediately
> > understand why would it be a hot spot when looking for fonts.
> 
> I don't either. However, when I profiled it on Fedora 27 (a low-level 
> profile, with --enable-profiling) it was a hotspot.

Does setting inhibit-compacting-font-caches non-nil changes anything
in this regard?

> Although I am not 
> observing the same horrible slowdown that Drew is (that is, first 
> display of password-word-equivalents help is noticeably slower than for 
> typical variables but it is not *horribly* slow), I'm suspicious of 
> free_realized_fontsets, which is why I suggested that he comment it out.

If some of the characters there end up being displayed as squares with
hex codepoints, i.e. you have no fonts capable of displaying them,
then this is expected.  I see this on my system as well, and I'm not
surprised.

> We should move password-word-equivalents to an i18n-related file anyway 
> (that is, regardless of Drew's problem), as password-word-equivalents 
> doesn't belong in simple.el.

That's a separate issue, and will just move the problem from one file
to another.

Re: Split `simple.el'?

[Paul Eggert]

On 04/04/2018 12:08 PM, Eli Zaretskii wrote:
> Does setting inhibit-compacting-font-caches non-nil changes anything
> in this regard?

Unfortunately, I can no longer run the same test, as this morning I 
changed my desktop configuration drastically (to fix a graphics driver 
freeze -- not an Emacs bug) and I now get quite-different performance 
results. It's a real hassle to change the configuration (it's easy to 
mess up and be left with a complete reinstall as the only real 
alternative) and I'd rather not go back, sorry.

In my new configuration I get different results. If I run:

emacs -Q -eval "(describe-variable 'password-word-equivalents)"

and immediately type C-x C-c to exit, gprof reports the following hot 
spots. Although it is a bit discouraging that so much garbage collection 
is occurring in such a simple use of Emacs, I suspect this is not Drew's 
problem.

Each sample counts as 0.01 seconds.
   %   cumulative   self              self total
  time   seconds   seconds    calls  ms/call  ms/call name
  40.00      0.06     0.06  1424609     0.00     0.00 mark_object
  13.33      0.08     0.02    49803     0.00     0.00 re_search_2
  13.33      0.10     0.02        6     3.33     3.33 sweep_conses
   6.67      0.11     0.01    12632     0.00     0.00 Fmake_string
   6.67      0.12     0.01    11471     0.00     0.00 allocate_vector
   6.67      0.13     0.01     5821     0.00     0.00 bidi_resolve_explicit
   6.67      0.14     0.01     1074     0.01     0.01 font_unparse_xlfd
   6.67      0.15     0.01 deliver_user_signal

> If some of the characters there end up being displayed as squares with
> hex codepoints, i.e. you have no fonts capable of displaying them,
> then this is expected.

The characters are all displayable. I see no squares.

>> We should move password-word-equivalents to an i18n-related file anyway
>> (that is, regardless of Drew's problem), as password-word-equivalents
>> doesn't belong in simple.el.
> That's a separate issue, and will just move the problem from one file
> to another.

Yes, of course.

Re: Split `simple.el'?

[Eli Zaretskii]

> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 4 Apr 2018 12:29:48 -0700
> 
> In my new configuration I get different results. If I run:
> 
> emacs -Q -eval "(describe-variable 'password-word-equivalents)"
> 
> and immediately type C-x C-c to exit, gprof reports the following hot 
> spots. Although it is a bit discouraging that so much garbage collection 
> is occurring in such a simple use of Emacs, I suspect this is not Drew's 
> problem.

You could perhaps remove GC from this equation by raising
gc-cons-threshold?

Re: Split `simple.el'?

[Juri Linkov]

>> How about splitting it up, in particular, moving the parts that
>> use oddball chars to a different file?
>
> As long as the VCS we use doesn't support well moving stuff from one
> file to another, I submit that we should resist the temptation of
> splitting files or moving stuff between files as much as we can.

What about moving whole files to subdirectories?  I suggest moving
all image related packages to the recently created directory ‘image’.

Re: Split `simple.el'?

[Paul Eggert]

On 04/04/2018 12:37 PM, Eli Zaretskii wrote:
> You could perhaps remove GC from this equation by raising 
> gc-cons-threshold?
It doesn't remove GC entirely, but it does speed things up considerably 
and the hot spots change. Here's a sample profile (the data are noisier 
now of course):

Each sample counts as 0.01 seconds.
   %   cumulative   self              self total
  time   seconds   seconds    calls  ms/call  ms/call name
  25.00      0.02     0.02   673036     0.00     0.00 mark_object
  12.50      0.03     0.01    69505     0.00     0.00 Fassq
  12.50      0.04     0.01    49803     0.00     0.00 re_search_2
  12.50      0.05     0.01    31578     0.00     0.00 Ffuncall
  12.50      0.06     0.01    10570     0.00     0.00 exec_byte_code
  12.50      0.07     0.01      225     0.04     0.04 map_sub_char_table
  12.50      0.08     0.01        3     3.33     3.33 sweep_strings

The command I used was:

emacs -Q -eval "(progn (setq gc-cons-threshold most-positive-fixnum) 
(describe-variable 'password-word-equivalents))"

and typing C-x C-c immediately. I suspect this is not helping us find 
Drew's problem.

Re: Split `simple.el'?

[John Wiegley]

>>>>> "DA" == Drew Adams <drew.adams@oracle.com> writes:

DA> Could we maybe split `simple.el'?
DA> Because there are a few parts of it that contain odd characters,
DA> when I visit it it takes almost 2 minutes (!) for it to finally
DA> get displayed.  I rarely need to access those parts of the file
DA> that are problematic - I typically just want to see the code for
DA> some relatively common function.

On a similar note, I've been meaning to rebind C-h h, because every time I
mistakenly hit it, Emacs completely locks up for about 30 seconds as it
fetches all those various Unicode characters to display. How long does that
take for you?

What would be nice is if all those characters rendered immediately as box
characters, and then filled in font-lock-style as the renderings were found.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Split `simple.el'?

[Clément Pit-Claudel]

On 2018-04-04 18:13, John Wiegley wrote:
> because every time I
> mistakenly hit it, Emacs completely locks up for about 30 seconds as it
> fetches all those various Unicode characters to display. How long does that
> take for you?

Fascinating. It takes less than a second here.

Re: Split `simple.el'?

[Jefferson Carpenter]

On 4/4/2018 10:13 PM, John Wiegley wrote> On a similar note, I've been 
meaning to rebind C-h h, because every time I
> mistakenly hit it, Emacs completely locks up for about 30 seconds as it
> fetches all those various Unicode characters to display. How long does that
> take for you?

When I hit C-h h inside Emacs on Windows 8, it hangs up for 30 seconds.

When I hit C-h h inside Putty, running Emacs on Linux over SSH, it takes 
less than a second.

Just my 2 cents.

Re: Split `simple.el'?

[John Wiegley]

>>>>> "CP" == Clément Pit-Claudel <cpitclaudel@gmail.com> writes:

CP> Fascinating. It takes less than a second here.

Interesting!  I'm using DejaVu Sans Mono, which has a lot of Unicode
characters too. What font are you using?  And I'm on a Mac, using Emacs 26.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

RE: Split `simple.el'?

[Drew Adams]

> > So yes, it seems to be that defcustom that is the
> > problem, for me.
> 
> Could you report this as a bug (where it belongs)?

What's the bug?

For my purposes (visiting simple.el for other stuff), it
would be enough to move that defcustom to some other file.

RE: Split `simple.el'?

[Drew Adams]

> > > Splitting a file to circumvent such a bug sounds like a bad idea (at
> > > least until we understand enough of the problem to decide that it's too
> > > hard to fix or something).
> >
> > Clearly we should fix his display bug.
> 
> I don't think there's a display bug.  Drew has a very large number of
> fonts installed, so searching for a font that supports some character
> can take a long time, if none of them support that character.
> 
> IOW, it's a system configuration bug: with such a large number of
> fonts, one should have at least one font that covers each Unicode
> codepoint, and one might need to customize their fontsets to find the
> fonts more quickly.
> 
> One possibility is to install the GNU Unifont, which AFAIK supports
> the entire Unicode range, albeit with glyphs that are in many areas
> simply ugly.

In the past you've told people with similar problems to
install Symbola.  I did that long ago.  Clearly just
installing it is not enough.

Now you suggest I add font GNU Unifont.  I just did that -
downloaded unifont-10.0.07.ttf, unifont_csur-10.0.07.ttf,
and unifont_csur-10.0.07.ttf from
http://unifoundry.com/unifont.html, and added them to my
Fonts folder.

No change - same 2 minutes to show simple.el.

RE: Split `simple.el'?

[Drew Adams]

> > But not until that part of the file is actually rendered.
> 
> Yes, of course (though redisplay many times looks also at characters a
> short ways above or below what will be visible).  Do we have any
> evidence that in this case the slowdown happens also when the
> problematic part of the file is nowhere near the visible part of the
> buffer?

Does it count as such evidence that visiting simple.el
(which shows only the beginning of the file, once it shows
anything) does nothing but show a blank Emacs window for
two minutes?  It's not trying to display the chars of
that defcustom value - that text is nowhere near the
part of the buffer that (eventually) is shown (at the
beginning of the file).

RE: Split `simple.el'?

[Drew Adams]

> > I don't immediately
> > understand why would it be a hot spot when looking for fonts.
> 
> I don't either. However, when I profiled it on Fedora 27 (a low-level
> profile, with --enable-profiling) it was a hotspot. Although I am not
> observing the same horrible slowdown that Drew is (that is, first
> display of password-word-equivalents help is noticeably slower than for
> typical variables but it is not *horribly* slow), I'm suspicious of
> free_realized_fontsets, which is why I suggested that he comment it out.

I have no way of commenting it out, AFAIK.
It appears nowhere in the Emacs Lisp sources I have,
and I don't build Emacs.

> We should move password-word-equivalents to an i18n-related file anyway
> (that is, regardless of Drew's problem), as password-word-equivalents
> doesn't belong in simple.el.

That would be great.  It might not solve the real
problem, but it would be helpful for at least some
of us.

RE: Split `simple.el'?

[Drew Adams]

> On a similar note, I've been meaning to rebind C-h h, because every time I
> mistakenly hit it, Emacs completely locks up for about 30 seconds as it
> fetches all those various Unicode characters to display. How long does that
> take for you?

emacs -Q: 1 min 15 sec
my setup: 1 min 30 sec

> What would be nice is if all those characters rendered immediately as box
> characters, and then filled in font-lock-style as the renderings were
> found.

Yes.

But for some situations it can make more sense to render all
together (less mystery/surprise), rather than bit by bit.

And in some contexts it makes sense to give users control
over whether to try to show the glyphs at all.

E.g., in my library `apu.el', because of the slowdown from
calling `char-displayable-p', and because showing the glyphs
is not always the most important thing for this library,
users have an option, `apu-match-only-displayable-chars-flag',
whose initial, default value is (< emacs-major-version 25).

,----
| apu-match-only-displayable-chars-flag is a variable defined in `apu.el'.
| Its value is nil
| 
| Documentation:
| Non-nil means filter out chars not displayable (`char-displayable-p').
|
| NOTE:
| 
| Starting with Emacs 25, `char-displayable-p' can be extremely slow if
| the new variable `inhibit-compacting-font-caches' is nil, which it is
| by default, and if you have many, or large, fonts installed.  For this
| reason the default value of `apu-match-only-displayable-chars-flag' is
| nil for Emacs 25 and later.  This seems to be the case if you have MS
| Windows TrueType fonts installed, for instance.
| 
| If you want to be able to exclude non-displayable characters in Emacs
| 25+, then set `apu-match-only-displayable-chars-flag' to non-nil.  If
| you find that this makes APU commands such as `apropos-char' extremely
| slow then set variable `inhibit-compacting-font-caches' to `t'.
| 
| You can customize this variable.
`----

Re: Split `simple.el'?

[Paul Eggert]

On 04/04/2018 04:44 PM, Drew Adams wrote:
>> We should move password-word-equivalents to an i18n-related file anyway
>> (that is, regardless of Drew's problem), as password-word-equivalents
>> doesn't belong in simple.el.
> That would be great.  It might not solve the real
> problem, but it would be helpful for at least some
> of us.

OK, I did that in master. Of course it does not solve the real problem.

Re: Split `simple.el'?

[Stefan Monnier]

>> Could you report this as a bug (where it belongs)?
> What's the bug?

Visiting a file should not suffer from such performance problems.

> For my purposes (visiting simple.el for other stuff), it
> would be enough to move that defcustom to some other file.

That just circumvents the bug.


        Stefan

Re: Split `simple.el'?

[Stefan Monnier]

This discussion belongs in a bug-report, not in emacs-devel.


        Stefan


Drew Adams <drew.adams@oracle.com> writes:

>> > But not until that part of the file is actually rendered.
>> 
>> Yes, of course (though redisplay many times looks also at characters a
>> short ways above or below what will be visible).  Do we have any
>> evidence that in this case the slowdown happens also when the
>> problematic part of the file is nowhere near the visible part of the
>> buffer?
>
> Does it count as such evidence that visiting simple.el
> (which shows only the beginning of the file, once it shows
> anything) does nothing but show a blank Emacs window for
> two minutes?  It's not trying to display the chars of
> that defcustom value - that text is nowhere near the
> part of the buffer that (eventually) is shown (at the
> beginning of the file).

Re: Split `simple.el'?

[Stefan Monnier]

> Maybe you mean a persistent cache on disk?

Yes, that's what I meant.  But in any case, there's no point
discussing/considering it until the actual source of the problem is
precisely characterized, and since his problem shows up long before
those chars are actually displayed, I'm inclined to believe that his
problem might depend on local configuration and the like.  IOW, for all
I know, it might not occur with `emacs -Q`.  So we're far from having
characterized the problem precisely.


        Stefan

RE: Split `simple.el'?

[Drew Adams]

> > Maybe you mean a persistent cache on disk?
> 
> Yes, that's what I meant.  But in any case, there's no point
> discussing/considering it until the actual source of the problem is
> precisely characterized, and since his problem shows up long before
> those chars are actually displayed, I'm inclined to believe that his
> problem might depend on local configuration and the like.  IOW, for all
> I know, it might not occur with `emacs -Q`.  So we're far from having
> characterized the problem precisely.

In reply to John's question about `C-h h' times, I noted:

  > How long does that take for you?

  emacs -Q: 1 min 15 sec
  my setup: 1 min 30 sec

But wrt visiting simple.el, where with my setup I see a
slowdown (~2 minutes) even without going to the part of
the file that has the problematic  defcustom, with
`emacs -Q' I do not see that same problem - there I see
a slowdown only when trying to show the defcustom.

Re: Split `simple.el'?

[Clément Pit-Claudel]

On 2018-04-04 18:49, John Wiegley wrote:
>>>>>> "CP" == Clément Pit-Claudel <cpitclaudel@gmail.com> writes:
> 
> CP> Fascinating. It takes less than a second here.
> 
> Interesting!  I'm using DejaVu Sans Mono, which has a lot of Unicode
> characters too. What font are you using?  And I'm on a Mac, using Emacs 26.

I see some FreeSerif, some FreeSans, Some Ubuntu Mono, some Symbola, some WenQuanYi Micro Hei, and some Noto Sans CJK.

That buffer is very strange.  Selecting parts of it causes the fonts to change in all sorts of surprising ways.

Re: Split `simple.el'?

[Eli Zaretskii]

> Cc: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 4 Apr 2018 12:59:16 -0700
> 
> Each sample counts as 0.01 seconds.
>    %   cumulative   self              self total
>   time   seconds   seconds    calls  ms/call  ms/call name
>   25.00      0.02     0.02   673036     0.00     0.00 mark_object
>   12.50      0.03     0.01    69505     0.00     0.00 Fassq
>   12.50      0.04     0.01    49803     0.00     0.00 re_search_2
>   12.50      0.05     0.01    31578     0.00     0.00 Ffuncall
>   12.50      0.06     0.01    10570     0.00     0.00 exec_byte_code
>   12.50      0.07     0.01      225     0.04     0.04 map_sub_char_table
>   12.50      0.08     0.01        3     3.33     3.33 sweep_strings

I think the relatively high use of Fassq is because we call it when
looking for appropriate fonts.

Btw, isn't it strange that changing your system configuration affects
the profile so much?  Did the change involve any significant
modifications in the fonts that are installed, or in the version or
configuration of fontconfig?

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Drew Adams <drew.adams@oracle.com>,  emacs-devel@gnu.org
> Date: Wed, 04 Apr 2018 22:45:08 +0300
> 
> > As long as the VCS we use doesn't support well moving stuff from one
> > file to another, I submit that we should resist the temptation of
> > splitting files or moving stuff between files as much as we can.
> 
> What about moving whole files to subdirectories?  I suggest moving
> all image related packages to the recently created directory ‘image’.

Moving of whole files is better supported by the VCS, but we still
should have a good reason for that.

What files are those you'd like to move, and why?

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Jefferson Carpenter <jeffersoncarpenter2@gmail.com>
> Date: Wed, 4 Apr 2018 22:45:14 +0000
> 
> On 4/4/2018 10:13 PM, John Wiegley wrote> On a similar note, I've been 
> meaning to rebind C-h h, because every time I
> > mistakenly hit it, Emacs completely locks up for about 30 seconds as it
> > fetches all those various Unicode characters to display. How long does that
> > take for you?
> 
> When I hit C-h h inside Emacs on Windows 8, it hangs up for 30 seconds.

Windows 8 added several fonts that are very large and cause slowdown
of the Emacs redisplay.  You can discover those fonts by moving the
cursor around in the buffer and watching for slowdown; then installing
alternative fonts for the same scripts will make the time much
shorter.  Start by installing Symbola, if you haven't already, as the
default Emacs fontset tells to use it for some Unicode blocks.

> When I hit C-h h inside Putty, running Emacs on Linux over SSH, it takes 
> less than a second.

When you do that, the fonts are served by your local X server, which
runs on Windows, right?  Then I'm guessing that the X server already
has the fonts loaded since the last session of Emacs or other
programs, so the comparison is "unfair".

It takes 5 to 6 seconds on my Windows system to display etc/HELLO,
when I start with "emacs -Q", FWIW.

Re: Split `simple.el'?

[Eli Zaretskii]

> Date: Wed, 4 Apr 2018 16:44:11 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > One possibility is to install the GNU Unifont, which AFAIK supports
> > the entire Unicode range, albeit with glyphs that are in many areas
> > simply ugly.
> 
> In the past you've told people with similar problems to
> install Symbola.  I did that long ago.  Clearly just
> installing it is not enough.
> 
> Now you suggest I add font GNU Unifont.  I just did that -
> downloaded unifont-10.0.07.ttf, unifont_csur-10.0.07.ttf,
> and unifont_csur-10.0.07.ttf from
> http://unifoundry.com/unifont.html, and added them to my
> Fonts folder.
> 
> No change - same 2 minutes to show simple.el.

If you haven't customized your fontset to use Unifont as fallback,
then the result can be expected, and depends to some extent on the
other fonts on your system that are examined before Unifont.

Try this:

  (set-fontset-font "fontset-default" nil "Unifont" nil 'append)

If that doesn't help, that probably means you have some font(s)
installed that cause this slowdown, and need to be deinstalled to fix
that.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Wed, 4 Apr 2018 22:37:25 -0400
> 
> That buffer is very strange.  Selecting parts of it causes the fonts to change in all sorts of surprising ways.

Anything like that is generally a bug: it means we don't supply enough
of the buffer to the shaping engine for it to return the correct
glyphs, which means some problem in the composition rules and/or in
the shaping engine itself (which in your case is libotf/libm17n-flt).

Re: Split `simple.el'?

[Eli Zaretskii]

> Date: Thu, 05 Apr 2018 09:26:21 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eggert@cs.ucla.edu, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > Now you suggest I add font GNU Unifont.  I just did that -
> > downloaded unifont-10.0.07.ttf, unifont_csur-10.0.07.ttf,
> > and unifont_csur-10.0.07.ttf from
> > http://unifoundry.com/unifont.html, and added them to my
> > Fonts folder.
> > 
> > No change - same 2 minutes to show simple.el.

Btw, your later post indicates that font configuration is not
necessarily the main issue here, because "emacs -Q" doesn't show the
slow visiting of simple.el.

Re: Split `simple.el'?

[Eli Zaretskii]

> Date: Wed, 4 Apr 2018 18:17:55 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> But wrt visiting simple.el, where with my setup I see a
> slowdown (~2 minutes) even without going to the part of
> the file that has the problematic  defcustom, with
> `emacs -Q' I do not see that same problem - there I see
> a slowdown only when trying to show the defcustom.

Then please identify the customization(s) which cause(s) the slowdown
when displaying only the beginning of simple.el, and tell what they
are.

That this doesn't happen in "emacs -Q" should have been told at the
very beginning of this discussion, because up to now all the responses
were completely irrelevant to the real problem.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: "John Wiegley" <johnw@gnu.org>
> Date: Wed, 04 Apr 2018 15:49:55 -0700
> Cc: emacs-devel@gnu.org
> 
> >>>>> "CP" == Clément Pit-Claudel <cpitclaudel@gmail.com> writes:
> 
> CP> Fascinating. It takes less than a second here.
> 
> Interesting!  I'm using DejaVu Sans Mono, which has a lot of Unicode
> characters too. What font are you using?  And I'm on a Mac, using Emacs 26.

Please take a look at fontset.el around lines 800 -- 900.  You will
see there several widely-available free fonts that Emacs
pre-configures for certain Unicode ranges.  make sure these fonts are
installed, then try again and see if there's any change in the time it
takes to show etc/HELLO.

Re: Split `simple.el'?

[Nick Helm]

On Thu, 05 Apr 2018 at 14:37:25 +1200, Clément Pit-Claudel wrote:

> That buffer is very strange. Selecting parts of it causes the fonts to
> change in all sorts of surprising ways.

Not sure if it's the same issue, but I see similar weirdness when
scrolling HELLO. It mainly affects Arabic text.

See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28312

Re: Split `simple.el'?

[Jefferson Carpenter]

On 4/5/2018 6:17 AM, Eli Zaretskii wrote:
>> From: Jefferson Carpenter <jeffersoncarpenter2@gmail.com>
>> Date: Wed, 4 Apr 2018 22:45:14 +0000
>>
>> On 4/4/2018 10:13 PM, John Wiegley wrote> On a similar note, I've been
>> meaning to rebind C-h h, because every time I
>>> mistakenly hit it, Emacs completely locks up for about 30 seconds as it
>>> fetches all those various Unicode characters to display. How long does that
>>> take for you?
>>
>> When I hit C-h h inside Emacs on Windows 8, it hangs up for 30 seconds.
> 
> Windows 8 added several fonts that are very large and cause slowdown
> of the Emacs redisplay.  You can discover those fonts by moving the
> cursor around in the buffer and watching for slowdown; then installing
> alternative fonts for the same scripts will make the time much
> shorter.  Start by installing Symbola, if you haven't already, as the
> default Emacs fontset tells to use it for some Unicode blocks.

Interesting, I'll look into that later if I have time.

>> When I hit C-h h inside Putty, running Emacs on Linux over SSH, it takes
>> less than a second.
> 
> When you do that, the fonts are served by your local X server, which
> runs on Windows, right?  Then I'm guessing that the X server already
> has the fonts loaded since the last session of Emacs or other
> programs, so the comparison is "unfair".
> 
> It takes 5 to 6 seconds on my Windows system to display etc/HELLO,
> when I start with "emacs -Q", FWIW.

No X server at all - just running in terminal mode over SSH.

Re: Split `simple.el'?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Jefferson Carpenter <jeffersoncarpenter2@gmail.com>
> Date: Thu, 5 Apr 2018 08:17:14 +0000
> 
> >> When I hit C-h h inside Putty, running Emacs on Linux over SSH, it takes
> >> less than a second.
> > 
> > When you do that, the fonts are served by your local X server, which
> > runs on Windows, right?  Then I'm guessing that the X server already
> > has the fonts loaded since the last session of Emacs or other
> > programs, so the comparison is "unfair".
> > 
> > It takes 5 to 6 seconds on my Windows system to display etc/HELLO,
> > when I start with "emacs -Q", FWIW.
> 
> No X server at all - just running in terminal mode over SSH.

Then your SSH timings are not relevant at all to this discussion,
since text-mode frames don't use any fonts at all.

Re: Split `simple.el'?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> Please take a look at fontset.el around lines 800 -- 900. You will see there
> several widely-available free fonts that Emacs pre-configures for certain
> Unicode ranges. make sure these fonts are installed, then try again and see
> if there's any change in the time it takes to show etc/HELLO.

Thanks, Eli, I'll give that try.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Split `simple.el'?

[Paul Eggert]

On 04/04/2018 10:55 PM, Eli Zaretskii wrote:
> Btw, isn't it strange that changing your system configuration affects
> the profile so much?  Did the change involve any significant
> modifications in the fonts that are installed, or in the version or
> configuration of fontconfig?

I'm not sure, but I suspect so. I changed my graphics driver and my 
windowing server (from Wayland to X). It was quite a big deal.

RE: Split `simple.el'?

[Drew Adams]

> In reply to John's question about `C-h h' times, I noted:
>   > How long does that take for you?
>   emacs -Q: 1 min 15 sec
>   my setup: 1 min 30 sec
> 
> But wrt visiting simple.el, where with my setup I see a
> slowdown (~2 minutes) even without going to the part of
> the file that has the problematic defcustom, with
> `emacs -Q' I do not see that same problem - there I see
> a slowdown only when trying to show the defcustom.

I understand the problem now, I think.  With my setup I
automatically fit frames with buffers to their text.
This happens after the buffer is visited (and thus
displayed).

The frame-fitting code moves point through the buffer,
at eol (`end-of-line'), within a `save-excursion', to
get the longest line length.  That movement presumably
means that fonts are looked for to render the chars in
each line.

Not yet sure what is the best way to work around this,
but that's the explanation why, with my setup but not
with emacs -Q, I see a slowdown before the user even
tries to display that problematic part of the file.

So no particular bug, it seems.  Behind the scene I'm
doing the equivalent of going to that problematic
defcustom, even though I don't see it displayed.

Re: Split `simple.el'?

[Achim Gratz]

John Wiegley writes:
> On a similar note, I've been meaning to rebind C-h h, because every time I
> mistakenly hit it, Emacs completely locks up for about 30 seconds as it
> fetches all those various Unicode characters to display. How long does that
> take for you?

About 10 seconds on a fairly speedy Haswell box; just the first time
around, when the fonts are already loaded and the glyphs cached the same
buffer appears almost instantaneaously.  A fresh second instance of
Emacs will load all fonts again, though.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

Re: Split `simple.el'?

[Eli Zaretskii]

> Date: Thu, 5 Apr 2018 10:25:27 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> I understand the problem now, I think.  With my setup I
> automatically fit frames with buffers to their text.
> This happens after the buffer is visited (and thus
> displayed).

You mean, before it is displayed, right?

> The frame-fitting code moves point through the buffer,
> at eol (`end-of-line'), within a `save-excursion', to
> get the longest line length.  That movement presumably
> means that fonts are looked for to render the chars in
> each line.

That must slow down visiting very large files, even if they don't have
special characters in them.

Why not just use a wide-enough frame?  Your monitor limits the width
you could set anyway.

RE: Split `simple.el'?

[Drew Adams]

> > I understand the problem now, I think.  With my setup I
> > automatically fit frames with buffers to their text.
> > This happens after the buffer is visited (and thus
> > displayed).
> 
> You mean, before it is displayed, right?

No, afterward.  I essentially tack on frame-fitting at the
end of `switch-to-buffer'.  (Among other reasons for this,
I use `image-display-size' to accommodate frames showing
images.)

> > The frame-fitting code moves point through the buffer,
> > at eol (`end-of-line'), within a `save-excursion', to
> > get the longest line length.  That movement presumably
> > means that fonts are looked for to render the chars in
> > each line.
> 
> That must slow down visiting very large files, even if they don't have
> special characters in them.

No, not noticeably.  As I said, if I remove that problematic
defcustom from `simple.el' then there is no delay at all -
the buffer appears instantly.  But sure, with a big enough
buffer it could be noticeable.  I could always add an option
to not fit when `buffer-size' is "too big", but I haven't yet
run into buffers that are too big in this regard.

> Why not just use a wide-enough frame?  Your monitor limits the width
> you could set anyway.

Dunno what you mean.  The idea is to shrink-fit the frame to
its buffer(s).  A one-window-p buffer with short lines has
its frame fit to the width of its widest line, resulting in
a narrow frame.

(There are user options to limit the max frame width and height
and to do other things, but within such constraints the frame
is shrink-fit to its content.)

If I could easily detect the presence of text that would prove
problematic to work with then I could avoid fitting in that
case.  Dunno how to do that.

I could search for a fancy (e.g. non-ASCII) char, but that
wouldn't test whether there is a potential problem, which
depends on the particular fonts installed and the particular
non-ASCII chars present.

Re: Split `simple.el'?

[Clément Pit-Claudel]

On 2018-04-05 02:33, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Wed, 4 Apr 2018 22:37:25 -0400
>>
>> That buffer is very strange.  Selecting parts of it causes the fonts to change in all sorts of surprising ways.
> 
> Anything like that is generally a bug: it means we don't supply enough
> of the buffer to the shaping engine for it to return the correct
> glyphs, which means some problem in the composition rules and/or in
> the shaping engine itself (which in your case is libotf/libm17n-flt).

Thanks. Then I'll report the issue if I can reproduce it reliably.

Re: Split `simple.el'?

[Stefan Monnier]

> The frame-fitting code moves point through the buffer,
> at eol (`end-of-line'), within a `save-excursion', to
> get the longest line length.  That movement presumably
> means that fonts are looked for to render the chars in
> each line.

The movement doesn't care about the chars themselves (it's just looking
for an end-of-line).  So it's probably the computation of the
line-length which triggers it.  You can probably avoid the problem by
using another way to compute the "length" (one which doesn't care about
fonts).


        Stefan

Re: Split `simple.el'?

[Stefan Monnier]

>> On a similar note, I've been meaning to rebind C-h h, because every time I
>> mistakenly hit it, Emacs completely locks up for about 30 seconds as it
>> fetches all those various Unicode characters to display.  How long does that
>> take for you?
> About 10 seconds on a fairly speedy Haswell box;

Wow!

It takes less than 2s on my Thinkpad X201s (8 years old, 2GHz Core i7-620LM)
running Debian stable (with a non-optimized build
using --enable-checking and all).


        Stefan

Re: Split `simple.el'?

[Stefan Monnier]

>> > I understand the problem now, I think.  With my setup I
>> > automatically fit frames with buffers to their text.
>> > This happens after the buffer is visited (and thus
>> > displayed).
>> You mean, before it is displayed, right?
> No, afterward.  I essentially tack on frame-fitting at the
> end of `switch-to-buffer'.

That's before it's displayed (the display only happens when Emacs goes
back to waiting for a key-press).


        Stefan

RE: Split `simple.el'?

[Drew Adams]

> > The frame-fitting code moves point through the buffer,
> > at eol (`end-of-line'), within a `save-excursion', to
> > get the longest line length.  That movement presumably
> > means that fonts are looked for to render the chars in
> > each line.
> 
> The movement doesn't care about the chars themselves (it's just looking
> for an end-of-line).  So it's probably the computation of the
> line-length which triggers it.  You can probably avoid the problem by
> using another way to compute the "length" (one which doesn't care about
> fonts).

So far, I've wanted to take the apparent (i.e., rendered)
char width into account.  I've just been counting columns,
to do that:

(while (not (eobp))
  (end-of-line)
  (setq max-win-width  (max (current-column) max-win-width))
  (when (zerop (forward-line 1))
    (setq max-win-height  (1+ max-win-height))))

(The file is here, if you want a bigger picture:
https://www.emacswiki.org/emacs/download/fit-frame.el)

`current-column' "is calculated by adding together the widths
of all the displayed representations of the character between
the start of the previous line and point (e.g., control
characters will have a width of 2 or 4, tabs will have a
variable width)."

That's TRT, I think: take into account rendered char widths.

But maybe I need an option, to calculate line width without
regard to rendering in some cases.  Even so, other than a
user choosing yes/no in general, I don't see a way for code
to tell whether or where it might makes sense to skip
requiring rendering.

Re: Split `simple.el'?

[Paul Eggert]

On 04/05/2018 01:52 PM, Stefan Monnier wrote:
>>> About 10 seconds on a fairly speedy Haswell box;
> Wow!
>
> It takes less than 2s on my Thinkpad X201s (8 years old, 2GHz Core i7-620LM)
> running Debian stable (with a non-optimized build
> using --enable-checking and all).

Similarly for my circa-2010 desktop with an AMD Phenom II X4 910e (2.6 
GHz) running Fedora 27, using --enable-checking with a non-optimized 
build. Typing 'C-h h' adds about 0.7 seconds to the user+system CPU time 
in a fresh emacs -Q. If I configure Emacs as usual (optimization, no 
checking) 'C-h h' adds only about 0.4 seconds. If it's taking 10 seconds 
in a default build, something is seriously wrong.

Re: Split `simple.el'?

[Stefan Monnier]

> `current-column' "is calculated by adding together the widths
> of all the displayed representations of the character between
> the start of the previous line and point (e.g., control
> characters will have a width of 2 or 4, tabs will have a
> variable width)."

This could be considered a bug: maybe `current-column` shouldn't need to
lookup any font information (after all, it doesn't actually give
pixel-precise information).


        Stefan

Re: Split `simple.el'?

[Andreas Schwab]

On Apr 05 2018, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> This could be considered a bug: maybe `current-column` shouldn't need to
> lookup any font information (after all, it doesn't actually give
> pixel-precise information).

But it needs to know whether the character is glyphless, so that it can
use glyphless-char-display.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Split `simple.el'?

[Andreas Schwab]

On Apr 05 2018, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> It takes less than 2s on my Thinkpad X201s (8 years old, 2GHz Core i7-620LM)
> running Debian stable (with a non-optimized build
> using --enable-checking and all).

How many fonts do you have installed?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Fri, 06 Apr 2018 08:22:51 +0200
> Cc: emacs-devel@gnu.org
> 
> On Apr 05 2018, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> > This could be considered a bug: maybe `current-column` shouldn't need to
> > lookup any font information (after all, it doesn't actually give
> > pixel-precise information).
> 
> But it needs to know whether the character is glyphless, so that it can
> use glyphless-char-display.

Indeed.  More generally, it needs to check character compositions, and
that requires involving the font back-end, which needs to look up a
font for each character from the beginning of the line to point.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Fri, 06 Apr 2018 08:23:47 +0200
> Cc: emacs-devel@gnu.org
> 
> On Apr 05 2018, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> > It takes less than 2s on my Thinkpad X201s (8 years old, 2GHz Core i7-620LM)
> > running Debian stable (with a non-optimized build
> > using --enable-checking and all).
> 
> How many fonts do you have installed?

Right, these comparisons are not really fair unless the number of
installed fonts is comparable.

Re: Split `simple.el'?

[Stefan Monnier]

> Right, these comparisons are not really fair unless the number of
> installed fonts is comparable.

[ I'm on another system now, but it's probably comparable to my X201s.  ]

How should I count them?

    % l -d /usr/share/fonts/*/* | wc

says there are 55 directories holding fonts.  Some of those contain
a single font in a single style, others contain a single font is various
styles, and others contain several fonts in various styles.

    % l /usr/share/fonts/**/*.?tf | wc

says I have 450 TTF or OTF font files installed.  Plus about 100 *.pfb.
And then there are also the *.pcf fonts (i.e. bitmap), of which there
are a bit more than 1000.


        Stefan

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 06 Apr 2018 08:52:13 -0400
> 
> > Right, these comparisons are not really fair unless the number of
> > installed fonts is comparable.
> 
> [ I'm on another system now, but it's probably comparable to my X201s.  ]
> 
> How should I count them?

Like this:

  M-: (length (x-list-fonts "*")) RET

Re: Split `simple.el'?

[Eli Zaretskii]

> Date: Fri, 06 Apr 2018 16:15:51 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > How should I count them?
> 
> Like this:
> 
>   M-: (length (x-list-fonts "*")) RET

Actually, this might be more fair:

  M-: (length (x-list-fonts "-*-*-normal-*-*-*-13-*-*-*-*-*-iso10646-1")) RET

where the "13" part could need a change, if (face-font 'default)
reports a different pixel-width of the default font on your system.

RE: Split `simple.el'?

[Drew Adams]

> > How should I count them?
> 
> Like this:
>   M-: (length (x-list-fonts "*"))

That gives 4525, for me, FWIW.

RE: Split `simple.el'?

[Drew Adams]

> >   M-: (length (x-list-fonts "*")) RET
> 
> Actually, this might be more fair:
> 
>   M-: (length (x-list-fonts "-*-*-normal-*-*-*-13-*-*-*-*-*-iso10646-1"))
> RET
> 
> where the "13" part could need a change, if (face-font 'default)
> reports a different pixel-width of the default font on your system.

That gives 3506, for me (substituting 14 for 13).  But only
when done in a frame where (face-font 'default) = 14.

I zoom frames all the time, by changing the font size
(changing 14 to 5, for a thumbnail frame, or incrementally
incrementing or decrementing the current font size.

IOW, `(face-font 'default)' changes, for me, depending on
the current degree of zooming.

Re: Split `simple.el'?

[Eli Zaretskii]

> Date: Fri, 6 Apr 2018 15:52:43 +0000 (UTC)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: emacs-devel@gnu.org
> 
> >   M-: (length (x-list-fonts "-*-*-normal-*-*-*-13-*-*-*-*-*-iso10646-1"))
> > RET
> > 
> > where the "13" part could need a change, if (face-font 'default)
> > reports a different pixel-width of the default font on your system.
> 
> That gives 3506, for me (substituting 14 for 13).

That's 3 times the number I get here.  Which could well be the reason
why you see so slow redislpay in some cases.

Re: Split `simple.el'?

[Stefan Monnier]

> Actually, this might be more fair:
>   M-: (length (x-list-fonts "-*-*-normal-*-*-*-13-*-*-*-*-*-iso10646-1")) RET
> where the "13" part could need a change, if (face-font 'default)
> reports a different pixel-width of the default font on your system.

OK, I just tried it on my office's 2006 mac-mini (Core 2 Duo T7600)
where the above says it has 437 fonts installed (removing the iso10646
constraint brings it up to 614):

    % time src/emacs -Q --geometry +10+10 -f view-hello-file --eval '(progn (redisplay t) (sleep-for 2) (kill-emacs))'
    src/emacs -Q --geometry +10+10 -f view-hello-file --eval   0.95s user 0.17s system 32% cpu 3.480 total
    % time src/emacs -Q --geometry +10+10 --eval '(progn (redisplay t) (sleep-for 2) (kill-emacs))'
    src/emacs -Q --geometry +10+10 --eval   0.46s user 0.05s system 19% cpu 2.661 total
    %

[ I just used a "sleep-for 2" so I could visually confirm that everything is
  indeed displayed.  ]
So a "normal" start takes ~0.7s and visiting the hello file brings it to
~1.5s hence adding less than a second.

Just for laughs, I tried it on my 2003-vintage Thinkpad X30 (Pentium III
@ 1.2GHz): the time goes from 1.9s to 5.2s, i.e. visiting the hello file
adds a bit more than 3s (and the above x-list-fonts says 462 on this
system).  Definitely not instantaneous but very tolerable.

I expect 500 fonts is not considered large, since I'm not particularly
interested in typography and have installed fairly few fonts above those
that get installed automatically with a "typical" Gnome desktop.

But to justify a delay of more than 10s on a non-ancient system, you'd
need to have at least 10 times as many fonts.  Is that really what's
going on, or is there something else at play?


        Stefan

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Fri, 06 Apr 2018 14:18:10 -0400
> 
> > Actually, this might be more fair:
> >   M-: (length (x-list-fonts "-*-*-normal-*-*-*-13-*-*-*-*-*-iso10646-1")) RET
> > where the "13" part could need a change, if (face-font 'default)
> > reports a different pixel-width of the default font on your system.
> 
> OK, I just tried it on my office's 2006 mac-mini (Core 2 Duo T7600)
> where the above says it has 437 fonts installed (removing the iso10646
> constraint brings it up to 614):

I get 1244 and 1703, respectively, almost 3 times as many.

> So a "normal" start takes ~0.7s and visiting the hello file brings it to
> ~1.5s hence adding less than a second.

I get 0.48s and 10.26s elapsed respectively, with 6s CPU time usage
for visiting HELLO.

> I expect 500 fonts is not considered large, since I'm not particularly
> interested in typography and have installed fairly few fonts above those
> that get installed automatically with a "typical" Gnome desktop.
> 
> But to justify a delay of more than 10s on a non-ancient system, you'd
> need to have at least 10 times as many fonts.  Is that really what's
> going on, or is there something else at play?

I have three time as many fonts, and Emacs on Windows checks 2 font
back-ends before it gives up on characters that don't have any font
supporting them.  The rest is OS differences regarding enumerating
fonts and caching them (or lack thereof) by the font back-end.

As for "ancient", this is a 6 year-old core i7 box.  But I don't think
CPU power is the main cost driver here, because I get the same times
from optimized and non-optimized builds.

Re: Split `simple.el'?

[Stefan Monnier]

>> OK, I just tried it on my office's 2006 mac-mini (Core 2 Duo T7600)
>> where the above says it has 437 fonts installed (removing the iso10646
>> constraint brings it up to 614):
> I get 1244 and 1703, respectively, almost 3 times as many.

And Drew seems to have yet another factor of 3 more.

>> So a "normal" start takes ~0.7s and visiting the hello file brings it to
>> ~1.5s hence adding less than a second.
> I get 0.48s and 10.26s elapsed respectively, with 6s CPU time usage
> for visiting HELLO.

Wow, so almost 10s to display HELLO, compared to less than 1s in my case.
That's a factor 10, i.e. significantly more than a factor 3.

> I have three time as many fonts, and Emacs on Windows checks 2 font
> back-ends before it gives up on characters that don't have any font
> supporting them.

Does this mean the same 1244 fonts (or a significant subset of them) get
considered twice?  If not, then it shouldn't necessarily make much of
a difference, right?

On X11 we also have several backends, and they probably have some
overlap, but I'm not sure which ones are actively in use in my case.

> The rest is OS differences regarding enumerating fonts and caching
> them (or lack thereof) by the font back-end.

That seems to impose a factor somewhere between 1.6 and 3.3 (depending
on the impact of the double font backends) of slowdown in your case
compared to mine.  That doesn't sound outlandish.

> As for "ancient", this is a 6 year-old core i7 box.  But I don't think
> CPU power is the main cost driver here, because I get the same times
> from optimized and non-optimized builds.

I think it is CPU-bound, but most of the CPU time is spent in libraries
rather than in our own code.


        Stefan

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Fri, 06 Apr 2018 15:25:55 -0400
> 
> > I have three time as many fonts, and Emacs on Windows checks 2 font
> > back-ends before it gives up on characters that don't have any font
> > supporting them.
> 
> Does this mean the same 1244 fonts (or a significant subset of them) get
> considered twice?

Not sure.  Font-specs are each checked twice, but I never tried to
trace all the fonts to see whether each back-end is tried with each
font.  I don't think it's a simple N fonts times M back-ends thing, as
one of the back-ends is more powerful and supports modern fonts, while
the other doesn't.

Re: Split `simple.el'?

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Like this:

EZ>   M-: (length (x-list-fonts "*")) RET

I have 3,084 apparently.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Split `simple.el'?

[Stefan Monnier]

> EZ> Like this:
> EZ>   M-: (length (x-list-fonts "*")) RET
> I have 3,084 apparently.

FWIW, my Thinkpad X30 says more than 4000 on this request, but these are
mostly bitmap fonts.


        Stefan

Re: Split `simple.el'?

[Juri Linkov]

>> What about moving whole files to subdirectories?  I suggest moving
>> all image related packages to the recently created directory ‘image’.
>
> Moving of whole files is better supported by the VCS, but we still
> should have a good reason for that.
>
> What files are those you'd like to move,

Everything that matches “image” in the file name:

ezimage.el
iimage.el
image-dired.el
image.el
image-file.el
image-mode.el

and maybe more image-related packages like doc-view.el.

> and why?

To group packages of the same category into one place,
to the created subdirectory “image” (if I correctly understand
the purpose of this new subdirectory).

Re: Split `simple.el'?

[Juri Linkov]

> On a similar note, I've been meaning to rebind C-h h, because every time I
> mistakenly hit it, Emacs completely locks up for about 30 seconds as it
> fetches all those various Unicode characters to display.

I remember some ideas to rebind ‘view-hello-file’ to ‘C-h H’
with the mnemonics of the first letter of the upper-case “HELLO”.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: drew.adams@oracle.com,  emacs-devel@gnu.org
> Date: Sat, 07 Apr 2018 23:29:20 +0300
> 
> ezimage.el
> iimage.el
> image-dired.el
> image.el
> image-file.el
> image-mode.el
> 
> and maybe more image-related packages like doc-view.el.

Do less than a dozen files justify a separate subdirectory?

Re: Split `simple.el'?

[Juri Linkov]

>> ezimage.el
>> iimage.el
>> image-dired.el
>> image.el
>> image-file.el
>> image-mode.el
>>
>> and maybe more image-related packages like doc-view.el.
>
> Do less than a dozen files justify a separate subdirectory?

Better to have a dozen files in a separate subdirectory than
just 2 files that already exist in the subdirectory “images”.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Date: Sun, 08 Apr 2018 22:51:38 +0300
> Cc: emacs-devel@gnu.org
> 
> >> ezimage.el
> >> iimage.el
> >> image-dired.el
> >> image.el
> >> image-file.el
> >> image-mode.el
> >>
> >> and maybe more image-related packages like doc-view.el.
> >
> > Do less than a dozen files justify a separate subdirectory?
> 
> Better to have a dozen files in a separate subdirectory than
> just 2 files that already exist in the subdirectory “images”.

Maybe we should move those 2 into the parent directory, and remove
'images'?

Re: Split `simple.el'?

[Juri Linkov]

>> > Do less than a dozen files justify a separate subdirectory?
>> 
>> Better to have a dozen files in a separate subdirectory than
>> just 2 files that already exist in the subdirectory “images”.
>
> Maybe we should move those 2 into the parent directory, and remove
> 'images'?

I believe more than a dozen files justifies a separate directory,
including the packages for PostScript images:

compface.el
doc-view.el
ezimage.el
gravatar.el
iimage.el
image.el
image-dired.el
image-file.el
image-mode.el
ps-bdf.el
ps-def.el
ps-mule.el
ps-print.el
ps-print-loaddefs.el
ps-samp.el
smiley.el
svg.el
thumbs.el

This list is open-ended for new files, such as for e.g. currently discussed
new mode for continuous image scrolling.

Re: Split `simple.el'?

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: emacs-devel@gnu.org
> Date: Mon, 09 Apr 2018 23:31:12 +0300
> 
> >> > Do less than a dozen files justify a separate subdirectory?
> >> 
> >> Better to have a dozen files in a separate subdirectory than
> >> just 2 files that already exist in the subdirectory “images”.
> >
> > Maybe we should move those 2 into the parent directory, and remove
> > 'images'?
> 
> I believe more than a dozen files justifies a separate directory,
> including the packages for PostScript images:
> 
> compface.el
> doc-view.el
> ezimage.el
> gravatar.el
> iimage.el
> image.el
> image-dired.el
> image-file.el
> image-mode.el
> ps-bdf.el
> ps-def.el
> ps-mule.el
> ps-print.el
> ps-print-loaddefs.el
> ps-samp.el
> smiley.el
> svg.el
> thumbs.el

ps-*.el have nothing to do with images, so that still leaves us with
too few, IMO.

> This list is open-ended for new files, such as for e.g. currently discussed
> new mode for continuous image scrolling.

Such scrolling will definitely work for tall fonts as well, so again
unrelated.