bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Chong Yidong]

> I can reproduce this. The malloc warnings always seem to occur in the 
> SAFE_ALLOCA call in font_sort_entities. The vec passed in to that 
> function contains all the fonts on my system, and len is over 3000, 
> requiring 24kB to be allocated.

FWIW, I can't reproduce this on GNU/Linux.  My default font is DejaVu
Sans Mono, and I tried following the recipe by inserting 0x2203 (which
displays in Kochi Gothic) repeatedly, and moving to the beginning and to
the end of the buffer repeatedly.

font_sort_entities is called a few times initially when I insert 0x2203,
but it is not called repeatedly thereafter.

Could you try to debug why font_sort_entities is called repeatedly in
your case?

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Michael Schierl]

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

- It is easier if you have one of the fonts Code2000 or Arial Unicode
  installed. I tried it with fonts that come with Windows XP (like
  Lucida Sans Unicode) but did not succeed in reproducing the bug.
  It might happen with other fonts as well, most likely ones that have
  lots of glyphs.
- You need to know a character that is in your font but not in the
  default font (0x2203 works for both Arial Unicode and Code2000, 0x2202
  would work for Lucida Sans Unicode, 0xFFFD works for Arial Unicode).
- Start emacs -Q
- Insert a few thousand characters of that kind, and move to the
  beginning and to the end of the buffer repeatedly. At first you
  receive warnings like "Emergency (alloc): Warning: past 95% of memory
  limit", then the memory is exhausted, and if you still go on, you will
  be asked if you want to debug Emacs. This is perfectly reproducable
  for me, both on my main machine and on a freshly installed WinXP
  virtual machine where I only added Emacs and one of the fonts.

You can use code like this to insert the characters:

 like (let ((str (char-to-string (decode-char 'ucs #x2203))))
  (dotimes (i 16)
    (setq str (concat str str)))
  (insert str)
  (goto-char (point-min))
  (goto-char (point-max))
  (goto-char (point-min))
  (goto-char (point-max))
  (goto-char (point-min))
  (goto-char (point-max)))



In GNU Emacs 23.0.93.1 (i386-mingw-nt5.1.2600)
 of 2009-05-02 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: DEU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  tool-bar-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
C-y C-x C-e M-x r e p o r t - e m a <tab> <return>

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
65903

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Michael Schierl wrote:
> - It is easier if you have one of the fonts Code2000 or Arial Unicode
>   installed. I tried it with fonts that come with Windows XP (like
>   Lucida Sans Unicode) but did not succeed in reproducing the bug.
>   It might happen with other fonts as well, most likely ones that have
>   lots of glyphs.
> - You need to know a character that is in your font but not in the
>   default font (0x2203 works for both Arial Unicode and Code2000, 0x2202
>   would work for Lucida Sans Unicode, 0xFFFD works for Arial Unicode).
> - Start emacs -Q
> - Insert a few thousand characters of that kind, and move to the
>   beginning and to the end of the buffer repeatedly. At first you
>   receive warnings like "Emergency (alloc): Warning: past 95% of memory
>   limit", then the memory is exhausted, and if you still go on, you will
>   be asked if you want to debug Emacs. This is perfectly reproducable
>   for me, both on my main machine and on a freshly installed WinXP
>   virtual machine where I only added Emacs and one of the fonts.
>   

I can reproduce this. The malloc warnings always seem to occur in the 
SAFE_ALLOCA call in font_sort_entities. The vec passed in to that 
function contains all the fonts on my system, and len is over 3000, 
requiring 24kB to be allocated. There appear to be two bugs here.

One is that this function is being called all the time when we have 
already found a font for the character we added to the buffer.

The other is that SAFE_FREE does not appear to call xfree to free the 
memory that SAFE_ALLOCA allocated with xmalloc. Maybe SAFE_ALLOCA is not 
used often to allocate more than 16kB (the threshold for using the heap 
instead of the stack) so this has gone unnoticed in the past.

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Jason Rumney wrote:
> The other is that SAFE_FREE does not appear to call xfree to free the 
> memory that SAFE_ALLOCA allocated with xmalloc. Maybe SAFE_ALLOCA is 
> not used often to allocate more than 16kB (the threshold for using the 
> heap instead of the stack) so this has gone unnoticed in the past.

I take that back, xfree is called from safe_alloca_unwind, the memory 
leak lies elsewhere.

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A005A64.5050908@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> Michael Schierl wrote:
> > - It is easier if you have one of the fonts Code2000 or Arial Unicode
> >   installed. I tried it with fonts that come with Windows XP (like
> >   Lucida Sans Unicode) but did not succeed in reproducing the bug.
> >   It might happen with other fonts as well, most likely ones that have
> >   lots of glyphs.
> > - You need to know a character that is in your font but not in the
> >   default font (0x2203 works for both Arial Unicode and Code2000, 0x2202
> >   would work for Lucida Sans Unicode, 0xFFFD works for Arial Unicode).
> > - Start emacs -Q
> > - Insert a few thousand characters of that kind, and move to the
> >   beginning and to the end of the buffer repeatedly. At first you
> >   receive warnings like "Emergency (alloc): Warning: past 95% of memory
> >   limit", then the memory is exhausted, and if you still go on, you will
> >   be asked if you want to debug Emacs. This is perfectly reproducable
> >   for me, both on my main machine and on a freshly installed WinXP
> >   virtual machine where I only added Emacs and one of the fonts.
> >   

> I can reproduce this. The malloc warnings always seem to occur in the 
> SAFE_ALLOCA call in font_sort_entities. The vec passed in to that 
> function contains all the fonts on my system, and len is over 3000, 
> requiring 24kB to be allocated. There appear to be two bugs here.

> One is that this function is being called all the time when we have 
> already found a font for the character we added to the buffer.

Oops, I've just fixed this typo:
  font_sort_entites -> font_sort_entities
Anyway, font_sort_entities is called from
font_find_for_lface (via font_select_entity).

Could you please find why font_find_for_lface is called so
releatedly by setting breakpoint in fontset_find_font (with
condition "c == 0x2203)?

> The other is that SAFE_FREE does not appear to call xfree to free the 
> memory that SAFE_ALLOCA allocated with xmalloc. Maybe SAFE_ALLOCA is not 
> used often to allocate more than 16kB (the threshold for using the heap 
> instead of the stack) so this has gone unnoticed in the past.

SAFE_FREE surely calls xfree via unbind_to because
SAFE_ALLOCA calls record_unwind_protect with
safe_alloca_unwind.

---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:

> Anyway, font_sort_entities is called from
> font_find_for_lface (via font_select_entity).
>
> Could you please find why font_find_for_lface is called so
> releatedly by setting breakpoint in fontset_find_font (with
> condition "c == 0x2203)?
>   

I am still investigating this, but I've found that the problem is 
specific to the uniscribe font backend on Windows.  I couldn't find the 
root cause by debugging, so I am now trying profiling to narrow the search.

With uniscribe font backend enabled, font_load_for_lface gets called 
124k times from 165k calls to fontset_find_font before the memory 
overflow. The glyph that gets displayed is a double-width katakana YO 
from MS Mincho font (it seems the font defines the same glyph to be used 
for mathematical THERE EXISTS).

With only the gdi font backend enabled, font_load_for_lface gets called 
only once from 399k calls to fontset_find_font, but the default Courier 
New font is used to display the characters even though it does not 
contain a glyph (so the default empty box glyph is displayed).

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:
> Could you please find why font_find_for_lface is called so
> releatedly by setting breakpoint in fontset_find_font (with
> condition "c == 0x2203)?
>   

What appears to be happening, is that font_find_for_lface is returning 
many fonts that match the requested spec, but do not contain the 
character required. Because has_char is effectively not implemented in 
the w32 backends, this isn't detected until late, and the negative 
result is either not cached, or is cached according to the original font 
spec which many unusable fonts match.  On subsequent calls, all the 
checking to see which fonts really contain the character required is 
repeated.


On first call to fontset_find_font:

    fontset_get_font_group returns nil (no spec in fontset)

Second call (with default fontset):

    fontset_get_font_group returns a single spec matching registry 
"iso10646-1", script "symbol"
    font_find_for_lface returns the font "Lucida Console", which does 
not contain the desired character.

Third call (with fallback):

    font_find_for_lface returns "Courier New", with registry "iso8859-1" 
then nil

Forth call (with fallback):

    font_find_for_lface returns nil

....
    eventually font_find_for_lface returns "MS Mincho" with registry 
"jisx0208", which does contain the corresponding character (albiet 
double width, looking suspiciously like katakana YO and not encodable by 
jisx0208). There may be an incompatibility in the w32 font handling 
here, because all truetype fonts are effectively unicode fonts, but we 
return them when other registries that the font can manage are 
requested. This is because Emacs requests iso8859-1 and other 8-bit 
registries before requesting iso10646-1, and if we only return bitmap 
fonts for those we will end up with an ugly display by default.

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A3F1B05.7030105@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> What appears to be happening, is that font_find_for_lface is returning 
> many fonts that match the requested spec, but do not contain the 
> character required. Because has_char is effectively not implemented in 
> the w32 backends, this isn't detected until late, and the negative 
> result is either not cached, or is cached according to the original font 
> spec which many unusable fonts match.  On subsequent calls, all the 
> checking to see which fonts really contain the character required is 
> repeated.

That will lead to slow display, but I don't understand why
that leads to memory full problem.  How many fonts do people
have on Windows roughly?  On GNU/Linux, I have about 700
fontconfig fonts and 4500 X fonts.

> On first call to fontset_find_font:

>     fontset_get_font_group returns nil (no spec in fontset)

> Second call (with default fontset):

>     fontset_get_font_group returns a single spec matching registry 
> "iso10646-1", script "symbol"
>     font_find_for_lface returns the font "Lucida Console", which does 
> not contain the desired character.

> Third call (with fallback):

>     font_find_for_lface returns "Courier New", with registry "iso8859-1" 
> then nil

> Forth call (with fallback):

>     font_find_for_lface returns nil

> ....
>     eventually font_find_for_lface returns "MS Mincho" with registry 
> "jisx0208", which does contain the corresponding character (albiet 
> double width, looking suspiciously like katakana YO and not encodable by 
> jisx0208). There may be an incompatibility in the w32 font handling 
> here, because all truetype fonts are effectively unicode fonts, but we 
> return them when other registries that the font can manage are 
> requested. This is because Emacs requests iso8859-1 and other 8-bit 
> registries before requesting iso10646-1, and if we only return bitmap 
> fonts for those we will end up with an ugly display by default.

??? JISX0208 surely contains U+2203 (THERE EXISTS).

By the way, in short, which part of the current code is
wrong?  Do you mean that there's a bug, or that the current
strategy doesn't work for Windows?

---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:
> That will lead to slow display, but I don't understand why
> that leads to memory full problem.  How many fonts do people
> have on Windows roughly?  On GNU/Linux, I have about 700
> fontconfig fonts and 4500 X fonts.
>   

I have 2092 fonts available in Emacs (1692 available through the gdi 
backend, 420 through the uniscribe backend - I'm not sure where the 
other 20 went). The problem occurs even without the gdi backend enabled 
- so with 420 fonts.

> ??? JISX0208 surely contains U+2203 (THERE EXISTS).
>   
My mistake, so that is no problem.

> By the way, in short, which part of the current code is
> wrong?  Do you mean that there's a bug, or that the current
> strategy doesn't work for Windows?
>   

I haven't found what is actually going wrong here yet. But it looks like 
the problem is caused by has_char returning -1, then later encode_char 
returns FONT_INVALID_CODE. The latter does not happen with the gdi 
backend, because it just keeps the code as unicode, rejecting only 
characters that are beyond the first and last characters covered by the 
font.
 

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A3F7058.902@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> > By the way, in short, which part of the current code is
> > wrong?  Do you mean that there's a bug, or that the current
> > strategy doesn't work for Windows?

> I haven't found what is actually going wrong here yet. But it looks like 
> the problem is caused by has_char returning -1, then later encode_char 
> returns FONT_INVALID_CODE. The latter does not happen with the gdi 
> backend, because it just keeps the code as unicode, rejecting only 
> characters that are beyond the first and last characters covered by the 
> font.

I think the same thing happens with x font-backend when
iso10646-1 font is requested.  But, it doesn't cause a
problem on GNU/Linux.

I managed to install mingw and msys, and built Emacs with
them.  Then, I started Emacs, set the defualt font to "Arial
Unicode MS", and insert many #x2203 characters.  But, I
can't reproduce the problem.

By the way, I also installed gdb-6.8-mingw-3.tar.bz2.  But,
with "M-x gdb", pp and pr commands doesn't work.  They print
nothing.  They do work when gdb is invoked from command
line.  Do you know what is wrong?

---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:
> I managed to install mingw and msys, and built Emacs with
> them.  Then, I started Emacs, set the defualt font to "Arial
> Unicode MS", and insert many #x2203 characters.  But, I
> can't reproduce the problem.
>   

Part of the recipe is that you don't set Arial Unicode MS as the default 
font, you let the font fallback find it (in the case of the original 
report, or MS Mincho in my case).

> By the way, I also installed gdb-6.8-mingw-3.tar.bz2.  But,
> with "M-x gdb", pp and pr commands doesn't work.  They print
> nothing.  They do work when gdb is invoked from command
> line.  Do you know what is wrong?
>   

I've often had problems in the past using debuggers under Emacs on 
Windows (under Emacs 21 and early 22 development versions though, so I 
haven't tried the new gdb-ui extensively), so I always use the command line.

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A3F81AC.1070404@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> Part of the recipe is that you don't set Arial Unicode MS as the default 
> font, you let the font fallback find it (in the case of the original 
> report, or MS Mincho in my case).

Although I still can't reproduce the problem (except for the
very slowness redisplay), I noticed some inefficiency in
fontset_font.  So, I've installed an improvement in the
TRUNK.  As a result, in the case of inserting many #x2203,
the redisplay got faster.

Could you please test it?

---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:
> Although I still can't reproduce the problem (except for the
> very slowness redisplay), I noticed some inefficiency in
> fontset_font.  So, I've installed an improvement in the
> TRUNK.  As a result, in the case of inserting many #x2203,
> the redisplay got faster.
>   

The memory full problem is still there. I am surprised you don't see it 
if you are seeing the slowness, since if things were working correctly, 
only the first character displayed should be slow while the fonts are 
searched, subsequent insertion of the same character should reuse the 
cached font.

Your change seems to have reduced the first time display of etc/HELLO 
from 12 seconds for the uniscribe backend to 10 seconds, vs 2 seconds 
for the gdi backend and Emacs 22 (though only uniscribe can display the 
complex scripts correctly). Subsequent redisplays are near 
instantaneous, so it still seems to be searching for fonts rather than 
displaying them that takes the time.

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A4201EF.7000901@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> Kenichi Handa wrote:
> > Although I still can't reproduce the problem (except for the
> > very slowness redisplay), I noticed some inefficiency in
> > fontset_font.  So, I've installed an improvement in the
> > TRUNK.  As a result, in the case of inserting many #x2203,
> > the redisplay got faster.

> The memory full problem is still there. I am surprised you don't see it 
> if you are seeing the slowness, since if things were working correctly, 
> only the first character displayed should be slow while the fonts are 
> searched, subsequent insertion of the same character should reuse the 
> cached font.

Even if Emacs once finds a font for a specific character C,
it doesn't directly cache the font for C.  Instead, the font
is recorded in a font-group that is shared with the other
characters (in a fontset).  So, each time we display C, we
must find a font that can display C within the cached fonts.
Of course, this process is, I think, far faster than
font-listing and opening, it is not instant especially when
the target font is at the end of the font-group.

The recipe of the original report inserts a very long line.
In that case, when Emacs displays a character at the end of
line, Emacs tries to check fonts of all characters in the
line.  This takes considerable amount of time (in my case 4
seconds before my patch and 2.5 seconds after the patch)
even if the font is cached.

> Your change seems to have reduced the first time display of etc/HELLO 

Actually it should reduce the second time display of a
specific group of characters.  But, as HELLO file contains
many different characters belonging to the same group,
display of some characters gets faster if a character of the
same group is already shown.

> from 12 seconds for the uniscribe backend to 10 seconds, vs 2 seconds 
> for the gdi backend and Emacs 22 (though only uniscribe can display the 
> complex scripts correctly). Subsequent redisplays are near 
> instantaneous, so it still seems to be searching for fonts rather than 
> displaying them that takes the time.

Anyway, it is not good to use HELLO file for comparing
because Emacs 23 knows more kinds of fonts can display a
specific character and thus tries more fonts.  So, it takes
more time to conclude that you system doesn't have a font
for that specific character.

---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:
> In article <4A3F81AC.1070404@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:
>
>   
>> Part of the recipe is that you don't set Arial Unicode MS as the default 
>> font, you let the font fallback find it (in the case of the original 
>> report, or MS Mincho in my case).
>>     
>
> Although I still can't reproduce the problem

I wonder if the default font contains the character U+2203 in your case, 
since you are using a Japanese locale. Can you try reproducing it using 
the other characters suggested in the original message: U+2202 and U+FFFD?

In my case I only see the problem with U+2203 with emacs -Q, but I did 
originally reproduce the problem with one of those other characters 
after changing the default font to something else.

(let ((str (char-to-string (decode-char 'ucs #xFFFD))))
  (dotimes (i 16)
    (setq str (concat str str)))
  (insert str)
  (goto-char (point-min))
  (goto-char (point-max))
  (goto-char (point-min))
  (goto-char (point-max))
  (goto-char (point-min))
  (goto-char (point-max)))

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A420357.8050706@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

>>> Part of the recipe is that you don't set Arial Unicode MS as the default 
>>> font, you let the font fallback find it (in the case of the original 
>>> report, or MS Mincho in my case).
> >
> > Although I still can't reproduce the problem

> I wonder if the default font contains the character U+2203 in your case, 
> since you are using a Japanese locale.

No.  My default font is:

uniscribe:-outline-Courier new-normal-normal-normal-mono-13-*-*-*-c-*-iso88591

and #x2203 is displayed by:

uniscribe:-outline-ＭＳ　ゴシック-normal-normal-normal-mono-13-*-*-*-c-*-jisx0208*-*
 
This font is found in the fallback group of the default
fontset (not in the frame's fontset).

By font-show-log, I confirmed that fontset_font() checked
these to find that font:

(1) the frame's fontset
(2) the default fontset
(3) the fallbacks of the frame's fontset
(4) the fallbacks of the default fontset.

So, isn't the situation the same as yours?

> Can you try reproducing it using 
> the other characters suggested in the original message: U+2202 and U+FFFD?

A font for U+2202 is found at (2) above, and a font for
U+FFFD is not found.  In both case, Emacs didn't cause
memory full.

By the way, my Emacs on Windows is as this:

GNU Emacs 23.1.50.1 (i386-mingw-nt5.1.2600) of 2009-06-24 on IBM-FSF27A21743
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'.


---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

I've just installed another patch for fontset.c in trunk.
With that, even in English language environment,
(insert-char #x2203 10000) is not that slow.

But, it may just hide the original problem of memory full.
I'll keep on finding why that happens.

---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:
> I've just installed another patch for fontset.c in trunk.
> With that, even in English language environment,
> (insert-char #x2203 10000) is not that slow.
>
> But, it may just hide the original problem of memory full.
> I'll keep on finding why that happens.
>   

It seems to avoid the problem in this case at least. I can now insert 
over a million of the characters from the original recipe without a 
problem (and within 5 seconds) by changing the 16 in the original recipe 
to 20.

I think this change should go in the release branch, then we can close 
the bug (or lower its priority if you would rather continue to investigate).

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A4379F7.4000100@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> It seems to avoid the problem in this case at least. I can now insert 
> over a million of the characters from the original recipe without a 
> problem (and within 5 seconds) by changing the 16 in the original recipe 
> to 20.

> I think this change should go in the release branch,

I think we need the previous change too for 23.1.  Yidon and
Stefan, may I install the attached change in EMACS_23_1_RC?
It seems that this is an effective workaround of the serious
problem on Windows.  Theoretically, the same problem happens
on GNU/Linux too.

> then we can close the bug (or lower its priority if you
> would rather continue to investigate).

Please just lower the priority.  I'll continue to
investigate.  I now have one hypothesis.

On Windows, both uniscribe and gdi font backends returns
many fonts.  When Emacs try to find a suitable font from
them, they are concatinated to a single vector and sorted.
This vector is created and abondoned many times.

If no GC happens while displaying all characters of that
long line, those temporarily created Lisp vectors accumulate
in memory without reused, which may lead to memory full.

What do you think?

---
Kenichi Handa
handa@m17n.org


2009-06-25  Kenichi Handa  <handa@m17n.org>

	* fontset.c (fontset_find_font): When a usable rfont_def is found
	in a fallback font-group, make it the first element of the group.
	(fontset_get_font_group): Return 0 if no font-group is set for C.
	(fontset_font): Record the availability of a font for C both in
	the realized fontsets of the current one and the default one.

Index: fontset.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/fontset.c,v
retrieving revision 1.173
diff -u -r1.173 fontset.c
--- fontset.c	8 Jun 2009 04:33:40 -0000	1.173
+++ fontset.c	26 Jun 2009 01:00:46 -0000
@@ -459,6 +459,12 @@
   XSETCAR (font_group, make_number (charset_ordered_list_tick));
 }
 
+/* Return a font-group (actually a cons (-1 . FONT-GROUP-VECTOR)) for
+   character C in FONTSET.  If C is -1, return a fallback font-group.
+   If C is not -1, the value may be Qt (FONTSET doesn't have a font
+   for C even in the fallback group, or 0 (a font for C may be found
+   only in the fallback group).  */
+
 static Lisp_Object
 fontset_get_font_group (Lisp_Object fontset, int c)
 {
@@ -480,9 +486,10 @@
     font_group = FONTSET_FALLBACK (base_fontset);
   if (NILP (font_group))
     {
+      font_group = make_number (0);
       if (c >= 0)
-	char_table_set_range (fontset, from, to, make_number (0));
-      return Qnil;
+	char_table_set_range (fontset, from, to, font_group);
+      return font_group;
     }
   font_group = Fcopy_sequence (font_group);
   for (i = 0; i < ASIZE (font_group); i++)
@@ -524,9 +531,10 @@
      int id, fallback;
 {
   Lisp_Object vec, font_group;
-  int i, charset_matched = -1;
+  int i, charset_matched = -1, found_index;
   FRAME_PTR f = (FRAMEP (FONTSET_FRAME (fontset)))
     ? XFRAME (selected_frame) : XFRAME (FONTSET_FRAME (fontset));
+  Lisp_Object rfont_def;
 
   font_group = fontset_get_font_group (fontset, fallback ? -1 : c);
   if (! CONSP (font_group))
@@ -546,9 +554,9 @@
 	   first.  */
 	for (i = 0; i < ASIZE (vec); i++)
 	  {
-	    Lisp_Object rfont_def = AREF (vec, i);
 	    Lisp_Object repertory;
 
+	    rfont_def = AREF (vec, i);
 	    if (NILP (rfont_def))
 	      break;
 	    repertory = FONT_DEF_REPERTORY (RFONT_DEF_FONT_DEF (rfont_def));
@@ -564,18 +572,22 @@
   /* Find the first available font in the vector of RFONT-DEF.  */
   for (i = 0; i < ASIZE (vec); i++)
     {
-      Lisp_Object rfont_def, font_def;
+      Lisp_Object font_def;
       Lisp_Object font_entity, font_object;
 
       if (i == 0 && charset_matched >= 0)
 	{
 	  /* Try the element matching with the charset ID at first.  */
 	  rfont_def = AREF (vec, charset_matched);
+	  found_index = charset_matched;
 	  charset_matched = -1;
 	  i--;
 	}
       else if (i != charset_matched)
-	rfont_def = AREF (vec, i);
+	{
+	  rfont_def = AREF (vec, i);
+	  found_index = i;
+	}
       else
 	continue;
 
@@ -623,7 +635,7 @@
 	}
 
       if (font_has_char (f, font_object, c))
-	return rfont_def;
+	goto found;
 
       /* Find a font already opened, maching with the current spec,
 	 and supporting C. */
@@ -637,7 +649,7 @@
 	    break;
 	  font_object = RFONT_DEF_OBJECT (AREF (vec, i));
 	  if (! NILP (font_object) && font_has_char (f, font_object, c))
-	    return rfont_def;
+	    goto found;
 	}
 
       /* Find a font-entity with the current spec and supporting C.  */
@@ -654,6 +666,7 @@
 					     Qnil);
 	  if (NILP (font_object))
 	    continue;
+	  found_index = i;
 	  RFONT_DEF_NEW (rfont_def, font_def);
 	  RFONT_DEF_SET_OBJECT (rfont_def, font_object);
 	  RFONT_DEF_SET_SCORE (rfont_def, RFONT_DEF_SCORE (rfont_def));
@@ -661,10 +674,12 @@
 	  for (j = 0; j < i; j++)
 	    ASET (new_vec, j, AREF (vec, j));
 	  ASET (new_vec, j, rfont_def);
+	  found_index = j;
 	  for (j++; j < ASIZE (new_vec); j++)
 	    ASET (new_vec, j, AREF (vec, j - 1));
 	  XSETCDR (font_group, new_vec);
-	  return rfont_def;
+	  vec = new_vec;
+	  goto found;
 	}
 
       /* No font of the current spec for C.  Try the next spec.  */
@@ -673,6 +688,19 @@
 
   FONTSET_SET (fontset, make_number (c), make_number (0));
   return Qnil;
+
+ found:
+  if (fallback && found_index > 0)
+    {
+      /* The order of fonts in the fallback font-group is not that
+	 important, and it is better to move the found font to the
+	 first of the group so that the next try will find it
+	 quickly. */
+      for (i = found_index; i > 0; i--)
+	ASET (vec, i, AREF (vec, i - 1));
+      ASET (vec, 0, rfont_def);
+    }
+  return rfont_def;
 }
 
 
@@ -683,15 +711,15 @@
      struct face *face;
      int id;
 {
-  Lisp_Object rfont_def;
+  Lisp_Object rfont_def, default_rfont_def;
   Lisp_Object base_fontset;
 
   /* Try a font-group of FONTSET. */
   rfont_def = fontset_find_font (fontset, c, face, id, 0);
   if (VECTORP (rfont_def))
     return rfont_def;
-  if (EQ (rfont_def, Qt))
-    goto no_font;
+  if (NILP (rfont_def))
+    FONTSET_SET (fontset, make_number (c), make_number (0));
 
   /* Try a font-group of the default fontset. */
   base_fontset = FONTSET_BASE (fontset);
@@ -700,32 +728,36 @@
       if (NILP (FONTSET_DEFAULT (fontset)))
 	FONTSET_DEFAULT (fontset)
 	  = make_fontset (FONTSET_FRAME (fontset), Qnil, Vdefault_fontset);
-      rfont_def = fontset_find_font (FONTSET_DEFAULT (fontset), c, face, id, 0);
-      if (VECTORP (rfont_def))
-	return rfont_def;
-      if (EQ (rfont_def, Qt))
-	goto no_font;
+      default_rfont_def
+	= fontset_find_font (FONTSET_DEFAULT (fontset), c, face, id, 0);
+      if (VECTORP (default_rfont_def))
+	return default_rfont_def;
+      if (NILP (default_rfont_def))
+	FONTSET_SET (FONTSET_DEFAULT (fontset), make_number (c),
+		     make_number (0));
     }
 
   /* Try a fallback font-group of FONTSET. */
-  rfont_def = fontset_find_font (fontset, c, face, id, 1);
-  if (VECTORP (rfont_def))
-    return rfont_def;
-  if (EQ (rfont_def, Qt))
-    goto no_font;
+  if (! EQ (rfont_def, Qt))
+    {
+      rfont_def = fontset_find_font (fontset, c, face, id, 1);
+      if (VECTORP (rfont_def))
+	return rfont_def;
+      /* Remember that FONTSET has no font for C.  */
+      FONTSET_SET (fontset, make_number (c), Qt);
+    }
 
-  /* Try a fallback font-group of the default fontset . */
-  if (! EQ (base_fontset, Vdefault_fontset))
+  /* Try a fallback font-group of the default fontset. */
+  if (! EQ (base_fontset, Vdefault_fontset)
+      && ! EQ (default_rfont_def, Qt))
     {
       rfont_def = fontset_find_font (FONTSET_DEFAULT (fontset), c, face, id, 1);
       if (VECTORP (rfont_def))
 	return rfont_def;
+      /* Remember that the default fontset has no font for C.  */
+      FONTSET_SET (FONTSET_DEFAULT (fontset), make_number (c), Qt);
     }
 
- no_font:
-  /* Remember that we have no font for C.  */
-  FONTSET_SET (fontset, make_number (c), Qt);
-
   return Qnil;
 }
 

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Jason Rumney]

Kenichi Handa wrote:
> If no GC happens while displaying all characters of that
> long line, those temporarily created Lisp vectors accumulate
> in memory without reused, which may lead to memory full.
>   

While I was profiling to try to identify a possible cause, there were 6 
GCs in the 25 seconds that Emacs was running (taking up 22% of the 
time). So GC does seem to happen, though maybe it was after the first 
warning was thrown.

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <4A44629C.9040209@gnu.org>, Jason Rumney <jasonr@gnu.org> writes:

> Kenichi Handa wrote:
> > If no GC happens while displaying all characters of that
> > long line, those temporarily created Lisp vectors accumulate
> > in memory without reused, which may lead to memory full.
> >   

> While I was profiling to try to identify a possible cause, there were 6 
> GCs in the 25 seconds that Emacs was running (taking up 22% of the 
> time). So GC does seem to happen, though maybe it was after the first 
> warning was thrown.

I added a static variable `vconcat_size' that accumulates
the size of vectors created by Fvconcat at the end of
font_list_entities.  After (insert-char #x2203 10000), GC is
called once while update_menu_bar is called.  After that,
until memory_full is called (in the Fvconcat above), no GC
is called, and vconcat_size increases by about 100M which
means at least 400M bytes is consumued by dead vectors.

So, I think the above hypothesis is correct.

---
Kenichi Handa
handa@m17n.org

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

Attached is a new patch I propose for 23.1 to solve the bug.
It is not the best one, but is simpler and safer.

I'll summarize the problem here.  Yidong and Stefan, please
consider installing it.

Emacs searches for a font for C in this order:

(1) search a font-group for C in the current fontset.
(2) search a font-group for C in the default fontset.
(3) search a fallback font-group of the current fontset.
(4) search a fallback font-group of the default fontset.

The problem occurs when there's a very long line that
contains characters whose fonts are found only in the second
or later font in the group at step (4).  Actually there are
two problems; memory full and extreme slowness.  They happen
typically on Windows because both uniscribe and gdi
font-backends returns many fonts, but theoretically it
happens also on GNU/Linux.

First, the reason of memory full is this.

When a set of fonts are found in multiple font-backends,
Emacs concatenates them into a single vector, sort elements
of the vector, then return the best matching font.  This is
done for each character in a line.  But, as Emacs doesn't
perform GC while displaying that long line, the created (and
then abandoned) vectors consume memory without being
re-used.  This results in memory full error.

The changes for font.c in the patch is to solve it.  The
strategy is to cache vectors of specific sizes in a weak
hash table for reuse until the next GC time.

Next, the reason of slowness is this.

When a font is found in the second or the following spec in
the fallback font-group, Emacs sorts fonts matching with the
first spec in the font-group, then check if fonts support C
one by one, even if it results in no-font-for-C.  This is
because Emacs doesn't remember which font-spec in the
font-group can return a suitable font for C.  We can't
remember such info for each character because it requires
lots of memory.  So, anyway this sorting and checking is
done for each character in that long line, and it's very
slow.

The changes for fontset.c in the patch is to solve it by
re-ordering the entries in the font-group so that the
lastly used entry comes first.  We do this re-ordering only
for fallback font-group assuming that the order is not that
important in a fallback font-group.  And, for a font-group
corresponding to a character, usually the font-specs are more
specific and the number of matching fonts is small, and thus
the above soring and checking is not that heavy.

By the way, I'm going to install a different change (bigger
but better) for 23.2.

---
Kenichi Handa
handa@m17n.org

2009-07-02  Kenichi Handa  <handa@m17n.org>

	* font.c (font_entity_vector_cache): New variable.
	(syms_of_font): Initialize it.
	(font_concat_entities): New function.
	(font_list_entities): Use font_concat_entities instead of
	Fvconcat.

	* fontset.c (fontset_find_font): Re-order a fallback font-group.
	(fontset_font): Treat return value t of fontset_find_font as a
	sign of no-font in that fontset, not as a sign of no-font
	anywhere.

Index: font.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/font.c,v
retrieving revision 1.133
diff -u -r1.133 font.c
--- font.c	10 Jun 2009 01:26:15 -0000	1.133
+++ font.c	2 Jul 2009 11:13:16 -0000
@@ -2763,6 +2763,33 @@
   return Fnreverse (val);
 }
 
+static Lisp_Object font_entity_vector_cache;
+
+/* Concatenate lists of font-entities in VEC_ENTITY_LIST or length LEN.  */
+
+static Lisp_Object
+font_concat_entities (Lisp_Object *vec_entity_list, int len)
+{
+  int i, j, num;
+  Lisp_Object vec, tail;
+  
+  for (i = 0, num = 0; i < len; i++)
+    num += XINT (Flength (vec_entity_list[i]));
+  vec = Fgethash (make_number (num), font_entity_vector_cache, Qnil);
+  if (NILP (vec))
+    {
+      vec = Fvconcat (len, vec_entity_list);
+      Fputhash (make_number (num), vec, font_entity_vector_cache);
+    }
+  else
+    {
+      for (i = 0, j = 0; i < len; i++)
+	for (tail = vec_entity_list[i]; CONSP (tail); tail = XCDR (tail), j++)
+	  ASET (vec, j, XCAR (tail));
+    }
+  return vec;
+}
+
 
 /* Return a vector of font-entities matching with SPEC on FRAME.  */
 
@@ -2831,7 +2858,7 @@
 	  vec[i++] = val;
       }
 
-  val = (i > 0 ? Fvconcat (i, vec) : null_vector);
+  val = (i > 0 ? font_concat_entities (vec, i) : null_vector);
   font_add_log ("list", spec, val);
   return (val);
 }
@@ -5178,6 +5205,14 @@
   staticpro (&Vfont_log_deferred);
   Vfont_log_deferred = Fmake_vector (make_number (3), Qnil);
 
+  staticpro (&font_entity_vector_cache);
+  { /* Here we rely on the fact that syms_of_font is called fairly
+       late, when QCweakness is known to be set.  */
+    Lisp_Object args[2];
+    args[0] = QCweakness;
+    args[1] = Qt;
+    font_entity_vector_cache = Fmake_hash_table (2, args);
+  }
 #if 0
 #ifdef HAVE_LIBOTF
   staticpro (&otf_list);
Index: fontset.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/fontset.c,v
retrieving revision 1.173
diff -u -r1.173 fontset.c
--- fontset.c	8 Jun 2009 04:33:40 -0000	1.173
+++ fontset.c	2 Jul 2009 11:06:14 -0000
@@ -525,6 +525,8 @@
 {
   Lisp_Object vec, font_group;
   int i, charset_matched = -1;
+  Lisp_Object rfont_def;
+  int found_index;
   FRAME_PTR f = (FRAMEP (FONTSET_FRAME (fontset)))
     ? XFRAME (selected_frame) : XFRAME (FONTSET_FRAME (fontset));
 
@@ -564,21 +566,22 @@
   /* Find the first available font in the vector of RFONT-DEF.  */
   for (i = 0; i < ASIZE (vec); i++)
     {
-      Lisp_Object rfont_def, font_def;
+      Lisp_Object font_def;
       Lisp_Object font_entity, font_object;
 
       if (i == 0 && charset_matched >= 0)
 	{
 	  /* Try the element matching with the charset ID at first.  */
-	  rfont_def = AREF (vec, charset_matched);
+	  found_index = charset_matched;
 	  charset_matched = -1;
 	  i--;
 	}
       else if (i != charset_matched)
-	rfont_def = AREF (vec, i);
+	found_index = i;
       else
 	continue;
 
+      rfont_def = AREF (vec, found_index);
       if (NILP (rfont_def))
 	/* This is a sign of not to try the other fonts.  */
 	return Qt;
@@ -623,7 +626,7 @@
 	}
 
       if (font_has_char (f, font_object, c))
-	return rfont_def;
+	goto found;
 
       /* Find a font already opened, maching with the current spec,
 	 and supporting C. */
@@ -637,7 +640,7 @@
 	    break;
 	  font_object = RFONT_DEF_OBJECT (AREF (vec, i));
 	  if (! NILP (font_object) && font_has_char (f, font_object, c))
-	    return rfont_def;
+	    goto found;
 	}
 
       /* Find a font-entity with the current spec and supporting C.  */
@@ -661,10 +664,12 @@
 	  for (j = 0; j < i; j++)
 	    ASET (new_vec, j, AREF (vec, j));
 	  ASET (new_vec, j, rfont_def);
+	  found_index = j;
 	  for (j++; j < ASIZE (new_vec); j++)
 	    ASET (new_vec, j, AREF (vec, j - 1));
 	  XSETCDR (font_group, new_vec);
-	  return rfont_def;
+	  vec = new_vec;
+	  goto found;
 	}
 
       /* No font of the current spec for C.  Try the next spec.  */
@@ -673,6 +678,20 @@
 
   FONTSET_SET (fontset, make_number (c), make_number (0));
   return Qnil;
+
+ found:
+  if (fallback && found_index > 0)
+    {
+      /* The order of fonts in the fallback font-group is not that
+	 important, and it is better to move the found font to the
+	 first of the group so that the next try will find it
+	 quickly. */
+      for (i = found_index; i > 0; i--)
+	ASET (vec, i, AREF (vec, i - 1));
+      ASET (vec, 0, rfont_def);
+      found_index = 0;
+    }
+  return rfont_def;
 }
 
 
@@ -685,13 +704,14 @@
 {
   Lisp_Object rfont_def;
   Lisp_Object base_fontset;
+  int try_fallback = 0, try_default_fallback = 0;
 
   /* Try a font-group of FONTSET. */
   rfont_def = fontset_find_font (fontset, c, face, id, 0);
   if (VECTORP (rfont_def))
     return rfont_def;
-  if (EQ (rfont_def, Qt))
-    goto no_font;
+  if (! EQ (rfont_def, Qt))
+    try_fallback = 1;
 
   /* Try a font-group of the default fontset. */
   base_fontset = FONTSET_BASE (fontset);
@@ -703,29 +723,30 @@
       rfont_def = fontset_find_font (FONTSET_DEFAULT (fontset), c, face, id, 0);
       if (VECTORP (rfont_def))
 	return rfont_def;
-      if (EQ (rfont_def, Qt))
-	goto no_font;
+      if (! EQ (rfont_def, Qt))
+	try_default_fallback = 1;
     }
 
   /* Try a fallback font-group of FONTSET. */
-  rfont_def = fontset_find_font (fontset, c, face, id, 1);
-  if (VECTORP (rfont_def))
-    return rfont_def;
-  if (EQ (rfont_def, Qt))
-    goto no_font;
+  if (try_fallback)
+    {
+      rfont_def = fontset_find_font (fontset, c, face, id, 1);
+      if (VECTORP (rfont_def))
+	return rfont_def;
+      /* Remember that FONTSET has no font for C.  */
+      FONTSET_SET (fontset, make_number (c), Qt);
+    }
 
   /* Try a fallback font-group of the default fontset . */
-  if (! EQ (base_fontset, Vdefault_fontset))
+  if (try_default_fallback)
     {
       rfont_def = fontset_find_font (FONTSET_DEFAULT (fontset), c, face, id, 1);
       if (VECTORP (rfont_def))
 	return rfont_def;
+      /* Remember that the default fontset has no font for C.  */
+      FONTSET_SET (FONTSET_DEFAULT (fontset), make_number (c), Qt);
     }
 
- no_font:
-  /* Remember that we have no font for C.  */
-  FONTSET_SET (fontset, make_number (c), Qt);
-
   return Qnil;
 }
 

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Stefan Monnier]

> Attached is a new patch I propose for 23.1 to solve the bug.
> It is not the best one, but is simpler and safer.

> I'll summarize the problem here.  Yidong and Stefan, please
> consider installing it.

I think it fixes an important enough new bug that it deserves to
be installed.  Thank you for working on this.


        Stefan

bug#3208: 23.0.93; Memory full / crash when displaying lots of characters from a large font (like Arial Unicode or Code2000) which is not explicitly selected (on Win32)

[Kenichi Handa]

In article <jwvws6qn4z4.fsf-monnier+emacsbugreports@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Attached is a new patch I propose for 23.1 to solve the bug.
> > It is not the best one, but is simpler and safer.

> > I'll summarize the problem here.  Yidong and Stefan, please
> > consider installing it.

> I think it fixes an important enough new bug that it deserves to
> be installed.  Thank you for working on this.

Ok, I've just installed them.

---
Kenichi Handa
handa@m17n.org