All platforms fail with Unicode in menus.

All platforms fail with Unicode in menus.

[David Kastrup]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]


Hi, considering that most Window systems can deal with iso10646 fonts
more or less properly, one should imagine that this would mean Unicode
to be supported in most environments.

The following package (a standalone excerpt from AUCTeX) can be loaded
with

  emacs -q -no-site-file unicode-menu.el

Pressing <f8> will add a "Math" menu to the menu bar.  This menu
contains TeX control sequences and corresponding Unicode.

Most platforms just produce some garbage that looks like an attempt at
Latin-1.  This includes Carbon, the standard X11 toolkits and Windows.
The best contender right now is GTK which gets it right, except that
it completely garbles the menus when garbage collection occurs during
their creation (the gc-cons-threshold setting at the end of this
example file more or less ensures that stuff gets garbled).

Since those kind of menus make for quite a bit of attraction, I'd
kindly ask the various platform programmer to consider accessing
Unicode support of their operating system/toolkit, and I'd appreciate
it if the GTK support of Unicode was made robust against garbage
collection.

Thanks,

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

[-- Attachment #2: unicode-menu.el --]
[-- Type: application/emacs-lisp, Size: 29747 bytes --]

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: All platforms fail with Unicode in menus.

[YAMAMOTO Mitsuharu]

>>>>> On 25 Aug 2004 23:48:23 +0200, David Kastrup <dak@gnu.org> said:

> Since those kind of menus make for quite a bit of attraction, I'd
> kindly ask the various platform programmer to consider accessing
> Unicode support of their operating system/toolkit, and I'd
> appreciate it if the GTK support of Unicode was made robust against
> garbage collection.

Here's a patch for Mac OS X, Carbon.  It also contains a fix for the
bug that the menu bar cannot be activated just after a pop-up menu is
used.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Index: src/macmenu.c
===================================================================
RCS file: /cvsroot/emacs/emacs/src/macmenu.c,v
retrieving revision 1.15
diff -c -r1.15 macmenu.c
*** src/macmenu.c	30 May 2004 00:18:41 -0000	1.15
--- src/macmenu.c	26 Aug 2004 05:37:37 -0000
***************
*** 163,168 ****
--- 163,174 ----
  
  extern Lisp_Object Qmenu_bar_update_hook;
  
+ #if TARGET_API_MAC_CARBON
+ #define ENCODE_MENU_STRING(str) ENCODE_UTF_8 (str)
+ #else
+ #define ENCODE_MENU_STRING(str) ENCODE_SYSTEM (str)
+ #endif
+ 
  void set_frame_menubar ();
  
  static void push_menu_item P_ ((Lisp_Object, Lisp_Object, Lisp_Object,
***************
*** 1246,1258 ****
  #ifndef HAVE_MULTILINGUAL_MENU
  	  if (STRING_MULTIBYTE (item_name))
  	    {
! 	      item_name = ENCODE_SYSTEM (item_name);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_NAME) = item_name;
  	    }
  
  	  if (STRINGP (descrip) && STRING_MULTIBYTE (descrip))
  	    {
! 	      descrip = ENCODE_SYSTEM (descrip);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_EQUIV_KEY) = descrip;
  	    }
  #endif /* not HAVE_MULTILINGUAL_MENU */
--- 1252,1264 ----
  #ifndef HAVE_MULTILINGUAL_MENU
  	  if (STRING_MULTIBYTE (item_name))
  	    {
! 	      item_name = ENCODE_MENU_STRING (item_name);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_NAME) = item_name;
  	    }
  
  	  if (STRINGP (descrip) && STRING_MULTIBYTE (descrip))
  	    {
! 	      descrip = ENCODE_MENU_STRING (descrip);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_EQUIV_KEY) = descrip;
  	    }
  #endif /* not HAVE_MULTILINGUAL_MENU */
***************
*** 1705,1716 ****
  #ifndef HAVE_MULTILINGUAL_MENU
            if (STRINGP (item_name) && STRING_MULTIBYTE (item_name))
  	    {
! 	      item_name = ENCODE_SYSTEM (item_name);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_NAME) = item_name;
  	    }
            if (STRINGP (descrip) && STRING_MULTIBYTE (descrip))
              {
! 	      descrip = ENCODE_SYSTEM (descrip);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_EQUIV_KEY) = descrip;
  	    }
  #endif /* not HAVE_MULTILINGUAL_MENU */
--- 1711,1722 ----
  #ifndef HAVE_MULTILINGUAL_MENU
            if (STRINGP (item_name) && STRING_MULTIBYTE (item_name))
  	    {
! 	      item_name = ENCODE_MENU_STRING (item_name);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_NAME) = item_name;
  	    }
            if (STRINGP (descrip) && STRING_MULTIBYTE (descrip))
              {
! 	      descrip = ENCODE_MENU_STRING (descrip);
  	      AREF (menu_items, i + MENU_ITEMS_ITEM_EQUIV_KEY) = descrip;
  	    }
  #endif /* not HAVE_MULTILINGUAL_MENU */
***************
*** 1764,1770 ****
  
  #ifndef HAVE_MULTILINGUAL_MENU
        if (STRING_MULTIBYTE (title))
! 	title = ENCODE_SYSTEM (title);
  #endif
        wv_title->name = (char *) SDATA (title);
        wv_title->enabled = TRUE;
--- 1770,1776 ----
  
  #ifndef HAVE_MULTILINGUAL_MENU
        if (STRING_MULTIBYTE (title))
! 	title = ENCODE_MENU_STRING (title);
  #endif
        wv_title->name = (char *) SDATA (title);
        wv_title->enabled = TRUE;
***************
*** 1813,1818 ****
--- 1819,1828 ----
    discard_mouse_events ();
  #endif
  
+   /* Must reset this manually because the button release event is not
+      passed to Emacs event loop. */
+   FRAME_MAC_DISPLAY_INFO (f)->grabbed = 0;
+ 
    /* Free the widget_value objects we used to specify the
       contents.  */
    free_menubar_widget_value_tree (first_wv);
***************
*** 2219,2226 ****
--- 2229,2246 ----
  	  strncat (item_name, wv->key, 255);
  	}
        item_name[255] = 0;
+ #if TARGET_API_MAC_CARBON
+       {
+ 	CFStringRef string =
+ 	  CFStringCreateWithCString (NULL, item_name, kCFStringEncodingUTF8);
+ 
+ 	SetMenuItemTextWithCFString (menu, pos, string);
+ 	CFRelease (string);
+       }
+ #else
        c2pstr (item_name);
        SetMenuItemText (menu, pos, item_name);
+ #endif
  
        if (wv->enabled && !force_disable)
  #if TARGET_API_MAC_CARBON

Re: All platforms fail with Unicode in menus.

[Jan D.]

David Kastrup wrote:

> The best contender right now is GTK which gets it right, except that
> it completely garbles the menus when garbage collection occurs during
> their creation (the gc-cons-threshold setting at the end of this
> example file more or less ensures that stuff gets garbled).

It seems that the string returned by ENCODE_UTF_8 gets collected during GC,
resulting in passing garbage to the GTK menu code.

I can work around the bug by this patch (the call to inhibit_garbage_collection 
is present in macmenu.c), but I am not sure it is the correct way to fix this:

*** xmenu.c.~1.255.~    2004-01-12 00:15:16.000000000 +0100
--- xmenu.c     2004-08-26 20:18:28.000000000 +0200
***************
*** 1930,1935 ****
--- 1930,1936 ----
         FRAME_MENU_BAR_ITEMS (f) = menu_bar_items (FRAME_MENU_BAR_ITEMS (f));

         items = FRAME_MENU_BAR_ITEMS (f);
+       inhibit_garbage_collection ();

         /* Save the frame's previous menu bar contents data.  */
         if (previous_menu_items_used)

Can someone that knows ENCODE_UTF_8 and garbage collection well comment on this 
patch?

Thanks,

	Jan D.

Re: All platforms fail with Unicode in menus.

[David Kastrup]

"Jan D." <jan.h.d@swipnet.se> writes:

> David Kastrup wrote:
> 
> > The best contender right now is GTK which gets it right, except that
> > it completely garbles the menus when garbage collection occurs during
> > their creation (the gc-cons-threshold setting at the end of this
> > example file more or less ensures that stuff gets garbled).
> 
> It seems that the string returned by ENCODE_UTF_8 gets collected during GC,
> resulting in passing garbage to the GTK menu code.
> 
> I can work around the bug by this patch (the call to
> inhibit_garbage_collection is present in macmenu.c), but I am not sure
> it is the correct way to fix this:

> +       inhibit_garbage_collection ();

> Can someone that knows ENCODE_UTF_8 and garbage collection well
> comment on this patch?

I don't know either well, but of course that is not the correct way to
fix it.  The correct way would be to add appropriate GCPRO and UNGCPRO
calls for temporary data structures that should not yet be collected.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: All platforms fail with Unicode in menus.

[Jan D.]

>
>> +       inhibit_garbage_collection ();
>
>> Can someone that knows ENCODE_UTF_8 and garbage collection well
>> comment on this patch?
>
> I don't know either well, but of course that is not the correct way to
> fix it.  The correct way would be to add appropriate GCPRO and UNGCPRO
> calls for temporary data structures that should not yet be collected.

I suspect this must be done in the coding functions.  This call to 
inhibit_garbage_collection was there about two years ago (1.234 removed 
it),
and it is still present in macmenu.c, which is the only reason the mac 
port
works better.  I suspect the w32 port also has this problem, but I can't
test that port.  I'll work a bit more on this, as a temporary solution I
checked in this patch.

	Jan D.

Re: All platforms fail with Unicode in menus.

[David Kastrup]

"Jan D." <jan.h.d@swipnet.se> writes:

> >
> >> +       inhibit_garbage_collection ();
> >
> >> Can someone that knows ENCODE_UTF_8 and garbage collection well
> >> comment on this patch?
> >
> > I don't know either well, but of course that is not the correct way to
> > fix it.  The correct way would be to add appropriate GCPRO and UNGCPRO
> > calls for temporary data structures that should not yet be collected.
> 
> I suspect this must be done in the coding functions.

Well, I glimpsed through the code, and it looked like there was quite
a bit of work involved.  Basically, whenever you have an automatic
Lisp_Object come into being and its life time reaches across some code
that might CONS memory (almost any Lisp code), and it may be the only
pointer to the respective Lisp Object, then you need to GCPRO it.

Glancing through the code looked like there was quite a bit of that.

The results of things like ENCODE_STRING or what it was need to get
into something GCPRO before garbage collection can strike.

> This call to inhibit_garbage_collection was there about two years
> ago (1.234 removed it), and it is still present in macmenu.c, which
> is the only reason the mac port works better.  I suspect the w32
> port also has this problem, but I can't test that port.  I'll work a
> bit more on this, as a temporary solution I checked in this patch.

Yes, as a temporary solution it will work, but it has the disadvantage
that during menu buildup, no old memory will be reclaimed for the
construction of the menus.  And I don't even know how long the effects
of inhibit_garbage_collection will last.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: All platforms fail with Unicode in menus.

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> Well, I glimpsed through the code, and it looked like there was quite
> a bit of work involved.  Basically, whenever you have an automatic
> Lisp_Object come into being and its life time reaches across some code
> that might CONS memory (almost any Lisp code), and it may be the only
> pointer to the respective Lisp Object, then you need to GCPRO it.

Fcons does not GC.  Only Feval, Ffuncall, Fbytecode and read_char can GC.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: All platforms fail with Unicode in menus.

[Stefan]

>> +       inhibit_garbage_collection ();
>> Can someone that knows ENCODE_UTF_8 and garbage collection well
>> comment on this patch?
> I don't know either well, but of course that is not the correct way to
> fix it.  The correct way would be to add appropriate GCPRO and UNGCPRO
> calls for temporary data structures that should not yet be collected.

On what system is that, and how is it compiled.
The thing is that GCPROs are almost never used nowadays: on most systems we
just conservatively scan the stack.  So unless his config happens to not do
the conservative stack scan (and thus use the GCPROs instead), adding GCPROs
won't do a thing.


        Stefan

Re: All platforms fail with Unicode in menus.

[Jan D.]

>>> +       inhibit_garbage_collection ();
>>> Can someone that knows ENCODE_UTF_8 and garbage collection well
>>> comment on this patch?
>> I don't know either well, but of course that is not the correct way to
>> fix it.  The correct way would be to add appropriate GCPRO and UNGCPRO
>> calls for temporary data structures that should not yet be collected.
>
> On what system is that, and how is it compiled.
> The thing is that GCPROs are almost never used nowadays: on most 
> systems we
> just conservatively scan the stack.  So unless his config happens to 
> not do
> the conservative stack scan (and thus use the GCPROs instead), adding 
> GCPROs
> won't do a thing.

I don't know what system David originally used, but I used GNU/Linux on 
x86,
with gcc 3.4.1.
If the GCPROs don't help, can for example static Lisp_Object:s be the 
source
of this problem?  Are there any other cases where Lisp_Object:s should
be protected?

	Jan D.

Re: All platforms fail with Unicode in menus.

[Stefan]

> I don't know what system David originally used, but I used GNU/Linux on x86,
> with gcc 3.4.1.

OK, so GCPROs won't help.

> If the GCPROs don't help, can for example static Lisp_Object:s be the source
> of this problem?

Yes, very much so.  You need to `staticpro' them.

> Are there any other cases where Lisp_Object:s should be protected?

Well, Lisp_Objects in the stack are protected by stack scanning or GCPRO,
Lisp_Objects in the static area are protected by staticpro, and Lisp_Objects
in the heap are protected by tracing, so you should make sure that any
Lisp_Objects in the heap are properly traced (typically, that means that
they are properly handled by mark_object).


        Stefan

Re: All platforms fail with Unicode in menus.

[YAMAMOTO Mitsuharu]

>>>>> On Thu, 26 Aug 2004 20:30:12 +0200, "Jan D." <jan.h.d@swipnet.se> said:

> It seems that the string returned by ENCODE_UTF_8 gets collected
> during GC, resulting in passing garbage to the GTK menu code.

In this case, the Lisp Object returned by ENCODE_UTF_8 is reachable
from the root set via `menu_items', which is staticpro-ed.  I think
the problem is that ENCODE_UTF_8 may cause GC, and that leads to
relocation of string contents by the compaction of small strings.
Without the call of inhibit_garbage_collection, the buffer list in the
"Buffers" menu is also corrupted if "Math" menu is present.

				     YAMAMOTO Mitsuharu
				mituharu@math.s.chiba-u.ac.jp

Re: All platforms fail with Unicode in menus.

[Jan D.]

>> It seems that the string returned by ENCODE_UTF_8 gets collected
>> during GC, resulting in passing garbage to the GTK menu code.
>
> In this case, the Lisp Object returned by ENCODE_UTF_8 is reachable
> from the root set via `menu_items', which is staticpro-ed.  I think
> the problem is that ENCODE_UTF_8 may cause GC, and that leads to
> relocation of string contents by the compaction of small strings.
> Without the call of inhibit_garbage_collection, the buffer list in the
> "Buffers" menu is also corrupted if "Math" menu is present.

Yes, you are correct.  The problem is that the menu code saves a pointer
to the strings in the menus, not the Lisp value, in its internal data
structure.  As it reaches the "Math" menu, the compaction of small 
strings
makes the previously saved pointers point to garbage as the strings has 
been
moved.

This problem has been considered in the code when the 
inhibit_garbage_collection was removed.  The pointers to the strings 
are
not stored until after the whole data structure is done, and no GC can 
happen.  But it is only done properly for the top level (i.e. menu bar)
entries.  Fixing the others should be easy, I'll do it within a day or
two.

	Jan D.