Items in FOR-RELEASE

Items in FOR-RELEASE

[Stefan]

I see in FOR-RELEASE the following items:

  * NEW FEATURES
  
  ** Face remapping.
  
  ** Let mouse-1 follow links.
  
  ** Make Rmail find the best version of movemail.
  To be done by Sergey Poznyakoff <gray@Mirddin.farlep.net>.
  
  ** Make VC-over-Tramp work where possible, or at least fail
  gracefully if something isn't supported over Tramp.
  To be done by Andre Spiegel <spiegel@gnu.org>.

Is that really correct?  I feel like the first two are not necessary for the
new release.  The new year is coming quickly and I feel like we should
really get a first pretest out in January.  So I'd like to try and trim all
the unnecessary stuff from the FOR-RELEASE, and then focus on what's left.


        Stefan

Re: Items in FOR-RELEASE

[Juri Linkov]

Stefan <monnier@iro.umontreal.ca> writes:
> I see in FOR-RELEASE the following items:
>
>   * NEW FEATURES
>   
>   ** Face remapping.
>
> Is that really correct?  I feel like the first two are not necessary
> for the new release.

There is one reason to include face remapping into the next release.
This would allow to get rid of all face variables that were introduced
after the previous release only for the sake of making face variables
buffer-local (i.e. face variables that were added with the same name
as face names).  With buffer-local faces such face variables are
useless, so it's better to remove them before the next release.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Items in FOR-RELEASE

[Stefan Monnier]

>> I see in FOR-RELEASE the following items:
>> 
>> * NEW FEATURES
>> 
>> ** Face remapping.
>> 
>> Is that really correct?  I feel like the first two are not necessary
>> for the new release.

> There is one reason to include face remapping into the next release.
> This would allow to get rid of all face variables that were introduced
> after the previous release only for the sake of making face variables
> buffer-local (i.e. face variables that were added with the same name
> as face names).  With buffer-local faces such face variables are
> useless, so it's better to remove them before the next release.

Huh?  What's your timeline for the next release? 2080?
Besides, what makes you think that face remapping is a better solution to
mode-specific faces?  I like face-remapping but it's not a panacea.


        Stefan

Re: Items in FOR-RELEASE

[Juri Linkov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> There is one reason to include face remapping into the next release.
>> This would allow to get rid of all face variables that were introduced
>> after the previous release only for the sake of making face variables
>> buffer-local (i.e. face variables that were added with the same name
>> as face names).  With buffer-local faces such face variables are
>> useless, so it's better to remove them before the next release.
>
> Huh?  What's your timeline for the next release? 2080?

Face remapping was announced long ago, and still not yet included
in Emacs.  Is it planned to add it to Emacs in 2080?

> Besides, what makes you think that face remapping is a better solution to
> mode-specific faces?  I like face-remapping but it's not a panacea.

Isn't something like below possible with face remapping?

(add-hook 'emacs-lisp-mode-hook
          (lambda ()
            (set (make-local-variable 'face-remappings)
                 '((font-lock-comment-face . bold)))))

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Items in FOR-RELEASE

[Stefan Monnier]

>> Besides, what makes you think that face remapping is a better solution to
>> mode-specific faces?  I like face-remapping but it's not a panacea.

> Isn't something like below possible with face remapping?

> (add-hook 'emacs-lisp-mode-hook
>           (lambda ()
>             (set (make-local-variable 'face-remappings)
>                  '((font-lock-comment-face . bold)))))

Beside the fact that you can't use Custom to set this up, the problem with
it is that it requires a preexistnig face with the required properties
(such as the `bold' face in your example).  If you want comments to appear
in "uglygreen" and that no other face uses "uglygreen" (or the other faces
the use "uglygreen" also use some other attribute that you don't want for
comments), you have to create your own with defface, ...

Hardly a a panacea,

Now, maybe we can provide a layer of code on top of it which will
transparently create faces when needed, etc... maybe as part of Custom, but
I don't see this happening any time soon.

In any case I'm not opposed at all to face-remapping (I've been asking for
something like it for a long time, actually), but I just think it should
either be installed right now or postponed to after the release.
I.e. it shouldn't be in FOR-RELEASE.

This release is coming *real* slow and if we keep going at this rate we'll
never get there.  Let's not forget that Emacs-CVS has not just some neat new
features, but it also has important changes that make it more usable on
newer/current systems (Mac OS X, utf-8, modifiers with new Xservers, PCL-CVS
support for CVS-1.12, ...).
It's more important that we release it soon than it is to include each and
every new feature.


        Stefan

Re: Items in FOR-RELEASE

[Juri Linkov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> Beside the fact that you can't use Custom to set this up, the problem with
> it is that it requires a preexistnig face with the required properties
> (such as the `bold' face in your example).  If you want comments to appear
> in "uglygreen" and that no other face uses "uglygreen" (or the other faces
> the use "uglygreen" also use some other attribute that you don't want for
> comments), you have to create your own with defface, ...

I don't know how face-remapping is implemented, but maybe it is possible
to use a property list of face attributes?

(add-hook 'emacs-lisp-mode-hook
          (lambda ()
            (set (make-local-variable 'face-remappings)
                 '((font-lock-comment-face . (:foreground "uglygreen"))))))

> In any case I'm not opposed at all to face-remapping (I've been
> asking for something like it for a long time, actually), but I just
> think it should either be installed right now or postponed to after
> the release.  I.e. it shouldn't be in FOR-RELEASE.

Maybe there is no need for a hurry in installing face-remapping.
But one reason I found was to get rid of face variables that duplicate
the faces with the same names, because it would be difficult to remove
them after the next release.  If it is desirable to have such
duplicate face variables, then there is still a problem.  The current
situation is inconsistent: some faces have corresponding face variables,
some not.  This should be fixed one way or the other.

-- 
Juri Linkov
http://www.jurta.org/emacs/

GC crashes (Was: Items in FOR-RELEASE)

[Juri Linkov]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> It's more important that we release it soon than it is to include
> each and every new feature.

It seems in its current state Emacs is too far from being ready for
the release.  In the last three months Emacs became too unstable:
I have experienced frequent crashes in GC (I had no crashes before
September 2004).  I haven't debugged them because debugging GC looks
to me like brain surgery.  However, unless this is a known problem,
I might try to debug it.

PS: The latest crash I got was just when composing the previous reply.
I attached the debug session below if it might help somehow.
The cause of the crash is the fact that symbol's `xname' contains
a vector.  I've looked at the contents of this vector, and it has the
values of variables `load-path' `user-init-file', `byte-boolean-vars',
etc.  The symbol's `value', `function' and `plist' have some absurd
values.  This looks like a memory corruption.  Could compiling with
ENABLE_CHECKING or other compiler options help detect where memory
corruption occurs?

(gdb) bt
#0  mark_object (arg=137262137) at alloc.c:5193
#1  0x0812490e in mark_object (arg=165544885) at alloc.c:5301
#2  0x0812136c in mark_interval (i=0x9e15a64, dummy=137262265) at alloc.c:1356
#3  0x0816bc39 in traverse_intervals_noorder (tree=0x9e15a10, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:207
#4  0x0816bc54 in traverse_intervals_noorder (tree=0x9e15394, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
#5  0x0816bc54 in traverse_intervals_noorder (tree=0x9e14abc, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
#6  0x0816bc54 in traverse_intervals_noorder (tree=0x9e16788, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
#7  0x0816bc54 in traverse_intervals_noorder (tree=0x9e12198, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
#8  0x0816bc54 in traverse_intervals_noorder (tree=0x9e131e8, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
#9  0x0812138c in mark_interval_tree (tree=0x9e131e8) at alloc.c:1371
#10 0x081249a4 in mark_buffer (buf=156504092) at alloc.c:5338
#11 0x0812401a in mark_object (arg=156504092) at alloc.c:5028
[...1500 more frames...]
(gdb) xba
"font-lock-fontify-keywords-region"
"font-lock-default-fontify-region"
"font-lock-fontify-region"
"run-hook-with-args"
"byte-code"
"jit-lock-fontify-now"
"byte-code"
"jit-lock-stealth-fontify"
"apply"
"byte-code"
"timer-event-handler"
(gdb) fr 0
#0  mark_object (arg=137262137) at alloc.c:5193
5193              MARK_STRING (XSTRING (ptr->xname));
(gdb) p ptr
$3 = (struct Lisp_Symbol *) 0x82a0ba0
(gdb) p *$
$4 = {
  gcmarkbit = 0, 
  indirect_variable = 0, 
  constant = 1, 
  interned = 0, 
  xname = 137228604, 
  value = 137199748, 
  function = 137199744, 
  plist = 137199772, 
  next = 0x82d7008
}
(gdb) p ptr->xname
$5 = 137228604
(gdb) xtype
Lisp_Vectorlike
167653757
(gdb) xvector
$6 = (struct Lisp_Vector *) 0x82df138
0
(gdb) p *$
$7 = {
  size = 167653757, 
  next = 0x82e7501, 
  contents = {143577373}
}
...

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: GC crashes

[Kim F. Storm]

Juri Linkov <juri@jurta.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> It's more important that we release it soon than it is to include
>> each and every new feature.
>
> It seems in its current state Emacs is too far from being ready for
> the release.  In the last three months Emacs became too unstable:
> I have experienced frequent crashes in GC (I had no crashes before
> September 2004).  I haven't debugged them because debugging GC looks
> to me like brain surgery.  However, unless this is a known problem,
> I might try to debug it.

Please do -- I know there is a memory corruption issue somewhere but
so far nobody's been able to identify under what circumstances they
happen.

>
> PS: The latest crash I got was just when composing the previous reply.
> I attached the debug session below if it might help somehow.
> The cause of the crash is the fact that symbol's `xname' contains
> a vector.  I've looked at the contents of this vector, and it has the
> values of variables `load-path' `user-init-file', `byte-boolean-vars',
> etc.  

Sounds like it points to the obarray or some such.  But probably the
pointer to the symbol is bogus, so someone wrote over that pointer,
so maybe you could look at where that pointer came from and see what
data is around there (to see if any other data there gives a clue
to what part of the code wrote over it).

>       The symbol's `value', `function' and `plist' have some absurd
> values.  This looks like a memory corruption.  Could compiling with
> ENABLE_CHECKING or other compiler options help detect where memory
> corruption occurs?

You can try changing the #if 0 at line 32 in lisp.h to #if 1, remove
all *.o files, rebuild emacs, and see if it catches the bug before it
happens (it will still abort emacs, but you can look at the data to
see what may be causing the memory overwrite).


>
> (gdb) bt
> #0  mark_object (arg=137262137) at alloc.c:5193
> #1  0x0812490e in mark_object (arg=165544885) at alloc.c:5301
> #2  0x0812136c in mark_interval (i=0x9e15a64, dummy=137262265) at alloc.c:1356
> #3  0x0816bc39 in traverse_intervals_noorder (tree=0x9e15a10, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:207
> #4  0x0816bc54 in traverse_intervals_noorder (tree=0x9e15394, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
> #5  0x0816bc54 in traverse_intervals_noorder (tree=0x9e14abc, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
> #6  0x0816bc54 in traverse_intervals_noorder (tree=0x9e16788, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
> #7  0x0816bc54 in traverse_intervals_noorder (tree=0x9e12198, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
> #8  0x0816bc54 in traverse_intervals_noorder (tree=0x9e131e8, function=0x8121354 <mark_interval>, arg=137262265) at intervals.c:212
> #9  0x0812138c in mark_interval_tree (tree=0x9e131e8) at alloc.c:1371
> #10 0x081249a4 in mark_buffer (buf=156504092) at alloc.c:5338
> #11 0x0812401a in mark_object (arg=156504092) at alloc.c:5028
> [...1500 more frames...]
> (gdb) xba
> "font-lock-fontify-keywords-region"
> "font-lock-default-fontify-region"
> "font-lock-fontify-region"
> "run-hook-with-args"
> "byte-code"
> "jit-lock-fontify-now"
> "byte-code"
> "jit-lock-stealth-fontify"
> "apply"
> "byte-code"
> "timer-event-handler"
> (gdb) fr 0
> #0  mark_object (arg=137262137) at alloc.c:5193
> 5193              MARK_STRING (XSTRING (ptr->xname));
> (gdb) p ptr
> $3 = (struct Lisp_Symbol *) 0x82a0ba0
> (gdb) p *$
> $4 = {
>   gcmarkbit = 0, 
>   indirect_variable = 0, 
>   constant = 1, 
>   interned = 0, 
>   xname = 137228604, 
>   value = 137199748, 
>   function = 137199744, 
>   plist = 137199772, 
>   next = 0x82d7008
> }
> (gdb) p ptr->xname
> $5 = 137228604
> (gdb) xtype
> Lisp_Vectorlike
> 167653757
> (gdb) xvector
> $6 = (struct Lisp_Vector *) 0x82df138
> 0
> (gdb) p *$
> $7 = {
>   size = 167653757, 
>   next = 0x82e7501, 
>   contents = {143577373}
> }
> ...
-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: GC crashes

[Paul Pogonyshev]

Kim F. Storm wrote:
> Please do -- I know there is a memory corruption issue somewhere but
> so far nobody's been able to identify under what circumstances they
> happen.

Somebody with a very fast machine with lots of RAM and lots of spare
time could try to run Emacs under Valgrind.

Paul

Re: GC crashes

[Jan D.]

2004-12-07 kl. 16.54 skrev Paul Pogonyshev:

> Kim F. Storm wrote:
>> Please do -- I know there is a memory corruption issue somewhere but
>> so far nobody's been able to identify under what circumstances they
>> happen.
>
> Somebody with a very fast machine with lots of RAM and lots of spare
> time could try to run Emacs under Valgrind.

 From Valgrind documentation:

    Programs which are known not to work are:
      * emacs starts up but immediately concludes it is out of memory and
        aborts. Emacs has it's own memory-management scheme, but I don't
        understand why this should interact so badly with Valgrind. Emacs
        works fine if you build it to use the standard malloc/free
        routines.


	Jan D.

Re: GC crashes

[Nick Roberts]

 > > Please do -- I know there is a memory corruption issue somewhere but
 > > so far nobody's been able to identify under what circumstances they
 > > happen.
 > 
 > Somebody with a very fast machine with lots of RAM and lots of spare
 > time could try to run Emacs under Valgrind.

Have you actually tried this? I can't get it to work with Emacs and the
documentation says:

doco> Emacs is known not to work with Valgrind because Emacs has its own
doco> memory-management scheme. Emacs works fine if you build it using the
doco> standard malloc/free routines.

and this is what I get:

==2325== Invalid free() / delete / delete[]
==2325==    at 0x1B900335: free (vg_replace_malloc.c:153)
==2325==    by 0x81467B6: (within /usr/bin/emacs-21.2)
==2325==    by 0x8054A15: (within /usr/bin/emacs-21.2)
==2325==    by 0x80F1A38: (within /usr/bin/emacs-21.2)
==2325==  Address 0x82B92D0 is not stack'd, malloc'd or (recently) free'd
emacs: Memory exhausted--use M-x save-some-buffers RET

I think it has something to do with the way that temacs dumps out emacs.

Recently I downloaded Valgrind 2.2.0, and this now works with temacs where
previously (2.0.0) it ended with a segmentation fault.

This runs like a dog though, and you need either a supercomputer or a lot of
patience.

Nick

Re: GC crashes

[Paul Pogonyshev]

Nick Roberts wrote:

>  > > Please do -- I know there is a memory corruption issue somewhere but
>  > > so far nobody's been able to identify under what circumstances they
>  > > happen.
>  >
>  > Somebody with a very fast machine with lots of RAM and lots of spare
>  > time could try to run Emacs under Valgrind.
>
> Have you actually tried this?

No, it was just a random idea.

> I can't get it to work with Emacs and the documentation says:
>
> doco> Emacs is known not to work with Valgrind because Emacs has its own
> doco> memory-management scheme. Emacs works fine if you build it using the
> doco> standard malloc/free routines.

I didn't know this.

> Recently I downloaded Valgrind 2.2.0, and this now works with temacs where
> previously (2.0.0) it ended with a segmentation fault.
>
> This runs like a dog though, and you need either a supercomputer or a lot
> of patience.

Programs under Valgrind run horribly slowly, but nevertheless I consider
Valgrind invaluable.  Under Valgrind your chances of spotting memory
corruption and the line it happens at go up like some 100 times (of course
it cannot notice if you write within allocated memory, but still to a
wrong place.)

Jan D. wrote:

> From Valgrind documentation:
>
>    Programs which are known not to work are:
>      * emacs starts up but immediately concludes it is out of memory and
>        aborts. Emacs has it's own memory-management scheme, but I don't
>        understand why this should interact so badly with Valgrind. Emacs
>        works fine if you build it to use the standard malloc/free
>        routines.

Then maybe someone should build it to use the standard malloc/free routines
and try?  (Yes, yes I should do it myself instead of bogging everybody else,
but I don't have that much CPU time and I have not seen the infamous memory
corruption yet, so I'm not that curious.)

Paul

Re: GC crashes

[Jan D.]

>>    Programs which are known not to work are:
>>      * emacs starts up but immediately concludes it is out of memory 
>> and
>>        aborts. Emacs has it's own memory-management scheme, but I 
>> don't
>>        understand why this should interact so badly with Valgrind. 
>> Emacs
>>        works fine if you build it to use the standard malloc/free
>>        routines.
>
> Then maybe someone should build it to use the standard malloc/free 
> routines
> and try?  (Yes, yes I should do it myself instead of bogging everybody 
> else,
> but I don't have that much CPU time and I have not seen the infamous 
> memory
> corruption yet, so I'm not that curious.)

I just tried that, but it failed in the same way.

	Jan D.

Re: GC crashes

[Nick Roberts]

 > > Then maybe someone should build it to use the standard malloc/free
 > > routines and try?  (Yes, yes I should do it myself instead of bogging
 > > everybody else, but I don't have that much CPU time and I have not seen
 > > the infamous memory corruption yet, so I'm not that curious.)
 > 
 > I just tried that, but it failed in the same way.

I recently asked on the Valgrind mailing list about this very matter. This
is what Julian Seward, the author of Valgrind said:

JS> I"ve had varying success running GNU emacs on recent V"s; one time
JS> it worked, but another time it didn"t.  I think it is to do with
JS> emacs doing something wildly non-portable with brk() and being
JS> confused by V"s behaviour.  xemacs doesn"t have any such stupidness.

XEmacs doesn't seem to work with Valgrind, but assuming it dumps the
executable in the same way, thats not surprising.

JS> "build it using the standard malloc/free routines" is just a ./configure
JS> option to emacs, IIRC.  Something like --use-system-malloc -- you"ll have
JS> to check, I don"t really remember.  Note I also have to use 
JS> --with-x-toolkit=NONE in order to get a build of emacs which doesn"t
JS> simply segfault at startup.

I tried building Emacs with -DSYSTEM_MALLOC, --with-x-toolkit=no, and even
--without-x (but not all combinations) to no avail.

I feel the code that causes a problem is probably in unexec.c, or possibly
emacs.c but this is just a guess and not a very educated one at that.

Nick

Re: GC crashes

[Eli Zaretskii]

> From: Nick Roberts <nickrob@snap.net.nz>
> Date: Wed, 8 Dec 2004 20:31:29 +1300
> Cc: "Kim F. Storm" <storm@cua.dk>, emacs-devel@gnu.org,
> 	Paul Pogonyshev <pogonyshev@gmx.net>
> 
> XEmacs doesn't seem to work with Valgrind, but assuming it dumps the
> executable in the same way, thats not surprising.

AFAIK, XEmacs uses a different dumping method, at least on modern
platforms.

> I feel the code that causes a problem is probably in unexec.c, or possibly
> emacs.c but this is just a guess and not a very educated one at that.

These problems were reported and discussed in the past, so if someone
would like to work on fixing these problems, they would be better off
if they first search the emacs-devel archives.

Re: GC crashes

[Nick Roberts]

 > > I feel the code that causes a problem is probably in unexec.c, or possibly
 > > emacs.c but this is just a guess and not a very educated one at that.
 > 
 > These problems were reported and discussed in the past, so if someone
 > would like to work on fixing these problems, they would be better off
 > if they first search the emacs-devel archives.

I don't recall much insight being shed on previous occasions and now that
Valgrind (2.2.0) does work with temacs, the focus can now shift to
looking at why emacs fails, i.e probably the dumping.

Nick

Re: GC crashes

[Richard Stallman]

It used to be possible to run Emacs with `temacs -l loadup'
and get a real session.

I tried this just now on a tty and it more or less worked, but there
were glaring problems in redisplay.  I would guess that a little
debugging would suffice to fix those problems.  Then maybe Emacs could
run under valgrind in this mode.

Re: Items in FOR-RELEASE

[Miles Bader]

> Beside the fact that you can't use Custom to set this up, the problem with
> it is that it requires a preexistnig face with the required properties

No it doesn't, you can specify face properties directly in the remapping list.

The reason it's not merged is because Richard didn't like one aspect of the
implementation and I didn't feel like arguing about it.

-Miles

Re: Items in FOR-RELEASE

[Stefan Monnier]

>> Beside the fact that you can't use Custom to set this up, the problem with
>> it is that it requires a preexistnig face with the required properties
> No it doesn't, you can specify face properties directly in the remapping list.

Hmm.... good point.

> The reason it's not merged is because Richard didn't like one aspect of the
> implementation and I didn't feel like arguing about it.

Too sad.


        Stefan

Re: Items in FOR-RELEASE

[Richard Stallman]

    > The reason it's not merged is because Richard didn't like one aspect of the
    > implementation and I didn't feel like arguing about it.

    Too sad.

It is a useful feature.  We need someone to make the implementation
changes that I asked for, so we can then install this feature.

Re: Items in FOR-RELEASE

[Kim F. Storm]

Miles Bader <snogglethorpe@gmail.com> writes:

>> Beside the fact that you can't use Custom to set this up, the problem with
>> it is that it requires a preexistnig face with the required properties
>
> No it doesn't, you can specify face properties directly in the remapping list.
>
> The reason it's not merged is because Richard didn't like one aspect of the
> implementation and I didn't feel like arguing about it.

Could you forward the objections to emacs-devel to give other a change
to argue about it :-)

I didn't see any major issues with your approach, so I would like to
know what was "wrong" with it.


-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Items in FOR-RELEASE

[Richard Stallman]

What is the status of implementing face remapping?
What work needs to be done to make it installable?