Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Joe Wells]

I have some issues with the Emacs Lisp manual's coverage of
overlays, display properties, margins, etc.  In the issues I
raise below, I'll assume this macro:

  (defmacro test-overlays (&rest body)
    `(let ((buf (get-buffer-create "xyzzy")))
      (with-current-buffer buf
        (display-buffer buf)
        (erase-buffer)
        (dolist (o (overlays-in (point-min) (point-max)))
          (delete-overlay o))
        ,@body)))

Issue 1:  The “height” display specification seems not to work at
all.  For example, in the following the B is not larger:

  (test-overlays
   (insert "ABC")
   (let ((o (make-overlay 2 3)))
     (overlay-put o 'display '(height 2))))

This seems to be a bug.  But perhaps merely the documentation is
wrong.  Can someone please check?

Issue 2:  RMS recently clarified the text in the “Display
Property” manual node (in the Emacs 22 CVS branch) to read like
this:

  The value of the @code{display} property should be a display
  specification, or a list or vector containing several display
  specifications.  Display specifications generally apply in parallel to
  the text they cover.

    Some kinds of @code{display} specifications specify something to
  display instead of the text that has the property.  If a list of
  display specifications includes more than one of this kind, the first
  is effective and the rest are ignored.

When it says that “display specifications generally apply in
parallel to the text they cover”, is that supposed to include
display properties of multiple overlays covering the same text?
For example, this works:

  (test-overlays
   (insert "A B C\nD E F")
   (let ((o1 (make-overlay 2 5)))
     (overlay-put o1 'display '((space-width 3) (raise 0.5)))))

In contrast, in the following, the display property of overlay o1
is completely ignored:

  (test-overlays
   (insert "A B C\nD E F")
   (let ((o1 (make-overlay 2 5))
         (o2 (make-overlay 2 5)))
     (overlay-put o2 'priority 1)
     (overlay-put o1 'display '(space-width 3))
     (overlay-put o2 'display '(raise 0.5))))

So “display specifications generally apply in parallel to the
text they cover” probably deserves a clarification to point out
that only the display specifications in one display property can
cover a piece of text.

Issue 3:  Does “display specifications generally apply in
parallel to the text they cover” mean that if one display spec
specifies a replacement, then the other display specs in parallel
with it cover it?  It seems not, because the Z in this is not
raised:

  (test-overlays
   (insert "ABC")
   (let ((o1 (make-overlay 2 3)))
     (overlay-put o1 'display '((raise 0.5) "Z"))))

Similarly, the space in "Y Z" in this is not wider:

  (test-overlays
   (insert "ABC")
   (let ((o1 (make-overlay 2 3)))
     (overlay-put o1 'display '((space-width 8) "Y Z"))))

So it would be helpful to clarify that a display spec never
covers replacement text in another display spec parallel to it.

Issue 4:  What does “display specifications generally apply in
parallel to the text they cover” mean in the presence of two
“slice” display specs?  I'm guessing there should be a
clarification here to specify that the leftmost one wins.  Am I
right?

Issue 5:  The “Display Margins” manual node says:

     To put text in the left or right display margin of the window, use a
  display specification of the form `(margin right-margin)' or `(margin
  left-margin)' on it.

However, this doesn't put anything in the margin:

  (test-overlays
   (insert "ABC")
   (let ((o1 (make-overlay 2 3)))
     (overlay-put o1 'display '(margin left-margin)))
   (set-window-margins (get-buffer-window (current-buffer)) 8))

As far as I can tell, (margin MARGIN) is not actually a valid
display spec, but only part of a display spec when in the
form ((margin MARGIN) SPEC) where SPEC must be a display spec
that replaces the covered text.  For example, this works:

  (test-overlays
   (insert "ABC")
   (let ((o1 (make-overlay 2 3)))
     (overlay-put o1 'display '((margin left-margin) "Z")))
   (set-window-margins (get-buffer-window (current-buffer)) 8))

But this doesn't put anything in the margin:

  (test-overlays
   (insert "ABC")
   (let ((o1 (make-overlay 2 3)))
     (overlay-put o1 'display '("Z" (margin left-margin))))
   (set-window-margins (get-buffer-window (current-buffer)) 8))

Can someone please check what the C code actually does and
clarify/correct this part of the manual accordingly?

Issue 6:  The “Overlay Properties” manual node says this on
overlay priorities:

  `priority'
       This property's value (which should be a nonnegative integer
       number) determines the priority of the overlay.  The priority
       matters when two or more overlays cover the same character and
       both specify the same property; the one whose `priority' value is
       larger takes priority over the other.  For the `face' property, ...

So what is the relative priority of an overlay which has a
nonnegative integer value of its priority property, and one which
has no priority property at all?  It seems that nil is below 0:

  (test-overlays
   (insert "ABC")
   (let ((o1 (make-overlay 2 3))
         (o2 (make-overlay 2 3)))
     (overlay-put o1 'display "123")
     (overlay-put o2 'priority 0)
     (overlay-put o2 'display "abc")))

This could beneficially be detailed in the manual.

I hope the points I made above can help in clarifying the manual (and
maybe fixing a bug in the “height” display specs).

Joe

======================================================================
In GNU Emacs 22.1.1 (i686-pc-linux-gnu, GTK+ Version 2.8.20)
 of 2007-06-27 on artemis
Windowing system distributor `The X.Org Foundation', version 11.0.70000000
configured using `configure  '--prefix=/home/jbw/local2' '--enable-debug' '--disable-nls' '--with-x-toolkit=gtk' 'CFLAGS=-O0 -g3 -ggdb''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: jbw
  value of $LANG: nil
  locale-coding-system: utf-8
  default-enable-multibyte-characters: t

Minor modes in effect:
  mml-mode: t
  flyspell-mode: t
  TeX-source-specials-mode: t
  desktop-save-mode: t
  url-handler-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  size-indication-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Johan Bockgård]

Joe Wells <jbw@macs.hw.ac.uk> writes:

> Issue 1:  The “height” display specification seems not to work at
> all.  For example, in the following the B is not larger:
>
>   (test-overlays
>    (insert "ABC")
>    (let ((o (make-overlay 2 3)))
>      (overlay-put o 'display '(height 2))))

This worked for me.

> Issue 6:  The “Overlay Properties” manual node says this on
> overlay priorities: [...]

> So what is the relative priority of an overlay which has a
> nonnegative integer value of its priority property, and one which
> has no priority property at all?  It seems that nil is below 0:
>
>   (test-overlays
>    (insert "ABC")
>    (let ((o1 (make-overlay 2 3))
>          (o2 (make-overlay 2 3)))
>      (overlay-put o1 'display "123")
>      (overlay-put o2 'priority 0)
>      (overlay-put o2 'display "abc")))

Both overlays have priority 0. o2 takes precedence because it is on top
(you get the same result without the second last line).

-- 
Johan Bockgård

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Joe Wells]

Johan Bockgård <bojohan+news@dd.chalmers.se> writes:
>Joe Wells <jbw@macs.hw.ac.uk> writes:
>
>> Issue 1:  The “height” display specification seems not to work at
>> all.  For example, in the following the B is not larger:
>>
>>   (test-overlays
>>    (insert "ABC")
>>    (let ((o (make-overlay 2 3)))
>>      (overlay-put o 'display '(height 2))))
>
> This worked for me.

For me, running all of the above in a fresh “emacs -Q”, all three
characters (A, B, and C) get displayed with font
“-Misc-Fixed-Medium-R-SemiCondensed--13-120-75-75-C-60-ISO8859-1”.  I
used “C-u C-x =” to find out what font is used.

By the way, I have no font settings for Emacs in my X resources.

Have you tried the example in a fresh “emacs -Q”?  If so, what fonts
do the 3 characters get displayed with?  Can you try it also with no X
resource font settings for Emacs?

>> Issue 6:  The “Overlay Properties” manual node says this on
>> overlay priorities: [...]
>
>> So what is the relative priority of an overlay which has a
>> nonnegative integer value of its priority property, and one which
>> has no priority property at all?  It seems that nil is below 0:
>>
>>   (test-overlays
>>    (insert "ABC")
>>    (let ((o1 (make-overlay 2 3))
>>          (o2 (make-overlay 2 3)))
>>      (overlay-put o1 'display "123")
>>      (overlay-put o2 'priority 0)
>>      (overlay-put o2 'display "abc")))
>
> Both overlays have priority 0. o2 takes precedence because it is on top
> (you get the same result without the second last line).

Can this please be documented?  Both that no “priority” property (or a
value of nil) is equivalent to a value of 0, and how priority works
for overlays with the same priority.

-- 
Joe

P.S.  Here is the test-overlays macro again:

  (defmacro test-overlays (&rest body)
    `(let ((buf (get-buffer-create "xyzzy")))
      (with-current-buffer buf
        (display-buffer buf)
        (erase-buffer)
        (dolist (o (overlays-in (point-min) (point-max)))
          (delete-overlay o))
        ,@body)))

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Johan Bockgård]

Joe Wells <jbw@macs.hw.ac.uk> writes:

> Johan Bockgård <bojohan+news@dd.chalmers.se> writes:
>>Joe Wells <jbw@macs.hw.ac.uk> writes:
>>
>>> Issue 1:  The “height” display specification seems not to work at
>>> all.  For example, in the following the B is not larger:
>>>
>>>   (test-overlays
>>>    (insert "ABC")
>>>    (let ((o (make-overlay 2 3)))
>>>      (overlay-put o 'display '(height 2))))
>>
>> This worked for me.
>
> For me, running all of the above in a fresh “emacs -Q”, all three
> characters (A, B, and C) get displayed with font
> “-Misc-Fixed-Medium-R-SemiCondensed--13-120-75-75-C-60-ISO8859-1”.  I
> used “C-u C-x =” to find out what font is used.

Ok, I too get no scaling with the "fixed" font.

I was using courier (FWIW, `C-u C-x =' reports the same thing for all 3
characters).

-- 
Johan Bockgård

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Richard Stallman]

    Can this please be documented?  Both that no ?priority? property (or a
    value of nil) is equivalent to a value of 0,

Ok, will do.

						 and how priority works
    for overlays with the same priority.

I am not sure which overlay wins when they have the same priority
and same span.  I think it is better not to depend on that.

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Joe Wells]

Richard Stallman <rms@gnu.org> writes:

>     Can this please be documented?  Both that no ?priority? property (or a
>     value of nil) is equivalent to a value of 0,
>
> Ok, will do.

Thanks!

> 						 and how priority works
>     for overlays with the same priority.
>
> I am not sure which overlay wins when they have the same priority
> and same span.  I think it is better not to depend on that.

Can this please be stated in the manual?

-- 
Joe

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Richard Stallman]

There seems to be something totally fishy with the way margin display
properties work:

      (test-overlays
       (insert "ABC")
       (let ((o1 (make-overlay 2 3)))
	 (overlay-put o1 'display '((margin left-margin) "Z")))
       (set-window-margins (get-buffer-window (current-buffer)) 8))

this causes Z to appear in the margin and causes B not to appear
in the body of the text.

That seems like a bug to me.  It seems to me that a property that puts
something in the margin should not alter the appearance of the text
it is on.

Does anyone think the current behavior is desirable?

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Johan Bockgård]

Richard Stallman <rms@gnu.org> writes:

> There seems to be something totally fishy with the way margin display
> properties work:
>
>       (test-overlays
>        (insert "ABC")
>        (let ((o1 (make-overlay 2 3)))
> 	 (overlay-put o1 'display '((margin left-margin) "Z")))
>        (set-window-margins (get-buffer-window (current-buffer)) 8))
>
> this causes Z to appear in the margin and causes B not to appear
> in the body of the text.
>
> That seems like a bug to me.  It seems to me that a property that puts
> something in the margin should not alter the appearance of the text
> it is on.
>
> Does anyone think the current behavior is desirable?

In any case, it follows the documentation

    If you put such a display specification directly on text in the
    buffer, the specified margin display appears _instead of_ that
    buffer text itself. To put something in the margin _in association
    with_ certain buffer text without preventing or altering the display
    of that text, put a `before-string' property on the text and put the
    display specification on the contents of the before-string.

(info "(elisp)Display Margins")

-- 
Johan Bockgård

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Joe Wells]

Richard Stallman <rms@gnu.org> writes:

> There seems to be something totally fishy with the way margin display
> properties work:
>
>       (test-overlays
>        (insert "ABC")
>        (let ((o1 (make-overlay 2 3)))
> 	 (overlay-put o1 'display '((margin left-margin) "Z")))
>        (set-window-margins (get-buffer-window (current-buffer)) 8))
>
> this causes Z to appear in the margin and causes B not to appear
> in the body of the text.
>
> That seems like a bug to me.  It seems to me that a property that puts
> something in the margin should not alter the appearance of the text
> it is on.

It seems to follow the definition though.  The current definition
seems to be that a display spec of the form ((margin MARGIN) SPEC)
works when SPEC is a display spec that causes something to be
displayed instead of the text that has the display property and the
(margin MARGIN) part acts as a modifier that additionally moves the
replacement material into the margin.

The documentation of margin display spec modifiers is a bit confusing.

> Does anyone think the current behavior is desirable?

I find the current definition of the margin display spec modifier
awkward.  I think part of the reason for the way the current margin
modifier definition is designed is that (image ...) display specs are
not strings and thus can not have display properties of their own.
Also, the only current way to display an image is to have it _replace_
characters (at least one) in a string or in the buffer.

Maybe there should be a margin property (distinct from the current
margin display spec) which specifies material to go in the margin
(like the way the overlay before-string property works).

-- 
Joe

P.S.  For people who want to try the sexp above, the test-overlays
macro is this:

  (defmacro test-overlays (&rest body)
    `(let ((buf (get-buffer-create "xyzzy")))
      (with-current-buffer buf
        (display-buffer buf)
        (erase-buffer)
        (dolist (o (overlays-in (point-min) (point-max)))
          (delete-overlay o))
        ,@body)))

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Stefan Monnier]

> Maybe there should be a margin property (distinct from the current
> margin display spec) which specifies material to go in the margin
> (like the way the overlay before-string property works).

IIRC Gerd decided to use a single property `display' rather than several
properties, for reasons of performance.  I do not know or remember what was
the particular performance issue(s), tho,


        Stefan

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Richard Stallman]

    > Maybe there should be a margin property (distinct from the current
    > margin display spec) which specifies material to go in the margin
    > (like the way the overlay before-string property works).

    IIRC Gerd decided to use a single property `display' rather than several
    properties, for reasons of performance.  I do not know or remember what was
    the particular performance issue(s), tho,

That is true, but we COULD have another kind of `margin' display spec
which works like `before-string'.  It would be more logical.  So I
wonder why it was not done that way.  Maybe there was a reason.

Re: Emacs Lisp manual issues regarding overlays, display properties, margins, etc.

[Richard Stallman]

    > That seems like a bug to me.  It seems to me that a property that puts
    > something in the margin should not alter the appearance of the text
    > it is on.
    >
    > Does anyone think the current behavior is desirable?

    In any case, it follows the documentation

Two parts of the Lisp manual seem to conflict with each other.  It
would be straightforward to fix the one that doesn't agree with the
code.  I will do that.  But is there a good reason for the current
behavior?

    Maybe there should be a margin property (distinct from the current
    margin display spec) which specifies material to go in the margin
    (like the way the overlay before-string property works).

We don't want to make another property, because each kind of property
that display needs to check for slows it down.

But we could make another kind of margin display specification
that combines the effect of before-string and margin.
That would do what users would naturally expect.