bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 4139 bytes --]

This bug is a regression from the emacs-25 branch build.

Here is how to reproduce this bug in the master build in an emacs -Q
session (and also to verify that this bug is not present in emacs-25 branch
build):

1. Launch emacs -Q
2. Type "(setq a " in the *scratch* buffer

In *master* build, you see this: http://i.imgur.com/YwBybGU.png
Note that the SYM argument is not highlighted (made bold).

Whereas this is what I see on the *emacs-25* branch after the same steps
above: http://i.imgur.com/DveB1le.png
Note that this time, the correct argument is seen in bold face in the eldoc
hint.



In GNU Emacs 25.1.50.3 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.23)
 of 2016-07-05 built on
Repository revision: a18baf565e872759de9281c3488c3981d19a15e1
Windowing system distributor 'The X.Org Foundation', version 11.0.60900000
System Description: Red Hat Enterprise Linux Workstation release 6.6
(Santiago)

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --with-modules
 --prefix=/home/kmodi/usr_local/apps/6/emacs/master
 'CPPFLAGS=-fgnu89-inline -I/home/kmodi/usr_local/6/include
 -I/usr/include/freetype2 -I/usr/include' 'CFLAGS=-ggdb3 -O0'
 'CXXFLAGS=-ggdb3 -O0' 'LDFLAGS=-L/home/kmodi/usr_local/6/lib
 -L/home/kmodi/usr_local/6/lib64 -ggdb3'
 PKG_CONFIG_PATH=/home/kmodi/usr_local/6/lib/pkgconfig:/home/kmodi/usr_local/6/lib64/pkgconfig:/cad/adi/apps/gnu/linux/x86_64/6/lib/pkgconfig:/cad/adi/apps/gnu/linux/x86_64/6/lib64/pkgconfig:/usr/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig:/lib/pkgconfig:/lib64/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK2 X11 MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message puny seq byte-opt gv bytecomp
byte-compile cl-extra help-mode cconv cl-loaddefs pcase cl-lib dired
dired-loaddefs format-spec rfc822 mml easymenu mml-sec password-cache
epa derived epg epg-config gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev
obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 98021 6511)
 (symbols 48 21150 0)
 (miscs 40 43 83)
 (strings 32 17964 4480)
 (string-bytes 1 590672)
 (vectors 16 14230)
 (vector-slots 8 465616 5414)
 (floats 8 187 88)
 (intervals 56 238 0)
 (buffers 976 11)
 (heap 1024 22229 700))

-- 

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 5459 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[martin rudalics]

 > Here is how to reproduce this bug in the master build in an emacs -Q
 > session (and also to verify that this bug is not present in emacs-25 branch
 > build):
 >
 > 1. Launch emacs -Q
 > 2. Type "(setq a " in the *scratch* buffer
 >
 > In *master* build, you see this: http://i.imgur.com/YwBybGU.png
 > Note that the SYM argument is not highlighted (made bold).
 >
 > Whereas this is what I see on the *emacs-25* branch after the same steps
 > above: http://i.imgur.com/DveB1le.png
 > Note that this time, the correct argument is seen in bold face in the eldoc
 > hint.

Confirmed.  Something seems to suppress highlighting in the minibuffer -
displaying eldoc in tooltips works as before.  Could you bisect to find
out what's been causing this?

martin

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 849 bytes --]

On 2016-07-07 12:18, martin rudalics wrote:
>> Here is how to reproduce this bug in the master build in an emacs -Q
>> session (and also to verify that this bug is not present in emacs-25 branch
>> build):
>>
>> 1. Launch emacs -Q
>> 2. Type "(setq a " in the *scratch* buffer
>>
>> In *master* build, you see this: http://i.imgur.com/YwBybGU.png
>> Note that the SYM argument is not highlighted (made bold).
>>
>> Whereas this is what I see on the *emacs-25* branch after the same steps
>> above: http://i.imgur.com/DveB1le.png
>> Note that this time, the correct argument is seen in bold face in the eldoc
>> hint.
> 
> Confirmed.  Something seems to suppress highlighting in the minibuffer -
> displaying eldoc in tooltips works as before.  Could you bisect to find
> out what's been causing this?

Confirmed here as well.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

Thanks for confirming.

Dumb question. Is the process of bisecting simply building emacs at
half-way commits between passing and failing builds till you narrow down to
the breaking commit?

My each build takes 5-10 minutes. So that would take a long time to narrow
down to the culprit.

On Thu, Jul 7, 2016 at 12:18 PM martin rudalics <rudalics@gmx.at> wrote:

> Confirmed.  Something seems to suppress highlighting in the minibuffer -
> displaying eldoc in tooltips works as before.  Could you bisect to find
> out what's been causing this?
>
> --

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 964 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Dmitry Gutov]

On 07/07/2016 07:57 PM, Kaushal Modi wrote:

> Dumb question. Is the process of bisecting simply building emacs at
> half-way commits between passing and failing builds till you narrow down
> to the breaking commit?

Yes. 'git bisect' makes it relatively straightforward, but it can take a 
lot of time (like an hour, maybe a few), depending on how many commits 
one has to examine.

You can switch to other work while it builds, though.

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 911 bytes --]

Phew, finally.

After git bisecting, I have narrowed down this bug to this commit:
http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0644e6f56d2be82dd716478eb65e7b3c761d813d

So I ended git bisecting at this point:

km²~/downloads/:git/emacs> git stash save && git bisect good

Saved working directory and index state WIP on (no branch): 1f55925 ; Fix
breakage from previous commit
HEAD is now at 1f55925 ; Fix breakage from previous commit
Bisecting: 0 revisions left to test after this (roughly 1 step)
[0644e6f56d2be82dd716478eb65e7b3c761d813d] Fix copying properties in
'format' when it produces padding

Building emacs using that commit shows that bug.

I don't understand what the C code does. But the comments make somewhat
sense as that code change has to do with the format function, and thus
related to the incorrect text properties of eldoc hints.
-- 

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1424 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

Actually I had to run git bisect one last time (I just became impatient).
But the conclusion in my previous email was right.

git bisect ended with

Saved working directory and index state WIP on (no branch): cfb3c61 Enable
dividers in NS (bug#22973)
HEAD is now at cfb3c61 Enable dividers in NS (bug#22973)
0644e6f56d2be82dd716478eb65e7b3c761d813d is the first bad commit
commit 0644e6f56d2be82dd716478eb65e7b3c761d813d
Author: Eli Zaretskii <eliz@gnu.org>
Date:   Tue Jun 28 19:03:43 2016 +0300

    Fix copying properties in 'format' when it produces padding

    * src/textprop.c (extend_property_ranges): Correct range extension
    when the new end is beyond the old end.  (Bug#23859)

:040000 040000 1e0b14a6241e1817a334ef67fdfef7555c2c8a4b
eb7f473f651a351ecbb6edc4bb8c0ab3473edb5b M      src

On Thu, Jul 7, 2016 at 3:31 PM Kaushal Modi <kaushal.modi@gmail.com> wrote:

> Phew, finally.
>
> After git bisecting, I have narrowed down this bug to this commit:
> http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0644e6f56d2be82dd716478eb65e7b3c761d813d
>
> So I ended git bisecting at this point:
>
> km²~/downloads/:git/emacs> git stash save && git bisect good
>
> Saved working directory and index state WIP on (no branch): 1f55925 ; Fix
> breakage from previous commit
> HEAD is now at 1f55925 ; Fix breakage from previous commit
> Bisecting: 0 revisions left to test after this (roughly 1 step)
> [0644e6f56d2be82dd716478eb65e7b3c761d813d] Fix copying properties in
> 'format' when it produces padding
>
> Building emacs using that commit shows that bug.
>
> I don't understand what the C code does. But the comments make somewhat
> sense as that code change has to do with the format function, and thus
> related to the incorrect text properties of eldoc hints.
> --
>
> --
> Kaushal Modi
>
-- 

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2878 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Thu, 07 Jul 2016 19:31:26 +0000
> Cc: martin rudalics <rudalics@gmx.at>, Dmitry Gutov <dgutov@yandex.ru>
> 
> [0644e6f56d2be82dd716478eb65e7b3c761d813d] Fix copying properties in 'format' when it produces padding
> 
> Building emacs using that commit shows that bug.

Then the fix will have to be in the code which calls format, because
the above commit is going to stay.

If no one beats me to it, I will look into this in a day or two.

Thanks for the analysis.

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

Looks like the fixes will be needed in major modes?

For instance, by adding the following debug statement in
elisp--highlight-function-argument function in elisp-mode.el,

=====
diff --git a/lisp/progmodes/elisp-mode.el b/lisp/progmodes/elisp-mode.el
index f360791..16365dd 100644
--- a/lisp/progmodes/elisp-mode.el
+++ b/lisp/progmodes/elisp-mode.el
@@ -1481,6 +1481,7 @@ elisp--highlight-function-argument
  (setq doc (copy-sequence args))
  (add-text-properties start end (list 'face argument-face) doc))
       (setq doc (eldoc-docstring-format-sym-doc prefix doc))
+      (message "debug: doc = %S" doc)
       doc)))

 ;; Return a string containing a brief (one-line) documentation string for
=====

I get the below when the cursor is after a defun:

debug: doc = #("defun: (NAME ARGLIST &optional DOCSTRING DECL &rest BODY)"
0 5 (face font-lock-keyword-face))

I get the same debug output in both emacs-25 and master builds. So I am
wondering if this doc output needs to be adjusted to the change in the
format function then ..

Also, I can see if debug of incorrect face display in both mode-line (when
I am using the minibuffer to eval stuff using M-: binding) and echo area.

On Thu, Jul 7, 2016 at 3:45 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Kaushal Modi <kaushal.modi@gmail.com>
> > Date: Thu, 07 Jul 2016 19:31:26 +0000
> > Cc: martin rudalics <rudalics@gmx.at>, Dmitry Gutov <dgutov@yandex.ru>
> >
> > [0644e6f56d2be82dd716478eb65e7b3c761d813d] Fix copying properties in
> 'format' when it produces padding
> >
> > Building emacs using that commit shows that bug.
>
> Then the fix will have to be in the code which calls format, because
> the above commit is going to stay.
>
> If no one beats me to it, I will look into this in a day or two.
>
> Thanks for the analysis.
>
-- 

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2865 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

Here's some more debug ..

If you eval the below (completely override eldoc-minibuffer-message for
debug purpose):

(defun eldoc-minibuffer-message (format-string &rest args)
  "Display messages in the mode-line when in the minibuffer.
Otherwise work like `message'."
  (setq args '(#("0123456789" 0 5 (face font-lock-keyword-face))))
  (let ((message-log-max 1000))
    (message "debug: format-string = %S" format-string)
    (message "debug: args = %S" args))
  (unless (minibufferp)
    (apply 'message format-string args)))

Then while in emacs-lisp-mode, with cursor after any keyword like defun or
setq or anything will show "0123456789" in the echo area. BUT the whole
face will be font-lock-keyword-face instead of just the first five chars.
Image: http://i.imgur.com/1tsp1rt.png

Here's the outcome of the same eldoc-minibuffer-message hack in emacs-25
build
http://i.imgur.com/1TfRxpx.png

This time, the first 5 chars only have the font-lock-keyword-face as
expected.


Now back to the master build, when I evaluate the below:

(let ((format-string "%s")
      (args '(#("0123456789" 0 5 (face font-lock-keyword-face)))))
  (apply #'message format-string args))

I get:

#("0123456789" 0 10 (face font-lock-keyword-face))

!! Notice that the end point 5 got updated to 10 by itself !!

On emacs-25 build, evalling the same keeps the end point at 5.

====

An unrelated thing, elisp related, that I don't understand is that

- The face-formatted string shown in the minibuffer when "(apply 'message
format-string args)" is called in the eldoc-minibuffer-message function.
- But "#("0123456789" 0 10 (face font-lock-keyword-face))" (in master
build) is shown verbatim without any face-formatting in the minibuffer when
the above let form (which has the exact same apply form with the exact same
message arguments) is evaluated.

Why is that?

Nonetheless this difference in behavior when evaluating let form is what
made me realize that the message function returns the end pointer updated
from 5 to 10 in the master build.


-- 

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 3312 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Thu, 07 Jul 2016 22:19:19 +0000
> Cc: 23897@debbugs.gnu.org, rudalics@gmx.at, dgutov@yandex.ru
> 
> Here's some more debug ..

Thanks.

For the future, please always try to find the simplest way to
demonstrate a problem, and preferably as close to the lowest-level
function that still exhibits it.  In particular, various fancy
constructs using 'apply', 'funcall', let alone cl-lib stuff are a
distraction that should only be present if the problem doesn't show
without them.

Specifically, in this case, the simplest way to demonstrate the
problem is this:

  (format "%s" (concat (propertize "01234" 'face 'bold) "56789"))
    => #("0123456789" 0 10 (face bold))

Clearly, the "0 10" part is not what is expected.

This has the advantage of showing that the problem is inside the
'format' primitive (you used 'message', but that just calls 'format'
and then displays the result, as you can see from its source).  Also,
propertize is a much simpler way of constructing a string with
properties than using the #("0123456789" 0 5 (face FACE)) read syntax.

> An unrelated thing, elisp related, that I don't understand is that 
> 
> - The face-formatted string shown in the minibuffer when "(apply 'message format-string args)" is called in the
> eldoc-minibuffer-message function.
> - But "#("0123456789" 0 10 (face font-lock-keyword-face))" (in master build) is shown verbatim without any
> face-formatting in the minibuffer when the above let form (which has the exact same apply form with the
> exact same message arguments) is evaluated. 
> Why is that?

The "#("0123456789" 0 10 (face font-lock-keyword-face))" stuff doesn't
have the face applied, that's why.  The face will be applied when it
is read by the Lisp reader.

(Avoiding these tricky constructs as much as possible is one way of
not getting distracted by unrelated issues.)

Thanks.

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Eli Zaretskii]

> Date: Fri, 08 Jul 2016 12:25:19 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 23897@debbugs.gnu.org, dgutov@yandex.ru
> 
>   (format "%s" (concat (propertize "01234" 'face 'bold) "56789"))
>     => #("0123456789" 0 10 (face bold))
> 
> Clearly, the "0 10" part is not what is expected.

I think I fixed this now, please test (commit d8a9c45 on master).

(The problem was that the bugfix in 0644e6f uncovered another bug,
which was hidden by the one that commit fixed.)

A problem in Isearch was also mentioned by Mark Oteiza in the context
of this bug, but I see no details for it.  Can someone please tell
what was the problem, so I could verify it no longer exists after the
fix?

Thanks.

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

The isearch problem was the same.. the face in the beginning of the string
got applied to the whole string.

So after C-s,

Instead of "Isearch: " showing in a different face than that of the use
input in the minibuffer, "Isearch: " prompt + user input search string
showed up in the same face.

On Fri, Jul 8, 2016, 3:40 PM Eli Zaretskii <eliz@gnu.org> wrote:

> >
> A problem in Isearch was also mentioned by Mark Oteiza in the context
> of this bug, but I see no details for it.  Can someone please tell
> what was the problem, so I could verify it no longer exists after the
> fix?
>
> Thanks.
>
-- 

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1027 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Fri, 08 Jul 2016 19:42:44 +0000
> Cc: 23897@debbugs.gnu.org, dgutov@yandex.ru
> 
> Instead of "Isearch: " showing in a different face than that of the use input in the minibuffer, "Isearch: " prompt
> + user input search string showed up in the same face. 

Ah, okay.  Then this is also fixed.

Thanks.

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

Thanks Eli! I confirm the fix in eldoc hints and isearch prompt.
Closing this bug.

On Fri, Jul 8, 2016 at 3:50 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Instead of "Isearch: " showing in a different face than that of the use
> input in the minibuffer, "Isearch: " prompt
> > + user input search string showed up in the same face.
>
> Ah, okay.  Then this is also fixed.
>
> --

-- 
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 775 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 559 bytes --]

Yup, fix confirmed here as well. Thanks Eli!

On 2016-07-08 22:01, Kaushal Modi wrote:
> Thanks Eli! I confirm the fix in eldoc hints and isearch prompt.
> Closing this bug.
> 
> On Fri, Jul 8, 2016 at 3:50 PM Eli Zaretskii <eliz@gnu.org <mailto:eliz@gnu.org>> wrote:
> 
>     > Instead of "Isearch: " showing in a different face than that of the use input in the minibuffer, "Isearch: " prompt
>     > + user input search string showed up in the same face.
> 
>     Ah, okay.  Then this is also fixed.
> 
> -- 
> 
> -- 
> Kaushal Modi
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

bug#23897: 25.1.50; Argument at point not being highlighted in eldoc hints

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 611 bytes --]

On 2016-07-07 13:06, Dmitry Gutov wrote:
> On 07/07/2016 07:57 PM, Kaushal Modi wrote:
> 
>> Dumb question. Is the process of bisecting simply building emacs at
>> half-way commits between passing and failing builds till you narrow down
>> to the breaking commit?
> 
> Yes. 'git bisect' makes it relatively straightforward, but it can take a lot of time (like an hour, maybe a few), depending on how many commits one has to examine.
> 
> You can switch to other work while it builds, though.

If you find a way to automate the test, you can also tell git to find the offending code automatically.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]