bug#28311: 25.2; Improve syntax highlighting in bat-mode

bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 5170 bytes --]

Hi!

I've noticed some problems with syntax highlighting in bat-mode. In
addition, I have two improvement suggestions.

Problem 1:

bat-mode highlights variables surrounded by "%":s, as in "%alpha%".
However, when the variable contains an underscore, as in "%alpha_beta%",
only "alpha" is highlighted.

Problem 2:

Similarly, bat-mode highlights variables defined using "set". As with
problem 1 above, it does not handle words with underscore. Concretely, in
"set alpha_beta=value", only "alpha" is highlighted.

Problem 3:

bat-mode highlights keywords like "in". Unfortunately, it also highlights
the word "in" when used as a file extension. For example "xxx alpha.in".

Improvement suggestion 1:

In bat scripts, the command "echo" can be used to print strings to the
console. Unlike in shell scripts, the argument to "echo" does not require
quotes. It would be nice if Emacs would highlight anything after "echo" as
a string (using font-lock-string-face).

Improvement suggestion 2:

In bat scripts, variable substitution is performed inside quoted strings,
for example:

    xxx "hello %alpha goodbye"

Unfortunately, emacs does not highlight this, which I hereby propose that
it should.


I had plans to fix all of the above myself. Unfortunately, my Emacs
todo-list is too long for my available time, so Richard S. suggested that I
report this here, in case someone else would have more time than I do.

Anyway, problem 1 and 2 can probably be fixed by adapting a regexp. Problem
3 should easily be fixed by using a custom font-lock syntax table, where
"." is seen as a word character, or possibly a symbol character.
Improvement suggestion 1 could probably be handled by a simple rule
matching echo + space + whatever and highlight "whatever" using
font-lock-string-face. All of these are easy to fix.

Improvement suggestion 2 raises the bar to "medium", as it would require
replacing the current rexexp rule with one written in elisp. The rule will
need to use the "prepend" (not "t") override flag to add
font-lock-variable-name-face to the variable name in top of the string
face. However, the code must ensure that it does not do this for comments.

Side note: Typically, the "prepend" override flag is preferred over "t", as
the end result will mix face properties from both the new and old faces.
Concretely, I use a string face with a red foreground and a grey
background. With the "prepend" flag, the variable name will be drawn using
yellow from font-lock-variable-name-face but the background will still be
grey, matching the rest of the string.

    -- Anders Lindgren



In GNU Emacs 25.2.1 (x86_64-apple-darwin13.4.0, NS appkit-1265.21 Version
10.9.5 (Build 13F1911))
 of 2017-04-21 built on builder10-9.porkrind.org
Windowing system distributor 'Apple', version 10.3.1504
Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES

Important settings:
  value of $LC_CTYPE: UTF-8
  value of $LANG: en_SE.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Bat

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
next-line: End of buffer

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode cl-loaddefs pcase
cl-lib mail-prsvr mail-utils vc-dispatcher vc-svn bat-mode easymenu
time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel ns-win ucs-normalize term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote kqueue cocoa ns
multi-tty make-network-process emacs)

Memory information:
((conses 16 196862 6811)
 (symbols 48 19646 0)
 (miscs 40 53 163)
 (strings 32 15415 5200)
 (string-bytes 1 442636)
 (vectors 16 33051)
 (vector-slots 8 651956 5657)
 (floats 8 162 94)
 (intervals 56 216 0)
 (buffers 976 19))

[-- Attachment #2: Type: text/html, Size: 6842 bytes --]

bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Achim Gratz]

Anders Lindgren writes:
> I've noticed some problems with syntax highlighting in bat-mode.

Consider joining this report with bug#18405.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html

bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Thu, 31 Aug 2017 21:27:43 +0200
> 
> Anders Lindgren writes:
> > I've noticed some problems with syntax highlighting in bat-mode.
> 
> Consider joining this report with bug#18405.

I've picked the lower-hanging fruit and fixed some of the problems
reported in these two bug reports on the emacs-26 branch.

Thanks.

bug#18405: bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Glenn Morris]

Running 5 tests (2017-09-22 09:57:54-0700)
   passed  1/5  bat-test-fill-paragraph-comment
hfy batch mode ( *temp*-973592:bat-mode)
Test bat-test-fontification-iter-var-1 backtrace:
  signal(ert-test-failed (((should (equal (bat-test-fontify "echo %%a\
  ert-fail(((should (equal (bat-test-fontify "echo %%a\necho %%~dp1\ne
  (if (unwind-protect (setq value-17 (apply fn-15 args-16)) (setq form
  (let (form-description-19) (if (unwind-protect (setq value-17 (apply
  (let ((value-17 'ert-form-evaluation-aborted-18)) (let (form-descrip
  (let* ((fn-15 (function equal)) (args-16 (condition-case err (let ((
  (closure (t) nil (let* ((fn-15 (function equal)) (args-16 (condition
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name bat-test-fontification-iter-var-1 :do
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests [#s(ert-test 
  ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
  ert-run-tests-batch(nil)
  ert-run-tests-batch-and-exit(nil)
  eval((ert-run-tests-batch-and-exit nil))
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/progmodes/bat-mode-t
  command-line()
  normal-top-level()
Test bat-test-fontification-iter-var-1 condition:
    (ert-test-failed
     ((should
       (equal
	(bat-test-fontify "echo %%a
echo %%~dp1
echo %%~$PATH:I")
	"<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">a</span>
<span class=\"builtin\">echo</span> %%~dp<span class=\"variable-name\">1</span>
<span class=\"builtin\">echo</span> %%~$<span class=\"variable-name\">PATH</span>:<span class=\"variable-name\">I</span>"))
      :form
      (equal
       #("<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">a</span>
<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~dp1</span>
<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~$PATH:I</span>" 22 26
(face font-lock-builtin-face)
64 65
(face font-lock-variable-name-face)
95 99
(face font-lock-builtin-face)
137 141
(face font-lock-variable-name-face)
171 175
(face font-lock-builtin-face)
213 221
(face font-lock-variable-name-face))
       "<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">a</span>
<span class=\"builtin\">echo</span> %%~dp<span class=\"variable-name\">1</span>
<span class=\"builtin\">echo</span> %%~$<span class=\"variable-name\">PATH</span>:<span class=\"variable-name\">I</span>")
      :value nil :explanation
      (arrays-of-different-length 228 263
				  #("<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">a</span>
<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~dp1</span>
<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~$PATH:I</span>" 22 26
(face font-lock-builtin-face)
64 65
(face font-lock-variable-name-face)
95 99
(face font-lock-builtin-face)
137 141
(face font-lock-variable-name-face)
171 175
(face font-lock-builtin-face)
213 221
(face font-lock-variable-name-face))
				  "<span class=\"builtin\">echo</span> %%<span class=\"variable-name\">a</span>
<span class=\"builtin\">echo</span> %%~dp<span class=\"variable-name\">1</span>
<span class=\"builtin\">echo</span> %%~$<span class=\"variable-name\">PATH</span>:<span class=\"variable-name\">I</span>" first-mismatch-at 109)))
   FAILED  2/5  bat-test-fontification-iter-var-1
hfy batch mode ( *temp*-454955:bat-mode)
   passed  3/5  bat-test-fontification-var-decl
hfy batch mode ( *temp*-493544:bat-mode)
   passed  4/5  bat-test-fontification-var-delayed-exp
hfy batch mode ( *temp*-832943:bat-mode)
   passed  5/5  bat-test-fontification-var-exp

Ran 5 tests, 4 results as expected, 1 unexpected (2017-09-22 09:57:54-0700)

1 unexpected results:
   FAILED  bat-test-fontification-iter-var-1

bug#18405: bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Achim Gratz <Stromeko@nexgo.de>,  Anders Lindgren <andlind@gmail.com>,  28311@debbugs.gnu.org,  18405@debbugs.gnu.org
> Date: Fri, 22 Sep 2017 13:01:03 -0400
> 
> 
> Running 5 tests (2017-09-22 09:57:54-0700)
>    passed  1/5  bat-test-fill-paragraph-comment
> hfy batch mode ( *temp*-973592:bat-mode)
> Test bat-test-fontification-iter-var-1 backtrace:

Thanks, fixed.

bug#18405: bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 5708 bytes --]

Hi!

Unfortunately, I can't comment on the actual error found, as I wasn't
involved in the fix...

Anyway, I am positively glad that you have tests for font-lock rules.
However, using htmlfontify for this doesn't seem like the best tool for
this. 1) The output of htmlfontify may change in the future, in which case
you will need to rewrite all your tests, 2) HTML is hard to read, and 3)
The ert explainer doesn't really help you here. If example, in this case
all it tells you is that the resulting html strings have different lengths.
On other words, it may work for snippets like this, but it would not scale
well.

A couple of years ago I wrote a tool, faceup (https://github.com/
Lindydancer/faceup), specifically for testing highlighting in Emacs using
ert. It uses a simple markup language, so the result is easier to read than
when using HTML. The reference text can be used inline (like in your test)
or stored in a corresponding ".faceup" file, which makes it possible to
test files thousands of lines long. When a test fails, the ert explainer
will tell you which lines contained errors, what that lines contained, and
what they should contain, so it's much easier to find errors. I have
successfully used this is several of my font-lock packages. In addition,
I've accumulated a number of source files in various programming languages
in a regression test suite for font-lock (https://github.com/
Lindydancer/font-lock-regression-suite).

If you would like to modernise your font-lock test, I would be happy to
contribute it to Emacs (as I suggested in bug#16063).

    -- Anders

On Fri, Sep 22, 2017 at 7:01 PM, Glenn Morris <rgm@gnu.org> wrote:

>
> Running 5 tests (2017-09-22 09:57:54-0700)
>    passed  1/5  bat-test-fill-paragraph-comment
> hfy batch mode ( *temp*-973592:bat-mode)
> Test bat-test-fontification-iter-var-1 backtrace:
>   signal(ert-test-failed (((should (equal (bat-test-fontify "echo %%a\
>   ert-fail(((should (equal (bat-test-fontify "echo %%a\necho %%~dp1\ne
>   (if (unwind-protect (setq value-17 (apply fn-15 args-16)) (setq form
>   (let (form-description-19) (if (unwind-protect (setq value-17 (apply
>   (let ((value-17 'ert-form-evaluation-aborted-18)) (let (form-descrip
>   (let* ((fn-15 (function equal)) (args-16 (condition-case err (let ((
>   (closure (t) nil (let* ((fn-15 (function equal)) (args-16 (condition
>   ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
>   ert-run-test(#s(ert-test :name bat-test-fontification-iter-var-1 :do
>   ert-run-or-rerun-test(#s(ert--stats :selector t :tests [#s(ert-test
>   ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
>   ert-run-tests-batch(nil)
>   ert-run-tests-batch-and-exit(nil)
>   eval((ert-run-tests-batch-and-exit nil))
>   command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/progmodes/bat-mode-t
>   command-line()
>   normal-top-level()
> Test bat-test-fontification-iter-var-1 condition:
>     (ert-test-failed
>      ((should
>        (equal
>         (bat-test-fontify "echo %%a
> echo %%~dp1
> echo %%~$PATH:I")
>         "<span class=\"builtin\">echo</span> %%<span
> class=\"variable-name\">a</span>
> <span class=\"builtin\">echo</span> %%~dp<span class=\"variable-name\">1</
> span>
> <span class=\"builtin\">echo</span> %%~$<span
> class=\"variable-name\">PATH</span>:<span class=\"variable-name\">I</
> span>"))
>       :form
>       (equal
>        #("<span class=\"builtin\">echo</span> %%<span
> class=\"variable-name\">a</span>
> <span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~dp1</
> span>
> <span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~$PATH:I</span>"
> 22 26
> (face font-lock-builtin-face)
> 64 65
> (face font-lock-variable-name-face)
> 95 99
> (face font-lock-builtin-face)
> 137 141
> (face font-lock-variable-name-face)
> 171 175
> (face font-lock-builtin-face)
> 213 221
> (face font-lock-variable-name-face))
>        "<span class=\"builtin\">echo</span> %%<span
> class=\"variable-name\">a</span>
> <span class=\"builtin\">echo</span> %%~dp<span class=\"variable-name\">1</
> span>
> <span class=\"builtin\">echo</span> %%~$<span
> class=\"variable-name\">PATH</span>:<span class=\"variable-name\">I</
> span>")
>       :value nil :explanation
>       (arrays-of-different-length 228 263
>                                   #("<span class=\"builtin\">echo</span>
> %%<span class=\"variable-name\">a</span>
> <span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~dp1</
> span>
> <span class=\"builtin\">echo</span> %%<span class=\"variable-name\">~$PATH:I</span>"
> 22 26
> (face font-lock-builtin-face)
> 64 65
> (face font-lock-variable-name-face)
> 95 99
> (face font-lock-builtin-face)
> 137 141
> (face font-lock-variable-name-face)
> 171 175
> (face font-lock-builtin-face)
> 213 221
> (face font-lock-variable-name-face))
>                                   "<span class=\"builtin\">echo</span>
> %%<span class=\"variable-name\">a</span>
> <span class=\"builtin\">echo</span> %%~dp<span class=\"variable-name\">1</
> span>
> <span class=\"builtin\">echo</span> %%~$<span
> class=\"variable-name\">PATH</span>:<span class=\"variable-name\">I</span>"
> first-mismatch-at 109)))
>    FAILED  2/5  bat-test-fontification-iter-var-1
> hfy batch mode ( *temp*-454955:bat-mode)
>    passed  3/5  bat-test-fontification-var-decl
> hfy batch mode ( *temp*-493544:bat-mode)
>    passed  4/5  bat-test-fontification-var-delayed-exp
> hfy batch mode ( *temp*-832943:bat-mode)
>    passed  5/5  bat-test-fontification-var-exp
>
> Ran 5 tests, 4 results as expected, 1 unexpected (2017-09-22 09:57:54-0700)
>
> 1 unexpected results:
>    FAILED  bat-test-fontification-iter-var-1
>
>

[-- Attachment #2: Type: text/html, Size: 7652 bytes --]

bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Eli Zaretskii]

> From: Anders Lindgren <andlind@gmail.com>
> Date: Fri, 22 Sep 2017 21:04:19 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, Achim Gratz <Stromeko@nexgo.de>, 28311@debbugs.gnu.org, 
> 	18405@debbugs.gnu.org
> 
> If you would like to modernise your font-lock test, I would be happy to contribute it to Emacs (as I suggested in
> bug#16063).

Why wouldn't we want that?  Please go ahead, and thanks.

bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 352 bytes --]

>
> > If you would like to modernise your font-lock test, I would be happy to
> contribute it to Emacs (as I suggested in
> > bug#16063).
>
> Why wouldn't we want that?  Please go ahead, and thanks.
>

Thanks! In which branch should I add it, master or Emacs-26?

I guess it should go into lisp/emacs-lisp, as "ert.el" is located there.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 800 bytes --]

bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Eli Zaretskii]

> From: Anders Lindgren <andlind@gmail.com>
> Date: Sat, 23 Sep 2017 14:50:27 +0200
> Cc: Glenn Morris <rgm@gnu.org>, Stromeko@nexgo.de, 28311@debbugs.gnu.org, 
> 	18405@debbugs.gnu.org
> 
>  Why wouldn't we want that? Please go ahead, and thanks.
> 
> Thanks! In which branch should I add it, master or Emacs-26?

How large is it?  Also, is it supposed to be used outside of the ert
context?

> I guess it should go into lisp/emacs-lisp, as "ert.el" is located there.

Probably.  Actually, why not make it part of ert, say, ert-faces.el?

bug#18405: bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

>
> > Thanks! In which branch should I add it, master or Emacs-26?
>
> How large is it?  Also, is it supposed to be used outside of the ert
> context?
>

The main source file is about 45 KB, with additional 15KB of test files.

It can be used outside the ERT context, as the markup language is generic.
I can think of many applications for it, like use it as the storage format
for a word-processor like mode, where the user can apply bold, italics etc.


> I guess it should go into lisp/emacs-lisp, as "ert.el" is located there.
>
> Probably.  Actually, why not make it part of ert, say, ert-faces.el?
>

As it can be used outside of ert, I would prefer the current name.

     -- Anders

[-- Attachment #2: Type: text/html, Size: 1260 bytes --]

bug#18405: bug#28311: 25.2; Improve syntax highlighting in bat-mode

[Eli Zaretskii]

> From: Anders Lindgren <andlind@gmail.com>
> Date: Sat, 23 Sep 2017 20:05:35 +0200
> Cc: Glenn Morris <rgm@gnu.org>, Achim Gratz <Stromeko@nexgo.de>, 28311@debbugs.gnu.org, 
> 	18405@debbugs.gnu.org
> 
>  > Thanks! In which branch should I add it, master or Emacs-26?
> 
>  How large is it? Also, is it supposed to be used outside of the ert
>  context?
> 
> The main source file is about 45 KB, with additional 15KB of test files.
> 
> It can be used outside the ERT context, as the markup language is generic. I can think of many applications
> for it, like use it as the storage format for a word-processor like mode, where the user can apply bold, italics
> etc.

In that case, I guess master is the right place.

Thanks.

bug#18405: 24.3.93; bat-mode.el

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> The main source file is about 45 KB, with additional 15KB of test files.
>> 
>> It can be used outside the ERT context, as the markup language is generic. I can think of many applications
>> for it, like use it as the storage format for a word-processor like mode, where the user can apply bold, italics
>> etc.
>
> In that case, I guess master is the right place.

This was added to Emacs a year later, as a0e5a02125a6.

A quick skim of this bug report seems to indicate that most of the
issues described here were fixed, so I'm closing this bug report.  If
there's anything more that should be done in bat-mode, then filing new
bug reports for any remaining issues would be the way forward.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no