bug#18405: 24.3.93; bat-mode.el

bug#18405: 24.3.93; bat-mode.el

[Achim Gratz]

In GNU Emacs 24.3.93.1 (i686-pc-linux-gnu, GTK+ Version 3.12.2)
 of 2014-08-24 on Rainer.invalid
Windowing system distributor `The X.Org Foundation', version 11.0.11403901
System Description:	openSUSE 13.1 (Bottle) (i586)

The fontlock pattern for environment variables ins bat-mode.el is

        ("%\\(\\w+\\)%?"
         (1 font-lock-variable-name-face))
        ("!\\(\\w+\\)!?"                ; delayed-expansion !variable!
         (1 font-lock-variable-name-face))

For CMD.EXE at least from Windows 7 onward (I've also tested with the
cmd.exe that comes with Wine), the name of an environment variable can
actually be any character including shell special characters (which
you'd then have to quote separately).  While it's possible to define a
variable that has a "%" character in its name, it can not be used it in
cmd itself since it never gets expanded, so the easiest replacement
pattern would be

        ("%\\([^%]+\\)%?"
         (1 font-lock-variable-name-face))
        ("!\\([^!%]+\\)!?"                ; delayed-expansion !variable!
         (1 font-lock-variable-name-face))

I've noticed this discrepancy when using variables with underscores in
their name, which only fontified the part up to the first underscore
correctly.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

bug#18405: 24.3.93; bat-mode.el

[Stefan Monnier]

>         ("%\\([^%]+\\)%?"
>          (1 font-lock-variable-name-face))
>         ("!\\([^!%]+\\)!?"                ; delayed-expansion !variable!
>          (1 font-lock-variable-name-face))

I know nothing of cmd.exe's syntax and the general thrust of the change
sounds good, but: the above matches "%foo\nbar", which is probably not
right.  When can variable refs be of the form %name or !name
(i.e. without the terminating % or !)?


        Stefan

bug#18405: 24.3.93; bat-mode.el

[Achim Gratz]

Stefan Monnier writes:
> I know nothing of cmd.exe's syntax and the general thrust of the change
> sounds good, but: the above matches "%foo\nbar", which is probably not
> right.  When can variable refs be of the form %name or !name
> (i.e. without the terminating % or !)?

I haven't tried embedded newlines (or rather \r\n as it were) in
variable names.  Just like the embedded '%' character they probably are
possible somehow, but mostly useless.  So it would make sense to exclude
them (embedded whitespace is also somewhat iffy, but it tends to work
surprisingly well within CMD itself).  The syntax for variable names in
CMD does not seem to be specified formally, at least I haven't found
anything.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

bug#18405: 24.3.93; bat-mode.el

[Eli Zaretskii]

> Date: Thu, 04 Sep 2014 16:39:25 -0400
> Cc: 18405@debbugs.gnu.org
> 
> When can variable refs be of the form %name or !name (i.e. without
> the terminating % or !)?

In the FOR loops.  The simplest example is (words in caps are
keywords, but don't need to appear in caps):

   FOR %f in (*.c) DO something %f

bug#18405: 24.3.93; bat-mode.el

[Eli Zaretskii]

> Date: Fri, 05 Sep 2014 10:03:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: Stromeko@nexgo.de, 18405@debbugs.gnu.org
> 
> > When can variable refs be of the form %name or !name (i.e. without
> > the terminating % or !)?
> 
> In the FOR loops.

And, of course, in references to batch file arguments, like %1, %2,
etc.

bug#18405: 24.3.93; bat-mode.el

[Achim Gratz]

Stefan Monnier writes:
> When can variable refs be of the form %name or !name
> (i.e. without the terminating % or !)?

These are so-called parameter variables; more specifically loop
variables and arguments, these must be single alpha character or
numerical.  There are also local variables when you enable extended
syntax, but they localize the environment variables and thus keep their
syntax.

I think that's the best syntax description (and most concise) I've found
so far:

http://ss64.com/nt/syntax-variables.html
http://ss64.com/nt/syntax-args.html


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

bug#18405: 24.3.93; bat-mode.el

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On 09/05/2014 11:39 AM, Achim Gratz wrote:
> Stefan Monnier writes:
>> When can variable refs be of the form %name or !name
>> (i.e. without the terminating % or !)?
> 
> These are so-called parameter variables; more specifically loop
> variables and arguments, these must be single alpha character or
> numerical.  There are also local variables when you enable extended
> syntax, but they localize the environment variables and thus keep their
> syntax.
> 
> I think that's the best syntax description (and most concise) I've found
> so far:
> 

Mind just importing http://www.emacswiki.org/emacs/ntcmd.el into trunk?
It handles a bunch of nasty cmd.exe syntactic corner cases.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#18405: 24.3.93; bat-mode.el

[Eli Zaretskii]

> Date: Sat, 06 Sep 2014 20:24:46 -0700
> From: Daniel Colascione <dancol@dancol.org>
> 
> Mind just importing http://www.emacswiki.org/emacs/ntcmd.el into trunk?

Would you mind some comments about it?

bug#18405: 24.3.93; bat-mode.el

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On 09/07/2014 08:17 AM, Eli Zaretskii wrote:
>> Date: Sat, 06 Sep 2014 20:24:46 -0700
>> From: Daniel Colascione <dancol@dancol.org>
>>
>> Mind just importing http://www.emacswiki.org/emacs/ntcmd.el into trunk?
> 
> Would you mind some comments about it?

Sorry --- I should have been more specific. I wrote ntcmd a while ago
because I was frustrated with existing modes for working with batch
files. ntcmd uses a parser to deal with cmd.exe's very weird syntax;
it's important to do that because cmd's syntax is highly irregular and
using regular font-lock patterns to match it is doomed --- in
particular, matches involving loop variables are complicated. (See
ntcmd-forward-generic-cmd and ntcmd-forward-for-cmd for some interesting
special cases and fossilized cmd.exe bugs.)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#18405: 24.3.93; bat-mode.el

[Eli Zaretskii]

> Date: Sun, 07 Sep 2014 18:04:15 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: Stromeko@nexgo.de, 18405@debbugs.gnu.org
> 
> On 09/07/2014 08:17 AM, Eli Zaretskii wrote:
> >> Date: Sat, 06 Sep 2014 20:24:46 -0700
> >> From: Daniel Colascione <dancol@dancol.org>
> >>
> >> Mind just importing http://www.emacswiki.org/emacs/ntcmd.el into trunk?
> > 
> > Would you mind some comments about it?
> 
> Sorry --- I should have been more specific. I wrote ntcmd a while ago
> because I was frustrated with existing modes for working with batch
> files. ntcmd uses a parser to deal with cmd.exe's very weird syntax;
> it's important to do that because cmd's syntax is highly irregular and
> using regular font-lock patterns to match it is doomed --- in
> particular, matches involving loop variables are complicated. (See
> ntcmd-forward-generic-cmd and ntcmd-forward-for-cmd for some interesting
> special cases and fossilized cmd.exe bugs.)

I've read the code, so yes, I've seen that.

My point was that I saw a few of its aspects that might benefit from
improvements, and if we are going to add that package to Emacs, I'd
like to see those happen.

bug#18405: 24.3.93; bat-mode.el

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]



On 09/07/2014 07:42 PM, Eli Zaretskii wrote:
>> Date: Sun, 07 Sep 2014 18:04:15 -0700
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: Stromeko@nexgo.de, 18405@debbugs.gnu.org
>>
>> On 09/07/2014 08:17 AM, Eli Zaretskii wrote:
>>>> Date: Sat, 06 Sep 2014 20:24:46 -0700
>>>> From: Daniel Colascione <dancol@dancol.org>
>>>>
>>>> Mind just importing http://www.emacswiki.org/emacs/ntcmd.el into trunk?
>>>
>>> Would you mind some comments about it?
>>
>> Sorry --- I should have been more specific. I wrote ntcmd a while ago
>> because I was frustrated with existing modes for working with batch
>> files. ntcmd uses a parser to deal with cmd.exe's very weird syntax;
>> it's important to do that because cmd's syntax is highly irregular and
>> using regular font-lock patterns to match it is doomed --- in
>> particular, matches involving loop variables are complicated. (See
>> ntcmd-forward-generic-cmd and ntcmd-forward-for-cmd for some interesting
>> special cases and fossilized cmd.exe bugs.)
> 
> I've read the code, so yes, I've seen that.
> 
> My point was that I saw a few of its aspects that might benefit from
> improvements, and if we are going to add that package to Emacs, I'd
> like to see those happen.

Like what? In 2011, you just mentioned that we should be using or
enhancing bat-generic-mode, which, as I explained, is insufficient.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#18405: 24.3.93; bat-mode.el

[Achim Gratz]

Daniel Colascione writes:
> Mind just importing http://www.emacswiki.org/emacs/ntcmd.el into trunk?
> It handles a bunch of nasty cmd.exe syntactic corner cases.

I've briefly looked at it and this seems like a better approach in the
long run.  However, I'd still like to have that particular bug in
bat-mode fixed in the next release, while ntcmd.el looks like something
that would have to be done in trunk.  So I suggest that this should be
separate bug report or split it off from the original one if possible.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

bug#18405: 24.3.93; bat-mode.el

[Eli Zaretskii]

> Date: Sun, 07 Sep 2014 19:47:52 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: Stromeko@nexgo.de, 18405@debbugs.gnu.org
> 
> > My point was that I saw a few of its aspects that might benefit from
> > improvements, and if we are going to add that package to Emacs, I'd
> > like to see those happen.
> 
> Like what?

See below.

> In 2011, you just mentioned that we should be using or enhancing
> bat-generic-mode

My goodness, I hope you don't expect me to forever hold my peace about
this just because I happened to write something 3 years ago.

Here are my comments:

 . There are some strange decisions about faces, for example:

   . the switch letters in Unix-style -x switches are highlighted in
     font-lock-type-face
   . redirection characters are highlighted in font-lock-constant-face
   . opening parenthesis in "echo" commands are given
     font-lock-warning-face
   . variable definitions in for loops (the 'x' part in "for %%x") are
     highlighted with different face from their references in the "do"
     part of the loop

 . Text after "echo" (up to redirection character, if any) is always a
   simple string, except if it's "on" or "off", but is not highlighted
   as a string, which then causes some random highlighting of the
   parts of the string that just happen to match other possible
   syntaxes

 . Something like this confuses fontification of the redirection
   character, probably because of the parentheses (the opening paren
   is given the font-lock-warning-face):

      echo int main()           >junk.c

 . In general, I don't understand what was intended to be the handling
   of parentheses, they seem to be either not highlighted at all or
   given the warning face.

 . A command of this form:

     if exist somefile dosomethingwith somefile

   has "dosomethingwith" sometimes in the font-lock-builtin-face and
   sometimes with no face at all, not sure why.  Looks like when
   'somefile' includes slashes, the command is not fontified.

 . The value to which a variable is set in "set foo-bar" doesn't have
   any face, which causes some funny fontifications if that value
   happens to match unrelated syntaxes, as in "set foo=-bar=baz"

 . The equals sign in "if foo == bar" is sometimes given the
   font-lock-keyword-face and sometimes no face at all.  Here's one
   example of the latter:

    if not (%svgsupport%) == (Y) goto :svgDone

 . This line has its fontifications after "(%distfiles%)" screwed:

    for %%d in (%distfiles%) do if not exist %%d set fileNotFound=%%d

   In particular, built-in commands are not fontified as such.

 . Some built-in commands are missing from ntcmd-shell-builtins:

     assoc, call, mklink

 . "dir" and "rem" appear twice in the list in ntcmd-shell-builtins

 . The support of "set /a" command is incomplete, e.g. try

     set /a y+=140

Thanks.

bug#18405: 24.3.93; bat-mode.el

[Stefan Monnier]

>    . variable definitions in for loops (the 'x' part in "for %%x") are
>      highlighted with different face from their references in the "do"
>      part of the loop

FWIW in sh-mode, for something like "for f in echo $f", we also
highlight the two occurrences of "f" differently.  I'm not saying it's
a feature, but it's definitely a common behavior to highlight the
"introduction" occurrence differently from the "reference" occurrences.


        Stefan

bug#18405: 24.3.93; bat-mode.el

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Daniel Colascione <dancol@dancol.org>, Stromeko@nexgo.de,
>         18405@debbugs.gnu.org
> Date: Tue, 09 Sep 2014 09:45:50 -0400
> 
> >    . variable definitions in for loops (the 'x' part in "for %%x") are
> >      highlighted with different face from their references in the "do"
> >      part of the loop
> 
> FWIW in sh-mode, for something like "for f in echo $f", we also
> highlight the two occurrences of "f" differently.  I'm not saying it's
> a feature, but it's definitely a common behavior to highlight the
> "introduction" occurrence differently from the "reference" occurrences.

Except that in the ntcmd case, that's not the intended behavior,
AFAICT.

bug#18405: 24.3.93; bat-mode.el

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> The main source file is about 45 KB, with additional 15KB of test files.
>> 
>> It can be used outside the ERT context, as the markup language is generic. I can think of many applications
>> for it, like use it as the storage format for a word-processor like mode, where the user can apply bold, italics
>> etc.
>
> In that case, I guess master is the right place.

This was added to Emacs a year later, as a0e5a02125a6.

A quick skim of this bug report seems to indicate that most of the
issues described here were fixed, so I'm closing this bug report.  If
there's anything more that should be done in bat-mode, then filing new
bug reports for any remaining issues would be the way forward.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no