asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 2357 bytes --]

I started work on rewriting cfengine.el for 3.x and realized the syntax
is too different.  In version 2 it was much simpler, while version 3 has
parameterized bundles, many variable types, new functions, etc.  The
cfengine.el code assumes a lot of things are simple and it's a pain to
extend it directly.  This is the first time I've started on a fairly
major progmode so please bear with me.

I wonder what progmode should I use as a starting point to support the
cfengine 3.x syntax; I've appended some samples at the end.  I also
attach my mostly-broken cfengine.el rewrite, which you can examine but
indentation is mostly wrong and as I worked on it I realized it was just
not easy to retrofit the indentation.

Basically there are two indentation-relevant lines:

1) (body|bundle).+ sets the indentation to 0

2) "x:" sets the indentation to `cfengine-indent' and "y::" sets it to
two times `cfengine-indent'.

...but the possibility of nested braces makes it all much more
interesting, and they have to be indented intelligently.

I could use CEDET but am not sure if it makes sense, considering the
existing code in cfengine.el doesn't.  So I'd like some recommendations
on where to start and what mode(s) could do it.

Ted

body classes context_sensitive(x)
{
  promise_repaired => { "$(x)_repaired" };
  promise_kept     => { "$(x)_kept"     };
  repair_failed    => { "$(x)_failed"   };
  repair_denied    => { "$(x)_denied"   };
  repair_timeout   => { "$(x)_timeout"  };
}

bundle edit_line block_append_if_no_line(data)
{
 insert_lines:
   any::
     "$(data)" insert_type => "preserve_block";
}

bundle agent runthis
{
  commands:

    "$(init) restart" 
    comment        =>    "Restart service $(s) with the initscript $(init).",
    classes        =>     context_yesnorepaired("$(service)_restart"),
    ifvarclass     =>    "$(service)_not_running&please_enable_$(service)";
}

bundle agent ypbind_start_conversion
{
  classes:
    "nis_islink" expression => islink("/etc/init.d/nis");

  commands:
    debian.!nis_islink::
      "/etc/init.d/nis stop";
      "/usr/sbin/service cron stop";

  reports:
    !nis_islink::
      "Stopped cron and ypbind for upstart conversion.";
}

bundle agent routes
{
  vars:
    "partition" slist => { "a", "q", "x" };

  methods:
    "dummy" usebundle => route_wrap("eth0", "any", "$(partition)");
}


[-- Attachment #2: cfengine.el --]
[-- Type: application/emacs-lisp, Size: 13487 bytes --]

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Stefan Monnier]

> 1) (body|bundle).+ sets the indentation to 0

I'd recommend you try the new SMIE thingy (me? biased?).
The syntax you show doesn't seem like a perfect fit, but neither does it
seem obviously difficult to accomodate.

Grep for smie-setup to see some example uses (2 uses in trunk right
now.  There is another one in an earlier version of prolog.el, plus one
in ProofGeneral's coq.el, and I have experimental ones for sh-script,
SML, and Pascal, if you're interested).

There is also documentation in the Elisp manual.


        Stefan

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Fri, 17 Jun 2011 18:02:32 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>> 1) (body|bundle).+ sets the indentation to 0
SM> I'd recommend you try the new SMIE thingy (me? biased?).
SM> The syntax you show doesn't seem like a perfect fit, but neither does it
SM> seem obviously difficult to accomodate.

SM> Grep for smie-setup to see some example uses (2 uses in trunk right
SM> now.  There is another one in an earlier version of prolog.el, plus one
SM> in ProofGeneral's coq.el, and I have experimental ones for sh-script,
SM> SML, and Pascal, if you're interested).

SM> There is also documentation in the Elisp manual.

SMIE is very new so I am concerned that relying on it will block Emacs
users that haven't upgraded.  Can it be distributed with my library?

It's also fairly hard to specify an operator precedence grammar for
CFEngine using SMIE.  I tried it and ended up uncertain how to debug the
parse tree of the current buffer or where my grammar's error lay.  See
below.  Perhaps I'm just too accustomed to LALR parsers, but they seem
easier and less ambiguous.  It's likely I misunderstood the examples, in
any case.

Thanks
Ted

#+begin_src lisp
(defconst cfengine3-operator-table
  '((assoc ",")
    (right "=>")
    (left ":")
    (assoc "||") (assoc "&&") (assoc ".")
    (right "!")))

(defconst cfengine3-smie-bnf-table
  '((classname)
    (class_condition (class_condition "&&" class_condition)
                     (class_condition "||" class_condition)
                     (class_condition "." class_condition)
                     ("!" class_condition)
                     (classname))
    (promise)
    (section)
    (bundle)
    (bundle_type)
    (function_parameter)
    (bundle_parameters (bundle_parameters "," bundle_parameters)
                       (function_parameter))
    (bundle_name (bundle)
                 (bundle bundle_parameters))
    (bundle_line ("bundle" bundle_type bundle_name))
    (promises (promise ";" promise) (promise))
    (bundle_section (section ":" "\n" promises)
                    (class_condition "::" "\n" promises))
    (bundle_body (bundle_section) (bundle_section "\n" bundle_section))
    (bundle_block (bundle_line "{" bundle_body "}"))))

(defvar cfengine3-smie-grammar
  (smie-prec2->grammar
   (smie-merge-prec2s
    (smie-bnf->prec2 cfengine3-smie-bnf-table
                     '((assoc "\n" ";")))
    (smie-precs->prec2 cfengine3-operator-table))))

#+end_src

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Stefan Monnier]

> SMIE is very new so I am concerned that relying on it will block Emacs
> users that haven't upgraded.

It's in Emacs-23.3.

> Can it be distributed with my library?

It's Free Software, so of course you can.

> It's also fairly hard to specify an operator precedence grammar for
> CFEngine using SMIE.

OPG are very limited, indeed, so they're difficult to use for most
languages.

> I tried it and ended up uncertain how to debug the parse tree of the
> current buffer or where my grammar's error lay.

The patch below will at least give you better feedback for the immediate
problem in your grammar.

> See below.  Perhaps I'm just too accustomed to LALR parsers, but they
> seem easier and less ambiguous.

Very likely.  With SMIE you shouldn't try to parse the language in an
exact manner, but think about parsing the minimum needed to indent
properly ;-)

Here are some concrete comments that I hope will help you.

> (defconst cfengine3-smie-bnf-table
>   '((classname)
>     (class_condition (class_condition "&&" class_condition)
>                      (class_condition "||" class_condition)
>                      (class_condition "." class_condition)
>                      ("!" class_condition)
>                      (classname))
>     (bundle_parameters (bundle_parameters "," bundle_parameters)
>                        (function_parameter))

Looks fine.

>     (bundle_name (bundle)
>                  (bundle bundle_parameters))

The second line above is outside the scope of OPG: you can't have
adjacent non-terminals in your grammar.

>     (bundle_line ("bundle" bundle_type bundle_name))

Same here.
Looking at your example, I would recommend you treat "bundle" as
an associative separator, as in

      (bundles (bundles "bundle" bundles))

This will surely strike you as really wrong, but OPG don't handle
"terminator" tokens or "start" tokens nearly as well as
separator tokens.
      
>     (promises (promise ";" promise) (promise))
>     (bundle_section (section ":" "\n" promises)
>                     (class_condition "::" "\n" promises))

Adjacent terminals are allowed in OPG, but they tend to work somewhat
poorly, so I'd recommend to try and avoid them.

>     (bundle_body (bundle_section) (bundle_section "\n" bundle_section))

I'm surprised to see \n here (and in the previous rule).  Are newlines
really significant in cfengine's syntax?

>     (bundle_block (bundle_line "{" bundle_body "}"))))

Better leave {...} as an expression of its own, so that { is an opener
and } is a closer.  I.e. let "bundle" be followed by a sequence of
<things>.
That means that

   bundle edit_line block_append_if_no_line(data)
   {
    insert_lines:
      any::
        "$(data)" insert_type => "preserve_block";
   }

will be parsed as "bundle" followed by 4 <things> (one of them being
"(data)"), which is structurally incorrect (IIUC "(data)" is actually
structurally a child of "block_append_if_no_line" rather than of
"bundle"), but we can hope that it won't matter for indentation.

> (defvar cfengine3-smie-grammar
>   (smie-prec2->grammar
>    (smie-merge-prec2s
>     (smie-bnf->prec2 cfengine3-smie-bnf-table
>                      '((assoc "\n" ";")))
>     (smie-precs->prec2 cfengine3-operator-table))))

You actually want to pass cfengine3-operator-table to smie-bnf->prec2
(as second parameter) because it's needed to disambiguate the
BNF grammar.


        Stefan


=== modified file 'lisp/emacs-lisp/smie.el'
--- lisp/emacs-lisp/smie.el	2011-06-17 01:46:30 +0000
+++ lisp/emacs-lisp/smie.el	2011-06-21 14:35:41 +0000
@@ -265,14 +265,18 @@
               ;; the trouble, and it lets the writer of the BNF
               ;; be a bit more sloppy by skipping uninteresting base
               ;; cases which are terminals but not OPs.
-              (assert (not (member (cadr rhs) nts)))
+              (when (member (cadr rhs) nts)
+                (error "Adjacent non-terminals: %s %s"
+                       (car rhs) (cadr rhs)))
               (pushnew (cadr rhs) first-ops)))
           (let ((shr (reverse rhs)))
             (if (not (member (car shr) nts))
                 (pushnew (car shr) last-ops)
               (pushnew (car shr) last-nts)
               (when (consp (cdr shr))
-                (assert (not (member (cadr shr) nts)))
+                (when (member (cadr rhs) nts)
+                  (error "Adjacent non-terminals: %s %s"
+                         (cadr rhs) (car rhs)))
                 (pushnew (cadr shr) last-ops)))))
         (push (cons nt first-ops) first-ops-table)
         (push (cons nt last-ops) last-ops-table)

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Tue, 21 Jun 2011 10:59:55 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> The patch below will at least give you better feedback for the immediate
SM> problem in your grammar.

I hope you can commit it eventually.

>> See below.  Perhaps I'm just too accustomed to LALR parsers, but they
>> seem easier and less ambiguous.

SM> Very likely.  With SMIE you shouldn't try to parse the language in an
SM> exact manner, but think about parsing the minimum needed to indent
SM> properly ;-)

I tried, but I can't examine the parser state at a buffer point or do
anything else to debug the SMIE grammar.  How do you do it?  I'm used to
the Perl Parse::RecDescent module, which can show line-by-line parsing
traces and the actions taken.

If you could post more SMIE parser examples, it would be helpful.  As I
said, I'm used to LALR grammars and the SMIE API plus the OPG grammars
themselves are a pretty big challenge.

I'm also surprised that Emacs doesn't have a decent LALR parser outside
Semantic, and Semantic itself is poorly documented as far as writing new
language support.

>> (bundle_body (bundle_section) (bundle_section "\n" bundle_section))

SM> I'm surprised to see \n here (and in the previous rule).  Are newlines
SM> really significant in cfengine's syntax?

I thought they were the best way to separate sections.  As far as the
syntax, newlines are separators but not operators, so I didn't do this
right.

SM> That means that

SM>    bundle edit_line block_append_if_no_line(data)
SM>    {
SM>     insert_lines:
SM>       any::
SM>         "$(data)" insert_type => "preserve_block";
SM>    }

SM> will be parsed as "bundle" followed by 4 <things> (one of them being
SM> "(data)"), which is structurally incorrect (IIUC "(data)" is actually
SM> structurally a child of "block_append_if_no_line" rather than of
SM> "bundle"), but we can hope that it won't matter for indentation.

Can you do a SMIE parser for the above to get me started?  It would
really help.  Or point me to an example that's similar to it.

Thanks
Ted

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Stefan Monnier]

SM> The patch below will at least give you better feedback for the immediate
SM> problem in your grammar.
> I hope you can commit it eventually.

Real soon now.

> I tried, but I can't examine the parser state at a buffer point or do
> anything else to debug the SMIE grammar.  How do you do it?

I use C-M-f and C-M-b (SMIE hooks into these commands so that it jumps
not only over (..) but also pays attention to the defined grammar), as
well as looking at smie-grammar.

> I'm also surprised that Emacs doesn't have a decent LALR parser outside
> Semantic, and Semantic itself is poorly documented as far as writing new
> language support.

LALR only works forward, so it requires parsing from the beginning of
the file, which basically forces you to use a cache or something
like that.  OPG on the other hand are bidirectional.

>>> (bundle_body (bundle_section) (bundle_section "\n" bundle_section))
SM> I'm surprised to see \n here (and in the previous rule).  Are newlines
SM> really significant in cfengine's syntax?

> I thought they were the best way to separate sections.

I'm not sure if that answers my question: are they syntactically
significant or are they simply "usually present"?

> As far as the syntax, newlines are separators but not operators, so
> I didn't do this right.

I'm not sure what's the difference between separators and operators,
from a syntax point of view.

SM> That means that

SM> bundle edit_line block_append_if_no_line(data)
SM> {
SM> insert_lines:
SM> any::
SM> "$(data)" insert_type => "preserve_block";
SM> }

SM> will be parsed as "bundle" followed by 4 <things> (one of them being
SM> "(data)"), which is structurally incorrect (IIUC "(data)" is actually
SM> structurally a child of "block_append_if_no_line" rather than of
SM> "bundle"), but we can hope that it won't matter for indentation.

> Can you do a SMIE parser for the above to get me started?  It would
> really help.  Or point me to an example that's similar to it.

I tried the appended code for a start and bumped into the following
problems:

- the indentation of {...} is only done relative to the first token
  after "bundle".  This can be addressed with a special indentation rule
  for (:before . "{") which walks back over sexps until finding
  a "bundle" or similar thingy, but I haven't taken the trouble to
  write it.
- more problematic: the ";" has various precedences, as in Pascal and
  Modula-2, because it can separate elements within "a:: b;c;d", and it
  can separate between "a::b ; c::d".  In Modula-2 I use a hack in the
  lexer which looks at the token after the ";" to decide what kind of
  ";" we're dealing with (see m2-smie-refine-semi), so I guess we could
  try something similar, but we may have to skip a whole
  "class_condition" before finding a "::", which is not great.
- code like

    "$(init) restart" 
    comment        =>    ...

  is parsed as
  
    ( "$(init) restart"
    comment )       =>    ...

  rather than
    
    "$(init) restart" 
    (comment        =>    ...)

  even though it appears that the second parse should be the right one
  (if I understand the intention behind the indentation used in your
  example).
  I don't understand enough of cfengine's grammar to know how best to
  handle this problem (what makes "$(init) restart" into something
  special, is it because it's a string, is it the \n that follows it,
  ...?).

    
        Stefan


(defvar foo-grammar
  (smie-prec2->grammar
   (smie-merge-prec2s
   (smie-bnf->prec2
    '((token)
      (decls (decls "body" decls)
             (decls "bundle" decls))
      (insts (token ":" insts)))
    '((assoc "body" "bundle")))
   (smie-precs->prec2
    '((right ":")
      (right "::")
      (assoc ";")
      (assoc ",")
      (right "=>"))))))

(defun foo-smie-rules (kind token)
  (pcase (cons kind token)
    (`(:elem . basic) 2)
    (`(:list-intro . ,(or `"body" `"bundle")) t)
    (`(:after . ":") 2)
    (`(:after . "::") 2)))

(define-derived-mode foo-mode prog-mode "Foo"
  "Docstring."
  (set (make-local-variable 'comment-start) "#")
  (smie-setup foo-grammar #'foo-smie-rules))

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Wed, 22 Jun 2011 13:44:02 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I tried, but I can't examine the parser state at a buffer point or do
>> anything else to debug the SMIE grammar.  How do you do it?

SM> I use C-M-f and C-M-b (SMIE hooks into these commands so that it jumps
SM> not only over (..) but also pays attention to the defined grammar), as
SM> well as looking at smie-grammar.

Yeesh...  I'm reading the "Parsing Techniques" book you referenced, but
understanding the SMIE grammar table will take me a while.

>> I'm also surprised that Emacs doesn't have a decent LALR parser outside
>> Semantic, and Semantic itself is poorly documented as far as writing new
>> language support.

SM> LALR only works forward, so it requires parsing from the beginning of
SM> the file, which basically forces you to use a cache or something
SM> like that.  OPG on the other hand are bidirectional.

...so you are saying OPG are the way you'd like to go in general?  I
thought syntax tables took care of most of the parsing and state
caching, but honestly I just don't know that area of Emacs well at all.

...
SM>   I don't understand enough of cfengine's grammar to know how best to
SM>   handle this problem (what makes "$(init) restart" into something
SM>   special, is it because it's a string, is it the \n that follows it,
SM>   ...?).

Rather than try to explain the grammar, I'll write it out.  That way the
terminology is clearer; I should have started there instead of giving
you ad-hoc examples.  Sorry about that.  I hope the explanation is
sufficient to clear things up.

There's a lexer and a parser for cfengine3 here:

http://source.cfengine.com/svn/core/trunk/src/cf3parse.y
http://source.cfengine.com/svn/core/trunk/src/cf3lex.l.in

I don't see a formal BNF grammar but the parser is pretty
understandable, standard top-down.  I rewrote it in a condensed form
from the .y file, hoping to make it easier to understand.  I think it's
pretty hard to parse with an OPG, since it's very context-sensitive.

Ted

specification: blocks
blocks: block | blocks block;
block:                 bundle typeid blockid bundlebody
                     | bundle typeid blockid usearglist bundlebody
                     | body typeid blockid bodybody
                     | body typeid blockid usearglist bodybody;

typeid: id
blockid: id
usearglist: '(' aitems ')';
aitems: aitem | aitem ',' aitems |;
aitem: id

bundlebody: '{' statements '}'
statements: statement | statements statement;
statement: category | classpromises;

bodybody: '{' bodyattribs '}'
bodyattribs: bodyattrib | bodyattribs bodyattrib;
bodyattrib: class | selections;
selections: selection | selections selection;
selection: id ASSIGN rval ';' ;

classpromises: classpromise | classpromises classpromise;
classpromise: class | promises;
promises: promise | promises promise;
category: CATEGORY
promise: promiser ARROW rval constraints ';' | promiser constraints ';';
constraints: constraint | constraints ',' constraint |;
constraint: id ASSIGN rval;
class: CLASS
id: ID
rval: ID | QSTRING | NAKEDVAR | list | usefunction
list: '{' litems '}' ;
litems: litem | litem ',' litems |;
litem: ID | QSTRING | NAKEDVAR | list | usefunction

functionid: ID | NAKEDVAR
promiser: QSTRING
usefunction: functionid givearglist
givearglist: '(' gaitems ')'
gaitems: gaitem | gaitems ',' gaitem |;
gaitem: ID | QSTRING | NAKEDVAR | list | usefunction

# from lexer:

bundle: "bundle"
body: "body"
COMMENT    #[^\n]*
NAKEDVAR   [$@][(][a-zA-Z0-9_\200-\377.]+[)]|[$@][{][a-zA-Z0-9_\200-\377.]+[}]
ID: [a-zA-Z0-9_\200-\377]+
ASSIGN: "=>"
ARROW: "->"
QSTRING: \"((\\\")|[^"])*\"|\'((\\\')|[^'])*\'|`[^`]*`
CLASS: [.|&!()a-zA-Z0-9_\200-\377]+::
CATEGORY: [a-zA-Z_]+:

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1973 bytes --]

On Wed, 22 Jun 2011 16:24:00 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Rather than try to explain the grammar, I'll write it out.  That way the
TZ> terminology is clearer; I should have started there instead of giving
TZ> you ad-hoc examples.  Sorry about that.  I hope the explanation is
TZ> sufficient to clear things up.

I included this "grammar" of sorts in cfengine3.el as comments.  The
SMIE stuff is commented out and not functional.  See attached.

A good, fairly large test file is at
http://source.cfengine.com/svn/copbl/trunk/cfengine_stdlib.cf 

This version does:

- much better font-lock (I only need to do in-string highlighting,
  e.g. "this is $(interpolated)" but haven't figured that part out
  yet).  The right faces are applied everywhere otherwise.

- intelligent indentation: 1 indent to "category:"; 2 indents to
  "class_selector::"; 3 indents to actual commands; appropriate
  indentation to nested lists; and 0 indentation to continued strings.
  The results look like this:

#+begin_src cfengine3
bundle edit_line syslog_init_append_oom
{
  delete_lines:
      "CFENGINE_ADD=1.*";

  insert_lines:
    debian::
      "CFENGINE_ADD=1 echo -17 > /proc/$PID/oom_adj" location => after("^\h+start\s+\"\$JOB\"");

    redhat::
      "CFENGINE_ADD=1 echo -17 > /proc/`cat /var/run/syslogd.pid`/oom_adj" 
      location => after("^\h+\[ \$RETVAL -eq 0 \] && touch /var/lock/subsys/syslog");
}

#+end_src

- everything is based on defuns, which in CFEngine3 are syntax blocks
  (older CFEngine versions didn't have them).

- the syntax table from cfengine.el is adjusted for the new version

As it is, except for highlighting string interpolation, it's great for
everyday work.  So I want to know if there's any reason to pursue
SMIE--what more can it do for this mode--and if there are any other
comments on the code.  If not, I'll remove the unnecessary parts, commit
it, and make a note in cfengine.el about the newer version.

Thanks
Ted


[-- Attachment #2: cfengine3.el --]
[-- Type: application/emacs-lisp, Size: 12135 bytes --]

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Thu, 23 Jun 2011 16:31:06 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> As [cfengine3.el stands], except for highlighting string
TZ> interpolation, it's great for everyday work.  So I want to know if
TZ> there's any reason to pursue SMIE--what more can it do for this
TZ> mode--and if there are any other comments on the code.  If not, I'll
TZ> remove the unnecessary parts, commit it, and make a note in
TZ> cfengine.el about the newer version.

Ping?  I'd like to release cfengine3.el before the pretest starts,
please.

Thanks
Ted

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Mon, 27 Jun 2011 10:44:59 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Thu, 23 Jun 2011 16:31:06 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 
TZ> As [cfengine3.el stands], except for highlighting string
TZ> interpolation, it's great for everyday work.  So I want to know if
TZ> there's any reason to pursue SMIE--what more can it do for this
TZ> mode--and if there are any other comments on the code.  If not, I'll
TZ> remove the unnecessary parts, commit it, and make a note in
TZ> cfengine.el about the newer version.

TZ> Ping?  I'd like to release cfengine3.el before the pretest starts,
TZ> please.

I'll commit cfengine3.el to the trunk tonight to beat the feature freeze
date.

Thanks
Ted

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Stefan Monnier]

TZ> As [cfengine3.el stands], except for highlighting string
TZ> interpolation, it's great for everyday work.  So I want to know if
TZ> there's any reason to pursue SMIE--what more can it do for this
TZ> mode--and if there are any other comments on the code.  If not, I'll
TZ> remove the unnecessary parts, commit it, and make a note in
TZ> cfengine.el about the newer version.
TZ> Ping?  I'd like to release cfengine3.el before the pretest starts,
TZ> please.
> I'll commit cfengine3.el to the trunk tonight to beat the feature freeze
> date.

Why not put it in the GNU ELPA instead, so it can keep evolving
independently from the feature freeze?


        Stefan

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Stefan Monnier]

> As it is, except for highlighting string interpolation, it's great for
> everyday work.  So I want to know if there's any reason to pursue
> SMIE--what more can it do for this mode--and if there are any other
> comments on the code.  If not, I'll remove the unnecessary parts, commit
> it, and make a note in cfengine.el about the newer version.

I don't have time to look further into it.  The strength of SMIE is that
it structures the indentation code in a way that makes it fairly robust
(in terms of indentation when the style is unusual, such as newlines
placed at unexpected places) once it works and also that the most
difficult part is the parsing, but that this then can be used for more
than indentation (e.g. for structural navigation, for blink-matching,
hopefully in the future it will also be used for alignment (in the
sense of M-x align)).
But SMIE is far from perfect and is not meant to be "the one and only
way to do it", so if it doesn't work for you, don't use it.


        Stefan

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Thu, 30 Jun 2011 12:03:12 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

TZ> As [cfengine3.el stands], except for highlighting string
TZ> interpolation, it's great for everyday work.  So I want to know if
TZ> there's any reason to pursue SMIE--what more can it do for this
TZ> mode--and if there are any other comments on the code.  If not, I'll
TZ> remove the unnecessary parts, commit it, and make a note in
TZ> cfengine.el about the newer version.
TZ> Ping?  I'd like to release cfengine3.el before the pretest starts,
TZ> please.
>> I'll commit cfengine3.el to the trunk tonight to beat the feature freeze
>> date.

SM> Why not put it in the GNU ELPA instead, so it can keep evolving
SM> independently from the feature freeze?

Because cfengine.el is in the trunk.  It doesn't make sense to put the
two files in different places--it's confusing for the users, they'll see
one and won't know about the other.  I also can't merge cfengine.el with
cfengine3.el, they are too different and the new code is IMHO much
simpler and cleaner.

Ted

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Thu, 30 Jun 2011 12:11:16 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> As it is, except for highlighting string interpolation, it's great for
>> everyday work.  So I want to know if there's any reason to pursue
>> SMIE--what more can it do for this mode--and if there are any other
>> comments on the code.  If not, I'll remove the unnecessary parts, commit
>> it, and make a note in cfengine.el about the newer version.

SM> I don't have time to look further into it.  The strength of SMIE is that
SM> it structures the indentation code in a way that makes it fairly robust
SM> (in terms of indentation when the style is unusual, such as newlines
SM> placed at unexpected places) once it works and also that the most
SM> difficult part is the parsing, but that this then can be used for more
SM> than indentation (e.g. for structural navigation, for blink-matching,
SM> hopefully in the future it will also be used for alignment (in the
SM> sense of M-x align)).

Well, that's why I did the work of extracting the full cfengine 3.x
grammar, so you could see if SMIE made sense or a standard parser was
better.  But it really seems like modifying the standard syntax tables
plus a little bit of indentation logic works OK without SMIE, so I was
asking what else would I get if I put the extra work into writing a SMIE
grammar.

SM> But SMIE is far from perfect and is not meant to be "the one and
SM> only way to do it", so if it doesn't work for you, don't use it.

I want to learn it and use it.  It's just not easy, so please have
patience.  I'm learning OPGs from scratch.

Ted

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Stefan Monnier]

TZ> As [cfengine3.el stands], except for highlighting string
TZ> interpolation, it's great for everyday work.  So I want to know if
TZ> there's any reason to pursue SMIE--what more can it do for this
TZ> mode--and if there are any other comments on the code.  If not, I'll
TZ> remove the unnecessary parts, commit it, and make a note in
TZ> cfengine.el about the newer version.
TZ> Ping?  I'd like to release cfengine3.el before the pretest starts,
TZ> please.
>>> I'll commit cfengine3.el to the trunk tonight to beat the feature freeze
>>> date.

SM> Why not put it in the GNU ELPA instead, so it can keep evolving
SM> independently from the feature freeze?

> Because cfengine.el is in the trunk.  It doesn't make sense to put the
> two files in different places--it's confusing for the users, they'll see
> one and won't know about the other.

Why not output a message from cfengine.el when visiting a version 3 file?


        Stefan

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Thu, 30 Jun 2011 17:10:50 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> I'll commit cfengine3.el to the trunk tonight to beat the feature freeze
>>>> date.

SM> Why not put it in the GNU ELPA instead, so it can keep evolving
SM> independently from the feature freeze?

>> Because cfengine.el is in the trunk.  It doesn't make sense to put the
>> two files in different places--it's confusing for the users, they'll see
>> one and won't know about the other.

SM> Why not output a message from cfengine.el when visiting a version 3 file?

In my experience that's not enough to make the users notice.  And if
they notice, they still have the problem of "now what?  do I really want
to bother installing this package?"  But even assuming they get over
that hurdle, it's still a confusing situation.  Let's either move
cfengine.el to the GNU ELPA as well, or keep them both in the trunk
please.

Ted

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Thu, 30 Jun 2011 13:25:10 -0500
> 
> >> I'll commit cfengine3.el to the trunk tonight to beat the feature freeze
> >> date.
> 
> SM> Why not put it in the GNU ELPA instead, so it can keep evolving
> SM> independently from the feature freeze?
> 
> Because cfengine.el is in the trunk.  It doesn't make sense to put the
> two files in different places--it's confusing for the users, they'll see
> one and won't know about the other.  I also can't merge cfengine.el with
> cfengine3.el, they are too different and the new code is IMHO much
> simpler and cleaner.

I see you committed it already.  However, cfengine3.el clashes with
cfengine.el in the 8+3 file namespace, so please rename it, if you
don't want to follow Stefan's advice and merge them into a single file
(which I personally think is TRT).

TIA

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Fri, 01 Jul 2011 11:19:55 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> Because cfengine.el is in the trunk.  It doesn't make sense to put the
>> two files in different places--it's confusing for the users, they'll see
>> one and won't know about the other.  I also can't merge cfengine.el with
>> cfengine3.el, they are too different and the new code is IMHO much
>> simpler and cleaner.

EZ> I see you committed it already.  However, cfengine3.el clashes with
EZ> cfengine.el in the 8+3 file namespace, so please rename it, if you
EZ> don't want to follow Stefan's advice and merge them into a single file
EZ> (which I personally think is TRT).

I realized I could just do that, even though `cfengine3-mode' remains a
separate function.  So I moved `cfengine3-mode' to cfengine.el and
abstracted the common functionality; I then removed cfengine3.el.

Ted

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Eli Zaretskii]

> From: Ted Zlatanov <tzz@lifelogs.com>
> Date: Fri, 01 Jul 2011 05:15:47 -0500
> 
> On Fri, 01 Jul 2011 11:19:55 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 
> 
> >> Because cfengine.el is in the trunk.  It doesn't make sense to put the
> >> two files in different places--it's confusing for the users, they'll see
> >> one and won't know about the other.  I also can't merge cfengine.el with
> >> cfengine3.el, they are too different and the new code is IMHO much
> >> simpler and cleaner.
> 
> EZ> I see you committed it already.  However, cfengine3.el clashes with
> EZ> cfengine.el in the 8+3 file namespace, so please rename it, if you
> EZ> don't want to follow Stefan's advice and merge them into a single file
> EZ> (which I personally think is TRT).
> 
> I realized I could just do that, even though `cfengine3-mode' remains a
> separate function.  So I moved `cfengine3-mode' to cfengine.el and
> abstracted the common functionality; I then removed cfengine3.el.

Thank you.

Re: asking for advice for changing the cfengine.el progmode to support CFEngine 3.x

[Ted Zlatanov]

On Fri, 01 Jul 2011 14:22:46 +0300 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Ted Zlatanov <tzz@lifelogs.com>
>> Date: Fri, 01 Jul 2011 05:15:47 -0500

EZ> I see you committed it already.  However, cfengine3.el clashes with
EZ> cfengine.el in the 8+3 file namespace, so please rename it, if you
EZ> don't want to follow Stefan's advice and merge them into a single file
EZ> (which I personally think is TRT).
>> 
>> I realized I could just do that, even though `cfengine3-mode' remains a
>> separate function.  So I moved `cfengine3-mode' to cfengine.el and
>> abstracted the common functionality; I then removed cfengine3.el.

EZ> Thank you.

I wish I had done this earlier to avoid hassling Stefan about including
the new cfengine3.el file!  Sometimes I don't see the obvious simple
solution even when people point it out :)

Ted