Code snippets/template consolidation and potential improvements

Code snippets/template consolidation and potential improvements

[Martin Marshall]

(Please accept my apologies in advance for the long message.  I've
tried to organize it to make it easier to digest or piece through, at
the reader's option.)

Hello, there's an item in the TODO file regarding Emacs's template
systems[1], but I'm not sure if I understand it the way it's intended.

What is meant by "consolidate", and what specific improvements should
be prioritized?

I've been looking through the templating packages for a while,
thinking about ways to improve them and writing some code.  I hope to
start a discussion, obtain guidance on improvements to submit, and
coordinate with anyone else who has the time and desire to work on
this.

Here are a few ideas I have, along with some background information:

- Make the same location-jumping commands usable without regard to
  which template package was used.[2]

  + Current situation: expand.el provides commands[3] for jumping
    within its templates.  Skeleton templates can use the same
    commands to a limited extent.[4]

    Observation: Only one template at a time can use the expand.el
    commands.  When a new template expands, it clears the markers
    stored by the last template.  On the other hand, this makes it
    possible for expand.el abbreviations to set markers in a
    non-linear order (since they don't need to be sorted with other
    templates' markers).

  + Current situation: tempo.el includes its own commands for jumping
    to locations within tempo templates.[5]

    Observation: Unlike the expand.el commands, the markers that
    tempo.el stores when expanding a template will persist between
    template expansions and the locations are sorted automatically.
    This allows you to nest a template inside of a surrounding
    template without losing the location-markers of the surrounding
    template.

  Would it be desirable to have one or both packages' jumping commands
  made usable for all 3 template systems?

  It seems they have different benefits and drawbacks.

- Improve the expand.el package.

  Expand.el makes an excellent simplified templating system, but it's
  unknown to most users and its out-of-the-box method for defining
  templates isn't convenient.

  It improves upon traditional abbrevs by adding a single very useful
  feature, the ability to jump to specified locations within the
  expanded text.

  + Current situation: Defining an expand.el abbrev involves calling
    'expand-add-abbrev' (or 'expand-add-abbrevs') and providing a
    manually-calculated list of location indices for use by
    expand.el's location-jumping commands.

    Some ideas to improve the method of defining expand.el abbrevs:

    * Provide an interactive method of defining expand.el templates.
      Since this is the simplest of the template systems, it should
      have the simplest method for defining templates.  This works
      much like the traditional interactive method of defining an
      abbrev.  You select the expansion text and call a command to
      define the abbrev.  But then a buffer pops up containing the
      expansion text, and you can designate the jump-locations on the
      text in this buffer.

      This would allow creation of templates without any specialized
      syntax, not even Elisp.

    * Provide a function for defining an expand.el abbrev that allows
      designating jump-locations by a special character in the
      expansion string.

      By default, this would use the "@" character, because that's
      what skeletons use.  But unlike a skeleton definition, this is a
      text character within the expansion string, not a symbol.  In
      case "@" is to be part of the expanded text, a different
      character can be specified by passing an optional argument to
      the function call.

  + Current situation: Expand.el doesn't come with prompting of any
    kind.  It would be more useful if there were some way to remind
    oneself what text to insert at a specific point, or provide some
    default text that could be easily replaced.

    I think expand.el's strength is its simplicity, but I also think
    it can be improved without making it overly complex.

    Idea:

    * Extend expand.el templates so that a jump-location can be either
      a single location or a region which becomes activated upon
      reaching it with one of the jumping commands.  Once activated,
      the region can be left as-is, overwritten by newly typed text,
      or deleted by simply pressing "DEL".  This is a feature included
      in many more modern template systems (e.g.\nosentn{} yasnippets, tempel
      templates).

      Caveat: Adding this feature might complicate consolidation of
      the location-jumping commands.  A potential way to alleviate
      that would be to add this to skeleton.el's and tempo.el's
      templates.

  + Expand.el isn't documented.

    Idea:

    * After adding new features and consolidating expand.el with the
      other template systems, document it in the Autotyping manual
      and/or the Emacs manual.

  There's almost no mention of expand.el online, but I find the
  simplicity of its templates quite attractive.  You expand an abbrev
  and can jump to locations within the expanded text.  I imagine this
  covers the requirements of 80-90% of templates.  Most of the time
  you just need some pre-written text with an easy way to change
  specific parts of it.

- Consolidation in general

  How should these template systems be consolidated?

  + Should there be a single function/macro for creating any type of
    template?

  + Should the definition syntax itself be consolidated?  (I would
    think not, but the meaning of "consolidate" might encompass that.)

From my perspective, Emacs's existing template options each fill a
role, because each comes with different levels of versatility, which
correlates to the learning curve of a user getting started with one of
them.

If you have read this far (or even if you just jumped to this
location), I thank you for your attention!


Footnotes:

[1] It reads, "** Improve the 'code snippets' support 
Consolidate skeleton.el, tempo.el, and expand.el (any other?) and then
advertise/use/improve it."

[2] This was the subject of a patch I submitted a couple of days ago,
but that may have been premature.

[3] They are 'expand-jump-to-next-slot' and
'expand-jump-to-previous-slot', bound by default to "C-x a n" and
"C-x a p" respectively.

[4] The locations are specified in the skeleton using '@' symbols.
The skeleton must be added to an abbrev-table using
'expand-add-abbrev', and the commands will only work when the skeleton
has been expanded by the abbrev.  They will not work when the skeleton
was expanded by calling its command (unless you've applied my patch to
expand.el and skeleton.el).

[5] The tempo.el commands are 'tempo-forward-mark' and
'tempo-backward-mark'.  They have no default keybindings.

-- 
Best regards,
Martin Marshall

Re: Code snippets/template consolidation and potential improvements

[Eli Zaretskii]

> From: Martin Marshall <law@martinmarshall.com>
> Date: Wed, 17 Jan 2024 16:48:52 -0500
> 
> (Please accept my apologies in advance for the long message.  I've
> tried to organize it to make it easier to digest or piece through, at
> the reader's option.)
> 
> Hello, there's an item in the TODO file regarding Emacs's template
> systems[1], but I'm not sure if I understand it the way it's intended.
> 
> What is meant by "consolidate", and what specific improvements should
> be prioritized?

I believe by "consolidate" the intent is to have a single package that
provides the features of those 3.  CC'ing Stefan, who added that TODO
item.

> I've been looking through the templating packages for a while,
> thinking about ways to improve them and writing some code.  I hope to
> start a discussion, obtain guidance on improvements to submit, and
> coordinate with anyone else who has the time and desire to work on
> this.

Thanks in advance for your work on these features.

Re: Code snippets/template consolidation and potential improvements

[Martin Marshall]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Martin Marshall <law@martinmarshall.com>
>> Date: Wed, 17 Jan 2024 16:48:52 -0500
>> 
>> Hello, there's an item in the TODO file regarding Emacs's template
>> systems[1], but I'm not sure if I understand it the way it's intended.
>> 
>> What is meant by "consolidate", and what specific improvements should
>> be prioritized?
>
> I believe by "consolidate" the intent is to have a single package that
> provides the features of those 3.  CC'ing Stefan, who added that TODO
> item.

Oh, that will make it easier to share code between them (of course).

> Thanks in advance for your work on these features.

Thanks for your response.  I look forward to getting started.

-- 
Best regards,
Martin Marshall

Re: Code snippets/template consolidation and potential improvements

[Stefan Kangas]

Martin Marshall <law@martinmarshall.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Martin Marshall <law@martinmarshall.com>
>>> Date: Wed, 17 Jan 2024 16:48:52 -0500
>>>
>>> Hello, there's an item in the TODO file regarding Emacs's template
>>> systems[1], but I'm not sure if I understand it the way it's intended.
>>>
>>> What is meant by "consolidate", and what specific improvements should
>>> be prioritized?
>>
>> I believe by "consolidate" the intent is to have a single package that
>> provides the features of those 3.  CC'ing Stefan, who added that TODO
>> item.

[I guess you meant to Cc Stefan Monnier, so I did that here.]

Why not start the work with considering both yasnippet and tempel, both of
which are on GNU ELPA.  Perhaps either of them are already good enough?

FWIW, I've used yasnippet since forever and I find it much easier to use
than anything in core.  I haven't really had occasion to consider any of
the other alternatives that exist.

This is from Tempel's documentation:

  List of alternatives (built-in or separate packages):

  • abbrev.el: Abbreviation expansion, builtin
  • expand.el: Abbreviation expansion, builtin
  • skeleton.el: Lisp syntax for templates, builtin
  • tempo.el: Lisp syntax for templates, builtin
  • srecode.el: CEDET template manager and code generator, builtin
  • [aas.el]: Auto activating snippets
  • [cdlatex.el]: Fast LaTeX insertion
  • [laas.el]: Latex auto activating snippets
  • [muban.el]: Lightweight template expansion
  • [placeholder.el]: Treat buffers as templates
  • [tempo-abbrev.el]: Abbrev integration for Tempo
  • [snippet.el]: Original snippet mode, with inline expansion
  • [tempo-snippets.el]: Interface like snippet.el for Tempo
  • [yasnippet.el]: Template system inspired by Textmate snippets

Re: Code snippets/template consolidation and potential improvements

[Stefan Monnier]

> Why not start the work with considering both yasnippet and tempel, both of
> which are on GNU ELPA.  Perhaps either of them are already good enough?

A previous effort in that direction is the `snippet-engine` branch in
YASnippet's Git repository, which aimed to reimplement YASnippet's core
functionality separately from the specific user-visible format of the
snippets, such that YASnipet could then be reimplemented on top of
snippet-engine (and hopefully other systems like tempo, tempel, skeleton
could be reimplemented on top of it as well).

AFAIK that branch was on the right track, but the implementer (João) lost
motivation at some point.


        Stefan

Re: Code snippets/template consolidation and potential improvements

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 18 Jan 2024 13:13:26 -0800
> Cc: emacs-devel@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	João Távora <joaotavora@gmail.com>, 
> 	Daniel Mendler <mail@daniel-mendler.de>
> 
> Martin Marshall <law@martinmarshall.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >>> From: Martin Marshall <law@martinmarshall.com>
> >>> Date: Wed, 17 Jan 2024 16:48:52 -0500
> >>>
> >>> Hello, there's an item in the TODO file regarding Emacs's template
> >>> systems[1], but I'm not sure if I understand it the way it's intended.
> >>>
> >>> What is meant by "consolidate", and what specific improvements should
> >>> be prioritized?
> >>
> >> I believe by "consolidate" the intent is to have a single package that
> >> provides the features of those 3.  CC'ing Stefan, who added that TODO
> >> item.
> 
> [I guess you meant to Cc Stefan Monnier, so I did that here.]

No, I meant you, see commit 317eb2d5b5.  But I see now that you just
re-arranged the file, and Git just duped me into thinking that commit
added this text.

Re: Code snippets/template consolidation and potential improvements

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

On Thu, Jan 18, 2024, 22:03 Stefan Monnier <monnier@iro.umontreal.ca> wrote:

A previous effort in that direction is the `snippet-engine` branch in
> YASnippet's Git repository, which aimed to reimplement YASnippet's core
> functionality separately from the specific user-visible format of the
> snippets, such that YASnipet could then be reimplemented on top of
> snippet-engine (and hopefully other systems like tempo, tempel, skeleton
> could be reimplemented on top of it as well).


Yes yes.

AFAIK that branch was on the right track,


You should fully disclose it was like 90% your design with my
implementation. But it's true, it was.

but the implementer (João) lost
> motivation at some point.


It's now a repo in github.com/joaotavora/snippet

Maybe I'll get a bit of motivation going again, some APIs are simpler and
I've learned some new tricks since.

Anyone can pick it up. There's a TODO list at the top, and a test suite.
It's a good candidate for test-driven programming.

João

[-- Attachment #2: Type: text/html, Size: 2076 bytes --]

Re: Code snippets/template consolidation and potential improvements

[Martin Marshall]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Jan 18, 2024, 22:03 Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> It's now a repo in github.com/joaotavora/snippet

This looks amazing--as does all of your work that I've seen.

It's also far more than the simple consolidation of
skeleton/tempo/expand resources, I'd imagined.  If those packages can be
reimplemented on top of this, that obviously makes a simpler
consolidation of the packages unnecessary.

-- 
Best regards,
Martin Marshall

Re: Code snippets/template consolidation and potential improvements

[João Távora]

On Fri, Jan 19, 2024 at 4:40 PM Martin Marshall <law@martinmarshall.com> wrote:

> > It's now a repo in github.com/joaotavora/snippet
>
> This looks amazing--as does all of your work that I've seen.

I'm very flattered, but here you should mostly think Stefan Monnier
who pitched the original idea and the "dynamic snippet" syntax,
as well as coaching me circa 2015 through the macrology-laden
implementation (which could probably see a tweak or twenty).

Anyway, I don't think the package is production-ready yet.
It's missing some notable and advanced features of the Yasnippet
engine some of which I'd have to get reacquainted with.  Notably
Yasnippet's undo capability is still missing.

I don't use Yasnippet much :-) except for one single trivial
snippet that inserts the date&name in comments.  And sometimes
via LSP (because LSP issues snippets in the original Textmate syntax
and Yasnippet groks that).

Anyway, if someone could write (failing) tests for those features
I would probably be motivated to implement them.


João

Re: Code snippets/template consolidation and potential improvements

[Martin Marshall]

João Távora <joaotavora@gmail.com> writes:

> Anyway, if someone could write (failing) tests for those features
> I would probably be motivated to implement them.

This will take some time for me, as I haven't used eieio or ERT before,
and it looks like I'll need to familiarize myself with those.

I'll update with any progress.  Of course, if anyone else is interested,
they should also feel free to join in.

-- 
Best regards,
Martin Marshall