Comment conventions, adding an explicit Header.

Comment conventions, adding an explicit Header.

[Phillip Lord]

Currently, emacs uses comments of the form ";;; Commentary;" to
effectively indicate section headers in the buffer.

However, the header of the file has no section indicator -- so, the
copyright, the ";; Author:" metadata and so on.

I was wondering how many things (if any) would break, if this were
changed. So:

;;; blah.el --- Dull file

;; This file is not part of Emacs

;; Author: Phillip Lord <phillip.lord@newcastle.ac.uk>

Would become:


;;; blah.el --- Dull file

;;; Header:

;; This file is not part of Emacs

;; Author: Phillip Lord <phillip.lord@newcastle.ac.uk>


Putting the header on the third or second line is deliberate; various
parts of emacs use the first line semantics, including  -*-
lexical-binding: t -*-.


Why do I ask?

I have written a mode which transforms an Emacs-Lisp file into an
org-mode file. So you can view (and edit) your comments in org-mode,
while maintaining a normal elisp file (i.e. it doesn't require tangling
as an org-mode babel file would). In this process ";;; Commentary:"
lines get transformed into Org mode section one headers. This works
nicely, but the lack of a ";;; Header:" line, means that the metadata
and copyright is outside of the org-mode structure. Adding a ";;;
Header:" is a simple way of circumventing this.

Phil

Re: Comment conventions, adding an explicit Header.

[Eric S. Raymond]

Phillip Lord <phillip.lord@newcastle.ac.uk>:
> I have written a mode which transforms an Emacs-Lisp file into an
> org-mode file. So you can view (and edit) your comments in org-mode,
> while maintaining a normal elisp file (i.e. it doesn't require tangling
> as an org-mode babel file would). In this process ";;; Commentary:"
> lines get transformed into Org mode section one headers. This works
> nicely, but the lack of a ";;; Header:" line, means that the metadata
> and copyright is outside of the org-mode structure. Adding a ";;;
> Header:" is a simple way of circumventing this.

I believe I was the person who made the ";;; Commentary" header
general across the Lisp library, back around '92 or '93. (Possibly I
generalized it from previous sporadic examples rather than inventing it;
I don't remember.)

When I did so, I was thinking explicitly of enabling text analysis by
introducing a structural semantic clue.  I did not anticipate org
mode, but you are extending the concept in the direction I intended.

+1
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Phillip Lord <phillip.lord@newcastle.ac.uk>:
>> I have written a mode which transforms an Emacs-Lisp file into an
>> org-mode file. So you can view (and edit) your comments in org-mode,
>> while maintaining a normal elisp file (i.e. it doesn't require tangling
>> as an org-mode babel file would). In this process ";;; Commentary:"
>> lines get transformed into Org mode section one headers. This works
>> nicely, but the lack of a ";;; Header:" line, means that the metadata
>> and copyright is outside of the org-mode structure. Adding a ";;;
>> Header:" is a simple way of circumventing this.
>
> I believe I was the person who made the ";;; Commentary" header
> general across the Lisp library, back around '92 or '93. (Possibly I
> generalized it from previous sporadic examples rather than inventing it;
> I don't remember.)
>
> When I did so, I was thinking explicitly of enabling text analysis by
> introducing a structural semantic clue.  I did not anticipate org
> mode, but you are extending the concept in the direction I intended.
>
> +1

Thanks. Org-mode seems like an obvious choice as a literate format; in
fact people are already using it for precisely this purpose. org-show is
a nice example, which uses org mode to do a slide show.

The source code is both an example slide show with org-show.el embedded.

https://raw.githubusercontent.com/jkitchin/jmax/master/org/org-show.org

But I dislike relegating the .el file to being generated, hence my
approach. I think it's working quite well. The transformation between
org and emacs-lisp is straight-forward.

On the emacs-lisp side, adding a "Header:" tag is all I need. On the
org-mode side, I need support for emacs-lisp `function-markup-syntax'
which is currently only highlighted in emacs-lisp-mode. So, in
emacs-lisp we have this...

https://github.com/phillord/linked-buffer/blob/master/linked-buffer-org.el

With the org-mode transformation looking like this:

http://homepages.cs.ncl.ac.uk/phillip.lord/scratch/linked-buffer-org.org

And exported to HTML by org-mode, like this:

http://homepages.cs.ncl.ac.uk/phillip.lord/scratch/linked-buffer-org.html

Still a work in progress, of course, but I am happy so far.

Phil

Re: Comment conventions, adding an explicit Header.

[Thorsten Jolitz]

phillip.lord@newcastle.ac.uk (Phillip Lord) writes:

Hallo, 

> Currently, emacs uses comments of the form ";;; Commentary;" to
> effectively indicate section headers in the buffer.

I once did a pretty exhaustive analysis of the 'pathologies' of current
emacs lisp conventions for file structuring:

,----
| https://lists.gnu.org/archive/html/emacs-devel/2013-10/msg00112.html
`----

and while maintainers did acknowledge the weaknesses I described, the
probability of introducing 'modern conventions' seems to tend towards
zero.

But for myself (always using outshine and navi-mode) I developed a
default file structure and a new way to store library meta-data, see
e.g. navi-mode.el:

,----
| 20 matches for "^;;;;? " in buffer: navi-mode.el
|       1:;;; navi-mode.el --- major-mode for easy buffer-navigation
|       7:;;;; MetaData
|      23:;;;; Commentary
|     270:;;;; ChangeLog
|     278:;;; Requires
|     283:;;; Mode Definitions
|     314:;;; Variables
|     315:;;;; Consts
|     316:;;;; Vars
|     335:;;;; Hooks
|     340:;;;; Fonts
|     341:;;;; Customs
|     960:;;; Defuns
|     961:;;;; Functions
|    1471:;;;; Commands
|    2156:;;; Menus and Keys
|    2157:;;;; Menus
|    2368:;;;; Keys
|    2516:;;; Run Hooks and Provide
|    2523:;;; navi-mode.el ends here
`----

with several subtrees under these headlines, e.g.

,----
|       1:;;; navi-mode.el --- major-mode for easy buffer-navigation
|       7:;;;; MetaData
|      23:;;;; Commentary
|      25:;;;;; About navi-mode
|      65:;;;;; Usage
|     245:;;;;; Installation
|     264:;;;;; Emacs Version
|     270:;;;; ChangeLog
|     278:;;; Requires [...]
`----

and this MetaData section, easily editable in Org-mode via outorg:

,----
| ;;;; MetaData
| ;;   :PROPERTIES:
| ;;   :copyright: Thorsten Jolitz
| ;;   :copyright-years: 2013+
| ;;   :version:  2.0
| ;;   :licence:  GPL 2 or later (free software)
| ;;   :licence-url: http://www.gnu.org/licenses/
| ;;   :part-of-emacs: no
| ;;   :author: Thorsten Jolitz
| ;;   :author_email: tjolitz AT gmail DOT com
| ;;   :git-repo: https://github.com/tj64/navi.git
| ;;   :git-clone: git://github.com/tj64/navi.git
| ;;   :inspiration:  occur-mode org-mode 
| ;;   :keywords: emacs navigation remote-buffer-control
| ;;   :END:
`----

[Note that I had to use oldschool instead of outshine headers and to
include the pathological last line ";;; navi-mode.el ends here", which
is not a headline at all, to please MELPA]

-- 
cheers,
Thorsten

Re: Comment conventions, adding an explicit Header.

[Stefan Monnier]

> Currently, emacs uses comments of the form ";;; Commentary;" to
> effectively indicate section headers in the buffer.

Section headers are defined as ";;; <something>".
And subsection headers as ";;;; <something>".

> ;;; blah.el --- Dull file
  ^^^^^^^^^^^^^^^^^^^^^^^^^

This *is* a section header.


        Stefan

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Currently, emacs uses comments of the form ";;; Commentary;" to
>> effectively indicate section headers in the buffer.
>
> Section headers are defined as ";;; <something>".
> And subsection headers as ";;;; <something>".
>
>> ;;; blah.el --- Dull file
>   ^^^^^^^^^^^^^^^^^^^^^^^^^
>
> This *is* a section header.


True. Currently, I handle the first line specially (as does emacs)
because otherwise you end up with level 1 header with, for example, the
lexical-binding instruction in it. 

Also, in the org-mode transformation I translate this into a org-mode
comment. The reason for this is that org-mode also has a "start of file"
semantics -- the lines before the first header is special. If the first
line of the file is a header, then there are no lines before the first
header.

Also, all the other headers that I see ("Commentary", "Status", "Code"
and so forth) are single word and end with an ":". So, currently, I use
this semantics also.

The ;;;; <something> headers I could support, but there aren't that many
files which use this consistently (calc does, so they are some). For
section 2 headers I use

;; ** Header 2

which in org-mode just translates to:

** Header 2

Deeper headers work likewise. This is inconsistent but works well,
because emacs-lisp files are consistent in their use of ";;; Header:"
and inconsistent in their use of ";;;;".


It's a work in progress, of course, but I am looking to DWIM with as
many existing files as possible.

Phil

Re: Comment conventions, adding an explicit Header.

[Stefan Monnier]

> True. Currently, I handle the first line specially (as does emacs)
> because otherwise you end up with level 1 header with, for example, the
> lexical-binding instruction in it.

outline-minor-mode doesn't handle it specially.  I don't mind having the
lexical-binding thingy in there, since it means that when the file
"fully folded" I can still immediately know whether the file is using
lexical-binding or not.

> Also, in the org-mode transformation I translate this into a org-mode
> comment.  The reason for this is that org-mode also has a "start of file"
> semantics -- the lines before the first header is special.

Not knowing what this "special" implies, it's hard to judge whether it's
important.

> Also, all the other headers that I see ("Commentary", "Status", "Code"
> and so forth) are single word and end with an ":". So, currently, I use
> this semantics also.

The few files that use ";;;" for sectioning *within* the code itself
don't seem to use "single word with a colon" very much.  E.g. a quick
grep shows things like:

lisp/wid-edit.el:;;; Utility functions.
lisp/wid-edit.el:;;; Widget text specifications.
lisp/wid-edit.el:;;; Widget Properties.
lisp/view.el:;;; Commands that enter or exit view mode.
lisp/view.el:;;; Some help routines.
lisp/view.el:;;; Commands for moving around in the buffer.
lisp/vc/ediff-util.el:;;; Build keymaps
lisp/vc/ediff-util.el:;;; Setup functions
lisp/vc/ediff-util.el:;;; Commands for working with Ediff
lisp/vc/ediff-util.el:;;; Copying diffs.


        Stefan

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> True. Currently, I handle the first line specially (as does emacs)
>> because otherwise you end up with level 1 header with, for example, the
>> lexical-binding instruction in it.
>
> outline-minor-mode doesn't handle it specially.  I don't mind having the
> lexical-binding thingy in there, since it means that when the file
> "fully folded" I can still immediately know whether the file is using
> lexical-binding or not.

This is true. Likewise, the same is true with the summary. Of course,
you can still see this in the org mode file, and it never folds. But as
a comment, it will not get rendered to any output.

It's not ideal, but I wanted to get something working, to see if it
makes sense first.

>> Also, in the org-mode transformation I translate this into a org-mode
>> comment.  The reason for this is that org-mode also has a "start of file"
>> semantics -- the lines before the first header is special.
>
> Not knowing what this "special" implies, it's hard to judge whether it's
> important.

I'm not a high-power org-mode user either, am afraid, so it's hard for
me to judge too. Things defined before the start, generally apply to the
whole document, so things like new "drawers" are defined here, or
keywords for TODO boxes.

Likewise, titles for output, latex headers and so on.



> The few files that use ";;;" for sectioning *within* the code itself
> don't seem to use "single word with a colon" very much.  E.g. a quick
> grep shows things like:
>
> lisp/wid-edit.el:;;; Utility functions.
> lisp/wid-edit.el:;;; Widget text specifications.
> lisp/wid-edit.el:;;; Widget Properties.
> lisp/view.el:;;; Commands that enter or exit view mode.
> lisp/view.el:;;; Some help routines.
> lisp/view.el:;;; Commands for moving around in the buffer.
> lisp/vc/ediff-util.el:;;; Build keymaps
> lisp/vc/ediff-util.el:;;; Setup functions
> lisp/vc/ediff-util.el:;;; Commands for working with Ediff
> lisp/vc/ediff-util.el:;;; Copying diffs.


Indeed. These will not currently be treated as sections. I could make
them so. In practice, files do need modifying to work with my code. Most
obviously, org-mode needs all the code blocks to be marked up.
I have been doing this by marking ALL of the file after the header,
and then splitting the org code blocks later. I fix sectioning as I go.

All good feedback. Can I conclude that so far there would be no show
stopping problem with adding a ";;; Header:" line where it was wanted?

Phil

Re: Comment conventions, adding an explicit Header.

[Stefan Monnier]

> Indeed.  These will not currently be treated as sections. I could make
> them so.

That would be good, since that's a common case and it also agrees with
the official convention (which says that anything that starts with
"^;;;+ [^ \n]" is a sectioning comment).

> All good feedback. Can I conclude that so far there would be no show
> stopping problem with adding a ";;; Header:" line where it was wanted?

I must say I don't like the idea of adding ";;; Header:" to every Elisp
file, no.  At least not without a very strong argument for it.


        Stefan

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Indeed.  These will not currently be treated as sections. I could make
>> them so.
>
> That would be good, since that's a common case and it also agrees with
> the official convention (which says that anything that starts with
> "^;;;+ [^ \n]" is a sectioning comment).
>
>> All good feedback. Can I conclude that so far there would be no show
>> stopping problem with adding a ";;; Header:" line where it was wanted?
>
> I must say I don't like the idea of adding ";;; Header:" to every Elisp
> file, no.  At least not without a very strong argument for it.


Oh, it doesn't need to be there. I am wondering whether it *can* be
there and iff it is, does it break everything.

Without it, in org-mode you would see this:

----
# # blah.el --- Dull file

This file is not part of Emacs

Author: Phillip Lord <phillip.lord@newcastle.ac.uk>
Other:

GPL Text

* Commentary...
* Code...
----

With it, you get this...


----
# # blah.el -- Dull file

* Header...
* Commentary...
* Code...
----


I think the latter is nicer, that's all.

Phil

Re: Comment conventions, adding an explicit Header.

[Stefan Monnier]

> Oh, it doesn't need to be there. I am wondering whether it *can* be
> there and iff it is, does it break everything.

These are just comments, so they won't break anything, no.
And yes, you can add them to your files.  But I think it'd be better to
find a way to make whateveryouredoing work on all files.

> I think the latter is nicer, that's all.

I'm pretty sure there's a way you can get "the latter" without having to
change the file.


        Stefan

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Oh, it doesn't need to be there. I am wondering whether it *can* be
>> there and iff it is, does it break everything.
>
> These are just comments, so they won't break anything, no.
> And yes, you can add them to your files.  But I think it'd be better to
> find a way to make whateveryouredoing work on all files.
>
>> I think the latter is nicer, that's all.
>
> I'm pretty sure there's a way you can get "the latter" without having to
> change the file.

Agreed, and it is something that I will think on.

Phil

Re: Comment conventions, adding an explicit Header.

[Stephen J. Turnbull]

Phillip Lord writes:
 > Stefan Monnier <monnier@iro.umontreal.ca> writes:
 > 
 > >> Currently, emacs uses comments of the form ";;; Commentary;" to
 > >> effectively indicate section headers in the buffer.
 > >
 > > Section headers are defined as ";;; <something>".
 > > And subsection headers as ";;;; <something>".
 > >
 > >> ;;; blah.el --- Dull file
 > >   ^^^^^^^^^^^^^^^^^^^^^^^^^
 > >
 > > This *is* a section header.

+1

 > True. Currently, I handle the first line specially (as does emacs)
 > because otherwise you end up with level 1 header with, for example, the
 > lexical-binding instruction in it.

I don't really see why this is a problem.  It may be a literate
programming style, but it's still a programming style.  I don't see
what's wrong with including pragmas in the documentation.  If you want
them removed, feel free to do so -- they're code, they have
well-defined syntax.  I personally think it a wart on such styles when
you need comments to mark a defun -- the format generators should do
that for you.  If you're going to do that much analysis, handling the
traditional lisp-mnt.el conventions shouldn't be all that hard.

IMHO YMMV of course, but pragmatically, I think your mode will be more
popular if it Just Works[tm] on the reams of lisp-mnt-formatted files
out there.

 > Also, in the org-mode transformation I translate this into a org-mode
 > comment. The reason for this is that org-mode also has a "start of file"
 > semantics -- the lines before the first header is special.

There aren't any in a Lisp file, and you're already treating this line
as special.  Why not just treat it as a special-case header?

 > Also, all the other headers that I see ("Commentary", "Status", "Code"
 > and so forth) are single word and end with an ":". So, currently, I use
 > this semantics also.

That's not semantics, that's syntax, and it's ad hoc: we (Eric?)
happened to standardize on those and Eric codified them in lisp-mnt,
but nobody has ever defined headers as matching "^;;;;* [A-za-z]+:$".

 > The ;;;; <something> headers I could support, but there aren't that many
 > files which use this consistently (calc does, so they are some). For
 > section 2 headers I use
 > 
 > ;; ** Header 2

Ugh.

 > It's a work in progress, of course, but I am looking to DWIM with as
 > many existing files as possible.

I don't see why defining headers as matching "^;;;;*\s-+.*\S-" or
similar doesn't DWYM.  Few files are long enough to really need
subsection headers; submit patches for the ones that do need them and
don't have them in the traditional format.

Steve

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  > True. Currently, I handle the first line specially (as does emacs)
>  > because otherwise you end up with level 1 header with, for example, the
>  > lexical-binding instruction in it.
>
> I don't really see why this is a problem.  It may be a literate
> programming style, but it's still a programming style.  I don't see
> what's wrong with including pragmas in the documentation.


Well, I agree, it's not a particular disaster, and it is certainly one
option that I considered. However, the problem remains that org-mode
uses the lines before the first header to specify a set of variables,
which is something that I cannot do if the first line is a header.


> If you want them removed, feel free to do so -- they're code, they
> have well-defined syntax. I personally think it a wart on such styles
> when you need comments to mark a defun -- the format generators should
> do that for you.

I would agree with this also. It would be nice to change org-mode so
that it recognises at lisp blocks as code automatically.

>  > Also, in the org-mode transformation I translate this into a org-mode
>  > comment. The reason for this is that org-mode also has a "start of file"
>  > semantics -- the lines before the first header is special.
>
> There aren't any in a Lisp file, and you're already treating this line
> as special.  Why not just treat it as a special-case header?

As I said, because it prevents the full use of org-mode.


>  > Also, all the other headers that I see ("Commentary", "Status", "Code"
>  > and so forth) are single word and end with an ":". So, currently, I use
>  > this semantics also.
>
> That's not semantics, that's syntax, and it's ad hoc: we (Eric?)
> happened to standardize on those and Eric codified them in lisp-mnt,
> but nobody has ever defined headers as matching "^;;;;* [A-za-z]+:$".

Yes, you are right, this is syntax. It can be changed.


>  > The ;;;; <something> headers I could support, but there aren't that many
>  > files which use this consistently (calc does, so they are some). For
>  > section 2 headers I use
>  > 
>  > ;; ** Header 2
>
> Ugh.

It's less ugly in the org-mode view of a file, where it just looks like
a normal org-mode header.


>  > It's a work in progress, of course, but I am looking to DWIM with as
>  > many existing files as possible.
>
> I don't see why defining headers as matching "^;;;;*\s-+.*\S-" or
> similar doesn't DWYM.  Few files are long enough to really need
> subsection headers; submit patches for the ones that do need them and
> don't have them in the traditional format.

I'll think on this. It is most likely possible.

Phil

Re: Comment conventions, adding an explicit Header.

[Stephen J. Turnbull]

Phillip Lord writes:

 > Well, I agree, it's not a particular disaster, and it is certainly one
 > option that I considered. However, the problem remains that org-mode
 > uses the lines before the first header to specify a set of variables,
 > which is something that I cannot do if the first line is a header.

That's a design bug in org-mode, then.  The obvious place to put those
variables is in a variables block in the *last* page, as Emacs has
been doing since before card-punched programs disappeared.

 > > Ugh.
 > 
 > It's less ugly in the org-mode view of a file, where it just looks like
 > a normal org-mode header.

I guess I'm just spoiled by ReST, where in many use cases viewing the
file as plain text is close to optimal (the main thing I'd change is
that I'd use // to delimit italics and ** for bold).

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Phillip Lord writes:
>
>  > Well, I agree, it's not a particular disaster, and it is certainly one
>  > option that I considered. However, the problem remains that org-mode
>  > uses the lines before the first header to specify a set of variables,
>  > which is something that I cannot do if the first line is a header.
>
> That's a design bug in org-mode, then.  The obvious place to put those
> variables is in a variables block in the *last* page, as Emacs has
> been doing since before card-punched programs disappeared.

Well, variables and other directives (in many cases, they do work just
by setting file locals). It makes sense for org-mode however; for
instance, the title of a document is set in this way, and that should,
for obvious reasons come at the top.

The deep problem is that org mode sections have an explicit start and an
implicit end. So "end of the file" is inside a section if there are any
sections at all.


>  > > Ugh.
>  > 
>  > It's less ugly in the org-mode view of a file, where it just looks like
>  > a normal org-mode header.
>
> I guess I'm just spoiled by ReST, where in many use cases viewing the
> file as plain text is close to optimal (the main thing I'd change is
> that I'd use // to delimit italics and ** for bold).


I'm trying to think about interacting with the text rather than just
viewing it. When writing comments, I want all the features that I expect
from emacs for writing documents. The only solution (in emacs and more
widely) to this is some sort of multiple/poly mode system.

Anyway, I shall stop, as I have strayed far from the original point. Not
the first person to do that on this mailing list, I realise.

Phil

Re: Comment conventions, adding an explicit Header.

[Gregor Zattler]

Hi Phillip,
* Phillip Lord <phillip.lord@newcastle.ac.uk> [30. Oct. 2014]:
> I have written a mode which transforms an Emacs-Lisp file into an
> org-mode file. So you can view (and edit) your comments in org-mode,
> while maintaining a normal elisp file (i.e. it doesn't require tangling
> as an org-mode babel file would).

How does this relate to outorg by Thorsten Jolitz?

https://github.com/tj64/outorg

Ciao, Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-

Re: Comment conventions, adding an explicit Header.

[Phillip Lord]

Gregor Zattler <telegraph@gmx.net> writes:

> Hi Phillip,
> * Phillip Lord <phillip.lord@newcastle.ac.uk> [30. Oct. 2014]:
>> I have written a mode which transforms an Emacs-Lisp file into an
>> org-mode file. So you can view (and edit) your comments in org-mode,
>> while maintaining a normal elisp file (i.e. it doesn't require tangling
>> as an org-mode babel file would).
>
> How does this relate to outorg by Thorsten Jolitz?
>
> https://github.com/tj64/outorg


From a user perspective, outorg allows you to edit your code and then
switch to a chunk of documentation and then back out again. This is much
the same paradigm as org-babel, except in reverse (where the
documentation is key, and you edit code a chunk at a time).

With my code (linked-buffer), you see both the code and the
documentation at the same time. So, for example, org-mode folding makes
sense in my case, because you see the whole buffer. You can navigate to
the bit of code you are interested in the folded documentation, and then
jump back to the code. You can publish your code using the org-mode
publication facilities. Instead of special commands to switch modes and
stop when you have finished, you just change buffer.

Set against this, outorg is mature and linked-buffer is not.
Linked-buffer is computationally intensive - which I am trying to
improve, but it may or may not be possible. And linked-buffer's do
strange things to undo.

It's an experiment. I didn't write it for emacs-lisp and org-mode, but
it should be adaptable to that purpose, so it seemed worth a try.

Phil