Release 6.17

Release 6.17

[Carsten Dominik]

Hi,

I am releasing version 6.17 of Org-mode.

Besides the footnote support discussed already extensively here,
this release also contains a new feature to make Org-mode
more usable for writing tutorials and similar documents with
code examples.  Lines in code examples can now be numbered,
and you can link to specific lines with magical highlighting
of the line when hovering the moue over the link.

Enjoy.

- Carsten


Changes in Version 6.17
=======================

Overview
=========

- Footnote support
- Line numbers and references in literal examples
- New hooks for export preprocessing
- Capture column view into a different file

Details
========

Footnote support
-----------------

Org-mode now directly supports the creation of footnotes.  In
contrast to the /footnote.el/ package, Org-mode's footnotes are
designed for work on a larger document, not only for one-off
documents like emails.  The basic syntax is similar to the one
used by /footnote.el/, i.e. a footnote is defined in a paragraph
that is started by a footnote marker in square brackets in column
0, no indentation allowed.  The footnote reference is simply the
marker in square brackets inside text.  For example:

   The Org homepage[fn:1] now looks a lot better than it used to.
   ...
   [fn:1] The link is: http://orgmode.org


Org-mode extends the number-based syntax to /named/ footnotes and
optional inline definition.  Using plain numbers as markers is
supported for backward compatibility, but not encouraged because
of possible conflicts with LaTeX syntax.  Here are the valid
references:

- [1] ::  A plain numeric footnote marker.

- [fn:name] :: A named footnote reference, where `name' is a
      unique label word or, for simplicity of automatic creation,
      a number.

- [fn:: This is the inline definition of this footnote] :: A
      LaTeX-like anonymous footnote where the definition is given
      directly at the reference point.

- [fn:name: a definition] :: An inline definition of a footnote,
      which also specifies a name for the note.  Since Org allows
      multiple references to the same note, you can then use use
      `[fn:name]' to create additional references.

Footnote labels can be created automatically, or you create names
yourself.  This is handled by the variable
`org-footnote-auto-label' and its corresponding `#+STARTUP'
keywords, see the docstring of that variable for details.

The following command handles footnotes:

- C-c C-x f :: The footnote action command.  When the cursor is
      on a footnote reference, jump to the definition.  When it is
      at a definition, jump to the (first) reference.  Otherwise,
      create a new footnote.  Depending on the variable
      `org-footnote-define-inline' (with associated `#+STARTUP'
      options `fninline' and `nofninline'), the definitions will
      be placed right into the text as part of the reference, or
      separately into the location determined by the variable
      `org-footnote-section'.
      When this command is called with a prefix argument, a menu
      of additional options is offered:
      - s :: Sort the footnote definitions by reference sequence.
             During editing, Org makes no effort to sort footnote
             definitions into a particular sequence.  If you want
             them sorted, use this command, which will also move
             entries according to `org-footnote-section'.
      - n :: Normalize the footnotes by collecting all
             definitions (including inline definitions) into a
             special section, and then numbering them in
             sequence.  The references will then also be
             numbers.  This is meant to be the final step before
             finishing a document (e.g. sending off an email).
             The exporters do this automatically, and so could
             something like `message-send-hook'.
      - d :: Delete the footnote at point, and all references to it.

- C-c C-c :: If the cursor is on a footnote reference, jump to
      the definition.  If it is a the definition, jump back to the
      reference.  When called with a prefix argument at either
      location, offer the same menu as `C-u C-c C-x f'.

- C-c C-o or mouse-1/2 :: Footnote labels are also links to the
      corresponding definition/reference, and you can use the
      usual commands to follow these links.

Org-mode's footnote support is designed so that it should also
work in buffers that are not in Org-mode, for example in email
messages.  Just bind `org-footnote-action' to a global key like
`C-c f'.

The main trigger for this development came from a hook function
written by Paul Rivier, to implement named footnotes and to
convert them to numbered ones before export.  Thanks, Paul!

Thanks also to Scot Becker for a thoughtful post bringing this
subject back onto the discussion table, and to Matt Lundin for
the idea of named footnotes and his prompt testing of the new
features.

Line numbers and references in literal examples
------------------------------------------------

Literal examples introduced with `#+BEGIN_EXAMPLE' or `#+BEGIN_SRC'
do now allow optional line numbering in the example.
Furthermore, links to specific code lines are supported, greatly
increasing Org-mode's utility for writing tutorials and other
similar documents.

Code references use special labels embedded directly into the
source code.  Such labels look like "((name))" and must be unique
within a document.  Org-mode links with the coderef cookie in the
link part will be correctly interpreted, both while working with
an Org file (internal links), and while exporting to the
different backends.  Line numbering and code references are
supported for all three major backends, HTML, LaTeX, and ASCII.
In the HTML backend, hovering the mouse over a link to a source
line will remote-highlight the referenced code line.

The options for the BEGIN lines are:

  - -n :: Number the lines in the example
  - +n :: Like -n, but continue numbering from where the previous
          example left off.
  - -r :: Remove the coderef cookies from the example, and replace
          links to this reference with line numbers.  This option
          takes only effect if either -n or +n are given as well.
          If -r is not given, coderefs simply use the label name.

Here is an example:

   #+begin_src emacs-lisp -n -r
   (defmacro org-unmodified (&rest body)                   ((def))
     "Execute body without changing `buffer-modified-p'."
     `(set-buffer-modified-p                               ((back))
       (prog1 (buffer-modified-p) ,@body)))
   #+end_src
   [[Line ((def))]] contains the macro name.  Later at line  
[[((back))]],
   backquoting is used.


When exported, this is translated to the following.  Make sure to
check this out in the HTML version of this document, at
        http://orgmode.org/Changes.html#sec-1.2.2
it looks much better there.

      1:  (defmacro org-unmodified (&rest body)
      2:    "Execute body without changing `buffer-modified-p'."
      3:    `(set-buffer-modified-p
      4:      (prog1 (buffer-modified-p) ,@body)))

    Line 1 contains the macro name.  Later at line 3,
    backquoting is used.


Thanks to Ilya Shlyakhter for proposing this feature set.  Thanks
to Sebastian Rose for the key Javascript element that made the
remote highlighting possible.

New hooks for export preprocessing
-----------------------------------
     The export preprocessor now runs more hooks, to allow
     better-timed tweaking by user functions:

- `org-export-preprocess-hook' ::
   Pretty much the first thing in the preprocessor.  But org-mode
   is already active in the preprocessing buffer.

- `org-export-preprocess-after-include-files-hook' ::
   This is run after the contents of included files have been inserted.

- `org-export-preprocess-after-tree-selection-hook' ::
   This is run after selection of trees to be exported has
   happened.  This selection includes tags-based selection, as
   well as removal of commented and archived trees.

- `org-export-preprocess-before-backend-specifics-hook' ::
   Hook run before backend-specific functions are called during  
preprocessing.

- `org-export-preprocess-final-hook' ::
   Hook for preprocessing an export buffer.  This is run as the
   last thing in the preprocessing buffer, just before returning
   the buffer string to the backend.

Capture column view into a different file
------------------------------------------

     The :id parameter for the dynamic block capturing column view
     can now truly be an ID that will also be found in a
     different file.  Also, it can be like =[file:path/to/file=], to
     capture the global column view from a different file.

     Thanks to Francois Lagarde for his report that IDs outside
     the current file would not work.

Re: Release 6.17

[Carsten Dominik]

On Jan 4, 2009, at 9:13 AM, Carsten Dominik wrote:

> Hi,
>
> I am releasing version 6.17 of Org-mode.

[...]

> Line numbers and references in literal examples
> ------------------------------------------------

[...]

> Here is an example:
>
>  #+begin_src emacs-lisp -n -r
>  (defmacro org-unmodified (&rest body)                   ((def))
>    "Execute body without changing `buffer-modified-p'."
>    `(set-buffer-modified-p                               ((back))
>      (prog1 (buffer-modified-p) ,@body)))
>  #+end_src
>  [[Line ((def))]] contains the macro name.  Later at line  
> [[((back))]],


This did not come out right, it must be

    [[((def))][Line ((def))]] contains the macro name.  Later at line  
[[((back))]]

i.e. the linke part of the link must be only the coderef cookie,
while the description part may contain additional text.

- Carsten

Re: Release 6.17

[Steven E. Harris]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Code references use special labels embedded directly into the source
> code.  Such labels look like "((name))" and must be unique within a
> document.

How does the parser know that, say, "((def))" is not a valid expression
in the surrounding Lisp forms? Is it important that it be separated by
space, or be the last token on the line?

Trying to concoct a motivating example, consider a structure represented
as nested lists:

,----
| '(a
|   ((b c) d)
|   (((e) f))    ((def))
|   g)
`----

Without knowing what the enclosing `quote' form means, how do know that
"((def))" is not part of it?

-- 
Steven E. Harris

Re: Re: Release 6.17

[Carsten Dominik]

On Jan 4, 2009, at 3:33 PM, Steven E. Harris wrote:

> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>> Code references use special labels embedded directly into the source
>> code.  Such labels look like "((name))" and must be unique within a
>> document.
>
> How does the parser know that, say, "((def))" is not a valid  
> expression
> in the surrounding Lisp forms? Is it important that it be separated by
> space, or be the last token on the line?
>
> Trying to concoct a motivating example, consider a structure  
> represented
> as nested lists:
>
> ,----
> | '(a
> |   ((b c) d)
> |   (((e) f))    ((def))
> |   g)
> `----
>
> Without knowing what the enclosing `quote' form means, how do know  
> that
> "((def))" is not part of it?

Hi Steven,

good question, and the answer is that is does not know,
cannot know, because this is a feature that is supposed
to work for any kind of example, an the parser cannot
know all possible syntaxes :-)

This idea is to make this work in a heuristic way, by using something
that is unlikely enough to occur in real code.

You are right that what I am using might be too
dangerous for emacs lisp or other lisp dialects, and
it could also show up in other languages like C.

What would be safer?

  <<name>>    like the other Org-mode targets?  That would make sense.
              Does anyone know a language where this would be used
              in real life?  It would make it harder to write about
              Org-mode, though.

Or do we need another option, so that, if needed, we could switch do
a different syntax?

Comments are very welcome.

- Carsten

Re: Release 6.17

[Steven E. Harris]

Carsten Dominik <dominik@science.uva.nl> writes:

> This idea is to make this work in a heuristic way, by using something
> that is unlikely enough to occur in real code.

And that is a tough problem, as code is usually defined as stuff that
contains all kinds of weird (and often paired) delimiters.

[...]

> What would be safer?
>
>  <<name>>    like the other Org-mode targets?  That would make sense.
>              Does anyone know a language where this would be used
>              in real life?  It would make it harder to write about
>              Org-mode, though.
>
> Or do we need another option, so that, if needed, we could switch do a
> different syntax?

This reminds me of the "leaning toothpick" problem with regular
expression syntax; Perl and some other languages adopted the flexibility
to accept any "matching" delimiters (either the same character used
twice or a balancing pair) in lieu of the default '/' delimiter
character. There was the need to have the delimiters be able to "get out
of the way" of the dominant syntax within that particular regular
expression. Here, too, I expect that we'd either need to define
language-specific escape hatches, or stop guessing and force the user to
define the active delimiters.

What if the user could specify before each code block some "dispatch
character" that then had to be followed by a more telling string, such
as "#line:def". In that example, the octothorpe is the dispatch
character, the "line:" is the belt-and-suspenders clarifying tag, and
the "def" is the named label for that line. Force it to be at the end of
the line (perhaps modulo trailing space), as there should only be one
definition per line.

A regular expression match would look for

  #line:([^)]+)\s*$
  ^
  |
  + (not fixed)

except that the dispatch character would need to be composed in and
regex-quoted appropriately. Also, that one would tolerate anything but a
closing parenthesis in a label; it could be more restrictive to tolerate
something more commonly expected of an identifier such as alphanumerics,
dashes, and underscores.

You could punt even further and just demand that the user provide a
suitable regex for finding the line labels unambiguously. I'm just leery
of trying to pick a default that's expected to work not just within
natural language, but within program source code.

-- 
Steven E. Harris

Re: Re: Release 6.17

[Carsten Dominik]

Hi Steven,

thank you for your thoughtful post and everyone else for chiming in with
useful suggestions.

I have just uploaded 6.17a which revamps the codeline references stuff,
in the following way:

1. The default label now looks like      (ref:name)

2. The default format is defined in org-coderef-label-format,
    with the default value "(ref:%s)".

3. You can change the format for each individual snippet with the -l  
switch:

       #+BEGIN_SRC pascal -n -r -l "((%s))"

4. Links to the labels have also changed, they are now

        [[(name)]]  or [[(name)][in line (name)]]

    instead of

        [[((name))]]  or [[((name))][in line ((name))]]

    i.e. only single parenthesis around the label name.

5. For technical reasons, there are currently some restrictions
    to what you can use as label format:

    - Do not use the character special in HTML:  `<', `>', and `&'.

    - Use something that will not be split up into sections
      with different fonts by font-lock/htmlize.  For example,
      in pascal-mode, "{{%s}}" will not work (I know nothing of
      Pascal, was just something I tried).

    The reason for both restrictions is that the current
    implementation looks for the labels only *after* htmlize has
    done its work on the example.  Clearly it would be good to
    change this, but it is non-trivial and I won't do it until
    I see that it is really necessary.

Let's see how ar we get with this.

- Carsten

On Jan 4, 2009, at 9:24 PM, Steven E. Harris wrote:

> Carsten Dominik <dominik@science.uva.nl> writes:
>
>> This idea is to make this work in a heuristic way, by using something
>> that is unlikely enough to occur in real code.
>
> And that is a tough problem, as code is usually defined as stuff that
> contains all kinds of weird (and often paired) delimiters.
>
> [...]
>
>> What would be safer?
>>
>> <<name>>    like the other Org-mode targets?  That would make sense.
>>             Does anyone know a language where this would be used
>>             in real life?  It would make it harder to write about
>>             Org-mode, though.
>>
>> Or do we need another option, so that, if needed, we could switch  
>> do a
>> different syntax?
>
> This reminds me of the "leaning toothpick" problem with regular
> expression syntax; Perl and some other languages adopted the  
> flexibility
> to accept any "matching" delimiters (either the same character used
> twice or a balancing pair) in lieu of the default '/' delimiter
> character. There was the need to have the delimiters be able to "get  
> out
> of the way" of the dominant syntax within that particular regular
> expression. Here, too, I expect that we'd either need to define
> language-specific escape hatches, or stop guessing and force the  
> user to
> define the active delimiters.
>
> What if the user could specify before each code block some "dispatch
> character" that then had to be followed by a more telling string, such
> as "#line:def". In that example, the octothorpe is the dispatch
> character, the "line:" is the belt-and-suspenders clarifying tag, and
> the "def" is the named label for that line. Force it to be at the  
> end of
> the line (perhaps modulo trailing space), as there should only be one
> definition per line.
>
> A regular expression match would look for
>
>  #line:([^)]+)\s*$
>  ^
>  |
>  + (not fixed)
>
> except that the dispatch character would need to be composed in and
> regex-quoted appropriately. Also, that one would tolerate anything  
> but a
> closing parenthesis in a label; it could be more restrictive to  
> tolerate
> something more commonly expected of an identifier such as  
> alphanumerics,
> dashes, and underscores.
>
> You could punt even further and just demand that the user provide a
> suitable regex for finding the line labels unambiguously. I'm just  
> leery
> of trying to pick a default that's expected to work not just within
> natural language, but within program source code.
>
> -- 
> Steven E. Harris
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Release 6.17

[Steven E. Harris]

"Steven E. Harris" <seh@panix.com> writes:

> Also, that one would tolerate anything but a closing parenthesis in a
> label;

That was a mistake to propose. I had forgotten that I intended the label
to run to the end of the line, not to a bounding parenthesis. So much
for writing code in haste without testing it -- or thinking it through
clearly.

-- 
Steven E. Harris

Re: Re: Release 6.17

[Eddward DeVilla]

On Sun, Jan 4, 2009 at 10:01 AM, Carsten Dominik <dominik@science.uva.nl> wrote:
>
> On Jan 4, 2009, at 3:33 PM, Steven E. Harris wrote:
>
>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>
>>> Code references use special labels embedded directly into the source
>>> code.  Such labels look like "((name))" and must be unique within a
>>> document.
>>
>> How does the parser know that, say, "((def))" is not a valid expression
>> in the surrounding Lisp forms? Is it important that it be separated by
>> space, or be the last token on the line?
>>
>> Trying to concoct a motivating example, consider a structure represented
>> as nested lists:
>>
>> ,----
>> | '(a
>> |   ((b c) d)
>> |   (((e) f))    ((def))
>> |   g)
>> `----
>>
>> Without knowing what the enclosing `quote' form means, how do know that
>> "((def))" is not part of it?
>
> Hi Steven,
>
> good question, and the answer is that is does not know,
> cannot know, because this is a feature that is supposed
> to work for any kind of example, an the parser cannot
> know all possible syntaxes :-)
>
> This idea is to make this work in a heuristic way, by using something
> that is unlikely enough to occur in real code.
>
> You are right that what I am using might be too
> dangerous for emacs lisp or other lisp dialects, and
> it could also show up in other languages like C.
>
> What would be safer?
>
>  <<name>>    like the other Org-mode targets?  That would make sense.
>             Does anyone know a language where this would be used
>             in real life?  It would make it harder to write about
>             Org-mode, though.
>
> Or do we need another option, so that, if needed, we could switch do
> a different syntax?
>
> Comments are very welcome.
>
> - Carsten

I think that is quote words in perl 6.

@list = <<$this is a 'list' of 7 strings>>  # in perl 6 is
@list = qw/$this is a 'list' of 7 strings/  # in perl 5.

It's looking like perl 6 will be a reality and that syntax is
recommend in several places like hash dereferences.

%hash<<bareword>>  # look up bareword in %hash

I can't remember enough off the top of my head, but I think <<name>>
will play merry heck with common(?) perl 6 code.  I can look up more
examples if needed.

Edd

Re: Re: Release 6.17

[David Lord]

[-- Attachment #1.1: Type: text/plain, Size: 427 bytes --]

Carsten,

2009/1/4 Carsten Dominik <dominik@science.uva.nl>:
>
>  <<name>>    like the other Org-mode targets?  That would make sense.
>             Does anyone know a language where this would be used
>             in real life?  It would make it harder to write about
>             Org-mode, though.

Yes, Oracle pl/sql uses that for loop labels.  You might also want to check
ADA since pl/sql is based on it.

-- David Lord

[-- Attachment #1.2: Type: text/html, Size: 1105 bytes --]

[-- Attachment #2: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: Release 6.17

[Rick Moynihan]

Carsten Dominik wrote:
> On Jan 4, 2009, at 3:33 PM, Steven E. Harris wrote:
> 
>> Carsten Dominik <carsten.dominik@gmail.com> writes:
>>
>>> Code references use special labels embedded directly into the source
>>> code.  Such labels look like "((name))" and must be unique within a
>>> document.
>> How does the parser know that, say, "((def))" is not a valid  
>> expression
>> in the surrounding Lisp forms? Is it important that it be separated by
>> space, or be the last token on the line?
>>
>> Trying to concoct a motivating example, consider a structure  
>> represented
>> as nested lists:
>>
>> ,----
>> | '(a
>> |   ((b c) d)
>> |   (((e) f))    ((def))
>> |   g)
>> `----
>>
>> Without knowing what the enclosing `quote' form means, how do know  
>> that
>> "((def))" is not part of it?
> 
> Hi Steven,
> 
> good question, and the answer is that is does not know,
> cannot know, because this is a feature that is supposed
> to work for any kind of example, an the parser cannot
> know all possible syntaxes :-)
> 
> This idea is to make this work in a heuristic way, by using something
> that is unlikely enough to occur in real code.
> 
> You are right that what I am using might be too
> dangerous for emacs lisp or other lisp dialects, and
> it could also show up in other languages like C.
> 
> What would be safer?
> 
>   <<name>>    like the other Org-mode targets?  That would make sense.
>               Does anyone know a language where this would be used
>               in real life?  It would make it harder to write about
>               Org-mode, though.
> 
> Or do we need another option, so that, if needed, we could switch do
> a different syntax?

Is a good work around not to simply supply the marker inside an inline 
comment, e.g.

,----
| '(a
|   ((b c) d)
|   (((e) f))   ;; ((def))
|   g)
`----

The advantage to this approach is that you can keep your code 
executable, which is really nice if you're writing documentation and 
want to be able to make sure the code always runs and is never broken. 
This solution seems to be more sensible than supporting different link 
markers etc... though the <<def>> does seem more consistent.

It might even be possible to link the ((def)) to the comment that 
describes it, so the links between code and comments are bidirectional.

Just some food for thought! :-)

R.

Re: Re: Release 6.17

[Tom Breton (Tehom)]

>
> On Jan 4, 2009, at 3:33 PM, Steven E. Harris wrote:
> [...]
>> Without knowing what the enclosing `quote' form means, how do know
>> that
>> "((def))" is not part of it?
>
> Hi Steven,
>
> good question, and the answer is that is does not know,
> cannot know, because this is a feature that is supposed
> to work for any kind of example, an the parser cannot
> know all possible syntaxes :-)
>
> This idea is to make this work in a heuristic way, by using something
> that is unlikely enough to occur in real code.
>
> You are right that what I am using might be too
> dangerous for emacs lisp or other lisp dialects, and
> it could also show up in other languages like C.
>
> What would be safer? [...]


Perhaps it would make sense to let the syntax vary by source language. 
Like, elisp could have something like ;;((def))\n and C something like
/*((def))*/.

Tom Breton

Re: Re: Release 6.17

[Samuel Wales]

I have a reply under the subject, "extensible syntax".

One possibility is this: if the syntax exists in a given language
(fairly unlikely), then you simply escape like this: \c = c for all c
(including \ itself).

-- 
For personal gain, myalgic encephalomyelitis denialists are knowingly
causing further suffering and death by grossly corrupting science.  Do
you care about the world?
http://www.meactionuk.org.uk/What_Is_ME_What_Is_CFS.htm