Error with fill-paragraph in my own major mode

Error with fill-paragraph in my own major mode

[Stefan Kamphausen]

Hi,

a good while back (actually I started this in 2004) I wrote a major
mode to write articles for the (German) linux magazin ([1]).  This
major mode can be found on my website at

http://www.skamphausen.de/cgi-bin/ska/linmag-mode

Today it has been brought to my attention that there is an error when
one tries to fill a paragraph (M-q) using that major mode in Gnu Emacs
22.  Back then I wrote that mode in XEmacs, today I seem to be a Gnu
Emacs user so I'd like to fix that issue.

Hitting M-q leads to an error
Args out of range: "", -1, 0

Using debug-on-error and edebug I could track that error down to the
following code in function fill-comment-paragraph in fill.el:

  (if (string-match comment-start-skip (concat "\0" commark "a"))
      (concat "[ \t]*" (regexp-quote commark)
              ;; Make sure we only match comments that
              ;; use the exact same comment marker.
              "[^" (substring commark -1) "]")
                   ^^^^^^^^^^^^^^^^^^^^^^

I tried to understand the workings there but to no avail and I would
appreciate any pointers to what I may be doing wrong in my major mode.

To reproduce:

* emacs -q
* Load linmag-mode.el (M-x load-file)
* Save the sample article from below ([2]) in a file with suffix
  .linmag
* Open the file, the buffer should be in linmag-mode.
* Move to the lorem ispum text
* Hit M-q.
* Voila.



Regards,
stefan



Footnotes: 
[1]  http://www.linux-magazin.de/
     The format is described (in German) on
     http://www.linux-magazin.de/heft_abo/autor_werden/format

[2]  Sample article text:
-snip-----------------------------------------------------------------
@#: ==================================================================
@#: ==                              HEAD                            ==
@#: ==================================================================
@R:Sample Rubric
@SW:Keyword

@D:Kicker
@T:This is the Title

@V: This is the lead text, kind of a teaser.
@A: The Author

@#: ==================================================================
@#:                                TEXT                             ==
@#: ==================================================================

@L: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Mauris
non eros. Phasellus mattis. Nunc libero enim, condimentum non, mollis
vitae, facilisis in, urna. Vivamus sit amet sem vitae est tempor
imperdiet. Cras quam odio, fermentum condimentum, sodales ac,
facilisis quis, odio. Etiam cursus sem ac lacus. Vestibulum lobortis,
nulla id pellentesque pharetra, sapien nunc commodo leo, eget
fermentum odio orci non leo. Sed dictum, nisi sit amet mattis
ullamcorper, eros elit lobortis purus, ut blandit tortor lacus ut
orci. Mauris tincidunt metus at purus. Quisque commodo. Aliquam magna
sapien, rutrum sit amet, interdum ut, vehicula id, orci. 
-snip-----------------------------------------------------------------


-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Error with fill-paragraph in my own major mode

[Andreas Röhler]

Am Mittwoch, 30. Januar 2008 17:41 schrieb Stefan Kamphausen:
> Hi,
>
> a good while back (actually I started this in 2004) I wrote a major
> mode to write articles for the (German) linux magazin ([1]).  This
> major mode can be found on my website at
>
> http://www.skamphausen.de/cgi-bin/ska/linmag-mode
>
> Today it has been brought to my attention that there is an error when
> one tries to fill a paragraph (M-q) using that major mode in Gnu Emacs
> 22.  Back then I wrote that mode in XEmacs, today I seem to be a Gnu
> Emacs user so I'd like to fix that issue.
>
> Hitting M-q leads to an error
> Args out of range: "", -1, 0
>
> Using debug-on-error and edebug I could track that error down to the
> following code in function fill-comment-paragraph in fill.el:
>
>   (if (string-match comment-start-skip (concat "\0" commark "a"))
>       (concat "[ \t]*" (regexp-quote commark)
>               ;; Make sure we only match comments that
>               ;; use the exact same comment marker.
>               "[^" (substring commark -1) "]")
>                    ^^^^^^^^^^^^^^^^^^^^^^
>
> I tried to understand the workings there but to no avail and I would
> appreciate any pointers to what I may be doing wrong in my major mode.
>
> To reproduce:
>
> * emacs -q
> * Load linmag-mode.el (M-x load-file)
> * Save the sample article from below ([2]) in a file with suffix
>   .linmag
> * Open the file, the buffer should be in linmag-mode.
> * Move to the lorem ispum text
> * Hit M-q.
> * Voila.
>
>
>
> Regards,
> stefan
>
...


Seems a clash with your comment-start setting.

Error occurs if form in line 871 from
`fill-comment-paragraph'

         "[^" (substring commark -1) "]")

is called.

A comment there says

,----
| 
|  The specialized regexp only works for "normal" comment
|               ;; syntax, not for Texinfo's "@c" (w
`----



BTW, when substring's `from' was set to 0, filling
worked. However, didn't check the results further,
other nasty things may happen than.

Andreas Röhler

Re: Error with fill-paragraph in my own major mode

[Stefan Monnier]

> Today it has been brought to my attention that there is an error when
> one tries to fill a paragraph (M-q) using that major mode in Gnu Emacs
> 22.  Back then I wrote that mode in XEmacs, today I seem to be a Gnu
> Emacs user so I'd like to fix that issue.

> Hitting M-q leads to an error
> Args out of range: "", -1, 0

> Using debug-on-error and edebug I could track that error down to the
> following code in function fill-comment-paragraph in fill.el:

>   (if (string-match comment-start-skip (concat "\0" commark "a"))
>       (concat "[ \t]*" (regexp-quote commark)
>               ;; Make sure we only match comments that
>               ;; use the exact same comment marker.
>               "[^" (substring commark -1) "]")
>                    ^^^^^^^^^^^^^^^^^^^^^^

The problem is the setting of comment-start-skip.
This regexp should match a "comment-start marker" (plus some optional
whitespace), where you set it to "".


        Stefan

Re: Error with fill-paragraph in my own major mode

[Stefan Kamphausen]

Hi Stefan,

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Today it has been brought to my attention that there is an error when
>> one tries to fill a paragraph (M-q) using that major mode in Gnu Emacs
>> 22.  Back then I wrote that mode in XEmacs, today I seem to be a Gnu
>> Emacs user so I'd like to fix that issue.
>
>> Hitting M-q leads to an error
>> Args out of range: "", -1, 0
>
>> Using debug-on-error and edebug I could track that error down to the
>> following code in function fill-comment-paragraph in fill.el:
>
>>   (if (string-match comment-start-skip (concat "\0" commark "a"))
>>       (concat "[ \t]*" (regexp-quote commark)
>>               ;; Make sure we only match comments that
>>               ;; use the exact same comment marker.
>>               "[^" (substring commark -1) "]")
>>                    ^^^^^^^^^^^^^^^^^^^^^^
>
> The problem is the setting of comment-start-skip.
> This regexp should match a "comment-start marker" (plus some optional
> whitespace), where you set it to "".

thanks.  I now have

 (setq comment-start "@#: "
        comment-end ""
        comment-start-skip "@#: +")


and that seems to work.  I will have to further test it, though.  The
new setting was 'inspired' by the one in texinfo-mode so I will read
some more of that code.

Nevertheless I must admit that I don't really grok that code in
fill.el.  What is that (concat "\0" commark "a") good for?

Anyway, thanks once more for your time.

Regards,
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Error with fill-paragraph in my own major mode

[Stefan Monnier]

> Nevertheless I must admit that I don't really grok that code in
> fill.el.  What is that (concat "\0" commark "a") good for?

The code in fill.el is fiendishly subtle (I should know, I wrote it).
This (concat "\0" commark "a") thingy is there to check whether or not
we can build a stricter comment-start-skip that will only match
a specific comment-marker rather than any comment marker.

E.g. we're looking at a line that starts with "//" and we want to find
all the lines that start with "//" and refill them together, but we
don't want to consider lines that start with "///" or "/*".

The `concat' builds a test case to check whether the comment-start-skip
regexp is restricting the comment starter to be in column 0 (as is the
case in Fortran), and whether a space (or similar) char is needed
between the comment-marker and the comment's content (as is the case
with Texinfo's "@c" and Basic's "Rem").

That's what the comment tries to explain:

              ;; A regexp more specialized than comment-start-skip, that only
              ;; matches the current commark rather than any valid commark.
              ;;
              ;; The specialized regexp only works for "normal" comment
              ;; syntax, not for Texinfo's "@c" (which can't be immediately
              ;; followed by word-chars) or Fortran's "C" (which needs to be
              ;; at bol), so check that comment-start-skip indeed allows the
              ;; commark to appear in the middle of the line and followed by
              ;; word chars.  The choice of "\0" and "a" is mostly arbitrary.


-- Stefan

Re: Error with fill-paragraph in my own major mode

[Stefan Kamphausen]

Hi Stefan,

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Nevertheless I must admit that I don't really grok that code in
>> fill.el.  What is that (concat "\0" commark "a") good for?
>
> The code in fill.el is fiendishly subtle (I should know, I wrote it).

"subtle" is too friendly a word ;-)

> [SNIP]
>
> That's what the comment tries to explain:

.. and what I tried to understand.

Thanks for your explanations.  I think one would have to look at all
the possible cases to fully get this.

As a (somewhat funny) sidenote: I know a special in-house language,
for which I also wrote a major mode back then) which uses # as the
comment character.  But only, and this is the funny part, if it does
not appear within the arguments of a command, where they are used
excessively.  

# valid comment
Command(arguments, man#y, of,wh#ich, may#contain, hash#es);
AnotherCommand(onearg); # valid comment, too

Try dealing with that.  I gave up ;-)

Regards
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Error with fill-paragraph in my own major mode

[Stefan Monnier]

> Thanks for your explanations.  I think one would have to look at all
> the possible cases to fully get this.

Nobody has time for that: the set of possible cases is unbounded.

> As a (somewhat funny) sidenote: I know a special in-house language,
> for which I also wrote a major mode back then) which uses # as the
> comment character.  But only, and this is the funny part, if it does
> not appear within the arguments of a command, where they are used
> excessively.  

Actually, sh comments are pretty much like that:

   % echo he#lo #world
   he#lo
   %


-- Stefan