Long lines prevent Org mode!

Long lines prevent Org mode!

[François Pinard]

Hi, Org maintainers.

This morning, while using my little org-grep tool, I got a "Stack
overflow in regexp matcher" diagnostic.  I first looked for a bug in my
things, but the problem lies in Org mode.  It cannot be activated if the
Org buffer happens to contain a long line.

In the *Scratch* buffer, I wrote this little function:

   (defun essai (n)
     (with-temp-buffer
       (insert (make-string n ?x))
       (org-mode)))

and called it with various values of N.  It works fine for small N, but
breaks when (on the machine here) N reaches 33333.

I wonder if the following article could help:

  [[https://groups.google.com/forum/#!topic/gnu.emacs.help/pxfMC5f7JHg][Stack overflow in regexp matcher - Google Groupes]]

François

P.S. Would someone be kind enough to teach me the wisdom about how to
insert Org mode fragments in an email so it is received the most legibly
possible at the other end?  Presumably, there is a Gnus way (which is
it?) yet many people do not use it.

Re: Long lines prevent Org mode!

[Nick Dokos]

François Pinard <pinard@iro.umontreal.ca> writes:

> Hi, Org maintainers.
>
> This morning, while using my little org-grep tool, I got a "Stack
> overflow in regexp matcher" diagnostic.  I first looked for a bug in my
> things, but the problem lies in Org mode.  It cannot be activated if the
> Org buffer happens to contain a long line.
>
> In the *Scratch* buffer, I wrote this little function:
>
>    (defun essai (n)
>      (with-temp-buffer
>        (insert (make-string n ?x))
>        (org-mode)))
>
> and called it with various values of N.  It works fine for small N, but
> breaks when (on the machine here) N reaches 33333.
>
> I wonder if the following article could help:
>
>   [[https://groups.google.com/forum/#!topic/gnu.emacs.help/pxfMC5f7JHg][Stack overflow in regexp matcher - Google Groupes]]
>

I think it is the same problem. The backtrace in this case is

,----
| Debugger entered--Lisp error: (error "Stack overflow in regexp matcher")
|   looking-at(".*:ARCHIVE:")
|   (if (looking-at (concat ".*:" org-archive-tag ":")) (message "%s" (substitute-command-keys "Subtree is archived and stays closed.  Use \\[org-force-cycle-archived] to cycle it anyway.")))
|   ...
|   org-mode()
|   (progn (insert (make-string n 120)) (org-mode))
|   (unwind-protect (progn (insert (make-string n 120)) (org-mode)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
|   (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert (make-string n 120)) (org-mode)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
|   (let ((temp-buffer (generate-new-buffer " *temp*"))) ...)
|   essai(33333)
|   eval((essai 33333) nil)
`----


and in fact, you can leave org out of the picture altogether: create an
empty buffer, insert 33333 x's by hand, go to the beginning and evaluate

   (looking-at ".*foo")

As Stefan says in the thread you pointed out, it's a limitation of the
emacs regexp engine.

> François
>
> P.S. Would someone be kind enough to teach me the wisdom about how to
> insert Org mode fragments in an email so it is received the most legibly
> possible at the other end?  Presumably, there is a Gnus way (which is
> it?) yet many people do not use it.
>
>

I cut and paste and then use either C-c m (bound to
message-mark-inserted-region) which produces this style:

--8<---------------cut here---------------start------------->8---
some code
--8<---------------cut here---------------end--------------->8---

or C-c q (bound to boxquote-region) which produces this style:

,----
| some quoted material
`----

I use the first one for things that a recipient might want to cut
and paste in order to experiment with them. I use the second to quote
immutable stuff (e.g. doc strings).

Of course, if your problem is that your mailer (or some mailer along the
way) mangles the content, then the safest thing to do is to put it in an
attachment or post it on some pastebin and send a link.
-- 
Nick

Re: Long lines prevent Org mode!

[François Pinard]

Nick Dokos <ndokos@gmail.com> writes:

> you can leave org out of the picture altogether: create an empty
> buffer, insert 33333 x's by hand, go to the beginning and evaluate

>    (looking-at ".*foo")

> As Stefan says in the thread you pointed out, it's a limitation of the
> emacs regexp engine.

Yes, I got this from his article.  However, it seems that "[^\nf]*foo"
would succeed, avoiding the stack explosion.  So, it is possible to
rework the regexpes around the limitation.

Surely not a fun job to do for Org regexpes, I know.  Maybe not worth
doing as very long lines are unusual, yet maybe worth doing nevertheless
for Org files because long lines are still possible.  The fact is that I
was hit by this problem in the Org buffers generated by org-grep.

However, as I am tempted to think that the problem is unlikely to be
solved soon, I tried to alleviate it.  org-grep now shortens the lines
by removing, within each line, a good part of the context between hits.
This does not guarantee that the above problem will never occur, but it
considerably decreases the chance that it occurs.

> I cut and paste and then use either C-c m (bound to
> message-mark-inserted-region) or C-c q (bound to boxquote-region)

Thanks for these hints, which I saved on keys here! :-)

> Of course, if your problem is that your mailer (or some mailer along
> the way) mangles the content, then the safest thing to do is to put it
> in an attachment or post it on some pastebin and send a link.

Sure.

My little problem was different.  If I use Org in-line markup, like
bold, italics, or links to Web sites, or even block markup like
fragments between #+BEGIN_SRC and #+END_SRC, say, I was wondering how
this is best transmitted within a message, in the text, without
resorting to an attachment.  Gnus as a reader surely has some capability
somewhere, as I think I saw messages displaying Org code nicely; I just
do not remember how it was done.  Outside Gnus, it might be more
problematic to find a useful way to have Org included.  Sending Org all
raw is sometimes a bit noisy, especially for links.

Someone recently mentioned org-mime.el, new to me, maybe this is the
proper avenue?  I should at least play and experiment with it! :-).

François

Re: Long lines prevent Org mode!

[Nick Dokos]

François Pinard <pinard@iro.umontreal.ca> writes:

> Nick Dokos <ndokos@gmail.com> writes:
>
>> you can leave org out of the picture altogether: create an empty
>> buffer, insert 33333 x's by hand, go to the beginning and evaluate
>
>>    (looking-at ".*foo")
>
>> As Stefan says in the thread you pointed out, it's a limitation of the
>> emacs regexp engine.
>
> Yes, I got this from his article.  However, it seems that "[^\nf]*foo"
> would succeed, avoiding the stack explosion.  So, it is possible to
> rework the regexpes around the limitation.
>
> Surely not a fun job to do for Org regexpes, I know.  Maybe not worth
> doing as very long lines are unusual, yet maybe worth doing nevertheless
> for Org files because long lines are still possible.  The fact is that I
> was hit by this problem in the Org buffers generated by org-grep.
>

It would be better to reimplement the regexp engine using e.g. an NDFA.
IIRC, it requires constant stack but it limits the regexps allowed. See
e.g.

        http://swtch.com/~rsc/regexp/regexp1.html

It might make sense to pair the NDFA with a backtracker. The pattern
compiler could then choose which one to use based on the regexp.

-- 
Nick

Re: Long lines prevent Org mode!

[Sebastien Vauban]

Hello Nick and François,

François Pinard wrote:
> Nick Dokos <ndokos-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>
>> I cut and paste and then use either C-c m (bound to
>> message-mark-inserted-region) or C-c q (bound to boxquote-region)
>
> Thanks for these hints, which I saved on keys here! :-)

Please note it is `C-c M-m' (bound in `message.el') for adding "cut
here" lines above and below the region.

OTOH, `C-c q' seems to be Nick's own binding.

>> Of course, if your problem is that your mailer (or some mailer along
>> the way) mangles the content, then the safest thing to do is to put it
>> in an attachment or post it on some pastebin and send a link.
>
> Sure.
>
> My little problem was different.  If I use Org in-line markup, like
> bold, italics, or links to Web sites, or even block markup like
> fragments between #+BEGIN_SRC and #+END_SRC, say, I was wondering how
> this is best transmitted within a message, in the text, without
> resorting to an attachment.  Gnus as a reader surely has some capability
> somewhere, as I think I saw messages displaying Org code nicely; I just
> do not remember how it was done.  Outside Gnus, it might be more
> problematic to find a useful way to have Org included.  Sending Org all
> raw is sometimes a bit noisy, especially for links.

Using `C-c M-m' will add "cut here" lines, and Gnus will highlight the
code inside it with one common color: so, no syntax highlighting
depending on the language.

Using `C-c C-v C-d' to demarcate your region as code, and give the
language, you'll get a nice highlighting of the code inside Gnus (for
the readers): that was on my first real patch for Org (in fact, for
Gnus), back in November 2010 or so.

> Someone recently mentioned org-mime.el, new to me, maybe this is the
> proper avenue?  I should at least play and experiment with it! :-).

This is mainly useful for sending HTML-export (or Org-export) of an Org
section.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Long lines prevent Org mode!

[Nick Dokos]

"Sebastien Vauban" <sva-news@mygooglest.com>
writes:

>>> I cut and paste and then use either C-c m (bound to
>>> message-mark-inserted-region) or C-c q (bound to boxquote-region)
>>
>> Thanks for these hints, which I saved on keys here! :-)
>
> Please note it is `C-c M-m' (bound in `message.el') for adding "cut
> here" lines above and below the region.
>
> OTOH, `C-c q' seems to be Nick's own binding.
>

Indeed C-C M-m is the "official" binding in message.el. The ones I
posted are both my own private bindings. Sorry for the confusion and
thanks for correcting me.
-- 
Nick

Re: Long lines prevent Org mode!

[François Pinard]

"Sebastien Vauban" <sva-news-D0wtAvR13HarG/iDocfnWg@public.gmane.org>
writes:

> Using `C-c C-v C-d' to demarcate your region as code, and give the
> language, you'll get a nice highlighting of the code inside Gnus

You mean that demarcated code

#+BEGIN_SRC elisp
((lambda (x y) (+ x y))(length "abc") 2)
#+END_SRC

will be automatically shown nicely by Gnus?  Trying it right in this
message! :-)

> (for the readers): that was on my first real patch for Org (in fact,
> for Gnus), back in November 2010 or so.

A nice one then!  Thanks!

François

Re: Long lines prevent Org mode!

[Sebastien Vauban]

Hi François,

François Pinard wrote:
> "Sebastien Vauban" writes:
>
>> Using `C-c C-v C-d' to demarcate your region as code, and give the
>> language, you'll get a nice highlighting of the code inside Gnus
>
> You mean that demarcated code
>
> #+BEGIN_SRC elisp
> ((lambda (x y) (+ x y))(length "abc") 2)
> #+END_SRC
>
> will be automatically shown nicely by Gnus?

Yes, I do!

> Trying it right in this message! :-)

Yep: see http://screencast.com/t/cgkg1gg24Wpg...

>> (for the readers): that was on my first real patch for Org (in fact,
>> for Gnus), back in November 2010 or so.
>
> A nice one then!  Thanks!

Thanks.

Best regards,
  Seb

-- 
Sebastien Vauban