narrow-to-here-document

narrow-to-here-document

[Masatake YAMATO]

I've written a new function, narrot-to-here-document.  This might be
useful to edit here document region in a script file.

How to you think to add this feature to emacs?

The implementation has two parts.

1. hdoc.el --- Generic here document support
2. sh-script.el and cperl-mode.el adapters

hdoc.el requires two function implementations, here-document-region
and here-document-marker for each major modes in which
narrot-to-here-document is enabled.

I've written the functions for sh-script.el and cperl-mode.el.

;;; hdoc.el --- Generic here document support

;; Copyright (C) 2003 ???
;;  

;; Author: Masatake YAMATO<jet@gyve.org> 

;;; Code:

(defvar here-document-region nil
  "If non-nil, function for `narrow-to-here-document' to call.
This is used to find the beginning and end of the here document region in a
script file (see `narrow-to-here-document'). Major modes can define this
to handle the mode's needs.

The function should return a cons cell; its car is the position of 
beginning and cdr is that of end if the point is in a here document region. 
The function should return `nil' if the point is out of a here document region. ")

(defvar here-document-marker nil
  "If non-nil, function for `narrow-to-here-document' and `make-indirect-buffer-for-here-document' to call.
This is used to find the marker of the here document region in a
script file (see `make-indirect-buffer-for-here-document' and 
`narrow-to-here-document').  A string \"EOF\" is used widely as the marker 
but other string is also OK. Major modes can define this to handle the 
mode's needs.

The function should return a marker string if the point is in a here document 
region. Else it should return `nil'. ")

(define-key ctl-x-map "nh" 'narrow-to-here-document)
(define-key ctl-x-4-map "h" 'make-indirect-buffer-for-here-document)

(defun narrow-to-here-document (&optional change-major-mode)
  "Make text outside current here document region invisible.
Local variable `here-document-region' must be defined in the major 
mode to use this function.

If CHANGE-MAJOR-MODE is non-nil and local variable `here-document-marker' 
is defined in the major mode, major mode for the current buffer will be
switched; new major mode is selected according to the result of 
`here-document-marker'. For example if `here-document-marker' returns
\"EOF.c\", `c-mode' will be selected. Major mode selection is done
by `set-auto-mode'. In `set-auto-mode', `here-document-marker' is used
as the buffer's file name temporally.

To make text outside of the region visible, use \[widen]."
  (interactive "P")
  (unless here-document-region
    (error "%s: `%s' is not supported in this major mode." 
	   mode-name
	   this-command))
  (let* ((marker (if here-document-marker (funcall here-document-marker)))
	 (r      (funcall here-document-region))
	 (b      (car r))
	 (e      (cdr r)))
    (unless r
      (error "Cannot find here document region"))
    (narrow-to-region b e)
    (if (and change-major-mode marker)
	(let ((buffer-file-name marker))
	  (set-auto-mode)))))

(defun make-indirect-buffer-for-here-document (&optional change-major-mode)
  "Create and return an indirect buffer for current here document
region.  In the new indirect buffer, the text outside current here
document region is invisible(narrowed). Local variables `here-document-region' 
and `here-document-marker' must be defined in the major mode to use this function.

CHANGE-MAJOR-MODE is passed to `narrow-to-here-document'.
See also `narrow-to-here-document'.

The new indirect buffer name will be \"original-buffername<<here-document-marker\"."
  (interactive "P")
  (unless here-document-marker
    (error "%s: `%s' is not supported in this major mode." 
	   mode-name
	   this-command))
  (let  ((marker  (funcall here-document-marker))
	 (p       (point))
	 buf 
	 bufname)
    (unless marker
      (error "Cannot find here document marker"))
    (setq bufname (format "%s<<%s" (buffer-name) marker))
    (setq buf (make-indirect-buffer (current-buffer) bufname t))
    (pop-to-buffer buf)
    (goto-char p)
    (narrow-to-here-document change-major-mode)))

(provide 'hdoc)
;; hdoc.el ends here

Index: lisp/progmodes/sh-script.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/progmodes/sh-script.el,v
retrieving revision 1.124
diff -u -r1.124 sh-script.el
--- lisp/progmodes/sh-script.el	16 Jun 2003 21:41:18 -0000	1.124
+++ lisp/progmodes/sh-script.el	20 Jun 2003 13:48:45 -0000
@@ -1332,6 +1332,8 @@
 	skeleton-newline-indent-rigidly t
 	sh-indent-supported-here nil)
   (set (make-local-variable 'parse-sexp-ignore-comments) t)
+  (set (make-local-variable 'here-document-region) 'sh-here-document-region)
+  (set (make-local-variable 'here-document-marker) 'sh-here-document-marker)
   ;; Parse or insert magic number for exec, and set all variables depending
   ;; on the shell thus determined.
   (let ((interpreter
@@ -3532,6 +3534,27 @@
   (interactive)
   (if (re-search-forward sh-end-of-command nil t)
       (goto-char (match-end 1))))
+
+(defun sh-here-document-region ()
+  "Return here document region around the point.
+Return nil if the point is not in a here document region."
+  (let ((pos (point)))
+    (if (eq 'sh-heredoc-face  (get-text-property pos 'face))
+	(let ((b (previous-single-property-change pos 'face))
+	      (e (next-single-property-change pos 'face)))
+	  (setq b (if b (1+ b) (point-min)))
+	  (setq e (if e (1- e) (point-max)))
+	  (cons b e)))))
+
+(defun sh-here-document-marker ()
+  "Return the marker of here document region around the point.
+Return nil if the point is not in a here document region. 
+'EOF' is a typical marker. "
+  (if (eq 'sh-heredoc-face  (get-text-property (point) 'face))
+      (save-excursion
+	(save-match-data
+	  (if (re-search-backward sh-here-doc-re (point-min) t) 
+	      (match-string 1)))))))
 
 (provide 'sh-script)
 
Index: lisp/progmodes/cperl-mode.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/progmodes/cperl-mode.el,v
retrieving revision 1.46
diff -u -r1.46 cperl-mode.el
--- lisp/progmodes/cperl-mode.el	6 May 2003 17:43:22 -0000	1.46
+++ lisp/progmodes/cperl-mode.el	20 Jun 2003 13:48:48 -0000
@@ -1540,6 +1540,10 @@
 	    (eval '(cperl-old-auto-fill-mode arg)) ; Avoid a warning
 	    (and auto-fill-function (memq major-mode '(perl-mode cperl-mode))
 		 (setq auto-fill-function 'cperl-do-auto-fill))))))
+  (make-local-variable 'here-document-region)
+  (setq here-document-region 'cperl-here-doc-region)
+  (make-local-variable 'here-document-marker)
+  (setq here-document-marker 'cperl-here-doc-delim)
   (if (cperl-enable-font-lock)
       (if (cperl-val 'cperl-font-lock)
 	  (progn (or cperl-faces-init (cperl-init-faces))
@@ -7268,6 +7272,30 @@
     (string-match ":\\s *\\([0-9.]+\\)" v)
     (substring v (match-beginning 1) (match-end 1)))
   "Version of IZ-supported CPerl package this file is based on.")
+
+(defun cperl-here-doc-region ()
+  "Return here document region around the point.
+Return nil if the point is not in a here document region."
+  (let ((pos (point)))
+    (if (eq 'here-doc  (get-text-property pos 'syntax-type))
+	(let ((b (previous-single-property-change pos 'syntax-type))
+	      (e (next-single-property-change pos 'syntax-type)))
+	  (setq b (if b b (point-min)))
+	  (setq e (if e (1- e) (point-max)))
+	  (cons b e)))))
+
+(defun cperl-here-doc-delim ()
+  "Return the delimiter of here document region around the point.
+Return nil if the point is not in a here document region. 
+'EOF' is a typical delimiter. "
+  (when (eq 'here-doc  (get-text-property (point) 'syntax-type))
+    (let* ((b (next-single-property-change (point) 'syntax-type))
+	   (e (if b (next-single-property-change b 'syntax-type))))      
+      (when b
+	(if e
+	    (buffer-substring b e)
+	  (buffer-substring b (point-max)))))))
+	  
 
 (provide 'cperl-mode)

Re: narrow-to-here-document

[Ilya Zakharevich]

On Fri, Jun 20, 2003 at 10:59:09PM +0900, Masatake YAMATO wrote:
> I've written a new function, narrot-to-here-document.  This might be
> useful to edit here document region in a script file.

> How to you think to add this feature to emacs?

> 1. hdoc.el --- Generic here document support
> 2. sh-script.el and cperl-mode.el adapters

But are there any other modes with here-docs?

I would do it like this:

a) defined a variable which keeps a function looking for a here-doc.

b) Merge your two adapters into one function (which makes a runtime
   dispatch basing on the mode - and croaks if the mode is not known);
   put the function into the variable.

c) delegate the other work to the mode writers - when a mode supports
   an interaction with your function, it should make your variable
   buffer-local, and will put its helper function into the variable.

Hope this helps,
Ilya

Re: narrow-to-here-document

[Masatake YAMATO]

Thank you for your comment.

> > I've written a new function, narrot-to-here-document.  This might be
> > useful to edit here document region in a script file.
> 
> > How to you think to add this feature to emacs?
> 
> > 1. hdoc.el --- Generic here document support
> > 2. sh-script.el and cperl-mode.el adapters
> 
> But are there any other modes with here-docs?

As far as I know ruby lang has "here document".
ruby-mode doesn't handle here document region now.

> I would do it like this:
> 
> a) defined a variable which keeps a function looking for a here-doc.
> 
> b) Merge your two adapters into one function (which makes a runtime
>    dispatch basing on the mode - and croaks if the mode is not known);
>    put the function into the variable.
> 
> c) delegate the other work to the mode writers - when a mode supports
>    an interaction with your function, it should make your variable
>    buffer-local, and will put its helper function into the variable.

a) and c) were done in the last patch I posted here.
To cover b), I'll move the adapters to hdoc.el itself.
Now only hdoc is needed. the patch for sh-script.el and cperl-mode.el
are not needed.

;;; hdoc.el --- Generic here document support

;; Copyright (C) 2003
;; 

;; Author: Masatake YAMATO<jet@gyve.org> 

;;; Commentary:

;;; Code:

(defvar here-document-region 'here-document-region-default
  "If non-nil, function for `narrow-to-here-document' to call.
This is used to find the beginning and end of the here document region in a
script file (see `narrow-to-here-document'). Major modes can define this
to handle the mode's needs.

The function should return a cons cell; its car is the position of 
beginning and cdr is that of end if the point is in a here document region. 
The function should return `nil' if the point is out of a here document region. ")

(defvar here-document-marker 'here-document-marker-default
  "If non-nil, function for `narrow-to-here-document' and `make-indirect-buffer-for-here-document' to call.
This is used to find the marker of the here document region in a
script file (see `make-indirect-buffer-for-here-document' and 
`narrow-to-here-document').  A string \"EOF\" is used widely as the marker 
but other string is also OK. Major modes can define this to handle the 
mode's needs.

The function should return a marker string if the point is in a here document 
region. Else it should return `nil'. ")

(define-key ctl-x-map "nh" 'narrow-to-here-document)
(define-key ctl-x-4-map "h" 'make-indirect-buffer-for-here-document)

(defun narrow-to-here-document (&optional change-major-mode)
  "Make text outside current here document region invisible.
Local variable `here-document-region' must be defined in the major 
mode to use this function.

If CHANGE-MAJOR-MODE is non-nil and local variable `here-document-marker' 
is defined in the major mode, major mode for the current buffer will be
switched; new major mode is selected according to the result of 
`here-document-marker'. For example if `here-document-marker' returns
\"EOF.c\", `c-mode' will be selected. Major mode selection is done
by `set-auto-mode'. In `set-auto-mode', `here-document-marker' is used
as the buffer's file name temporally.

To make text outside of the region visible, use \[widen]."
  (interactive "P")
  (let* ((marker (funcall here-document-marker))
	 (r      (funcall here-document-region))
	 (b      (car r))
	 (e      (cdr r)))
    (unless r
      (error "Cannot find here document region"))
    (narrow-to-region b e)
    (if (and change-major-mode marker)
	(let ((buffer-file-name marker))
	  (set-auto-mode)))))

(defun make-indirect-buffer-for-here-document (&optional change-major-mode)
  "Create and return an indirect buffer for current here document
region.  In the new indirect buffer, the text outside current here
document region is invisible(narrowed). Local variables `here-document-region' 
and `here-document-marker' must be defined in the major mode to use this function.

CHANGE-MAJOR-MODE is passed to `narrow-to-here-document'.
See also `narrow-to-here-document'.

The new indirect buffer name will be \"original-buffername<<here-document-marker\"."
  (interactive "P")
  (let  ((marker  (funcall here-document-marker))
	 (p       (point))
	 buf 
	 bufname)
    (unless marker
      (error "Cannot find here document marker"))
    (setq bufname (format "%s<<%s" (buffer-name) marker))
    (setq buf (make-indirect-buffer (current-buffer) bufname t))
    (pop-to-buffer buf)
    (goto-char p)
    (narrow-to-here-document change-major-mode)))

(defun here-document-region-default ()
  (cond 
   ((eq major-mode 'sh-mode)
    (sh-here-document-region))
   ((eq major-mode 'cperl-mode)
    (cperl-here-doc-region))
   (t
    (error "%s: `%s' is not supported in this major mode." 
	   mode-name
	   this-command))))

(defun here-document-marker-default ()
  (cond 
   ((eq major-mode 'sh-mode)
    (sh-here-document-marker))
   ((eq major-mode 'cperl-mode)
    (cperl-here-doc-delim))
   (t
    (error "%s: `%s' is not supported in this major mode." 
	   mode-name
	   this-command))))

;;
;; sh-mode
;;
(defun sh-here-document-region ()
  "Return here document region around the point.
Return nil if the point is not in a here document region."
  (let ((pos (point)))
    (if (eq 'sh-heredoc-face  (get-text-property pos 'face))
	(let ((b (previous-single-property-change pos 'face))
	      (e (next-single-property-change pos 'face)))
	  (setq b (if b (1+ b) (point-min)))
	  (setq e (if e (1- e) (point-max)))
	  (cons b e)))))

(defun sh-here-document-marker ()
  "Return the marker of here document region around the point.
Return nil if the point is not in a here document region. 
'EOF' is a typical marker. "
  (if (eq 'sh-heredoc-face  (get-text-property (point) 'face))
      (save-excursion
	(save-match-data
	  (if (re-search-backward sh-here-doc-re (point-min) t) 
	      (match-string 1))))))
;;
;; cperl-mode
;;
(defun cperl-here-doc-region ()
  "Return here document region around the point.
Return nil if the point is not in a here document region."
  (let ((pos (point)))
    (if (eq 'here-doc  (get-text-property pos 'syntax-type))
	(let ((b (previous-single-property-change pos 'syntax-type))
	      (e (next-single-property-change pos 'syntax-type)))
	  (setq b (if b b (point-min)))
	  (setq e (if e (1- e) (point-max)))
	  (cons b e)))))

(defun cperl-here-doc-delim ()
  "Return the delimiter of here document region around the point.
Return nil if the point is not in a here document region. 
'EOF' is a typical delimiter. "
  (when (eq 'here-doc  (get-text-property (point) 'syntax-type))
    (let* ((b (next-single-property-change (point) 'syntax-type))
	   (e (if b (next-single-property-change b 'syntax-type))))      
      (when b
	(if e
	    (buffer-substring b e)
	  (buffer-substring b (point-max)))))))

(provide 'hdoc)
;; hdoc.el ends here

Re: narrow-to-here-document

[Stephen J. Turnbull]

>>>>> "Masatake" == Masatake YAMATO <jet@gyve.org> writes:

    >> But are there any other modes with here-docs?

    Masatake> As far as I know ruby lang has "here document".
    Masatake> ruby-mode doesn't handle here document region now.

Any language (Python or Emacs-Lisp, for examples) with multi-line
strings effectively has here-docs.  My version of Emacs doesn't
support either narrow-to-string or mark-string AFAICT, so maybe this
would be a good time to add those, too.



-- 
Institute of Policy and Planning Sciences     http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: narrow-to-here-document

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> >>>>> "Masatake" == Masatake YAMATO <jet@gyve.org> writes:
> 
>     >> But are there any other modes with here-docs?
> 
>     Masatake> As far as I know ruby lang has "here document".
>     Masatake> ruby-mode doesn't handle here document region now.
> 
> Any language (Python or Emacs-Lisp, for examples) with multi-line
> strings effectively has here-docs.  My version of Emacs doesn't
> support either narrow-to-string or mark-string AFAICT, so maybe this
> would be a good time to add those, too.

TeX is one particular contestant.  There are forms of "Literate
Programming" with a heavy mixture of code and comments, but every
reference manual contains C code passages, shell scripts and similar
stuff.  An excerpt:

<snip>
%
That way, several specialized sort routines may be defined with
different names in the same module.  C++ templates the ugly way.

\begin{lstlisting}{listsort.c}
listptr *listsort(listptr *head,   /* pointer to head of list */
                  size_t n,    /* number of elements to sort */
                  size_t linkoffs, /* offset of link in structure */
                  int (*compar)(const void *, const void *)
                                   /* comparison routine */
               )
{
\end{lstlisting}
%
The body will have to start with a few declarations.

\begin{lstlisting}{listsort.c}
  size_t underpow,n1,n2,bitrev;
  int lev, kmax;

</snip>

You get the drift.  And inside of those passages, indentation and
editing should reign that is governed by C mode.  In essence, what
would be best is if editing effectively could happen in a virtual
buffer constituted of all the code passages in C, with C indentation
over the continued passages, and keybindings.

Literate programming can even mean TeX immersed into TeX: so-called
dtx documents look like this:

%\begin{macro}{\PreviewSnarfEnvironment}
%  This is a nuisance since we have to advise \emph{both} the
%  environment and its end.
%    \begin{macrocode}
\newcommand{\PreviewSnarfEnvironment}[2][]{%
  \expandafter\pr@advise
   \csname #2\endcsname{\pr@snarfafter#1\pr@endparse}%
 \expandafter\pr@advise
   \csname end#2\endcsname{\endgroup}}
%</!active>
%    \end{macrocode}
%\end{macro}
%\begin{macro}{\pr@snarfafter}
%\begin{macro}{\pr@startsnarf}
%  Ok, this looks complicated, but we have to start a group in order
%  to be able to hook \cmd{\pr@endbox} into the game only when

Again, indentation and other context (environment open and closing,
something akin to verbose brace matching) should effectually happen in
virtual buffers that contain only a selected subset of the file in
question.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: narrow-to-here-document

[Ilya Zakharevich]

On Sat, Jun 21, 2003 at 10:01:31AM +0200, David Kastrup wrote:
> TeX is one particular contestant.  There are forms of "Literate
> Programming" with a heavy mixture of code and comments, but every
> reference manual contains C code passages, shell scripts and similar
> stuff.  An excerpt:

TeX has no HERE documents.  Period.

There are *programming styles* in TeX (as far as can call fighting
with TeX deficiencies programming ;-[) which provide a poor-man
approximations to HERE documents.  However, the target of the
discussion is how to add mode-specific functions which find boundaries
of the given HERE documents.

The problem with TeX is that given no standard way to define HERE
docs, one cannot define such a function - without loading of some
highly specialized minor modes.  Hmm, maybe one can do as I did with
indentation styles in Perl: they may be loaded from a menu, so it is
easy to change it on a document-by-document base.

Ilya

Re: narrow-to-here-document

[David Kastrup]

Ilya Zakharevich <ilya@Math.Berkeley.EDU> writes:

> On Sat, Jun 21, 2003 at 10:01:31AM +0200, David Kastrup wrote:
> > TeX is one particular contestant.  There are forms of "Literate
> > Programming" with a heavy mixture of code and comments, but every
> > reference manual contains C code passages, shell scripts and similar
> > stuff.  An excerpt:
> 
> TeX has no HERE documents.  Period.

An interesting style of discussion, given that I provided examples.

> There are *programming styles* in TeX (as far as can call fighting
> with TeX deficiencies programming ;-[) which provide a poor-man
> approximations to HERE documents.

The examples given were not concerned with programming in TeX.  They
demonstrated various ways to embed program code into documents
typeset with TeX.

> However, the target of the discussion is how to add mode-specific
> functions which find boundaries of the given HERE documents.
> 
> The problem with TeX is that given no standard way to define HERE
> docs, one cannot define such a function

listings.sty provides one of various ways for including documents in
particular languages, and AUCTeX has hooks for packages.  It could
easily use one for listings.sty.

Knuth's WEB system of structured documentation (aka Literate
Programming) has well defined semantics for including code pieces in
Pascal or (for CWEB) C in what amounts to a TeX document

noweb has well-defined ways to do the same (illustrated source code
in some programming language, using TeX for the typesetting).

So I think that the potential applicability of such functionality
can't be completely summarized with "TeX has no HERE documents.
Period."  For the sake of discussing the matter at hand, I don't see
that it would make sense not to consider the various TeX-based
applications here, too.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: narrow-to-here-document

[Ilya Zakharevich]

On Sat, Jun 21, 2003 at 04:48:09PM +0200, David Kastrup wrote:
> > > TeX is one particular contestant.  There are forms of "Literate
> > > Programming" with a heavy mixture of code and comments, but every
> > > reference manual contains C code passages, shell scripts and similar
> > > stuff.  An excerpt:
> > 
> > TeX has no HERE documents.  Period.
> 
> An interesting style of discussion, given that I provided examples.

> > There are *programming styles* in TeX (as far as can call fighting
> > with TeX deficiencies programming ;-[) which provide a poor-man
> > approximations to HERE documents.

You did not provide any example of HERE-doc in TeX (only of constructs
which may be used as substitutes for a language feature); as I said,
there is no such thing.  But this is pointless to discuss; what
matters is how things are used, and given an engine which detects
loaded packages, one can recognize the substitutes instead.

> So I think that the potential applicability of such functionality
> can't be completely summarized with "TeX has no HERE documents.
> Period."  For the sake of discussing the matter at hand, I don't see
> that it would make sense not to consider the various TeX-based
> applications here, too.

I agreed with this in the end of the previous posting already. ;-)

Yours,
Ilya

Re: narrow-to-here-document

[Kevin Rodgers]

Masatake YAMATO wrote:

> Thank you for your comment.
> 
> 
>>>I've written a new function, narrot-to-here-document.  This might be
>>>useful to edit here document region in a script file.
>>>
>>>How to you think to add this feature to emacs?
>>>
>>>1. hdoc.el --- Generic here document support
>>>2. sh-script.el and cperl-mode.el adapters
>>>
>>But are there any other modes with here-docs?

Would INCLUDE/IGNORE/CDATA marked sections in SGML/XML/HTML qualify?

<![CDATA[
No <tag>s or &entity; references are recognized here.
]]>

-- 
<a href="mailto:&lt;kevin.rodgers&#64;ihs.com&gt;">Kevin Rodgers</a>

Re: narrow-to-here-document

[Masatake YAMATO]

I've reflected suggestions to my code.

Richard Stallman: 
1. The first line of a doc string is supposed to be complete in itself.
2. "here-document" is too long. 

Kevin Rodgers:
3. 
    Would INCLUDE/IGNORE/CDATA marked sections in SGML/XML/HTML qualify?

    <![CDATA[
    No <tag>s or &entity; references are recognized here.
    ]]>

About narrow-to-string, I do nothing yet.
I wonder what should I do.

Regards,
Masatake
p.s. Next time, I'll use diff&patch. 

;;; hdoc.el --- Generic here document support

;; Copyright (C) 2003
;;  

;; Author: Masatake YAMATO<jet@gyve.org> 

;;; Commentary:

;;; Code:

(defvar hdoc-region 'here-document-region-default
  "If non-nil, function for `narrow-to-hdoc' to call.
This is used to find the beginning and end of the here document region in a
script file (see `narrow-to-hdoc'). Major modes can define this
to handle the mode's needs.

The function should return a cons cell; its car is the position of 
beginning and cdr is that of end if the point is in a here document region. 
The function should return `nil' if the point is out of a here document region. ")

(defvar hdoc-marker 'here-document-marker-default
  "If non-nil, function for `narrow-to-hdoc' and `make-indirect-buffer-for-hdoc' to call.
This is used to find the marker of the here document region in a
script file (see `make-indirect-buffer-for-hdoc' and 
`narrow-to-hdoc').  A string \"EOF\" is used widely as the marker 
but other string is also OK. Major modes can define this to handle the 
mode's needs.

The function should return a marker string if the point is in a here document 
region. Else it should return `nil'. ")

(define-key ctl-x-map "nh" 'narrow-to-hdoc)
(define-key ctl-x-4-map "h" 'make-indirect-buffer-for-hdoc)

(defun narrow-to-hdoc (&optional change-major-mode)
  "Make text outside current here document region invisible.
Local variable `hdoc-region' must be defined in the major 
mode to use this function.

If CHANGE-MAJOR-MODE is non-nil and local variable `hdoc-marker' 
is defined in the major mode, major mode for the current buffer will be
switched; new major mode is selected according to the result of 
`hdoc-marker'. For example if `hdoc-marker' returns
\"EOF.c\", `c-mode' will be selected. Major mode selection is done
by `set-auto-mode'. In `set-auto-mode', `hdoc-marker' is used
as the buffer's file name temporally.

To make text outside of the region visible, use \[widen]."
  (interactive "P")
  (let* ((marker (funcall hdoc-marker))
	 (r      (funcall hdoc-region))
	 (b      (car r))
	 (e      (cdr r)))
    (unless r
      (error "Cannot find here document region"))
    (narrow-to-region b e)
    (if (and change-major-mode marker)
	(let ((buffer-file-name marker))
	  (set-auto-mode)))))

(defun make-indirect-buffer-for-hdoc (&optional change-major-mode)
  "Create and return an indirect buffer for current here document region.  
In the new indirect buffer, the text outside current here
document region is invisible(narrowed). Local variables `hdoc-region' 
and `hdoc-marker' must be defined in the major mode to use this function.

CHANGE-MAJOR-MODE is passed to `narrow-to-hdoc'.
See also `narrow-to-hdoc'.

The new indirect buffer name will be \"original-buffername<<hdoc-marker\"."
  (interactive "P")
  (let  ((marker  (funcall hdoc-marker))
	 (p       (point))
	 buf 
	 bufname)
    (unless marker
      (error "Cannot find here document marker"))
    (setq bufname (format "%s<<%s" (buffer-name) marker))
    (setq buf (make-indirect-buffer (current-buffer) bufname t))
    (pop-to-buffer buf)
    (goto-char p)
    (narrow-to-hdoc change-major-mode)))

(defun here-document-region-default ()
  (cond 
   ((eq major-mode 'sh-mode)
    (sh-here-document-region))
   ((eq major-mode 'cperl-mode)
    (cperl-here-doc-region))
   ((or (eq major-mode 'html-mode)
	(eq major-mode 'sgml-mode))
    (sgml-here-document-region))
   (t
    (error "%s: `%s' is not supported in this major mode." 
	   mode-name
	   this-command))))

(defun here-document-marker-default ()
  (cond 
   ((eq major-mode 'sh-mode)
    (sh-here-document-marker))
   ((eq major-mode 'cperl-mode)
    (cperl-here-doc-delim))
   ((or (eq major-mode 'html-mode)
	(eq major-mode 'sgml-mode))
    (sgml-here-document-marker))
   (t
    (error "%s: `%s' is not supported in this major mode." 
	   mode-name
	   this-command))))

;;
;; sh-mode
;;
(defun sh-here-document-region ()
  "Return here document region around the point.
Return nil if the point is not in a here document region."
  (let ((pos (point)))
    (if (eq 'sh-heredoc-face  (get-text-property pos 'face))
	(let ((b (previous-single-property-change pos 'face))
	      (e (next-single-property-change pos 'face)))
	  (setq b (if b (1+ b) (point-min)))
	  (setq e (if e (1- e) (point-max)))
	  (cons b e)))))

(defun sh-here-document-marker ()
  "Return the marker of here document region around the point.
Return nil if the point is not in a here document region. 
'EOF' is a typical marker. "
  (if (eq 'sh-heredoc-face  (get-text-property (point) 'face))
      (save-excursion
	(save-match-data
	  (if (re-search-backward sh-here-doc-re (point-min) t) 
	      (match-string 1))))))
;;
;; cperl-mode
;;
(defun cperl-here-doc-region ()
  "Return here document region around the point.
Return nil if the point is not in a here document region."
  (let ((pos (point)))
    (if (eq 'here-doc  (get-text-property pos 'syntax-type))
	(let ((b (previous-single-property-change pos 'syntax-type))
	      (e (next-single-property-change pos 'syntax-type)))
	  (setq b (if b b (point-min)))
	  (setq e (if e (1- e) (point-max)))
	  (cons b e)))))

(defun cperl-here-doc-delim ()
  "Return the delimiter of here document region around the point.
Return nil if the point is not in a here document region. 
'EOF' is a typical delimiter. "
  (when (eq 'here-doc  (get-text-property (point) 'syntax-type))
    (let* ((b (next-single-property-change (point) 'syntax-type))
	   (e (if b (next-single-property-change b 'syntax-type))))      
      (when b
	(if e
	    (buffer-substring b e)
	  (buffer-substring b (point-max)))))))

;;
;; sgml-mode
;;
(defvar sgml-here-document-beginning-regexp "<!\\[\\([A-Z]+\\)\\[")
(defvar sgml-here-document-end-regexp "]]>")
(defun sgml-here-document-region ()
  (let (beginning end)
    (save-excursion
      (save-match-data
	(when (re-search-backward
	       sgml-here-document-beginning-regexp
	       (point-min) t)
	  (setq beginning (match-end 0))
	  (when (re-search-forward 
		 sgml-here-document-end-regexp
		 (point-max) t)
	    (setq end (match-beginning 0))))))    
    (if (and beginning end (<= (point) end))
	(cons beginning end))))

(defun sgml-here-document-marker ()
  (if (sgml-here-document-region)
      (save-excursion
	(save-match-data
	  (when (re-search-backward
		 sgml-here-document-beginning-regexp
		 (point-min) t)
	    (match-string 1))))))

(provide 'hdoc)
;; hdoc.el ends here

Re: narrow-to-here-document

[David Kastrup]

Masatake YAMATO <jet@gyve.org> writes:

> I've reflected suggestions to my code.
> 

Actually, what one really would want is to have a
narrow-to-here-document which is not apparent in the display (leaving
the context around the here document), and to have hooks that will
revert the narrowing once the _user_ (as opposed to indentation code
and the like) tries leaving the narrowed area.  We would need some
functionality inside of Emacs in order to deal with that, probably.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: narrow-to-here-document

[Masatake YAMATO]

> Actually, what one really would want is to have a
> narrow-to-here-document which is not apparent in the display (leaving
> the context around the here document), and to have hooks that will
> revert the narrowing once the _user_ (as opposed to indentation code
> and the like) tries leaving the narrowed area.  We would need some
> functionality inside of Emacs in order to deal with that, probably.

Really great idea! 

I'll try to implement.
post-command-hook's time to the stage.

Masatake

Re: narrow-to-here-document

[Kevin Rodgers]

Masatake YAMATO wrote:

> I've reflected suggestions to my code.
...
> Kevin Rodgers:
> 3. 
>     Would INCLUDE/IGNORE/CDATA marked sections in SGML/XML/HTML qualify?
> 
>     <![CDATA[
>     No <tag>s or &entity; references are recognized here.
>     ]]>
...
> ;;
> ;; sgml-mode
> ;;
> (defvar sgml-here-document-beginning-regexp "<!\\[\\([A-Z]+\\)\\[")


SGML actually allows zero or more status keywords (separated by one or
more parameter separators, and followed by zero or more parameter
separators) between the marked section start (<![) and the declaration
subset open ([).  The status keywords are CDATA, IGNORE, INCLUDE,
and RCDATA, and TEMP.  A parameter separator is a whitespace character
(space, linefeed, carriage return, or tab), a parameter entity
reference, or a comment:

(defvar sgml-status-keyword-regexp
   ;; 10.4.2 Status Keyword Specification [100]
   "CDATA\\|IGNORE\\|INCLUDE\\|RCDATA")

(defvar sgml-s-regexp "[ \r\n\t]")	; 6.2.1 s Separator [5]

(defvar sgml-parameter-entity-reference-regexp
   ;; 9.4.4 Named Entity References [60]
   "%[a-zA-Z][a-zA-Z0-9---.]*[;\r\n]")

(defvar sgml-comment-regexp "--.*--")	; 10.3 Comment Declaration [92]

(defvar sgml-parameter-separator	; 10.1.1 Parameter Separator [65]
   (format "%s\\|%s\\|\\%s"
	  sgml-s-regexp
	  sgml-parameter-entity-reference-regexp
	  sgml-comment-regexp))

(defvar sgml-here-document-beginning-regexp
   ;; 10.4 Marked Section Declaration [93], [94]
   ;; 10.4.2 Status Keyword Specification [97]
   (format "<!\\[\\(\\(%s\\)+\\(%s\\|TEMP\\)\\)*\\(%s\\)*\\["
	  sgml-parameter-separator-reference-regexp
	  sgml-status-keyword-regexp
	  sgml-parameter-separator-reference-regexp))

XML only allows "<![CDATA[" (with no intervening characters at all):

(defvar xml-here-document-beginning-regexp ; 2.1 CDATA Sections [18], [19]
  "<!\\[\\(CDATA\\)\\[")

> (defvar sgml-here-document-end-regexp "]]>")
> (defun sgml-here-document-region ()
>   (let (beginning end)
>     (save-excursion
>       (save-match-data
> 	(when (re-search-backward
> 	       sgml-here-document-beginning-regexp
> 	       (point-min) t)
> 	  (setq beginning (match-end 0))
> 	  (when (re-search-forward 
> 		 sgml-here-document-end-regexp
> 		 (point-max) t)
> 	    (setq end (match-beginning 0))))))    
>     (if (and beginning end (<= (point) end))
> 	(cons beginning end))))


-- 
<a href="mailto:&lt;kevin.rodgers&#64;ihs.com&gt;">Kevin Rodgers</a>

Re: narrow-to-here-document

[Stefan Daschek]

David Kastrup <dak@gnu.org> writes:

> Actually, what one really would want is to have a
> narrow-to-here-document which is not apparent in the display (leaving
> the context around the here document), and to have hooks that will
> revert the narrowing once the _user_ (as opposed to indentation code
> and the like) tries leaving the narrowed area.  We would need some
> functionality inside of Emacs in order to deal with that, probably.

Well, that sounds a bit like what mmm-mode.el [1] already does (or at
least tries to do).

[1] http://mmm-mode.sourceforge.net/

ciao,
noniq

Re: narrow-to-here-document

[Kim F. Storm]

Masatake YAMATO <jet@gyve.org> writes:

> > Actually, what one really would want is to have a
> > narrow-to-here-document which is not apparent in the display (leaving
> > the context around the here document), and to have hooks that will
> > revert the narrowing once the _user_ (as opposed to indentation code
> > and the like) tries leaving the narrowed area.  We would need some
> > functionality inside of Emacs in order to deal with that, probably.
> 
> Really great idea! 
> 
> I'll try to implement.
> post-command-hook's time to the stage.

Or the `point-left' and `point-entered' text properties... ?


-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: narrow-to-here-document

[Richard Stallman]

The term "here document" is used for shell programming only.
You've explained that this mode is intended for broader use.
So the term "here document" has two problems: it is too long
in function names, and it is too narrow.

This suggests we should use some other more general term instead.
How about "literal"?  Any other suggestions?

    (defun make-indirect-buffer-for-hdoc (&optional change-major-mode)...

I suggest the name edit-literal for that function.
We should not rigidly put all relevant entities into the function name.

Re: narrow-to-here-document

[Richard Stallman]

    > Actually, what one really would want is to have a
    > narrow-to-here-document which is not apparent in the display (leaving
    > the context around the here document), and to have hooks that will
    > revert the narrowing once the _user_ (as opposed to indentation code
    > and the like) tries leaving the narrowed area.  We would need some
    > functionality inside of Emacs in order to deal with that, probably.

    Really great idea! 

    I'll try to implement.
    post-command-hook's time to the stage.

I have an idea for how to do this cleanly; I will try to implement it.

mmm-mode.el(Re: narrow-to-here-document)

[Masatake YAMATO]

> [1] http://mmm-mode.sourceforge.net/
I've just downloaded. It is a big package; it consists of 12 el files.

from README of mmm-mode.el:
                          MMM Mode for Emacs
                          ==================

OVERVIEW

  MMM Mode is a minor mode for Emacs that allows Multiple Major Modes
  (hence the name) to coexist in one buffer.  It is particularly
  well-suited to editing embedded code, such as Mason server-side
  Perl, or HTML output in CGI scripts.

This might be useful as the base for the narrow-to-here-document.
I'll report more later.

Masatake YAMATO
p.s. I've just remember I tried to write flex-mode.el...
mmm-mode.el was what I needed at that time:-P

Re: narrow-to-here-document

[David Kastrup]

Stefan Daschek <noniq-usenet@noniq.at> writes:

> David Kastrup <dak@gnu.org> writes:
> 
> > Actually, what one really would want is to have a
> > narrow-to-here-document which is not apparent in the display (leaving
> > the context around the here document), and to have hooks that will
> > revert the narrowing once the _user_ (as opposed to indentation code
> > and the like) tries leaving the narrowed area.  We would need some
> > functionality inside of Emacs in order to deal with that, probably.
> 
> Well, that sounds a bit like what mmm-mode.el [1] already does (or at
> least tries to do).
> 
> [1] http://mmm-mode.sourceforge.net/

AFAIR, it does not narrow, but merely switches major modes.  As an
example, it won't be useful for a shell script as a here document in
a shell script.  And indentation and stuff will look beyond the
beginning of the language area.

Perhaps indentation and brace matching and font locking and filling
and other syntactical properties could be made to depend on a text
property syntactic-realm or so that needs to be eq for belonging to
the same area as point, and maybe could contain a syntax table and a
local variable list or so?

Just brainstorming, ignore this if it seems too wildly absurd.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: narrow-to-here-document

[Miles Bader]

Richard Stallman <rms@gnu.org> writes:
> This suggests we should use some other more general term instead.
> How about "literal"?  Any other suggestions?
> 
>     (defun make-indirect-buffer-for-hdoc (&optional change-major-mode)...
> 
> I suggest the name edit-literal for that function.
> We should not rigidly put all relevant entities into the function name.

I find the name `literal' not clear for the opposite reason -- it seems
too broad, and literal seems like a work that might have many other
potential meanings.  Since the crucial point is that it's a literal field
embedded in a buffer, how about something like `edit-embedded-literal' or
`edit-literal-field'?

-Miles
-- 
`To alcohol!  The cause of, and solution to,
 all of life's problems' --Homer J. Simpson

Re: narrow-to-here-document

[Alan Shutko]

David Kastrup <dak@gnu.org> writes:

> AFAIR, it does not narrow, but merely switches major modes.  As an
> example, it won't be useful for a shell script as a here document in
> a shell script.

Actually, it doesn't just switch major modes, it does a lot of work
putting variables on text properties so that it can try to get
everything right.  It is useful for here documents, although I don't
know why it would be less useful for a shell script as a here
document in a shell script.

> And indentation and stuff will look beyond the beginning of the
> language area.

This is a problem.  For most modes it doesn't matter, but for CC mode
it can cause weirdness.  That's an ongoing problem, but narrowing
doesn't help, since CC mode widens before it parses the buffer.

-- 
Alan Shutko <ats@acm.org> - I am the rocks.
Looking for a software developer in St. Louis? http://web.springies.com/~ats/

Re: narrow-to-here-document

[Tak Ota]

26 Jun 2003 16:19:57 +0900: Miles Bader <miles@lsi.nec.co.jp> wrote:

> Richard Stallman <rms@gnu.org> writes:
> > This suggests we should use some other more general term instead.
> > How about "literal"?  Any other suggestions?
> > 
> >     (defun make-indirect-buffer-for-hdoc (&optional change-major-mode)...
> > 
> > I suggest the name edit-literal for that function.
> > We should not rigidly put all relevant entities into the function name.
> 
> I find the name `literal' not clear for the opposite reason -- it seems
> too broad, and literal seems like a work that might have many other
> potential meanings.  Since the crucial point is that it's a literal field
> embedded in a buffer, how about something like `edit-embedded-literal' or
> `edit-literal-field'?

It is a bit pity loosing the name 'here document' since its use is so
popular that any other naming would be an indirect description of
'here document'.  I can imagine such a conversation that:

"What is this embedded literal stuff?"  "That really is a here
document"

I Googled {perl|python|ruby} "here document", and found that the name
was more generically used now than when it was originally coined.

-Tak

Re: narrow-to-here-document

[David Kastrup]

Tak Ota <Takaaki.Ota@am.sony.com> writes:

> 26 Jun 2003 16:19:57 +0900: Miles Bader <miles@lsi.nec.co.jp> wrote:
> 
> > Richard Stallman <rms@gnu.org> writes:
> > > This suggests we should use some other more general term instead.
> > > How about "literal"?  Any other suggestions?
> > > 
> > >     (defun make-indirect-buffer-for-hdoc (&optional change-major-mode)...
> > > 
> > > I suggest the name edit-literal for that function.
> > > We should not rigidly put all relevant entities into the function name.
> > 
> > I find the name `literal' not clear for the opposite reason -- it seems
> > too broad, and literal seems like a work that might have many other
> > potential meanings.  Since the crucial point is that it's a literal field
> > embedded in a buffer, how about something like `edit-embedded-literal' or
> > `edit-literal-field'?
> 
> It is a bit pity loosing the name 'here document' since its use is so
> popular that any other naming would be an indirect description of
> 'here document'.

But "here" is a particularly bad keyword for apropos searches.  I'd
rather recommend something like "embedded", in particularly since
"here documents" are only a small part of the equation.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: narrow-to-here-document

[Miles Bader]

Tak Ota <Takaaki.Ota@am.sony.com> writes:
> It is a bit pity loosing the name 'here document' since its use is so
> popular that any other naming would be an indirect description of
> 'here document'.

I'm very skeptical of that claim; I was completely confused by the
original name.

-Miles
-- 
Run away!  Run away!

Re: narrow-to-here-document

[Richard Stallman]

      Since the crucial point is that it's a literal field
    embedded in a buffer, how about something like `edit-embedded-literal' or
    `edit-literal-field'?

Those are good ideas, and not terribly long.

Re: narrow-to-here-document

[Miles Bader]

On Thu, Jun 26, 2003 at 08:00:14AM -0500, Alan Shutko wrote:
> This is a problem.  For most modes it doesn't matter, but for CC mode
> it can cause weirdness.  That's an ongoing problem, but narrowing
> doesn't help, since CC mode widens before it parses the buffer.

Presumably CC mode could be changed to check the value of some variable
and not widen if it's non-nil.

-Miles
-- 
Somebody has to do something, and it's just incredibly pathetic that it
has to be us.  -- Jerry Garcia

Re: narrow-to-here-document

[Kai Großjohann]

Alan Shutko <ats@acm.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> And indentation and stuff will look beyond the beginning of the
>> language area.
>
> This is a problem.  For most modes it doesn't matter, but for CC mode
> it can cause weirdness.  That's an ongoing problem, but narrowing
> doesn't help, since CC mode widens before it parses the buffer.

I've been wondering if a workable alternative would be to make the
point motion functions look at a variable.  If the variable is set,
they ignore text with a certain property.

Suppose that we set the (hypothetical) text property `language' to
`c' on all the C bits, and then we let-bind the variable to the value
`c'.  Then all point motion commands would skip all text where the
text property language is not set to c.  So backward-char, for
example, would skip all that text.

This *might* cause the C mode syntax parsing functions to do the
right thing out of the box, since they can't look at the non-C parts
anymore.  But I don't know if it really works, it's just an idea.
-- 
~/.signature

Re: narrow-to-here-document

[David Kastrup]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> I've been wondering if a workable alternative would be to make the
> point motion functions look at a variable.  If the variable is set,
> they ignore text with a certain property.
> 
> Suppose that we set the (hypothetical) text property `language' to
> `c' on all the C bits, and then we let-bind the variable to the value
> `c'.  Then all point motion commands would skip all text where the
> text property language is not set to c.  So backward-char, for
> example, would skip all that text.

The intangible property does this sort of skipping, but it would be
pretty dangerous to fool around with this when one has code not
tailored for it: it is a reasonable assumption when programming that
(1- (point)) and (save-excursion (backward-char) (point)) are the
same thing, basically.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: narrow-to-here-document

[Kai Großjohann]

David Kastrup <dak@gnu.org> writes:

> it is a reasonable assumption when programming that (1- (point)) and
> (save-excursion (backward-char) (point)) are the same thing,
> basically.

It used to be that this was not the case.  (In Emacs 20.2 I think.)

I agree that it would be better if they were the same, but I can't
think of a less disruptive method for having two major modes in the
same buffer, and to allow the usual parsing functions for each of the
modes to function more-or-less normally.

Another approach might be to have two buffers, one for each mode, and
to merge them into a third buffer.  Then each command in the third
buffer would be actually executed in one of the two buffers.  But this
approach gives difficulties in finding out how to merge the two
buffers.
-- 
~/.signature

Re: narrow-to-here-document

[Richard Stallman]

The idea I had recently was to create an indirect buffer for each mode
that we want to use in the buffer, and then have a way of saying
"temporarily select the bindings of indirect buffer FOO for this
buffer".  That way, each mode could have its own set of local bindings,
and Emacs would switch between sets.

The switching could be done under the control of a text property, or
explicitly by post-command-hook.

You can think of this as themes for local variable values.

Re: narrow-to-here-document

[Stefan Monnier]

> I agree that it would be better if they were the same, but I can't
> think of a less disruptive method for having two major modes in the
> same buffer, and to allow the usual parsing functions for each of the
> modes to function more-or-less normally.

I think it's pretty disruptive (`intangible' properties already tend to
break a lot of code).  Also, when there are two blocks of C code separated
by something else, it's not clear whether the first should influence the
second or not.
I expect that mmm-mode can be extended to use point-motion hooks like
point-enter and point-leave to simulate your suggestions.  I think such
experimentation is necessary before being able to decide what kind
of additional support is needed, if any.  Sadly I haven't heard much from
mmm-mode developers on this list.  Maybe we should invite them so they
can share their horror stories and wishes.


	Stefan

Re: narrow-to-here-document

[Stefan Monnier]

> The idea I had recently was to create an indirect buffer for each mode
> that we want to use in the buffer, and then have a way of saying
> "temporarily select the bindings of indirect buffer FOO for this
> buffer".  That way, each mode could have its own set of local bindings,
> and Emacs would switch between sets.
> 
> The switching could be done under the control of a text property, or
> explicitly by post-command-hook.

That's pretty much what mmm-mode does already, except that it doesn't
use indirect buffers to store the various contexts (it just stores them
in plain variables, using `buffer-local-variables' and things like that).

I think we need feedback from the mmm-mode people before making any
decision on such an issue.  I haven't looked at mmm-mode recently, 
but it used to try and deal with font-lock, indentation and stuff like
that.  I'd actually expect them to need/want ad-hoc support from
font-lock.el (for example) but haven't heard anything (maybe they
complain about such things in their own .el files rather than here).


	Stefan

Re: narrow-to-here-document

[Kai Großjohann]

"Stefan Monnier" <monnier+gnu/emacs@cs.yale.edu> writes:

> I expect that mmm-mode can be extended to use point-motion hooks like
> point-enter and point-leave to simulate your suggestions.  I think such
> experimentation is necessary before being able to decide what kind
> of additional support is needed, if any.  Sadly I haven't heard much from
> mmm-mode developers on this list.  Maybe we should invite them so they
> can share their horror stories and wishes.

Good point.  Yeah, let us hear stories!
-- 
~/.signature

Re: [MMM] Re: narrow-to-here-document

[Michael A. Shulman]

On Thu, Jul 03, 2003 at 08:32:43PM -0400, Stefan Monnier wrote:
> I think we need feedback from the mmm-mode people before making any
> decision on such an issue.  I haven't looked at mmm-mode recently, 
> but it used to try and deal with font-lock, indentation and stuff like
> that.  I'd actually expect them to need/want ad-hoc support from
> font-lock.el (for example) but haven't heard anything (maybe they
> complain about such things in their own .el files rather than here).

Our current implementation is able to do reasonably well by using
font-lock-fontify-region-function and font-lock-beginning-of-syntax-function,
but we'd love to have better support from font-lock (and other
packages) for this sort of thing.  Thanks, Stefan, for crossposting to
the mmm list, since I wasn't following emacs-devel.

Briefly, the way mmm-mode works is to parse the buffer for regions
that it thinks (based on its configuration) ought to be edited using a
different major mode, and create overlays to mark those positions.
(It's not currently as good as we'd like at updating these overlays
dynamically as text is edited, but I believe this is just because no
one has sat down to write good code to do it.)  Then it uses
post-command-hook to check which region the point is in and make the
buffer look (almost) like it's in the associated major mode.  It
primarily does this by changing keymaps, syntax tables, and lots of
other buffer-local variables.  Some of these values are stored
buffer-locally, and others region-locally (all managed manually, of
course).

I don't mean to co-opt this disucssion for mmm, since it started on a
somewhat different (though related) topic.  But a "wish list" of
possible features in emacs to make mmm and related packages easier has
been on my mind for a while, and the discussion has been moving in
that direction.  So here are a few things I think emacs could do to
make this sort of stuff easier and more effective.

1. Have a "registry" of some sort for major modes, where they can
store enough information for mmm to turn them on effectively in small
regions.  Currently, the way mmm finds out what local variables and so
on to set for each mode is to create temporary buffers and save the
values of individual local variables.  Among other things, this
requires mmm mode knowing about all local variables that any major
mode could want to set, and so we have a huge list of what variables
to look for and what modes to look for them in.  In addition, some
major modes expect things like a file name set in all buffers, so we
have to do kludgy things to the temporary buffer to try to make them
work.

2. Have some sort of built-in support for variables that are localized
to a text region within a buffer, such as an overlay or a text
property.  I'm not sure how or if this would work in general, but it's
effectively what mmm does already to some extent.  In particular, some
variables want to keep different values in different regions even of
the same major mode, in particular variables that store the state of
syntax parsing.  This leads to...

3. Better support in the syntax parsing routines allowing the caller
to specify starting and ending positions and states.  One of
our biggest problems is that submodes end up trying to parse their
syntax beyond the beginning of their region and getting confused, so
ideally we would be able to tell them where to start their parsing,
and what state to start parsing in.  Sometimes the start state needs
to be a continuation of the end state of the previous region in that
mode (as with some literate programming, PHP, etc.), but sometimes
it ought to be an independent start for each region.

This is similar to the 'syntactic realm' idea that David Kastrup
suggested.  I don't think we've tried using intangible properties, for
the reasons mentioned and perhaps others, but Joe Kelsey has had some
success using syntax-table text properties to make the text that's not
supposed to be parsed be treated as a single word or as whitespace, as
appropriate.  Joe, want to comment?

4. Effectively the same thing for font-lock, which uses the syntax
parsing functions but also its own parsing functions.  In general
we've had more success hacking around font-lock than around syntax,
because font-lock provides more customization variables and its state
is easier to understand.

If we want to continue this discussion, should it stay crossposted, or
only on emacs-devel?

-- 
Michael A. Shulman
http://www.ugcs.net/~shulman

The goal of psychological, as of biological, development is
self-realization, or individuation.  But since man knows himself only
as an ego, and the self, as a totality, is indescribable and
indistinguishable from a God-image, self-realization... amounts to
God's incarnation.
        -- Carl Gustav Jung

Re: [MMM] Re: narrow-to-here-document

[Kai Großjohann]

"Michael A. Shulman" <shulman@ugcs.caltech.edu> writes:

> 3. Better support in the syntax parsing routines allowing the caller
> to specify starting and ending positions and states.  One of
> our biggest problems is that submodes end up trying to parse their
> syntax beyond the beginning of their region and getting confused, so
> ideally we would be able to tell them where to start their parsing,
> and what state to start parsing in.  Sometimes the start state needs
> to be a continuation of the end state of the previous region in that
> mode (as with some literate programming, PHP, etc.), but sometimes
> it ought to be an independent start for each region.

Maybe it is sufficient to just allow for ignoring parts of the
buffer.  Let's say you are mixing modes A and B.  Let's further say
that each A chunk needs to be considered separately, and all the B
chunks should be considered to be concatenated (as in the literate
programming case).  Then you would arrange things so that when point
is in an A region all the rest of the buffer is ignored.  And when
point is in a B region then all A regions are ignored, but the other
B regions are not ignored.

The above is based on having a big buffer and to ignore some parts of
it, depending on the state we're in.

The complementary approach is to have many small buffers and do the
editing there, but have one big buffer which serves as a view into
the many small buffers.  This approach has been suggested by
Richard.  I feel that it is promising -- it avoids having to tweak
the individual modes.

The user does C-x C-f mixedfile RET.  Then mmm-mode divides the buffer
into chunks and puts each chunk into its own buffer.  The mixedfile
buffer is set up such that all keys are bound to forwarding functions.
That is, when you hit a key, mmm-mode figures out which buffer to go
to.  Then it goes to that chunk-buffer and executes the corresponding
command.  Then it looks at the changes in the chunk-buffer and
propagates them back to the mixedfile buffer.

That way, no troubles with syntax, no troubles with font-lock, no
trouble with parsing states.

When you want to start a new chunk, that might be a bit difficult.
But mmm-mode could offer a command mmm-split-chunk or something like
this.

In the above I've used the term mmm-mode somewhat generically for the
desired feature; I didn't mean the current implementation
specifically.

Kai
-- 
~/.signature

mmm-mode needs (was: narrow-to-here-document)

[Stefan Monnier]

> Our current implementation is able to do reasonably well by using
> font-lock-fontify-region-function and font-lock-beginning-of-syntax-function,
> but we'd love to have better support from font-lock (and other
> packages) for this sort of thing.  Thanks, Stefan, for crossposting to
> the mmm list, since I wasn't following emacs-devel.

I wish package developers were more proactive in requesting features.
Especially since oftentimes the feature already exists (although not
in your case, I'm afraid) but also because it gives us more information
about what direction Elisp should be heading into.

> Briefly, the way mmm-mode works is to parse the buffer for regions
> that it thinks (based on its configuration) ought to be edited using a
> different major mode, and create overlays to mark those positions.

Is the parsing done "eagerly" when opening a file, or is it done
on-demand, à la jit-lock ?  If done eagerly, has the parsing time
ever been a problem (as in font-lock where the solution was to have
font-lock-maximum-size before jit-lock) ?

What kind of granularity has mmm-mode been used at ?
I.e. has it been used with hundreds of "regions" ?
What about treating strings and comments as separate regions ?

> (It's not currently as good as we'd like at updating these overlays
> dynamically as text is edited, but I believe this is just because no
> one has sat down to write good code to do it.)

I tend to think of strings and comments as being in the same category
as here-documents, C-in-yacc, javascript-in-html, ...
For strings and comments, the boundaries are recognized by the syntax-tables
and syntax-ppss can be used to efficiently determine the "current mode"
as well as the starting point of that mode (and together with
parse-partial-sexp is can also give you the end point).  It doesn't use
overlays, but the dynamic updating works rather well.

What do you think about extending syntax-tables to recognize an extensible
set of "mode switch" markers (rather than just comment-start, comment-end,
and string).

> I don't mean to co-opt this disucssion for mmm, since it started on a
> somewhat different (though related) topic.  But a "wish list" of

I'm highly interested in Emacs's support for language syntax and mmm-mode
is an important aspect of it.

> 1. Have a "registry" of some sort for major modes, where they can
> store enough information for mmm to turn them on effectively in small
> regions.  Currently, the way mmm finds out what local variables and so
> on to set for each mode is to create temporary buffers and save the
> values of individual local variables.  Among other things, this
> requires mmm mode knowing about all local variables that any major
> mode could want to set, and so we have a huge list of what variables
> to look for and what modes to look for them in.  In addition, some

Why not use `buffer-local-variables' ?
[ Don't take that to mean I think you're an idiot or something:
  I'm just trying to better understand where the current features
  are failing. ]

> major modes expect things like a file name set in all buffers, so we
> have to do kludgy things to the temporary buffer to try to make them
> work.

Why do you have to use a temporary buffer rather than the current
buffer exactly ?

> 2. Have some sort of built-in support for variables that are localized
> to a text region within a buffer, such as an overlay or a text
> property.  I'm not sure how or if this would work in general, but it's
> effectively what mmm does already to some extent.

Is there a particular problem with the way mmm-mode does it ?

> In particular, some
> variables want to keep different values in different regions even of
> the same major mode, in particular variables that store the state of
> syntax parsing.  This leads to...

I don't understand this part.  Since currently such region-local variables
don't exist, packages use other ways to store the info than a variable:
either in a list (that's what I use in syntax-ppss) or in overlays or
text-properties, ...

> 3. Better support in the syntax parsing routines allowing the caller
> to specify starting and ending positions and states.  One of

That's generally between very difficult and impossible: many/most
indentation algorithms don't have an explicit notion of `state'
(they actually work backwards from the indentation-point trying to find
a place w.r.t which they can indent).

> our biggest problems is that submodes end up trying to parse their
> syntax beyond the beginning of their region and getting confused, so
> ideally we would be able to tell them where to start their parsing,
> and what state to start parsing in.  Sometimes the start state needs
> to be a continuation of the end state of the previous region in that
> mode (as with some literate programming, PHP, etc.), but sometimes
> it ought to be an independent start for each region.
> 
> This is similar to the 'syntactic realm' idea that David Kastrup
> suggested.  I don't think we've tried using intangible properties, for
> the reasons mentioned and perhaps others, but Joe Kelsey has had some
> success using syntax-table text properties to make the text that's not
> supposed to be parsed be treated as a single word or as whitespace, as
> appropriate.  Joe, want to comment?

I think making the "outside" text intangible or whitespace-like or word-like
or comment-like is indeed the best approach.  Fiddling with syntax-tables
is not 100% because many indentation algorithms don't use syntax-tables or
at least not exclusively, so they may not always expect to have to jump over
a comment, or they use "[ \t]" rather than "\\s-" or ...

Has anybody tried to play with point-entered/point-left or with
intangible ?

In any case it seems difficult with the current Emacs to make it work
reliably without having to adjust the indentation code in a few places,
so the Joe's approach might indeed be the best option (together with
a bit of collaboration from the major mode).

> 4. Effectively the same thing for font-lock, which uses the syntax
> parsing functions but also its own parsing functions.  In general
> we've had more success hacking around font-lock than around syntax,
> because font-lock provides more customization variables and its state
> is easier to understand.

Also font-lock doesn't keep much state between lines.
Is mmm-mode line-oriented as well ?

> If we want to continue this discussion, should it stay crossposted, or
> only on emacs-devel?

It should definitely be on emacs-devel.  Whether it should be crossposted
to mmm-mode, I don't have an opinion.


	Stefan

Re: [MMM] Re: narrow-to-here-document

[Richard Stallman]

    Briefly, the way mmm-mode works is to parse the buffer for regions
    that it thinks (based on its configuration) ought to be edited using a
    different major mode, and create overlays to mark those positions.
    (It's not currently as good as we'd like at updating these overlays
    dynamically as text is edited, but I believe this is just because no
    one has sat down to write good code to do it.)  Then it uses
    post-command-hook to check which region the point is in and make the
    buffer look (almost) like it's in the associated major mode.

This fits in well with my idea for pseudo-indirect buffers.
The overlays could have a special property that automatically
directs the command loop to switch to a different pseudo-indirect buffer,
which would bring the appropriate keymaps, syntax tables, etc.
into force.

Re: [MMM] Re: narrow-to-here-document

[Richard Stallman]

    The complementary approach is to have many small buffers and do the
    editing there, but have one big buffer which serves as a view into
    the many small buffers.  This approach has been suggested by
    Richard.

I think I see a misunderstanding, because this is not what I meant to
suggest.  I suggested pseudo-indirect buffers as a way of being able
to deal with the same text using different key bindings, different
buffer-local values, different major modes.  In effect, they are
themes for bindings.  They would all share the same text, just as
user-visible indirect buffers do.

The idea of splitting up the text into many small buffers which would
be virtually concatenated is a completely different one.  I think that
idea is a non-starter because it would require changing nearly every
Emacs primitive in a very complex way.

    Maybe it is sufficient to just allow for ignoring parts of the
    buffer.  Let's say you are mixing modes A and B.  Let's further say
    that each A chunk needs to be considered separately, and all the B
    chunks should be considered to be concatenated (as in the literate
    programming case).  Then you would arrange things so that when point
    is in an A region all the rest of the buffer is ignored.  And when
    point is in a B region then all A regions are ignored, but the other
    B regions are not ignored.

This has the merit that we could implement it for syntax parsing
and font lock without changing most of Emacs at all.