enriched-mode and switching major modes.

enriched-mode and switching major modes.

[Luc Teirlinck]

I noted in an earlier message that when visiting a file that Emacs
earlier saved in text/enriched format, say enriched.text (which forces
Text mode), Enriched mode is turned on, but `use-hard-newlines' is nil
(and hence hard newlines do not work).  I suggested solving that
problem by making `use-hard-newlines' a permanent local and Richard
agreed.  I still believe that `use-hard-newlines' should be a
permanent local, regardless of the problems with Enriched mode, but I
will wait a little bit before installing, since some related issues
need to be discussed..

There are several other similar problems.  In the same enriched.text
buffer, `default-text-properties' is nil, which also messes up some
stuff.  Also, `indent-line-function' is `indent-relative' instead of
`indent-to-left-margin'.  `buffer-display-table' is nil.
`enriched-old-bindings' is nil.  These problems have the same cause as
the one I noted earlier for `use-hard-newlines'.  `enriched-mode' sets
buffer-local values for these variables, but when the major mode is
set by `after-find-file', these variables are killed.

Of course, similar problems arise whenever the _user_ switches major
modes in a buffer already in Enriched mode: all local variables are
killed.

Enriched mode is currently a minor mode and a permanent local as a
variable.  It can not possibly set local variables that are
non-permanent locals for the reasons described above.  If it _has_ to
do that, it should be a major mode.

One solution is to make Enriched mode into a major mode.  It already
very much "feels" like one.

The other solution would be to make all variables that Enriched mode
needs to set permanent locals.  For `use-hard-newlines' we already
decided that there was no problem. 

`default-text-properties' is currently not even an automatically local
variable, but maybe it should be made into a permanent local.  It is
_only_ used by enriched mode, so making it into a permanent local
should not cause problems.

`buffer-display-table' is used by several other packages, but all seem
to use it in buffers where changing the major mode does not seem to
make any sense to begin with.  So it could be made into a permanent local.

I believe that there is no way to make `indent-line-function' into a
permanent local.  Here, I believe that the solution is to no longer
have Enriched mode set the variable.  Setting `indent-line-function'
is the prerogative of the major mode.  If Enriched mode absolutely
_needs_ to set `indent-line-function' to `indent-to-left-margin', then
Enriched mode _needs_ to be a major mode (and then the other problems
automatically disappear).

`enriched-old-bindings' should not be a permanent local.  Its
implementation is iffy to begin with.  Here I believe all we can do is
rely on the fact that disabling Enriched mode only seems to make sense
if it was enabled by mistake.  Thus, problems arising from
`enriched-old-bindings' having the wrong value, though they
theoretically exist, are unlikely to arise in practice.  Making
Enriched mode into a major mode would make `enriched-old-bindings'
completely unnecessary.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Richard Stallman]

      `enriched-mode' sets
    buffer-local values for these variables, but when the major mode is
    set by `after-find-file', these variables are killed.

It sounds like enriched-mode is called too early.
What is the command that you give?

    One solution is to make Enriched mode into a major mode.  It already
    very much "feels" like one.

    The other solution would be to make all variables that Enriched mode
    needs to set permanent locals.  For `use-hard-newlines' we already
    decided that there was no problem. 

The former is acceptable, but not really nice.  The latter is not
acceptable at all.  So I am looking for some other solution.

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Richard Stallman wrote:

	 `enriched-mode' sets
       buffer-local values for these variables, but when the major mode is
       set by `after-find-file', these variables are killed.

   It sounds like enriched-mode is called too early.
   What is the command that you give?

C-x C-f on a file called enriched.text.  But any time one switches
major modes in a buffer in Enriched mode, the same problems arise.

I now believe that Enriched mode should be temporarily disabled when
switching major modes and later re-enabled by post-command-hook.  That
not only resets all killed local variables, it also seems to be the
only way to correctly update `enriched-old-bindings', which is
necessary when switching major modes.  The following patch implements
this.  It should work OK even in case the user quits or an error
occurs.  I can install it if it looks OK.

===File ~/enriched.el-diff==================================
*** enriched.el	02 Jul 2004 21:39:16 -0500	1.4
--- enriched.el	05 Sep 2004 19:16:50 -0500	
***************
*** 1,6 ****
  ;;; enriched.el --- read and save files in text/enriched format
  
! ;; Copyright (c) 1994, 1995, 1996, 2002 Free Software Foundation, Inc.
  
  ;; Author: Boris Goldowsky <boris@gnu.org>
  ;; Keywords: wp, faces
--- 1,6 ----
  ;;; enriched.el --- read and save files in text/enriched format
  
! ;; Copyright (c) 1994, 1995, 1996, 2002, 2004 Free Software Foundation, Inc.
  
  ;; Author: Boris Goldowsky <boris@gnu.org>
  ;; Keywords: wp, faces
***************
*** 155,160 ****
--- 155,164 ----
  The value is a list of \(VAR VALUE VAR VALUE...).")
  (make-variable-buffer-local 'enriched-old-bindings)
  
+ ;; A non-nil value indicates that Enriched mode was temporarily disabled
+ ;; by `enriched-before-change-major-mode' while switching major modes.
+ (defvar enriched-change-major-mode-flag nil)
+ 
  ;;;
  ;;; Define the mode
  ;;;
***************
*** 197,203 ****
  	 (make-local-variable 'default-text-properties)
  	 (setq indent-line-function 'indent-to-left-margin ;WHY??  -sm
  	       buffer-display-table  enriched-display-table)
! 	 (use-hard-newlines 1 nil)
  	 (let ((sticky (plist-get default-text-properties 'front-sticky))
  	       (p enriched-par-props))
  	   (dolist (x p)
--- 201,207 ----
  	 (make-local-variable 'default-text-properties)
  	 (setq indent-line-function 'indent-to-left-margin ;WHY??  -sm
  	       buffer-display-table  enriched-display-table)
! 	 (use-hard-newlines 1 (if enriched-change-major-mode-flag 'never nil))
  	 (let ((sticky (plist-get default-text-properties 'front-sticky))
  	       (p enriched-par-props))
  	   (dolist (x p)
***************
*** 205,211 ****
  	   (if sticky
  	       (setq default-text-properties
  		     (plist-put default-text-properties
! 				'front-sticky sticky)))))))
  
  ;;;
  ;;; Keybindings
--- 209,234 ----
  	   (if sticky
  	       (setq default-text-properties
  		     (plist-put default-text-properties
! 				'front-sticky sticky))))
! 	 (add-hook 'change-major-mode-hook
! 		   'enriched-before-change-major-mode nil t)))
!   (setq enriched-change-major-mode-flag nil))
! 
! ;; Enriched mode sets several local variables that are killed when
! ;; changing major modes.  They need to be reset.  Also, when the major
! ;; mode changes, `enriched-old-bindings' needs to be correctly
! ;; updated.  Therefore, we disable Enriched mode before changing the
! ;; major mode and enable it back afterward.
! (defun enriched-before-change-major-mode ()
!   (when enriched-mode
!     (let ((inhibit-quit t))
!       (enriched-mode 0)
!       (setq enriched-change-major-mode-flag t))))
! 
! (defun enriched-after-change-major-mode ()
!   (when enriched-change-major-mode-flag (enriched-mode 1)))
! 
! (add-hook 'post-command-hook 'enriched-after-change-major-mode)
  
  ;;;
  ;;; Keybindings
============================================================

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>       `enriched-mode' sets
>     buffer-local values for these variables, but when the major mode is
>     set by `after-find-file', these variables are killed.
>
> It sounds like enriched-mode is called too early.
> What is the command that you give?
>
>     One solution is to make Enriched mode into a major mode.  It already
>     very much "feels" like one.
>
>     The other solution would be to make all variables that Enriched mode
>     needs to set permanent locals.  For `use-hard-newlines' we already
>     decided that there was no problem. 
>
> The former is acceptable, but not really nice.

IMO opinion enriched-mode actually serves two purposes: (1) editing of
(very restricted) WP-like documents in the format text/enriched (2) a
means to save the colours that appear in a buffer in a file.

The difference is subtle. A usage scenario of type (1) would be, for
example, that I'd want to keep notes with some text properties in a
file. I'd create that file for this purpose, use text-mode as a major
mode, turn on enriched-mode and then specify myself explicitely which
text properties to use with M-g. A usage scenario of type (2) would
be, that I have a buffer that got its text properties from some Elisp
function and I want to save that buffer to a file, while also saving
the text-properties -- or at least the colours. For instance, when I
save my IRC logs (from ERC), I turn on enriched-mode before that.

As (1) enriched-mode should clearly be a major mode IMO, a derivative
of text-mode. If, for instance, a user wants to keep notes in a file
in the format text/enriched, then I can't actually see the point of
having text/enriched as a minor mode that is used together with
text-mode instead of just being a major mode.

As (2) it should be a minor mode, of course. However, currently some
of enriched-mode's features that make it suitable for (1) get in the
way here. For example I can't just turn on enriched-mode in an ERC
buffer, because enriched-mode rebinds RET. As a minor mode, it should
only affect the way a buffer is saved to a file and nothing else.

So I'd suggest to make enriched-mode a major mode and create a new
additional minor mode for the purpose of (2).

There is a technical problem with making it a major mode: currently
format.el expects modes specified in `format-alist' to be minor
modes.  However, fixing this would be an improvement anyways.

But maybe this all can wait until after the release.  I believe that
(1) is the way it is used most of the time.

    Oliver
-- 
Oliver Scholz               21 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Stefan]

> I now believe that Enriched mode should be temporarily disabled when
> switching major modes and later re-enabled by post-command-hook.  That

I agree about switching it off in before-change-major-mode, but I see no
reason to switch it back on afterwards.  This would basically mean to make
it permanant-local .


        Stefan

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Stefan Monnier wrote:

   > I now believe that Enriched mode should be temporarily disabled when
   > switching major modes and later re-enabled by post-command-hook.  That

   I agree about switching it off in before-change-major-mode, but I see no
   reason to switch it back on afterwards.  This would basically mean to make
   it permanant-local .

It already is supposed to be permanent-local:

    (put 'enriched-mode 'permanent-local t)
    ;;;###autoload
    (define-minor-mode enriched-mode

The problem is that some of the variables it sets are not
permanent-local, which makes Enriched mode survive mode changes in an
inconsistent state.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Alex Schroeder]

Oliver Scholz <epameinondas@gmx.de> writes:

> IMO opinion enriched-mode actually serves two purposes: (1) editing of
> (very restricted) WP-like documents in the format text/enriched (2) a
> means to save the colours that appear in a buffer in a file.

I agree with the analysis and also prefer to make enriched-mode a
major-mode based on text-mode.  

As to (2) -- assume this is a programming language: The markup added
by enriched-mode will essentially change the file and make it
unusable.  Therefore I think (2) could be served with a new defun M-x
save-buffer-as-enriched-text or something like that.  This will prompt
the user for a new file-name, making it clear that the original file
should not be overwritten.  Maybe default to current-name +
".enriched".  Reloading the file will the use the new major-mode
enriched-mode.

True, the current implementation allows future derivatives of
text-mode -- such as a major mode to write movie scripts -- to add
persistent colors to the text.  But how often is this power used?  And
at what cost does this power come?  Even if we wanted this power,
wouldn't it be better if such modes could just add some defun to a
hook or two that would automatically do the right thing: Automatically
encode and decode text properties upon saving and reading files.
Without all the other features of enriched mode which are rarely
useful in this case, such as the the treatment of newlines or the
keybindings.

Alex.
-- 
.O.  http://www.emacswiki.org/alex/
..O  Schroeder's fifth law:
OOO  Never accept more work than you can handle in one night of hacking.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    True, the current implementation allows future derivatives of
    text-mode -- such as a major mode to write movie scripts -- to add
    persistent colors to the text.

It also allows existing simple variants of text mode, such as
paragraph-indent-text-mode, to work with enriched mode.

    But how often is this power used?

Enriched mode is not used much, now.  Part of the reason is that we
don't have code to save in the formats users really use, and part is
that we have not implemented enough of the display formatting features
that people want in a word processor.  This is an unfinished feature.

But does the lack of usage really make a difference here?

				       And
    at what cost does this power come?

I don't see the point here.  What do you think the cost is?
At present, what we see a bug.  I admit that the bug is not
trivial to fix, but I don't see that the "power" of enriched
mode has a "cost" in some inherent sense.

      Even if we wanted this power,
    wouldn't it be better if such modes could just add some defun to a
    hook or two that would automatically do the right thing: Automatically
    encode and decode text properties upon saving and reading files.
    Without all the other features of enriched mode which are rarely
    useful in this case, such as the the treatment of newlines or the
    keybindings.

I am not really sure what feature you have in mind.
It might be a useful feature, but it would not replace
the existing feature of Enriched mode.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    I now believe that Enriched mode should be temporarily disabled when
    switching major modes and later re-enabled by post-command-hook.

I think you are on the right track, except for the detail that
post-command-hook does the job too late.  enriched-mode ought to get
reenabled as soon as the new major mode is set up; it should not wait
until the command finishes.

One way to make this work is by adding a feature to run-mode-hooks.
enriched-mode's change-major-mode-hook could then do someting 
to tell run-mode-hooks to reenable enriched-mode.

Want to try that?

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     True, the current implementation allows future derivatives of
>     text-mode -- such as a major mode to write movie scripts -- to add
>     persistent colors to the text.
>
> It also allows existing simple variants of text mode, such as
> paragraph-indent-text-mode, to work with enriched mode.

But for things like this, enriched-mode-the-minor-mode should /not/
affect anything but encoding/decoding.  It should leave everything
else to the major mode.

>     But how often is this power used?
>
> Enriched mode is not used much, now.  Part of the reason is that we
> don't have code to save in the formats users really use, and part is
> that we have not implemented enough of the display formatting features
> that people want in a word processor.  This is an unfinished feature.

Indeed, I believe that in the long run Emacs' display engine should
support a real block model.

These two belong together, IMO.  It is very difficult to make Emacs
support formats that people really use in a WP-style way. Word
processors assign properties to paragraphs, including defaults for
character styles (like the font, the weight etc.); and they support
style sheets for that. It is very, very hairy to keep paragraphs,
their properties and their representation in an Emacs buffer in sync,
not to talk about style sheets.  In fact I do think that getting a WP
UI right in Emacs is currently impossible.  (However, I start to think
that some 60%-70% solution could be feasible. But I have not explored
that yet.)

Interesting coincidence, btw: I lately had a discussion via e-mail
with somebody who stumbled over a prototype RTF reader of mine on the
web.  He asked me to make enriched-mode support RTF as a format.  I told
him that this is not possible for the reasons above; even an
incomplete, possibly fragile, partial solution would require a major
mode.  It turned out that that person would be satisfied with something
which I call "bad RTF", that specifies font, font height, font weight
etc. everywhere explicitely; i.e. indeed something like text/enriched
but with a file format that is supported by more programs.  This,
however, could not really be called "word processing".  Personally, I
am not interested in such things and I think it is very wrong.

I do think that enriched-mode as a minor mode is a dead end.  If it
would restricted itself to just affecting visiting and saving, it
would be nice to have (I disagree with Alex Schröder here).  But this
is not how WP in Emacs could be like.  WP in Emacs should be based on
a major mode that handels every aspect of editing and representing
and that would support different file formats (RTF, HTML, TEI XML,
Docbook, maybe Texinfo) as a target.  And most importantly that major
mode would be responsible for providing an user interface to a block
model -- either a real one supported by the display engine, or a
fake one based on some half-obscure hackery.

> But does the lack of usage really make a difference here?
>
> 				       And
>     at what cost does this power come?
>
> I don't see the point here.  What do you think the cost is?
> At present, what we see a bug.  I admit that the bug is not
> trivial to fix, but I don't see that the "power" of enriched
> mode has a "cost" in some inherent sense.
>
>       Even if we wanted this power,
>     wouldn't it be better if such modes could just add some defun to a
>     hook or two that would automatically do the right thing: Automatically
>     encode and decode text properties upon saving and reading files.
>     Without all the other features of enriched mode which are rarely
>     useful in this case, such as the the treatment of newlines or the
>     keybindings.
>
> I am not really sure what feature you have in mind.
> It might be a useful feature, but it would not replace
> the existing feature of Enriched mode.

I think that Alex talks about writing new major modes that also use
text/enriched as a file format.  I think that this feature is already
there.  We have `after-insert-file-functions' and
`write-region-annotate-functions' (or `format-alist') and we have
`enriched-encode' and `enriched-decode'.


    Oliver
-- 
Oliver Scholz               25 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Richard Stallman wrote:

   I think you are on the right track, except for the detail that
   post-command-hook does the job too late.  enriched-mode ought to get
   reenabled as soon as the new major mode is set up; it should not wait
   until the command finishes.

   One way to make this work is by adding a feature to run-mode-hooks.
   enriched-mode's change-major-mode-hook could then do someting 
   to tell run-mode-hooks to reenable enriched-mode.

   Want to try that?

My previous patch somehow forgot that the command could switch
buffers.  The new patch below fixes that.  I believe that we still
have to use post-command-hook to avoid possibly very annoying and
confusing inconsistencies that could otherwise arise in the case of
error or quit (as well as for major modes that do not use
`run-mode-hooks').  With the new function `with-local-quit', there can
actually be several quits and hence several inconsistencies as a
result of executing a single command.  The patches below take all of
this into account.

Note that `post-command-hook' will only have some non-trivial work to
do in case of really exceptional situations, like errors or quits at
the wrong moment.  In normal circumstances, all it does is a dolist
over an empty list.

Of course, if we would make Enriched mode into a major mode, then none
of this would be necessary.

===File ~/subr.el-diff======================================
*** subr.el	08 Sep 2004 11:29:29 -0500	1.407
--- subr.el	10 Sep 2004 14:39:21 -0500	
***************
*** 1937,1942 ****
--- 1937,1946 ----
  (make-variable-buffer-local 'delayed-mode-hooks)
  (put 'delay-mode-hooks 'permanent-local t)
  
+ (defvar after-change-major-mode-hook nil
+   "Mode independent hook run after changing major modes.
+ This is run just before the mode dependent hooks.")
+ 
  (defun run-mode-hooks (&rest hooks)
    "Run mode hooks `delayed-mode-hooks' and HOOKS, or delay HOOKS.
  Execution is delayed if `delay-mode-hooks' is non-nil.
***************
*** 1948,1953 ****
--- 1952,1958 ----
      ;; Normal case, just run the hook as before plus any delayed hooks.
      (setq hooks (nconc (nreverse delayed-mode-hooks) hooks))
      (setq delayed-mode-hooks nil)
+     (run-hooks 'after-change-major-mode-hook)
      (apply 'run-hooks hooks)))
  
  (defmacro delay-mode-hooks (&rest body)
============================================================

===File ~/enriched.el-diff==================================
*** enriched.el	02 Jul 2004 21:39:16 -0500	1.4
--- enriched.el	10 Sep 2004 20:25:21 -0500	
***************
*** 1,6 ****
  ;;; enriched.el --- read and save files in text/enriched format
  
! ;; Copyright (c) 1994, 1995, 1996, 2002 Free Software Foundation, Inc.
  
  ;; Author: Boris Goldowsky <boris@gnu.org>
  ;; Keywords: wp, faces
--- 1,6 ----
  ;;; enriched.el --- read and save files in text/enriched format
  
! ;; Copyright (c) 1994, 1995, 1996, 2002, 2004 Free Software Foundation, Inc.
  
  ;; Author: Boris Goldowsky <boris@gnu.org>
  ;; Keywords: wp, faces
***************
*** 141,147 ****
  
  ;;; Internal variables
  
- 
  (defcustom enriched-mode-hook nil
    "Hook run after entering/leaving Enriched mode.
  If you set variables in this hook, you should arrange for them to be restored
--- 141,146 ----
***************
*** 155,160 ****
--- 154,170 ----
  The value is a list of \(VAR VALUE VAR VALUE...).")
  (make-variable-buffer-local 'enriched-old-bindings)
  
+ ;; A non-nil value indicates that Enriched mode was temporarily disabled
+ ;; by `enriched-before-change-major-mode' while switching major modes.
+ (defvar enriched-change-major-mode-flag nil)
+ (make-local-variable 'enriched-change-major-mode-flag)
+ (put 'enriched-change-major-mode-flag 'permanent-local t)
+ 
+ ;; List of buffers in which Enriched mode was disabled by
+ ;; `enriched-before-change-major-mode' and was not properly re-enabled.
+ ;; `post-command-hook' checks this.
+ (defvar enriched-marked-buffers nil)
+ 
  ;;;
  ;;; Define the mode
  ;;;
***************
*** 197,203 ****
  	 (make-local-variable 'default-text-properties)
  	 (setq indent-line-function 'indent-to-left-margin ;WHY??  -sm
  	       buffer-display-table  enriched-display-table)
! 	 (use-hard-newlines 1 nil)
  	 (let ((sticky (plist-get default-text-properties 'front-sticky))
  	       (p enriched-par-props))
  	   (dolist (x p)
--- 207,213 ----
  	 (make-local-variable 'default-text-properties)
  	 (setq indent-line-function 'indent-to-left-margin ;WHY??  -sm
  	       buffer-display-table  enriched-display-table)
! 	 (use-hard-newlines 1 (if enriched-change-major-mode-flag 'never nil))
  	 (let ((sticky (plist-get default-text-properties 'front-sticky))
  	       (p enriched-par-props))
  	   (dolist (x p)
***************
*** 205,211 ****
  	   (if sticky
  	       (setq default-text-properties
  		     (plist-put default-text-properties
! 				'front-sticky sticky)))))))
  
  ;;;
  ;;; Keybindings
--- 215,254 ----
  	   (if sticky
  	       (setq default-text-properties
  		     (plist-put default-text-properties
! 				'front-sticky sticky))))
! 	 (add-hook 'change-major-mode-hook
! 		   'enriched-before-change-major-mode nil t)))
!   (setq enriched-change-major-mode-flag nil)
!   (setq enriched-marked-buffers
! 	(delete (current-buffer) enriched-marked-buffers)))
! 
! ;; Enriched mode sets several local variables that are killed when
! ;; changing major modes.  They need to be reset.  Also, when the major
! ;; mode changes, `enriched-old-bindings' needs to be correctly
! ;; updated.  Therefore, we disable Enriched mode before changing the
! ;; major mode and enable it back afterward.
! (defun enriched-before-change-major-mode ()
!   (when enriched-mode
!     (let ((inhibit-quit t))
!       (enriched-mode 0)
!       (setq enriched-change-major-mode-flag t)
!       (add-to-list 'enriched-marked-buffers (current-buffer)))))
! 
! (defun enriched-after-change-major-mode ()
!   (when enriched-change-major-mode-flag (enriched-mode 1)))
! 
! (add-hook 'after-change-major-mode-hook 'enriched-after-change-major-mode)
! 
! ;; Have `post-command-hook' check for buffers left in an inconsistent
! ;; state as a result of an error, quit or "local quit" (that is, by
! ;; using `with-local-quit').
! (defun enriched-check-mode-change-buffers ()
!   (dolist (buf enriched-marked-buffers)
!     (with-current-buffer buf
!       (when enriched-change-major-mode-flag (enriched-mode 1))))
!   (setq enriched-marked-buffers nil))
! 
! (add-hook 'post-command-hook 'enriched-check-mode-change-buffers)
  
  ;;;
  ;;; Keybindings
============================================================

Re: enriched-mode and switching major modes.

[Stefan]

> !     (let ((inhibit-quit t))

Please don't mess with inhibit-quit unless it's *really* *really* necessary.
I'd recommend you just stop worrying about `quit' here or else use
`unwind-protect' if there are cleanup operations that need to be done.


        Stefan

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Stefan Monnier wrote:

   > !     (let ((inhibit-quit t))

   Please don't mess with inhibit-quit unless it's *really* *really* necessary.
   I'd recommend you just stop worrying about `quit' here or else use
   `unwind-protect' if there are cleanup operations that need to be done.

There is nothing time consuming done inside that `inhibit-quit'.
Actually, after this some code in post-command-hook will run (which
automatically prevents quitting) but nothing time consuming is done
there either.

There are several problems.  I could take care of the very bad problem
of `enriched-change-major-mode-flag' not being reset to nil with no
code _explicitly_ inside inhibit-quit and even less time consuming
code in post-command-hook.  But doing this could leave Enriched mode
disabled in buffers in which it should be enabled and these buffers
may not even be current when the user returns.  The disabling was done
behind the user's back, without even notifying the user.  C-x s will
save the buffer in the wrong format.  My code completely avoids this
risk.

Auto Revert mode runs tons of relatively more time consuming stuff
inside an implicit inhibit-quit, and before I started using Tramp, as
well as after we disabled auto-reverting for remote files, this has
never caused me any problems, nor did I see anybody else complain
about such problems.  (Again, except for the Tramp stuff.)

Of course, if somebody accidentally introduces an infinite loop in
`enriched-mode', my inhibit-quit (as well as the use of
post-command-hook) will make a bad problem even worse.  But the same
holds for any function run from a timer, pre- or post-command-hook.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Stefan]

> There are several problems.  I could take care of the very bad problem
> of `enriched-change-major-mode-flag' not being reset to nil with no
> code _explicitly_ inside inhibit-quit and even less time consuming
> code in post-command-hook.  But doing this could leave Enriched mode
> disabled in buffers in which it should be enabled and these buffers
> may not even be current when the user returns.  The disabling was done
> behind the user's back, without even notifying the user.  C-x s will
> save the buffer in the wrong format.  My code completely avoids this
> risk.

It seems you don't realize that a lot of elisp code suffers from such
"not 100% correct state if C-g is hit at the wrong moment".  It's not such
a big deal.

And even if you want to solve such problems, inhibit-quit is usually not the
best way to solve them.  Making sure that the state is consistent at all
times is a better one (when available).
E.g. instead of

!     (let ((inhibit-quit t))
!       (enriched-mode 0)
!       (setq enriched-change-major-mode-flag t)
!       (add-to-list 'enriched-marked-buffers (current-buffer)))))

you can do

!       (add-to-list 'enriched-marked-buffers (current-buffer))))
!       (setq enriched-change-major-mode-flag t)
!       (enriched-mode 0)

Oh, and please don't go through details like mentioning with-local-quit
when it's just one of many possible cases (and especially since the cause
of the "problem" is not with-local-quit, but just the `quit' itself).

> Of course, if somebody accidentally introduces an infinite loop in
> `enriched-mode', my inhibit-quit (as well as the use of
> post-command-hook) will make a bad problem even worse.  But the same
> holds for any function run from a timer, pre- or post-command-hook.

Of course, if inhibit-quit poses problem, it's a bug, and bugs are "rare"
(thanks to our debugging efforts), so inhibit-quit rarely poses problems.
But when it does it's a real pain.

BTW, looking at the enriched-mode code, I'm wondering whether all this
fiddling is necessary.  After all the variables it sets are:
- buffer-file-format: already permanent-local.
- use-hard-newlines: we agreed this should also be permanent-local.
- indent-line-function: I'd argue that enriched-mode should *not* change
  this variable.  After all, enriched-mode is supposed to be a minor mode
  that can be used with various major modes and the indentation behavior
  should be determined by the major mode.
- default-text-properties: the changes applied to this var seem to be
  unrelated to enriched-mode and is just always useful/necessary.  They should
  be moved out of enriched-mode.
- buffer-display-table: OK, this one is trickier.  It's used to turn ^L
  (i.e. form feed) chars into a line of dashes.  We could get rid of the
  feature or replace it with a display property, or keep it as is
  (i.e. works only as long as the major mode is not changed).

Alternatively, we can go through the trouble that Luc proposes, but then we
should make it generic, so that enriched-mode can just be added to a list of
"permanent-local minor modes that need to be turned back on after a major
mode change".



        Stefan

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Stefan Monnier wrote:

   It seems you don't realize that a lot of elisp code suffers from such
   "not 100% correct state if C-g is hit at the wrong moment".  It's not such
   a big deal.

It is a big deal if the nature of the inconsistent state is not obvious.
In this case the situation is particularly bad, because even
immediately doing C-x C-c after the quit may result in buffers being
saved in the wrong format.

   you can do

   !       (add-to-list 'enriched-marked-buffers (current-buffer))))
   !       (setq enriched-change-major-mode-flag t)
   !       (enriched-mode 0)

No, because enriched-mode sets enriched-change-major-mode-flag to nil,
(It has to.)

   !       (add-to-list 'enriched-marked-buffers (current-buffer))))
   !       (setq enriched-change-major-mode-flag t)
   !       (let (enriched-change-major-mode-flag)
   !         (enriched-mode 0))

would be less bad.  But even that would give problems if a quit happened
during the execution of enriched-mode.

Ayway, why try to find half-baked solutions to a completely
_non-existent_ problem?  `inhibit-quit' is only a problem around code
that can take a non-trivial time to execute.

   Oh, and please don't go through details like mentioning with-local-quit
   when it's just one of many possible cases (and especially since the cause
   of the "problem" is not with-local-quit, but just the `quit' itself).

It is because of `with-local-quit' that more than one buffer can be
involved.  That could not happen with an error or a regular quit.
(Unless the error was handled in a way similar to `with-local-quit'.)

   BTW, looking at the enriched-mode code, I'm wondering whether all this
   fiddling is necessary.  After all the variables it sets are:

Eventually Enriched mode will need to set more variables.  It is not a
major mode but feels very much like one.

>From other people's reactions it would seem that the two major
alternatives are my patch (or some variation thereof) and simply
making enriched-mode into a major mode.

I do not object against making enriched-mode into a major mode.  It
would be a lot _simpler_ than the solution I implemented.  But note
that after my patch you can use Enriched mode as a major mode _anyway_
(for all practical purposes) by combining it with fundamental mode.
But you can also combine it with any other major mode that is
"compatible" with Enriched mode.  So you get extra flexibility for the
added complexity.

Details:

   - use-hard-newlines: we agreed this should also be permanent-local.

It now is.  (I installed that.)

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Stefan Monnier wrote:

   Alternatively, we can go through the trouble that Luc proposes, but then we
   should make it generic, so that enriched-mode can just be added to a list of
   "permanent-local minor modes that need to be turned back on after a major
   mode change".

I believe it is a mistake to try to find a generic solution for a
problem that affects one single function, just because it is
theoretically possible that at some point in the future there might be
a second example.  Especially since we might not really want to
encourage the creation of other examples, unless really needed.
`enriched-mode' is kind of peculiar in that it is a minor mode that
really needs to be a kind of second major mode.  Actually, an
alternative to my patches would be to make it a major mode.  But that
would take away the present feature that it can be combined with other
major modes.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Richard Stallman]

      I believe that we still
    have to use post-command-hook to avoid possibly very annoying and
    confusing inconsistencies that could otherwise arise in the case of
    error or quit (as well as for major modes that do not use
    `run-mode-hooks').

I want to avoid using post-command-hook for this, because it is not
clean and reliable.  It more or less does the right thing, in most
cases, but not exactly, and not always.

The right thing to do in case of an error would be to restore these
bindings as part of error handling.

Or, alternatively, there could be a list of local variables not to
kill when changing major modes.  This list would itself be local
(that's how it differs from making the variables permanent).  Enriched
mode would put them onto this list.  That way, it would not have to do
anything like turn itself off and on.

As regards the key bindings, it ought to do them with a minor mode
keymap instead of its current method.  That would be much cleaner
anyway.  Probably it was written before the minor mode keymap feature
existed.

Want to try it that way?

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Richard Stallman wrote:

   Or, alternatively, there could be a list of local variables not to
   kill when changing major modes.  This list would itself be local
   (that's how it differs from making the variables permanent).  Enriched
   mode would put them onto this list.  That way, it would not have to do
   anything like turn itself off and on.

Even if `kill-all-local-variables' does not kill these variables, the
major mode can override them.  In the case of `indent-line-function',
most modes that are usually combined with Enriched mode will.

I think we are getting into problems that are going to get worse and
worse as Enriched mode gets more extensive.

I more and more agree with Oliver that we should have two Enriched
modes.  The minor mode Enriched mode would be a true minor mode.  It
would essentially only set ` buffer-file-format' and maybe do some
additional minor stuff.  The major mode would replace the current
"major mode in a minor mode".

If necessary, we could have several derived modes of the new major mode
Enriched mode, that would essentially have the same effect as enabling
a second major mode, but in a way that does not try to "fight" with
Enriched mode.

Sincerely,

Luc. 

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

[...]
> I more and more agree with Oliver that we should have two Enriched
> modes.

Just in case this would seem weird to somebody: with `outline-mode'
and `outline-minor-mode' there is already a precedent for having
functionality both as a major mode and as a minor mode.

    Oliver
-- 
Oliver Scholz               27 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Stefan]

>    !       (add-to-list 'enriched-marked-buffers (current-buffer))))
>    !       (setq enriched-change-major-mode-flag t)
>    !       (enriched-mode 0)

> No, because enriched-mode sets enriched-change-major-mode-flag to nil,
> (It has to.)

Easy to fix.  BTW, why bother turning it off?
Just make sure that (enriched-mode 1) correctly sets up variables even if
the enriched-mode variable was already non-nil (that's good practice in
general anyway).

> Ayway, why try to find half-baked solutions to a completely _non-existent_
> problem?  `inhibit-quit' is only a problem around code that can take
> a non-trivial time to execute.

Right.  For example, code that calls encode-mode since encode-mode will run
the minor-mode hook which will run ...


        Stefan

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> I more and more agree with Oliver that we should have two Enriched
> modes.  The minor mode Enriched mode would be a true minor mode.  It
> would essentially only set ` buffer-file-format' and maybe do some
> additional minor stuff.  The major mode would replace the current
> "major mode in a minor mode".

Following the discussions on this thread, I agree with this.

The question is: how old are these problems with enriched-mode ?

If they were also in 21.1, I think we should just install Luc's patch
for 21.4 (as an interrim improvement), and plan a proper fix (separate
major and minor modes) for 22.1.

WDYT?

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Kim Storm wrote:

   The question is: how old are these problems with enriched-mode ?

They were present in 20.7.  In current CVS, the worst of the problems,
the fact that hard newlines failed to work, has been fixed already by
making `use-hard-newlines' a permanent local.  The fact that nobody
apparently complained about this probably means that currently people
only use Enriched mode to add some color and fontification to their text
and save it to file.  People who use it this way may actually consider
lack of hard newlines a _good_ thing.  I believe that the new _minor_
mode should not enable hard newlines by default.  Of course, the new
_major_ mode should.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    It seems you don't realize that a lot of elisp code suffers from such
    "not 100% correct state if C-g is hit at the wrong moment".  It's not such
    a big deal.

We do try to fix such problems when we find them.
Anyway, what do you think of the other solution
I proposed?

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Kim Storm wrote:
>
>    The question is: how old are these problems with enriched-mode ?
>
> They were present in 20.7.  In current CVS, the worst of the problems,
> the fact that hard newlines failed to work, has been fixed already by
> making `use-hard-newlines' a permanent local.  The fact that nobody
> apparently complained about this probably means that currently people
> only use Enriched mode to add some color and fontification to their text
> and save it to file.  People who use it this way may actually consider
> lack of hard newlines a _good_ thing.  I believe that the new _minor_
> mode should not enable hard newlines by default.  Of course, the new
> _major_ mode should.

So the second question is:  Do we really need to rework this for 21.4?

I suggest that we keep things as they are now for 21.4 (with your
patches installed), and add an entry to the TODO list for 22.1:

Rework enriched-mode into a major mode, enriched-mode, and a minor
mode, enriched-minor-mode.  [you can add your specific thoughts about
this split here].

Just my 2¢.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Richard Stallman]

    Even if `kill-all-local-variables' does not kill these variables, the
    major mode can override them.  In the case of `indent-line-function',
    most modes that are usually combined with Enriched mode will.

Maybe that one shouldn't be set by Enriched mode.
I think the motive for setting it was to make it easy
to preserve the left-margin indent of a paragraph.

Without this particular setting, how easy is it to do that?


It is useful to set 

    If necessary, we could have several derived modes of the new major mode
    Enriched mode, that would essentially have the same effect as enabling
    a second major mode, but in a way that does not try to "fight" with
    Enriched mode.

That is a very bad idea.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    Word
    processors assign properties to paragraphs, including defaults for
    character styles (like the font, the weight etc.); and they support
    style sheets for that.

Why can't we do something like that in Emacs using text properties?
We could perhaps have a text property on the whole paragraph
that indirects to a list of default properties, and then have other overriding
properties on specific characters in the paragraph.

Aside from data format, what would be the difference between a "style
sheet" and that list of default properties?

    It is very, very hairy to keep paragraphs,
    their properties and their representation in an Emacs buffer in sync,
    not to talk about style sheets.  In fact I do think that getting a WP
    UI right in Emacs is currently impossible.

Since you're saying something negative, I think you should fill in the
argument for this conclusion.  What methods have you considered?

    Indeed, I believe that in the long run Emacs' display engine should
    support a real block model.

Could you explain more clearly what you mean by that?

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Richard Stallman wrote:

       Even if `kill-all-local-variables' does not kill these variables, the
       major mode can override them.  In the case of `indent-line-function',
       most modes that are usually combined with Enriched mode will.

   Maybe that one shouldn't be set by Enriched mode.
   I think the motive for setting it was to make it easy
   to preserve the left-margin indent of a paragraph.

   Without this particular setting, how easy is it to do that?

It would not properly update `enriched-old-bindings'.  Also, while it
is unlikely that a major mode would try to override
`buffer-display-table' or `default-text-properties', the fact remains
that if `enriched-mode' continues to grow, the danger of one of its
variables being overridden by the major mode increases.

The following improved version of my original patch avoids these two
problems.  The improvements are that it is a lot simpler and no longer
relies on post-command-hook.  I noticed though that fundamental-mode
needs to be treated specially, since it does not run `run-mode-hooks'.
I guess that if any other major mode does not run `run-mode-hooks',
then this is a bug that needs to be fixed.

The patch to subr.el is unchanged.

===File ~/enriched.el-new-diff==============================
*** enriched.el	02 Jul 2004 21:39:16 -0500	1.4
--- enriched.el	13 Sep 2004 21:34:52 -0500	
***************
*** 1,6 ****
  ;;; enriched.el --- read and save files in text/enriched format
  
! ;; Copyright (c) 1994, 1995, 1996, 2002 Free Software Foundation, Inc.
  
  ;; Author: Boris Goldowsky <boris@gnu.org>
  ;; Keywords: wp, faces
--- 1,6 ----
  ;;; enriched.el --- read and save files in text/enriched format
  
! ;; Copyright (c) 1994, 1995, 1996, 2002, 2004 Free Software Foundation, Inc.
  
  ;; Author: Boris Goldowsky <boris@gnu.org>
  ;; Keywords: wp, faces
***************
*** 141,147 ****
  
  ;;; Internal variables
  
- 
  (defcustom enriched-mode-hook nil
    "Hook run after entering/leaving Enriched mode.
  If you set variables in this hook, you should arrange for them to be restored
--- 141,146 ----
***************
*** 155,160 ****
--- 154,164 ----
  The value is a list of \(VAR VALUE VAR VALUE...).")
  (make-variable-buffer-local 'enriched-old-bindings)
  
+ ;; Technical internal variable.  Bound to t if `enriched-mode' is
+ ;; being rerun by a major mode to allow it to restore buffer-local
+ ;; variables and to correctly adjust `enriched-old-bindings'.
+ (defvar rerun-flag nil)
+ 
  ;;;
  ;;; Define the mode
  ;;;
***************
*** 181,203 ****
  	 (while enriched-old-bindings
  	   (set (pop enriched-old-bindings) (pop enriched-old-bindings))))
  
! 	((memq 'text/enriched buffer-file-format)
  	 ;; Mode already on; do nothing.
  	 nil)
  
  	(t				; Turn mode on
! 	 (push 'text/enriched buffer-file-format)
  	 ;; Save old variable values before we change them.
  	 ;; These will be restored if we exit Enriched mode.
  	 (setq enriched-old-bindings
  	       (list 'buffer-display-table buffer-display-table
- 		     'indent-line-function indent-line-function
  		     'default-text-properties default-text-properties))
- 	 (make-local-variable 'indent-line-function)
  	 (make-local-variable 'default-text-properties)
! 	 (setq indent-line-function 'indent-to-left-margin ;WHY??  -sm
! 	       buffer-display-table  enriched-display-table)
! 	 (use-hard-newlines 1 nil)
  	 (let ((sticky (plist-get default-text-properties 'front-sticky))
  	       (p enriched-par-props))
  	   (dolist (x p)
--- 185,205 ----
  	 (while enriched-old-bindings
  	   (set (pop enriched-old-bindings) (pop enriched-old-bindings))))
  
! 	((and (memq 'text/enriched buffer-file-format)
! 	      (not rerun-flag))
  	 ;; Mode already on; do nothing.
  	 nil)
  
  	(t				; Turn mode on
! 	 (add-to-list 'buffer-file-format 'text/enriched)
  	 ;; Save old variable values before we change them.
  	 ;; These will be restored if we exit Enriched mode.
  	 (setq enriched-old-bindings
  	       (list 'buffer-display-table buffer-display-table
  		     'default-text-properties default-text-properties))
  	 (make-local-variable 'default-text-properties)
! 	 (setq buffer-display-table  enriched-display-table)
! 	 (use-hard-newlines 1 (if rerun-flag 'never nil))
  	 (let ((sticky (plist-get default-text-properties 'front-sticky))
  	       (p enriched-par-props))
  	   (dolist (x p)
***************
*** 207,212 ****
--- 209,228 ----
  		     (plist-put default-text-properties
  				'front-sticky sticky)))))))
  
+ (defun enriched-before-change-major-mode ()
+   (when enriched-mode
+     (while enriched-old-bindings
+       (set (pop enriched-old-bindings) (pop enriched-old-bindings)))))
+ 
+ (add-hook 'change-major-mode-hook 'enriched-before-change-major-mode)
+ 
+ (defun enriched-after-change-major-mode ()
+   (when enriched-mode
+     (let ((rerun-flag t))
+       (enriched-mode 1))))
+ 
+ (add-hook 'after-change-major-mode-hook 'enriched-after-change-major-mode)
+ 
  ;;;
  ;;; Keybindings
  ;;;
============================================================

===File ~/simple.el-diff====================================
*** simple.el	12 Sep 2004 15:43:59 -0500	1.660
--- simple.el	13 Sep 2004 22:16:08 -0500	
***************
*** 282,288 ****
    "Major mode not specialized for anything in particular.
  Other major modes are defined by comparison with this one."
    (interactive)
!   (kill-all-local-variables))
  
  ;; Making and deleting lines.
  
--- 282,289 ----
    "Major mode not specialized for anything in particular.
  Other major modes are defined by comparison with this one."
    (interactive)
!   (kill-all-local-variables)
!   (run-hooks 'after-change-major-mode-hook))
  
  ;; Making and deleting lines.
  
============================================================

===File ~/subr.el-diff======================================
*** subr.el	08 Sep 2004 11:29:29 -0500	1.407
--- subr.el	10 Sep 2004 14:39:21 -0500	
***************
*** 1937,1942 ****
--- 1937,1946 ----
  (make-variable-buffer-local 'delayed-mode-hooks)
  (put 'delay-mode-hooks 'permanent-local t)
  
+ (defvar after-change-major-mode-hook nil
+   "Mode independent hook run after changing major modes.
+ This is run just before the mode dependent hooks.")
+ 
  (defun run-mode-hooks (&rest hooks)
    "Run mode hooks `delayed-mode-hooks' and HOOKS, or delay HOOKS.
  Execution is delayed if `delay-mode-hooks' is non-nil.
***************
*** 1948,1953 ****
--- 1952,1958 ----
      ;; Normal case, just run the hook as before plus any delayed hooks.
      (setq hooks (nconc (nreverse delayed-mode-hooks) hooks))
      (setq delayed-mode-hooks nil)
+     (run-hooks 'after-change-major-mode-hook)
      (apply 'run-hooks hooks)))
  
  (defmacro delay-mode-hooks (&rest body)
============================================================

Re: enriched-mode and switching major modes.

[Stefan Monnier]

>        Even if `kill-all-local-variables' does not kill these variables, the
>        major mode can override them.  In the case of `indent-line-function',
>        most modes that are usually combined with Enriched mode will.
> 
>    Maybe that one shouldn't be set by Enriched mode.

I obviously agree (since I all but requested it in a previous message).

>    Without this particular setting, how easy is it to do that?
> It would not properly update `enriched-old-bindings'.

I lost some of the context, but enriched-old-bindings is (or should be)
permanent-local (at least as long as eriched-mode is).

> Also, while it is unlikely that a major mode would try to override
> `buffer-display-table' or `default-text-properties', the fact remains that
> if `enriched-mode' continues to grow, the danger of one of its variables
> being overridden by the major mode increases.

I wouldn't worry so much about the future: it leads to overengineering.
Also if you look at the past, you'd see that enriched-mode hasn't been
growing that much, so the risk is indeed not as clear as you seem to imply.


        Stefan

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     Word
>     processors assign properties to paragraphs, including defaults for
>     character styles (like the font, the weight etc.); and they support
>     style sheets for that.
>
> Why can't we do something like that in Emacs using text properties?
> We could perhaps have a text property on the whole paragraph
> that indirects to a list of default properties, and then have other overriding
> properties on specific characters in the paragraph.
>
> Aside from data format, what would be the difference between a "style
> sheet" and that list of default properties?
>
>     It is very, very hairy to keep paragraphs,
>     their properties and their representation in an Emacs buffer in sync,
>     not to talk about style sheets.  In fact I do think that getting a WP
>     UI right in Emacs is currently impossible.

The impossible part are tables (which I consider to be important).
table.el is a very nifty package for tables in documents that are
basically text/plain; but think of a table where the table cells of a
row contain text with different character and paragraph formatting
properties, for instance, column 1 has text with height 24 pt, column
2 only 12 pt, both in a proportional font and both having the same
line spacing.

The hairy part is whitespace formatting. The problems arise from the
fact that I can't tell Emacs: "Display this text from position POS1 to
POS2 as a paragraph with a left margin of 20 pt and a right margin of
40 pt with 20 pt above and below -- *without* adding any character to
the buffer." I am going to expand a little bit on these difficulties
below with a practical example. Parts of it are probably solvable; I
am not yet sure how reliable those solutions are.

> Since you're saying something negative, I think you should fill in the
> argument for this conclusion.  What methods have you considered?
>
>     Indeed, I believe that in the long run Emacs' display engine should
>     support a real block model.
>
> Could you explain more clearly what you mean by that?

I actually meant "box model". I am thinking of something like
specified by CSS 2 or CSS 3
<URL: http://www.w3.org/TR/2002/WD-css3-box-20021024/> (draft).

In short: a box model is an abstract way to specify the formatting of
a piece of character data on screen. Emacs' text-properties (those
affecting the display of text) could be regarded as "inline boxes" in
the terminology of that model, because they do not force the text to
which they apply to be displayed as a block (a "paragraph").

Block boxes are missing. CSS's block box model specifies margins
(between borders and surounding boxes), borders, padding (between
borders and content area) and content area as the four components of a
block box. In a picture:

+ - - - - - - - - - - - - - - - - - - - - - - - +
               Margin (Top)
|   +--------------------------------------+    |
    |          Padding (Top)               |
|   |   + - - - - - - - - - - - - - - - +  |    |
    |PL |                               |PR|
|ML |          Content Area                | MR |
    |   |                               |  |
|   |                                      |    |
    |   + - - - - - - - - - - - - - - - +  |
|   |          Padding (Bottom)            |    |
    +--------------------------------------+
|                                               |
               Margin (Bottom)
+ - - - - - - - - - - - - - - - - - - - - - - - +


If Emacs' display engine would support this, e.g. as a `block' text
property, then I could write:

(progn (switch-to-buffer (generate-new-buffer "*tmp*"))
       (insert "Example text. Example paragraph. Example text.")
       (put-text-property 15 33
                          'block
                          '(:margin (4 1 1 1) :border nil :padding nil)))

And then the text "Example paragraph" would get displayed as a
paragraph on its own with a left margin of four canonical character
units etc..  No inserting of newline characters or inserting of
spaces for the left margin involved here.

Other box types of the CSS include `list-item' for numbered or
bulleted lists or various table-boxes for specifying tables.

I am not bound to this particular model of CSS.  But I do think that
in the long run Emacs' display engine should support a visual
formatting model that is equally powerful.  The reason being, that I
envision Emacs-the-Word-Processor as an XML-centric application.
Even non-XML formats like RTF should be parsed into a data structure
that is an instance of the XML infoset (DOM or SXML, probably).  So
that users have a nice API for writing extensions to that WP in Emacs
Lisp.

So much for the answer to your question what I mean by "box model".
Now for the more concrete problem of implementing WP functionality for
Emacs with its current capabilities. The difficult part here is the
relation of data structure ("the document"), visual appearance ("the
formatting") and user interface.

With text/plain their relation is so simple that we hardly
distinguish them at all.  The visual appearance is determined by
control characters like space and newline, which are part of the
document (i.e. part of the data structure).  The user interface is
also simple: to change the (whitespace) formatting, we just insert
spaces and newlines where appropriate, which in turn become part of
the data structure.  To some extend this also works for
text/enriched.

But it stops to work for more elaborate, more widely used and --
IMNSHO -- more interesting document types and document formats.

Consider the following RTF document:

{\rtf1\ansi\deff0
{\fonttbl{\f0\froman Times;}{\f1\fswiss Helvetica;}}
{\stylesheet{\s1\f0\fs24\snext1 Standard;}
{\s2\keepn\f1\sb400\sa200\fs48 Headline;}
{\s3\sbasedon1\i\sb100\sa100\fs20\lin709 Motto;}}
{\*\listtable
{\list\listtemplateid1
{\listlevel\levelnfc23\leveljc0\levelstartat1\levelfollow2
{\leveltext\'01\u8226 ?;}} \listid1}}
{\listoverridetable{\listoverride\listid1\listoverridecount0\ls0}}
{\s2 Lirum larum (A Headline)}
{\par\pard\s3 "Mariage is the chief cause of divorce."}
\par\pard\plain\s1 This is just ordinary {\fs48 paragraph} text.
 Nothing special here.
\par\pard\plain\ls0\ilvl0 This is a list item. It contains two subitems:
\par\pard\plain\ls0\ilvl1 One and
\par\pard\plain\ls0\ilvl1 Two.
\par\pard\plain\ls0\ilvl0 This is another list item.}

A short explanation: Brackets group stuff together. Everything up to
line 10 ("{\listoverridetable ...") is header information. The
\fonttbl group specifies the fonts to use in the document. Each font
definition starts with \fN where N is a decimal number which is used
to refer to that font. The \stylesheet group defines stylesheets. Here
I only define paragraph stylesheets whose definition is started with
\sN. I define three paragraph styles here, "Standard", "Headline" and
"Motto". For example for the "Headline" style this specifies that a
"Headline"-paragraph should use the font "Helvetica" (\f1) with a
height of 24pt (\fs48), that it should be preceded by 20 pt vertical
whitespace (\sb400 -- the units are "Twips") and followed by 10 pt
vertical whitespace.  The rest of the header is important for
bulleted or numbered lists; I won't go into details here, because
that is a black art, which I have not yet fully mastered myself.

In the document itself \par starts a new paragraph and \sN refers to
a stylesheet.  \lsN\ilvlN is for list-items, again.

A plain/text approximation to the whitespace formatting of the
document (e.g. how it would be rendered on a tty) could look like
this:

---------------------- Start Document --------------------------------
Lirum larum (A headline)

    "Mariage is the chief cause of divorce."

This is just ordinary paragraph text. Nothing special here.
* This is a list item. It contains two subitems:
    1. One and
    2. Two
* This is another list item.
---------------------- End Document ----------------------------------

If Emacs display engine would support a block model, we would just
tell the display engine how to render the paragraphs. There is not a
single newline chars and no space between paragraphs that would be
part of the character data.  I.e.
`(buffer-substring-no-properties (point-min) (point-max))' would
return:

"Lirum larum (A headline)\"Mariage is the chief cause of divorce.\"\
This is just ordinary paragraph text. Nothing special here. This is\
 a list item. It contains two subitems:One and Two This is another \
list item."

(Note that the bullets and the numbers of the lists are not part of
the character data, either.)

Without a block model supported by the display engine, we have to
fake it by inserting newline characters and space (probably with a
`display' property) where appropriate.

In this case we would have to make sure that the UI is right. For
instance a user must not be able insert characters in a place where
"no character data are".  For instance, here:

Lirum larum (A Headline)
-!-
    "Mariage is the chief cause of divorce."


Or here:

* This is a list item. It contains two subitems:
-!- 1. One and
    2. Two.


The UI in typical word processors simply inhibits to move the cursor
to these places.  If the cursor is after "subitems:" and the user
hits <left>, the cursor would move before "One".

To get the same effect in Emacs we would have to make everything from
the newline after "subitems:" up to "1." intangible.  For this we
need a specialised fill function.  If we store the paragraph
properties in a text property, then this fill-function would


1) determine how far the paragraph extents, this could be, for
   instance, every text with an `eq' paragraph text property.

2) Remove every newline or space character that was inserted
   programatically by any previous filling.  Those newlines and
   spaces were not entered by the user and she does not want them to
   be part of her document.  They were added to the buffer only for
   visual rendering.

3) Determine the whitespace formatting properties of the paragraph.
   They may be specified via a stylesheet or directly or both (direct
   specification which overrides the defaults of a style sheet).

4) Add newline chars (word wrapping) and spaces (indentation) where
   appropriate to get a visual approximation to the paragraph
   properties specified in step 3). Those programatically added spaces
   and newlines should probably marked with a text property in order
   to make them distinguishable in step 2) from spaces that were
   entered by the user.


So far I have only talked about vertical and horizontal whitespace.
Character formatting information is another issue.  Take for example
this part from the RTF above:

\par\pard\plain\s1 This is just ordinary {\fs48 paragraph} text.

\s1 says: use paragraph stylesheet #1: Font: Helvetica; font-height:
12pt.  But this default for the paragraph is overriden by \fs48 for
the single word paragraph, it is meant to displayed with a font
height of 24 pt; however, this overrides only the height, all other
properties of the stylesheet do apply.

I guess this is best solved by letting font-lock look at the
paragraph properties, resolve all style information and then put an
according anonymous face on the `face' property.

Large parts of a WP may be possible in this or similar ways. Tables,
borders (and border styles), embedded vector graphics, multiple column
text are probably not feasible; but with the exeption of tables they
are IMO not /that/ important for now.

However, about one thing I am positiv: there is absolutely no room
for a minor mode here.  That's why I say that enriched-mode (as a
minor mode) is a dead end.

    Oliver
--
Oliver Scholz               29 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

A few minor corrections.

Oliver Scholz <alkibiades@gmx.de> writes:
[...]
> If Emacs' display engine would support this, e.g. as a `block' text
> property, then I could write:
>
> (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
>        (insert "Example text. Example paragraph. Example text.")
>        (put-text-property 15 33
>                           'block
>                           '(:margin (4 1 1 1) :border nil :padding nil)))

It should be noted, that a good box model should allow for nested
block boxes.  However, this all is blue sky anyways ...

[...]
> With text/plain their relation is so simple that we hardly
> distinguish them at all.  The visual appearance is determined by
> control characters like space and newline, which are part of the
> document (i.e. part of the data structure).  The user interface is
  ^^^^^^^^
> also simple: to change the (whitespace) formatting, we just insert
> spaces and newlines where appropriate, which in turn become part of
> the data structure.  To some extend this also works for
      ^^^^^^^^^^^^^^
> text/enriched.

I mean "the document's character data" here.  The important point is
that formats suitable for WP (RTF, HTML ...) separate character data
from formatting information entirely.  In text/plain on the other hand
the formatting information (LF, SPC, TAB, FF, VT) /is/ part of the
character data.

[...]
> Or here:
>
> * This is a list item. It contains two subitems:
> -!- 1. One and
>     2. Two.
>
>
> The UI in typical word processors simply inhibits to move the cursor
> to these places.  If the cursor is after "subitems:" and the user
> hits <left>, the cursor would move before "One".
       ^^^^^

I meant "<right>" here; or `C-f', even.


    Oliver
-- 
Oliver Scholz               29 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Stefan Monnier wrote:

   I lost some of the context, but enriched-old-bindings is (or should be)
   permanent-local (at least as long as eriched-mode is).

`enriched-old-bindings' should definitely _not_ be permanent local.
It is supposed to restore the major mode bindings overridden by
Enriched mode.  If the major mode changes, `enriched-old-bindings'
should be updated.

   I wouldn't worry so much about the future: it leads to overengineering.
   Also if you look at the past, you'd see that enriched-mode hasn't been
   growing that much, so the risk is indeed not as clear as you seem to imply.

What I am worried about is consistency.  If one sees (Foo Enriched)
in the mode line,  one should not have to worry about which of
Foo mode or Enriched mode was enabled first.  With my patch it does
not matter.  This is the only solid and predictable solution.

It is not just the future but also user customizations of Enriched
mode.  Also, if there are not going to be major future additions, then
we can quite as well quit worrying about Enriched mode altogether.  It
is very much an unfinished feature.  Enriched mode is supposed to grow
into a full blown word processor, supporting a variety of file
formats.  It is far from that right now.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Oliver Scholz wrote:

   However, about one thing I am positiv: there is absolutely no room
   for a minor mode here.  That's why I say that enriched-mode (as a
   minor mode) is a dead end.

Note that with my latest patch you can essentially treat Enriched mode
as a second major mode.  You are guaranteed that it runs _after_ the
major mode, even if the user changes the major mode with Enriched mode
enabled.  Hence you know that the major mode will not override any
setting that is essential for proper functioning of Enriched mode.

I do not believe that there is any requirement that Enriched mode be
compatible with every single major mode.  That would be completely
unrealistic and is currently already by no means the case.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

>From my previous message:

   Note that with my latest patch you can essentially treat Enriched mode
   as a second major mode.

I should clarify that this patch is currently under discussion and is
not installed in CVS at present.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Oliver Scholz wrote:
>
>    However, about one thing I am positiv: there is absolutely no room
>    for a minor mode here.  That's why I say that enriched-mode (as a
>    minor mode) is a dead end.
>
> Note that with my latest patch you can essentially treat Enriched mode
> as a second major mode.  You are guaranteed that it runs _after_ the
> major mode, even if the user changes the major mode with Enriched mode
> enabled.  Hence you know that the major mode will not override any
> setting that is essential for proper functioning of Enriched mode.

But it would be a second major mode that steps on the other major
mode's toes, hits it in the stomach and says: "Heya, buddie! If ya
don't move, don't breath and do what I tell you, we'll be friends.
Otherwise you better get outa here."

When it is supposed to meet the requirements that I described, then
enriched-mode would need to be in full control of filling and maybe
even of font-lock; I am even tempted to say that it needs to be in
full control of the buffer contents and the interface to the user
(since it would introduce a significant difference between the buffer
contents and the abstract document).  At any rate the other major
mode must not be allowed to programatically change the buffer
contents in a way that is supposed to affect the visual appearance,
because that would mean that the document data structure and its
visual appearance get out of sync, which is tantamount to corrupting
the file, if the user falls for it.

> I do not believe that there is any requirement that Enriched mode be
> compatible with every single major mode.  That would be completely
> unrealistic and is currently already by no means the case.

I see. But while I appreciate your solution, I don't see anymore what
problem it is supposed to solve.  It turned out in this discussion that
the reason why enriched-mode should be more than the usual minor mode
is its future role in WP, while it should still be possible to use it
together with some of the text-modes.  I tried to explain the
difficulties the WP in Emacs will encounter.  Based on this reasoning,
I maintain, that this future WP mode is by no means compatible with
any of the text modes.  No way here.

Maybe I am missing some particular usage scenario.  But with what
major modes would such a fully fledged future WP mode be supposed to
work together?

You might argue that especially with what I described switching major
modes would mean lossage of data for the user.  So either there
should be a yes-or-no query before switching major modes or said data
should be preserved (in this case at the cost that the major mode
being switched to is not fully functional).  I'd argue that it is not
clear that blurrying the major-minor mode distinction is the best
solution to this.  At any rate it is really a future problem.

    Oliver
-- 
Oliver Scholz               30 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Richard Stallman]

    It would not properly update `enriched-old-bindings'.

Yes, you are right.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    The hairy part is whitespace formatting. The problems arise from the
    fact that I can't tell Emacs: "Display this text from position POS1 to
    POS2 as a paragraph with a left margin of 20 pt and a right margin of
    40 pt with 20 pt above and below -- *without* adding any character to
    the buffer."

The idea of Emacs is that the appearance is represented by text in the
buffer.  We designed text properties so that the text in th ebuffer
could be something more than a sequence of characters.

Any extensions have to preserve this principle, or we will lose
the benefits of Emacs.

    If Emacs' display engine would support this, e.g. as a `block' text
    property, then I could write:

    (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
	   (insert "Example text. Example paragraph. Example text.")
	   (put-text-property 15 33
			      'block
			      '(:margin (4 1 1 1) :border nil :padding nil)))

If the block parameters are specified as a text property on the entire
contents of the block, that might solve the problem.  However, there
are some details here that are nontrivial problems.

1. How to distinguish between two similar boxes with the same specs
and a single longer box.

2. How to represent line breaks.  Saying "break one long line at
display time" would work ok for display, but all the commands that
operate on lines would see just one long line there.

3. How to represent indentation.  If the indentation appears only
in redisplay, Lisp code that looks at the text will think it is not
indented at all.

I think we need to look for a hybrid solution where there could be a
text property saying how this box is supposed to look, but that works
by inserting newlines, indentation, etc., so that the text can be seen
in Lisp without need to decode the box.

    If Emacs display engine would support a block model, we would just
    tell the display engine how to render the paragraphs. There is not a
    single newline chars and no space between paragraphs that would be
    part of the character data.  I.e.
    `(buffer-substring-no-properties (point-min) (point-max))' would
    return:

    "Lirum larum (A headline)\"Mariage is the chief cause of divorce.\"\
    This is just ordinary paragraph text. Nothing special here. This is\
     a list item. It contains two subitems:One and Two This is another \
    list item."

This model fails to address those problems.  It would work as a way
of grafting a separate word processing facility into Emacs, but it
would not integrate well with the existing Emacs Lisp world.

However, later you talk about an implementation more like what I have
in mind, where the boxes and lists would be rendered by changing the
buffer text; therefore, the buffer text would show what's really
there.

    I mean "the document's character data" here.  The important point is
    that formats suitable for WP (RTF, HTML ...) separate character data
    from formatting information entirely.

My point is that this is exactly what we must not do in Emacs, lest it
ruin everything in a subtle way.

    However, about one thing I am positiv: there is absolutely no room
    for a minor mode here.  That's why I say that enriched-mode (as a
    minor mode) is a dead end.

I don't see that that follows from the rest of your points.

Re: enriched-mode and switching major modes.

[Kai Grossjohann]

Richard Stallman <rms@gnu.org> writes:

> However, there are some details here that are nontrivial problems.
>
> 1. How to distinguish between two similar boxes with the same specs
> and a single longer box.

That's true.  Perhaps one could have a block-begin text property and a
block-end text property, both attached to the gap between two
characters?

> 2. How to represent line breaks.  Saying "break one long line at
> display time" would work ok for display, but all the commands that
> operate on lines would see just one long line there.

I think this is not too much of a problem.  Kim has done a great job
of changing the behavior of some Emacs commands to do cua-like
things.  I expect that a similar method would work to change the
behavior of beginning-of-line, next-line and so on.

> 3. How to represent indentation.  If the indentation appears only
> in redisplay, Lisp code that looks at the text will think it is not
> indented at all.

Here again, I think it is not a problem.  Why does the Lisp code need
to see the newline characters?  Why does it need to see the leading
whitespace?

Imagine that you have a feature which wraps long lines at word
boundaries at display time.  Imagine that this feature was configured
to wrap to the window width.  Now imagine that you make the window
narrower.  This introduces more wrapping points.  But which Lisp code
needs to see more newlines?

If we have source code, then we do need newline characters, but we're
talking about editing text.

Kai

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     I mean "the document's character data" here.  The important point is
>     that formats suitable for WP (RTF, HTML ...) separate character data
>     from formatting information entirely.
>
> My point is that this is exactly what we must not do in Emacs, lest it
> ruin everything in a subtle way.

[I moved that quote to the top, because the issue it adresses seems to
me to be at the root of our disagreement.  Everything I am going to
write below needs to refer to the distinction between the abstract
document and its visual rendering.  I am sorry if I am stealing your
time by flooding you with unsolicited information; but since we
discuss this prematurely, i.e. without me showing the code
implementing it, I want at least to clearly express my points.]

The distinction between character data and formatting information,
mentioned in the paragraph you quoted, is inherent in the type of
documents we discuss here.  You can enter as many newline characters
in the source of an RTF document as you want, an RTF reader will
simply remove them.  You can also put as many newline characters and
spaces into a HTML document as you want, a browser will just treat
each sequence of such whitespace control characters as a single space.

But probably you challenge my statement that it is important to keep
the distinction between the abstract document and its visual
representation in mind when discussion the implementation of WP
functionality.

This distinction is also inherent in the types of documents we discuss
here.  But first I should make clear what I mean by those terms.  I
say that we need to distinguish, at least conceptually, between: the
abstract document, the data structure that represents the abstract
document, the encoded file and the visual representation of the
abstract document.

1) the abstract document

   The abstract document is what the user of a word processor wants.
   She doesn't specify: "I want four space characters before each line
   in this area." nor "I want an <indent> tag before and an </indent>
   tag after this piece of text."  The former would refer to the way
   the visual representation could happen to be implemented in Emacs,
   the latter would refer to the document file format.  The user may
   be---and users of word processors typically are---blissfully
   ignorant of both.

   Instead the user says: "This piece of text is a paragraph, because
   I hit RET when I finished writing it.  I want it to be a paragraph
   of the type "Standard Text".  I want "Standard Text" paragraphs to
   be indented on the left by 1 cm and to have a font size of 10 pt.
   Exept here, I want this paragraph to have an indentation of 2 cm."

   I call the ensemble of character data and this latter specification
   of how the character data should "be like" the abstract document.
   It is what the user has expressed by typing text and interacting
   with the UI.  (It is thus important to design a good UI so that a
   user may express her wishes most clearly.)


2) the data structure used to represent that abstract document
   internally

   I list this explicetly, because there are quite some glitches here
   when implementing WP functionality in Emacs.  Among other things I
   suspect that I have failed to make one or two requirements for
   implementing the data structure in Emacs clear, if that data
   structure is meant to express abstract documents that are suitable
   for processing RTF (HTML ...).  More on this below.


3) the encoded document file

   This is the document as it would be written to a file: the concrete
   RTF or HTML file.  Not every file format is capable of expressing
   every abstract document.  The expressiveness of the available file
   formats would limit---or even shape---the possible abstract
   documents that a user may specify in a word processor.


4) the visual (aural) representation of the abstract document

   This is how the document is rendered on screen (or by a speech
   synthesizer; I won't further discuss aural rendering for now.)

   It is important to keep two things in mind:

       a) For a given word processor application, two different
          documents could look exactly the same when rendered on the
          display.

          For instance, in the document there could be specified as a
          paragraph property that the first line of a paragraph should
          be indented by 1 cm.  While in another document a user could
          get exactly the same visual effect by entering a number of
          space characters.  (Notabene: for a given application; when
          transfered to another application or to another machine,
          the appearances to those two documents could -- and probably
          would -- differ.)


       b) For a given document, two different applications or the
          same application on two different machines/operating
          systems might render two different visual representations.

          For example the font for a paragraph could be "Times New
          Roman" on MS Windows, but "Times" or even "New Century
          Schoolbook" on XFree.  Or the rendering device could be a
          tty. In that case word wrapping would happen at different
          places.  Or the user might have not specified a width for
          paragraphs at all, in that case the rendering could depend
          on application specific defaults or even on the size the
          application window happens to have.



             +---------------------+
             |  abstract document  |
             +---------------------+
                          ^
                          |
                          v
          +-----------------------------+
          |  data structure             |
          +-----------------------------+
            ^             |           |
            |             |           |
            |             v           v
+-------------+ +-----------------+ +------------+
|user commands| |visual appearance| |encoded file|
+-------------+ +-----------------+ +------------+



Even if we put these things together in Emacs, for one reason or
another, we need to keep said distinctions in mind---again: at least
conceptually---or else confusion will arise.  Any implementation of WP
functionality will have to account for how it deals with each of these
aspects or else Emacs would become infamous as the "so-called word
processor that provides a lot of surprises for its users."

The implementation that I suggested in my last posting (that one which
would work without a box model supported by the display engine),
strives to preserve the distinction by a) implementing a data
structure that stores those parts of the abstract document which
affect the formatting properties of paragraphs in a special text
property of the paragraph text, b) using font-lock and filling to
make these formatting properties visible.  Since filling would add
characters to the buffer content, my proposed implementation would
distinguish those characters added for visual appearance from the
characters which are part of the abstract document by means of
another special text property.

This is very important: If a user enters space characters into an
Emacs buffers, she wants there to be space characters.  Those
characters would have to become part of the character data in the
encoded file.  But if a user just specifies: I want this paragraph to
be indented, then the space characters used to display the left
margin _must_not_ become part of the encoded file.

I can not put too much emphasis on this.  Imagine the space characters
and the newlines as they appear in the buffer would be preserved.  A
encoded RTF file could look like this:

{\rtf\ansi
    This is just a paragraph.  It contains pointless example text.
    Habe nun ach, Philosophie, Juristerei und Medizin und leider auch
    Theologie durchaus studiert mit heissem Bemuehn. Da steh' ich nun
    ich armer Tor und bin so klug als wie zuvor.  To be or not to be
    this is the question.  Whether it is nobler in the mind ...}

If the user visits that file again with Emacs on the same system and
has not changed any of her customisations, then it would look the
same again.  So she would be content.

But try it!  Write that text above to a file and open it with
OpenOffice or AbiWord or whatever.  If I open it in OpenOffice on my
system, it looks like this (copied & pasted line by line, adding a
newline after each line):


    This is just a paragraph.  It contains pointless example text.    Habe nun ach, Philosophie, 
Juristerei und Medizin und leider auch    Theologie durchaus studiert mit heissem Bemuehn. Da 
steh' ich nun    ich armer Tor und bin so klug als wie zuvor.  To be or not to be    this is the question.  
Whether it is nobler in the mind ...


Obviously this is not an option.  `enriched-mode' has a nice UI bug
here.  If a range of text has a `left-margin' text property, say with
a value of 4, it removes the spaces on the left when saving and puts
according "<indent> ... </indent>" tags around it.  But the fill
function of text-mode also allows the user to insert just four spaces
at the beginning of a paragraph, auto-fill will then fill it with four
spaces indentation.  In the file, this will result exactly in what I
described for RTF above. This is by no means distinguishable from the
case where there is a `left-margin' text property, except by carefully
examining several pieces of the buffer text with `C-u C-x ='.

This bug has a great change to go unnoticed for a while or even
forever, because a) text/enriched is an extremely simple markup
format, b) there is probably not other application that deals with
text/enriched files (contrary to text/enriched e-mail) and c) the
text/enriched files of Emacs specify the value of fill-column in their
"header".  So even if the document is transfered to another user who
has a different default value for fill-column, the buffer's
fill-column will be the same as in the Emacs were the document was
created.  The circumstances under which the bug could show up are
probably rather rare.

We can by absolutely no means rely on things like this for RTF or HTML
or DocBook or TEI XML.  Those formats are far too complex.  Those
formats are widely used.  Such inconsitencies in the user interface
would inevitably result in more bugs of this kind which would
inevitably show up.  We do have to design WP functionality very
carefully in order to distinguish between the abstract document and
its visual appearance.  And the user interface must at each point
provide enough feedback about what is part of which.

This bug is due to an ambiguous user interface in enriched-mode. There
is an easy fix here: let the fill function indent a paragraph if and
only if there is a `left-margin' text property.

Thank goodness, text/enriched has no concept of character formatting
properties applying to a paragraph (a block box), and thus the
abstract document that can be produced in enriched-mode does not
provide that either; which fits well to Emacs, because the display
engine also does not know that concept.  In an Emacs buffer as well as
in a text/enriched file paragraph boundaries are defined by the
newline characters that precede and follow it; character formatting
properties in an Emacs buffer are specified with text properties and
in a text/enriched file with tags.  So these fit together.

Not so RTF.  Not so HTML.  A paragraph is specified not by newline
characters before or after it, but by syntactical markup.  Character
formatting properties may apply not only to a range of characters (an
"inline box") but also to a whole paragraph (a "block box").

So where am I supposed to store those character formatting properties
applying to the whole paragraph?  The only option I have are text
properties.  But since text properties do not define a paragraph
visually (the display engine does not support this), I have to make
sure that a range of text defined to be a paragraph by means of text
properties, does /appear/ as a paragraph in the buffer.  That means
that I have to define a fill-function as described in my last posting.

Perhaps I did not make one point clear enough: this fill function must
have the _total_control_ over the whitespace formatting in the buffer.
There must not any other Lisp function interfere here.  Or else the
abstract document/the data structure and the appearance on the screen
would get out of sync.  If the user wrongly thinks that the visual
appearance matches the abstract document as specified in the data
structure and saves the file, she will be painfully surprised the next
time she visits it.  In other words: we would have introduced an
indirect way of corrupting the user's files.  We would have introduced
a myriad of possibilities for bugs like the one in enriched-mode
described above.

>     The hairy part is whitespace formatting. The problems arise from the
>     fact that I can't tell Emacs: "Display this text from position POS1 to
>     POS2 as a paragraph with a left margin of 20 pt and a right margin of
>     40 pt with 20 pt above and below -- *without* adding any character to
>     the buffer."
>
> The idea of Emacs is that the appearance is represented by text in the
> buffer.  We designed text properties so that the text in th ebuffer
> could be something more than a sequence of characters.
>
> Any extensions have to preserve this principle, or we will lose
> the benefits of Emacs.

I obviously disagree, since I argued for an enhancement that goes
well beyond this principle.  I fail too see at all how we would lose
the benefits of Emacs this way.  As I see it, it would only add to
Emacs capabilities, extend the domain of documents it is suitable for
and enlarge its benefits this way.

>     If Emacs' display engine would support this, e.g. as a `block' text
>     property, then I could write:
>
>     (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
>          (insert "Example text. Example paragraph. Example text.")
>          (put-text-property 15 33
>                             'block
>                             '(:margin (4 1 1 1) :border nil :padding nil)))
>
> If the block parameters are specified as a text property on the entire
> contents of the block, that might solve the problem.  However, there
> are some details here that are nontrivial problems.
>
> 1. How to distinguish between two similar boxes with the same specs
> and a single longer box.

I have not spent much thought on this;  I have only spent quite some
time trying to grok the display engine (in vain, so far) and have not
come to a stage where I could make some reasonably funded proposals
for the design of that feature.  Maybe they could programatically be
distinguished by checking for `eq'uality rather than `equal'ity.
Maybe for the user this wouldn't suffice, because they would look
the same after `C-x =' which could or could not be confusing.

I have no idea how to specify /nested/ block boxes with text
properties, either.  This is also a problem.

> 2. How to represent line breaks.  Saying "break one long line at
> display time" would work ok for display, but all the commands that
> operate on lines would see just one long line there.

`C-a', `C-p' and the like would need to be enhanced to deal with this.
Since it should of course be possible to check from Lisp whether some
buffer text is the content of a box, I don't see any problems here
which would be greater then the problem of implementing the display
feature itself.  In fact, there are at least three packages that
already implement this for wrapped long lines:  screen-lines.el,
screenline.el and window-lines.el. (Ouch!  They stopped to work with
the current CVS version ...)

> 3. How to represent indentation.  If the indentation appears only
> in redisplay, Lisp code that looks at the text will think it is not
> indented at all.

I actually regard this as a feature.  In WP documents the left margin
has no more significance than the right margin, which is not currently
implemented by adding space characters, either.  Functions that, for
instance, match on the space characters used to display a left margin,
would match on something which is not part of the character data of
the document: if the user sends that document to somebody else using
another word processor or uses another word processor herself, they
simply won't be there.

And with the implementation that I proposed in my last mail: if an
arbitrary Lisp function not part of the WP mode actually changes the
contents of a buffer containing a WP document, they better change only
the character data, not any spaces or newlines added programatically
for whitespace formatting or else they could wreak havoc in an
intolerable way.  

But the display feature I proposed would greatly increase the chance
that existing Emacs Lisp functions would seamlessly do the right thing
in a buffer containing a such a WP document.  Right now they work on
character data only and they should continue to work on character data
only in a WP document.

> I think we need to look for a hybrid solution where there could be a
> text property saying how this box is supposed to look, but that works
> by inserting newlines, indentation, etc., so that the text can be seen
> in Lisp without need to decode the box.

You mean so that those characters are added and updated
automagically?   Hmm, I seem to recall that I have experimented a bit
with using jit-lock to do the whitespace formatting, when I first
started to think about WP in Emacs.  But I don't remember how well
that worked.  I did not follow that approach further, because it
seemed unorthodox to me at that time.

However, something like this could at least deal with some of the
problems of keeping visual representation and the abstract document in
sync, so it would be at any rate a big improvement.

However, the implementation strategy that I described is meant as a
temporary solution to implement a box model in Lisp without support
from the display engine.  If you see it as a final state, moreover,
if you think that it is fundamentally different from a box model,
then I have failed to explain it.

>     If Emacs display engine would support a block model, we would just
>     tell the display engine how to render the paragraphs. There is not a
>     single newline chars and no space between paragraphs that would be
>     part of the character data.  I.e.
>     `(buffer-substring-no-properties (point-min) (point-max))' would
>     return:
>
>     "Lirum larum (A headline)\"Mariage is the chief cause of divorce.\"\
>     This is just ordinary paragraph text. Nothing special here. This is\
>      a list item. It contains two subitems:One and Two This is another \
>     list item."
>
> This model fails to address those problems.  It would work as a way
> of grafting a separate word processing facility into Emacs, but it
> would not integrate well with the existing Emacs Lisp world.

I don't understand why you say that.  And I don't know which parts of
the existing Lisp world you mean.

`query-replace' and isearch, for example, would do the right thing.
Not so with the replacement for a real box model that I described.
`query-replace' could even indirectly lead to file corruption, as
explained above.

Please note also, that a real box model supported by the display
engine is the only way to get tables with the capabilities that
people will expect (I am talking about columns with different font
heights here and about table borders).  It is not possible to mimic
it without.

> However, later you talk about an implementation more like what I have
> in mind, where the boxes and lists would be rendered by changing the
> buffer text; therefore, the buffer text would show what's really
> there.

Erm, what does the concept of "what's really there" in that context
mean?  In the buffer, or more generally spoken: in the data structure
a containing block box, or a text property storing formatting
information is, of course, no less there than any space or newline
character added for whitespace formatting.  But when Emacs writes the
WP document out to a file a again, say an RTF file, then it would
/remove/ those space and newline characters.  It definitely has to;
everything else would be a most serious bug.  So, spoken from the point
of view of the file format, those space and newline characters are not
"there" at all.

>     However, about one thing I am positiv: there is absolutely no room
>     for a minor mode here.  That's why I say that enriched-mode (as a
>     minor mode) is a dead end.
>
> I don't see that that follows from the rest of your points.

I don't understand why you say that.  I described an implementation
that needs to be in full control of every aspect of whitespace
formatting (at the very least).  I don't see how that could
justifiedly be implemented as a minor mode.  And if it were, I don't
see with what major modes it should be able to work together.

I can think of a third implementation strategy for editing RTF files
that /could/ be implemented as a minor mode; and if enriched-mode
would be slightly changed, it could even be part of
enriched-the-minor-mode.  I don't want to undermine my own position,
but here it is:

It would be an heuristical approach.  The encoding function (when
saving to a file) could determin whether a paragraph (defined by its
preceding and following newline characters) is indented by space
characters.  It could then write the nessary markup to the file
accordingly.  The problem of character formatting properties applying
to a paragraph could be solved in a similar, heuristical way:
determine whether---let's say---a `paragraph-stylesheet' with a
particular property predominates a paragraph; if so, write the
according markup again.

There are two important consequences implied in this approach: 

About the first one I am not 100% sure. But it is quite possible that
We would not even try to come close to the visual appearance of the
document in other word processing applications. One might say: we
would try to preserve only the syntactial properties of the document
when reading it.

The other consequence is that this approach will invariable fail
sometimes; this is implied in the fact that this approach is
heuristical.  If it is carefully implemented, it might work well
enough in common cases to be useful.  I don't know; that area would
need to be explored.  Bulleted and numbered lists would probably have
to be implemented as an exeption to the heuristical approach in a way
that is similar to what table.el does for tables.

However: it would be an incredible exaggeration to call Emacs a "word
processor" if it followed such an heuristical approach. And personally
I am not interested in pursuing it.  Not the least reason is that some
particular aspects of what I personally want for WP in Emacs would not
be feasible this way.  This is not the time to discuss the whole abyss
of my ideas.  Yet I want, for instance, an API that lets people access
the contents of a WP document syntactically by a query language (to
name the devil: XPath); so that people could specify in Lisp: give me
all text areas that are of the type "headline 2" etc.  I mention this
approach only for completeness.

    Oliver
--
Oliver Scholz               30 Fructidor an 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Oliver Scholz <alkibiades@gmx.de>
> Date: Thu, 16 Sep 2004 19:04:53 +0200
> Cc: boris@gnu.org, emacs-devel@gnu.org, alex@emacswiki.org
> 
> 2) the data structure used to represent that abstract document
>    internally

Note that in Emacs, that data structure is the buffer text (with
associated text properties and any Lisp code that those properties
evaluate).  This is very important thing to understand: there are no
data structures built by Emacs from the text it reads except what it
puts into the buffer.

> > The idea of Emacs is that the appearance is represented by text in the
> > buffer.  We designed text properties so that the text in th ebuffer
> > could be something more than a sequence of characters.
> >
> > Any extensions have to preserve this principle, or we will lose
> > the benefits of Emacs.
> 
> I obviously disagree, since I argued for an enhancement that goes
> well beyond this principle.  I fail too see at all how we would lose
> the benefits of Emacs this way.

I think Richard refers to the fact that your suggestions are so
incompatible with the design principles and the current workings of
the Emacs display engine that implementing them would mean working
against the Emacs design.

The solutions you propose for the problems raised by Richard (how to
deal with line breaks and indentation from Lisp) already show a
potential to require changes in too many parts of Emacs, including in
lots of Lisp code.  That is already an evidence that trying to display
too many things that are not in the buffer is a path we should avoid.

> > 3. How to represent indentation.  If the indentation appears only
> > in redisplay, Lisp code that looks at the text will think it is not
> > indented at all.
> 
> I actually regard this as a feature.  In WP documents the left margin
> has no more significance than the right margin, which is not currently
> implemented by adding space characters, either.

That's not what Richard was thinking and talking about, AFAIU.  One
example of where the indentation _does_ matter, even in a word
processor, is when the mouse pointer hovers above a displayed portion
of text: you need to be able to compute the buffer character position
which is displayed below the mouse pointer.  If the code that computes
that doesn't know about indentation, you will never get that right.
This breaks the mouse-highlight feature and the tooltips, for
starters.

> > This model fails to address those problems.  It would work as a way
> > of grafting a separate word processing facility into Emacs, but it
> > would not integrate well with the existing Emacs Lisp world.
> 
> I don't understand why you say that.  And I don't know which parts of
> the existing Lisp world you mean.

The parts that do their work by walking the buffer text and making
decisions on how that text is displayed.

> > However, later you talk about an implementation more like what I have
> > in mind, where the boxes and lists would be rendered by changing the
> > buffer text; therefore, the buffer text would show what's really
> > there.
> 
> Erm, what does the concept of "what's really there" in that context
> mean?

What's really there in the buffer.

Re: enriched-mode and switching major modes.

[Richard Stallman]

This change seems good to me.  Please install it.

Could you also add after-change-major-mode-hook to NEWS
and to lispref/modes.texi?

Re: enriched-mode and switching major modes.

[Oliver Scholz]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> From: Oliver Scholz <alkibiades@gmx.de>
>> Date: Thu, 16 Sep 2004 19:04:53 +0200
>> Cc: boris@gnu.org, emacs-devel@gnu.org, alex@emacswiki.org
>> 
>> 2) the data structure used to represent that abstract document
>>    internally
>
> Note that in Emacs, that data structure is the buffer text (with
> associated text properties and any Lisp code that those properties
> evaluate).  This is very important thing to understand: there are no
> data structures built by Emacs from the text it reads except what it
> puts into the buffer.

Fine. In that case Emacs can not become a word processor. Or maybe it
already is: the first "word processor" which supports only the file
formats text/plain and text/enriched.

That means I can stop thinking about implementing word processor
functionality for Emacs right now and go back to use it just as a
programmer's editor.  It is not the application then, that I thought
it would become.

Sorry, for having been a nuisance, then.


    Oliver
-- 
Oliver Scholz               Jour du Génie de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Robert J. Chassell]

   >     I mean "the document's character data" here.  The important point is
   >     that formats suitable for WP (RTF, HTML ...) separate character data
   >     from formatting information entirely.
   >
   > My point is that this is exactly what we must not do in Emacs, lest it
   > ruin everything in a subtle way.

I think there is a confusion here:  internally, Emacs works with a
`deep representation'.  People mostly work with `surface expressions'.
Nowadays, the two can be different.  In the old days, they were similar.

(Oliver Scholz calls a deep representation the `encoded document file'
and the surface expression the `visual (aural) representation' or
`rendering'.)

An Info file has a `deep representation' that you can see with
`find-file-literally'.  

But many people look at Info files using `info' -- and what they see
(or hear) is a `surface expression'.  The Info `rendering' may convert
a chapter heading with a line of asterisks below it to a heading
without the asterisks and with a different font or voice.

In its earliest days, Emacs always used what we now call
`find-file-literally'.  The rendering you saw was the deep
representation.

But ever since text properties and overlays were introduced, the deep
representation has become different, or potentially different, from
its rendering.  (Images also look different than their deep
representation.)

The question is how far to go?

  * Should an HTML formatted line be displayed with <em>
    formatting </em> commands in it, as it is when using `find-file'
    or `find-file-literally', but with a different font or voice for
    the emphasized word when in W3 mode?  (This is how it is now.)

  * Should a surface expression or rendering that is quite different
    from the deep representation be presented only in a read-only
    buffer?  Or should a person be able to edit that rendering and be
    confident that the underlying `encoded document file' or deep
    representation will incorporate the correct tag?

    Put another way, when you type a command such as `C-c C-c i'
    (tempo-template-html-emphasized) in a buffer in HTML Helper mode,
    the string `<em></em>' is inserted.

    Should you also be able to type that same command into a buffer in
    W3 mode and have that string be inserted properly in the `encoded
    document file'?

Enriched mode illustrates this very well.  When you run `find-file' on
etc/enriched.doc you will see a line like this

    Colors: anything your screen can display...

but with many colors.  However, if you run `find-file-literally' on
that same file, that line looks like this:

    <bold>Colors:</bold> <x-color><param>red</param><x-bg-color><param>DarkSlateGray</param><indent>any</indent></x-bg-color></x-color><x-bg-color><param>DarkSlateGray</param><indent><x-color><param>orange</param>thing</x-color> <x-color><param>yellow</param>your</x-color><x-color><param>green</param> screen</x-color><x-color><param>blue</param> </x-color><x-color><param>light blue</param>can</x-color><x-color><param>violet</param> display...</x-color></indent></x-bg-color>

The surface expression or rendering is different from the deep
representation.

Emacs already has shifted away from its beginnings.

The question is not whether

    ... the appearance is represented by text in the buffer.  

That misses the point.  One kind of appearance should always be shown
when you run `find-file-literally'.  That should represent what is
saved, the `encoded document file'.  That should be what is shown when
you run `cat' on the file in a command line interface.

The other kind of appearance should be shown when a display mode is
invoked, as with Enriched mode and `find-file' or  W3 mode and `w3'.

The basic question is whether to install writing capabilities in
surface expression buffers.  Should typing `C-c C-c i' in a W3 mode
buffer insert `<em></em>' in a deep representation buffer?

(If the latter, you should be able to look at the same HTML encoded
buffer in several different modes in different windows:  in W3 mode,
in HTML Helper mode, and in whatever minor mode `find-file-literally'
uses.)

Texinfo already has one deep representation or `encoded document' form
that is shown by `find-file-literally' and five `surface expressions'.
(The surface renderings, however, do not come from the same file as
the deep representation.)

    Texinfo mode, which is read-writable 
    Info mode (and Emacspeak aural from Info)
    HTML
    XML
    DVI (and other printed)

In the old days, surface expression rendering came directly from the
rendition of the `encoded document' form, just like 
`find-file-literally' does now.

Info showed files with asterisks below a chapter title.  In those
days, the only way to display the same document differently in
different modes was to use different files.  That ended and so
`find-file-literally' had to be invented.

At the moment, in Texinfo, the Texinfo mode (or `find-file-literally')
file is writable.  You cannot edit an Info expression and have the
results show well in a printed book.

Incidentally, I regularly use four of these renderings.  I write and
edit in Texinfo mode and I publish the resulting document in Info,
HTML, DVI, PDF, and PostScript.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: enriched-mode and switching major modes.

[Kai Grossjohann]

Oliver Scholz <alkibiades@gmx.de> writes:

> That means I can stop thinking about implementing word processor
> functionality for Emacs right now and go back to use it just as a
> programmer's editor.  It is not the application then, that I thought
> it would become.

What a pity.


I agree with Oliver and with Bob: Emacs already displays something
that's different than what is present in the file, and people like
that.  Oliver is just proposing to go further in that direction.
Those working on bidi are also proposing to go further in that
direction.

Perhaps one can make small steps.  One step could be to allow
word-wrap in the display engine.  Then one could start to work out the
issues with missing newlines in the buffer text.

For this part, we already have another implementation, longlines.el,
that works by frobbing the buffer text.

I do not expect the display engine implementation to be inferior.

Kai

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Kai Grossjohann <kai@emptydomain.de> writes:

> I agree with Oliver and with Bob: Emacs already displays something
> that's different than what is present in the file, and people like
> that.  Oliver is just proposing to go further in that direction.
> Those working on bidi are also proposing to go further in that
> direction.


I have various wild ideas how to improve emacs towards a block structure
which is don't require a lot of changes to emacs commands.

One idea, is to allow stacking of images and text flow around images.

Another idea is to create "floating" windows which are inserted just
like an image -- but with all the normal emacs editing features of a
normal emacs window (except modeline, scrollbars, fringes, etc) 

Combine these with display word wrap, you can get pretty close to WP
capabilities.

But that's AFTER 21.4 :-)

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Richard Stallman]

    That's true.  Perhaps one could have a block-begin text property and a
    block-end text property, both attached to the gap between two
    characters?

Text properties are attached to characters, not to gaps.
I don't think this idea could make sense.

Another idea is to give each box a "name", a Lisp object,
which has to be eq in order for them to be the same box.
It could be a list of data that defines the box characteristics.

    > 2. How to represent line breaks.  Saying "break one long line at
    > display time" would work ok for display, but all the commands that
    > operate on lines would see just one long line there.

    I think this is not too much of a problem.  Kim has done a great job
    of changing the behavior of some Emacs commands to do cua-like
    things.  I expect that a similar method would work to change the
    behavior of beginning-of-line, next-line and so on.

Sorry, I do not want to have to change each command to
explicitly recognize lines in two different ways.

    Imagine that you have a feature which wraps long lines at word
    boundaries at display time.  Imagine that this feature was configured
    to wrap to the window width.  Now imagine that you make the window
    narrower.  This introduces more wrapping points.  But which Lisp code
    needs to see more newlines?

Whatever code operates on lines.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    The distinction between character data and formatting information,
    mentioned in the paragraph you quoted, is inherent in the type of
    documents we discuss here.  You can enter as many newline characters
    in the source of an RTF document as you want, an RTF reader will
    simply remove them.

That is not relevant to the issue, as it concerns the detailed
specifications of RTF format, rather than the detailed specifications
of Emacs' internal representation.  We can easily implement this
behavior in the format translators for RTF, but it doesn't really
relate to the issue here.

       Instead the user says: "This piece of text is a paragraph, because
       I hit RET when I finished writing it.  I want it to be a paragraph
       of the type "Standard Text".  I want "Standard Text" paragraphs to
       be indented on the left by 1 cm and to have a font size of 10 pt.
       Exept here, I want this paragraph to have an indentation of 2 cm."

Now you're talking about the user interface concepts.  That too is
distinct from the question of representing text inside Emacs.  The
Emacs text representation needs to include a representation of the
specification of a type of paragraph, in order to implement this user
interface.  But it also needs to have spaces in the buffer where there
is indentation, so as to be compatible with how the whole Emacs Lisp
world understands indentation.

	   a) For a given word processor application, two different
	      documents could look exactly the same when rendered on the
	      display.

	      For instance, in the document there could be specified as a
	      paragraph property that the first line of a paragraph should
	      be indented by 1 cm.  While in another document a user could
	      get exactly the same visual effect by entering a number of
	      space characters.

It is probably inevitable that such indeterminacy will occur,
but let's try to avoid it when we can.  For instance, by keeping
indentation expressed as paragraph style specifications in sync
with indentation expressed as spaces in the buffer.

	   b) For a given document, two different applications or the
	      same application on two different machines/operating
	      systems might render two different visual representations.

This is not specifically a problem, and may even give us extra
flexibility.

    This is very important: If a user enters space characters into an
    Emacs buffers, she wants there to be space characters.  Those
    characters would have to become part of the character data in the
    encoded file.  But if a user just specifies: I want this paragraph to
    be indented, then the space characters used to display the left
    margin _must_not_ become part of the encoded file.

Why do you think so?  It seems to me that these two different user actions
should both produce spaces in the buffer--in one case, inserted manually,
in the other, caused by the format specification.

We can recompute line breaks automatically, and represent them by
newline characters in the buffer, which will be removed and replaced
by the next recomputation.  In this mode, the user would not manually
enter newline characters except to create breaks (hard newlines).

    Erm, what does the concept of "what's really there" in that context
    mean?  In the buffer, or more generally spoken: in the data structure
    a containing block box, or a text property storing formatting
    information is, of course, no less there than any space or newline
    character added for whitespace formatting.

Something is "really there" if other Lisp programs will see it in the
way that they are written to look for it.

    It would be an heuristical approach.  The encoding function (when
    saving to a file) could determin whether a paragraph (defined by its
    preceding and following newline characters) is indented by space
    characters.  It could then write the nessary markup to the file
    accordingly.

I agree with you there.  I'm not suggesting anything like that.

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> Cc: boris@gnu.org,  emacs-devel@gnu.org,  alex@emacswiki.org
> From: Oliver Scholz <alkibiades@gmx.de>
> Date: Fri, 17 Sep 2004 16:34:01 +0200
> 
> > Note that in Emacs, that data structure is the buffer text (with
> > associated text properties and any Lisp code that those properties
> > evaluate).  This is very important thing to understand: there are no
> > data structures built by Emacs from the text it reads except what it
> > puts into the buffer.
> 
> Fine. In that case Emacs can not become a word processor. Or maybe it
> already is: the first "word processor" which supports only the file
> formats text/plain and text/enriched.
> 
> That means I can stop thinking about implementing word processor
> functionality for Emacs right now and go back to use it just as a
> programmer's editor.  It is not the application then, that I thought
> it would become.
> 
> Sorry, for having been a nuisance, then.

That's a very far cry from what I intended.  I didn't mean to say that
the idea of having more word-processing features in Emacs was futile,
much less that your messages were a nuisance.  All I was saying was
that Emacs design has some basic assumptions which are better left
alone.

It is possible that what I wrote was misunderstood to mean that
there's nothing in the buffer but plain text.  Others in this thread
pointed out something that I assumed was known and obvious: that text
properties are also part of buffer text, and that they can be used to
hold the additional meta-information required for word-processing
features.

FWIW, I think there _is_ a way to add sophisticated word-processing
capabilities to Emacs without breaking the basic Emacs design
assumptions.  One good idea was already suggested in this thread: when
Emacs decodes a file which has embedded information about the text
layout, it should convert that information into a combination of
characters and text properties, such that Lisp programs and C code
that look at the buffer text could extract the layout information from
that text alone.  In other words, convert indentation to the
appropriate amount of whitespace, add newlines where line-wraps are
required (and put some text properties on those added characters to be
able to treat them specially, e.g. at save-file time), add text
properties for paragraph-level style information, etc.

If you think that this approach will not work, can you tell why?

Re: enriched-mode and switching major modes.

[David Kastrup]

"Eli Zaretskii" <eliz@gnu.org> writes:

> FWIW, I think there _is_ a way to add sophisticated word-processing
> capabilities to Emacs without breaking the basic Emacs design
> assumptions.  One good idea was already suggested in this thread: when
> Emacs decodes a file which has embedded information about the text
> layout, it should convert that information into a combination of
> characters and text properties, such that Lisp programs and C code
> that look at the buffer text could extract the layout information from
> that text alone.  In other words, convert indentation to the
> appropriate amount of whitespace, add newlines where line-wraps are
> required (and put some text properties on those added characters to be
> able to treat them specially, e.g. at save-file time), add text
> properties for paragraph-level style information, etc.
>
> If you think that this approach will not work, can you tell why?

If I do search&replace on text that is in a run-in paragraph, I don't
want to cater for CR and multiple spaces that are just added for the
sake of appearance.  If my text is conceptually a long line separated
by single spaces (but displayed as a formatted block), then I want my
buffer modification attempts to work on exactly that, and nothing
else.

The user-accessible cursor commands, of course, should try to do
_visual_ motion usually.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> Cc: Oliver Scholz <alkibiades@gmx.de>,  boris@gnu.org,  alex@emacswiki.org,
> 	  emacs-devel@gnu.org
> From: David Kastrup <dak@gnu.org>
> Date: Sat, 18 Sep 2004 14:04:54 +0200
> 
> If I do search&replace on text that is in a run-in paragraph, I don't
> want to cater for CR and multiple spaces that are just added for the
> sake of appearance.

I agree; these commands will need to know about the special text
properties and skip/ignore such characters.

However, similar modifications to search/replace commands are required
anyway for Unicode-compatible text editing (e.g., canonical
decompositions and combining sequences, see UAX#15).

Also, it seems to me that they (the modifications) are less intrusive
than what we will need if the displayed text is different from what a
Lisp program can find in the buffer.

Re: enriched-mode and switching major modes.

[David Kastrup]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Cc: Oliver Scholz <alkibiades@gmx.de>,  boris@gnu.org,  alex@emacswiki.org,
>> 	  emacs-devel@gnu.org
>> From: David Kastrup <dak@gnu.org>
>> Date: Sat, 18 Sep 2004 14:04:54 +0200
>> 
>> If I do search&replace on text that is in a run-in paragraph, I don't
>> want to cater for CR and multiple spaces that are just added for the
>> sake of appearance.
>
> I agree; these commands will need to know about the special text
> properties and skip/ignore such characters.
>
> However, similar modifications to search/replace commands are required
> anyway for Unicode-compatible text editing (e.g., canonical
> decompositions and combining sequences, see UAX#15).
>
> Also, it seems to me that they (the modifications) are less intrusive
> than what we will need if the displayed text is different from what a
> Lisp program can find in the buffer.

I can't see that.  The entities I want to manipulate with Lisp
programs are typically the contents, not the appearance of the text.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: enriched-mode and switching major modes.

[Robert J. Chassell]

As Kai Grossjohann said,

    Emacs already displays something that's different than what is present
    in the file

Yes, indeed.  For a high resolution, word-processor-style display, two
questions arise.  But first, a look at the past.

Version 18 Emacs had:

  * the same rendering device (a character-only display screen) 
     that used different Emacs rendering programs
      to produce differently displayed expressions
        of the same info formatted file; for example:

            in Emacs, find-file  
            in Emacs, Info

  * different rendering devices (a character-only display screen, paper)
    that used different rendering programs, some not Emacs, 
      to produce differently displayed expressions
        of the same DVI formatted file; for example:

            in Emacs,     find-file
            not in Emacs, xdvi in X
            not in Emacs, printer

  * different conversion programs, some not Emacs,
        to produce different files 
        from the same Texinfo formatted file; for example:

            in Emacs,     texinfo-format-* to convert Texinfo to Info
            not in Emacs, tex to convert Texinfo to DVI

Version 21 Emacs supports three rendering devices:

  * character-only display screen
  * bit-mapped display screen, as in X
  * sound device

Two questions arise.  In Emacs, in a bit-mapped display screen, as in
X, should a future Emacs support a high resolution,
word-processor-style display

 1. by breaking current, basic Emacs design assumptions?  This would
    require a different display engine, as `xdvi' or `mozilla' now
    are, but integrated within Emacs.  Or,

 2. by using current, basic Emacs design assumptions?  This might mean
    using two new modules (which are perhaps seen by a novice user as
    making up one program, evoked as one Emacs command):

      * a module to convert from the deep representation or `encoded
        document file' to an intermediate representation containing,
        as Eli Zaretskii suggested,

            "a combination of characters and text properties, such
            that Lisp programs and C code that look at the buffer
            text could extract the layout information from that
            text alone"

        (This is similar to what `makeinfo --html' now does, with two
        exceptions:
        -- `makeinfo' is not part of Emacs, although two rendering
        programs, `find-file' and `w3' are a part; and, 
        -- `makeinfo' creates a stand-alone output file rather than a
        temporary file used within Emacs.)

      * a module to render from the intermediate representation to the
        display, as `find-file' with Enriched mode or `w3' or Info now
        do, but with a more word-processor-like look.

For read-only displays, either solution fits the
single-deep-representation-input/multiple-surface-expression-outputs
model I discussed on this list some time ago.

However, people like to edit the various different surface expressions
for paper, for a graphic display, and for a sound device, as well as
the deep representation that holds them.  So people prefer read-write
capability.

(A long time ago, I proposed a user interface design to handle the
display of the results of a slow, asynchronous conversion program
within Emacs without requiring the Emacs display to halt.  For
example, this design -- showing a moving `wave of change' in a scroll
bar -- would enable an Emacs user both to view two parts of a long
file, one near its beginning and the other near its end, and to edit
them even after starting a `tex' update near the beginning and not
have its changes progress to the end for many seconds.)

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> Cc: alkibiades@gmx.de,  boris@gnu.org,  alex@emacswiki.org,
> 	  emacs-devel@gnu.org
> From: David Kastrup <dak@gnu.org>
> Date: Sat, 18 Sep 2004 15:46:27 +0200
> 
> The entities I want to manipulate with Lisp programs are typically
> the contents, not the appearance of the text.

They are almost the same in Emacs today.  Perhaps that is why it's
hard to see how the difference matters.

However, it's enough to think about what <down-arrow> should do to see
that difference.  Do you really mean to say that a word processor
should move the cursor to the next _physical_ line (ignoring
line-wrap), not the next screen line?

Re: enriched-mode and switching major modes.

[Oliver Scholz]

[-- Attachment #1: Type: text/plain, Size: 4939 bytes --]

Richard Stallman <rms@gnu.org> writes:

[...]
>        Instead the user says: "This piece of text is a paragraph, because
>        I hit RET when I finished writing it.  I want it to be a paragraph
>        of the type "Standard Text".  I want "Standard Text" paragraphs to
>        be indented on the left by 1 cm and to have a font size of 10 pt.
>        Exept here, I want this paragraph to have an indentation of 2 cm."
>
> Now you're talking about the user interface concepts.  That too is
> distinct from the question of representing text inside Emacs.  The
> Emacs text representation needs to include a representation of the
> specification of a type of paragraph, in order to implement this user
> interface.

Exactly!  However, it is important to discuss implementation issues
together with the user interface concepts in order to account for the
semantics of the representation.

> But it also needs to have spaces in the buffer where there is
> indentation, so as to be compatible with how the whole Emacs Lisp
> world understands indentation.

This is where I disagree.  The box model I was talking about is an
abstact model, it can be implemented by different means.  For
instance, its left margin can indeed be implemented by spaces.  I have
been working with nothing but that so far.  So, to a certain extent, I
agree with you here.  But a few notes:

-  The semantics of space characters used to implement a box model and
   space chars used for indentation in a non-WP Emacs buffer (like in
   this message buffer in this paragraph) are different.  The former
   don't belong to the abstract document, they are only a means of
   rendering it visually.  The latter are both part of the abstract
   document (the text/plain e-mail) and of its visual representation,
   because for text/plain there is no meaningful difference between
   both.

   Because of this semantic difference, existing Emacs Lisp functions
   would not do the right thing when working on them.  I wouldn't
   expect them either to do the right thing.  `fill-paragraph' does
   not work well in a dired buffer.  `query-replace' does not work in
   a Gnus *group* buffer.


-  Even when working with space characters for indentation, it would
   probably not be as you seem to expect.  When working on a
   graphical user interface, we would have to deal with proportional
   fonts of varying sizes.  This is what people expect from a word
   processor.

   For this reason, all implementation techniques I have examined so
   far work by putting a single space character with a display
   property specifying the indentation column in canonical character
   units.  Like this:

   (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
          (insert (propertize " " 'display '(space :align-to 20)))
          (insert (propertize "lirum larum" 'face
                              '(face :height 140 :inherit variable-pitch))))

   Lisp functions, not intentionally written to deal with paragraphs
   in a WP buffer, would not expect this.  The reason being that
   proportional fonts are up to today are hardly used in Emacs.  (I
   might add, that it is also rather a bit tedious to use them
   without a box model supported by the display engine ...)


-  Implementing a box model by means of space and newline characters
   works for some common cases, but it doesn't scale well to the full
   capabilities of all abstract documents that can be expressed by
   various formats.  I already mentioned tables as a point where it
   does not scale at all.  But if we want to implement XML/HTML + CSS
   (and I definitely want), then some nested boxes with borders and
   background colours are not possible to display, but it would be an
   understatement that implementing a way to let the user interact
   with them (for example by typing text) would be "difficult".  It is
   not that I have not tried ...

   I don't worry about that much yet!  I am content with implementing
   a box model by means of space and newline characters for now.  The
   Emacs word processor can do useful work without it.  If I am not
   able to hack the display engine, then that's life.  The Emacs word
   processor can do useful work without it.  That's why I talked about
   "in the long run", meaning: years.  But it won't be complete!


I am not religious about the particular implementation of a box model!
All I do care about are the design principles of WP functionality.  I
must say that I have felt very frustrated in the last days, because it
seemed to me that we have a fundamental disagreement about concepts
and design principles.  Right now I am not sure.  Maybe what you
called a "hybrid solution" would be perfectly in order, depending on
the specific features that you have in mind.

Maybe we should discuss this on a concrete implementation.  I have
assembled a quick example from (now abandoned) code that I have
written almost a year ago:


[-- Attachment #2: wp-example.el --]
[-- Type: application/emacs-lisp, Size: 26871 bytes --]

[-- Attachment #3: Type: text/plain, Size: 6474 bytes --]


Load this file and then type `M-x wp-example RET' you will then see a
buffer with some example text. You can type text; you can change some
of its properties with `M-x wp-set-paragraph-property'. `M-x
wp-example-encode RET' will encode the contents of that WP buffer and
show the encoded document in a temp-buffer.

Please bear with me.  The code has probably a lot of bugs; and the
`wp-.*' functions are just quickly hacked together right now.  I have
abandoned this approach, because the data structure is modelled for
RTF alone, which quickly seemed wrong to me.  I have been thinking
about a more general design since then.

But maybe it provides an example for what I have in mind. You have to
hit M-q to fill a paragraph -- I have never come to implement
something for auto-fill or even refill. M-q calls
`epos-fill-paragraph' which in turn calls
`epos-fill-region-as-paragraph'. This latter function is the heart of
the whitespace formatting. The important data-structure are two
defstructs stored in the text properties `epos-character' and
`epos-paragraph'.  Characters which are inserted by
`epos-fill-region-as-paragraph' are marked with a text property
`epos-transient', those characters are removed whenever appropriate.

You shouldn't hit RET, btw.  RET would have to be bound to a function
which would create a new paragraph.  I also never implemented this.
Right now RET would just insert a rogue newline, which will be
removed after M-q.

[It seems that this thread is becoming infamous as the "thread with
the 30 kb mails".  I am to blame for that. My apologies. I have been
thinking about word processing in Emacs for years, reading
specifications, writing prototypes and experiments, acquiring coding
skills.  Right now I'd like to find out whether I should abandon that
project altogether now, because I fear that it won't be welcome
because of its fundamental design principles and that it will never be
complete.]

>          b) For a given document, two different applications or the
>             same application on two different machines/operating
>             systems might render two different visual representations.
>
> This is not specifically a problem, and may even give us extra
> flexibility.

Yes.  In fact, I regard it as one of the strengths of word processing.
This is where the difference between word processing and "desktop
publishing" (DTP) becomes most manifest.  Emacs could really excel
here by carrying that principle further to media that are not catered
for yet by traditional word processors: character consoles.

[I didn't meant everything I said to be problematical.  I just wanted
to make the concepts clear.  The term "word processing" is used
ambigously in the wild and part of my work in the past has been to
draw clear conceptual distinctions.]

>     This is very important: If a user enters space characters into an
>     Emacs buffers, she wants there to be space characters.  Those
>     characters would have to become part of the character data in the
>     encoded file.  But if a user just specifies: I want this paragraph to
>     be indented, then the space characters used to display the left
>     margin _must_not_ become part of the encoded file.
>
> Why do you think so?  It seems to me that these two different user actions
> should both produce spaces in the buffer--in one case, inserted manually,
> in the other, caused by the format specification.
>
> We can recompute line breaks automatically, and represent them by
> newline characters in the buffer, which will be removed and replaced
> by the next recomputation.  In this mode, the user would not manually
> enter newline characters except to create breaks (hard newlines).

How do you want to solve the problem of distinguishing manually
inserted spaces from spaces added programatically for visual
rendering when the document is encoded and written to a file?  I
explained this problem in my last mail with an example RTF.  If the
user may create indentation both by inserting space characters and by
adding a text property via a command, then we have an ambiguity in
the user interface, which makes it hazardous.

About newlines: I have thought about using hard newlines for
paragraph separation.  The problem is that I need a place to store
properties belonging to the whole paragraph.  I have thought about
(and experimented with) putting those properties on the final hard
newline.  But I found it inelegant and there is also some hair in
there.  The example code identifies as paragraphs every sequence of
characters with an `eq' `epos-paragraph' property; the fill function
then takes care of the whitespace.  I think that this works better.

>     Erm, what does the concept of "what's really there" in that context
>     mean?  In the buffer, or more generally spoken: in the data structure
>     a containing block box, or a text property storing formatting
>     information is, of course, no less there than any space or newline
>     character added for whitespace formatting.
>
> Something is "really there" if other Lisp programs will see it in the
> way that they are written to look for it.

That's what I would regard as "being there" also.  But I don't see
why this would exclude a box model supported by the display engine
(i.e., as you stated more precisely: a box that is rendered at
display time).  No matter how it happens to be implemented, there
would have to be means to check for a box from Lisp and to examine
its properties.

[This goes actually well beyond the issue at hand: But I'd like to
note: we have an open issue here: we have no way to determine the
screen column of a buffer position (in canonical character units) from
Lisp.  This function is a requirement for implementing filling that
DTRT with proportional fonts or with fonts that have a different width
than the default font.  I am by no means competent here, so I may be
horribly wrong: but when I looked into it, it seemed to me that this
would require help from the display engine and thus short-circuit the
boundaries between the display engine and Lisp functions that examine
the buffer text.  Again: I may be horribly wrong.  Nevertheless, with
a box model that specifies how to render text at display time, we
would have no need for such a function.]

    Oliver
-- 
Oliver Scholz               Jour du Travail de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: enriched-mode and switching major modes.

[Oliver Scholz]

"Eli Zaretskii" <eliz@gnu.org> writes:

[...]
> It is possible that what I wrote was misunderstood to mean that
> there's nothing in the buffer but plain text.

O.k. my mistake then.  I misunderstood you as saying that we should not
build more complex data structures than buffer-text and faces and at
the very most a `left-margin' text property.

[...]
> FWIW, I think there _is_ a way to add sophisticated word-processing
> capabilities to Emacs without breaking the basic Emacs design
> assumptions.  One good idea was already suggested in this thread: when
> Emacs decodes a file which has embedded information about the text
> layout, it should convert that information into a combination of
> characters and text properties, such that Lisp programs and C code
> that look at the buffer text could extract the layout information from
> that text alone.  In other words, convert indentation to the
> appropriate amount of whitespace, add newlines where line-wraps are
> required (and put some text properties on those added characters to be
> able to treat them specially, e.g. at save-file time), add text
> properties for paragraph-level style information, etc.
>
> If you think that this approach will not work, can you tell why?

It works to some extend, as long as you are able to identify those
newlines and space characters when the document is encoded again and
written to a file.  I mentioned this in one of my other longish mails
(sorry for this) as a possible approach for the time that the display
engine does not provide support for block boxes.

This has some limitations.  David already mentioned one problem.  You
are right, of course, that `next-line' & Co. would need to be changed
to DTRT on screen lines rather than what you called physical lines.
But I really think that this is the least of all problems.  Kai
mentioned that cua.el provides such functionality; I counted three
other packages that implement this.

And then there is also the point that this approach fails for the more
complicated things: tables with different font heights, paragraphs
with background colours and borders ...

    Oliver
-- 
Oliver Scholz               Jour du Travail de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Oliver Scholz <alkibiades@gmx.de> writes:

[wp-example.el]
> Load this file and then type `M-x wp-example RET' you will then see a
> buffer with some example text.

I should note that it would be good if you did this on a GUI.  I have
never come to make that code DTRT for ttys.

    Oliver
-- 
Oliver Scholz               Jour du Travail de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Oliver Scholz]

"Robert J. Chassell" <bob@rattlesnake.com> writes:

[...]
> (If the latter, you should be able to look at the same HTML encoded
> buffer in several different modes in different windows:  in W3 mode,
> in HTML Helper mode, and in whatever minor mode `find-file-literally'
> uses.)

I think there is room for some niftiness here.  When working with a
document in what you call the surface expression, then it would be
nice, if a command toggle would show what you call the deep
representation of a paragraph (a block box) or a region.  Then you
could make changes to that and issue the command again to get the
updated surface representation.

This would extent the editing model of `preview-latex' to file formats
like HTML.  With the difference that the "surface expression" is not
just a preview, but also editable, just by different means.

This is just an idea that has sprung to my mind one day.  I have not
pursued this thought.  There might be problems, I don't know.

    Oliver
-- 
Oliver Scholz               Jour du Travail de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Oliver Scholz <alkibiades@gmx.de>
> Date: Sat, 18 Sep 2004 19:08:17 +0200
> Cc: boris@gnu.org, alex@emacswiki.org, emacs-devel@gnu.org
> 
> It works to some extend, as long as you are able to identify those
> newlines and space characters when the document is encoded again and
> written to a file.

If those characters have a special text property (put there when the
file is decoded), then such an identification at save time is
possible.

> This has some limitations.  David already mentioned one problem.  You
> are right, of course, that `next-line' & Co. would need to be changed
> to DTRT on screen lines rather than what you called physical lines.
> But I really think that this is the least of all problems.  Kai
> mentioned that cua.el provides such functionality; I counted three
> other packages that implement this.

That is where I think we disagree: I think that the amount of Lisp
code that relies on being able to deduce lots of layout-related issues
by merely looking at the buffer is HUGE.  You really do NOT want to
modify every Emacs command like CUA Mode did.

> And then there is also the point that this approach fails for the more
> complicated things: tables with different font heights, paragraphs
> with background colours and borders ...

I suggest to discuss that and try to identify the specific problems
that you think will cause such an approach to fail.  Then we might
have a better idea of the limitations of this approach, and could talk
about solutions.

In other words, the current design might indeed have to be changed,
but I think you will agree that such changes need to be kept to a
minimum, lest we end up redesigning Emacs.  Talking about specific
problems will help to come up with such a minimal set of changes.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    I have various wild ideas how to improve emacs towards a block structure
    which is don't require a lot of changes to emacs commands.

I proposed one yesterday: you just need to adjust the characters to
reflect the specifications of the blocks.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

[-- Attachment #1: Type: text/plain, Size: 1910 bytes --]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> From: Oliver Scholz <alkibiades@gmx.de>
>> Date: Sat, 18 Sep 2004 19:08:17 +0200
>> Cc: boris@gnu.org, alex@emacswiki.org, emacs-devel@gnu.org
>> 
>> It works to some extend, as long as you are able to identify those
>> newlines and space characters when the document is encoded again and
>> written to a file.
>
> If those characters have a special text property (put there when the
> file is decoded), then such an identification at save time is
> possible.

Yes, that's the technique I have been using so far.  It seems to me
that Richard would disagree here.

[...]
> I suggest to discuss that and try to identify the specific problems
> that you think will cause such an approach to fail.  Then we might
> have a better idea of the limitations of this approach, and could talk
> about solutions.

Easy.  Just consider the HTML document attachted below.  RTF has
similar features, but I chose HTML, because that is more widely
known.  To see where the problem is, you need a CSS capable browser
on a GUI.

However, I don't think we need to talk about specific solutions /now/,
for this reason:

> In other words, the current design might indeed have to be changed,
> but I think you will agree that such changes need to be kept to a
> minimum, lest we end up redesigning Emacs.  Talking about specific
> problems will help to come up with such a minimal set of changes.

Yes, I agree.  If all this is basically a question of whether a box
model (in the display engine) is too much work and trouble, then in
fact, I suggest to delay such decisions until basic functionality is
implemented by means of space and newline characters.  Then we'll see
whether what's left is worth it.

I am thinking about an architecture (for word processing) that should
be modular enough to replace the spaces-and-newline box model with
something real without trouble.

    Oliver



[-- Attachment #2: example.html --]
[-- Type: text/html, Size: 2914 bytes --]

[-- Attachment #3: Type: text/plain, Size: 173 bytes --]


-- 
Oliver Scholz               Jour du Travail de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Oliver Scholz <alkibiades@gmx.de>
> Date: Sat, 18 Sep 2004 22:02:04 +0200
> Cc: boris@gnu.org, emacs-devel@gnu.org, alex@emacswiki.org
> 
> > I suggest to discuss that and try to identify the specific problems
> > that you think will cause such an approach to fail.  Then we might
> > have a better idea of the limitations of this approach, and could talk
> > about solutions.
> 
> Easy.  Just consider the HTML document attachted below.

I looked at it, but I can't say I understand which specific problems
you refer to.  Assuming that the Emacs buffer has the text layed out
as you want it on the screen, plus whatever text properties that are
necessary to preserve the style information, what specifically would
cause Emacs to fail to display this as you want?

> I am thinking about an architecture (for word processing) that should
> be modular enough to replace the spaces-and-newline box model with
> something real without trouble.

Why should we even consider such replacement if we are not yet sure
that the spaces-and-newline model will fail to do the job?

Re: enriched-mode and switching major modes.

[Oliver Scholz]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> From: Oliver Scholz <alkibiades@gmx.de>
>> Date: Sat, 18 Sep 2004 22:02:04 +0200
>> Cc: boris@gnu.org, emacs-devel@gnu.org, alex@emacswiki.org
>> 
>> > I suggest to discuss that and try to identify the specific problems
>> > that you think will cause such an approach to fail.  Then we might
>> > have a better idea of the limitations of this approach, and could talk
>> > about solutions.
>> 
>> Easy.  Just consider the HTML document attachted below.
>
> I looked at it, but I can't say I understand which specific problems
> you refer to.  Assuming that the Emacs buffer has the text layed out
> as you want it on the screen, plus whatever text properties that are
> necessary to preserve the style information, what specifically would
> cause Emacs to fail to display this as you want?

The nested blocks example:

The critical point here are the borders. As the comment says it is not
strictly speaking impossible: If I have a function to determine the
height of a face in pixels, I /could/ determine the height of a line,
programatically generate an PPM picture of the height of that line in
pixels, which is corresponds to a fraction of a vertical border line.
Doing this for each line I /could/ render the border. That's o.k. so
far. Then I have to update this whenever the user types something in
that paragraph. This seems to my like a reliable source for headaches.
But if you ask for impossibility, then you have a point here.


The table example:

The lines are not continous.  The font sizes differ widely, but the
relative line-spacing doesn't.

Maybe I can demonstrate it with another piece of code. If you evaluate
the code at this URL (I should have thought of posting links earlier):

<URL: http://home.arcor.de/utis/table-example.el>

you'll see a buffer with three table cells in a row. The problem here:
all table cells have the same line-spacing in pixels; they should have
the same /relative/ line spacing.  The middle cell, with the
smallest font size, should occupy much less vertical space than the
right column containing the largest font.


    Oliver
-- 
Oliver Scholz               Jour du Travail de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Kim F. Storm]

"Eli Zaretskii" <eliz@gnu.org> writes:

> That is where I think we disagree: I think that the amount of Lisp
> code that relies on being able to deduce lots of layout-related issues
> by merely looking at the buffer is HUGE.  You really do NOT want to
> modify every Emacs command like CUA Mode did.

I think you confuse CUA with s-region.el and pc-select.el which does
modify a lot of commands, or make strange key bindings to achieve the 
"use shift to start/extend the region" behaviour.

CUA mode does NOT modify any "ordinary" commands.

Instead, CUA has a list of commands which it should recognize as "cursor
movement", so it can know when a shifted cursor movement key should
activate/extend the region.

>
>> And then there is also the point that this approach fails for the more
>> complicated things: tables with different font heights, paragraphs
>> with background colours and borders ...
>
> I suggest to discuss that and try to identify the specific problems
> that you think will cause such an approach to fail.  Then we might
> have a better idea of the limitations of this approach, and could talk
> about solutions.
>
> In other words, the current design might indeed have to be changed,
> but I think you will agree that such changes need to be kept to a
> minimum, lest we end up redesigning Emacs.  Talking about specific
> problems will help to come up with such a minimal set of changes.


I can see one such example:

Consider

 A
 A

to mean ONE double height A, while

 b
 c

are two normal height characters.

With the current emacs display engine, there is no way
to display something simple like

 AAAbbbbbbb
 AAAccccccc

The best emacs can do is:

 AAAbbbbbbb
 AAA
    ccccccc

as it has a primitive "no overlapping line" display model.

Even worse if you have two 1.5 height characters on two lines (top half
of X belongs to the A, bottom half of X belongs to H)

 A   < line 1
 X
 H   < line 2
   
You cannot display a mix of 2 and 3 lines like this:

 AAAbbbbbbb
 XXXccccccc
 HHHddddddd

Again, the best emacs can do is something like

 AAAbbbbbbb
 VVV
 ^^^ccccccc
 HHH
    ddddddd

Still, in all of these examples, it is quite easy to see what <down>
arrow should do, and not too difficult to see what <right> should do
either.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Richard Stallman]

    If I do search&replace on text that is in a run-in paragraph, I don't
    want to cater for CR and multiple spaces that are just added for the
    sake of appearance.

This is a good point.  However, it wouldn't be hard to make search
overcome this.  So we have to weigh the number of commands for which
we want to disregard the soft newlines, against the number of commands
for which would look at the newlines.  I think the former will be few
and the latter many, but if someone can demonstrate it is the other way,
you could change my mind.

Re: enriched-mode and switching major modes.

[Robert J. Chassell]

   ... When working with a document in what you call the surface
   expression, then it would be nice, if a command toggle would show
   what you call the deep representation of a paragraph (a block box)
   or a region.  Then you could make changes to that and issue the
   command again to get the updated surface representation.

If I understand you rightly, the idea here is that you look at one of
the surface representations is a read-only display and maintain the
same position in the read-write deep representation buffer.  You can
either keep the two buffers visible at the same time or switch from
one to the other.

This sounds good.  Moreover, it is less complex than writing a program
to edit a sufficiently complex surface expression.

Generally, I would expect you to look at at least two different
surface expressions at the same time, such as Info and DVI.  This way
you could avoid accidentally focusing too much on just one output
format.  With the deep representation, this means three visible
windows.  You might set things up so that by default a frame is laid
out with three windows above each other.

For example:

         +-----------------------------+
         |                             |
         |   Info surface expression   |        (Read only)
         |                             |
         +-----------------------------+
         |                             |
         |    DVI surface expression   |        (Read only)
         |                             |
         +-----------------------------+
         |                             |
         | Texinfo deep representation |        (Read-write)
         |                             |
         +-----------------------------+

--
    Robert J. Chassell
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> Cc: Oliver Scholz <alkibiades@gmx.de>, boris@gnu.org,
> 	emacs-devel@gnu.org, alex@emacswiki.org
> From: storm@cua.dk (Kim F. Storm)
> Date: Sun, 19 Sep 2004 00:11:42 +0200
> 
> With the current emacs display engine, there is no way
> to display something simple like
> 
>  AAAbbbbbbb
>  AAAccccccc

Yes, I know about the limtations of the display engine, but those
limitations will need to be lifted anyhow, for Emacs to be able to
display examples that Oliver talked about.

What I asked was about the idea of rearranging the text in the buffer
as dictated by the style information in the file, not about current
limitations of the display engine.  I suggested to discuss specific
problems in using that idea to implement word-processing features.
Obviously, display of arbitrary mixtures of variable-size characters
doesn't belong to the class of problems that this idea could solve.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

"Robert J. Chassell" <bob@rattlesnake.com> writes:

>    ... When working with a document in what you call the surface
>    expression, then it would be nice, if a command toggle would show
>    what you call the deep representation of a paragraph (a block box)
>    or a region.  Then you could make changes to that and issue the
>    command again to get the updated surface representation.
>
> If I understand you rightly, the idea here is that you look at one of
> the surface representations is a read-only display and maintain the
> same position in the read-write deep representation buffer.  You can
> either keep the two buffers visible at the same time or switch from
> one to the other.

Well, no. The idea is rather that you "fold" and "unfold" parts of a
buffer. Say, you edit the surface expression of a HTML document; then
you "unfold" the surface expression of a table and edit the HMTL code
(the deep representation) of that table directly. The rest of the
document stays in its surface expression. If you "fold" the table
again, you come to see the updated surface expression of the table.

`preview-latex' works similar (to a certain extent).  You edit the
LaTeX source (the deep representation), but `preview-latex' will
display certain (configurable) parts of it, such as tables, formulas
etc. in their surface expression /instead/ of the deep representation
/in the same buffer/.  If you want to edit the formula, you issue a
command to get the deep representation again.

My idea is to have /both/ the deep representation and the surface
expression read-write, as you say.  Again: I have not worked this out.
There may be pitfalls; I will spend thought on this only when
everything else is ready.  There are some fundamental differences to
editing LaTeX with preview-latex.  But it is too early too worry
about it.

There is btw. an interesting difference in our terminology. What you
call "the deep representation" is indeed what I call the "encoded
document file" and your "surface expression" is my "visual (aural)
appearance".  But it seems to me that the emphasis expressed in the
choice of words is different; not better or worse, just different.  In
the editing model that I pursue various deep representations (as well
as various surface expression) are just instances of an abstract
document.  In your editing model we have a particular deep
representation and a surface expression for that; a read-write surface
expression would just be another means to edit the deep
representation: Editing the deep representation is what one really
wants in this editing model.  Contrary to that, in the editing model I
prefer the deep representation is just a means to freeze the abstract
document on disc or to transfer it to another person[1].  I think that
some people would prefer your view, and so it seems a good idea to me
to spend some thought---when the time is ripe---on how both could be
made compatible, so that people can get the best of both worlds at
their choice.  This is another area where Emacs could excel compared
to traditional word processors.

> This sounds good.  Moreover, it is less complex than writing a program
> to edit a sufficiently complex surface expression.

Hey!  I am the guy who desperately wants to edit really complex
surface expressions. :-)

> Generally, I would expect you to look at at least two different
> surface expressions at the same time, such as Info and DVI.  This way
> you could avoid accidentally focusing too much on just one output
> format.  With the deep representation, this means three visible
> windows.  You might set things up so that by default a frame is laid
> out with three windows above each other.
[...]

This is quite different.  David Kastrup has written a paper discussing
the merits of various preview models, which could be of interest here.
I don't recall the URL, though.

    Oliver


Footnotes: 

[1] There is an interesting analogy to what Emacs provides at the
character level.  I found out that my terminology corresponds to the
structure of Emacs' handling of character encoding.  But there is not
enough space left on this e-mail to prove that statement.

-- 
Oliver Scholz               Jour de la Raison de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[David Kastrup]

Oliver Scholz <epameinondas@gmx.de> writes:

[...]

>> Generally, I would expect you to look at at least two different
>> surface expressions at the same time, such as Info and DVI.  This way
>> you could avoid accidentally focusing too much on just one output
>> format.  With the deep representation, this means three visible
>> windows.  You might set things up so that by default a frame is laid
>> out with three windows above each other.
> [...]
>
> This is quite different.  David Kastrup has written a paper discussing
> the merits of various preview models, which could be of interest here.
> I don't recall the URL, though.

It's focused about WYSIWYG and LaTeX, and the paper not only focuses
on the paradigms itself, but also on the quality of existing (free)
implementations.  So it should probably be considered as relevant in
this discussion only with an appropriately sized grain of salt.

Try <URL:http://preview-latex.sourceforge.net/wysiwyg-draft.pdf>.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: enriched-mode and switching major modes.

[Robert J. Chassell]

"Robert J. Chassell" <bob@rattlesnake.com> writes:

   > If I understand you rightly, the idea here is that you look at one of
   > the surface representations is a read-only display and maintain the
   > same position in the read-write deep representation buffer.  You can
   > either keep the two buffers visible at the same time or switch from
   > one to the other.

Oliver Scholz <epameinondas@gmx.de> writes:

   Well, no. The idea is rather that you "fold" and "unfold" parts of a
   buffer. Say, you edit the surface expression of a HTML document; then
   you "unfold" the surface expression of a table and edit the HMTL code
   (the deep representation) of that table directly. 

This is a matter of preference.  20 years ago, When I started with
Emacs, I remember a colleague saying that she always switched from one
window to the other.  She seldom displayed two windows on the screen
at the same time.  Thus, when she moved a paragraph from one end of a
document to the other, she search for the paragraph, found it, copied
it, then switched to another window that took up her whole screen.
That was her preferred mode.

On the other hand, I preferred then, as I do now, to see different
windows at the same time.  When I copy a paragraph from one spot to
another, I tend to switch between two visible windows.

It is easy enough in Emacs to satisfy either preference.  Either you
set up your frame with two (or more) visible windows or you toggle
among them, showing only one at a time.

A pure toggle format may work for some features.  For example, I think
it makes sense to toggle among the different ways to insert
mathematical expressions into a buffer using Emacs Calc mode.  You
only can insert into only one buffer at a time.  But toggling fails
when you want to look at different renderings of that buffer.

   My idea is to have /both/ the deep representation and the surface
   expression read-write, as you say.

Yes, that is good.  (It is also difficult... )

Also, David Kastrup makes the point that he found the TeXmacs 

    ... `concertina effect' distracting: constant reformatting during
    text entry causes the line spacing to shrink until the line gets
    wrapped differently again.

This concern may vary widely among people or may be intrinsic to any
decent typesetting system.  I don't know.

In 1984, I was not bothered by the `concertina effect' from text that
reformatted on a line by line basis.  (I was using a WYSIWYG editor at
that time.)  Indeed, I rather enjoyed it when I noticed it.  But this
type of formatting is much less good than that done by TeX, although
it is much faster.  

At that time, the machines I was using could reformat a line in real
time, but Tex took seconds to reformat a page.  I have not used
TeXmacs enough to determine whether I am bothered by its more
extensive `concertina effect'.  I invented the `moving wave of change'
interface notion to deal with TeX's instrinsic slowness.  

Even now it takes about 25 milliseconds for TeX to format a page on my
machine.  (I just checked.)  If you were writing a 100 page book, and
reformatting `in real time', you would need `moving wave of change' to
be able to look at a document in more than one window (whether
toggling a single screenful or viewing the parts simultaneously).

In any event, a typeset output is only one of the multiple outputs
that a system can produce.

   There is btw. an interesting difference in our terminology.  ....
   Editing the deep representation is what one really wants in [Bob's]
   editing model.  Contrary to that, in the editing model I prefer the
   deep representation is just a means to freeze the abstract document
   on disc or to transfer it to another person ...

This is *exactly* what I mean by a `deep representation': the means to
freeze the abstract document.  That is what goes on disk or to another
person.

I do not think there is any difference at all.

   > This sounds good.  Moreover, it is less complex than writing a program
   > to edit a sufficiently complex surface expression.

   Hey!  I am the guy who desperately wants to edit really complex
   surface expressions. :-)

Yes, I understand.  I would prefer to edit the typeset output or HTML
output of a document than the format in which its abstract document is
written, and have the changes appear in all three representations.
But that is impossible at this time with Emacs.  I must edit the
`abstract document' (except in enriched mode, which I do not use).
(Mostly, I prepare stuff for printing as a typeset book, for viewing
on the Web, for efficient online reading and listening, and for
copying for plain text formats, like email.)

   > Generally, I would expect you to look at at least two different
   > surface expressions at the same time, such as Info and DVI.  This way
   > you could avoid accidentally focusing too much on just one output
   > format.  With the deep representation, this means three visible
   > windows.  You might set things up so that by default a frame is laid
   > out with three windows above each other.
   [...]

   This is quite different.  

How is it different?  You are not just running a 1970s computer
typesetter?  Or are you?  (I well remember a visit to a newspaper
printing shop in 1970.  In that visit, I saw both the men and their
`hot metal' typsetting machines and the young woman working on the
shop's only, and at that time experimental, computer typsetting
machine.  It was clear to all that the computer typsetting machine was
coming.)

Certainly for Emacs, we don't want to recreate a paper-only universe,
in which the `universal machine' does no more than typeset for
printing.  After all, remote connections were first made in the 1950s
or early 1960s.  (I met a man who said he soldered the first
connection; but I do not remember the date he told me.)  The Internet
was first installed in 1969.  Nowadays, many people through out the
world can transfer documents without needing to put them on paper.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: enriched-mode and switching major modes.

[Kai Grossjohann]

"Eli Zaretskii" <eliz@gnu.org> writes:

> However, it's enough to think about what <down-arrow> should do to see
> that difference.  Do you really mean to say that a word processor
> should move the cursor to the next _physical_ line (ignoring
> line-wrap), not the next screen line?

I think that cursor motion is the only part that needs to be aware of
how the text is displayed, whereas everything else should be happy
thinking that a paragraph is a single line of text (and thus would
miss the newlines inserted for display and would also miss the space
used for indentation, and would also miss the numbers used for an
enumeration, and so on).

But I can't prove it.

Kai

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Richard Stallman wrote:

   This change seems good to me.  Please install it.

   Could you also add after-change-major-mode-hook to NEWS
   and to lispref/modes.texi?

I installed the change and a NEWS entry.  In as far as the changes to
the Elisp manual (and docstrings) are concerned, what about the
following patches?

===File ~/modes.texi-diff===================================
*** modes.texi	22 Aug 2004 13:05:50 -0500	1.79
--- modes.texi	19 Sep 2004 14:15:07 -0500	
***************
*** 1,6 ****
  @c -*-texinfo-*-
  @c This is part of the GNU Emacs Lisp Reference Manual.
! @c Copyright (C) 1990, 1991, 1992, 1993, 1994, 1995, 1998, 1999, 2003
  @c   Free Software Foundation, Inc.
  @c See the file elisp.texi for copying conditions.
  @setfilename ../info/modes
--- 1,6 ----
  @c -*-texinfo-*-
  @c This is part of the GNU Emacs Lisp Reference Manual.
! @c Copyright (C) 1990, 1991, 1992, 1993, 1994, 1995, 1998, 1999, 2003, 2004
  @c   Free Software Foundation, Inc.
  @c See the file elisp.texi for copying conditions.
  @setfilename ../info/modes
***************
*** 287,292 ****
--- 287,293 ----
  @code{delay-mode-hooks} around its entire body, including the call to
  the parent mode command and the final call to @code{run-mode-hooks}.
  (Using @code{define-derived-mode} does this automatically.)
+ @xref{Derived Modes}, and @ref{Hooks}.
  
  @item
  If something special should be done if the user switches a buffer from
***************
*** 575,581 ****
  in particular.  Other major modes are defined in effect by comparison
  with this one---their definitions say what to change, starting from
  Fundamental mode.  The @code{fundamental-mode} function does @emph{not}
! run any hooks; you're not supposed to customize it.  (If you want Emacs
  to behave differently in Fundamental mode, change the @emph{global}
  state of Emacs.)
  @end deffn
--- 576,582 ----
  in particular.  Other major modes are defined in effect by comparison
  with this one---their definitions say what to change, starting from
  Fundamental mode.  The @code{fundamental-mode} function does @emph{not}
! run any mode hooks; you're not supposed to customize it.  (If you want Emacs
  to behave differently in Fundamental mode, change the @emph{global}
  state of Emacs.)
  @end deffn
***************
*** 2563,2588 ****
  a symbol with a function definition), it is called.  If it is a list
  that isn't a function, its elements are called, consecutively.  All
  the hook functions are called with no arguments.
- 
- For example, here's how @code{emacs-lisp-mode} runs its mode hook:
- 
- @example
- (run-hooks 'emacs-lisp-mode-hook)
- @end example
  @end defun
  
  @defun run-mode-hooks &rest hookvars
! Like @code{run-hooks}, but is affected by the @code{delay-mode-hooks}
! macro.
  @end defun
  
  @defmac delay-mode-hooks body...
! This macro executes the @var{body} forms but defers all calls to
! @code{run-mode-hooks} within them until the end of @var{body}.
! This macro enables a derived mode to arrange not to run
! its parent modes' mode hooks until the end.
  @end defmac
  
  @defun run-hook-with-args hook &rest args
  This function is the way to run an abnormal hook and always call all
  of the hook functions.  It calls each of the hook functions one by
--- 2564,2600 ----
  a symbol with a function definition), it is called.  If it is a list
  that isn't a function, its elements are called, consecutively.  All
  the hook functions are called with no arguments.
  @end defun
  
  @defun run-mode-hooks &rest hookvars
! Major modes should run their mode hook using this function.  It is
! similar to @code{run-hooks}, but if the variable
! @code{delay-mode-hooks} is non-@code{nil}, this function delays
! running @var{hookvars}, as well as any previously delayed hooks, until
! a later call.  If that variable is @code{nil}, @code{run-mode-hooks}
! runs @code{after-change-major-mode-hook} (see below) before running
! any delayed hooks in order and finally @var{hookvars}.
  @end defun
  
+ @defvar delay-mode-hooks
+ If the value of this variable is non-@code{nil}, @code{run-mode-hooks}
+ delays running any hooks until a later call..
+ @end defvar
+ 
  @defmac delay-mode-hooks body...
! This macro executes the @var{body} forms with the variable
! @code{delay-mode-hooks} buffer locally bound to @code{t} in the
! current buffer.  This macro enables a derived mode to arrange not to
! run its parent modes' mode hooks until the end.
  @end defmac
  
+ @defvar after-change-major-mode-hook
+ @code{run-mode-hooks} runs this mode independent normal hook before
+ the mode hooks.  This hook is also called at the end of
+ @code{fundamental-mode} (which is the only major mode function that
+ does not run @code{run-mode-hooks} at its end).
+ @end defvar
+ 
  @defun run-hook-with-args hook &rest args
  This function is the way to run an abnormal hook and always call all
  of the hook functions.  It calls each of the hook functions one by
============================================================

===File ~/hooks.texi-diff===================================
*** hooks.texi	06 Aug 2004 20:46:37 -0500	1.17
--- hooks.texi	19 Sep 2004 12:58:27 -0500	
***************
*** 35,40 ****
--- 35,41 ----
  @table @code
  @item activate-mark-hook
  @item after-change-functions
+ @item after-change-major-mode-hook
  @item after-init-hook
  @item after-insert-file-functions
  @item after-make-frame-functions
============================================================

===File ~/subr.el-diff======================================
*** subr.el	19 Sep 2004 09:52:28 -0500	1.410
--- subr.el	19 Sep 2004 13:43:41 -0500	
***************
*** 1944,1949 ****
--- 1944,1951 ----
  (defun run-mode-hooks (&rest hooks)
    "Run mode hooks `delayed-mode-hooks' and HOOKS, or delay HOOKS.
  Execution is delayed if `delay-mode-hooks' is non-nil.
+ If `delay-mode-hooks' is nil, run `after-change-major-mode-hook'
+ before running the mode hooks.
  Major mode functions should use this."
    (if delay-mode-hooks
        ;; Delaying case.
============================================================

Re: enriched-mode and switching major modes.

[Richard Stallman]

    > If those characters have a special text property (put there when the
    > file is decoded), then such an identification at save time is
    > possible.

    Yes, that's the technique I have been using so far.  It seems to me
    that Richard would disagree here.

I suspect that is a misunderstanding, because what Eli is talking
about sounds like what I'm in favor of.  Automatically-inserted
newlines should be "soft", while newlines inserted by the user should
be "hard".  On saving, the soft newlines can be discarded, since
line-breaking will regenerate them anyway.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    With the current emacs display engine, there is no way
    to display something simple like

     AAAbbbbbbb
     AAAccccccc

    The best emacs can do is:

     AAAbbbbbbb
     AAA
	ccccccc

    as it has a primitive "no overlapping line" display model.

In a word processor, what constructs would give you something like
that?  I have no idea what HTML constructs could produce such a
layout; if there is a way, it must go beyond the simple HTML markup
that I know.

It seems to me that Emacs would be an adequate word processor for
editing things like manuals, and web pages that are mainly substance,
even if it could not do anything like this.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    Exactly!  However, it is important to discuss implementation issues
    together with the user interface concepts in order to account for the
    semantics of the representation.

I agree we need to discuss them both; the point is, they are not the
same question.  They are related questions.

    -  The semantics of space characters used to implement a box model and
       space chars used for indentation in a non-WP Emacs buffer (like in
       this message buffer in this paragraph) are different.  The former
       don't belong to the abstract document, they are only a means of
       rendering it visually.

I agree they would be a means of rendering the box, but not just
a way of rendering it visually.  They would be a way of rendering
it for the Lisp world.

Most programming languages have a textual representation and a
behavior.  Lisp is different: everything has a textual representation,
a Lisp object representation, and a behavior.  When designing a Lisp
construct, or any Lisp data, one must think about all three levels.

Because of Lisp, a similar thing happens for text in an Emacs buffer.
In a typical word processor, text has a visual appearance and a way it
is saved in a file, that's all.  In Emacs, text has a visual
appearance, a way it is saved in a file, and the way it appears as
Lisp data.  We have to design all three.

The visual rendering of paragraph indentation on the screen could be
the same, regardless of whether the buffer contains space characters.
And the file representation can also be the same (no space characters
for indentation there).  Despite that, it makes a difference what the
buffer contains.

       Because of this semantic difference, existing Emacs Lisp functions
       would not do the right thing when working on them.  I wouldn't
       expect them either to do the right thing.  `fill-paragraph' does
       not work well in a dired buffer.  `query-replace' does not work in
       a Gnus *group* buffer.

fill-paragraph would do exactly the right thing with these spaces.  Of
course, we would tell it to treat them as the fill prefix in such a
paragraph.  This feature *already* exists and works this way.

This implementation makes all kinds of cursor motion commands, and
commands that only look at the buffer contents, do the right thing.

    -  Even when working with space characters for indentation, it would
       probably not be as you seem to expect.  When working on a
       graphical user interface, we would have to deal with proportional
       fonts of varying sizes.

Yes, this will make matters harder.  We will need to use some
spaces that have only partial width, or something like that.

       (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
	      (insert (propertize " " 'display '(space :align-to 20)))
	      (insert (propertize "lirum larum" 'face
				  '(face :height 140 :inherit variable-pitch))))

Why not this?

       (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
       	      (insert-char ?\s 3)
	      (insert (propertize " " 'display '(space :align-to 20)))
	      (insert (propertize "lirum larum" 'face
				  '(face :height 140 :inherit variable-pitch))))

supposing that aligning to pixel 20 means 3 and a fraction spaces?

However, even having one very wide space to represent the indentation
is better than having none at all.  That one space will at least
show Lisp code that some indentation is present.

      But if we want to implement XML/HTML + CSS
       (and I definitely want), then some nested boxes with borders and
       background colours are not possible to display, but it would be an
       understatement that implementing a way to let the user interact
       with them (for example by typing text) would be "difficult".

I won't argue with this statement, but I don't know what cases you
mean.

    How do you want to solve the problem of distinguishing manually
    inserted spaces from spaces added programatically for visual
    rendering when the document is encoded and written to a file?

With text properties.  That is how we distinguish hard and soft
newlines now.  Hard newlines represent user-specified breaks, and are
saved as such; soft newlines are meant to be recomputed.
We could have hard and soft spaces too.

    About newlines: I have thought about using hard newlines for
    paragraph separation.  The problem is that I need a place to store
    properties belonging to the whole paragraph.

Those should be stored on all the text of the paragraph, right?

What are the jobs you want to do with these properties?
What is the semantics?  (One example may be enough.)

Re: enriched-mode and switching major modes.

[Richard Stallman]

    The nested blocks example:

    The critical point here are the borders.

I don't recall seeing this nested blocks example.
Was it buried in the epos code you sent?  I am too overloaded
to read all that code; if you could describe the example
in a more readable way, I could understand I guess.

Re: enriched-mode and switching major modes.

[Stefan]

[ Note; I'm pretty happy with Emacs's current word-processing facilities
  right now and don't see any need for it to develop into yet-another
  WYSIWYG thingy, so I'm generally not too intereted in this thread,
  especially since I don't see anyone actually working on an implementation
  of what you're talking about. ]

I just think that WhyzzyTeX is also an interesting design point: you get to
edit the deep representation (which, AFAIC, is not "just the disk format"
but is the format where I can expresss my *intents*, i.e. where I can
distinguish between two concepts even if they happen to be rendered
identically on the currently used output mode), while seeing the typeset
output "on the fly".


        Stefan

Re: enriched-mode and switching major modes.

[David Kastrup]

Stefan <monnier@iro.umontreal.ca> writes:

> [ Note; I'm pretty happy with Emacs's current word-processing facilities
>   right now and don't see any need for it to develop into yet-another
>   WYSIWYG thingy, so I'm generally not too intereted in this thread,
>   especially since I don't see anyone actually working on an implementation
>   of what you're talking about. ]
>
> I just think that WhyzzyTeX is also an interesting design point: you
> get to edit the deep representation (which, AFAIC, is not "just the
> disk format" but is the format where I can expresss my *intents*,
> i.e. where I can distinguish between two concepts even if they
> happen to be rendered identically on the currently used output
> mode), while seeing the typeset output "on the fly".

But as its display is outside of the scope of Emacs, it is mostly
academical for this discussion.

More interesting would be speculating about giving "calc" a better
formatted formula display: if you have compositions like in "Big mode"
(d B), then you can still put the cursor on elements and do selections
and so on.  If one were to render the stuff with TeX or similar
instead, how would one want to implement cursor navigation?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: enriched-mode and switching major modes.

[Stefan]

>> I just think that WhyzzyTeX is also an interesting design point: you
[...]
> But as its display is outside of the scope of Emacs, it is mostly
> academical for this discussion.

I'm not talking about the specific implementation, but about the
general design.

> More interesting would be speculating about giving "calc" a better
> formatted formula display: if you have compositions like in "Big mode"
> (d B), then you can still put the cursor on elements and do selections
> and so on.  If one were to render the stuff with TeX or similar
> instead, how would one want to implement cursor navigation?

WhyzzyTeX + ADvi shows you a cursor in the DVI output which tracks the
location of the cursor in the Emacs buffer, and if you click on the DVI
screen, point is moved to the corresponding source location.  It's not
perfect, but that's a detail of the implementation: after all, TeX was not
designed for interactive use.


        Stefan

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Stefan <monnier@iro.umontreal.ca> writes:

> [ Note; I'm pretty happy with Emacs's current word-processing facilities
>   right now and don't see any need for it to develop into yet-another
>   WYSIWYG thingy
[...]

I don't talk about WYSIWYG.  I abhor the very concept of it in the
context of word processing.  "WYSIWYG" means to say something about
the relationship between the rendering of the document on screen and
on printed paper.

We can't have that in Emacs.  It would mean that if Emacs is started
on, say the linux tty, we would have to print the document in light
gray on black in a font which looks like the linux console font.  It
would mean that the printed output would look different when I start
Emacs on a character console, a terminal emulator or on the GUI.

Traditional word processors are conceptually ambigous here.  WYSIWYG
makes sense only for DTP, where you specify the exact position of text
on the paper.

I believe that "WYSIWYG" is a relic of the times when "ordinary
people" did not trust computers and tried to look at them as a special
kind of typewriter.  Nowadays there are other buzzwords to make up for
the initial confusion: "paperless office", "e-document exchange" and
the like.

[...]
> the deep representation (which, AFAIC, is not "just the disk format"
> but is the format where I can expresss my *intents*, i.e. where I
> can distinguish between two concepts even if they happen to be
> rendered identically on the currently used output mode)
[...]

Yes, that's a different editing model.  Just as I said to Robert
Chassel: some people would prefer an editing model, where the "deep
representation" is more than just the encoding of the abstract
document as frozen on disc.

    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     With the current emacs display engine, there is no way
>     to display something simple like
>
>      AAAbbbbbbb
>      AAAccccccc
>
>     The best emacs can do is:
>
>      AAAbbbbbbb
>      AAA
> 	ccccccc
>
>     as it has a primitive "no overlapping line" display model.
>
> In a word processor, what constructs would give you something like
> that?  I have no idea what HTML constructs could produce such a
> layout; if there is a way, it must go beyond the simple HTML markup
> that I know.

If I understand Kim correctly, then simple tables where the first
column contains a caption for the row would be an example where it
would matter.

       __   __
  /\  |__) /     ABC
 /__\ |  \ |     ABC
/    \|__/ \__   ABC


as opposed to:

       __   __
  /\  |__) /     
 /__\ |  \ |     
/    \|__/ \__   ABC
                 ABC
                 ABC


    Oliver
--
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

[...]
>     -  The semantics of space characters used to implement a box model and
>        space chars used for indentation in a non-WP Emacs buffer (like in
>        this message buffer in this paragraph) are different.  The former
>        don't belong to the abstract document, they are only a means of
>        rendering it visually.
>
> I agree they would be a means of rendering the box, but not just
> a way of rendering it visually.  They would be a way of rendering
> it for the Lisp world.
[...]

I don't think that looking at indentation and newlines could at any
rate be enough for something as complex as a WP document.  Where word
wrapping happens to take place is insignificant in WP documents.  It
might happen at very different places when I change the font or send
the document to somebody else or even when I adjust only the size of
the window.  When writing Lisp functions to examine and change the
text, then I have nothing gained here.  On the contrary, I have to
take extra care that my functions will ignore those characters.

> Because of Lisp, a similar thing happens for text in an Emacs buffer.
> In a typical word processor, text has a visual appearance and a way it
> is saved in a file, that's all.  In Emacs, text has a visual
> appearance, a way it is saved in a file, and the way it appears as
> Lisp data.  We have to design all three.

I agree wholeheartedly.  And I have quite some ideas here, which are
not fully worked out, though.  We need something by which Lisp can
access the content of such a document syntactically.  For example: a
Lisp program might have to access all paragraphs of the type
"Headline" that contain the string "foobar" and make changes there. In
short: I intend to make the whole /abstract/ document accessible as a
Lisp object in a structured way.

[...]
> fill-paragraph would do exactly the right thing with these spaces.  Of
> course, we would tell it to treat them as the fill prefix in such a
> paragraph.  This feature *already* exists and works this way.

I honestly can't think of any implementation---save the heuristical
one---that could work with `fill-paragraph' (because whatever data
structure we use, `fill-paragraph' will have to examine properties
that belong to the whole paragraph).

> This implementation makes all kinds of cursor motion commands, and
> commands that only look at the buffer contents, do the right thing.

In the implementation designs that I favour (for reasons we are
currently discussing), only `next-line' and `previous-line' would
continue to do the right thing.

[...]
>        (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
> 	      (insert (propertize " " 'display '(space :align-to 20)))
> 	      (insert (propertize "lirum larum" 'face
> 				  '(face :height 140 :inherit variable-pitch))))
>
> Why not this?
>
>        (progn (switch-to-buffer (generate-new-buffer "*tmp*"))
>        	      (insert-char ?\s 3)
> 	      (insert (propertize " " 'display '(space :align-to 20)))
> 	      (insert (propertize "lirum larum" 'face
> 				  '(face :height 140 :inherit variable-pitch))))

Agreed.


>     How do you want to solve the problem of distinguishing manually
>     inserted spaces from spaces added programatically for visual
>     rendering when the document is encoded and written to a file?
>
> With text properties.  That is how we distinguish hard and soft
> newlines now.  Hard newlines represent user-specified breaks, and are
> saved as such; soft newlines are meant to be recomputed.
> We could have hard and soft spaces too.

Ah, maybe I start to understand what you mean: depending on a minor
mode, `fill-paragraph' would add that text property, when it
determines that a paragraph should be indented.  I agree that it would
work, then, for that particular problem.  I can't say that I like this
UI, but it could work.  I am not sure yet, though, whether it would
scale well---or scale at all---to other paragraph formatting
properties.  But let's discuss that other problem first:

>     About newlines: I have thought about using hard newlines for
>     paragraph separation.  The problem is that I need a place to store
>     properties belonging to the whole paragraph.
>
> Those should be stored on all the text of the paragraph, right?
>
> What are the jobs you want to do with these properties?
> What is the semantics?  (One example may be enough.)

Consider this HTML fragment:

<h1>Some <i>meaningless</i> heading</h1>

The <i> element maps directly to text properties, of course.  But the
h1 element both demands that its contents be rendered as a paragraph
(a block) /and/ specifies certain character formatting properties for
the whole of it, e.g. a large bold font.

When encoding a buffer, I need to identify the whole paragraph as
being of the type "h1".  I.e. I have to distinguish it from:

<p><font size=7><b>Some <i>meaningless</i> heading</font></p>

In RTF it is a bit more complicated.  Each paragraph may specify a
stylesheet that defines both character formatting properties (font,
font size, colour ...) applying to the whole of the paragraph text, as
well as paragraph formatting properties proper such as left/right
indentation, first line indentation, line spacing, space before/after
the paragraph ... In addition to that each paragraph may specify each
single one of these properties directly, thus overriding the defaults
of the stylesheet.  This has to be resolved not only when encoding the
file, but also at, so to say, editing time, when filling or applying
faces.

In XML it will get even more complicated, since we can have nested
paragraphs (nested block boxes), which I can not express with hard
newlines alone.  In fact the whole document here is tree-like; I have
to conserve this in the data structure.

    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Oliver Scholz <alkibiades@gmx.de> writes:

> Consider this HTML fragment:
>
> <h1>Some <i>meaningless</i> heading</h1>
>
> When encoding a buffer, I need to identify the whole paragraph as
> being of the type "h1".  I.e. I have to distinguish it from:

Maybe we can get some of the way with Emacs' category text properties?!

In this case, you could put a 'category 'H1 text property on the heading.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     The nested blocks example:
>
>     The critical point here are the borders.
>
> I don't recall seeing this nested blocks example.
> Was it buried in the epos code you sent?  I am too overloaded
> to read all that code; if you could describe the example
> in a more readable way, I could understand I guess.

It is not in wp-example.el; I have never implemented it, I have only
played with the thought how it could be implemented.  The nested
blocks example was part of the HTML document that I sent in response
to Eli.  Here is the relevant part:

<?xml version="1.0" encoding="us-ascii" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
  <head>
    <title>Lirum larum</title>
    <style type="text/css">
      div.lirum {
                  margin: 5ex 10ex 5ex 10ex;
                  padding: 5ex 10ex 5ex 10ex;
                  background: purple;
                  border-style: solid;
                  border-width: medium;
                  border-color: black;
                }
      div.larum {
                  margin: 3ex 2ex 3ex 2ex;
                  padding: 3ex 2ex 3ex 2ex;
                  background: gray;
                  border-style: dashed;
                  border-color: black;
                  border-width: medium;
                }
    </style>
  </head>
  <body>
    <div class="lirum">
      <div class="larum">
        Some text.
      </div>
      <div class="larum">
        Some other text.
      </div>
    </div>
  </body>
</html>


When rendered by a graphical, CSS2-enabled browser, you'll see two
paragraphs on a gray background sourounded by a dashed border.  Those
two paragraphs are again contained in a larger paragraph on a purple
background surounded by a solid border.


    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Oliver Scholz <epameinondas@gmx.de> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> In a word processor, what constructs would give you something like
>> that?  I have no idea what HTML constructs could produce such a
>> layout; if there is a way, it must go beyond the simple HTML markup
>> that I know.

E.g. tables with different font sizes.  Try this:

<html><body>
<table>
<tr>
<td><font size=7>AAA</font></td>
<td>bbbbbb<br>cccccc</td>
</tr>
</table>
</body></html>


>
> If I understand Kim correctly, then simple tables where the first
> column contains a caption for the row would be an example where it
> would matter.
>
>        __   __
>   /\  |__) /     ABC
>  /__\ |  \ |     ABC
> /    \|__/ \__   ABC
>
>
> as opposed to:
>
>        __   __
>   /\  |__) /
>  /__\ |  \ |
> /    \|__/ \__   ABC
>                  ABC
>                  ABC
>

Exactly.

--
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Kai Grossjohann]

Richard Stallman <rms@gnu.org> writes:

> In a word processor, what constructs would give you something like
> that?  I have no idea what HTML constructs could produce such a
> layout; if there is a way, it must go beyond the simple HTML markup
> that I know.

It could happen in a table, if a row contained a cell with a large
font and a cell with a small font.

It is also often used in old-style books for the first letter of a
chapter.  The first letter would be larger and look fancier than the
others.

If an image could also be a "character", then I can see how
positioning images in the text flow and letting the text flow around
them would produce something like this.

> It seems to me that Emacs would be an adequate word processor for
> editing things like manuals, and web pages that are mainly substance,
> even if it could not do anything like this.

I think especially the table example above is relevant for this.

Kai

Re: enriched-mode and switching major modes.

[Stefan Monnier]

> I don't talk about WYSIWYG.  I abhor the very concept of it in the
> context of word processing.  "WYSIWYG" means to say something about
> the relationship between the rendering of the document on screen and
> on printed paper.

My understanding of WYSIWYG is not really that "the printer version is equal
to the screen version" (after all, TeX gets that: the xdvi rendering is the
same as the printer's).  Instead, the underlying idea is that there is no
"deep representation", or at least that you can at most manipulate
it implicitly.

> We can't have that in Emacs.  It would mean that if Emacs is started
> on, say the linux tty, we would have to print the document in light
> gray on black in a font which looks like the linux console font.  It
> would mean that the printed output would look different when I start
> Emacs on a character console, a terminal emulator or on the GUI.

No: most WYSIWYG word-processors will happily accept (and print) colors in
your document, even when your screen is black&white.  I.e. the way it
typically works is that there *is* a deep representation but it can only be
manipulated implicitly through the superficial representation which is
displayed with "the best possible rendering on the current output device".

>> the deep representation (which, AFAIC, is not "just the disk format"
>> but is the format where I can expresss my *intents*, i.e. where I
>> can distinguish between two concepts even if they happen to be
>> rendered identically on the currently used output mode)
> [...]

> Yes, that's a different editing model.

I guess I still don't understand the model you're thinking of, which doesn't
involve an explicit deep representation and yet isn't WYSIWYG either.


        Stefan

Re: enriched-mode and switching major modes.

[Oliver Scholz]

storm@cua.dk (Kim F. Storm) writes:

> Oliver Scholz <alkibiades@gmx.de> writes:
>
>> Consider this HTML fragment:
>>
>> <h1>Some <i>meaningless</i> heading</h1>
>>
>> When encoding a buffer, I need to identify the whole paragraph as
>> being of the type "h1".  I.e. I have to distinguish it from:
>
> Maybe we can get some of the way with Emacs' category text properties?!
>
> In this case, you could put a 'category 'H1 text property on the heading.
[...]

<blush/> I have to admit that I have never thought of the `category'
property, instead I have always experimented with implementing its
functionality myself.  Thanks for the reminder.  This might indeed be
useful.

We still have have a problem if we mean to distinguish paragraphs by
hard newlines instead of a text property covering the region of
characters that make the content of the paragraph:

(progn (put 'h1 'face 'Info-title-1-face)
       (put 'h2 'face 'Info-title-2-face)
       (switch-to-buffer (generate-new-buffer "*tmp*"))
       (use-hard-newlines 1 'never)
       (insert (propertize "Lirum larum"
                           'category 'h1))
       (newline 1)
       (save-excursion
         (insert (propertize "lirum larum"
                             'category 'h2))))

We have to deal with the case that a user deletes the hard newline (if
you evaluate the code above: just hit backspace).  Is the resulting
paragraph of type 'h1 or of type 'h2?  What about this:

(progn (put 'h1 'face 'Info-title-1-face)
       (switch-to-buffer (generate-new-buffer "*tmp*"))
       (use-hard-newlines 1 'never)
       (insert (propertize "lirum larum" 'category 'h1))
       (insert " lirum larum") ; no `category' property.
       (newline 1))

This might happen as the result of yanking something in.

To use the `category' property for paragraphs (blocks), we have to do
something like:

(progn (put 'h1 'block t)
       (put 'h1 'space-before XX)
       (put 'h1 'left-margin XX)
       etc. ...)

I can see four ways to deal with that.  The first two imply that we
ditch the semantics of hard newlines altogether in WP documents.
(`use-hard-newlines' would remain useful for text/plain;
format=flowed.)  In order of decreasing preference and in order of
increasing fragility:

1.  A box model supported by the display engine, of course.

2.  A special fill function that examines the properties and 
    DTRT.  This is basically how `epos-fill-paragraph' in
    wp-example.el works.

3.  Make sure that each region covered by a `category' property which
    identifies the text as content of a block is separeted by hard
    newlines.  For example by letting a function run by jit-lock,
    which updates that stuff constantly.  That would mean that in the
    first example hitting backspace would have this effect:
    `delete-char-backward' would remove the hard newline, but the
    updating function would insert it again immediately.

4.  Make every hard newline read-only, whenever the user says "this
    paragraph is of type h1".  Advice `yank' to DTRT.


I don't like 3. at all.  (I think that 4. is rather silly, mentioned
only for completeness.)  IMO it short circuits two different semantics
of distinguishing paragraphs.  And my gut feeling is that exactly this
is a potential source for a lot of subtle bugs.

The problem of XML's nested blocks is still another, unresolved issue.

    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I don't talk about WYSIWYG.  I abhor the very concept of it in the
>> context of word processing.  "WYSIWYG" means to say something about
>> the relationship between the rendering of the document on screen and
>> on printed paper.
>
> My understanding of WYSIWYG is not really that "the printer version is equal
> to the screen version"
[...]

Well, we are trying to make sense of a marketing term here.  My
understanding is that the "what you see" refers to the computer
display and the "what you get" refers to the printed paper.
Otherwise, I think, it should be called "what you see is what you
already have, of course, because that is what you see."
(WYSIWYAHOCBTIWYS).

I'd rather avoid the term "WYSIWYG" in this discussion altogether and
talk about clearer concepts.

[different printed output when Emacs is started on a tty]
> No: most WYSIWYG word-processors will happily accept (and print) colors in
> your document, even when your screen is black&white.  I.e. the way it
> typically works is that there *is* a deep representation but it can only be
> manipulated implicitly through the superficial representation which is
> displayed with "the best possible rendering on the current output device".

I meant the tty example as a reductio ad absurdum of the idea that the
printed output should resemble the rendering on screen.  I do think
that each visual (aural) represantation of an abstract document should
make the best use of the rendering device.

For example on a tty a user might specify: "If somebody displays this
document on a GUI, then display every 'Headline 1' paragraph in bold
with a size of 36pt.  But since I am editing on a tty, display it in
blue instead, so that I can distinguish it from a normal paragraph."

And on a GUI, on a low resolution computer screen, somebody might
prefer a sans serif font, while for the higher resolution printed
output the same person might prefer a font with serifes.

HTML/XML + CSS 2 already supports this in the document encoding.  For
RTF I still have to design a good solution.

>>> the deep representation (which, AFAIC, is not "just the disk format"
>>> but is the format where I can expresss my *intents*, i.e. where I
>>> can distinguish between two concepts even if they happen to be
>>> rendered identically on the currently used output mode)
>> [...]
>
>> Yes, that's a different editing model.
>
> I guess I still don't understand the model you're thinking of, which doesn't
> involve an explicit deep representation and yet isn't WYSIWYG either.

It is basically as you said: you express your intents.  But rather
than doing this via a command or markup language, you do it by
interacting with a UI.  This abstracts your intent from a specific
file format ("deep representation").  That is why I talk about
"abstract documents".

The benefit is that you do this while looking at the "surface
expression".  When I am working on a document this is the time where I
specially /want/ the additional visual clues provided by fonts, font
sizes, whitespace formatting ... whatever.  I do not want to edit a
representation (the deep one) that differs from the representation
that is meant for reading (the surface expression).  Your mileage may
vary; for either editing models there are people who prefer it.

[When the time is ripe, I'd like to think about different means to
make that as compatible as possible with a POV where emphasis is on
the "deep representation".  That would, for instance, involve clever
pretty printers that produce clean, human readable encoded document
files.]


    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Some of the existing hook properties may also be used to handle
"delete hard newline betwee paragraphs with different categories" in
some sensible way.

If nothing else, such hooks could set some global value which causes a
post-command-hook to fix whatever conflicts are created by the last
modification -- at least I think that would be more efficient that
relying on jit-lock to run "all the time".

`modification-hooks'
     If a character has the property `modification-hooks', then its
     value should be a list of functions; modifying that character
     calls all of those functions.  Each function receives two
     arguments: the beginning and end of the part of the buffer being
     modified.  Note that if a particular modification hook function
     appears on several characters being modified by a single
     primitive, you can't predict how many times the function will be
     called.

`insert-in-front-hooks'
`insert-behind-hooks'
     The operation of inserting text in a buffer also calls the
     functions listed in the `insert-in-front-hooks' property of the
     following character and in the `insert-behind-hooks' property of
     the preceding character.  These functions receive two arguments,
     the beginning and end of the inserted text.  The functions are
     called _after_ the actual insertion takes place.

     See also *Note Change Hooks::, for other hooks that are called
     when you change text in a buffer.

`point-entered'
`point-left'
     The special properties `point-entered' and `point-left' record
     hook functions that report motion of point.  Each time point
     moves, Emacs compares these two property values:

        * the `point-left' property of the character after the old
          location, and

        * the `point-entered' property of the character after the new
          location.

     If these two values differ, each of them is called (if not `nil')
     with two arguments: the old value of point, and the new one.

     The same comparison is made for the characters before the old and
     new locations.  The result may be to execute two `point-left'
     functions (which may be the same function) and/or two
     `point-entered' functions (which may be the same function).  In
     any case, all the `point-left' functions are called first,
     followed by all the `point-entered' functions.

     It is possible with `char-after' to examine characters at various
     buffer positions without moving point to those positions.  Only an
     actual change in the value of point runs these hook functions.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Stefan Monnier]

> It is basically as you said: you express your intents.  But rather
> than doing this via a command or markup language, you do it by
> interacting with a UI.

It's a false dichotomy: Emacs on a tty is a UI.
I guess what you mean to say here is that rather than representing the
intent as text you'd represent it somehow graphically.

Of course this graphical representation depends on the rendering device, so
the user can't know the intent unless he remembers how it's represented
graphically on this particular device.
That might be OK, of course, if the mapping is obvious enough from context.

> This abstracts your intent from a specific file format ("deep
> representation").

So you don't actually get to see the abstract representation, even though
that's really what you're editing.  I.e. you're still editing somewhat
blind-folded.

You're trying to strike a balance between WYSIWYG and plain text.

> The benefit is that you do this while looking at the "surface
> expression".

With something like WhyzzyTeX I get to edit while seeing (rather than
"looking at") the surface expression.


        Stefan

Re: enriched-mode and switching major modes.

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> You're trying to strike a balance between WYSIWYG and plain text.
>
>> The benefit is that you do this while looking at the "surface
>> expression".
>
> With something like WhyzzyTeX I get to edit while seeing (rather
> than "looking at") the surface expression.

I find that this does not work for most creative work (except
fine-tuning the page layout), and there are several reasons for that:
the fonts that are used are intended for printing, not the screen.
They have visual cues that help for reading at about 600dpi (at that
resolution, the hairlines of cm fonts get fine enough not to disturb
the leading character of the thicker serifs).  The screen can't render
that, and antialiasing, while correcting the worst grey level sins,
muddies things up at those resolutions.  As a result, creating text
with TeX fonts interactively is a pain.  For running text, I much
prefer my 10x20 high contrast, 100dpi-hand-designed pixel font.

But while I am creating text, I want to be able to focus my eyes on
the text I am entering, not vascillating between text entry and
typeset window.  I want to be able to focus on one cursor entry
position.

That's what preview-latex <URL:http://preview-latex.sourceforge.net>
provides me with: terminal fonts where I am interested in the text,
antialiased WYSIWYG where I am interested in more complex compositions
like math formulas, graphs and similar stuff, where the input
representation is so far from the represented content that thinking
about the content without visual translation becomes difficult.

preview-latex just works on-demand.  In contrast X-Symbol
<URL:http://x-symbol.sourceforge.net> works automatically and the deep
representation \pm is replaces by ± the moment that I type a space or
otherwise create a separate token.  From that time on, the character
in the buffer really _is_ ± and not the token \pm.  The conversion to
tokens is only done when the file is saved or read in.  This way of
working is inherently dangerous: when things go wrong, your whole file
may be en/decoded in some weird coding.  Also doing isearch for \pm
will not work.  With preview-latex, which still has a separate deep
representation underlying the graphics, it will.

The fine lines when you want to work with deep and surface
representation will be different from application to application.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Oliver Scholz <alkibiades@gmx.de> writes:

> <blush/> I have to admit that I have never thought of the `category'
> property, instead I have always experimented with implementing its
> functionality myself.  Thanks for the reminder.  This might indeed be
> useful.

I think we need a way for merging categories, if those are supposed to
be used.

In RTF we have:


(section stylesheets: specifying both character and paragraph
formatting properties)

paragraph stylesheets: specifying both character and paragraph
formatting properties

character stylesheets: specifying only character formatting
properties

explicit setting of propeties, bother character and paragraph
formatting.


These four need to be merged in that order.  I'd like to use the
`category' text property for both character and paragraph formatting,
but I don't know how.

I used to solve that problem in my past prototypes by maintaining two
text properties, one for character and one for paragraph properties.
These text properties would hold a vector each with all necessary
formatting properties, including a reference to a stylesheet, if any.
A fontification function in `jit-lock' would then resolve the
formatting information accordingly and create an anonymous face.  This
works.  But finding a way to use the `category' text property would be
nicer.

Here is a make-up example for the issue in XHTML:

<?xml version="1.0" encoding="us-ascii" ?>
<html>
  <head>
    <title>Lirum larum</title>
    <style type="text/css">
       h2 { text-decoration: underline }
       span.charstyle
           {
             font-style: italic;
             color: blue;
           }
        span.special { font-size: 12pt; }
       <!-- Think of span.charstyle as the character formatting
           stylesheet in RTF and of span.special as explicit
           overwriting of a character formatting property. -->
    </style>
  </head>
  <body>
    <h2>The headline plus <span class="charstyle">character formatting
    stylesheet plus
      <span class="special">explicitely added
      properties.</span></span></h2>
  </body>
</html>


    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Oliver Scholz]

storm@cua.dk (Kim F. Storm) writes:

> Some of the existing hook properties may also be used to handle
> "delete hard newline betwee paragraphs with different categories" in
> some sensible way.
>
> If nothing else, such hooks could set some global value which causes a
> post-command-hook to fix whatever conflicts are created by the last
> modification -- at least I think that would be more efficient that
> relying on jit-lock to run "all the time".

I agree that it would work.  But why is it better than the solution
that I listed as #2, i.e. letting a special fill-function handle all
whitespace formatting issues?  I have experimented with this and it
works nicely.  It can also handle the problem of putting whitespace of
a specific height before and after a paragraph rather elegantly.

The way I see it we first create two different semantics for
paragraphs: one via the `category' property, one via hard new lines.
And then we take special pain to keep both representations in sync.
To me this seems like asking for trouble.

The only benefit I can see, is that `forward-paragraph' and its like
would DTRT.  But writing a version that works correctly for the other
solution and binding it to the appropriate keys is rather trivial.


...!


Oh, no!  I see now, this is a way to make a slightly modified
`fill-paragraph' work.  Some of my major concerns could be adressed
this way and /technically/ it could be handled by a minor mode.
(Though I don't see what good that should be.  How a paragraph should
be indented in an RTF document depends on its properties, not on
paragraph-indent-text-mode nor on paragraph-indent-minor-mode.)  It
could be a way to provide a user interface for RTF where I can type
four spaces to indent a paragraph like in text mode.

I don't like the resulting user interface.  I don't like it at all.
In fact I find it really, really horrible.

Would it be possible to handle XML with its tree-like structure this
way?  My own thoughts have let into an entirely different direction
(not explained in this thread); offhand I don't see how nested block
boxes would be possible with hard newline semantics.


    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It is basically as you said: you express your intents.  But rather
>> than doing this via a command or markup language, you do it by
>> interacting with a UI.
>
> It's a false dichotomy: Emacs on a tty is a UI.
> I guess what you mean to say here is that rather than representing the
> intent as text you'd represent it somehow graphically.

This is nitpicking.  By UI I meant to say that you issue a command M-x
set-the-type-of-this-paragraph.  Or hit `C-u 4 C-c C-i'
(indent-this-paragraph).  Or click on the "Make this italic" button in
the tool bar.  Or whatever.  Since things like this can be---and in
fact are---provided by modes for editing your "deep representation",
you might say that the difference is that this is the /only/ way to
express your intent.  The point is that you express your intent while
not caring about the particular encoding.

[...]
>> This abstracts your intent from a specific file format ("deep
>> representation").
>
> So you don't actually get to see the abstract representation, even though
> that's really what you're editing.  I.e. you're still editing somewhat
> blind-folded.

"somewhat blind-folded" is a vast exaggeration.

> You're trying to strike a balance between WYSIWYG and plain text.

I can not parse that sentence.  It makes absolutely no sense to me.

>> The benefit is that you do this while looking at the "surface
>> expression".
>
> With something like WhyzzyTeX I get to edit while seeing (rather than
> "looking at") the surface expression.

I think I already understood that you like this way of editing.  Now,
it seems that I also have to understand that in your view this is the
Only True Way of Editing.

Do you really think, that /I/ or anybody else who wants "word
processing" in Emacs would be content with WhyzzyTeX-style functionality?

A gedankenexperiment: Suppose we have the inverse of WhizzyTeX: you
edit the visual appearance WP-style and Emacs would constantly update
the encoded document in a read-only buffer and show it in another
window.  Would /you/ be content with that?  Well, /I/ would like it,
though I'd probably hit C-x 1.

O.k. more precise rephrasing: the benefit is that all editing actions
take place in a buffer that only shows the "surface expression".


    Oliver
-- 
Oliver Scholz               Jour des Récompenses de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Stefan Monnier]

>> With something like WhyzzyTeX I get to edit while seeing (rather
>> than "looking at") the surface expression.

> I find that this does not work for most creative work (except

I'm not sure what "this" refers to, but I completely agree: I don't want to
use the same font while editing and while printing (I typically like my
screen fonts to be sans-serif, while I usually want to print with something
like computer-modern or times-roman).

> That's what preview-latex <URL:http://preview-latex.sourceforge.net>
> provides me with: terminal fonts where I am interested in the text,
> antialiased WYSIWYG where I am interested in more complex compositions
> like math formulas, graphs and similar stuff, where the input
> representation is so far from the represented content that thinking
> about the content without visual translation becomes difficult.

Yes, it's also a very interested point in the design space.
I don't think WhyzzyTeX and preview-latex should be considered as necessarily
opposed designs.


        Stefan

Re: enriched-mode and switching major modes.

[Stefan Monnier]

>>> It is basically as you said: you express your intents.  But rather
>>> than doing this via a command or markup language, you do it by
>>> interacting with a UI.
>> 
>> It's a false dichotomy: Emacs on a tty is a UI.
>> I guess what you mean to say here is that rather than representing the
>> intent as text you'd represent it somehow graphically.

> This is nitpicking.

Oh, sorry, I misunderstood (nitpicked as well, of course).  I was thinking
in terms of *reading*, not *writing*.  I find the visual representation of
the intent to be very important: if it weren't for that, indeed I wouldn't
care as much whether I was seeing the deep or surface representation.

> The point is that you express your intent while
> not caring about the particular encoding.

But getting the intent from the user to the data is "easy".  It's getting it
the other way around which is the harder part (unless you display the deep
representation, that is).

>> So you don't actually get to see the abstract representation, even though
>> that's really what you're editing.  I.e. you're still editing somewhat
>> blind-folded.

> "somewhat blind-folded" is a vast exaggeration.

I hope you now better understand what I meant by that.

>>> The benefit is that you do this while looking at the "surface
>>> expression".
>> With something like WhyzzyTeX I get to edit while seeing (rather than
>> "looking at") the surface expression.
> I think I already understood that you like this way of editing.  Now,
> it seems that I also have to understand that in your view this is the
> Only True Way of Editing.

I don't think it's the one true way.  I just think it's an interesting point
in the design space, like preview-latex, tex-mode, Lyx, TeXmacs, FrameMaker,
OpenOffice, ...

> Do you really think, that /I/ or anybody else who wants "word
> processing" in Emacs would be content with WhyzzyTeX-style functionality?

I'm sorry if you got the impression I thought the answer is yes.

> A gedankenexperiment: Suppose we have the inverse of WhizzyTeX: you edit
> the visual appearance WP-style and Emacs would constantly update the
> encoded document in a read-only buffer and show it in another window.

It's not the inverse.  WhyzzyTeX (assuming it's implemented differently so
as to work around the severely rough edges) does exactly that, both ways:
the only detail is whether the window manager's focus is on one window
rather than another.


        Stefan

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Oliver Scholz <epameinondas@gmx.de> writes:

> I think we need a way for merging categories, if those are supposed to
> be used.
>
> (section stylesheets: specifying both character and paragraph
> formatting properties)
>
> paragraph stylesheets: specifying both character and paragraph
> formatting properties
>
> character stylesheets: specifying only character formatting
> properties
>
> explicit setting of propeties, bother character and paragraph
> formatting.

Maybe we could allow 'category to take a value which is a list
of category symbols -- and then use those categories in order, i.e.
when looking for a specific property, use the first occurrence.

Just like an explicit property should (does?) take precedence over
a property from a category.

Like this:

     'category '(char-style para-style sect-style)


-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Robert J. Chassell]

Oliver Scholz <epameinondas@gmx.de> wrote,

    The benefit is that you do this while looking at the "surface
    expression".

Please be more precise and say `one of the surface expressions'.

Otherwise, someone may think that a particular surface expression is
prime.  This happens not only when people work on LaTeX to produce
papers that are to be printed, but with many word processing programs.

Unfortunately, even Emacs people think this way sometimes.  Yes, it is
less of a problem since people who use Emacs tend to read Texinfo
documents and know they may be printed or in Info or in HTML (or they
listen to them).  But even so, I have run into people editing a
Texinfo source file who think it is primarily for `the book', i.e.,
for printed form typeset with TeX, or for `Info'.  I have done this
myself.  I fear this tendency towards over simplification is
universal.  It needs to be fought.


Stefan <monnier@iro.umontreal.ca> made the good point that 

    ... the deep representation . .. is the format where I can
    expresss my *intents*, i.e. where I can distinguish between two
    concepts even if they happen to be rendered identically on the
    currently used output mode

This is worth remembering.  One surface representation provides you
with only one way to specify your intent.  Pretty obviously, this may
not be the only intent you want to save.

Nonetheless, this intent must be incorporated into the deep
representation along with the others.  (This is all very hard to do;
and some surface representations, such as Info, simply cannot provide
all the data used by a representation such as that produced by TeX.)

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The point is that you express your intent while
>> not caring about the particular encoding.
>
> But getting the intent from the user to the data is "easy".  It's getting it
> the other way around which is the harder part (unless you display the deep
> representation, that is).

Ah, yes, I see.  Yeah, the editing model where display the deep
representation has its undeniable strengths.

I have a few ideas how to improve a word processing UI a little bit.
>From putting additional (optional) constraints on it ("I don't want to
format my text by means of literal space characters.  Stop me from
accidentally inserting more than one, except at the end of a sentence.
I don't want to use explicit formatting properties, let me use
stylsheets only.")  up to displaying additional information in the
margin.  It would not be the same as when editing the deep
representation directly, but it could help.

[...]
> I don't think it's the one true way.  I just think it's an
> interesting point in the design space, like preview-latex, tex-mode,
> Lyx, TeXmacs, FrameMaker, OpenOffice, ...

I agree.

Word processing functionality would require a lot of code for parsing
and rendering.  If the architecture is modular enough, that same code
base could be used to implement something like preview-latex for HTML
and XML as well, or to implement WhizzyHTML, where you can edit both
the deep representation and the displayed surface expression.  If done
well, you can users can switch between several editing models at their
leisure.  It is all blue sky, though, but it is worth keeping such
things in mind, anyways.

I really like the word processing model, but even I could imagine that
I sometimes would want to frob the encoded HTML markup of a table
directly.  That's why I would like the WP to provide a means to
"unfold" the visual rendering of a block and edit its markup encoding
directly.  The problematic spot in this model, unlike in preview-latex
or WhizzyTeX, is source preservation.  Source preservation is probably
the spot where both fundamental models ("deep representation" <->
"surface expression" vs. "encoded document file" <-> "abstract
document" <-> "visual (aural) appearance") are not 100% compatible.

    Oliver
-- 
Oliver Scholz               Jour de la Révolution de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Oliver Scholz]

storm@cua.dk (Kim F. Storm) writes:

[...]
> Maybe we could allow 'category to take a value which is a list
> of category symbols -- and then use those categories in order, i.e.
> when looking for a specific property, use the first occurrence.
>
> Just like an explicit property should (does?) take precedence over
> a property from a category.
>
> Like this:
>
>      'category '(char-style para-style sect-style)
[...]

Faces would need to be merged.

(progn
  (put 'h1 'face 'Info-title-1-face)
  (put 'italic 'face '(face :slant italic))
  (switch-to-buffer (generate-new-buffer "*tmp*"))
  (insert (propertize "lirum larum"
                      'category '(italic h1)
                      'face '(face :foreground "Blue"))))

This should get an italic `Info-title-1-face' in blue.

    Oliver
-- 
Oliver Scholz               Jour de la Révolution de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Kai Grossjohann]

Richard Stallman <rms@gnu.org> writes:

> Because of Lisp, a similar thing happens for text in an Emacs buffer.
> In a typical word processor, text has a visual appearance and a way it
> is saved in a file, that's all.  In Emacs, text has a visual
> appearance, a way it is saved in a file, and the way it appears as
> Lisp data.  We have to design all three.

Talking about the Lisp representation, consider an enumerated list
like this:

1. First item.  This item has a lot of text to show what happens when
   it is multi-line.

2. Second item.

3. Third item.

What should Lisp see and what shouldn't it see?

I would expect Emacs to compute the numbers automatically, so that
inserting an item into the middle would recompute the numbers.  To me,
this means it makes no sense to make the numbers themselves accessible
via Lisp.  But Lisp should see that there is an enumerated list using
the "1." numbering style (as opposed to "1)" or "1/", say), and Lisp
should see that the list has three items.

I would also expect the line spacing between the items to vary
depending on the style sheet.  That is, Lisp shouldn't see two newline
characters after "multi-line.", but just the end of the first item.

Now let's talk about the newlines and spaces between "when" and "it"
(in the first item).  Suppose you decide that you want the item
numbers to come out bold.  Often, bold weight runs longer than medium
weight.  This means that the indentation of the "it" line might grow a
bit.  Does it really make sense to insert more spaces in front of "it"
just because the user has changed the style sheet of the document to
specify that enumerations should use bold numbers?  Additionally,
depending on the exact text, making the numbers wider could mean that
the "when" does not fit on the line anymore.  Making the newlines
accessible via Lisp would mean that Lisp would suddenly see a newline
in front of "when", instead of after "when".

Now let's talk about cursor movement.  I think that positioning point
after "multi-line." and then hitting C-f should position point before
"Second".  The "2." part is not meaningfully editable, as it is
computed automatically.  The whitespace isn't meaningfully editable,
either, since the amount depends on the stylesheet in use.


I think that Lisp should have access to the stylesheet information,
though.  That is much more useful than knowing that "when" happens to
be at the end of a line.  For instance, I want to be able to search
for enumeration items containing the word "when".  (The search should
ignore "when" occurring outside of enumerations.)


What do people think?

Kai

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Oliver Scholz <alkibiades@gmx.de> writes:

> storm@cua.dk (Kim F. Storm) writes:
>
> [...]
>> Maybe we could allow 'category to take a value which is a list
>> of category symbols -- and then use those categories in order, i.e.
>> when looking for a specific property, use the first occurrence.
>>
>> Just like an explicit property should (does?) take precedence over
>> a property from a category.
>>
>> Like this:
>>
>>      'category '(char-style para-style sect-style)
> [...]
>
> Faces would need to be merged.

True -- but emacs already knows how to merge faces, so
I think that would be easy.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Kai Grossjohann <kai@emptydomain.de> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> Because of Lisp, a similar thing happens for text in an Emacs buffer.
>> In a typical word processor, text has a visual appearance and a way it
>> is saved in a file, that's all.  In Emacs, text has a visual
>> appearance, a way it is saved in a file, and the way it appears as
>> Lisp data.  We have to design all three.
>
> Talking about the Lisp representation, consider an enumerated list
> like this:
>
> 1. First item.  This item has a lot of text to show what happens when
>    it is multi-line.
>
> 2. Second item.
>
> 3. Third item.
>
> What should Lisp see and what shouldn't it see?
>


It depends on which "level" of Lisp you are talking about.

On the level which must take the "deep representation" and
show it on the screen, Lisp definitely need to look at 1.,
2., line wrapping, etc.

On the level which navigates "on the screen" in response to
cursor movement, or searching, only looking at the "text" while
ignoring the formatting makes sense.  But it will probably have
to be aware of it anyway to DTRT...

I don't think you can have a surface presentation which is
radically different from the deep representation without
writing a suitable set of cursor movement commands to
accomplish it.

Maybe the 'line-move' framework could be extended to allow a specific
surface presentation to mark things to be skipped over (it already
does that in various ways, but maybe there is a way to generalize
that so a minor/major mode can modify the behaviour).

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Robert J. Chassell]

Previously, when talking about transforming documents, I have focused
on two terms for thinking from an end user's (not a programmer's)
point of view, "deep representation" and "surface expression".

But three terms fit the situation more closely:

    "deep representation"  
    "intermediate expression" 
    "surface expression"

Among other things, the additon enables us to talk about the Lisp
object representation more readily.

Suppose we want to display an enumerated list, such as Kai Grossjohann
provided.  In plain text it looks like this:

    1. First item.  This item has a lot of text to show what happens when
       it is multi-line.

    2. Second item.

    3. Third item.

In Texinfo, the list looks like this:

    @enumerate
    @item First item.  This item has a lot of text to show what happens when
    it is multi-line.

    @item Second item.  

    @item Third item.  
    @end enumerate

The Texinfo source is the "deep representation" which is also the
"encoded document file".

For Info, makeinfo produces this:

  1. First item.  This item has a lot of text to show what happens
     when it is multi-line.

  2. Second item.

  3. Third item.

(In the past, the Info file has been a permanent file rather than a
temporary data structure.  That does not matter for this discussion.)

The Info output is a `surface expression' for one kind of
human/computer interaction but the source for another.  When talking
about it as a source, it is an `intermediate expression'.

Similarly for HTML.  `makeinfo' currently produces this ugly looking
output for that Texinfo source:

     <ol type=1 start=1>
<li>First item.  This item has a lot of text to show what happens when
it is multi-line.

     <li>Second item.

     <li>Third item.
        </ol>

(The HTML should look more like

<ol>
  <li> First item.  ....
  <li> Second item.  
  <li> Third item.  
</ol>

but this is a digression.)

Again the HTML is a surface expression in one circumstance (such as
here in the mail message) and a source in another, such as when I look
at my `test.html' file in Firefox.  The term `intermediate expression'
helps clarify this.

A Lisp object representation is, from this point of view, another
`intermediate expression'.  It is not the "encoded document file" nor
is it what a novice user looks at or listens to.  (When a programmer
works with the Lisp object representation, then it is to him a
`surface expression.)

In current Emacs Enriched mode, the deep representation is the version
that is seen by `find-file-literally'.  It has lots of angle brackets.
The surface expression has colors (or if you listen to it, it has
different voices, I think).

In Enriched mode, the Lisp object representation is an `intermediate
expression'.

Of course, there are more steps to each process.  For example, when
you view an HTML file in Emacs W3 mode, the (simpliefied) sequence
goes like this:


test.texi           -->        test.html        -->     W3 buffer

deep representation --> intermediate expression --> surface expression
                    |                           | 
                    V                           V
      the `makeinfo' transform       the W3 mode transform uses 
        uses temporary data           a Lisp object representation 
                                      as an `intermediate expression'

And depending on how you look at it, `test.html' may serve as a `deep
representation' or as a `surface expression'.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: enriched-mode and switching major modes.

[Richard Stallman]

    If I understand Kim correctly, then simple tables where the first
    column contains a caption for the row would be an example where it
    would matter.

Is it common practice to use "a caption for a row" in a document?

    E.g. tables with different font sizes.  Try this:

    <html><body>
    <table>
    <tr>
    <td><font size=7>AAA</font></td>
    <td>bbbbbb<br>cccccc</td>
    </tr>
    </table>
    </body></html>

Is this a useful thing to do?  How important is it
for a word processor to support this sort of thing?

Re: enriched-mode and switching major modes.

[Richard Stallman]

    > When encoding a buffer, I need to identify the whole paragraph as
    > being of the type "h1".  I.e. I have to distinguish it from:

    Maybe we can get some of the way with Emacs' category text properties?!

The `category' property is meant for things like this.  But it is not
used much if at all.  If its existing definition is not quite right,
we can change it, or we could add a new feature that DTRT.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    When rendered by a graphical, CSS2-enabled browser, you'll see two
    paragraphs on a gray background sourounded by a dashed border.  Those
    two paragraphs are again contained in a larger paragraph on a purple
    background surounded by a solid border.

It will be very hard to implement this in a way that fits in with
Emacs.

The text property feature has been designed to work well with all
kinds of commands that copy text.  What makes it work well is that
text properties are per character.  So if you copy each character with
its properties, everything works ok.  Emacs primitives that copy text
all do this.

I am not sure how to represent nested blocks with per-character text
properties.  In principle we could have other kinds of data associated
with the text to supplement text properties, but what could we design
that would fit in properly with cut and paste?

What does it *mean* to copy a character from inside environment
`larum' which is inside environment `lirum' and insert it somewere
else?  What should that character look like in its new location?

One possible approach is to use markup characters, put the properties
on the markup characters (or express them in the text), and make these
markup characters more or less invisible in formatted display mode.
Then we would use text props only for fontification, etc., within a
paragraph.

However, it is worth doing this only if the result is an improvement.
I'm not sure of that.  For instance:

    <h1>Some <i>meaningless</i> heading</h1>

    The <i> element maps directly to text properties, of course.  But the
    h1 element both demands that its contents be rendered as a paragraph
    (a block) /and/ specifies certain character formatting properties for
    the whole of it, e.g. a large bold font.

    When encoding a buffer, I need to identify the whole paragraph as
    being of the type "h1".  I.e. I have to distinguish it from:

    <p><font size=7><b>Some <i>meaningless</i> heading</font></p>

Why do you have to distinguish them?  Why wouldn't it work simply to
put these properties on the whole text of the paragraph?  What aspect
would work differently as a result of doing one or the other, and why
is it better if the properties are attached to paragraphs?

    We have to deal with the case that a user deletes the hard newline (if
    you evaluate the code above: just hit backspace).  Is the resulting
    paragraph of type `h1' or of type `h2'?

Why ask the question?  Why not just accept that it's a paragraph
of partly h1 text and partly h2 text?

    I don't think that looking at indentation and newlines could at any
    rate be enough for something as complex as a WP document.

It's a matter of what the specific command does.  We're talking about
commands not written specifically for word-processing documents,
general-purpose code.

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Kai Grossjohann <kai@emptydomain.de>
> Date: Tue, 21 Sep 2004 11:53:41 +0200
> 
> I would expect Emacs to compute the numbers automatically, so that
> inserting an item into the middle would recompute the numbers. To me,
> this means it makes no sense to make the numbers themselves accessible
> via Lisp.

That's not a contradiction: Emacs could recompute them each time a new
item is inserted or deleted, and keep the result in the buffer.

> But Lisp should see that there is an enumerated list using
> the "1." numbering style (as opposed to "1)" or "1/", say), and Lisp
> should see that the list has three items.

If we keep the numbers in the buffer, it becomes almost trivial for a
Lisp program to see them.  That is why keeping them does make sense,
IMHO.

> I would also expect the line spacing between the items to vary
> depending on the style sheet.  That is, Lisp shouldn't see two newline
> characters after "multi-line.", but just the end of the first item.

As Kim pointed out, that depends on what the Lisp program wants to do.
In many cases, Lisp programs walk the buffer in order to decide how
the displayed text looks (or will look when it comes into the visible
portion of the buffer).  Such programs will want to see a very close
approximation of the actual display.

> Now let's talk about the newlines and spaces between "when" and "it"
> (in the first item).  Suppose you decide that you want the item
> numbers to come out bold.  Often, bold weight runs longer than medium
> weight.  This means that the indentation of the "it" line might grow a
> bit.  Does it really make sense to insert more spaces in front of "it"
> just because the user has changed the style sheet of the document to
> specify that enumerations should use bold numbers?

That's a design decision we will have to make.  Ideally, it wouild be
nice to adjust the indentation; in the specific example you've given,
the effect might be almost invisible, but I can think of similar
situations where if we don't adjust the indentation, the text will
look ugly on the screen.  (Try to use a proportional font in a C-Mode
buffer, and you will see what I mean.)

For this indentation to work, Emacs needs to support fractional
spaces.

> Additionally, depending on the exact text, making the numbers wider
> could mean that the "when" does not fit on the line anymore.  Making
> the newlines accessible via Lisp would mean that Lisp would suddenly
> see a newline in front of "when", instead of after "when".

It will see a newline with a special text property.  A Lisp program
that cares about soft newlines can use the property to find that out
and DTRT.

> Now let's talk about cursor movement.  I think that positioning point
> after "multi-line." and then hitting C-f should position point before
> "Second".  The "2." part is not meaningfully editable, as it is
> computed automatically.  The whitespace isn't meaningfully editable,
> either, since the amount depends on the stylesheet in use.

This "smart" cursor motion is not hard to implement (we already do
something similar with invisible/intangible text, don't we?): cursor
motion is part of the display engine, which already examines text
properties as part of its routine operation.  The complications you
mention pale in comparison to what Emacs will need to do after C-f to
support bidirectional editing, for example: there, C-f would sometimes
go backwards or jump over arbitrarily large sequences of characters.

> For instance, I want to be able to search
> for enumeration items containing the word "when".  (The search should
> ignore "when" occurring outside of enumerations.)

If the enumerated items have a suitable text property on them, such a
feature will not be too hard to implement, I think.

Btw, one limitation of text properties is that they cannot overlap.
So, if we base these features on text properties, we need to solve the
nesting problem somehow (or prove to ourselves that nesting is not
required).

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 21 Sep 2004 14:30:53 -0400
> Cc: boris@gnu.org, eliz@gnu.org, alex@emacswiki.org, emacs-devel@gnu.org
> 
> I am not sure how to represent nested blocks with per-character text
> properties.  In principle we could have other kinds of data associated
> with the text to supplement text properties, but what could we design
> that would fit in properly with cut and paste?

Perhaps such portions of text should only be cut-n-pasted as
rectangles.

> What does it *mean* to copy a character from inside environment
> `larum' which is inside environment `lirum' and insert it somewere
> else?  What should that character look like in its new location?

This is a hard problem.  I don't have solutions, but I can say that
one very widespread non-free word processor sometimes does _HORRIBLE_
things when text is copied from a fragment that uses some non-trivial
style to another portion of the document.  My suggestion would be to
stick to the KISS principle as much as it's feasible.

Re: enriched-mode and switching major modes.

[Stefan Monnier]

>> What does it *mean* to copy a character from inside environment
>> `larum' which is inside environment `lirum' and insert it somewere
>> else?  What should that character look like in its new location?

> This is a hard problem.

Indeed, if your markup is implicit, it's a hard problem because the user
can't actually see what he's selected for copy&paste.

If it's explicit in the form of markup commands (as is the case in
tex-mode), then it's a non-issue (i.e. it's now an issue for the user
rather than for the word processor.  And that's good, because the user
hopefully knows what he wants, whereas the word processor can't guess).


        Stefan

Re: enriched-mode and switching major modes.

[Miles Bader]

On Tue, Sep 21, 2004 at 02:30:52PM -0400, Richard Stallman wrote:
>     Maybe we can get some of the way with Emacs' category text properties?!
> 
> The `category' property is meant for things like this.  But it is not
> used much if at all.

buttons uses it... :-)

-Miles
-- 
`Life is a boundless sea of bitterness'

Re: enriched-mode and switching major modes.

[Miles Bader]

On Tue, Sep 21, 2004 at 09:53:58PM +0300, Eli Zaretskii wrote:
> Btw, one limitation of text properties is that they cannot overlap.
> So, if we base these features on text properties, we need to solve the
> nesting problem somehow (or prove to ourselves that nesting is not
> required).

But they _can_ be any lisp value, including a list, so application code could
do its own merging.

-Miles
-- 
"1971 pickup truck; will trade for guns"

Re: enriched-mode and switching major modes.

[David Kastrup]

"Eli Zaretskii" <eliz@gnu.org> writes:

> As Kim pointed out, that depends on what the Lisp program wants to
> do.  In many cases, Lisp programs walk the buffer in order to decide
> how the displayed text looks (or will look when it comes into the
> visible portion of the buffer).  Such programs will want to see a
> very close approximation of the actual display.

"Many cases" meaning almost exclusively the display engine, and maybe
cursor motion.  Which is not the majority of Lisp programs.  In
particular, it is almost exclusively Emacs-internal code, not that
from external packages or written by a user.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: enriched-mode and switching major modes.

[James Clark]

On Wed, 2004-09-22 at 01:30, Richard Stallman wrote:
>     > When encoding a buffer, I need to identify the whole paragraph as
>     > being of the type "h1".  I.e. I have to distinguish it from:
> 
>     Maybe we can get some of the way with Emacs' category text properties?!
> 
> The `category' property is meant for things like this.  But it is not
> used much if at all.  If its existing definition is not quite right,
> we can change it, or we could add a new feature that DTRT.

I wonder whether the value of the `category' property should be allowed
to be a list of atoms somewhat like with `invisible'.  This would allow
multiple emacs modes to use the `category' property without collision,
and would also allow one to handle a case like  <b>foo<i>bar</i></b>:
foo would have the property (category (b)) and bar would the property
(category (i b); i would have property face (:slant italic), b would
have property face (:weight bold) and the net result would be that foo
would be bold and bar would be bold and italic, which is what one would
want.

James
-- 
To send me mail, replace auth-only by public in the from address. 

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> Cc: rms@gnu.org, boris@gnu.org, alex@emacswiki.org,
> 	emacs-devel@gnu.org, alkibiades@gmx.de
> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 21 Sep 2004 16:06:34 -0400
> 
> If it's explicit in the form of markup commands (as is the case in
> tex-mode), then it's a non-issue (i.e. it's now an issue for the user
> rather than for the word processor.

Not really, not with complex styles.  A style can have lots of
complicated settings, and it could be very hard to undesrtand which
ones are in effect, even with explicit markup, since some directives
that set the style can be very far away of the text you are copying.
It is even harder to understand what will happen when the copied text
removes some of the directives or stretches into another style area.

As the first approximation to guessing ``what the user wants'' I'd
simply remove all style settings from the copied text and applied the
settings of the target text portion when the text is pasted.  It is
not always the 100% right thing to do, but it is usually close, and
simple commands (such as renumber an itemized list) can be added to
fix whatever becomes broken as the result.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    Maybe we could allow 'category to take a value which is a list
    of category symbols -- and then use those categories in order, i.e.
    when looking for a specific property, use the first occurrence.

That seems clean to me; if it does the job, I would say yes.

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

>     If I understand Kim correctly, then simple tables where the first
>     column contains a caption for the row would be an example where it
>     would matter.
>
> Is it common practice to use "a caption for a row" in a document?

Maybe not, but it shows the general concept of what is displayed in
one table cell can be _completely_ unrelated to what is displayed in
another table cell.

>
>     E.g. tables with different font sizes.  Try this:
>
>     <html><body>
>     <table>
>     <tr>
>     <td><font size=7>AAA</font></td>
>     <td>bbbbbb<br>cccccc</td>
>     </tr>
>     </table>
>     </body></html>
>
> Is this a useful thing to do?  How important is it
> for a word processor to support this sort of thing?

Very important.

Consider also the example that instead of AAA we have a tall image,
and bbbb ccccc etc is text that is displayed next to that image like
this:

+---------+
|         | this text
|         | is shown
|         | on multiple
|         | lines on the
|         | right side of
|         | an image
+---------+


I still think my idea to use "anonymous" windows as the base of a
block display model would be a good solution (and we already have 95%
of the code to do it).

It could be done like this:

(let* ((b (get-buffer-create "X"))
       (w (make-anonymous-window b)))
  (with-current-buffer b
    (insert 
"this text    
is shown     
on multiple  
lines on the 
right side of
an image"))
  (insert-image IMAGE)
  (insert-anonymous-window w))


The insert-anonymous-window would do something like

(defun insert-anonymous-window (w)
  (insert (propertize "*" 'display `(window ,w))))

Of course, cursor movement need to know how to move in and out of the
anonymous window when we move through the display -- that might be
doable with point-enter and point-leave properties, or we need some
other mechanism.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     When rendered by a graphical, CSS2-enabled browser, you'll see two
>     paragraphs on a gray background sourounded by a dashed border.  Those
>     two paragraphs are again contained in a larger paragraph on a purple
>     background surounded by a solid border.
>
> It will be very hard to implement this in a way that fits in with
> Emacs.

Okay, maybe this is the time to lay out the design on which I am
spending thought and code since I ditched the approach that I already
mentioned (as in `wp-example.el').  I fear, though, that you won't
like it.

The idea crossed my mind when I thought about how to implement a
data structure fit for XML + CSS in Emacs Lisp.  In other words: how
to make Emacs a /rendering/ XML editor.

XML is by nature a tree-like format.  The W3C has specified the
structure of information contained in XML documents in a way that
abstracts from the pointy brackets syntax; this abstract data set is
called the "XML Information Set":

http://www.w3.org/TR/xml-infoset/

For simplicity, I focus on elements and character data here and talk
about them as "nodes" in a tree; thus we have "element nodes" and
"text nodes".  An XHTML fragment like

<h1>Some <em>meaningless</em> text</h1>

Would be regarded as a `h1' element node which has three children: a
text node "Some ", an element node `em' (which has itself a text node
as its single child) an another text node " text".

I found out that I can translate any RTF document into an instance of
the XML info set.  So I can reduce the problem of designing a data
structure for word processing in Emacs to the question of how to
implement the XML info set in a way that text nodes are stored in a
buffer rather than in a string and that they are /editable/.  And that
question I can reduce to: how can I implement a tree-like data
structure with text properties?

So far I have considered two ways to do this. Both have specific
disadvantages.  But I'll come to that in a minute.

[If desired, I have prototype code for each of those two approaches to
experiment with. :-/ ]

One way is to have a single, unique Lisp object, a vector for example,
stored in a text property, say `text-node'.  That vector (or list)
would store a reference to its immeditate parent (which is always an
element node).  That parent would have a reference both to its
children and to its own parent and so on.  In addition, a buffer-local
variable would store the root element.

This has the advantage that I have two views of the document: One as a
Lisp Object, a tree of vectors or of lists, stored in a variable; the
other one as the content of a buffer with specific text properties.
The former allows to implement an API for accessing the contents of
the document and modifying it---I am thinking of XPath, DOM and other
W3C standards here that many people are familiar with.  If I have a
text node (the vector or list), then I can find its text in the buffer
with

`(text-property-any (point-min) (point-max) 'text-node TEXT-NODE)'

This should be fast.

But this solution has an undesirable fragility: care must be taken,
when killing and yanking, that both the text properties and the tree
be updated accordingly (for example if the killing results in the
entire deletion of a text node).  And if the tree is modified directly
(via the API), then the buffer contents need to be updated, too (for
example when this leads to transfering text nodes to another place in
the tree).  Basically this is again the problem of keeping two
structures in sync again.

The other way is to have a text property `parents' on each text
node in the buffer.  This would hold a list of all ancestor nodes
in the tree, starting with the immediate parent.  The
disadvantage here is that finding nodes takes much more time.
Especially finding all the children or descendants of a node
takes time.  Whereas in #1 I have a reference to the children in
the node, here I have to scan several ranges of text properties
to determine the children, e.g.

    1. Find the first position in the buffer where NODE is a
       member of the value of the text property `parents'.

    2. Push the value of `parents' to a list.

    3. Find the next single property change of `parents'.
    
    4. Determine if NODE is a member of the value of `parents'.
       If yes, goto 2.  If no, got 5.

    5. Determine children or descendants from the collected
       values.

Some care is necessary with copying and inserting text.  But we avoid
to keep to separate structures in sync at the cost that the access of
nodes (and thus the API) is inefficient.


So how to handle formatting in the buffer?  The element nodes would
store formatting information---either after applying a CSS stylesheet
to the tree, or, in the case of RTF, right away when parsing the file.
Functions that apply the formatting in the buffer (i.e. filling and
jit-lock) scan the tree upwards until they find the information they
need.

I have not yet determined whether #2 requires too much time for this.
The idea of #2 is rather new and not fully thought out and tested.  I
am not certain if I am aware of all possible pitfalls.  Moreover, I
have not yet figured out every detail of handling formatting
information in general.  I am still in the process of reading
specifications in order to get an overview.  I have to admit that I
have also been wondering, whether something could be done on the C
level to provide for such tree-like documents in an Emacs buffer.  I
don't have a clue here, though.

[Yesterday or so, a third way how to handle nested blocks crossed my
mind.  Maybe each paragraph could have a `nesting-level' text property
whose value is an integer.  For each nesting-level N, with N > 0, the
first preceeding block with a nesting level N - 1 is the immediate
parent.  I have no idea yet how, if at all, that would translate to
the XML info set, though.]


    Oliver
--
Oliver Scholz               1 Vendémiaire an 213 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

I split my answer to this mail in order to adress some issues
separately.

Richard Stallman <rms@gnu.org> writes:

[nested blocks]
> What does it *mean* to copy a character from inside environment
> `larum' which is inside environment `lirum' and insert it somewere
> else?  What should that character look like in its new location?

Two things could make sense here:

* Copy the properties of the /immediate/ containing block.

* Ignore the block formatting properties and copy only normal text
  properties.

I definitely prefer the second one.  I think it would be the Right
Thing.  If I copy text from a H1 paragraph and insert it into a H2
paragraph, then it should get all character formatting properties that
are specified at the paragraph level from the H2 environment.  But if
the text has /additional/ character formatting properties specified,
like it contains some italic words, those should be preserved.

[...]
>     <h1>Some <i>meaningless</i> heading</h1>
>
>     The <i> element maps directly to text properties, of course.  But the
>     h1 element both demands that its contents be rendered as a paragraph
>     (a block) /and/ specifies certain character formatting properties for
>     the whole of it, e.g. a large bold font.
>
>     When encoding a buffer, I need to identify the whole paragraph as
>     being of the type "h1".  I.e. I have to distinguish it from:
>
>     <p><font size=7><b>Some <i>meaningless</i> heading</font></p>
>
> Why do you have to distinguish them? 

It is about preserving the user's intent.

Word processors as well as the file formats used in word processing
typically provide several ways to apply character formatting
properties on text:

*  paragraph formatting stylesheets
   - RTF: \sN
   - HTML: block elements like h1, h2 ... 

*  character formatting stylesheets
   - RTF: \csN
   - HTML: inline elements like em

*  direct specification of character formatting properties
   - RTF: \fN, \fsN, \b ...
   - HTML: i, b, font ...

The first two provide an layer of indirection which allows to specify
the user's /semantical/ intent on the document text.  Some
users---well, /I/ for example---would prefer /not/ to work with direct
specification of formatting properties at all.

It is a matter of what is the intent that the user has expressed.  Did
she specify "I want this to be a top level headline" or did she
specify "I want this to be large, bold text"?

The difference will show up, when the document is transfered to
another rendering device or when the user changes her mind and changes
the stylesheet for "level 1 headlines".  We have to preserve that
intent of the user in the data structure.  That's why I introduced the
concept of the abstract document and distinguished it from the
appearance.  The abstract document is the aggregation of the user's
intent.

Specifying only the appearance ("This should be large, bold text") is
considered bad practice in word processing.  Some users do it this
way; but many, at least most people /I/ know, prefer stylesheets.  If
Emacs would fail to preserve the semantical intents, it would get a
very bad reputation as a word processor.  Even worse, we would have to
expect that sophisticated users would recommend /not/ to use Emacs in
document exchange.  This must not happen.  Emacs has the potential to
be much better than any existing word processor; I would be very sad
if it happens to become worse.

> Why wouldn't it work simply to put these properties on the whole
> text of the paragraph?  What aspect would work differently as a
> result of doing one or the other, and why is it better if the
> properties are attached to paragraphs?

When encoding the document, I have to determine the type of a
paragraph, so that the encoded document file conserves the user's
semantical intent.  I have to get that information from somewhere.

If we can guarantee, that text properties affecting the paragraph
/always/ cover the whole of text of a paragraph, then this o.k.  When
encoding, I first distuinguish the paragraph; then I look at the text
property.  Kim has hinted at some ways of guaranteeing this.  Offhand
I believe that this would work for non-nested paragraphs (blocks).  I
dislike that approach, though, partly because I don't trust its
robustness, partly because it does not scale to handle nested blocks.

This whole affair is partly an UI problem.  The functions that encode
the document must be able to unambigously determine the type of a
paragraph as well as its other features from the data structure.  But
also the user must get feedback on how her actions affected the
abstract document (as expressed in said data structure):

>     We have to deal with the case that a user deletes the hard newline (if
>     you evaluate the code above: just hit backspace).  Is the resulting
>     paragraph of type `h1' or of type `h2'?
>
> Why ask the question?  Why not just accept that it's a paragraph
> of partly h1 text and partly h2 text?

In HTML there is no such thing as a paragraph that is partly H1 and H2
text.  What you suggest would result in this:

<h1>lirum larum</h1><h2>lirum larum</h2>

Any user agent (web browser, another word processor) would render this
as two paragraphs (blocks).  But the user in Emacs saw it as a single
paragraph when she saved that document.  Due to the commands she has
issued (maybe accidentally) the data structure treats it as two
separate paragraphs and encodes it accordingly when writing to the
file; but the user does not get any visual feedback on this.  She will
be surprised.  If she knows that things like this could happen, she
could feel the urge to examine the encoded document file before she
transers it to somebody else.  Eventually she could even stop to use
the word processing facilities and edit the raw HTML from the
beginning; or use another word processor.

Of course, treating "h1" and "h2" always as character formatting types
only would avoid the "one paragraph that suddenly becomes two
paragraphs" effect:

<p><font size=7><b>lirum larum</b></font><font size=6><b>lirum larum</b><font></p>

But then we fail again to preserve any semantical intent.



    Oliver
-- 
Oliver Scholz               Jour de la Révolution de l'Année 212 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Stefan Monnier]

> `(text-property-any (point-min) (point-max) 'text-node TEXT-NODE)'
> This should be fast.

It's linear time in the size of your tree (aka in the number of
text-property-transitions).


        Stefan

Re: enriched-mode and switching major modes.

[Stefan Monnier]

> buffer rather than in a string and that they are /editable/.  And that
> question I can reduce to: how can I implement a tree-like data
> structure with text properties?

Without thinking too much about it, I'd say that overlays seem to be
a better fit: each node is an overlay, the node's attributes can be stored
in the overlay's properties, the text corresponding to a node can be stored
directly in the buffer and retrieved by (buffer-substring (overlay-start o)
(overlay-end o)), ...


        Stefan

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> buffer rather than in a string and that they are /editable/.  And that
>> question I can reduce to: how can I implement a tree-like data
>> structure with text properties?
>
> Without thinking too much about it, I'd say that overlays seem to be
> a better fit: each node is an overlay, the node's attributes can be stored
> in the overlay's properties, the text corresponding to a node can be stored
> directly in the buffer and retrieved by (buffer-substring (overlay-start o)
> (overlay-end o)), ...

I did not spend any thought about overlays, because I thought they
were too expensive?  The documents could be large and have many nodes.

Of course, if overlays work better: maybe they could be made more
efficient in the long run?

    Oliver
-- 
Oliver Scholz               1 Vendémiaire an 213 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Richard Stallman]

    I would expect Emacs to compute the numbers automatically, so that
    inserting an item into the middle would recompute the numbers.  To me,
    this means it makes no sense to make the numbers themselves accessible
    via Lisp.

Generating characters such as these numbers at display time would be
a fine method in some other program that does not have the Lisp level.
It would be absurd in Emacs.  These digits have to be in the buffer
when Lisp programs look at the buffer.

They could be updated automatically at certain times, normalized as it
were, but they must be real text.

    I would also expect the line spacing between the items to vary
    depending on the style sheet.  That is, Lisp shouldn't see two newline
    characters after "multi-line.", but just the end of the first item.

It would be useful to put two newlines there if that is possible.

      Does it really make sense to insert more spaces in front of "it"
    just because the user has changed the style sheet of the document to
    specify that enumerations should use bold numbers?

Yes.

      Additionally,
    depending on the exact text, making the numbers wider could mean that
    the "when" does not fit on the line anymore.  Making the newlines
    accessible via Lisp would mean that Lisp would suddenly see a newline
    in front of "when", instead of after "when".

Yes, just as if you were to type M-q.

Re: enriched-mode and switching major modes.

[Stefan Monnier]

> Of course, if overlays work better: maybe they could be made more
> efficient in the long run?

Better ignore such efficiency issues at first.  There's no fundamental
reasons why a facility along the lines of overlays can't be made efficient.


        Stefan

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> Cc: Oliver Scholz <epameinondas@gmx.de>,  boris@gnu.org,  eliz@gnu.org,
> 	  alex@emacswiki.org,  emacs-devel@gnu.org
> From: no-spam@cua.dk (Kim F. Storm)
> Date: Wed, 22 Sep 2004 09:44:19 +0200
> 
> I still think my idea to use "anonymous" windows as the base of a
> block display model would be a good solution

But a window is more than just a portion of the display; for example,
you cannot get out of a window without typing a special command, like
"C-x o".  Wouldn't it be hard to lift this kind of limitations?

Re: enriched-mode and switching major modes.

[Richard Stallman]

    > I am not sure how to represent nested blocks with per-character text
    > properties.  In principle we could have other kinds of data associated
    > with the text to supplement text properties, but what could we design
    > that would fit in properly with cut and paste?

    Perhaps such portions of text should only be cut-n-pasted as
    rectangles.

How would that help?

Re: enriched-mode and switching major modes.

[Richard Stallman]

    > Is it common practice to use "a caption for a row" in a document?

    Maybe not, but it shows the general concept of what is displayed in
    one table cell can be _completely_ unrelated to what is displayed in
    another table cell.

I understand the concept.  What I'm getting at is whether it is
important for Emacs to support such tables.  All else being equal, of
course we might as well support them.  But how important should that
be as a factor in difficult design decisions?

    Consider also the example that instead of AAA we have a tall image,
    and bbbb ccccc etc is text that is displayed next to that image like
    this:

Hmm, that is important.  But it might be easier to support this
with just images than to support this with editable text inside
the box.  The cursor would not have to be able to move thru the
image.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    [nested blocks]
    > What does it *mean* to copy a character from inside environment
    > `larum' which is inside environment `lirum' and insert it somewere
    > else?  What should that character look like in its new location?

    * Ignore the block formatting properties and copy only normal text
      properties.

In that case, you get a behavior that is equivalent to putting the
block properties into the buffer as markup.  If that's what we want,
I suggest we look at representing the information with markup
that consists of text in the buffer.

An implementation using text properties that state the paragraph
properties might also be possible, as you have been investigating.

    Word processors as well as the file formats used in word processing
    typically provide several ways to apply character formatting
    properties on text:

    *  paragraph formatting stylesheets
       - RTF: \sN
       - HTML: block elements like h1, h2 ... 

    *  character formatting stylesheets
       - RTF: \csN
       - HTML: inline elements like em

    *  direct specification of character formatting properties
       - RTF: \fN, \fsN, \b ...
       - HTML: i, b, font ...

The latter two can be implemented easily with text properties.  In
fact, Emacs supports them both now with text properties.  For the
present discussion, we can treat them as two variants of the same
thing.

So that is one issue we need not spend time on.

    > Why ask the question?  Why not just accept that it's a paragraph
    > of partly h1 text and partly h2 text?

    In HTML there is no such thing as a paragraph that is partly H1 and H2
    text.

That doesn't mean it can't exist in an Emacs buffer.  Our imaginations
need not be limited by what HTML can comprehend ;-).

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: alkibiades@gmx.de, boris@gnu.org, alex@emacswiki.org,
> 	emacs-devel@gnu.org
> Date: Wed, 22 Sep 2004 14:20:18 -0400
> 
>     Perhaps such portions of text should only be cut-n-pasted as
>     rectangles.
> 
> How would that help?

It might help if the copied text never crosses the boundaries of
styles, i.e. if all of the copied text has the same style properties.
Forcing rectangle copy could help us make sure such crossing never
happens.

Re: enriched-mode and switching major modes.

[Kim F. Storm]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Cc: Oliver Scholz <epameinondas@gmx.de>,  boris@gnu.org,  eliz@gnu.org,
>> 	  alex@emacswiki.org,  emacs-devel@gnu.org
>> From: no-spam@cua.dk (Kim F. Storm)
>> Date: Wed, 22 Sep 2004 09:44:19 +0200
>> 
>> I still think my idea to use "anonymous" windows as the base of a
>> block display model would be a good solution
>
> But a window is more than just a portion of the display; for example,
> you cannot get out of a window without typing a special command, like
> "C-x o".  Wouldn't it be hard to lift this kind of limitations?

One fairly trivial solution:

We could add before-beginning-of-buffer-hook and
after-end-of-buffer-hook which are hooks which are run when point is
moved backwards when (bolp) or forwards at (eobp).  These could be
setup to jump to the proper position in the "parent window".

There could also be explicit bindings for this, e.g. M-left and M-right.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

>     > Is it common practice to use "a caption for a row" in a document?
>
>     Maybe not, but it shows the general concept of what is displayed in
>     one table cell can be _completely_ unrelated to what is displayed in
>     another table cell.
>
> I understand the concept.  What I'm getting at is whether it is
> important for Emacs to support such tables.  All else being equal, of
> course we might as well support them.  But how important should that
> be as a factor in difficult design decisions?

It depends on which level of WP functionality (and rendering of e.g. HTML)
we want to support in emacs.

Once 21.4 is out, we can think more about ways to do this...

>
>     Consider also the example that instead of AAA we have a tall
>image, and bbbb ccccc etc is text that is displayed next to that
>image like this:
>
> Hmm, that is important.  But it might be easier to support this
> with just images than to support this with editable text inside
> the box.  The cursor would not have to be able to move thru the
> image.
>

For images, we can already achieve this with image slices, but that's
a work-around, not a real solution.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Luc Teirlinck]

Oliver Scholz wrote:

   I did not spend any thought about overlays, because I thought they
   were too expensive?  The documents could be large and have many nodes.

   Of course, if overlays work better: maybe they could be made more
   efficient in the long run?

Note that, unless you take special steps, there are important
user-visible differences between using text properties and overlays
(for better or for worse, depending on the situation).

Text properties are part of the text, overlays are not.  Hence
changing text properties marks the buffer modified, playing around
with overlays not.  Of course, this has numerous important consequences
in terms of auto-save files, asking or not asking the user to save the
file, auto reverting and so on...

It has been a while ago that I played around extensively with many
overlays in huge buffers, but I found that functions that do that
could be made incredibly faster by executing:

(overlay-recenter (point-max))

before starting to play around with overlays.  I do not know whether
this is still the case, since the overlay code could have changed in
the meantime.

Sincerely,

Luc.

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> Cc: epameinondas@gmx.de, boris@gnu.org, alex@emacswiki.org,
> 	emacs-devel@gnu.org
> From: storm@cua.dk (Kim F. Storm)
> Date: Wed, 22 Sep 2004 23:53:29 +0200
> 
> We could add before-beginning-of-buffer-hook and
> after-end-of-buffer-hook which are hooks which are run when point is
> moved backwards when (bolp) or forwards at (eobp).  These could be
> setup to jump to the proper position in the "parent window".

What about "C-a" and "C-e"? shouldn't they go to the left and right
margin of the surrounding parent window?

What about "C-x o"? it will allow jumping into what the user thinks is
a part of the current window, which is confusing.

I can see many similar complications.

Re: enriched-mode and switching major modes.

[Kim F. Storm]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Cc: epameinondas@gmx.de, boris@gnu.org, alex@emacswiki.org,
>> 	emacs-devel@gnu.org
>> From: storm@cua.dk (Kim F. Storm)
>> Date: Wed, 22 Sep 2004 23:53:29 +0200
>> 
>> We could add before-beginning-of-buffer-hook and
>> after-end-of-buffer-hook which are hooks which are run when point is
>> moved backwards when (bolp) or forwards at (eobp).  These could be
>> setup to jump to the proper position in the "parent window".
>
> What about "C-a" and "C-e"? shouldn't they go to the left and right
> margin of the surrounding parent window?

Perhaps -- it depends on how things a layed out.  E.g if I have
text and image (IIII) like this:

  abc      IIII
  defgh    IIII
  jkl      IIII

and cursor is on `f' I would probably prefer if C-e moved cursor to
the position after the 'h' rather than to the right edge of the
IIIImage.

>
> What about "C-x o"? it will allow jumping into what the user thinks is
> a part of the current window, which is confusing.

C-x o (and C-x 4 f etc) should ignore anonymous windows.

I think most of the relevant commands use a small number of
functions to select the target window -- we only need to
fix those functions, not each command.

>
> I can see many similar complications.

Sure, it need to be thought about before implementing anything.

In a block model used for WP, there are probably lots of similar
problems that we would have to solve.  I don't think that can
be avoided no matter which approach we choose.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: enriched-mode and switching major modes.

[Richard Stallman]

    Without thinking too much about it, I'd say that overlays seem to be
    a better fit: each node is an overlay, the node's attributes can be stored
    in the overlay's properties, the text corresponding to a node can be stored
    directly in the buffer and retrieved by (buffer-substring (overlay-start o)
    (overlay-end o)), ...

Overlays are no good for this because they will be lost completely
if you cut and paste.  If you copy the text of a buffer
to another buffer, the overlays won't come along.  (That is the purpose
of overlays.)

This information has to be *part of the text*.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    I wonder whether the value of the `category' property should be allowed
    to be a list of atoms somewhat like with `invisible'.

If this is really usable for nested blocks, I see no difficulty in it.

Re: enriched-mode and switching major modes.

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     Without thinking too much about it, I'd say that overlays seem
>     to be a better fit: each node is an overlay, the node's
>     attributes can be stored in the overlay's properties, the text
>     corresponding to a node can be stored directly in the buffer and
>     retrieved by (buffer-substring (overlay-start o) (overlay-end
>     o)), ...
>
> Overlays are no good for this because they will be lost completely
> if you cut and paste.  If you copy the text of a buffer to another
> buffer, the overlays won't come along.  (That is the purpose of
> overlays.)
>
> This information has to be *part of the text*.

I think that where we have structural information particular to the
buffer (like "this is a section heading"), it could be preferable if
this information got lost on copy&paste.  And also if I have some
indentation or formatting that is active due to the current document
structure, I would not want to get this pasted as well.  At least if
the structure is not copied completely.

OTOH, _if_ some entity is copied completely, then it would make sense
to carry over its structure.  So the task of the yank handler would be
to rip out everything which is not complete, but leave things that
_are_.

Ok, text properties after all, for most things.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: enriched-mode and switching major modes.

[Stefan]

>     Without thinking too much about it, I'd say that overlays seem to be
>     a better fit: each node is an overlay, the node's attributes can be stored
>     in the overlay's properties, the text corresponding to a node can be stored
>     directly in the buffer and retrieved by (buffer-substring (overlay-start o)
>     (overlay-end o)), ...

> Overlays are no good for this because they will be lost completely
> if you cut and paste.  If you copy the text of a buffer
> to another buffer, the overlays won't come along.  (That is the purpose
> of overlays.)

> This information has to be *part of the text*.

XEmacs's version of overlays (called extents) can be part of the text
(which is a property they call `duplicable').  We could do the same.


        Stefan

Re: enriched-mode and switching major modes.

[David Kastrup]

Stefan <monnier@iro.umontreal.ca> writes:

>>     Without thinking too much about it, I'd say that overlays seem
>>     to be a better fit: each node is an overlay, the node's
>>     attributes can be stored in the overlay's properties, the text
>>     corresponding to a node can be stored directly in the buffer
>>     and retrieved by (buffer-substring (overlay-start o)
>>     (overlay-end o)), ...
>
>> Overlays are no good for this because they will be lost completely
>> if you cut and paste.  If you copy the text of a buffer to another
>> buffer, the overlays won't come along.  (That is the purpose of
>> overlays.)
>
>> This information has to be *part of the text*.
>
> XEmacs's version of overlays (called extents) can be part of the text
> (which is a property they call `duplicable').  We could do the same.

Extents are actually used for implementing text properties with
XEmacs, if I understand their stuff correctly.  They are just extents
with a few special properties (like duplicable).  So in effect, XEmacs
does not force you to decide whether you want to use one or the other:
every feature that would be different can be chosen separately with an
appropriate property.

This simplifies implementation in some circumstances.  The main
problem I see with the XEmacs model is that they have not bothered
overly much to add useful abstractions (like overlays and text
properties are) on top of the low-level basics.  For example, for
getting some properties or overlays at point, you basically have to
use the map-extents function, for which I'll append at the end of the
posting.

There are not really convenience functions like "overlays-at" and
similar, since you supposedly can do everything with the likes of
map-extents.  For keeping the number of internal data structures
manageable, the idea is probably not bad, but the practice turns out
to be a bit of a nuisance for the typical application programmer.  Not
because raw power would be missing, but because it turns out to be
raw: you have to focus on many details in order to get something that
does just what you want.

The actual amount of C code underlying the implementation of extents
could well be less than the combined code for text properties and
overlays in Emacs, and maybe can serve more purposes.  Still, the sort
of do-everything-for-a-reasonably-complicated-set-of-arguments
interface is something that could certainly make use of some more
abstractions in the form of Lisp functions accessing it.

David


`map-extents' is a built-in function
(map-extents FUNCTION &optional OBJECT FROM TO MAPARG FLAGS PROPERTY VALUE)

Documentation:
Map FUNCTION over the extents which overlap a region in OBJECT.
OBJECT is normally a buffer or string but could be an extent (see below).
The region is normally bounded by [FROM, TO) (i.e. the beginning of the
region is closed and the end of the region is open), but this can be
changed with the FLAGS argument (see below for a complete discussion).

FUNCTION is called with the arguments (extent, MAPARG).  The arguments
OBJECT, FROM, TO, MAPARG, and FLAGS are all optional and default to
the current buffer, the beginning of OBJECT, the end of OBJECT, nil,
and nil, respectively.  `map-extents' returns the first non-nil result
produced by FUNCTION, and no more calls to FUNCTION are made after it
returns non-nil.

If OBJECT is an extent, FROM and TO default to the extent's endpoints,
and the mapping omits that extent and its predecessors.  This feature
supports restarting a loop based on `map-extents'.  Note: OBJECT must
be attached to a buffer or string, and the mapping is done over that
buffer or string.

An extent overlaps the region if there is any point in the extent that is
also in the region. (For the purpose of overlap, zero-length extents and
regions are treated as closed on both ends regardless of their endpoints'
specified open/closedness.) Note that the endpoints of an extent or region
are considered to be in that extent or region if and only if the
corresponding end is closed.  For example, the extent [5,7] overlaps the
region [2,5] because 5 is in both the extent and the region.  However, (5,7]
does not overlap [2,5] because 5 is not in the extent, and neither [5,7] nor
(5,7] overlaps the region [2,5) because 5 is not in the region.

The optional FLAGS can be a symbol or a list of one or more symbols,
modifying the behavior of `map-extents'.  Allowed symbols are:

end-closed		The region's end is closed.

start-open		The region's start is open.

all-extents-closed	Treat all extents as closed on both ends for the
			purpose of determining whether they overlap the
			region, irrespective of their actual open- or
			closedness.
all-extents-open	Treat all extents as open on both ends.
all-extents-closed-open	Treat all extents as start-closed, end-open.
all-extents-open-closed	Treat all extents as start-open, end-closed.

start-in-region		In addition to the above conditions for extent
			overlap, the extent's start position must lie within
			the specified region.  Note that, for this
			condition, open start positions are treated as if
			0.5 was added to the endpoint's value, and open
			end positions are treated as if 0.5 was subtracted
			from the endpoint's value.
end-in-region		The extent's end position must lie within the
			region.
start-and-end-in-region	Both the extent's start and end positions must lie
			within the region.
start-or-end-in-region	Either the extent's start or end position must lie
			within the region.

negate-in-region	The condition specified by a `*-in-region' flag
			must NOT hold for the extent to be considered.


At most one of `all-extents-closed', `all-extents-open',
`all-extents-closed-open', and `all-extents-open-closed' may be specified.

At most one of `start-in-region', `end-in-region',
`start-and-end-in-region', and `start-or-end-in-region' may be specified.

If optional arg PROPERTY is non-nil, only extents with that property set
on them will be visited.  If optional arg VALUE is non-nil, only extents
whose value for that property is `eq' to VALUE will be visited.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Stefan <monnier@iro.umontreal.ca> writes:

[overlays for implementing a document tree]
>
>> Overlays are no good for this because they will be lost completely
>> if you cut and paste.  If you copy the text of a buffer
>> to another buffer, the overlays won't come along.  (That is the purpose
>> of overlays.)
>
>> This information has to be *part of the text*.

I agree, in general, for conceptual reasons. Yet, overlays in many
ways fit the job description: they are first class Lisp objects
specifying properties for a range of text.  For implementing a tree
data structure, they don't fit in the one respect that overlays on the
same level (depth) of the tree may overlap.  But maybe that won't be a
problem.

What I also like is that I can delay the decision whether to find the
children or the parent of a node in the buffer each time I want to
access them.  Or whether I keep references to both on the node.
Overlays allow for both.

And finally, they allow me to store character formatting properties on
tree nodes in a way that Emacs will automagically resolve them for me.

That text is copied without the overlays is not that much of a
problem.  With text properties I'd have to remove those applying to a
block only anyways.  Thus for overlays it is just the inverse. On the
contrary, if in doubt, the lesser evil is to copy the pure characters
without properties.

> XEmacs's version of overlays (called extents) can be part of the text
> (which is a property they call `duplicable').  We could do the same.

After spending some more thought on it, I think I can hide the actual
implementation of the abstract tree-like data structure behind an
interface. Thus we can delay any decision in this point.  There won't
happen anything before the next release, anyways.  For now I think,
I am going to work with overlays.


                        implementation of the
                        data structure
                                ^
 document                       v
 structure              Specification of the abstract
    +-----------------> tree data structure and      ----+
    |                   interface to it                  |
    |                           ^                        |
    |                           |                        |
    |                           |                        v
file decoding --------> applying stylesheets   resolving formatting
               formatt.                        properties for visual
               inform.                         appearance
                                                 - character formatting
                                                 - whitespace formatting


    Oliver
--
Oliver Scholz               2 Vendémiaire an 213 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    It might help if the copied text never crosses the boundaries of
    styles, i.e. if all of the copied text has the same style properties.
    Forcing rectangle copy could help us make sure such crossing never
    happens.

I don't think that limiting to rectangle copy would have that effect.
It is easy to select a rectangle that crosses paragraphs.

Re: enriched-mode and switching major modes.

[Richard Stallman]

    I think that where we have structural information particular to the
    buffer (like "this is a section heading"), it could be preferable if
    this information got lost on copy&paste.

This must be a misunderstanding of the issue.  Consider C-x h C-w C-x
u.  Should it forget the structure of your document?  Clearly not.

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Oliver Scholz <alkibiades@gmx.de>
> Date: Wed, 22 Sep 2004 12:01:27 +0200
> Cc: boris@gnu.org, eliz@gnu.org, alex@emacswiki.org, emacs-devel@gnu.org
> 
> For simplicity, I focus on elements and character data here and talk
> about them as "nodes" in a tree; thus we have "element nodes" and
> "text nodes".  An XHTML fragment like
> 
> <h1>Some <em>meaningless</em> text</h1>
> 
> Would be regarded as a `h1' element node which has three children: a
> text node "Some ", an element node `em' (which has itself a text node
> as its single child) an another text node " text".

I'm not sure I understand why you want a tree-like representation for
that.  Is it only so you can find what you call ``the immeditate
parent of the current node''?

Perhaps it would be better to step back for a moment and describe the
operations you will need to perform on fragments of the document text,
and only _then_ think about an appropriate data structure.  The fact
that the XML people came up with a tree does not mean you should
follow suit; there might be other appropriate data structures.  For
example, why not a list (which is a native data structure in ELisp)?

Re: enriched-mode and switching major modes.

[Oliver Scholz]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> From: Oliver Scholz <alkibiades@gmx.de>
>> Date: Wed, 22 Sep 2004 12:01:27 +0200
>> Cc: boris@gnu.org, eliz@gnu.org, alex@emacswiki.org, emacs-devel@gnu.org
>> 
>> For simplicity, I focus on elements and character data here and talk
>> about them as "nodes" in a tree; thus we have "element nodes" and
>> "text nodes".  An XHTML fragment like
>> 
>> <h1>Some <em>meaningless</em> text</h1>
>> 
>> Would be regarded as a `h1' element node which has three children: a
>> text node "Some ", an element node `em' (which has itself a text node
>> as its single child) an another text node " text".
>
> I'm not sure I understand why you want a tree-like representation for
> that.  Is it only so you can find what you call ``the immeditate
> parent of the current node''?

Since I don't want to render a given XML document /only/, but to make
it /editable/ also, I have to preserve the structure/the information
contained in the original document.

There is no requirement that the data structure is a tree in a strict
sense immediately (in fact, the XML information set spec explicitely
states that this is not required).  But it must be possible to map a
tree into that data structure and to map that data structure into a
tree; so that the original tree is regenerated after doing both.  The
data structure that I presented as #2 is actually not a tree in itself
(no link to the parents and children on a node); but I can compute
tree-like relations from it.

> Perhaps it would be better to step back for a moment and describe the
> operations you will need to perform on fragments of the document text,
> and only _then_ think about an appropriate data structure. 

I am not sure that I understand what you mean here.  First and
foremost I want that data structure to preserve the information
contained in a specific file format (here: XML).  

Then, to implement CSS I need an interface to whatever data structure
I use that presents it as a tree.

> The fact that the XML people came up with a tree does not mean you
> should follow suit; there might be other appropriate data
> structures.  For example, why not a list (which is a native data
> structure in ELisp)?

Lists are fine in general:

'(h1 "some " (em "meaningless") " text")

SXML uses lists in a similar way to represent the XML information
set. <URL: http://okmij.org/ftp/Scheme/SXML.html>
Emacs Lisp XML parsers that generate similar lists include:
nxml-parse.el, xml.el and xml-parse.el.

That nodes have no reference to their parent in a non-circular list is
not much of a problem, because for a given node, I can find its parent
by scanning the list.  So the information is preserved.

The solution I presented as #1 could be implemented this way by
representing text nodes as a unique Lisp object that is put into a
`text-node' text property.  In fact, I intended for a long time to do
it exactly like this (using SXML).  But I distrust the robustness of
this solution.  (After some thought I also realized that using SXML is
a bit pointless, but that is another issue, probably not relevant
here.)

    Oliver
-- 
Oliver Scholz               3 Vendémiaire an 213 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Richard Stallman]

    XEmacs's version of overlays (called extents) can be part of the text
    (which is a property they call `duplicable').  We could do the same.

We already did.  That's what text properties are.

The reason I decided that text properties should not belong to extents
which are objects is that there is no consistent way to preserve such
identities through cutting and pasting.  Every way of doing it, that I
could see, gives inconsistent behavior in some cases.  That I do not
want.

If you can design a system to attach text properties to extent objects
that gives consistent behavior, I would not object to implementing it.

Re: enriched-mode and switching major modes.

[Stefan]

>     XEmacs's version of overlays (called extents) can be part of the text
>     (which is a property they call `duplicable').  We could do the same.

> We already did.  That's what text properties are.

It's not the same because:

1 - it does not preserve the identity: the text-propeties belong to
    characters, not to some external object.  I.e. I can't do the equivalent
    of overlays-at, all I can do is to get all the text-properties whithout
    being able to distinguish "where they come from" and which ones were
    added together, ...

2 - I can't (in general) add a text-property to a given piece of text, and
    hours later say "what are the current start and end points of this
    property I added".

3 - An extent/overlay is limited to a single contiguous range, so when you add
    an extent/overlay with a `field' property, you can be sure that this
    field will stay as a single field.  With text-propertis OTOH, if text is
    inserted in the middle of the field without inheriting properties, the
    field could be split into two.

For Olivier's purpose, the model offered by overlays/extents is much easier
to use.

> The reason I decided that text properties should not belong to extents
> which are objects is that there is no consistent way to preserve such
> identities through cutting and pasting.  Every way of doing it, that
> I could see, gives inconsistent behavior in some cases.  That I do
> not want.

Indeed, it's tricky business.

> If you can design a system to attach text properties to extent objects
> that gives consistent behavior, I would not object to implementing it.

I think the semantics of `duplicable' extents is clear when the ducplicated
text includes the whole extent, the problems only come up when trying to
duplicate part of the text of an extent.  Since the required behavior
depends on the extent, we can't handle it automatically, and I just the
`duplicate' property should thus be a function which will handle those
"corner" cases.  I.e. it doesn't solve the problem, but just pushes it to
the package that decided to use a duplicable extent.

My guess is that in practice, if a "partial copy" should duplicate the
extent, text-properties will be a better interface, so in most cases the
`duplicate' function will be set to something like `ignore'.


        Stefan

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Oliver Scholz <alkibiades@gmx.de> writes:

[...]
> I am not sure that I understand what you mean here.  First and
> foremost I want that data structure to preserve the information
> contained in a specific file format (here: XML).  
>
> Then, to implement CSS I need an interface to whatever data structure
> I use that presents it as a tree.
[...]

On a second thought: this latter is---strictly speaking---not quite
true.  If I have a data structure that is not a tree, but for whose
parts I can compute tree-like relations on demand (that is thus an
instance of the XML info set), then there is no apriori requirement to
always work with an interface that presents that data structure as a
tree.  Thus it might be possible to implement an engine applying and
querying formatting properties, that works by interfacing to the
underlying data structure (like solution #2, for instance) directly
(though I have no idea yet how).

So, the exact specifica of the underlying data structure and the
interface to it are not as irrelevant to the rest of the architecture,
efficiency-wise, as it might have seemed in one of my last posting.
Maybe that is what you meant.

But I have to start somewhere.  The tree interface to this kind of
documents is well known and tested in practice.  Almost every
specification talks about trees.  So if there are no principal
objections to improving Emacs to better work with trees-in-a-buffer in
the long run, I think that I fare better by assuming a tree-interface.


    Oliver
-- 
Oliver Scholz               3 Vendémiaire an 213 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Alex Schroeder]

Oliver Scholz <epameinondas@gmx.de> writes:

> But I have to start somewhere.  The tree interface to this kind of
> documents is well known and tested in practice.  Almost every
> specification talks about trees.  So if there are no principal
> objections to improving Emacs to better work with trees-in-a-buffer in
> the long run, I think that I fare better by assuming a tree-interface.

One of the problems you described was keeping a tree and the buffer in
sync.  If you had a way of regenerating the tree based on the text in
the buffer when saving a file, would that not be enough?

Alex.
-- 
.O.  http://www.emacswiki.org/alex/
..O  Schroeder's fifth law:
OOO  Never accept more work than you can handle in one night of hacking.

Re: enriched-mode and switching major modes.

[Oliver Scholz]

Alex Schroeder <alex@emacswiki.org> writes:

> Oliver Scholz <epameinondas@gmx.de> writes:
>
>> But I have to start somewhere.  The tree interface to this kind of
>> documents is well known and tested in practice.  Almost every
>> specification talks about trees.  So if there are no principal
>> objections to improving Emacs to better work with trees-in-a-buffer in
>> the long run, I think that I fare better by assuming a tree-interface.
>
> One of the problems you described was keeping a tree and the buffer in
> sync.  If you had a way of regenerating the tree based on the text in
> the buffer when saving a file, would that not be enough?

Yes, it could be enough.  It just makes implementing everything else
much more difficult and non-intuitive.  Nearly every specification
that could be directly or indirectly important (CSS, XPath, XSL even,
XForms, maybe various schemas, maybe others) describes the information
set as a tree.  When implementing I would have to translate that to
the custom data structure for reasons of efficiency.  And first I
would have to find that data structure---maybe #2 is it, maybe not.
(True, it is wrong to worry about efficiency prematurely, but since
this application must respond to user actions interactively, it must
not make efficiency impossible by design.)

Thus, if the outcome of this thread is that using a tree interface is
o.k., because later Emacs can be enhanced to deal with
trees-in-a-buffer in a way that avoids the syncing problem, then this
is like christmas for me.  (It is not important then, whether the
actual implementation hidden behind the interface relies on improved
overlays, extents or even an entirely new data structure.)

    Oliver
-- 
Oliver Scholz               3 Vendémiaire an 213 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: enriched-mode and switching major modes.

[Richard Stallman]

    1 - it does not preserve the identity: the text-propeties belong to
	characters, not to some external object.

I already explained why I made that change.

    I think the semantics of `duplicable' extents is clear when the ducplicated
    text includes the whole extent, the problems only come up when trying to
    duplicate part of the text of an extent.

Yes, that's right.

      Since the required behavior
    depends on the extent, we can't handle it automatically, and I just the
    `duplicate' property should thus be a function which will handle those
    "corner" cases.

You're just moving the unsolved problem inside the `duplicate'
property.  Nobody has ever proposed a good solution to this that I
have seen, so I think none exists.  If you can present one, that
will change my mind.  However, I will not agree to simply sweep
the problem under a hook and presume someone will solve it later.

Re: enriched-mode and switching major modes.

[Eli Zaretskii]

> From: Oliver Scholz <epameinondas@gmx.de>
> Date: Fri, 24 Sep 2004 17:51:56 +0200
> Cc: boris@gnu.org, emacs-devel@gnu.org, alex@emacswiki.org
> 
> But I have to start somewhere.  The tree interface to this kind of
> documents is well known and tested in practice.  Almost every
> specification talks about trees.  So if there are no principal
> objections to improving Emacs to better work with trees-in-a-buffer in
> the long run, I think that I fare better by assuming a tree-interface.

I simply didn't understand why you converged on the tree right from
the beginning.  I thought that the specification of the data structure
might benefit from some hind-sight discussion, since (at least in my
experience) trees are not very often used in ELisp code, oprobably for
a good reason.