Re: Saving markup formats

Re: Saving markup formats

[Reiner Steib]

[ I'd suggest to move this discussion to emacs-devel ]

On Sat, Jun 16 2007, Richard Stallman wrote:

[ Juri Linkov: ]
>     What are the most preferable formats to save markup?  One variant is
>     Enriched text.  It was designed for using in e-mail, but actually nobody
>     uses it nowadays.  Most people prefer HTML in e-mail as a replacement of
>     plain text.
>
> HTML would be useful.  RTF would be useful.  XML would be useful.
> The old Word format would be useful, at least to be able to read it.
> If you implement any of these, it will be a nice contribution.
> If you implement more than one, even better.

Oliver Scholz <epameinondas@gmx.de> (Cc-ed) has done some work on an
RTF reader for Emacs a couple of years ago.  Maybe his work is useful
in the current context.  Oliver has signed papers for Emacs (past and
future changes) and I'm quite sure he'd assign this code as well.

Here's the summary from the project page:

,----[ http://savannah.nongnu.org/projects/emacs-rtf/ ]
| This package aims to implement the "Rich Text Format" (RTF), version
| 1.5, as a file format for GNU Emacs. It also provides the necessary
| framework to edit RTF documents in an Emacs buffer. It is thus
| intended to enhance the word processing capabilities of the Emacs
| environment. The goal is to enable a user to edit RTF documents
| without launching a non-Emacs application like OpenOffice or abiword.
| 
| This package is in a very early state of development. Currently only
| the reader is in the work, and only the main structure of it is
| implemented. Some RTF documents are already rendered correctly,
| though.
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Saving markup formats

[Juri Linkov]

> [ I'd suggest to move this discussion to emacs-devel ]
> On Sat, Jun 16 2007, Richard Stallman wrote:
> [ Juri Linkov: ]
>>     What are the most preferable formats to save markup?  One variant is
>>     Enriched text.  It was designed for using in e-mail, but actually nobody
>>     uses it nowadays.  Most people prefer HTML in e-mail as a replacement of
>>     plain text.
>>
>> HTML would be useful.  RTF would be useful.  XML would be useful.
>> The old Word format would be useful, at least to be able to read it.
>> If you implement any of these, it will be a nice contribution.
>> If you implement more than one, even better.
>
> Oliver Scholz <epameinondas@gmx.de> (Cc-ed) has done some work on an
> RTF reader for Emacs a couple of years ago.  Maybe his work is useful
> in the current context.  Oliver has signed papers for Emacs (past and
> future changes) and I'm quite sure he'd assign this code as well.

I suppose the most important question is not whether Emacs can read/write
files in some particular format, but rather a question whether Emacs can
render format elements correctly on screen.  That's whether Emacs is ready
to be a WYSIWYG editor with all its required features or not.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Saving markup formats

[Nic James Ferrier]

Juri Linkov <juri@jurta.org> writes:

> I suppose the most important question is not whether Emacs can read/write
> files in some particular format, but rather a question whether Emacs can
> render format elements correctly on screen.  That's whether Emacs is ready
> to be a WYSIWYG editor with all its required features or not.

I certainly don't care about wysiwyg. I'd just like to have good
editing support for documents written in word or openoffice.

A good embedded XML parser would be needed. I've thought about taking
time out to add libxml2 support natively to emacs. That would do the
job very well.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Lennart Borgman (gmail)]

Nic James Ferrier wrote:
> Juri Linkov <juri@jurta.org> writes:
> 
>> I suppose the most important question is not whether Emacs can read/write
>> files in some particular format, but rather a question whether Emacs can
>> render format elements correctly on screen.  That's whether Emacs is ready
>> to be a WYSIWYG editor with all its required features or not.
> 
> I certainly don't care about wysiwyg. I'd just like to have good
> editing support for documents written in word or openoffice.
> 
> A good embedded XML parser would be needed. I've thought about taking
> time out to add libxml2 support natively to emacs. That would do the
> job very well.

Did you try nxml-mode?

Re: Saving markup formats

[Juri Linkov]

>> A good embedded XML parser would be needed. I've thought about taking
>> time out to add libxml2 support natively to emacs. That would do the
>> job very well.
>
> Did you try nxml-mode?

Do you mean `nxml-parse-file'?  I find it more restrictive than
`xml-parse-file' from xml.el.  It fails on more XML files.

Do you know a tidy-like XML parser for Emacs that would correct
not well-formed XML and not choke on them?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Saving markup formats

[Lennart Borgman (gmail)]

Juri Linkov wrote:
>>> A good embedded XML parser would be needed. I've thought about taking
>>> time out to add libxml2 support natively to emacs. That would do the
>>> job very well.
>> Did you try nxml-mode?
> 
> Do you mean `nxml-parse-file'?  I find it more restrictive than
> `xml-parse-file' from xml.el.  It fails on more XML files.

No, I meant using nxml-mode. It parses the buffer as you type.

But maybe that does not fit what you are searching for.

> Do you know a tidy-like XML parser for Emacs that would correct
> not well-formed XML and not choke on them?

Sorry, no. But nxml-mode has the ability to continue parsing after an 
error. It just marks the errors (with red underlines) in the buffer 
however, it does not correct them.

That parser, an rng-parser, is very tightly coupled with nxml-mode 
however. I suggested earlier to break it up to make it easier to handle 
multiple modes.

Re: Saving markup formats

[Nic James Ferrier]

Juri Linkov <juri@jurta.org> writes:

>>> A good embedded XML parser would be needed. I've thought about taking
>>> time out to add libxml2 support natively to emacs. That would do the
>>> job very well.
>>
>> Did you try nxml-mode?
>
> Do you mean `nxml-parse-file'?  I find it more restrictive than
> `xml-parse-file' from xml.el.  It fails on more XML files.
>
> Do you know a tidy-like XML parser for Emacs that would correct
> not well-formed XML and not choke on them?

This is the thing isn't it? Emacs has lots of good elisp based XML
support but none of it quite cuts the mustard for all the different
things one might do with XML.

nxml is a very good editor... but not so good for DOM programming.

xml.el has *some* good DOM stuff, but it's parser is a little weak and
there's poor error support.

Binding a really good XML library into Emacs would solve a bunch of
low level problems. It would mean that we could reasonably write
programs to manipulate a whole host of different documents and data
formats. It would also mean that nxml-mode type functionality would be
much easier to do.

In my view what emacs needs in the word processing arena is the
ability to read and write a lot of file formats (which are often XML
based now) and manipulate those DOMs in a way that is consistent with
the native editor.

For example, I don't care if Emacs renders an OO document as something
like an nxml document as long as I have ways of quickly adding a list
item, making a new paragraph, inserting a formula into a table, etc...

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Lennart Borgman (gmail)]

Nic James Ferrier wrote:
> For example, I don't care if Emacs renders an OO document as something
> like an nxml document as long as I have ways of quickly adding a list
> item, making a new paragraph, inserting a formula into a table, etc...

Using the completion hook in nxml-mode such things can rather easily be 
done today. (See nxhtml-mode for an example of some little things of 
this kind.)

Re: Saving markup formats

[Nic James Ferrier]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> Nic James Ferrier wrote:
>> For example, I don't care if Emacs renders an OO document as something
>> like an nxml document as long as I have ways of quickly adding a list
>> item, making a new paragraph, inserting a formula into a table, etc...
>
> Using the completion hook in nxml-mode such things can rather easily be 
> done today. (See nxhtml-mode for an example of some little things of 
> this kind.)

Only when the specification of the action is available in the schema.

nxml-mode doesn't have DOM manipulation tools, does it?

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Lennart Borgman (gmail)]

Nic James Ferrier wrote:
> "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:
> 
>> Nic James Ferrier wrote:
>>> For example, I don't care if Emacs renders an OO document as something
>>> like an nxml document as long as I have ways of quickly adding a list
>>> item, making a new paragraph, inserting a formula into a table, etc...
>> Using the completion hook in nxml-mode such things can rather easily be 
>> done today. (See nxhtml-mode for an example of some little things of 
>> this kind.)
> 
> Only when the specification of the action is available in the schema.

Eh, yes. The result should fit into the schema, of course.

But nothing prevents you from doing more complex things. Insert first 
result, check completion again etc.

> nxml-mode doesn't have DOM manipulation tools, does it?

We might be miscommunicating, but here is a attempt to answer:

- See above first.
- You have primitives for moving to parent, next/previous (paired) 
element, first child.

Re: Saving markup formats

[Nic James Ferrier]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

>> Only when the specification of the action is available in the schema.
>
> Eh, yes. The result should fit into the schema, of course.

It's not just that. A word processor might have non-schema defined
constraints on a document.


> But nothing prevents you from doing more complex things. Insert first 
> result, check completion again etc.

Sure. Maybe that is more powerful than I'm giving it credit for.

But it's too difficult because it's not a DOM.

If it weren't too difficult people (me?) would have already done a lot
of this stuff.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Lennart Borgman (gmail)]

Nic James Ferrier wrote:
> "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:
> 
>>> Only when the specification of the action is available in the schema.
>> Eh, yes. The result should fit into the schema, of course.
> 
> It's not just that. A word processor might have non-schema defined
> constraints on a document.

How is that? Do you mean that a document following the schema might not 
be valid as a word processor document?

That is clearly a messy state, isn't it? What about standardization 
committées view of this?

>> But nothing prevents you from doing more complex things. Insert first 
>> result, check completion again etc.
> 
> Sure. Maybe that is more powerful than I'm giving it credit for.
> 
> But it's too difficult because it's not a DOM.
> 
> If it weren't too difficult people (me?) would have already done a lot
> of this stuff.

Could you give me a more concrete example of what you want to do? (Or 
did you give that before?)

Re: Saving markup formats

[Nic James Ferrier]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> How is that? Do you mean that a document following the schema might not 
> be valid as a word processor document?

No. I mean that a schema might not express totally the constraints
that a word processor requires.

Actually, I think this is quite common.

OpenOffice, last time I mucked about with the files, seemed to have
some funny constraints about where the style information could go. It
seemed like certain styles were possible in the document, but others
weren't. The schema didn't seem to describe that stuff, it was just
abritary constraints on top of what the schema specified.


> Could you give me a more concrete example of what you want to do? (Or 
> did you give that before?)

I am talking about editing a word processor file from OpenOffice or
something like that and being able to edit the file in such a way that
I (or someone else) can open it up in the native editor and it will
work.

I don't care much about WYSIWYG facilities. I just want to be able to
edit it safely.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Lennart Borgman (gmail)]

Nic James Ferrier wrote:
> "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:
> 
>> How is that? Do you mean that a document following the schema might not 
>> be valid as a word processor document?
> 
> No. I mean that a schema might not express totally the constraints
> that a word processor requires.
> 
> Actually, I think this is quite common.


I just read that XML Schema could express more constraints. Is that what 
is used?

Re: Saving markup formats

[Nic James Ferrier]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> I just read that XML Schema could express more constraints. Is that what 
> is used?

No. I don't think so. Some of the word processors use XMLSchema (I
don't think XMLSchema is particularly more capable btw).

But the constraints I am speaking of are implementation defined.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Dave Pawson]

On 18/06/07, Nic James Ferrier <nferrier@tapsellferrier.co.uk> wrote:

> OpenOffice, last time I mucked about with the files, seemed to have
> some funny constraints about where the style information could go. It
> seemed like certain styles were possible in the document, but others
> weren't. The schema didn't seem to describe that stuff, it was just
> abritary constraints on top of what the schema specified.

Other tools (Schematron comes to mind) are possibly more
appropriate for such checking which is beyond the schema.
Though if you've specific complaints about ODF schema I can
forward them to that group.


>
>
> > Could you give me a more concrete example of what you want to do? (Or
> > did you give that before?)
>
> I am talking about editing a word processor file from OpenOffice or
> something like that and being able to edit the file in such a way that
> I (or someone else) can open it up in the native editor and it will
> work.
>

I've worked with ODF content.xml files in nxml-mode quite happily,
using the ODF rng schemas?

regards


-- 
Dave Pawson
XSLT XSL-FO FAQ.
http://www.dpawson.co.uk

Re: Saving markup formats

[Juri Linkov]

>> Could you give me a more concrete example of what you want to do?
>> (Or did you give that before?)
>
> I am talking about editing a word processor file from OpenOffice or
> something like that and being able to edit the file in such a way that
> I (or someone else) can open it up in the native editor and it will
> work.
>
> I don't care much about WYSIWYG facilities. I just want to be able to
> edit it safely.

Editing a word processor file can mean different things.  I see at least
three levels:

- editing raw XML files (e.g. with the help of nxml);
- editing DOM trees (like in graphical XML editors: expand/collapse
  subtrees, add/delete a node, change its attributes);
- editing on the visual representation of the XML file (WYSIWYG).

Which one from these you are talking about?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Saving markup formats

[Nic James Ferrier]

Juri Linkov <juri@jurta.org> writes:

>>> Could you give me a more concrete example of what you want to do?
>>> (Or did you give that before?)
>>
>> I am talking about editing a word processor file from OpenOffice or
>> something like that and being able to edit the file in such a way that
>> I (or someone else) can open it up in the native editor and it will
>> work.
>>
>> I don't care much about WYSIWYG facilities. I just want to be able to
>> edit it safely.
>
> Editing a word processor file can mean different things.  I see at least
> three levels:
>
> - editing raw XML files (e.g. with the help of nxml);
> - editing DOM trees (like in graphical XML editors: expand/collapse
>   subtrees, add/delete a node, change its attributes);
> - editing on the visual representation of the XML file (WYSIWYG).
>
> Which one from these you are talking about?

Not one of those.

What I am talking about is something not much above editing raw
XML... but above it none the less.

For example, here's a use case.

I want to provide my CV in Word because agents over here use that (I
always give them an HTML version as well... but for some reason they
like to mail each other Word files).

I would like to be able to open the word file directly in Emacs and
see some view of it. It doesn't matter if it's structural rather than
WYSIWYG but I don't want to look at Word XML thanks... it's awful.

I want to add a Job Description for my recent job to the list. It is
formatted as a heading followed by a list of technologies used and
then a paragraph or two.

I should be able to simply:

- add a heading and mark it as such (by picking from the style list) 

- add a list (same thing) and some list items

- add a paragraph

It doesn't have to be WYSIWYG, it could be more like WordPerfect used
to be (or more complex than that as long as the text is quite
central). 

But it does have to deal with the document natively and allow me to
use it in the way that the document format intends (adding list items,
paragraphs, etc...)


I'm not sure this is desirable or possible with Word. But with opendoc
it certainly is.

It seems to me that the key thing is not the WYSIWYG bit... but the
rendering of one form (in this case XML) into something not directly
derived.

I guess that's why there's resistance here to adding libxml2 to emacs,
because a DOM model would supplant a traditional buffer?

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Dave Pawson]

On 18/06/07, Nic James Ferrier <nferrier@tapsellferrier.co.uk> wrote:

> > - editing raw XML files (e.g. with the help of nxml);
> > - editing DOM trees (like in graphical XML editors: expand/collapse
> >   subtrees, add/delete a node, change its attributes);
> > - editing on the visual representation of the XML file (WYSIWYG).
> >
> > Which one from these you are talking about?
>
> Not one of those.
> For example, here's a use case.

> I would like to be able to open the word file directly in Emacs and
> see some view of it. It doesn't matter if it's structural rather than
> WYSIWYG but I don't want to look at Word XML thanks... it's awful.
>

I agree it's an aweful form of XML, and possibly that's the reason
not to choose to operate on it?

Why not use one of the many CV XML formats around, then transform
from that into Office XML?

That way emacs is still the tool of choice, and you can go to html
and pdf formats for the users that still want a sensible format.

regards

-- 
Dave Pawson
XSLT XSL-FO FAQ.
http://www.dpawson.co.uk

Re: Saving markup formats

[Nic James Ferrier]

"Dave Pawson" <dave.pawson@gmail.com> writes:

> I agree it's an aweful form of XML, and possibly that's the reason
> not to choose to operate on it?
>
> Why not use one of the many CV XML formats around, then transform
> from that into Office XML?

What I actually do is keep my CV in org mode and then transform it
from there, as you say, then I get OO, PDF, HTML and whatever I like.


But that's not the point. Making an edit to an existing document is a
useful thing.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Jason Rumney]

Nic James Ferrier wrote:
> nxml-mode doesn't have DOM manipulation tools, does it?
>   

No. A mode for editing raw XML has different needs than a higher-level
mode that happens to use XML as its storage format. The best way forward
would be to improve xml.el, and add xpath functionality for accessing
the DOM.

XML editing wishlist (was: Saving markup formats)

[Drew Adams]

Some thoughts on Emacs support for editing XML...

I use nxml mode now when I edit XML with Emacs 22. I use SGML mode for that
when I use Emacs 20 (which is still most of the time ;-)). I'm not very
familiar with either mode, sad to say. So please ignore or forgive (and
correct) any misconceptions about them that I might demonstrate.

I edit XML code quite often, in spite of not having taken the time to check
out nxml mode well. Sometimes the XML I edit is within SQL or PL/SQL code,
as an argument to a SQL function. Similarly, I use XQuery and XPath
expressions quite often - likewise, within calls to SQL functions such as
XMLQuery and XMLTable. I really should try a multi-mode approach such as mmm
or mumamo, no doubt, but I haven't had time to do that yet either - I just
use SQL mode.

I know that there is an XQuery Emacs mode that exists. I downloaded it, but
I haven't yet tried it. I see zero commentary in the file and I haven't
found any Info file for it.

I often use XML Schema - that is, most of the XML I use is XML Schema-based.
I do not use any other type of schema for XML, except sometimes DTDs. In
particular, I don't use Relax NG. In the database world, it's all about XML
Schema and XQuery - and becoming more so (no flames, please).

I would love to see a really handy XML editing mode. I don't claim it
doesn't exist, but I haven't yet noticed some things I'd like to see. `C-h
m' in nXML mode gives only a cursory description, the XML menu-bar menu
seems quite sparse, there are no contextual pop-up menus (that I have
noticed), and the nXML Info manual (which I have only skimmed) seems also
quite skimpy. In short, where's the beef? ;-)

XML is huge. It is an ugly cousin of Lisp, in some ways. We really should
have something more to offer the _many_ different kinds of XML users, IMO.
Again, maybe we do, and I'm just unaware of it.

FWIW, I also use Framemaker, for work, and it is a surprisingly good WYSIWYG
round-trip editor for XML documents. It has great search and replace for XML
elements and attributes, and good visualization and manipulation of XML
structure (it's easy to change the structure interactively and not get lost
in what you're doing). It also has a good validation interface, but it is
limited to DTDs (or to simple XML Schemas that are limited to what DTDs can
do).

Some things I'd like to see in Emacs for XML (they might be there, but I
haven't noticed):

1. Double-click the mouse  on `<' or `>' to pick up a complete sexp, just as
in Lisp. Or, better, double-click to pick up a complete tag and triple-click
to pick up the complete element. Currently, as soon as there is any
whitespace (e.g. there is an attribute), the complete tag is not selected.
Double-click, then right-click to extend to the next element boundary (or
something more significant than just a word). Other mouse selection
possibilities could also be useful. The mouse now seems almost useless for
XML.

2. XPath navigation.

3. XQuery navigation.

4. XML Schema support.

5. XML-specific search capabilities, even unrelated to XPath and XQuery:
search (and replace) for elements with certain attribute values etc. - at
least as good as what Framemaker has.

You can imagine different things for #2 and #3, from adding, deleting, and
selecting nodes to constructing more XML data using XQuery. Really, the
possibilities are limitless. I'd like the editor to let me use XQuery and
XPath to edit XML. I'd also like editor support for XQuery code itself; that
is, for editing XQuery and XPath expressions. Neither is XML (unlike XSLT).

XPath and XQuery (which includes XPath) are the natural matching languages
for XML. Regular expressions are completely general; they cannot take
advantage of anything defined in the XML language.

I've thought about adding something for #2 and #3 to Icicles, and I might
yet do so when I get some time. The idea would be to type an XPath or XQuery
expression as minibuffer input and have all of the matching nodes become
completion candidates (yes, whole nodes). More precisely, the result of an
XQuery expression is a sequence of items, which can be atoms or nodes. The
sequence would be used as the completion candidates set.

Different commands would do different things with those candidates. I would
use an Icicles search command to navigate among them. I would use another
command to select one or more, or transform one or more. And so on. IOW, use
XML-aware matching instead of regexp matching.

For that, I would add a different kind of completion to Icicles, in addition
to prefix completion and apropos (regexp) completion. But I would need
something to parse the XML and come up with the set of matching candidates
for the current input. I have no time to work on that parsing code myself -
perhaps such code exists and I could tap into it; I don't know.

I've had this project in the back of my head almost since I started working
on Icicles. Put it this way - soon after I had the idea of adding regexp
completion, I thought of adding XPath or XQuery completion. After all, the
basic infrastructure of Icicles can use any kind of matching. The hard part
(now) is doing the XQuery/XPath matching.

Anyway, just throwing this out as food for thought, in case someone finds it
interesting. I've kept it to myself long enough, without yet finding the
time to do anything about it. Actually, I did mention it on some XML lists
long ago, asking for some info about existing Emacs packages, but I never
got any reply.

IIUC, the current limited form of Emacs XML support is both a pain and an
opportunity to do something better.

Re: XML editing wishlist

[Lennart Borgman (gmail)]

Drew Adams wrote:
> 2. XPath navigation.

There is a little module in nXhtml showing the path (nxml-where.el), but 
I guess you mean something like giving the path and moving to the 
corresponding point?

RE: XML editing wishlist

[Drew Adams]

> > 2. XPath navigation.
>
> There is a little module in nXhtml showing the path (nxml-where.el), but
> I guess you mean something like giving the path and moving to the
> corresponding point?

1. Yes.

2. How can nxhtml "show _the_ path"? There are limitless XPath expressions
to represent the same node. I suppose you mean that it shows one such path.

Re: XML editing wishlist

[Lennart Borgman (gmail)]

Drew Adams wrote:
>>> 2. XPath navigation.
>> There is a little module in nXhtml showing the path (nxml-where.el), but
>> I guess you mean something like giving the path and moving to the
>> corresponding point?
> 
> 1. Yes.
> 
> 2. How can nxhtml "show _the_ path"? There are limitless XPath expressions
> to represent the same node. I suppose you mean that it shows one such path.


It just shows the parents up to the root.

Re: XML editing wishlist

[Lennart Borgman (gmail)]

Drew Adams wrote:
> I would love to see a really handy XML editing mode. I don't claim it
> doesn't exist, but I haven't yet noticed some things I'd like to see. `C-h
> m' in nXML mode gives only a cursory description, the XML menu-bar menu
> seems quite sparse, there are no contextual pop-up menus (that I have
> noticed), and the nXML Info manual (which I have only skimmed) seems also
> quite skimpy. In short, where's the beef? ;-)

In the function nxml-complete.

Internally nXml uses rng, but there are converters available from dtd 
and xsd.


> 1. Double-click the mouse  on `<' or `>' to pick up a complete sexp, just as
> in Lisp. Or, better, double-click to pick up a complete tag and triple-click
> to pick up the complete element. Currently, as soon as there is any
> whitespace (e.g. there is an attribute), the complete tag is not selected.
> Double-click, then right-click to extend to the next element boundary (or
> something more significant than just a word). Other mouse selection
> possibilities could also be useful. The mouse now seems almost useless for
> XML.

There are no mouse bindings, but the needed functions are there in nXml
AFAICS.

Re: XML editing wishlist

[Vagn Johansen]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> Drew Adams wrote:
>> 1. Double-click the mouse  on `<' or `>' to pick up a complete sexp, just as
>> in Lisp. Or, better, double-click to pick up a complete tag and triple-click
>> to pick up the complete element. Currently, as soon as there is any
>> whitespace (e.g. there is an attribute), the complete tag is not selected.
>> Double-click, then right-click to extend to the next element boundary (or
>> something more significant than just a word). Other mouse selection
>> possibilities could also be useful. The mouse now seems almost useless for
>> XML.
>
> There are no mouse bindings, but the needed functions are there in nXml
> AFAICS.


With nxml-sexp-element-flag configured to t you can use the normal sexp
functions: C-M-SPC, C-M-F, etc.

-- 
Vagn Johansen

Re: Saving markup formats

[Juri Linkov]

> I certainly don't care about wysiwyg. I'd just like to have good
> editing support for documents written in word or openoffice.

If you don't need WYSIWYG, you can edit them just visiting files in Emacs
in their source format.  OpenOffice has two XML formats .odt and .sxw,
and AbiWord has the format .abw which is XML as well.  How else do you want
to edit these documents without WYSIWYG?

> A good embedded XML parser would be needed. I've thought about taking
> time out to add libxml2 support natively to emacs. That would do the
> job very well.

Emacs already has a XML parser (`xml-parse-file' in xml.el), so this is
not a problem.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Saving markup formats

[Richard Stallman]

    I suppose the most important question is not whether Emacs can read/write
    files in some particular format, but rather a question whether Emacs can
    render format elements correctly on screen.

Both are crucial for this; they both need to implemented, mostly
separately.

Re: Saving markup formats

[Oliver Scholz]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> [ I'd suggest to move this discussion to emacs-devel ]
>
> On Sat, Jun 16 2007, Richard Stallman wrote:
>
> [ Juri Linkov: ]
>>     What are the most preferable formats to save markup?  One variant is
>>     Enriched text.  It was designed for using in e-mail, but actually nobody
>>     uses it nowadays.  Most people prefer HTML in e-mail as a replacement of
>>     plain text.
>>
>> HTML would be useful.  RTF would be useful.  XML would be useful.
>> The old Word format would be useful, at least to be able to read it.
>> If you implement any of these, it will be a nice contribution.
>> If you implement more than one, even better.
>
> Oliver Scholz <epameinondas@gmx.de> (Cc-ed) has done some work on an
> RTF reader for Emacs a couple of years ago.  Maybe his work is useful
> in the current context.  Oliver has signed papers for Emacs (past and
> future changes) and I'm quite sure he'd assign this code as well.
>
> Here's the summary from the project page:
>
> ,----[ http://savannah.nongnu.org/projects/emacs-rtf/ ]
> | This package aims to implement the "Rich Text Format" (RTF), version
[...]

This is a very old project of mine, and an abandoned one, I am afraid.
Of course, anybody is free to make use of the codebase, but I for
myself am convinced that it is the wrong approach. I did some further
work on the issue of processing XML or RTF or HTML documents in Emacs,
but it is an uphill struggle. And because it is---if done well---a
little too big for somebody with a notorious lack of time, this has
been sleeping for quite a while. Parsing is not the problem. (Though
for XML: none of the existing XML parsers would do the job. You need
CDATA in a buffer, not as strings in a list.)


    Oliver
-- 
Oliver Scholz               30 Prairial an 215 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: Saving markup formats

[Juri Linkov]

> This is a very old project of mine, and an abandoned one, I am afraid.
> Of course, anybody is free to make use of the codebase, but I for
> myself am convinced that it is the wrong approach.

Could you tell why do you think it is the wrong approach?  This would help
someone who will do something similar to avoid mistakes you think you made.
How would you do things over if you had enough time?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Saving markup formats

[Oliver Scholz]

Juri Linkov <juri@jurta.org> writes:

>> This is a very old project of mine, and an abandoned one, I am afraid.
>> Of course, anybody is free to make use of the codebase, but I for
>> myself am convinced that it is the wrong approach.
>
> Could you tell why do you think it is the wrong approach?  This would help
> someone who will do something similar to avoid mistakes you think you made.

Basically the approach was too naive. Basically I started like this:
"Hey, implementing RTF can't be too hard. Let's just take the RTF
spec, write a parser for it, get the text with some text properties
into a buffer and write a major mode for editing it."

Until I realised, a) if you want word processing in Emacs this SHOULD
be designed with different target formats right from the start and b)
even for RTF alone this is not sufficient. IIRC I was originally
planning for flat data structure: RTF's paragraph formatting
properties and character formatting properties each stored as lists or
vectors in a separate text property. Font-lock then would resolve
character formatting properties and apply faces, Fill-paragraph would
resolve whitespace formatting properties. This would work for simple
cases. But it wouldn't in all cases preserve the logical structure of
the original document, if you got it from somebody using a different
word processor. This is a very bad thing; a _reliable_ word
processor---as opposed to an unreliable hack---shouldn't make any
changes to the logical structure of a document unless explicitly
ordered to do it. Also, while it is o.k. to implement only a subset of
RTF in the beginning, the design (or lack thereof) of the data
structure would eventually lead to a dead end.

> How would you do things over if you had enough time?

I'd start designing the data structure. I would do it with an eye on
the various specifications for XML (most notably: the XML info set and
the style properties in CSS), for the simple reason that they were
designed to cover a wide range of needs for text/data representation,
formatting, text processing etc. and that in this area they are tested
by a lot of people. So looking at XML right from the start could help
avoiding shortcomings in the design that lead to dead ends or crude
kludges later. Also, for a word processing suite in Emacs, XML file
formats would be the major target formats besides RTF: XHTML, TEI XML,
DocBook, eventually techinfo XML.

So, IMNSHO spending thought about text representation and rendering in
Emacs is the _very first_ thing to do. Once you have a capable data
structure, parsing RTF is not too hard. You can regard an RTF document
as some sort of weird s-expression, with "{" and "}" instead of
parentheses. It is still a dreadful file format, because of its lack
of constraints, but, again, if and only if you have a well designed
data structure, you have a fighting chance to deal with those dreads.

I'd start with designing a data structure that is a realisation of the
XML info set. (This has nothing to do with pointy brackets. The XML
info set is a specification of requirements for a data structure. IIRC
there is a W3C technical report out there.) This doesn't have to be
DOM. I am pretty confident, that it is rather straight forward to
parse RTF into an instance of the XML info set.

Unfortunately, this is were the trouble starts. With the XML info set
the logical structure of the internal data structure is clear (except
for style properties). But the specifics depend on Emacs' ability to
render text on-screen. Of course, there are ways to implement a
tree-like data structure even with CDATA as text in a buffer right
now---I was experimenting with overlays, for instance. But eventually,
if you really go for word processing, you'd have to enhance the
display engine anyways to deal with certain style properties. So you'd
might as well design both, the tree-like data structure and how the
display engine deals with it, right from the beginning---thus gaining,
possibly, maximum reliability.

Also, after I discarded the naive approach I have spent a lot of
thought on UI issues and I would advice anybody to do the
same---again, right from the beginning, before you write a single line
of code. What is word processing exactly? Current word processors are
hybrid beasts, undecided between DTP software (like Adobe Indesign,
Quark Express, Macromedia Freehand, Inkscape, Scribus) and programmes
dealing the logical structure of documents. Word processors have an
ambiguous editing model, resulting from a long history starting with
their origin as replacements for typewriters. Emacs' editing model on
the other hand is mostly about dealing with text/plain, even were it
goes beyond that. So if you come from the Emacs world alone, you are
bound to unwillingly design your UI according to your best known
editing model (especially because it is easiest to implement in Emacs)
and maybe you even make decisions for the design of the data structure
which make that mandatory. Thus, you'd add to the confusion that's
already there. The result being probably the worst word processor
ever, not the best. In that case, would be better to stick with
packages like Muse or emacs-wiki or something similar, which are
one-way (no reading of other file formats), for your own documents
(that's what I do); and use Abiword or OpenOffice if you receive
documents from somebody else. Word processors are good for document
exchange, that's why they are more popular than DTP software. So,
should the UI make the logical structure which would be stored in the
file explicit? What are the objects a user wants to interact with?
I.e. if there are four spaces at the right margin, and if the user can
put the cursor on those spaces, copy them etc., then those spaces are
an object, the use can interact with. They are _there_. But is this
what he or she wants? Or does she just want a left margin that is 4em
wide? How do you distinguish a 4em margin from four spaces at the
left?

If somebody is seriously going to implement WP with a good large-scale
battle plan, then please drop me a line. I'd might find a little time
to contribute.

    Oliver
-- 
1 Messidor an 215 de la Révolution
Liberté, Egalité, Fraternité!

RE: Saving markup formats

[Drew Adams]

Interesting post, Oliver. Lots of food for thought. Thanks.

> I'd start designing the data structure. I would do it with an eye on
> the various specifications for XML (most notably: the XML info set and
> the style properties in CSS)

What about the XQuery and XPath data model (XDM), instead of the Infoset? It's essentially the XML Infoset + XML Schema types. From the W3C XDM spec (http://www.w3.org/TR/xpath-datamodel/): 

 The data model is based on the [Infoset] (henceforth "Infoset"),
 but it requires the following new features to meet the [XPath
 2.0 Requirements] and [XML Query Requirements]:

 * Support for XML Schema types. The XML Schema recommendations
   define features, such as structures ([Schema Part 1]) and
   simple data types ([Schema Part 2]), that extend the XML
   Information Set with precise type information.

 * Representation of collections of documents and of complex values.
   ([XML Query Requirements])

 * Support for typed atomic values.

 * Support for ordered, heterogeneous sequences.

Re: Saving markup formats

[Stefan Monnier]

> I.e. if there are four spaces at the right margin, and if the user can put
> the cursor on those spaces, copy them etc., then those spaces are an
> object, the use can interact with. They are _there_. But is this what he
> or she wants? Or does she just want a left margin that is 4em wide? How do
> you distinguish a 4em margin from four spaces at the left?

Which is why I much prefer something like LaTeX.


        Stefan

Re: Saving markup formats

[Richard Stallman]

    This is a very bad thing; a _reliable_ word
    processor---as opposed to an unreliable hack---shouldn't make any
    changes to the logical structure of a document unless explicitly
    ordered to do it.

I reject that position (as well as the gratuitous insult tacked onto
it).  It is an important feature of Emacs that text is a sequence of
characters, not structured.  That is why you can kill text anywhere
and yank it anywhere.

To save files as RTF does not require that we use the structured
nesting features of RTF.  We want to be able to read such RTF files,
and when doing som we can either flatten the structure or record it by
means of markup text (effectively, open and close braces in the text
itself).

Re: Saving markup formats

[Nic James Ferrier]

Richard Stallman <rms@gnu.org> writes:

> To save files as RTF does not require that we use the structured
> nesting features of RTF.  We want to be able to read such RTF files,
> and when doing som we can either flatten the structure or record it by
> means of markup text (effectively, open and close braces in the text
> itself).

That's interesting. You see emacs buffers as a primary contruct in the
implementation of a word processor.


I'd prefer to be talking about something other than RTF, say OpenDoc,
because it is acutally more useful now than RTF.


I'm not an expert, but I've done a little hacking with OpenDoc. It is
an XML format but comes wrapped up in a zip file.

In otherwords if I give you my CV in OpenDoc I'm giving you a zip file
with a number of XML files in it.

There always seems to be one XML file with the content of the document
in it. It consists of a number of logical style definitions followed
by markup describing the content of the document and applying styles.

Here's a little bit of content from the top of an OpenDoc document:

    <office:body>
      <office:text>
        <office:forms form:automatic-focus="false" form:apply-design-mode="false"/>
        <text:sequence-decls>
          <text:sequence-decl text:display-outline-level="0"
                              text:name="Illustration"/>
          <text:sequence-decl text:display-outline-level="0" 
                              text:name="Table"/>
          <text:sequence-decl text:display-outline-level="0" 
                              text:name="Text"/>
          <text:sequence-decl text:display-outline-level="0" 
                              text:name="Drawing"/>
        </text:sequence-decls>
  Nic Ferrier
  <text:h text:style-name="Heading_20_1">Nic Ferrier - Curriculum Vitae</text:h>
  <text:p text:style-name="Standard">Nic is a hacker.</text:p><text:p text:style-name="Standard"/>

It's mostly gobbledegook, as you can see. Checking the schema for
OpenDoc I note that most of the above (the office:forms element and
the text:sequence-decls element) is optional.


Now, an emacs textual representation of a styled document would seem
to be a really useful thing to have. This would be capable of
attaching styles to text via properties. The styles would come from
one of a number of sets of predefined styles; there might be an HTML
set or an OpenDoc set.

The definition of a style would have to be very loose. Probably just a
text string as an identifier and then something attached to the text
string.


Then we could write importers to the styled form and "displayers" of
the styled form, like the current emacs rich text mode. The styles
would be defined by the import/export programs and the "displayer".

It would be a start.


I think such a plan would require nxml-mode, at least James Clark's
xmltok lisp library which comes with nxml-mode.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Lennart Borgman (gmail)]

Nic James Ferrier wrote:
> Then we could write importers to the styled form and "displayers" of
> the styled form, like the current emacs rich text mode. The styles
> would be defined by the import/export programs and the "displayer".
> 
> It would be a start.

What is it you are trying to do? I get an impression that you want to 
Write OpenOffice in Emacs, but I guess I am exaggerating.

Maybe some ideas for how to find and edit important parts of an 
OpenOffice doc would be useful?

Re: Saving markup formats

[Nic James Ferrier]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

> What is it you are trying to do? I get an impression that you want to 
> Write OpenOffice in Emacs, but I guess I am exaggerating.

Heh.

I find it frustrating that I sometimes have to use OpenOffice or
Word. I would prefer to use Emacs for editing Word and OpenDoc files
(and Abiword and ....)

But I do think it is a massive exaggeration to say that I want to
write OpenOffice in Emacs... I don't.

It seems to me that _most_ of a word processing program like OO or
Word is:

- wysiwyg display code
- integration with other apps
- definition of styles

and I don't see that Emacs has to do any of those things to have a
useful editor for OpenDoc files (for example).



> Maybe some ideas for how to find and edit important parts of an 
> OpenOffice doc would be useful?

Yeah. Sure. That'd be great.

Or were you asking me? I have a clue... but I'm not really interested
in editing them as XML files.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Lennart Borgman (gmail)]

Nic James Ferrier wrote:
>> Maybe some ideas for how to find and edit important parts of an 
>> OpenOffice doc would be useful?
> 
> Yeah. Sure. That'd be great.
> 
> Or were you asking me? 

Yes ;-)

> I have a clue... but I'm not really interested
> in editing them as XML files.

I guess that would be the price (to edit them as XML). I understand it 
as if you want some intermediate view. I think that is reasonable only 
on an abstract level (like navigating etc).

Re: Saving markup formats

[Nic James Ferrier]

"Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:

>> I have a clue... but I'm not really interested
>> in editing them as XML files.
>
> I guess that would be the price (to edit them as XML). I understand it 
> as if you want some intermediate view. I think that is reasonable only 
> on an abstract level (like navigating etc).

I want emacs to be my word processor. I don't care about wysiwyg. I
just want to be able to edit the files. I don't think this is
unreasonable since most of the information is in well-marked up
documents.

Let me tell you about how I make my CV right now:

- I edit the CV in org mode
- I transform org mode to XOXO/XHTML using elisp
- I run an XSLT transform to turn the XOXO into XHTML
- I run an XSLT transform to turn the XHTML into an OpenDoc
  content.xml file
- I zip the new content.xml file up with a standard template for the
  other files that come with an OpenDoc file

So essentially I'm doing there what I'm talking about here. Just in a
long winded way and using more than just elisp.

org-mode is providing stylistic markup which has syntax and can be
serialized to some syntax not it's own (XOXO in this case).


Instead of this I don't see why I shouldn't be able to write a mode
that stores text in properties as a set of named styles (in the case
of OpenDoc these would be derived from the Relax-NG Schema and from a
template of common styles); eg in elisp manual text property syntax:


#("Nic Ferrier - CV" 0 16 (element h2))
#("Nic is a hacker with some crazy ideas" 0 37 (element para style p1))


this would be quite simple to display nicely in Emacs (in a variety of
ways) and it has all the syntactic information necessary to serialize
back to an OpenDoc document.

I just need some way of getting the XML into the text properties. And
xmltok might help with that.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

PS in *principle* I don't see any reason why word is any different than
OpenDoc. It's just XML at the end of the day. Wouldn't it be a coup to
be able to edit Word files in Emacs?

Re: Saving markup formats

[Jeremy Maitin-Shepard]

Nic James Ferrier <nferrier@tapsellferrier.co.uk> writes:

> "Lennart Borgman (gmail)" <lennart.borgman@gmail.com> writes:
>>> I have a clue... but I'm not really interested
>>> in editing them as XML files.
>> 
>> I guess that would be the price (to edit them as XML). I understand it 
>> as if you want some intermediate view. I think that is reasonable only 
>> on an abstract level (like navigating etc).

> I want emacs to be my word processor. I don't care about wysiwyg. I
> just want to be able to edit the files. I don't think this is
> unreasonable since most of the information is in well-marked up
> documents.

> Let me tell you about how I make my CV right now:

> - I edit the CV in org mode
> - I transform org mode to XOXO/XHTML using elisp
> - I run an XSLT transform to turn the XOXO into XHTML
> - I run an XSLT transform to turn the XHTML into an OpenDoc
>   content.xml file
> - I zip the new content.xml file up with a standard template for the
>   other files that come with an OpenDoc file

That seems like a very long a difficult process for something that could
be done directly, with a more convenient syntax, and with better looking
output, using LaTeX.  ;)

Of course that has nothing to do with this thread, except perhaps the
fact that many people, including myself, use Emacs exclusively for word
processing by using it to edit LaTeX.  That may well be the best way to
use Emacs as a word processor.

[snip]

-- 
Jeremy Maitin-Shepard

Re: Saving markup formats

[David House]

Nic James Ferrier writes:
 > - I run an XSLT transform to turn the XOXO into XHTML

You realise XOXO is a subset of XHTML, right?

-- 
-David House, dmhouse@gmail.com

Re: Saving markup formats

[Nic James Ferrier]

David House <dmhouse@gmail.com> writes:

> Nic James Ferrier writes:
>  > - I run an XSLT transform to turn the XOXO into XHTML
>
> You realise XOXO is a subset of XHTML, right?

You missed what I was trying to say, XOXO is a data format (yes, I
know it's XHTML). I am transforming the data format into a document,
my CV.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[David House]

Nic James Ferrier writes:
 > David House <dmhouse@gmail.com> writes:
 > > You realise XOXO is a subset of XHTML, right?
 > 
 > You missed what I was trying to say, XOXO is a data format (yes, I
 > know it's XHTML). I am transforming the data format into a document,
 > my CV.

Yes, but I would have thought you could simplify the process a little:

org file -> XHTML + XOXO (org-mode export)
XHTML + XOXO -> content.xml (XSLT transform)
content.xml -> .odt file (zipping etc.)

Although it's likely I'm just missing some details of the process and it's
actually more complex than I understand.

-- 
-David House, dmhouse@gmail.com

Re: Saving markup formats

[Nic James Ferrier]

David House <dmhouse@gmail.com> writes:

> Yes, but I would have thought you could simplify the process a little:
>
> org file -> XHTML + XOXO (org-mode export)
> XHTML + XOXO -> content.xml (XSLT transform)
> content.xml -> .odt file (zipping etc.)

That moves the complexity of the transport to the code doing the org
mode export.

It's simpler to have that code as XSLT.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Mathias Dahl]

> Let me tell you about how I make my CV right now:
>
> - I edit the CV in org mode
> - I transform org mode to XOXO/XHTML using elisp
> - I run an XSLT transform to turn the XOXO into XHTML
> - I run an XSLT transform to turn the XHTML into an OpenDoc
>  content.xml file
> - I zip the new content.xml file up with a standard template for the
>  other files that come with an OpenDoc file

I do something similar to get stuff into Word, but I just open the
HTML exported by org-mode in Word (with a small elisp hack added to
org-mode), and voila! Maybe OpenOffice can do that as well? But maybe
you didn't want to need to use an external program (OpenOffice in this
case) to create the OpenDoc file?

Re: Saving markup formats

[Nic James Ferrier]

"Mathias Dahl" <mathias.dahl@gmail.com> writes:

>> Let me tell you about how I make my CV right now:
>>
>> - I edit the CV in org mode
>> - I transform org mode to XOXO/XHTML using elisp
>> - I run an XSLT transform to turn the XOXO into XHTML
>> - I run an XSLT transform to turn the XHTML into an OpenDoc
>>  content.xml file
>> - I zip the new content.xml file up with a standard template for the
>>  other files that come with an OpenDoc file
>
> I do something similar to get stuff into Word, but I just open the
> HTML exported by org-mode in Word (with a small elisp hack added to
> org-mode), and voila! Maybe OpenOffice can do that as well? But maybe
> you didn't want to need to use an external program (OpenOffice in this
> case) to create the OpenDoc file?

That's right.

The process above works really well because a CV is a very structured
document. But it wouldn't work generally.


Another point is that if you open an HTML file in Word of OpenOffice
they seem to get stuck treating it as an HTML file. That's no good. I
want real OpenDoc files.


And anyway... that doesn't help me edit an OpenDoc file that someone
else sent me or stored in a repository somewhere.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Mathias Dahl]

> Another point is that if you open an HTML file in Word of OpenOffice
> they seem to get stuck treating it as an HTML file. That's no good. I
> want real OpenDoc files.

At least in Word I can do Save As... and it seems to become a valid .doc file.

> And anyway... that doesn't help me edit an OpenDoc file that someone
> else sent me or stored in a repository somewhere.

True. I was just commenting on this "export" process in case you did
not know about it.

Re: Saving markup formats

[Nic James Ferrier]

"Mathias Dahl" <mathias.dahl@gmail.com> writes:

>> Another point is that if you open an HTML file in Word of OpenOffice
>> they seem to get stuck treating it as an HTML file. That's no good. I
>> want real OpenDoc files.
>
> At least in Word I can do Save As... and it seems to become a valid
> .doc file.

It does but I recall that the styles remain the same. I'm sure someone
experienced in those tools could pull the normal document styles
in. But how many of us are that experienced?

I guess we better ask Jimi Hendrix.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Oliver Scholz]

Richard Stallman <rms@gnu.org> writes:

>     This is a very bad thing; a _reliable_ word
>     processor---as opposed to an unreliable hack---shouldn't make any
>     changes to the logical structure of a document unless explicitly
>     ordered to do it.
>
> I reject that position (as well as the gratuitous insult tacked onto
> it)

It was not meant as an insult; by "unreliable hack" I was refering to
my own first implementation. Of course, I was also referring to a
general implementation strategy, still "insult" seems a bit strong.

I'll try to refrain from further discussing this issue. I have to
admit that from memory I can't think of any case where the approach
you describe would fail with RTF---at least with the help of markup
text in a buffer. My personal opinions are of no concern here.


    Oliver
-- 
2 Messidor an 215 de la Révolution
Liberté, Egalité, Fraternité!

Re: Saving markup formats

[David Kastrup]

Oliver Scholz <alkibiades@gmx.de> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>     This is a very bad thing; a _reliable_ word
>>     processor---as opposed to an unreliable hack---shouldn't make any
>>     changes to the logical structure of a document unless explicitly
>>     ordered to do it.
>>
>> I reject that position (as well as the gratuitous insult tacked onto
>> it)
>
> It was not meant as an insult; by "unreliable hack" I was refering to
> my own first implementation. Of course, I was also referring to a
> general implementation strategy, still "insult" seems a bit strong.
>
> I'll try to refrain from further discussing this issue. I have to
> admit that from memory I can't think of any case where the approach
> you describe would fail with RTF---at least with the help of markup
> text in a buffer. My personal opinions are of no concern here.

I find myself agreeing with you here: documents should preserve
structure.  Word has so-called "style sheets" as a structuring method,
and it means that you can change the layout of a document consistently
by changing the style sheet.  Ignoring the structure of the RTF and
saving something visually equivalent is breaking the document.  While
it does not much harm to the documents of _naive_ Word users, it would
be horrible to load a complex file, change a few words, and have it
saved basically as a seemingly same-looking but unmaintainable mess.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Saving markup formats

[Oliver Scholz]

David Kastrup <dak@gnu.org> writes:

> Oliver Scholz <alkibiades@gmx.de> writes:
>
>> Richard Stallman <rms@gnu.org> writes:
>>
>>>     This is a very bad thing; a _reliable_ word
>>>     processor---as opposed to an unreliable hack---shouldn't make any
>>>     changes to the logical structure of a document unless explicitly
>>>     ordered to do it.
>>>
>>> I reject that position (as well as the gratuitous insult tacked onto
>>> it)
>>
>> It was not meant as an insult; by "unreliable hack" I was refering to
>> my own first implementation. Of course, I was also referring to a
>> general implementation strategy, still "insult" seems a bit strong.
>>
>> I'll try to refrain from further discussing this issue. I have to
>> admit that from memory I can't think of any case where the approach
>> you describe would fail with RTF---at least with the help of markup
>> text in a buffer. My personal opinions are of no concern here.
>
> I find myself agreeing with you here: documents should preserve
> structure.  Word has so-called "style sheets" as a structuring method,
> and it means that you can change the layout of a document consistently
> by changing the style sheet.  Ignoring the structure of the RTF and
> saving something visually equivalent is breaking the document.  While
> it does not much harm to the documents of _naive_ Word users, it would
> be horrible to load a complex file, change a few words, and have it
> saved basically as a seemingly same-looking but unmaintainable mess.

Yes, exactly. Another thing is that RTF is used as document exchange
format: for instance, people send the exposé for their PhD thesis to
friends and teachers and get it sent back with changes. If you did
once take part in such a process, you may have noticed the
inconsistent formatting (paragraphs or even sentences with different
font hight and the like) in such documents? This often comes from one
of the involved programmes having its own ideas about stylesheets and
document structure, thus introducing slight chances at some spots when
saving. The last step in such an editorial process is always fixing
the formatting.

Technically, though, you can preserve the nested structure in such
cases by using a special markup text. This is also true for features
like---just from the top of my head--- "track changes (revision
marks)", i.e. changes made in a special mode for editorial revisions
(I forgot the English name, the German name is "Änderungsmodus"). This
feature is very popular among certain users in any collaborative
editing process. Those "revision marks" can, and probably will be
nested, if more than one editor other than the original author is
involved. But again, you can---technically---address this by means of
markup text in the buffer.

What I, personally, think about this UI-wise doesn't really matter.
Though, matters of taste aside, it will be an interesting task to make
this UI secure against inadvertent and unnoticed changes by the user
to the document structure (for instance by yanking text at the wrong
spot) that could cause trouble. And by "trouble" I mean things like:
the formatting of a 200p. master thesis going south half an hour
before the final dead line.

Sorry. I should really shut up now. I don't see the paradagm shift
coming that I deem necessary.


    Oliver
-- 
2 Messidor an 215 de la Révolution
Liberté, Egalité, Fraternité!

Re: Saving markup formats

[Nic James Ferrier]

Oliver Scholz <alkibiades@gmx.de> writes:

> What I, personally, think about this UI-wise doesn't really matter.
> Though, matters of taste aside, it will be an interesting task to make
> this UI secure against inadvertent and unnoticed changes by the user
> to the document structure (for instance by yanking text at the wrong
> spot) that could cause trouble. And by "trouble" I mean things like:
> the formatting of a 200p. master thesis going south half an hour
> before the final dead line.
>
> Sorry. I should really shut up now. I don't see the paradagm shift
> coming that I deem necessary.

What do you think about the idea of an Emacs mode based on properties
representing styles as I outlined earlier?

The tiny OpenDoc example:

#("Nic Ferrier - CV" 0 16 (element h2))
#("Nic is a hacker with some crazy ideas" 0 37 (element para style p1))

I think this would work for RTF as well. Clearly we wouldn't need an
XML parser for RTF.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Oliver Scholz]

Nic James Ferrier <nferrier@tapsellferrier.co.uk> writes:

[...]
> What do you think about the idea of an Emacs mode based on properties
> representing styles as I outlined earlier?
>
> The tiny OpenDoc example:
>
> #("Nic Ferrier - CV" 0 16 (element h2))
> #("Nic is a hacker with some crazy ideas" 0 37 (element para style p1))

Unless I am missing something, this is just one possible internal
representation of a tree-like data structure. In fact, I have once
experimented with this. Its main problem is that the data structure
would get corrupted in certain cases, especially when yanking text or
when some Lisp function is using `insert'. One way to deal with that,
would be to let font-lock automatically fix it. But this seems ugly.

More importantly, I understand that Richard is strongly opposed to any
such kind of data structure. So this wouldn't make any difference. If
a tree-like data structure is o.k., then---given the sensibility of
maintaining the integrity of said data structure---I, as a user, would
very much prefer some solution at the C-level. At least in the long
run. But actually, the specific internal representation is a secondary
issue. The general architecture and the UI is much more important and
I don't see any room for agreement there.


    Oliver
-- 
Oliver Scholz               2 Messidor an 215 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: Saving markup formats

[Nic James Ferrier]

Oliver Scholz <alkibiades@gmx.de> writes:

>> #("Nic Ferrier - CV" 0 16 (element h2))
>> #("Nic is a hacker with some crazy ideas" 0 37 (element para style p1))
>
> Unless I am missing something, this is just one possible internal
> representation of a tree-like data structure. In fact, I have once
> experimented with this. 

I thought the rms was saying "why store data somewhere other than a
buffer?". In other words, why add DOM (for example) to Emacs so that
we can store text in it when we already have text buffers?

The above snippet puts the document structure into the buffer. But it
doesn't tie the document structure to one form. It can be anything
that you can represent with elements and styles.


> Its main problem is that the data structure
> would get corrupted in certain cases, especially when yanking text or
> when some Lisp function is using `insert'. One way to deal with that,
> would be to let font-lock automatically fix it. But this seems ugly.

Ok. I'll have to look at that. Thanks.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Oliver Scholz]

Nic James Ferrier <nferrier@tapsellferrier.co.uk> writes:

> Oliver Scholz <alkibiades@gmx.de> writes:
>
>>> #("Nic Ferrier - CV" 0 16 (element h2))
>>> #("Nic is a hacker with some crazy ideas" 0 37 (element para style p1))
>>
>> Unless I am missing something, this is just one possible internal
>> representation of a tree-like data structure. In fact, I have once
>> experimented with this. 
>
> I thought the rms was saying "why store data somewhere other than a
> buffer?". In other words, why add DOM (for example) to Emacs so that
> we can store text in it when we already have text buffers?

If you use buffers and text-properties (or overlays) as an internal
represenation of the XML infoset (or the XDL of XPath) and then add an
according API, then you have DOM, or something very DOM-like. On the
other hand, if you do something at the C-level (whatever it is) and
make sure that the integration with the rest of Emacs is right, i.e.
that killing text in a WP buffer and yanking text in a source code
buffer DTRT (whatever the right thing is ...) and vice versa, then
"you can kill text anywhere and yank it anywhere". My reading of
Richards post was different, I sense a deep conceptual disagreement
behind the sentence: "It is an important feature of Emacs that text is
a sequence of characters, not structured." Otherwise, I really think
that the internal representation is a secondary issue, the decision
about which depending on possibly conflicting parameters like
robustness, performance, aesthetics and work needed for
implementation. The UI is relevant there, too, but that is another can
of worms.


    Oliver
-- 
Oliver Scholz               2 Messidor an 215 de la Révolution
Ostendstr. 61               Liberté, Egalité, Fraternité!
60314 Frankfurt a. M.       

Re: Saving markup formats

[Richard Stallman]

I am against structured data formats for an Emacs buffer because there
is no way to reconcile that cleanly with the Emacs Lisp model of a
buffer as a string of text.  I thought about this thoroughly 15 years
ago.

The problem is not just a matter of implementation.  It is a problem
of irreconcilable design concepts.  In an ordinary word processor,
the only goals are at the UI level.  In Emacs, though, we need to make
any design fit the Emacs programming model.

    I thought the rms was saying "why store data somewhere other than a
    buffer?".

No, I was talking about what the buffer contents can consist of.

	      In other words, why add DOM (for example) to Emacs so that
    we can store text in it when we already have text buffers?

I don't know the term "DOM", so I have to guess just what this means.
If it means that a buffer would consist of multiple strings, which are
put together through some other kind of structure, that is just no
good.  It can't make kill and yank work.

Re: Saving markup formats

[Nic James Ferrier]

Richard Stallman <rms@gnu.org> writes:

> I am against structured data formats for an Emacs buffer because there
> is no way to reconcile that cleanly with the Emacs Lisp model of a
> buffer as a string of text.  I thought about this thoroughly 15 years
> ago.

I don't *think* I'm talking about a data format in an emacs buffer.

I'm talking about using properties to mark the syntax of pieces of
text (in an ordinary buffer) as they are described in a word processor
schema instance (a document).


I think I'm just going to have to knock up an example and see whether
it's viable.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Nic James Ferrier]

Nic James Ferrier <nferrier@tapsellferrier.co.uk> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>> I am against structured data formats for an Emacs buffer because there
>> is no way to reconcile that cleanly with the Emacs Lisp model of a
>> buffer as a string of text.  I thought about this thoroughly 15 years
>> ago.
>
> I don't *think* I'm talking about a data format in an emacs buffer.
>
> I'm talking about using properties to mark the syntax of pieces of
> text (in an ordinary buffer) as they are described in a word processor
> schema instance (a document).
>
>
> I think I'm just going to have to knock up an example and see whether
> it's viable.

Ok. I looked into it a bit more.

I don't think I'm talking about something that breaks the primacy of
the buffer. In fact, what I'm talking about has practically already
been done.

I am talking about writing a mode that works primarily on text
properties specified via what most word processors call style. Emacs
has a term for this as well: the category property.

  `category'
     If a character has a `category' property, we call it the
     "category" of the character.  It should be a symbol.  The
     properties of the symbol serve as defaults for the properties of
     the character.

That's really all I'm talking about.

Importing an OpenDoc (or in principle a Word document) into Emacs in a
way that would ensure that you could save it back to something not too
dissimilar would mean:

- read the opendoc XML

- convert all the styles to elisp symbols;

  some of these styles ("list-item", "p", etc...) would have well understood
  display; for example a heading needs to be display in bold or
  something. We could map standard Emacs display stuff onto the
  category symbols.

- read in the document content inserting just the text into the
  buffer

- propertize each piece of text, as you insert, with the name of the
  appropriate category; eg: if you have the following XML:

    <text:p text:style-name="Standard">Designed a REST binding for the
            SAML specification for single sign on web applications.</text:p>

  then that would be converted into the following text buffer content:

    Designed a REST binding for the SAML specification for single sign
    on web applications.

  and the text would have a property assigned:

    category Standard

  so as to record the origin structure we might also add the following
  property:

    element p

  or:

    element text:p

- let the user edit the text, editing would have to include the
  ability to add, remove or change the category property from/to text.

  cut and paste is not a problem. the text carries with it the
  properties it was assigned. That's fine. That's all word processor
  users expect.


This doesn't deal with more complex structure (say, where you have a
paragraph embedded in a list item) but I think that is possible quite
naturally as well.

I hope this answers some concerns. I think it shows that Emacs _could_
be a viable word processor for modern wp formats (and RTF as well).

But if people are still doubtfull I'll have to go away and do it (in
the 10seconds of free time I have every week).

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Davis Herring]

> - propertize each piece of text, as you insert, with the name of the
>   appropriate category; eg: if you have the following XML:
>
>     <text:p text:style-name="Standard">Designed a REST binding for the
>             SAML specification for single sign on web
> applications.</text:p>
>
>   then that would be converted into the following text buffer content:
>
>     Designed a REST binding for the SAML specification for single sign
>     on web applications.
>
>   and the text would have a property assigned:
>
>     category Standard
>
>   so as to record the origin structure we might also add the following
>   property:
>
>     element p
>
>   or:
>
>     element text:p
>
> - let the user edit the text, editing would have to include the
>   ability to add, remove or change the category property from/to text.
>
>   cut and paste is not a problem. the text carries with it the
>   properties it was assigned. That's fine. That's all word processor
>   users expect.

I'm not sure that this property-set really captures the structure of the
document in a way that fits what properties do: if someone kills an entire
paragraph and yanks it inside another paragraph, Emacs won't know that the
text properties "element -> text:p" are from different paragraphs.  If
those paragraphs had different styles or so, you risk losing that
information in the most obvious implementations I can see.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Saving markup formats

[Nic James Ferrier]

"Davis Herring" <herring@lanl.gov> writes:

> I'm not sure that this property-set really captures the structure of the
> document in a way that fits what properties do: if someone kills an entire
> paragraph and yanks it inside another paragraph, Emacs won't know that the
> text properties "element -> text:p" are from different paragraphs.  If
> those paragraphs had different styles or so, you risk losing that
> information in the most obvious implementations I can see.

No. A style is unique. More than one paragraph might carry the same
style, but in that case they'll both have the same style
properties.

So if you have a paragraph with a style attached to it and you kill it
and then yank it to somewhere else and it has the same styles attached
to it.


The problem you're talking about does occur when you have 'embedded
structure', when you have a paragraph inside a list item for example.

But to be honest, I'm not sure how much that happens with OpenDoc and
Word. It certainly happens in these situations:

- tables
- lists

but both of those will require a little extra display work anyway. I
think anything embedded in a list or table will be free of any list or
table properties.


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Thien-Thi Nguyen]

() Nic James Ferrier <nferrier@tapsellferrier.co.uk>
() Thu, 21 Jun 2007 08:56:36 +0100

   I think anything embedded in a list or table will be free of
   any list or table properties.

that thinking doesn't scale, unfortunately.  better to think of
lists in lists, lists in tables, tables in lists, and tables in
tables.  generally, most things can logically be nested (and
sometimes illogically as well -- just look at what people publish
on the net :-).

here is an example i hope we can use as a "working exercise":

  lists
  * item1
  * ===========BIG==================
    =  foo         =  zzz          =
    =  * bar       =  * yyy        =
    =  * baz       =  * ==SMALL==  =
    =              =    = 3 = 5 =  =
    =  wow         =    = 4 = 2 =  =   <--
    =  * zow       =    = 6 = A =  =
    =    * yow     =    =========  =
    ================================
  * item3

each of these pieces of text (chosen to be unique for clarity) has
a location and a function: lists, item1, BIG, foo, zzz, bar, baz,
yyy, SMALL, 3, 5, wow, 4, 2, zow, 6, A, yow, item3.  additionally,
each "=", "*" and " " has its coordinate.

deriving a tree from this is easy.  rendering the tree is
moderately easy.  the hard questions are:
 - what text properties to put on each character?
 - if we do C-k at various places on the <-- (indicated) line:
   - what should happen (to text / properties)?
   - what should we do if what should happen doesn't?
 - simiarly for C-y anywhere, w/ and w/o prior kill.

thi

Re: Saving markup formats

[Nic James Ferrier]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> here is an example i hope we can use as a "working exercise":
>
>   lists
>   * item1
>   * ===========BIG==================
>     =  foo         =  zzz          =
>     =  * bar       =  * yyy        =
>     =  * baz       =  * ==SMALL==  =
>     =              =    = 3 = 5 =  =
>     =  wow         =    = 4 = 2 =  =   <--
>     =  * zow       =    = 6 = A =  =
>     =    * yow     =    =========  =
>     ================================
>   * item3
>
> each of these pieces of text (chosen to be unique for clarity) has
> a location and a function: lists, item1, BIG, foo, zzz, bar, baz,
> yyy, SMALL, 3, 5, wow, 4, 2, zow, 6, A, yow, item3.  additionally,
> each "=", "*" and " " has its coordinate.

The trouble is (as you point out) this is not going to work because
doing kill-line at char 0,line 5 will presumably kill the entire table
row including the start of the table in column 2.

So the answer to this is not to do wysiwyg layout to this
extent. Colouring or some such could be used to indicate that a
flatter rendering actually represents something else.


It is, as you are suggesting I think, impossible to do the above
without completly changing the way kill, yank, etc... work which rms
doesn't want.

-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Kim F. Storm]

Nic James Ferrier <nferrier@tapsellferrier.co.uk> writes:

>
> It is, as you are suggesting I think, impossible to do the above
> without completly changing the way kill, yank, etc... work which rms
> doesn't want.

We have the yank handler functionality which may help DTRT.

BTW, I think table.el already does some of the magic needed here.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Saving markup formats

[Thien-Thi Nguyen]

() Nic James Ferrier <nferrier@tapsellferrier.co.uk>
() Thu, 21 Jun 2007 11:31:46 +0100

   It is, as you are suggesting I think, impossible to do the
   above without completly changing the way kill, yank,
   etc... work which rms doesn't want.

right.  the trick is not to change completely, but just enough,
preserving respect for history and its lumps.

some pieces already mentioned: xml.el, yank handlers, table.el.

thi

Re: Saving markup formats

[Davis Herring]

> So if you have a paragraph with a style attached to it and you kill it
> and then yank it to somewhere else and it has the same styles attached
> to it.
>
> The problem you're talking about does occur when you have 'embedded
> structure', when you have a paragraph inside a list item for example.

I still think it can happen even without nesting; consider the document
(rendered for email as if it were HTML)

<p class="bigbold">This is important!</p>
<p class="fineprint">This is even more important.</p>

Now that will be rendered in Emacs as

This is important!
This is even more important.

with appropriate faces, and perhaps one more newline between the
sentences.  Now kill from "imp..." to "...more" and what happens?  If the
result (in the buffer, not the kill ring) is a new paragraph with a
mixture of styles, we have a problem because we then cannot tell whether a
style is applied to a paragraph or merely to every character in it.  If
it's something else, we have a problem because the nearby text properties
don't jive.  What am I missing?

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Saving markup formats

[Nic James Ferrier]

"Davis Herring" <herring@lanl.gov> writes:

> <p class="bigbold">This is important!</p>
> <p class="fineprint">This is even more important.</p>
>
> Now that will be rendered in Emacs as
>
> This is important!
> This is even more important.
>
> with appropriate faces, and perhaps one more newline between the
> sentences.  Now kill from "imp..." to "...more" and what happens?  If the
> result (in the buffer, not the kill ring) is a new paragraph with a
> mixture of styles, we have a problem because we then cannot tell whether a
> style is applied to a paragraph or merely to every character in it.  If
> it's something else, we have a problem because the nearby text properties
> don't jive.  What am I missing?

What OpenOffice does when you cut the same text out is strip the style
from "important!" and leave the style on "This is even more".

If you yank it into a para with another style however the new style
wins.

That's what we want I think. It doesn't seem _that_ hard.

Or am I missing something?


-- 
Nic Ferrier
http://www.tapsellferrier.co.uk   

Re: Saving markup formats

[Richard Stallman]

    mixture of styles, we have a problem because we then cannot tell whether a
    style is applied to a paragraph or merely to every character in it.

In Emacs, these two have to be equivalent.

Re: Saving markup formats

[Richard Stallman]

    I'm talking about using properties to mark the syntax of pieces of
    text (in an ordinary buffer) as they are described in a word processor
    schema instance (a document).

Maybe I misunderstood your message.  Using text properties is the
right way to indicate markup, when it can be handled character by
character (which is what text properties do).