`texinfo-format-buffer' doesn't fold long lines

`texinfo-format-buffer' doesn't fold long lines

[Katsumi Yamaoka]

Hi,

In some cases, `texinfo-format-buffer' doesn't fold long lines.
For example, try the following and you can see how it does.

;--8<---------------cut here---------------start------------->8---
(with-temp-buffer
  (insert "@setfilename testing
@node Top
@itemize @bullet
@item
foo
@enumerate 1
@item
bar
@enumerate a
@item
baz
The quick brown fox jumps over the lazy dog. The quick brown fox jumps
over the lazy dog.
@end enumerate
@end enumerate
@end itemize
")
  (texinfo-format-buffer)
  (setq buffer-file-name nil))
;--8<---------------cut here---------------end--------------->8---

There's no such problem in Emacs 21.3.  It happens because someone
added "itemize\\|" to `texinfo-no-refill-regexp' as follows:

(defvar texinfo-no-refill-regexp
  (concat
   "^@"
   "\\("
   ;; add "itemize\\|"   (from experiment of 2001 Nov 28)
   ;;     because of a problem with @end itemize@refill
   ;;     I don't know if this causes other problems.
   ;;     I suspect itemized lists don't get filled properly and a
   ;;     more precise fix is required.  Bob
   "itemize\\|"
   "direntry\\|"

[...]

Although I don't have a right understanding of the reason it is
necessary, I hope for it to be mended anyway.  It is because
texinfmt is essential to format Japanese Info files.

Regards,

Re: `texinfo-format-buffer' doesn't fold long lines

[Robert J. Chassell]

Katsumi Yamaoka <yamaoka@jpl.org> wrote,

   In some cases, `texinfo-format-buffer' doesn't fold long lines.

   ... It happens because someone added "itemize\\|" to
   `texinfo-no-refill-regexp' as follows:

   (defvar texinfo-no-refill-regexp

Yes, the inclusion of "itemize\\|" tells Emacs not to refill lines in
an itemized list.

I have commented out that inclusion and committed the change.

So `texinfo-format-buffer' should work again as you expect.

As I wrote at the time I included it

    ... I don't think this fix should be used as is.

Evidentally, back in November 2001, a user in FSF India could not
format the then new edition of `emacs-lisp-intro.texi' using
`texinfo-format-buffer' because `texinfmt.el' created the bad line

    @end itemize@refill

He was using a Microsoft operating system and thought (erroneously,
according to Eli Zaretskii) that `makeinfo' had not yet been ported.
That is why he was using the deprecated `texinfo-format-buffer'
command.

In any event, I completely forgot about it.

You write that 

    ... texinfmt is essential to format Japanese Info files.

What needs to be changed for `makeinfo' to format them?

As Eli wrote back then,

    I'd rather advise that users to use makeinfo.  texinfmt.el doesn't
    support quite a few of the newer features ...

We should modify `makeinfo' to handle Japanese Texinfo, if need be.
`makeinfo' is faster and `texinfmt.el' was deprecated a long time ago.

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: `texinfo-format-buffer' doesn't fold long lines

[Katsumi Yamaoka]

>>>>> In <m1D5mYY-0004QmC@rattlesnake.com> Robert J. Chassell wrote:

> I have commented out that inclusion and committed the change.
> So `texinfo-format-buffer' should work again as you expect.

Thank you very much for that.  It works fine now.

> As I wrote at the time I included it

>     ... I don't think this fix should be used as is.

> Evidentally, back in November 2001, a user in FSF India could not
> format the then new edition of `emacs-lisp-intro.texi' using
> `texinfo-format-buffer' because `texinfmt.el' created the bad line

>     @end itemize@refill

> He was using a Microsoft operating system and thought (erroneously,
> according to Eli Zaretskii) that `makeinfo' had not yet been ported.
> That is why he was using the deprecated `texinfo-format-buffer'
> command.

> In any event, I completely forgot about it.

I see, and I will have to keep in mind that there might still be
need to fix the `@end itemize@refill' problem.

> You write that 

>     ... texinfmt is essential to format Japanese Info files.

> What needs to be changed for `makeinfo' to format them?

It is to support text encoded with the iso-2022-7bit coding
system, etc.  The raw data of those will contain `@', `{', and
so forth, and makeinfo will misidentify them as the commands.
Please note, coding systems used to encode Japanese text
includes not only iso-2022-7bit but also euc-jp, shift_jis and
utf-8, etc.  Furthermore, Chinese and Korean might be similar,
though I'm not familiar with them.  There will be 8bit data
which makeinfo does not expect.

Probably, what makeinfo should do first will be to decode text,
before formatting them.  And the means to know the coding system
which is used to decode text will be necessary.

> As Eli wrote back then,

>     I'd rather advise that users to use makeinfo.  texinfmt.el doesn't
>     support quite a few of the newer features ...

> We should modify `makeinfo' to handle Japanese Texinfo, if need be.
> `makeinfo' is faster and `texinfmt.el' was deprecated a long time ago.

It is just what I am hoping!  I always use the makeinfo command
in order to format texi files written by US-ASCII and Latin text
and rate it highly.

Regards,

Re: `texinfo-format-buffer' doesn't fold long lines

[Eli Zaretskii]

> Date: Tue, 01 Mar 2005 08:57:07 +0900
> From: Katsumi Yamaoka <yamaoka@jpl.org>
> Cc: emacs-devel@gnu.org
> 
> >     ... I don't think this fix should be used as is.
> 
> > Evidentally, back in November 2001, a user in FSF India could not
> > format the then new edition of `emacs-lisp-intro.texi' using
> > `texinfo-format-buffer' because `texinfmt.el' created the bad line
> 
> >     @end itemize@refill
> 
> > He was using a Microsoft operating system and thought (erroneously,
> > according to Eli Zaretskii) that `makeinfo' had not yet been ported.
> > That is why he was using the deprecated `texinfo-format-buffer'
> > command.
> 
> > In any event, I completely forgot about it.
> 
> I see, and I will have to keep in mind that there might still be
> need to fix the `@end itemize@refill' problem.

I'd rather not have us fix a problem by reintroducing another problem
that we fixed in the past.  Isn't there another way to solve both of
them at once?

> Probably, what makeinfo should do first will be to decode text,
> before formatting them.  And the means to know the coding system
> which is used to decode text will be necessary.

There's already such a means: the @documentencoding directive.  What
is missing is support for multibyte encodings.

Re: `texinfo-format-buffer' doesn't fold long lines

[Katsumi Yamaoka]

>>>>> In <01c51e18$Blat.v2.4$512429a0@zahav.net.il> Eli Zaretskii wrote:

>> I see, and I will have to keep in mind that there might still be
>> need to fix the `@end itemize@refill' problem.

> I'd rather not have us fix a problem by reintroducing another problem
> that we fixed in the past.  Isn't there another way to solve both of
> them at once?

>> Probably, what makeinfo should do first will be to decode text,
>> before formatting them.  And the means to know the coding system
>> which is used to decode text will be necessary.

> There's already such a means: the @documentencoding directive.  What
> is missing is support for multibyte encodings.

I already tried it and failed.  The line

@documentencoding ISO-2022-JP

is not effective in the Japanese text encoded by the iso-2022-jp
coding system.  euc-jp and euc-japan or shift_jis, shift-jis and
sjis also do not the trick.  They cause an error as follows:

emacs-w3m-ja.texi:8: warning: unrecognized encoding name `ISO-2022-JP'.

Is the makeinfo v4.8 too old?  It misidentifies `@' contained in
the raw file contents as the commands.

Regards,

Re: `texinfo-format-buffer' doesn't fold long lines

[Eli Zaretskii]

> Date: Tue, 01 Mar 2005 14:32:31 +0900
> From: Katsumi Yamaoka <yamaoka@jpl.org>
> Cc: bob@rattlesnake.com, emacs-devel@gnu.org
> 
> > There's already such a means: the @documentencoding directive.  What
> > is missing is support for multibyte encodings.
> 
> I already tried it and failed.  The line
> 
> @documentencoding ISO-2022-JP
> 
> is not effective in the Japanese text encoded by the iso-2022-jp
> coding system.

But I just told you: the support for multibyte encodings is missing.
In other words, I already know that ISO-2022-JP is not supported;
someone should add that support, before ISO-2022-JP encoded files will
begin to work in Texinfo.

> Is the makeinfo v4.8 too old?

It's the latest, but ISO-2022-JP support is not part of it (and I
don't think anyone is working on adding such a support).  Maybe you
could find a volunteer among Japanese users of Texinfo.

Re: `texinfo-format-buffer' doesn't fold long lines

[Robert J. Chassell]

    > I see, and I will have to keep in mind that there might still be
    > need to fix the `@end itemize@refill' problem.

    I'd rather not have us fix a problem by reintroducing another
    problem that we fixed in the past.  Isn't there another way to
    solve both of them at once?

Here is a fix to both at once; at least I hope so.

Basically, in emacs/lisp/textmodes/texinfmt.el, near the end of
texinfo-append-refill, I replaced the current search for

    @refill\\|@bye

with

    @refill\\|^[ \t]*@

This defines the types of line to which @refill is *not* appended.
(`@bye' should be on a line of its own; `@refill' need not be.)
The change enables a writer to nest @itemize and similar entries like
this:

    @itemize @bullet
    @item foo
      @itemize @minus
      @item bar
      @end itemize
    @end itemize

I also updated the texinfmt-version number and date.


Katsumi Yamaoka: does this change work with your files?


diff -rc2P /usr/local/src/emacs/lisp/textmodes/texinfmt.el.\~1.76.\~ /usr/local/src/emacs/lisp/textmodes/texinfmt.el
*** /usr/local/src/emacs/lisp/textmodes/texinfmt.el.~1.76.~	Tue Mar  1 12:26:36 2005
--- /usr/local/src/emacs/lisp/textmodes/texinfmt.el	Tue Mar  1 15:19:50 2005
***************
*** 38,42 ****
        `(defvar ,var ,value ,doc)))
  
! (defvar texinfmt-version "2.40 of  6 Dec 2002")
  
  (defun texinfmt-version (&optional here)
--- 38,42 ----
        `(defvar ,var ,value ,doc)))
  
! (defvar texinfmt-version "2.41 of  1 Mar 2005")
  
  (defun texinfmt-version (&optional here)
***************
*** 638,642 ****
  	    (unless (re-search-backward "@c[ \t\n]\\|@comment[ \t\n]" line-beg t)
  	      (forward-char -1))
! 	    (unless (re-search-backward "@refill\\|@bye" line-beg t)
  	      (insert "@refill")))
            (forward-line 1))))))
--- 638,642 ----
  	    (unless (re-search-backward "@c[ \t\n]\\|@comment[ \t\n]" line-beg t)
  	      (forward-char -1))
! 	    (unless (re-search-backward "@refill\\|^[ \t]*@" line-beg t)
  	      (insert "@refill")))
            (forward-line 1))))))

-- 
    Robert J. Chassell                         
    bob@rattlesnake.com                         GnuPG Key ID: 004B4AC8
    http://www.rattlesnake.com                  http://www.teak.cc

Re: `texinfo-format-buffer' doesn't fold long lines

[Richard Stallman]

    > We should modify `makeinfo' to handle Japanese Texinfo, if need be.
    > `makeinfo' is faster and `texinfmt.el' was deprecated a long time ago.

    It is just what I am hoping!

Have you reported this problem to bug-texinfo@gnu.org?
Would you like to work on code for Makeinfo to do this?

Re: `texinfo-format-buffer' doesn't fold long lines

[Katsumi Yamaoka]

>>>>> In <E1D6RwK-0004zV-8M@fencepost.gnu.org> Richard Stallman wrote:

>> We should modify `makeinfo' to handle Japanese Texinfo, if need be.
>> `makeinfo' is faster and `texinfmt.el' was deprecated a long time ago.

>     It is just what I am hoping!

> Have you reported this problem to bug-texinfo@gnu.org?

No, I haven't done.

> Would you like to work on code for Makeinfo to do this?

I'm not equal to that.  I heard someone has tried to modify it
so as to support Japanese text years ago, however it didn't seem
to have been successful.  There are many programs which don't
support Japanese in practice.  `makeinfo' is merely one of them.

Re: `texinfo-format-buffer' doesn't fold long lines

[Richard Stallman]

    > Have you reported this problem to bug-texinfo@gnu.org?

    No, I haven't done.

Please send a clear and specific bug report
with a test case, so they can work on this.

Re: `texinfo-format-buffer' doesn't fold long lines

[Katsumi Yamaoka]

>>>>> In <E1D6o7U-0001xW-4u@fencepost.gnu.org> Richard Stallman wrote:

>> Have you reported this problem to bug-texinfo@gnu.org?

>     No, I haven't done.

> Please send a clear and specific bug report
> with a test case, so they can work on this.

I'll do that tomorrow.  Thanks.

support of Japanese in makeinfo

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 2290 bytes --]

Hello,

I was encouraged by Richard Stallman and send this message.  I'm
participating in the development of some Emacs Lisp softwares and
have the chance to write the TexInfo manuals in Japanese.  Since
the makeinfo command doesn't support Japanese text, I always put
the rule which uses the Emacs Lisp based makeinfo, that is, the
texinfo-format-buffer function, into those Makefiles.  Probably,
developers in China and Korea will do the same because of the
same reason, although I'm not familiar with those languages.

Is there a chance that the makeinfo command supports Japanese
text?  I heard someone in Japan has tried to modify it so as to
support Japanese text years ago, however it didn't seem to have
been successful.  I wish the makeinfo command could support
languages which correspond to many locales defined in the
/usr/lib/locale directory.  Unfortunately, I'm not good at C.

There are four encodings which are generally used for Japanese
text; they are ISO-2022-JP, EUC-JP, SHIFT_JIS and UTF-8.
ISO-2022-JP uses only 7-bit data, but some characters which are
special to makeinfo, for example, `@' and `{', will appear in
the raw data.  EUC-JP uses only 8-bit data, is popular in the
Unix-like systems as well as ISO-2022-JP.  SHIFT_JIS uses a
mixture of 7-bit and 8-bit data, mainly used in MS Windows.

In addition, there are special conventions used in Japanese
text.  The typical ones are as follows:

1. Japanese words are not separated by spaces as if they look
   "Japanesewordsarenotseparatedbyspaces".

2. Lines can be broken even in the middle of a word.  However,
   ASCII words embedded in Japanese sentences cannot be broken.

3. There's no space between the Japanese comma or the Japanese
   period and the beginning of the next sentence.

4. There are special rules called `kinsoku'.  Some characters
   including Japanese comma, period and dash, small kana
   characters and the right parenthesis should not be places at
   the beginning of lines.  Contrarily, the left parenthesis and
   so forth should not be placed at the end of lines.  In Emacs,
   they are defined in the lisp/international/kinsoku.el file.

I would greatly appreciate making the makeinfo command support
them.  I attach an example of the Japanese TexInfo file below.

Best regards,


[-- Attachment #2: jugemu.texi.gz --]
[-- Type: application/x-gzip, Size: 380 bytes --]

\x1f

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: `texinfo-format-buffer' doesn't fold long lines

[Richard Stallman]

    In some cases, `texinfo-format-buffer' doesn't fold long lines.

texinfo-format-buffer is more or less obsolete; you should use
makeinfo instead.

Re: `texinfo-format-buffer' doesn't fold long lines

[Stefan Monnier]

>     In some cases, `texinfo-format-buffer' doesn't fold long lines.
> texinfo-format-buffer is more or less obsolete; you should use
> makeinfo instead.

Should we move it lisp/obsolete?


        Stefan

Re: `texinfo-format-buffer' doesn't fold long lines

[Katsumi Yamaoka]

>>>>> In <87bra49w0p.fsf-monnier+emacs@gnu.org> Stefan Monnier wrote:

>>     In some cases, `texinfo-format-buffer' doesn't fold long lines.
>> texinfo-format-buffer is more or less obsolete; you should use
>> makeinfo instead.

> Should we move it lisp/obsolete?

No, please never do so!  The makeinfo command doesn't support
yet texi files written by non-ASCII text.  I can submit an
example of its not working if it is necessary.  Let me write
again; texinfmt is essential to format at least Japanese Info
files.

Regards,

Re: `texinfo-format-buffer' doesn't fold long lines

[Richard Stallman]

    Should we move it lisp/obsolete?

Apparently it is still useful for something, so I guess not.
At least not now.