makeinfo swallows page breaks

makeinfo swallows page breaks

[Ludovic Courtès]

Hello,

Guile (ab)uses ‘makeinfo’ to generate a file, guile-procedures.txt, from
Texinfo.  That file is then read at run-time to get docstrings for
Guile’s primitives.

The input looks like this (note the page breaks, aka. ^L):

--8<---------------cut here---------------start------------->8---

\facons
@c snarfed from alist.c:40
@deffn {Scheme Procedure} acons key value alist
Add a new key-value pair to @var{alist}.  A new pair is
created whose car is @var{key} and whose cdr is @var{value}, and the
pair is consed onto @var{alist}, and the new list is returned.  This
function is @emph{not} destructive; @var{alist} is not modified.
@end deffn

\fsloppy-assq
@c snarfed from alist.c:54
@deffn {Scheme Procedure} sloppy-assq key alist
Behaves like @code{assq} but does not do any error checking.
Recommended only for use in Guile internals.
@end deffn

--8<---------------cut here---------------end--------------->8---

With Texinfo 4.3, page breaks are preserved:

--8<---------------cut here---------------start------------->8---
This is guile-procedures.txt, produced by makeinfo version 4.13 from
guile-procedures.texi.

\facons

 -- Scheme Procedure: acons key value alist
     Add a new key-value pair to ALIST.  A new pair is created whose
     car is KEY and whose cdr is VALUE, and the pair is consed onto
     ALIST, and the new list is returned.  This function is _not_
     destructive; ALIST is not modified.

   \fsloppy-assq

 -- Scheme Procedure: sloppy-assq key alist
     Behaves like `assq' but does not do any error checking.
     Recommended only for use in Guile internals.

--8<---------------cut here---------------end--------------->8---

With Texinfo 5.[01], page breaks vanish:

--8<---------------cut here---------------start------------->8---
This is guile-procedures.txt, produced by makeinfo version 5.1 from
guile-procedures.texi.

acons
 -- Scheme Procedure: acons key value alist
     Add a new key-value pair to ALIST.  A new pair is created whose car
     is KEY and whose cdr is VALUE, and the pair is consed onto ALIST,
     and the new list is returned.  This function is _not_ destructive;
     ALIST is not modified.

   sloppy-assq
 -- Scheme Procedure: sloppy-assq key alist
     Behaves like 'assq' but does not do any error checking.
     Recommended only for use in Guile internals.
--8<---------------cut here---------------end--------------->8---

Unfortunately, those page breaks are used by Guile’s docstring parser to
determine procedure boundaries.

Would it be possible to reinstate previous behavior–i.e., to preserve
page breaks?

Alternately, would you suggest using a different approach?

Thanks,
Ludo’.

Re: makeinfo swallows page breaks

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

ludo@gnu.org (Ludovic Courtès) skribis:

> Alternately, would you suggest using a different approach?

Attached is a patch to use Guile’s Texinfo support [0] to build said
file.  Guile’s Texinfo parser is incomplete but sufficient to handle
those docstrings.

It solves an actual bug for anyone using Texinfo 5.x.  It also seems to
be the most profitable solution for us Guile hackers in the long run.

And, it’s much faster:

  makeinfo 4.13 takes 0.1s
  makeinfo 5.1 takes 11.7s
  Guile takes         1.3s

OK to commit?

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 5902 bytes --]

From 84b2457ed9fdf866a354465c67e282f8fbb30eb3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 21 Mar 2013 19:17:13 +0100
Subject: [PATCH] Build `guile-procedures.txt' using (texinfo) instead of
 `makeinfo'.

* Makefile.am (schemelibdir, schemelib_DATA): New variables.
  (libguile/guile-procedures.txt): New target.
  (EXTRA_DIST): Add libguile/texi-fragments-to-docstrings.
* libguile/Makefile.am (guile-procedures.txt): Remove target.
  (schemelibdir, schemelib_DATA): Remove.
* libguile/texi-fragments-to-docstrings: New file.
---
 Makefile.am                           |   17 ++++++++--
 libguile/Makefile.am                  |   16 ----------
 libguile/texi-fragments-to-docstrings |   55 +++++++++++++++++++++++++++++++++
 3 files changed, 70 insertions(+), 18 deletions(-)
 create mode 100644 libguile/texi-fragments-to-docstrings

diff --git a/Makefile.am b/Makefile.am
index 3aa5ddd..737897b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,7 +1,7 @@
 ## Process this file with automake to produce Makefile.in.
 ##
 ##  	Copyright (C) 1998, 1999, 2000, 2001, 2002, 2006, 2007,
-##        2008, 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+##        2008, 2009, 2010, 2011, 2012, 2013 Free Software Foundation, Inc.
 ##
 ##   This file is part of GUILE.
 ##
@@ -42,6 +42,18 @@ SUBDIRS =					\
 libguileincludedir = $(pkgincludedir)/$(GUILE_EFFECTIVE_VERSION)
 libguileinclude_HEADERS = libguile.h
 
+schemelibdir = $(pkgdatadir)/$(GUILE_EFFECTIVE_VERSION)
+schemelib_DATA = libguile/guile-procedures.txt
+
+# Build it from here so that all the modules are compiled by the time we
+# build it.
+libguile/guile-procedures.txt: libguile/guile-procedures.texi
+	$(AM_V_GEN)						\
+	$(top_builddir)/meta/guile --no-auto-compile		\
+	  "$(srcdir)/libguile/texi-fragments-to-docstrings"	\
+	  "$(builddir)/libguile/guile-procedures.texi"		\
+	  > libguile/guile-procedures.txt
+
 EXTRA_DIST = LICENSE HACKING GUILE-VERSION			\
 	     m4/ChangeLog-2008					\
 	     m4/gnulib-cache.m4					\
@@ -50,7 +62,8 @@ EXTRA_DIST = LICENSE HACKING GUILE-VERSION			\
 	     gnulib-local/lib/localcharset.h.diff		\
 	     gnulib-local/lib/localcharset.c.diff		\
 	     gnulib-local/m4/clock_time.m4.diff			\
-	     gnulib-local/build-aux/git-version-gen.diff
+	     gnulib-local/build-aux/git-version-gen.diff	\
+	     libguile/texi-fragments-to-docstrings
 
 TESTS = check-guile
 TESTS_ENVIRONMENT = @LOCALCHARSET_TESTS_ENVIRONMENT@
diff --git a/libguile/Makefile.am b/libguile/Makefile.am
index d77bdfe..450d955 100644
--- a/libguile/Makefile.am
+++ b/libguile/Makefile.am
@@ -713,25 +713,9 @@ guile.texi: $(alldotdocfiles) guile$(EXEEXT)
 guile-procedures.texi: $(alldotdocfiles) guile$(EXEEXT)
 	$(AM_V_GEN)$(dotdoc2texi)          > $@ || { rm $@; false; }
 
-if HAVE_MAKEINFO
-
-guile-procedures.txt: guile-procedures.texi
-	rm -f $@
-	makeinfo --force -o $@ guile-procedures.texi || test -f $@
-
-else
-
-guile-procedures.txt: guile-procedures.texi
-	cp guile-procedures.texi $@
-
-endif
-
 c-tokenize.c: c-tokenize.lex
 	flex -t $(srcdir)/c-tokenize.lex > $@ || { rm $@; false; }
 
-schemelibdir = $(pkgdatadir)/$(GUILE_EFFECTIVE_VERSION)
-schemelib_DATA = guile-procedures.txt
-
 ## Add -MG to make the .x magic work with auto-dep code.
 MKDEP = gcc -M -MG $(DEFS) $(AM_CPPFLAGS) $(CPPFLAGS) $(CFLAGS)
 
diff --git a/libguile/texi-fragments-to-docstrings b/libguile/texi-fragments-to-docstrings
new file mode 100644
index 0000000..b72390b
--- /dev/null
+++ b/libguile/texi-fragments-to-docstrings
@@ -0,0 +1,55 @@
+;;; -*- mode: scheme; coding: utf-8; -*-
+;;;
+;;; Copyright (C) 2013 Free Software Foundation, Inc.
+;;;
+;;; This library is free software; you can redistribute it and/or
+;;; modify it under the terms of the GNU Lesser General Public
+;;; License as published by the Free Software Foundation; either
+;;; version 3 of the License, or (at your option) any later version.
+;;;
+;;; This library is distributed in the hope that it will be useful,
+;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+;;; Lesser General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU Lesser General Public
+;;; License along with this library; if not, write to the Free Software
+;;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+\f
+;;;
+;;; Read Texinfo fragments from stdin (docstrings of Guile's primitives
+;;; in the format of `guile-procedures.texi'), and write to stdout a
+;;; textual rendering thereof.  The output preserves page breaks (^L)
+;;; found in the input, as per the Guile Documentation Format
+;;; version 2---see (ice-9 documentation).
+;;;
+
+(use-modules (texinfo)
+             (texinfo plain-text)
+             (srfi srfi-1)
+             (ice-9 match)
+             (rnrs io ports))
+
+(define (docstring-fragments->strings str)
+  "Return the list resulting from the split of STR at each page
+break (^L)"
+  (string-tokenize str (char-set-complement (char-set #\page))))
+
+(match (command-line)
+  ((_ texi-file)
+   (let* ((fragments (remove (compose string-null? string-trim-both)
+                             (call-with-input-file texi-file
+                               (compose docstring-fragments->strings
+                                        get-string-all))))
+          (stexi     (map texi-fragment->stexi fragments)))
+     (format #t "Produced by GNU Guile ~a from `~a'.~%~%"
+             (version) texi-file)
+     (for-each (lambda (stexi)
+                 (display #\page)
+                 (display (stexi->plain-text stexi)))
+               stexi)))
+  ((command args ...)
+   (format (current-error-port) "invalid arguments: ~s~%" args)
+   (format (current-error-port) "Usage: ~a TEXINFO-FILE~%" command)
+   (exit 1)))
-- 
1.7.10.4

Re: makeinfo swallows page breaks

[Mike Gran]

Hi Ludo



> Attached is a patch to use Guile’s Texinfo support [0] to build said
> file.  Guile’s Texinfo parser is incomplete but sufficient to handle
> those docstrings.

> OK to commit?

If Guile depends on Guile for multiple stages the build, it becomes
difficult to recover from a problem when the build is failing.

So maybe this commit will lead to better, more methodical hacking; maybe
it will make fixing a broken Guile more annoying.

Dunno.

-Mike

Re: makeinfo swallows page breaks

[Ludovic Courtès]

Mike Gran <spk121@yahoo.com> skribis:

>> Attached is a patch to use Guile’s Texinfo support [0] to build said
>> file.  Guile’s Texinfo parser is incomplete but sufficient to handle
>> those docstrings.
>
>> OK to commit?
>
> If Guile depends on Guile for multiple stages the build, it becomes
> difficult to recover from a problem when the build is failing.

Well, that’s nothing compared to its bootstrapping procedure.  :-)

> So maybe this commit will lead to better, more methodical hacking; maybe
> it will make fixing a broken Guile more annoying.

It will force us to maintain the Texinfo modules at a reasonable level,
which I think it’s good.  Hopefully, if something breaks, it will break
deterministically.

Conversely, here people with Texinfo 5.x get no docstrings (and thus
failing tests), people with an older Texinfo don’t have any problems,
and people without Texinfo have no problem but their docstrings include
unprocessed markup.

Overall, I really think it’s an improvement.

Ludo’.

Re: makeinfo swallows page breaks

[Karl Berry]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

    Guileœôòùs Texinfo parser

Argh.  The idea of a full second Texinfo parser in GNU is fundamentally
wrong.  If you want to call it a "Guile docstring parser, whose language
happens to bear some resemblance to a subset of Texinfo", fine.

Anyway, your change to use more Guile in the Guile build procedures
makes sense to me.

k

Re: makeinfo swallows page breaks

[Andy Wingo]

On Thu 21 Mar 2013 22:56, karl@freefriends.org (Karl Berry) writes:

>     Guile’s Texinfo parser
>
> Argh.  The idea of a full second Texinfo parser in GNU is fundamentally
> wrong.

I understand where you come from.  For some context on why I wrote this
parser back in 2004:

  http://wingolog.org/archives/2006/11/17/high-on-sodium-vapor

I even had a gtk browser at one point with help:

  http://wingolog.org/pub/texinfo-browser.png

I had a GTK app then with context-sensitive interactive help.  So I
needed to treat docs programmatically.

> If you want to call it a "Guile docstring parser, whose language
> happens to bear some resemblance to a subset of Texinfo", fine.

It is intended to be a strict subset.  To do otherwise we would have to
make a new format, and that would be bad.

Cheers,

Andy
-- 
http://wingolog.org/

Re: makeinfo swallows page breaks

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Attached is a patch to use Guile’s Texinfo support [0] to build said
> file.  Guile’s Texinfo parser is incomplete but sufficient to handle
> those docstrings.

I went ahead and pushed this after discussion with Mark and Andy on IRC.

Regarding rendering: Andy noted that spaces after a sentence-ending
periods are not preserved.  I’ve investigated, but there are tests that
assume that whitespace is collapsed, so that will require more thought.

Ludo’.

Re: makeinfo swallows page breaks

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Regarding rendering: Andy noted that spaces after a sentence-ending
> periods are not preserved.  I’ve investigated, but there are tests that
> assume that whitespace is collapsed, so that will require more thought.

More precisely, Texinfo says (info "(texinfo) Multiple Spaces"):

  Ordinarily, TeX collapses multiple whitespace characters (space, tab,
  and newline) into a single space.  Info output, on the other hand,
  preserves whitespace as you type it, except for changing a newline into
  a space; this is why it is important to put two spaces at the end of
  sentences in Texinfo documents.

So I believe that it means that stexi should preserve whitespace (which
it doesn’t currently), and that it’s the renderer’s job to trim
whitespace or not.

Tests in texinfo.test will have to be adjusted accordingly.

Currently, the parser consumes whitespace:

  scheme@(guile-user)> (texi-fragment->stexi "foo.  bar.")
  $2 = (*fragment* (para "foo. bar."))

WDYT?

Ludo’.

Re: makeinfo swallows page breaks

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Regarding rendering: Andy noted that spaces after a sentence-ending
> periods are not preserved.  I’ve investigated, but there are tests that
> assume that whitespace is collapsed, so that will require more thought.

After discussion on IRC, I committed 43c2a48, which just modifies line
wrapping (used by the plain-text renderer) to ensure there are two
spaces after end-of-sentence periods.

Ludo’.