bug#49719: 27.2; make-manuals fails

bug#49719: 27.2; make-manuals fails

[Eli Zaretskii]

With the current emacs-27 branch of the Emacs repository, the
'make-manuals' script fails:

  eliz@fencepost:~/git/emacs/pretest$ ./admin/make-manuals
  Making manuals (slow)...
  Search failed: "<ul>"
  make-manuals: error running make-manuals

The error message doesn't help to understand in which file did the
search fail.

This could be due to using Texinfo 6.8: maybe the format of the
produced HTML have changed?

Let me know what other information I can provide to help resolve
this.  TIA.


In GNU Emacs 27.2 (build 2, i686-pc-mingw32)
 of 2021-03-25 built on HOME-C4E4A596F7
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Recent messages:
Saving file d:/gnu/git/emacs/branch/doc/emacs/book-spine.texi...
Wrote d:/gnu/git/emacs/branch/doc/emacs/book-spine.texi
When done with a buffer, type C-x #
Auto-saving...done
Saving file d:/gnu/git/emacs/branch/.git/COMMIT_EDITMSG...
Wrote d:/gnu/git/emacs/branch/.git/COMMIT_EDITMSG
Mark set
Beginning of buffer [2 times]
Quit
Mark saved where search started [3 times]

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int 'CFLAGS=-O2 -gdwarf-4
 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY W32NOTIFY ACL GNUTLS LIBXML2
HARFBUZZ ZLIB TOOLKIT_SCROLL_BARS MODULES THREADS JSON PDUMPER LCMS2
GMP

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: RMAIL

Minor modes in effect:
  shell-dirtrack-mode: t
  desktop-save-mode: t
  save-place-mode: t
  show-paren-mode: t
  display-battery-mode: t
  display-time-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  buffer-read-only: t
  line-number-mode: t

Load-path shadows:
d:/usr/share/emacs/site-lisp/soap-inspect hides d:/usr/share/emacs/27.2/lisp/net/soap-inspect
d:/usr/share/emacs/site-lisp/soap-client hides d:/usr/share/emacs/27.2/lisp/net/soap-client

Features:
(shadow emacsbug lao-util descr-text ibuf-ext ibuffer ibuffer-loaddefs
mule-diag apropos eieio-opt speedbar sb-image ezimage dframe rng-xsd
xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse
nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode
nxml-outln nxml-rap sgml-mode nxml-util nxml-enc xmltok sh-script
executable rx org-element avl-tree ol-eww eww mm-url url-queue
ol-rmail ol-mhe ol-irc ol-info ol-gnus nnir gnus-sum url url-proxy
url-privacy url-expand url-methods url-history mailcap gnus-group
gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc
nnoo parse-time iso8601 gnus-spec gnus-int gnus-range gnus-win gnus
nnheader ol-docview doc-view image-mode exif ol-bibtex bibtex ol-bbdb
ol-w3m org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-footnote org-src ob-comint org-pcomplete org-list org-faces
org-entities org-version ob-emacs-lisp ob-core ob-eval org-table ol
org-keys org-compat advice org-macs org-loaddefs find-func cal-menu
calendar cal-loaddefs tar-mode arc-mode archive-mode repeat help-fns
radix-tree ebuff-menu skeleton rmailout dabbrev rfc2104 gnutls
network-stream nsm mail-extr smtpmail mailalias sendmail pulse
thingatpt etags fileloop generator misearch multi-isearch shell
pcomplete cl-extra help-mode cc-awk conf-mode jka-compr vc-dispatcher
vc-bzr bat-mode rst compile noutline outline make-mode gud octave smie
comint ansi-color texinfo flyspell add-log mule-util info rmailsum
shr-color color shr url-cookie url-domsuf url-util url-parse
auth-source json map url-vars svg xml dom browse-url qp rmailmm
message rmc puny format-spec rfc822 mml mml-sec password-cache epa epg
epg-config gnus-util text-property-search seq mm-decode mm-bodies
mm-encode mailabbrev gmm-utils mailheader mail-parse rfc2231 rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils vc-cvs time-date subr-x vc-git diff-mode easy-mmode
bug-reference desktop frameset server find-lisp dired dired-loaddefs
filecache mairix cus-edit cus-start cus-load wid-edit saveplace
midnight ispell derived generic-x cc-mode cc-fonts easymenu cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs paren
xref cl-seq project ring eieio byte-opt bytecomp byte-compile cconv
eieio-core cl-macs gv eieio-loaddefs cl-loaddefs cl-lib battery time
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese
eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese composite charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads w32notify w32 lcms2
multi-tty make-network-process emacs)

Memory information:
((conses 16 6493449 666594)
 (symbols 48 38544 84)
 (strings 16 297360 30618)
 (string-bytes 1 8745813)
 (vectors 16 91305)
 (vector-slots 8 2087017 575530)
 (floats 8 455 854)
 (intervals 40 1445288 4093)
 (buffers 888 408))

bug#49719: 27.2; make-manuals fails

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> With the current emacs-27 branch of the Emacs repository, the
> 'make-manuals' script fails:
>
>   eliz@fencepost:~/git/emacs/pretest$ ./admin/make-manuals
>   Making manuals (slow)...
>   Search failed: "<ul>"
>   make-manuals: error running make-manuals
>
> The error message doesn't help to understand in which file did the
> search fail.
>
> This could be due to using Texinfo 6.8: maybe the format of the
> produced HTML have changed?

It seems to work fine on Debian/bullseye:

[larsi@stories ~/src/emacs/emacs-27]$ ./admin/make-manuals 
Making manuals (slow)...
Adding compressed html files...
Making manual tarfiles...
Doing emacs...
...created emacs-manual-27.2.50.tar
Doing lispref...
...created emacs-lispref-27.2.50.tar
Doing lispintro...
...created emacs-lispintro-27.2.50.tar
Doing misc...
...created emacs-misc-27.2.50.tar
Making refcards...

But this is with an older makeinfo:

[larsi@stories ~/src/emacs/emacs-27]$ makeinfo --version
texi2any (GNU texinfo) 6.7

Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49719: 27.2; make-manuals fails

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 49719@debbugs.gnu.org
> Date: Sat, 24 Jul 2021 18:37:00 +0200
> 
> [larsi@stories ~/src/emacs/emacs-27]$ ./admin/make-manuals 
> Making manuals (slow)...
> Adding compressed html files...
> Making manual tarfiles...
> Doing emacs...
> ...created emacs-manual-27.2.50.tar
> Doing lispref...
> ...created emacs-lispref-27.2.50.tar
> Doing lispintro...
> ...created emacs-lispintro-27.2.50.tar
> Doing misc...
> ...created emacs-misc-27.2.50.tar
> Making refcards...
> 
> But this is with an older makeinfo:
> 
> [larsi@stories ~/src/emacs/emacs-27]$ makeinfo --version
> texi2any (GNU texinfo) 6.7

Yes, with that version it worked for me as well, back when I updated
the manuals for the v27.2 release.

bug#49719: 27.2; make-manuals fails

[Eli Zaretskii]

> Date: Sat, 24 Jul 2021 19:46:27 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 49719@debbugs.gnu.org
> 
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: 49719@debbugs.gnu.org
> > Date: Sat, 24 Jul 2021 18:37:00 +0200
> > 
> > [larsi@stories ~/src/emacs/emacs-27]$ ./admin/make-manuals 
> > Making manuals (slow)...
> > Adding compressed html files...
> > Making manual tarfiles...
> > Doing emacs...
> > ...created emacs-manual-27.2.50.tar
> > Doing lispref...
> > ...created emacs-lispref-27.2.50.tar
> > Doing lispintro...
> > ...created emacs-lispintro-27.2.50.tar
> > Doing misc...
> > ...created emacs-misc-27.2.50.tar
> > Making refcards...
> > 
> > But this is with an older makeinfo:
> > 
> > [larsi@stories ~/src/emacs/emacs-27]$ makeinfo --version
> > texi2any (GNU texinfo) 6.7
> 
> Yes, with that version it worked for me as well, back when I updated
> the manuals for the v27.2 release.

Nine months later, the script (admin/make-manuals) still doesn't work
with the latest Texinfo.  The end result is that right now, I'm unable
to update the Emacs on-line docs to the version that was released with
Emacs 28.1.  (And no, I don't want to downgrade my Texinfo just to do
that.)

I looked at the code in admin.el, and it seems to be trying to make
the produced HTML more pretty than what Texinfo produces.  While I'm
not objected to having prettier HTML manuals, this is not really our
job, and playing catch-up with the ever-changing HTML that comes from
Texinfo is a maintenance burden.  Moreover, the documentation of the
various "fixes" is minimal at best, and in many places nonexistent (in
the sense that there's no description of the original HTML, the
"improved" one, and the rationale for each "fix"); reverse-engineering
that from code which runs in batch mode is a non-trivial job,
especially since some "fixes" need old versions of Texinfo to be used
to even understand what they are doing.

So I don't think we should continue making these "fixes", unless
Someone™ steps forward and volunteers to keep maintaining this script,
keeping it up-to-date with the latest Texinfo developments, and
generally making sure the script is usable at all times.

I will wait for a day or two for someone to volunteer; failing that I
will remove most or all of the "fixes" from the script, so that it
works with the latest Texinfo, and so I could again refresh our
on-line documentation when needed.

TIA

bug#49719: 27.2; make-manuals fails

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I agree that Emacs should use the output that Texinfo generates.
If we have ideas for how the HTML output could be better,
let's suggest them to the Texinfo developers.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#49719: 27.2; make-manuals fails

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I will wait for a day or two for someone to volunteer; failing that I
> will remove most or all of the "fixes" from the script, so that it
> works with the latest Texinfo, and so I could again refresh our
> on-line documentation when needed.

Removing the "fixes" is fine by me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49719: 27.2; make-manuals fails

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 49719@debbugs.gnu.org
> Date: Wed, 06 Apr 2022 11:21:17 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I will wait for a day or two for someone to volunteer; failing that I
> > will remove most or all of the "fixes" from the script, so that it
> > works with the latest Texinfo, and so I could again refresh our
> > on-line documentation when needed.
> 
> Removing the "fixes" is fine by me.

I eventually went with the minimal change: the function that fails due
to changes in how Texinfo 6.8 produces HTML was modified to skip the
problematic step of rewriting the TOC menu as a table (something that
wasn't possible to keep, because some of the information used for the
table is no longer there in the new HTML).  The rest of the "fixes" I
kept.

With that, I'm closing this bug.