Re: master 20fa40d: Improve doc for web pages; reproducible tarballs

Re: master 20fa40d: Improve doc for web pages; reproducible tarballs

[Glenn Morris]

Paul Eggert wrote:

> branch: master
> commit 20fa40ddd3e49b73cf2127a147d3b4ee03a3b5ba

> --- a/admin/make-tarball.txt
> +++ b/admin/make-tarball.txt
[...]
> +tar='tar --numeric-owner --owner=0 --group=0 --mode=go+u,go-w --sort=name'

> --- a/make-dist
> +++ b/make-dist

> +  taropt='--numeric-owner --owner=0 --group=0 --mode=go+u,go-w --sort=name'

This breaks make-dist on platforms where tar does not support --sort=name.
Eg RHEL 7.4 with tar (GNU tar) 1.26; and on hydra:

https://hydra.nixos.org/build/72309149

(The build bogusly reports success, but the created tar file is empty
and all derived builds fail.)

Re: master 20fa40d: Improve doc for web pages; reproducible tarballs

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 174 bytes --]

Glenn Morris wrote:
> This breaks make-dist on platforms where tar does not support --sort=name.

Thanks for reporting that. I installed the attached, which I hope fixes it.

[-- Attachment #2: 0001-Port-make-dist-to-tar-1.27-and-earlier.patch --]
[-- Type: text/x-patch, Size: 1617 bytes --]

From 69e3af0f6df4093c3e587261dbee64c3f749d47f Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 1 Apr 2018 13:53:15 -0700
Subject: [PATCH] Port make-dist to tar 1.27 and earlier

Problem reported by Glenn Morris in:
https://lists.gnu.org/r/emacs-devel/2018-03/msg00984.html
* make-dist: Use --sort=name only if Tar supports it.
---
 admin/make-tarball.txt | 3 ++-
 make-dist              | 4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/admin/make-tarball.txt b/admin/make-tarball.txt
index bdf7b8c..f9ad217 100644
--- a/admin/make-tarball.txt
+++ b/admin/make-tarball.txt
@@ -210,7 +210,8 @@ longer present.
 
 Tar up the generated html_node/emacs/ and elisp/ directories and update
 the files manual/elisp.html_node.tar.gz and emacs.html_node.tar.gz.
-Use GNU Tar as follows so that the tarballs are reproducible:
+Use GNU Tar 1.28 or later so that the tarballs are more reproducible,
+as follows:
 
   cd manual
   tar='tar --numeric-owner --owner=0 --group=0 --mode=go+u,go-w --sort=name'
diff --git a/make-dist b/make-dist
index 48c7fb4..d651e18 100755
--- a/make-dist
+++ b/make-dist
@@ -643,7 +643,9 @@ files=
        *)  gzip_extension= ;;
   esac
   echo "Creating tar file"
-  taropt='--numeric-owner --owner=0 --group=0 --mode=go+u,go-w --sort=name'
+  taropt='--numeric-owner --owner=0 --group=0 --mode=go+u,go-w'
+  tar --sort=name -cf /dev/null $tempparent/$emacsname/src/lisp.h 2>/tmp/out &&
+    taropt="$taropt --sort=name"
   [ "$verbose" = "yes" ] && taropt="$taropt --verbose"
 
   (cd ${tempparent} ; tar $taropt -cf - ${emacsname} ) \
-- 
2.7.4

Re: master 20fa40d: Improve doc for web pages; reproducible tarballs

[Glenn Morris]

Thanks.
I think this demonstrates that make-dist should exit with a non-zero
status if the final tar command failed.

Re: master 20fa40d: Improve doc for web pages; reproducible tarballs

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On 04/02/2018 12:13 PM, Glenn Morris wrote:
> make-dist should exit with a non-zero
> status if the final tar command failed.

Good point, and there are a lot of other places where make-dist 
incorrectly ignores exit status. I attempted to fix low-hanging fruit in 
this area by installing the attached patch into master.


[-- Attachment #2: 0001-make-dist-check-exit-statuses-more-carefully.patch --]
[-- Type: text/x-patch, Size: 14116 bytes --]

From 394dc613521e7d2f4074b3311a5c8da6945e655e Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 9 Apr 2018 13:34:44 -0700
Subject: [PATCH] make-dist: check exit statuses more carefully

* make-dist: Do a better job checking for subprocess failure.
---
 make-dist | 250 ++++++++++++++++++++++++++++++++++++--------------------------
 1 file changed, 144 insertions(+), 106 deletions(-)

diff --git a/make-dist b/make-dist
index d651e18b83..7fd227862b 100755
--- a/make-dist
+++ b/make-dist
@@ -346,7 +346,7 @@ tempparent=
 
 echo "Creating staging directory: '${tempparent}'"
 
-mkdir ${tempparent}
+mkdir ${tempparent} || exit
 tempdir="${tempparent}/${emacsname}"
 
 ### This trap ensures that the staging directory will be cleaned up even
@@ -356,7 +356,7 @@ tempdir=
 fi
 
 echo "Creating top directory: '${tempdir}'"
-mkdir ${tempdir}
+mkdir ${tempdir} || exit
 
 if [ "$changelog" = yes ]; then
   if test -r .git; then
@@ -379,10 +379,13 @@ tempdir=
 ### tar file; this means that people can start reading the INSTALL and
 ### README while the rest of the tar file is still unpacking.  Whoopee.
 echo "Making links to top-level files"
-ln INSTALL README BUGS ${tempdir}
-ln ChangeLog.*[0-9] Makefile.in autogen.sh configure configure.ac ${tempdir}
-ln config.bat make-dist .dir-locals.el ${tempdir}
-ln aclocal.m4 CONTRIBUTE ChangeLog ${tempdir}
+top_level='
+  INSTALL README BUGS
+  ChangeLog.*[0-9] Makefile.in autogen.sh configure configure.ac
+  config.bat make-dist .dir-locals.el
+  aclocal.m4 CONTRIBUTE ChangeLog
+'
+ln $top_level $tempdir || exit
 
 echo "Creating subdirectories"
 for subdir in site-lisp \
@@ -412,132 +415,151 @@ tempdir=
   [ "$subdir" = "site-lisp" ] || [ -d "$subdir" ] || \
       echo "WARNING: $subdir not found, making anyway"
   [ "$verbose" = "yes" ] && echo "  ${tempdir}/${subdir}"
-  mkdir ${tempdir}/${subdir}
+  mkdir ${tempdir}/${subdir} || exit
 done
 
 echo "Making links to 'lisp' and its subdirectories"
 files=`find lisp \( -name '*.el' -o -name '*.elc' -o -name 'ChangeLog*' \
- -o -name 'README' \)`
+ -o -name 'README' \)` || exit
 
 ### Don't distribute site-init.el, site-load.el, or default.el.
 for file in lisp/Makefile.in $files; do
   case $file in
     */site-init*|*/site-load*|*/default*) continue ;;
   esac
-  ln $file $tempdir/$file
+  ln $file $tempdir/$file || exit
 done
 
 echo "Making links to 'leim' and its subdirectories"
-(cd leim
- ln ChangeLog.*[0-9] README ../${tempdir}/leim
- ln CXTERM-DIC/README CXTERM-DIC/*.tit ../${tempdir}/leim/CXTERM-DIC
- ln SKK-DIC/README SKK-DIC/SKK-JISYO.L ../${tempdir}/leim/SKK-DIC
- ln MISC-DIC/README MISC-DIC/*.* ../${tempdir}/leim/MISC-DIC
- ln Makefile.in ../${tempdir}/leim/Makefile.in
- ln leim-ext.el ../${tempdir}/leim/leim-ext.el)
+(cd leim &&
+ ln ChangeLog.*[0-9] README ../${tempdir}/leim &&
+ ln CXTERM-DIC/README CXTERM-DIC/*.tit ../${tempdir}/leim/CXTERM-DIC &&
+ ln SKK-DIC/README SKK-DIC/SKK-JISYO.L ../${tempdir}/leim/SKK-DIC &&
+ ln MISC-DIC/README MISC-DIC/*.* ../${tempdir}/leim/MISC-DIC &&
+ ln Makefile.in ../${tempdir}/leim/Makefile.in &&
+ ln leim-ext.el ../${tempdir}/leim/leim-ext.el &&
+:) || exit
 
 ## FIXME Can we not just use the "find -type f" method for this one?
 echo "Making links to 'build-aux'"
-(cd build-aux
- ln config.guess config.sub msys-to-w32 ../${tempdir}/build-aux
- ln gitlog-to-changelog gitlog-to-emacslog ../${tempdir}/build-aux
- ln install-sh move-if-change ../${tempdir}/build-aux
- ln update-copyright update-subdirs ../${tempdir}/build-aux
- ln dir_top make-info-dir ../${tempdir}/build-aux)
+(cd build-aux &&
+ ln config.guess config.sub msys-to-w32 ../${tempdir}/build-aux &&
+ ln gitlog-to-changelog gitlog-to-emacslog ../${tempdir}/build-aux &&
+ ln install-sh move-if-change ../${tempdir}/build-aux &&
+ ln update-copyright update-subdirs ../${tempdir}/build-aux &&
+ ln dir_top make-info-dir ../${tempdir}/build-aux &&
+:) || exit
 
 echo "Making links to 'src'"
 ### Don't distribute the configured versions of
 ### config.in, paths.in, buildobj.h, or Makefile.in.
-(cd src
- echo "  (It is ok if ln fails in some cases.)"
- ln [a-zA-Z]*.[chm] ../${tempdir}/src
- ln [a-zA-Z]*.in ../${tempdir}/src
- ln deps.mk ../${tempdir}/src
- ln README ChangeLog.*[0-9] ../${tempdir}/src
- ln .gdbinit .dbxinit ../${tempdir}/src
- cd ../${tempdir}/src
- rm -f globals.h config.h epaths.h Makefile buildobj.h)
+(cd src &&
+ ln [a-zA-Z]*.[chm] ../${tempdir}/src &&
+ ln [a-zA-Z]*.in ../${tempdir}/src &&
+ ln deps.mk ../${tempdir}/src &&
+ ln README ChangeLog.*[0-9] ../${tempdir}/src &&
+ ln .gdbinit .dbxinit ../${tempdir}/src &&
+ cd ../${tempdir}/src &&
+ rm -f globals.h config.h epaths.h Makefile buildobj.h &&
+:) || exit
 
 echo "Making links to 'src/bitmaps'"
-(cd src/bitmaps
- ln README *.xbm ../../${tempdir}/src/bitmaps)
+(cd src/bitmaps &&
+ ln README *.xbm ../../${tempdir}/src/bitmaps &&
+:) || exit
 
 echo "Making links to 'lib'"
-(cd lib
- ln [a-zA-Z_]*.[ch] ../${tempdir}/lib
- ln gnulib.mk.in Makefile.in ../${tempdir}/lib
- cd ../${tempdir}/lib
- script='/[*]/d; s/\.in\.h$/.h/'
- rm -f `ls *.in.h | sed "$script"`)
+(cd lib &&
+ ln [a-zA-Z_]*.[ch] ../${tempdir}/lib &&
+ ln gnulib.mk.in Makefile.in ../${tempdir}/lib &&
+ cd ../${tempdir}/lib &&
+ script='/[*]/d; s/\.in\.h$/.h/' &&
+ rm -f `ls *.in.h | sed "$script"` &&
+:) || exit
 
 echo "Making links to 'lib-src'"
-(cd lib-src
- ln [a-zA-Z]*.[ch] ../${tempdir}/lib-src
- ln ChangeLog.*[0-9] Makefile.in README ../${tempdir}/lib-src
- ln rcs2log ../${tempdir}/lib-src)
+(cd lib-src &&
+ ln [a-zA-Z]*.[ch] ../${tempdir}/lib-src &&
+ ln ChangeLog.*[0-9] Makefile.in README ../${tempdir}/lib-src &&
+ ln rcs2log ../${tempdir}/lib-src &&
+:) || exit
 
 echo "Making links to 'm4'"
-(cd m4
- ln *.m4 ../${tempdir}/m4)
+(cd m4 &&
+ ln *.m4 ../${tempdir}/m4 &&
+:) || exit
 
 echo "Making links to 'modules'"
-(cd modules
- ln *.py ../${tempdir}/modules
-)
+(cd modules &&
+ ln *.py ../${tempdir}/modules &&
+:) || exit
 
 echo "Making links to 'nt'"
-(cd nt
- ln emacs-x86.manifest emacs-x64.manifest ../${tempdir}/nt
- ln [a-z]*.bat [a-z]*.[ch] ../${tempdir}/nt
- ln *.in gnulib-cfg.mk ../${tempdir}/nt
- ln mingw-cfg.site epaths.nt INSTALL.W64 ../${tempdir}/nt
- ln ChangeLog.*[0-9] INSTALL README README.W32 ../${tempdir}/nt)
+(cd nt &&
+ ln emacs-x86.manifest emacs-x64.manifest ../${tempdir}/nt &&
+ ln [a-z]*.bat [a-z]*.[ch] ../${tempdir}/nt &&
+ ln *.in gnulib-cfg.mk ../${tempdir}/nt &&
+ ln mingw-cfg.site epaths.nt INSTALL.W64 ../${tempdir}/nt &&
+ ln ChangeLog.*[0-9] INSTALL README README.W32 ../${tempdir}/nt &&
+:) || exit
 
 echo "Making links to 'nt/inc' and its subdirectories"
 for f in `find nt/inc -type f -name '[a-z]*.h'`; do
-  ln $f $tempdir/$f
+  ln $f $tempdir/$f || exit
 done
 
 echo "Making links to 'nt/icons'"
-(cd nt/icons
- ln README [a-z]*.ico ../../${tempdir}/nt/icons
- ln [a-z]*.cur ../../${tempdir}/nt/icons)
+(cd nt/icons &&
+ ln README [a-z]*.ico ../../${tempdir}/nt/icons &&
+ ln [a-z]*.cur ../../${tempdir}/nt/icons &&
+:) || exit
 
 echo "Making links to 'msdos'"
-(cd msdos
- ln ChangeLog.*[0-9] INSTALL README emacs.ico emacs.pif ../${tempdir}/msdos
- ln depfiles.bat inttypes.h ../${tempdir}/msdos
- ln mainmake.v2 sed*.inp ../${tempdir}/msdos)
+(cd msdos &&
+ ln ChangeLog.*[0-9] INSTALL README emacs.ico emacs.pif ../${tempdir}/msdos &&
+ ln depfiles.bat inttypes.h ../${tempdir}/msdos &&
+ ln mainmake.v2 sed*.inp ../${tempdir}/msdos &&
+:) || exit
 
 echo "Making links to 'nextstep'"
-(cd nextstep
- ln ChangeLog.*[0-9] README INSTALL Makefile.in ../${tempdir}/nextstep)
+(cd nextstep &&
+ ln ChangeLog.*[0-9] README INSTALL Makefile.in ../${tempdir}/nextstep &&
+:) || exit
 
 echo "Making links to 'nextstep/templates'"
-(cd nextstep/templates
- ln Emacs.desktop.in Info-gnustep.plist.in Info.plist.in InfoPlist.strings.in ../../${tempdir}/nextstep/templates)
+(cd nextstep/templates &&
+ ln Emacs.desktop.in Info-gnustep.plist.in Info.plist.in InfoPlist.strings.in \
+   ../../${tempdir}/nextstep/templates &&
+:) || exit
 
 echo "Making links to 'nextstep/Cocoa/Emacs.base/Contents'"
-(cd nextstep/Cocoa/Emacs.base/Contents
- ln PkgInfo ../../../../${tempdir}/nextstep/Cocoa/Emacs.base/Contents)
+(cd nextstep/Cocoa/Emacs.base/Contents &&
+ ln PkgInfo ../../../../${tempdir}/nextstep/Cocoa/Emacs.base/Contents &&
+:) || exit
 
 echo "Making links to 'nextstep/Cocoa/Emacs.base/Contents/Resources'"
-(cd nextstep/Cocoa/Emacs.base/Contents/Resources
- ln Credits.html *.icns ../../../../../${tempdir}/nextstep/Cocoa/Emacs.base/Contents/Resources)
+(cd nextstep/Cocoa/Emacs.base/Contents/Resources &&
+ ln Credits.html *.icns \
+   ../../../../../${tempdir}/nextstep/Cocoa/Emacs.base/Contents/Resources &&
+:) || exit
 
 echo "Making links to 'nextstep/GNUstep/Emacs.base/Resources'"
-(cd nextstep/GNUstep/Emacs.base/Resources
- ln README emacs.tiff ../../../../${tempdir}/nextstep/GNUstep/Emacs.base/Resources )
+(cd nextstep/GNUstep/Emacs.base/Resources &&
+ ln README emacs.tiff \
+   ../../../../${tempdir}/nextstep/GNUstep/Emacs.base/Resources &&
+:) || exit
 
 echo "Making links to 'oldXMenu'"
-(cd oldXMenu
- ln *.[ch] *.in *.mk ../${tempdir}/oldXMenu
- ln README ChangeLog.*[0-9] ../${tempdir}/oldXMenu)
+(cd oldXMenu &&
+ ln *.[ch] *.in *.mk ../${tempdir}/oldXMenu &&
+ ln README ChangeLog.*[0-9] ../${tempdir}/oldXMenu &&
+:) || exit
 
 echo "Making links to 'lwlib'"
-(cd lwlib
- ln *.[ch] *.in *.mk ../${tempdir}/lwlib
- ln README ChangeLog.*[0-9] ../${tempdir}/lwlib)
+(cd lwlib &&
+ ln *.[ch] *.in *.mk ../${tempdir}/lwlib &&
+ ln README ChangeLog.*[0-9] ../${tempdir}/lwlib &&
+:) || exit
 
 ## It is important to distribute admin/ because it contains sources
 ## for generated lisp/international/uni-*.el files.
@@ -546,7 +568,7 @@ files=
   case $f in
     */Makefile) [ -f $f.in ] && continue ;;
   esac
-  ln $f $tempdir/$f
+  ln $f $tempdir/$f || exit
 done
 
 if [ "$with_tests" = "yes" ]; then
@@ -557,7 +579,7 @@ files=
     case $f in
         */Makefile) [ -f $f.in ] && continue ;;
     esac
-    ln $f $tempdir/$f
+    ln $f $tempdir/$f || exit
   done
 fi
 
@@ -569,45 +591,50 @@ files=
     etc/refcards/*.aux|etc/refcards/*.dvi|etc/refcards/*.log|etc/refcards/*.ps)
       continue ;;
   esac
-  ln $f $tempdir/$f
+  ln $f $tempdir/$f || exit
 done
 
 echo "Making links to 'info'"
-ln `find info -type f -print` ${tempdir}/info
+ln `find info -type f -print` ${tempdir}/info || exit
 
 echo "Making links to 'doc/emacs'"
-(cd doc/emacs
- ln *.texi *.in ChangeLog.*[0-9] ../../${tempdir}/doc/emacs)
+(cd doc/emacs &&
+ ln *.texi *.in ChangeLog.*[0-9] ../../${tempdir}/doc/emacs &&
+:) || exit
 
 echo "Making links to 'doc/misc'"
-(cd doc/misc
+(cd doc/misc &&
  ln *.texi *.tex *.in gnus-news.el ChangeLog.*[0-9] \
-    ../../${tempdir}/doc/misc)
+   ../../${tempdir}/doc/misc &&
+:) || exit
 
 echo "Making links to 'doc/lispref'"
-(cd doc/lispref
- ln *.texi *.in README ChangeLog.*[0-9] ../../${tempdir}/doc/lispref
- ln spellfile ../../${tempdir}/doc/lispref
- ln two-volume.make two-volume-cross-refs.txt ../../${tempdir}/doc/lispref)
+(cd doc/lispref &&
+ ln *.texi *.in README ChangeLog.*[0-9] ../../${tempdir}/doc/lispref &&
+ ln spellfile ../../${tempdir}/doc/lispref &&
+ ln two-volume.make two-volume-cross-refs.txt ../../${tempdir}/doc/lispref &&
+:) || exit
 
 echo "Making links to 'doc/lispintro'"
-(cd doc/lispintro
- ln *.texi *.in *.eps *.pdf ../../${tempdir}/doc/lispintro
- ln README ChangeLog.*[0-9] ../../${tempdir}/doc/lispintro
- cd ../../${tempdir}/doc/lispintro)
+(cd doc/lispintro &&
+ ln *.texi *.in *.eps *.pdf ../../${tempdir}/doc/lispintro &&
+ ln README ChangeLog.*[0-9] ../../${tempdir}/doc/lispintro &&
+ cd ../../${tempdir}/doc/lispintro &&
+:) || exit
 
 echo "Making links to 'doc/man'"
-(cd doc/man
- ln *.*[0-9] *.in ../../${tempdir}/doc/man
- cd ../../${tempdir}/doc/man
- rm -f emacs.1)
+(cd doc/man &&
+ ln *.*[0-9] *.in ../../${tempdir}/doc/man &&
+ cd ../../${tempdir}/doc/man &&
+ rm -f emacs.1 &&
+:) || exit
 
 ### It would be nice if they could all be symlinks to top-level copy, but
 ### you're not supposed to have any symlinks in distribution tar files.
 echo "Making sure copying notices are all copies of 'COPYING'"
 for subdir in . etc leim lib lib-src lisp lwlib msdos nt src; do
-  rm -f ${tempdir}/${subdir}/COPYING
-  cp COPYING ${tempdir}/${subdir}
+  rm -f ${tempdir}/${subdir}/COPYING || exit
+  cp COPYING ${tempdir}/${subdir} || exit
 done
 
 if [ "${newer}" ]; then
@@ -615,12 +642,13 @@ files=
   ## We remove .elc files unconditionally, on the theory that anyone picking
   ## up an incremental distribution already has a running Emacs to byte-compile
   ## them with.
-  find ${tempparent} \( -name '*.elc' -o ! -newer ${newer} \) -exec rm -f {} \;
+  find ${tempparent} \( -name '*.elc' -o ! -newer ${newer} \) \
+    -exec rm -f {} \; || exit
 fi
 
 ## Don't distribute backups, autosaves, etc.
 echo "Removing unwanted files"
-find ${tempparent} \( -name '*~' -o -name '#*#' -o -name '.*ignore' -o -name '=*' -o -name 'TAGS' \) -exec rm -f {} \;
+find ${tempparent} \( -name '*~' -o -name '#*#' -o -name '.*ignore' -o -name '=*' -o -name 'TAGS' \) -exec rm -f {} \; || exit
 
 if [ "${make_tar}" = yes ]; then
   echo "Looking for $default_gzip"
@@ -648,13 +676,23 @@ files=
     taropt="$taropt --sort=name"
   [ "$verbose" = "yes" ] && taropt="$taropt --verbose"
 
-  (cd ${tempparent} ; tar $taropt -cf - ${emacsname} ) \
-    | ${default_gzip} \
-    > ${emacsname}.tar${gzip_extension}
+  (cd $tempparent &&
+   case $default_gzip in
+     cat) tar $taropt -cf - $emacsname;;
+     *) if tar $taropt -cf /dev/null --use-compress-program="$default_gzip" \
+	       $emacsname/src/lisp.h
+	then
+	  tar $taropt -cf - --use-compress-program="$default_gzip" $emacsname
+	else
+	  tar $taropt -cf $emacsname.tar $emacsname &&
+	  $default_gzip <$emacsname.tar
+        fi;;
+   esac
+  ) >$emacsname.tar$gzip_extension || exit
 fi
 
 if [ "${clean_up}" != yes ]; then
-  (cd ${tempparent}; mv ${emacsname} ..)
+  (cd ${tempparent} && mv ${emacsname} ..) &&
   rm -rf ${tempparent}
 fi
 
-- 
2.14.3