Re: what make-dist should include [was Re: Emacs pretest 26.1.91 is out]

[Paul Eggert <eggert@cs.ucla.edu>]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

On 1/16/19 2:30 AM, Glenn Morris wrote:
> I see zero practical advantages to a switch to a "distribute everything"
> approach (which Emacs has never done, and nor when I check do eg tar,
> texinfo, coreutils); unless it will drastically simplify make-dist and
> avoid problems like the one fixed in 021f34b. If it does then I am all
> for it. :)

OK, proposed make-dist patch attached. This patch drastically simplifies 
the list of files in make-dist that needs to be maintained manually, by 
relying on Git to keep track of files under version control. This should 
help avoid problems like the one you mentioned.


[-- Attachment #2: 0001-Make-make-dist-more-automatic.txt --]
[-- Type: text/plain, Size: 13847 bytes --]

From 2325172f59faa4f3c1d41262ccf1ebd520d5b736 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 22 Jan 2019 13:52:24 -0800
Subject: [PATCH] Make make-dist more automatic
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Simplify make-dist maintenance by having it generate its
list of files more automatically.  Put the list of distributed
files into a file MANIFEST that can be used in the unusual
situations when you’re making a distribution without having
access to a Git repository.
* make-dist (top_level_ChangeLog): Now nonempty if the
distribution tarball will contain a ChangeLog, instead of
being nonempty when a ChangeLog is requested, Git is present
and a readable ChangeLog exists.  The new interpretation makes
the script a bit easier to follow.
(possibly_non_vc_files, info_files, mkdir_verbose)
(file_to_skip, MANIFEST_subdir_sed, tempsubdirs):
New variables.
(MANIFEST): Update and use this file, which now records what
files are distributed.
(top_level, subdir, files, file): Remove.
---
 make-dist | 346 ++++++++++++------------------------------------------
 1 file changed, 75 insertions(+), 271 deletions(-)

diff --git a/make-dist b/make-dist
index 20be62da1f..62e47b4d14 100755
--- a/make-dist
+++ b/make-dist
@@ -358,6 +358,50 @@ tempparent=
   $EMACS -batch -f batch-byte-recompile-directory lisp
 fi                              # $update = yes
 
+if [ "$changelog" = yes ] && [ -r .git ]; then
+  top_level_ChangeLog=ChangeLog
+else
+  top_level_ChangeLog=
+fi
+
+# Files to distribute that might not be under version control.
+# Don't distribute site-init.el, site-load.el, or default.el.
+possibly_non_vc_files="
+  $top_level_ChangeLog
+  MANIFEST aclocal.m4 configure
+  admin/charsets/jisx2131-filter
+  src/config.in src/dmpstruct.h src/emacs-module.h
+  src/fingerprint.c src/stamp-h.in
+"$(
+  find admin doc etc lisp \
+    \( -name '*.el' -o -name '*.elc' -o -name '*.map' -o -name '*.stamp' \
+       -o -name '*.texi' -o -name '*.tex' -o -name '*.txt' \) \
+    ! -name 'site-init*' ! -name 'site-load*' ! -name 'default*'
+) || exit
+
+if [ $with_info = yes ]; then
+  info_files="info/dir $(echo info/*.info)" || exit
+else
+  info_files=
+fi
+
+# If Git is in use update the file MANIFEST, which can substitute for
+# 'git ls-files' later (e.g., after extraction from a tarball).
+# Otherwise, rely on the existing MANIFEST, which should be maintained some
+# other way when adding or deleting a distributed file while not using Git.
+if [ $update = yes ] && [ -r .git ]; then
+  echo "Updating MANIFEST"
+  if [ $with_tests = yes ]; then
+    git ls-files >MANIFEST
+  else
+    git ls-files | grep -v '^test' >MANIFEST
+  fi || exit
+  printf '%s\n' $possibly_non_vc_files $info_files >>MANIFEST || exit
+  sort -u -o MANIFEST MANIFEST || exit
+fi
+
+<MANIFEST || exit
+
 echo "Creating staging directory: '${tempparent}'"
 
 mkdir ${tempparent} || exit
@@ -370,290 +414,52 @@ tempdir=
 fi
 
 echo "Creating top directory: '${tempdir}'"
-mkdir ${tempdir} || exit
+if [ $verbose = yes ] && (mkdir --verbose ${tempdir}) >/dev/null 2>&1; then
+  mkdir_verbose='mkdir --verbose'
+else
+  mkdir $tempdir || exit
+  mkdir_verbose=mkdir
+fi
+
+# file_to_skip is normally empty to link every file,
+# but it can be 'ChangeLog' if we do not want to link the
+# top-level ChangeLog.
+file_to_skip=
 
-top_level_ChangeLog=
 if [ "$changelog" = yes ]; then
   if test -r .git; then
     ## When making a release or pretest the ChangeLog should already
     ## have been created and edited as needed.  Don't ignore it.
-    if test -r ChangeLog; then
+    if [ -r ChangeLog ] && [ ! -L ChangeLog ]; then
       echo "Using existing top-level ChangeLog"
-      top_level_ChangeLog=ChangeLog
     else
       echo "Making top-level ChangeLog"
       make ChangeLog CHANGELOG=${tempdir}/ChangeLog || \
         { x=$?; echo "make ChangeLog FAILED (try --no-changelog?)" >&2; exit $x; }
+      file_to_skip=ChangeLog
     fi
   else
     echo "No repository, so omitting top-level ChangeLog"
   fi
 fi
 
-### We copy in the top-level files before creating the subdirectories in
-### hopes that this will make the top-level files appear first in the
-### tar file; this means that people can start reading the INSTALL and
-### README while the rest of the tar file is still unpacking.  Whoopee.
-echo "Making links to top-level files"
-top_level='
-  INSTALL README BUGS
-  ChangeLog.*[0-9] Makefile.in autogen.sh configure configure.ac
-  config.bat make-dist .dir-locals.el
-  aclocal.m4 CONTRIBUTE
-'
-ln $top_level $top_level_ChangeLog $tempdir || exit
-
 echo "Creating subdirectories"
-for subdir in site-lisp \
-	      leim leim/CXTERM-DIC leim/MISC-DIC leim/SKK-DIC \
-	      build-aux \
-	      src src/bitmaps lib lib-src oldXMenu lwlib \
-	      nt nt/inc nt/inc/sys nt/inc/arpa nt/inc/netinet nt/icons \
-	      `find etc lisp admin test -type d` \
-	      doc doc/emacs doc/misc doc/man doc/lispref doc/lispintro \
-	      info m4 modules msdos \
-	      nextstep nextstep/templates \
-	      nextstep/Cocoa nextstep/Cocoa/Emacs.base \
-	      nextstep/Cocoa/Emacs.base/Contents \
-	      nextstep/Cocoa/Emacs.base/Contents/Resources \
-	      nextstep/GNUstep \
-	      nextstep/GNUstep/Emacs.base \
-	      nextstep/GNUstep/Emacs.base/Resources
-do
-
-  if [ "$with_tests" != "yes" ]; then
-    case $subdir in
-        test*) continue ;;
-    esac
-  fi
-
-  ## site-lisp for in-place installs (?).
-  [ "$subdir" = "site-lisp" ] || [ -d "$subdir" ] || \
-      echo "WARNING: $subdir not found, making anyway"
-  [ "$verbose" = "yes" ] && echo "  ${tempdir}/${subdir}"
-  mkdir ${tempdir}/${subdir} || exit
-done
-
-echo "Making links to 'lisp' and its subdirectories"
-files=`find lisp \( -name '*.el' -o -name '*.elc' -o -name 'ChangeLog*' \
- -o -name 'README' \)` || exit
-
-### Don't distribute site-init.el, site-load.el, or default.el.
-for file in lisp/Makefile.in $files; do
-  case $file in
-    */site-init*|*/site-load*|*/default*) continue ;;
-  esac
-  ln $file $tempdir/$file || exit
-done
-
-echo "Making links to 'leim' and its subdirectories"
-(cd leim &&
- ln ChangeLog.*[0-9] README ../${tempdir}/leim &&
- ln CXTERM-DIC/README CXTERM-DIC/*.tit ../${tempdir}/leim/CXTERM-DIC &&
- ln SKK-DIC/README SKK-DIC/SKK-JISYO.L ../${tempdir}/leim/SKK-DIC &&
- ln MISC-DIC/README MISC-DIC/*.* ../${tempdir}/leim/MISC-DIC &&
- ln Makefile.in ../${tempdir}/leim/Makefile.in &&
- ln leim-ext.el ../${tempdir}/leim/leim-ext.el &&
-:) || exit
-
-## FIXME Can we not just use the "find -type f" method for this one?
-echo "Making links to 'build-aux'"
-(cd build-aux &&
- ln config.guess config.sub msys-to-w32 ../${tempdir}/build-aux &&
- ln gitlog-to-changelog gitlog-to-emacslog ../${tempdir}/build-aux &&
- ln install-sh move-if-change ../${tempdir}/build-aux &&
- ln update-copyright update-subdirs ../${tempdir}/build-aux &&
- ln dir_top make-info-dir ../${tempdir}/build-aux &&
-:) || exit
-
-echo "Making links to 'src'"
-### Don't distribute the configured versions of
-### config.in, paths.in, buildobj.h, or Makefile.in.
-(cd src &&
- ln [a-zA-Z]*.[chm] ../${tempdir}/src &&
- ln [a-zA-Z]*.in ../${tempdir}/src &&
- ln deps.mk dmpstruct.awk ../${tempdir}/src &&
- ln README ChangeLog.*[0-9] ../${tempdir}/src &&
- ln .gdbinit .dbxinit ../${tempdir}/src &&
- cd ../${tempdir}/src &&
- rm -f globals.h config.h epaths.h Makefile buildobj.h &&
-:) || exit
-
-echo "Making links to 'src/bitmaps'"
-(cd src/bitmaps &&
- ln README *.xbm ../../${tempdir}/src/bitmaps &&
-:) || exit
-
-echo "Making links to 'lib'"
-(cd lib &&
- ln [a-zA-Z_]*.[ch] ../${tempdir}/lib &&
- ln gnulib.mk.in Makefile.in ../${tempdir}/lib &&
- cd ../${tempdir}/lib &&
- script='/[*]/d; s/\.in\.h$/.h/' &&
- rm -f `ls *.in.h | sed "$script"` &&
-:) || exit
-
-echo "Making links to 'lib-src'"
-(cd lib-src &&
- ln [a-zA-Z]*.[ch] ../${tempdir}/lib-src &&
- ln ChangeLog.*[0-9] Makefile.in README ../${tempdir}/lib-src &&
- ln rcs2log ../${tempdir}/lib-src &&
-:) || exit
-
-echo "Making links to 'm4'"
-(cd m4 &&
- ln *.m4 ../${tempdir}/m4 &&
-:) || exit
-
-echo "Making links to 'modules'"
-(cd modules &&
- ln *.py ../${tempdir}/modules &&
-:) || exit
-
-echo "Making links to 'nt'"
-(cd nt &&
- ln emacs-x86.manifest emacs-x64.manifest ../${tempdir}/nt &&
- ln [a-z]*.bat [a-z]*.[ch] ../${tempdir}/nt &&
- ln *.in gnulib-cfg.mk ../${tempdir}/nt &&
- ln mingw-cfg.site epaths.nt INSTALL.W64 ../${tempdir}/nt &&
- ln ChangeLog.*[0-9] INSTALL README README.W32 ../${tempdir}/nt &&
-:) || exit
-
-echo "Making links to 'nt/inc' and its subdirectories"
-for f in `find nt/inc -type f -name '[a-z]*.h'`; do
-  ln $f $tempdir/$f || exit
-done
-
-echo "Making links to 'nt/icons'"
-(cd nt/icons &&
- ln README [a-z]*.ico ../../${tempdir}/nt/icons &&
- ln [a-z]*.cur ../../${tempdir}/nt/icons &&
-:) || exit
-
-echo "Making links to 'msdos'"
-(cd msdos &&
- ln ChangeLog.*[0-9] INSTALL README emacs.ico emacs.pif ../${tempdir}/msdos &&
- ln depfiles.bat inttypes.h ../${tempdir}/msdos &&
- ln mainmake.v2 sed*.inp ../${tempdir}/msdos &&
-:) || exit
-
-echo "Making links to 'nextstep'"
-(cd nextstep &&
- ln ChangeLog.*[0-9] README INSTALL Makefile.in ../${tempdir}/nextstep &&
-:) || exit
-
-echo "Making links to 'nextstep/templates'"
-(cd nextstep/templates &&
- ln Emacs.desktop.in Info-gnustep.plist.in Info.plist.in InfoPlist.strings.in \
-   ../../${tempdir}/nextstep/templates &&
-:) || exit
-
-echo "Making links to 'nextstep/Cocoa/Emacs.base/Contents'"
-(cd nextstep/Cocoa/Emacs.base/Contents &&
- ln PkgInfo ../../../../${tempdir}/nextstep/Cocoa/Emacs.base/Contents &&
-:) || exit
-
-echo "Making links to 'nextstep/Cocoa/Emacs.base/Contents/Resources'"
-(cd nextstep/Cocoa/Emacs.base/Contents/Resources &&
- ln Credits.html *.icns \
-   ../../../../../${tempdir}/nextstep/Cocoa/Emacs.base/Contents/Resources &&
-:) || exit
-
-echo "Making links to 'nextstep/GNUstep/Emacs.base/Resources'"
-(cd nextstep/GNUstep/Emacs.base/Resources &&
- ln README emacs.tiff \
-   ../../../../${tempdir}/nextstep/GNUstep/Emacs.base/Resources &&
-:) || exit
-
-echo "Making links to 'oldXMenu'"
-(cd oldXMenu &&
- ln *.[ch] *.in *.mk ../${tempdir}/oldXMenu &&
- ln README ChangeLog.*[0-9] ../${tempdir}/oldXMenu &&
-:) || exit
-
-echo "Making links to 'lwlib'"
-(cd lwlib &&
- ln *.[ch] *.in *.mk ../${tempdir}/lwlib &&
- ln README ChangeLog.*[0-9] ../${tempdir}/lwlib &&
-:) || exit
-
-## It is important to distribute admin/ because it contains sources
-## for generated lisp/international/uni-*.el files.
-echo "Making links to 'admin' and its subdirectories"
-for f in `find admin -type f`; do
-  case $f in
-    */Makefile) [ -f $f.in ] && continue ;;
-  esac
-  ln $f $tempdir/$f || exit
-done
-
-if [ "$with_tests" = "yes" ]; then
-  echo "Making links to 'test' and its subdirectories"
-  for f in `find test -type f ! -name '*.log' ! -name a.out \
-	      ! -name '*.so' ! -name '*.dll' ! -name '*.o'
-  `; do
-    case $f in
-        */Makefile) [ -f $f.in ] && continue ;;
-    esac
-    ln $f $tempdir/$f || exit
-  done
-fi
-
-echo "Making links to 'etc' and its subdirectories"
-for f in `find etc -type f`; do
-  case $f in
-    etc/DOC*|etc/*.pyc) continue ;;
-    ## Arguably we should not exclude *.ps.
-    etc/refcards/*.aux|etc/refcards/*.dvi|etc/refcards/*.log|etc/refcards/*.ps)
-      continue ;;
-  esac
-  ln $f $tempdir/$f || exit
-done
-
-if [ "$with_info" = "yes" ]; then
-  echo "Making links to 'info'"
-  ln `find info -type f -print` ${tempdir}/info || exit
-fi
+MANIFEST_subdir_sed='
+  $a\
+'$tempdir'/info\
+'$tempdir'/site-lisp
+  s,[^/]*$,,
+  s,/$,,
+  /^$/d
+  s,^,'$tempdir'/,
+'
+tempsubdirs=$(sed "$MANIFEST_subdir_sed" MANIFEST | sort -u)
+$mkdir_verbose -p $tempsubdirs || exit
 
-echo "Making links to 'doc/emacs'"
-(cd doc/emacs &&
- ln *.texi *.in ChangeLog.*[0-9] ../../${tempdir}/doc/emacs &&
-:) || exit
-
-echo "Making links to 'doc/misc'"
-(cd doc/misc &&
- ln *.texi *.tex *.in gnus-news.el ChangeLog.*[0-9] \
-   ../../${tempdir}/doc/misc &&
-:) || exit
-
-echo "Making links to 'doc/lispref'"
-(cd doc/lispref &&
- ln *.texi *.in README ChangeLog.*[0-9] ../../${tempdir}/doc/lispref &&
- ln spellfile ../../${tempdir}/doc/lispref &&
- ln two-volume.make two-volume-cross-refs.txt ../../${tempdir}/doc/lispref &&
-:) || exit
-
-echo "Making links to 'doc/lispintro'"
-(cd doc/lispintro &&
- ln *.texi *.in *.eps *.pdf ../../${tempdir}/doc/lispintro &&
- ln README ChangeLog.*[0-9] ../../${tempdir}/doc/lispintro &&
- cd ../../${tempdir}/doc/lispintro &&
-:) || exit
-
-echo "Making links to 'doc/man'"
-(cd doc/man &&
- ln *.*[0-9] *.in ../../${tempdir}/doc/man &&
- cd ../../${tempdir}/doc/man &&
- rm -f emacs.1 &&
-:) || exit
-
-### It would be nice if they could all be symlinks to top-level copy, but
-### you're not supposed to have any symlinks in distribution tar files.
-echo "Making sure copying notices are all copies of 'COPYING'"
-for subdir in . etc leim lib lib-src lisp lwlib msdos nt src; do
-  rm -f ${tempdir}/${subdir}/COPYING || exit
-  cp COPYING ${tempdir}/${subdir} || exit
-done
+echo "Making links to files"
+while read file; do
+  [ $file = "$file_to_skip" ] || ln $file $tempdir/$file || exit
+done <MANIFEST
 
 if [ "${newer}" ]; then
   printf '%s\n' "Removing files older than $newer"
@@ -664,10 +470,6 @@ files=
     -exec rm -f {} \; || exit
 fi
 
-## Don't distribute backups, autosaves, etc.
-echo "Removing unwanted files"
-find ${tempdir} \( -name '*~' -o -name '#*#' -o -name '.*ignore' -o -name '=*' -o -name 'TAGS' \) -exec rm -f {} \; || exit
-
 if [ "${make_tar}" = yes ]; then
   echo "Looking for $default_gzip"
   found=0
@@ -715,4 +517,6 @@ files=
   rm -rf ${tempparent}
 fi
 
-### make-dist ends here
+# Local Variables:
+# sh-basic-offset: 2
+# End:
-- 
2.20.1