bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Ulrich Mueller]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 1064 bytes --]

Tags: patch

With parallel make (-j3), some elisp files, for example lisp/select.el
and lisp/scroll-bar.el, are byte-compiled twice. From the build log:

   Compiling /var/tmp/portage/app-editors/emacs-23.3/work/emacs-23.3/src/../lisp/select.el
   [...]
   Wrote /var/tmp/portage/app-editors/emacs-23.3/work/emacs-23.3/lisp/select.elc
   [...]
   Compiling ../lisp/select.el
   [...]
   Wrote /var/tmp/portage/app-editors/emacs-23.3/work/emacs-23.3/lisp/select.elc

The reason seems to be that select.elc occurs twice in
src/Makefile.in: in variable "lisp" as $(lispsource)select.elc and in
variable "SOME_MACHINE_LISP" as ../lisp/select.elc, so that make
doesn't recognise them as identical targets.

Attached patch makes the problem go away for me. My humble solution is
to change SOME_MACHINE_LISP to use $(lispsource), and introduce a new
variable SOME_MACHINE_SHORTLISP for the ../lisp/ entries. (Quite the
same as for the lisp and shortlisp variables).

As a side remark, mouse.elc is listed twice in the variable
assignment. Attached patch fixes this too.


[-- Attachment #2: emacs-23.3-parallel-make.patch --]
[-- Type: text/plain, Size: 2047 bytes --]

--- emacs-23.3-orig/src/Makefile.in
+++ emacs-23.3/src/Makefile.in
@@ -887,9 +887,26 @@
 	../lisp/window.elc \
 	../lisp/version.el
 
+SOME_MACHINE_LISP = $(lispsource)mouse.elc \
+  $(lispsource)select.elc $(lispsource)scroll-bar.elc \
+  $(lispsource)ls-lisp.elc $(lispsource)dos-fns.elc \
+  $(lispsource)w32-fns.elc $(lispsource)dos-w32.elc \
+  $(lispsource)disp-table.elc $(lispsource)dos-vars.elc \
+  $(lispsource)tooltip.elc $(lispsource)image.elc \
+  $(lispsource)fringe.elc $(lispsource)dnd.elc \
+  $(lispsource)mwheel.elc $(lispsource)tool-bar.elc \
+  $(lispsource)x-dnd.elc $(lispsource)font-setting.elc \
+  $(lispsource)international/ccl.elc \
+  $(lispsource)international/fontset.elc \
+  $(lispsource)term/common-win.elc \
+  $(lispsource)term/x-win.elc \
+  $(lispsource)term/pc-win.elc $(lispsource)term/internal.elc \
+  $(lispsource)term/ns-win.elc $(lispsource)term/w32-win.elc \
+  $(lispsource)emacs-lisp/easymenu.elc
+
 /* Like $shortlisp, but includes only those files from $lisp that are loaded
    conditionally (i.e., only on some platforms).  */
-SOME_MACHINE_LISP = ../lisp/mouse.elc \
+SOME_MACHINE_SHORTLISP = ../lisp/mouse.elc \
   ../lisp/select.elc ../lisp/scroll-bar.elc \
   ../lisp/ls-lisp.elc ../lisp/dos-fns.elc \
   ../lisp/w32-fns.elc ../lisp/dos-w32.elc \
@@ -900,7 +917,6 @@
   ../lisp/x-dnd.elc ../lisp/font-setting.elc \
   ../lisp/international/ccl.elc \
   ../lisp/international/fontset.elc \
-  ../lisp/mouse.elc \
   ../lisp/term/common-win.elc \
   ../lisp/term/x-win.elc \
   ../lisp/term/pc-win.elc ../lisp/term/internal.elc \
@@ -955,7 +971,7 @@
 ${etc}DOC: ${libsrc}make-docfile${EXEEXT} ${obj} ${lisp} ${SOME_MACHINE_LISP}
 	-rm -f ${etc}DOC
 	${libsrc}make-docfile -d ${srcdir} ${SOME_MACHINE_OBJECTS} ${obj} > ${etc}DOC
-	${libsrc}make-docfile -a ${etc}DOC -d ${srcdir} ${SOME_MACHINE_LISP} ${shortlisp}
+	${libsrc}make-docfile -a ${etc}DOC -d ${srcdir} ${SOME_MACHINE_SHORTLISP} ${shortlisp}
 
 ${libsrc}make-docfile${EXEEXT}:
 	cd ${libsrc}; ${MAKE} ${MFLAGS} make-docfile${EXEEXT}

bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Glenn Morris]

Ulrich Mueller wrote:

> With parallel make (-j3), some elisp files, for example lisp/select.el
> and lisp/scroll-bar.el, are byte-compiled twice.
[...]
> The reason seems to be that select.elc occurs twice in
> src/Makefile.in: in variable "lisp" as $(lispsource)select.elc and in
> variable "SOME_MACHINE_LISP" as ../lisp/select.elc, so that make
> doesn't recognise them as identical targets.
>
> Attached patch makes the problem go away for me. My humble solution is
> to change SOME_MACHINE_LISP to use $(lispsource), and introduce a new
> variable SOME_MACHINE_SHORTLISP for the ../lisp/ entries.

I haven't applied this because I was reluctant to add yet another
list'o'filenames to src/Makefile.in, for a relatively minor issue.

On reflection, I think the current src/Makefile is needlessly complex.
It draws a distinction between lisp files that are preloaded on all
platforms, and those that are preloaded only on some platforms.
However, AFAICS, there is no need to do this. All the possible preloaded
lisp files are prerequisites of the DOC file (so that the DOC file can
be platform independent), and the emacs executable depends on the DOC
file, therefore it depends on all the possible preloaded lisp files, not
just the ones used by the platform in question.

So I think we can get rid of the "some_machine" lisp distinction.
Does anyone see a problem with that?

It nicely simplifies src/Makefile, and makes some configure output
variables no longer necessary.

The essential part of the patch is below. I have not included the
associated simplifications to configure.in, and the msdos and W32
makefiles.


*** src/Makefile.in	2011-04-24 09:00:03 +0000
--- src/Makefile.in	2011-05-19 01:06:45 +0000
***************
*** 237,254 ****
  MSDOS_OBJ =
  ## w16select.o termcap.o if MSDOS && HAVE_X_WINDOWS.
  MSDOS_X_OBJ =
- MSDOS_SUPPORT_REAL = $(lispsource)/ls-lisp.elc $(lispsource)/disp-table.elc \
-  $(lispsource)/dos-fns.elc $(lispsource)/dos-w32.elc $(lispsource)/dos-vars.elc \
-  $(lispsource)/term/internal.elc $(lispsource)/term/pc-win.elc
- ## $MSDOS_SUPPORT_REAL if MSDOS.
- MSDOS_SUPPORT =
  
  ns_appdir=@ns_appdir@
  ns_appbindir=@ns_appbindir@
  ns_appsrc=@ns_appsrc@
  NS_OBJ=@NS_OBJ@
  NS_OBJC_OBJ=@NS_OBJC_OBJ@
- NS_SUPPORT=@NS_SUPPORT@
  ## Only set if NS_IMPL_GNUSTEP.
  GNU_OBJC_CFLAGS=@GNU_OBJC_CFLAGS@
  
--- 237,248 ----
***************
*** 258,285 ****
  ## else xfont.o
  FONT_OBJ=@FONT_OBJ@
  
- ## Used if HAVE_MOUSE.
- REAL_MOUSE_SUPPORT=$(lispsource)/mouse.elc $(lispsource)/select.elc \
-   $(lispsource)/scroll-bar.elc
- ## Used if HAVE_GPM && !HAVE_MOUSE
- GPM_MOUSE_SUPPORT=$(lispsource)/mouse.elc
  LIBGPM = @LIBGPM@
- ## Either of the two preceding options, or empty.
- MOUSE_SUPPORT=@MOUSE_SUPPORT@
- 
- ## $(lispsource)/tooltip.elc if HAVE_WINDOW_SYSTEM, else empty.
- TOOLTIP_SUPPORT=@TOOLTIP_SUPPORT@
- 
- BASE_WINDOW_SUPPORT=$(lispsource)/fringe.elc $(lispsource)/image.elc \
-   $(lispsource)/international/fontset.elc $(lispsource)/dnd.elc \
-   $(lispsource)/tool-bar.elc $(lispsource)/mwheel.elc
- 
- X_WINDOW_SUPPORT=$(lispsource)/x-dnd.elc $(lispsource)/term/common-win.elc \
-   $(lispsource)/term/x-win.elc $(lispsource)/dynamic-setting.elc
- 
- ## If HAVE_X_WINDOWS, both the above
- ## else if HAVE_WINDOW_SYSTEM (ie, HAVE_NS) just the former; else empty.
- WINDOW_SUPPORT=@WINDOW_SUPPORT@
  
  ## -lresolv, or empty.
  LIBRESOLV = @LIBRESOLV@
--- 252,258 ----
***************
*** 385,439 ****
  otherobj= $(TERMCAP_OBJ) $(PRE_ALLOC_OBJ) $(GMALLOC_OBJ) $(RALLOC_OBJ) \
    $(POST_ALLOC_OBJ) $(VMLIMIT_OBJ) $(WIDGET_OBJ) $(LIBOBJS)
  
! ## This is the platform-specific list of Lisp files loaded into the
! ## dumped Emacs.  It is arranged like this because it is easier to generate
! ## it semi-mechanically from loadup.el this way.
  ##
  ## Note that this list should not include lisp files which might not
  ## be present, like site-load.el and site-init.el; this makefile
  ## expects them all to be either present or buildable.
  ##
- ## Files which are loaded unconditionally (i.e., on all platforms) should
- ## also be in shortlisp.  Files which are loaded conditionally (i.e., only
- ## on some platforms) should instead be in SOME_MACHINE_LISP.
- ##
  ## Place loaddefs.el first, so it gets generated first, since it is on
  ## the critical path (relevant in parallel compilations).
  lisp= \
  	$(lispsource)/loaddefs.el \
! 	$(lispsource)/abbrev.elc \
! 	$(lispsource)/buff-menu.elc \
! 	$(lispsource)/button.elc \
  	$(lispsource)/emacs-lisp/byte-run.elc \
- 	$(lispsource)/composite.elc \
- 	$(lispsource)/cus-face.elc \
- 	$(lispsource)/cus-start.elc \
- 	$(lispsource)/custom.elc \
  	$(lispsource)/emacs-lisp/backquote.elc \
! 	$(lispsource)/emacs-lisp/lisp-mode.elc \
! 	$(lispsource)/emacs-lisp/lisp.elc \
! 	$(lispsource)/env.elc \
! 	$(lispsource)/faces.elc \
! 	$(lispsource)/files.elc \
! 	$(lispsource)/format.elc \
! 	$(lispsource)/facemenu.elc \
! 	$(MOUSE_SUPPORT) \
! 	$(lispsource)/emacs-lisp/float-sup.elc \
! 	$(lispsource)/frame.elc \
! 	$(lispsource)/help.elc \
! 	$(lispsource)/indent.elc \
! 	$(lispsource)/isearch.elc \
! 	$(lispsource)/rfn-eshadow.elc \
! 	$(lispsource)/loadup.el \
! 	$(lispsource)/bindings.elc \
  	$(lispsource)/emacs-lisp/map-ynp.elc \
! 	$(lispsource)/menu-bar.elc \
  	$(lispsource)/international/mule.elc \
  	$(lispsource)/international/mule-conf.elc \
  	$(lispsource)/international/mule-cmds.elc \
  	$(lispsource)/international/characters.elc \
  	$(lispsource)/international/charprop.el \
- 	$(lispsource)/case-table.elc \
  	$(lispsource)/language/chinese.elc \
  	$(lispsource)/language/cyrillic.elc \
  	$(lispsource)/language/indian.elc \
--- 358,415 ----
  otherobj= $(TERMCAP_OBJ) $(PRE_ALLOC_OBJ) $(GMALLOC_OBJ) $(RALLOC_OBJ) \
    $(POST_ALLOC_OBJ) $(VMLIMIT_OBJ) $(WIDGET_OBJ) $(LIBOBJS)
  
! ## This is the list of all Lisp files that might be loaded into the
! ## dumped Emacs.  Some of them are not loaded on all platforms, but
! ## the DOC file on every platform uses them (because the DOC file is
! ## supposed to be platform-independent).
! ## It is arranged like this because it is easier to generate it
! ## semi-mechanically from loadup.el this way.
! ## Eg something like:
! ##   sed -e 's/"[ )].*//' -n -e '/(load "/ s/.*load "//p' loadup.el | \
! ##   grep -vE 'site-|ldefs-boot'
! ## minus any duplicates.
! ##
! ## Confusingly, term/internal is not in loadup, but is unconditionally
! ## loaded by pc-win, which is.
  ##
  ## Note that this list should not include lisp files which might not
  ## be present, like site-load.el and site-init.el; this makefile
  ## expects them all to be either present or buildable.
  ##
  ## Place loaddefs.el first, so it gets generated first, since it is on
  ## the critical path (relevant in parallel compilations).
  lisp= \
  	$(lispsource)/loaddefs.el \
! 	$(lispsource)/loadup.el \
  	$(lispsource)/emacs-lisp/byte-run.elc \
  	$(lispsource)/emacs-lisp/backquote.elc \
! 	$(lispsource)/subr.elc \
! 	$(lispsource)/version.el \
! 	$(lispsource)/widget.elc \
! 	$(lispsource)/custom.elc \
  	$(lispsource)/emacs-lisp/map-ynp.elc \
! 	$(lispsource)/cus-start.elc \
  	$(lispsource)/international/mule.elc \
  	$(lispsource)/international/mule-conf.elc \
+ 	$(lispsource)/env.elc \
+ 	$(lispsource)/format.elc \
+ 	$(lispsource)/bindings.elc \
+ 	$(lispsource)/files.elc \
+ 	$(lispsource)/cus-face.elc \
+ 	$(lispsource)/faces.elc \
+ 	$(lispsource)/button.elc \
+ 	$(lispsource)/startup.elc \
+ 	$(lispsource)/minibuffer.elc \
+ 	$(lispsource)/abbrev.elc \
+ 	$(lispsource)/simple.elc \
+ 	$(lispsource)/help.elc \
+ 	$(lispsource)/jka-cmpr-hook.elc \
+ 	$(lispsource)/epa-hook.elc \
  	$(lispsource)/international/mule-cmds.elc \
+ 	$(lispsource)/case-table.elc \
  	$(lispsource)/international/characters.elc \
+ 	$(lispsource)/composite.elc \
  	$(lispsource)/international/charprop.el \
  	$(lispsource)/language/chinese.elc \
  	$(lispsource)/language/cyrillic.elc \
  	$(lispsource)/language/indian.elc \
***************
*** 449,455 ****
  	$(lispsource)/language/japanese.el \
  	$(lispsource)/language/korean.el \
  	$(lispsource)/language/lao.el \
- 	$(lispsource)/language/cham.el \
  	$(lispsource)/language/tai-viet.el \
  	$(lispsource)/language/thai.el \
  	$(lispsource)/language/tibetan.elc \
--- 425,430 ----
***************
*** 459,530 ****
  	$(lispsource)/language/georgian.el \
  	$(lispsource)/language/khmer.el \
  	$(lispsource)/language/burmese.el \
! 	$(lispsource)/paths.el \
! 	$(lispsource)/register.elc \
! 	$(lispsource)/replace.elc \
! 	$(lispsource)/simple.elc \
! 	$(lispsource)/minibuffer.elc \
! 	$(lispsource)/startup.elc \
! 	$(lispsource)/subr.elc \
  	$(lispsource)/term/tty-colors.elc \
  	$(lispsource)/font-core.elc \
  	$(lispsource)/emacs-lisp/syntax.elc \
  	$(lispsource)/font-lock.elc \
  	$(lispsource)/jit-lock.elc \
! 	$(lispsource)/textmodes/fill.elc \
  	$(lispsource)/textmodes/page.elc \
  	$(lispsource)/textmodes/paragraphs.elc \
  	$(lispsource)/textmodes/text-mode.elc \
! 	$(lispsource)/emacs-lisp/timer.elc \
! 	$(lispsource)/jka-cmpr-hook.elc \
  	$(lispsource)/vc/vc-hooks.elc \
  	$(lispsource)/vc/ediff-hook.elc \
! 	$(lispsource)/epa-hook.elc \
! 	$(TOOLTIP_SUPPORT) \
! 	$(MSDOS_SUPPORT) \
! 	$(WINDOW_SUPPORT) \
! 	$(NS_SUPPORT) \
! 	$(lispsource)/widget.elc \
! 	$(lispsource)/window.elc \
! 	$(lispsource)/version.el
  
! ## List of relative names for those files from $lisp that are loaded
! ## unconditionally (i.e. on all platforms).  Files from $lisp that
! ## are only loaded on some platforms should instead be placed in
! ## SOME_MACHINE_LISP.  The only reason this variable exists is to prevent
! ## the make-docfile command-line getting too long for some systems.
  shortlisp= \
  	../lisp/loaddefs.el \
! 	../lisp/abbrev.elc \
! 	../lisp/buff-menu.elc \
! 	../lisp/button.elc \
  	../lisp/emacs-lisp/byte-run.elc \
- 	../lisp/composite.elc \
- 	../lisp/cus-face.elc \
- 	../lisp/cus-start.elc \
- 	../lisp/custom.elc \
  	../lisp/emacs-lisp/backquote.elc \
! 	../lisp/emacs-lisp/lisp-mode.elc \
! 	../lisp/emacs-lisp/lisp.elc \
! 	../lisp/facemenu.elc \
! 	../lisp/faces.elc \
! 	../lisp/files.elc \
! 	../lisp/emacs-lisp/float-sup.elc \
! 	../lisp/format.elc \
! 	../lisp/frame.elc \
! 	../lisp/help.elc \
! 	../lisp/indent.elc \
! 	../lisp/isearch.elc \
! 	../lisp/rfn-eshadow.elc \
! 	../lisp/loadup.el \
! 	../lisp/bindings.elc \
  	../lisp/emacs-lisp/map-ynp.elc \
! 	../lisp/env.elc \
  	../lisp/international/mule.elc \
  	../lisp/international/mule-conf.elc \
  	../lisp/international/mule-cmds.elc \
- 	../lisp/international/characters.elc \
  	../lisp/case-table.elc \
  	../lisp/language/chinese.elc \
  	../lisp/language/cyrillic.elc \
  	../lisp/language/indian.elc \
--- 434,528 ----
  	$(lispsource)/language/georgian.el \
  	$(lispsource)/language/khmer.el \
  	$(lispsource)/language/burmese.el \
! 	$(lispsource)/language/cham.el \
! 	$(lispsource)/indent.elc \
! 	$(lispsource)/window.elc \
! 	$(lispsource)/frame.elc \
  	$(lispsource)/term/tty-colors.elc \
  	$(lispsource)/font-core.elc \
+ 	$(lispsource)/facemenu.elc \
  	$(lispsource)/emacs-lisp/syntax.elc \
  	$(lispsource)/font-lock.elc \
  	$(lispsource)/jit-lock.elc \
! 	$(lispsource)/mouse.elc \
! 	$(lispsource)/scroll-bar.elc \
! 	$(lispsource)/select.elc \
! 	$(lispsource)/emacs-lisp/timer.elc \
! 	$(lispsource)/isearch.elc \
! 	$(lispsource)/rfn-eshadow.elc \
! 	$(lispsource)/menu-bar.elc \
! 	$(lispsource)/paths.el \
! 	$(lispsource)/emacs-lisp/lisp.elc \
  	$(lispsource)/textmodes/page.elc \
+ 	$(lispsource)/register.elc \
  	$(lispsource)/textmodes/paragraphs.elc \
+ 	$(lispsource)/emacs-lisp/lisp-mode.elc \
  	$(lispsource)/textmodes/text-mode.elc \
! 	$(lispsource)/textmodes/fill.elc \
! 	$(lispsource)/replace.elc \
! 	$(lispsource)/buff-menu.elc \
! 	$(lispsource)/fringe.elc \
! 	$(lispsource)/image.elc \
! 	$(lispsource)/international/fontset.elc \
! 	$(lispsource)/dnd.elc \
! 	$(lispsource)/tool-bar.elc \
! 	$(lispsource)/dynamic-setting.elc \
! 	$(lispsource)/x-dnd.elc \
! 	$(lispsource)/term/common-win.elc \
! 	$(lispsource)/term/x-win.elc \
! 	$(lispsource)/w32-vars.elc \
! 	$(lispsource)/term/w32-win.elc \
! 	$(lispsource)/ls-lisp.elc \
! 	$(lispsource)/disp-table.elc \
! 	$(lispsource)/dos-w32.elc \
! 	$(lispsource)/w32-fns.elc \
! 	$(lispsource)/dos-fns.elc \
! 	$(lispsource)/dos-vars.elc \
! 	$(lispsource)/term/pc-win.elc \
! 	$(lispsource)/term/internal.elc \
! 	$(lispsource)/term/ns-win.elc \
! 	$(lispsource)/mwheel.elc \
! 	$(lispsource)/emacs-lisp/float-sup.elc \
  	$(lispsource)/vc/vc-hooks.elc \
  	$(lispsource)/vc/ediff-hook.elc \
! 	$(lispsource)/tooltip.elc
  
! ## The same as $lisp, but using relative file names.  The only reason
! ## this variable exists is to prevent the make-docfile command-line
! ## getting too long for some systems.  Unfortunately, no-one has any
! ## idea how long "too long" is.
  shortlisp= \
  	../lisp/loaddefs.el \
! 	../lisp/loadup.el \
  	../lisp/emacs-lisp/byte-run.elc \
  	../lisp/emacs-lisp/backquote.elc \
! 	../lisp/subr.elc \
! 	../lisp/version.el \
! 	../lisp/widget.elc \
! 	../lisp/custom.elc \
  	../lisp/emacs-lisp/map-ynp.elc \
! 	../lisp/cus-start.elc \
  	../lisp/international/mule.elc \
  	../lisp/international/mule-conf.elc \
+ 	../lisp/env.elc \
+ 	../lisp/format.elc \
+ 	../lisp/bindings.elc \
+ 	../lisp/files.elc \
+ 	../lisp/cus-face.elc \
+ 	../lisp/faces.elc \
+ 	../lisp/button.elc \
+ 	../lisp/startup.elc \
+ 	../lisp/minibuffer.elc \
+ 	../lisp/abbrev.elc \
+ 	../lisp/simple.elc \
+ 	../lisp/help.elc \
+ 	../lisp/jka-cmpr-hook.elc \
+ 	../lisp/epa-hook.elc \
  	../lisp/international/mule-cmds.elc \
  	../lisp/case-table.elc \
+ 	../lisp/international/characters.elc \
+ 	../lisp/composite.elc \
+ 	../lisp/international/charprop.el \
  	../lisp/language/chinese.elc \
  	../lisp/language/cyrillic.elc \
  	../lisp/language/indian.elc \
***************
*** 540,546 ****
  	../lisp/language/japanese.el \
  	../lisp/language/korean.el \
  	../lisp/language/lao.el \
- 	../lisp/language/cham.el \
  	../lisp/language/tai-viet.el \
  	../lisp/language/thai.el \
  	../lisp/language/tibetan.elc \
--- 538,543 ----
***************
*** 550,599 ****
  	../lisp/language/georgian.el \
  	../lisp/language/khmer.el \
  	../lisp/language/burmese.el \
! 	../lisp/menu-bar.elc \
! 	../lisp/paths.el \
! 	../lisp/register.elc \
! 	../lisp/replace.elc \
! 	../lisp/simple.elc \
! 	../lisp/minibuffer.elc \
! 	../lisp/startup.elc \
! 	../lisp/subr.elc \
  	../lisp/term/tty-colors.elc \
  	../lisp/font-core.elc \
  	../lisp/emacs-lisp/syntax.elc \
  	../lisp/font-lock.elc \
  	../lisp/jit-lock.elc \
! 	../lisp/textmodes/fill.elc \
  	../lisp/textmodes/page.elc \
  	../lisp/textmodes/paragraphs.elc \
  	../lisp/textmodes/text-mode.elc \
! 	../lisp/emacs-lisp/timer.elc \
! 	../lisp/vc/vc-hooks.elc \
! 	../lisp/vc/ediff-hook.elc \
! 	../lisp/jka-cmpr-hook.elc \
! 	../lisp/epa-hook.elc \
! 	../lisp/widget.elc \
! 	../lisp/window.elc \
! 	../lisp/version.el
! 
! ## Like $shortlisp, but includes only those files from $lisp that are loaded
! ## conditionally (i.e., only on some platforms).
! ## Confusingly, term/internal is not in loadup, but is unconditionally
! ## loaded by pc-win, which is.
! SOME_MACHINE_LISP = ../lisp/mouse.elc \
!   ../lisp/select.elc ../lisp/scroll-bar.elc \
!   ../lisp/ls-lisp.elc ../lisp/dos-fns.elc \
!   ../lisp/w32-fns.elc ../lisp/dos-w32.elc \
!   ../lisp/disp-table.elc ../lisp/dos-vars.elc ../lisp/w32-vars.elc \
!   ../lisp/tooltip.elc ../lisp/image.elc \
!   ../lisp/fringe.elc ../lisp/dnd.elc \
!   ../lisp/mwheel.elc ../lisp/tool-bar.elc \
!   ../lisp/x-dnd.elc ../lisp/dynamic-setting.elc \
    ../lisp/international/fontset.elc \
    ../lisp/term/common-win.elc \
    ../lisp/term/x-win.elc \
!   ../lisp/term/pc-win.elc ../lisp/term/internal.elc \
!   ../lisp/term/ns-win.elc ../lisp/term/w32-win.elc
  
  ## Construct full set of libraries to be linked.
  ## Note that SunOS needs -lm to come before -lc; otherwise, you get
--- 547,604 ----
  	../lisp/language/georgian.el \
  	../lisp/language/khmer.el \
  	../lisp/language/burmese.el \
! 	../lisp/language/cham.el \
! 	../lisp/indent.elc \
! 	../lisp/window.elc \
! 	../lisp/frame.elc \
  	../lisp/term/tty-colors.elc \
  	../lisp/font-core.elc \
+ 	../lisp/facemenu.elc \
  	../lisp/emacs-lisp/syntax.elc \
  	../lisp/font-lock.elc \
  	../lisp/jit-lock.elc \
! 	../lisp/mouse.elc \
! 	../lisp/scroll-bar.elc \
! 	../lisp/select.elc \
! 	../lisp/emacs-lisp/timer.elc \
! 	../lisp/isearch.elc \
! 	../lisp/rfn-eshadow.elc \
! 	../lisp/menu-bar.elc \
! 	../lisp/paths.el \
! 	../lisp/emacs-lisp/lisp.elc \
  	../lisp/textmodes/page.elc \
+ 	../lisp/register.elc \
  	../lisp/textmodes/paragraphs.elc \
+ 	../lisp/emacs-lisp/lisp-mode.elc \
  	../lisp/textmodes/text-mode.elc \
! 	../lisp/textmodes/fill.elc \
! 	../lisp/replace.elc \
! 	../lisp/buff-menu.elc \
! 	../lisp/fringe.elc \
! 	../lisp/image.elc \
  	../lisp/international/fontset.elc \
+ 	../lisp/dnd.elc \
+ 	../lisp/tool-bar.elc \
+ 	../lisp/dynamic-setting.elc \
+ 	../lisp/x-dnd.elc \
  	../lisp/term/common-win.elc \
  	../lisp/term/x-win.elc \
! 	../lisp/w32-vars.elc \
! 	../lisp/term/w32-win.elc \
! 	../lisp/ls-lisp.elc \
! 	../lisp/disp-table.elc \
! 	../lisp/dos-w32.elc \
! 	../lisp/w32-fns.elc \
! 	../lisp/dos-fns.elc \
! 	../lisp/dos-vars.elc \
! 	../lisp/term/pc-win.elc \
! 	../lisp/term/internal.elc \
! 	../lisp/term/ns-win.elc \
! 	../lisp/mwheel.elc \
! 	../lisp/emacs-lisp/float-sup.elc \
! 	../lisp/vc/vc-hooks.elc \
! 	../lisp/vc/ediff-hook.elc \
! 	../lisp/tooltip.elc
  
  ## Construct full set of libraries to be linked.
  ## Note that SunOS needs -lm to come before -lc; otherwise, you get
***************
*** 630,645 ****
  ## $(SOME_MACHINE_OBJECTS).  Since a file is processed when it is mentioned
  ## for the first time, this prevents any variation between configurations
  ## in the contents of the DOC file.
- ## Likewise for $(SOME_MACHINE_LISP).
  ## Most of this Makefile refers to Lisp files via $(lispsource), so
  ## we also use $(lisp) rather than $(shortlisp) for the dependency since
  ## the Makefile uses string equality to decide when we talk about identical
  ## files.  Apparently we pass $(shortlisp) rather than $(lisp) to make-docfile
  ## only in order to reduce the command line length.  --Stef
! $(etc)/DOC: $(libsrc)/make-docfile$(EXEEXT) $(obj) $(lisp) $(SOME_MACHINE_LISP)
  	-rm -f $(etc)/DOC
  	$(libsrc)/make-docfile -d $(srcdir) $(SOME_MACHINE_OBJECTS) $(obj) > $(etc)/DOC
! 	$(libsrc)/make-docfile -a $(etc)/DOC -d $(srcdir) $(SOME_MACHINE_LISP) $(shortlisp)
  
  $(libsrc)/make-docfile$(EXEEXT):
  	cd $(libsrc); $(MAKE) $(MFLAGS) make-docfile$(EXEEXT)
--- 635,649 ----
  ## $(SOME_MACHINE_OBJECTS).  Since a file is processed when it is mentioned
  ## for the first time, this prevents any variation between configurations
  ## in the contents of the DOC file.
  ## Most of this Makefile refers to Lisp files via $(lispsource), so
  ## we also use $(lisp) rather than $(shortlisp) for the dependency since
  ## the Makefile uses string equality to decide when we talk about identical
  ## files.  Apparently we pass $(shortlisp) rather than $(lisp) to make-docfile
  ## only in order to reduce the command line length.  --Stef
! $(etc)/DOC: $(libsrc)/make-docfile$(EXEEXT) $(obj) $(lisp)
  	-rm -f $(etc)/DOC
  	$(libsrc)/make-docfile -d $(srcdir) $(SOME_MACHINE_OBJECTS) $(obj) > $(etc)/DOC
! 	$(libsrc)/make-docfile -a $(etc)/DOC -d $(srcdir) $(shortlisp)
  
  $(libsrc)/make-docfile$(EXEEXT):
  	cd $(libsrc); $(MAKE) $(MFLAGS) make-docfile$(EXEEXT)
***************
*** 797,803 ****
  	                     THEFILE=$< EMACS=$(bootstrap_exe)
  
  ## Since the .el.elc rule cannot specify an extra dependency, we do it here.
! $(lisp) $(SOME_MACHINE_LISP): $(BOOTSTRAPEMACS)
  
  ## VCSWITNESS points to the file that holds info about the current checkout.
  ## We use it as a heuristic to decide when to rebuild loaddefs.el.
--- 801,807 ----
  	                     THEFILE=$< EMACS=$(bootstrap_exe)
  
  ## Since the .el.elc rule cannot specify an extra dependency, we do it here.
! $(lisp): $(BOOTSTRAPEMACS)
  
  ## VCSWITNESS points to the file that holds info about the current checkout.
  ## We use it as a heuristic to decide when to rebuild loaddefs.el.

*** lisp/emacs-lisp/autoload.el	2011-05-04 15:38:41 +0000
--- lisp/emacs-lisp/autoload.el	2011-05-19 00:27:36 +0000
***************
*** 769,820 ****
  Definitions are written to `generated-autoload-file' (which
  should be non-nil)."
    ;; For use during the Emacs build process only.
    (unless autoload-excludes
!     (let* ((ldir (file-name-directory generated-autoload-file))
! 	   (default-directory
! 	     (file-name-as-directory
! 	      (expand-file-name (if (eq system-type 'windows-nt)
! 				    "../lib-src"
! 				  "../src") ldir)))
! 	   (mfile "Makefile")
! 	   (tmpfile "echolisp.tmp")
! 	   lim)
!       ;; Windows uses the 'echolisp' approach because:
!       ;; i) It does not have $lisp as a single simple definition, so
!       ;; it would be harder to parse the Makefile.
!       ;; ii) It can, since it already has $lisp broken up into pieces
!       ;; that the command-line can handle.
!       ;; Non-Windows builds do not use the 'echolisp' approach because
!       ;; no-one knows (?) the maximum safe command-line length on all
!       ;; supported systems.  $lisp is much longer there since it uses
!       ;; absolute paths, and it would seem a shame to split it just for this.
!       (when (file-readable-p mfile)
! 	(if (eq system-type 'windows-nt)
! 	    (when (ignore-errors
! 		   (if (file-exists-p tmpfile) (delete-file tmpfile))
! 		   ;; FIXME call-process is better, if it works.
! 		   (shell-command (format "%s echolisp > %s"
! 					  autoload-make-program tmpfile))
! 		   (file-readable-p tmpfile))
  	      (with-temp-buffer
! 		(insert-file-contents tmpfile)
! 		;; FIXME could be a single while loop.
! 		(while (not (eobp))
! 		  (setq lim (line-end-position))
! 		  (while (re-search-forward "\\([^ ]+\\.el\\)c?\\>" lim t)
! 		    (push (expand-file-name (match-string 1))
! 			  autoload-excludes))
! 		  (forward-line 1))))
! 	  (with-temp-buffer
! 	    (insert-file-contents mfile)
! 	    (when (re-search-forward "^shortlisp= " nil t)
! 	      (while (and (not lim)
! 			  (re-search-forward "\\.\\./lisp/\\([^ ]+\\.el\\)c?\\>"
! 					     nil t))
! 		(push (expand-file-name (match-string 1) ldir)
! 		      autoload-excludes)
! 		(skip-chars-forward " \t")
! 		(if (eolp) (setq lim t)))))))))
    (let ((args command-line-args-left))
      (setq command-line-args-left nil)
      (apply 'update-directory-autoloads args)))
--- 769,791 ----
  Definitions are written to `generated-autoload-file' (which
  should be non-nil)."
    ;; For use during the Emacs build process only.
+   ;; Exclude those files that are preloaded on ALL platforms.
+   ;; These are the ones in loadup.el where "(load" is at the start
+   ;; of the line (crude, but it works).
    (unless autoload-excludes
!     (let ((default-directory (file-name-directory generated-autoload-file))
! 	  file)
!       (when (file-readable-p "loadup.el")
! 	(dolist (f '("loadup.el" "loaddefs.el"))
! 	  (push (expand-file-name f) autoload-excludes))
  	(with-temp-buffer
! 	  (insert-file-contents "loadup.el")
! 	  (while (re-search-forward "^(load \"\\([^\"]+\\)\"" nil t)
! 	    (setq file (match-string 1))
! 	    (or (string-match "\\.el\\'" file)
! 		(setq file (format "%s.el" file)))
! 	    (or (string-match "\\`site-" file)
! 		(push (expand-file-name file) autoload-excludes)))))))
    (let ((args command-line-args-left))
      (setq command-line-args-left nil)
      (apply 'update-directory-autoloads args)))

bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Stefan Monnier]

> So I think we can get rid of the "some_machine" lisp distinction.
> Does anyone see a problem with that?

If it works correctly, that's great.
Eliminating the shortlisp-vs-lisp duplication would be the next
logical step.


        Stefan

bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Glenn Morris]

Stefan Monnier wrote:

> If it works correctly, that's great.

It works for me... Let's see if it works for everyone else.

> Eliminating the shortlisp-vs-lisp duplication would be the next
> logical step.

Still can't see how to do that. Hopefully it's a bit easier to keep
up-to-date now it is almost a literal copy of $lisp.

Something clever with xargs?

Or, $lisp could be in a separate lisp.mk file that configure could
include into src/Makefile, and then for DOC use something like:

  $(libsrc)/make-docfile -a $(etc)/DOC \
    -d $(srcdir) `sed 's|$(lispsource)|../lisp|' $(srcdir)/lisp.mk`

?

bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Dan Nicolaescu]

Glenn Morris <rgm@gnu.org> writes:

> Stefan Monnier wrote:
>
>> If it works correctly, that's great.
>
> It works for me... Let's see if it works for everyone else.
>
>> Eliminating the shortlisp-vs-lisp duplication would be the next
>> logical step.
>
> Still can't see how to do that. Hopefully it's a bit easier to keep
> up-to-date now it is almost a literal copy of $lisp.
>
> Something clever with xargs?
>
> Or, $lisp could be in a separate lisp.mk file that configure could
> include into src/Makefile, and then for DOC use something like:
>
>   $(libsrc)/make-docfile -a $(etc)/DOC \
>     -d $(srcdir) `sed 's|$(lispsource)|../lisp|' $(srcdir)/lisp.mk`

Or even better make make-docfile accept arguments from a file, using the
@FILE syntax used by other GNU tools.  That would deal with the command
line length limitations.

bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Glenn Morris]

Dan Nicolaescu wrote:

>> Or, $lisp could be in a separate lisp.mk file that configure could
>> include into src/Makefile, and then for DOC use something like:
>>
>>   $(libsrc)/make-docfile -a $(etc)/DOC \
>>     -d $(srcdir) `sed 's|$(lispsource)|../lisp|' $(srcdir)/lisp.mk`

Works for me; committed.

> Or even better make make-docfile accept arguments from a file, using the
> @FILE syntax used by other GNU tools.  That would deal with the command
> line length limitations.

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7397

By itself, I don't see how this helps, since you still need to find a
way to write the input file without overflowing the command line length.

bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Dan Nicolaescu]

Glenn Morris <rgm@gnu.org> writes:

> Dan Nicolaescu wrote:
>
>>> Or, $lisp could be in a separate lisp.mk file that configure could
>>> include into src/Makefile, and then for DOC use something like:
>>>
>>>   $(libsrc)/make-docfile -a $(etc)/DOC \
>>>     -d $(srcdir) `sed 's|$(lispsource)|../lisp|' $(srcdir)/lisp.mk`
>
> Works for me; committed.
>
>> Or even better make make-docfile accept arguments from a file, using the
>> @FILE syntax used by other GNU tools.  That would deal with the command
>> line length limitations.
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7397
>
> By itself, I don't see how this helps, since you still need to find a
> way to write the input file without overflowing the command line length.

Use >> ?

BTW, the @FILE syntax is in wide use, especially for linking, where
command lines can be humongous.

[This might be a moot point now that a solution has been implemented, so
please don't keep this bug open just for this reason]

bug#7397: bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Glenn Morris]

( Moving comments from bug#8302 to bug#7397 )

Dan Nicolaescu wrote:

>>> Or even better make make-docfile accept arguments from a file, using the
>>> @FILE syntax used by other GNU tools.  That would deal with the command
>>> line length limitations.
>>
>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7397
>>
>> By itself, I don't see how this helps, since you still need to find a
>> way to write the input file without overflowing the command line length.
>
> Use >> ?

I still don't get it. You cannot do:
    
    for file in $lisp; do
      sed '<blah>' $lisp >> input-file
    done
    
    make-docfile @input-file

because the "for file..." line could be too long for the shell.

> [This might be a moot point now that a solution has been implemented, so
> please don't keep this bug open just for this reason]

I may close #7397 in a bit then.

bug#7397: bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Glenn Morris]

Glenn Morris wrote:

> I still don't get it.

Oh, I guess the src/Makefile .el.elc rule could also have appended to
the make-docfile input file as it compiled each .el file, or something.

bug#7397: bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Stefan Monnier]

>> I still don't get it.
> Oh, I guess the src/Makefile .el.elc rule could also have appended to
> the make-docfile input file as it compiled each .el file, or something.

This would require running the byte-compiler without a DOC file.
IIRC this may require a few tweaks.


        Stefan

bug#7397: make-docfile should support the @FILE command line option

[Dan Nicolaescu]

src/Makefile.in contains some ugliness just in order to reduce the
command line length for make-docfile.

Instead make-docfile could support the @FILE syntax.  That would
remove the command line length limitation.

Snarfed from the GNU ld info page:

`@FILE'
     Read command-line options from FILE.  The options read are
     inserted in place of the original @FILE option. 

bug#7397: make-docfile should support the @FILE command line option

[Eli Zaretskii]

> From: Dan Nicolaescu <dann@gnu.org>
> Date: Sun, 14 Nov 2010 04:01:28 -0500
> Cc: 
> 
> src/Makefile.in contains some ugliness just in order to reduce the
> command line length for make-docfile.
> 
> Instead make-docfile could support the @FILE syntax.  That would
> remove the command line length limitation.

I'm not against adding this feature to make-docfile, but are we sure
this problem is real nowadays?  What systems cannot grok the full list
of Lisp files?  shortlisp is 2.2KB long; how much longer can the full
list be?  I don't think we have any system supported that cannot
handle at least 10K long commands, do we?

Maybe we should simply dump this ugliness, and see if someone hollers?

bug#7397: make-docfile should support the @FILE command line option

[Dan Nicolaescu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dan Nicolaescu <dann@gnu.org>
>> Date: Sun, 14 Nov 2010 04:01:28 -0500
>> Cc: 
>> 
>> src/Makefile.in contains some ugliness just in order to reduce the
>> command line length for make-docfile.
>> 
>> Instead make-docfile could support the @FILE syntax.  That would
>> remove the command line length limitation.
>
> I'm not against adding this feature to make-docfile, but are we sure
> this problem is real nowadays?  What systems cannot grok the full list
> of Lisp files?  shortlisp is 2.2KB long; how much longer can the full
> list be?  I don't think we have any system supported that cannot
> handle at least 10K long commands, do we?
>
> Maybe we should simply dump this ugliness, and see if someone hollers?

That's an option, but it might cause problems down the road when the
length increases again...
@FILE guarantees that we won't have any problems in the future.

bug#7397: bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Glenn Morris]

Stefan Monnier wrote:

> This would require running the byte-compiler without a DOC file.
> IIRC this may require a few tweaks.

I don't think this needs to be implemented, and I think I am probably
missing some obvious way how it would work, so this is all academic
anyway, but:

Huh? The DOC file requires the .elc files to be compiled first.
bootstrap-emacs compiles files just fine without a DOC file.

bug#7397: bug#8302: 23.3; Parallel make: some *.el files are byte-compiled twice

[Stefan Monnier]

> Huh? The DOC file requires the .elc files to be compiled first.
> bootstrap-emacs compiles files just fine without a DOC file.

I guess I'm just remembering the problems I encountered when
implementing the bootstrap, but that I did fix them all at that time.
Good, thanks,


        Stefan

bug#7397: make-docfile should support the @FILE command line option

[Glenn Morris]

I don't think we need to keep this open.
The src/Makefile ugliness is hopefully reduced/gone now.