Fixing parallel byte-compilation

Fixing parallel byte-compilation

[Glenn Morris]

Parallel bootstrap sometimes fails if one Emacs process tries to read
an .elc file that another is in the middle of writing. See eg

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=4196
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=6858

and recent mails on this list.

One suggested solution was to compile to a temp-file, then move it in
place. That seems simple to do...?


*** lisp/emacs-lisp/bytecomp.el	2010-09-08 16:02:38 +0000
--- lisp/emacs-lisp/bytecomp.el	2010-09-10 23:42:25 +0000
***************
*** 1698,1714 ****
  	  (insert "\n")			; aaah, unix.
  	    (if (file-writable-p target-file)
  		;; We must disable any code conversion here.
! 		(let ((coding-system-for-write 'no-conversion))
  		  (if (memq system-type '(ms-dos 'windows-nt))
  		      (setq buffer-file-type t))
! 		  (when (file-exists-p target-file)
! 		    ;; Remove the target before writing it, so that any
! 		    ;; hard-links continue to point to the old file (this makes
! 		    ;; it possible for installed files to share disk space with
! 		    ;; the build tree, without causing problems when emacs-lisp
! 		    ;; files in the build tree are recompiled).
! 		    (delete-file target-file))
! 		  (write-region (point-min) (point-max) target-file))
  	      ;; This is just to give a better error message than write-region
  	      (signal 'file-error
  		      (list "Opening output file"
--- 1698,1720 ----
  	  (insert "\n")			; aaah, unix.
  	    (if (file-writable-p target-file)
  		;; We must disable any code conversion here.
! 		(let ((coding-system-for-write 'no-conversion)
! 		      ;; Write to a tempfile so that if another Emacs
! 		      ;; process is trying to load target-file (eg in a
! 		      ;; parallel bootstrap), it does not risk getting a
! 		      ;; half-finished file.  (Bug#4196)
! 		      (tempfile (make-temp-name target-file)))
  		  (if (memq system-type '(ms-dos 'windows-nt))
  		      (setq buffer-file-type t))
! 		  ;; The old code used to: delete target-file before
! 		  ;; writing it, so that any hard-links continue to
! 		  ;; point to the old file (this makes it possible
! 		  ;; for installed files to share disk space with
! 		  ;; the build tree, without causing problems when
! 		  ;; emacs-lisp files in the build tree are
! 		  ;; recompiled).  Renaming works the same way.
! 		  (write-region (point-min) (point-max) tempfile)
! 		  (rename-file tempfile target-file t))
  	      ;; This is just to give a better error message than write-region
  	      (signal 'file-error
  		      (list "Opening output file"

Re: Fixing parallel byte-compilation

[Chad Brown]

I'll hazard a guess that this solves the `exact-same-time' problem,
but still leaves dueling emacs builds with potentially incompatible
.elc files if there's a multi-file change, and so it wasn't deemed
worth the trouble.  Of course, applying a patch someone already
wrote is less effort, so... :-)

*Chad

On Sep 10, 2010, at 4:51 PM, Glenn Morris wrote:

> 
> Parallel bootstrap sometimes fails if one Emacs process tries to read
> an .elc file that another is in the middle of writing. See eg
> 
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=4196
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=6858
> 
> and recent mails on this list.
> 
> One suggested solution was to compile to a temp-file, then move it in
> place. That seems simple to do...?
> 

Re: Fixing parallel byte-compilation

[Glenn Morris]

Chad Brown wrote:

> I'll hazard a guess that this solves the `exact-same-time' problem,
> but still leaves dueling emacs builds with potentially incompatible
> .elc files if there's a multi-file change,

Sorry, I don't understand what you mean by the second part.
Bootstrapping starts by deleting all .elc files, if that's relevant.

Re: Fixing parallel byte-compilation

[Stefan Monnier]

> One suggested solution was to compile to a temp-file, then move it in
> place. That seems simple to do...?

Thanks, feel free to install it,


        Stefan
        

Re: Fixing parallel byte-compilation

[Glenn Morris]

Bah, just noticed it makes the build log rather ugly:

 Compiling org/org-freemind.el
 Wrote /path/to/lisp/org/org-freemind.elc8242rG9

Maybe I should disable the "Wrote file" message from write-region, and
have byte-compile-file print it instead? It already prints any error
message by hand.

Re: Fixing parallel byte-compilation

[Stefan Monnier]

> Bah, just noticed it makes the build log rather ugly:

>  Compiling org/org-freemind.el
>  Wrote /path/to/lisp/org/org-freemind.elc8242rG9

> Maybe I should disable the "Wrote file" message from write-region, and
> have byte-compile-file print it instead?

Yup,


        Stefan

Re: Fixing parallel byte-compilation

[Davis Herring]

> ! 		  ;; The old code used to: delete target-file before
> ! 		  ;; writing it, so that any hard-links continue to
> ! 		  ;; point to the old file (this makes it possible
> ! 		  ;; for installed files to share disk space with
> ! 		  ;; the build tree, without causing problems when
> ! 		  ;; emacs-lisp files in the build tree are
> ! 		  ;; recompiled).  Renaming works the same way.
> ! 		  (write-region (point-min) (point-max) tempfile)
> ! 		  (rename-file tempfile target-file t))

Does `rename-file' work outside of POSIX (that is, on Windows) if the
target file exists?

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Fixing parallel byte-compilation

[Eli Zaretskii]

> Date: Mon, 13 Sep 2010 09:33:21 -0700 (PDT)
> From: "Davis Herring" <herring@lanl.gov>
> Cc: emacs-devel@gnu.org
> 
> > ! 		  ;; The old code used to: delete target-file before
> > ! 		  ;; writing it, so that any hard-links continue to
> > ! 		  ;; point to the old file (this makes it possible
> > ! 		  ;; for installed files to share disk space with
> > ! 		  ;; the build tree, without causing problems when
> > ! 		  ;; emacs-lisp files in the build tree are
> > ! 		  ;; recompiled).  Renaming works the same way.
> > ! 		  (write-region (point-min) (point-max) tempfile)
> > ! 		  (rename-file tempfile target-file t))
> 
> Does `rename-file' work outside of POSIX (that is, on Windows) if the
> target file exists?

Not for free, but it does.  Windows has sys_rename, which see in
w32.c, while the DOS build uses a standard C library whose `rename'
does that under the hood.

Of course, this could still fail if the target is open by some other
program, but the old code uses delete-file which would fail in that
case as well.  So we didn't make the situation worse than it was
already.

Re: Fixing parallel byte-compilation

[Glenn Morris]

Eli Zaretskii wrote:

> So we didn't make the situation worse than it was already.

Adopting this as my motto: "not making the situation worse than it was
already."