Possibility of a ${srcdir}/info/dir target race?

Possibility of a ${srcdir}/info/dir target race?

[Rob Browning]

I'm currently investigating a 24.4 build failure on s390x which is
beginning to look like it might be a make -j2 race (the Debian buildds
use -j2) with respect to the info/dir target:

  https://buildd.debian.org/status/package.php?p=emacs24&suite=experimental

  Success: https://buildd.debian.org/status/fetch.php?pkg=emacs24&arch=i386&ver=24.4%2B1-1&stamp=1413961402
  Failure: https://buildd.debian.org/status/fetch.php?pkg=emacs24&arch=s390x&ver=24.4%2B1-1&stamp=1413962421

Running the build in a sid chroot on zelenka (s390x porter box) by hand,
both with (and once) without -j2 works fine, but of course that doesn't
provide any definitive answer.

And please note that the problem might well be the fault of (my) Debian
modifications, but I thought I'd go ahead and ask here (while still
investigating) in case this rings a bell.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibility of a ${srcdir}/info/dir target race?

[Stefan Monnier]

> I'm currently investigating a 24.4 build failure on s390x which is
> beginning to look like it might be a make -j2 race (the Debian buildds
> use -j2) with respect to the info/dir target:

Such races are possible, of course, but lots of us use "-j2" or more to
build Emacs, I expect, so if that's the problem, you probably
got very unlucky.


        Stefan

Re: Possibility of a ${srcdir}/info/dir target race?

[Glenn Morris]

Stefan Monnier wrote:

>> I'm currently investigating a 24.4 build failure on s390x which is
>> beginning to look like it might be a make -j2 race (the Debian buildds
>> use -j2) with respect to the info/dir target:
>
> Such races are possible, of course, but lots of us use "-j2" or more to
> build Emacs, I expect, so if that's the problem, you probably
> got very unlucky.

I don't even see how it's possible, since this rule uses $$ to get a
different random number in the file name each time it runs.

(I also don't see why this rule runs at all, since info/ is pre-built in
release tarfiles. Presumably Debian-specific changes.)

Re: Possibility of a ${srcdir}/info/dir target race?

[Rob Browning]

Glenn Morris <rgm@gnu.org> writes:

> (I also don't see why this rule runs at all, since info/ is pre-built in
> release tarfiles. Presumably Debian-specific changes.)

Right -- that part is Debian specific (and expected).

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibility of a ${srcdir}/info/dir target race?

[Rob Browning]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Such races are possible, of course, but lots of us use "-j2" or more to
> build Emacs, I expect, so if that's the problem, you probably
> got very unlucky.

So far I'm not sure how the buildd output in question could have been
produced, unless make-info-dir is failing internally.  (I suppose that's
possible, if unlikely, since it doesn't check for failures when invoking
cp, etc.)

In the output we have:

  (cd . && ./build-aux/make-info-dir ${thisdir}/${tempfile}); \
  ./build-aux/move-if-change ${tempfile} ./info/dir
  Creating /«BUILDDIR»/emacs24-24.4+1/debian/build-nox/info-dir.30059...
  [...]
  make[4]: Entering directory '/«BUILDDIR»/emacs24-24.4+1/debian/build-nox/doc/misc'
  [...]
  Created /«BUILDDIR»/emacs24-24.4+1/debian/build-nox/info-dir.30059

At this point, unless the unconditional "cp $basefile $outfile" command
in make-info-dir failed, then info-dir.30059 should exist.  But then
immediately afterward we see:

  mv: cannot move 'info-dir.30059' to './info/dir': No such file or directory

Which means that either the file never existed, or it was somehow
deleted before move-if-change (assuming that's the source of the mv)
gets to it.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibility of a ${srcdir}/info/dir target race?

[Paul Eggert]

Rob Browning wrote:
> mv: cannot move 'info-dir.30059' to './info/dir': No such file or directory
>
> Which means that either the file never existed, or it was somehow
> deleted

... or the directory './info' doesn't exist, and that indeed is a race.

I fixed this bug in trunk bzr 118181, along with another bug (wildcards weren't 
expanded in dependencies) and some performance tuneups.  It's not urgent to 
apply these changes to the emacs-24 branch as the problems shouldn't occur when 
building from an Emacs source distribution.

Re: Possibility of a ${srcdir}/info/dir target race?

[Glenn Morris]

Paul Eggert wrote:

> ... or the directory './info' doesn't exist, and that indeed is a race.

Why would info/, which is included in release tarfiles and not deleted
by any rule AFAICS, go missing?

Re: Possibility of a ${srcdir}/info/dir target race?

[Glenn Morris]

Paul Eggert wrote:

> I fixed this bug in trunk bzr 118181, along with another bug
> (wildcards weren't expanded in dependencies) and some performance
> tuneups.  It's not urgent to apply these changes to the emacs-24
> branch as the problems shouldn't occur when building from an Emacs
> source distribution.

Surely the actual relevant change is only to add ${MKDIR_P}
${srcdir}/info, which could easily have been applied to emacs-24, only
it's mixed in with unrelated performance improvements?

And the "wildcards" thing is only relevant if you've deleted
every texi file from doc/misc (which even Debian doesn't do)?
Similar constructs have been used for years in eg lisp/Makefile.in
lisptagsfiles etc.

Re: Possibility of a ${srcdir}/info/dir target race?

[Glenn Morris]

Glenn Morris wrote:

> Paul Eggert wrote:
>
>> ... or the directory './info' doesn't exist, and that indeed is a race.
>
> Why would info/, which is included in release tarfiles and not deleted
> by any rule AFAICS, go missing?

Oh, bootstrap-clean deletes it.

No idea why that would be run when building from a tarfile.

(Does the Debian build process really need to diverge from the normal
tarfile build process so much?)

Re: Possibility of a ${srcdir}/info/dir target race?

[Paul Eggert]

Glenn Morris wrote:
> Surely the actual relevant change is only to add ${MKDIR_P}
> ${srcdir}/info, which could easily have been applied to emacs-24, only
> it's mixed in with unrelated performance improvements?

I didn't bother, because I didn't think any part of the change should be applied 
to emacs-24.  As you mentioned, an 'info' dir shouldn't go missing with any 
normal use of an Emacs distribution tarball.  The Debian build process is quite 
specialized and surely the Debian folks can get it to work themselves, e.g., by 
doing a "mkdir info" after the "make bootstrap-clean" (or better yet, omit the 
bootstrap-clean).

> Similar constructs have been used for years in eg lisp/Makefile.in
> lisptagsfiles etc.

Ah, sorry, I was wrong about the wildcards.  I didn't know that GNU make 
expanded wildcards in prerequisites.  So that part of the change didn't fix any 
bugs (though the use of the 'wildcard' function is still helpful).  I have fixed 
the corresponding ChangeLog entry accordingly, in trunk bzr 118182.  That bzr 
also removes a couple of other calls to 'wildcard' that were present only 
because of the same misunderstanding of how wildcards work in prerequisites.

Re: Possibility of a ${srcdir}/info/dir target race?

[Rob Browning]

Glenn Morris <rgm@gnu.org> writes:

> (Does the Debian build process really need to diverge from the normal
> tarfile build process so much?)

It's quite likely that the way I handle things can be improved, but the
current approach is due to:

  - The DFSG split (unfortunately)

  - History/accretion[1]

  - Debian's need to keep regularly updated auto* files (architectures)

  - My desire to maintain the package as branches of the upstream git
    repository (via git-dpm)

[1] As an example (if I don't misremember), at least in the past the
    build process might alter the shipped .elc files in the source tree,
    which caused difficulties with respect to the Debian diff (also
    affected by the fact that Debian patches some .el files -- though
    I've been steadily reducing that).

    However I eventually just switched to a brute-force approach, where
    we make three entire copies of the source in
    debian/build-{x,nox,lucid}, so that may not be an issue anymore.

Hope this helps
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibility of a ${srcdir}/info/dir target race?

[Rob Browning]

Glenn Morris <rgm@gnu.org> writes:

> And the "wildcards" thing is only relevant if you've deleted
> every texi file from doc/misc (which even Debian doesn't do)?

We don't -- we currently leave efaq*.texi.  The rest move to the
non-dfsg package(s).

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibility of a ${srcdir}/info/dir target race?

[Rob Browning]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I fixed this bug in trunk bzr 118181, along with another bug (wildcards weren't 
> expanded in dependencies) and some performance tuneups.  It's not urgent to 
> apply these changes to the emacs-24 branch as the problems shouldn't occur when 
> building from an Emacs source distribution.

Great, and thanks.  I'll add the mkdir to Debian's 24.4+1-3.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4