Spurious "file errors" when making master

Spurious "file errors" when making master

[martin rudalics]

I recently get spurious "file errors" when building master.  The first one
is from Windows XP and goes as

 >>Error occurred processing ../../lisp/gnus/gnus-group.el: File error (("Renaming" "Permission denied" "c:/emacs-git/trunk/lisp/gnus/gnus-group.elc2552Dme" "c:/emacs-git/trunk/lisp/gnus/gnus-group.elc"))

That file still has its extension .elc2552Dme.

The second one is from Debian and goes as

Error running git rev-parse: (file-error "Searching for program" "No such file or directory" "git")

Any ideas?

Thanks, martin

Re: Spurious "file errors" when making master

[Paul Eggert]

martin rudalics wrote:
> The second one is from Debian and goes as
>
> Error running git rev-parse: (file-error "Searching for program" "No such file
> or directory" "git")

I don't observe this on Ubuntu 16.04. When are you seeing this? autogen.sh? 
gitlog-to-changelog? emacs-repository-version-git?

I assume you have git installed; is it on a network directory that is flaky, or 
perhaps something is setting PATH to a value that does not include the directory 
where git is?

Re: Spurious "file errors" when making master

[martin rudalics]

 >> Error running git rev-parse: (file-error "Searching for program" "No such file
 >> or directory" "git")
 >
 > I don't observe this on Ubuntu 16.04. When are you seeing this? autogen.sh? gitlog-to-changelog? emacs-repository-version-git?

Before dumping.  The context is:

...
Loading dynamic-setting...
Loading x-dnd...
Loading term/common-win...
Loading term/x-win...
Loading mwheel...
Loading emacs-lisp/float-sup...
Loading vc/vc-hooks...
Loading vc/ediff-hook...
Loading uniquify...
Loading electric...
Loading emacs-lisp/eldoc...
Loading cus-start...
Loading tooltip...
Loading leim/leim-list.el (source)...
Waiting for git...
Error running git rev-parse: (file-error "Searching for program" "No such file or directory" "git")
Finding pointers to doc strings...
Finding pointers to doc strings...done
Pure-hashed: 24257 strings, 3832 vectors, 38724 conses, 3753 bytecodes, 103 others
Dumping under the name emacs
2798849 pure bytes used
Adding name emacs-25.1.50.18
ln -f emacs bootstrap-emacs
make[1]: Leaving directory `/home/martin/emacs-git/quick/obj-lucid/src'
make -C lisp all
make[1]: Entering directory `/home/martin/emacs-git/quick/obj-lucid/lisp'
make -C ../leim all EMACS="../src/emacs"
make -C ../admin/grammars all EMACS="../../src/emacs"
...

 > I assume you have git installed; is it on a network directory that is flaky, or perhaps something is setting PATH to a value that does not include the directory where git is?

git is on my machine in usr/bin and PATH includes usr/bin.  I'm using
git all the time without problems.

martin

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Thu, 26 May 2016 09:30:39 +0200
> From: martin rudalics <rudalics@gmx.at>
> 
> I recently get spurious "file errors" when building master.  The first one
> is from Windows XP and goes as
> 
>  >>Error occurred processing ../../lisp/gnus/gnus-group.el: File error (("Renaming" "Permission denied" "c:/emacs-git/trunk/lisp/gnus/gnus-group.elc2552Dme" "c:/emacs-git/trunk/lisp/gnus/gnus-group.elc"))
> 
> That file still has its extension .elc2552Dme.

Having a temporary file with such an extension is normal, you can see
that in the code.

Do you have anti-virus software installed on that machine, and if so,
did you lately change it configuration or upgraded to a different
version?

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Thu, 26 May 2016 10:20:00 +0200
> From: martin rudalics <rudalics@gmx.at>
> 
>  >> Error running git rev-parse: (file-error "Searching for program" "No such file
>  >> or directory" "git")
>  >
>  > I don't observe this on Ubuntu 16.04. When are you seeing this? autogen.sh? gitlog-to-changelog? emacs-repository-version-git?
> 
> Before dumping.  The context is:
> 
> ...
> Loading dynamic-setting...
> Loading x-dnd...
> Loading term/common-win...
> Loading term/x-win...
> Loading mwheel...
> Loading emacs-lisp/float-sup...
> Loading vc/vc-hooks...
> Loading vc/ediff-hook...
> Loading uniquify...
> Loading electric...
> Loading emacs-lisp/eldoc...
> Loading cus-start...
> Loading tooltip...
> Loading leim/leim-list.el (source)...
> Waiting for git...
> Error running git rev-parse: (file-error "Searching for program" "No such file or directory" "git")
> Finding pointers to doc strings...
> Finding pointers to doc strings...done

I'd suggest to instrument the place where this happens to display the
value of exec-path.

Re: Spurious "file errors" when making master

[Paul Eggert]

On 05/26/2016 01:20 AM, martin rudalics wrote:
> Loading leim/leim-list.el (source)...
> Waiting for git...
> Error running git rev-parse: (file-error "Searching for program" "No 
> such file or directory" "git") 

Ah, this appears to be Bug#20799. It's odd that you run into it while I 
do not. I wonder why that would be? For what it's worth I'm running in a 
full clone, where "git diff" reports no differences.

Re: Spurious "file errors" when making master

[martin rudalics]

> Do you have anti-virus software installed on that machine, and if so,
> did you lately change it configuration or upgraded to a different
> version?

No such software on that machine.

martin

Re: Spurious "file errors" when making master

[martin rudalics]

 >> Waiting for git...
 >> Error running git rev-parse: (file-error "Searching for program" "No such file or directory" "git")
 >> Finding pointers to doc strings...
 >> Finding pointers to doc strings...done
 >
 > I'd suggest to instrument the place where this happens to display the
 > value of exec-path.

Can someone please tell me how.

TIA, martin

Re: Spurious "file errors" when making master

[martin rudalics]

 >> Loading leim/leim-list.el (source)...
 >> Waiting for git...
 >> Error running git rev-parse: (file-error "Searching for program" "No such file or directory" "git")
 >
 > Ah, this appears to be Bug#20799. It's odd that you run into it while I do not. I wonder why that would be? For what it's worth I'm running in a full clone, where "git diff" reports no differences.

Hmm.... I thought bug#20799 was Windows-only.  And I don't see this on
Windows but only on Debian.

I'm not sure whether mine was a full clone or had changes.  I'll tell
you the next time this happens,

martin

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Fri, 27 May 2016 15:17:35 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
>  >> Waiting for git...
>  >> Error running git rev-parse: (file-error "Searching for program" "No such file or directory" "git")
>  >> Finding pointers to doc strings...
>  >> Finding pointers to doc strings...done
>  >
>  > I'd suggest to instrument the place where this happens to display the
>  > value of exec-path.
> 
> Can someone please tell me how.

(message "%s" exec-path)

Re: Spurious "file errors" when making master

[martin rudalics]

>> Can someone please tell me how.
>
> (message "%s" exec-path)

Where do I put that?

martin

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Fri, 27 May 2016 15:34:43 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> >> Can someone please tell me how.
> >
> > (message "%s" exec-path)
> 
> Where do I put that?

In emacs-repository-version-git, I guess: that's where the "Waiting
for git" message is coming from.

My crystal ball says exec-path will come out nil, btw.

Re: Spurious "file errors" when making master

[martin rudalics]

 >>> (message "%s" exec-path)
 >>
 >> Where do I put that?
 >
 > In emacs-repository-version-git, I guess: that's where the "Waiting
 > for git" message is coming from.
 >
 > My crystal ball says exec-path will come out nil, btw.

Your crystal ball is right, as usual btw.

martin

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Fri, 27 May 2016 15:59:29 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
>  >>> (message "%s" exec-path)
>  >>
>  >> Where do I put that?
>  >
>  > In emacs-repository-version-git, I guess: that's where the "Waiting
>  > for git" message is coming from.
>  >
>  > My crystal ball says exec-path will come out nil, btw.
> 
> Your crystal ball is right, as usual btw.

So it's bug#20799, allright.  Hmm...

I'm not sure I understand why we invoke Git in this case: the
source-directory is known and used by emacs-repository-get-version, so
why cannot it do the same it does for an in-tree build, i.e. access
the files in .git/ directly, just relative to source-directory?  What
am I missing?

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Fri, 27 May 2016 15:17:55 +0200
> From: martin rudalics <rudalics@gmx.at>
> 
> Hmm.... I thought bug#20799 was Windows-only.  And I don't see this on
> Windows but only on Debian.

No, it's not Windows-only.  (Glenn doesn't run Windows ;-)

Re: Spurious "file errors" when making master

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

Eli Zaretskii wrote:
> I'm not sure I understand why we invoke Git in this case: the
> source-directory is known and used by emacs-repository-get-version, so
> why cannot it do the same it does for an in-tree build, i.e. access
> the files in .git/ directly, just relative to source-directory?

Yes, that's how it works for me.  If I try the attached debugging patch, 'make' 
chatters as follows:

Loading leim/leim-list.el (source)...
emacs-repository--version-git-1 
/home/eggert/src/gnu/emacs/static-checking/.git/HEAD 
/home/eggert/src/gnu/emacs/static-checking/.git
emacs-repository--version-git-1 
/home/eggert/src/gnu/emacs/static-checking/.git/refs/heads/master 
/home/eggert/src/gnu/emacs/static-checking/.git
Finding pointers to doc strings...
Finding pointers to doc strings...done

which is what I'd expect: git does not get executed. Martin, can you investigate 
why git is executed in your case?

[-- Attachment #2: t.patch --]
[-- Type: text/x-diff, Size: 435 bytes --]

diff --git a/lisp/version.el b/lisp/version.el
index dc22887..ec04154 100644
--- a/lisp/version.el
+++ b/lisp/version.el
@@ -118,6 +118,7 @@ emacs-repository-version-git
 
 (defun emacs-repository--version-git-1 (file dir)
   "Internal subroutine of `emacs-repository-get-version'."
+  (message "emacs-repository--version-git-1 %s %s" file dir)
   (when (file-readable-p file)
     (with-temp-buffer
       (insert-file-contents file)

Re: Spurious "file errors" when making master

[martin rudalics]

 > Yes, that's how it works for me.  If I try the attached debugging patch, 'make' chatters as follows:
 >
 > Loading leim/leim-list.el (source)...
 > emacs-repository--version-git-1 /home/eggert/src/gnu/emacs/static-checking/.git/HEAD /home/eggert/src/gnu/emacs/static-checking/.git
 > emacs-repository--version-git-1 /home/eggert/src/gnu/emacs/static-checking/.git/refs/heads/master /home/eggert/src/gnu/emacs/static-checking/.git
 > Finding pointers to doc strings...
 > Finding pointers to doc strings...done
 >
 > which is what I'd expect: git does not get executed. Martin, can you investigate why git is executed in your case?

I have some 10 separate sub-directories in my emacs directories each for
the builds of the corresponding toolkit (Gtk, lucid, motif, ...).  These
are not tracked by git.  Could that be the cause?  The messages printed
by your patch are:

emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/HEAD /home/martin/emacs-git/release/.git
emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/refs/heads/master /home/martin/emacs-git/release/.git

martin

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Sun, 29 May 2016 15:44:31 +0200
> From: martin rudalics <rudalics@gmx.at>
> Cc: emacs-devel@gnu.org
> 
> I have some 10 separate sub-directories in my emacs directories each for
> the builds of the corresponding toolkit (Gtk, lucid, motif, ...).  These
> are not tracked by git.  Could that be the cause?

I don't see why it would: what matters is the source-directory, and
that is the same for all of those build sub-directories, right?

> The messages printed by your patch are:
> 
> emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/HEAD /home/martin/emacs-git/release/.git
> emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/refs/heads/master /home/martin/emacs-git/release/.git

Are these file names incorrect?

Re: Spurious "file errors" when making master

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

martin rudalics wrote:
> The messages printed
> by your patch are:
>
> emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/HEAD
> /home/martin/emacs-git/release/.git
> emacs-repository--version-git-1
> /home/martin/emacs-git/release/.git/refs/heads/master
> /home/martin/emacs-git/release/.git

What is the contents of /home/martin/emacs-git/release/.git/refs/heads/master? 
What happens if you try the attached patch instead? On my system, it outputs this:

...
Loading leim/leim-list.el (source)...
emacs-repository--version-git-1 
/home/eggert/src/gnu/emacs/static-checking/.git/HEAD 
/home/eggert/src/gnu/emacs/static-checking/.git
emacs-repository--version-git-1 
/home/eggert/src/gnu/emacs/static-checking/.git/refs/heads/master 
/home/eggert/src/gnu/emacs/static-checking/.git
emacs-repository--version-git-1 returns 062757dca01cff127583438ae16237af55b6253c
emacs-repository--version-git-1 returns 062757dca01cff127583438ae16237af55b6253c
Finding pointers to doc strings...
Finding pointers to doc strings...done
...


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: version.patch --]
[-- Type: text/x-diff; name="version.patch", Size: 886 bytes --]

diff --git a/lisp/version.el b/lisp/version.el
index dc22887..1b18e23 100644
--- a/lisp/version.el
+++ b/lisp/version.el
@@ -118,6 +118,8 @@ emacs-repository-version-git
 
 (defun emacs-repository--version-git-1 (file dir)
   "Internal subroutine of `emacs-repository-get-version'."
+  (message "emacs-repository--version-git-1 %s %s" file dir)
+  (let ((result
   (when (file-readable-p file)
     (with-temp-buffer
       (insert-file-contents file)
@@ -127,6 +129,9 @@ emacs-repository--version-git-1
              (emacs-repository--version-git-1
               (expand-file-name (match-string 1) dir)
               dir))))))
+        )
+    (message "emacs-repository--version-git-1 returns %s" result)
+    result))
 
 (defun emacs-repository-get-version (&optional dir external)
   "Try to return as a string the repository revision of the Emacs sources.

Re: Spurious "file errors" when making master

[martin rudalics]

 > I don't see why it would: what matters is the source-directory, and
 > that is the same for all of those build sub-directories, right?

It's the same source-directory for all of them, yes.

 >> The messages printed by your patch are:
 >>
 >> emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/HEAD /home/martin/emacs-git/release/.git
 >> emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/refs/heads/master /home/martin/emacs-git/release/.git
 >
 > Are these file names incorrect?

The directories and files exist and can be read if it's that what you
mean.

martin

Re: Spurious "file errors" when making master

[martin rudalics]

 > What is the contents of /home/martin/emacs-git/release/.git/refs/heads/master?

16be3e90545972dec16014253a843229d5bdf388

 > What happens if you try the attached patch instead? On my system, it outputs this:
 >
 > ...
 > Loading leim/leim-list.el (source)...
 > emacs-repository--version-git-1 /home/eggert/src/gnu/emacs/static-checking/.git/HEAD /home/eggert/src/gnu/emacs/static-checking/.git
 > emacs-repository--version-git-1 /home/eggert/src/gnu/emacs/static-checking/.git/refs/heads/master /home/eggert/src/gnu/emacs/static-checking/.git
 > emacs-repository--version-git-1 returns 062757dca01cff127583438ae16237af55b6253c
 > emacs-repository--version-git-1 returns 062757dca01cff127583438ae16237af55b6253c
 > Finding pointers to doc strings...
 > Finding pointers to doc strings...done
 > ...
 >

Here it outputs

...
Loading leim/leim-list.el (source)...
emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/HEAD /home/martin/emacs-git/release/.git
emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/refs/heads/master /home/martin/emacs-git/release/.git
emacs-repository--version-git-1 returns 16be3e90545972dec16014253a843229d5bdf388
emacs-repository--version-git-1 returns 16be3e90545972dec16014253a843229d5bdf388
Finding pointers to doc strings...
Finding pointers to doc strings...done
...

martin

Re: Spurious "file errors" when making master

[Paul Eggert]

martin rudalics wrote:
> Loading leim/leim-list.el (source)...
> emacs-repository--version-git-1 /home/martin/emacs-git/release/.git/HEAD
> /home/martin/emacs-git/release/.git
> emacs-repository--version-git-1
> /home/martin/emacs-git/release/.git/refs/heads/master
> /home/martin/emacs-git/release/.git
> emacs-repository--version-git-1 returns 16be3e90545972dec16014253a843229d5bdf388
> emacs-repository--version-git-1 returns 16be3e90545972dec16014253a843229d5bdf388
> Finding pointers to doc strings...
> Finding pointers to doc strings...do


So -- adding calls to 'message' fixes the bug?!?

I'm afraid you will have to debug this more directly; I don't see how to do more 
from a distance than I already have.

Re: Spurious "file errors" when making master

[martin rudalics]

 > So -- adding calls to 'message' fixes the bug?!?

Silly me.  Obviously not!!

 > I'm afraid you will have to debug this more directly; I don't see how to do more from a distance than I already have.

I was confused and applied your patch and tested it on the emacs 25
branch where the problem does _not_ happen.  Please disregard all my
prior messages where the word "release" appears in a path or file name.
Sorry.

On master I'm getting with your latest version.patch:

...
Loading leim/leim-list.el (source)...
emacs-repository--version-git-1 /home/martin/emacs-git/quick/.git/HEAD /home/martin/emacs-git/quick/.git
emacs-repository--version-git-1 /home/martin/emacs-git/quick/.git/refs/heads/master /home/martin/emacs-git/quick/.git
emacs-repository--version-git-1 returns nil
emacs-repository--version-git-1 returns nil
emacs-repository--version-git-1 /home/martin/emacs-git/quick/.git/refs/heads/master /home/martin/emacs-git/quick/.git
emacs-repository--version-git-1 returns nil
Waiting for git...
Error running git rev-parse: (file-error "Searching for program" "No such file or directory" "git")
Finding pointers to doc strings...
Finding pointers to doc strings...done
...

Also

/home/martin/emacs-git/quick/.git/refs/heads

is empty and

/home/martin/emacs-git/quick/.git/HEAD

contains the single line

ref: refs/heads/master

martin

Re: Spurious "file errors" when making master

[Paul Eggert]

On 05/30/2016 11:52 PM, martin rudalics wrote:
> /home/martin/emacs-git/quick/.git/refs/heads
>
> is empty

Aha! That's it.

I guess you've garbage-collected your repository, either by hand ('git 
gc') or automatically, and that there is a file .git/packed-refs that 
contains a line like this:

8e22067d59f59ee3a39f5d4f95c27e8ea3c935d6 refs/heads/master

If so, someone (perhaps you?) needs to modify the 
emacs-repository-get-version function in lisp/version.el to handle this 
situation.

Re: Spurious "file errors" when making master

[martin rudalics]

 >> /home/martin/emacs-git/quick/.git/refs/heads
 >>
 >> is empty
 >
 > Aha! That's it.
 >
 > I guess you've garbage-collected your repository, either by hand ('git gc') or automatically,

Not by hand but during a pull git told me something about doing some
collection or packing.

 > and that there is a file .git/packed-refs that contains a line like this:
 >
 > 8e22067d59f59ee3a39f5d4f95c27e8ea3c935d6 refs/heads/master

There is such a file and the number shown on that line is that returned
by ‘emacs-repository-get-version’.

 > If so, someone (perhaps you?) needs to modify the emacs-repository-get-version function in lisp/version.el to handle this situation.

On Windows I have done manual packing frequently and the ../heads
directory is not empty.  And on GNU/Linux if I manually call
‘emacs-repository-get-version’ it does return the number of packed-refs.
Queer.

martin

Re: Spurious "file errors" when making master

[Paul Eggert]

On 05/31/2016 10:26 AM, martin rudalics wrote:
> There is such a file and the number shown on that line is that returned
> by ‘emacs-repository-get-version’. 

OK, thanks, I installed something into master which should work for your 
setup.

This area is a house of cards if you ask me (Emacs should not be fooling 
around with Git internals!) but I didn't attack the larger problem.

Re: Spurious "file errors" when making master

[martin rudalics]

 > OK, thanks, I installed something into master which should work for your setup.

Thanks.

 > This area is a house of cards if you ask me (Emacs should not be
 > fooling around with Git internals!) but I didn't attack the larger
 > problem.

IIUC you nevertheless did that in the meantime.  But it seems that I
can't test any fix because git now seems to have populated
/home/martin/emacs-git/quick/.git/refs/heads again.  Maybe that
directory is empty only following an (automatic?) GC until the next
pull.  I will try to watch the behavior of git in this regard in the
future.

Thanks again for your efforts, martin

Re: Spurious "file errors" when making master

[Eli Zaretskii]

> Date: Tue, 31 May 2016 19:26:57 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org
> 
> On Windows I have done manual packing frequently and the ../heads
> directory is not empty.

You cannot have automatic GC enabled on Windows, IME, since then a
pull will typically fail due to file sharing violations.

> And on GNU/Linux if I manually call ‘emacs-repository-get-version’
> it does return the number of packed-refs.

Because when you call it manually, exec-path is already set, so the
Git invocation works.