Magit reports binary files that don't exist

Magit reports binary files that don't exist

[Thorsten Jolitz]

Hi List, 

using magit, I see this quite often now

,----
| [<<header>>]
| 
|    Do something
|     
| 1 file changed, 0 insertions(+), 0 deletions(-)
|  src/foo.el | Bin 59711 -> 58056 bytes
| 
| 	Modified   src/foo.el
`----

although

 - there are no binary versions of foo.el (i.e. no foo.elc nowhere)

 - I certainly did insertions and deletions before the commit

Am I the only one seeing this?

PS

#+BEGIN_SRC emacs-lisp
(emacs-version)
#+END_SRC

#+results:
: GNU Emacs 24.3.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.12.2)
:  of 2014-06-11 on var-lib-archbuild-staging-x86_64-jgc

#+BEGIN_SRC emacs-lisp
(magit-version)
#+END_SRC

#+results:
: 20140118.1540

,----
| git version 2.1.0
`----


-- 
cheers,
Thorsten

Re: Magit reports binary files that don't exist

[Óscar Fuentes]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> using magit, I see this quite often now
>
> ,----
> | [<<header>>]
> | 
> |    Do something
> |     
> | 1 file changed, 0 insertions(+), 0 deletions(-)
> |  src/foo.el | Bin 59711 -> 58056 bytes
> | 
> | 	Modified   src/foo.el
> `----
>
> although
>
>  - there are no binary versions of foo.el (i.e. no foo.elc nowhere)
>
>  - I certainly did insertions and deletions before the commit
>
> Am I the only one seeing this?

Let me try: src/foo.el has 1655 lines.

Re: Magit reports binary files that don't exist

[Yuri Khan]

On Fri, Aug 22, 2014 at 8:43 PM, Óscar Fuentes <ofv@wanadoo.es> wrote:
> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> |  src/foo.el | Bin 59711 -> 58056 bytes
>
> Let me try: src/foo.el has 1655 lines.

1656, if you also count the last one with its line ending missing.

Re: Magit reports binary files that don't exist

[Thorsten Jolitz]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> using magit, I see this quite often now

I  actually reposted this on the magit mailing list, but any hints are
welcome anyway. 

>> ,----
>> | [<<header>>]
>> | 
>> |    Do something
>> |     
>> | 1 file changed, 0 insertions(+), 0 deletions(-)
>> |  src/foo.el | Bin 59711 -> 58056 bytes
>> | 
>> | 	Modified   src/foo.el
>> `----
>>
>> although
>>
>>  - there are no binary versions of foo.el (i.e. no foo.elc nowhere)
>>
>>  - I certainly did insertions and deletions before the commit
>>
>> Am I the only one seeing this?
>
> Let me try: src/foo.el has 1655 lines.

I shouldn't make my examples too abstract, because I forgot what foo.el
was ;)

But none of the two likely files has that number of lines, rather 624 or
1923 lines. Not quite sure how a correct estimate would/should enlighten
me?

-- 
cheers,
Thorsten

Re: Magit reports binary files that don't exist

[Óscar Fuentes]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> But none of the two likely files has that number of lines, rather 624 or
> 1923 lines. Not quite sure how a correct estimate would/should enlighten
> me?

Possible line ending conversion due to configuration differences on
checkin/checkout machines.

I've just read your message on the magit ml. I manage thousands of .el
files with magit/git (the emacs repo, for instance) and never observed
that message.

Re: Magit reports binary files that don't exist

[Thorsten Jolitz]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> But none of the two likely files has that number of lines, rather 624 or
>> 1923 lines. Not quite sure how a correct estimate would/should enlighten
>> me?
>
> Possible line ending conversion due to configuration differences on
> checkin/checkout machines.

I used only one (Archlinux) machine for these repos, and I'm the only
committer. 

Could emacs-server be responsable for several problems I'm
seeing right now (access to my gmail-accounts denied for gnus, magit
treats .el files as binary)? Because a few days ago I (accidentally)
started an emacs-server with the same init-file of an already running
emacs instance, and that was more or less the time when the problems
began. 

But its hard to see how this could be related, its probably pure
coincidence. But maybe not, because I remember that after starting the
server every file I opened in dired was "changed" and had to be saved,
although I did not modify most of them. 

> I've just read your message on the magit ml. I manage thousands of .el
> files with magit/git (the emacs repo, for instance) and never observed
> that message.

Its relatively new for me too, and I'm managing hundreds of .el files too
for quite some time now. 

-- 
cheers,
Thorsten

Re: Magit reports binary files that don't exist

[Thorsten Jolitz]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:

>> I've just read your message on the magit ml. I manage thousands of .el
>> files with magit/git (the emacs repo, for instance) and never observed
>> that message.
>
> Its relatively new for me too, and I'm managing hundreds of .el files too
> for quite some time now. 

Just for the record, on the magit mailing list I was pointed to NULL
chars as possible culprits for this behaviour, and in fact the affected
.el files contain regexps with this construct borrowed from Org-mode
sources:

,----
| "[^^@]+" 
`----

where ^@ is actually the NULL char entered with C-q. 

-- 
cheers,
Thorsten

Re: Magit reports binary files that don't exist

[Yuri Khan]

On Fri, Aug 22, 2014 at 11:06 PM, Thorsten Jolitz <tjolitz@gmail.com> wrote:

> Its relatively new for me too, and I'm managing hundreds of .el files too
> for quite some time now.

Open up the log in gitk. Find a file that is detected as binary.
Right-click it in the bottom right frame, select “Highlight this
only”. This will highlight all commits where this file is modified
with bold in the top frame. (Why gitk and not magit? Because gitk
shows you a list of commits and the diff in one window, and you can
just press the down arrow until you hit the commit you’re looking for.
Magit only shows you the list of commits and you have to press Enter
or Space on each in order to see diffs.)

Now follow the trail. Several topmost commits will report that the
file is binary and that it changed. Eventually, though, you will find
the first commit that introduced binariness to this file.

Check out that commit. Look at the file. Maybe do a ‘diff -au’ against
its previous version (when it was still text). Look what changed.
Think which specific change could make a text file binary.

Re: Magit reports binary files that don't exist

[Nicolas Richard]

Thorsten Jolitz <tjolitz@gmail.com> writes:
> ,----
> | "[^^@]+" 
> `----
>
> where ^@ is actually the NULL char entered with C-q. 

Using "\000" in lisp strings instead of an explicit NUL might help, then.

-- 
Nico.

Re: Magit reports binary files that don't exist

[Thorsten Jolitz]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Fri, Aug 22, 2014 at 11:06 PM, Thorsten Jolitz <tjolitz@gmail.com>
> wrote:
>
>> Its relatively new for me too, and I'm managing hundreds of .el files too
>> for quite some time now.
>
> Open up the log in gitk. Find a file that is detected as binary.
> Right-click it in the bottom right frame, select “Highlight this
> only”. This will highlight all commits where this file is modified
> with bold in the top frame. (Why gitk and not magit? Because gitk
> shows you a list of commits and the diff in one window, and you can
> just press the down arrow until you hit the commit you’re looking for.
> Magit only shows you the list of commits and you have to press Enter
> or Space on each in order to see diffs.)
>
> Now follow the trail. Several topmost commits will report that the
> file is binary and that it changed. Eventually, though, you will find
> the first commit that introduced binariness to this file.
>
> Check out that commit. Look at the file. Maybe do a ‘diff -au’ against
> its previous version (when it was still text). Look what changed.
> Think which specific change could make a text file binary.

I'm only working on the console, so I cannot use the mouse. But I will
follow your recipe anyway, although it seems likely now that my use of
literal NUL chars in regexps convinced git that my .el files are binary
files. 

Thanks for the tip. 

-- 
cheers,
Thorsten

Re: Magit reports binary files that don't exist

[Thorsten Jolitz]

Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>> ,----
>> | "[^^@]+" 
>> `----
>>
>> where ^@ is actually the NULL char entered with C-q. 
>
> Using "\000" in lisp strings instead of an explicit NUL might help, then.

Yes, in org-mode they write this as 

,----
| "[^\000]+" 
`----

however I (and even somebody else at that time) tried it on a certain
problem and it was not reliable, only the literal NUL worked
always. There must be a thread on the Org mailing-list about this. But
maybe we did something wrong, if it works in Org, it should work
everywhere ...

-- 
cheers,
Thorsten

Re: Magit reports binary files that don't exist

[Yuri Khan]

On Sat, Aug 23, 2014 at 5:05 AM, Thorsten Jolitz <tjolitz@gmail.com> wrote:

> I'm only working on the console, so I cannot use the mouse. But I will
> follow your recipe anyway, although it seems likely now that my use of
> literal NUL chars in regexps convinced git that my .el files are binary
> files.

Yes, NULs are certainly a reason to consider a file binary.

In console, the recipe is "git log --patch <filename>". This gives you
all commits that modified <filename>, and how they modified it.

Re: Magit reports binary files that don't exist

[Nicolas Richard]

Thorsten Jolitz <tjolitz@gmail.com> writes:
> however I (and even somebody else at that time) tried it on a certain
> problem and it was not reliable, only the literal NUL worked
> always. 
> There must be a thread on the Org mailing-list about this. 

I did not manage to find it (searched for the term "NUL" and your email
address on the gmane interface).

Do you have a reproducible recipe ?

-- 
Nico.

Re: Magit reports binary files that don't exist

[Thorsten Jolitz]

Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>> however I (and even somebody else at that time) tried it on a certain
>> problem and it was not reliable, only the literal NUL worked
>> always. 
>> There must be a thread on the Org mailing-list about this. 
>
> I did not manage to find it (searched for the term "NUL" and your email
> address on the gmane interface).

Its this thread (you were even involved ;):

,----
| http://lists.gnu.org/archive/html/help-gnu-emacs/2014-04/msg00042.html
`----

> Do you have a reproducible recipe ?

No, and never had, just a few unexplicable cases were the regexp did not
match. 

-- 
cheers,
Thorsten

Re: Magit reports binary files that don't exist

[Nicolas Richard]

Thorsten Jolitz <tjolitz@gmail.com> writes:
> Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:
>
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>> however I (and even somebody else at that time) tried it on a certain
>>> problem and it was not reliable, only the literal NUL worked
>>> always. 
>>> There must be a thread on the Org mailing-list about this. 
>>
>> I did not manage to find it (searched for the term "NUL" and your email
>> address on the gmane interface).
>
> Its this thread (you were even involved ;):

Err, that's embarassing. Thanks for finding it.

Re-reading the thread, I don't see where a difference between "\000" and
"<actual NUL>" is mentionned.

I do, however, see some confusion between "\000" (a string of one NUL
char) and "\\000" (a string of 4 chars : backslash, followed by three
0's), e.g. the regexp "^aPDU[^\\000]+[}]+?$" is mentionned (at
http://lists.gnu.org/archive/html/help-gnu-emacs/2014-04/msg00077.html)

Perhaps that would explain the oddities ?

-- 
Nico.

Re: Magit reports binary files that don't exist

[Thorsten Jolitz]

Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>> Nicolas Richard <theonewiththeevillook@yahoo.fr> writes:
>>
>>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>>> however I (and even somebody else at that time) tried it on a certain
>>>> problem and it was not reliable, only the literal NUL worked
>>>> always. 
>>>> There must be a thread on the Org mailing-list about this. 
>>>
>>> I did not manage to find it (searched for the term "NUL" and your email
>>> address on the gmane interface).
>>
>> Its this thread (you were even involved ;):
>
> Err, that's embarassing. Thanks for finding it.

I don't think so, forgot about it too ...

> Re-reading the thread, I don't see where a difference between "\000" and
> "<actual NUL>" is mentionned.
>
> I do, however, see some confusion between "\000" (a string of one NUL
> char) and "\\000" (a string of 4 chars : backslash, followed by three
> 0's), e.g. the regexp "^aPDU[^\\000]+[}]+?$" is mentionned (at
> http://lists.gnu.org/archive/html/help-gnu-emacs/2014-04/msg00077.html)
>
> Perhaps that would explain the oddities ?

Yes, thats quite likely, because at that time I just discovered this
(most useful) 'regexp trick', and maybe I automatically escaped the
backslash  at some point without noticing. 

I switched to the 'Org version' again now (from literal NUL). Its not
really such a good idea to have literal NULs in source files ...

-- 
cheers,
Thorsten