How to see the raw line endings in a file?

How to see the raw line endings in a file?

[Steinar Bang]

Is there a way in emacs where I can be sure that I see that the files
have line endings of a particular form? 

Ie. DOS style cr lf or unix-ish style with just lf?

I'm normally looking for emacs telling me with a "DOS" in the status
line, that a file actually has cr lf line endings.

But now I have some files with no "DOS" in the status line where weird
stuff is happening:
 1. Everytime I do something with the maven-release-plugin on this
    project, line endings are messed up (most of the lines, but not all,
    get cr lf)
     https://github.com/steinarb/authservice
 2. When I have googled I haven't found any relevant matches.  The only
    matches have been when trying to run releases on Windows, where one
    of the suggestions have been "do the release in a GNU/linux VM"
 3. However, I do the releases on debian GNU/linux

My conclusion from that is that there is something weird with the line
endings on my project's pom.xml files.  However, if that is the case,
then emacs is hiding the issues from me.

So I'm looking for a way to track down any line ending issues in a
file.

Any ideas?

Thanks!


- Steinar

Re: How to see the raw line endings in a file?

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Sun, 31 Mar 2019 18:47:25 +0200
> 
> Is there a way in emacs where I can be sure that I see that the files
> have line endings of a particular form? 
> 
> Ie. DOS style cr lf or unix-ish style with just lf?
> 
> I'm normally looking for emacs telling me with a "DOS" in the status
> line, that a file actually has cr lf line endings.
> 
> But now I have some files with no "DOS" in the status line where weird
> stuff is happening:
>  1. Everytime I do something with the maven-release-plugin on this
>     project, line endings are messed up (most of the lines, but not all,
>     get cr lf)
>      https://github.com/steinarb/authservice
>  2. When I have googled I haven't found any relevant matches.  The only
>     matches have been when trying to run releases on Windows, where one
>     of the suggestions have been "do the release in a GNU/linux VM"
>  3. However, I do the releases on debian GNU/linux
> 
> My conclusion from that is that there is something weird with the line
> endings on my project's pom.xml files.  However, if that is the case,
> then emacs is hiding the issues from me.

Emacs never hides CRLF EOLs unless it says "DOS" in the mode line.  If
your file has mixed EOL format, then either you will see "DOS" in the
mode line or some of the lines will have ^M characters at their end.

If you see neither of these, then it isn't Emacs that converts some
lines to DOS CRLF format, it's some other program.  And you didn't
describe in enough detail what you do to try guessing more.

Re: How to see the raw line endings in a file?

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Emacs never hides CRLF EOLs unless it says "DOS" in the mode line.  If
> your file has mixed EOL format, then either you will see "DOS" in the
> mode line or some of the lines will have ^M characters at their end.

This is what I have been trusting to be happening for many years.
 1. I see no "DOS" in the mode line before running the commands
 2. I see a mix of lines with and without ^M after running the commands

> If you see neither of these, then it isn't Emacs that converts some
> lines to DOS CRLF format, it's some other program.  And you didn't
> describe in enough detail what you do to try guessing more.

I think there must be an issue in the maven-release-plugin, even though
google haven't dug up anyone else with the same problem, and even though
the maven-release-plugin hasn't had a release since 2015.

So I've opened up a bug on the maven-release-plugin:
 https://issues.apache.org/jira/browse/MRELEASE-1021

Re: How to see the raw line endings in a file?

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Sun, 31 Mar 2019 19:33:53 +0200
> 
> > Emacs never hides CRLF EOLs unless it says "DOS" in the mode line.  If
> > your file has mixed EOL format, then either you will see "DOS" in the
> > mode line or some of the lines will have ^M characters at their end.
> 
> This is what I have been trusting to be happening for many years.
>  1. I see no "DOS" in the mode line before running the commands
>  2. I see a mix of lines with and without ^M after running the commands

Then why are you saying that Emacs hides CR characters from you?  It
does show them as those ^Ms.

In general, when a  file has mixed EOLs,
what will show in the mode line and what in the buffer is determined
by sheer luck, because Emacs examines only a small portion of the file
to determine the EOL format, and depending on where you have CRLF and
where just LF you can end up with either "DOS" or ^M characters.

Re: How to see the raw line endings in a file?

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Then why are you saying that Emacs hides CR characters from you?  It
> does show them as those ^Ms.

I'm not saying that emacs hides CR characters from me.  I'm asking the
question: is it a possibility that emacs hides characters from me?

(And I see from your reply that the answer to that question is "no")

The reason I'm asking this question is that I am encountering a problem
noone else seems to be encountering, and I was wondering if I could
trust what emacs is telling me (ie. that the files originally were lf
eol only).

(It could be that my google searches are insufficient to find others
with the same problem, but by my experience is that weird problems that
noone else has, is usually caused by a silly problem with my own code or
setup)

> In general, when a file has mixed EOLs, what will show in the mode
> line and what in the buffer is determined by sheer luck, because Emacs
> examines only a small portion of the file to determine the EOL format,
> and depending on where you have CRLF and where just LF you can end up
> with either "DOS" or ^M characters.

Hm... so, if the is not detected as "DOS" and I mainly look at the start
of the file, then it looks like a unix file, but could have an CR
further down...?

Ok, thanks for the info.  I will investigate further and report back.

Re: How to see the raw line endings in a file?

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:
>>>>> Eli Zaretskii <eliz@gnu.org>:

>> In general, when a file has mixed EOLs, what will show in the mode
>> line and what in the buffer is determined by sheer luck, because
>> Emacs examines only a small portion of the file to determine the EOL
>> format, and depending on where you have CRLF and where just LF you
>> can end up with either "DOS" or ^M characters.

> Hm... so, if the is not detected as "DOS" and I mainly look at the
> start of the file, then it looks like a unix file, but could have an
> CR further down...?

> Ok, thanks for the info.  I will investigate further and report back.

Hm... looks like existing cr's are not the cause, I grepped the source
and this is all that was found:
 sb@lorenzo:~/workspaces/ws03/authservice$ grep -r $'\r' * | grep -v node_modules | grep -v target
 Binary file authservice.web.security/src/main/resources/web/favicon.ico matches
 Binary file authservice.web.users.frontend/src/main/frontend/public/favicon.ico matches
 sb@lorenzo:~/workspaces/ws03/authservice$

(the node_modules and target directories I'm excluding, contains
downloaded files and/or files created by the maven build process)

Thanks for your help!

Re: How to see the raw line endings in a file?

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Sun, 31 Mar 2019 20:57:55 +0200
> 
> The reason I'm asking this question is that I am encountering a problem
> noone else seems to be encountering, and I was wondering if I could
> trust what emacs is telling me

Yes, you can.

> > In general, when a file has mixed EOLs, what will show in the mode
> > line and what in the buffer is determined by sheer luck, because Emacs
> > examines only a small portion of the file to determine the EOL format,
> > and depending on where you have CRLF and where just LF you can end up
> > with either "DOS" or ^M characters.
> 
> Hm... so, if the is not detected as "DOS" and I mainly look at the start
> of the file, then it looks like a unix file, but could have an CR
> further down...?

Yes, it could.

RE: How to see the raw line endings in a file?

[Drew Adams]

> Is there a way in emacs where I can be sure that I see that the files
> have line endings of a particular form?

Eli has answered your question, I think.

Additionally, you can use command `hc-highlight-chars'
from library `highlight-chars.el' to highlight any set
of chars you like, in any face you like.

To highlight control chars newline (`C-j') and carriage
return (`C-m'), you type `C-q C-j C-q C-m'.

With a prefix arg the command unhighlights.


https://www.emacswiki.org/emacs/download/highlight-chars.el

https://www.emacswiki.org/emacs/ShowWhiteSpace#HighlightChars

Re: How to see the raw line endings in a file?

[Stefan Monnier]

>> Emacs never hides CRLF EOLs unless it says "DOS" in the mode line.  If
>> your file has mixed EOL format, then either you will see "DOS" in the
>> mode line or some of the lines will have ^M characters at their end.
>
> This is what I have been trusting to be happening for many years.
>  1. I see no "DOS" in the mode line before running the commands
>  2. I see a mix of lines with and without ^M after running the commands

So your input file is using Unix line-endings and apparently the command
you run copies some of the input with those (LF) line-endings to the
output and also outputs some of its own text with its own (CRLF)
line-endings line-endings, hence a mix of the two in the output.

You can probably circumvent the problem either by tweaking the command
so that it uses LF line-endings for its own text, or changing the input
so that it uses CRLF line-endings.


        Stefan

Re: How to see the raw line endings in a file?

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Sun, 31 Mar 2019 21:09:52 +0200
> 
> Hm... looks like existing cr's are not the cause, I grepped the source
> and this is all that was found:
>  sb@lorenzo:~/workspaces/ws03/authservice$ grep -r $'\r' * | grep -v node_modules | grep -v target
>  Binary file authservice.web.security/src/main/resources/web/favicon.ico matches
>  Binary file authservice.web.users.frontend/src/main/frontend/public/favicon.ico matches
>  sb@lorenzo:~/workspaces/ws03/authservice$

"Binary file" usually means there are null bytes in the file.  When
Emacs sees null bytes, it by default inhibits decoding, including
detection of EOL format, so you see a Unix file with ^M characters, if
there are any.