Checking in files with trailing white space

Checking in files with trailing white space

[Lars Magne Ingebrigtsen]

I'm adding test cases for the shr HTML rendering, and in some of the
tests there should really be trailing white space in the result files.
Is there a way to say that the test/data/shr directory should be allowed
to have trailing spaces in our git/checkin thing?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Checking in files with trailing white space

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm adding test cases for the shr HTML rendering, and in some of the
> tests there should really be trailing white space in the result files.
> Is there a way to say that the test/data/shr directory should be allowed
> to have trailing spaces in our git/checkin thing?

Never mind; found it...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Checking in files with trailing white space

[Stefan Monnier]

> I'm adding test cases for the shr HTML rendering, and in some of the
> tests there should really be trailing white space in the result files.
> Is there a way to say that the test/data/shr directory should be allowed
> to have trailing spaces in our git/checkin thing?

How 'bout you don't store files with trailing whitespace, but instead
store files which contain code which creates those trailing whitespaces.
E.g.

  (ert-deftest ...
    (with-temp-buffer
      (insert "foo   ")
      <dosomething>))


-- Stefan

Re: Checking in files with trailing white space

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> How 'bout you don't store files with trailing whitespace, but instead
> store files which contain code which creates those trailing whitespaces.
> E.g.
>
>   (ert-deftest ...
>     (with-temp-buffer
>       (insert "foo   ")
>       <dosomething>))

Having the test and result files be HTML and text files is much easier
to work with.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Checking in files with trailing white space

[Paul Eggert]

Lars Magne Ingebrigtsen wrote:
> Having the test and result files be HTML and text files is much easier
> to work with.

Another possibility is to design your own escape convention for trailing white 
space and remove it just before the test.  You could strip backslash just before 
newline, say.  This will be more robust in the presence of tools that routinely 
strip trailing white space.

If you do keep trailing whitespace, please check that it survives editing the 
file by an Emacs configured something like this:

(setq whitespace-action '(auto-cleanup))
(setq whitespace-style '(trailing empty))
(global-whitespace-mode t)

as this is how I edit Emacs source code. I assume you can do this by setting 
local variables in the affected files.

Re: Checking in files with trailing white space

[Lars Magne Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Another possibility is to design your own escape convention for
> trailing white space and remove it just before the test.  You could
> strip backslash just before newline, say.  This will be more robust in
> the presence of tools that routinely strip trailing white space.

I kinda wanted to keep the HTML files and the results as close to the
input/output from eww as possible.  That is, you can just say `M-x
eww-open-file' on the HTML test files and twiddle the shr code until it
looks right, and then you copy the output to the .txt file.

That's a really convenient way to create the regression test cases.

> If you do keep trailing whitespace, please check that it survives
> editing the file by an Emacs configured something like this:
>
> (setq whitespace-action '(auto-cleanup))
> (setq whitespace-style '(trailing empty))
> (global-whitespace-mode t)
>
> as this is how I edit Emacs source code.

Yeah, the files do not survive editing with those values.

> I assume you can do this by setting local variables in the affected
> files.

I've never played around with local variables in text files.  Can these
local variables be put into a dot file of some kind?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Checking in files with trailing white space

[Paul Eggert]

On 04/04/2016 01:05 PM, Lars Magne Ingebrigtsen wrote:
> I've never played around with local variables in text files.  Can these
> local variables be put into a dot file of some kind?

Yes, you can modify .dir-locals.el. But that's intended for generic 
rules, not for specific files that are exceptions. Generally file-local 
variables are a better match for quirks like this. It should be pretty 
easy; e.g., see the file 'test/manual/etags/html-src/software.html'.

Re: Checking in files with trailing white space

[Lars Magne Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Yes, you can modify .dir-locals.el. But that's intended for generic
> rules, not for specific files that are exceptions. Generally
> file-local variables are a better match for quirks like this. It
> should be pretty easy; e.g., see the file
> 'test/manual/etags/html-src/software.html'.

Hm...  since this only affects few of the rendered HTML files, I think
that perhaps your original suggestion of using some kind of encoding
scheme is easier.  I'll do that instead.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Checking in files with trailing white space

[Alan Mackenzie]

Hello, Emacs.

On Sun, Mar 20, 2016 at 02:08:51PM +0100, Lars Magne Ingebrigtsen wrote:
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> > I'm adding test cases for the shr HTML rendering, and in some of the
> > tests there should really be trailing white space in the result files.
> > Is there a way to say that the test/data/shr directory should be allowed
> > to have trailing spaces in our git/checkin thing?

> Never mind; found it...

Would some kind soul please tell me the recipe.  I've just merged master
into a nine-month old branch, and I get lots of these whitespace errors
when I try to commit the merge.

Thanks!

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Checking in files with trailing white space

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

>> > I'm adding test cases for the shr HTML rendering, and in some of the
>> > tests there should really be trailing white space in the result files.
>> > Is there a way to say that the test/data/shr directory should be allowed
>> > to have trailing spaces in our git/checkin thing?
>
>> Never mind; found it...
>
> Would some kind soul please tell me the recipe.  I've just merged master
> into a nine-month old branch, and I get lots of these whitespace errors
> when I try to commit the merge.

Hm...  sorry, I've forgotten again.  But I don't think what you're
seeing is related to what I was talking about.  The git/checkin script
refuses to check in files with whitespace at the end of lines, while it
sounds like you have something different going on...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Checking in files with trailing white space

[Alan Mackenzie]

Hello, Lars.

On Sun, Dec 11, 2016 at 07:50:13PM +0100, Lars Ingebrigtsen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> >> > I'm adding test cases for the shr HTML rendering, and in some of the
> >> > tests there should really be trailing white space in the result files.
> >> > Is there a way to say that the test/data/shr directory should be allowed
> >> > to have trailing spaces in our git/checkin thing?

> >> Never mind; found it...

> > Would some kind soul please tell me the recipe.  I've just merged master
> > into a nine-month old branch, and I get lots of these whitespace errors
> > when I try to commit the merge.

> Hm...  sorry, I've forgotten again.  But I don't think what you're
> seeing is related to what I was talking about.  The git/checkin script
> refuses to check in files with whitespace at the end of lines, while it
> sounds like you have something different going on...

No, that was precisely my problem.  The files with the trailing spaces
(there must have been ~30 - 100 of them) might well have been the test
files you created, as reported in the paragraph above.

However, this absolute prohibition on trailing spaces is fascistic.
Some files (for example, test files) have essential trailing spaces in
them, and for git absolutely to refuse to commit them I think is
unacceptable.

I eventually found what was causing the problem, although it cost me a
precious 2 - 3 hours out of my Emacs hacking time.  :-(  This problem
was the last line in ..../.git/hooks/pre-commit, which looked like this:

    exec git diff-index --check --cached HEAD --

.  Do you see "trailing spaces" in there anywhere?  No, neither did I,
and the fact that there's no comment to say so got me somewhat angry
after I'd found out.  Negligent obfuscation, particularly given the dire
state of git's documentation.

Anyhow, I've commented out that line, and all the files with trailing
spaces got committed OK.  It would be nice to get a warning along the
lines of "The following files contain trailing whitespace, do you wish
to proceed with the commit?  [Y/n]: ....".

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Checking in files with trailing white space

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

On 12/12/2016 02:16 PM, Alan Mackenzie wrote:
> However, this absolute prohibition on trailing spaces is fascistic.
> Some files (for example, test files) have essential trailing spaces in
> them

I added a comment to the pre-commit file to help out others in your 
situation; see the attached patch.

As it is all too easy to overlook trailing white space when reading 
source code, I'd rather avoid having significant whitespace in committed 
files. If I were writing those test cases, I'd commit files that didn't 
have trailing whitespace, and convert them to 
files-with-trailing-whitespace while running the test.

Similarly, I'd rather avoid committing binary data. It's easier to read 
and review text files, given the tools we have. Any tests that require 
binary data can generate the data when the test is run.


[-- Attachment #2: 0001-build-aux-git-hooks-pre-commit-Add-whitespace-commen.patch --]
[-- Type: application/x-patch, Size: 1040 bytes --]

Re: Checking in files with trailing white space

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 12 Dec 2016 14:57:42 -0800
> Cc: emacs-devel@gnu.org
> 
> Similarly, I'd rather avoid committing binary data. It's easier to read 
> and review text files, given the tools we have. Any tests that require 
> binary data can generate the data when the test is run.

How would you generate the likes of test/manual/etags/ETAGS.good_1?

Re: Checking in files with trailing white space

[Paul Eggert]

Eli Zaretskii wrote:
>> I'd rather avoid committing binary data. It's easier to read
>> and review text files, given the tools we have. Any tests that require
>> binary data can generate the data when the test is run.

> How would you generate the likes of test/manual/etags/ETAGS.good_1?

That one is not a problem. By "binary data" I meant files that are not valid 
UTF-8 text, and that file consists entirely of UTF-8 text.

An example of trouble is test/manual/etags/f-src/entry.strange.gz. Instead of 
committing the gzipped file, I'd commit the original text file 'entry.strange', 
and compress a copy as part of the test.

To see an example problem when binary data is committed, visit this URL:

http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=6db72771cc08e719a08f56db8abd7cefef9c8c99

Because that commit installed some Latin-1 text, the body of that web page 
contains a mixture of UTF-8 and Latin-1. The web page is labeled as UTF-8, so 
the result works poorly with browsers (Firefox displays mojibake lines like 
"+2�4�bb*�abbbc�"). I fixed this particular problem just now in master by 
changing that file to use UTF-8, which works just as well for the Emacs test in 
question.

Re: Checking in files with trailing white space

[Eli Zaretskii]

> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 12 Dec 2016 23:31:42 -0800
> 
> An example of trouble is test/manual/etags/f-src/entry.strange.gz. Instead of 
> committing the gzipped file, I'd commit the original text file 'entry.strange', 
> and compress a copy as part of the test.

But what about data files that cannot be easily produced from text on
any garden-variety system?

> To see an example problem when binary data is committed, visit this URL:
> 
> http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=6db72771cc08e719a08f56db8abd7cefef9c8c99
> 
> Because that commit installed some Latin-1 text, the body of that web page 
> contains a mixture of UTF-8 and Latin-1.

Sometimes this is necessary.  We could, of course, recode the file
using Emacs, but only in most cases, not all of them.  E.g., producing
a file with invalid bytes might not be easy at all.

And for other use cases, doing what you want might be next to
impossible, except if we write a C or Lisp program to generate the
data, something that might make the job significantly harder than just
grabbing a file.

So I think it would be good to avoid binary data when feasible, but I
won't make that a hard requirement.  We don't want to make the job of
adding tests any harder than it already is.

Re: Checking in files with trailing white space

[Alan Mackenzie]

Hello, Paul.

On Mon, Dec 12, 2016 at 02:57:42PM -0800, Paul Eggert wrote:
> On 12/12/2016 02:16 PM, Alan Mackenzie wrote:
> > However, this absolute prohibition on trailing spaces is fascistic.
> > Some files (for example, test files) have essential trailing spaces in
> > them

> I added a comment to the pre-commit file to help out others in your 
> situation; see the attached patch.

Thanks.  That looks just the job!

> As it is all too easy to overlook trailing white space when reading 
> source code, I'd rather avoid having significant whitespace in committed 
> files. If I were writing those test cases, I'd commit files that didn't 
> have trailing whitespace, and convert them to 
> files-with-trailing-whitespace while running the test.

Alternatively, could the pre-commit git hook be adapted to warn the
committer about trailing space and ask him for confirmation?  I couldn't
see an easy way over the weekend, but it might be possible, somehow.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Checking in files with trailing white space

[Paul Eggert]

On 12/13/2016 08:01 AM, Eli Zaretskii wrote:
> what about data files that cannot be easily produced from text on
> any garden-variety system?

An Elisp program can easily produce data files in any format, so this 
should not be a problem in practice.

> We don't want to make the job of adding tests any harder than it 
> already is. 

We can tolerate low-quality tests, I suppose. But it's better if tests 
are readable and work well with standard tools like "git diff". So we 
should encourage replacing binary data with text in the repository, as 
this will make the tests more maintainable.

This is not merely a practical point; it's also a philosophical one. The 
GPL distinguishes source code from object code. Typically, binary data 
files are "object code" in the GPL sense, in that they are not the 
preferred form of a test case for software-development tools like Git. 
Whenever we distribute object code, the GPL implies that we should also 
distribute the corresponding source code that is easy for Git etc. to 
work with. This source code belongs in the Git repository, and the 
corresponding object code need not and should not be in the Git repository.

Re: Checking in files with trailing white space

[Paul Eggert]

On 12/13/2016 10:28 AM, Alan Mackenzie wrote:
> Alternatively, could the pre-commit git hook be adapted to warn the
> committer about trailing space and ask him for confirmation?  I couldn't
> see an easy way over the weekend, but it might be possible, somehow.

I don't see an easy way either. Part of the problem is that some 
developers commit with vc and some do not. If you use vc, perhaps you 
could cook up something that works with vc; as I don't use vc to commit, 
I'm not the best person to give advice in that area. Plus, I'd rather 
not have to deal with source files containing trailing white space (it 
is *such* a pain in so many ways), so my motivation to help in this area 
is somewhat limited....

Re: Checking in files with trailing white space

[Eli Zaretskii]

> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 13 Dec 2016 10:33:07 -0800
> 
> > what about data files that cannot be easily produced from text on
> > any garden-variety system?
> 
> An Elisp program can easily produce data files in any format, so this 
> should not be a problem in practice.

Producing arbitrary output from a text description might not be easy.
Unless you are willing to accept ASCII-ized binary files in the form
of hex dump or something.  That might make some files harder to read,
not easier, though.  IOW, Git will be happier, but we the humans might
not be.

> > We don't want to make the job of adding tests any harder than it 
> > already is. 
> 
> We can tolerate low-quality tests, I suppose. But it's better if tests 
> are readable and work well with standard tools like "git diff". So we 
> should encourage replacing binary data with text in the repository, as 
> this will make the tests more maintainable.

I'm okay with encouraging that, I just don't think we should require
it.

> This is not merely a practical point; it's also a philosophical one. The 
> GPL distinguishes source code from object code. Typically, binary data 
> files are "object code" in the GPL sense, in that they are not the 
> preferred form of a test case for software-development tools like Git. 
> Whenever we distribute object code, the GPL implies that we should also 
> distribute the corresponding source code that is easy for Git etc. to 
> work with. This source code belongs in the Git repository, and the 
> corresponding object code need not and should not be in the Git repository.

I'm not sure this is relevant here.  E.g., we distribute Emacs sources
as a compressed archive, not as a set of plain-text files.  So stuff
like entry.strange.gz is not an issue in this respect, I think.

Re: Checking in files with trailing white space

[Paul Eggert]

On 12/13/2016 11:00 AM, Eli Zaretskii wrote:
> Producing arbitrary output from a text description might not be easy.

It might not be; but ordinarily it is, for the test data that Emacs needs.

> I'm not sure this is relevant here. E.g., we distribute Emacs sources
> as a compressed archive, not as a set of plain-text files.

That is merely an issue of efficient data transfer. We use a convenient 
format (compressed tarball) that is universally understood to stand for 
a set of source files, and it's OK to do that.

It would become not-OK if we had a complicated or error-prone procedure 
for deriving the source code, or expected people to reverse-engineer 
object code within the tarball. It's common practice for closed-source 
software developers to do that sort of thing. For example, they may 
purposely obscure the source code, and ship object files (in the GPL 
sense) that they call "source files" (because you can feed them to a 
compiler). Although we are obviously not doing that here, we should 
avoid even a whiff of doing so, so as to set a good example.

Re: Checking in files with trailing white space

[Eli Zaretskii]

> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 13 Dec 2016 14:39:23 -0800
> 
> > I'm not sure this is relevant here. E.g., we distribute Emacs sources
> > as a compressed archive, not as a set of plain-text files.
> 
> That is merely an issue of efficient data transfer. We use a convenient 
> format (compressed tarball) that is universally understood to stand for 
> a set of source files, and it's OK to do that.

My point is that providing compressed text files in the distribution
doesn't violate the GPL or its spirit.  Legal obligations are more
important than efficiency considerations, so if compressing sources
would violate the GPL, we would already be violating it.

Re: Checking in files with trailing white space

[Paul Eggert]

Eli Zaretskii wrote:
> providing compressed text files in the distribution
> doesn't violate the GPL or its spirit.

Sure, but we are talking about binary data files that are in the repository, 
which is not the same thing. Most of these files are not in .gz form; they are 
typically object files (in the GPL sense) of some other sort. These are the 
more-serious issue, as far as the GPL goes. Again, I'm saying we're violating 
the GPL -- merely that we are not setting a good example of best practice.

Even if we restrict our attention to the repository's compressed text files, 
there seems little point to compressing a file before putting it into the 
repository. Compression makes it unnecessarily hard to maintain a text file, as 
standard maintenance commands do not work as well with such files. Although I 
agree this practice doesn't violate the GPL, it is unnecessarily awkward and the 
cost of compression typically outweighs the benefits.

Re: Checking in files with trailing white space

[Eli Zaretskii]

> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 13 Dec 2016 23:47:02 -0800
> 
> Eli Zaretskii wrote:
> > providing compressed text files in the distribution
> > doesn't violate the GPL or its spirit.
> 
> Sure, but we are talking about binary data files that are in the repository, 
> which is not the same thing. Most of these files are not in .gz form; they are 
> typically object files (in the GPL sense) of some other sort.

Which files are those?  A gzip-compressed file was the only example
you brought up, but now you are saying it isn't representative, so I'm
unsure which files exactly are the case in point.

> Even if we restrict our attention to the repository's compressed text files, 
> there seems little point to compressing a file before putting it into the 
> repository.

Depends on the compression method, I guess.  For some of them, we
cannot rely on the respective tools being available, so if that
particular kind of compression is needed for some test, having the
source file might not be good enough.  (This is all theoretical at
this point, of course.)

Re: Checking in files with trailing white space

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1938 bytes --]

On 12/14/2016 07:32 AM, Eli Zaretskii wrote:
>> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Tue, 13 Dec 2016 23:47:02 -0800
>>
>> Eli Zaretskii wrote:
>>> providing compressed text files in the distribution
>>> doesn't violate the GPL or its spirit.
>> Sure, but we are talking about binary data files that are in the repository,
>> which is not the same thing. Most of these files are not in .gz form; they are
>> typically object files (in the GPL sense) of some other sort.
> Which files are those?

I've attached a list of repository-controlled files that are not UTF-8 
text files in the POSIX sense. Of the 433 such files, I count 181 Netpbm 
image data, 107 gzipped files (mostly gzipped text), 57 PNG files, 11 
PDF files, and there is a hodgepodge of assorted other formats. A few 
files are in the list merely because they lack a trailing newline (a 
problem for some tools, and POSIX says such files are not text files). 
(For a sense of the scale here, all this is out of 3695 files total in 
Emacs master right now.)

Most of the files in this list are merely awkward and are not a GPL 
issue; these include the gzipped text files, the text files that use 
non-UTF-8 encodings, and the no-trailing-newline files. And many files 
are there because we have put object files into the source repository, 
where we really should be generate them as part of the build and limit 
the repository to source code; these include the PBM and PNG files (or 
at least I hope they do; I haven't checked).

> Depends on the compression method, I guess. For some of them, we
> cannot rely on the respective tools being available, so if that
> particular kind of compression is needed for some test, having the
> source file might not be good enough.

We can assume gzip; it's universally available and is needed anyway to 
build tarballs. This will handle all the compressed files.


[-- Attachment #2: not-UTF-8-txt.ls --]
[-- Type: text/plain, Size: 13545 bytes --]

admin/charsets/glibc/BIG5.gz
admin/charsets/glibc/BIG5-HKSCS.gz
admin/charsets/glibc/CP10007.gz
admin/charsets/glibc/CP1125.gz
admin/charsets/glibc/CP1250.gz
admin/charsets/glibc/CP1251.gz
admin/charsets/glibc/CP1252.gz
admin/charsets/glibc/CP1253.gz
admin/charsets/glibc/CP1254.gz
admin/charsets/glibc/CP1255.gz
admin/charsets/glibc/CP1256.gz
admin/charsets/glibc/CP1257.gz
admin/charsets/glibc/CP1258.gz
admin/charsets/glibc/CP737.gz
admin/charsets/glibc/CP775.gz
admin/charsets/glibc/CP949.gz
admin/charsets/glibc/EBCDIC-UK.gz
admin/charsets/glibc/EBCDIC-US.gz
admin/charsets/glibc/EUC-JISX0213.gz
admin/charsets/glibc/EUC-JP.gz
admin/charsets/glibc/EUC-JP-MS.gz
admin/charsets/glibc/EUC-KR.gz
admin/charsets/glibc/EUC-TW.gz
admin/charsets/glibc/GB18030.gz
admin/charsets/glibc/GB2312.gz
admin/charsets/glibc/GBK.gz
admin/charsets/glibc/GEORGIAN-ACADEMY.gz
admin/charsets/glibc/GEORGIAN-PS.gz
admin/charsets/glibc/HP-ROMAN8.gz
admin/charsets/glibc/IBM037.gz
admin/charsets/glibc/IBM038.gz
admin/charsets/glibc/IBM1004.gz
admin/charsets/glibc/IBM1026.gz
admin/charsets/glibc/IBM1047.gz
admin/charsets/glibc/IBM256.gz
admin/charsets/glibc/IBM273.gz
admin/charsets/glibc/IBM274.gz
admin/charsets/glibc/IBM275.gz
admin/charsets/glibc/IBM277.gz
admin/charsets/glibc/IBM278.gz
admin/charsets/glibc/IBM280.gz
admin/charsets/glibc/IBM281.gz
admin/charsets/glibc/IBM284.gz
admin/charsets/glibc/IBM285.gz
admin/charsets/glibc/IBM290.gz
admin/charsets/glibc/IBM297.gz
admin/charsets/glibc/IBM420.gz
admin/charsets/glibc/IBM423.gz
admin/charsets/glibc/IBM424.gz
admin/charsets/glibc/IBM437.gz
admin/charsets/glibc/IBM500.gz
admin/charsets/glibc/IBM850.gz
admin/charsets/glibc/IBM851.gz
admin/charsets/glibc/IBM852.gz
admin/charsets/glibc/IBM855.gz
admin/charsets/glibc/IBM856.gz
admin/charsets/glibc/IBM857.gz
admin/charsets/glibc/IBM860.gz
admin/charsets/glibc/IBM861.gz
admin/charsets/glibc/IBM862.gz
admin/charsets/glibc/IBM863.gz
admin/charsets/glibc/IBM864.gz
admin/charsets/glibc/IBM865.gz
admin/charsets/glibc/IBM866.gz
admin/charsets/glibc/IBM868.gz
admin/charsets/glibc/IBM869.gz
admin/charsets/glibc/IBM870.gz
admin/charsets/glibc/IBM871.gz
admin/charsets/glibc/IBM874.gz
admin/charsets/glibc/IBM875.gz
admin/charsets/glibc/IBM880.gz
admin/charsets/glibc/IBM891.gz
admin/charsets/glibc/IBM903.gz
admin/charsets/glibc/IBM904.gz
admin/charsets/glibc/IBM905.gz
admin/charsets/glibc/IBM918.gz
admin/charsets/glibc/ISO-8859-10.gz
admin/charsets/glibc/ISO-8859-11.gz
admin/charsets/glibc/ISO-8859-13.gz
admin/charsets/glibc/ISO-8859-14.gz
admin/charsets/glibc/ISO-8859-15.gz
admin/charsets/glibc/ISO-8859-16.gz
admin/charsets/glibc/ISO-8859-2.gz
admin/charsets/glibc/ISO-8859-3.gz
admin/charsets/glibc/ISO-8859-4.gz
admin/charsets/glibc/ISO-8859-5.gz
admin/charsets/glibc/ISO-8859-6.gz
admin/charsets/glibc/ISO-8859-7.gz
admin/charsets/glibc/ISO-8859-8.gz
admin/charsets/glibc/ISO-8859-9.gz
admin/charsets/glibc/JIS_X0201.gz
admin/charsets/glibc/JOHAB.gz
admin/charsets/glibc/KOI-8.gz
admin/charsets/glibc/KOI8-R.gz
admin/charsets/glibc/KOI8-T.gz
admin/charsets/glibc/KOI8-U.gz
admin/charsets/glibc/KSC5636.gz
admin/charsets/glibc/MACINTOSH.gz
admin/charsets/glibc/MIK.gz
admin/charsets/glibc/NEXTSTEP.gz
admin/charsets/glibc/TCVN5712-1.gz
admin/charsets/glibc/TIS-620.gz
admin/charsets/glibc/VISCII.gz
admin/charsets/mapfiles/cns2ucsdkw.txt
doc/lispintro/cons-1.pdf
doc/lispintro/cons-2a.pdf
doc/lispintro/cons-2.pdf
doc/lispintro/cons-3.pdf
doc/lispintro/cons-4.pdf
doc/lispintro/cons-5.pdf
doc/lispintro/drawers.pdf
doc/lispintro/lambda-1.pdf
doc/lispintro/lambda-2.pdf
doc/lispintro/lambda-3.pdf
etc/e/eterm-color
etc/HELLO
etc/images/attach.pbm
etc/images/back-arrow.pbm
etc/images/bookmark_add.pbm
etc/images/cancel.pbm
etc/images/close.pbm
etc/images/connect.pbm
etc/images/contact.pbm
etc/images/copy.pbm
etc/images/custom/down.pbm
etc/images/custom/down-pushed.pbm
etc/images/custom/right.pbm
etc/images/custom/right-pushed.pbm
etc/images/cut.pbm
etc/images/data-save.pbm
etc/images/delete.pbm
etc/images/describe.pbm
etc/images/diropen.pbm
etc/images/disconnect.pbm
etc/images/exit.pbm
etc/images/ezimage/bitsbang.pbm
etc/images/ezimage/bits.pbm
etc/images/ezimage/box-minus.pbm
etc/images/ezimage/box.pbm
etc/images/ezimage/box-plus.pbm
etc/images/ezimage/checkmark.pbm
etc/images/ezimage/dir-minus.pbm
etc/images/ezimage/dir.pbm
etc/images/ezimage/dir-plus.pbm
etc/images/ezimage/doc-minus.pbm
etc/images/ezimage/doc.pbm
etc/images/ezimage/doc-plus.pbm
etc/images/ezimage/info.pbm
etc/images/ezimage/key.pbm
etc/images/ezimage/label.pbm
etc/images/ezimage/lock.pbm
etc/images/ezimage/mail.pbm
etc/images/ezimage/page-minus.pbm
etc/images/ezimage/page.pbm
etc/images/ezimage/page-plus.pbm
etc/images/ezimage/tag-gt.pbm
etc/images/ezimage/tag-minus.pbm
etc/images/ezimage/tag.pbm
etc/images/ezimage/tag-plus.pbm
etc/images/ezimage/tag-type.pbm
etc/images/ezimage/tag-v.pbm
etc/images/ezimage/unlock.pbm
etc/images/fwd-arrow.pbm
etc/images/gnus/catchup.pbm
etc/images/gnus/cu-exit.pbm
etc/images/gnus/describe-group.pbm
etc/images/gnus/exit-gnus.pbm
etc/images/gnus/exit-summ.pbm
etc/images/gnus/followup.pbm
etc/images/gnus/fuwo.pbm
etc/images/gnus/get-news.pbm
etc/images/gnus/gnntg.pbm
etc/images/gnus/gnus.png
etc/images/gnus/gnus-pointer.xpm
etc/images/gnus/important.pbm
etc/images/gnus/kill-group.pbm
etc/images/gnus/mail-reply.pbm
etc/images/gnus/mail-send.pbm
etc/images/gnus/next-ur.pbm
etc/images/gnus.pbm
etc/images/gnus/post.pbm
etc/images/gnus/prev-ur.pbm
etc/images/gnus/receipt.pbm
etc/images/gnus/reply.pbm
etc/images/gnus/reply-wo.pbm
etc/images/gnus/rot13.pbm
etc/images/gnus/save-aif.pbm
etc/images/gnus/save-art.pbm
etc/images/gnus/subscribe.pbm
etc/images/gnus/toggle-subscription.pbm
etc/images/gnus/unimportant.pbm
etc/images/gnus/unsubscribe.pbm
etc/images/gnus/uu-decode.pbm
etc/images/gnus/uu-post.pbm
etc/images/gud/all.pbm
etc/images/gud/break.pbm
etc/images/gud/cont.pbm
etc/images/gud/down.pbm
etc/images/gud/finish.pbm
etc/images/gud/go.pbm
etc/images/gud/nexti.pbm
etc/images/gud/next.pbm
etc/images/gud/pp.pbm
etc/images/gud/print.pbm
etc/images/gud/pstar.pbm
etc/images/gud/rcont.pbm
etc/images/gud/recstart.pbm
etc/images/gud/recstop.pbm
etc/images/gud/remove.pbm
etc/images/gud/rfinish.pbm
etc/images/gud/rnexti.pbm
etc/images/gud/rnext.pbm
etc/images/gud/rstepi.pbm
etc/images/gud/rstep.pbm
etc/images/gud/run.pbm
etc/images/gud/stepi.pbm
etc/images/gud/step.pbm
etc/images/gud/stop.pbm
etc/images/gud/thread.pbm
etc/images/gud/until.pbm
etc/images/gud/up.pbm
etc/images/gud/watch.pbm
etc/images/help.pbm
etc/images/home.pbm
etc/images/icons/allout-widgets/dark-bg/closed.png
etc/images/icons/allout-widgets/dark-bg/empty.png
etc/images/icons/allout-widgets/dark-bg/end-connector.png
etc/images/icons/allout-widgets/dark-bg/extender-connector.png
etc/images/icons/allout-widgets/dark-bg/leaf.png
etc/images/icons/allout-widgets/dark-bg/locked-encrypted.png
etc/images/icons/allout-widgets/dark-bg/mid-connector.png
etc/images/icons/allout-widgets/dark-bg/opened.png
etc/images/icons/allout-widgets/dark-bg/skip-descender.png
etc/images/icons/allout-widgets/dark-bg/through-descender.png
etc/images/icons/allout-widgets/dark-bg/unlocked-encrypted.png
etc/images/icons/allout-widgets/light-bg/closed.png
etc/images/icons/allout-widgets/light-bg/empty.png
etc/images/icons/allout-widgets/light-bg/end-connector.png
etc/images/icons/allout-widgets/light-bg/extender-connector.png
etc/images/icons/allout-widgets/light-bg/leaf.png
etc/images/icons/allout-widgets/light-bg/locked-encrypted.png
etc/images/icons/allout-widgets/light-bg/mid-connector.png
etc/images/icons/allout-widgets/light-bg/opened.png
etc/images/icons/allout-widgets/light-bg/skip-descender.png
etc/images/icons/allout-widgets/light-bg/through-descender.png
etc/images/icons/allout-widgets/light-bg/unlocked-encrypted.png
etc/images/icons/hicolor/128x128/apps/emacs23.png
etc/images/icons/hicolor/128x128/apps/emacs.png
etc/images/icons/hicolor/16x16/apps/emacs22.png
etc/images/icons/hicolor/16x16/apps/emacs23.png
etc/images/icons/hicolor/16x16/apps/emacs.png
etc/images/icons/hicolor/24x24/apps/emacs22.png
etc/images/icons/hicolor/24x24/apps/emacs23.png
etc/images/icons/hicolor/24x24/apps/emacs.png
etc/images/icons/hicolor/32x32/apps/emacs22.png
etc/images/icons/hicolor/32x32/apps/emacs23.png
etc/images/icons/hicolor/32x32/apps/emacs.png
etc/images/icons/hicolor/48x48/apps/emacs22.png
etc/images/icons/hicolor/48x48/apps/emacs23.png
etc/images/icons/hicolor/48x48/apps/emacs.png
etc/images/icons/hicolor/scalable/mimetypes/emacs-document.svg
etc/images/index.pbm
etc/images/info.pbm
etc/images/jump-to.pbm
etc/images/left-arrow.pbm
etc/images/letter.pbm
etc/images/lock-broken.pbm
etc/images/lock-ok.pbm
etc/images/lock.pbm
etc/images/mail/compose.pbm
etc/images/mail/copy.pbm
etc/images/mail/flag-for-followup.pbm
etc/images/mail/forward.pbm
etc/images/mail/inbox.pbm
etc/images/mail/move.pbm
etc/images/mail/not-spam.pbm
etc/images/mail/outbox.pbm
etc/images/mail/preview.pbm
etc/images/mail/repack.pbm
etc/images/mail/reply-all.pbm
etc/images/mail/reply-from.pbm
etc/images/mail/reply.pbm
etc/images/mail/reply-to.pbm
etc/images/mail/save-draft.pbm
etc/images/mail/send.pbm
etc/images/mh-logo.pbm
etc/images/mpc/add.pbm
etc/images/mpc/ffwd.pbm
etc/images/mpc/next.pbm
etc/images/mpc/pause.pbm
etc/images/mpc/play.pbm
etc/images/mpc/prev.pbm
etc/images/mpc/rewind.pbm
etc/images/mpc/stop.pbm
etc/images/new.pbm
etc/images/newsticker/rss-feed.png
etc/images/next-node.pbm
etc/images/next-page.pbm
etc/images/open.pbm
etc/images/paste.pbm
etc/images/preferences.pbm
etc/images/prev-node.pbm
etc/images/print.pbm
etc/images/redo.pbm
etc/images/refresh.pbm
etc/images/right-arrow.pbm
etc/images/saveas.pbm
etc/images/save.pbm
etc/images/search.pbm
etc/images/search-replace.pbm
etc/images/separator.pbm
etc/images/show.pbm
etc/images/smilies/blink.pbm
etc/images/smilies/braindamaged.pbm
etc/images/smilies/cry.pbm
etc/images/smilies/dead.pbm
etc/images/smilies/evil.pbm
etc/images/smilies/forced.pbm
etc/images/smilies/frown.pbm
etc/images/smilies/grin.pbm
etc/images/smilies/indifferent.pbm
etc/images/smilies/sad.pbm
etc/images/smilies/smile.pbm
etc/images/smilies/wry.pbm
etc/images/sort-ascending.pbm
etc/images/sort-column-ascending.pbm
etc/images/sort-criteria.pbm
etc/images/sort-descending.pbm
etc/images/sort-row-ascending.pbm
etc/images/spell.pbm
etc/images/splash.pbm
etc/images/splash.png
etc/images/tree-widget/default/close.png
etc/images/tree-widget/default/empty.png
etc/images/tree-widget/default/end-guide.png
etc/images/tree-widget/default/guide.png
etc/images/tree-widget/default/handle.png
etc/images/tree-widget/default/leaf.png
etc/images/tree-widget/default/no-guide.png
etc/images/tree-widget/default/no-handle.png
etc/images/tree-widget/default/open.png
etc/images/tree-widget/folder/close.png
etc/images/tree-widget/folder/empty.png
etc/images/tree-widget/folder/end-guide.png
etc/images/tree-widget/folder/guide.png
etc/images/tree-widget/folder/handle.png
etc/images/tree-widget/folder/leaf.png
etc/images/tree-widget/folder/no-guide.png
etc/images/tree-widget/folder/no-handle.png
etc/images/tree-widget/folder/open.png
etc/images/unchecked.pbm
etc/images/undo.pbm
etc/images/up-arrow.pbm
etc/images/up-node.pbm
etc/images/zoom-in.pbm
etc/images/zoom-out.pbm
etc/package-keyring.gpg
etc/refcards/gnus-logo.pdf
etc/tutorials/TUTORIAL.ja
leim/CXTERM-DIC/4Corner.tit
leim/CXTERM-DIC/ARRAY30.tit
leim/CXTERM-DIC/CCDOSPY.tit
leim/CXTERM-DIC/ECDICT.tit
leim/CXTERM-DIC/ETZY.tit
leim/CXTERM-DIC/Punct-b5.tit
leim/CXTERM-DIC/Punct.tit
leim/CXTERM-DIC/PY-b5.tit
leim/CXTERM-DIC/QJ-b5.tit
leim/CXTERM-DIC/QJ.tit
leim/CXTERM-DIC/SW.tit
leim/CXTERM-DIC/TONEPY.tit
leim/CXTERM-DIC/ZOZY.tit
leim/MISC-DIC/cangjie-table.b5
leim/MISC-DIC/cangjie-table.cns
leim/MISC-DIC/CTLau-b5.html
leim/MISC-DIC/CTLau.html
leim/MISC-DIC/pinyin.map
leim/MISC-DIC/ziranma.cin
leim/SKK-DIC/SKK-JISYO.L
lisp/international/ja-dic-cnv.el
lisp/international/ja-dic-utl.el
lisp/international/kinsoku.el
lisp/international/kkc.el
lisp/international/titdic-cnv.el
lisp/language/ethiopic.el
lisp/language/ethio-util.el
lisp/language/ind-util.el
lisp/language/japanese.el
lisp/language/japan-util.el
lisp/language/tibetan.el
lisp/language/tibet-util.el
lisp/leim/quail/cyril-jis.el
lisp/leim/quail/ethiopic.el
lisp/leim/quail/hanja-jis.el
lisp/leim/quail/japanese.el
lisp/leim/quail/pypunct-b5.el
lisp/leim/quail/py-punct.el
lisp/leim/quail/tibetan.el
msdos/emacs.ico
msdos/emacs.pif
nextstep/Cocoa/Emacs.base/Contents/PkgInfo
nextstep/Cocoa/Emacs.base/Contents/Resources/document.icns
nextstep/Cocoa/Emacs.base/Contents/Resources/Emacs.icns
nextstep/GNUstep/Emacs.base/Resources/emacs.tiff
nt/icons/emacs22.ico
nt/icons/emacs.ico
nt/icons/gnu2a32.ico
nt/icons/gnu2a32t.ico
nt/icons/gnu2b48.ico
nt/icons/gnu2b48t.ico
nt/icons/gnu3b32.ico
nt/icons/gnu3b32t.ico
nt/icons/gnu4g48.ico
nt/icons/gnu4g48t.ico
nt/icons/gnu5w32.ico
nt/icons/gnu5w32t.ico
nt/icons/gnu6w48.ico
nt/icons/gnu6w48t.ico
nt/icons/gnu7.ico
nt/icons/gnu8.ico
nt/icons/gnu9.ico
nt/icons/hand.cur
src/msdos.c
test/data/decompress/foo.gz
test/data/files-bug18141.el.gz
test/data/shr/ul-empty.txt
test/lisp/emacs-lisp/package-resources/multi-file-0.2.3.tar
test/lisp/emacs-lisp/package-resources/signed/archive-contents.sig
test/lisp/emacs-lisp/package-resources/signed/signed-bad-1.0.el.sig
test/lisp/emacs-lisp/package-resources/signed/signed-good-1.0.el.sig
test/manual/etags/cp-src/clheir.cpp.gz
test/manual/etags/f-src/entry.strange.gz
test/manual/etags/tex-src/nonewline.tex

Re: Checking in files with trailing white space

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 14 Dec 2016 10:15:55 -0800
> Cc: acm@muc.de, larsi@gnus.org, emacs-devel@gnu.org
> 
> I've attached a list of repository-controlled files that are not UTF-8 
> text files in the POSIX sense. Of the 433 such files, I count 181 Netpbm 
> image data, 107 gzipped files (mostly gzipped text), 57 PNG files, 11 
> PDF files, and there is a hodgepodge of assorted other formats. A few 
> files are in the list merely because they lack a trailing newline (a 
> problem for some tools, and POSIX says such files are not text files). 
> (For a sense of the scale here, all this is out of 3695 files total in 
> Emacs master right now.)
> 
> Most of the files in this list are merely awkward and are not a GPL 
> issue; these include the gzipped text files, the text files that use 
> non-UTF-8 encodings, and the no-trailing-newline files. And many files 
> are there because we have put object files into the source repository, 
> where we really should be generate them as part of the build and limit 
> the repository to source code; these include the PBM and PNG files (or 
> at least I hope they do; I haven't checked).

There are no licensing issues with any of those, each one was verified
at the time, AFAIR.

As for generating them from text: feel free to scratch that itch, if
you want to.

> > Depends on the compression method, I guess. For some of them, we
> > cannot rely on the respective tools being available, so if that
> > particular kind of compression is needed for some test, having the
> > source file might not be good enough.
> 
> We can assume gzip

Yes.  Gzip was not what I had in mind.