Files from gnulib

Files from gnulib

[Eli Zaretskii]

Some of the new files added from gnulib use file names that are
invalid on 8+3 (aka MSDOS) filesystems.  These files are:

 c++defs.h -- the `+' character is not allowed in file names
 lib/stddef.in.h lib/time.in.h lib/unistd.in.h -- 2 dots are not allowed

In addition, several file names in the m4/ subdirectory clash after
8+3 truncation:

 gnulib-cache.m4  gnulib-common.m4  gnulib-comp.m4

This will cause problems when unpacking the emacs tarball on those
filesystems, something we managed to avoid until now.

Can these files be renamed, please?  I suggest cxxdefs.h for the first
and FOO.in for the FOO.in.h files.  For the files in m4/, how about
transposing the "gnulib" part with the rest, i.e. cache-gnulib.m4
etc.?

(I would remove the files myself, but I don't know what changes in the
gnulib imports will that require.)

TIA

Re: Files from gnulib

[Eli Zaretskii]

> Date: Sun, 23 Jan 2011 14:15:43 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
>  lib/stddef.in.h lib/time.in.h lib/unistd.in.h -- 2 dots are not allowed

Forgot getopt.in.h.

Re: Files from gnulib

[Paul Eggert]

[Adding bug-gnulib to the thread that started in
 <http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00761.html>]:

On 01/23/2011 04:15 AM, Eli Zaretskii wrote:
> Some of the new files added from gnulib use file names that are
> invalid on 8+3 (aka MSDOS) filesystems.  These files are:
> 
>  c++defs.h -- the `+' character is not allowed in file names

I am sympathetic to the proposal to rename c++defs.h to cxxdefs.h,
as "+" is not in the POSIX portable file name character set; see
<http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_276>.

>  lib/stddef.in.h lib/time.in.h lib/unistd.in.h -- 2 dots are not allowed

The Emacs trunk already has seventeen other files
with 2 dots in their file names, with names like
lisp/gnus/.dir-locals.el and admin/charsets/mapfiles/symbol.txt.gz.
Since these haven't been a problem, why would file names
like getopt.in.h be a problem?

> In addition, several file names in the m4/ subdirectory clash after
> 8+3 truncation:
> 
>  gnulib-cache.m4  gnulib-common.m4  gnulib-comp.m4

Again, the Emacs trunk already has several instances of truncation
after 8+3 limits, such as lisp/org/org-compat.el versus
lisp/org/org-complete.el, and test/cedet/semantic-ia-utest.el
versus test/cedet/semantic-tests.el, and I don't see why
files imported from gnulib would be different.

> This will cause problems when unpacking the emacs tarball on those
> filesystems, something we managed to avoid until now.
> 
> Can these files be renamed, please?  I suggest cxxdefs.h for the first
> and FOO.in for the FOO.in.h files.  For the files in m4/, how about
> transposing the "gnulib" part with the rest, i.e. cache-gnulib.m4
> etc.?
> 
> (I would remove the files myself, but I don't know what changes in the
> gnulib imports will that require.)

Developers on Microsoft Windows platforms all have access to
better file systems these days, surely.  And as shown above
Emacs does not respect those limitations in other parts
of its source tree.  So, other than c++defs.h, I don't see
the need for renaming these files.

Re: Files from gnulib

[Bruno Haible]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

Paul Eggert replied to Eli Zaretskii:
> > Some of the new files added from gnulib use file names that are
> > invalid on 8+3 (aka MSDOS) filesystems.  These files are:
> > 
> >  c++defs.h -- the `+' character is not allowed in file names
> 
> I am sympathetic to the proposal to rename c++defs.h to cxxdefs.h,
> as "+" is not in the POSIX portable file name character set; see
> <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_276>.

I'm not sympathetic to this proposal. gnulib supports POSIX platforms
and mingw, and all known POSIX platforms and mingw support '+' in file names.

Eli's point is that he wants support for MSDOS file system. This is outside
the scope of gnulib, and therefore is best solved within Emacs. It can be
done by adding 2 files to the Emacs tree (see attached .tar.gz file) and
by adding the option --local-dir=gnulib-local to the gnulib-tool invocation.
See the gnulib documentation [1] for details.

Bruno

[1] <http://www.gnu.org/software/gnulib/manual/html_node/Extending-Gnulib.html>

[-- Attachment #2: new-files.tar.gz --]
[-- Type: application/x-tgz, Size: 1301 bytes --]

Re: Files from gnulib

[Stefan Monnier]

> The Emacs trunk already has seventeen other files
> with 2 dots in their file names, with names like
> lisp/gnus/.dir-locals.el

I don't know how MSDOS handles names that start with dot, so either this
is a special case that's OK, or it's indeed a problem which we hadn't
noticed yet.

> and admin/charsets/mapfiles/symbol.txt.gz.

This one, OTOH is OK because the `admin' subdir is not included in the
tarball, so its name is irrelevant.

>> In addition, several file names in the m4/ subdirectory clash after
>> 8+3 truncation:
>> gnulib-cache.m4  gnulib-common.m4  gnulib-comp.m4

> Again, the Emacs trunk already has several instances of truncation
> after 8+3 limits, such as lisp/org/org-compat.el versus
> lisp/org/org-complete.el,

Good point.  I guess Eli hadn't noticed it yet.

> and test/cedet/semantic-ia-utest.el
> versus test/cedet/semantic-tests.el, and I don't see why
> files imported from gnulib would be different.

Here, again, the `test' subdir is not included in the tarball, so it's
not an issue.


        Stefan

Re: Files from gnulib

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  bug-gnulib <bug-gnulib@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 23 Jan 2011 22:26:52 -0500
> 
> > The Emacs trunk already has seventeen other files
> > with 2 dots in their file names, with names like
> > lisp/gnus/.dir-locals.el
> 
> I don't know how MSDOS handles names that start with dot, so either this
> is a special case that's OK, or it's indeed a problem which we hadn't
> noticed yet.

The program used to unpack the .tar.gz archives automatically renames
ant .FOO files to _FOO while unpacking, so that's not a problem.
(This program, called djtar.exe, is part of the DJGPP package which is
used to build Emacs on MSDOS.)

> > and admin/charsets/mapfiles/symbol.txt.gz.
> 
> This one, OTOH is OK because the `admin' subdir is not included in the
> tarball, so its name is irrelevant.

Right.

> > Again, the Emacs trunk already has several instances of truncation
> > after 8+3 limits, such as lisp/org/org-compat.el versus
> > lisp/org/org-complete.el,
> 
> Good point.  I guess Eli hadn't noticed it yet.

I did notice that.  These files appeared very recently, and I didn't
yet have time to talk to Org mode maintainers about renaming them.
In the past a couple of other Org files were renamed for this reason,
at my request.

> > and test/cedet/semantic-ia-utest.el
> > versus test/cedet/semantic-tests.el, and I don't see why
> > files imported from gnulib would be different.
> 
> Here, again, the `test' subdir is not included in the tarball, so it's
> not an issue.

Right.

Re: Files from gnulib

[Paul Eggert]

On 01/23/11 20:01, Eli Zaretskii wrote:
> The program used to unpack the .tar.gz archives automatically renames
> ant .FOO files to _FOO while unpacking, so that's not a problem.

OK, so it sounds like there's already part of a solution here,
in that files are automatically renamed to avoid the MS-DOS
restrictions.

How about if we expand on that solution, as follows:

* Just before creating a tarball for MS-DOS, apply
  the following substitutions to the contents of all
  files that go into the tarball:

	s/c++defs\.h/cxxdefs\.h/g
	s/\([a-zA-Z0-9_]*\)\.in\.h/_\1.h/g

  The latter substitution, for example, replaces all
  instances of unistd.in.h with _unistd.h.

* Similarly, just before creating the tarball, rename
  all the source files according to the above patterns.

* The above can be done in the make-dist implementation
  for MS-DOS.

* The conflicting names in m4/* don't matter, for the same
  reason that conflicts in admin/* and tests/* don't matter:
  these files are not used in an MS-DOS build.

With this approach, the problem of MS-DOS names is handled
entirely in the MS-DOS port.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Mon, 24 Jan 2011 15:26:06 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Stefan Monnier <monnier@iro.umontreal.ca>, bug-gnulib@gnu.org, 
>  emacs-devel@gnu.org
> 
> On 01/23/11 20:01, Eli Zaretskii wrote:
> > The program used to unpack the .tar.gz archives automatically renames
> > ant .FOO files to _FOO while unpacking, so that's not a problem.
> 
> OK, so it sounds like there's already part of a solution here,
> in that files are automatically renamed to avoid the MS-DOS
> restrictions.
> 
> How about if we expand on that solution, as follows:
> 
> * Just before creating a tarball for MS-DOS, apply
>   the following substitutions to the contents of all
>   files that go into the tarball:

There are no special tarballs for MSDOS.  People use the tarball from
the GNU FTP site.  So this solution will not work.

> * The conflicting names in m4/* don't matter, for the same
>   reason that conflicts in admin/* and tests/* don't matter:
>   these files are not used in an MS-DOS build.

The files in m4/ still matter because you need to unpack the tarball,
and the utility that does that won't silently overwrite files due to
file-name clashes.

Re: Files from gnulib

[Paul Eggert]

On 01/24/2011 08:00 PM, Eli Zaretskii wrote:
> There are no special tarballs for MSDOS.

Ah, sorry, then I misunderstood.  Well, then, instead of
doing the stuff I mentioned before the tarball is created,
we can add a shell script to be run after the tarball is extracted.
For example, on MS-DOS the c++defs.h file is automatically renamed
to cxxdefs.h by the extractor, so the script can uniformly substitute
"cxxdefs.h" for "c++defs.h" in all the text files.  Hopefully
a similar idea works for all the other files with non-MS-DOS
names.

I assume that the "configure" procedure for MS-DOS is already
different, so this new script can be folded into that procedure.

> The files in m4/ still matter because you need to unpack the tarball,
> and the utility that does that won't silently overwrite files due to
> file-name clashes.

That's OK.  People can ignore those diagnostics, just as I assume
they already ignore the diagnostics for the files whose names
start with ".".  It doesn't matter whether those files are
extracted correctly, as the MS-DOS build doesn't use them.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 00:48:12 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: monnier@iro.umontreal.ca, bug-gnulib@gnu.org, emacs-devel@gnu.org
> 
> instead of doing the stuff I mentioned before the tarball is
> created, we can add a shell script to be run after the tarball is
> extracted.  For example, on MS-DOS the c++defs.h file is
> automatically renamed to cxxdefs.h by the extractor, so the script
> can uniformly substitute "cxxdefs.h" for "c++defs.h" in all the text
> files.  Hopefully a similar idea works for all the other files with
> non-MS-DOS names.

The magic coded into the extractor utility is limited.  It indeed
handles file names with `+' in them, but does not handle arbitrary
file names with multiple dots as a human would.  It uses some
convoluted algorithm to replace the extra dot with a `_' or a `-';
sometimes it replaces the first dot, sometimes the second.  The
results are often unpredictable or surprising, especially if, as it
often happens, the modified names also clash in the 8+3 namespace (see
below).

So going this way means a much more complex and error-prone
arrangement than a one-time rename of a small number of files.

> I assume that the "configure" procedure for MS-DOS is already
> different

Yes, see config.bat in the top-level directory.  But that's not the
issue here.

> > The files in m4/ still matter because you need to unpack the tarball,
> > and the utility that does that won't silently overwrite files due to
> > file-name clashes.
> 
> That's OK.  People can ignore those diagnostics

The result is not ignorable diagnostics, but a prompt for the user to
provide an alternate name.  Since the user does not generally know
whether these files are needed by the build, she will not be able to
deal with the prompt.

> just as I assume they already ignore the diagnostics for the files
> whose names start with ".".

No, there are no diagnostics for these conversions, they are done
silently.

Re: Files from gnulib

[Bastien ROUCARIES]

Le mardi 25 janvier 2011 12:24:56, Eli Zaretskii a écrit :
> > Date: Tue, 25 Jan 2011 00:48:12 -0800
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > CC: monnier@iro.umontreal.ca, bug-gnulib@gnu.org, emacs-devel@gnu.org
> > 
> > instead of doing the stuff I mentioned before the tarball is
> > created, we can add a shell script to be run after the tarball is
> > extracted.  For example, on MS-DOS the c++defs.h file is
> > automatically renamed to cxxdefs.h by the extractor, so the script
> > can uniformly substitute "cxxdefs.h" for "c++defs.h" in all the text
> > files.  Hopefully a similar idea works for all the other files with
> > non-MS-DOS names.
> 
> The magic coded into the extractor utility is limited.  It indeed
> handles file names with `+' in them, but does not handle arbitrary
> file names with multiple dots as a human would.  It uses some
> convoluted algorithm to replace the extra dot with a `_' or a `-';
> sometimes it replaces the first dot, sometimes the second.  The
> results are often unpredictable or surprising, especially if, as it
> often happens, the modified names also clash in the 8+3 namespace (see
> below).

As I have mentionned previously, does doslfn is an option ?
 
Bastien

Re: Files from gnulib

[Eli Zaretskii]

> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Date: Tue, 25 Jan 2011 12:32:41 +0100
> Cc: Paul Eggert <eggert@cs.ucla.edu>,
>  monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> 
> As I have mentionned previously, does doslfn is an option ?

Yeah, only 3 hours ago.  Don't you think you are a bit impatient?  The
question wasn't exactly a trivial one.

Re: Files from gnulib

[Paul Eggert]

On 01/25/11 03:24, Eli Zaretskii wrote:

> It uses some
> convoluted algorithm to replace the extra dot with a `_' or a `-';
> sometimes it replaces the first dot, sometimes the second.  The
> results are often unpredictable or surprising, especially if, as it
> often happens, the modified names also clash in the 8+3 namespace (see
> below).

The results may seem convoluted, but I doubt whether they are actually
unpredictable.  I suspect we can accommodate whatever programmatic scheme
djtar defaults to.  Or we can tell djtar to use our own scheme, as
discussed below.

>> That's OK.  People can ignore those diagnostics
> 
> The result is not ignorable diagnostics, but a prompt for the user to
> provide an alternate name.  Since the user does not generally know
> whether these files are needed by the build, she will not be able to
> deal with the prompt.

We already provide instructions for people who want to build Emacs
on MS-DOS platforms.  Our instructions can be expanded slightly
to tell them how to extract the files in the first place, and how
to deal with such prompts.  They can be asked to just type RETURN,
for example.

Or, if that's too much trouble, the MS-DOS build instructions could
instead tell people to fetch a small file "emacs-25.chg", and then
execute

   djtar -n emacs-25.chg emacs-25.tgz

where emacs-25.chg is a list of desired file name conversions.
That is not much trouble, and it also addresses the extraction
name-change issues that have been raised so far.  If we don't
want to distribute this small file separately, we can bundle it
as part of the Emacs tarball, and give instructions on how to
extract it separately first.

> So going this way means a much more complex and error-prone
> arrangement than a one-time rename of a small number of files.

It is a bit more complex and error-prone for MS-DOS, yes.  But
it has the advantage of compartmentalizing the MS-DOS restrictions
into the MS-DOS build instructions, rather than having these restrictions
spread to Emacs more generally, where they complicate software
integration efforts there.  There is a significant advantage to
modularization and separation of concerns, one that outweighs minor
increases in complexity for individual components.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 10:07:45 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: monnier@iro.umontreal.ca, bug-gnulib@gnu.org, emacs-devel@gnu.org
> 
> We already provide instructions for people who want to build Emacs
> on MS-DOS platforms.  Our instructions can be expanded slightly
> to tell them how to extract the files in the first place, and how
> to deal with such prompts.  They can be asked to just type RETURN,
> for example.

To read the instructions, you need to unpack the archive first.

>    djtar -n emacs-25.chg emacs-25.tgz
> ...
> It is a bit more complex and error-prone for MS-DOS, yes.  But
> it has the advantage of compartmentalizing the MS-DOS restrictions
> into the MS-DOS build instructions

Sorry, but these complications are unacceptable for me.  We use
something like that in GDB, and the result is extremely fragile and
error-prone, I find problems with missing renames every time I build
GDB.  I'm not going to make the same kind or mistake again in Emacs.

Stefan and Chong, please make a decision regarding this issue.  If the
decision is not to rename these few files in the Emacs distribution,
and instead ask me to cope with these complications, I will understand
that the knee-jerk reaction of too many members of this community when
they hear "MS-DOS" is more important that any voice of reason, and I
will therefore resign from everything I do for Emacs development.

Re: Files from gnulib

[Stefan Monnier]

>> djtar -n emacs-25.chg emacs-25.tgz
>> ...
>> It is a bit more complex and error-prone for MS-DOS, yes.  But
>> it has the advantage of compartmentalizing the MS-DOS restrictions
>> into the MS-DOS build instructions

> Sorry, but these complications are unacceptable for me.  We use
> something like that in GDB, and the result is extremely fragile and
> error-prone, I find problems with missing renames every time I build
> GDB.  I'm not going to make the same kind or mistake again in Emacs.

Here's a compromise:
- let's do the renames that need to be done (they're really not
  problematic, AFAICT).
- let's try and get this "djtar -n emacs-25.chg emacs-25.tgz" to work
  reliably, e.g. by auto-building the emacs-25.chg file as part of
  make-dist, so there's no way for that file to be "missing renames".


        Stefan

Re: Files from gnulib

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Paul Eggert <eggert@cs.ucla.edu>, cyd@stupidchicken.com,
>         bug-gnulib@gnu.org, emacs-devel@gnu.org
> Date: Tue, 25 Jan 2011 16:03:16 -0500
> 
> - let's try and get this "djtar -n emacs-25.chg emacs-25.tgz" to work
>   reliably, e.g. by auto-building the emacs-25.chg file as part of
>   make-dist, so there's no way for that file to be "missing renames".

Based on the experience from GDB, here are the problems with this
approach:

  . The remapping of file names cannot be fully automated.  Once all
    limits on file names are removed ("we don't want even the
    slightest impediment", says Jim), sooner or later you get to the
    point where the number of files to be renamed becomes large and a
    human needs to define how files will be renamed, because you want
    names that at least remotely resemble the original ones, but still
    don't clash in the 8+3 namespace.  (I hope no one is seriously
    entertaining the idea of producing meaningless names like
    foo~123.c or ABC123EF.c, from some hash or whatever).

  . Once the remapping is maintained by humans, it becomes unreliable.

  . The unpacking instructions are part of the tarball, and need to be
    extracted separately before the "main" extraction begins.  More
    importantly, the remapping file needs to be extracted before that
    as well.  This makes the entire unpacking procedure extremely
    complicated and thus error-prone, except if the person who does
    that is the one who designed it and wrote the instructions in the
    first place.

In addition, there will be a need to deal with something that the GDB
distribution doesn't.  In GDB, the files that are renamed during
unpacking are only those that are not used for the DOS build.  By
contrast, here we want to rename files that are used during the build.
So there will be a need to edit all the files that reference the
renamed ones, before the build can begin.  This can be done with Sed,
but the DOS shell doesn't have any way of recursively descending
through a directory tree.  So, if we want to make this editing as part
of config.bat, the subdirectories of the Emacs tree will have to be
hard-coded in config.bat, which is yet another source of errors, when
a subdirectory is added or removed.  (The alternative, of using a port
of GNU Find, would mean addition of another tool that is not required
today for building Emacs.)

Perhaps it's possible to solve all this in a satisfactory manner, but
doing so would require a lot of work, much more than I am willing to
invest.  (I still want to finish development of bidirectional editing,
remember?)  Given the small (yes, small!) fraction of files that ever
need to be renamed, the significant effort needed for getting this
particular alternative right is not justified.

Re: Files from gnulib

[Stefan Monnier]

> Based on the experience from GDB, here are the problems with this
> approach:

>   . The remapping of file names cannot be fully automated.  Once all
>     limits on file names are removed ("we don't want even the
>     slightest impediment", says Jim), sooner or later you get to the
>     point where the number of files to be renamed becomes large and a
>     human needs to define how files will be renamed, because you want
>     names that at least remotely resemble the original ones, but still
>     don't clash in the 8+3 namespace.  (I hope no one is seriously
>     entertaining the idea of producing meaningless names like
>     foo~123.c or ABC123EF.c, from some hash or whatever).

Yes, I was assuming that there would still be few files, and that the
renaming would be human-controlled (but the make-dist script signals an
error if the human-provided rules don't cover all cases).  So it
wouldn't help for cases like CEDET.

>   . Once the remapping is maintained by humans, it becomes unreliable.

Why is that?

>   . The unpacking instructions are part of the tarball, and need to be
>     extracted separately before the "main" extraction begins.  More
>     importantly, the remapping file needs to be extracted before that
>     as well.  This makes the entire unpacking procedure extremely
>     complicated and thus error-prone, except if the person who does
>     that is the one who designed it and wrote the instructions in the
>     first place.

Rather than distribute a file that needs to be passed to djtar, I was
thinking of distributing a script tailored to MS-DOS, run instead of
djtar, and which would run djtar insternally.  So this script can
provide the instructions.

> In addition, there will be a need to deal with something that the GDB
> distribution doesn't.  In GDB, the files that are renamed during
> unpacking are only those that are not used for the DOS build.  By
> contrast, here we want to rename files that are used during the build.

Yes, that's a much bigger problem.

In the mean time, please rename the files, thank you,


        Stefan

Re: Files from gnulib

[Paul Eggert]

On 01/25/11 14:15, Stefan Monnier wrote:
> In the mean time, please rename the files, thank you,

I'm afraid it's not as simple as merely renaming the files.
One must also change the contents of all files referring to
the renamed files, and change Makefile.in so that when
fresh copies are imported from gnulib, the files' names
and contents are consistently renamed.  And in the end,
one will end up with files that differ from gnulib,
which will make it a bit harder to port patches from Emacs
back to gnulib.

I'd like to pursue the idea of renaming the files only
on the MS-DOS platform.  This approach shouldn't take
that much work now, and it should save time in the future
(including lessening the need to discuss 8+3 issues on
this mailing list :-).

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 17:05:28 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, cyd@stupidchicken.com, 
>  bug-gnulib@gnu.org, emacs-devel@gnu.org
> 
> I'm afraid it's not as simple as merely renaming the files.
> One must also change the contents of all files referring to
> the renamed files, and change Makefile.in so that when
> fresh copies are imported from gnulib, the files' names
> and contents are consistently renamed.

These are all parts of the job that needs to be done under your
suggestion as well--to rename and edit when Emacs is built on MSDOS.

> I'd like to pursue the idea of renaming the files only
> on the MS-DOS platform.  This approach shouldn't take
> that much work now, and it should save time in the future

That approach as suggested (modeled on what GDB does) is unacceptable,
sorry.  I already said what it means for me personally if it will be
accepted here.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Wed, 26 Jan 2011 06:12:55 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: cyd@stupidchicken.com, bug-gnulib@gnu.org, monnier@IRO.UMontreal.CA,
> 	emacs-devel@gnu.org
> 
> > I'd like to pursue the idea of renaming the files only
> > on the MS-DOS platform.  This approach shouldn't take
> > that much work now, and it should save time in the future
> 
> That approach as suggested (modeled on what GDB does) is unacceptable,
> sorry.

Here's a compromise that at least I can live with; hope others can,
too:

 1) For c++defs.h and the lib/*.in.h files: rely on the automatic
    renaming by djtar.  Files that reference those will be edited by
    config.bat as part of configuring Emacs for the MS-DOS build.

 2) For the 3 m4/gnulib-*.m4 files: rename them.  I suggested one way
    of renaming, but there's nothing sacred about that; any renaming
    that will get rid of the name clash is fine with me.

As long as the list of files that get handled by 1) is relatively
small and kept under control, this is manageable.  This means no "open
season" for disregarding the issue altogether, as in "we already have
such problems elsewhere, why not add a few more".

I hope that as long as the list of files that are handled by 2) is
short (3 for now), that would be regarded as manageable and acceptable
by gnulib people.

[Make no mistake: supporting 1) is still a complication.  There's more
there than meets the eye.  For example, when djtar runs on Windows, it
doesn't perform any automatic renames, because it knows that long file
names are available (DJGPP has built-in support for them in the
Standard C library).  So, to support the Emacs build on both plain DOS
and Windows, config.bat will need to do slightly different things in
each case, which means more scripting and more testing.  But these
complications are relatively minor, don't affect the end users, and I
have good experience with solving such problems, in Emacs and
elsewhere.]

Proposed gnulib renames [was: Files from gnulib]

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3980 bytes --]

[intentionally breaking threading and retitling this, to try and make it
easier to see replies to just this aspect of the thread]

On 01/25/2011 11:01 PM, Eli Zaretskii wrote:
> Here's a compromise that at least I can live with; hope others can,
> too:
> 
>  1) For c++defs.h and the lib/*.in.h files: rely on the automatic
>     renaming by djtar.  Files that reference those will be edited by
>     config.bat as part of configuring Emacs for the MS-DOS build.

Paul already suggested renaming c++defs.h to cxxdefs.h in gnulib, which
makes sense to me in light of POSIX restrictions on portable filenames;
however, this module belongs to Bruno, so it is his call.

As for the *.in.h files, the ONLY other files that refer to those at
build time are the Makefile snippets in module files that convert them
over to *.h replacement headers.  Here's a link to some of the
back-discussion where we first settled on the name *.in.h in the first
place in Oct 2007 (we were previously using *_.h, but underscores are
painful to type in daily use; and also on the table at the time was the
rejected idea of *.h.in and *.hin):

http://thread.gmane.org/gmane.comp.lib.gnulib.bugs/11185/focus=11206

and given that the renaming from *_.h -> *.in.h was practically
mechanical, a conversion from *.in.h -> *-in.h would likewise be
mechanical, but I'm not sure whether to make that jump yet:

http://thread.gmane.org/gmane.comp.lib.gnulib.bugs/11182/focus=11352

that thread also included a quote from Eli, predicting the death of an
emacs DOS build (for good or for bad, that prediction has not come true):

http://thread.gmane.org/gmane.comp.lib.gnulib.bugs/11182/focus=11234

> 
>  2) For the 3 m4/gnulib-*.m4 files: rename them.  I suggested one way
>     of renaming, but there's nothing sacred about that; any renaming
>     that will get rid of the name clash is fine with me.

This would involve a change mainly to gnulib-tool (2 of the 3 gnulib-c*
files are generated; there's also gnulib-tool.m4 to avoid clashing
with).  Changing gnulib-cache.m4 would have downstream effects on all
gnulib clients; it could be done with a NEWS entry, but I'd rather avoid
that.  But the other two files (gnulib-common.m4 and gnulib-comp.m4)
seem like fair game; how about the names gnulib-prereq.m4 (since all the
common code is prerequisite to the rest of gnulib) and gnulib-list.m4
(since comp contains the computed list of files/macros installed by
gnulib-tool).

> As long as the list of files that get handled by 1) is relatively
> small and kept under control, this is manageable.

Right now, there is the c++defs naming issue (with user-visible changes
to all gnulib clients, but worth making).

Also, there are currently 63 *.in.h files available in gnulib (I'm not
sure how many will ever be used by emacs), but I'd rather see the same
change made to all 63 files (and their corresponding modules) at once if
the change is made upstream in gnulib.  I agree that use of gnulib-tool
--local-dir won't quite work; it could easily patch the module
descriptions (and makefile snippets) to refer to *-in.h files with
minimal risk of too many upstream changes to track, but the rename
itself is not possible via simple patch(1) files (yes, GNU patchutils is
adding support for git-style rename patches, which would be much easier
to maintain, but I don't know if we should rely on that yet).  But if
gnulib makes no change to these 63 files, then that's an upper bound on
the possible complexity of Eli hand-maintaining the rename within the
DOS build of emacs.

> I hope that as long as the list of files that are handled by 2) is
> short (3 for now), that would be regarded as manageable and acceptable
> by gnulib people.

It seems reasonable to me, but I'm not the primary author of
gnulib-tool.  Bruno?

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Proposed gnulib renames

[Eli Zaretskii]

> Date: Wed, 26 Jan 2011 08:19:13 -0700
> From: Eric Blake <eblake@redhat.com>
> CC: cyd@stupidchicken.com, eggert@cs.ucla.edu, bug-gnulib@gnu.org,
>         monnier@IRO.UMontreal.CA, emacs-devel@gnu.org,
>         Bruno Haible <bruno@clisp.org>
> 
> >  1) For c++defs.h and the lib/*.in.h files: rely on the automatic
> >     renaming by djtar.  Files that reference those will be edited by
> >     config.bat as part of configuring Emacs for the MS-DOS build.
> 
> Paul already suggested renaming c++defs.h to cxxdefs.h in gnulib, which
> makes sense to me in light of POSIX restrictions on portable filenames;
> however, this module belongs to Bruno, so it is his call.

And Bruno already voiced his objections, so I guess I will have to
live will that.  It's not a big deal to edit with Sed the few files
that refer to c++defs.h, as part of the config.bat run.  Of course, if
Bruno will change his mind, it will be even better.

> and given that the renaming from *_.h -> *.in.h was practically
> mechanical, a conversion from *.in.h -> *-in.h would likewise be
> mechanical, but I'm not sure whether to make that jump yet:
> 
> http://thread.gmane.org/gmane.comp.lib.gnulib.bugs/11182/focus=11352

I'm okay with renaming them to use underscores.  But if gnulib
maintainers object, the djtar utility will take care of these, too.
Editing Makefile.in to refer to them in config.h should not be a big
issue.

> that thread also included a quote from Eli, predicting the death of an
> emacs DOS build (for good or for bad, that prediction has not come true):

Life has its surprises.  I found some free time to work on support for
bidirectional editing in Emacs, and as a side effect, I could invest
some of that time into reviving the DOS build.

> >  2) For the 3 m4/gnulib-*.m4 files: rename them.  I suggested one way
> >     of renaming, but there's nothing sacred about that; any renaming
> >     that will get rid of the name clash is fine with me.
> 
> This would involve a change mainly to gnulib-tool (2 of the 3 gnulib-c*
> files are generated; there's also gnulib-tool.m4 to avoid clashing
> with).  Changing gnulib-cache.m4 would have downstream effects on all
> gnulib clients; it could be done with a NEWS entry, but I'd rather avoid
> that.  But the other two files (gnulib-common.m4 and gnulib-comp.m4)
> seem like fair game; how about the names gnulib-prereq.m4 (since all the
> common code is prerequisite to the rest of gnulib) and gnulib-list.m4
> (since comp contains the computed list of files/macros installed by
> gnulib-tool).

Of course, it is enough to rename only 2 of the 3, to avoid file-name
clashes.  So if your suggestion to rename gnulib-common.m4 and
gnulib-comp.m4 is accepted, that would solve the problem entirely.

Thanks!

Re: Proposed gnulib renames

[Bruno Haible]

Hi Eric, Eli,

Remember that Eli said that the entire discussion is about renaming files
in the Emacs repository, not in gnulib [1]. I also pointed out that MSDOS
support is outside of gnulib's scope [2].

On this background, for me the goal is not to do any renamings in gnulib,
but rather to help the Emacs maintainers do these renamings easily when
they pull from gnulib.


1) About c++defs.h:

I have proposed in [2] a modification to the Emacs sources that will ensure
that
  - No c++defs.h is contained in the build.
  - A gnulib-local/modules/c++defs.orig file is contained in the tarball but
    not used in the build. According to Eli [3][4] the djtar program can
    deal with such a file automatically.


2) About *.in.h include files:

Eric Blake wrote:
> I agree that use of gnulib-tool
> --local-dir won't quite work; it could easily patch the module
> descriptions (and makefile snippets) to refer to *-in.h files with
> minimal risk of too many upstream changes to track, but the rename
> itself is not possible via simple patch(1) files (yes, GNU patchutils is
> adding support for git-style rename patches, which would be much easier
> to maintain, but I don't know if we should rely on that yet).

The simplest way to deal with this is that the script for updating the
Emacs sources from gnulib
  1. runs "gnulib-tool"
  2. looks at list of lib/*.in.h files,
  3. performs some 'mv' commands to rename them,
  4. performs some sed replacements on the generated Makefile.am.
Compared to the sed processing that is done by gnulib-tool, this is really
minor.

Eli Zaretskii says in [5]:

> I'm okay with renaming them to use underscores.  But if gnulib
> maintainers object, the djtar utility will take care of these, too.
> Editing Makefile.in to refer to them in config.h should not be a big
> issue.

I agree with Eli that a bit of hand-written renaming in the script
is feasible and maintainable.


3) About the collision of gnulib-cache.m4, gnulib-comp.m4, gnulib-common.m4:

Only one of these files is in gnulib. The other two are generated, and these
files are used by two programs: 'gnulib-tool' and 'aclocal'. For
'gnulib-tool', the file names matter, but 'gnulib-tool' is only invoked
from a specific script or Makefile in the Emacs sources. For 'aclocal',
the names of the .m4 files don't matter.

So, I would propose that you change the script from

   gnulib-tool ...<many options>

to

   if test -f m4/gl-cache.m4; then mv m4/gl-cache.m4 m4/gnulib-cache.m4; done
   if test -f m4/gl-comp.m4; then mv m4/gl-comp.m4 m4/gnulib-comp.m4; done
   gnulib-tool ...<many options>
   mv m4/gnulib-cache.m4 m4/gl-cache.m4
   mv m4/gnulib-comp.m4 m4/gl-comp.m4

Eric Blake wrote:
> Changing gnulib-cache.m4 would have downstream effects on all
> gnulib clients; it could be done with a NEWS entry, but I'd rather avoid
> that.  But the other two files (gnulib-common.m4 and gnulib-comp.m4)
> seem like fair game; how about the names gnulib-prereq.m4 (since all the
> common code is prerequisite to the rest of gnulib) and gnulib-list.m4
> (since comp contains the computed list of files/macros installed by
> gnulib-tool).

It is out of question to rename things in a way that would affect all
gnulib users. If necessary, we could introduce a special option or two
in gnulib-tool, for Emacs developers to use. But even this is not needed,
given that it can be done with two 'mv' commands before and two 'mv'
commands after the invocation of gnulib-tool.


Eli, do these three proposals solve the issues? If not, what's remaining?

Bruno


[1] http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00865.html
[2] http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00779.html
[3] http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00827.html
[4] http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00866.html
[5] http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00922.html

Re: Proposed gnulib renames

[Eli Zaretskii]

> From: Bruno Haible <bruno@clisp.org>
> Date: Wed, 26 Jan 2011 18:33:20 +0100
> Cc: cyd@stupidchicken.com,
>  eggert@cs.ucla.edu,
>  bug-gnulib@gnu.org,
>  monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> 
> Remember that Eli said that the entire discussion is about renaming files
> in the Emacs repository, not in gnulib [1].

That's true.  I have no intention to force any undesired changes on
gnulib.

> 1) About c++defs.h:
> 
> I have proposed in [2] a modification to the Emacs sources that will ensure
> that
>   - No c++defs.h is contained in the build.
>   - A gnulib-local/modules/c++defs.orig file is contained in the tarball but
>     not used in the build. According to Eli [3][4] the djtar program can
>     deal with such a file automatically.

IIUC, the file gnulib-local/modules/c++defs.diff is a patch to be
applied by gnulib-tool when it reads c++defs from the gnulib
directory.  But what is the file gnulib-local/build-aux/cxxdefs.h for?

Also, the patch in gnulib-local/modules/c++defs.diff would need to be
updated from time to time, when c++defs in gnulib changes
significantly, is that right?

Finally, what is the file c++defs that will be patched by that patch?
I see no such file in Emacs at the moment.

> 2) About *.in.h include files:
> [...]
> The simplest way to deal with this is that the script for updating the
> Emacs sources from gnulib
>   1. runs "gnulib-tool"
>   2. looks at list of lib/*.in.h files,
>   3. performs some 'mv' commands to rename them,
>   4. performs some sed replacements on the generated Makefile.am.

That's fine with me, but these Sed replacements could be done by
config.bat only for the DOS build.  Of course, it's slightly more
complicated to do that with DOS shells, but it's certainly doable.

> 3) About the collision of gnulib-cache.m4, gnulib-comp.m4, gnulib-common.m4:
> [...]
> So, I would propose that you change the script from
> 
>    gnulib-tool ...<many options>
> 
> to
> 
>    if test -f m4/gl-cache.m4; then mv m4/gl-cache.m4 m4/gnulib-cache.m4; done
>    if test -f m4/gl-comp.m4; then mv m4/gl-comp.m4 m4/gnulib-comp.m4; done
>    gnulib-tool ...<many options>
>    mv m4/gnulib-cache.m4 m4/gl-cache.m4
>    mv m4/gnulib-comp.m4 m4/gl-comp.m4

That's fine with me.

> Eli, do these three proposals solve the issues?

I think so.

> If not, what's remaining?

Assuming this is acceptable to Stefan and Paul, all that's left is for
me to understand the details about which I ask above, and then do all
these changes.

Thanks a lot for taking time to describe these solutions.

Re: Proposed gnulib renames

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2584 bytes --]

On 01/26/2011 11:40 AM, Eli Zaretskii wrote:
>> I have proposed in [2] a modification to the Emacs sources that will ensure
>> that
>>   - No c++defs.h is contained in the build.
>>   - A gnulib-local/modules/c++defs.orig file is contained in the tarball but
>>     not used in the build. According to Eli [3][4] the djtar program can
>>     deal with such a file automatically.
> 
> IIUC, the file gnulib-local/modules/c++defs.diff is a patch to be
> applied by gnulib-tool when it reads c++defs from the gnulib
> directory.  But what is the file gnulib-local/build-aux/cxxdefs.h for?

The counterpart of the c++defs.diff patch that provides the contents of
the file under its desired new name.  That is, the combination of the
two patches together performs the rename of c++defs over to cxxdefs when
the module is imported into the emacs tree.

> 
> Also, the patch in gnulib-local/modules/c++defs.diff would need to be
> updated from time to time, when c++defs in gnulib changes
> significantly, is that right?

Correct, although given the stability of c++defs, that is not likely to
be very often.

> 
> Finally, what is the file c++defs that will be patched by that patch?
> I see no such file in Emacs at the moment.

It's the file gnulib/modules/c++defs that tells gnulib-module how to
import all files related to the c++defs module.  But you are changing it
to be the cxxdefs module.  Gnulib-tool uses the combination of its own
module files, as patched by your local files, to decide what to apply to
your local tree.  After gnulib-tool is complete, you no longer need any
of the gnulib/modules files stored in the emacs tree, but just the
output.  Which means that after a all is said and done, the only
remaining file with + in the name is your instructions to gnulib-tool
(not part of the normal emacs build, but necessary for whoever reruns
the gnulib synchronization script), and everything else in the emacs
tree (which is actually part of the build) has already been renamed to
cxxdefs during the gnulib-tool synchronization.

>> If not, what's remaining?
> 
> Assuming this is acceptable to Stefan and Paul, all that's left is for
> me to understand the details about which I ask above, and then do all
> these changes.

And to make sure that in the future, none of the other 8.3 conflicts
that were pointed out by Jim end up being imported into emacs during a
subsequent gnulib-tool synchronization run.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Proposed gnulib renames

[Bruno Haible]

Hi Eli,

> > 1) About c++defs.h:
> > 
> > I have proposed in [2] a modification to the Emacs sources that will ensure
> > that
> >   - No c++defs.h is contained in the build.
> >   - A gnulib-local/modules/c++defs.diff file is contained in the tarball but
> >     not used in the build. According to Eli [3][4] the djtar program can
> >     deal with such a file automatically.
> 
> IIUC, the file gnulib-local/modules/c++defs.diff is a patch to be
> applied by gnulib-tool when it reads c++defs from the gnulib
> directory.  But what is the file gnulib-local/build-aux/cxxdefs.h for?
> 
> Also, the patch in gnulib-local/modules/c++defs.diff would need to be
> updated from time to time, when c++defs in gnulib changes
> significantly, is that right?
> 
> Finally, what is the file c++defs that will be patched by that patch?
> I see no such file in Emacs at the moment.

This was an approach that meant to use gnulib-tool built-in functionality,
with no post-processing of the generated Makefile.am. The cxxdefs.h is
the replacement for c++defs.h (which would be a file with an undesired
name, needed during the build). This file and gnulib-local/modules/c++defs.diff
would indeed have to change when major changes are being done to this module,
but this is not going to happen frequently. The c++defs.diff file is a patch,
that would be stored in the Emacs repository, for a file c++defs that is
stored in the gnulib repository.

Another approach, that may be simpler to put in place, is to change the
script that invokes gnulib-tool so that it
  1) renames build-aux/c++defs.h to build-aux/cxxdefs.h
  2) post-processes the generated Makefile.am, replacing "c++defs" with
     "cxxdefs".
This approach requires post-processing, but will require less frequent
updates after changes in gnulib.

> > 2) About *.in.h include files:
> > [...]
> > The simplest way to deal with this is that the script for updating the
> > Emacs sources from gnulib
> >   1. runs "gnulib-tool"
> >   2. looks at list of lib/*.in.h files,
> >   3. performs some 'mv' commands to rename them,
> >   4. performs some sed replacements on the generated Makefile.am.
> 
> That's fine with me, but these Sed replacements could be done by
> config.bat only for the DOS build.  Of course, it's slightly more
> complicated to do that with DOS shells, but it's certainly doable.

I was under the impression that you wanted to have this renaming done
already in the tarball and, in consequence, also in the Emacs repository.
If the renaming can be limited to the DOS based checkouts, that's of course
more work for you but better for everyone else.

Bruno

Re: Proposed gnulib renames

[Eli Zaretskii]

> From: Bruno Haible <bruno@clisp.org>
> Date: Wed, 26 Jan 2011 20:01:30 +0100
> Cc: eblake@redhat.com,
>  cyd@stupidchicken.com,
>  eggert@cs.ucla.edu,
>  bug-gnulib@gnu.org,
>  monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> 
> > >   1. runs "gnulib-tool"
> > >   2. looks at list of lib/*.in.h files,
> > >   3. performs some 'mv' commands to rename them,
> > >   4. performs some sed replacements on the generated Makefile.am.
> > 
> > That's fine with me, but these Sed replacements could be done by
> > config.bat only for the DOS build.  Of course, it's slightly more
> > complicated to do that with DOS shells, but it's certainly doable.
> 
> I was under the impression that you wanted to have this renaming done
> already in the tarball and, in consequence, also in the Emacs repository.

Files whose names are FOOBAR.in.h are automatically renamed by djtar,
the unpacking utility, to something like FOOBAR.in-h (and will appear
as foobar.in- on plain DOS), so they are no problem.  What I wanted to
avoid were files whose name clash after 8+3 truncation, because these
are not renamed automatically, but need user intervention for
providing an alternative name, something end users will be stumped
about.

Re: Proposed gnulib renames

[Paul Eggert]

On 01/26/2011 09:33 AM, Bruno Haible wrote:

> I agree with Eli that a bit of hand-written renaming in the script
> is feasible and maintainable.

A bit is, yes.  However, we should strive to put the burden of
renaming-for-DOS into the DOS-related source files, and so this bit
shouldn't be precedent for much more renaming.  On the contrary, I'd
like to remove this bit once a more-general solution can be found.

>    if test -f m4/gl-cache.m4; then mv m4/gl-cache.m4 m4/gnulib-cache.m4; done
>    if test -f m4/gl-comp.m4; then mv m4/gl-comp.m4 m4/gnulib-comp.m4; done
>    gnulib-tool ...<many options>
>    mv m4/gnulib-cache.m4 m4/gl-cache.m4
>    mv m4/gnulib-comp.m4 m4/gl-comp.m4

Thanks for the suggestion.  We can simplify that part further as shown
in the patch below (which I just put into the Emacs trunk).  There's
no need to keep gnulib-cache.m4, since gnulib-tool is invoked only
from Makefile.in.

However, some extra stuff is needed as well, as we need to modify
ACLOCAL_INPUTS to reflect the MS-DOS-oriented name for the renamed
file.

=== modified file 'ChangeLog'
--- ChangeLog	2011-01-26 08:36:39 +0000
+++ ChangeLog	2011-01-27 07:26:48 +0000
@@ -1,3 +1,18 @@
+2011-01-27  Paul Eggert  <eggert@cs.ucla.edu>
+
+	fix two m4/gnulib-*.m4 file names that clashed under MS-DOS
+	* Makefile.in (DOS-gnulib-comp.m4): New macro.
+	(sync-from-gnulib): Rename m4/gnulib-comp.m4 to m4/gl-comp.m4 to avoid
+	problems with MS-DOS 8+3 file name restrictions.
+	Remove m4/gnulib-cache.m4, as we can live without it.  If we kept
+	it, it would also cause problems when extracting Emacs distribution
+	tarballs on MS-DOS hosts.
+	(ACLOCAL_INPUTS): Adjust to file renaming.
+	* aclocal.m4, configure, lib/Makefile.in, src/config.in: Regenerate.
+	* config.guess, config.sub: Sync from gnulib.
+	* m4/gnulib-cache.m4: Remove from repository.
+	* m4/gl-comp.m4: Rename from m4/gnulib-comp.m4.
+
 2011-01-25  Glenn Morris  <rgm@gnu.org>
 
 	* README: Add a note about ranges in copyright years.

=== modified file 'Makefile.in'
--- Makefile.in	2011-01-26 08:36:39 +0000
+++ Makefile.in	2011-01-27 07:26:48 +0000
@@ -324,6 +324,9 @@
 $(gnulib_srcdir):
 	git clone git://git.savannah.gnu.org/gnulib.git $@
 
+# A shorter name that satisfies MS-DOS 8+3 constraints.
+DOS-gnulib-comp.m4 = gl-comp.m4
+
 # Update modules from gnulib, for maintainers, who should have it in
 # $(gnulib_srcdir) (relative to $(srcdir) and should have build tools
 # as per $(gnulib_srcdir)/DEPENDENCIES.
@@ -333,7 +336,8 @@
 sync-from-gnulib: $(gnulib_srcdir)
 	cd $(srcdir) && \
 	  $(gnulib_srcdir)/gnulib-tool $(GNULIB_TOOL_FLAGS) $(GNULIB_MODULES)
-	rm $(srcdir)/m4/warn-on-use.m4
+	cd $(srcdir)/m4 && rm gnulib-cache.m4 warn-on-use.m4
+	cd $(srcdir)/m4 && mv gnulib-comp.m4 $(DOS-gnulib-comp.m4)
 	cp $(gnulib_srcdir)/build-aux/texinfo.tex $(srcdir)/doc/misc
 	cp \
 	  $(gnulib_srcdir)/build-aux/config.sub \
@@ -406,7 +410,7 @@
 $(srcdir)/configure: $(AUTOCONF_INPUTS)
 	cd ${srcdir} && autoconf
 
-ACLOCAL_INPUTS = @MAINT@ $(srcdir)/m4/gnulib-comp.m4
+ACLOCAL_INPUTS = @MAINT@ $(srcdir)/m4/$(DOS-gnulib-comp.m4)
 $(srcdir)/aclocal.m4: $(ACLOCAL_INPUTS)
 	cd $(srcdir) && aclocal -I m4
 

[The remaining part of the patch is automatically generated.]

Re: Proposed gnulib renames

[Eli Zaretskii]

> Date: Wed, 26 Jan 2011 23:37:29 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, Eric Blake <eblake@redhat.com>, 
>  cyd@stupidchicken.com, bug-gnulib@gnu.org, monnier@iro.umontreal.ca, 
>  emacs-devel@gnu.org
> 
> @@ -333,7 +336,8 @@
>  sync-from-gnulib: $(gnulib_srcdir)
>  	cd $(srcdir) && \
>  	  $(gnulib_srcdir)/gnulib-tool $(GNULIB_TOOL_FLAGS) $(GNULIB_MODULES)
> -	rm $(srcdir)/m4/warn-on-use.m4
> +	cd $(srcdir)/m4 && rm gnulib-cache.m4 warn-on-use.m4
> +	cd $(srcdir)/m4 && mv gnulib-comp.m4 $(DOS-gnulib-comp.m4)
>  	cp $(gnulib_srcdir)/build-aux/texinfo.tex $(srcdir)/doc/misc
>  	cp \
>  	  $(gnulib_srcdir)/build-aux/config.sub \

Don't we need the opposite "mv" before running gnulib-tool?  Apologies
if this is a dumb question, I know almost nothing about gnulib.

Re: Proposed gnulib renames

[Eli Zaretskii]

> Date: Wed, 26 Jan 2011 23:37:29 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, Eric Blake <eblake@redhat.com>, 
>  cyd@stupidchicken.com, bug-gnulib@gnu.org, monnier@iro.umontreal.ca, 
>  emacs-devel@gnu.org
> 
> Thanks for the suggestion.  We can simplify that part further as shown
> in the patch below (which I just put into the Emacs trunk).

Thank you.

Re: Proposed gnulib renames

[Bruno Haible]

Hi Paul,

Paul Eggert wrote:
> +       * Makefile.in (DOS-gnulib-comp.m4): New macro.

A Makefile macro with a minus sign in its name?! This is not portable
according to POSIX [1]:
  "Applications shall select macro names from the set of characters
   consisting solely of periods, underscores, digits, and alphabetics
   from the portable character set"
Likewise text is also in the autoconf documentation [2].

Bruno

[1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html
[2] http://www.gnu.org/software/autoconf/manual/html_node/Special-Chars-in-Names.html

Re: Proposed gnulib renames

[Paul Eggert]

On 01/27/2011 01:59 AM, Bruno Haible wrote:

>> +       * Makefile.in (DOS-gnulib-comp.m4): New macro.
> 
> A Makefile macro with a minus sign in its name?! This is not portable
> according to POSIX [1]:

Thanks for mentioning that.  Other Emacs makefiles had it, so I naively
assumed it was OK; but you're right, we shouldn't be using unportable
Makefile identifiers.  After fixing the above problem in my previous patch
I fixed the previously-existing identifiers, as follows.

This will most likely affect the MS-DOS build.

=== modified file 'etc/ChangeLog'
--- etc/ChangeLog	2011-01-26 08:36:39 +0000
+++ etc/ChangeLog	2011-01-28 01:52:26 +0000
@@ -1,3 +1,15 @@
+2011-01-28  Paul Eggert  <eggert@cs.ucla.edu>
+
+	Redo spelling of Makefile variables to conform to POSIX.
+	POSIX does not allow "-" in Makefile variable names.
+	Reported by Bruno Haible in
+	<http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00990.html>.
+	* refcards/Makefile (DIRED_REFCARDS_PDF): Renamed from
+	DIRED-REFCARDS_PDF.
+	(MISC_REFCARDS_PDF): Renamed from MISC-REFCARDS_PDF.
+	(SURVIVAL_CARDS_PDF): Renamed from SURVIVAL-CARDS_PDF.
+	(VIPER_CARDS_PDF): Renamed from VIPER-CARDS_PDF.
+
 2011-01-18  Glenn Morris  <rgm@gnu.org>
 
 	* PROBLEMS: Add note about svn+ssh.  (Bug#7791)

=== modified file 'etc/refcards/Makefile'
--- etc/refcards/Makefile	2011-01-25 04:08:28 +0000
+++ etc/refcards/Makefile	2011-01-28 01:52:26 +0000
@@ -28,24 +28,24 @@
 refcards_pdf: ${REFCARDS_PDF}
 refcards_ps: ${REFCARDS_PDF:.pdf=.ps}
 
-DIRED-REFCARDS_PDF = dired-ref.pdf cs-dired-ref.pdf fr-dired-ref.pdf \
+DIRED_REFCARDS_PDF = dired-ref.pdf cs-dired-ref.pdf fr-dired-ref.pdf \
 		sk-dired-ref.pdf
-dired-refcards_pdf: ${DIRED-REFCARDS_PDF}
-dired-refcards_ps: ${DIRED-REFCARDS_PDF:.pdf=.ps}
+dired-refcards_pdf: ${DIRED_REFCARDS_PDF}
+dired-refcards_ps: ${DIRED_REFCARDS_PDF:.pdf=.ps}
 
-MISC-REFCARDS_PDF = calccard.pdf gnus-booklet.pdf gnus-refcard.pdf orgcard.pdf
-misc-refcards_pdf: ${MISC-REFCARDS_PDF}
-misc-refcards_ps: ${MISC-REFCARDS_PDF:.pdf=.ps}
+MISC_REFCARDS_PDF = calccard.pdf gnus-booklet.pdf gnus-refcard.pdf orgcard.pdf
+misc-refcards_pdf: ${MISC_REFCARDS_PDF}
+misc-refcards_ps: ${MISC_REFCARDS_PDF:.pdf=.ps}
 
 
 ## The following files are not included with Emacs.
-SURVIVAL-CARDS_PDF = survival.pdf cs-survival.pdf sk-survival.pdf
-survival-cards_pdf: ${SURVIVAL-CARDS_PDF}
-survival-cards_ps: ${SURVIVAL-CARDS_PDF:.pdf=.ps}
+SURVIVAL_CARDS_PDF = survival.pdf cs-survival.pdf sk-survival.pdf
+survival-cards_pdf: ${SURVIVAL_CARDS_PDF}
+survival-cards_ps: ${SURVIVAL_CARDS_PDF:.pdf=.ps}
 
-VIPER-CARDS_PDF = vipcard.pdf viperCard.pdf
-viper-cards_pdf: ${VIPER-CARDS_PDF}
-viper-cards_ps: ${VIPER-CARDS_PDF:.pdf=.ps}
+VIPER_CARDS_PDF = vipcard.pdf viperCard.pdf
+viper-cards_pdf: ${VIPER_CARDS_PDF}
+viper-cards_ps: ${VIPER_CARDS_PDF:.pdf=.ps}
 
 
 ## PDF files.
@@ -74,7 +74,7 @@
 #gnus-logo.pdf: %.pdf: %.eps
 #	ps2pdf $<
 
-gnus-refcard.pdf: %.pdf: %.tex gnus-logo.pdf 
+gnus-refcard.pdf: %.pdf: %.tex gnus-logo.pdf
 	pdflatex $<
 
 gnus-booklet.pdf: gnus-refcard.tex gnus-logo.pdf

=== modified file 'leim/ChangeLog'
--- leim/ChangeLog	2011-01-25 04:08:28 +0000
+++ leim/ChangeLog	2011-01-28 01:52:26 +0000
@@ -1,3 +1,22 @@
+2011-01-28  Paul Eggert  <eggert@cs.ucla.edu>
+
+	Redo spelling of Makefile variables to conform to POSIX.
+	POSIX does not allow "-" in Makefile variable names.
+	Reported by Bruno Haible in
+	<http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00990.html>.
+	* Makefile.in (BUILT_EMACS): Renamed from BUILT-EMACS.
+	(TIT_GB): Renamed from TIT-GB.
+	(CHINESE_TIT): Renamed from CHINESE-TIT.
+	(NON_TIT_GB): Renamed from NON-TIT-GB.
+	(NON_TIT_BIG5): Renamed from NON-TIT-BIG5.
+	(CHINESE_NON_TIT): Renamed from CHINESE-NON-TIT.
+	(CHINESE_GB): Renamed from CHINESE-GB.
+	(CHINESE_BIG5): Renamed from CHINESE-BIG5.
+	(TIT_MISC): Renamed from TIT-MISC.
+	(NON_TIT_MISC): Renamed from NON-TIT-MISC.
+	(TIT_SOURCES): Renamed from TIT-SOURCES.
+	(MISC_SOURCES): Renamed from MISC-SOURCES.
+
 2011-01-08  Glenn Morris  <rgm@gnu.org>
 
 	* makefile.w32-in (RUN_EMACS):

=== modified file 'leim/Makefile.in'
--- leim/Makefile.in	2011-01-26 08:36:39 +0000
+++ leim/Makefile.in	2011-01-28 01:52:26 +0000
@@ -41,20 +41,20 @@
 
 # Which Emacs to use to convert TIT files to Emacs Lisp files,
 # byte-compile Emacs Lisp files, and generate the file leim-list.el.
-BUILT-EMACS = ../src/emacs
+BUILT_EMACS = ../src/emacs
 
 buildlisppath=${srcdir}/../lisp
 
 # How to run Emacs.
-RUN-EMACS = EMACSLOADPATH=$(buildlisppath) LC_ALL=C \
-	${BUILT-EMACS} -batch --no-site-file --no-site-lisp
+RUN_EMACS = EMACSLOADPATH=$(buildlisppath) LC_ALL=C \
+	${BUILT_EMACS} -batch --no-site-file --no-site-lisp
 
 # Subdirectories to be made if ${srcdir} is different from the current
 # directory.
 SUBDIRS=quail
 
 # Files generated from TIT dictionaries for Chinese GB character set.
-TIT-GB=\
+TIT_GB=\
 	quail/CCDOSPY.elc	\
 	quail/Punct.elc		\
 	quail/QJ.elc		\
@@ -72,17 +72,17 @@
 	quail/QJ-b5.elc		\
 	quail/ZOZY.elc
 
-CHINESE-TIT=${TIT-GB} ${TIT-BIG5}
-
-NON-TIT-GB=${srcdir}/quail/py-punct.elc
-
-NON-TIT-BIG5=${srcdir}/quail/pypunct-b5.elc
-
-CHINESE-NON-TIT=${NON-TIT-GB} ${NON-TIT-BIG5}
-
-CHINESE-GB=${TIT-GB} ${NON-TIT-GB}
-
-CHINESE-BIG5=${TIT-BIG5} ${NON-TIT-BIG5}
+CHINESE_TIT=${TIT_GB} ${TIT_BIG5}
+
+NON_TIT_GB=${srcdir}/quail/py-punct.elc
+
+NON_TIT_BIG5=${srcdir}/quail/pypunct-b5.elc
+
+CHINESE_NON_TIT=${NON_TIT_GB} ${NON_TIT_BIG5}
+
+CHINESE_GB=${TIT_GB} ${NON_TIT_GB}
+
+CHINESE_BIG5=${TIT_BIG5} ${NON_TIT_BIG5}
 
 JAPANESE=${srcdir}/quail/japanese.elc ${srcdir}/ja-dic/ja-dic.elc
 
@@ -138,22 +138,22 @@
 	quail/CTLau.elc		\
 	quail/CTLau-b5.elc
 
-CHINESE=${CHINESE-GB} ${CHINESE-BIG5}
+CHINESE=${CHINESE_GB} ${CHINESE_BIG5}
 EASTASIA=${CHINESE} ${JAPANESE} ${KOREAN}
 ASIA=${EASTASIA} ${THAI} ${VIETNAMESE} ${LAO} ${INDIAN} ${TIBETAN}
 EUROPEAN=${LATIN} ${SLAVIC} ${GREEK} ${RUSSIAN}
 WORLD=${ASIA} ${EUROPEAN} ${OTHERS} ${MISC} ${UNICODE}
 
-TIT-MISC=${CHINESE-TIT} ${MISC}
-NON-TIT-MISC=${CHINESE-NON-TIT} ${JAPANESE} ${KOREAN} ${EUROPEAN} ${OTHERS}
+TIT_MISC=${CHINESE_TIT} ${MISC}
+NON_TIT_MISC=${CHINESE_NON_TIT} ${JAPANESE} ${KOREAN} ${EUROPEAN} ${OTHERS}
 
 .SUFFIXES: .elc .el
 
 .el.elc:
 	@echo Compiling $<
-	@${RUN-EMACS} -f batch-byte-compile $<
+	@${RUN_EMACS} -f batch-byte-compile $<
 
-all: ${BUILT-EMACS} ${SUBDIRS} leim-list.el ${WORLD}
+all: ${BUILT_EMACS} ${SUBDIRS} leim-list.el ${WORLD}
 
 # To ensure that we can run Emacs.  This target is ignored (never
 # being hit) if a user changes default value of EMACS.
@@ -164,7 +164,7 @@
 	mkdir $@
 	touch stamp-subdir
 
-TIT-SOURCES= \
+TIT_SOURCES= \
 	${srcdir}/CXTERM-DIC/4Corner.tit \
 	${srcdir}/CXTERM-DIC/ARRAY30.tit \
 	${srcdir}/CXTERM-DIC/CCDOSPY.tit \
@@ -179,15 +179,15 @@
 	${srcdir}/CXTERM-DIC/TONEPY.tit \
 	${srcdir}/CXTERM-DIC/ZOZY.tit
 
-${CHINESE-TIT:.elc=.el}: changed.tit
+${CHINESE_TIT:.elc=.el}: changed.tit
 	@true
 
-changed.tit: ${TIT-SOURCES}
-	${RUN-EMACS} -l ${buildlisppath}/international/titdic-cnv \
+changed.tit: ${TIT_SOURCES}
+	${RUN_EMACS} -l ${buildlisppath}/international/titdic-cnv \
 	  -f batch-titdic-convert -dir quail ${srcdir}/CXTERM-DIC; \
 	  echo "changed" > $@
 
-MISC-SOURCES= \
+MISC_SOURCES= \
 	${srcdir}/MISC-DIC/CTLau-b5.html \
 	${srcdir}/MISC-DIC/CTLau.html \
 	${srcdir}/MISC-DIC/cangjie-table.b5 \
@@ -198,20 +198,20 @@
 ${MISC:.elc=.el}: changed.misc
 	@true
 
-changed.misc: ${MISC-SOURCES}
-	${RUN-EMACS} -l ${buildlisppath}/international/titdic-cnv \
+changed.misc: ${MISC_SOURCES}
+	${RUN_EMACS} -l ${buildlisppath}/international/titdic-cnv \
 	  -f batch-miscdic-convert -dir quail ${srcdir}/MISC-DIC; \
 	  echo "changed" > $@
 
-leim-list.el: ${SUBDIRS} ${TIT-MISC} changed.tit changed.misc ${srcdir}/leim-ext.el
+leim-list.el: ${SUBDIRS} ${TIT_MISC} changed.tit changed.misc ${srcdir}/leim-ext.el
 	rm -f leim-list.el
-	${RUN-EMACS}  -l ${buildlisppath}/international/quail \
-	  -f batch-byte-compile-if-not-done ${TIT-MISC:.elc=.el}
+	${RUN_EMACS}  -l ${buildlisppath}/international/quail \
+	  -f batch-byte-compile-if-not-done ${TIT_MISC:.elc=.el}
 	if [ x`(cd ${srcdir} && /bin/pwd)` = x`(/bin/pwd)` ] ; then \
-	  ${RUN-EMACS} -l ${buildlisppath}/international/quail \
+	  ${RUN_EMACS} -l ${buildlisppath}/international/quail \
 	    --eval "(update-leim-list-file \".\")" ; \
 	else \
-	  ${RUN-EMACS} -l ${buildlisppath}/international/quail \
+	  ${RUN_EMACS} -l ${buildlisppath}/international/quail \
 	    --eval "(update-leim-list-file \".\" \"${srcdir}\")" ; \
 	fi
 	sed -n '/^[^;]/ p' < ${srcdir}/leim-ext.el >> $@
@@ -264,7 +264,7 @@
 	else true ; fi
 
 clean mostlyclean:
-	rm -f ${TIT-MISC} ${TIT-MISC:.elc=.el} \
+	rm -f ${TIT_MISC} ${TIT_MISC:.elc=.el} \
 		leim-list.el changed.tit changed.misc
 
 # The following target is needed because the `clean' target only removes
@@ -287,5 +287,5 @@
 .PHONY: check-declare
 
 check-declare:
-	$(RUN-EMACS) -l $(buildlisppath)/emacs-lisp/check-declare \
+	$(RUN_EMACS) -l $(buildlisppath)/emacs-lisp/check-declare \
 	  --eval '(check-declare-directory "$(srcdir)")'

Re: Proposed gnulib renames

[Bruno Haible]

Paul Eggert wrote:
> There's no need to keep gnulib-cache.m4, since gnulib-tool is invoked only
> from Makefile.in.

I disagree. The gnulib documentation ([1], paragraph 1) recommends to put also
gnulib-cache.m4 into the repository.

The reason is that when a file is renamed in gnulib or some module dependency
is removed in gnulib, what happens at the next invocation of sync-from-gnulib?
  - When gnulib-cache.m4 is present, gnulib-tool will add a file under the
    new name and remove the old file. Because it knows that the file came from
    gnulib.
  - When gnulib-cache.m4 is missing, gnulib-tool will just add a file under
    the new name, but leave the old file around, because it looks like that
    file was genuine Emacs source.

So, if you routinely remove gnulib-cache.m4, over time the repository will
accumulate garbage files. Sometimes they don't hurt (if it's just a .c file),
but it can really get in the way and cause trouble (if it's a .h file).

Bruno

[1] http://www.gnu.org/software/gnulib/manual/html_node/VCS-Issues.html

Re: Proposed gnulib renames

[Bruno Haible]

> I disagree. The gnulib documentation ([1], paragraph 1) recommends to put also
> gnulib-cache.m4 into the repository.
> 
> The reason is that when a file is renamed in gnulib or some module dependency
> is removed in gnulib, what happens at the next invocation of sync-from-gnulib?

Oops, I misspoke. The gnulib documentation does recommend to put
gnulib-cache.m4 into the repository, but this is only a convenience for
someone who wants to use the --add-import or --remove-import options
and therefore not really necessary. The file that needs to be kept so
that old files don't accumulate as garbage is gnulib-comp.m4.

Sorry for the confusion.

Bruno

Re: Proposed gnulib renames

[Paul Eggert]

On 01/27/11 02:23, Bruno Haible wrote:
> The file that needs to be kept so
> that old files don't accumulate as garbage is gnulib-comp.m4.

Thanks, I installed this into the Emacs trunk.  I was leery about
using 'mv' as the first step, because what if 'make' is interrupted
right after the 'mv'? gl-comp.m4 will be missing and a later
'make' will fail because aclocal.m4 can't be built.  So I instead
used 'cp', as follows.

=== modified file 'ChangeLog'
--- ChangeLog	2011-01-27 07:24:57 +0000
+++ ChangeLog	2011-01-28 00:25:24 +0000
@@ -1,3 +1,12 @@
+2011-01-28  Paul Eggert  <eggert@cs.ucla.edu>
+
+	improve fix for MS-DOS file name clash
+	* Makefile.in (DOS_gnulib_comp.m4): Renamed from DOS-gnulib-comp.m4,
+	for portability to POSIX make.  Reported by Bruno Haible.
+	(sync-from-gnulib): Copy gl-comp.m4 (if present) back to
+	gnulib-comp.m4 before running gnulib-tool, to prevent old gnulib
+	files from accumulating as garbage.  Also reported by Bruno Haible.
+
 2011-01-27  Paul Eggert  <eggert@cs.ucla.edu>
 
 	fix two m4/gnulib-*.m4 file names that clashed under MS-DOS

=== modified file 'Makefile.in'
--- Makefile.in	2011-01-27 07:24:57 +0000
+++ Makefile.in	2011-01-28 00:25:24 +0000
@@ -325,7 +325,7 @@
 	git clone git://git.savannah.gnu.org/gnulib.git $@
 
 # A shorter name that satisfies MS-DOS 8+3 constraints.
-DOS-gnulib-comp.m4 = gl-comp.m4
+DOS_gnulib_comp.m4 = gl-comp.m4
 
 # Update modules from gnulib, for maintainers, who should have it in
 # $(gnulib_srcdir) (relative to $(srcdir) and should have build tools
@@ -334,10 +334,11 @@
 GNULIB_TOOL_FLAGS = \
  --import --no-changelog --no-vc-files --makefile-name=gnulib.mk
 sync-from-gnulib: $(gnulib_srcdir)
+	-cd $(srcdir)/m4 && cp $(DOS_gnulib_comp.m4) gnulib-comp.m4
 	cd $(srcdir) && \
 	  $(gnulib_srcdir)/gnulib-tool $(GNULIB_TOOL_FLAGS) $(GNULIB_MODULES)
 	cd $(srcdir)/m4 && rm gnulib-cache.m4 warn-on-use.m4
-	cd $(srcdir)/m4 && mv gnulib-comp.m4 $(DOS-gnulib-comp.m4)
+	cd $(srcdir)/m4 && mv gnulib-comp.m4 $(DOS_gnulib_comp.m4)
 	cp $(gnulib_srcdir)/build-aux/texinfo.tex $(srcdir)/doc/misc
 	cp \
 	  $(gnulib_srcdir)/build-aux/config.sub \
@@ -410,7 +411,7 @@
 $(srcdir)/configure: $(AUTOCONF_INPUTS)
 	cd ${srcdir} && autoconf
 
-ACLOCAL_INPUTS = @MAINT@ $(srcdir)/m4/$(DOS-gnulib-comp.m4)
+ACLOCAL_INPUTS = @MAINT@ $(srcdir)/m4/$(DOS_gnulib_comp.m4)
 $(srcdir)/aclocal.m4: $(ACLOCAL_INPUTS)
 	cd $(srcdir) && aclocal -I m4
 

Re: Files from gnulib

[Stefan Monnier]

> Here's a compromise that at least I can live with; hope others can, too:

>  1) For c++defs.h and the lib/*.in.h files: rely on the automatic
>     renaming by djtar.  Files that reference those will be edited by
>     config.bat as part of configuring Emacs for the MS-DOS build.

>  2) For the 3 m4/gnulib-*.m4 files: rename them.  I suggested one way
>     of renaming, but there's nothing sacred about that; any renaming
>     that will get rid of the name clash is fine with me.

Sounds good to me.


        Stefan

Re: Files from gnulib

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: eggert@cs.ucla.edu, cyd@stupidchicken.com, bug-gnulib@gnu.org,
>         emacs-devel@gnu.org
> Date: Tue, 25 Jan 2011 17:15:37 -0500
> 
> >   . Once the remapping is maintained by humans, it becomes unreliable.
> 
> Why is that?

Because people err and don't always pay attention to all changes that
affect this issue.

> >   . The unpacking instructions are part of the tarball, and need to be
> >     extracted separately before the "main" extraction begins.  More
> >     importantly, the remapping file needs to be extracted before that
> >     as well.  This makes the entire unpacking procedure extremely
> >     complicated and thus error-prone, except if the person who does
> >     that is the one who designed it and wrote the instructions in the
> >     first place.
> 
> Rather than distribute a file that needs to be passed to djtar, I was
> thinking of distributing a script tailored to MS-DOS, run instead of
> djtar, and which would run djtar insternally.  So this script can
> provide the instructions.

How will that script be distributed?

> > In addition, there will be a need to deal with something that the GDB
> > distribution doesn't.  In GDB, the files that are renamed during
> > unpacking are only those that are not used for the DOS build.  By
> > contrast, here we want to rename files that are used during the build.
> 
> Yes, that's a much bigger problem.

There's more to it.  If the affected files are mentioned in some Lisp
files, then those Lisp files will need to be recompiled after being
edited.  That means the Emacs build process on MSDOS will have to run
Make in the lips/ subdirectory, which until now it didn't.
lisp/Makefile requires a Unixy shell (and associated utilities, like
Coreutils), which would be now another prerequisite for building
Emacs.

I'm sure we will discover more difficulties, because even GDB didn't
rename files used in building on DOS.

> In the mean time, please rename the files, thank you,

Thank you.

Re: Files from gnulib

[Jason Rumney]

On 26/01/2011 05:03, Stefan Monnier wrote:
> Here's a compromise:
> - let's do the renames that need to be done (they're really not
>    problematic, AFAICT).
> - let's try and get this "djtar -n emacs-25.chg emacs-25.tgz" to work
>    reliably, e.g. by auto-building the emacs-25.chg file as part of
>    make-dist, so there's no way for that file to be "missing renames".
>    

It might also be a good idea to add an alternate-load-library-alist 
variable which require and load could consult when attempts to load a 
file fail.  That would also be useful on other platforms for the CEDET 
rename and similar cases.

Re: Files from gnulib

[Stefan Monnier]

>> Here's a compromise:
>> - let's do the renames that need to be done (they're really not
>> problematic, AFAICT).
>> - let's try and get this "djtar -n emacs-25.chg emacs-25.tgz" to work
>> reliably, e.g. by auto-building the emacs-25.chg file as part of
>> make-dist, so there's no way for that file to be "missing renames".
> It might also be a good idea to add an alternate-load-library-alist variable
> which require and load could consult when attempts to load a file fail.
> That would also be useful on other platforms for the CEDET rename and
> similar cases.

Indeed.  This could even be auto-filled by autoload.el if we add support
in it for entries like:

   ;;;###autoload
   (provide 'foo)


-- Stefan

Re: Files from gnulib

[Paul Eggert]

On 01/25/11 11:02, Eli Zaretskii wrote:

> To read the instructions, you need to unpack the archive first.

That may have been true years ago, when the tarballs themselves were
the main way that one could find out how to do maintenance.  But that
long ago stopped being true for Emacs.  If I wanted to come up to
speed on how to build Emacs for MS-DOS, the first thing I'd do would
be a Google search, which would point me at places like
<http://www.emacswiki.org/emacs-fr/EmacsForDOS> and
<http://www.gnu.org/software/emacs/manual/html_node/emacs/MS_002dDOS.html>.
If these places contain extraction instructions, that's good enough.


>>    djtar -n emacs-25.chg emacs-25.tgz

> We use something like that in GDB, and the result is extremely
> fragile and error-prone,

Even if -n is currently error-prone in GDB, that does not mean
that the approach is inherently error-prone, or that it must be
error-prone in Emacs.  For example, it should be pretty easy to check
emacs-25.chg automatically; is that done with GDB?  If not, and if
checking is done by hand, I can understand why it might be error-prone;
but an automated check should substantially reduce the number
of errors.

> If the decision is not to rename these few files in the Emacs
> distribution, and instead ask me to cope with these complications, I
> will understand that the knee-jerk reaction of too many members of
> this community when they hear "MS-DOS" is more important that any
> voice of reason

I hope that you don't include me in members whose knees are jerking.
Personally I would just rename the files in gnulib and be done with
it, as none of the name changes seem to be onerous.  However, we don't
seem to have consensus for that now; I seem to be the only gnulib
developer who would go that route.  Also, the problem of non-8+3 file
names does not seem to be limited to gnulib-derived files.

All in all it sounds like automating the renaming on the MS-DOS side
would be a reasonable thing to do.  This is a bit of work but doesn't
seem that hard.  And if we get the automation working well with Emacs
we could then apply similar ideas to GDB as well, and make GDB
development less error-prone on MS-DOS.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 13:24:11 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: cyd@stupidchicken.com, bug-gnulib@gnu.org, monnier@iro.umontreal.ca, 
>  emacs-devel@gnu.org
> 
> On 01/25/11 11:02, Eli Zaretskii wrote:
> 
> > To read the instructions, you need to unpack the archive first.
> 
> That may have been true years ago, when the tarballs themselves were
> the main way that one could find out how to do maintenance.  But that
> long ago stopped being true for Emacs.  If I wanted to come up to
> speed on how to build Emacs for MS-DOS, the first thing I'd do would
> be a Google search, which would point me at places like
> <http://www.emacswiki.org/emacs-fr/EmacsForDOS> and
> <http://www.gnu.org/software/emacs/manual/html_node/emacs/MS_002dDOS.html>.
> If these places contain extraction instructions, that's good enough.

And what about the emacs-25.chg file?  Would you expect users to
google for it as well, and copy-paste it into their shell window?  I
hope you will agree that it's a very bad idea.  So the file will have
to be extracted first, and you are back at the same problem as with
the instructions again.

> For example, it should be pretty easy to check emacs-25.chg
> automatically; is that done with GDB?

Yes, it is done.  But it doesn't catch all the errors.  More
importantly, the remapping file is maintained manually.  See my
response to Stefan.

> > If the decision is not to rename these few files in the Emacs
> > distribution, and instead ask me to cope with these complications, I
> > will understand that the knee-jerk reaction of too many members of
> > this community when they hear "MS-DOS" is more important that any
> > voice of reason
> 
> I hope that you don't include me in members whose knees are jerking.

I no longer know who is and who isn't.  Oscar's was the only message
that sounded like a glimpse of light in the darkness.

> Personally I would just rename the files in gnulib and be done with
> it, as none of the name changes seem to be onerous.  However, we don't
> seem to have consensus for that now; I seem to be the only gnulib
> developer who would go that route.

We are talking about renaming files in the Emacs repo.  Why would
gnulib developers have any say in that?

> Also, the problem of non-8+3 file names does not seem to be limited
> to gnulib-derived files.

Yes, they are limited to gnulib-derived files.  If you mean Org, I'm
sure those files will be renamed.

> All in all it sounds like automating the renaming on the MS-DOS side
> would be a reasonable thing to do.

Theoretically, yes.  It sounded like that years ago, when it was
introduced into GDB.  I feel much better now, thank you.

> This is a bit of work but doesn't seem that hard.  And if we get the
> automation working well with Emacs we could then apply similar ideas
> to GDB as well, and make GDB development less error-prone on MS-DOS.

Who is "we" here, I wonder.

Re: Files from gnulib

[Paul Eggert]

On 01/25/11 13:54, Eli Zaretskii wrote:

>     This makes the entire unpacking procedure extremely
>     complicated and thus error-prone

I don't see why.  With GDB it's two commands:

  djtar -x -p -o gdb-7.2/djunpack.bat gdb-7.2.tar.gz > djunpack.bat
  djunpack gdb-7.2.tar.gz

Why would it be more complicated than that for Emacs?

>   . Once the remapping is maintained by humans, it becomes unreliable.

No, not if it's checked.  Surely the check can be automated reliably.

> the subdirectories of the Emacs tree will have to be hard-coded in
> config.bat, which is yet another source of errors, when a
> subdirectory is added or removed.  (The alternative, of using a port
> of GNU Find, would mean addition of another tool that is not
> required today for building Emacs.)

That should not be a problem.  'find' is already required to build
Emacs; for example, Makefile.in uses it.  It is easy to run 'find'
as part of the process that makes a distribution, and to put its
output into config.bat or the equivalent, so there is no need to run
'find' under MS-DOS.

> Perhaps it's possible to solve all this in a satisfactory manner, but
> doing so would require a lot of work,

I don't think it'd take much work to do the above.  I can write
scripts to do the check and to do the find, since they can all be done
on a standard GNU platform.  What else is needed?

On 01/25/11 14:06, Eli Zaretskii wrote:

> And what about the emacs-25.chg file?  Would you expect users to
> google for it as well, and copy-paste it into their shell window?

No, I would expect users to extract it from the tarball much as
is already done with GDB and djunpack.bat.  That's simple, and it
would work.

>> For example, it should be pretty easy to check emacs-25.chg
>> automatically; is that done with GDB?
> 
> Yes, it is done.  But it doesn't catch all the errors.

How is that checking done, and what errors doesn't it catch?
I don't see the checking in the GDB 7.2 distribution.

> We are talking about renaming files in the Emacs repo.  Why would
> gnulib developers have any say in that?

Because we automatically sync files from gnulib into Emacs.
What I think you are suggesting, is that we rename gnulib files,
and edit their contents, after copying them from gnulib into Emacs.
But this will break the existing "make sync-from-gnulib", or require
"make sync-from-gnulib" to be considerably more complicated.

The renaming and copying is needed only on MS-DOS; it's not needed
for any other platform.  It makes sense to do it only on MS-DOS.
This will simplify maintenance on non-MS-DOS platforms; for example,
it will make it easier to propagate fixes from Emacs back to gnulib.

>> Also, the problem of non-8+3 file names does not seem to be limited
>> to gnulib-derived files.
> 
> Yes, they are limited to gnulib-derived files.  If you mean Org, I'm
> sure those files will be renamed.

I meant all the other files that have 8+3 issues.  These are all
problems, even if the solutions differ for different cases.

> Who is "we" here, I wonder.

I can write the above scripts for Emacs.  The ideas can be propagated
into GDB later, as needed.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 16:54:16 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: cyd@stupidchicken.com, bug-gnulib@gnu.org, monnier@iro.umontreal.ca, 
>  emacs-devel@gnu.org
> 
> On 01/25/11 13:54, Eli Zaretskii wrote:
> 
> >     This makes the entire unpacking procedure extremely
> >     complicated and thus error-prone
> 
> I don't see why.  With GDB it's two commands:
> 
>   djtar -x -p -o gdb-7.2/djunpack.bat gdb-7.2.tar.gz > djunpack.bat
>   djunpack gdb-7.2.tar.gz
> 
> Why would it be more complicated than that for Emacs?

This is the complexity I want to avoid.  Don't you think it's
complicated enough?  And how about the issue with using slashes in the
argument to djunpack? does that kind of subtlety mean something to
you?

> >   . Once the remapping is maintained by humans, it becomes unreliable.
> 
> No, not if it's checked.  Surely the check can be automated reliably.

The reliability of this is determined by the scripts that do it.
Scripts are written by people, who tend to err or miss something.

> > the subdirectories of the Emacs tree will have to be hard-coded in
> > config.bat, which is yet another source of errors, when a
> > subdirectory is added or removed.  (The alternative, of using a port
> > of GNU Find, would mean addition of another tool that is not
> > required today for building Emacs.)
> 
> That should not be a problem.  'find' is already required to build
> Emacs; for example, Makefile.in uses it.

Only lisp/Makefile.in, which is not used when a release is built on
DOS (all the files are already compiled).

> It is easy to run 'find' as part of the process that makes a
> distribution, and to put its output into config.bat or the
> equivalent, so there is no need to run 'find' under MS-DOS.

More complications.  This means, for example, that to test an
arbitrary revision of the development tree, I will need to run
make-dist on Unix, create a tarball, copy it to a DOS machine, then
build, find problems, go back to the Unix machine, etc.

> > Perhaps it's possible to solve all this in a satisfactory manner, but
> > doing so would require a lot of work,
> 
> I don't think it'd take much work to do the above.  I can write
> scripts to do the check and to do the find, since they can all be done
> on a standard GNU platform.  What else is needed?

Maintenance.

> > And what about the emacs-25.chg file?  Would you expect users to
> > google for it as well, and copy-paste it into their shell window?
> 
> No, I would expect users to extract it from the tarball much as
> is already done with GDB and djunpack.bat.  That's simple, and it
> would work.

How can instructions that need to be googled for be simple and
reliable?

> >> For example, it should be pretty easy to check emacs-25.chg
> >> automatically; is that done with GDB?
> > 
> > Yes, it is done.  But it doesn't catch all the errors.
> 
> How is that checking done, and what errors doesn't it catch?

It's done by the ARI script.  All I know about the errors is that some
files still clash.

> The renaming and copying is needed only on MS-DOS; it's not needed
> for any other platform.  It makes sense to do it only on MS-DOS.
> This will simplify maintenance on non-MS-DOS platforms

But the price will be unacceptable complications for MS-DOS.  No,
thanks, not unless we find a simpler way.

> >> Also, the problem of non-8+3 file names does not seem to be limited
> >> to gnulib-derived files.
> > 
> > Yes, they are limited to gnulib-derived files.  If you mean Org, I'm
> > sure those files will be renamed.
> 
> I meant all the other files that have 8+3 issues.

Which ones?

Re: Files from gnulib

[Jim Meyering]

Eli Zaretskii wrote:
>> From: Paul Eggert <eggert@cs.ucla.edu>
...
>> It is easy to run 'find' as part of the process that makes a
>> distribution, and to put its output into config.bat or the
>> equivalent, so there is no need to run 'find' under MS-DOS.
>
> More complications.  This means, for example, that to test an
> arbitrary revision of the development tree, I will need to run
> make-dist on Unix, create a tarball, copy it to a DOS machine, then
> build, find problems, go back to the Unix machine, etc.

Nah.  You wouldn't have to test at all,
because it could be automated.  See below.

>> > Perhaps it's possible to solve all this in a satisfactory manner, but
>> > doing so would require a lot of work,
>>
>> I don't think it'd take much work to do the above.  I can write
>> scripts to do the check and to do the find, since they can all be done
>> on a standard GNU platform.  What else is needed?
>
> Maintenance.

[assuming something like doslfn is too buggy -- have you tried it? ]

We could write an 8.3 conflict-checking Makefile rule and add
it as a dependent of "check".  Then any new file that conflicts
would be detected immediately, and whoever added it would
deal with the failed "make check" by renaming their file or
by adding a DOS-renaming rule.

>> > And what about the emacs-25.chg file?  Would you expect users to
>> > google for it as well, and copy-paste it into their shell window?
>>
>> No, I would expect users to extract it from the tarball much as
>> is already done with GDB and djunpack.bat.  That's simple, and it
>> would work.
>
> How can instructions that need to be googled for be simple and
> reliable?
>
>> >> For example, it should be pretty easy to check emacs-25.chg
>> >> automatically; is that done with GDB?
>> >
>> > Yes, it is done.  But it doesn't catch all the errors.
>>
>> How is that checking done, and what errors doesn't it catch?
>
> It's done by the ARI script.

I didn't find any script in GDB named ARI, but do see
many references to ARI in ChangeLogs.

> All I know about the errors is that some files still clash.

All that means is that there's room for improvement.
No need to reject that solution because it's not yet perfect.

Re: Files from gnulib

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  cyd@stupidchicken.com,  bug-gnulib@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 26 Jan 2011 12:13:28 +0100
> 
> >> It is easy to run 'find' as part of the process that makes a
> >> distribution, and to put its output into config.bat or the
> >> equivalent, so there is no need to run 'find' under MS-DOS.
> >
> > More complications.  This means, for example, that to test an
> > arbitrary revision of the development tree, I will need to run
> > make-dist on Unix, create a tarball, copy it to a DOS machine, then
> > build, find problems, go back to the Unix machine, etc.
> 
> Nah.  You wouldn't have to test at all,
> because it could be automated.  See below.

We are miscommunicating.  Paul suggested that config.bat, the script
used instead of the Posix `configure' when building the MS-DOS port,
will have the list of files/directories to rename/edit hardcoded into
it by make-dist.  Which means that to update config.bat, I will need
to run make-dist on a Unix host, then copy the results to the machine
where I build the DOS port, and only then build it.  Just testing
doesn't solve the issue, because if testing detects a problem, the way
to correct it is very complicated (compared to what I do today -- just
edit and try again).

> >> I don't think it'd take much work to do the above.  I can write
> >> scripts to do the check and to do the find, since they can all be done
> >> on a standard GNU platform.  What else is needed?
> >
> > Maintenance.
> 
> [assuming something like doslfn is too buggy -- have you tried it? ]
> 
> We could write an 8.3 conflict-checking Makefile rule and add
> it as a dependent of "check".  Then any new file that conflicts
> would be detected immediately, and whoever added it would
> deal with the failed "make check" by renaming their file or
> by adding a DOS-renaming rule.

"make check" is jut the part that detects problems.  They also need to
be fixed, and that's where most of the maintenance effort goes.  And
what about the need to remember to say "make check"?  Emacs does not
yet have a test suite that can be run as a single large test in batch
mode, so "make check" is not a routine part of building it.

> >> How is that checking done, and what errors doesn't it catch?
> >
> > It's done by the ARI script.
> 
> I didn't find any script in GDB named ARI, but do see
> many references to ARI in ChangeLogs.

I think you will find it on the GDB Web site.  If not, ask on the
gdb-patches mailing list.  I once asked for it and the core
maintainers sent it to me.

> > All I know about the errors is that some files still clash.
> 
> All that means is that there's room for improvement.
> No need to reject that solution because it's not yet perfect.

Either you underestimate the complexity of the task, or you somehow
believe that any complexity can be dealt with reliably.  In both
cases, I really have no practical way of convincing you (and Paul),
because this is no longer about facts, it's about experience and gray
hair.  After so many years of bad experience, I concluded that this
kind of solution will never work reliably, unless someone works full
time on maintaining it.  You are welcome to learn the same lesson the
hard way, but I've learned mine.

Re: Files from gnulib

[Jim Meyering]

Eli Zaretskii wrote:
>> From: Jim Meyering <jim@meyering.net>
>> >> It is easy to run 'find' as part of the process that makes a
>> >> distribution, and to put its output into config.bat or the
>> >> equivalent, so there is no need to run 'find' under MS-DOS.
>> >
>> > More complications.  This means, for example, that to test an
>> > arbitrary revision of the development tree, I will need to run
>> > make-dist on Unix, create a tarball, copy it to a DOS machine, then
>> > build, find problems, go back to the Unix machine, etc.
>>
>> Nah.  You wouldn't have to test at all,
>> because it could be automated.  See below.
>
> We are miscommunicating.  Paul suggested that config.bat, the script
> used instead of the Posix `configure' when building the MS-DOS port,
> will have the list of files/directories to rename/edit hardcoded into
> it by make-dist.  Which means that to update config.bat, I will need

There would be a rule (always run at least by "make dist") that would
update config.bat.  You could conceivably just run that rule, which
would surely be very quick.  You would not have to run the full "make dist".

However, you might not have to do even that.
You're presuming that people will be adding new files with conflicting
name, but without updating the renaming rules required for DOS.
That's where the automation comes in.
If some rename-shim is required for DOS, then any build target (even "all")
can require it, if it is deemed important enough.  Then, if someone
adds a conflicting file and does not also update the list of rename
pairs, a regular "make" could fail with a diagnostic telling them
what's required.

That would make it the responsibility of each person adding a new
conflicting file to tend to this small infrequent task, not you.

> to run make-dist on a Unix host, then copy the results to the machine
> where I build the DOS port, and only then build it.  Just testing
> doesn't solve the issue, because if testing detects a problem, the way
> to correct it is very complicated (compared to what I do today -- just
> edit and try again).

Re: Files from gnulib

[Lennart Borgman]

On Wed, Jan 26, 2011 at 2:23 PM, Jim Meyering <jim@meyering.net> wrote:
>
> However, you might not have to do even that.

At the moment it is not even possible to build on w32. Could we please
fix this first?

I do not know very much about the topic discussed here, but from my
experience things often does not work as well as those who have not
really tried assume. Maybe it would be could if those proposing a
change actually tried it themselves - in the environment where they
suggest it?

Re: Files from gnulib

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Wed, 26 Jan 2011 14:29:31 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, cyd@stupidchicken.com, eggert@cs.ucla.edu, 
> 	bug-gnulib@gnu.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> At the moment it is not even possible to build on w32. Could we please
> fix this first?

That's a separate issue.  I will fix the w32 build when I have time;
volunteers are welcome to beat me to it.

Re: Files from gnulib

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Cc: eggert@cs.ucla.edu,  cyd@stupidchicken.com,  bug-gnulib@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 26 Jan 2011 14:23:51 +0100
> 
> There would be a rule (always run at least by "make dist") that would
> update config.bat.  You could conceivably just run that rule, which
> would surely be very quick.  You would not have to run the full "make dist".

You cannot run "make ANYTHING" before config.bat, because the latter
edits Makefile.in files into Makefile files.

> You're presuming that people will be adding new files with conflicting
> name, but without updating the renaming rules required for DOS.
> That's where the automation comes in.

You cannot maintain the DB or renaming automatically, unless you are
willing to settle for meaningless file names.

> That would make it the responsibility of each person adding a new
> conflicting file to tend to this small infrequent task, not you.

People don't care about this now (and I cannot and don't blame them).
Why would they care more under this new scheme?

Re: Files from gnulib

[Jim Meyering]

Eli Zaretskii wrote:
>> From: Jim Meyering <jim@meyering.net>
>> Cc: eggert@cs.ucla.edu, cyd@stupidchicken.com, bug-gnulib@gnu.org,
>> monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Wed, 26 Jan 2011 14:23:51 +0100
>>
>> There would be a rule (always run at least by "make dist") that would
>> update config.bat.  You could conceivably just run that rule, which
>> would surely be very quick.  You would not have to run the full "make dist".
>
> You cannot run "make ANYTHING" before config.bat, because the latter
> edits Makefile.in files into Makefile files.

You would run it on a gnu/linux system.

>> You're presuming that people will be adding new files with conflicting
>> name, but without updating the renaming rules required for DOS.
>> That's where the automation comes in.
>
> You cannot maintain the DB or renaming automatically, unless you are
> willing to settle for meaningless file names.

Adding rename pairs would have to be done manually,
and as I said, infrequently, typically by the person who
has added the offending name into the repository.

>> That would make it the responsibility of each person adding a new
>> conflicting file to tend to this small infrequent task, not you.
>
> People don't care about this now (and I cannot and don't blame them).
> Why would they care more under this new scheme?

If the build rules (and policy) make it their concern,
and it is well enough documented, they will do it.
We're talking about only a few minutes of added work, here,
and then only infrequently.  How often do people add new
files to emacs?  Of those, how many fail the 8.3 uniqueness test?
Very few.
No big deal.

If this is the price people must pay to be rid of the 8.3
name restrictions, few will complain.

Re: Files from gnulib

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> More complications.  This means, for example, that to test an
> arbitrary revision of the development tree, I will need to run
> make-dist on Unix, create a tarball, copy it to a DOS machine, then
> build, find problems, go back to the Unix machine, etc.

There is also DOSBox which should obviate the need to use a separate DOS
machine.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Files from gnulib

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  cyd@stupidchicken.com,  bug-gnulib@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 26 Jan 2011 13:27:09 +0100
> 
> There is also DOSBox which should obviate the need to use a separate DOS
> machine.

Not if you develop software that needs to run on bare metal.

Re: Files from gnulib

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Paul Eggert <eggert@cs.ucla.edu>,  cyd@stupidchicken.com,  bug-gnulib@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Wed, 26 Jan 2011 13:27:09 +0100
>> 
>> There is also DOSBox which should obviate the need to use a separate DOS
>> machine.
>
> Not if you develop software that needs to run on bare metal.

Since when does Emacs run on bare metal?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Files from gnulib

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Wed, 26 Jan 2011 14:24:03 +0100
> Cc: cyd@stupidchicken.com, eggert@cs.ucla.edu, bug-gnulib@gnu.org,
> 	monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Since when does Emacs run on bare metal?

On plain DOS? since 1994.

Anyway, the DJGPP project supports all environments that include a
reasonable implementation of the DOS system calls.  Asking that one of
the most important packages in the DJGPP suite will only run on 1 or
two select environments, not even on the original DOS, is ridiculous.

Re: Files from gnulib

[Paul Eggert]

On 01/25/2011 08:10 PM, Eli Zaretskii wrote:

> It's done by the ARI script.  All I know about the errors is that some
> files still clash.

I looked into that, and the ARI script itself
<http://sourceware.org/gdb/current/ari/gdb_ari.sh> doesn't know
anything about 8+3 file names, so some other program must be doing it.
My guess is that they're using GNU doschk
<http://www.gnu.org/software/doschk/> and filtering the output.

If I run "make dist" on the Emacs trunk, and then ask GNU doschk
to report all 8+3 file name clashes in the resulting distribution, it
reports the following:

   ORG-COMP.EL    : ./lisp/org/org-compat.el
		    ./lisp/org/org-complete.el
   ORG-COMP.ELC   : ./lisp/org/org-compat.elc
		    ./lisp/org/org-complete.elc
   SEMANTIC.EL    : ./test/cedet/semantic-ia-utest.el
		    ./test/cedet/semantic-tests.el
		    ./test/cedet/semantic-utest-c.el
		    ./test/cedet/semantic-utest.el
   TESTSPPR.C     : ./test/cedet/tests/testsppreplace.c
		    ./test/cedet/tests/testsppreplaced.c

Do you see any errors in this report?  If so, we should fix GNU
doschk.  If not, that suggests that doschk is good enough for us
to use with Emacs, as part of a maintainer-mode check for problems
in this area.

>> From: Paul Eggert <eggert@cs.ucla.edu>
>> With GDB it's two commands:
>>
>>   djtar -x -p -o gdb-7.2/djunpack.bat gdb-7.2.tar.gz > djunpack.bat
>>   djunpack gdb-7.2.tar.gz
>>
>> Why would it be more complicated than that for Emacs?
> 
> This is the complexity I want to avoid.  Don't you think it's
> complicated enough?

Having MS-DOS builders type two commands to extract, rather than one,
is not complicated.  It would be a tiny price to pay, compared to
the hassles for all developers who have to shoehorn file names into
the 8+3 straitjacket.  People who build for MS-DOS can be expected to
understand minor workarounds like this.

> And how about the issue with using slashes in the
> argument to djunpack?

What issue is that?  In the above instructions, djunpack's argument
does not contain any slashes.

> Scripts are written by people, who tend to err or miss something.

Sure, but a reasonable script will greatly lower the error rate to
something that is manageable.  That's all that one can ask of any
build system.

>> 'find' is already required to build Emacs; for example, Makefile.in
>> uses it.
> 
> Only lisp/Makefile.in, which is not used when a release is built on
> DOS (all the files are already compiled).

No, 'find' is used in other places too; for example it is used
the top level Makefile.in, and in leim/Makefile.in.  But my point,
which I think you're agreeing with, is that it's OK to use 'find'
in maintainer 'make' rules, since maintainers are expected to have hosts
with a decent toolset.

> This means, for example, that to test an
> arbitrary revision of the development tree, I will need to run
> make-dist on Unix, create a tarball, copy it to a DOS machine, then
> build, find problems, go back to the Unix machine, etc.

That's OK.  It's normal, even on Unix-like hosts to do that.
I do it all the time.  I've caught multiple bugs recently in
Emacs by doing that.

The goal is not to make _maintenance_ practical on MS-DOS; that
would be far too ambitious.  The goal is only to make _building_
practical on MS-DOS.

>> What else is needed?
> 
> Maintenance.

No matter what solution we adopt, some maintenance will be required.
However, if we can automate most of it, we will lessen the overall
maintenance burden.

> How can instructions that need to be googled for be simple and
> reliable?

Instructions don't *need* to be Googled for.  One can visit the Emacs
web page, and navigate to the installation instructions, which will be
one of the web pages that I mentioned.  However, it is quite common to
use Google nowadays, more common than traditional navigation, and
there's nothing wrong with that.

>>>> Also, the problem of non-8+3 file names does not seem to be limited
>>>> to gnulib-derived files.
>>>
>>> Yes, they are limited to gnulib-derived files.  If you mean Org, I'm
>>> sure those files will be renamed.
>>
>> I meant all the other files that have 8+3 issues.
> 
> Which ones?

CEDET as well (see the above).  This is a continuing issue, with files
coming from multiple sources, and the problem is likely to keep
cropping up.  We need a better solution than what we've got now.

Re: Files from gnulib

[Eli Zaretskii]

Note: I removed gnulib from the list of addressees, as this is no
longer a gnulib issue.

> Date: Thu, 27 Jan 2011 00:32:03 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: cyd@stupidchicken.com, bug-gnulib@gnu.org, monnier@iro.umontreal.ca, 
>  emacs-devel@gnu.org
> 
> On 01/25/2011 08:10 PM, Eli Zaretskii wrote:
> 
> > It's done by the ARI script.  All I know about the errors is that some
> > files still clash.
> 
> I looked into that, and the ARI script itself
> <http://sourceware.org/gdb/current/ari/gdb_ari.sh> doesn't know
> anything about 8+3 file names, so some other program must be doing it.
> My guess is that they're using GNU doschk
> <http://www.gnu.org/software/doschk/> and filtering the output.

AFAIR, ARI calls doschk, or maybe there was some wrapper script that
did that.  But doschk is definitely used for this.

> If I run "make dist" on the Emacs trunk, and then ask GNU doschk
> to report all 8+3 file name clashes in the resulting distribution, it
> reports the following:
> 
>    ORG-COMP.EL    : ./lisp/org/org-compat.el
> 		    ./lisp/org/org-complete.el
>    ORG-COMP.ELC   : ./lisp/org/org-compat.elc
> 		    ./lisp/org/org-complete.elc

I emailed the Org mode maintainers yesterday (off-list), and they
agreed to rename org-complete.el to org-pcomplete.el (which is a more
correct name anyway, IMO).  The Emacs trunk will get that change when
it is sync'ed with Org next time.

>    SEMANTIC.EL    : ./test/cedet/semantic-ia-utest.el
> 		    ./test/cedet/semantic-tests.el
> 		    ./test/cedet/semantic-utest-c.el
> 		    ./test/cedet/semantic-utest.el
>    TESTSPPR.C     : ./test/cedet/tests/testsppreplace.c
> 		    ./test/cedet/tests/testsppreplaced.c

These clashes are not important, because the test suite should not be
part of the release tarball.  (I know that currently it is, but Stefan
said that this is a bug that should be fixed.)

> Do you see any errors in this report?  If so, we should fix GNU
> doschk.  If not, that suggests that doschk is good enough for us
> to use with Emacs, as part of a maintainer-mode check for problems
> in this area.

doschk is definitely good enough.

I do run doschk from time to time, and it seemed to be frequently
enough.  The issue is only important when a release is close.

In any case, the problem is not with having a Make target (the command
is trivial), but to arrange for someone or something to run it and
report the results.  Frankly, I never pushed that issue because people
tend to get annoyed by 8+3, so having something automatic that would
print messages about that didn't seem like a good idea...

> >>   djtar -x -p -o gdb-7.2/djunpack.bat gdb-7.2.tar.gz > djunpack.bat
> >>   djunpack gdb-7.2.tar.gz
> >>
> >> Why would it be more complicated than that for Emacs?
> > 
> > This is the complexity I want to avoid.  Don't you think it's
> > complicated enough?
> 
> Having MS-DOS builders type two commands to extract, rather than one,
> is not complicated.

I'm not talking about the number of commands, I'm talking about the
complexity of the first one and the potential to make mistakes in it.
Few DJGPP users even know about the -p and -o switches to djtar.
(djtar is generally used with only -x.)  If you use the command-line
arguments in a different order, chances are the command will not work
as intended.  I myself need to consult the README when I unpack the
GDB tarball, for fear of mistakes (which did happen), and I wrote that
stuff!


> > And how about the issue with using slashes in the
> > argument to djunpack?
> 
> What issue is that?  In the above instructions, djunpack's argument
> does not contain any slashes.

The argument could include slashes if the tarball is in a different
directory.  See the comments near the beginning of djunpack.bat.

> >> 'find' is already required to build Emacs; for example, Makefile.in
> >> uses it.
> > 
> > Only lisp/Makefile.in, which is not used when a release is built on
> > DOS (all the files are already compiled).
> 
> No, 'find' is used in other places too; for example it is used
> the top level Makefile.in

Not relevant, as the DOS build doesn't use the top-level Makefile.in.
See config.bat and msdos/mainmake.v2.

> and in leim/Makefile.in

Not relevant, since this use in the `install' target which isn't run
on DOS (because "make install" in mainmake.v2 installs the binary
in-place, under the source tree, and thus doesn't recurse into leim/).

> But my point,
> which I think you're agreeing with, is that it's OK to use 'find'
> in maintainer 'make' rules, since maintainers are expected to have hosts
> with a decent toolset.

Yes, I agree.  However, those comments of mine were to your suggestion
to rename the files as part of running config.bat, which is run at
configure time on the end-user machines.  If you now have something
else in mind, those comments are probably not pertinent anymore.

> > This means, for example, that to test an
> > arbitrary revision of the development tree, I will need to run
> > make-dist on Unix, create a tarball, copy it to a DOS machine, then
> > build, find problems, go back to the Unix machine, etc.
> 
> That's OK.  It's normal, even on Unix-like hosts to do that.
> I do it all the time.

You don't realize how little time I have to work on Emacs in general
and on the MS-DOS port in particular.  Each such complication steals
precious time that I don't have.  The Unix machine I use is fencepost,
so copying files involves moving them through the Internet.  Which
means I cannot do this offline (yes, connectivity problems do happen
in my corner of the world) or when fencepost is down, and also slows
down things quite a lot even if the connection is in good shape.

> > How can instructions that need to be googled for be simple and
> > reliable?
> 
> Instructions don't *need* to be Googled for.  One can visit the Emacs
> web page, and navigate to the installation instructions, which will be
> one of the web pages that I mentioned.

So now I'm supposed to lobby Emacs maintainers to have MS-DOS build
instructions on the Web site?  We've just heard several people ask
when it will be possible to drop the MS-DOS support entirely; how
would they react, you think, if I made such a request?  I generally
tend to avoid unnecessary friction wrt the MS-DOS port; I'm sure you
will understand after the experience of this thread.  msdos/INSTALL is
all I need, thank you.  It lies low and out of sight, and doesn't
bother anyone.

> However, it is quite common to use Google nowadays, more common than
> traditional navigation, and there's nothing wrong with that.

If you are looking for precise and correct instructions, googling is
your enemy, because it brings hits that cannot be guaranteed to be
correct.  Someone said something a year ago, and it's there forever,
even if in the meantime things happened that made it false.

Look, it's pointless to try to push further the possibility that Emacs
will adopt the same way of unpacking as GDB.  I invented that stuff in
the first place, but after all these years my experience with it is so
bad that I will never agree to have it in Emacs.  If the issue of
file-name clashes becomes so unbearable for the Emacs developers that
they will request that all limitations on file names are
unconditionally lifted, it will be easier for me to declare that the
MS-DOS port is dead, as soon as the conflicts hit the point where it
cannot be handled by simple tricks.

> >> I meant all the other files that have 8+3 issues.
> > 
> > Which ones?
> 
> CEDET as well (see the above).

Not an issue (see above).

> This is a continuing issue, with files coming from multiple sources,
> and the problem is likely to keep cropping up.  We need a better
> solution than what we've got now.

Experience shows that these issues crop up very rarely, although the
nervous reactions and the futile prolonged debates make it sound like
a grave problem.  I'm willing to consider any reasonable solutions to
that nonetheless, just not this one, because unlike any others, this
one is tested in practice, and it failed that test.

Re: Files from gnulib

[Paul Eggert]

`On 01/27/2011 03:08 AM, Eli Zaretskii wrote:

> I do run doschk from time to time, and it seemed to be frequently
> enough.  The issue is only important when a release is close.

If file renaming code needs to be checked only when a release is
close, that's fine.  We can run the checks then.

> In any case, the problem is not with having a Make target (the command
> is trivial), but to arrange for someone or something to run it and
> report the results.

This can be done when a release is close.  It's better if it's
automated.

>> Having MS-DOS builders type two commands to extract, rather than one,
>> is not complicated.
> 
> I'm not talking about the number of commands, I'm talking about the
> complexity of the first one and the potential to make mistakes in it.

Cutting and pasting two lines is pretty reliable.  Anyone who wants to
build Emacs for MS-DOS already should be referring to the
instructions, to learn other things.  For example, they need to know
that they should use djtar and not some other extractor.  These
instructions, which are needed anyway, can tell them to cut and paste
two lines.  There is no fundamental problem with this approach.

> The argument could include slashes if the tarball is in a different
> directory.

If people cute and paste two lines, as suggested, this won't be a
problem, because the two lines won't include slashes in the wrong
spot.

> So now I'm supposed to lobby Emacs maintainers to have MS-DOS build
> instructions on the Web site?

That's easy.  All we have to do is to put the two lines into the Emacs
manual (enough to extract the files), and then point people at the
extracted readme file.  That will not be a problem.

> You don't realize how little time I have to work on Emacs in general
> and on the MS-DOS port in particular.

Everybody who contributes to Emacs has limited time.  We are trying to
lessen the total amount of developer time being consumed by this problem.
It's possible that this may require spending a bit more of your time,
so that we collectively save time.  Even so, this can be a tradeoff
that is well worth making.

> If the issue of file-name clashes becomes so unbearable for the
> Emacs developers that they will request that all limitations on file
> names are unconditionally lifted, it will be easier for me to
> declare that the MS-DOS port is dead, as soon as the conflicts hit
> the point where it cannot be handled by simple tricks.

If that is the best alternative available, then we should do that.
However, it doesn't sound that hard to do something that is similar to
what GDB does, but is considerably more reliable because it is checked
systematically.

> Look, it's pointless to try to push further the possibility that Emacs
> will adopt the same way of unpacking as GDB.

I am not suggesting that.  I'm suggesting something better.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Thu, 27 Jan 2011 23:30:12 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> Cc: cyd@stupidchicken.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > Look, it's pointless to try to push further the possibility that Emacs
> > will adopt the same way of unpacking as GDB.
> 
> I am not suggesting that.  I'm suggesting something better.

Then perhaps I don't understand your suggestion.  Could you describe
it in its entirety, starting with how file-name clashes would be
detected, how their alternative names would be determined, and how all
this would work with make-dist and configuring and building the MS-DOS
port?

Compartmentalizing the 8.3 problem into the msdos directory

[Paul Eggert]

[Renaming the thread from "Files from gnulib".]

On 01/28/2011 06:20 AM, Eli Zaretskii wrote:

> Then perhaps I don't understand your suggestion.  Could you describe
> it in its entirety, starting with how file-name clashes would be
> detected, how their alternative names would be determined, and how all
> this would work with make-dist and configuring and building the MS-DOS
> port?

Here's how it works.

File name clashes are detected by doschk, since that's reliable.

Alternative names are determined by a file msdos/fnchange.prs
that contains lines that look like this:

   semantic-utest-c.el     sem-ut-c.el
   semantic-utest.el       sem-ut.el

where the left column gives the original base name, and the right
column gives the base name as renamed under MS-DOS.

After make-dist creates the distribution tree, it calls a new script
msdos/fncheck.sh that does the MS-DOS-specific file name checking and
generation; make-dist fails if msdos/fncheck.sh fails.

msdos/fncheck.sh creates a file msdos/fnchange.lst that is suitable for
use by djtar to unpack the tarball on MS-DOS.  msdos/fncheck.sh
guarantees that djtar, when used with msdos/fnchange.lst, will not
generate any name clashes; so if there's a typo in fnchange.prs this
is caught at "make dist" time.

Once the files are unpacked on MS-DOS, the MS-DOS build procedure
first finds all instances of the left column in fnchange.prs in the
source code, and substitutes the right column.  Then it builds as usual.

The general outline is similar to what's done with GDB.  But the
improvement over GDB is that "make dist" checks for clashes reliably,
and this catches the usual glitches where people forget to update
fnchange.prs after creating clashing names.

Here's a draft implementation of all of the above (except that the
MS-DOS side is left as an exercise for the reader :-).  I haven't
committed this.

=== modified file 'make-dist'
--- make-dist	2011-01-31 08:12:52 +0000
+++ make-dist	2011-01-31 09:05:07 +0000
@@ -413,6 +413,7 @@
 echo "Making links to \`msdos'"
 (cd msdos
  ln ChangeLog INSTALL README emacs.ico emacs.pif ../${tempdir}/msdos
+ ln fncheck.sh fnchange.prs ../${tempdir}/msdos
  ln is_exec.c sigaction.c mainmake.v2 sed*.inp ../${tempdir}/msdos)
 
 echo "Making links to \`nextstep'"
@@ -525,6 +526,9 @@
 echo "Removing unwanted files"
 find ${tempparent} \( -name '*~' -o -name '#*#' -o -name '.*ignore' -o -name '=*' -o -name 'TAGS' \) -exec rm -f {} \;
 
+echo "Making file map for MS-DOS"
+(cd ${tempdir} && msdos/fncheck.sh) || exit
+
 if [ "${make_tar}" = yes ]; then
   echo "Looking for $default_gzip"
   found=0

::::::::::::::
msdos/fnchange.prs
::::::::::::::
# List of file base names that need to be renamed in MS-DOS due to its
# 8+3 limitations.  The second column is the base name under MS-DOS.
# The assumption is that each such file is renamed within its
# directory under MS-DOS, and that all directory names are OK.

org-complete.el         o-complete.el
org-complete.elc        o-complete.elc
semantic-ia-utest.el    sem-ia-utest.el
semantic-utest-c.el     sem-ut-c.el
semantic-utest.el       sem-ut.el
testsppreplaced.c       tpprplcd.c


::::::::::::::
msdos/fncheck.sh (this file should be executable)
::::::::::::::
#! /bin/sh

export LC_ALL=C

if (doschk) </dev/null >/dev/null 2>&1; then
  doschk=doschk
else
  echo >&2 "$0: warning: doschk not installed; skipping MS-DOS checks"
  doschk=true
fi

<msdos/fnchange.prs || exit

find .??* * -print | sort | awk '
  BEGIN {
    while (0 < (getline <"msdos/fnchange.prs")) {
      if ($1 ~ /^[^#]/ && $2) renamed[$1] = $2
    }
  }
  {
    file = $0
    basename = $file
    sub(/.*\//, "", basename)
    if (renamed[basename]) {
      renamed_file = file
      sub(/[^/]*$/, "", renamed_file)
      renamed_file = renamed_file renamed[basename]
      printf "@V@/%s @V@/%s\n", file, renamed_file > "msdos/fnchange.lst"
      print renamed_file
    } else {
      print file
    }
  }
' | $doschk | awk '
  /^The following files are not valid DOS file names:$/,  /^$/ { next }
  /^The following resolve to the same SysV file names:$/, /^$/ { next }
  /^The following file names are too long for SysV:$/,    /^$/ { next }
  { print; status = 1 }
  END { exit status }
'

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Eli Zaretskii]

> Date: Mon, 31 Jan 2011 01:29:18 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: cyd@stupidchicken.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On 01/28/2011 06:20 AM, Eli Zaretskii wrote:
> 
> > Then perhaps I don't understand your suggestion.  Could you describe
> > it in its entirety, starting with how file-name clashes would be
> > detected, how their alternative names would be determined, and how all
> > this would work with make-dist and configuring and building the MS-DOS
> > port?
> 
> Here's how it works.

Thanks.  Here are the main difficulties I see with this method:

 . It will effectively block a release, until msdos/fnchange.prs is
   fixed.  I doubt that release managers will like that.  To unblock
   the release, they will have to negotiate with me, because alternate
   file names cannot be arbitrary.  (This is unlike in GDB, where the
   renamed files are not used in the MS-DOS build, and so their
   alternate names can be anything at all.)  I don't think I want to
   be a road block for Emacs releases: what if I'm sick or off line
   for several days?

 . The proposed fncheck.sh script will rename files regardless of
   their directories, which might cause gratuitous problems, if the
   same basename happens in more than one directory, but causes
   file-name clashes only in one of them.

 . The script that is to be run at MS-DOS configure time, the one you
   left as an exercise, is tricky to write at best, for the following
   reasons:

   . It needs to run on plain DOS, and so can only use features
     supported by the extremely primitive command.com shell available
     there.  For example, it cannot even recurse through arbitrary
     directory trees.

     The Emacs build procedure for MS-DOS makes a point of not using
     any ported GNU software, such as Bash or Findutils, for which
     replacements are not easily available, because Emacs is one of
     the few packages that need to be built first to bootstrap the
     DJGPP development environment.  Currently, only rm, mv, cp, and
     Sed are required, in addition to Make and a working compiler.
     (This is unlike GDB, which already requires Bash, Coreutils, and
     Findutils.)

     To make this script possible without requiring more ported tools,
     we will need to have some database of directories into which to
     look and files in those directories to edit.  This means more
     manually maintained files and more potential for errors.

   . It's not clear, in general, how to replace original names with
     alternative ones.  Different types of files (Makefile.in, C
     sources, Lisp sources, auto-generated files) that refer to
     renamed ones might require different ways of editing them.

     For example, files in etc/themes assume that the name of the file
     matches the name of the theme itself, so we will need to edit the
     `provide' line as well, and then any `require' lines elsewhere.
     
     As another example, various Makefile's could use shell trickery,
     such as mentioning "*.info-*" wildcards, which needs implicitly
     to be edited without any file by that name present anywhere, so
     doschk will not catch that.

     The rules to support those non-trivial issues will have to be
     maintained manually: more potential for errors.

   . Editing files generally changes their time stamps, which could
     then trigger Make rules not intended to run at end-user build
     time (e.g., generation of Makefile.in from Makefile.am).  Keeping
     the original file times requires additional tools such as GNU
     `touch'.

So, by and large, I'd prefer to avoid these complications as much as
possible.  If I'm faced with the choice to drop MS-DOS build support
or use this method, I might reconsider, but even then it's quite
possible I will decide to give up rather than plunge into this
adventure.  As long as the head maintainers don't force me to face
that choice, I'm unwilling to add all this complexity to the task of
maintaining the DOS port, which currently takes a small fraction of my
time (even if we count in the occasional dispute such as the one which
led to this thread ;-).

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Paul Eggert]

On 02/05/2011 02:59 AM, Eli Zaretskii wrote:

>  . It will effectively block a release, until msdos/fnchange.prs is
>    fixed.

We can easily give an option to override that, if necessary.  But the
idea is that such an option should not be needed, because the normal
state of affairs is that fnchange.prs should be sufficiently
up-to-date.  Even if (due to some last-second screwup) it's not
up-to-date, fnchange.prs is easy enough for the releaser to fix
without knowing all the ins and outs of MS-DOS 8+3 rules.

>    . It needs to run on plain DOS, and so can only use features
>      supported by the extremely primitive command.com shell available
>      there.  For example, it cannot even recurse through arbitrary
>      directory trees.

That can be handled in the same way that the file-renaming is handled:
do the 'find' on Linux as part of the release process, distribute the
output of 'find' as a file, and have the DOS build procedure read that file.
This essentially removes the potential for error that you mentioned.

>    . Editing files generally changes their time stamps, which could
>      then trigger Make rules not intended to run at end-user build
>      time

If the DOS procedure edits files in the proper order, the resulting
time stamps should be consistent with what 'make' expects.


The remaining problems you mentioned seem to be largely theoretical.
They might occur if we start to use unusually tricky file names, but
these problems don't exist now and are unlikely to occur.  If they do
happen, we can deal with those problems by hand, as they come up.
Ordinarily long file names shouldn't trigger any of those problems.

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Eli Zaretskii]

> Date: Sat, 05 Feb 2011 03:20:30 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: cyd@stupidchicken.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On 02/05/2011 02:59 AM, Eli Zaretskii wrote:
> 
> >  . It will effectively block a release, until msdos/fnchange.prs is
> >    fixed.
> 
> We can easily give an option to override that, if necessary.

Then it _will_ be used, and the whole point is moot.

> >    . It needs to run on plain DOS, and so can only use features
> >      supported by the extremely primitive command.com shell available
> >      there.  For example, it cannot even recurse through arbitrary
> >      directory trees.
> 
> That can be handled in the same way that the file-renaming is handled:
> do the 'find' on Linux as part of the release process, distribute the
> output of 'find' as a file, and have the DOS build procedure read that file.

More complications for the release process.

> >    . Editing files generally changes their time stamps, which could
> >      then trigger Make rules not intended to run at end-user build
> >      time
> 
> If the DOS procedure edits files in the proper order, the resulting
> time stamps should be consistent with what 'make' expects.

I very much doubt that there exists a "proper order" that would
satisfy these requirements.  And even it does, finding it will be
non-trivial and error-prone.

> The remaining problems you mentioned seem to be largely theoretical.

Not if limitations on file names are lifted entirely.

> If they do happen, we can deal with those problems by hand, as they
> come up.

We do that now, and see where it got us.

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Paul Eggert]

On 02/05/2011 03:26 AM, Eli Zaretskii wrote:
>> If they do happen, we can deal with those problems by hand, as they
>> come up.
> 
> We do that now, and see where it got us.

Yes, right now "make dist" in the Emacs trunk doesn't obey the 8+3
restrictions.  My proposal would fix all six violations, automatically.

I'm trying to make it easier for mainstream developers to be largely
unaffected by DOS filename limits.  With a bit of luck this method
will work for all the file names developers are likely to add later.
But even if there are a few exceptions that require fixing by hand,
we'll still be much better off than we are now.

>> The remaining problems you mentioned seem to be largely theoretical.
> 
> Not if limitations on file names are lifted entirely.

I'm not proposing lifting *all* naming limitations.  Just lifting them
enough so that as a practical matter, non-DOS developers can almost
always stop worrying about this stuff.

>> We can easily give an option to override that, if necessary.
> 
> Then it _will_ be used, and the whole point is moot.

It's not moot.  It will be up to the person making the release to
decide whether a last-second 8+3 problem needs to be fixed before the
release is actually cut.  Releasers already have to make these sorts
of decisions; the change would help them do their job better, by
automating the checks.

>> That can be handled in the same way that the file-renaming is handled:
>> do the 'find' on Linux as part of the release process, distribute the
>> output of 'find' as a file, and have the DOS build procedure read that file.
> 
> More complications for the release process.

It's a small script, easily written, and reliable once written.

>> If the DOS procedure edits files in the proper order, the resulting
>> time stamps should be consistent with what 'make' expects.
> 
> I very much doubt that there exists a "proper order"

I don't see why not.  Any order that satisfies Make's dependencies
will do.  Just generate the files in the same order that 'make' does.

I see no significant technical impediment to the proposal.  If the
MS-DOS port is important enough to keep, then it's important enough to
improve its release process in this way, so that its 8+3 limits don't
continue to impede mainstream development.

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Glenn Morris]

Paul Eggert wrote:

> It will be up to the person making the release to decide whether a
> last-second 8+3 problem needs to be fixed before the release is
> actually cut.

Pretesting of releases goes on for months. It seems highly unlikely that
a brand new entire file will be added at the least minute. Thus I don't
think any such problems will occur "at the last second" in practice.

If I read `bzr log' correctly, the last time a file was added or renamed
in emacs-23, for example, was 3 months ago.

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Miles Bader]

Glenn Morris <rgm@gnu.org> writes:
> If I read `bzr log' correctly, the last time a file was added or renamed
> in emacs-23, for example, was 3 months ago.

Also, of course, not all new files cause 8.3 conflicts...

-Miles

-- 
Education, n. That which discloses to the wise and disguises from the foolish
their lack of understanding.

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Eli Zaretskii]

> Date: Sat, 05 Feb 2011 15:30:54 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: cyd@stupidchicken.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> I see no significant technical impediment to the proposal.

I gave technical reasons why it will complicate things.  We just
disagree wrt their significance.

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Paul Eggert]

On 02/05/2011 08:01 PM, Eli Zaretskii wrote:

> I gave technical reasons why it will complicate things.  We just
> disagree wrt their significance.

The technical reasons are clearly minor, and are not enough to prevent
the change.

There's no need to make the change now, as the MS-DOS build evidently
works well enough now even though several source file names clash
after 8+3 truncation.  But if in the future there are more 8+3
clashes, and if these new clashes cause significant problems on
MS-DOS, then the change will become relevant, as we'll be able to use
it rather than worry about renaming Emacs source files to keep MS-DOS
happy.

The bottom line is that there is no longer any significant technical
reason to restrict all Emacs source file names to be unique after 8+3
truncation.

Re: Compartmentalizing the 8.3 problem into the msdos directory

[Eli Zaretskii]

> Date: Sat, 05 Feb 2011 23:30:24 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: cyd@stupidchicken.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> The bottom line is that there is no longer any significant technical
> reason to restrict all Emacs source file names to be unique after 8+3
> truncation.

According to you; clearly, I disagree.

Re: Files from gnulib

[Glenn Morris]

Stefan Monnier wrote:

> Here, again, the `test' subdir is not included in the tarball, so it's
> not an issue.

Except of course, now it is included:

http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00445.html

Re: Files from gnulib

[Stefan Monnier]

>> Here, again, the `test' subdir is not included in the tarball, so it's
>> not an issue.
> Except of course, now it is included:
> http://lists.gnu.org/archive/html/emacs-devel/2011-01/msg00445.html

But it's an error, so it's not really relevant.


        Stefan

Re: Files from gnulib

[Eli Zaretskii]

> Date: Sun, 23 Jan 2011 11:29:37 -0800
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org, bug-gnulib <bug-gnulib@gnu.org>
> 
> Developers on Microsoft Windows platforms all have access to
> better file systems these days, surely.

I'm talking only about the MSDOS port.  The Windows port has no
problems with these file names.

It must be possible to build the MSDOS port on plain DOS, because
Emacs is one of a couple of GNU programs (another one is Make) that
are built first, because without them you have no usable development
environment.  That is why Emacs on MSDOS is built using a minimal set
of tools, although DJGPP has an excellent port of Bash, which could be
used to run the configure script.

> So, other than c++defs.h, I don't see the need for renaming these
> files.

Given the responses from Stefan and myself, please reconsider.  Again,
I'm willing to do the job myself if needed, if you could guide me
through the gnulib maze.

TIA

RE : Re: Files from gnulib

[Bastien ROUCARIES]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

Btw does doslfn not available these days?

Bastien

Le 25 janv. 2011 05:00, "Eli Zaretskii" <eliz@gnu.org> a écrit :

> Date: Mon, 24 Jan 2011 15:26:06 -0800

> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Stefan Monnier <monnier@iro.umontreal.ca>, bug-gnulib@gnu.org,
>  emacs-devel@gnu.org
>

> On 01/23/11 20:01, Eli Zaretskii wrote:
> > The program used to unpack the .tar.gz archives automa...
There are no special tarballs for MSDOS.  People use the tarball from
the GNU FTP site.  So this solution will not work.


> * The conflicting names in m4/* don't matter, for the same
> reason that conflicts in admin/* a...
The files in m4/ still matter because you need to unpack the tarball,
and the utility that does that won't silently overwrite files due to
file-name clashes.

[-- Attachment #2: Type: text/html, Size: 1284 bytes --]

Re: RE : Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 09:20:48 +0100
> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca, bug-gnulib@gnu.org, 
> 	Paul Eggert <eggert@cs.ucla.edu>
> 
> Btw does doslfn not available these days?

It can be downloaded (although unmaintained since 2006), but I don't
know whether it can be a viable solution.  It was never widely used
with DJGPP, so its compatibility with the Long File Name (LFN)
functions used by DJGPP is unknown.  DJGPP implementation of the
Standard C library assumes that any LFN support implements the full
range of LFN functions that Windows 9X provided.  There isn't a single
LFN function that is not used by some library function in DJGPP.

If someone wants LFN support nowadays, they can easily enough install
Windows (where DJGPP programs can run and support long file names).
Which may explain why doslfn is not used too much.

So I'd rather not request users to install doslfn.  None of the other
DJGPP ports requires LFN as a prerequisite for building it, so doing
so for Emacs sounds like a bad idea to me, when the alternative is to
rename a few files.

Re: RE : Re: Files from gnulib

[Jim Meyering]

Eli Zaretskii wrote:
>> Date: Tue, 25 Jan 2011 09:20:48 +0100
>> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
>> Cc: emacs-devel@gnu.org, monnier@iro.umontreal.ca, bug-gnulib@gnu.org,
>> 	Paul Eggert <eggert@cs.ucla.edu>
>>
>> Btw does doslfn not available these days?
>
> It can be downloaded (although unmaintained since 2006), but I don't
> know whether it can be a viable solution.  It was never widely used
> with DJGPP, so its compatibility with the Long File Name (LFN)
> functions used by DJGPP is unknown.  DJGPP implementation of the
> Standard C library assumes that any LFN support implements the full
> range of LFN functions that Windows 9X provided.  There isn't a single
> LFN function that is not used by some library function in DJGPP.
>
> If someone wants LFN support nowadays, they can easily enough install
> Windows (where DJGPP programs can run and support long file names).
> Which may explain why doslfn is not used too much.
>
> So I'd rather not request users to install doslfn.  None of the other
> DJGPP ports requires LFN as a prerequisite for building it, so doing
> so for Emacs sounds like a bad idea to me, when the alternative is to
> rename a few files.

Forcing current and future emacs development into the archaic 8.3 mold has
a significant cost (just look at how long this thread is), yet provides
relatively little benefit.  If something like doslfn is reliable enough
and not hard to install, then requiring it makes sense: then all emacs
developers will be freed of this onerous file-naming constraint.

This is like various shims we've used over the years. (remember ansi2knr?)
Imposing small relatively transparent requirements on users of less common
systems is actually a good practice, when doing so permits improvements
in the development process.

Re: RE : Re: Files from gnulib

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Cc: Bastien ROUCARIES <roucaries.bastien@gmail.com>,  eggert@cs.ucla.edu,  bug-gnulib@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 25 Jan 2011 15:51:00 +0100
> 
> Forcing current and future emacs development into the archaic 8.3 mold has
> a significant cost

The costs are generally mine, and mine alone.  I offered to do the
renaming job myself, provided some guidance from people who know their
way through gnulib.

> (just look at how long this thread is)

It could be much shorter if my original request was granted.  It is
long because people ask me questions and I respect them enough to
answer those questions in detail.  I don't mind keeping answering
them, but please don't hold that against me, or present that as
incurring significant costs on Emacs development.  If we want to cut
our losses, why not accept my suggestion, and be done with that?  For
that matter, how about presenting some technical reasons for objecting
the renaming I suggested, or any alternative renaming?  I explained
why proposed alternatives were problematic, but didn't yet see any
explanation of the reasons behind the apparent objection.

> yet provides relatively little benefit.

See, you are wrong here.  The number of times I found bugs in Emacs
that are of importance to Posix platforms, just by building the DOS
port, is not negligible.  The reasons are that the DOS build is very
similar to the Unix build --without-x (which evidently not many people
who track the development try these days), and its use of menus is
exactly identical to the no-toolkit X build.  These are evidently rare
configurations, but they are still supported.

I think that the occasional hour or two I invest once in a few weeks
when the DOS build becomes broken and I need to fix it is well payed
by the benefits that brings to Emacs development in general, by
uncovering bugs in those rare configurations.  And if it does some
service to a niche user community while at that, what's wrong with
that?

> If something like doslfn is reliable enough
> and not hard to install, then requiring it makes sense: then all emacs
> developers will be freed of this onerous file-naming constraint.

It's impossible for me to say if doslfn is reliable.  I never used it
myself, nor was it ever used widely enough by DJGPP users.

As for the onerous file-naming constraint, we have more than 3000
files in the Emacs tree, and the problem is limited to just 7 or so,
all of them recent additions.

> Imposing small relatively transparent requirements on users of less common
> systems is actually a good practice, when doing so permits improvements
> in the development process.

I'm not aware of any improvements in the development process that the
DOS port imposes.

Re: Files from gnulib

[Ulrich Mueller]

>>>>> On Tue, 25 Jan 2011, Eli Zaretskii wrote:

>> Forcing current and future emacs development into the archaic 8.3
>> mold has a significant cost

> The costs are generally mine, and mine alone.

Unfortunately, that is not entirely true.

For example, most files of CEDET were renamed to 8+3 some time ago,
breaking other elisp packages (e.g. company-mode, ecb, jde, matlab,
speechd-el) depending on them.

Ulrich

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 17:50:59 +0100
> From: Ulrich Mueller <ulm@gentoo.org>
> Cc: eggert@cs.ucla.edu, bug-gnulib@gnu.org, Jim Meyering <jim@meyering.net>,
> 	emacs-devel@gnu.org, monnier@iro.umontreal.ca, roucaries.bastien@gmail.com
> 
> For example, most files of CEDET were renamed to 8+3 some time ago,
> breaking other elisp packages (e.g. company-mode, ecb, jde, matlab,
> speechd-el) depending on them.

Reality check: there were many changes made in CEDET due to admission
into the Emacs distribution.  File-name changes were only a small part
of them, and the changes just moved some files into subdirectories,
such that foo-bar.el became foo/bar.el.

Re: Files from gnulib

[Ulrich Mueller]

>>>>> On Tue, 25 Jan 2011, Eli Zaretskii wrote:

>> For example, most files of CEDET were renamed to 8+3 some time ago,
>> breaking other elisp packages (e.g. company-mode, ecb, jde, matlab,
>> speechd-el) depending on them.

> Reality check: there were many changes made in CEDET due to admission
> into the Emacs distribution.  File-name changes were only a small part
> of them, and the changes just moved some files into subdirectories,
> such that foo-bar.el became foo/bar.el.

Which still breaks compatibility, because (require 'foo-bar) in any
elisp package will fail now.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 20:31:30 +0100
> From: Ulrich Mueller <ulm@gentoo.org>
> Cc: eggert@cs.ucla.edu, bug-gnulib@gnu.org, jim@meyering.net,
> 	emacs-devel@gnu.org, monnier@iro.umontreal.ca, roucaries.bastien@gmail.com
> 
> > Reality check: there were many changes made in CEDET due to admission
> > into the Emacs distribution.  File-name changes were only a small part
> > of them, and the changes just moved some files into subdirectories,
> > such that foo-bar.el became foo/bar.el.
> 
> Which still breaks compatibility, because (require 'foo-bar) in any
> elisp package will fail now.

And that was the single backward-incompatible change in CEDET?

Re: Files from gnulib

[Ulrich Mueller]

>>>>> On Tue, 25 Jan 2011, Eli Zaretskii wrote:

>> > Reality check: there were many changes made in CEDET due to admission
>> > into the Emacs distribution.  File-name changes were only a small part
>> > of them, and the changes just moved some files into subdirectories,
>> > such that foo-bar.el became foo/bar.el.
>> 
>> Which still breaks compatibility, because (require 'foo-bar) in any
>> elisp package will fail now.

> And that was the single backward-incompatible change in CEDET?

No, but your original statement was: "The costs are generally mine,
and mine alone."

These renamed files have to be taken account for in all packages using
CEDET, and that means additional work for the upstreams of these
packages and for distro builders. Therefore, at least in the case of
CEDET, the costs are not yours alone.

Re: Files from gnulib

[Eli Zaretskii]

> Date: Tue, 25 Jan 2011 21:00:16 +0100
> Cc: eggert@cs.ucla.edu, bug-gnulib@gnu.org, jim@meyering.net,
>         emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>         roucaries.bastien@gmail.com
> From: Ulrich Mueller <ulm@gentoo.org>
> 
> >>>>> On Tue, 25 Jan 2011, Eli Zaretskii wrote:
> 
> >> > Reality check: there were many changes made in CEDET due to admission
> >> > into the Emacs distribution.  File-name changes were only a small part
> >> > of them, and the changes just moved some files into subdirectories,
> >> > such that foo-bar.el became foo/bar.el.
> >> 
> >> Which still breaks compatibility, because (require 'foo-bar) in any
> >> elisp package will fail now.
> 
> > And that was the single backward-incompatible change in CEDET?
> 
> No, but your original statement was: "The costs are generally mine,
> and mine alone."

The word "generally" is there for a reason.

Re: Files from gnulib

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> For example, most files of CEDET were renamed to 8+3 some time ago,
>> breaking other elisp packages (e.g. company-mode, ecb, jde, matlab,
>> speechd-el) depending on them.
>
> Reality check: there were many changes made in CEDET due to admission
> into the Emacs distribution.  File-name changes were only a small part
> of them, and the changes just moved some files into subdirectories,
> such that foo-bar.el became foo/bar.el.

Those changes were a burden the day I tried to use an external package
which adds CEDET-based C# code completion to Emacs. I thought that as
the supposedly difficult part of installing CEDET was already done
thanks to its integration into the Emacs distribution, applying the
simple instructions for adding the C# package would be a piece of
cake. But soon discovered that the instructions and the .el file
referenced several files that apparently were missing from Emacs. After
some investigation I learned that those files were moved. It was not fun
to locate each file and patch the external package. Moving the files
will cause extra work to those who maintain external CEDET packages, so
they must introduce changes for supporting the new places while keeping
compatibility with the old ones.

This is just to demonstrate that the 8+3 issue is not as irrelevant as
you say. That said, I'm glad that the DOS port provides motivation for
keeping you as an Emacs maintainer, some renaming from time to time is a
little price to pay for that.

Re: Files from gnulib

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Cc: Ulrich Mueller <ulm@gentoo.org>,  eggert@cs.ucla.edu,  jim@meyering.net,  emacs-devel@gnu.org,  monnier@iro.umontreal.ca,  roucaries.bastien@gmail.com
> Date: Tue, 25 Jan 2011 20:52:22 +0100
> 
> This is just to demonstrate that the 8+3 issue is not as irrelevant as
> you say.

Well, I didn't really mean to say they were irrelevant, just that they
are not as significant as this thread seems to indicate.

> That said, I'm glad that the DOS port provides motivation for
> keeping you as an Emacs maintainer, some renaming from time to time is a
> little price to pay for that.

Thank you for your kind words.

Re: Files from gnulib

[Stefan Monnier]

> For example, most files of CEDET were renamed to 8+3 some time ago,
> breaking other elisp packages (e.g. company-mode, ecb, jde, matlab,
> speechd-el) depending on them.

Indeed the CEDET renaming was (and still is) a source of problems
because there is still an "upstream" that hasn't done the same renaming
yet (or has been done, finally?).
But in this particular case, I'd argue that the restructuring was for
the better.

This said, I'd be happy if we could solve the 8+3 issue differently.
Eli, what do you think of Paul's suggestion to use

  "djtar -n emacs-25.chg emacs-25.tgz"


-- Stefan

Re: Files from gnulib

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Eli Zaretskii <eliz@gnu.org>, eggert@cs.ucla.edu, bug-gnulib@gnu.org,
>         Jim Meyering <jim@meyering.net>, emacs-devel@gnu.org,
>         roucaries.bastien@gmail.com
> Date: Tue, 25 Jan 2011 14:04:44 -0500
> 
> This said, I'd be happy if we could solve the 8+3 issue differently.

I will go with any solution that does not unduly increases the burden
and does not involve tedious manual work down the line.  For now, the
only alternatives I heard more or less told me to go screw myself.

> Eli, what do you think of Paul's suggestion to use
> 
>   "djtar -n emacs-25.chg emacs-25.tgz"

I already replied: this kind of scheme is used by GDB, and it works
very badly.  I don't want to go that way in Emacs.

Re: Files from gnulib

[Bastien ROUCARIES]

Le mardi 25 janvier 2011 16:33:40, Eli Zaretskii a écrit :
> I think that the occasional hour or two I invest once in a few weeks
> when the DOS build becomes broken and I need to fix it is well payed
> by the benefits that brings to Emacs development in general, by
> uncovering bugs in those rare configurations.  And if it does some
> service to a niche user community while at that, what's wrong with
> that?
> 
> > If something like doslfn is reliable enough
> > and not hard to install, then requiring it makes sense: then all emacs
> > developers will be freed of this onerous file-naming constraint.
> 
> It's impossible for me to say if doslfn is reliable.  I never used it
> myself, nor was it ever used widely enough by DJGPP users.
> 
> As for the onerous file-naming constraint, we have more than 3000
> files in the Emacs tree, and the problem is limited to just 7 or so,
> all of them recent additions.
> 
> > Imposing small relatively transparent requirements on users of less
> > common systems is actually a good practice, when doing so permits
> > improvements in the development process.
> 
> I'm not aware of any improvements in the development process that the
> DOS port imposes.
A quick search on google show that doslfn is used by DJGPP user. 

Moreover, instead of fixing bugs in source package, it will allow a final fix for dos. And instead of renaming and thus using your 
time doing every time the same stuff, you could fix the doslfn package is buggy. If fixed once, it will definitvly fix the 8.3 problem 
(if and only if doslfn is buggy, and it does not seems according to a quick search).

Bastien

Re: Files from gnulib

[Eli Zaretskii]

> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Date: Tue, 25 Jan 2011 23:37:08 +0100
> Cc: Jim Meyering <jim@meyering.net>,
>  eggert@cs.ucla.edu,
>  bug-gnulib@gnu.org,
>  monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> 
> (if and only if doslfn is buggy, and it does not seems according to a quick search).

Your search was too quick.

Re: Files from gnulib

[Jim Meyering]

Eli Zaretskii wrote:
>> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
>> Date: Tue, 25 Jan 2011 23:37:08 +0100
>>
>> (if and only if doslfn is buggy, and it does not seems according to
>> a quick search).
>
> Your search was too quick.

Considering your wish to continue supporting emacs on DOS,
I would have thought you would jump at a possible solution like this.
If it works (and indications are that it does), then it defines
away the whole problem.  Wouldn't you welcome the idea of a DOS
port with no risk of 8.3 collisions?

What if the solution really is as easy as it appears?  Here is a
recently-updated FAQ:

    http://www-user.tu-chemnitz.de/~heha/hs_freeware/what_lfn.htm.en

Please give it an honest try before dismissing it.

Just because someone wrote about a problem does not
mean using it to build Emacs will trigger the same one.
And even if it does, report it nicely and someone might
even fix it right away.

Re: Files from gnulib

[Bastien ROUCARIES]

Le mercredi 26 janvier 2011 12:02:47, Jim Meyering a écrit :
> Eli Zaretskii wrote:
> >> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> >> Date: Tue, 25 Jan 2011 23:37:08 +0100
> >> 
> >> (if and only if doslfn is buggy, and it does not seems according to
> >> a quick search).
> > 
> > Your search was too quick.
> 
> Considering your wish to continue supporting emacs on DOS,
> I would have thought you would jump at a possible solution like this.
> If it works (and indications are that it does), then it defines
> away the whole problem. 

Know bug list is pretty short note

> Wouldn't you welcome the idea of a DOS
> port with no risk of 8.3 collisions?
> 
> What if the solution really is as easy as it appears?  Here is a
> recently-updated FAQ:
> 
>     http://www-user.tu-chemnitz.de/~heha/hs_freeware/what_lfn.htm.en
> 
> Please give it an honest try before dismissing it.
> 
> Just because someone wrote about a problem does not
> mean using it to build Emacs will trigger the same one.
> And even if it does, report it nicely and someone might
> even fix it right away.

Moreover freedos seems to use routinly doslfn. So it is a least a little bit tested.

Bastien

Re: Files from gnulib

[Bastien ROUCARIES]

Le mercredi 26 janvier 2011 12:52:33, Bastien ROUCARIES a écrit :
> Le mercredi 26 janvier 2011 12:02:47, Jim Meyering a écrit :
> > Eli Zaretskii wrote:
> > >> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> > >> Date: Tue, 25 Jan 2011 23:37:08 +0100
> > >> 
> > >> (if and only if doslfn is buggy, and it does not seems according to
> > >> a quick search).
> > > 
> > > Your search was too quick.
> > 
> > Considering your wish to continue supporting emacs on DOS,
> > I would have thought you would jump at a possible solution like this.
> > If it works (and indications are that it does), then it defines
> > away the whole problem.
> 
> Know bug list is pretty short note
> 
> > Wouldn't you welcome the idea of a DOS
> > port with no risk of 8.3 collisions?
> > 
> > What if the solution really is as easy as it appears?  Here is a
> > 
> > recently-updated FAQ:
> >     http://www-user.tu-chemnitz.de/~heha/hs_freeware/what_lfn.htm.en
> > 
> > Please give it an honest try before dismissing it.
> > 
> > Just because someone wrote about a problem does not
> > mean using it to build Emacs will trigger the same one.
> > And even if it does, report it nicely and someone might
> > even fix it right away.
> 
> Moreover freedos seems to use routinly doslfn. So it is a least a little
> bit tested.
and used by 4dos one of the most usuable dos file manager
http://4dos.isgreat.org

Bastien

Re: Files from gnulib

[Eli Zaretskii]

> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Date: Wed, 26 Jan 2011 12:52:33 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  emacs-devel@gnu.org,
>  eggert@cs.ucla.edu,
>  bug-gnulib@gnu.org,
>  monnier@iro.umontreal.ca
> 
> Moreover freedos seems to use routinly doslfn. So it is a least a
> little bit tested.

FreeDOS has known issues with DJGPP.

Re: Files from gnulib

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Cc: Bastien ROUCARIES <roucaries.bastien@gmail.com>,  emacs-devel@gnu.org,  eggert@cs.ucla.edu,  bug-gnulib@gnu.org,  monnier@iro.umontreal.ca
> Date: Wed, 26 Jan 2011 12:02:47 +0100
> 
> Eli Zaretskii wrote:
> >> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> >> Date: Tue, 25 Jan 2011 23:37:08 +0100
> >>
> >> (if and only if doslfn is buggy, and it does not seems according to
> >> a quick search).
> >
> > Your search was too quick.
> 
> Considering your wish to continue supporting emacs on DOS,
> I would have thought you would jump at a possible solution like this.

The solution would need to be rock solid, for me to jump at it.  DJGPP
is very stable on plain DOS, and its users expect stability.

>     http://www-user.tu-chemnitz.de/~heha/hs_freeware/what_lfn.htm.en
> 
> Please give it an honest try before dismissing it.

It obviously has problems with protected-mode programs and
environments, and with DOS extenders.  DJGPP programs switch the
machine to 32-bit protected mode, so any programs that have
difficulties with that environment are not promising.  And the fact
that DJGPP is not mentioned as a solution for DOS programs that
support LFN does not help to remove the cloud.

Re: Files from gnulib

[Bastien ROUCARIES]

> > Considering your wish to continue supporting emacs on DOS,
> > I would have thought you would jump at a possible solution like this.
> 
> The solution would need to be rock solid, for me to jump at it.  DJGPP
> is very stable on plain DOS, and its users expect stability.
> 
> >     http://www-user.tu-chemnitz.de/~heha/hs_freeware/what_lfn.htm.en
> > 
> > Please give it an honest try before dismissing it.
> 
> It obviously has problems with protected-mode programs and
> environments, and with DOS extenders.  DJGPP programs switch the
> machine to 32-bit protected mode, so any programs that have
> difficulties with that environment are not promising.  And the fact
> that DJGPP is not mentioned as a solution for DOS programs that
> support LFN does not help to remove the cloud.

according to http://www.delorie.com/djgpp/doc/kb/kb_10.html
"The DJGPP library features transparent and automatic support for long file names on Windows 9X(1) The DJGPP startup code queries 
the system for the availability of the LFN API, and if it's available, all low-level file-oriented primitives are automatically 
switched to using the special LFN-aware functions. This run-time detection of the LFN support means that the same executable will 
run on DOS and on Windows, and will automatically support long file names when it runs on Windows 9X"

They are a faq entry on lfn:
http://www.delorie.com/djgpp/v2faq/faq22_16.html
And it seems that lfn is supported since djgpp 2.01 so since October 19, 1996.

according to 
http://www.unet.univie.ac.at/~a0503736/php/drdoswiki/index.php?n=Main.DevelCppBeginners
It could be used under drdos and with djgpp

I agree with you about bug but they seems minor like this
http://permalink.gmane.org/gmane.comp.emulators.freedos.kernel/1372
and could be fixed

doslfn is moreover well supported due to adoption of win95 even on fat12. 

Therefore since 2001 (creation of doslfn), 8.3 name problem should not exist. 

Instead of tackling the main problem lack of 8.3 support and use the new api, you are stick in the 90's. 

Thanks 

Bastien

Re: Files from gnulib

[Eli Zaretskii]

> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Date: Wed, 26 Jan 2011 14:33:54 +0100
> Cc: Jim Meyering <jim@meyering.net>,
>  emacs-devel@gnu.org,
>  eggert@cs.ucla.edu,
>  bug-gnulib@gnu.org,
>  monnier@iro.umontreal.ca
> 
> according to http://www.delorie.com/djgpp/doc/kb/kb_10.html
> "The DJGPP library features transparent and automatic support for long file names on Windows 9X(1) The DJGPP startup code queries 
> the system for the availability of the LFN API, and if it's available, all low-level file-oriented primitives are automatically 
> switched to using the special LFN-aware functions. This run-time detection of the LFN support means that the same executable will 
> run on DOS and on Windows, and will automatically support long file names when it runs on Windows 9X"

Yes, I know.  I wrote some of those "low-level file-oriented primitives".

> They are a faq entry on lfn:
> http://www.delorie.com/djgpp/v2faq/faq22_16.html

Did you see who wrote that FAQ?

> And it seems that lfn is supported since djgpp 2.01 so since October 19, 1996.

That's not the issue, you misunderstood my comment.  The issue is that
DJGPP programs switch the CPU to 32-bit protected mode, and use a DPMI
server to issue real-mode DOS and BIOS system calls from protected
mode.  doslfn seems to have problems with such programs (which is not
surprising, since any TSR that hooks interrupts might have problems in
protected mode).

Re: Files from gnulib

[Bastien ROUCARIES]

Le mercredi 26 janvier 2011 14:48:18, Eli Zaretskii a écrit :
> > "The DJGPP library features transparent and automatic support for long
> > file names on Windows 9X(1) The DJGPP startup code queries the system
> > for the availability of the LFN API, and if it's available, all
> > low-level file-oriented primitives are automatically switched to using
> > the special LFN-aware functions. This run-time detection of the LFN
> > support means that the same executable will run on DOS and on Windows,
> > and will automatically support long file names when it runs on Windows
> > 9X"
> 
> Yes, I know.  I wrote some of those "low-level file-oriented primitives".
>
> > They are a faq entry on lfn:
> > http://www.delorie.com/djgpp/v2faq/faq22_16.html
> 
> Did you see who wrote that FAQ?

Yes you :)

> > And it seems that lfn is supported since djgpp 2.01 so since October 19,
> > 1996.
> 
> That's not the issue, you misunderstood my comment.  The issue is that
> DJGPP programs switch the CPU to 32-bit protected mode, and use a DPMI
> server to issue real-mode DOS and BIOS system calls from protected
> mode.  doslfn seems to have problems with such programs (which is not
> surprising, since any TSR that hooks interrupts might have problems in
> protected mode).

This is the real bug, and I suppose "might have problems" means this bug could be fixed. 

And do not talk about bare metal. DPMI is not bare metal dos*. So we could add another layer, that is doslfn in order to compile 
emacs...

Bastien

* at least in my sense

Re: Files from gnulib

[Eli Zaretskii]

> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Date: Wed, 26 Jan 2011 16:26:05 +0100
> Cc: jim@meyering.net,
>  emacs-devel@gnu.org,
>  eggert@cs.ucla.edu,
>  bug-gnulib@gnu.org,
>  monnier@iro.umontreal.ca
> 
> And do not talk about bare metal. DPMI is not bare metal dos*.

Yes, it is, because DJGPP includes a DPMI host in its development kit,
and each DJGPP program (including Emacs) loads that DPMI host into
memory if it is not already there at startup time.  So you only need a
plain stock DOS system to run DJGPP.

Re: Files from gnulib

[Bastien ROUCARIES]

Le mercredi 26 janvier 2011 16:46:29, Eli Zaretskii a écrit :
> > From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> > Date: Wed, 26 Jan 2011 16:26:05 +0100
> > Cc: jim@meyering.net,
> > 
> >  emacs-devel@gnu.org,
> >  eggert@cs.ucla.edu,
> >  bug-gnulib@gnu.org,
> >  monnier@iro.umontreal.ca
> > 
> > And do not talk about bare metal. DPMI is not bare metal dos*.
> 
> Yes, it is, because DJGPP includes a DPMI host in its development kit,
> and each DJGPP program (including Emacs) loads that DPMI host into
> memory if it is not already there at startup time.  So you only need a
> plain stock DOS system to run DJGPP.

For sure and why not supply a doslfn module (fixed) in djgpp ? 

And you do not answer about:
>This is the real bug, and I suppose "might have problems" means this bug could be fixed. 
I suppose the problem could be fixed because some scsi disk/network disk of this era use tsr and could be browsed from dpmi mode.

Bastien

Re: Files from gnulib

[Eli Zaretskii]

> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Date: Wed, 26 Jan 2011 16:57:10 +0100
> Cc: jim@meyering.net,
>  emacs-devel@gnu.org,
>  eggert@cs.ucla.edu,
>  bug-gnulib@gnu.org,
>  monnier@iro.umontreal.ca
> 
> why not supply a doslfn module (fixed) in djgpp ? 

Because DJGPP has seen its last release 8.5 years ago, and will not
see another one.  Its development is finished.  (There are a couple of
enthusiasts who still work on development, but the development code
will never leave the alpha status, for lack of heavy users to test it
well enough for it to get to release quality.)

> And you do not answer about:
> >This is the real bug, and I suppose "might have problems" means this bug could be fixed. 
> I suppose the problem could be fixed because some scsi disk/network disk of this era use tsr and could be browsed from dpmi mode.

I doubt we will find someone to fix that.

Re: Files from gnulib

[Andy Moreton]

On Wed 26 Jan 2011, Bastien ROUCARIES wrote:

> Therefore since 2001 (creation of doslfn), 8.3 name problem should not
> exist.
>
> Instead of tackling the main problem lack of 8.3 support and use the
> new api, you are stick in the 90's.

It seems odd to me that the developers of a GNU library fight so
hard to avoid their code being portable. The gnulib manual [1] says:

  Gnulib is useful to enhance various aspects of a package:

    * Portability: With Gnulib, a package maintainer can program
      against the POSIX and GNU libc APIs and nevertheless expect good
      portability to platforms that don't implement POSIX.

    * Maintainability: When a package uses modules from Gnulib instead
      of code written specifically for that package, the maintainer has
      less code to maintain.

It seems that manual promotes exactly the features that Eli needs.

    AndyM


[1] http://www.gnu.org/software/gnulib/manual/html_node/Benefits.html#Benefits

Re: Files from gnulib

[Bastien ROUCARIES]

Le mercredi 26 janvier 2011 15:35:52, Andy Moreton a écrit :
> On Wed 26 Jan 2011, Bastien ROUCARIES wrote:
> > Therefore since 2001 (creation of doslfn), 8.3 name problem should not
> > exist.
> > 
> > Instead of tackling the main problem lack of 8.3 support and use the
> > new api, you are stick in the 90's.
> 
> It seems odd to me that the developers of a GNU library fight so
> hard to avoid their code being portable. The gnulib manual [1] says:
> 
>   Gnulib is useful to enhance various aspects of a package:
> 
>     * Portability: With Gnulib, a package maintainer can program
>       against the POSIX and GNU libc APIs and nevertheless expect good
>       portability to platforms that don't implement POSIX.
> 
>     * Maintainability: When a package uses modules from Gnulib instead
>       of code written specifically for that package, the maintainer has
>       less code to maintain.
> 
> It seems that manual promotes exactly the features that Eli needs.
> 
>     AndyM

8.3 limitation is really of another age, and could be lifted on dos is doslfn is fixed. DJGPP need dpmi adding a prerequist of 
doslfn is not so hard in order to compile (not run) emacs.

Bastien

Re: Files from gnulib

[Miles Bader]

Bastien ROUCARIES <roucaries.bastien@gmail.com> writes:
>> It seems odd to me that the developers of a GNU library fight so
>> hard to avoid their code being portable. The gnulib manual [1] says:
>
> 8.3 limitation is really of another age, and could be lifted on dos is
> doslfn is fixed. DJGPP need dpmi adding a prerequist of
> doslfn is not so hard in order to compile (not run) emacs.

Indeed:  "portability" is a good goal in general, but it's a vague word.
There are obviously points at which it can be taken too far, where the
benefit isn't enough to justify the effort required and resulting
obfuscation of the code base.

Where these points are, of course, shifts over time.

[E.g., the once fairly widespread goal of supporting pre-ANSI C
compilers seems to have been largely abandoned these days.]

I'm not saying MSDOS has reached that point yet, just that a goal of
"portability" needs to be judged in the appropriate context.

-miles

-- 
Joy, n. An emotion variously excited, but in its highest degree arising from
the contemplation of grief in another.

Re: Files from gnulib

[Eli Zaretskii]

> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Date: Wed, 26 Jan 2011 16:31:04 +0100
> Cc: Andy Moreton <andrewjmoreton@gmail.com>, emacs-devel@gnu.org
> 
> DJGPP need dpmi adding a prerequist of doslfn is not so hard in
> order to compile (not run) emacs.

Not true.  Once we allow any clashes of file names anywhere in Emacs,
soon enough we will have them in the Lisp files, and then LFNs will be
a necessity for Emacs to run as well, not just to be built.

Re: Files from gnulib

[Simon Josefsson]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Wed 26 Jan 2011, Bastien ROUCARIES wrote:
>
>> Therefore since 2001 (creation of doslfn), 8.3 name problem should not
>> exist.
>>
>> Instead of tackling the main problem lack of 8.3 support and use the
>> new api, you are stick in the 90's.
>
> It seems odd to me that the developers of a GNU library fight so
> hard to avoid their code being portable. The gnulib manual [1] says:
>
>   Gnulib is useful to enhance various aspects of a package:
>
>     * Portability: With Gnulib, a package maintainer can program
>       against the POSIX and GNU libc APIs and nevertheless expect good
>       portability to platforms that don't implement POSIX.
>
>     * Maintainability: When a package uses modules from Gnulib instead
>       of code written specifically for that package, the maintainer has
>       less code to maintain.
>
> It seems that manual promotes exactly the features that Eli needs.

This could probably be clarified in the gnulib manual, but my impression
is that gnulib has always been agreesive about abandoning support for
platforms that are not commercially supported and that we can't access
and test patches on.  For example, I believe SunOS 4 falls into this
category.

/Simon

Re: Files from gnulib

[Paul Eggert]

On 01/27/2011 02:54 AM, Simon Josefsson wrote:

> my impression is that gnulib has always been agreesive about
> abandoning support for platforms that are not commercially supported
> and that we can't access and test patches on.

I'm not sure what "aggressive" means, but the general rule is that we
try not to waste resources maintaining code that cannot easily be
tested and that increases the maintenance burden for ordinary
development.  Another way of putting it is, if the original platform
distributor no longer supports something, that's a good indication
that we shouldn't bother either, as it indicates that the audience is
too small to be worth committing our scarce resources to.

> For example, I believe SunOS 4 falls into this category.

SunOS 4 is still documented as being a target, though this
may change as per the above.  I personally haven't used
SunOS 4 in years; has anyone else?

> This could probably be clarified in the gnulib manual

Here's what it currently says.  Can you suggest which
clarifications would be helpful?

   Originally much of the Gnulib code was portable to ancient hosts like
   4.2BSD, but it is a maintenance hassle to maintain compatibility with
   unused hosts, so currently we assume at least a freestanding C89
   compiler, possibly operating with a C library that predates C89.  The
   oldest environment currently ported to is probably SunOS 4 + GCC 1.x,
   though we haven't tested this exact combination.  SunOS 4 last shipped
   on 1998-09-30, and Sun dropped support for it on 2003-10-01, so at
   some point we may start assuming a C89 library as well.

   ...

   Even if the include files conform to C89, the library itself may not.
   For example, SunOS 4's (free (NULL)) can dump core, so Gnulib code
   must avoid freeing a null pointer, even though C89 allows it.
   You can work around some of these problems by requiring the relevant
   modules, e.g., the Gnulib 'free' module supplies a conforming 'free'.

Re: Files from gnulib

[Bastien ROUCARIES]

[sorry for retrieving this old thread]
On Wed, Jan 26, 2011 at 1:02 PM, Jim Meyering <jim@meyering.net> wrote:
> Eli Zaretskii wrote:
>>> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
>>> Date: Tue, 25 Jan 2011 23:37:08 +0100
>>>
>>> (if and only if doslfn is buggy, and it does not seems according to
>>> a quick search).
>>
>> Your search was too quick.
>
> Considering your wish to continue supporting emacs on DOS,
> I would have thought you would jump at a possible solution like this.
> If it works (and indications are that it does), then it defines
> away the whole problem.  Wouldn't you welcome the idea of a DOS
> port with no risk of 8.3 collisions?
>
> What if the solution really is as easy as it appears?  Here is a
> recently-updated FAQ:
>
>     http://www-user.tu-chemnitz.de/~heha/hs_freeware/what_lfn.htm.en
>
> Please give it an honest try before dismissing it

Moreover doslfn is activelly maintened. The upsteam release a new
version in february. (http://adoxa.3eeweb.com/doslfn/), so time to
drop 8.3 limitation  in emacs build ?

Bastien

Re: Files from gnulib

[Eli Zaretskii]

> Date: Thu, 23 Aug 2012 13:36:16 +0200
> From: Bastien ROUCARIES <roucaries.bastien@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org, eggert@cs.ucla.edu, 
> 	bug-gnulib@gnu.org, monnier@iro.umontreal.ca
> 
> [sorry for retrieving this old thread]

I suggest to let the sleeping dogs lie.