Could we include diff and grep (etc.) executables on Windows?

Could we include diff and grep (etc.) executables on Windows?

[Mathias Dahl]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

Hi,

I sometimes help friends at work trying out Emacs. We're mostly on Windows
and one thing that makes things harder is that some of the useful features
of Emacs uses external tools. I'm thinking specifically about small
utilities like diff, grep, etc. So, on Windows, useful features like Ediff
and the different grep-based tools do not work out of the box.

I see that we bundle some exe files in the bin directory of Emacs (bunzip2,
bzcat, etc.), could we include some of the other core utilities as well?

Of course, if you commit to use Emacs you can make sure to install these
tools in one of several ways it can be done today (Cygwin. MSYS, natively
compiled, etc.) but users could be up and running much faster if they did
not have to do this. If you are trying out Emacs and compare it with other
editors today, where some of this are also built in but work without these
external tools, you might give up.

Any comments and opinions on this? Does anything stop us from bundling a
few more useful utilities to make Emacs even more useful by default? I
think it should benefit many users.

Thanks!

/Mathias

PS. Of course, Emacs has support for many external tools, and one might
ask: where do we stop? But, I think the core features, mentioned above, are
good candidates to include by default.

[-- Attachment #2: Type: text/html, Size: 1521 bytes --]

Re: Could we include diff and grep (etc.) executables on Windows?

[Eli Zaretskii]

> From: Mathias Dahl <mathias.dahl@gmail.com>
> Date: Sun, 17 Nov 2019 12:50:19 +0100
> 
> I see that we bundle some exe files in the bin directory of Emacs (bunzip2, bzcat, etc.), could we include
> some of the other core utilities as well?

It all depends on the good will and free resources of the people who
prepare these binary packages.  (The executables you see are there
because Emacs is linked against the respective libraries, and so all
the contents of the respective binary package is present.)  Adding
more packages means more work for those volunteers.

Re: Could we include diff and grep (etc.) executables on Windows?

[Stefan Monnier]

> Of course, if you commit to use Emacs you can make sure to install these
> tools in one of several ways it can be done today (Cygwin. MSYS, natively
> compiled, etc.) but users could be up and running much faster if they did
> not have to do this. If you are trying out Emacs and compare it with other
> editors today, where some of this are also built in but work without these
> external tools, you might give up.

Maybe streamlining the MSYS way to install Emacs is the easiest/best way
to solve this problem?


        Stefan

Re: Could we include diff and grep (etc.) executables on Windows?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 17 Nov 2019 11:53:42 -0500
> Cc: emacs-devel@gnu.org
> 
> Maybe streamlining the MSYS way to install Emacs is the easiest/best way
> to solve this problem?

Depends on what you mean by that (and what you mean when you say
"MSYS").  In general, quite a few useful programs don't have native
Windows ports in MSYS2 repository, so you will get MSYS ports instead,
and that will cause subtle problems, similar to what happens when one
mixes Cygwin programs with a native w32 Emacs.

Re: Could we include diff and grep (etc.) executables on Windows?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Maybe streamlining the MSYS way to install Emacs is the easiest/best way
>> to solve this problem?
>
> Depends on what you mean by that (and what you mean when you say
> "MSYS").  In general, quite a few useful programs don't have native
> Windows ports in MSYS2 repository, so you will get MSYS ports instead,
> and that will cause subtle problems, similar to what happens when one
> mixes Cygwin programs with a native w32 Emacs.

For the case of diff and grep MSYS2 has native binaries, but I agree
with you in general, although it is perfectly ok to run Emacs without
the MSYS2 binaries in sight.

The real problem is that the MSYS2 approach would add some order of
magnitude to the complexity of packaging and publishing Emacs, so there
is no gain over "simply" adding the bare binaries to the current
packages, quite the contrary.

Re: Could we include diff and grep (etc.) executables on Windows?

[Stefan Monnier]

>>> Maybe streamlining the MSYS way to install Emacs is the easiest/best way
>>> to solve this problem?
>> Depends on what you mean by that (and what you mean when you say "MSYS").

God question.  IIUC one of the main problems with bundling tools is
having to compile them, keep the versions up-to-date, distributing the
sources, etc....

So what I meant mostly was: don't actually bundle them, but just provide
some straightforward way to install Emacs+tools where the tools are
actually fetched from some other place that handles the job of
compiling, keeping them up-to-date, distributing the sources, ...

I assumed MSYS could be that "other place" but apparently that's not
the case.  Maybe that doesn't invalidate the underlying idea, tho.


        Stefan

Re: Could we include diff and grep (etc.) executables on Windows?

[Phillip Lord]

Mathias Dahl <mathias.dahl@gmail.com> writes:
> I sometimes help friends at work trying out Emacs. We're mostly on Windows
> and one thing that makes things harder is that some of the useful features
> of Emacs uses external tools. I'm thinking specifically about small
> utilities like diff, grep, etc. So, on Windows, useful features like Ediff
> and the different grep-based tools do not work out of the box.
>
> I see that we bundle some exe files in the bin directory of Emacs (bunzip2,
> bzcat, etc.), could we include some of the other core utilities as well?
>
> Of course, if you commit to use Emacs you can make sure to install these
> tools in one of several ways it can be done today (Cygwin. MSYS, natively
> compiled, etc.) but users could be up and running much faster if they did
> not have to do this. If you are trying out Emacs and compare it with other
> editors today, where some of this are also built in but work without these
> external tools, you might give up.
>
> Any comments and opinions on this? Does anything stop us from bundling a
> few more useful utilities to make Emacs even more useful by default? I
> think it should benefit many users.
>
> Thanks!
>
> /Mathias
>
> PS. Of course, Emacs has support for many external tools, and one might
> ask: where do we stop? But, I think the core features, mentioned above, are
> good candidates to include by default.

I think, a priori, there would be no particular problem with doing this,
in terms of the packaging and bundling. But there would be a problem in
terms of curating the list of things we added. In addition, there is the
difficulty that the size of the Emacs download will go up; so, in fact,
between Emacs-26 and -27 the number of external packages has dropped
(so, for example, python was in Emacs-26, but is not in Emacs-27). I
made this change because of complaints about size.

Still, the build scripts that I have written could be modified to
support adding additional binaries relatively easily. So, it would be
straight-forward to add diff and grep and see what difference it makes.
Other things on my list would be gunzip and aspell.

But I'd really need a good set of critera for making a decision. I did
think about just greping source for "executable-find", but that fails as
soon as you hit interpreters (python, perl and latex for example). I
could say "called by emacs and relatively small for some value of
small".

Phil

Re: Could we include diff and grep (etc.) executables on Windows?

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Sun, 17 Nov 2019 11:53:42 -0500
>> Cc: emacs-devel@gnu.org
>> 
>> Maybe streamlining the MSYS way to install Emacs is the easiest/best way
>> to solve this problem?
>
> Depends on what you mean by that (and what you mean when you say
> "MSYS").  In general, quite a few useful programs don't have native
> Windows ports in MSYS2 repository, so you will get MSYS ports instead,
> and that will cause subtle problems, similar to what happens when one
> mixes Cygwin programs with a native w32 Emacs.

How do you tell the difference between the two? Are the ones in the
Emacs windows download native?

Phil

Re: Could we include diff and grep (etc.) executables on Windows?

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Maybe streamlining the MSYS way to install Emacs is the easiest/best way
>>>> to solve this problem?
>>> Depends on what you mean by that (and what you mean when you say "MSYS").
>
> God question.  IIUC one of the main problems with bundling tools is
> having to compile them, keep the versions up-to-date, distributing the
> sources, etc....
>
> So what I meant mostly was: don't actually bundle them, but just provide
> some straightforward way to install Emacs+tools where the tools are
> actually fetched from some other place that handles the job of
> compiling, keeping them up-to-date, distributing the sources, ...
>
> I assumed MSYS could be that "other place" but apparently that's not
> the case.  Maybe that doesn't invalidate the underlying idea, tho.


For the current windows downloads I don't actually compile the tools,
just download them (except where source for the current version of a
binary does not exist, which happens rarely but often enough to
irritate).

I don't do anything for "keeping-up-to-date"; in fact, I keep the
dependencies pinnned at an old version for the entire release
cycle. This is obviously both sensible and problematic at the same time.

Phil

Re: Could we include diff and grep (etc.) executables on Windows?

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org,
>   mathias.dahl@gmail.com
> Date: Mon, 18 Nov 2019 15:49:44 +0000
> 
> > Depends on what you mean by that (and what you mean when you say
> > "MSYS").  In general, quite a few useful programs don't have native
> > Windows ports in MSYS2 repository, so you will get MSYS ports instead,
> > and that will cause subtle problems, similar to what happens when one
> > mixes Cygwin programs with a native w32 Emacs.
> 
> How do you tell the difference between the two?

The native ports have "mingw" in their package names.

> Are the ones in the Emacs windows download native?

Yes, of course.  You cannot link a native Windows build of Emacs
against MSYS libraries, you'd get unresolved externals because the
MSYS DLL is not present on the link command line.

Re: Could we include diff and grep (etc.) executables on Windows?

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Mon, 18 Nov 2019 15:48:48 +0000
> Cc: emacs-devel@gnu.org
> 
> But I'd really need a good set of critera for making a decision. I did
> think about just greping source for "executable-find", but that fails as
> soon as you hit interpreters (python, perl and latex for example).

Grep for "-program".

Re: Could we include diff and grep (etc.) executables on Windows?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Maybe streamlining the MSYS way to install Emacs is the easiest/best way
>>>> to solve this problem?
>>> Depends on what you mean by that (and what you mean when you say "MSYS").
>
> God question.  IIUC one of the main problems with bundling tools is
> having to compile them, keep the versions up-to-date, distributing the
> sources, etc....
>
> So what I meant mostly was: don't actually bundle them, but just provide
> some straightforward way to install Emacs+tools where the tools are
> actually fetched from some other place that handles the job of
> compiling, keeping them up-to-date, distributing the sources, ...
>
> I assumed MSYS could be that "other place" but apparently that's not
> the case.  Maybe that doesn't invalidate the underlying idea, tho.

There is an emacs package currently in MSYS2, with both mingw-w64 and
msys2 versions (for emacs 26.2). I have not tried them (mainly because
they didn't used to be there), so I don't know what other packages/tools
they install.

There are also emacs packages in Cygwin, for text and X11 (for emacs
26.3); I have not tried those in a long time.

So "die-hard" mingw or cygwin users are already covered. That leaves the
Gnu distribution of Emacs for Windows to cover the remaining users,
which are probably a very diverse lot.

I compile emacs from source (usually master, sometimes the release
branch), and use MSYS2/mingw64 for all tools not provided by emacs (as
do many others). For tools not yet in MSYS2 (primarily LaTeX), I use
Cygwin. This process does have some downsides, but I find it the best
way.

-- 
-- Stephe

Re: Could we include diff and grep (etc.) executables on Windows?

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> God question.  IIUC one of the main problems with bundling tools is
>> having to compile them, keep the versions up-to-date, distributing the
>> sources, etc....
>>
>> So what I meant mostly was: don't actually bundle them, but just provide
>> some straightforward way to install Emacs+tools where the tools are
>> actually fetched from some other place that handles the job of
>> compiling, keeping them up-to-date, distributing the sources, ...
>>
>> I assumed MSYS could be that "other place" but apparently that's not
>> the case.  Maybe that doesn't invalidate the underlying idea, tho.
>
> There is an emacs package currently in MSYS2, with both mingw-w64 and
> msys2 versions (for emacs 26.2). I have not tried them (mainly because
> they didn't used to be there), so I don't know what other packages/tools
> they install.
>
> There are also emacs packages in Cygwin, for text and X11 (for emacs
> 26.3); I have not tried those in a long time.
>
> So "die-hard" mingw or cygwin users are already covered. That leaves the
> Gnu distribution of Emacs for Windows to cover the remaining users,
> which are probably a very diverse lot.
>
> I compile emacs from source (usually master, sometimes the release
> branch), and use MSYS2/mingw64 for all tools not provided by emacs (as
> do many others). For tools not yet in MSYS2 (primarily LaTeX), I use
> Cygwin. This process does have some downsides, but I find it the best
> way.


Be good to know what you mean by "all the tools". That would form the
basis of a list that we might install.

I have created a branch called feature/windows-with-utils. I am happy
for people to add new packages like so:

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index 5698f51..5df7694 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -40,6 +40,11 @@ mingw-w64-x86_64-libxml2
 mingw-w64-x86_64-xpm-nox'''.split()
 
 
+## This list is some extra things that we want to add because they are useful
+PKG_REQ=PKG_REQ+'''
+mingw-w64-x86_64-diffutils'''.split()
+
+

That would give me an idea of how much bigger the download would get. If
it's just a bit, then good. If it's a lot, I think, we need two yet more
optional downloads which would be a little unfortunate.

Phil

Re: Could we include diff and grep (etc.) executables on Windows?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 17 Nov 2019 18:49:46 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Maybe streamlining the MSYS way to install Emacs is the easiest/best way
> >> to solve this problem?
> >
> > Depends on what you mean by that (and what you mean when you say
> > "MSYS").  In general, quite a few useful programs don't have native
> > Windows ports in MSYS2 repository, so you will get MSYS ports instead,
> > and that will cause subtle problems, similar to what happens when one
> > mixes Cygwin programs with a native w32 Emacs.
> 
> For the case of diff and grep MSYS2 has native binaries

But that's about all, so it seems.  There are no Coreutils, no
Findutils, no Grep, no Sed, etc.

> The real problem is that the MSYS2 approach would add some order of
> magnitude to the complexity of packaging and publishing Emacs, so there
> is no gain over "simply" adding the bare binaries to the current
> packages, quite the contrary.

That's another problem, yes.

Re: Could we include diff and grep (etc.) executables on Windows?

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> For the case of diff and grep MSYS2 has native binaries
>
> But that's about all, so it seems.  There are no Coreutils, no
> Findutils, no Grep, no Sed, etc.

MSYS2 already has native Sed, Grep and Diff. Findutils would be doable.
Dunno about Coreutils.

Nowadays I just install some new fancy searcher (such as ag, the Silver
Searcher) and that covers the explicit Findutils+Grep case. IIRC there
are places on Emacs that use Findutils+Grep under the hood (Dired?
Ediff?). AFAIR never missed Sed or Coreutils on Emacs on Windows.

Re: Could we include diff and grep (etc.) executables on Windows?

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Tue, 19 Nov 2019 21:32:40 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> For the case of diff and grep MSYS2 has native binaries
> >
> > But that's about all, so it seems.  There are no Coreutils, no
> > Findutils, no Grep, no Sed, etc.
> 
> MSYS2 already has native Sed, Grep

It does?  Maybe I'm blind, but "pacman -Ss" disagrees, it only shows
msys packages for these.

> Findutils would be doable.

Doable, yes.  But not easy, if you want 'find', 'locate', and 'xargs'
to work: they use fork/exec/waitpid, take inode equality to mean
identical files/directories, expect you to be able to pass quoted
wildcards, etc.  That's why years ago I had to do my own port, because
the existing ones were either broken, or abysmally slow (or both).

> Nowadays I just install some new fancy searcher (such as ag, the Silver
> Searcher) and that covers the explicit Findutils+Grep case. IIRC there
> are places on Emacs that use Findutils+Grep under the hood (Dired?
> Ediff?).

grep-find and find-dired come to mind, and I think Cedet.  But we also
call xargs here and there.

> AFAIR never missed Sed or Coreutils on Emacs on Windows.

You don't use Dired?