64 bit official Windows builds

64 bit official Windows builds

[Sam Halliday]

Dear Emacs Users,

Are there any plans for GNU to create official 64 bit builds of Emacs releases? I  understand we use mingw to cross-compile, I'm used it to build 64 bit Windows applications in the past (from a GNU/Linux build machine) with great success. Is there any technical reason why we can't do this for Emacs?

(I'm aware of unofficial builds, and I am not interested because they are hosted on unreliable websites such as sourceforge.)

Best regards,
Sam

Re: 64 bit official Windows builds

[Eli Zaretskii]

> Date: Thu, 24 Dec 2015 15:01:53 -0800 (PST)
> From: Sam Halliday <sam.halliday@gmail.com>
> 
> Are there any plans for GNU to create official 64 bit builds of
> Emacs releases?

You mean, for Windows?  Or for all 64-bit platforms?

If the former, this simply needs a volunteer who'd be prepared to
produce the binaries, package them in a compressed archive, set up
uploading rights to the GNU servers, and upload the stuff whenever a
new release is out.

> I  understand we use mingw to cross-compile

MinGW is not a cross compiler, at least not when it runs on Windows
natively.  It's a native Windows port of GNU development tools (GCC
and Binutils) that runs on MS-Windows, targets MS-Windows, and uses
the Windows runtime libraries for its C library.

> I'm used it to build 64 bit Windows applications in the past (from a GNU/Linux build machine) with great success. Is there any technical reason why we can't do this for Emacs?

A 64-bit build using MinGW64 already works, and several people use it
all the time.  If you have MinGW64 installed, you can compile Emacs
with it any time you want.  Uploading the binaries requires a
volunteer to step forward, but there are no technical problems that
would prevent this.

Re: 64 bit official Windows builds

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 24 Dec 2015 15:01:53 -0800 (PST)
>> From: Sam Halliday <sam.halliday@gmail.com>
>> 
>> Are there any plans for GNU to create official 64 bit builds of
>> Emacs releases?
>
> You mean, for Windows?  Or for all 64-bit platforms?
>
> If the former, this simply needs a volunteer who'd be prepared to
> produce the binaries, package them in a compressed archive, set up
> uploading rights to the GNU servers, and upload the stuff whenever a
> new release is out.

IIRC the problematic part is to create and distribute a source tarball
with all the libraries included on the binary package (graphic
libraries, SSL, etc). This was discussed on the past and can't recall
the outcome.

>> I  understand we use mingw to cross-compile
>
> MinGW is not a cross compiler, at least not when it runs on Windows
> natively.  It's a native Windows port of GNU development tools (GCC
> and Binutils) that runs on MS-Windows, targets MS-Windows, and uses
> the Windows runtime libraries for its C library.

MinGW is used on GNU/Linux as a cross compiler for creating Windows
binaries. Maybe the OP is thinking on this scenario. He is not aware of
the peculiar build procedure of Emacs, which requires dumping a running
binary image (maybe this could be achieved running temacs under Wine,
but modifications to the build scripts are required).

[snip]

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 25 Dec 2015 14:35:46 +0100
> 
> IIRC the problematic part is to create and distribute a source tarball
> with all the libraries included on the binary package (graphic
> libraries, SSL, etc). This was discussed on the past and can't recall
> the outcome.

It's up to the person who volunteers for the job.  It's quite okay to
upload only the Emacs binary that was compiled with support for the
optional libraries, and expect the end users to download and itsall
the DLLs separately.  It is also okay to upload the DLLs as part of
the binary distro, but then the corresponding sources should be on the
same site.

> MinGW is used on GNU/Linux as a cross compiler for creating Windows
> binaries. Maybe the OP is thinking on this scenario. He is not aware of
> the peculiar build procedure of Emacs, which requires dumping a running
> binary image (maybe this could be achieved running temacs under Wine,
> but modifications to the build scripts are required).

It's a general problem with cross-compiling Emacs, yes.

Re: 64 bit official Windows builds

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>> MinGW is used on GNU/Linux as a cross compiler for creating Windows
>> binaries. Maybe the OP is thinking on this scenario. He is not aware of
>> the peculiar build procedure of Emacs, which requires dumping a running
>> binary image (maybe this could be achieved running temacs under Wine,
>> but modifications to the build scripts are required).
>
> It's a general problem with cross-compiling Emacs, yes.

Would it be viable to shift this step to an installer process,
for cross-compiled builds? Maybe even in general, rather than
just for Windows.

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Fri, 25 Dec 2015 10:32:29 -0500
> 
> Would it be viable to shift this step to an installer process,
> for cross-compiled builds? Maybe even in general, rather than
> just for Windows.

You mean, distribute temacs, and make it dump emacs at installation
time?  It could be done, I see no technical issues with that, although
I think it would be a very unusual thing to do.

I don't see the motivation, though, certainly not for Windows: the
existing build procedure works reliably on the target host, so making
changes to distributions for the benefit of cross-compilation makes
little sense to me.  But I have no experience in packaging.

Re: 64 bit official Windows builds

[Sam Halliday]

On Friday, 25 December 2015 07:35:25 UTC, Eli Zaretskii  wrote:
> > Date: Thu, 24 Dec 2015 15:01:53 -0800 (PST)
> > From: Sam Halliday <sam.halliday@gmail.com>
> > 
> > Are there any plans for GNU to create official 64 bit builds of
> > Emacs releases?
> 
> this simply needs a volunteer who'd be prepared to
> produce the binaries, package them in a compressed archive, set up
> uploading rights to the GNU servers, and upload the stuff whenever a
> new release is out.

I'm happy (this is a relative term when Windoze is involved) to create the builds. I have a virtual Windows image that I use for testing. Can somebody please point me at the exact instructions that I need to follow? Most of my computing experience is with GNU/Linux, so knowing exactly how to set up the build environment would be useful, although I can fumble my way to getting the GNU chain installed, I'm sure.


Incidentally, http://www.appveyor.com/ can be used in continuous integration for testing on Windows. I use it in my free software projects and its a real time saver. If GNU Emacs was hosted on github it would be possible to trigger a Windows test on every push to master (this might be possible as a mirror). I use a combination of appveyor and a self-hosted https://github.com/drone/drone/ to test, build and deploy our continual delivery releases for all major platforms.

So, ideally, I'd like to get the GNU Emacs windows builds running off an appveyor script so it should work with the click of a button.

Re: 64 bit official Windows builds

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Fri, 25 Dec 2015 14:35:46 +0100
>> 
>> IIRC the problematic part is to create and distribute a source tarball
>> with all the libraries included on the binary package (graphic
>> libraries, SSL, etc). This was discussed on the past and can't recall
>> the outcome.
>
> It's up to the person who volunteers for the job.  It's quite okay to
> upload only the Emacs binary that was compiled with support for the
> optional libraries, and expect the end users to download and itsall
> the DLLs separately.  It is also okay to upload the DLLs as part of
> the binary distro, but then the corresponding sources should be on the
> same site.

I think that providing bare Emacs binaries without the corresponding
dll's is not really user friendly.  I build Emacs on my Win 64bit
machine with Msys2/MinGW-w64 and it would be pain if I had to collect all
dll's myself somehow.  OTOH, collecting and providing all the sources
along with the dll's is also not fun.  Can there be a compromise?  For
Msys2/MinGW-w64, all PKGBUILD files contain references the sources, e.g.:

    https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-libidn/PKGBUILD

So one could say: Consult the PKGBUILD files for the sources
(incl. dependencies) for

- mingw-w64-libtiff
- mingw-w64-giflib
- mingw-w64-libpng
- mingw-w64-libjpeg-turbo
- mingw-w64-librsvg
- mingw-w64-libxml2
- mingw-w64-gnutls
- mingw-w64-xpm-nox

Best, Arash

Re: 64 bit official Windows builds

[Rasmus]

Hi Sam,

Sam Halliday <sam.halliday@gmail.com> writes:

> If GNU Emacs was hosted on github it would be possible to trigger a
> Windows test on every push to master (this might be possible as a
> mirror).

What CI have to do with github?

Org and Gnus are hosted on their own git servers, but have continues
integration via buildbot, curtsy of David Engster.

Rasmus

-- 
In theory, practice and theory are the same. In practice they are not

Re: 64 bit official Windows builds

[Eli Zaretskii]

> Date: Thu, 7 Jan 2016 14:38:38 -0800 (PST)
> From: Sam Halliday <sam.halliday@gmail.com>
> 
> On Friday, 25 December 2015 07:35:25 UTC, Eli Zaretskii  wrote:
> > > Date: Thu, 24 Dec 2015 15:01:53 -0800 (PST)
> > > From: Sam Halliday <sam.halliday@gmail.com>
> > > 
> > > Are there any plans for GNU to create official 64 bit builds of
> > > Emacs releases?
> > 
> > this simply needs a volunteer who'd be prepared to
> > produce the binaries, package them in a compressed archive, set up
> > uploading rights to the GNU servers, and upload the stuff whenever a
> > new release is out.
> 
> I'm happy (this is a relative term when Windoze is involved) to create the builds. I have a virtual Windows image that I use for testing. Can somebody please point me at the exact instructions that I need to follow? Most of my computing experience is with GNU/Linux, so knowing exactly how to set up the build environment would be useful, although I can fumble my way to getting the GNU chain installed, I'm sure.

You will find the instructions in nt/INSTALL and nt/INSTALL.W64 in the
Emacs Git repository.

> Incidentally, http://www.appveyor.com/ can be used in continuous integration for testing on Windows. I use it in my free software projects and its a real time saver. If GNU Emacs was hosted on github it would be possible to trigger a Windows test on every push to master (this might be possible as a mirror). I use a combination of appveyor and a self-hosted https://github.com/drone/drone/ to test, build and deploy our continual delivery releases for all major platforms.
> 
> So, ideally, I'd like to get the GNU Emacs windows builds running off an appveyor script so it should work with the click of a button.

Thanks, but these issues are better discussed on emacs-devel@gnu.org,
not here.

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Arash Esbati <esbati@gmx.de>
> Date: Thu, 07 Jan 2016 23:47:00 +0100
> 
> I think that providing bare Emacs binaries without the corresponding
> dll's is not really user friendly.  I build Emacs on my Win 64bit
> machine with Msys2/MinGW-w64 and it would be pain if I had to collect all
> dll's myself somehow.

If you build your own Emacs, you already have all those DLLs
installed, right?  So you don't need to collect them, right?

> OTOH, collecting and providing all the sources
> along with the dll's is also not fun.  Can there be a compromise?  For
> Msys2/MinGW-w64, all PKGBUILD files contain references the sources, e.g.:
> 
>     https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-libidn/PKGBUILD
> 
> So one could say: Consult the PKGBUILD files for the sources
> (incl. dependencies) for
> 
> - mingw-w64-libtiff
> - mingw-w64-giflib
> - mingw-w64-libpng
> - mingw-w64-libjpeg-turbo
> - mingw-w64-librsvg
> - mingw-w64-libxml2
> - mingw-w64-gnutls
> - mingw-w64-xpm-nox

No, this compromise contradicts the GPL.  The sources must be
available from the same place as the binaries, because otherwise it
isn't practical for the user who wants to rebuild a DLL (e.g., to fix
a bug in it or add a feature) of the exact version used to build
Emacs.  (If she tries to do that with a different version, that
version might be incompatible with the specific version of Emacs she
uses.)  For the same reason, the source distribution found near the
binary should be of the exact same version used to produce the binary,
and include any changes done by whoever built the binary.

This indeed is not trivial to do, which is why the decision whether to
provide the optional libraries together with Emacs is entirely up to
the person who volunteers for the job.  It's not an easy job even if
the libraries aren't included.

Re: 64 bit official Windows builds

[Sam Halliday]

On Friday, 8 January 2016 09:12:20 UTC, Eli Zaretskii  wrote:
> No, this compromise contradicts the GPL.  The sources must be
> available from the same place as the binaries


Are you sure about this? My understanding is that source code is to be provided if it is requested:

  http://www.gnu.org/licenses/gpl-faq.en.html#WhatDoesWrittenOfferValid


I believe a GNU Emacs 64bit Windows download should be self-contained and include all runtime DLLs that are necessary for it to run.

Re: 64 bit official Windows builds

[Eli Zaretskii]

> Date: Fri, 8 Jan 2016 02:44:26 -0800 (PST)
> From: Sam Halliday <sam.halliday@gmail.com>
> Injection-Date: Fri, 08 Jan 2016 10:44:26 +0000
> 
> On Friday, 8 January 2016 09:12:20 UTC, Eli Zaretskii  wrote:
> > No, this compromise contradicts the GPL.  The sources must be
> > available from the same place as the binaries
> 
> 
> Are you sure about this?

Yes, quite sure.  Other ways are theoretically possible, but they are
all impractical.

> My understanding is that source code is to be provided if it is requested:
> 
>   http://www.gnu.org/licenses/gpl-faq.en.html#WhatDoesWrittenOfferValid

Are you (you personally) really going to provide a service of this
kind?  And request that anyone who receives the binaries will have to
provide the same service as well?  Sounds unlikely.  Actually, sounds
like a lip service to me.  And it still requires you to have the
sources handy, for when someone requests them, because you cannot
control what is available at any given time on other sites.

> I believe a GNU Emacs 64bit Windows download should be self-contained and include all runtime DLLs that are necessary for it to run.

That's fine, and I don't disagree.  You just have to provide the
sources used to compile those DLLs from the same place.  Which
shouldn't be a problem, at least not theoretically, since either you
have built them yourself, or you downloaded the binaries from some
GPL-compatible site, which then must provide the sources they used,
right?

FWIW, I do this for 32-bit builds of all the optional libraries needed
by Emacs, see here:

  https://sourceforge.net/projects/ezwinports/files/?source=navbar

It's possible, and it's practically doable.  It is more work than just
building Emacs, though.

Re: 64 bit official Windows builds

[Arash Esbati]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arash Esbati <esbati@gmx.de>
>> Date: Thu, 07 Jan 2016 23:47:00 +0100
>> 
>> I think that providing bare Emacs binaries without the corresponding
>> dll's is not really user friendly.  I build Emacs on my Win 64bit
>> machine with Msys2/MinGW-w64 and it would be pain if I had to collect all
>> dll's myself somehow.
>
> If you build your own Emacs, you already have all those DLLs
> installed, right?  So you don't need to collect them, right?

My apologies for the unclear sentence.  I meant: I build Emacs on my Win
64bit machine with Msys2/MinGW-w64 and have access to the necessary
DLLs; it would be a pain if I only had bare Emacs binaries and had to
collect all those DLLs myself somehow.

>> [...]
>> So one could say: Consult the PKGBUILD files for the sources
>> (incl. dependencies) for
>> 
>> - mingw-w64-libtiff
>> - mingw-w64-giflib
>> - mingw-w64-libpng
>> - mingw-w64-libjpeg-turbo
>> - mingw-w64-librsvg
>> - mingw-w64-libxml2
>> - mingw-w64-gnutls
>> - mingw-w64-xpm-nox
>
> No, this compromise contradicts the GPL.  The sources must be
> available from the same place as the binaries,

Hmm, I read the FAQ differently:

--8<---------------cut here---------------start------------->8---
Can I put the binaries on my Internet server and put the source on a
different Internet site? (#SourceAndBinaryOnDifferentSites)

    Yes. Section 6(d) allows this. However, you must provide clear
    instructions people can follow to obtain the source, and you must
    take care to make sure that the source remains available for as long
    as you distribute the object code.

http://www.gnu.org/licenses/gpl-faq.en.html#SourceAndBinaryOnDifferentSites
--8<---------------cut here---------------end--------------->8---

I admit that the "you must take care to make sure that the source
remains available for as long as you distribute the object code." part
makes things more complicated, but still ...

> because otherwise it isn't practical for the user who wants to rebuild
> a DLL (e.g., to fix a bug in it or add a feature) of the exact version
> used to build Emacs.  (If she tries to do that with a different
> version, that version might be incompatible with the specific version
> of Emacs she uses.)  For the same reason, the source distribution
> found near the binary should be of the exact same version used to
> produce the binary, and include any changes done by whoever built the
> binary.

True, from a developer point of view.  But not required by GPL if I get
it correctly.

Please, don't get me wrong here, I do understand your point.  But from a
user point of view, I think Emacs becomes more attractive on Windows
if it is provided as a self-contained binary package.

Best, Arash

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Arash Esbati <esbati@gmx.de>
> Date: Fri, 08 Jan 2016 22:42:20 +0100
> 
> My apologies for the unclear sentence.  I meant: I build Emacs on my Win
> 64bit machine with Msys2/MinGW-w64 and have access to the necessary
> DLLs; it would be a pain if I only had bare Emacs binaries and had to
> collect all those DLLs myself somehow.

It's some work, but I wouldn't describe that as "pain".  Depends on
how the sites that offer those DLLs are organized and what tools they
offer for downloading and installation.

> > No, this compromise contradicts the GPL.  The sources must be
> > available from the same place as the binaries,
> 
> Hmm, I read the FAQ differently:
> 
> --8<---------------cut here---------------start------------->8---
> Can I put the binaries on my Internet server and put the source on a
> different Internet site? (#SourceAndBinaryOnDifferentSites)
> 
>     Yes. Section 6(d) allows this. However, you must provide clear
>     instructions people can follow to obtain the source, and you must
>     take care to make sure that the source remains available for as long
>     as you distribute the object code.
> 
> http://www.gnu.org/licenses/gpl-faq.en.html#SourceAndBinaryOnDifferentSites
> --8<---------------cut here---------------end--------------->8---
> 
> I admit that the "you must take care to make sure that the source
> remains available for as long as you distribute the object code." part
> makes things more complicated, but still ...

Not just "complicated", practically impossible.  You have no control
on what another site holds, and for how long.  They could decide to
upload a "fixed" archive, which no longer builds for that version of
Emacs, or is no longer compatible to it.

> > because otherwise it isn't practical for the user who wants to rebuild
> > a DLL (e.g., to fix a bug in it or add a feature) of the exact version
> > used to build Emacs.  (If she tries to do that with a different
> > version, that version might be incompatible with the specific version
> > of Emacs she uses.)  For the same reason, the source distribution
> > found near the binary should be of the exact same version used to
> > produce the binary, and include any changes done by whoever built the
> > binary.
> 
> True, from a developer point of view.  But not required by GPL if I get
> it correctly.

The GPL gives users and developers the same rights.  A user who cannot
by herself change the program can hire someone who can.  That someone
will be a developer who will need the exact sources you used.

> Please, don't get me wrong here, I do understand your point.  But from a
> user point of view, I think Emacs becomes more attractive on Windows
> if it is provided as a self-contained binary package.

I agree.  I'm just saying that providing such a self-contained binary
means more work on the part of the person who provides that.  I know
that, because that's what I do when I upload packages to the
ezwinports site -- each binary zip contains all of its dependency
DLLs, and there's always a source zip for each of those dependencies,
in the same directory, or sometimes in the sibling directory.  That's
what the GPL requires.

Re: 64 bit official Windows builds

[Sam Halliday]

Arash,

It sounds like you know how to do this a lot better than I do.

Is there any chance you could package this up so that everybody can get your 64 bit builds of Emacs for Windows from the FSF site? If the sources are listed already, maybe you could just create a tarball containing all the sources of the compiler and required libraries? 

It would be really good from an automation point of view if you could write a little recipe (maybe an appveyor script?) that could be triggered when new versions of emacs (or the compiler) are released.

Best regards,
Sam

On Thursday, 7 January 2016 22:48:10 UTC, Arash Esbati  wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Óscar Fuentes <ofv@wanadoo.es>
> >> Date: Fri, 25 Dec 2015 14:35:46 +0100
> >> 
> >> IIRC the problematic part is to create and distribute a source tarball
> >> with all the libraries included on the binary package (graphic
> >> libraries, SSL, etc). This was discussed on the past and can't recall
> >> the outcome.
> >
> > It's up to the person who volunteers for the job.  It's quite okay to
> > upload only the Emacs binary that was compiled with support for the
> > optional libraries, and expect the end users to download and itsall
> > the DLLs separately.  It is also okay to upload the DLLs as part of
> > the binary distro, but then the corresponding sources should be on the
> > same site.
> 
> I think that providing bare Emacs binaries without the corresponding
> dll's is not really user friendly.  I build Emacs on my Win 64bit
> machine with Msys2/MinGW-w64 and it would be pain if I had to collect all
> dll's myself somehow.  OTOH, collecting and providing all the sources
> along with the dll's is also not fun.  Can there be a compromise?  For
> Msys2/MinGW-w64, all PKGBUILD files contain references the sources, e.g.:
> 
>     https://github.com/Alexpux/MINGW-packages/blob/master/mingw-w64-libidn/PKGBUILD
> 
> So one could say: Consult the PKGBUILD files for the sources
> (incl. dependencies) for
> 
> - mingw-w64-libtiff
> - mingw-w64-giflib
> - mingw-w64-libpng
> - mingw-w64-libjpeg-turbo
> - mingw-w64-librsvg
> - mingw-w64-libxml2
> - mingw-w64-gnutls
> - mingw-w64-xpm-nox
> 
> Best, Arash

Re: 64 bit official Windows builds

[Arash Esbati]

Sam Halliday <sam.halliday@gmail.com> writes:

> It sounds like you know how to do this a lot better than I do.

No, not really.  I just managed to build Emacs after a recipe you can
find here:

    http://sourceforge.net/p/emacsbinw64/wiki/Build%20guideline%20for%20MSYS2-MinGW-w64%20system/

and now here as well:

    http://git.savannah.gnu.org/cgit/emacs.git/tree/nt/INSTALL.W64?h=emacs-25

After the discussion here, I looked closer at how Msys2/MinGW-w64
manages its packages.

    http://sourceforge.net/p/msys2/wiki/Contributing%20to%20MSYS2/
    https://github.com/Alexpux/MINGW-packages

The optional packages for building Emacs are:

    mingw-w64-libtiff
    mingw-w64-giflib
    mingw-w64-libpng
    mingw-w64-libjpeg-turbo
    mingw-w64-librsvg
    mingw-w64-libxml2
    mingw-w64-gnutls
    mingw-w64-xpm-nox

and their dependencies are: (`pactree --help')

    mingw-w64-bzip2
    mingw-w64-cairo
    mingw-w64-expat
    mingw-w64-fontconfig
    mingw-w64-freetype
    mingw-w64-gdk-pixbuf2
    mingw-w64-gettext
    mingw-w64-glib2
    mingw-w64-gmp
    mingw-w64-harfbuzz
    mingw-w64-jasper
    mingw-w64-libcroco
    mingw-w64-libffi
    mingw-w64-libiconv
    mingw-w64-lzo2
    mingw-w64-pango
    mingw-w64-pixman
    mingw-w64-xz
    mingw-w64-zlib

It is possible to build source archives incl. the patches coming from
Msys2 project with `makepkg-mingw' command.  The process is start the
`msys2_shell' and do:

--8<---------------cut here---------------start------------->8---
   git clone "https://github.com/Alexpux/MINGW-packages"
   cd MINGW-packages/mingw-w64-libtiff
   MINGW_INSTALLS=mingw64 makepkg-mingw --nobuild -sL
   MINGW_INSTALLS=mingw64 makepkg-mingw --allsource -sLf
--8<---------------cut here---------------end--------------->8---

I tried this and it worked for all the packages except for
`mingw-w64-gettext', `mingw-w64-gmp' and `mingw-w64-gnutls'.  I have to
take a closer look.  So I think it would be possible to provide source
tarballs for the DLLs.

> Is there any chance you could package this up so that everybody can
> get your 64 bit builds of Emacs for Windows from the FSF site? If the
> sources are listed already, maybe you could just create a tarball
> containing all the sources of the compiler and required libraries?

Sources of the compiler?  Do you mean gcc?  No, I won't go there :-)

> It would be really good from an automation point of view if you could
> write a little recipe (maybe an appveyor script?) that could be
> triggered when new versions of emacs (or the compiler) are released.

I'm not familiar with appveyor.

Best, Arash

Re: 64 bit official Windows builds

[moocow062]

On Friday, December 25, 2015 at 12:01:56 AM UTC+1, Sam Halliday wrote:
 
> Are there any plans for GNU to create official 64 bit builds of Emacs releases? I  understand we use mingw to cross-compile, I'm used it to build 64 bit Windows applications in the past (from a GNU/Linux build machine) with great success. Is there any technical reason why we can't do this for Emacs?

Could I ask, what is the benefit of 64 bits compared to 32 bits for windows ?
I can see that if you have to edit files bigger than 2Gb, it is necessary, but otherwise doubling the size of the pointers just makes it slower (halves the cache size) and takes more memory.

Re: 64 bit official Windows builds

[Eli Zaretskii]

> Date: Mon, 8 Feb 2016 09:44:53 -0800 (PST)
> From: moocow062@gmail.com
> 
> Could I ask, what is the benefit of 64 bits compared to 32 bits for windows ?

It will run roughly twice faster.

> I can see that if you have to edit files bigger than 2Gb, it is necessary, but otherwise doubling the size of the pointers just makes it slower (halves the cache size) and takes more memory.

First, you can have several large buffers, each one smaller than 2GB,
but all of them together could weigh in at, like, 4GB or more,
something a 32-bit executable, even when running on a 64-bit Windows,
cannot have.

Second, large files happen more frequently nowadays than you may
think.

Third, when Emacs is built with a 64-bit compiler, it runs faster, not
slower, because running a 32-bit executable on a 64-bit Windows
requires expensive thunking for every call to any Windows API,
something that happens a lot.

Btw, Emacs 25 can be built as a 32-bit binary that supports 64-bit
ints, which allows large buffers (up to 2GB), at a price of being
slower by about 30%.

Re: 64 bit official Windows builds

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> Third, when Emacs is built with a 64-bit compiler, it runs faster, not
> slower, because running a 32-bit executable on a 64-bit Windows
> requires expensive thunking for every call to any Windows API,
> something that happens a lot.

For a pointer-chasing program like Emacs, data cache effects are orders
of magnitude more expensive than thunking. That's what I observe with
similar applications. As for Emacs, I see no performance difference
among 32 bit and 64 bit executables, for ordinary use. That's how it
should be: a good interactive application is never supposed to make the
user wait, and Emacs does a decent job at that.

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 08 Feb 2016 19:29:29 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Third, when Emacs is built with a 64-bit compiler, it runs faster, not
> > slower, because running a 32-bit executable on a 64-bit Windows
> > requires expensive thunking for every call to any Windows API,
> > something that happens a lot.
> 
> For a pointer-chasing program like Emacs, data cache effects are orders
> of magnitude more expensive than thunking. That's what I observe with
> similar applications. As for Emacs, I see no performance difference
> among 32 bit and 64 bit executables, for ordinary use. That's how it
> should be: a good interactive application is never supposed to make the
> user wait, and Emacs does a decent job at that.

Try some CPU intensive processing, or a command that does a lot of
disk I/O.

A program such as GNU Find runs more than twice faster when compiled
as a 64-bit application.

Re: 64 bit official Windows builds

[djc]

> Could I ask, what is the benefit of 64 bits compared to 32 bits for windows ?

In emacs I count things that total more than a 32-bit counter can hold, and I routinely edit multiple files of hundreds of megabytes.  Pure 32-bit emacs on Windows simply can't do the former, and is v-e-r-r-y slow at the latter, to the point of being unusable.  As a 64-bit program it does just fine, and overall it is, as Eli has point out, a whole lot faster.  64-bit Windows builds came along just in time to save me enormous effort programming around the limitations of the 32-bit version.

Re: 64 bit official Windows builds

[Óscar Fuentes]

djc <peter.kaiser@gmail.com> writes:

>> Could I ask, what is the benefit of 64 bits compared to 32 bits for windows ?
>
> In emacs I count things that total more than a 32-bit counter can
> hold, and I routinely edit multiple files of hundreds of megabytes.
> Pure 32-bit emacs on Windows simply can't do the former, and is
> v-e-r-r-y slow at the latter, to the point of being unusable.

Are you using the same versions for your 32/64 bits performance
comparisions. IIRC handling of large buffers and large long lines
received some improvements not too long ago.

I can't not depict an scenario where changing to the 64 bits ISA could
change Emacs' speed from very slow to acceptable. If something like that
were observed (on the same Emacs source code version) I would suspect
some difference on the C runtime or on the compiler (intrinsics not
lowered, optimizations not applied on 32 bit code, etc.)

Re: 64 bit official Windows builds

[Stefan Monnier]

>> Could I ask, what is the benefit of 64 bits compared to 32 bits for windows ?
> It will run roughly twice faster.
[...]
> slower, because running a 32-bit executable on a 64-bit Windows
> requires expensive thunking for every call to any Windows API,

Holy crap?  Really?  It's hard to believe that 64-bit Windows's
emulation of the 32-bit API is so inefficient that it causes a "rough
slowdown" by a factor 2 in an application like Emacs.

Do you see such a factor-of-2 difference when doing "rm lisp/**/*.elc;
make"?  Or in which kind of circumstance have you seen such a slowdown?


        Stefan

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 11 Feb 2016 08:58:45 -0500
> 
> >> Could I ask, what is the benefit of 64 bits compared to 32 bits for windows ?
> > It will run roughly twice faster.
> [...]
> > slower, because running a 32-bit executable on a 64-bit Windows
> > requires expensive thunking for every call to any Windows API,
> 
> Holy crap?  Really?  It's hard to believe that 64-bit Windows's
> emulation of the 32-bit API is so inefficient that it causes a "rough
> slowdown" by a factor 2 in an application like Emacs.
> 
> Do you see such a factor-of-2 difference when doing "rm lisp/**/*.elc;
> make"?  Or in which kind of circumstance have you seen such a slowdown?

I measured that by running GNU Find compiled from the same sources on
a large and deep directory tree on the same Windows 7 box.

That thunking is the culprit is my theory, not a fact; however, I
cannot find any other explanation.  If someone does, I'm all ears.

Re: 64 bit official Windows builds

[Nicolas Petton]

[-- Attachment #1: Type: text/plain, Size: 230 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
>  Uploading the binaries requires a volunteer to step forward, but
> there are no technical problems that would prevent this.

I will upload them if someone is willing to do the builds.

Nico

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: 64 bit official Windows builds

[Kaushal Modi]

Copying Chris Zheng (emacsbinw64 sourceforge project) to see if he is
interested in providing the binaries.

https://sourceforge.net/p/emacsbinw64/tickets/9/

On Thu, Feb 11, 2016 at 4:22 PM Nicolas Petton <nicolas@petton.fr> wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
> >  Uploading the binaries requires a volunteer to step forward, but
> > there are no technical problems that would prevent this.
>
> I will upload them if someone is willing to do the builds.
>
> Nico
>

Re: 64 bit official Windows builds

[John Mastro]

Kaushal Modi <kaushal.modi@gmail.com> wrote:
> Copying Chris Zheng (emacsbinw64 sourceforge project) to see if he is
> interested in providing the binaries.
>
> https://sourceforge.net/p/emacsbinw64/tickets/9/

I'll take the opportunity to say thank you, Chris, for providing those
builds. I've been using them for several months and they've been great.

-- 
john

Re: 64 bit official Windows builds

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Do you see such a factor-of-2 difference when doing "rm lisp/**/*.elc;
>> make"?  Or in which kind of circumstance have you seen such a slowdown?
>
> I measured that by running GNU Find compiled from the same sources on
> a large and deep directory tree on the same Windows 7 box.
>
> That thunking is the culprit is my theory, not a fact; however, I
> cannot find any other explanation.  If someone does, I'm all ears.

I mentioned some possibilities on a previous message. Did you use the
same toolset and libraries for the 32 and 64 bits build? The MinGW and
MinGW-w64 (32/64 bits) runtimes diverged quite a bit.

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 11 Feb 2016 23:16:35 +0100
> 
> > I measured that by running GNU Find compiled from the same sources on
> > a large and deep directory tree on the same Windows 7 box.
> >
> > That thunking is the culprit is my theory, not a fact; however, I
> > cannot find any other explanation.  If someone does, I'm all ears.
> 
> I mentioned some possibilities on a previous message. Did you use the
> same toolset and libraries for the 32 and 64 bits build?

No.  The program was compiled by mingw.org's MinGW for 32 bits and by
MinGW64 for 64 bits.

> The MinGW and MinGW-w64 (32/64 bits) runtimes diverged quite a bit.

GNU Find uses only msvcrt.dll, no other runtime libraries are involved
in any significant way.

Re: 64 bit official Windows builds

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > That thunking is the culprit is my theory, not a fact; however, I
>> > cannot find any other explanation.  If someone does, I'm all ears.
>> 
>> I mentioned some possibilities on a previous message. Did you use the
>> same toolset and libraries for the 32 and 64 bits build?
>
> No.  The program was compiled by mingw.org's MinGW for 32 bits and by
> MinGW64 for 64 bits.

There you have a strong candidate for explaining the difference. That
probably also means that they were different compiler versions.

>> The MinGW and MinGW-w64 (32/64 bits) runtimes diverged quite a bit.
>
> GNU Find uses only msvcrt.dll, no other runtime libraries are involved
> in any significant way.

As you know, there are other code pieces that are linked into the
executable besides the C runtime (which MinGW(-w64) supersede by
providing their implementations for certain functions, plus other
features missing from msvcrt.dll). IIRC some *stat functions are very
slow on Mingw, maybe the MinGW-w64 guys introduced improvements, just a
guess.

Why don't you build both 32 and 64 bits executables of GNU Find with
MinGW-w64 (same toolset version) for comparing its performance?

Not saying that GNU Find will be representative of what you can expect
from Emacs. (GNU Find: I/O bound; Emacs: user bound.)

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 12 Feb 2016 08:16:37 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > That thunking is the culprit is my theory, not a fact; however, I
> >> > cannot find any other explanation.  If someone does, I'm all ears.
> >> 
> >> I mentioned some possibilities on a previous message. Did you use the
> >> same toolset and libraries for the 32 and 64 bits build?
> >
> > No.  The program was compiled by mingw.org's MinGW for 32 bits and by
> > MinGW64 for 64 bits.
> 
> There you have a strong candidate for explaining the difference. That
> probably also means that they were different compiler versions.

I find it hard to believe that compiler version differences can
explain a factor of two.  It contradicts every bit of my experience
with GCC over the last 30 years.

> >> The MinGW and MinGW-w64 (32/64 bits) runtimes diverged quite a bit.
> >
> > GNU Find uses only msvcrt.dll, no other runtime libraries are involved
> > in any significant way.
> 
> As you know, there are other code pieces that are linked into the
> executable besides the C runtime (which MinGW(-w64) supersede by
> providing their implementations for certain functions, plus other
> features missing from msvcrt.dll). IIRC some *stat functions are very
> slow on Mingw, maybe the MinGW-w64 guys introduced improvements, just a
> guess.

Not according to the current MinGW64's Git repository.  They basically
simply call the msvcrt _stat.

And I doubt such a speedup is really possible at all, given the
simplistic implementation of 'stat' in msvcrt -- you cannot do less,
really.

> Why don't you build both 32 and 64 bits executables of GNU Find with
> MinGW-w64 (same toolset version) for comparing its performance?

Sorry, I don't have time for that.  But anyone who is interested can
do this experiment, the sources (and the binaries) are on the
ezwinports site.  FWIW, I'd be very glad to hear that my measurements
were some fluke and should be disregarded.

> Not saying that GNU Find will be representative of what you can expect
> from Emacs. (GNU Find: I/O bound; Emacs: user bound.)

Performance only matters when you do prolonged operations.  One such
prolonged operation in Emacs is reading a directory in Dired, in which
case what Emacs does is quite similar to what Find does.  For someone
who uses Dired extensively, the GNU Find example is not irrelevant.

Memory- and CPU-intensive operations is another matter.  But here,
too, I'd welcome actual measurements more than theories.  Measurements
can and do surprise, as is known to anyone who ever profiled a
real-life program.

Re: 64 bit official Windows builds

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > No.  The program was compiled by mingw.org's MinGW for 32 bits and by
>> > MinGW64 for 64 bits.
>> 
>> There you have a strong candidate for explaining the difference. That
>> probably also means that they were different compiler versions.
>
> I find it hard to believe that compiler version differences can
> explain a factor of two.  It contradicts every bit of my experience
> with GCC over the last 30 years.

You missed the "also". The compiler version can have a dramatic impact
on some cases (it is quite common to find 10x differences on some
microbenchmarks) but I'm more prone to point fingers to what is around
of the compiler, something that I hinted on my previous message.

>> As you know, there are other code pieces that are linked into the
>> executable besides the C runtime (which MinGW(-w64) supersede by
>> providing their implementations for certain functions, plus other
>> features missing from msvcrt.dll). IIRC some *stat functions are very
>> slow on Mingw, maybe the MinGW-w64 guys introduced improvements, just a
>> guess.
>
> Not according to the current MinGW64's Git repository.  They basically
> simply call the msvcrt _stat.

Okay, let's scratch that hypothesis then.

>> Why don't you build both 32 and 64 bits executables of GNU Find with
>> MinGW-w64 (same toolset version) for comparing its performance?
>
> Sorry, I don't have time for that.  But anyone who is interested can
> do this experiment, the sources (and the binaries) are on the
> ezwinports site.  FWIW, I'd be very glad to hear that my measurements
> were some fluke and should be disregarded.

Maybe I'll try the comparision, but at the moment I only have virtual
machines with Windows 64 bits and those are not specially reliable for
performance measurements, although a 2x difference on real metal should
have some impact on the VM too.

>> Not saying that GNU Find will be representative of what you can expect
>> from Emacs. (GNU Find: I/O bound; Emacs: user bound.)
>
> Performance only matters when you do prolonged operations.  One such
> prolonged operation in Emacs is reading a directory in Dired, in which
> case what Emacs does is quite similar to what Find does.  For someone
> who uses Dired extensively, the GNU Find example is not irrelevant.

Are there reports about Dired being slow on Windows 32 bits? Just
curious.

> Memory- and CPU-intensive operations is another matter.  But here,
> too, I'd welcome actual measurements more than theories.  Measurements
> can and do surprise, as is known to anyone who ever profiled a
> real-life program.

I'm glad you think this way. So now we have agreed that the existence of
a dramatic performance gap between 32 and 64 bits Emacs executables and
its cause being API thunking is just a theory of yours based on limited
evidence :-)

My also (limited) evidence is that 64 bit Windows binaries can be a bit
faster than 32 bit ones, and vice-versa. For instance: my experience
building complex C++ code with GCC is that the 32 bit compiler runs a
bit faster than its 64 bit version. For Emacs, I see no difference, it
is responsive on both systems.

Re: 64 bit official Windows builds

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 12 Feb 2016 09:16:13 +0100
> 
> >> Not saying that GNU Find will be representative of what you can expect
> >> from Emacs. (GNU Find: I/O bound; Emacs: user bound.)
> >
> > Performance only matters when you do prolonged operations.  One such
> > prolonged operation in Emacs is reading a directory in Dired, in which
> > case what Emacs does is quite similar to what Find does.  For someone
> > who uses Dired extensively, the GNU Find example is not irrelevant.
> 
> Are there reports about Dired being slow on Windows 32 bits? Just
> curious.

Compared to what? a Posix host that runs 'ls' instead? you betcha!
Comparing that to a 64-bit Emacs would be a good measurement, I think.
I didn't try that.

> > Memory- and CPU-intensive operations is another matter.  But here,
> > too, I'd welcome actual measurements more than theories.  Measurements
> > can and do surprise, as is known to anyone who ever profiled a
> > real-life program.
> 
> I'm glad you think this way. So now we have agreed that the existence of
> a dramatic performance gap between 32 and 64 bits Emacs executables and
> its cause being API thunking is just a theory of yours based on limited
> evidence :-)

I actually said that already in my reply to Stefan, didn't I?

> My also (limited) evidence is that 64 bit Windows binaries can be a bit
> faster than 32 bit ones, and vice-versa. For instance: my experience
> building complex C++ code with GCC is that the 32 bit compiler runs a
> bit faster than its 64 bit version. For Emacs, I see no difference, it
> is responsive on both systems.

Responsiveness is not what is at stake here, as I already wrote.  You
need some prolonged operation, either I/O intensive or CPU- or
memory-intensive (or both).  Try repeatedly searching a regexp through
a large buffer, or run XML validation on a large XML document, or
anything similar.