HarfBuzz is available on MS-Windows

HarfBuzz is available on MS-Windows

[Eli Zaretskii]

The harfbuzz branch can now be built on MS-Windows, and will support
the new 'harfbuzz' font backend if the requisite DLLs are installed.

Work still continues on the branch in preparation for landing it on
master at a later date, but I'd like at this time to ask people to try
building the branch on Windows and report any problems they see.  Once
the branch is built, you can test that it's working by typing "C-h h"
and seeing whether HELLO displays correctly.

By default, the 'harfbuzz' backend is preferred to the other ones, so
if everything is OK, "C-u C-x =" on any character in HELLO should show
"harfbuzz" at the beginning of the font name used to display the
character.  To force Emacs to use another backend (e.g., for
comparison of the display), you can use the -xrm command line option.
For example:

   emacs -Q -xrm Emacs.fontBackend:uniscribe

will force Emacs to use Uniscribe, the previous default text shaping
engine, as the backend.

HarfBuzz DLLs are available from the MSYS2 project and from ezwinports
(the latter only for 32-bit builds).

Thanks in advance for helping to test the branch.

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks in advance for helping to test the branch.

What advantages does this backend provide over the previous one?

What can I do to test that harfbuzz is working as expected on areas
where the difference should be observable?

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 31 May 2019 16:25:32 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Thanks in advance for helping to test the branch.
> 
> What advantages does this backend provide over the previous one?

Its shaping engine has fewer bugs and is actively developed to keep it
up to date with recent developments in fonts and support for complex
scripts.

In the future, I hope we will be able to enable advanced features like
ligatures and color emoji, but this is not yet in the code base.

> What can I do to test that harfbuzz is working as expected on areas
> where the difference should be observable?

It depends on what scripts you use routinely.  If you are using Latin
scripts and little else, just verifying that you see no strange
display artifacts, and that HELLO displays correctly, is enough for
now.

TIA

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Thanks in advance for helping to test the branch.
>> 
>> What advantages does this backend provide over the previous one?
>
> Its shaping engine has fewer bugs and is actively developed to keep it
> up to date with recent developments in fonts and support for complex
> scripts.
>
> In the future, I hope we will be able to enable advanced features like
> ligatures and color emoji, but this is not yet in the code base.
>
>> What can I do to test that harfbuzz is working as expected on areas
>> where the difference should be observable?
>
> It depends on what scripts you use routinely.  If you are using Latin
> scripts and little else, just verifying that you see no strange
> display artifacts, and that HELLO displays correctly, is enough for
> now.

Thanks. Just built and installed emacs on MSYS2/Mingw-w64/64 and it
doesn't start, no output whatsoever. Tried gdb and the output was quite
strange:

$ gdb emacs.exe
GNU gdb (GDB) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from emacs.exe...
(No debugging symbols found in emacs.exe)
(gdb) r
Starting program: C:\apps\msys64\mingw64\bin\emacs.exe
[New Thread 2480.0x7b0]
warning: C
warning: a
warning: n
warning: n
warning: o
warning: t
warning:
warning: o
warning: p
warning: e
warning: n
warning:
warning: l
warning: o
warning: a
warning: d
warning:
warning: f
warning: i
warning: l
warning: e
warning: :
warning:
warning: N
warning: o
warning:
warning: s
warning: u
warning: c
warning: h
warning:
warning: f
warning: i
warning: l
warning: e
warning:
warning: o
warning: r
warning:
warning: d
warning: i
warning: r
warning: e
warning: c
warning: t
warning: o
warning: r
warning: y
warning: ,
warning:
warning: u
warning: r
warning: l
warning: -
warning: h
warning: a
warning: n
warning: d
warning: l
warning: e
warning: r
warning: s
warning:
Cannot open load file: No such file or directory, url-handlers
[Thread 2480.0x7b0 exited with code 0]
[Inferior 1 (process 2480) exited with code 037777777777]
(gdb)


However, -Q works. Will bisect my .emacs later tonight.

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Thanks. Just built and installed emacs on MSYS2/Mingw-w64/64 and it
> doesn't start, no output whatsoever. Tried gdb and the output was quite
> strange:
>
> $ gdb emacs.exe
> GNU gdb (GDB) 8.3
> Copyright (C) 2019 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-w64-mingw32".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
>     <http://www.gnu.org/software/gdb/documentation/>.
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from emacs.exe...
> (No debugging symbols found in emacs.exe)
> (gdb) r
> Starting program: C:\apps\msys64\mingw64\bin\emacs.exe
> [New Thread 2480.0x7b0]
> warning: C
> warning: a
> warning: n
> warning: n
> warning: o
> warning: t
> warning:
> warning: o
> warning: p
> warning: e
> warning: n
> warning:
> warning: l
> warning: o
> warning: a
> warning: d
> warning:
> warning: f
> warning: i
> warning: l
> warning: e
> warning: :
> warning:
> warning: N
> warning: o
> warning:
> warning: s
> warning: u
> warning: c
> warning: h
> warning:
> warning: f
> warning: i
> warning: l
> warning: e
> warning:
> warning: o
> warning: r
> warning:
> warning: d
> warning: i
> warning: r
> warning: e
> warning: c
> warning: t
> warning: o
> warning: r
> warning: y
> warning: ,
> warning:
> warning: u
> warning: r
> warning: l
> warning: -
> warning: h
> warning: a
> warning: n
> warning: d
> warning: l
> warning: e
> warning: r
> warning: s
> warning:
> Cannot open load file: No such file or directory, url-handlers
> [Thread 2480.0x7b0 exited with code 0]
> [Inferior 1 (process 2480) exited with code 037777777777]
> (gdb)
>
>
> However, -Q works. Will bisect my .emacs later tonight.

The problem persist after commenting-out all my .emacs. However, noticed
that on `emacs -Q .emacs` if I evaluate (package initialize) (which is
the first line in my .emacs, courtesy of Package.el) this error is
thrown:


Debugger entered--Lisp error: (file-missing "Cannot open load file" #("No such file or directory" 0 25 (charset windows-1252)) "url-handlers")
  require(url-handlers)
  byte-code("\301\302!\210\301\303!\210\301\304!\210\301\305!\210\301\306!\210\307\310\311\312\313\314\315\316&\7\210\317\320\321\322\323DD\324\325\326\315\316&\7\210\317\327\321\322..." [package-user-dir require cl-lib seq tabulated-list macroexp url-handlers custom-declare-group package nil "Manager for Emacs Lisp packages." :group applications :version "24.1" custom-declare-variable package-enable-at-startup funcall function #f(compiled-function () #<bytecode 0xbc8589>) "Whether to make installed packages available when ..." :type boolean package-load-list #f(compiled-function () #<bytecode 0xbcffa9>) "List of packages for `package-initialize' to make ..." (repeat (choice (const all) (list :tag "Specific package" (symbol :tag "Package name") (choice :tag "Version" (const :tag "disable" nil) (const :tag "most recent" t) (string :tag "specific version"))))) :risky t package-archives #f(compiled-function () #<bytecode 0xbdcd31>) "An alist of archives from which to fetch.\nThe defa..." (alist :key-type (string :tag "Archive name") :value-type (string :tag "URL or directory name")) "26.1" package-menu-hide-low-priority #f(compiled-function () #<bytecode 0xbdf1b1>) "If non-nil, hide low priority packages from the pa..." (choice (const :tag "Don't hide anything" nil) (const :tag "Hide per package-archive-priorities" archive) (const :tag "Hide per archive and version number" t)) "25.1" package-archive-priorities #f(compiled-function () #<bytecode 0xbe7471>) "An alist of priorities for packages.\n\nEach element..." (alist :key-type (string :tag "Archive name") :value-type (integer :tag "Priority (default is 0)")) package-pinned-packages #f(compiled-function () #<bytecode 0xbe742d>) "An alist of packages that are pinned to specific a..." (alist :key-type (symbol :tag "Package") :value-type (string :tag "Archive name")) "24.4" #f(compiled-function () #<bytecode 0xbe7805>) "Directory containing the user's Emacs Lisp package..." ...] 12)
  (package-initialize)
  eval-region(254 274 t #f(compiled-function (ignore) #<bytecode 0x15f7cfd>))  ; Reading at buffer position 279
  elisp--eval-defun()
  eval-defun(nil)
  funcall-interactively(eval-defun nil)
  call-interactively(eval-defun nil nil)
  command-execute(eval-defun)

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 31 May 2019 19:13:13 +0200
> 
> if I evaluate (package initialize) (which is
> the first line in my .emacs, courtesy of Package.el) this error is
> thrown:
> 
> 
> Debugger entered--Lisp error: (file-missing "Cannot open load file" #("No such file or directory" 0 25 (charset windows-1252)) "url-handlers")
>   require(url-handlers)

Do you have that file in that tree?

On my system, evaluating (require 'url-handlers) works without any
problems in that branch.

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Fri, 31 May 2019 19:13:13 +0200
>> 
>> if I evaluate (package initialize) (which is
>> the first line in my .emacs, courtesy of Package.el) this error is
>> thrown:
>> 
>> 
>> Debugger entered--Lisp error: (file-missing "Cannot open load file" #("No such file or directory" 0 25 (charset windows-1252)) "url-handlers")
>>   require(url-handlers)
>
> Do you have that file in that tree?
>
> On my system, evaluating (require 'url-handlers) works without any
> problems in that branch.

It fails on this build. The file is present (and its correspondent .elc)
in the source tree, it is installed on the i686 build, but not in the
x86_64 build.

This is not the first weird thing that I experience building packages on
this machine on the last weeks. Will try a different one.

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Fri, 31 May 2019 18:44:14 +0200
> 
> Thanks. Just built and installed emacs on MSYS2/Mingw-w64/64 and it
> doesn't start, no output whatsoever. Tried gdb and the output was quite
> strange:

It reads: Cannot open load file: No such file or directory, url-handlers

So you have lisp/url/url-handlers.el{c} in that tree?

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> The harfbuzz branch can now be built on MS-Windows, and will support
> the new 'harfbuzz' font backend if the requisite DLLs are installed.
>
> Work still continues on the branch in preparation for landing it on
> master at a later date, but I'd like at this time to ask people to try
> building the branch on Windows and report any problems they see.  Once
> the branch is built, you can test that it's working by typing "C-h h"
> and seeing whether HELLO displays correctly.
>
> By default, the 'harfbuzz' backend is preferred to the other ones, so
> if everything is OK, "C-u C-x =" on any character in HELLO should show
> "harfbuzz" at the beginning of the font name used to display the
> character.

Finally got it working, but the font name on M-x describe-char still is
prepended by "uniscribe". Harfbuzz is installed and the configure script
knows about it [1], although I haven't seen the point where it is
specifically tested for presence.

1: this is a typical gcc invocation taken from config.log. Note -isystem
... harfbuzz:

  16123:configure:19686: gcc -I /d/dev/other/MINGW-packages/mingw-w64-emacs-git/src/emacs/nt/inc -o conftest.exe -march=x86-64 -mtune=generic -O2 -pipe -pthread -mms-bitfields -isystem C:/apps/msys64/mingw64/include/librsvg-2.0 -isystem C:/apps/msys64/mingw64/include/gdk-pixbuf-2.0 -isystem C:/apps/msys64/mingw64/include/libpng16 -isystem C:/apps/msys64/mingw64/include -isystem C:/apps/msys64/mingw64/include/cairo -isystem C:/apps/msys64/mingw64/include -isystem C:/apps/msys64/mingw64/lib/libffi-3.2.1/include -isystem C:/apps/msys64/mingw64/include/pixman-1 -isystem C:/apps/msys64/mingw64/include -isystem C:/apps/msys64/mingw64/include/freetype2 -isystem C:/apps/msys64/mingw64/include -isystem C:/apps/msys64/mingw64/include/harfbuzz -isystem C:/apps/msys64/mingw64/include/glib-2.0 -isystem C:/apps/msys64/mingw64/lib/glib-2.0/include -isystem C:/apps/msys64/mingw64/include -isystem C:/apps/msys64/mingw64/include/libpng16 -isystem C:/apps/msys64/mingw64/include -mtune=generic  -D_FORTIFY_SOURCE=2 -D__USE_MINGW_ANSI_STDIO=1 -pipe conftest.c   >&5 

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Finally got it working, but the font name on M-x describe-char still is
> prepended by "uniscribe".

Nevermind.  Building the wrong branch.

HELLO looks normal. Everything looks the same so far.

As for the install problems on my other e-mail, it seems that a second
build on the same source tree causes some files to be missed by `make
install'. If I delete the src directory, the resulting package works. If
not, term/w32-terminal.el(c) (possibly among others) are missing.

Dunno if this is a MSYS2 problem or the build is buggy.

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 01 Jun 2019 00:10:12 +0200
> 
> HELLO looks normal. Everything looks the same so far.

OK, thanks.

Re: HarfBuzz is available on MS-Windows

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

> The harfbuzz branch can now be built on MS-Windows, and will support
> the new 'harfbuzz' font backend if the requisite DLLs are installed.
>
> HarfBuzz DLLs are available from the MSYS2 project and from ezwinports
> (the latter only for 32-bit builds).


What are those dependencies? I'd like to add them the scripts that build
the dependency zips for windows on that branch, so that I don't have to
remember to do it when the merge happens!

Phil

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Cc: emacs-devel@gnu.org
> Date: Sun, 02 Jun 2019 19:46:28 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The harfbuzz branch can now be built on MS-Windows, and will support
> > the new 'harfbuzz' font backend if the requisite DLLs are installed.
> >
> > HarfBuzz DLLs are available from the MSYS2 project and from ezwinports
> > (the latter only for 32-bit builds).
> 
> What are those dependencies? I'd like to add them the scripts that build
> the dependency zips for windows on that branch, so that I don't have to
> remember to do it when the merge happens!

Sorry, I don't understand the question.  The HarfBuzz DLLs aren't
different from any other DLL you download from MSYS2, so the same
procedure should do, and should bring you all the DLLs on which
HarfBuzz depends.  Which are those depends on how the MSYS2 people
built HarfBuzz; they generally specify everything that's possible,
even if it makes little sense on Windows, but that's their decision.

If you refer to "the requisite DLLs" I mentioned, that was a reference
to libharfbuzz-0.dll itself, not to its dependencies.  The code loads
only that single DLL.

Re: HarfBuzz is available on MS-Windows

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> What are those dependencies? I'd like to add them the scripts that build
>> the dependency zips for windows on that branch, so that I don't have to
>> remember to do it when the merge happens!
>
> Sorry, I don't understand the question.  The HarfBuzz DLLs aren't
> different from any other DLL you download from MSYS2, so the same
> procedure should do, and should bring you all the DLLs on which
> HarfBuzz depends.  Which are those depends on how the MSYS2 people
> built HarfBuzz; they generally specify everything that's possible,
> even if it makes little sense on Windows, but that's their decision.
>
> If you refer to "the requisite DLLs" I mentioned, that was a reference
> to libharfbuzz-0.dll itself, not to its dependencies.  The code loads
> only that single DLL.

Sorry for lack of clarity. Yes, I meant the direct dependencies -- I
just need to add the package of libharfbuzz-0.dll, if that is the only
DLL loaded. pacman will do the rest.

Phil

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Sorry for lack of clarity. Yes, I meant the direct dependencies -- I
> just need to add the package of libharfbuzz-0.dll, if that is the only
> DLL loaded. pacman will do the rest.

The package is mingw-w64-[x86_64/i686]-harfbuzz. I updated the PKGBUILD
for emacs-git but my PR is still pending.

I also moved to optdepends almost every library dependency of
emacs[-git].

Looking at their respective PKGBUILDs, it seems that there is a cyclic
dependency among harfbuzz and freetype, which brings in libpng and
bzip2. HarfBuzz itself also depends on glib2.

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 02 Jun 2019 22:36:51 +0200
> 
> Looking at their respective PKGBUILDs, it seems that there is a cyclic
> dependency among harfbuzz and freetype

This circular dependency is a known issue; Freetype needs to be
rebuilt once HarfBuzz is built and installed, if you need Freetype as
a separate library, to be called directly.  We don't need to call
Freetype directly from Emacs on Windows, so this isn't an issue for
Emacs on Windows.

> which brings in libpng and bzip2.

libpng and bzip2 are dependencies of Freetype regardless of HarfBuzz,
AFAIK.

> HarfBuzz itself also depends on glib2.

That's an optional dependency that Emacs doesn't need at all, but
MSYS2 folks decided to include it.  In general, Glib is needed by
HarfBuzz only in order to run its test suite.

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Looking at their respective PKGBUILDs, it seems that there is a cyclic
>> dependency among harfbuzz and freetype
>
> This circular dependency is a known issue; Freetype needs to be
> rebuilt once HarfBuzz is built and installed, if you need Freetype as
> a separate library, to be called directly.  We don't need to call
> Freetype directly from Emacs on Windows, so this isn't an issue for
> Emacs on Windows.
>
>> which brings in libpng and bzip2.
>
> libpng and bzip2 are dependencies of Freetype regardless of HarfBuzz,
> AFAIK.

Thanks.

>> HarfBuzz itself also depends on glib2.
>
> That's an optional dependency that Emacs doesn't need at all, but
> MSYS2 folks decided to include it.  In general, Glib is needed by
> HarfBuzz only in order to run its test suite.

I looked at Debian and Arch Linux package descpriptions and it seems
that glib2 is a runtime dependency for providing GObject bindings.
HarfBuzz can be compiled without glib2, but then those bindings are not
available.

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

On June 3, 2019 6:27:40 AM GMT+03:00, "Óscar Fuentes" <ofv@wanadoo.es> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> HarfBuzz itself also depends on glib2.
> >
> > That's an optional dependency that Emacs doesn't need at all, but
> > MSYS2 folks decided to include it.  In general, Glib is needed by
> > HarfBuzz only in order to run its test suite.
> 
> I looked at Debian and Arch Linux package descpriptions and it seems
> that glib2 is a runtime dependency for providing GObject bindings.
> HarfBuzz can be compiled without glib2, but then those bindings are
> not
> available.

AFAIK, GObject bindings are needed only for Pango, something that's unlikely to be useful on Windows.

Re: HarfBuzz is available on MS-Windows

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> I looked at Debian and Arch Linux package descpriptions and it seems
>> that glib2 is a runtime dependency for providing GObject bindings.
>> HarfBuzz can be compiled without glib2, but then those bindings are
>> not
>> available.
>
> AFAIK, GObject bindings are needed only for Pango, something that's
> unlikely to be useful on Windows.

Well, MSYS2 has Pango as a package. There are several other packages
that depend on it.

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

On June 3, 2019 7:07:22 AM GMT+03:00, "Óscar Fuentes" <ofv@wanadoo.es> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I looked at Debian and Arch Linux package descpriptions and it
> seems
> >> that glib2 is a runtime dependency for providing GObject bindings.
> >> HarfBuzz can be compiled without glib2, but then those bindings are
> >> not
> >> available.
> >
> > AFAIK, GObject bindings are needed only for Pango, something that's
> > unlikely to be useful on Windows.
> 
> Well, MSYS2 has Pango as a package. There are several other packages
> that depend on it.

Yes, and everybody else pays for that.  It might be educational to find out which of those other packages actually need complex script shaping with HarfBuzz, and cannot be satisfied by Uniscribe; I'm guessing zero.

That's what the "one fits all" methods always boil down to: 99.99% of users pay for the needs of 0.01%.

Re: HarfBuzz is available on MS-Windows

[Tak Kunihiro]

The build with MSYS2 works good on my PC. I see no problem.

MSYS2$ pacman -S  mingw-w64-x86_64-harfbuzz
CMD> C:\emacs\emacs-27\bin\runemacs -Q
M-x about-emacs
GNU Emacs 27.0.50 (build 14, x86_64-w64-mingw32)
M-x view-hello-file
C-u C-x =
harfbuzz:-outline-MS Gothic-normal-normal-normal-mono-13-*-*-*-c-*-jisx0208*-* (#x887)

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Tak Kunihiro <homeros.misasa@gmail.com>
> Date: Tue, 04 Jun 2019 13:51:58 +0900
> Cc: tkk@misasa.okayama-u.ac.jp, emacs-devel@gnu.org
> 
> The build with MSYS2 works good on my PC. I see no problem.
> 
> MSYS2$ pacman -S  mingw-w64-x86_64-harfbuzz
> CMD> C:\emacs\emacs-27\bin\runemacs -Q
> M-x about-emacs
> GNU Emacs 27.0.50 (build 14, x86_64-w64-mingw32)
> M-x view-hello-file
> C-u C-x =
> harfbuzz:-outline-MS Gothic-normal-normal-normal-mono-13-*-*-*-c-*-jisx0208*-* (#x887)

Thank you very much for trying the branch.

Re: HarfBuzz is available on MS-Windows

[Andy Moreton]

On Fri 31 May 2019, Eli Zaretskii wrote:

> The harfbuzz branch can now be built on MS-Windows, and will support
> the new 'harfbuzz' font backend if the requisite DLLs are installed.
>
> Work still continues on the branch in preparation for landing it on
> master at a later date, but I'd like at this time to ask people to try
> building the branch on Windows and report any problems they see.  Once
> the branch is built, you can test that it's working by typing "C-h h"
> and seeing whether HELLO displays correctly.

I've tested this with an MSYS2 64bit build on Windows 10. It appears to
work ok, but is noticeably slower compared to the master branch. I also
found the following minor issues:

a) On harfbuzz and master branches from "emacs -Q", a machine without
the Symbola font does not display the emoji U+1F44B WAVING HAND SIGN (no
font available). Babelmap shows that this character is available using
the built-in "Segoe UI Symbol" or "Segoe UI Emoji" fonts (or by
installing Symbola).

b) On the harfbuzz branch from "emacs -Q", the lao U+EC3 LAO VOWEL SIGN
AY and U+EC3 LAO VOWEL SIGN O characters are not displayed (no font
available). Babelmap shows that this character is available using the
built-in "Leelawadee UI" font.


I use the following to speed up finding built-in fonts on Windows 10:

  (pcase-dolist
      (`(,font-spec . ,targets)
       '(;; Unicode blocks ---------------------------------------
         ("Segoe UI Emoji"
          (#x1f900 . #x1f9ff)) ; Supplemental Symbols and Pictographs
         ;; Unicode scripts --------------------------------------
         ("Segoe UI Symbol"      braille mathematical symbol)
         ("Leelawadee UI"        khmer thai lao)
         ("Nirmala UI"           bengali devanagari gujarati kannada
          malayalam oriya sinhala tamil telugu)
         ("Microsoft Himalaya"   tibetan)
         ("Myanmar Text"         burmese)
         ("Ebrima"               ethiopic)
         ("Gadugi"               canadian-aboriginal cherokee)))
    (dolist (target targets)
      (set-fontset-font "fontset-default" target font-spec nil 'prepend)))

Perhaps the built in fonts should be added to the default mappings for
Windows 10.

    AndyM

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 07 Jun 2019 18:05:23 +0100
> 
> I've tested this with an MSYS2 64bit build on Windows 10.

Thanks, you can now just build the master branch.

> It appears to work ok, but is noticeably slower compared to the
> master branch.

I didn't see any tangible slowdown with HarfBuzz on my system.  Can
you post some benchmarks with timings?  (I assume both branches were
built using the same optimization switches.)

> a) On harfbuzz and master branches from "emacs -Q", a machine without
> the Symbola font does not display the emoji U+1F44B WAVING HAND SIGN (no
> font available). Babelmap shows that this character is available using
> the built-in "Segoe UI Symbol" or "Segoe UI Emoji" fonts (or by
> installing Symbola).
> 
> b) On the harfbuzz branch from "emacs -Q", the lao U+EC3 LAO VOWEL SIGN
> AY and U+EC3 LAO VOWEL SIGN O characters are not displayed (no font
> available). Babelmap shows that this character is available using the
> built-in "Leelawadee UI" font.

BabelMap just shows coverage, but Emacs also tests additional features
of the fonts (although I don't think we have any special requirements
for Emoji; Lao certainly does require some OTF features).

In any case, the font backend has nothing whatsoever to do with how
Emacs searches for a suitable font, at least on Windows.  What the
above means is that Leelawadee somehow doesn't fit the criteria for
the Lao script and/or the features bits these fonts exhibit don't
announce that they cover the respective codepoint ranges.  The way to
improve the font search is to customize the fontsets.

> I use the following to speed up finding built-in fonts on Windows 10:
> 
>   (pcase-dolist
>       (`(,font-spec . ,targets)
>        '(;; Unicode blocks ---------------------------------------
>          ("Segoe UI Emoji"
>           (#x1f900 . #x1f9ff)) ; Supplemental Symbols and Pictographs
>          ;; Unicode scripts --------------------------------------
>          ("Segoe UI Symbol"      braille mathematical symbol)
>          ("Leelawadee UI"        khmer thai lao)
>          ("Nirmala UI"           bengali devanagari gujarati kannada
>           malayalam oriya sinhala tamil telugu)
>          ("Microsoft Himalaya"   tibetan)
>          ("Myanmar Text"         burmese)
>          ("Ebrima"               ethiopic)
>          ("Gadugi"               canadian-aboriginal cherokee)))
>     (dolist (target targets)
>       (set-fontset-font "fontset-default" target font-spec nil 'prepend)))
> 
> Perhaps the built in fonts should be added to the default mappings for
> Windows 10.

I think the policy is not to mention non-free fonts in our fontsets.

Re: HarfBuzz is available on MS-Windows

[Andy Moreton]

On Fri 07 Jun 2019, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Fri, 07 Jun 2019 18:05:23 +0100
>> 
>> I've tested this with an MSYS2 64bit build on Windows 10.
>
> Thanks, you can now just build the master branch.
>
>> It appears to work ok, but is noticeably slower compared to the
>> master branch.
>
> I didn't see any tangible slowdown with HarfBuzz on my system.  Can
> you post some benchmarks with timings?  (I assume both branches were
> built using the same optimization switches.)

My feeling was only subjective based on running from "emacs -Q", but
that may be problems with font lookup making everything slow.

I've now bootstrapped the master branch after you merged the harfbuzz
support, and everything appears to be working fine so far.

> In any case, the font backend has nothing whatsoever to do with how
> Emacs searches for a suitable font, at least on Windows.  What the
> above means is that Leelawadee somehow doesn't fit the criteria for
> the Lao script and/or the features bits these fonts exhibit don't
> announce that they cover the respective codepoint ranges.  The way to
> improve the font search is to customize the fontsets.

Agreed, but it would be nicer for the average Windows user if the
built-in fonts were used by default, and that configuring the fontset
would only be needed to override the default selection.

>> Perhaps the built in fonts should be added to the default mappings for
>> Windows 10.
>
> I think the policy is not to mention non-free fonts in our fontsets.

That's reasonable when recommending additional fonts to install, but it
is unhelpful to avoid using fonts that are installed by default.

    AndyM

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 07 Jun 2019 22:13:36 +0100
> 
> >> It appears to work ok, but is noticeably slower compared to the
> >> master branch.
> >
> > I didn't see any tangible slowdown with HarfBuzz on my system.  Can
> > you post some benchmarks with timings?  (I assume both branches were
> > built using the same optimization switches.)
> 
> My feeling was only subjective based on running from "emacs -Q", but
> that may be problems with font lookup making everything slow.

If it's font lookup, then it has nothing to do with HarfBuzz.  The
font lookup is entirely our own code, and for HarfBuzz we actually
use the same code as for Uniscribe.  IOW, the speed of finding a
suitable font should be unaltered in the current master.

> I've now bootstrapped the master branch after you merged the harfbuzz
> support, and everything appears to be working fine so far.

Good to know, thanks.

> > In any case, the font backend has nothing whatsoever to do with how
> > Emacs searches for a suitable font, at least on Windows.  What the
> > above means is that Leelawadee somehow doesn't fit the criteria for
> > the Lao script and/or the features bits these fonts exhibit don't
> > announce that they cover the respective codepoint ranges.  The way to
> > improve the font search is to customize the fontsets.
> 
> Agreed, but it would be nicer for the average Windows user if the
> built-in fonts were used by default, and that configuring the fontset
> would only be needed to override the default selection.

That's the situation, AFAIK.  With the qualification that "the default
selection" depends also on the order in which the installed fonts are
enumerated by the system APIs -- we have no control on this order.
Emacs chooses the first matching font produced by the enumeration, so
if a font comes later in the enumeration order that is better in some
sense, Emacs might not select it without fontset customizations.

> >> Perhaps the built in fonts should be added to the default mappings for
> >> Windows 10.
> >
> > I think the policy is not to mention non-free fonts in our fontsets.
> 
> That's reasonable when recommending additional fonts to install, but it
> is unhelpful to avoid using fonts that are installed by default.

There are 2 issues here: one is about the speed of finding fonts for
certain characters, the other is about finding a suitable font at
all.  I was under the impression that your fontset customizations were
meant to solve the speed issue, in which case I don't think it's an
issue with the defaults, and in any case we shouldn't assume anything
about the users' installation: they could have other fonts installed
that they prefer, so it would be improper for Emacs to force them to
use the built-in fonts.  In addition, quality of built-in fonts
changes with time, so a font that comes with the system today might be
less desirable to use tomorrow.  Thus, including those fonts in our
sources would mean we need to track the development of system fonts,
update the fonts in our default fontsets, and perhaps make the fontset
dependent on the OS version.  This would be a maintenance burden.

However, above you seem now to be talking about failure to find a font
although some fonts do support the character.  That is a separate
issue.  The question in this case is why doesn't Emacs find and select
these fonts by itself, without any hints in the fontset in addition to
what we already have there.  To answer this question, one needs to
step through the code which enumerates the fonts on your system and
determines whether a given font matches the criteria we require for
supporting that particular character.  For example, we treat Emoji
like we treat symbols; maybe that is incorrect?  For Lao, we require
that the font supports certain OTF features; perhaps there's a mistake
there, or the criteria need some adjustments?  If you'd like to try
debugging this, I can help by pointing to the code where this
happens.  In general, start with the fontset as defined in fontset.el,
and then look in w32font.c:w32font_list_internal and its subroutines.

Re: HarfBuzz is available on MS-Windows

[Andy Moreton]

On Sat 08 Jun 2019, Eli Zaretskii wrote:
> There are 2 issues here: one is about the speed of finding fonts for
> certain characters, the other is about finding a suitable font at
> all.

Agreed.

> I was under the impression that your fontset customizations were
> meant to solve the speed issue, in which case I don't think it's an
> issue with the defaults, and in any case we shouldn't assume anything
> about the users' installation: they could have other fonts installed
> that they prefer, so it would be improper for Emacs to force them to
> use the built-in fonts.

The problem with incorrect character display is fixed by part of my
fontset customisation: the rest of it is to improve font lookup speed.
All of the mentioned fonts are installed by default on Windows 10.

Emacs should be capable of finding a font to display all of the 
characters in HELLO, as it can do so given some fontset customisation.
It should do so by default, so charatcers are displayed properly but
users can add further customisation to express a preference for
different fonts.

> In addition, quality of built-in fonts changes with time, so a font
> that comes with the system today might be less desirable to use
> tomorrow. Thus, including those fonts in our sources would mean we
> need to track the development of system fonts, update the fonts in our
> default fontsets, and perhaps make the fontset dependent on the OS
> version. This would be a maintenance burden.

It is more important to give users a working tool than to reduce
maintenance effort. The selection of fonts shipped with each platform
changes fairly slowly, so this should not present an undue burden.

> However, above you seem now to be talking about failure to find a font
> although some fonts do support the character.  That is a separate
> issue.  The question in this case is why doesn't Emacs find and select
> these fonts by itself, without any hints in the fontset in addition to
> what we already have there.

Yes, and this was the first part of what I wrote. This needs fixing.

> If you'd like to try debugging this, I can help by pointing to the
> code where this happens. In general, start with the fontset as defined
> in fontset.el, and then look in w32font.c:w32font_list_internal and
> its subroutines.

I can try debugging from there - any further hints are welcome.

    AndyM

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 08 Jun 2019 13:11:46 +0100
> 
> Emacs should be capable of finding a font to display all of the 
> characters in HELLO, as it can do so given some fontset customisation.

Only if fonts to support all of the characters are available on the
user's system.  It could be that all of the fonts are available on
Windows 10 (I don't have such a system handy to verify that), but this
certainly isn't so on older versions of Windows.

> > In addition, quality of built-in fonts changes with time, so a font
> > that comes with the system today might be less desirable to use
> > tomorrow. Thus, including those fonts in our sources would mean we
> > need to track the development of system fonts, update the fonts in our
> > default fontsets, and perhaps make the fontset dependent on the OS
> > version. This would be a maintenance burden.
> 
> It is more important to give users a working tool than to reduce
> maintenance effort.

This is a noble goal, but it's impractical to expect that, at least
for Windows.  Guess how many developers actually touch the related
code, or even have a good enough understanding of what happens there.

> The selection of fonts shipped with each platform changes fairly
> slowly, so this should not present an undue burden.

My experience is very different.  Specifically wrt Windows, we have a
major new version every 2 to 3 years, with minor updates in-between.
This is more frequent than Emacs releases its major versions.  And if
you want to get an idea regarding the amount of font-related changes
in each major release of Windows, look here:

  https://docs.microsoft.com/en-us/globalization/input/font-support

Just to study and understand what each new font means will take a
significant amount of time.

Besides, even the information about which fonts are available on what
versions of Windows, and which scripts they cover reasonably well, is
currently not available in the form that can be directly compared with
our fontset setup (the above page only lists _new_ fonts, but
significant changes are also made in existing fonts).

So we are nowhere near being on top of this.  If you want to
contribute to improving this situation, please consider coming
on-board and collecting the necessary information, then we could start
talking about where we have inadequate support and how to make things
better.  Otherwise, at best things will be left where they are now,
because no one works on this on a routine basis, and (speaking about
myself personally) I doubt if we even have enough expert knowledge,
let alone free time and energy, for doing the job.

> > If you'd like to try debugging this, I can help by pointing to the
> > code where this happens. In general, start with the fontset as defined
> > in fontset.el, and then look in w32font.c:w32font_list_internal and
> > its subroutines.
> 
> I can try debugging from there - any further hints are welcome.

The function add_font_entity_to_list is the first place to look.  It
performs all the checks to determine whether a font might fit the
requirements of the font spec.  For Uniscribe and HarfBuzz, the
match_data->opentype_only flag is set.  The subroutines of
add_font_entity_to_list, font_matches_spec and font_supported_scripts
also perform important checks.

If you have more specific questions, I will try my best to answer
them.  TIA.

Re: HarfBuzz is available on MS-Windows

[Angelo Graziosi]

Eli Zaretskii wrote:
>
> The harfbuzz branch can now be built on MS-Windows, and will support
> the new 'harfbuzz' font backend if the requisite DLLs are installed.
>

I have built it on MSYS2/MinGW64 and seems to work as expected: I tested "C-h h", and for what I can see, HELLO displays just fine..

Ciao,
 Angelo.

Re: HarfBuzz is available on MS-Windows

[Eli Zaretskii]

> Date: Fri, 31 May 2019 22:24:27 +0200 (CEST)
> From: Angelo Graziosi <angelo.g0@libero.it>
> Cc: Eli Zaretskii <eliz@gnu.org>
> 
> I have built it on MSYS2/MinGW64 and seems to work as expected: I tested "C-h h", and for what I can see, HELLO displays just fine..

Thank you very much for testing the branch.