Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust <corwin@bru.st>]

[-- Attachment #1: Type: text/plain, Size: 3567 bytes --]

On Wed, Feb 2, 2022 at 4:07 PM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
> Corwin Brust <corwin@bru.st> writes:
>
> > On Mon, Jan 31, 2022 at 9:58 AM H. Dieter Wilhelm
> > <dieter@duenenhof-wilhelm.de> wrote:
> >>
> >> Currently we have 19 DLLs in the /bin folder (28.0.91), whereas 27.2 has
> >> 77 DLLs!  Corwin could you please check your build configuration, maybe
> >> together with Phil?  (I'm sorry, I'm still of no help here.)
> >
> > Excellent observation.
> >
> > I hope Phil will chime-in with suggestions of what I may have
> > missed/broken.  I'll investigate also, obviously, and others'
> > kibitzing most welcome too.
>
>
> Strange. What are the differences?

Firstly:  I'm not too sure yet.  I still need to download the Emacs
27.2 zip from the official source and do a direct "gap" comparison.

Overall, I'm leaning toward ditching automation of dependency
detection in favor of adding a separate file.   All efforts to
automating dependency recursion seem britle, ultimating causing more
work than they save us as we have to track down one mystery after
another. The proposed new file will
 - exhaustively lists each DLL we ship
 - be referenced in by documentation (such as README.W32)
 - be loaded by scripts, e.g. build-dep-zips.py
 - have additional "columns" to name the msys package, and
 - document why the dependency is required.

I have attached a txt file which is the complete list of what I would
start out including in that file.  Obviously, it's missing the msys
package names and has no commentary, so far.   I would probably call
it admin/nt/dist-build/deps.txt but I don't feel strongly about the
filename.

WDYT?

Meanwhile, in mesier news:

I wrote a perl script (also attached but don't bother) to find second
(and etc) order dependances using LDD that I believe should have been
(but have not been) getting pulled into the build.  I thought that the
msys installation folder path change Dieter caught would sort this,
but the numbers seem to don't add up (yet), probably because of the
undependability of the msys dependency checking that you mentioned.

I thought this would get me up to the same count (77) as Dieter reported, but:

  $ ls *.dll | wc -l
  19

  $ perl ../../../find-deps.pl | xargs -I '{}' cp '{}' .
  searched: 19, found: 392, of which 43 are missing in
/c/Users/corwi/emacs-build/install/emacs-28.0.91/bin

  $ ls *.dll | wc -l
  62

Note, these are the 62 DLLs listed in the attached textt file.

I also used this rather henus string of pipelined commands to
"cross-check" for missing dependencies and only came up with one,
which appears to be an optional dependency of harfbuzz:

  $  find ./ -name '*.dll' -print0 | xargs -0 -I '{}' cygcheck '{}'
2>&1 | grep 'could not' | cut -d ' ' -f 6 | sort | uniq
  libicuuc69.dll

Finally, I double-checked using an even more rediculous set of
pipeline commands. seemingly confirming everything the prior finds are
in the bin directory:

  $ find ./ -name '*.dll' -print0 | xargs -0 -I '{}' cygcheck '{}'
2>&1 | grep -v 'could not' | cut -d ' ' -f 6 | sort | uniq | xargs -0
-L1 echo | xargs -0 cygpath | perl -npe 's|c:|/c|g' | xargs -L1
basename | xargs ls >/dev/null

All this has me convinced some DLLs that shipped with Emacs 27.2
aren't findable as dependencies of anything we currently need for
Emacs 28, but not why.  And that leads me to the suggestion, above:  I
think we should hard-code the DLLs along with their source packages.
We can consider automatically generating/updating the deps file once
we get that far.

Warmest regards,
Corwin

[-- Attachment #2: dll-list.txt --]
[-- Type: text/plain, Size: 1062 bytes --]

libLerc.dll
libXpm-noX4.dll
libbrotlicommon.dll
libbrotlidec.dll
libbz2-1.dll
libcairo-2.dll
libcairo-gobject-2.dll
libdatrie-1.dll
libdeflate.dll
libexpat-1.dll
libffi-7.dll
libfontconfig-1.dll
libfreetype-6.dll
libfribidi-0.dll
libgcc_s_seh-1.dll
libgdk_pixbuf-2.0-0.dll
libgif-7.dll
libgio-2.0-0.dll
libglib-2.0-0.dll
libgmodule-2.0-0.dll
libgmp-10.dll
libgmpxx-4.dll
libgnutls-30.dll
libgnutls-openssl-27.dll
libgnutlsxx-28.dll
libgobject-2.0-0.dll
libgraphite2.dll
libharfbuzz-0.dll
libharfbuzz-gobject-0.dll
libharfbuzz-icu-0.dll
libharfbuzz-subset-0.dll
libhogweed-6.dll
libiconv-2.dll
libidn2-0.dll
libintl-8.dll
libjansson-4.dll
libjbig-0.dll
libjpeg-8.dll
liblcms2-2.dll
liblzma-5.dll
libnettle-8.dll
libp11-kit-0.dll
libpango-1.0-0.dll
libpangocairo-1.0-0.dll
libpangoft2-1.0-0.dll
libpangowin32-1.0-0.dll
libpcre-1.dll
libpixman-1-0.dll
libpng16-16.dll
librsvg-2-2.dll
libstdc++-6.dll
libtasn1-6.dll
libthai-0.dll
libtiff-5.dll
libtiffxx-5.dll
libturbojpeg.dll
libunistring-2.dll
libwebp-7.dll
libwinpthread-1.dll
libxml2-2.dll
libzstd.dll
zlib1.dll

[-- Attachment #3: find-deps.pl --]
[-- Type: application/octet-stream, Size: 2749 bytes --]

#!/usr/bin/env perl
#
# find-deps - find DLLs required by (but not in) the DLLs in CWD
#
# Copyright (C) 2022 Corwin Brust
#
# Quick and dirty check for missing dependancies of the DLLs included
# with a recently packaged build of Emacs. This should work given
# building Emacs works.  Run it from the install folder used for
# packaging, e.g.:
#
#   cd ~/emacs-build/install/emacs-28.0.91/bin
#   perl ../../../find-deps.pl > ~/emacs-build/dep-search.txt
# Or:
#  perl ../../../find-deps.pl | xargs -I '{}' cp '{}' .
#
# this rather obtuse alternative catches a couple more:
#   find ./ -name '*.dll' -print0 | \
#    xargs -0 -I '{}' cygcheck '{}' 2>&1 | \
#    grep 'could not' | cut -d ' ' -f 6 | \
#    sort | uniq | xargs -I '{}' cp '{}' .

## This file is not part of GNU Emacs.

## This program is free software: you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 3 of the License, or
## (at your option) any later version.

## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
## GNU General Public License for more details.

## You should have received a copy of the GNU General Public License
## along with this program.  If not, see <https://www.gnu.org/licenses/>.

use strict;
use warnings qw[all];

use File::Basename;

my %deps;

# use ldd to find deps
sub deps_of(_) {
  local $_ = `ldd $_`;
  s/^\s+//msg;
  map [ split / => / ]->[1] || $_,
    map { s/ \(.*?\)//; $_ }
    grep !/not found/,  # ignore optional
    grep !m|/Windows/|i,# ignore */Windows/*
    split /\n/
}

# call deps_of for each arg (unless we already have)
sub walk_deps(@) {
  my $found_count = 0;
  for (@_) {
    my $key = fileparse($_);
    next if exists $deps{$key} or !/.dll$/;
    my @found = deps_of $_;
    if(@found) {
      $deps{$key} = \@found;
      $found_count += 1;
    }
  }
  return $found_count;
}

# get a list of DLLs to start with
my @libs = glob('*.dll');

# search for deps of DLLs we included
walk_deps @libs;

# search for deps of our deps
my $added_count = 0;
do {
  $added_count = walk_deps map @$_, values %deps
} while($added_count > 0);

# de-dup
my %uniq;
for(map @$_, values %deps) {
  my $dll = fileparse($_);
  next if -e $dll; # skip those we included
  $uniq{$_} += 1;
}

# report
$\ = "\n";
print for sort keys %uniq;
warn
  "searched: ",  scalar(@libs),
  ", found: ",   scalar( map @$_, values %deps),
  ", of which ", scalar(keys %uniq),
  " are missing in ", `pwd`;