Re: Emacs 27.1 Windows Binaries -- testing wanted

[phillip.lord@russet.org.uk]

On 2020-08-22 14:08, Eli Zaretskii wrote:
>> Date: Sat, 22 Aug 2020 13:52:22 +0100
>> From: phillip.lord@russet.org.uk
>> Cc: emacs-devel@gnu.org
>> 
>> > I'm not talking about using ert or not, I'm talking about determining
>> > which of the features should check out and which shouldn't.  Are you
>> > going to edit w32-features.el by hand each time you add or remove an
>> > optional library from the build?  Or will the list of the features
>> > which should check out be constructed automagically somehow?
>> 
>> It would be manual. The list of features changes rarely enough that 
>> it's
>> hard to automate this and know that it will work. The automation is as
>> likely to break as a manual system
> 
> I don't understand why it must break, if you base it on
> system-configuration-features, or on some new file, to be generated by
> the build procedure.  Did you consider one of these alternatives?  If
> yes, why did you reject them?

I didn't consider it much. I cannot see how to go from this to something 
I can test. Say, `system-configuration-features` lists "XPM", I cannot 
turn that into something I could test.

I could modify by test something like

(ert-deftest
   (skip-unless (string-match-p "XPM" system-configuration-features))
   (should (image-type-available 'xpm))
)


>> If I think back to the bugs with Emacs binary releases, this would 
>> have
>> stopped only a few: harfbuzz and svg failure in this release. It would
>> not have prevented missing dependencies (I have missed all of 
>> harfbuzz,
>> lcms and libjansson till late in the day), nor the failed optimization
>> or stripping.
> 
> I don't understand this, either.  If the dependencies of HarfBuzz are
> not installed, then the HarfBuzz font backend will not be available,
> and the way I suggested for testing its availability will tell you
> HarfBuzz doesn't work, which is AFAIU what you wanted.  And the same
> with any other DLLs: if their dependencies are not available, they
> will fail to load when the feature test runs, and the test will fail.

It would work if, when harfbuzz was installed w32-features was updated, 
but not if it wasn't. As the harfbuzz merge didn't update build-deps.zh 
which it arguable should have, then why would a test file be updated.


>> > If the latter, I didn't see you describe how will that happen.  If the
>> > former, it means a file in the distribution will depend on the details
>> > of how you build the official binaries, which could be different from
>> > what end-users do on their systems (those who build their own Emacs),
>> > so the tests will fail for them, AFAIU.
>> 
>> That is true with all of our tests, I think.
> 
> No, our tests use skip-unless to avoid running tests that need
> features or programs which are unavailable.
> 
>> json parsing is tested by make check and it will not work on msys2
>> if the dependencies are not installed.
> 
> I didn't look into this, but if json testing fails instead of skipping
> all the tests, that's a bug that should be fixed.

I think you are correct about all of these. In either case, the tests 
will not pass!


>> In this case, I am not suggesting adding ert tests to make check,
>> but a manually run test specifically for understanding the features
>> of a binary build. I am guessing anyone running this would be
>> understand what they are doing enough to know why. And I will
>> document the entry function.
> 
> That's not what worries me.  What worries me is the fact that a file
> distributed as part of the Lisp libraries will have its contents
> depend on the last release you personally did, with whatever quirks
> that required from the tests in that file.  It doesn't sound right to
> me.


Not sure I understand this. A lisp library with "w32-" in it's name will 
have behaviour which reflects how the official binaries are supposed to 
behave. If this worries you, as I say, I can add it to `data-directory' 
instead. But, it has to be in the distribution.


>> > It might also mean problems in merging from the release branch to
>> > master,
>> > e.g. if we remove some optional library from master that is still being
>> > used on the release
>> > branch.
>> 
>> That problem is there anyway in build-deps.py
> 
> But build-deps.py is under admin, whereas you are suggesting to do
> this under lisp/.

Having a test functionality under lisp/ does, I agree, seem morally 
wrong. Likewise `data-directory'. But, the reasons for it are clear, and 
what harm would it cause in practice?


Phil