Re: master 32e790f: Implement NTLM server for ntlm.el testing

[Michael Albinus <michael.albinus@gmx.de>]

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

Hi Thomas,

>> Not a big deal, after pulling and recompiling recent web-server ELPA
>> package it was fixed. However, it makes me nervous that a checkout of
>> emacs master can fail tests, w/o any reason inside the sources.
>
> I realize this isn't ideal, but it's better than nothing, i.e., not
> being able to use GNU ELPA packages as test dependencies.  And if you
> expand "inside the sources" to include GNU ELPA, which I assume Emacs
> maintainers who have an elpa checkout do, then I think it's OK (maybe
> even desirable?) to have to look into GNU ELPA-dependent test failures.

Of course I appreciate to use GNU ELPA packages for tests. But you
cannot assume that everybody has an ELPA git checkout. Given, that the
packages live in their own branches or repositories.

> That said, if there is no elpa.git checkout at ../elpa, or if
> GNU_ELPA_DIRECTORY=/nonexistent, then the tests are just skipped.

There are just three Emacs maintainers, and they might run an ELPA
checkout. But ordinary Emacs developers might live w/o an ELPA checkout,
and use just packages provided by the default mechanism, and selected by
them. We shall be prepared also for this case (and maybe extend the way
how GNU ELPA packages are accessed when running the Emacs tests).

> (Maybe as a result of the bundling discussions, GNU ELPA will become a
> submodule of emacs.git which would make the dependency more reliable.)

Maybe. But we aren't at this point yet. And since packages have their
own repository life, I doubt that "git pull" from the Emacs repository
will bring us also fresh packages checkout.

>> Could you pls provide more advanced checks for dependencies? For
>> example, instead of
>>
>>  (and (featurep 'url-http-ntlm) (featurep 'web-server))
>>
>> you could check a proper version of the packages?
>
> I'd rather not add the complexity, e.g., depending on package.el.
> Beyond this first requirement to update web-server to the latest, I
> don't see how a version check would be useful.  I tried to implement a
> specific check for the ws-parse functionality, but I couldn't find an
> easy way of confirming the new NTLM path.

That's just *this* case. Other changes might apply to your packages in
the future, which require newer package code. And I'm thinking also
about a more general approach for other Emacs tests, how to use GNU ELPA
packages as dependency.

And you don't need package.el. Require lisp-mnt, then you could do
something like

--8<---------------cut here---------------start------------->8---
(defvar ntlm-tests--dependencies-present
  (and (featurep 'url-http-ntlm)
       (version<= "2.0.4" (lm-version (locate-file "url-http-ntlm.el" load-path)))
       (featurep 'web-server)
       (version<= "0.1.2" (lm-version (locate-file "web-server.el" load-path))))

  "Non-nil if GNU ELPA test dependencies were loaded.")
--8<---------------cut here---------------end--------------->8---

>> Warning (emacs): Cannot find one or more GNU ELPA packages
>> Warning (emacs): Skipping NTLM authentication tests
>> Warning (emacs): See GNU_ELPA_DIRECTORY in test/README
>> Running 3 tests (2021-02-19 10:26:00+0100, selector `(not (tag :unstable))')
>>   skipped  1/3  ntlm-authentication (0.000133 sec)
>>   skipped  2/3  ntlm-authentication-old-compatibility-level (0.000097 sec)
>>    passed  3/3  ntlm-time-to-timestamp (0.000136 sec)
>>
>> Is it necessary to be such chatty? The tests are skipped (like other
>> tests), fine.
>
> I debated not putting in those warnings.  I wanted to point out why the
> tests are being skipped, and in particular that it's easy to have them
> not be skipped if you have a GNU ELPA checkout (versus tests that are
> unavoidably skipped because they depend on a different OS or
> architecture).
>
> Basically, I'd like to encourage Emacs maintainers to not skip these
> tests; the reason I wrote them was to prevent regressions like Bug#43566
> (merged with Bug#44195 and Bug#44439), which broke all uses of NTLM in
> an Emacs point release.

Again, not every Emacs developer has an ELPA checkout. And we shouldn't
be bossy to them.

> Thomas

Best regards, Michael.