Collation tests in fns-tests.el

Collation tests in fns-tests.el

[Ken Brown]

Hi Michael,

I'm curious why you put the following test in fns-tests.el:

;; Punctuation and whitespace characters are not taken into account
   ;; for collation in other locales.
   (should
    (equal
     (sort '("11" "12" "1 1" "1 2" "1.1" "1.2")
	  (lambda (a b)
	    (let ((w32-collate-ignore-punctuation t))
	      (string-collate-lessp
	       a b (if (eq system-type 'windows-nt) "enu_USA" "en_US.UTF-8")))))
     '("11" "1 1" "1.1" "12" "1 2" "1.2")))

This suggests that punctuation and whitespace should definitely not be 
taken into account in non-POSIX locales.  But the docstring of 'sort' is 
much less definitive:

"This function obeys the conventions for collation order in your locale 
settings.  For example, punctuation and whitespace characters *might* be 
considered less significant for sorting."  [My emphasis.]

Is there some place where emacs relies on punctuation and whitespace 
being ignored?  That certainly isn't the case on all supported systems, 
nor is it mandated by POSIX.

Ken

P.S. My question is motivated by the fact that punctuation and 
whitespace are not ignored on Cygwin in non-POSIX locales, and it does 
not seem to be easy to make this happen.  If you're interested in the 
gory details, start here:

   https://www.cygwin.com/ml/cygwin/2015-10/msg00516.html

Re: Collation tests in fns-tests.el

[Eli Zaretskii]

> From: Ken Brown <kbrown@cornell.edu>
> Date: Fri, 30 Oct 2015 13:51:45 -0400
> Cc: Emacs <emacs-devel@gnu.org>
> 
> I'm curious why you put the following test in fns-tests.el:
> 
> ;; Punctuation and whitespace characters are not taken into account
>    ;; for collation in other locales.
>    (should
>     (equal
>      (sort '("11" "12" "1 1" "1 2" "1.1" "1.2")
> 	  (lambda (a b)
> 	    (let ((w32-collate-ignore-punctuation t))
> 	      (string-collate-lessp
> 	       a b (if (eq system-type 'windows-nt) "enu_USA" "en_US.UTF-8")))))
>      '("11" "1 1" "1.1" "12" "1 2" "1.2")))
> 
> This suggests that punctuation and whitespace should definitely not be 
> taken into account in non-POSIX locales.

They were found to be ignored in all the cases we tested until now.

> But the docstring of 'sort' is much less definitive:
> 
> "This function obeys the conventions for collation order in your locale 
> settings.  For example, punctuation and whitespace characters *might* be 
> considered less significant for sorting."  [My emphasis.]
> 
> Is there some place where emacs relies on punctuation and whitespace 
> being ignored?

Listing of files generally ignores them, as one example.  ls-lisp.el
relies on that to emulate what 'ls' the program does on Posix hosts.

> P.S. My question is motivated by the fact that punctuation and 
> whitespace are not ignored on Cygwin in non-POSIX locales, and it does 
> not seem to be easy to make this happen.  If you're interested in the 
> gory details, start here:
> 
>    https://www.cygwin.com/ml/cygwin/2015-10/msg00516.html

You already said in that discussion what I'd suggest ;-)

Since Cygwin tries to be compatible to GNU/Linux (i.e. glibc), it
should indeed use some non-zero flags in its implementation of string
collation-dependent comparison.  IMO, it makes no sense not to do
that, since users expect that to happen.  Then the above test will
work for it, and moreover, ls-lisp.el will, too.

Re: Collation tests in fns-tests.el

[Eli Zaretskii]

> Date: Fri, 30 Oct 2015 22:28:09 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> > From: Ken Brown <kbrown@cornell.edu>
> > Date: Fri, 30 Oct 2015 13:51:45 -0400
> > Cc: Emacs <emacs-devel@gnu.org>
> > 
> > I'm curious why you put the following test in fns-tests.el:
> > 
> > ;; Punctuation and whitespace characters are not taken into account
> >    ;; for collation in other locales.
> >    (should
> >     (equal
> >      (sort '("11" "12" "1 1" "1 2" "1.1" "1.2")
> > 	  (lambda (a b)
> > 	    (let ((w32-collate-ignore-punctuation t))
> > 	      (string-collate-lessp
> > 	       a b (if (eq system-type 'windows-nt) "enu_USA" "en_US.UTF-8")))))
> >      '("11" "1 1" "1.1" "12" "1 2" "1.2")))
> > 
> > This suggests that punctuation and whitespace should definitely not be 
> > taken into account in non-POSIX locales.
> 
> They were found to be ignored in all the cases we tested until now.

I should have added "in UTF-8 locales" here, sorry.

But since Cygwin nowadays behaves like a UTF-8 locale (AFAIK), this
doesn't change the conclusions regarding Cygwin behavior, I think.

Re: Collation tests in fns-tests.el

[Ken Brown]

On 10/30/2015 4:28 PM, Eli Zaretskii wrote:
> You already said in that discussion what I'd suggest ;-)
>
> Since Cygwin tries to be compatible to GNU/Linux (i.e. glibc), it
> should indeed use some non-zero flags in its implementation of string
> collation-dependent comparison.  IMO, it makes no sense not to do
> that, since users expect that to happen.

Yes, I agree completely.  The issue is implementation.  Simply using the 
NORM_IGNORESYMBOLS flag yields comparison functions that can return 0 on 
unequal strings.  Eric pointed out the problem with that; moreover, it 
seriously violates users' expectations and compatibility with glibc.  I 
thought I had a way around that, but Corinna pointed out in 
https://www.cygwin.com/ml/cygwin/2015-10/msg00559.html why my suggestion 
doesn't work.  At this point I'm out of ideas.

Ken

Re: Collation tests in fns-tests.el

[Eli Zaretskii]

> Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Fri, 30 Oct 2015 17:10:48 -0400
> 
> On 10/30/2015 4:28 PM, Eli Zaretskii wrote:
> > You already said in that discussion what I'd suggest ;-)
> >
> > Since Cygwin tries to be compatible to GNU/Linux (i.e. glibc), it
> > should indeed use some non-zero flags in its implementation of string
> > collation-dependent comparison.  IMO, it makes no sense not to do
> > that, since users expect that to happen.
> 
> Yes, I agree completely.  The issue is implementation.  Simply using the 
> NORM_IGNORESYMBOLS flag yields comparison functions that can return 0 on 
> unequal strings.  Eric pointed out the problem with that; moreover, it 
> seriously violates users' expectations and compatibility with glibc.  I 
> thought I had a way around that, but Corinna pointed out in 
> https://www.cygwin.com/ml/cygwin/2015-10/msg00559.html why my suggestion 
> doesn't work.  At this point I'm out of ideas.

I don't see why that conclusion is the only reasonable one (the
"seriously violates users' expectation" part surprises me), but I
don't really consider myself an expert on this, certainly not in
Cygwin.

If Cygwin's implementation of strcoll cannot be fixed, then we should
treat this test on Cygwin as expected failure.

Re: Collation tests in fns-tests.el

[Ken Brown]

On 10/30/2015 5:35 PM, Eli Zaretskii wrote:
>> Cc: michael.albinus@gmx.de, emacs-devel@gnu.org
>> From: Ken Brown <kbrown@cornell.edu>
>> Date: Fri, 30 Oct 2015 17:10:48 -0400
>>
>> On 10/30/2015 4:28 PM, Eli Zaretskii wrote:
>>> You already said in that discussion what I'd suggest ;-)
>>>
>>> Since Cygwin tries to be compatible to GNU/Linux (i.e. glibc), it
>>> should indeed use some non-zero flags in its implementation of string
>>> collation-dependent comparison.  IMO, it makes no sense not to do
>>> that, since users expect that to happen.
>>
>> Yes, I agree completely.  The issue is implementation.  Simply using the
>> NORM_IGNORESYMBOLS flag yields comparison functions that can return 0 on
>> unequal strings.  Eric pointed out the problem with that; moreover, it
>> seriously violates users' expectations and compatibility with glibc.  I
>> thought I had a way around that, but Corinna pointed out in
>> https://www.cygwin.com/ml/cygwin/2015-10/msg00559.html why my suggestion
>> doesn't work.  At this point I'm out of ideas.
>
> I don't see why that conclusion is the only reasonable one (the
> "seriously violates users' expectation" part surprises me), but I
> don't really consider myself an expert on this, certainly not in
> Cygwin.
>
> If Cygwin's implementation of strcoll cannot be fixed, then we should
> treat this test on Cygwin as expected failure.

I'll probably do that, but I'll wait to see if Michael has anything to add.

Ken

Re: Collation tests in fns-tests.el

[Michael Albinus]

Ken Brown <kbrown@cornell.edu> writes:

>> If Cygwin's implementation of strcoll cannot be fixed, then we should
>> treat this test on Cygwin as expected failure.
>
> I'll probably do that, but I'll wait to see if Michael has anything to add.

I have no other idea, sorry. Let's mark the test case as expected to
fail for Cygwin. And maybe a note about this behaviour could be added in
doc/lispref/strings.texi.

> Ken

Best regards, Michael.

Re: Collation tests in fns-tests.el

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 31 Oct 2015 09:49:30 +0100
> 
> Ken Brown <kbrown@cornell.edu> writes:
> 
> >> If Cygwin's implementation of strcoll cannot be fixed, then we should
> >> treat this test on Cygwin as expected failure.
> >
> > I'll probably do that, but I'll wait to see if Michael has anything to add.
> 
> I have no other idea, sorry. Let's mark the test case as expected to
> fail for Cygwin. And maybe a note about this behaviour could be added in
> doc/lispref/strings.texi.

Well, one other idea is for Cygwin's Emacs build to have its private
implementation of strcoll, similar to what w32_compare_strings does.

Re: Collation tests in fns-tests.el

[Ken Brown]

On 10/31/2015 5:07 AM, Eli Zaretskii wrote:
>> From: Michael Albinus <michael.albinus@gmx.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Sat, 31 Oct 2015 09:49:30 +0100
>>
>> Ken Brown <kbrown@cornell.edu> writes:
>>
>>>> If Cygwin's implementation of strcoll cannot be fixed, then we should
>>>> treat this test on Cygwin as expected failure.
>>>
>>> I'll probably do that, but I'll wait to see if Michael has anything to add.
>>
>> I have no other idea, sorry. Let's mark the test case as expected to
>> fail for Cygwin. And maybe a note about this behaviour could be added in
>> doc/lispref/strings.texi.
>
> Well, one other idea is for Cygwin's Emacs build to have its private
> implementation of strcoll, similar to what w32_compare_strings does.

I  could do this in the future if necessary.  For now, I've just 
followed Michael's suggestion.

Ken