From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Michal Nazarewicz Newsgroups: gmane.emacs.bugs Subject: bug#24603: [PATCHv6 6/6] Implement special sigma casing rule (bug#24603) Date: Tue, 21 Mar 2017 02:27:09 +0100 Message-ID: <20170321012709.19402-7-mina86@mina86.com> References: <20170309215150.9562-1-mina86@mina86.com> <20170321012709.19402-1-mina86@mina86.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1490059703 9911 195.159.176.226 (21 Mar 2017 01:28:23 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 21 Mar 2017 01:28:23 +0000 (UTC) To: Eli Zaretskii , Andreas Schwab , 24603@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Mar 21 02:28:19 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq8aw-0001rZ-Sb for geb-bug-gnu-emacs@m.gmane.org; Tue, 21 Mar 2017 02:28:19 +0100 Original-Received: from localhost ([::1]:36105 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cq8az-0006Zx-JH for geb-bug-gnu-emacs@m.gmane.org; Mon, 20 Mar 2017 21:28:21 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:46854) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cq8ak-0006WH-B6 for bug-gnu-emacs@gnu.org; Mon, 20 Mar 2017 21:28:08 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cq8ah-0006p0-5m for bug-gnu-emacs@gnu.org; Mon, 20 Mar 2017 21:28:06 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:38813) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cq8ah-0006oq-2B for bug-gnu-emacs@gnu.org; Mon, 20 Mar 2017 21:28:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1cq8ag-0008Od-Tp for bug-gnu-emacs@gnu.org; Mon, 20 Mar 2017 21:28:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Michal Nazarewicz Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 21 Mar 2017 01:28:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 24603 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 24603-submit@debbugs.gnu.org id=B24603.149005964832175 (code B ref 24603); Tue, 21 Mar 2017 01:28:02 +0000 Original-Received: (at 24603) by debbugs.gnu.org; 21 Mar 2017 01:27:28 +0000 Original-Received: from localhost ([127.0.0.1]:36998 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq8a7-0008Mm-PQ for submit@debbugs.gnu.org; Mon, 20 Mar 2017 21:27:28 -0400 Original-Received: from mail-wr0-f175.google.com ([209.85.128.175]:34997) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cq8a3-0008M5-Mc for 24603@debbugs.gnu.org; Mon, 20 Mar 2017 21:27:24 -0400 Original-Received: by mail-wr0-f175.google.com with SMTP id g10so103075898wrg.2 for <24603@debbugs.gnu.org>; Mon, 20 Mar 2017 18:27:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OzB5AXRp712RQQ6Yg6vitZNWWWtPhU4XH9jhGHJ3RGo=; b=gP9DH123C28zkU3Slo59wOiegMQ4wshAeuMTxjwqjtYGtUlFSpUREF6/oE011gGXDZ x6+VXsmx4IwZdDMl5xF8AFUNxV79I41ot0yFVMU/xPVJuKKIeELjHM/HrnSoALwOSE9U 1mdvymEjjHU7BLjw6Let2l7VCkGgJx26Ih4cjNHq6wac+m1OoLaGoO6e2mwe4h3zzKse waJZfI5pX8lpNMfEwDx88w3RGuLmPadeWTi2SnaebLgny63XL9WVrUzZ3kbxAcpEv/pH cui92eKZyf0Fku8FXXy4aJELUDOP/YDHIiwUkaJGAwtoIz4aEaeQp0mtkZcNUISGrMRa ojLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=OzB5AXRp712RQQ6Yg6vitZNWWWtPhU4XH9jhGHJ3RGo=; b=sHTzLPCAi9UYtgoiWQ2LFWB/cEf5qspgRHyHOrIXhHcFyPbXzPwTIjTOOfrgtloj2b G/W8zHq/nN5SgfCQ8zOXbjkTCPPTwcjBaeJ5BXSej7+BE2KTsiyjapLCV0fKWSNis4DY aMwCFP2Tdi19zWxlH9iKYOrUolvn+Dd9DsMi6puGBYNtYZbgYvuSLzRwX57KP2rncndB e5ZgIpjNCklkTYjvamYdNqVsZNeoh2mrQlBYh6gIPhOOPDbMk3/RDxuHYdoS1tcXmkkA H5t4YwTXAMLTPZ6rSvA+kfB+ai/tY55qb4fohXSqEPEQKY/to04j+PNJLrCqJoWWahZm /OKg== X-Gm-Message-State: AFeK/H125A1H0xtoF9DCtb0MV+f0XMgHfKGNm8HRk/52FnIxbuey50IgyP8O3zAx4lQGPM4z X-Received: by 10.223.177.207 with SMTP id r15mr31192711wra.127.1490059637847; Mon, 20 Mar 2017 18:27:17 -0700 (PDT) Original-Received: from mpn.zrh.corp.google.com ([172.16.115.43]) by smtp.gmail.com with ESMTPSA id m201sm15512562wmd.19.2017.03.20.18.27.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Mar 2017 18:27:15 -0700 (PDT) Original-Received: by mpn.zrh.corp.google.com (Postfix, from userid 126942) id 061571E026C; Tue, 21 Mar 2017 02:27:14 +0100 (CET) X-Mailer: git-send-email 2.12.1.500.gab5fba24ee-goog In-Reply-To: <20170321012709.19402-1-mina86@mina86.com> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:130766 Archived-At: In Greek, a sigma character has two lower case forms which depend on their position in the word. Implement logic determining it. * src/casefiddle.c (struct casing_context, case_character_impl): Don’t assume inword is true when flag is CASE_UP and false when flag is CASE_DOWN. For final sigma detection we need this information tracked reliably;. (CAPITAL_SIGMA, SMALL_SIGMA, SMALL_FINAL_SIGMA): New macros defining Unicode code point of different forms of sigma letter. (case_character): Implement support for final sigma casing. (do_casify_multibyte_string, do_casify_multibyte_region): Update after changes to case_character. * test/src/casefiddle-tests.el (casefiddle-tests-casing): Add test cases for final sigma. --- etc/NEWS | 5 +++ src/casefiddle.c | 73 +++++++++++++++++++++++++++++++++----------- test/src/casefiddle-tests.el | 15 +++++---- 3 files changed, 69 insertions(+), 24 deletions(-) diff --git a/etc/NEWS b/etc/NEWS index af1981b1453..fb5cc6095bb 100644 --- a/etc/NEWS +++ b/etc/NEWS @@ -367,6 +367,11 @@ of incorrect DŽungla). *** Characters which turn into multiple ones when cased are correctly handled. For example, fi ligature is converted to FI when upper cased. +*** Greek small sigma is correctly handled when at the end of the word. +Strings such as ΌΣΟΣ are now correctly converted to Όσος when +capitalized instead of incorrect Όσοσ (compare lowercase sigma at the +end of the word). + * Changes in Specialized Modes and Packages in Emacs 26.1 diff --git a/src/casefiddle.c b/src/casefiddle.c index df94c970539..8a47641c4bb 100644 --- a/src/casefiddle.c +++ b/src/casefiddle.c @@ -45,9 +45,7 @@ struct casing_context { When run on a buffer, syntax_prefix_flag_p is taken into account when determined inword flag. */ bool inbuffer; - /* Conceptually, this denotes whether we are inside of a word except - that if flag is CASE_UP it’s always false and if flag is CASE_DOWN - this is always true. */ + /* Whether we are inside of a word. */ bool inword; }; @@ -58,7 +56,7 @@ prepare_casing_context (struct casing_context *ctx, { ctx->flag = flag; ctx->inbuffer = inbuffer; - ctx->inword = flag == CASE_DOWN; + ctx->inword = false; ctx->titlecase_char_table = (int)flag < (int)CASE_CAPITALIZE ? Qnil : uniprop_table (intern_c_string ("titlecase")); ctx->specialcase_char_tables[CASE_UP] = flag == CASE_DOWN ? Qnil : @@ -100,15 +98,16 @@ case_character_impl (struct casing_str_buf *buf, /* Update inword state */ was_inword = ctx->inword; - if ((int) ctx->flag >= (int) CASE_CAPITALIZE) - ctx->inword = SYNTAX (ch) == Sword && - (!ctx->inbuffer || was_inword || !syntax_prefix_flag_p (ch)); + ctx->inword = SYNTAX (ch) == Sword && + (!ctx->inbuffer || was_inword || !syntax_prefix_flag_p (ch)); /* Normalise flag so its one of CASE_UP, CASE_DOWN or CASE_CAPITALIZE. */ - if (!was_inword) - flag = ctx->flag == CASE_UP ? CASE_UP : CASE_CAPITALIZE; + if (ctx->flag == CASE_CAPITALIZE) + flag = (enum case_action)((int)ctx->flag - was_inword); else if (ctx->flag != CASE_CAPITALIZE_UP) - flag = CASE_DOWN; + flag = ctx->flag; + else if (!was_inword) + flag = CASE_CAPITALIZE; else { cased = ch; @@ -149,7 +148,18 @@ case_character_impl (struct casing_str_buf *buf, buf->len_bytes = CHAR_STRING (cased, buf->data); return cased != ch; } + +/* In Greek, lower case sigma has two forms: one when used in the middle and one + when used at the end of a word. Below is to help handle those cases when + casing. + + The rule does not conflict with any other casing rules so while it is + a conditional one, it is independent on language. */ +#define CAPITAL_SIGMA 0x03A3 +#define SMALL_SIGMA 0x03C3 +#define SMALL_FINAL_SIGMA 0x03C2 + /* Based on CTX, case character CH accordingly. Update CTX as necessary. Return cased character. @@ -163,12 +173,34 @@ case_single_character (struct casing_context *ctx, int ch) } /* Save in BUF result of casing character CH. Return whether casing changed the - character. This is like case_single_character but also handles one-to-many - casing rules. */ -static inline bool -case_character (struct casing_str_buf *buf, struct casing_context *ctx, int ch) + character. + + If not-NULL, NEXT points to the next character in the cased string. If NULL, + it is assumed current character is the last one being cased. This is used to + apply some rules which depend on proceeding state. + + This is like case_single_character but also handles one-to-many casing + rules. */ +static bool +case_character (struct casing_str_buf *buf, struct casing_context *ctx, + int ch, const unsigned char *next) { - return case_character_impl (buf, ctx, ch); + bool changed, was_inword; + + was_inword = ctx->inword; + changed = case_character_impl (buf, ctx, ch); + + /* If we have just down-cased a capital sigma and the next character no longer + has a word syntax (i.e. current character is end of word), use final + sigma. */ + if (was_inword && ch == CAPITAL_SIGMA && changed && + (!next || SYNTAX (STRING_CHAR (next)) != Sword)) + { + buf->len_bytes = CHAR_STRING (SMALL_FINAL_SIGMA, buf->data); + buf->len_chars = 1; + } + + return changed; } static Lisp_Object @@ -230,7 +262,7 @@ do_casify_multibyte_string (struct casing_context *ctx, Lisp_Object obj) if (dst_end - o < sizeof(struct casing_str_buf)) string_overflow (); ch = STRING_CHAR_ADVANCE (src); - case_character ((void *)o, ctx, ch); + case_character ((void *)o, ctx, ch, size > 1 ? src : NULL); n += ((struct casing_str_buf *)o)->len_chars; o += ((struct casing_str_buf *)o)->len_bytes; } @@ -381,12 +413,17 @@ do_casify_multibyte_region (struct casing_context *ctx, ptrdiff_t pos = *startp, pos_byte = CHAR_TO_BYTE (pos), size = *endp - pos; ptrdiff_t opoint = PT, added = 0; struct casing_str_buf buf; - int ch, cased, len; + bool changed; + int ch, len; for (; size; --size) { ch = STRING_CHAR_AND_LENGTH (BYTE_POS_ADDR (pos_byte), len); - if (!case_character (&buf, ctx, ch)) + changed = case_character ( + &buf, ctx, ch, + size > 1 ? BYTE_POS_ADDR (pos_byte + len) : NULL); + + if (!changed) { pos_byte += len; ++pos; diff --git a/test/src/casefiddle-tests.el b/test/src/casefiddle-tests.el index dd260633f4c..234d233c71a 100644 --- a/test/src/casefiddle-tests.el +++ b/test/src/casefiddle-tests.el @@ -195,13 +195,16 @@ casefiddle-tests--test-casing ("define" "DEFINE" "define" "Define" "Define") ("fish" "FISH" "fish" "Fish" "Fish") ("Straße" "STRASSE" "straße" "Straße" "Straße") - ;; FIXME(bug#24603): Everything below is broken at the moment. - ;; Here’s what should happen: - ;;("ΌΣΟΣ" "ΌΣΟΣ" "όσος" "Όσος" "Όσος") - ;; And here’s what is actually happening: - ("ΌΣΟΣ" "ΌΣΟΣ" "όσοσ" "Όσοσ" "ΌΣΟΣ") - ("όσος" "ΌΣΟΣ" "όσος" "Όσος" "Όσος")))))) + ;; The word repeated twice to test behaviour at the end of a word + ;; inside of an input string as well as at the end of the string. + ("ΌΣΟΣ ΌΣΟΣ" "ΌΣΟΣ ΌΣΟΣ" "όσος όσος" "Όσος Όσος" "ΌΣΟΣ ΌΣΟΣ") + ;; What should be done with sole sigma? It is ‘final’ but on the + ;; other hand it does not form a word. We’re using regular sigma. + ("Σ Σ" "Σ Σ" "σ σ" "Σ Σ" "Σ Σ") + ("όσος" "ΌΣΟΣ" "όσος" "Όσος" "Όσος") + ;; If sigma is already lower case, we don’t want to change it. + ("όσοσ" "ΌΣΟΣ" "όσοσ" "Όσοσ" "Όσοσ")))))) (ert-deftest casefiddle-tests-casing-byte8 () (should-not -- 2.12.0.367.g23dc2f6d3c-goog