RFC: locale-sensitive Emacs functions

RFC: locale-sensitive Emacs functions

[Ted Zlatanov]

Eli and I discussed adding commas to separate digits of a number, a very
common need mentioned here and also addressed in the Emacs Wiki etc.:
https://www.reddit.com/r/emacs/comments/61mhyx/creating_a_function_to_add_commasseparators_to/dfga1h4/

I mentioned that commas are only used in some regions, and the C
libraries often support "%'d" to do this in a locale-sensitive way.

I think it makes sense to provide this specifier in `format' as well,
even though it's locale-sensitive. Eli wasn't sure.

What do you think?

Thanks
Ted

Re: RFC: locale-sensitive Emacs functions

[Gdobbins]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

In Common Lisp, such directives use : rather than ', but more importantly they also take arguments to specify the grouping and the separation character. They default to 3 and comma respectively. If Emacs' format were to allow similar arguments to be provided, it would allow the exact format to be specified where it is important and the user could set global variables like default-digit-grouping and default-digit-separator or some such to get behavior matching their locale. For example:

(format "%:d" 100000) => "100,000"

(let ((default-digit-grouping 4)
(default-digit-separator ?.))
(format "%:d" 100000)) => "10.0000"

-- Graham Dobbins

[-- Attachment #2: Type: text/html, Size: 907 bytes --]

Re: RFC: locale-sensitive Emacs functions

[Paul Eggert]

Gdobbins wrote:
> In Common Lisp, such directives use : rather than ', but more importantly they also take arguments to specify the grouping and the separation character.

The ' flag is the only printf flag that the POSIX printf function supports but 
the Emacs 'format' function does not. When I looked into this a while ago I 
remember concluding that I was glad I didn't have to write code to support the ' 
flag, as it would require messing with the LC_NUMERIC locale setting (currently 
hardwired to "C" for other good reasons) and would make for some work redoing 
carefully-calculated output buffer sizes internal to 'format', not to mention 
the character set conversion that would be required.

For what it's worth, the Common Lisp approach cannot handle the Indian numbering 
system, which has a comma every two digits except that the last grouping 
contains three digits (e.g., "12,34,56,789"). This is something that POSIX can do.

Re: RFC: locale-sensitive Emacs functions

[Gdobbins]

[-- Attachment #1.1: Type: text/plain, Size: 1350 bytes --]

>I remember concluding that I was glad I didn't have to write code to support the '
>flag, as it would require messing with the LC_NUMERIC locale setting (currently
>hardwired to "C" for other good reasons) and would make for some work redoing
>carefully-calculated output buffer sizes internal to 'format', not to mention
>the character set conversion that would be required.

The way I've proposed should obviate those problems. Since the variables governing the behavior are set by the user, LC_NUMERIC need not be consulted. If it is deemed necessary a function like set-locale-dependent-vars could be created to set these (and potentially other) variables, and the user could put it in their init if they so choose.

>For what it's worth, the Common Lisp approach cannot handle the Indian numbering
>system, which has a comma every two digits except that the last grouping
>contains three digits (e.g., "12,34,56,789").

The literal CL approach can't handle it, but it can be adapted to do so. The attached patch makes it so the format-digit-grouping variable can be a list. The first element of the list controls the grouping size of the least significant figures etc. until the last element controls all remaining groups. Using your example:

(let ((format-digit-grouping '(3 2)))
(format "%'d" 123456789)) => "12,34,56,789"

-- Graham Dobbins

[-- Attachment #1.2: Type: text/html, Size: 1823 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-src-editfns.c-styled_format-Add-d-format-specifier.patch --]
[-- Type: text/x-patch; name="0001-src-editfns.c-styled_format-Add-d-format-specifier.patch", Size: 4933 bytes --]

From b4a7b52f3044ebc51b4529cfcd46682815f69545 Mon Sep 17 00:00:00 2001
From: Graham Dobbins <gdobbins@protonmail.com>
Date: Tue, 28 Mar 2017 07:48:58 -0400
Subject: [PATCH] * src/editfns.c (styled_format): Add %'d format specifier

---
 etc/NEWS      |  5 +++++
 src/editfns.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 70 insertions(+), 2 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index cd98f53399..da593b81d8 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1055,6 +1055,11 @@ its window gets deleted by 'delete-other-windows'.
 *** New command 'window-swap-states' swaps the states of two live
 windows.
 
+---
+*** 'format' now accepts the specifier "%'d" to format integers into
+groups of size 'format-digit-grouping' with separator
+'format-digit-separator' which default to 3 and comma respectively.
+
 \f
 * Changes in Emacs 26.1 on Non-Free Operating Systems
 
diff --git a/src/editfns.c b/src/editfns.c
index 65c0c721d1..b6e8c489c0 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -3884,7 +3884,7 @@ specifiers, as follows:
 
   %<flags><width><precision>character
 
-where flags is [+ #-0]+, width is [0-9]+, and precision is a literal
+where flags is [+ #-0\\=']+, width is [0-9]+, and precision is a literal
 period "." followed by [0-9]+
 
 The + flag character inserts a + before any positive number, while a
@@ -3900,6 +3900,10 @@ the precision is zero; for %g, it causes a decimal point to be
 included even if the the precision is zero, and also forces trailing
 zeros after the decimal point to be left in place.
 
+The \\=' flag is ignored for all arguments except %d. If the argument
+is an integer it is printed in groupings whose size is dictated by
+`format-digit-grouping' and are separated by `format-digit-separator'.
+
 The width specifier supplies a lower limit for the length of the
 printed representation.  The padding, if any, normally goes on the
 left, but it goes on the right if the - flag is present.  The padding
@@ -4059,6 +4063,7 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 	  bool space_flag = false;
 	  bool sharp_flag = false;
 	  bool  zero_flag = false;
+	  bool  apos_flag = false;
 
 	  for (; ; format++)
 	    {
@@ -4069,6 +4074,7 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 		case ' ': space_flag = true; continue;
 		case '#': sharp_flag = true; continue;
 		case '0':  zero_flag = true; continue;
+		case '\'': apos_flag = true; continue;
 		}
 	      break;
 	    }
@@ -4381,6 +4387,49 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 			}
 		    }
 		  sprintf_bytes = sprintf (sprintf_buf, convspec, prec, x);
+
+		  if (apos_flag && INTEGERP (args[n]))
+		    {
+		      Lisp_Object real_grouping = Vformat_digit_grouping;
+		      EMACS_INT grouping;
+		      if (CONSP (real_grouping))
+			{
+			  CHECK_RANGED_INTEGER (XCAR (real_grouping),	\
+						1, MOST_POSITIVE_FIXNUM);
+			  grouping = XINT (XCAR (real_grouping));
+			}
+		      else
+			{
+                          CHECK_RANGED_INTEGER (real_grouping,		\
+						1, MOST_POSITIVE_FIXNUM);
+			  grouping = XINT (real_grouping);
+			}
+
+		      CHECK_CHARACTER (Vformat_digit_separator);
+
+		      char last_char;
+		      char temp_char;
+		      for (ptrdiff_t i = sprintf_bytes - grouping;	\
+			   i > 0; i -= grouping)
+			{
+			  last_char = XFASTINT (Vformat_digit_separator);
+                          for (ptrdiff_t j = i; j <= sprintf_bytes; j++)
+			    {
+			      temp_char = sprintf_buf[j];
+			      sprintf_buf[j] = last_char;
+			      last_char = temp_char;
+			    }
+			  sprintf_bytes++;
+			  if (CONSP (real_grouping)		\
+			      && !NILP (XCDR (real_grouping)))
+			    {
+			      real_grouping = XCDR (real_grouping);
+			      CHECK_RANGED_INTEGER (XCAR (real_grouping), \
+						    1, MOST_POSITIVE_FIXNUM);
+			      grouping = XINT (XCAR (real_grouping));
+			    }
+			}
+		    }
 		}
 	      else
 		{
@@ -5171,7 +5220,21 @@ functions if all the text being accessed has this property.  */);
   DEFVAR_LISP ("operating-system-release", Voperating_system_release,
 	       doc: /* The release of the operating system Emacs is running on.  */);
 
-  defsubr (&Spropertize);
+  DEFVAR_LISP ("format-digit-grouping",
+	       Vformat_digit_grouping,
+	       doc: /* Number of digits in each group of a formatted
+number in `format'. If a list, the first number applies to the least
+significant grouping and so on, with the last number applying to all
+remaining groupings.  */);
+  Vformat_digit_grouping = make_number (3);
+
+  DEFVAR_LISP ("format-digit-separator",
+	       Vformat_digit_separator,
+	       doc: /* Character to use as grouping separator for
+formatted numbers in `format'.  */);
+  Vformat_digit_separator = make_number (',');
+
+ defsubr (&Spropertize);
   defsubr (&Schar_equal);
   defsubr (&Sgoto_char);
   defsubr (&Sstring_to_char);
-- 
2.12.1

Re: RFC: locale-sensitive Emacs functions

[Ted Zlatanov]

On Tue, 28 Mar 2017 07:59:25 -0400 Gdobbins <gdobbins@protonmail.com> wrote: 

>> I remember concluding that I was glad I didn't have to write code to support the '
>> flag, as it would require messing with the LC_NUMERIC locale setting (currently
>> hardwired to "C" for other good reasons) and would make for some work redoing
>> carefully-calculated output buffer sizes internal to 'format', not to mention
>> the character set conversion that would be required.

G> The way I've proposed should obviate those problems. Since the variables
G> governing the behavior are set by the user, LC_NUMERIC need not be consulted. If
G> it is deemed necessary a function like set-locale-dependent-vars could be
G> created to set these (and potentially other) variables, and the user could put
G> it in their init if they so choose.

Right, so you're *ignoring* the locale, and making "%'d" an
Emacs-specific thing because it doesn't follow the C behavior. I agree
that's a clean approach.

>> For what it's worth, the Common Lisp approach cannot handle the Indian numbering
>> system, which has a comma every two digits except that the last grouping
>> contains three digits (e.g., "12,34,56,789").

G> The literal CL approach can't handle it, but it can be adapted to do so. The
G> attached patch makes it so the format-digit-grouping variable can be a list. The
G> first element of the list controls the grouping size of the least significant
G> figures etc. until the last element controls all remaining groups. Using your
G> example:

G> (let ((format-digit-grouping '(3 2)))
G> (format "%'d" 123456789)) => "12,34,56,789"

I like the patch and the proposed approach, and if there are no
objections, and with some tests, I think it would be good to go.

Ted

Re: RFC: locale-sensitive Emacs functions

[Ted Zlatanov]

On Tue, 28 Mar 2017 08:45:13 -0400 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> I like the patch and the proposed approach, and if there are no
TZ> objections, and with some tests, I think it would be good to go.

Gdobbins, can you take Paul's suggestions and make a new patch? Or if
you are not able to work on it, please let us know so we can pick up the
task.

Thanks
Ted

Re: RFC: locale-sensitive Emacs functions

[Gdobbins]

[-- Attachment #1.1: Type: text/plain, Size: 81 bytes --]

Attached is a new patch which implements everything requested.

-- Graham Dobbins

[-- Attachment #1.2: Type: text/html, Size: 314 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-new-d-specifier-to-format.patch --]
[-- Type: text/x-patch; name="0001-Add-new-d-specifier-to-format.patch", Size: 9349 bytes --]

From e52eb215cb7dcace3a97180dbc744522db98f4d0 Mon Sep 17 00:00:00 2001
From: Graham Dobbins <gdobbins@protonmail.com>
Date: Fri, 31 Mar 2017 00:16:17 -0400
Subject: [PATCH] Add new %'d specifier to format.

* src/editfns.c (styled_format): Add the new functionality.
* test/src/editfns-tests.el: Add tests for new functionality.
* doc/lispref/strings.texi: Document new specifier.
---
 doc/lispref/strings.texi  |  16 ++++++
 etc/NEWS                  |   5 ++
 src/editfns.c             | 125 ++++++++++++++++++++++++++++++++++++++++++++--
 test/src/editfns-tests.el |  14 ++++++
 4 files changed, 157 insertions(+), 3 deletions(-)

diff --git a/doc/lispref/strings.texi b/doc/lispref/strings.texi
index ae2b31c541..9d5786e26b 100644
--- a/doc/lispref/strings.texi
+++ b/doc/lispref/strings.texi
@@ -1030,8 +1030,24 @@ Formatting Strings
 If both @samp{-} and @samp{0} are present, the @samp{0} flag is
 ignored.
 
+  The flag @samp{'} only has defined behavior when combined with the
+@samp{%d} specifier. It causes the number to be printed in groups with
+width determined by @code{format-digit-grouping} and separated by the
+character @code{format-digit-separator}. This action takes place
+before any padding is applied. This flag is typically used to achieve
+more human readable representations of numbers such as
+@samp{"1,234,567"}.
+
 @example
 @group
+(format "%'d" 123456789)
+     @result{} "123,456,789"
+
+(let ((format-digit-grouping 4)
+      (format-digit-separator ?.))
+   (format "%'d" 123456789))
+     @result{} "1.2345.6789"
+
 (format "%06d is padded on the left with zeros" 123)
      @result{} "000123 is padded on the left with zeros"
 
diff --git a/etc/NEWS b/etc/NEWS
index cd98f53399..5105904687 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1055,6 +1055,11 @@ its window gets deleted by 'delete-other-windows'.
 *** New command 'window-swap-states' swaps the states of two live
 windows.
 
++++
+*** 'format' now accepts the specifier "%'d" to format integers into
+groups of size 'format-digit-grouping' with separator
+'format-digit-separator' which default to 3 and comma respectively.
+
 \f
 * Changes in Emacs 26.1 on Non-Free Operating Systems
 
diff --git a/src/editfns.c b/src/editfns.c
index 2dafd8e7b1..eb7d1a4829 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -3884,7 +3884,7 @@ specifiers, as follows:
 
   %<flags><width><precision>character
 
-where flags is [+ #-0]+, width is [0-9]+, and precision is a literal
+where flags is [+ #-0\\=']+, width is [0-9]+, and precision is a literal
 period "." followed by [0-9]+
 
 The + flag character inserts a + before any positive number, while a
@@ -3900,6 +3900,12 @@ the precision is zero; for %g, it causes a decimal point to be
 included even if the the precision is zero, and also forces trailing
 zeros after the decimal point to be left in place.
 
+The behavior of the \\=' flag is only defined for %d. The argument is
+printed in groupings whose size is dictated by `format-digit-grouping'
+and are separated by `format-digit-separator'. This is typically used
+to print more human readable representations of numbers like
+"1,000." This action takes place before any padding is done.
+
 The width specifier supplies a lower limit for the length of the
 printed representation.  The padding, if any, normally goes on the
 left, but it goes on the right if the - flag is present.  The padding
@@ -4042,7 +4048,7 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 
 	     where
 
-	     flags ::= [-+0# ]+
+	     flags ::= [-+0# ']+
 	     field-width ::= [0-9]+
 	     precision ::= '.' [0-9]*
 
@@ -4059,6 +4065,7 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 	  bool space_flag = false;
 	  bool sharp_flag = false;
 	  bool  zero_flag = false;
+	  bool  apos_flag = false;
 
 	  for (; ; format++)
 	    {
@@ -4069,6 +4076,7 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 		case ' ': space_flag = true; continue;
 		case '#': sharp_flag = true; continue;
 		case '0':  zero_flag = true; continue;
+		case '\'': apos_flag = true; continue;
 		}
 	      break;
 	    }
@@ -4274,7 +4282,10 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 		   precision is no more than USEFUL_PRECISION_MAX.
 		   On all practical hosts, %f is the worst case.  */
 		SPRINTF_BUFSIZE =
-		  sizeof "-." + (LDBL_MAX_10_EXP + 1) + USEFUL_PRECISION_MAX,
+		max (sizeof "-." + (LDBL_MAX_10_EXP + 1) + USEFUL_PRECISION_MAX,
+		     (INT_BUFSIZE_BOUND (printmax_t)
+		      + (INT_STRLEN_BOUND (printmax_t) - 2)
+		      * MAX_MULTIBYTE_LENGTH)),
 
 		/* Length of pM (that is, of pMd without the
 		   trailing "d").  */
@@ -4381,6 +4392,100 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 			}
 		    }
 		  sprintf_bytes = sprintf (sprintf_buf, convspec, prec, x);
+
+		  if (apos_flag)
+		    {
+		      CHECK_CHARACTER (Vformat_digit_separator);
+		      int separator = XFASTINT (Vformat_digit_separator);
+
+                      int separator_size = CHAR_BYTES (separator);
+		      if (separator_size != 1 && !multibyte)
+			{
+			  multibyte = true;
+			  goto retry;
+			}
+
+		      ptrdiff_t beg_pos = 0;
+		      for (; !('0' <= sprintf_buf[beg_pos]
+			       && sprintf_buf[beg_pos] <= '9');
+			   ++beg_pos);
+
+		      Lisp_Object real_grouping = Vformat_digit_grouping;
+		      EMACS_INT grouping;
+		      ptrdiff_t separator_count;
+
+		      if (CONSP (real_grouping))
+			{
+			  EMACS_INT temp_bytes = sprintf_bytes;
+			  EMACS_INT temp_grouping;
+			  separator_count = 0;
+			  FOR_EACH_TAIL (real_grouping)
+			    {
+			      CHECK_RANGED_INTEGER (XCAR (real_grouping),
+						    1, MOST_POSITIVE_FIXNUM);
+			      temp_grouping = XINT (XCAR (real_grouping));
+			      if (temp_bytes > beg_pos + temp_grouping)
+				{
+				  temp_bytes -= temp_grouping;
+				  ++separator_count;
+				  if (NILP (XCDR (real_grouping)))
+				    {
+				      separator_count
+					+= ((temp_bytes - beg_pos - 1)
+					    / temp_grouping);
+				    }
+				}
+			      else
+				break;
+			    }
+			  real_grouping = Vformat_digit_grouping;
+			  grouping = XINT (XCAR (real_grouping));
+			}
+		      else
+			{
+			  CHECK_RANGED_INTEGER (real_grouping,
+						1, MOST_POSITIVE_FIXNUM);
+			  grouping = XINT (real_grouping);
+			  separator_count = (sprintf_bytes - beg_pos - 1) / grouping;
+			}
+		      separator_count *= separator_size;
+
+		      EMACS_INT group_count = -2;
+
+		      for (ptrdiff_t i = sprintf_bytes + 1; i > beg_pos; --i)
+			{
+                          sprintf_buf[i + separator_count] = sprintf_buf[i];
+			  ++group_count;
+
+			  if (group_count == grouping)
+			    {
+			      separator_count -= separator_size;
+
+			      if (separator_size == 1)
+				sprintf_buf[i + separator_count] = separator;
+			      else
+				{
+				  CHAR_STRING (separator,
+					       (unsigned char *)
+					       (sprintf_buf + i  + separator_count));
+				  nchars -= (separator_size - 1);
+				}
+
+			      if (CONSP (real_grouping)
+				  && !NILP (XCDR (real_grouping)))
+				{
+				  real_grouping = XCDR (real_grouping);
+				  CHECK_RANGED_INTEGER
+				    (XCAR (real_grouping),
+				     1, MOST_POSITIVE_FIXNUM);
+				  grouping = XINT (XCAR (real_grouping));
+				}
+
+			      group_count = 0;
+			      sprintf_bytes += separator_size;
+			    }
+			}
+		    }
 		}
 	      else
 		{
@@ -5171,6 +5276,20 @@ functions if all the text being accessed has this property.  */);
   DEFVAR_LISP ("operating-system-release", Voperating_system_release,
 	       doc: /* The release of the operating system Emacs is running on.  */);
 
+  DEFVAR_LISP ("format-digit-grouping",
+	       Vformat_digit_grouping,
+	       doc: /* Number of digits in each group of a formatted
+number in `format'. If a list, the first number applies to the least
+significant grouping and so on, with the last number applying to all
+remaining groupings.  */);
+  Vformat_digit_grouping = make_number (3);
+
+  DEFVAR_LISP ("format-digit-separator",
+	       Vformat_digit_separator,
+	       doc: /* Character to use as grouping separator for
+formatted numbers in `format'.  */);
+  Vformat_digit_separator = make_number (',');
+
   defsubr (&Spropertize);
   defsubr (&Schar_equal);
   defsubr (&Sgoto_char);
diff --git a/test/src/editfns-tests.el b/test/src/editfns-tests.el
index 14124ef85f..231c72ead6 100644
--- a/test/src/editfns-tests.el
+++ b/test/src/editfns-tests.el
@@ -136,4 +136,18 @@ transpose-test-get-byte-positions
 (ert-deftest format-c-float ()
   (should-error (format "%c" 0.5)))
 
+(ert-deftest format-quote-d ()
+  (let ((format-digit-grouping 3)
+        (format-digit-separator ?,))
+    (should (string= (format "%'d" 123456789) "123,456,789"))
+    (should (string= (format "%+'d" 123456789) "+123,456,789"))
+    (should (string= (format "% 'd" 123456789) " 123,456,789"))
+    (should (string= (format "%0'15d" 123456789) "0000123,456,789"))
+    (let ((format-digit-grouping 1))
+      (should (string= (format "%'d" -123456789) "-1,2,3,4,5,6,7,8,9")))
+    (let ((format-digit-grouping '(3 2)))
+      (should (string= (format "%'d" 123456789) "12,34,56,789")))
+    (let ((format-digit-separator ?Ĭ))
+      (should (string= (format "%'d" 123456789) "123Ĭ456Ĭ789")))))
+
 ;;; editfns-tests.el ends here
-- 
2.12.1

Re: RFC: locale-sensitive Emacs functions

[Davis Herring]

> +  The flag @samp{'} only has defined behavior when combined with the

English nitpick: better to write "The flag ' has defined behavior only 
when combined..." (there and in the docstring).

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: RFC: locale-sensitive Emacs functions

[Paul Eggert]

Thanks, this looks like a reasonable way to proceed. Some comments:

There is no need for any of those backslashes at line end. Also, please 
indent consistently.

The manual needs to be updated, and there needs to be some test cases.

> +The \\=' flag is ignored for all arguments except %d.

This places too many constraints on the implementation. Instead, please 
say only that the behavior is defined for %d, and leave things undefined 
if the ' flag is used for any other conversion. We may want to add 
support for %'x at some point, for example.

This should document what happens when the 0 and ' flags are both used. 
Are grouping characters inserted before zero-padding, or after? I think 
it's before (as in POSIX), but you should check this.

> +		  if (apos_flag && INTEGERP (args[n]))

I don't see why that "&& INTEGERP (args[n])" is needed. Please remove it.

The code does not appear to do the right thing when the space or + flags 
are used, or when the integer is negative. It treats the leading sign or 
space as if it were a digit.

The code assumes that format-digit-separator is ASCII; it might not be.

Add test cases for the above corner cases.

Adjust SPRINTF_BUFSIZE to account for the worst-case buffer size when 
%'d is used. I suppose it should be the maximum of its current value, 
and (INT_BUFSIZE_BOUND (printmax_t) + (INT_STRLEN_BOUND (printmax_t) - 
2) * MAX_MULTIBYTE_LENGTH), but you should check this.

Use an O(N) rather than an O(N**2) algorithm when inserting grouping 
characters.