bug#32252: [PATCH] %o and %x now format signed numbers

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Treat integers as signed numbers when using the %o
and %x format specifiers, instead of treating them as
a machine-dependent two’s complement representation.
This should make Emacs more machine-independent, and
better-insulated for future changes involving bignums.
The new compatibility variable ‘binary-as-unsigned’
enables the old machine-dependent behavior.
This is a simplified version of the change proposed in:
https://lists.gnu.org/r/emacs-devel/2018-07/msg00763.html
I simplified that proposal by omitting bitwidth modifiers, as
I could not find an any example uses in the Emacs source code
that needed them and doing them correctly would have been
quite a bit more work for apparently little benefit.
* doc/lispref/strings.texi (Formatting Strings):
* etc/NEWS: Document change.
* src/editfns.c (styled_format): Treat integers as signed
numbers even with %o and %x.  Support the + and space
flags with %o and %x, since they’re about signs.
(syms_of_editfns): New compatibility variable binary-as-signed.
* test/src/editfns-tests.el (read-large-integer):
Test that maximal integers can be read after printing
with all integer formats.
(format-%o-invalid-float): Adjust test to match new
behavior, which allows negative octal.
---
 doc/lispref/strings.texi  |  6 +++---
 etc/NEWS                  |  8 ++++++++
 src/editfns.c             | 42 ++++++++++++++++++++++++++++++++++-----
 test/src/editfns-tests.el | 11 +++++-----
 4 files changed, 54 insertions(+), 13 deletions(-)

diff --git a/doc/lispref/strings.texi b/doc/lispref/strings.texi
index 2fff3c7c75..582561fe42 100644
--- a/doc/lispref/strings.texi
+++ b/doc/lispref/strings.texi
@@ -922,7 +922,7 @@ Formatting Strings
 @item %o
 @cindex integer to octal
 Replace the specification with the base-eight representation of an
-unsigned integer.  The object can also be a nonnegative floating-point
+integer.  The object can also be a floating-point
 number that is formatted as an integer, dropping any fraction, if the
 integer does not exceed machine limits.
 
@@ -935,8 +935,8 @@ Formatting Strings
 @itemx %X
 @cindex integer to hexadecimal
 Replace the specification with the base-sixteen representation of an
-unsigned integer.  @samp{%x} uses lower case and @samp{%X} uses upper
-case.  The object can also be a nonnegative floating-point number that
+integer.  @samp{%x} uses lower case and @samp{%X} uses upper
+case.  The object can also be a floating-point number that
 is formatted as an integer, dropping any fraction, if the integer does
 not exceed machine limits.
 
diff --git a/etc/NEWS b/etc/NEWS
index fc2a5d4c03..b735cf49b3 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -697,6 +697,14 @@ To get the old behavior, set the new, experimental variable
 read-integer-overflow-as-float to t and please email
 30408@debbugs.gnu.org if you need that.  (Bug#30408).
 
++++
+** Numbers formatted via %o or %x are now formatted as signed integers.
+Formerly they were formatted using machine-dependent two's complement
+representations.  To get the old behavior, set the new, experimental
+variable binary-as-unsigned to t and please email emacs-devel@gnu.org
+if you need that.  Because %o and %x now format signed integers, they
+now support the + and space flags.
+
 ---
 ** Some functions and variables obsolete since Emacs 22 have been removed:
 archive-mouse-extract, assoc-ignore-case, assoc-ignore-representation,
diff --git a/src/editfns.c b/src/editfns.c
index 09f836c3eb..a8fc499d63 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -4196,8 +4196,8 @@ contain either numbered or unnumbered %-sequences but not both, except
 that %% can be mixed with numbered %-sequences.
 
 The + flag character inserts a + before any nonnegative number, while a
-space inserts a space before any nonnegative number; these flags only
-affect %d, %e, %f, and %g sequences, and the + flag takes precedence.
+space inserts a space before any nonnegative number; these flags
+affect only numeric %-sequences, and the + flag takes precedence.
 The - and 0 flags affect the width specifier, as described below.
 
 The # flag means to use an alternate display form for %o, %x, %X, %e,
@@ -4736,19 +4736,42 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 		}
 	      else
 		{
-		  /* Don't sign-extend for octal or hex printing.  */
 		  uprintmax_t x;
+		  bool negative;
 		  if (INTEGERP (arg))
-		    x = XUINT (arg);
+		    {
+		      EMACS_INT i;
+		      if (binary_as_unsigned)
+			{
+			  i = XUINT (arg);
+			  negative = false;
+			}
+		      else
+			{
+			  i = XINT (arg);
+			  negative = i < 0;
+			  if (negative)
+			    i = -i;
+			}
+		      x = i;
+		    }
 		  else
 		    {
 		      double d = XFLOAT_DATA (arg);
 		      double uprintmax = TYPE_MAXIMUM (uprintmax_t);
+		      negative = d < 0;
+		      if (negative)
+			d = -d;
 		      if (! (0 <= d && d < uprintmax + 1))
 			xsignal1 (Qoverflow_error, arg);
 		      x = d;
+		      negative &= x != 0;
 		    }
-		  sprintf_bytes = sprintf (sprintf_buf, convspec, prec, x);
+		  sprintf_buf[0] = negative ? '-' : plus_flag ? '+' : ' ';
+		  bool signedp = negative | plus_flag | space_flag;
+		  sprintf_bytes = sprintf (sprintf_buf + signedp,
+					   convspec, prec, x);
+		  sprintf_bytes += signedp;
 		}
 
 	      /* Now the length of the formatted item is known, except it omits
@@ -5558,6 +5581,15 @@ functions if all the text being accessed has this property.  */);
   DEFVAR_LISP ("operating-system-release", Voperating_system_release,
 	       doc: /* The release of the operating system Emacs is running on.  */);
 
+  DEFVAR_BOOL ("binary-as-unsigned",
+	       binary_as_unsigned,
+	       doc: /* Non-nil means `format' %x and %o treat numbers as unsigned.
+Nil (the default) means to treat them as signed.
+Treating them as unsigned has machine-dependent results.
+
+This variable is experimental; email emacs-devel@gnu.org if you need it.  */);
+  binary_as_unsigned = false;
+
   defsubr (&Spropertize);
   defsubr (&Schar_equal);
   defsubr (&Sgoto_char);
diff --git a/test/src/editfns-tests.el b/test/src/editfns-tests.el
index c828000bb4..57a2081fc5 100644
--- a/test/src/editfns-tests.el
+++ b/test/src/editfns-tests.el
@@ -165,15 +165,16 @@ transpose-test-get-byte-positions
                 :type 'overflow-error)
   (should-error (read (substring (format "%d" most-negative-fixnum) 1))
                 :type 'overflow-error)
-  (should-error (read (format "#x%x" most-negative-fixnum))
-                :type 'overflow-error)
-  (should-error (read (format "#o%o" most-negative-fixnum))
-                :type 'overflow-error)
+  (dolist (fmt '("%d" "%s" "#o%o" "#x%x"))
+    (dolist (val (list most-negative-fixnum (1+ most-negative-fixnum)
+                       -1 0 1
+                       (1- most-positive-fixnum) most-positive-fixnum))
+      (should (eq val (read (format fmt val))))))
   (should-error (read (format "#32rG%x" most-positive-fixnum))
                 :type 'overflow-error))
 
 (ert-deftest format-%o-invalid-float ()
-  (should-error (format "%o" -1e-37)
+  (should-error (format "%o" -1e+INF)
                 :type 'overflow-error))
 
 ;; Bug#31938
-- 
2.17.1

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

>  Replace the specification with the base-eight representation of an
> -unsigned integer.  The object can also be a nonnegative floating-point
> +integer.  The object can also be a floating-point
>  number that is formatted as an integer, dropping any fraction, if the
>  integer does not exceed machine limits.

Why breaking longstanding behavior instead of introducing this with a
different format specifier.  Like %a.

> +  DEFVAR_BOOL ("binary-as-unsigned",
> +	       binary_as_unsigned,
> +	       doc: /* Non-nil means `format' %x and %o treat numbers as unsigned.
 
This is the same crap as text-quoting-style.  Nil should mean treat
numbers in the tradtional way.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Drew Adams]

1. I don't see where the doc now mentions variable `binary-as-signed'.  And I don't see where the doc mentions what the output is for %o and %x - for a negative number or a positive number.

2. Should `binary-as-signed' perhaps be a user option?

3. Why the change in the default behavior?

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

On 07/23/2018 12:49 PM, Drew Adams wrote:
> 1. I don't see where the doc now mentions variable `binary-as-signed'.

binary-as-signed is intentionally documented only in its doc string. The 
intent is that it be like the variable read-integer-overflow-as-float. 
That is, although it's present if someone actually needs it, we're 
hoping actual use cases are so rare that we can obsolete the variable 
soon. This is also why it's not a user option.

> I don't see where the doc mentions what the output is for %o and %x - for a negative number or a positive number.

For %o the Elisp manual says "Replace the specification with the 
base-eight representation of an
integer", and there is similar wording for %x and %X. It might be 
helpful to nail down negative-integer handling with wording like "Format 
a negative integer with a leading @samp{-} followed by the absolute value."

> 3. Why the change in the default behavior?

The primary motivation is that the old behavior can't reasonably be 
extended to bignums. Also, it's long been an annoyance that the old 
behavior was machine dependent: (format "%x" -1) yields different 
answers depending on what machine you're running on. The proposed change 
would make this usage portable, and would make it obvious what bignums 
should do.

bug#32252: [PATCH] %o and %x now format signed numbers

[Drew Adams]

> > 3. Why the change in the default behavior?
> 
> The primary motivation is that the old behavior can't reasonably be
> extended to bignums. Also, it's long been an annoyance that the old
> behavior was machine dependent: (format "%x" -1) yields different
> answers depending on what machine you're running on. The proposed change
> would make this usage portable, and would make it obvious what bignums
> should do.

Put that motivation in NEWS.  Users deserve to be told why
things change incompatibly.

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Drew Adams wrote:
> Put that motivation in NEWS.  Users deserve to be told why
> things change incompatibly.

OK, here's a revised NEWS entry. It also mentions avoiding read/print 
misbehavior, a more-immediate prompt for this change that I forgot to mention 
previously.

** Numbers formatted via %o or %x are now formatted as signed integers.
Formerly they were formatted using machine-dependent two's complement
representations.  The new behavior is machine-independent, avoids
misbehavior in calls like (read (format "#x%x" -1)), and is more
compatible with bignums, a planned feature.  To get the old behavior,
set the new, experimental variable binary-as-unsigned to t and please
email 32252@debbugs.gnu.org if you need that.  Because %o and %x now
format signed integers, they now support the + and space flags.
(Bug#32252).

bug#32252: [PATCH] %o and %x now format signed numbers

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 23 Jul 2018 19:04:48 -0700
> 
> ** Numbers formatted via %o or %x are now formatted as signed integers.
> Formerly they were formatted using machine-dependent two's complement
> representations.  The new behavior is machine-independent, avoids
> misbehavior in calls like (read (format "#x%x" -1)), and is more
> compatible with bignums, a planned feature.  To get the old behavior,
> set the new, experimental variable binary-as-unsigned to t and please
> email 32252@debbugs.gnu.org if you need that.  Because %o and %x now
> format signed integers, they now support the + and space flags.
> (Bug#32252).

Please don't put bug numbers in NEWS.  If some bug is important for
NEWS (I doubt that it ever could be), describe it instead.

NEWS is for people who want to grasp changes at a glance, we shouldn't
bother them with looking up bugs.

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Eli Zaretskii wrote:
> Please don't put bug numbers in NEWS.  If some bug is important for
> NEWS (I doubt that it ever could be), describe it instead.

Sorry, I didn't know that. Should I remove the existing bug numbers in NEWS? 
There are a half-dozen that could easily be removed. (One would need to stay 
since it's part of a quotation.)

bug#32252: [PATCH] %o and %x now format signed numbers

[Eli Zaretskii]

> Cc: drew.adams@oracle.com, 32252@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 23 Jul 2018 19:44:33 -0700
> 
> Eli Zaretskii wrote:
> > Please don't put bug numbers in NEWS.  If some bug is important for
> > NEWS (I doubt that it ever could be), describe it instead.
> 
> Sorry, I didn't know that. Should I remove the existing bug numbers in NEWS? 

Yes, please, but if the details were left off the rest of the text
because the bug reference was supposed to serve instead, please now
add some minimal details to make the entry speak for itself.  (I can
do this myself, if you prefer that.)

Thanks.

P.S. I never said anything to this effect until now because I didn't
realize bug numbers are being added to NEWS in the absence of any
guidance to the contrary.

bug#32252: [PATCH] %o and %x now format signed numbers

[Drew Adams]

> > Put that motivation in NEWS.  Users deserve to be told why
> > things change incompatibly.
> 
> OK, here's a revised NEWS entry. It also mentions avoiding read/print
> misbehavior, a more-immediate prompt for this change that I forgot to
> mention
> previously.
> 
> ** Numbers formatted via %o or %x are now formatted as signed integers.
> Formerly they were formatted using machine-dependent two's complement
> representations.  The new behavior is machine-independent, avoids
> misbehavior in calls like (read (format "#x%x" -1)), and is more
> compatible with bignums, a planned feature.  To get the old behavior,
> set the new, experimental variable binary-as-unsigned to t and please
> email 32252@debbugs.gnu.org if you need that.  Because %o and %x now
> format signed integers, they now support the + and space flags.
> (Bug#32252).

Thank you for providing the rationale.  A minor suggestion would
be to drop "new, experimental".  There's no way of knowing now
whether the variable will be only temporary/experimental, and
no reason to broadcast that supposition.  I'd also suggest that
you remove the part about emailing that bug thread.  Also, maybe
"space" should be `SPC' (dunno).

IMO it's good to let us know why the change was made.

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

> Why breaking longstanding behavior instead of introducing this with a
> different format specifier.  Like %a.

Although it is indeed an incompatible change, I surveyed the existing 
Emacs source code and couldn't find anything that the change would 
break. Almost all uses of %x, %X, and %o were with nonnegative integers 
where the behavior would not change. In the few places I found where the 
integers could be negative, the resulting strings were things like 
nonces where the change didn't break anything. Of course I could have 
missed some uses, and there are uses outside of Emacs source code; 
still, it seems that this change would break very little user code (and 
the rare users who might run into a problem have a simple workaround).

%a has quite a different meaning in C11, so we'd need to use some other 
letter (or set of letters, if we want something for %x, %X, %o) if we 
went that route. This seems like overkill for such a small change to 
functionality that hardly ever matters; plus, %x, %X and %o would 
continue to have problematic machine-dependent semantics and would not 
be sensibly extensible to bignums.

bug#32252: [PATCH] %o and %x now format signed numbers

[Drew Adams]

> Although it is indeed an incompatible change, I surveyed the existing
> Emacs source code and couldn't find anything that the change would
> break.

Inadequate.  Never wholly irrelevant, but certainly not
sufficient.  You surveyed something mostly irrelevant.

Emacs is about users.  Users use Elisp.  Lisp code delivered
with Emacs is not the same as the set of existing Emacs Lisp
code. Far, far from it.  This is a classic mistake.

> Almost all uses ... would not change.

What should be done, usually (not always):

1. Keep the default behavior - at least for a while
   (typically quite a while).

2. Ask users.  Take a poll - beyond this list.

What should not be done, usually:

* Just do what you feel like - make the default behavior
  whatever you like.

(Note: I don't have a horse in this race.  I don't care
what the default behavior is in this case, other than,
other things being equal it should be kept as is, as
that usually perturbs users least.)

bug#32252: [PATCH] %o and %x now format signed numbers

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Although it is indeed an incompatible change, I surveyed the existing
  > > Emacs source code and couldn't find anything that the change would
  > > break.

  > Inadequate.  Never wholly irrelevant, but certainly not
  > sufficient.  You surveyed something mostly irrelevant.

How about saying "incomplete"?  It states the valid point
but is far less harsh.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#32252: [PATCH] %o and %x now format signed numbers

[Drew Adams]

>   > > Although it is indeed an incompatible change, I surveyed the existing
>   > > Emacs source code and couldn't find anything that the change would
>   > > break.
> 
>   > Inadequate.  Never wholly irrelevant, but certainly not
>   > sufficient.  You surveyed something mostly irrelevant.
> 
> How about saying "incomplete"?  It states the valid point but is far less harsh.

Incomplete, of course; thank you.

But also inadequate, IMO - or insufficient, if you prefer - perhaps that is less harsh than inadequate.

The intention was not to be harsh but to say that, while it is good to check the sources distributed with Emacs, it is not good enough.

"Incomplete" tells part of the story, but it doesn't imply that there is a problem. (Unless there is agreement on a completeness requirement -  I wasn't assuming such agreement.)

Saying it's not sufficient to just check the Emacs sources says both that doing so is incomplete _and_ that such an incomplete search is not enough.

But you're no doubt right that there is likely more agreement that such a search is incomplete than there is that it's also not enough.

bug#32252: [PATCH] %o and %x now format signed numbers

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The intention was not to be harsh

I know that, so I am not rebuking you.  I'm asking you to make
a little extra effort so that you won't inadvertently hurt people's
feelings.

				      but to say that, while it is
  > good to check the sources distributed with Emacs, it is not good
  > enough.

Said that way, it's not harsh at all.

Can you see how to say things the non-harsh way the first time?


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Mon, Jul 23 2018, Paul Eggert wrote:

> %a has quite a different meaning in C11, so we'd need to use some

With your change %x will also have quite a different meaning in C11.

> other letter (or set of letters, if we want something for %x, %X, %o)
> if we went that route. This seems like overkill for such a small
> change to functionality that hardly ever matters; plus, %x, %X and %o
> would continue to have problematic machine-dependent semantics and
> would not be sensibly extensible to bignums.

If it doesn't matter to you then why make the change/break at all?  You
could just create a function bignum-to-hex-string.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Helmut Eller wrote:
> With your change %x will also have quite a different meaning in C11.

Not really, as Emacs (format "%x" N) agrees with C11 printf ("%x", N) in all 
values of N that are valid in both languages. In C11, negative values are not 
valid, as printf ("%x", N) has undefined behavior when N is negative. So we are 
discussing an area where Emacs Lisp can define behavior without introducing 
incompatibilities with C11.

If we changed (format "%x" -1) to signal an error instead, that would also be 
upward-compatible with C11. However, it's more useful for something like (format 
"#x%x" -1) to output a string that can 'read' can scan to get -1, something 
that's not true of Emacs now.

>> This seems like overkill for such a small
>> change to functionality that hardly ever matters; plus, %x, %X and %o
>> would continue to have problematic machine-dependent semantics and
>> would not be sensibly extensible to bignums.
> 
> If it doesn't matter to you then why make the change/break at all?

It does matter to me, actually. I think Emacs should have sensible behavior even 
in corner cases that hardly ever arise in real programs.

bug#32252: [PATCH] %o and %x now format signed numbers

[Andreas Schwab]

On Jul 24 2018, Paul Eggert <eggert@cs.ucla.edu> wrote:

> If we changed (format "%x" -1) to signal an error instead, that would also
> be upward-compatible with C11.

Unlike printf, format can convert arguments as needed, so an error is
wrong here.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Tue, Jul 24 2018, Paul Eggert wrote:

>> With your change %x will also have quite a different meaning in C11.
>
> Not really, as Emacs (format "%x" N) agrees with C11 printf ("%x", N)
> in all values of N that are valid in both languages. In C11, negative
> values are not valid, as printf ("%x", N) has undefined behavior when
> N is negative. So we are discussing an area where Emacs Lisp can
> define behavior without introducing incompatibilities with C11.

In practice, printf ("%x", (int)N) prints the bits in N as unsigned
integer not as signed integer.

> If we changed (format "%x" -1) to signal an error instead, that would
> also be upward-compatible with C11. However, it's more useful for
> something like (format "#x%x" -1) to output a string that can 'read'
> can scan to get -1, something that's not true of Emacs now.

#x-1 maybe interesting to the read function but it's not interesting to
humans.  Humans want to see #x3fffffffffffffff.

>>> This seems like overkill for such a small
>>> change to functionality that hardly ever matters; plus, %x, %X and %o
>>> would continue to have problematic machine-dependent semantics and
>>> would not be sensibly extensible to bignums.
>>
>> If it doesn't matter to you then why make the change/break at all?
>
> It does matter to me, actually.

If it matters to you then you should have no problem with introducing an
new specifier for this.

> I think Emacs should have sensible behavior even in corner cases that
> hardly ever arise in real programs.

That %x prints the two's complement representation for negative fixnums
is fairly sensible behavior.  Emacs should not change such basic things
or for that matter start to print ’ instead of '.  Such changes would be
the opposite of sensible.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Helmut Eller wrote:

> In practice, printf ("%x", (int)N) prints the bits in N as unsigned
> integer not as signed integer.

That's two operations not one, and even there the results disagree with what 
(format "%x" N) does. On every Emacs platform I've ever used the C output begins 
with a different hex digit than the Emacs Lisp output. Emacs has never done %x 
just like that unportable C code, and likely will never do so.

> #x-1 maybe interesting to the read function but it's not interesting to
> humans.  Humans want to see #x3fffffffffffffff.

I doubt very much that most humans really want to see an error-prone notation 
like that (can you easily spot the difference between it and 
#x3ffffffffffffffff? I can't). And even if some humans did want it, they won't 
get it on 32-bit Emacs. This whole idea is a bad approach for Emacs.

What I expect most users would prefer is an approach where 'read' and 'format' 
are inverses, e.g., (read (printf "#x%x" N)) returns N for every integer N. 
Emacs doesn't do that now, and the proposed patch fixes things so that it does. 
No other approach has been proposed that would make much sense (that is, be 
portable, be easily extendible to bignums, that sort of thing).

> you should have no problem with introducing an
> new specifier for this.

I'd rather not, as the hex stuff has never worked right in Emacs and we 
shouldn't be codifying old bugs.

bug#32252: [PATCH] %o and %x now format signed numbers

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 24 Jul 2018 17:50:46 -0700
> Cc: 32252@debbugs.gnu.org
> 
> > #x-1 maybe interesting to the read function but it's not interesting to
> > humans.  Humans want to see #x3fffffffffffffff.
> 
> I doubt very much that most humans really want to see an error-prone notation 
> like that (can you easily spot the difference between it and 
> #x3ffffffffffffffff? I can't).

I don't know if "most" is accurate, but I'm definitely in that camp.

As for counting the number of 'f's, how is that different from
counting leading zeros?

> And even if some humans did want it, they won't get it on 32-bit
> Emacs.

I don't see any problem: the user always knows in what Emacs she is
running.

> This whole idea is a bad approach for Emacs.

This bad approach was with us since about forever.  It is a de-facto
correct behavior by now

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

On 07/24/2018 07:41 PM, Eli Zaretskii wrote:
> As for counting the number of 'f's, how is that different from
> counting leading zeros?

The 'f's contribute to the value, the leading zeros do not.  For 
example, in Emacs 26 #x3fffffffffffffff is 2**62 (since it is rounded to 
a floating point number) whereas the visually-similar number 
#x3ffffffffffffff is (2**58 - 1). Although the rounding error should go 
away once we have bignums, the notational problems will become worse if 
%x and %o continue to print in a machine-dependent way.

That being said, it appears that there are enough qualms about the 
change that I plan to install it with the new variable set the other 
way. I.e., the default will be the current behavior and people can set 
the new variable to get the new behavior, to try it this behavior out to 
see whether they have problems.

bug#32252: [PATCH] %o and %x now format signed numbers

[Eli Zaretskii]

> Cc: eller.helmut@gmail.com, 32252@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 25 Jul 2018 10:21:43 -0700
> 
> That being said, it appears that there are enough qualms about the 
> change that I plan to install it with the new variable set the other 
> way. I.e., the default will be the current behavior and people can set 
> the new variable to get the new behavior, to try it this behavior out to 
> see whether they have problems.

Thank you.

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

Eli Zaretskii wrote:
>> Cc: eller.helmut@gmail.com, 32252@debbugs.gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>>
>> That being said, it appears that there are enough qualms about the
>> change that I plan to install it with the new variable set the other
>> way. I.e., the default will be the current behavior and people can set
>> the new variable to get the new behavior, to try it this behavior out to
>> see whether they have problems.
> 
> Thank you.

OK, I installed the attached patch which does that, and am closing the bug 
report. I plan to run with binary-as-unsigned set to nil and to report any 
problems I encounter. I encourage others (particularly skeptics :-) to do the same.

[-- Attachment #2: 0001-o-and-x-can-now-format-signed-integers.txt --]
[-- Type: text/plain, Size: 8770 bytes --]

From 4a56ca5bbfabbb9c581828cd91648346e6b03844 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 26 Jul 2018 00:34:10 -0700
Subject: [PATCH] %o and %x can now format signed integers
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Optionally treat integers as signed numbers with %o
and %x format specifiers, instead of treating them as
a machine-dependent two’s complement representation.
This option is more machine-independent, allows formats
like "#x%x" to be useful for reading later, and is
better-insulated for future changes involving bignums.
Setting the new variable ‘binary-as-unsigned’ to nil
enables the new behavior (Bug#32252).
This is a simplified version of the change proposed in:
https://lists.gnu.org/r/emacs-devel/2018-07/msg00763.html
I simplified that proposal by omitting bitwidth modifiers, as
I could not find an any example uses in the Emacs source code
that needed them and doing them correctly would have been
quite a bit more work for apparently little benefit.
* doc/lispref/strings.texi (Formatting Strings):
Document that %x and %o format negative integers in a
platform-dependent way.  Also, document how to format
numbers so that the same values can be read back in.
* etc/NEWS: Document the change.
* src/editfns.c (styled_format): Treat integers as signed
numbers even with %o and %x, if binary-as-unsigned is nil.
Support the + and space flags with %o and %x, since they’re
about signs.
(syms_of_editfns): New variable binary-as-unsigned.
* test/src/editfns-tests.el (read-large-integer):
Test that maximal integers can be read after printing
with all integer formats, if binary-as-unsigned is nil.
---
 doc/lispref/strings.texi  | 17 +++++++++++++++--
 etc/NEWS                  |  9 +++++++++
 src/editfns.c             | 43 ++++++++++++++++++++++++++++++++++++++-----
 test/src/editfns-tests.el | 10 ++++++----
 4 files changed, 68 insertions(+), 11 deletions(-)

diff --git a/doc/lispref/strings.texi b/doc/lispref/strings.texi
index 2fff3c7..3558f17 100644
--- a/doc/lispref/strings.texi
+++ b/doc/lispref/strings.texi
@@ -922,7 +922,8 @@ Formatting Strings
 @item %o
 @cindex integer to octal
 Replace the specification with the base-eight representation of an
-unsigned integer.  The object can also be a nonnegative floating-point
+integer.  Negative integers are formatted in a platform-dependent
+way.  The object can also be a nonnegative floating-point
 number that is formatted as an integer, dropping any fraction, if the
 integer does not exceed machine limits.
 
@@ -935,7 +936,8 @@ Formatting Strings
 @itemx %X
 @cindex integer to hexadecimal
 Replace the specification with the base-sixteen representation of an
-unsigned integer.  @samp{%x} uses lower case and @samp{%X} uses upper
+integer.  Negative integers are formatted in a platform-dependent
+way.  @samp{%x} uses lower case and @samp{%X} uses upper
 case.  The object can also be a nonnegative floating-point number that
 is formatted as an integer, dropping any fraction, if the integer does
 not exceed machine limits.
@@ -1108,6 +1110,17 @@ Formatting Strings
 precision is what the local library functions of the @code{printf}
 family produce.
 
+@cindex formatting numbers for rereading later
+  If you plan to use @code{read} later on the formatted string to
+retrieve a copy of the formatted value, use a specification that lets
+@code{read} reconstruct the value.  To format numbers in this
+reversible way you can use @samp{%s} and @samp{%S}, to format just
+integers you can also use @samp{%d}, and to format just nonnegative
+integers you can also use @samp{#x%x} and @samp{#o%o}.  Other formats
+may be problematic; for example, @samp{%d} and @samp{%g} can mishandle
+NaNs and can lose precision and type, and @samp{#x%x} and @samp{#o%o}
+can mishandle negative integers.  @xref{Input Functions}.
+
 @node Case Conversion
 @section Case Conversion in Lisp
 @cindex upper case
diff --git a/etc/NEWS b/etc/NEWS
index 995ceb6..089fc40 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -812,6 +812,15 @@ between two strings.
 ** 'print-quoted' now defaults to t, so if you want to see
 (quote x) instead of 'x you will have to bind it to nil where applicable.
 
++++
+** Numbers formatted via %o or %x may now be formatted as signed integers.
+This avoids problems in calls like (read (format "#x%x" -1)), and is
+more compatible with bignums, a planned feature.  To get this
+behavior, set the experimental variable binary-as-unsigned to nil,
+and if the new behavior breaks your code please email
+32252@debbugs.gnu.org.  Because %o and %x can now format signed
+integers, they now support the + and space flags.
+
 ** To avoid confusion caused by "smart quotes", the reader signals an
 error when reading Lisp symbols which begin with one of the following
 quotation characters: ‘’‛“”‟〞＂＇.  A symbol beginning with such a
diff --git a/src/editfns.c b/src/editfns.c
index 1d6040d..df25721 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -4196,8 +4196,8 @@ contain either numbered or unnumbered %-sequences but not both, except
 that %% can be mixed with numbered %-sequences.
 
 The + flag character inserts a + before any nonnegative number, while a
-space inserts a space before any nonnegative number; these flags only
-affect %d, %e, %f, and %g sequences, and the + flag takes precedence.
+space inserts a space before any nonnegative number; these flags
+affect only numeric %-sequences, and the + flag takes precedence.
 The - and 0 flags affect the width specifier, as described below.
 
 The # flag means to use an alternate display form for %o, %x, %X, %e,
@@ -4736,10 +4736,22 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 		}
 	      else
 		{
-		  /* Don't sign-extend for octal or hex printing.  */
 		  uprintmax_t x;
+		  bool negative;
 		  if (INTEGERP (arg))
-		    x = XUINT (arg);
+		    {
+		      if (binary_as_unsigned)
+			{
+			  x = XUINT (arg);
+			  negative = false;
+			}
+		      else
+			{
+			  EMACS_INT i = XINT (arg);
+			  negative = i < 0;
+			  x = negative ? -i : i;
+			}
+		    }
 		  else
 		    {
 		      double d = XFLOAT_DATA (arg);
@@ -4747,8 +4759,13 @@ styled_format (ptrdiff_t nargs, Lisp_Object *args, bool message)
 		      if (! (0 <= d && d < uprintmax + 1))
 			xsignal1 (Qoverflow_error, arg);
 		      x = d;
+		      negative = false;
 		    }
-		  sprintf_bytes = sprintf (sprintf_buf, convspec, prec, x);
+		  sprintf_buf[0] = negative ? '-' : plus_flag ? '+' : ' ';
+		  bool signedp = negative | plus_flag | space_flag;
+		  sprintf_bytes = sprintf (sprintf_buf + signedp,
+					   convspec, prec, x);
+		  sprintf_bytes += signedp;
 		}
 
 	      /* Now the length of the formatted item is known, except it omits
@@ -5558,6 +5575,22 @@ functions if all the text being accessed has this property.  */);
   DEFVAR_LISP ("operating-system-release", Voperating_system_release,
 	       doc: /* The release of the operating system Emacs is running on.  */);
 
+  DEFVAR_BOOL ("binary-as-unsigned",
+	       binary_as_unsigned,
+	       doc: /* Non-nil means `format' %x and %o treat integers as unsigned.
+This has machine-dependent results.  Nil means to treat integers as
+signed, which is portable; for example, if N is a negative integer,
+(read (format "#x%x") N) returns N only when this variable is nil.
+
+This variable is experimental; email 32252@debbugs.gnu.org if you need
+it to be non-nil.  */);
+  /* For now, default to true if bignums exist, false in traditional Emacs.  */
+#ifdef lisp_h_FIXNUMP
+  binary_as_unsigned = true;
+#else
+  binary_as_unsigned = false;
+#endif
+
   defsubr (&Spropertize);
   defsubr (&Schar_equal);
   defsubr (&Sgoto_char);
diff --git a/test/src/editfns-tests.el b/test/src/editfns-tests.el
index c828000..2951270 100644
--- a/test/src/editfns-tests.el
+++ b/test/src/editfns-tests.el
@@ -165,10 +165,12 @@ transpose-test-get-byte-positions
                 :type 'overflow-error)
   (should-error (read (substring (format "%d" most-negative-fixnum) 1))
                 :type 'overflow-error)
-  (should-error (read (format "#x%x" most-negative-fixnum))
-                :type 'overflow-error)
-  (should-error (read (format "#o%o" most-negative-fixnum))
-                :type 'overflow-error)
+  (let ((binary-as-unsigned nil))
+    (dolist (fmt '("%d" "%s" "#o%o" "#x%x"))
+      (dolist (val (list most-negative-fixnum (1+ most-negative-fixnum)
+                         -1 0 1
+                         (1- most-positive-fixnum) most-positive-fixnum))
+        (should (eq val (read (format fmt val)))))))
   (should-error (read (format "#32rG%x" most-positive-fixnum))
                 :type 'overflow-error))
 
-- 
2.7.4

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

> OK, I installed the attached patch which does that, and am closing the
> bug report. I plan to run with binary-as-unsigned set to nil and to
> report any problems I encounter. I encourage others (particularly
> skeptics :-) to do the same.

So you actually changed the default behavior even if you said you
wouldn't?

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

Helmut Eller wrote:
> So you actually changed the default behavior even if you said you
> wouldn't?

That was not my intent. Sorry, I messed up the default settings in a last-minute 
change. Thanks for catching that so quickly. I installed the attached to fix it.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-src-editfns.c-syms_of_editfns-Fix-typo-in-previous-c.patch --]
[-- Type: text/x-patch; name="0001-src-editfns.c-syms_of_editfns-Fix-typo-in-previous-c.patch", Size: 826 bytes --]

From e4d6ebee213e2d6f37de839c09c05ab0a5346ce3 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Thu, 26 Jul 2018 01:14:31 -0700
Subject: [PATCH] * src/editfns.c (syms_of_editfns): Fix typo in previous
 change.

---
 src/editfns.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/editfns.c b/src/editfns.c
index df25721..522cb5d 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -5586,9 +5586,9 @@ This variable is experimental; email 32252@debbugs.gnu.org if you need
 it to be non-nil.  */);
   /* For now, default to true if bignums exist, false in traditional Emacs.  */
 #ifdef lisp_h_FIXNUMP
-  binary_as_unsigned = true;
-#else
   binary_as_unsigned = false;
+#else
+  binary_as_unsigned = true;
 #endif
 
   defsubr (&Spropertize);
-- 
2.7.4

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Tue, Jul 24 2018, Paul Eggert wrote:

> Helmut Eller wrote:
>
>> In practice, printf ("%x", (int)N) prints the bits in N as unsigned
>> integer not as signed integer.
>
> That's two operations not one, and even there the results disagree
> with what (format "%x" N) does. On every Emacs platform I've ever used
> the C output begins with a different hex digit than the Emacs Lisp
> output. Emacs has never done %x just like that unportable C code, and
> likely will never do so.

Nobody is arguing for printing the tag-bits.  And what Emacs has done is
fairly obvious to everybody.

>> #x-1 maybe interesting to the read function but it's not interesting to
>> humans.  Humans want to see #x3fffffffffffffff.
>
> I doubt very much that most humans really want to see an error-prone
> notation like that (can you easily spot the difference between it and
> #x3ffffffffffffffff? I can't). And even if some humans did want it,
> they won't get it on 32-bit Emacs. This whole idea is a bad approach
> for Emacs.

What's more interesting:
(format "%x" (lognot 8)) => "-9"
or
(format "%x" (lognot 8)) => "3ffffffffffffff7"

For me, the first version is totally useless.

> What I expect most users would prefer is an approach where 'read' and
> 'format' are inverses, e.g., (read (printf "#x%x" N)) returns N for
> every integer N. Emacs doesn't do that now, and the proposed patch
> fixes things so that it does.

The inverse of read is print, not format or printf. 

> No other approach has been proposed that
> would make much sense (that is, be portable, be easily extendible to
> bignums, that sort of thing).

Of course there have been proposals: Do your bignum stuff with a
different format specifier.

Here is another proposal: Add a read syntax for unsigned fixnums like
#x3fffffffffffffffu or alternatively #xu3fffffffffffffff.

>> you should have no problem with introducing an
>> new specifier for this.
>
> I'd rather not, as the hex stuff has never worked right in Emacs and
> we shouldn't be codifying old bugs.

It was already codified in the documentation, even the much more dubios
flonum case.  It's really annoying when documented behavior changes.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Helmut Eller wrote:

> What's more interesting:
> (format "%x" (lognot 8)) => "-9"
> or
> (format "%x" (lognot 8)) => "3ffffffffffffff7"
> 
> For me, the first version is totally useless.

Shrug. It's what Common Lisp and Scheme do, and it works pretty well once you 
get used to it. Programs that need negative integers displayed modulo some power 
of 2 can use the mod or logand functions; that's the mathematically right way to 
do it anyway, and it's machine-independent.

> Of course there have been proposals: Do your bignum stuff with a
> different format specifier.

And prohibit %x on bignums? That would make little sense. Common Lisp and Scheme 
don't have any such prohibition; why should Emacs Lisp? Again, programs that 
need just the low-order bits of a negative integer can use 'mod' or a mask.

> Here is another proposal: Add a read syntax for unsigned fixnums like
> #x3fffffffffffffffu or alternatively #xu3fffffffffffffff.

That's heading down the wrong path. Emacs Lisp does not have unsigned fixnums, 
so why add a syntax for a data type that does not exist? And Emacs Lisp should 
not add such a data type, as it is a low-level machine concept unsuitable for 
Lisp, is not needed in Emacs Lisp, and would cause unnecessary complexity in 
documentation and implementation.

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Thu, Jul 26 2018, Paul Eggert wrote:

> Helmut Eller wrote:
>
>> What's more interesting:
>> (format "%x" (lognot 8)) => "-9"
>> or
>> (format "%x" (lognot 8)) => "3ffffffffffffff7"
>>
>> For me, the first version is totally useless.
>
> Shrug. It's what Common Lisp and Scheme do, and it works pretty well
> once you get used to it.

It's because I happen to know a thing or two about Common Lisp that I
know that I almost always would prefer the way that Emacs Lisp prints
negative fixnums.

> Programs that need negative integers
> displayed modulo some power of 2 can use the mod or logand functions;
> that's the mathematically right way to do it anyway, and it's
> machine-independent.

Do you think I don't know that?  It's always possible to write a helper
function that does the formatting in the variant that you need.

>> Of course there have been proposals: Do your bignum stuff with a
>> different format specifier.
>
> And prohibit %x on bignums? That would make little sense.

It would prohibit only negative bignums just like negative flonums are
forbidden.  Besides I still think that it would make much more sense to
print flonums the way %a does in C; another reason for having %a or
another format specifier to do the hex conversion in a world with
bignums.

> Common Lisp and Scheme don't have any such prohibition; why should
> Emacs Lisp?

Because Emacs Lisp was very successful without bignums.  Even more
successful than most other Lisps with all their fancy bignums, complex
numbers, ratios and other numeric stuff that hardly anyone needs.  And
the stuff that almost everybody would like to have like, fast unboxed
arithmetic, is not offered in a reliable and portable way.

>> Here is another proposal: Add a read syntax for unsigned fixnums like
>> #x3fffffffffffffffu or alternatively #xu3fffffffffffffff.
>
> That's heading down the wrong path. Emacs Lisp does not have unsigned
> fixnums, so why add a syntax for a data type that does not exist?  And
> Emacs Lisp should not add such a data type, as it is a low-level
> machine concept unsuitable for Lisp, is not needed in Emacs Lisp, and
> would cause unnecessary complexity in documentation and
> implementation.

In my mind Emacs does have negative fixnums and there are various ways
to write negative fixnums.  Of course a special syntax is not absolutely
needed but it may be convenient.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Helmut Eller wrote:
>> And prohibit %x on bignums? That would make little sense.
> It would prohibit only negative bignums just like negative flonums are
> forbidden.

So under this proposal %x would generate ambiguous output once we add bignums? 
For example, on a 32-bit platform (format "%x" -1) would generate the same 
output as (format "%x" #x3fffffff), even though -1 and #x3fffffff would be 
different integers? That doesn't sound like a good idea.

> it would make much more sense to print flonums the way %a does in C

It would make sense to add an %a specification, like C's %a. But C's %a formats 
floating-point numbers, and surely Emacs Lisp's %a should be similar. That is, 
%a should treat integers the way the other floating-point specs do (%e, %f, %g).

>> Common Lisp and Scheme don't have any such prohibition; why should
>> Emacs Lisp?
> 
> Because Emacs Lisp was very successful without bignums.

This appears to be more an argument against bignums than anything else. I've 
used Emacs Lisp without bignums for many years, and it's *always* been a 
hindrance. I would much rather have bignums, as they would make a lot of things 
simpler. I agree that Emacs Lisp does not much need complex numbers or ratios.

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Thu, Jul 26 2018, Paul Eggert wrote:

> Helmut Eller wrote:
>>> And prohibit %x on bignums? That would make little sense.
>> It would prohibit only negative bignums just like negative flonums are
>> forbidden.
>
> So under this proposal %x would generate ambiguous output once we add
> bignums? For example, on a 32-bit platform (format "%x" -1) would
> generate the same output as (format "%x" #x3fffffff), even though -1
> and #x3fffffff would be different integers? That doesn't sound like a
> good idea.

Indeed, had forgotten that case.

>>> Common Lisp and Scheme don't have any such prohibition; why should
>>> Emacs Lisp?
>>
>> Because Emacs Lisp was very successful without bignums.
>
> This appears to be more an argument against bignums than anything
> else. I've used Emacs Lisp without bignums for many years, and it's
> *always* been a hindrance. I would much rather have bignums, as they
> would make a lot of things simpler.

Well, it's an argument to promote the special role of fixnums.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Andreas Schwab]

On Jul 26 2018, Helmut Eller <eller.helmut@gmail.com> wrote:

> Because Emacs Lisp was very successful without bignums.

The lack of bignums requires workarounds like the first two elements of
(current-time) or the 11th and 12th element of (file-attributes).

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#32252: [PATCH] %o and %x now format signed numbers

[Robert Pluim]

Andreas Schwab <schwab@suse.de> writes:

> On Jul 26 2018, Helmut Eller <eller.helmut@gmail.com> wrote:
>
>> Because Emacs Lisp was very successful without bignums.
>
> The lack of bignums requires workarounds like the first two elements of
> (current-time) or the 11th and 12th element of (file-attributes).

Not to mention having to represent IP addresses as vectors, thus
forcing operations on them to be byte by byte.

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Thu, Jul 26 2018, Andreas Schwab wrote:

> On Jul 26 2018, Helmut Eller <eller.helmut@gmail.com> wrote:
>
>> Because Emacs Lisp was very successful without bignums.
>
> The lack of bignums requires workarounds like the first two elements of
> (current-time) or the 11th and 12th element of (file-attributes).

Yes, that's one of the few places where bignums would be useful. But
having a time data-type, maybe just a struct and functions like time-add
etc, would also be good solution.  How often does one need to print a
time value in hex notation?  Probably so rareley that format-time-string
doesn't even support that.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

On 07/26/2018 02:56 AM, Helmut Eller wrote:
> How often does one need to print a
> time value in hex notation?

It happens every time I run Emacs, in standard code shipped with Emacs. 
I discovered this while doing compatibility checking before I proposed 
this change. (The code in question continues to work just fine when 
binary-as-unsigned is nil.)


> format-time-string doesn't even support that.

We're talking about 'format' not 'format-time-string', and Emacs code 
does apply 'format' with %x to the tricky little integers that 
'current-time' returns. It's a real hassle, and this hassle is pervasive 
in Emacs. Having bignums would simplify it significantly.

And this problem is not limited to times. It's also file sizes, buffer 
and string sizes (currently limited to fixnums, but they shouldn't be), 
glyph codes, inode numbers, device numbers, file descriptor numbers, 
inotify masks, windowing parameters, and other stuff I don't even know 
about. It's pervasive and is a real hassle.

By the way, the format %x compatibility checking I mentioned is partly 
why I have confidence that the practical effects of this change will be 
minor. Many of the problems that I thought might happen, don't in fact 
happen, because people who attempt to format negative numbers with %x 
have already run into portability hassles on 32- vs 64-bit platforms, 
and so they have written their code so that the platform choice doesn't 
matter, either by ensuring that the integer is nonnegative or by not 
caring what string is generated so long as it doesn't lose information. 
Code written like this (and so far, that's all the code I've found) 
works just fine when 'format'  %x works like it does in Common Lisp or 
Scheme.

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Thu, Jul 26 2018, Paul Eggert wrote:

> We're talking about 'format' not 'format-time-string', and Emacs code
> does apply 'format' with %x to the tricky little integers that
> 'current-time' returns. It's a real hassle, and this hassle is
> pervasive in Emacs. Having bignums would simplify it significantly.

Where is that code?  And why would it be difficult to teach
format-time-string to deal with the tricky bits that current-time
returns.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

On 07/26/2018 10:16 AM, Helmut Eller wrote:
> Where is that code?

One example is url-digest-auth-make-cnonce. This was the first I ran 
into. No doubt there are others.

> And why would it be difficult to teach
> format-time-string to deal with the tricky bits that current-time
> returns.

?! format-time-string already does that. And it's a hassle. Other Emacs 
Lisp code inspects timestamps directly too. And it's a hassle for that 
code too. Bignums would significantly lessen this hassle. Surely this is 
obvious.

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Thu, Jul 26 2018, Paul Eggert wrote:

> On 07/26/2018 10:16 AM, Helmut Eller wrote:
>> Where is that code?
>
> One example is url-digest-auth-make-cnonce. This was the first I ran
> into. No doubt there are others.

And how would url-digest-auth-make-cnonce look with bignums?

>> And why would it be difficult to teach
>> format-time-string to deal with the tricky bits that current-time
>> returns.
>
> ?! format-time-string already does that. And it's a hassle. Other
> Emacs Lisp code inspects timestamps directly too. And it's a hassle
> for that code too. Bignums would significantly lessen this
> hassle. Surely this is obvious.

To me it's not so obvious.  E.g. current-time is supposed to "return the
current time, as the number of seconds since [epoch]".  So, how are you
going to represent micro-seconds with bignums?

Or, e.g. disassemble_lisp_time deals with formats that are the form
(HIGH LOW USEC PSEC) or formats that were used in previous Emacs
versions.  With bignums, disassemble_lisp_time has to support the old
formats and bignums too.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Helmut Eller wrote:
> On Thu, Jul 26 2018, Paul Eggert wrote:
> 
>> On 07/26/2018 10:16 AM, Helmut Eller wrote:
>>> Where is that code?
>>
>> One example is url-digest-auth-make-cnonce. This was the first I ran
>> into. No doubt there are others.
> 
> And how would url-digest-auth-make-cnonce look with bignums?

Just like it does now. It would not need to change.

> With bignums, disassemble_lisp_time has to support the old
> formats and bignums too.

Yes, of course. That's not much trouble. The point is that other Lisp code can 
simply deal with just one integer rather than a tricky list of four integers 
with mixed bases.

bug#32252: [PATCH] %o and %x now format signed numbers

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Date: Tue, 24 Jul 2018 06:49:34 +0200
> Cc: 32252@debbugs.gnu.org
> 
> On Mon, Jul 23 2018, Paul Eggert wrote:
> 
> > %a has quite a different meaning in C11, so we'd need to use some
> 
> With your change %x will also have quite a different meaning in C11.
> 
> > other letter (or set of letters, if we want something for %x, %X, %o)
> > if we went that route. This seems like overkill for such a small
> > change to functionality that hardly ever matters; plus, %x, %X and %o
> > would continue to have problematic machine-dependent semantics and
> > would not be sensibly extensible to bignums.
> 
> If it doesn't matter to you then why make the change/break at all?  You
> could just create a function bignum-to-hex-string.

Do we really need to have identical or consistent behavior for fixnums
and bignums?  They are different beasts, so the behavior could be
different, provided that it makes sense for each of the varieties.

More generally, given the controversy, maybe we should collect some
real-life experience before we start making incompatible behavior
changes?  I mean, the bignum branch is not even merged yet, and we are
already changing the behavior with fixnums in incompatible ways.
Maybe we should merge the branch first, let the dust settle and let
people use the new functionality, then revisit this stuff with more
experience on our hands.

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Eli Zaretskii wrote:

> Do we really need to have identical or consistent behavior for fixnums
> and bignums?  They are different beasts, so the behavior could be
> different, provided that it makes sense for each of the varieties.

I don't know what inconsistent behavior would make sense here. There's a 
longstanding tradition in Lisp that integer arithmetic just works, and I don't 
see why Emacs would want to fight against that tradition here. Although there 
may be some low-level functions that distinguish between fixnums and bignums for 
efficiency reasons, the vast majority of Lisp functions should not distinguish 
them, and 'format' should be one of those functions.
> Maybe we should merge the branch first, let the dust settle and let
> people use the new functionality, then revisit this stuff with more
> experience on our hands.

This particular patch is not primarily about bignums. It's more about (read 
(printf "#x%x" N)) returning N, which is basic functionality that should work 
even if we never add bignums. The fact that the patch also will make bignums 
easier is just icing on the cake.

Although I understand the concern about the patch, the concern doesn't seem to 
be warranted, and any compatibility issues can easily be addressed by setting 
binary-as-signed.

bug#32252: [PATCH] %o and %x now format signed numbers

[Andreas Schwab]

On Jul 24 2018, Paul Eggert <eggert@cs.ucla.edu> wrote:

> It's more about (read (printf "#x%x" N)) returning N, which is basic
> functionality that should work

Where is that documented?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Andreas Schwab wrote:
>> It's more about (read (printf "#x%x" N)) returning N, which is basic
>> functionality that should work
> Where is that documented?

You're right, it's not. It's not even documented that (read (printf "%s" N)) 
should work for any number N. I'll add something along these lines. I'll also 
warn that (read (printf "#x%x" N)) might not work.

bug#32252: [PATCH] %o and %x now format signed numbers

[Eli Zaretskii]

> Cc: 32252@debbugs.gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 24 Jul 2018 17:54:57 -0700
> 
> Eli Zaretskii wrote:
> 
> > Do we really need to have identical or consistent behavior for fixnums
> > and bignums?  They are different beasts, so the behavior could be
> > different, provided that it makes sense for each of the varieties.
> 
> I don't know what inconsistent behavior would make sense here. There's a 
> longstanding tradition in Lisp that integer arithmetic just works, and I don't 
> see why Emacs would want to fight against that tradition here. Although there 
> may be some low-level functions that distinguish between fixnums and bignums for 
> efficiency reasons, the vast majority of Lisp functions should not distinguish 
> them, and 'format' should be one of those functions.

I agree that this would be nice to have, but I'm not sure we should
absolutely require that, at least not yet.  Not before we have enough
experience to be sure we have good reasons for changing long-standing
behavior.

> Although I understand the concern about the patch, the concern doesn't seem to 
> be warranted, and any compatibility issues can easily be addressed by setting 
> binary-as-signed.

Well, we've seen such movies in the past, and we know we've been wrong
in at least some cases.

bug#32252: [PATCH] %o and %x now format signed numbers

[Brett Gilio]

Paul Eggert writes:
> I don't know what inconsistent behavior would make sense here. 
> There's a
> longstanding tradition in Lisp that integer arithmetic just 
> works, and I don't
> see why Emacs would want to fight against that tradition here. 
> Although there
> may be some low-level functions that distinguish between fixnums 
> and bignums for
> efficiency reasons, the vast majority of Lisp functions should 
> not distinguish
> them, and 'format' should be one of those functions.
>> Maybe we should merge the branch first, let the dust settle and 
>> let
>> people use the new functionality, then revisit this stuff with 
>> more
>> experience on our hands.

Paul, I agree with you here. However, I wonder what you mean by 
"just
works" in contrast to what Eli suggested about the inconsistency 
in
behavior between the different function definitions for fixnums 
and
bignums.

How would that be working against the lisp tradition?



-- 
Brett M. Gilio
Free Software Foundation, Member
https://parabola.nu | https://emacs.org

bug#32252: [PATCH] %o and %x now format signed numbers

[Paul Eggert]

Brett Gilio wrote:
> I wonder what you mean by "just
> works" in contrast to what Eli suggested about the inconsistency in
> behavior between the different function definitions for fixnums and
> bignums.

I meant that ordinarily Emacs Lisp programmers should not worry whether an 
integer's internal representation is a bignum or a fixnum. Functions like +, -, 
*, /,  format, read, etc. should accept and generate mathematically correct 
integer values regardless of whether their inputs are fixnums or bignums. If an 
integer fits in a fixnum it is presumably represented that way, but by and large 
Emacs programs should not care about this detail.

That is how other Lisps do it, it works well for them, and Emacs Lisp should do 
likewise.

bug#32252: [PATCH] %o and %x now format signed numbers

[Andy Moreton]

On Tue 24 Jul 2018, Paul Eggert wrote:

> Helmut Eller wrote:
>> With your change %x will also have quite a different meaning in C11.
>
> Not really, as Emacs (format "%x" N) agrees with C11 printf ("%x", N) in all
> values of N that are valid in both languages. In C11, negative values are not
> valid, as printf ("%x", N) has undefined behavior when N is negative. So we
> are discussing an area where Emacs Lisp can define behavior without
> introducing incompatibilities with C11.

As emacs fixnums are signed, and the C printf conversion specifier "%x"
takes an unsigned int argument, there are expected to be differences in
behaviour.

However what matters in C (and in elisp) is not what the spec says, but
what the codebase of existing users does, and what existing library
implementations do.

When printing values with base!=10, it is always the underlying
representation (i.e. bit pattern) that is of interest, not the value
interpreted as a signed number. Long standing practice in many languages
shows values printed in hex, octal and binary as unsigned.

> If we changed (format "%x" -1) to signal an error instead, that would also be
> upward-compatible with C11. However, it's more useful for something like
> (format "#x%x" -1) to output a string that can 'read' can scan to get -1,
> something that's not true of Emacs now.

I agree that there is a problem, but not with your solution.

I think it would be better to change the reader to treat non-base10
values as if they were unsigned representations, so that `format' is
unchanged, and (read (format "#x%x" -1)) evaluates to -1.

That is consistent wth other languages and user expectation.

> It does matter to me, actually. I think Emacs should have sensible behavior
> even in corner cases that hardly ever arise in real programs.

I agree: a consistent model of behaviour is important, even if it may
take a while to fix buggy behaviour that disagrees with the model.

    AndyM

bug#32252: [PATCH] %o and %x now format signed numbers

[Andy Moreton]

On Tue 24 Jul 2018, Paul Eggert wrote:

> Eli Zaretskii wrote:
>
>> Do we really need to have identical or consistent behavior for fixnums
>> and bignums?  They are different beasts, so the behavior could be
>> different, provided that it makes sense for each of the varieties.
>
> I don't know what inconsistent behavior would make sense here. There's a
> longstanding tradition in Lisp that integer arithmetic just works, and I don't
> see why Emacs would want to fight against that tradition here. Although there
> may be some low-level functions that distinguish between fixnums and bignums
> for efficiency reasons, the vast majority of Lisp functions should not
> distinguish them, and 'format' should be one of those functions.
>> Maybe we should merge the branch first, let the dust settle and let
>> people use the new functionality, then revisit this stuff with more
>> experience on our hands.
>
> This particular patch is not primarily about bignums. It's more about (read
> (printf "#x%x" N)) returning N, which is basic functionality that should work
> even if we never add bignums. The fact that the patch also will make bignums
> easier is just icing on the cake.

It makes things more consistent at a cost of incomatibility with long
standing behaviour, and of making (format "%x") output useless for
humans to read. That is far too high a price for consistency.

> Although I understand the concern about the patch, the concern doesn't seem to
> be warranted, and any compatibility issues can easily be addressed by setting
> binary-as-signed.

No. The default value for this option should preserve backward
compatible behaviour.

    AndyM

bug#32252: [PATCH] %o and %x now format signed numbers

[Andy Moreton]

On Wed 25 Jul 2018, Paul Eggert wrote:

> Andreas Schwab wrote:
>>> It's more about (read (printf "#x%x" N)) returning N, which is basic
>>> functionality that should work
>> Where is that documented?
>
> You're right, it's not. It's not even documented that (read (printf "%s" N))
> should work for any number N. I'll add something along these lines. I'll also
> warn that (read (printf "#x%x" N)) might not work.

I think you meant (read (format "%s" N)).

    AndyM

bug#32252: [PATCH] %o and %x now format signed numbers

[Andy Moreton]

On Thu 26 Jul 2018, Paul Eggert wrote:

> Brett Gilio wrote:
>> I wonder what you mean by "just
>> works" in contrast to what Eli suggested about the inconsistency in
>> behavior between the different function definitions for fixnums and
>> bignums.
>
> I meant that ordinarily Emacs Lisp programmers should not worry whether an
> integer's internal representation is a bignum or a fixnum. Functions like +,
> -, 
> *, /,  format, read, etc. should accept and generate mathematically correct
> integer values regardless of whether their inputs are fixnums or bignums. If
> an integer fits in a fixnum it is presumably represented that way, but by and
> large Emacs programs should not care about this detail.
>
> That is how other Lisps do it, it works well for them, and Emacs Lisp should
> do likewise.

That is uncontroversial for arithmetic operations, but how are bitwise
logical operations handled in other lisps with bignums ?

    AndyM

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Thu, Jul 26 2018, Andy Moreton wrote:

>> That is how other Lisps do it, it works well for them, and Emacs Lisp should
>> do likewise.
>
> That is uncontroversial for arithmetic operations, but how are bitwise
> logical operations handled in other lisps with bignums ?

In Common Lisp those are defined to behave as if integers were
represented in two's complement form, regardless of how integers are
represented internally.

Interestinlgy, Common Lisp has ASH but no direct equivalent to Emacs
Lisp's lsh.  Let's see how the bignum proponents extend lsh.

Helmut

bug#32252: [PATCH] %o and %x now format signed numbers

[Helmut Eller]

On Thu, Jul 26 2018, Paul Eggert wrote:

> Helmut Eller wrote:
>> So you actually changed the default behavior even if you said you
>> wouldn't?
>
> That was not my intent. Sorry, I messed up the default settings in a
> last-minute change. Thanks for catching that so quickly. I installed
> the attached to fix it.

So you actually changed the default behavior again even if you said you
wouldn't?

Helmut

bug#32252: i find binary-as-unsigned to be very helpful

[snickerbockers]

hello,

I use binary-as-unsigned in my .emacs, and I saw that the help
page
said to e-mail you if I find it useful.  Below is a function i
made that
displays a 32-bit two's complement of a given integer in hex.  I
do a
lot of reverse-engineering of low-level programs, so it is very
useful
for me to have a quick way to see how a given integer is actually
represented in the hardware.  if binary-as-unsigned wasn't
available to
make format do most of the work for me, this function would be a
lot
longer and more complicated.

(defun to-hex(value)
 "convert a signed int to its 32-bit two's complement in hex
 notation"
 (interactive "svalue:")
(let ((binary-as-unsigned t))
 (message (substring (format "%08x" (cl-parse-integer value))
 -8))))

(global-set-key (kbd "C-c h") 'to-hex)

Thanks,
snickerbockers