Re: issues with recent doprnt-related changes

Re: issues with recent doprnt-related changes

[Paul Eggert]

On 05/04/11 02:52, Eli Zaretskii wrote:
>> It seems from that discussion that strings can contain MOST_POSITIVE_FIXNUM bytes.
> I think the conclusion was that it can contain MOST_POSITIVE_FIXNUM
> _including_the_terminating_null_.

Hmm, that's not how I read
<http://lists.gnu.org/archive/html/emacs-devel/2011-04/msg00923.html>.

I understood the argument to be that a buffer must contain at most
MOST_POSITIVE_FIXNUM - 1 bytes due to other reasons, but it's OK
to have "a string whose length is MOST_POSITIVE_FIXNUM", i.e.,
(= (length STRING) most-positive-fixnum), because we already check
buffer sizes before inserting strings.  If you
count the trailing byte, the length of the underlying C character
array would be MOST_POSITIVE_FIXNUM + 1.

I'll CC: this to emacs-devel just in case I misinterpreted that.

Re: issues with recent doprnt-related changes

[Eli Zaretskii]

> Date: Wed, 04 May 2011 07:56:43 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: 8545@debbugs.gnu.org, Emacs Development <emacs-devel@gnu.org>
> 
> On 05/04/11 02:52, Eli Zaretskii wrote:
> >> It seems from that discussion that strings can contain MOST_POSITIVE_FIXNUM bytes.
> > I think the conclusion was that it can contain MOST_POSITIVE_FIXNUM
> > _including_the_terminating_null_.
> 
> Hmm, that's not how I read
> <http://lists.gnu.org/archive/html/emacs-devel/2011-04/msg00923.html>.
> 
> I understood the argument to be that a buffer must contain at most
> MOST_POSITIVE_FIXNUM - 1 bytes due to other reasons, but it's OK
> to have "a string whose length is MOST_POSITIVE_FIXNUM", i.e.,
> (= (length STRING) most-positive-fixnum), because we already check
> buffer sizes before inserting strings.  If you
> count the trailing byte, the length of the underlying C character
> array would be MOST_POSITIVE_FIXNUM + 1.

Stefan, could you please chime in?

Re: issues with recent doprnt-related changes

[Stefan Monnier]

>> >> It seems from that discussion that strings can contain
>> >> MOST_POSITIVE_FIXNUM bytes.
>> > I think the conclusion was that it can contain MOST_POSITIVE_FIXNUM
>> > _including_the_terminating_null_.
>> 
>> Hmm, that's not how I read
>> <http://lists.gnu.org/archive/html/emacs-devel/2011-04/msg00923.html>.
>> 
>> I understood the argument to be that a buffer must contain at most
>> MOST_POSITIVE_FIXNUM - 1 bytes due to other reasons, but it's OK
>> to have "a string whose length is MOST_POSITIVE_FIXNUM", i.e.,
>> (= (length STRING) most-positive-fixnum), because we already check
>> buffer sizes before inserting strings.  If you
>> count the trailing byte, the length of the underlying C character
>> array would be MOST_POSITIVE_FIXNUM + 1.

> Stefan, could you please chime in?

I'm not sure I understand the details more than you do.  AFAIK the
MOST_POSITIVE_FIXNUM limit only comes into play when the number goes
through Lisp_Object somewhere.  And we never show "length, including
terminating nul-byte" to Elisp, so the nul-byte shouldn't matter: while
the C array will have size MOST_POSITIVE_FIXNUM + 1, this will only be
represented in EMACS_INT which can accommodate it just fine.


        Stefan

Re: issues with recent doprnt-related changes

[Paul Eggert]

On 05/06/11 06:33, Stefan Monnier wrote:
> I'm not sure I understand the details more than you do.  AFAIK the
> MOST_POSITIVE_FIXNUM limit only comes into play when the number goes
> through Lisp_Object somewhere.  And we never show "length, including
> terminating nul-byte" to Elisp, so the nul-byte shouldn't matter: while
> the C array will have size MOST_POSITIVE_FIXNUM + 1, this will only be
> represented in EMACS_INT which can accommodate it just fine.

First, you say you're not sure you understand the details;
then, you explain the situation perfectly!  :-)

Thanks.

Re: issues with recent doprnt-related changes

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  8545@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Fri, 06 May 2011 10:33:21 -0300
> 
> And we never show "length, including
> terminating nul-byte" to Elisp, so the nul-byte shouldn't matter

If you are sure of that, then I agree it's fine to use
MOST_POSITIVE_FIXNUM+1.  I wasn't sure.  I do see in C variables that
explicitly use the string position of the null byte; I was afraid that
they could somehow leak to Lisp.

Re: issues with recent doprnt-related changes

[Stefan Monnier]

>> And we never show "length, including
>> terminating nul-byte" to Elisp, so the nul-byte shouldn't matter

> If you are sure of that, then I agree it's fine to use
> MOST_POSITIVE_FIXNUM+1.  I wasn't sure.  I do see in C variables that
> explicitly use the string position of the null byte; I was afraid that
> they could somehow leak to Lisp.

I don't know enough to guarantee you that they can never leak to Lisp,
but I can't think of any reason why they should.  And note that "the
position of the nul byte" is the same as "the length of the list", so
it's still <= MOST_POSITIVE_FIXNUM.  It's only the position after the
nul byte that would overflow.


        Stefan

Re: issues with recent doprnt-related changes

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: eggert@cs.ucla.edu,  8545@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Fri, 06 May 2011 14:13:06 -0300
> 
> note that "the position of the nul byte" is the same as "the length
> of the list", so it's still <= MOST_POSITIVE_FIXNUM.  It's only the
> position after the nul byte that would overflow.

But what about this code and its commentary (from
next_element_from_c_string):

  /* IT's position can be greater IT->string_nchars in case a field
     width or precision has been specified when the iterator was
     initialized.  */
  if (IT_CHARPOS (*it) >= it->end_charpos)
    {
      /* End of the game.  */
      ...

This happens when the iterator is initialized by reseat_to_string.
Admittedly, it's not very practical to have such huge strings that are
padded to an even larger width...

Re: issues with recent doprnt-related changes

[Stefan Monnier]

>> note that "the position of the nul byte" is the same as "the length
>> of the list", so it's still <= MOST_POSITIVE_FIXNUM.  It's only the
>> position after the nul byte that would overflow.

> But what about this code and its commentary (from
> next_element_from_c_string):

>   /* IT's position can be greater IT->string_nchars in case a field
>      width or precision has been specified when the iterator was
>      initialized.  */
>   if (IT_CHARPOS (*it) >= it->end_charpos)
>     {
>       /* End of the game.  */
>       ...

Do these ever make it into a Lisp_Object?


        Stefan

Re: issues with recent doprnt-related changes

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: eggert@cs.ucla.edu,  8545@debbugs.gnu.org,  emacs-devel@gnu.org
> Date: Sat, 07 May 2011 00:18:47 -0300
> 
> >> note that "the position of the nul byte" is the same as "the length
> >> of the list", so it's still <= MOST_POSITIVE_FIXNUM.  It's only the
> >> position after the nul byte that would overflow.
> 
> > But what about this code and its commentary (from
> > next_element_from_c_string):
> 
> >   /* IT's position can be greater IT->string_nchars in case a field
> >      width or precision has been specified when the iterator was
> >      initialized.  */
> >   if (IT_CHARPOS (*it) >= it->end_charpos)
> >     {
> >       /* End of the game.  */
> >       ...
> 
> Do these ever make it into a Lisp_Object?

Well, the resulting string can be returned by format-mode-line, for
example.