bug#17507: Documentation for `add-text-property' and relatives

bug#17507: Documentation for `add-text-property' and relatives

[Eli Barzilay]

It was unclear to me how to use `add-text-property' and its relatives
(I actually started with `font-lock-append-text-property') with
strings, since they don't return the string.  It also looks like it
won't be practical to ask for a change where they do return the string
(if given one) since `add-text-properties' actually has a specified
return value (which is arguably not as useful as getting back a
string...).

In any case, I eventually found out what I need to do, but it took me
a while because I was confused.  This is a bit verbose, but please
bear with me...

What I was surprised to find out is that

    (let ((s "foo")) (put-text-property 0 3 'foo 'bar s) s)

works.  It's probably obvious for elisp hackers, but coming from
most other lisps/schemes, I kind of assumed that the #("foo" ...)
thing is made of the "foo" string in some vector-like container that
holds the properties.  Given that unconscious assumption, I assumed
that it wouldn't work, and that I'll need some uglier hack like

    (let ((s (propertize "foo" 'foo nil)))
      (put-text-property 0 3 'foo 'bar s) s)

so that `s' points to an object that can be mutated inside.

So I think that it would be a good idea to add a comment in the docs
(of all of these functions) that notes that when given a string, they
mutate it, and it works even if the input is property-less.  Or just
make it easier with showing a example as the above.

(Hopefully the above makes sense -- I wonder how many lispers who are
not native e-lispers have that assumption...)

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                    http://barzilay.org/                   Maze is Life!

bug#17507: Documentation for `add-text-property' and relatives

[Lars Ingebrigtsen]

Eli Barzilay <eli@barzilay.org> writes:

> What I was surprised to find out is that
>
>     (let ((s "foo")) (put-text-property 0 3 'foo 'bar s) s)
>
> works.  It's probably obvious for elisp hackers, but coming from
> most other lisps/schemes, I kind of assumed that the #("foo" ...)
> thing is made of the "foo" string in some vector-like container that
> holds the properties.  Given that unconscious assumption, I assumed
> that it wouldn't work, and that I'll need some uglier hack like
>
>     (let ((s (propertize "foo" 'foo nil)))
>       (put-text-property 0 3 'foo 'bar s) s)
>
> so that `s' points to an object that can be mutated inside.
>
> So I think that it would be a good idea to add a comment in the docs
> (of all of these functions) that notes that when given a string, they
> mutate it, and it works even if the input is property-less.  Or just
> make it easier with showing a example as the above.
>
> (Hopefully the above makes sense -- I wonder how many lispers who are
> not native e-lispers have that assumption...)

(I'm going through old bug reports that have unfortunately gotten no
responses yet.)

Hm...  I think the doc string is pretty clear here:

---
If the optional fifth argument OBJECT is a buffer (or nil, which means
the current buffer), START and END are buffer positions (integers or
markers).  If OBJECT is a string, START and END are 0-based indices into it.
---

I understand that if you come from a C background (or something like
that) your assumption may be that a string is just a bunch of bytes with
a null at the end, so you can't add any properties to it -- but that's
not the case in most languages, where a string is an object with lots of
different qualities in addition to the text itself.

So I don't think this is something that's desirable to explain in this
specific context, and I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#17507: Documentation for `add-text-property' and relatives

[Eli Barzilay]

Um...  Note that I referred *explicitly* to other lisps and schemes.
The assumption that I'm talking about would hold for people who are
used to CL too: a string is indeed more than just a pointer bytes with
a NUL at the end, yet there is no way to change a *property* of "foo"
and end up with #("foo" stuff...).

On Thu, Aug 15, 2019 at 2:57 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> Eli Barzilay <eli@barzilay.org> writes:
>
> > What I was surprised to find out is that
> >
> >     (let ((s "foo")) (put-text-property 0 3 'foo 'bar s) s)
> >
> > works.  It's probably obvious for elisp hackers, but coming from
> > most other lisps/schemes, I kind of assumed that the #("foo" ...)
> > thing is made of the "foo" string in some vector-like container that
> > holds the properties.  Given that unconscious assumption, I assumed
> > that it wouldn't work, and that I'll need some uglier hack like
> >
> >     (let ((s (propertize "foo" 'foo nil)))
> >       (put-text-property 0 3 'foo 'bar s) s)
> >
> > so that `s' points to an object that can be mutated inside.
> >
> > So I think that it would be a good idea to add a comment in the docs
> > (of all of these functions) that notes that when given a string, they
> > mutate it, and it works even if the input is property-less.  Or just
> > make it easier with showing a example as the above.
> >
> > (Hopefully the above makes sense -- I wonder how many lispers who are
> > not native e-lispers have that assumption...)
>
> (I'm going through old bug reports that have unfortunately gotten no
> responses yet.)
>
> Hm...  I think the doc string is pretty clear here:
>
> ---
> If the optional fifth argument OBJECT is a buffer (or nil, which means
> the current buffer), START and END are buffer positions (integers or
> markers).  If OBJECT is a string, START and END are 0-based indices into it.
> ---
>
> I understand that if you come from a C background (or something like
> that) your assumption may be that a string is just a bunch of bytes with
> a null at the end, so you can't add any properties to it -- but that's
> not the case in most languages, where a string is an object with lots of
> different qualities in addition to the text itself.
>
> So I don't think this is something that's desirable to explain in this
> specific context, and I'm closing this bug report.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no



-- 
                   ((x=>x(x))(x=>x(x)))                  Eli Barzilay:
                   http://barzilay.org/                  Maze is Life!

bug#17507: Documentation for `add-text-property' and relatives

[Eli Zaretskii]

> From: Eli Barzilay <eli@barzilay.org>
> Date: Thu, 15 Aug 2019 06:41:17 +0300
> Cc: 17507@debbugs.gnu.org
> 
> Um...  Note that I referred *explicitly* to other lisps and schemes.
> The assumption that I'm talking about would hold for people who are
> used to CL too: a string is indeed more than just a pointer bytes with
> a NUL at the end, yet there is no way to change a *property* of "foo"
> and end up with #("foo" stuff...).

Can you explain in more detail what did you think was missing from the
manual?  I've read the original report, and I admit I didn't
understand what was the essence of your difficulty.

Thanks.

bug#17507: Documentation for `add-text-property' and relatives

[Eli Barzilay]

In lisps you can mutate properties of an object, but you cannot change
the pointer itself.  I think that another way to clarify it would be
to say that "foo" is actually shorthand for #("foo") but the #() is
omitted for brevity.  But at this point this is subjective enough to
be ignorable...

On Thu, Aug 15, 2019 at 5:30 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Eli Barzilay <eli@barzilay.org>
> > Date: Thu, 15 Aug 2019 06:41:17 +0300
> > Cc: 17507@debbugs.gnu.org
> >
> > Um...  Note that I referred *explicitly* to other lisps and schemes.
> > The assumption that I'm talking about would hold for people who are
> > used to CL too: a string is indeed more than just a pointer bytes with
> > a NUL at the end, yet there is no way to change a *property* of "foo"
> > and end up with #("foo" stuff...).
>
> Can you explain in more detail what did you think was missing from the
> manual?  I've read the original report, and I admit I didn't
> understand what was the essence of your difficulty.
>
> Thanks.



-- 
                   ((x=>x(x))(x=>x(x)))                  Eli Barzilay:
                   http://barzilay.org/                  Maze is Life!

bug#17507: Documentation for `add-text-property' and relatives

[Eli Zaretskii]

> From: Eli Barzilay <eli@barzilay.org>
> Date: Fri, 16 Aug 2019 04:35:35 +0300
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 17507@debbugs.gnu.org
> 
> In lisps you can mutate properties of an object, but you cannot change
> the pointer itself.  I think that another way to clarify it would be
> to say that "foo" is actually shorthand for #("foo") but the #() is
> omitted for brevity.  But at this point this is subjective enough to
> be ignorable...

Sorry for prolonging this discussion, but I still don't think I
understand the difficulty.

Allow me to step back a notch and look at this from a higher-level
perspective.

In Emacs, text properties are stored separately from the text itself.
In particular, text properties of a string are not part of the string
text, they live in a separate structure, and a string only maintains a
reference to that structure.  When Emacs needs to display the string,
it consults that structure and renders the string characters
accordingly.  When the Lisp interpreter needs to print the string, it
prints the properties as well, to make them visible.

It sounds like your mental model of a string was that it included the
properties, and therefore you were surprised that add-text-property
etc. appeared to modify the string by side effect.  Is that what
surprised you?  If so, would saying that these functions modify the
properties of a string without mutating the string itself resolve the
difficulty?

bug#17507: Documentation for `add-text-property' and relatives

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> In Emacs, text properties are stored separately from the text itself.
> In particular, text properties of a string are not part of the string
> text, they live in a separate structure, and a string only maintains a
> reference to that structure.

Well, isn't that the same thing conceptually as the properties being in
the string object?  However things are arranged internally (with a
reference to a different struct or being in the in the same struct),
from the Emacs Lisp side of thing, the string object has properties.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#17507: Documentation for `add-text-property' and relatives

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Barzilay <eli@barzilay.org>,  17507@debbugs.gnu.org
> Date: Fri, 16 Aug 2019 13:04:34 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > In Emacs, text properties are stored separately from the text itself.
> > In particular, text properties of a string are not part of the string
> > text, they live in a separate structure, and a string only maintains a
> > reference to that structure.
> 
> Well, isn't that the same thing conceptually as the properties being in
> the string object?

It depends on your concepts.  I'm trying to determine which part of
the related concepts contributed to the confusion reported in the bug
report.

bug#17507: Documentation for `add-text-property' and relatives

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Well, isn't that the same thing conceptually as the properties being in
  > the string object?  However things are arranged internally (with a
  > reference to a different struct or being in the in the same struct),
  > from the Emacs Lisp side of thing, the string object has properties.

The string object has properties.  20 years ago, that was not merely
conceptually true: the tree of properties was stored in a slot in the
string.

I don't know whether the structure has changed since then, but the
concept has not changed.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#17507: Documentation for `add-text-property' and relatives

[Eli Barzilay]

To clarify what I said, it sounds like a string is a tuple of the
actual string *and* its properties, so it's always #("xyz" ...) where
the quotes denote the raw string contents, except that when there's no
properties, then you see just the "xyz" part -- but that's *not* a raw
string, it just doesn't show the (empty) properties.

Yet another way to show this is the fact that if you evaluate
#("xyz"), the result is *displayed* as "xyz".

Maybe adding some note to that effect (perhaps rephrased, to point at
the fact that all strings have properties, there are no raw strings)
would have resolved my original confusion.

On Fri, Aug 16, 2019 at 11:46 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Eli Barzilay <eli@barzilay.org>
> > Date: Fri, 16 Aug 2019 04:35:35 +0300
> > Cc: Lars Ingebrigtsen <larsi@gnus.org>, 17507@debbugs.gnu.org
> >
> > In lisps you can mutate properties of an object, but you cannot change
> > the pointer itself.  I think that another way to clarify it would be
> > to say that "foo" is actually shorthand for #("foo") but the #() is
> > omitted for brevity.  But at this point this is subjective enough to
> > be ignorable...
>
> Sorry for prolonging this discussion, but I still don't think I
> understand the difficulty.
>
> Allow me to step back a notch and look at this from a higher-level
> perspective.
>
> In Emacs, text properties are stored separately from the text itself.
> In particular, text properties of a string are not part of the string
> text, they live in a separate structure, and a string only maintains a
> reference to that structure.  When Emacs needs to display the string,
> it consults that structure and renders the string characters
> accordingly.  When the Lisp interpreter needs to print the string, it
> prints the properties as well, to make them visible.
>
> It sounds like your mental model of a string was that it included the
> properties, and therefore you were surprised that add-text-property
> etc. appeared to modify the string by side effect.  Is that what
> surprised you?  If so, would saying that these functions modify the
> properties of a string without mutating the string itself resolve the
> difficulty?



-- 
                   ((x=>x(x))(x=>x(x)))                  Eli Barzilay:
                   http://barzilay.org/                  Maze is Life!

bug#17507: Documentation for `add-text-property' and relatives

[Eli Zaretskii]

> From: Eli Barzilay <eli@barzilay.org>
> Date: Sat, 17 Aug 2019 02:49:54 +0300
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 17507@debbugs.gnu.org
> 
> To clarify what I said, it sounds like a string is a tuple of the
> actual string *and* its properties, so it's always #("xyz" ...) where
> the quotes denote the raw string contents, except that when there's no
> properties, then you see just the "xyz" part -- but that's *not* a raw
> string, it just doesn't show the (empty) properties.

That's not what the printed representation is trying to convey.  It is
simply trying to show you the properties, because that's useful in
many situations.

> Yet another way to show this is the fact that if you evaluate
> #("xyz"), the result is *displayed* as "xyz".
> 
> Maybe adding some note to that effect (perhaps rephrased, to point at
> the fact that all strings have properties, there are no raw strings)
> would have resolved my original confusion.

I really don't see how this is related to what you originally wrote,
sorry.  Also, "all strings have properties" would be misleading
because most strings have no properties.

Can we please return to the original issue, which was the
documentation of add-text-property?  I don't see how the printed
representation of a Lisp string is relevant to the documentation of
that function (and similar functions that modify text properties of
strings).  Maybe that's what I'm missing, in which case please help me
connect the dots.

Thanks.

bug#17507: Documentation for `add-text-property' and relatives

[Eli Barzilay]

On Sat, Aug 17, 2019 at 9:41 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Eli Barzilay <eli@barzilay.org>
> > Date: Sat, 17 Aug 2019 02:49:54 +0300
> > Cc: Lars Ingebrigtsen <larsi@gnus.org>, 17507@debbugs.gnu.org
> >
> > To clarify what I said, it sounds like a string is a tuple of the
> > actual string *and* its properties, so it's always #("xyz" ...) where
> > the quotes denote the raw string contents, except that when there's no
> > properties, then you see just the "xyz" part -- but that's *not* a raw
> > string, it just doesn't show the (empty) properties.
>
> That's not what the printed representation is trying to convey.  It is
> simply trying to show you the properties, because that's useful in
> many situations.
>
> > Yet another way to show this is the fact that if you evaluate
> > #("xyz"), the result is *displayed* as "xyz".
> >
> > Maybe adding some note to that effect (perhaps rephrased, to point at
> > the fact that all strings have properties, there are no raw strings)
> > would have resolved my original confusion.
>
> I really don't see how this is related to what you originally wrote,
> sorry.  Also, "all strings have properties" would be misleading
> because most strings have no properties.

"All strings have a property list pointer."


> Can we please return to the original issue, which was the
> documentation of add-text-property?  I don't see how the printed
> representation of a Lisp string is relevant to the documentation of
> that function (and similar functions that modify text properties of
> strings).  Maybe that's what I'm missing, in which case please help me
> connect the dots.

This is the issue: if "xyz" is a raw string in the sense that there is
no plist pointer, then adding one looks confusing.

But I'll just give up.


-- 
                   ((x=>x(x))(x=>x(x)))                  Eli Barzilay:
                   http://barzilay.org/                  Maze is Life!

bug#17507: Documentation for `add-text-property' and relatives

[Eli Zaretskii]

> From: Eli Barzilay <eli@barzilay.org>
> Date: Sat, 17 Aug 2019 15:40:09 +0300
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 17507@debbugs.gnu.org
> 
> But I'll just give up.

Then so will I.

bug#17507: Documentation for `add-text-property' and relatives

[Lars Ingebrigtsen]

Eli Barzilay <eli@barzilay.org> writes:

> Um...  Note that I referred *explicitly* to other lisps and schemes.
> The assumption that I'm talking about would hold for people who are
> used to CL too: a string is indeed more than just a pointer bytes with
> a NUL at the end, yet there is no way to change a *property* of "foo"
> and end up with #("foo" stuff...).

If there were a way to put properties on string objects in a Common Lisp
implementation, I would expect that it would work just as it does in
Emacs: Not wrapping the string in some other object, but just extend the
object itself.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no