buffer substring of only visible text

buffer substring of only visible text

[Samuel Wales]

i want to copy visible text from a region in a buffer, into
a string.

if any invisible regions are in the region, i want them to
not be in the string.

that is, i want the visible intervals in the copied region
to be merely appended.

when i INSERT the resulting string into the new buffer, i
want any text properties that i specify to still exist.


htmlize.el has a function, buffer-substring-no-invisible,
which does not include any text properties at all.

it works, but i'd like to include at least one text
property, which i want to specify.

that is the problem that i want to solve.  i am looking for a
know a cookbook formula to do this really straightforwardly.

Re: buffer substring of only visible text

[Emanuel Berg]

Samuel Wales wrote:

> i want to copy visible text from a region in a buffer, into
> a string.

`buffer-substring'

-- 
underground experts united
https://dataswamp.org/~incal

Re: buffer substring of only visible text

[Felix Dietrich]

Hello Samuel,

Samuel Wales <samologist@gmail.com> writes:

> i want to copy visible text from a region in a buffer, into a string.
> if any invisible regions are in the region, i want them to not be in
> the string.  that is, i want the visible intervals in the copied
> region to be merely appended.  when i INSERT the resulting string into
> the new buffer, i want any text properties that i specify to still
> exist.  htmlize.el has a function, buffer-substring-no-invisible,
> which does not include any text properties at all.  it works, but i'd
> like to include at least one text property, which i want to specify.
>
> i am looking for a know a cookbook formula to do this really
> straightforwardly.

I do not know any cookbook solution for this problem, but you have
already found a (fairly short and straightforward) function that almost
does what you described; with a few minor modifications you should be
able to get what you want (almost untested):

#+begin_src emacs-lisp
  (defun my/buffer-substring-with-properties (start end props)
    ;; Copy the specified property from the specified region of the
    ;; buffer to the target string.  We cannot rely on Emacs to copy the
    ;; property because we want to handle properties coming from both
    ;; text properties and overlays.
    (let ((text (buffer-substring-no-properties start end)))
      (dolist (p props)
        (htmlize-copy-prop p start end text))
      text))
  
  
  (defun my/buffer-substring-no-invisible (beg end &optional props)
    ;; Like buffer-substring-no-properties, but don't copy invisible
    ;; parts of the region.  Where buffer-substring-no-properties
    ;; mandates an ellipsis to be shown, htmlize-ellipsis is inserted.
    (let ((pos beg)
          visible-list invisible show last-show next-change)
      ;; Iterate over the changes in the `invisible' property and filter
      ;; out the portions where it's non-nil, i.e. where the text is
      ;; invisible.
      (while (< pos end)
        (setq invisible (get-char-property pos 'invisible)
              next-change (htmlize-next-change pos 'invisible end)
              show (htmlize-decode-invisibility-spec invisible))
        (cond ((eq show t)
               (push (my/buffer-substring-with-properties pos next-change
                                                          props)
                     visible-list))
              ((and (eq show 'ellipsis)
                    (not (eq last-show 'ellipsis))
                    ;; Conflate successive ellipses.
                    (push htmlize-ellipsis visible-list))))
        (setq pos next-change last-show show))
      (htmlize-concat (nreverse visible-list))))
#+end_src

The function ‘my/buffer-substring-no-invisible’ only adds an additional
parameter PROPS to ‘htmlize-buffer-substring-no-invisible’ and replaces
the call to ‘htmlize-get-text-with-display’ with a call to
‘my/buffer-substring-with-properties’ that takes the PROPS parameter to
copy not only the two fixed properties ‘display’ and ‘htmlize-link’, as
does ‘htmlize-get-text-with-display’, but allow specifying the desired
properties to copy when calling the function.

Text that has the ‘invisible’ property may be replaced with an ellipsis
(depending on ‘buffer-invisibility-spec’).  If you preferred there
rather be no ellipsis, adjust the cond-form or bind ‘htmlize-ellipsis’
to the empty string "".

As the comment at the top of ‘htmlize-copy-prop’ suggests, Emacs may
provide more efficient means to copy text properties if you do not need
the overlay properties.  I cannot help you with the details of that,
though, but maybe the sections on text and overlay properties in the
Emacs Lisp handbook can provide you with the necessary clues [1][2].

Footnotes:
[1]  (info "(elisp) Text Properties")
     <https://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Properties.html>

[2]  (info "(elisp) Overlay Properties")
     <https://www.gnu.org/software/emacs/manual/html_node/elisp/Overlay-Properties.html>

-- 
Felix Dietrich

Re: buffer substring of only visible text

[Samuel Wales]

thank you for your response.  very interesting and thorough.

i was hoping there would be just a built in function; thank you for
suggesting there is not.

it is surely a useful function regardless of what it is used for so i
will hang on to it.  maybe can use.

this problem with or without manuals has made my head swim ever since
text properties arrived.  i'm sure the manuals are good but my broken
brain is not really getting text properties.  results are sometimes
off, i don't know the special ones, etc.

for me it would be better if thre were a tool more powerful than c-u
c-x = to inspect them.  for example, if there were a tool to show all
text properties and their intervals in a region or buffer or string.
and also links to where any special text properties are documented.
but that's a wish list i guess.  it would help me understand emacs
anyway.

i forgot to mention more explicitly that i was hoping to not depend on
htmlize as it is not in core, but that's a side point, as i will use
htmlize if necessary, and also i think now, maybe, it /might/ be
better to go line by line?  but i am not sure.

more of the x for which this is possibly an xy problem: as noted, i am
trying to copy visible text from buffer a to b.  and i want buffer b
to have text properties that include markers to jump to a for specific
lines.  kinda like magit jumps to files.

and so, those text properties in buffer b would include, in b, the
markers in a.  or... something.   you can probably tell i am a little
lost here.  but with such a funciton like yours, i would put the
markers in a, put the text properties redundantly in a, then copy over
using the function.  the redundant text properties seems wrong
however.  it seems efficient.

so idk.  one alternative seems to be to go line by line through a,
insert markers in a, then create a string for each line with a text
property containing the marker, then insert that string into b, then
continue with the next matching visible line in a.

which seems inefficient.  idk if this is a common problem: copy parts
of buffer a to buffer b such that you can RET to get to a.


On 9/21/22, Felix Dietrich <felix.dietrich@sperrhaken.name> wrote:
> Hello Samuel,
>
> Samuel Wales <samologist@gmail.com> writes:
>
>> i want to copy visible text from a region in a buffer, into a string.
>> if any invisible regions are in the region, i want them to not be in
>> the string.  that is, i want the visible intervals in the copied
>> region to be merely appended.  when i INSERT the resulting string into
>> the new buffer, i want any text properties that i specify to still
>> exist.  htmlize.el has a function, buffer-substring-no-invisible,
>> which does not include any text properties at all.  it works, but i'd
>> like to include at least one text property, which i want to specify.
>>
>> i am looking for a know a cookbook formula to do this really
>> straightforwardly.
>
> I do not know any cookbook solution for this problem, but you have
> already found a (fairly short and straightforward) function that almost
> does what you described; with a few minor modifications you should be
> able to get what you want (almost untested):
>
> #+begin_src emacs-lisp
>   (defun my/buffer-substring-with-properties (start end props)
>     ;; Copy the specified property from the specified region of the
>     ;; buffer to the target string.  We cannot rely on Emacs to copy the
>     ;; property because we want to handle properties coming from both
>     ;; text properties and overlays.
>     (let ((text (buffer-substring-no-properties start end)))
>       (dolist (p props)
>         (htmlize-copy-prop p start end text))
>       text))
>
>
>   (defun my/buffer-substring-no-invisible (beg end &optional props)
>     ;; Like buffer-substring-no-properties, but don't copy invisible
>     ;; parts of the region.  Where buffer-substring-no-properties
>     ;; mandates an ellipsis to be shown, htmlize-ellipsis is inserted.
>     (let ((pos beg)
>           visible-list invisible show last-show next-change)
>       ;; Iterate over the changes in the `invisible' property and filter
>       ;; out the portions where it's non-nil, i.e. where the text is
>       ;; invisible.
>       (while (< pos end)
>         (setq invisible (get-char-property pos 'invisible)
>               next-change (htmlize-next-change pos 'invisible end)
>               show (htmlize-decode-invisibility-spec invisible))
>         (cond ((eq show t)
>                (push (my/buffer-substring-with-properties pos next-change
>                                                           props)
>                      visible-list))
>               ((and (eq show 'ellipsis)
>                     (not (eq last-show 'ellipsis))
>                     ;; Conflate successive ellipses.
>                     (push htmlize-ellipsis visible-list))))
>         (setq pos next-change last-show show))
>       (htmlize-concat (nreverse visible-list))))
> #+end_src
>
> The function ‘my/buffer-substring-no-invisible’ only adds an additional
> parameter PROPS to ‘htmlize-buffer-substring-no-invisible’ and replaces
> the call to ‘htmlize-get-text-with-display’ with a call to
> ‘my/buffer-substring-with-properties’ that takes the PROPS parameter to
> copy not only the two fixed properties ‘display’ and ‘htmlize-link’, as
> does ‘htmlize-get-text-with-display’, but allow specifying the desired
> properties to copy when calling the function.
>
> Text that has the ‘invisible’ property may be replaced with an ellipsis
> (depending on ‘buffer-invisibility-spec’).  If you preferred there
> rather be no ellipsis, adjust the cond-form or bind ‘htmlize-ellipsis’
> to the empty string "".
>
> As the comment at the top of ‘htmlize-copy-prop’ suggests, Emacs may
> provide more efficient means to copy text properties if you do not need
> the overlay properties.  I cannot help you with the details of that,
> though, but maybe the sections on text and overlay properties in the
> Emacs Lisp handbook can provide you with the necessary clues [1][2].
>
> Footnotes:
> [1]  (info "(elisp) Text Properties")
>
> <https://www.gnu.org/software/emacs/manual/html_node/elisp/Text-Properties.html>
>
> [2]  (info "(elisp) Overlay Properties")
>
> <https://www.gnu.org/software/emacs/manual/html_node/elisp/Overlay-Properties.html>
>
> --
> Felix Dietrich
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: buffer substring of only visible text

[Felix Dietrich]

[-- Attachment #1: Type: text/plain, Size: 2329 bytes --]

Hello Samuel,

Samuel Wales <samologist@gmail.com> writes:

> thank you for your response.  very interesting and thorough.
>
> this problem with or without manuals has made my head swim ever since
> text properties arrived.  i'm sure the manuals are good but my broken
> brain is not really getting text properties.  results are sometimes
> off, i don't know the special ones, etc.

If you have a concrete problem, some small snippet you were
experimenting with and that does not do what you were expecting then,
maybe, someone on the list could help you with understanding it.  Just
play around with some code, compare the actual results with your
expectations, and try to figure out your misapprehensions either
yourself or by asking others.  This way, after some time and
frustration, things will hopefully become clearer and you develop an
intuition for Emacsʼ idiosyncrasies.

> i forgot to mention more explicitly that i was hoping to not depend on
> htmlize as it is not in core, but that's a side point, as i will use
> htmlize if necessary, and also i think now, maybe, it /might/ be
> better to go line by line?  but i am not sure.

The little snippet only uses a few utility functions that you could
easily copy and paste from htmlize and give your own prefix.

> for me it would be better if thre were a tool more powerful than c-u
> c-x = to inspect them.  for example, if there were a tool to show all
> text properties and their intervals in a region or buffer or string.
> and also links to where any special text properties are documented.
> but that's a wish list i guess.  it would help me understand emacs
> anyway.

I dabbled around a little with your idea and came up with two commands.
They do not do exactly what you want, have issues (only some of them I
am vaguely aware of) and might be at least a little broken
(‘hl-line-mode’ for example leaves partially highlighted lines around),
but you might still find them interesting as toys to play around with.
In any case, I had fun and learned a little about Emacs and its Lisp.
One shows in another buffer the active properties at point; the buffer
is updated when point is moved, the other shows active properties within
a region.  Properties in overlapping overlays with lower priority are
not shown.


[-- Attachment #2: Commands to display text properties in a buffer --]
[-- Type: application/emacs-lisp, Size: 10727 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1672 bytes --]



> more of the x for which this is possibly an xy problem: as noted, i am
> trying to copy visible text from buffer a to b.  and i want buffer b
> to have text properties that include markers to jump to a for specific
> lines.  kinda like magit jumps to files.

I donʼt know how magit does it.  Maybe it uses the line information
provided in the diff hunks?  You could read itʼs source.

> and so, those text properties in buffer b would include, in b, the
> markers in a.  or... something.   you can probably tell i am a little
> lost here.  but with such a funciton like yours, i would put the
> markers in a, put the text properties redundantly in a, then copy over
> using the function.  the redundant text properties seems wrong
> however.  it seems efficient.

What is in those text properties that creates the need to copy and
duplicate them?

> so idk.  one alternative seems to be to go line by line through a,
> insert markers in a, then create a string for each line with a text
> property containing the marker, then insert that string into b, then
> continue with the next matching visible line in a.
>
> which seems inefficient.  idk if this is a common problem: copy parts
> of buffer a to buffer b such that you can RET to get to a.

I donʼt know either.  If no one else chimes in and your search engine
does not provide you with an answer, just start writing some code and
playing around with your ideas to develop a better understanding of the
problem and possible solutions.  Donʼt start worrying to much about
efficiency at the beginning unless your Emacs or your Computer grind to
a halt.

-- 
Felix Dietrich

Re: buffer substring of only visible text

[Michael Heerdegen]

Samuel Wales <samologist@gmail.com> writes:

> i want to copy visible text from a region in a buffer, into
> a string.
>
> if any invisible regions are in the region, i want them to
> not be in the string.

Not a complicated task - and it seems that `backtrace--filter-visible'
is implementing exactly that:

#+begin_src emacs-lisp
(defun backtrace--filter-visible (beg end &optional _delete)
  "Return the visible text between BEG and END."
  (let ((result ""))
    (while (< beg end)
      (let ((next (next-single-char-property-change beg 'invisible)))
        (unless (get-char-property beg 'invisible)
          (setq result (concat result (buffer-substring beg (min end next)))))
        (setq beg next)))
    result))
#+end_src

If you don't want to rely on an internal function you might just copy
these few lines.

Michael.