Implementing image support for kitty terminal

Implementing image support for kitty terminal

[Jose Antonio Ortega Ruiz]

Hi,

The kitty terminal emulator (which runs under X11 and wayland) offers a
simple protocol for displaying images, fully described at
<https://sw.kovidgoyal.net/kitty/graphics-protocol/>.  

In a nutshell, it accepts an escape sequence that make it enter "graphic
mode", followed by either encoded image data or a path to an image file
or a shared memory location to display.  Among several other niceties,
the protocol allows drawing to rectangles specified in cell units. As a
simple example, the sequence:

   <ESC>_Gf=100,t=f,c=50,r=100;<encoded /path/to/file.png><ESC>\

would make kitty draw the image in file.png rescaling it to 50 columns
and 100 rows.  By default, the current cursor position is used, but it's
also possible to specify pixel offsets and sizes.

At first sight, it looks as if adding support for this protocol to
emacs's tty terminal (when kitty, or the capability (it seems other
terminals support the same protocol) is detected) shouldn't be too
complex, and with that, perhaps, provide direct support for the
elisp-level image- API for these terminals (so that, for instance,
doc-view or pdf-tools or displaying images in eww buffers would work out
of the box).  Am i wrong?

On a personal note, if that were possible it would put emacs on a kitty
terminal on the same league as the full graphical version for my needs,
with the added benefit of dramatically reduced RAM footprint, faster
display and, last but not least, a truly great alternative to pgtk in
wayland.  So, if the implementation is feasible, i'd be willing to help
if needed.

Thanks,
jao
-- 
千里之行始於足下
The journey of a thousand miles begins with a single step.
  — 老子 (Lao Tsu) in Chapter 64 of 道德经 (Tao Te Ching)

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Wed 07 Sep 2022 at 16:50, Jose Antonio Ortega Ruiz <jao@gnu.org> wrote:
> At first sight, it looks as if adding support for this protocol to
> emacs's tty terminal (when kitty, or the capability (it seems other
> terminals support the same protocol) is detected) shouldn't be too
> complex, and with that, perhaps, provide direct support for the
> elisp-level image- API for these terminals (so that, for instance,
> doc-view or pdf-tools or displaying images in eww buffers would work out
> of the box).  Am i wrong?

Any work in this direction would be great.

It would help me move emacs-framebuffer
<https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html>
forward.  I use w3mimgdisplay there which works on the Linux console.

w3mimgdisplay uses quite similar protocol and also caches the images for
faster redrawing.

There was discussion about some aspects, for example:
id:87zh9hrxfj.fsf@logand.com
id:87czzu7y5r.fsf@logand.com
id:87mtyw9c7f.fsf@logand.com

Unfortunatelly, I think it would require quite a lot of changes to the C
code.

(image-size (create-image "/tmp/a.jpg"))
=> (error "Window system frame should be used")

> On a personal note, if that were possible it would put emacs on a kitty
> terminal on the same league as the full graphical version for my needs,
> with the added benefit of dramatically reduced RAM footprint, faster
> display and, last but not least, a truly great alternative to pgtk in
> wayland.  So, if the implementation is feasible, i'd be willing to help
> if needed.

That was also my motivation for emacs-framebuffer.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Jose Antonio Ortega Ruiz <jao@gnu.org>
> Date: Wed, 07 Sep 2022 16:50:08 +0100
> 
> The kitty terminal emulator (which runs under X11 and wayland) offers a
> simple protocol for displaying images, fully described at
> <https://sw.kovidgoyal.net/kitty/graphics-protocol/>.  
> 
> In a nutshell, it accepts an escape sequence that make it enter "graphic
> mode", followed by either encoded image data or a path to an image file
> or a shared memory location to display.  Among several other niceties,
> the protocol allows drawing to rectangles specified in cell units. As a
> simple example, the sequence:
> 
>    <ESC>_Gf=100,t=f,c=50,r=100;<encoded /path/to/file.png><ESC>\
> 
> would make kitty draw the image in file.png rescaling it to 50 columns
> and 100 rows.  By default, the current cursor position is used, but it's
> also possible to specify pixel offsets and sizes.
> 
> At first sight, it looks as if adding support for this protocol to
> emacs's tty terminal (when kitty, or the capability (it seems other
> terminals support the same protocol) is detected) shouldn't be too
> complex, and with that, perhaps, provide direct support for the
> elisp-level image- API for these terminals (so that, for instance,
> doc-view or pdf-tools or displaying images in eww buffers would work out
> of the box).  Am i wrong?

It's hard to tell, because you haven't described your ideas of
implementing that.  In particular, the current model of image display
in the Emacs display engine is that an image is basically considered a
single very large character, from the screen layout POV.  I guess
that's not what you have in mind for the above, so IMO it's important
to come up with an alternative model that would somehow fit with the
current display code with only minor changes, if we want this not to
be too complex.

> On a personal note, if that were possible it would put emacs on a kitty
> terminal on the same league as the full graphical version for my needs,
> with the added benefit of dramatically reduced RAM footprint, faster
> display and, last but not least, a truly great alternative to pgtk in
> wayland.  So, if the implementation is feasible, i'd be willing to help
> if needed.

I don't think anyone will disagree that extending the capabilities of
TTY frames in this direction will be a very good and useful feature.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Wed, Sep 07 2022, Eli Zaretskii wrote:

>> From: Jose Antonio Ortega Ruiz <jao@gnu.org>
>> Date: Wed, 07 Sep 2022 16:50:08 +0100
>> 
>> The kitty terminal emulator (which runs under X11 and wayland) offers a
>> simple protocol for displaying images, fully described at
>> <https://sw.kovidgoyal.net/kitty/graphics-protocol/>.  
>> 
>> In a nutshell, it accepts an escape sequence that make it enter "graphic
>> mode", followed by either encoded image data or a path to an image file
>> or a shared memory location to display.  Among several other niceties,
>> the protocol allows drawing to rectangles specified in cell units. As a
>> simple example, the sequence:
>> 
>>    <ESC>_Gf=100,t=f,c=50,r=100;<encoded /path/to/file.png><ESC>\
>> 
>> would make kitty draw the image in file.png rescaling it to 50 columns
>> and 100 rows.  By default, the current cursor position is used, but it's
>> also possible to specify pixel offsets and sizes.
>> 
>> At first sight, it looks as if adding support for this protocol to
>> emacs's tty terminal (when kitty, or the capability (it seems other
>> terminals support the same protocol) is detected) shouldn't be too
>> complex, and with that, perhaps, provide direct support for the
>> elisp-level image- API for these terminals (so that, for instance,
>> doc-view or pdf-tools or displaying images in eww buffers would work out
>> of the box).  Am i wrong?
>
> It's hard to tell, because you haven't described your ideas of
> implementing that.  In particular, the current model of image display
> in the Emacs display engine is that an image is basically considered a
> single very large character, from the screen layout POV.

Would it be possible to keep that model and only alter it when the
display actually draws the character in the underlying terminal?  

For an image-enhanced tty that'd mean that when we are going to write
this "single very large character" (whose data can be either an array of
bytes or simply a path in the filesystem, stored in an associated text
property perhaps) what we do is write a ESC character sequence starting
with _G, as the one shown above, and specifying not only the data to
draw, but also its size in rows and columns.  Then, according to the
kitty API promises, the image will be rendered in that rectangle for us.

I am assuming here that the display engine (for a tty) knows somehow
that it's printing a very lage "character" that is going to spawn
several rows and columns, and also that we can always tell how wide and
tall the window around point is when we display this "character".  We
could then decide to tell kitty to either scale the image if it doesn't
fit or display only part of it.  That's done by passing x,y,w,h in the
escape sequence instead or r,c... something like

      <ESC>_Gf=100,t=f,x=0,y=0,w=100,h=20;<encoded /path/to/file.png><ESC>\

As you can see, the API offered is quite powerful and would do a lot of
work for us.  It's even able to cache images and reuse them: according
to their documentation, every transmitted image can be displayed an
arbitrary number of times on the screen, in different locations, using
different parts of the source image, as needed.

As the above ideas most probably demonstrate, i don't know anything
concrete about how emacs works at this level, so apologies in advance if
i'm not making sense.

Thanks,
jao
-- 
They are ill discoverers that think there is no land, when they can
see nothing but sea. -Francis Bacon, philosopher (1561-1626)

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Wed 07 Sep 2022 at 21:11, Eli Zaretskii <eliz@gnu.org> wrote:
> In particular, the current model of image display
> in the Emacs display engine is that an image is basically considered a
> single very large character, from the screen layout POV.

Are you talking about GUI emacs?
Is that not different in tty emacs?
I thought that the tty emacs has no concept of image (so far?).

> I guess
> that's not what you have in mind for the above, so IMO it's important
> to come up with an alternative model that would somehow fit with the
> current display code with only minor changes, if we want this not to
> be too complex.

For example, the tty based w3m web browser optionally supports images.
It reserves space w*h chars on the tty and then uses external program
to draw the image in the right place.

I guess tty emacs would also need to reserve a w*h chars rectangle on
the tty and then send the right escape code to fill that gap with the
image.

First, tty emacs would need to understand images instead of throwing
error.  At least their dimensions.  Maybe using some library or even in
elisp, for example like this
https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html#l286
I have not figured out, how to actually plug it into upstream emacs, as
this seems to require a lot of changes to C code.

Second, tty emacs would need to somehow reserve rectangle for the image.
This might be lots of work I guess.

Third, there would have to be a piece of code which actually draws the
image.  In case of kitty terminal, the image path and coordinates would
be sent to the tty using the kitty terminal protocol.  In case of Linux
console, the image path and coordinates would be sent to an external
program like w3mimgdisplay.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Jose A Ortega Ruiz <jao@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 07 Sep 2022 19:49:34 +0100
> 
> > It's hard to tell, because you haven't described your ideas of
> > implementing that.  In particular, the current model of image display
> > in the Emacs display engine is that an image is basically considered a
> > single very large character, from the screen layout POV.
> 
> Would it be possible to keep that model and only alter it when the
> display actually draws the character in the underlying terminal?  
> 
> For an image-enhanced tty that'd mean that when we are going to write
> this "single very large character" (whose data can be either an array of
> bytes or simply a path in the filesystem, stored in an associated text
> property perhaps) what we do is write a ESC character sequence starting
> with _G, as the one shown above, and specifying not only the data to
> draw, but also its size in rows and columns.  Then, according to the
> kitty API promises, the image will be rendered in that rectangle for us.

This is a different level: the level of actually delivering the stuff
to the glass.  I think it's much easier than what was bothering me.  I
was talking about the level of layout: the one that knows how much
space each "display element" (character, image, stretch of white
space, etc.) takes on display.  On TTY frames, there's a built-in
assumption that every display element takes just one pixel in the
vertical direction, and each screen line takes one pixel on display.
That will have to change for image support, I presume.

> I am assuming here that the display engine (for a tty) knows somehow
> that it's printing a very lage "character" that is going to spawn
> several rows and columns, and also that we can always tell how wide and
> tall the window around point is when we display this "character".

That's exactly the problem: it currently assumes that no display
element takes more than one row, and that each row is exactly one
pixel (= one character height) tall.

> We
> could then decide to tell kitty to either scale the image if it doesn't
> fit or display only part of it.  That's done by passing x,y,w,h in the
> escape sequence instead or r,c... something like
> 
>       <ESC>_Gf=100,t=f,x=0,y=0,w=100,h=20;<encoded /path/to/file.png><ESC>\

I think that's the easy part.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 07 Sep 2022 20:56:09 +0200
> 
> On Wed 07 Sep 2022 at 21:11, Eli Zaretskii <eliz@gnu.org> wrote:
> > In particular, the current model of image display
> > in the Emacs display engine is that an image is basically considered a
> > single very large character, from the screen layout POV.
> 
> Are you talking about GUI emacs?

Yes.  That's the only kind of frame where we currently support
images.

> Is that not different in tty emacs?

Not in principle, no.  The only difference -- and that is important
for this discussion -- is that on TTY frames we assume that all the
screen lines have the same height.  Which contradicts the basic
requirement for displaying images.

> I thought that the tty emacs has no concept of image (so far?).

It indeed doesn't support image display, but the layout level, which
is what I'm talking about, doesn't care _how_ to display stuff, it
only cares about the dimensions on the screen.

> For example, the tty based w3m web browser optionally supports images.
> It reserves space w*h chars on the tty and then uses external program
> to draw the image in the right place.

I have no doubt that this is possible.  We are not talking about how
to display images on text-mode screens _in_principle_, we are talking
about how to add that to the existing Emacs TTY display with minimum
cost and complexity.

> I guess tty emacs would also need to reserve a w*h chars rectangle on
> the tty and then send the right escape code to fill that gap with the
> image.

The first part is the main problem to solve.  Currently, TTY display
cannot do that, it's outside its basic design, which assumes that each
screen line has the same height, no display element can be split
between screen lines (the latter design assumption is common to the
GUI display).

> First, tty emacs would need to understand images instead of throwing
> error.  At least their dimensions.  Maybe using some library or even in
> elisp, for example like this
> https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html#l286
> I have not figured out, how to actually plug it into upstream emacs, as
> this seems to require a lot of changes to C code.
> 
> Second, tty emacs would need to somehow reserve rectangle for the image.
> This might be lots of work I guess.
> 
> Third, there would have to be a piece of code which actually draws the
> image.  In case of kitty terminal, the image path and coordinates would
> be sent to the tty using the kitty terminal protocol.  In case of Linux
> console, the image path and coordinates would be sent to an external
> program like w3mimgdisplay.

The second part is the problem I'm talking about.  The other two are
much simpler to add, I think.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Wed, Sep 07 2022, Tomas Hlavaty wrote:

> On Wed 07 Sep 2022 at 21:11, Eli Zaretskii <eliz@gnu.org> wrote:
>> In particular, the current model of image display
>> in the Emacs display engine is that an image is basically considered a
>> single very large character, from the screen layout POV.
>
> Are you talking about GUI emacs?
> Is that not different in tty emacs?
> I thought that the tty emacs has no concept of image (so far?).
>
>> I guess
>> that's not what you have in mind for the above, so IMO it's important
>> to come up with an alternative model that would somehow fit with the
>> current display code with only minor changes, if we want this not to
>> be too complex.
>
> For example, the tty based w3m web browser optionally supports images.
> It reserves space w*h chars on the tty and then uses external program
> to draw the image in the right place.
>
> I guess tty emacs would also need to reserve a w*h chars rectangle on
> the tty and then send the right escape code to fill that gap with the
> image.
>
> First, tty emacs would need to understand images instead of throwing
> error.  At least their dimensions.  Maybe using some library or even in
> elisp, for example like this
> https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html#l286
> I have not figured out, how to actually plug it into upstream emacs, as
> this seems to require a lot of changes to C code.

fwiw, in the kitty case, image libraries are going to be almost for sure
available, it being an x11/wayland application.  so an emacs compiled
with image support (but not necessarily linking against any x libs)
doesn't seem like a strong requirement.  in that scenario, determining
the size of an image wouldn't needed a lot of C code, would it? and, if
those libraries are not available, a call to "file foo.png" is (at least
in my system) telling me the size of foo.png... kind of hacky, of
course, but as a first step maybe good enough.

> Second, tty emacs would need to somehow reserve rectangle for the image.
> This might be lots of work I guess.

i was hoping this wouldn't be the case... at least when running inside
terminal emulators, emacs nox seems pretty dexterous lying out windows
and filly them with content, and i reckoned it must have a pretty good
idea of their sizes.  but maybe i'm too optimistic :)

> Third, there would have to be a piece of code which actually draws the
> image.  In case of kitty terminal, the image path and coordinates would
> be sent to the tty using the kitty terminal protocol.

(as an aside, this would also work for other terminals like konsole or
wayst, which support the protocol).

cheers,
jao
-- 
If we all think alike then nobody’s thinking at all.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Wed, Sep 07 2022, Eli Zaretskii wrote:

[...]

> This is a different level: the level of actually delivering the stuff
> to the glass.  I think it's much easier than what was bothering me.  I
> was talking about the level of layout: the one that knows how much
> space each "display element" (character, image, stretch of white
> space, etc.) takes on display.  On TTY frames, there's a built-in
> assumption that every display element takes just one pixel in the
> vertical direction, and each screen line takes one pixel on display.
> That will have to change for image support, I presume.

in x terminal emulators, by "pixel" here you mean one terminal "cell"
(the extend of a displayed ascii char), right?

>> I am assuming here that the display engine (for a tty) knows somehow
>> that it's printing a very lage "character" that is going to spawn
>> several rows and columns, and also that we can always tell how wide and
>> tall the window around point is when we display this "character".
>
> That's exactly the problem: it currently assumes that no display
> element takes more than one row, and that each row is exactly one
> pixel (= one character height) tall.

aha, i see.  so what's needed, for instance, is a way of managing blocks
of such display elements, that are always rendered all at once... or are
there better ways of accomplishing the same?

jao
-- 
Be regular and orderly in your life so that you may be violent and original
in your work.
 -Gustave Flaubert, novelist (1821-1880)

Re: Implementing image support for kitty terminal

[Stefan Monnier]

> That's exactly the problem: it currently assumes that no display
> element takes more than one row, and that each row is exactly one
> pixel (= one character height) tall.

So the change will/would need to make the glyph matrices (and their
generation) more like those used in the GUI code.


        Stefan

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Jose A Ortega Ruiz <jao@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 07 Sep 2022 16:11:01 -0400
> 
> > That's exactly the problem: it currently assumes that no display
> > element takes more than one row, and that each row is exactly one
> > pixel (= one character height) tall.
> 
> So the change will/would need to make the glyph matrices (and their
> generation) more like those used in the GUI code.

That's the "hard way".  It will require changes in many places that
assume the current TTY geometry of the glyph rows.  The problem, I
think, is not with generating the matrices, it's with using them, and
with adjusting features like scrolling, that currently assume all
screen lines have the same height, to this change.  I hoped we could
avoid that, but maybe not.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Jose A Ortega Ruiz <jao@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Wed, 07 Sep 2022 21:09:40 +0100
> 
> > This is a different level: the level of actually delivering the stuff
> > to the glass.  I think it's much easier than what was bothering me.  I
> > was talking about the level of layout: the one that knows how much
> > space each "display element" (character, image, stretch of white
> > space, etc.) takes on display.  On TTY frames, there's a built-in
> > assumption that every display element takes just one pixel in the
> > vertical direction, and each screen line takes one pixel on display.
> > That will have to change for image support, I presume.
> 
> in x terminal emulators, by "pixel" here you mean one terminal "cell"
> (the extend of a displayed ascii char), right?

Yes.

> >> I am assuming here that the display engine (for a tty) knows somehow
> >> that it's printing a very lage "character" that is going to spawn
> >> several rows and columns, and also that we can always tell how wide and
> >> tall the window around point is when we display this "character".
> >
> > That's exactly the problem: it currently assumes that no display
> > element takes more than one row, and that each row is exactly one
> > pixel (= one character height) tall.
> 
> aha, i see.  so what's needed, for instance, is a way of managing blocks
> of such display elements, that are always rendered all at once...

Something like that would be a significant change in the current
design, which handles each screen line separately.

> or are there better ways of accomplishing the same?

That's exactly the question: how to do better?

Re: Implementing image support for kitty terminal

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]

Jose Antonio Ortega Ruiz <jao@gnu.org> writes:

> Hi,
>
> The kitty terminal emulator (which runs under X11 and wayland) offers a
> simple protocol for displaying images, fully described at
> <https://sw.kovidgoyal.net/kitty/graphics-protocol/>.  

Many text terminals also happen to show images.  But I think Kitty is a
video terminal that also happen to show texts.

>
> In a nutshell, it accepts an escape sequence that make it enter "graphic
> mode", followed by either encoded image data or a path to an image file
> or a shared memory location to display.  Among several other niceties,
> the protocol allows drawing to rectangles specified in cell units. As a
> simple example, the sequence:
>
>    <ESC>_Gf=100,t=f,c=50,r=100;<encoded /path/to/file.png><ESC>\
>
> would make kitty draw the image in file.png rescaling it to 50 columns
> and 100 rows.  By default, the current cursor position is used, but it's
> also possible to specify pixel offsets and sizes.
>
> At first sight, it looks as if adding support for this protocol to
> emacs's tty terminal (when kitty, or the capability (it seems other
> terminals support the same protocol) is detected) shouldn't be too
> complex, and with that, perhaps, provide direct support for the
> elisp-level image- API for these terminals (so that, for instance,
> doc-view or pdf-tools or displaying images in eww buffers would work out
> of the box).  Am i wrong?

IMHO, showing images by converting to text would be a better choice.  It
would also allow to show image on Linux console.  Because the idea you
propose would probably make the Emacs C source more complex.

>
> On a personal note, if that were possible it would put emacs on a kitty
> terminal on the same league as the full graphical version for my needs,
> with the added benefit of dramatically reduced RAM footprint, faster
> display and, last but not least, a truly great alternative to pgtk in
> wayland.  So, if the implementation is feasible, i'd be willing to help
> if needed.

You can use Xwayland to run X applications on Wayland.  But Emacs has a
pgtk port, so you shouldn't need that.

Is there any X server for sixel terminals (like Kitty)?  If there is any
then you can that to run graphical Emacs in terminal.

And about the RAM footprint, every new feature usually increases RAM
footprint; so implementing this would also probably increase Emacs's
memory footprint.

>
> Thanks,
> jao

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Implementing image support for kitty terminal

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]

Tomas Hlavaty <tom@logand.com> writes:

> On Wed 07 Sep 2022 at 21:11, Eli Zaretskii <eliz@gnu.org> wrote:
>> In particular, the current model of image display
>> in the Emacs display engine is that an image is basically considered a
>> single very large character, from the screen layout POV.
>
> Are you talking about GUI emacs?
> Is that not different in tty emacs?
> I thought that the tty emacs has no concept of image (so far?).

Yes.  There is no image in text terminals.

>
>> I guess
>> that's not what you have in mind for the above, so IMO it's important
>> to come up with an alternative model that would somehow fit with the
>> current display code with only minor changes, if we want this not to
>> be too complex.
>
> For example, the tty based w3m web browser optionally supports images.
> It reserves space w*h chars on the tty and then uses external program
> to draw the image in the right place.
>
> I guess tty emacs would also need to reserve a w*h chars rectangle on
> the tty and then send the right escape code to fill that gap with the
> image.
>
> First, tty emacs would need to understand images instead of throwing
> error.  At least their dimensions.  Maybe using some library or even in
> elisp, for example like this
> https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html#l286
> I have not figured out, how to actually plug it into upstream emacs, as
> this seems to require a lot of changes to C code.
>
> Second, tty emacs would need to somehow reserve rectangle for the image.
> This might be lots of work I guess.
>
> Third, there would have to be a piece of code which actually draws the
> image.  In case of kitty terminal, the image path and coordinates would
> be sent to the tty using the kitty terminal protocol.  In case of Linux
> console, the image path and coordinates would be sent to an external
> program like w3mimgdisplay.
>

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Implementing image support for kitty terminal

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 3130 bytes --]

Jose A Ortega Ruiz <jao@gnu.org> writes:

> On Wed, Sep 07 2022, Tomas Hlavaty wrote:
>
>> On Wed 07 Sep 2022 at 21:11, Eli Zaretskii <eliz@gnu.org> wrote:
>>> In particular, the current model of image display
>>> in the Emacs display engine is that an image is basically considered a
>>> single very large character, from the screen layout POV.
>>
>> Are you talking about GUI emacs?
>> Is that not different in tty emacs?
>> I thought that the tty emacs has no concept of image (so far?).
>>
>>> I guess
>>> that's not what you have in mind for the above, so IMO it's important
>>> to come up with an alternative model that would somehow fit with the
>>> current display code with only minor changes, if we want this not to
>>> be too complex.
>>
>> For example, the tty based w3m web browser optionally supports images.
>> It reserves space w*h chars on the tty and then uses external program
>> to draw the image in the right place.
>>
>> I guess tty emacs would also need to reserve a w*h chars rectangle on
>> the tty and then send the right escape code to fill that gap with the
>> image.
>>
>> First, tty emacs would need to understand images instead of throwing
>> error.  At least their dimensions.  Maybe using some library or even in
>> elisp, for example like this
>> https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html#l286
>> I have not figured out, how to actually plug it into upstream emacs, as
>> this seems to require a lot of changes to C code.
>
> fwiw, in the kitty case, image libraries are going to be almost for sure
> available, it being an x11/wayland application.  so an emacs compiled
> with image support (but not necessarily linking against any x libs)
> doesn't seem like a strong requirement.  in that scenario, determining
> the size of an image wouldn't needed a lot of C code, would it? and, if
> those libraries are not available, a call to "file foo.png" is (at least
> in my system) telling me the size of foo.png... kind of hacky, of
> course, but as a first step maybe good enough.
>
>> Second, tty emacs would need to somehow reserve rectangle for the image.
>> This might be lots of work I guess.
>
> i was hoping this wouldn't be the case... at least when running inside
> terminal emulators, emacs nox seems pretty dexterous lying out windows
> and filly them with content, and i reckoned it must have a pretty good
> idea of their sizes.  but maybe i'm too optimistic :)
>
>> Third, there would have to be a piece of code which actually draws the
>> image.  In case of kitty terminal, the image path and coordinates would
>> be sent to the tty using the kitty terminal protocol.
>
> (as an aside, this would also work for other terminals like konsole or
> wayst, which support the protocol).
>
> cheers,
> jao

What?  You need to send the path of the image?  How would this work for
a SSH session?

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

 > So the change will/would need to make the glyph matrices (and their
 > generation) more like those used in the GUI code.

Or go a step further and treat this as a new kind of "GTUI" backend, for 
which everything is like in X and other GUIs, except that all fonts are 
size 1x1, images have sizes that are computed to match that notion and 
what else might be necessary.

Maybe.

And I'd leave the "normal" terminal code out of this.  There lie dragons.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Thu, Sep 08 2022, Akib Azmain Turja wrote:

[...]

> What?  You need to send the path of the image?  How would this work for
> a SSH session?

it's just one of the options.  you can also send the image data.  take a
look at the link above detailing the protocol, it's not too long.

jao
-- 
Don’t worry about people stealing your ideas. If your ideas are any
good, you’ll have to ram them down people’s throats.
  -Howard Aiken

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Akib Azmain Turja <akib@disroot.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Jose Antonio Ortega Ruiz <jao@gnu.org>,
>   emacs-devel@gnu.org
> Date: Thu, 08 Sep 2022 16:59:53 +0600
> 
> > On Wed 07 Sep 2022 at 21:11, Eli Zaretskii <eliz@gnu.org> wrote:
> >> In particular, the current model of image display
> >> in the Emacs display engine is that an image is basically considered a
> >> single very large character, from the screen layout POV.
> >
> > Are you talking about GUI emacs?
> > Is that not different in tty emacs?
> > I thought that the tty emacs has no concept of image (so far?).
> 
> Yes.  There is no image in text terminals.

No, we just explicitly ignore display properties whose values are
images when the frame is not a GUI frame.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> Date: Thu, 8 Sep 2022 14:47:48 +0200
> Cc: eliz@gnu.org, emacs-devel@gnu.org, jao@gnu.org
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> 
> And I'd leave the "normal" terminal code out of this.

Including frame-based redisplay in update_frame?

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

On 22-09-08 15:54 , Eli Zaretskii wrote:
>> Date: Thu, 8 Sep 2022 14:47:48 +0200
>> Cc: eliz@gnu.org, emacs-devel@gnu.org, jao@gnu.org
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>
>> And I'd leave the "normal" terminal code out of this.
> 
> Including frame-based redisplay in update_frame?

Yes, I'd leave that as-is.  The frame-based updates used to be quite
important for relatively dumb and slow terminals because it increases 
the possibility of reusing parts of the display that are alreay 
displayed on the terminal.

I guess terminal emulators allowing images and such are less likely to 
benefit from that.

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

On 22-09-08 16:03 , Gerd Möllmann wrote:
> On 22-09-08 15:54 , Eli Zaretskii wrote:
>>> Date: Thu, 8 Sep 2022 14:47:48 +0200
>>> Cc: eliz@gnu.org, emacs-devel@gnu.org, jao@gnu.org
>>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>>
>>> And I'd leave the "normal" terminal code out of this.
>>
>> Including frame-based redisplay in update_frame?
> 
> Yes, I'd leave that as-is.  The frame-based updates used to be quite
> important for relatively dumb and slow terminals because it increases 
> the possibility of reusing parts of the display that are alreay 
> displayed on the terminal.
> 
> I guess terminal emulators allowing images and such are less likely to 
> benefit from that.

Or even, do we still have terminals connected with, don't know, say 1200 
baud or some such?  If not, or not many, maybe such a new backend that 
is not frame-based could replace the frame-based one eventually?

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> Date: Thu, 8 Sep 2022 16:03:16 +0200
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org, jao@gnu.org
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> 
> On 22-09-08 15:54 , Eli Zaretskii wrote:
> >> Date: Thu, 8 Sep 2022 14:47:48 +0200
> >> Cc: eliz@gnu.org, emacs-devel@gnu.org, jao@gnu.org
> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> >>
> >> And I'd leave the "normal" terminal code out of this.
> > 
> > Including frame-based redisplay in update_frame?
> 
> Yes, I'd leave that as-is.  The frame-based updates used to be quite
> important for relatively dumb and slow terminals because it increases 
> the possibility of reusing parts of the display that are alreay 
> displayed on the terminal.

So you basically are saying that this should be a new kind of
window-system, yes?  Because we will need to distinguish it both in
places other than write_glyphs, and also in Lisp.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> Date: Thu, 8 Sep 2022 16:24:33 +0200
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org, jao@gnu.org
> 
> On 22-09-08 16:03 , Gerd Möllmann wrote:
> > On 22-09-08 15:54 , Eli Zaretskii wrote:
> >>> Date: Thu, 8 Sep 2022 14:47:48 +0200
> >>> Cc: eliz@gnu.org, emacs-devel@gnu.org, jao@gnu.org
> >>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> >>>
> >>> And I'd leave the "normal" terminal code out of this.
> >>
> >> Including frame-based redisplay in update_frame?
> > 
> > Yes, I'd leave that as-is.  The frame-based updates used to be quite
> > important for relatively dumb and slow terminals because it increases 
> > the possibility of reusing parts of the display that are alreay 
> > displayed on the terminal.
> > 
> > I guess terminal emulators allowing images and such are less likely to 
> > benefit from that.
> 
> Or even, do we still have terminals connected with, don't know, say 1200 
> baud or some such?

I don't think so, but we do have slow net links and ssh logins.

> If not, or not many, maybe such a new backend that is not
> frame-based could replace the frame-based one eventually?

That's possible.  Someone should take measurements and present the
results.

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Wed 07 Sep 2022 at 22:51, Eli Zaretskii <eliz@gnu.org> wrote:
> From: Tomas Hlavaty <tom@logand.com>
>> Second, tty emacs would need to somehow reserve rectangle for the image.
>> This might be lots of work I guess.
> The second part is the problem I'm talking about.  The other two are
> much simpler to add, I think.

Maybe the difficulty comes from the idea that tty emacs should somehow
match GUI emacs capabilities when displaying images.

The original use-case in this thread was using kitty terminal to view
documents like doc-view or pdf-tools.  There is no need to implement
full GUI like graphics capabilities for that.  In emacs-framebuffer, I
simply use the dimensions and position of the buffer to display images
without changing any layouting in tty emacs.  Adding images to eww would
be more complex, but I guess the change could be to shr layouting
written in elisp instead of changing emacs C code.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: jao@gnu.org, emacs-devel@gnu.org
> Date: Thu, 08 Sep 2022 20:30:39 +0200
> 
> On Wed 07 Sep 2022 at 22:51, Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Tomas Hlavaty <tom@logand.com>
> >> Second, tty emacs would need to somehow reserve rectangle for the image.
> >> This might be lots of work I guess.
> > The second part is the problem I'm talking about.  The other two are
> > much simpler to add, I think.
> 
> Maybe the difficulty comes from the idea that tty emacs should somehow
> match GUI emacs capabilities when displaying images.

No, it comes from the basic design of the display engine, which is
common to GUI and TTY frames, as far as layout is concerned.

> The original use-case in this thread was using kitty terminal to view
> documents like doc-view or pdf-tools.  There is no need to implement
> full GUI like graphics capabilities for that.  In emacs-framebuffer, I
> simply use the dimensions and position of the buffer to display images
> without changing any layouting in tty emacs.  Adding images to eww would
> be more complex, but I guess the change could be to shr layouting
> written in elisp instead of changing emacs C code.

You are again thinking about a completely different layer than the one
which bothers me the most.  As soon as you want to display images
together with text, or even scroll through tall images displayed alone
in a buffer, the basic layout issues come into play, because Emacs
scrolls and updates the display by considering internally which
portions of the window need to be redrawn and how, and it does that by
comparing old coordinates of the stuff that was on the screen with the
new coordinates of the stuff that should be on the screen.  For this,
it must now which part of the display occupies what screen location,
and this must be done in the coordinate system used by the display
code, which represents the screen as "glyph matrices": a series of
glyph rows, and each display element as a glyph in some glyph row.
This design is identical in both GUI and TTY windows, and the code
which deals with the glyph matrices is the same in both cases.

Of course, if all you want is to display a single image on the screen,
and don't want to support scrolling of images nor display of images
mixed with text, then these issues pretty much disappear.  But I very
much doubt that this will be good enough for our users.  It is
definitely much less interesting from my POV, because it basically
means we implement some kludgey and severely restricted semi-solution.

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Wed 07 Sep 2022 at 20:59, Jose A Ortega Ruiz <jao@gnu.org> wrote:
> On Wed, Sep 07 2022, Tomas Hlavaty wrote:
>> First, tty emacs would need to understand images instead of throwing
>> error.  At least their dimensions.  Maybe using some library or even in
>> elisp, for example like this
>> https://logand.com/sw/emacs-framebuffer/file/emacs-framebuffer.el.html#l286
>> I have not figured out, how to actually plug it into upstream emacs, as
>> this seems to require a lot of changes to C code.
>
> fwiw, in the kitty case, image libraries are going to be almost for sure
> available, it being an x11/wayland application.  so an emacs compiled
> with image support (but not necessarily linking against any x libs)
> doesn't seem like a strong requirement.  in that scenario, determining
> the size of an image wouldn't needed a lot of C code, would it?

Note the difference between
"a lot of changes to C code" vs "a lot of C code".

I already linked an elisp function which does that.
It would be easy to write it in C.
Emacs already has the capability anyway.
The problem is, that that capability is not available in tty emacs:

   (image-size (create-image "/tmp/a.jpg"))
   => (error "Window system frame should be used")

Would it be possible to fix that?

> and, if
> those libraries are not available, a call to "file foo.png" is (at least
> in my system) telling me the size of foo.png... kind of hacky, of
> course, but as a first step maybe good enough.

This is the easy part.  Then you need to change the existing emacs C
code to actually call that.

(btw isn't the file program potential security issue?)

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Thu 08 Sep 2022 at 17:13, Akib Azmain Turja <akib@disroot.org> wrote:
> What?  You need to send the path of the image?  How would this work for
> a SSH session?

The image needs to get to the kitty terminal somehow obviously.
If you read the kitty protocol, you'll find some options.
In emacs-framebuffer, remote images and documents are cached locally.
There are cases that break displaying remote images completely,
for example when the emacs and the framebuffer are on a different
machine than your terminal tunnel with tmux or screen.  I do not think
it is possible to fix that.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 08 Sep 2022 21:15:37 +0200
> 
> The problem is, that that capability is not available in tty emacs:

Yes, it _is_ available.  We just suppress it.

>    (image-size (create-image "/tmp/a.jpg"))
>    => (error "Window system frame should be used")

This is us deliberately failing image-size early on, because we don't
want complications down the road, and because we know images cannot be
displayed on TTY frames.

> Would it be possible to fix that?

I don't see why it would be hard.  E.g., this works just fine in a -nw
session:

  (create-image "/tmp/a.jpg")  

> > and, if
> > those libraries are not available, a call to "file foo.png" is (at least
> > in my system) telling me the size of foo.png... kind of hacky, of
> > course, but as a first step maybe good enough.
> 
> This is the easy part.  Then you need to change the existing emacs C
> code to actually call that.
> 
> (btw isn't the file program potential security issue?)

We don't need 'file', Emacs already knows how to compute the size of
an image.

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Thu 08 Sep 2022 at 15:13, Akib Azmain Turja <akib@disroot.org> wrote:
> IMHO, showing images by converting to text would be a better choice.

What do you mean by "converting images to text"?

> It would also allow to show image on Linux console.

That is already possible, see emacs-framebuffer.

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Thu 08 Sep 2022 at 21:45, Eli Zaretskii <eliz@gnu.org> wrote:
> Of course, if all you want is to display a single image on the screen,

Yes, that is enough for me and for the use-cases the author of this
thread mentioned.

> and don't want to support scrolling of images

I think this might be doable.

> nor display of images mixed with text,

I do not need that.

And in case of eww/shr, this has another layouting elisp code which
might be more convenient to change.

> then these issues pretty much disappear.

Yes, that was my point.

> But I very much doubt that this will be good enough for our users.  It
> is definitely much less interesting from my POV, because it basically
> means we implement some kludgey and severely restricted semi-solution.

It would not be a perfect solution but it might be simple and good enough.
Certainly better than nothing at all.

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Thu 08 Sep 2022 at 22:30, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tomas Hlavaty <tom@logand.com>
>> Cc: emacs-devel@gnu.org
>> Date: Thu, 08 Sep 2022 21:15:37 +0200
>> 
>> The problem is, that that capability is not available in tty emacs:
>
> Yes, it _is_ available.  We just suppress it.

Interesting, is it possible to somehow "un-suppress" it?

>>    (image-size (create-image "/tmp/a.jpg"))
>>    => (error "Window system frame should be used")
>
> This is us deliberately failing image-size early on, because we don't
> want complications down the road, and because we know images cannot be
> displayed on TTY frames.

I am asking emacs to tell me the size of the image.
I am not asking emacs to display it.

>> Would it be possible to fix that?
>
> I don't see why it would be hard.  E.g., this works just fine in a -nw
> session:
>
>   (create-image "/tmp/a.jpg")

Why does create-image work and image-size does not?

Why is this not "us deliberately failing create-image early on, because
we don't want complications down the road, and because we know images
cannot be displayed on TTY frames."?

Also this returns:

   (image :type jpeg :file "/tmp/a.jpg" :scale 1 :transform-smoothing t)

which does not tell me image width and height that I'd like to know.

>> > and, if
>> > those libraries are not available, a call to "file foo.png" is (at least
>> > in my system) telling me the size of foo.png... kind of hacky, of
>> > course, but as a first step maybe good enough.
>> 
>> This is the easy part.  Then you need to change the existing emacs C
>> code to actually call that.
>> 
>> (btw isn't the file program potential security issue?)
>
> We don't need 'file',

Yes, that was my point.

> Emacs already knows how to compute the size of an image.

So how do I get the size of an image?

My tty emacs does not know that, I get this error:

   (image-size (create-image "/tmp/a.jpg"))
   => (error "Window system frame should be used")

Maybe tty emacs already knows how to compute the size of an image, but I
have not figured a way of asking my tty emacs to share that information
with me.  Is there a way?

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Thu, Sep 08 2022, Tomas Hlavaty wrote:

> On Wed 07 Sep 2022 at 22:51, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tomas Hlavaty <tom@logand.com>
>>> Second, tty emacs would need to somehow reserve rectangle for the image.
>>> This might be lots of work I guess.
>> The second part is the problem I'm talking about.  The other two are
>> much simpler to add, I think.
>
> Maybe the difficulty comes from the idea that tty emacs should somehow
> match GUI emacs capabilities when displaying images.
>
> The original use-case in this thread was using kitty terminal to view
> documents like doc-view or pdf-tools.  There is no need to implement

fwiw, as the OP, no, that's not the original use-case i had in mind,
only part of it.  i want also to be able to display images in documents
rendered by eww, or by chat clients such as telega or circe that have
excellent support for it.  i'd also like to use image dired in these
kind of terminals.  in fact, i almost never use a pure tty or the
console.  what this would allow me is to have the same functionality
working at once in X11 and Wayland without suffering the complexities of
more heavy-weight graphical libraries and toolkits.

of course, ymmv :)

jao

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Thu, Sep 08 2022, Tomas Hlavaty wrote:

> On Thu 08 Sep 2022 at 21:45, Eli Zaretskii <eliz@gnu.org> wrote:
>> Of course, if all you want is to display a single image on the screen,
>
> Yes, that is enough for me and for the use-cases the author of this
> thread mentioned.

it's certainly not enough for me.  maybe i didn't explain myself clearly
at all, but i think Eli understood what i meant.  in fact, the idea of
new window-like "TGUI" backend mentioned by Gerd makes lots of sense to
me.

jao
-- 
The universe is full of magical things, patiently waiting for our
wits to grow sharper.
 -Eden Philpotts

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Thu 08 Sep 2022 at 21:29, Jose A Ortega Ruiz <jao@gnu.org> wrote:
> On Thu, Sep 08 2022, Tomas Hlavaty wrote:
>> The original use-case in this thread was using kitty terminal to view
>> documents like doc-view or pdf-tools.  There is no need to implement
>
> fwiw, as the OP, no, that's not the original use-case i had in mind,
> only part of it.  i want also to be able to display images in documents
> rendered by eww, or by chat clients such as telega or circe that have
> excellent support for it.  i'd also like to use image dired in these
> kind of terminals.  in fact, i almost never use a pure tty or the
> console.  what this would allow me is to have the same functionality
> working at once in X11 and Wayland without suffering the complexities of
> more heavy-weight graphical libraries and toolkits.

How is kitty not heavy-weight?
A graphical emacs already has all that.
Also there is no-toolkit emacs.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Thu, Sep 08 2022, Tomas Hlavaty wrote:

> On Thu 08 Sep 2022 at 21:29, Jose A Ortega Ruiz <jao@gnu.org> wrote:
>> On Thu, Sep 08 2022, Tomas Hlavaty wrote:
>>> The original use-case in this thread was using kitty terminal to view
>>> documents like doc-view or pdf-tools.  There is no need to implement
>>
>> fwiw, as the OP, no, that's not the original use-case i had in mind,
>> only part of it.  i want also to be able to display images in documents
>> rendered by eww, or by chat clients such as telega or circe that have
>> excellent support for it.  i'd also like to use image dired in these
>> kind of terminals.  in fact, i almost never use a pure tty or the
>> console.  what this would allow me is to have the same functionality
>> working at once in X11 and Wayland without suffering the complexities of
>> more heavy-weight graphical libraries and toolkits.
>
> How is kitty not heavy-weight?
> A graphical emacs already has all that.

After 4-5 days of continous use, a graphical emacs is consuming around
2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
workload except displaying images, is under 500Mb.  foot is even
slimmer.

> Also there is no-toolkit emacs.

not in wayland.

jao
-- 
Tact is the ability to tell a man he has an open mind when he has a
hole in his head.

Re: Implementing image support for kitty terminal

[Óscar Fuentes]

Jose A Ortega Ruiz <jao@gnu.org> writes:

>> How is kitty not heavy-weight?
>> A graphical emacs already has all that.
>
> After 4-5 days of continous use, a graphical emacs is consuming around
> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
> workload except displaying images, is under 500Mb.  foot is even
> slimmer.

In my experience of many years, Emacs RSS usage stabilizes around ~500
MB, which is what it is using right now in this box with M-x
emacs-uptime of 36 days. When it grows larger is because of some
extension eating memory or, more rarely, a bug in Emacs.

Take a look at the output of M-x memory-report.

If that is not enlightening, look for extensions that you use in GUI
Emacs and not in TUI Emacs, disable them and see if GUI Emacs' memory
keeps growing.

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Thu 08 Sep 2022 at 22:47, Jose A Ortega Ruiz <jao@gnu.org> wrote:
> After 4-5 days of continous use, a graphical emacs is consuming around
> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
> workload except displaying images, is under 500Mb.

Which graphical emacs exactly?  There are several options.

Maybe "except displaying images" makes the difference?
Have you tried the same workload both not displaying images?
Maybe if you open images in gui emacs and do not close
the buffers, they do not free the memory?
Or some other issue?

(I stopped using eshell because it eats my RAM but that is not graphics
related.)

>> Also there is no-toolkit emacs.
>
> not in wayland.

Under Xwayland?

I do not use wayland, but iirc pgtk backend should work there.

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Fri 09 Sep 2022 at 00:21, Óscar Fuentes <ofv@wanadoo.es> wrote:
> Take a look at the output of M-x memory-report.

great function, thank you

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Tomas Hlavaty wrote:

> On Thu 08 Sep 2022 at 22:47, Jose A Ortega Ruiz <jao@gnu.org> wrote:
>> After 4-5 days of continous use, a graphical emacs is consuming around
>> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
>> workload except displaying images, is under 500Mb.
>
> Which graphical emacs exactly?  There are several options.

all of them. 2Gb is usually for X11 with no-toolkit or lucid.  gtk is
easily in the 2.5Gb.  in wayland with pgtk, we're on the 3Gb range.  and
it doesn't seem to be bounded: every day of usage will add footprint (i
usually restart because i compile every week or so).

> Maybe "except displaying images" makes the difference?

maybe.  i'll know for sure when the new TGUI backend is implemented :)

> Have you tried the same workload both not displaying images?

yes.  i still see the effect.

> Maybe if you open images in gui emacs and do not close
> the buffers, they do not free the memory?
> Or some other issue?

believe me, i've tried lots of combinations, including selectively
excluding from my workflow some packages for a while, but haven't been
able to pinpoint a single culprit, except that when running in a
terminal RAM is stable (and low)..

> (I stopped using eshell because it eats my RAM but that is not graphics
> related.)

yeah, it could be anything in principle (i have around 60 packages
installed and use at least a couple dozen of them frequently), but in my
case the trouble seems to be triggered only in graphical emacsen (i've
also tried with and without imagemagick, but it seems to make no
difference)..

>> not in wayland.
>
> Under Xwayland?
>
> I do not use wayland, but iirc pgtk backend should work there.

it does, but the pgtk build, besides needing 3Gb RAM for continuous use,
is based on GTK, one of the "complicated toolkits" i'd be happy to
avoid.

also, a term emacs running inside kitty or foot in wayland is very
noticeably faster than the graphical counterpart (also in X11, but there
the difference is less remarkable), and i actually prefer using the term
and open my pdfs, images, etc. with an external viewer than using the
pgtk build.

all in all, i am a quite content emacs-inside-a-graphical-terminal user,
but also having the images in there, without needing any other program,
would be the bee's knees.

cheers,
jao
-- 
People hate as they love, unreasonably. -William M. Thackeray

Re: Implementing image support for kitty terminal

[Po Lu]

Jose A Ortega Ruiz <jao@gnu.org> writes:

> fwiw, as the OP, no, that's not the original use-case i had in mind,
> only part of it.  i want also to be able to display images in documents
> rendered by eww, or by chat clients such as telega or circe that have
> excellent support for it.  i'd also like to use image dired in these
> kind of terminals.  in fact, i almost never use a pure tty or the
> console.  what this would allow me is to have the same functionality
> working at once in X11 and Wayland without suffering the complexities of
> more heavy-weight graphical libraries and toolkits.

I'm going to bet that any terminal emulator that displays images will be
more heavyweight than Emacs on X --with-x-toolkit=no.

Re: Implementing image support for kitty terminal

[Po Lu]

Jose A Ortega Ruiz <jao@gnu.org> writes:

> After 4-5 days of continous use, a graphical emacs is consuming around
> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
> workload except displaying images, is under 500Mb.  foot is even
> slimmer.

And what is allocating that much memory?
Sounds like a bug to me.  A leak was recently fixed, could you please
try again?

Re: Implementing image support for kitty terminal

[Po Lu]

Jose A Ortega Ruiz <jao@gnu.org> writes:

> On Fri, Sep 09 2022, Tomas Hlavaty wrote:
>
>> On Thu 08 Sep 2022 at 22:47, Jose A Ortega Ruiz <jao@gnu.org> wrote:
>>> After 4-5 days of continous use, a graphical emacs is consuming around
>>> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
>>> workload except displaying images, is under 500Mb.
>>
>> Which graphical emacs exactly?  There are several options.
>
> all of them. 2Gb is usually for X11 with no-toolkit or lucid.  gtk is
> easily in the 2.5Gb.  in wayland with pgtk, we're on the 3Gb range.  and
> it doesn't seem to be bounded: every day of usage will add footprint (i
> usually restart because i compile every week or so).

Would you please run Emacs inside Masstif (if you have a sufficiently
large machine) for that amount of time, and find out what is allocating
so much memory?

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> So you basically are saying that this should be a new kind of
> window-system, yes?  Because we will need to distinguish it both in
> places other than write_glyphs, and also in Lisp.

Yes, exactly.

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> Or even, do we still have terminals connected with, don't know, say 1200 
>> baud or some such?
>
> I don't think so, but we do have slow net links and ssh logins.

That's true, I didn't think of that.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: jao@gnu.org, emacs-devel@gnu.org
> Date: Thu, 08 Sep 2022 22:03:41 +0200
> 
> On Thu 08 Sep 2022 at 22:30, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Tomas Hlavaty <tom@logand.com>
> >> Cc: emacs-devel@gnu.org
> >> Date: Thu, 08 Sep 2022 21:15:37 +0200
> >> 
> >> The problem is, that that capability is not available in tty emacs:
> >
> > Yes, it _is_ available.  We just suppress it.
> 
> Interesting, is it possible to somehow "un-suppress" it?

Yes, by changing the C code not to do that.

> >>    (image-size (create-image "/tmp/a.jpg"))
> >>    => (error "Window system frame should be used")
> >
> > This is us deliberately failing image-size early on, because we don't
> > want complications down the road, and because we know images cannot be
> > displayed on TTY frames.
> 
> I am asking emacs to tell me the size of the image.
> I am not asking emacs to display it.

But the code involved reuses some subroutines that are used for
display as well, and those signal an error.

It is never a problem with the current code because only GUI frames
can display images.  To get TTY frames be capable of some of that
functionality, we need changes on the C level (and perhaps also in
Lisp), but those changes are rather mechanical -- find out which code
signals the error and modify it to do that on some new condition, or
not at all.

> >> Would it be possible to fix that?
> >
> > I don't see why it would be hard.  E.g., this works just fine in a -nw
> > session:
> >
> >   (create-image "/tmp/a.jpg")
> 
> Why does create-image work and image-size does not?

See above.

> Also this returns:
> 
>    (image :type jpeg :file "/tmp/a.jpg" :scale 1 :transform-smoothing t)
> 
> which does not tell me image width and height that I'd like to know.

Of course.  create-image isn't supposed to return the size.  It was
just an example to show to you that some image-related functionality
does work on TTY frames; I didn't say that create-image is enough to
get the size.

> > Emacs already knows how to compute the size of an image.
> 
> So how do I get the size of an image?
> 
> My tty emacs does not know that, I get this error:
> 
>    (image-size (create-image "/tmp/a.jpg"))
>    => (error "Window system frame should be used")

See above.

> Maybe tty emacs already knows how to compute the size of an image, but I
> have not figured a way of asking my tty emacs to share that information
> with me.  Is there a way?

See above: you need changes on the C level.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Óscar Fuentes wrote:

> Jose A Ortega Ruiz <jao@gnu.org> writes:
>
>>> How is kitty not heavy-weight?
>>> A graphical emacs already has all that.
>>
>> After 4-5 days of continous use, a graphical emacs is consuming around
>> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
>> workload except displaying images, is under 500Mb.  foot is even
>> slimmer.
>
> In my experience of many years, Emacs RSS usage stabilizes around ~500
> MB, which is what it is using right now in this box with M-x
> emacs-uptime of 36 days. When it grows larger is because of some
> extension eating memory or, more rarely, a bug in Emacs.
>
> Take a look at the output of M-x memory-report.
>
> If that is not enlightening, look for extensions that you use in GUI
> Emacs and not in TUI Emacs, disable them and see if GUI Emacs' memory
> keeps growing.

thanks for your advice, but i've already tried all that and there's no
obvious candidate.  my conclusion is that it's the rendering to x
surfaces that somehow is leaking memory, maybe when it refreshes, or
colors or some other graphical resource (but not specially images,
memory raises without displaying them).  there is no extension that i
don't use in one place and not the other, with the exception of
doc-view, but i know it's not the cause because memory grows up in X
even when i don't use it.  likewise, i know gnus is not the cause, nor
notmuch, because i see the effect with either.

judging from your experience, it must be an external package that causes
the leak, in graphical mode only, but i haven't been able to pinpoint
one so far.  perhaps i'll try with vertico/corfu next.

all that said, i'd be a bit surprised if the leak came from a pure elisp
package...

thanks,
jao
-- 
The enjoyment of one's tools is an essential ingredient of successful
work. – Donald E. Knuth

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Po Lu wrote:

> Jose A Ortega Ruiz <jao@gnu.org> writes:
>
>> On Fri, Sep 09 2022, Tomas Hlavaty wrote:
>>
>>> On Thu 08 Sep 2022 at 22:47, Jose A Ortega Ruiz <jao@gnu.org> wrote:
>>>> After 4-5 days of continous use, a graphical emacs is consuming around
>>>> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
>>>> workload except displaying images, is under 500Mb.
>>>
>>> Which graphical emacs exactly?  There are several options.
>>
>> all of them. 2Gb is usually for X11 with no-toolkit or lucid.  gtk is
>> easily in the 2.5Gb.  in wayland with pgtk, we're on the 3Gb range.  and
>> it doesn't seem to be bounded: every day of usage will add footprint (i
>> usually restart because i compile every week or so).
>
> Would you please run Emacs inside Masstif (if you have a sufficiently
> large machine) for that amount of time, and find out what is allocating
> so much memory?

pardon my ignorance: what's masstif? search engines are giving me back
just pictures of dogs!

thanks,
jao
-- 
I shall be telling this with a sigh / Somewhere ages and ages hence: / Two
roads diverged in a wood, and I -- / I took the one less traveled by, / And
that has made all the difference.
 -Robert Frost, poet (1874-1963)

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Po Lu wrote:

> Jose A Ortega Ruiz <jao@gnu.org> writes:
>
>> After 4-5 days of continous use, a graphical emacs is consuming around
>> 2Gb for me, sometimes 3Gb; emacs-nox+kitty, combined, with the same
>> workload except displaying images, is under 500Mb.  foot is even
>> slimmer.
>
> And what is allocating that much memory?

i wish i knew :)

> Sounds like a bug to me.  A leak was recently fixed, could you please
> try again?

sure (last time i tried was a couple of weeks ago).  thanks for the
heads-up.

jao
-- 
They say when you play a Microsoft CD backwards you can hear satanic
messages...but that's nothing, if you play it forward it will install
windows!

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Po Lu wrote:

> Jose A Ortega Ruiz <jao@gnu.org> writes:
>
>> fwiw, as the OP, no, that's not the original use-case i had in mind,
>> only part of it.  i want also to be able to display images in documents
>> rendered by eww, or by chat clients such as telega or circe that have
>> excellent support for it.  i'd also like to use image dired in these
>> kind of terminals.  in fact, i almost never use a pure tty or the
>> console.  what this would allow me is to have the same functionality
>> working at once in X11 and Wayland without suffering the complexities of
>> more heavy-weight graphical libraries and toolkits.
>
> I'm going to bet that any terminal emulator that displays images will be
> more heavyweight than Emacs on X --with-x-toolkit=no.

well, and i'm going to be that, in wayland, the terminal emulator with
images is going to consume less than half the memory of the pgtk
build...  but if we never try, we'll never know.

jao
-- 
It is perfectly monstrous the way people go about, nowadays, saying
things against one behind one’s back that are absolutely and entirely
true. —Oscar Wilde, The Picture of Dorian Gray

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Jose A Ortega Ruiz <jao@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 09 Sep 2022 00:32:32 +0100
> 
> also, a term emacs running inside kitty or foot in wayland is very
> noticeably faster than the graphical counterpart

That advantage will almost certainly go away, at least partially,
if/when we implement the new backend as proposed by Gerd, because some
of that advantage is due to the simplicity of the TTY frame geometry
and text layout, something that will be lost as soon as we support
variable-height screen lines on TTY frames.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Eli Zaretskii wrote:

>> From: Jose A Ortega Ruiz <jao@gnu.org>
>> Cc: emacs-devel@gnu.org
>> Date: Fri, 09 Sep 2022 00:32:32 +0100
>> 
>> also, a term emacs running inside kitty or foot in wayland is very
>> noticeably faster than the graphical counterpart
>
> That advantage will almost certainly go away, at least partially,
> if/when we implement the new backend as proposed by Gerd, because some
> of that advantage is due to the simplicity of the TTY frame geometry
> and text layout, something that will be lost as soon as we support
> variable-height screen lines on TTY frames.

hmm. in the image display protocol that kitty supports, everything can
be done, if desired, in terms of rows and columns, and the terminal
itself works, as far as i understand, in terms of character cells:
couldn't we retain that modus operandi?  for instance, displaying an
image would amount (logically) to displaying a bunch of lines of text,
all of the same length, with the only peculiarity that maybe the cells
on the borders wouldn't look completely filled.  (i was at first even
imagining an admittedly obnoxious hack consisting of simply writing
first n rows of m arbitrary characters, and then using the image
protocol to ask kitty to fill that character rectangle with the image,
because i thought that for the TTY terminal it was easy to determine the
rows and columns available in a given window around a given position).

jao

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Jose A Ortega Ruiz <jao@gnu.org>
> Cc: tom@logand.com, emacs-devel@gnu.org
> Date: Fri, 09 Sep 2022 07:21:34 +0100
> 
> On Fri, Sep 09 2022, Eli Zaretskii wrote:
> 
> >> From: Jose A Ortega Ruiz <jao@gnu.org>
> >> Cc: emacs-devel@gnu.org
> >> Date: Fri, 09 Sep 2022 00:32:32 +0100
> >> 
> >> also, a term emacs running inside kitty or foot in wayland is very
> >> noticeably faster than the graphical counterpart
> >
> > That advantage will almost certainly go away, at least partially,
> > if/when we implement the new backend as proposed by Gerd, because some
> > of that advantage is due to the simplicity of the TTY frame geometry
> > and text layout, something that will be lost as soon as we support
> > variable-height screen lines on TTY frames.
> 
> hmm. in the image display protocol that kitty supports, everything can
> be done, if desired, in terms of rows and columns, and the terminal
> itself works, as far as i understand, in terms of character cells:
> couldn't we retain that modus operandi?

I think you misunderstand what Gerd proposed.  He proposed abandoning
the fixed-line-height assumption of the text-mode layout calculations.
Thus, TTY frames could have screen lines of variable height, although
their granularity will still be the height of the TTY character cell.

Once screen lines are of variable height, some of the advantages of
TTY display go away, because we no longer can assume that the next
or previous screen line is exactly one character cell away.

> for instance, displaying an image would amount (logically) to
> displaying a bunch of lines of text, all of the same length, with
> the only peculiarity that maybe the cells on the borders wouldn't
> look completely filled.

That won't work, because Emacs must know the screen layout; you cannot
lie to it with impunity.  For example, with your idea above, if point
is in the first line of a window, and you type C-n, and the next line
displays an image which takes more than one character cell's height,
then what will be the line number, as it's known to Emacs, when point
moves to that second line?  If you tell Emacs that there are 10 lines
of text in-between, instead of a single tall line, Emacs will think
the next line is line 11, whereas from the user POV it's line 2.

A lot of Emacs features depend on Emacs knowing exactly what's on the
screen, so you cannot easily lie to it about that.

Re: Implementing image support for kitty terminal

[Po Lu]

Jose A Ortega Ruiz <jao@gnu.org> writes:

> pardon my ignorance: what's masstif? search engines are giving me back
> just pictures of dogs!

Sorry, autocorrect problem.  Massif is the Valgrind heap analyzer tool.
Running Emacs under that will let you know if some C code is allocating
excessive amounts from the heap.

But to run Emacs under it for a long period of time, you need a fast
computer with lots of RAM; it's not for everyone.

Thanks.

Re: Implementing image support for kitty terminal

[Stefan Monnier]

>> Or even, do we still have terminals connected with, don't know, say 1200 
>> baud or some such?
> I don't think so, but we do have slow net links and ssh logins.

I wonder how this interacts with our display optimizations, to be
honest: slow network connections tend to suffer mostly in terms of
"packets per second" rather than "bytes per second", so an optimization
makes a difference only if it changes the size of a display update
from N+1 packets to N packets (or fewer).

E.g. when our display updates (counted in bytes and without frame-based
redisplay) fall below the "1 packet" limit (which I presume is around
1.5kB), then using frame-based redisplay doesn't make any noticeable
difference to "ssh logins over congested/slow networks".


        Stefan

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>   emacs-devel@gnu.org,
>   jao@gnu.org
> Date: Fri, 09 Sep 2022 08:19:44 -0400
> 
> >> Or even, do we still have terminals connected with, don't know, say 1200 
> >> baud or some such?
> > I don't think so, but we do have slow net links and ssh logins.
> 
> I wonder how this interacts with our display optimizations, to be
> honest: slow network connections tend to suffer mostly in terms of
> "packets per second" rather than "bytes per second", so an optimization
> makes a difference only if it changes the size of a display update
> from N+1 packets to N packets (or fewer).
> 
> E.g. when our display updates (counted in bytes and without frame-based
> redisplay) fall below the "1 packet" limit (which I presume is around
> 1.5kB), then using frame-based redisplay doesn't make any noticeable
> difference to "ssh logins over congested/slow networks".

The theory is well-understood, I think.  What we need is measurements
in various usage scenarios.

Btw, slow/congested network suffer not only from packet per second
issue, but also from packet loss.

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Or even, do we still have terminals connected with, don't know, say 1200 
>>> baud or some such?
>> I don't think so, but we do have slow net links and ssh logins.
>
> I wonder how this interacts with our display optimizations, to be
> honest: slow network connections tend to suffer mostly in terms of
> "packets per second" rather than "bytes per second", so an optimization
> makes a difference only if it changes the size of a display update
> from N+1 packets to N packets (or fewer).

Where the size of packets differs in interactive use, I think.  I mean,
I can't imagine how to use SSH interactively if it waited until packets
are large enough.

Hm.  Hard to tell what effect that packeting has.  How does SSH decide
to send something over the network in interactive mode?

Re: Implementing image support for kitty terminal

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]

On Fri, Sep 09, 2022 at 02:59:19PM +0200, Gerd Möllmann wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >>> Or even, do we still have terminals connected with, don't know, say 1200 
> >>> baud or some such?
> >> I don't think so, but we do have slow net links and ssh logins.
> >
> > I wonder how this interacts with our display optimizations, to be
> > honest: slow network connections tend to suffer mostly in terms of
> > "packets per second" rather than "bytes per second", so an optimization
> > makes a difference only if it changes the size of a display update
> > from N+1 packets to N packets (or fewer).
> 
> Where the size of packets differs in interactive use, I think.  I mean,
> I can't imagine how to use SSH interactively if it waited until packets
> are large enough.
> 
> Hm.  Hard to tell what effect that packeting has.  How does SSH decide
> to send something over the network in interactive mode?

Nagle's algorithm and TCP_NODELAY are the keywords here, AFAIK.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Implementing image support for kitty terminal

[Stefan Monnier]

Eli Zaretskii [2022-09-09 15:53:12] wrote:
> The theory is well-understood, I think.  What we need is measurements
> in various usage scenarios.

Indeed.

> Btw, slow/congested network suffer not only from packet per second
> issue, but also from packet loss.

That still only affects the "per packet" behavior.

Gerd Möllmann [2022-09-09 14:59:19] wrote:
> Where the size of packets differs in interactive use, I think.  I mean,
> I can't imagine how to use SSH interactively if it waited until packets
> are large enough.

Indeed, but that tends to affect the other direction (from the keyboard
to the remote) more significantly.  For redisplay, we can send the whole
byte sequence as a single "write".  Most simple redisplays should result
in a single packet.

> Hm.  Hard to tell what effect that packeting has.  How does SSH decide
> to send something over the network in interactive mode?

I assume it sends it "as soon as possible", modulo something like
Nagle's algorithm.

BTW, nowadays I tend to use `mosh` for many of my "ssh" connections,
where the behavior may be yet different since `mosh` does the "terminal
emulation" rendering on the remote host (into the equivalent of our
glyph matrix) and then uses an ad-hoc protocol (over UDP) to update the
local "glass".  So it does its own optimizations (comparing its "glyph
matrix" to what (it thinks) is currently on the glass) and I'm not sure
how/if it's affected by ours.


        Stefan

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

 > Nagle's algorithm and TCP_NODELAY are the keywords here, AFAIK.

Yup, SSH must be using TCP_NODELAY interactively.  I can't see how else 
that would work.  And OpenSSH docs say it uses TCP_NODELAY for X 
forwarding, so that makes sense.

Which means that our "intelligent" update on terminals, which does 
potentially many writes may produce as many packets.  Which may be a 
problem sometimes :-).

It wouldn't be hard to buffer the writes until the update is done, I 
think.  But is it worth it as an option?  Don't know.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> Date: Fri, 9 Sep 2022 16:10:07 +0200
> Cc: emacs-devel@gnu.org
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> 
> It wouldn't be hard to buffer the writes until the update is done, I 
> think.

I think we already do, see this part of update_frame_1:

  /* Now update the rest of the lines.  */
  for (i = 0; i < desired_matrix->nrows - 1 && (force_p || !input_pending); i++)
    {
      if (MATRIX_ROW_ENABLED_P (desired_matrix, i))
	{
	  if (FRAME_TERMCAP_P (f))
	    {
	      /* Flush out every so many lines.
		 Also flush out if likely to have more than 1k buffered
		 otherwise.   I'm told that some telnet connections get
		 really screwed by more than 1k output at once.  */
	      FILE *display_output = FRAME_TTY (f)->output;
	      if (display_output)
		{
		  ptrdiff_t outq = __fpending (display_output);
		  if (outq > 900
		      || (outq > 20 && ((i - 1) % preempt_count == 0)))
		    fflush (display_output);
		}
	    }

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

On 22-09-09 16:20 , Eli Zaretskii wrote:
>> Date: Fri, 9 Sep 2022 16:10:07 +0200
>> Cc: emacs-devel@gnu.org
>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>>
>> It wouldn't be hard to buffer the writes until the update is done, I
>> think.
> 
> I think we already do, see this part of update_frame_1:
> 
>    /* Now update the rest of the lines.  */
>    for (i = 0; i < desired_matrix->nrows - 1 && (force_p || !input_pending); i++)
>      {
>        if (MATRIX_ROW_ENABLED_P (desired_matrix, i))
> 	{
> 	  if (FRAME_TERMCAP_P (f))
> 	    {
> 	      /* Flush out every so many lines.
> 		 Also flush out if likely to have more than 1k buffered
> 		 otherwise.   I'm told that some telnet connections get
> 		 really screwed by more than 1k output at once.  */
> 	      FILE *display_output = FRAME_TTY (f)->output;
> 	      if (display_output)
> 		{
> 		  ptrdiff_t outq = __fpending (display_output);
> 		  if (outq > 900
> 		      || (outq > 20 && ((i - 1) % preempt_count == 0)))
> 		    fflush (display_output);
> 		}
> 	    }

Oh, right.

The comment there is also interesting.  Looks like some connections 
don't like larger buffers, some benefit from it, some don't care. 
Whatever we do, it's always wrong :-).

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Eli Zaretskii wrote:

[...]

>> for instance, displaying an image would amount (logically) to
>> displaying a bunch of lines of text, all of the same length, with
>> the only peculiarity that maybe the cells on the borders wouldn't
>> look completely filled.
>
> That won't work, because Emacs must know the screen layout; you cannot
> lie to it with impunity.  For example, with your idea above, if point
> is in the first line of a window, and you type C-n, and the next line
> displays an image which takes more than one character cell's height,
> then what will be the line number, as it's known to Emacs, when point
> moves to that second line?  If you tell Emacs that there are 10 lines
> of text in-between, instead of a single tall line, Emacs will think
> the next line is line 11, whereas from the user POV it's line 2.

Say that, in that line, i have an image, which emacs knows about because
there's, say, a text property with 4 rows x 5 columns as the image
extent (which we derive from knowing the image size and the terminal
cell size, in pixels).  For movement and display purpose, we tell Emacs
to interpret that as the block of pure text

xxxx
xxxx
xxxx
xxxx
xxxx

and emacs, initially, displays that, as a normal tty would, with point
at the beginning of the block.  Now, it computes how many lines are
visible after any required scrolling, and then asks kitty to draw, on
the visible rectangle, the corresponding part of the image.

All movements and positioning would happen as if all that is being
displayed are blocks of text.  Of course that means that line numbers
won't treat images as spawning just one line, but several.  I find that
actually an advantage from the user's point of view, because the images
are taking more than one line in the glass, so to me it's very natural
that emacs tells me that, at the beginning of the image, line is no. 2,
and, at the end, line is no. 20.  and images could be scrolled line by
line in that schema. These terminals use the GPU for rendering when
available, so my expectation is that frequent redraws of the displayed
image sections will be fast enough, but i could well be wrong.

> A lot of Emacs features depend on Emacs knowing exactly what's on the
> screen, so you cannot easily lie to it about that.

Yes.  I am proposing we tell emacs that what's on the screen is some
lines of xs-text and making that, actually, true in every sense except
that we also tell kitty to display those rows differently at the very
end, when emacs is done with its text display duties.

jao
-- 
Whenever you commend, add your reasons for doing so; it is this which
distinguishes the approbation of a man of sense from the flattery of
sycophants and admiration of fools. -Richard Steele (1672-1729)

Re: Implementing image support for kitty terminal

[Stefan Monnier]

>> 		 Also flush out if likely to have more than 1k buffered
>> 		 otherwise.   I'm told that some telnet connections get
>> 		 really screwed by more than 1k output at once.  */
[...]
> The comment there is also interesting.  Looks like some connections don't
> like larger buffers, some benefit from it, some don't care. Whatever we do,
> it's always wrong :-).

Note that the comment dates back to:

    commit 4588ec205730239596486e8ad4d18d541917199a
    Author: Jim Blandy <jimb@redhat.com>
    Date:   Wed Jul 3 12:10:07 1991 +0000
    
        Initial revision

so there's a chance it's ... slightly dated?


        Stefan

Re: Implementing image support for kitty terminal

[chad]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

On Fri, Sep 9, 2022 at 10:57 AM Jose A Ortega Ruiz <jao@gnu.org> wrote:

> [...]
>
>

> xxxx

> xxxx

> xxxx  How will you handle

> xxxx  this text?

> xxxx

Note that users have to reach that text, for example with C-n, including
both lines. Also, it needs to wrap between "handle" and "this". This
ignores a bunch of other issues like "what happens if the image is sliced
into more lines than will fit on the screen at once?".

This is a commonly recurring idea. In the 90's, HTML went through these
concerns with a model that was mostly (Grail, Amaya, etc.) read-only, and
ended up with CSS and the DOM, which are conceptually close to emacs'
redisplay engine and glyph matrix.

It is certainly possible to forbid/ignore most of the complexity in favor
of a simple hack, as people have mentioned already. That doesn't help
figure out the issues that would be necessary to make the feature be fully
(or even mostly) emacs-capable, and if that's what you want, doesn't
emacs-framebuffer already give you that?

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 2632 bytes --]

Re: Implementing image support for kitty terminal

[Stefan Monnier]

> cell size, in pixels).  For movement and display purpose, we tell Emacs
> to interpret that as the block of pure text
>
> xxxx
> xxxx
> xxxx
> xxxx
> xxxx

That's no easier than interpreting it as an image displayed via kitty :-)
IOW, the difficulty resides elsewhere.

I think there's no real technical difficulty.  There's simply work to be
done on the code.  Either by defining a new "TGUI" window-system which
uses the GUI part of the display rendering code, or by changing the tty
code to be closer to that of the GUI code.

Which option is simplest and/or best, I don't know.
And I suspect we won't know for sure until someone tries to do it.


        Stefan

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, chad wrote:

> On Fri, Sep 9, 2022 at 10:57 AM Jose A Ortega Ruiz <jao@gnu.org> wrote:
>
>> [...]
>>
>>
>
>> xxxx
>
>> xxxx
>
>> xxxx  How will you handle
>
>> xxxx  this text?
>
>> xxxx
>
> Note that users have to reach that text, for example with C-n, including
> both lines. Also, it needs to wrap between "handle" and "this". This
> ignores a bunch of other issues like "what happens if the image is sliced
> into more lines than will fit on the screen at once?".

well, i simply wouldn't.  text alongside images in a window wouldn't be
allowed, and it would appear below the image (that's actually how eww
behaves with images, even in graphical terminals, at least in some
cases).  i don't want to convert term emacs in a graphical wyswig
editor, just to display images every now and then.  perfect enemy of the
better and all that :)

[...]

> It is certainly possible to forbid/ignore most of the complexity in favor
> of a simple hack, as people have mentioned already. That doesn't help
> figure out the issues that would be necessary to make the feature be fully
> (or even mostly) emacs-capable, and if that's what you want, doesn't
> emacs-framebuffer already give you that?

i've never used emacs-framebuffer, but i don't think so.  i want, for
instance, to run emacs in a wayland or x11 term, possibly interacting
with other applications there (e.g., i want to watch videos, or, every
now and then, open complex webpages in firefox).

cheers,
jao
-- 
Spend the afternoon. You can't take it with you.
  -Annie Dillard, author (b. 1945)

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Jose A Ortega Ruiz <jao@gnu.org>
> Cc: tom@logand.com, emacs-devel@gnu.org
> Date: Fri, 09 Sep 2022 15:56:35 +0100
> 
> Say that, in that line, i have an image, which emacs knows about because
> there's, say, a text property with 4 rows x 5 columns as the image
> extent (which we derive from knowing the image size and the terminal
> cell size, in pixels).  For movement and display purpose, we tell Emacs
> to interpret that as the block of pure text
> 
> xxxx
> xxxx
> xxxx
> xxxx
> xxxx
> 
> and emacs, initially, displays that, as a normal tty would, with point
> at the beginning of the block.  Now, it computes how many lines are
> visible after any required scrolling, and then asks kitty to draw, on
> the visible rectangle, the corresponding part of the image.
> 
> All movements and positioning would happen as if all that is being
> displayed are blocks of text.  Of course that means that line numbers
> won't treat images as spawning just one line, but several.  I find that
> actually an advantage from the user's point of view, because the images
> are taking more than one line in the glass, so to me it's very natural
> that emacs tells me that, at the beginning of the image, line is no. 2,
> and, at the end, line is no. 20.

But you do want to support text and images in the same buffer, don't
you?  So the following situation:

  tttttttttttttttttttt (1)
       IIIIIII
       IIIIIII
       IIIIIII
  tttt IIIIIII ttttttt (2)
  tttttttttttttttttttt (3)

where "t" is text is "I" is an image, should be supported, yes?  So
now, if point is on the top-most text line (1), and the user types
C-n, don't you want point to the text on the line that displays the
image, line (2)?  This is what happens on GUI frames, and I very much
doubt that users will be happy to see the cursor somewhere between
lines (1) and (2).  And even if they do like that, what would be the
position of point in that case, i.e. what will "C-x =" display?

> Yes.  I am proposing we tell emacs that what's on the screen is some
> lines of xs-text and making that, actually, true in every sense except
> that we also tell kitty to display those rows differently at the very
> end, when emacs is done with its text display duties.

That will break too much of both user expectations and Emacs features,
whereby every position on the screen where we put the cursor has some
buffer address.  What you suggest is a trick that may make display
easier (and I'm not even sure in this part), but it will require many
changes all over the rest of Emacs, because Emacs isn't prepared to
deal with such disconnect between the screen coordinates and the text
it displays.

Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz]

On Fri, Sep 09 2022, Eli Zaretskii wrote:

[...]

> But you do want to support text and images in the same buffer, don't
> you?  So the following situation:
>
>   tttttttttttttttttttt (1)
>        IIIIIII
>        IIIIIII
>        IIIIIII
>   tttt IIIIIII ttttttt (2)
>   tttttttttttttttttttt (3)
>
> where "t" is text is "I" is an image, should be supported, yes?

no, not necessarily.  i'd be already pretty happy even if that wouldn't
be possible. for my uses cases, rendering that as

>   tttttttttttttttttttt (1)
>        IIIIIII
>        IIIIIII
>        IIIIIII
>        IIIIIII 
>   tttt ttttttt         (2)
>   tttttttttttttttttttt (3)

would be good enough.  moreover, there are few cases of images inserted
by even current graphical emacs (at least in my use cases) with text
flowing on the sides (i remember even having to advice eww at some point
to get inline images).

> So now, if point is on the top-most text line (1), and the user types
> C-n, don't you want point to the text on the line that displays the
> image, line (2)?  This is what happens on GUI frames, and I very much
> doubt that users will be happy to see the cursor somewhere between
> lines (1) and (2).  And even if they do like that, what would be the
> position of point in that case, i.e. what will "C-x =" display?

well, i cannot speak for other users.  for me, something like the above,
even if far from perfect, would be a big improvement over no images at
all.

>> Yes.  I am proposing we tell emacs that what's on the screen is some
>> lines of xs-text and making that, actually, true in every sense except
>> that we also tell kitty to display those rows differently at the very
>> end, when emacs is done with its text display duties.
>
> That will break too much of both user expectations and Emacs features,
> whereby every position on the screen where we put the cursor has some
> buffer address.  What you suggest is a trick that may make display
> easier (and I'm not even sure in this part), but it will require many
> changes all over the rest of Emacs, because Emacs isn't prepared to
> deal with such disconnect between the screen coordinates and the text
> it displays.

fair enough.  that's the kind of warning i was looking for, to avoid
wasting time chasing wild geese.  if even an imperfect hack is going to
need such an effort, it's not worth it.

thanks,
jao
-- 
There are two ways to write error-free programs; only the third one
works.
  - Alan Perlis, Epigrams on Programming

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Fri 09 Sep 2022 at 16:34, Jose A Ortega Ruiz <jao@gnu.org> wrote:
> i want to watch videos

unless you want to somehow watch videos in an emacs buffer (is that
possible?); it is easy.  see for example the framebuffer-fbmpv function.
it runs mpv on the linux framebuffer.  you could trivially change it to
invoke a video player of your choice

> open complex webpages in firefox

firefox does not support framebuffer as far as i know

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Fri 09 Sep 2022 at 08:43, Eli Zaretskii <eliz@gnu.org> wrote:
> Yes, by changing the C code not to do that.
>
>> >>    (image-size (create-image "/tmp/a.jpg"))
>> >>    => (error "Window system frame should be used")
>> >
>> > This is us deliberately failing image-size early on, because we don't
>> > want complications down the road, and because we know images cannot be
>> > displayed on TTY frames.
>> 
>> I am asking emacs to tell me the size of the image.
>> I am not asking emacs to display it.
>
> But the code involved reuses some subroutines that are used for
> display as well, and those signal an error.
>
> It is never a problem with the current code because only GUI frames
> can display images.  To get TTY frames be capable of some of that
> functionality, we need changes on the C level (and perhaps also in
> Lisp), but those changes are rather mechanical -- find out which code
> signals the error and modify it to do that on some new condition, or
> not at all.

Understand.
Thank you for the explanation.

Re: Implementing image support for kitty terminal

[Visuwesh]

[சனி செப்டம்பர் 10, 2022] Tomas Hlavaty wrote:

> On Fri 09 Sep 2022 at 16:34, Jose A Ortega Ruiz <jao@gnu.org> wrote:
>> i want to watch videos
>
> unless you want to somehow watch videos in an emacs buffer (is that
> possible?)  

In GUI Emacs, it is.  https://lars.ingebrigtsen.no/2021/11/09/finally-videos-in-eww/

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Sat 10 Sep 2022 at 09:05, Visuwesh <visuweshm@gmail.com> wrote:
> [சனி செப்டம்பர் 10, 2022] Tomas Hlavaty wrote:
>> On Fri 09 Sep 2022 at 16:34, Jose A Ortega Ruiz <jao@gnu.org> wrote:
>>> i want to watch videos
>>
>> unless you want to somehow watch videos in an emacs buffer (is that
>> possible?)  
>
> In GUI Emacs, it is.  https://lars.ingebrigtsen.no/2021/11/09/finally-videos-in-eww/

Interesting.

in no-toolkit Emacs, I get:

   Content-type video/mp4 is unsupported
   Direct link to the document

I guess it works under very special circumstances.

It would likely not work under kitta terminal either.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> From: Jose A Ortega Ruiz <jao@gnu.org>
> Cc: tom@logand.com, emacs-devel@gnu.org
> Date: Sat, 10 Sep 2022 01:45:53 +0100
> 
> On Fri, Sep 09 2022, Eli Zaretskii wrote:
> 
> [...]
> 
> > But you do want to support text and images in the same buffer, don't
> > you?  So the following situation:
> >
> >   tttttttttttttttttttt (1)
> >        IIIIIII
> >        IIIIIII
> >        IIIIIII
> >   tttt IIIIIII ttttttt (2)
> >   tttttttttttttttttttt (3)
> >
> > where "t" is text is "I" is an image, should be supported, yes?
> 
> no, not necessarily.  i'd be already pretty happy even if that wouldn't
> be possible. for my uses cases, rendering that as
> 
> >   tttttttttttttttttttt (1)
> >        IIIIIII
> >        IIIIIII
> >        IIIIIII
> >        IIIIIII 
> >   tttt ttttttt         (2)
> >   tttttttttttttttttttt (3)
> 
> would be good enough.  moreover, there are few cases of images inserted
> by even current graphical emacs (at least in my use cases) with text
> flowing on the sides (i remember even having to advice eww at some point
> to get inline images).

You misunderstand what I described, I think.  It is a simple case of
text mixed with an image, i.e. a line which has both text and images.
It is not "text flowing on the sides of the image", which Emacs
currently doesn't support even on GUI frames, not unless the image is
displayed in slices.  Try looking at some URL with images with EWW on
a GUI frame, and you will see what I described.

You said you did want to be able to have EWW display pages with
images.  If you do, what I described will be needed.  By contrast,
what you describe is a line that has only a single image, and no text
(not even some whitespace) nor another image near it.  Which is
possible, but is just a subset of what needs to be supported.

Re: Implementing image support for kitty terminal

[Po Lu]

Tomas Hlavaty <tom@logand.com> writes:

> Interesting.
>
> in no-toolkit Emacs, I get:
>
>    Content-type video/mp4 is unsupported
>    Direct link to the document
>
> I guess it works under very special circumstances.

It only works if you build with xwidgets, which requires GTK 3 and
WebKitGTK.

In addition it currently does not work on many systems due to a WebKit
bug.

Re: Implementing image support for kitty terminal

[Gerd Möllmann]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Note that the comment dates back to:
>
>     commit 4588ec205730239596486e8ad4d18d541917199a
>     Author: Jim Blandy <jimb@redhat.com>
>     Date:   Wed Jul 3 12:10:07 1991 +0000
>     
>         Initial revision
>
> so there's a chance it's ... slightly dated?

Yup, the telnet part at least looks to me like a coprolith :-).

For the rest, it's hard to tell.  That's another heuristic where it's
impossible to tell (for me) if it's (still) needed or not.

Maybe we could remove it, and wait for complaints?  I wouldm't mind.

BTW, do you have a slow connection you could use to test larger stream
buffer sizes with, at least anecdotally?  If we remove the flushes,
except at the end of the update, and give the output stream a large
buffer (setvbuf, or something), we could at least get some impression if
that may benefit users.

I don't have any SSH access to other computers, alas.

Re: Implementing image support for kitty terminal

[Kyaw Thu Soe]

test 

Re: Implementing image support for kitty terminal

[xenodasein--- via Emacs development discussions.]

Are you considering to change the text terminal infrastructure for
Emacs just to accommodate one wayward desktop app that claims to be
a terminal and a Kindle at the same time?  Isn't frame handling
already Frankenstein-ized enough between mixing graphical and text
modes?  Wouldn't it be better to focus on how to solve that instead;
supporting singular apps would then become much easier.

Re: Implementing image support for kitty terminal

[Eli Zaretskii]

> Date: Sat, 10 Sep 2022 12:15:32 +0200 (CEST)
> From: xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> Are you considering to change the text terminal infrastructure for
> Emacs just to accommodate one wayward desktop app that claims to be
> a terminal and a Kindle at the same time?

Possibly.  However, at least one proposal for implementing that leaves
most of the infrastructure intact.

> Isn't frame handling already Frankenstein-ized enough between mixing
> graphical and text modes?

Not really, no.  Not to such an extreme degree.

> Wouldn't it be better to focus on how to solve that instead;
> supporting singular apps would then become much easier.

If someone comes up with a suggestion for how to refactor the display
code to make its maintenance easier, I think we will welcome that.
But we cannot reject other enhancements because of the complexity of
what we have.  That complexity didn't prevent us from implementing
several display-related enhancements in the recent years, and I don't
see why this particular one should be an exception.

Re: Implementing image support for kitty terminal

[xenodasein--- via Emacs development discussions.]

>> Are you considering to change the text terminal infrastructure for
>> Emacs just to accommodate one wayward desktop app that claims to be
>> a terminal and a Kindle at the same time?

> Possibly. However, at least one proposal for implementing that leaves
> most of the infrastructure intact.

...

> If someone comes up with a suggestion for how to refactor the display
> code to make its maintenance easier, I think we will welcome that.
> But we cannot reject other enhancements because of the complexity of
> what we have. That complexity didn't prevent us from implementing
> several display-related enhancements in the recent years, and I don't
> see why this particular one should be an exception.

I guess the hard part is to balance the amount of new enhancements
to not build up to a point where it isn't too big of a deterrent to
a possible future, uh, modernization?  If supporting Kitty is easy
within the current framework, why not; this sounds too good though.

Re: Implementing image support for kitty terminal

[Akib Azmain Turja]

[-- Attachment #1: Type: text/plain, Size: 1558 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 10 Sep 2022 12:15:32 +0200 (CEST)
>> From: xenodasein--- via "Emacs development discussions." <emacs-devel@gnu.org>
>> 
>> Are you considering to change the text terminal infrastructure for
>> Emacs just to accommodate one wayward desktop app that claims to be
>> a terminal and a Kindle at the same time?
>
> Possibly.  However, at least one proposal for implementing that leaves
> most of the infrastructure intact.
>
>> Isn't frame handling already Frankenstein-ized enough between mixing
>> graphical and text modes?
>
> Not really, no.  Not to such an extreme degree.
>
>> Wouldn't it be better to focus on how to solve that instead;
>> supporting singular apps would then become much easier.
>
> If someone comes up with a suggestion for how to refactor the display
> code to make its maintenance easier, I think we will welcome that.
> But we cannot reject other enhancements because of the complexity of
> what we have.  That complexity didn't prevent us from implementing
> several display-related enhancements in the recent years, and I don't
> see why this particular one should be an exception.
>

I don't have any objection if someone implements this, but I think it
would be better to focus on other things (e.g. long lines?, making Emacs
Lisp faster, etc).

-- 
Akib Azmain Turja

Find me on Mastodon at @akib@hostux.social.

This message is signed by me with my GnuPG key.  Its fingerprint is:

    7001 8CE5 819F 17A3 BBA6  66AF E74F 0EFA 922A E7F5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Implementing image support for kitty terminal

[Tomas Hlavaty]

On Sat 10 Sep 2022 at 14:51, Po Lu <luangruo@yahoo.com> wrote:
> Tomas Hlavaty <tom@logand.com> writes:
>> Interesting.
>>
>> in no-toolkit Emacs, I get:
>>
>>    Content-type video/mp4 is unsupported
>>    Direct link to the document
>>
>> I guess it works under very special circumstances.
>
> It only works if you build with xwidgets, which requires GTK 3 and
> WebKitGTK.
>
> In addition it currently does not work on many systems due to a WebKit
> bug.

i see, thank you