Re: Implementing image support for kitty terminal

[Jose A Ortega Ruiz <jao@gnu.org>]

On Wed, Sep 07 2022, Eli Zaretskii wrote:

>> From: Jose Antonio Ortega Ruiz <jao@gnu.org>
>> Date: Wed, 07 Sep 2022 16:50:08 +0100
>> 
>> The kitty terminal emulator (which runs under X11 and wayland) offers a
>> simple protocol for displaying images, fully described at
>> <https://sw.kovidgoyal.net/kitty/graphics-protocol/>.  
>> 
>> In a nutshell, it accepts an escape sequence that make it enter "graphic
>> mode", followed by either encoded image data or a path to an image file
>> or a shared memory location to display.  Among several other niceties,
>> the protocol allows drawing to rectangles specified in cell units. As a
>> simple example, the sequence:
>> 
>>    <ESC>_Gf=100,t=f,c=50,r=100;<encoded /path/to/file.png><ESC>\
>> 
>> would make kitty draw the image in file.png rescaling it to 50 columns
>> and 100 rows.  By default, the current cursor position is used, but it's
>> also possible to specify pixel offsets and sizes.
>> 
>> At first sight, it looks as if adding support for this protocol to
>> emacs's tty terminal (when kitty, or the capability (it seems other
>> terminals support the same protocol) is detected) shouldn't be too
>> complex, and with that, perhaps, provide direct support for the
>> elisp-level image- API for these terminals (so that, for instance,
>> doc-view or pdf-tools or displaying images in eww buffers would work out
>> of the box).  Am i wrong?
>
> It's hard to tell, because you haven't described your ideas of
> implementing that.  In particular, the current model of image display
> in the Emacs display engine is that an image is basically considered a
> single very large character, from the screen layout POV.

Would it be possible to keep that model and only alter it when the
display actually draws the character in the underlying terminal?  

For an image-enhanced tty that'd mean that when we are going to write
this "single very large character" (whose data can be either an array of
bytes or simply a path in the filesystem, stored in an associated text
property perhaps) what we do is write a ESC character sequence starting
with _G, as the one shown above, and specifying not only the data to
draw, but also its size in rows and columns.  Then, according to the
kitty API promises, the image will be rendered in that rectangle for us.

I am assuming here that the display engine (for a tty) knows somehow
that it's printing a very lage "character" that is going to spawn
several rows and columns, and also that we can always tell how wide and
tall the window around point is when we display this "character".  We
could then decide to tell kitty to either scale the image if it doesn't
fit or display only part of it.  That's done by passing x,y,w,h in the
escape sequence instead or r,c... something like

      <ESC>_Gf=100,t=f,x=0,y=0,w=100,h=20;<encoded /path/to/file.png><ESC>\

As you can see, the API offered is quite powerful and would do a lot of
work for us.  It's even able to cache images and reuse them: according
to their documentation, every transmitted image can be displayed an
arbitrary number of times on the screen, in different locations, using
different parts of the source image, as needed.

As the above ideas most probably demonstrate, i don't know anything
concrete about how emacs works at this level, so apologies in advance if
i'm not making sense.

Thanks,
jao
-- 
They are ill discoverers that think there is no land, when they can
see nothing but sea. -Francis Bacon, philosopher (1561-1626)