>> Isn’t it possible to use the 'keymap text/overlay property on the image to do what you described? At least for zooming or
>> moving around view port, I think you can create some simple +/-/C-npfb bindings for the image and send the command to the backend.
>> Or write a lisp package for any more complex interaction.
> 
> That's probably possible to get some limited interaction, however I'm not sure it would very satisfactory or complete. For example is it possible to have a live updating rectangle displayed as one drags the mouse to select a region to inspect? Is it possible to display a slider which can be dragged by the mouse to update values on the plot or have a drop down menu to select a data series? It seems it may also be difficult to find a way to have elisp interact with the matplotlib fast enough to not introduce noticable delay as such actions update the view.

I agree that displaying a live updating rectangle/slider following the mouse is near impossible.

For selecting region, supporting two mouse click should be totally doable. For selecting
data series, read-from-minibuffer together with user’s favorite completion/fuzzy search package
should be suffice. Those are discrete event. But do we really need continuous event that
much? E.g. what useful information is exchanged between computer/user when drawing a 
rectangle following user’s mouse? It seems to me that only the starting and ending position
matters. More here http://cat-v.mit.edu/2019-10-15-mouse-considered-harmful.html <http://cat-v.mit.edu/2019-10-15-mouse-considered-harmful.html>

However, if everyone decides that every event mouse spits out matters (which is a lot),
then it is the case that image display won’t work good enough.