image scaling

image scaling

[Stefan Monnier]

Looking at the Elisp manual's "38.16.2 Image Descriptors", it seems that
we do not support image scaling.

I'd like to add such a feature to image-mode.el, but am wondering how we
should go about doing it.  Would there be some advantage to provide it
at the C level (e.g. if some of the image libs we use already provide
the necessary functionality) or should we just bite the bullet and call
some external tool like "convert -scale ..." (with the disadvantage that
flushing the image cache will not automatically cause the image to be
re-scaled).


        Stefan

Re: image scaling

[Mathias Dahl]

>  Looking at the Elisp manual's "38.16.2 Image Descriptors", it seems that
>  we do not support image scaling.
>
>  I'd like to add such a feature to image-mode.el, but am wondering how we
>  should go about doing it.  Would there be some advantage to provide it
>  at the C level (e.g. if some of the image libs we use already provide
>  the necessary functionality) or should we just bite the bullet and call
>  some external tool like "convert -scale ..." (with the disadvantage that
>  flushing the image cache will not automatically cause the image to be
>  re-scaled).

It would be really good to have such a feature as I think it would
make browsing images in image-dired's thumbnail buffer (display some
large images there and use SPC and DEL to go back and forth to see
what I mean) much quicker. At least I guess it would be quicker to do
this natively inside Emacs than using convert to scale the image as we
do now. I assume the image libraries that Emacs use support scaling in
some way. When on the subject of images, it would be really neat if we
had built-in EXIF (et al) support as well, although I understand if
people do not think it fits well in Emacs.

Re: image scaling

[Miles Bader]

"Mathias Dahl" <mathias.dahl@gmail.com> writes:
>>  I'd like to add such a feature to image-mode.el, but am wondering how we
>>  should go about doing it.  Would there be some advantage to provide it
>>  at the C level (e.g. if some of the image libs we use already provide
>>  the necessary functionality) or should we just bite the bullet and call
>>  some external tool like "convert -scale ..." (with the disadvantage that
>>  flushing the image cache will not automatically cause the image to be
>>  re-scaled).
>
> It would be really good to have such a feature as I think it would
> make browsing images in image-dired's thumbnail buffer (display some
> large images there and use SPC and DEL to go back and forth to see
> what I mean) much quicker. At least I guess it would be quicker to do
> this natively inside Emacs than using convert to scale the image as we
> do now.

In addition, I think it's a bit dodgy to rely on specific programs --
"convert", for instance is part of imagemagick, which is a
humongous/bloated package, so I don't have it installed on my home
system (even though it's a GNU/Linux system, where you might otherwise
expect to find it installed).  I imagine it's simply not installed at
all on many non-GNU systems, or hard to find even when it is installed.

Scaling images isn't all that hard, even if you try to do a good job
(using a filter to avoid aliasing etc), so I think it would be fine to
do it in C code.

[We could optionally try to use library calls to do it when those
libraries will be linked into Emacs anyway, e.g., gtk-related libs.]

-Miles

-- 
The key to happiness
 is having dreams.      [from a fortune cookie]

Re: image scaling

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Looking at the Elisp manual's "38.16.2 Image Descriptors", it seems that
> we do not support image scaling.
>
> I'd like to add such a feature to image-mode.el, but am wondering how we
> should go about doing it.  Would there be some advantage to provide it
> at the C level (e.g. if some of the image libs we use already provide
> the necessary functionality) or should we just bite the bullet and call
> some external tool like "convert -scale ..." (with the disadvantage that
> flushing the image cache will not automatically cause the image to be
> re-scaled).

I think we should refactor image support somewhat: most image libraries
offer some way of loading/displaying/scaling/gamma-correcting whole
images (and where they don't, things like gdk might come in).  Yet our
code can't make use of that.  There is a reason we need an "image cache"
even though the normal image library functions would be quite fast
enough to cope with stuff on the fly.

Without refactoring the API, adding hand-spun scaling support will
further complicate future refactoring.

-- 
David Kastrup

Re: image scaling

[joakim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Looking at the Elisp manual's "38.16.2 Image Descriptors", it seems that
> we do not support image scaling.

> I'd like to add such a feature to image-mode.el, but am wondering how we
> should go about doing it.  Would there be some advantage to provide it
> at the C level (e.g. if some of the image libs we use already provide
> the necessary functionality) or should we just bite the bullet and call
> some external tool like "convert -scale ..." (with the disadvantage that
> flushing the image cache will not automatically cause the image to be
> re-scaled).

I've looked at this briefly while preparing the SVG patch for inclusion.

I think it would be apropriate to do this at the C level, but what would
the Lisp level interface look like?

>
>
>         Stefan
>
-- 
Joakim Verona

Re: image scaling

[David O'Toole]

I would like to suggest basic image-composition as well. I wrote a
graphical roguelike engine in Emacs that uses the external "convert"
program to resize and compose (i.e. overlay on top of one another)
tiny graphical tiles that represent the player, monsters, walls,
objects to be picked up, et cetera.

This works fine with Imagemagick, but it would be wonderful to be able
to compose these images directly within Emacs.

Some screenshots here: http://dto.mamalala.org/notebook/rlx.el

Currently won't compile due to refactoring, but the screenshots are real.

On Thu, Feb 21, 2008 at 5:06 AM,  <joakim@verona.se> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>
>
> > Looking at the Elisp manual's "38.16.2 Image Descriptors", it seems that
>  > we do not support image scaling.
>
>  > I'd like to add such a feature to image-mode.el, but am wondering how we
>  > should go about doing it.  Would there be some advantage to provide it
>  > at the C level (e.g. if some of the image libs we use already provide
>  > the necessary functionality) or should we just bite the bullet and call
>  > some external tool like "convert -scale ..." (with the disadvantage that
>  > flushing the image cache will not automatically cause the image to be
>  > re-scaled).
>
>  I've looked at this briefly while preparing the SVG patch for inclusion.
>
>  I think it would be apropriate to do this at the C level, but what would
>  the Lisp level interface look like?
>
>  >
>  >
>  >         Stefan
>  >
>  --
>  Joakim Verona
>
>
>

Re: image scaling

[Stefan Monnier]

> It would be really good to have such a feature as I think it would
> make browsing images in image-dired's thumbnail buffer (display some
> large images there and use SPC and DEL to go back and forth to see
> what I mean) much quicker.

Hmm... Seeing how other programs that deal with thumbnails also do
"convert to a new file", it looks like that example isn't very
compelling: it would require not just the ability to scale an image, but
also to then save (i.e. re-encode, compressed in a format like png or
jpeg) it.  Running `convert' doesn't sound that bad of an idea in this
case (especially you can run several `convert' processes at the same
time, benefitting from multi-cores).

> At least I guess it would be quicker to do this natively inside Emacs
> than using convert to scale the image as we do now.

Could be, but I'm not sure.


        Stefan

Re: image scaling

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Looking at the Elisp manual's "38.16.2 Image Descriptors", it seems that
> we do not support image scaling.
>
> I'd like to add such a feature to image-mode.el, but am wondering how we
> should go about doing it.  Would there be some advantage to provide it
> at the C level (e.g. if some of the image libs we use already provide
> the necessary functionality) or should we just bite the bullet and call
> some external tool like "convert -scale ..." (with the disadvantage that
> flushing the image cache will not automatically cause the image to be
> re-scaled).

I think the cleanest (but not necessarily easiest) approach is to add
a C level function to rescale an image.  This C function would act on
the internal pixmap structures and implement its own image scaling
algorithm.  The trouble with using external library functions is that
not all of them support rescaling.

Then we could simply add an extra "scale" argument to create-image
(and the underlying image descriptors), which, when non-nil, says to
scale the image by that amount.

Re: image scaling

[Mathias Dahl]

> > It would be really good to have such a feature as I think it would
>  > make browsing images in image-dired's thumbnail buffer (display some
>  > large images there and use SPC and DEL to go back and forth to see
>  > what I mean) much quicker.
>
>  Hmm... Seeing how other programs that deal with thumbnails also do
>  "convert to a new file", it looks like that example isn't very
>  compelling:

I think you misunderstand what I meant; I wasn't talking about making
the thumbnails, I was talking about the scaled (to fit window) image
used when browsing images in the thumbnail buffer. Here is a recipe:

 $ emacs -Q

Then (output from view-lossage):

M-x i m a g e SPC d i r e d <return> g f x <tab> p i c s <return> SPC
SPC SPC ...

Each time you do SPC or DEL in the thumbnail buffer a scaled version
of the original image is created and displayed.

Just so we talk about the same thing.

Re: image scaling

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> Looking at the Elisp manual's "38.16.2 Image Descriptors", it seems that
>> we do not support image scaling.
>>
>> I'd like to add such a feature to image-mode.el, but am wondering how we
>> should go about doing it.  Would there be some advantage to provide it
>> at the C level (e.g. if some of the image libs we use already provide
>> the necessary functionality) or should we just bite the bullet and call
>> some external tool like "convert -scale ..." (with the disadvantage that
>> flushing the image cache will not automatically cause the image to be
>> re-scaled).
>
> I think the cleanest (but not necessarily easiest) approach is to add
> a C level function to rescale an image.  This C function would act on
> the internal pixmap structures and implement its own image scaling
> algorithm.  The trouble with using external library functions is that
> not all of them support rescaling.

Since lot more people concerned about graphics work on those library
functions than on Emacs' graphics, we should be factoring Emacs' image
code in a manner that will allow us to make use of that functionality if
it is given (and possibly be able to use gdk if it is available).
Really: how pixels are passed one by one through Emacs before reaching
the display is a mess.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: image scaling

[Miles Bader]

David Kastrup <dak@gnu.org> writes:
> Really: how pixels are passed one by one through Emacs before reaching
> the display is a mess.

Yup.  It would be nice to see large swathes of the Emacs image support
code hidden behind an abstract interface that lets us simply hand off
90% of it when linked against, say, GTK libs.

-Miles

-- 
"1971 pickup truck; will trade for guns"

Re: image scaling

[Richard Stallman]

    I'd like to add such a feature to image-mode.el, but am wondering how we
    should go about doing it.  Would there be some advantage to provide it
    at the C level (e.g. if some of the image libs we use already provide
    the necessary functionality)

It would be far better for users to do it at the C level
because that way changing the size would work instantly.

Re: image scaling

[Chong Yidong]

David Kastrup <dak@gnu.org> writes:

>> I think the cleanest (but not necessarily easiest) approach is to add
>> a C level function to rescale an image.  This C function would act on
>> the internal pixmap structures and implement its own image scaling
>> algorithm.  The trouble with using external library functions is that
>> not all of them support rescaling.
>
> Since lot more people concerned about graphics work on those library
> functions than on Emacs' graphics, we should be factoring Emacs' image
> code in a manner that will allow us to make use of that functionality if
> it is given (and possibly be able to use gdk if it is available).
> Really: how pixels are passed one by one through Emacs before reaching
> the display is a mess.

My main concern with outsourcing particular bits of the Emacs image
functionality to external libraries (which may or may not be linked
in) is how this will look at the Lisp level.  This problem is already
present to a small degree; for example, the `image-type-available-p'
function checks if a particular image format can be displayed.  But it
would be a mess if we force users (Lisp coders) to perform a similar
test before performing things like image rescaling---or, worse still,
having such functionality quietly disappear, by functions not being
defined or becoming no-ops.

Of course, we could take the approach of writing our own "fallback"
code for when the libraries aren't available, but that negates the
point of using the libraries in the first place.

As for refactoring the Emacs image code, I'm not sure if there's any
urgency for such a project.  The code isn't elegant, but it's simple
enough to understand and work on; and I don't know of any performance
problem in the way Emacs handles image pixmaps.

Re: image scaling

[Jan Djärv]

Miles Bader skrev:
> David Kastrup <dak@gnu.org> writes:
>> Really: how pixels are passed one by one through Emacs before reaching
>> the display is a mess.
> 
> Yup.  It would be nice to see large swathes of the Emacs image support
> code hidden behind an abstract interface that lets us simply hand off
> 90% of it when linked against, say, GTK libs.
> 

I tried this and it works quite well.  However, the image handling that uses
bare X calls only caters for one display, so it is not usable for Emacs.

	Jan D.

Re: image scaling

[Miles Bader]

Jan Djärv <jan.h.d@swipnet.se> writes:
> I tried this and it works quite well.  However, the image handling that uses
> bare X calls only caters for one display, so it is not usable for Emacs.

Couldn't we just use cairo?  That seems fairly well-designed to be be
independent of the actual display...

-Miles

-- 
`There are more things in heaven and earth, Horatio,
 Than are dreamt of in your philosophy.'

Re: image scaling

[Jan Djärv]

Miles Bader skrev:
> Jan Djärv <jan.h.d@swipnet.se> writes:
>> I tried this and it works quite well.  However, the image handling that uses
>> bare X calls only caters for one display, so it is not usable for Emacs.
> 
> Couldn't we just use cairo?  That seems fairly well-designed to be be
> independent of the actual display...
> 

Sure, it would help printing also (we could use the Gtk+ print API).  But it
is a bigger effort.

	Jan D.

Re: image scaling

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> Of course, we could take the approach of writing our own "fallback"
> code for when the libraries aren't available, but that negates the
> point of using the libraries in the first place.

Not if the fallback code sucks.  Like pretty much all of Emacs'
image-handling code does now.

> As for refactoring the Emacs image code, I'm not sure if there's any
> urgency for such a project.  The code isn't elegant,

The understatement of the year.  Have you followed the code path for
displaying the various pixels of an image?  Please do so if you have not
done so yet, and then repeat that statement with a straight face.

> but it's simple enough to understand and work on; and I don't know of
> any performance problem in the way Emacs handles image pixmaps.

You must be kidding.  We have an image cache exactly because Emacs can't
handle images close to reasonably efficiently.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: image scaling

[Juanma Barranquero]

On Fri, Feb 22, 2008 at 5:57 AM, Chong Yidong <cyd@stupidchicken.com> wrote:

> for example, the `image-type-available-p'
>  function checks if a particular image format can be displayed.  But it
>  would be a mess if we force users (Lisp coders) to perform a similar
>  test before performing things like image rescaling

Why would Emacs do rescaling of images that can not be displayed in
the running instance of Emacs?

 Juanma

Re: image scaling

[Thien-Thi Nguyen]

() "Juanma Barranquero" <lekktu@gmail.com>
() Fri, 22 Feb 2008 11:23:41 +0100

   Why would Emacs do rescaling of images that can not
   be displayed in the running instance of Emacs?

Maybe the scaling is part of a series of operations,
the end of which would be displayable but not the middle.

thi

Re: image scaling

[Juanma Barranquero]

On Fri, Feb 22, 2008 at 11:55 AM, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:

>  Maybe the scaling is part of a series of operations,
>  the end of which would be displayable but not the middle.

I'm not sure I understand. A series of operations (presumably, image
transformations of some sort) without the image library support? We do
that today, manipulating for example PNG images without (statically or
dynamically) linking libpng?

 Juanma

Re: image scaling

[Chong Yidong]

David Kastrup <dak@gnu.org> writes:

>> but it's simple enough to understand and work on; and I don't know of
>> any performance problem in the way Emacs handles image pixmaps.
>
> You must be kidding.  We have an image cache exactly because Emacs can't
> handle images close to reasonably efficiently.

Don't other programs use caches for images?

Re: image scaling

[Chong Yidong]

"Juanma Barranquero" <lekktu@gmail.com> writes:

>>  Maybe the scaling is part of a series of operations,
>>  the end of which would be displayable but not the middle.
>
> I'm not sure I understand. A series of operations (presumably, image
> transformations of some sort) without the image library support? We do
> that today, manipulating for example PNG images without (statically or
> dynamically) linking libpng?

Suppose, for example, we "outsource" image transformations to a
specific external library.  For the sake of argument, let's say this
library is libpng.

One way to do this is to have Emacs use libpng internally to store and
manipulate (e.g. rescale) images, instead of the pixel buffer system
we currently use.  Alternatively, Emacs might keep the pixel buffer
system, but perform image manipulations by converting the image into a
png internally, calling the relevant libpng function to rescale it or
whatever, and then converting it back.

We would thus be using libpng to perform image manipulations, for
arbitrary images and not just pngs.  The trouble is that if libpng is
not available, all this functionality would disappear (not just for
pngs, but also for gifs or whatever other images *are* supported).  In
that case, how would we handle this at the Lisp interface level?

Re: image scaling

[Juanma Barranquero]

On Fri, Feb 22, 2008 at 9:38 PM, Chong Yidong <cyd@stupidchicken.com> wrote:

>  We would thus be using libpng to perform image manipulations, for
>  arbitrary images and not just pngs.  The trouble is that if libpng is
>  not available, all this functionality would disappear (not just for
>  pngs, but also for gifs or whatever other images *are* supported).  In
>  that case, how would we handle this at the Lisp interface level?

Well, what would be the motivation to use the libraries? Because if it
is speed, you can define a list of libraries, sorted by functionality
(speed, or whatever), and if no library is available, you default to
an Emacs-internal implementation...

             Juanma

Re: image scaling

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>>> but it's simple enough to understand and work on; and I don't know of
>>> any performance problem in the way Emacs handles image pixmaps.
>>
>> You must be kidding.  We have an image cache exactly because Emacs can't
>> handle images close to reasonably efficiently.
>
> Don't other programs use caches for images?

XEmacs, for example, does not have the equivalent, and works fine
(actually quite faster than Emacs) when using preview-latex.

The point is that when the image data is loaded into memory, Emacs still
can't display it fast.  It converts it with a painfully slow conversion
chain called for every pixel into a pixmap and caches that.  Considering
that the binary data is pretty much identical a pixmap to start with
(maybe modulo a gamma correction lookup table), this is somewhat
ridiculous.  The code paths are so muddled, however, that it is not easy
to rip this out.

Have you actually looked at the code?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: image scaling

[Miles Bader]

Chong Yidong <cyd@stupidchicken.com> writes:
> We would thus be using libpng to perform image manipulations, for
> arbitrary images and not just pngs.  The trouble is that if libpng is
> not available, all this functionality would disappear (not just for
> pngs, but also for gifs or whatever other images *are* supported).  In
> that case, how would we handle this at the Lisp interface level?

I dunno about other people, but my thought was simply to make the
interface abstract enough that we could easily use a thin wrapper around
an external library for image handling, and keep a "portable" version
(mostly, the current code) for cases when such is not available.

-Miles

-- 
Quack, n. A murderer without a license.

Re: image scaling

[Chong Yidong]

David Kastrup <dak@gnu.org> writes:

>>>> but it's simple enough to understand and work on; and I don't know of
>>>> any performance problem in the way Emacs handles image pixmaps.
>>>
>>> You must be kidding.  We have an image cache exactly because Emacs can't
>>> handle images close to reasonably efficiently.
>>
>> Don't other programs use caches for images?
>
> XEmacs, for example, does not have the equivalent, and works fine
> (actually quite faster than Emacs) when using preview-latex.
>
> The point is that when the image data is loaded into memory, Emacs still
> can't display it fast.  It converts it with a painfully slow conversion
> chain called for every pixel into a pixmap and caches that.  Considering
> that the binary data is pretty much identical a pixmap to start with
> (maybe modulo a gamma correction lookup table), this is somewhat
> ridiculous.  The code paths are so muddled, however, that it is not easy
> to rip this out.
>
> Have you actually looked at the code?

I'm fairly familiar with the code.

If Emacs can't quickly display images that are already loaded into
memory, it's not clear to me that the problem lies in the image.c code
rather than redisplay.  Could you elucidate?  When you say the image
is already "loaded into memory", do you mean that the image is
supplied using a Lisp string representing binary data (i.e., passing a
DATA-P argument to create-image), or are you simply referring to time
Emacs takes to run the png_load function (i.e., from calling the
libpng functions parsing the file until Emacs finishes populating and
caching the pixmaps)?

Re: image scaling

[David Kastrup]

Chong Yidong <cyd@stupidchicken.com> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> The point is that when the image data is loaded into memory, Emacs
>> still can't display it fast.  It converts it with a painfully slow
>> conversion chain called for every pixel into a pixmap and caches
>> that.  Considering that the binary data is pretty much identical a
>> pixmap to start with (maybe modulo a gamma correction lookup table),
>> this is somewhat ridiculous.  The code paths are so muddled, however,
>> that it is not easy to rip this out.
>>
>> Have you actually looked at the code?
>
> I'm fairly familiar with the code.

It's been some time since I looked at it, and it was pretty inscrutable.
I doubt it has improved much.

> If Emacs can't quickly display images that are already loaded into
> memory, it's not clear to me that the problem lies in the image.c code
> rather than redisplay.

Oh, I was not trying to assign blame to a particular piece of code.  It
is just what happens in totality is pretty ugly, almost to the point of
being absurd.  It may well be that each part, in isolation, does things
as good as can be expected, given the designed APIs and pathways.

> Could you elucidate?  When you say the image is already "loaded into
> memory", do you mean that the image is supplied using a Lisp string
> representing binary data (i.e., passing a DATA-P argument to
> create-image),

For example.

> or are you simply referring to time Emacs takes to run the png_load
> function (i.e., from calling the libpng functions parsing the file
> until Emacs finishes populating and caching the pixmaps)?

It has been some time since I last looked at the code and finally gave
up in disgust (twisty maze of passages...).  So I can't be much more
specific.  The gist is that Emacs is orders of magnitude slower than
programs which basically just tie libraries together without serious
data manipulation of their own.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: image scaling

[Richard Stallman]

    My main concern with outsourcing particular bits of the Emacs image
    functionality to external libraries (which may or may not be linked
    in) is how this will look at the Lisp level.

The natural and convenient way is to put something in the text
properties that specify scaling, etc.  That makes it very easy
for Lisp programs to specify scaling.

      But it
    would be a mess if we force users (Lisp coders) to perform a similar
    test before performing things like image rescaling---or, worse still,
    having such functionality quietly disappear, by functions not being
    defined or becoming no-ops.

The tremendous advantage of using library facilities for this
outweighs that disadvantage.

We should not write our own "fallback" code, because that is
duplication of effort.  Let's use the libraries to save our time so we
can make some other improvement.

    As for refactoring the Emacs image code, I'm not sure if there's any
    urgency for such a project.  The code isn't elegant, but it's simple
    enough to understand and work on; and I don't know of any performance
    problem in the way Emacs handles image pixmaps.

I agree it is not urgent, but it is quite desirable.
We could release Emacs 23 without doing this, but we would very
much like someone to do it after that.

Re: image scaling

[Richard Stallman]

    I tried this and it works quite well.  However, the image handling that uses
    bare X calls only caters for one display, so it is not usable for Emacs.

We could perhaps write extensions to that GTK library
and offer it for installation.

Re: image scaling

[Richard Stallman]

    We would thus be using libpng to perform image manipulations, for
    arbitrary images and not just pngs.  The trouble is that if libpng is
    not available, all this functionality would disappear (not just for
    pngs, but also for gifs or whatever other images *are* supported).

It is ok if libpng is required.  We need not bend over backwards to
support its absence.  Anyone can install it, after all.

Or we could make libpng required for image support.