Why not zlib-compress-region?

Why not zlib-compress-region?

[Leo Liu]

Would be nice to have this as well. Any objection to adding it? Some
specs allow zlib-compressed data and this function can be very handy.

Thanks,
Leo

Re: Why not zlib-compress-region?

[Dmitry Antipov]

On 06/26/2014 01:20 PM, Leo Liu wrote:

> Would be nice to have this as well. Any objection to adding it? Some
> specs allow zlib-compressed data and this function can be very handy.

Some time ago I proposed full zlib/bz2/xz (de)compression support.
IIRC Stefan has rejected it to avoid creeping featurism and extra
external library dependencies; see
https://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00564.html.

Dmitry

Re: Why not zlib-compress-region?

[Stefan Monnier]

>> Would be nice to have this as well. Any objection to adding it? Some
>> specs allow zlib-compressed data and this function can be very handy.
> Some time ago I proposed full zlib/bz2/xz (de)compression support.
> IIRC Stefan has rejected it to avoid creeping featurism and extra
> external library dependencies; see
> https://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00564.html.

An FFI could make this a non-issue ;-)


        Stefan

Re: Why not zlib-compress-region?

[Leo Liu]

On 2014-06-26 09:27 -0400, Stefan Monnier wrote:
> An FFI could make this a non-issue ;-)
>
>
>         Stefan

Indeed. But it would be nice to make the API symmetric for zlib which
incurs no extra external dependencies.

BTW, what is required to get this FFI done? Anyone working on it?

Leo

Re: Why not zlib-compress-region?

[Stefan Monnier]

> BTW, what is required to get this FFI done? Anyone working on it?

Exactly: what is required is "anyone working on it".


        Stefan

Re: Why not zlib-compress-region?

[Aurélien Aptel]

On Thu, Jun 26, 2014 at 6:43 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> Exactly: what is required is "anyone working on it".

Christopher Wellons had made some progress:

http://nullprogram.com/blog/2014/04/26/

Re: Why not zlib-compress-region?

[Stefan Monnier]

>> Exactly: what is required is "anyone working on it".
> Christopher Wellons had made some progress:
> http://nullprogram.com/blog/2014/04/26/

I didn't know this one.
Of course, there's Dave Love's
http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

I think Dave's work is a good starting point.  It needs a GPL-check
(i.e. it needs to check the dyn-loaded code to see if it has a special
symbol stating that it's gpl-compatible), and it needs to be updated for
current Emacs code, but it should only touch code that hasn't
fundamentally changed very much (famous last word).


        Stefan

Re: Why not zlib-compress-region?

[Dmitry Antipov]

On 06/27/2014 05:28 PM, Stefan Monnier wrote:

> Of course, there's Dave Love's
> http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

IIUC this is a completely different thing. Dave's approach is to
dlopen() specially designed shared objects to add pre-defined
DEFVARs and DEFUNs; FFI is about _generating_ DEFUNs "on the fly"
from _any_ (well, almost any) shared object.

Dmitry

Re: Why not zlib-compress-region?

[Ted Zlatanov]

On Thu, 26 Jun 2014 12:43:54 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> BTW, what is required to get this FFI done? Anyone working on it?
SM> Exactly: what is required is "anyone working on it".

I haven't been able to give it time but saw the two approaches (Dave
Love and Christopher Wellons).

Ted

Re: Why not zlib-compress-region?

[Stefan Monnier]

>> Of course, there's Dave Love's
>> http://www.loveshack.ukfsn.org/emacs/dynamic-loading/
> IIUC this is a completely different thing. Dave's approach is to
> dlopen() specially designed shared objects to add pre-defined
> DEFVARs and DEFUNs; FFI is about _generating_ DEFUNs "on the fly"
> from _any_ (well, almost any) shared object.

It's not that different.  In either case, you need some glue which
describes how to construct the arguments to pass to those dl-loaded
functions and what to do with their return value.

In the case of Dave's approach, this glue is written in C.  In more
modern FFIs (or in Christopher Wellons's approach), this is written in
some DSL.

Writing the glue in C has its advantages: more people know C than know
the binding-DSL, and C lets you do "anything" whereas the typical
binding DSL always has various limitations as to which kinds of objects
it supports, etc...

Of course, writing the glue in C means it needs to be passed to
a C compiler, which means that the user who wants to use a package that
uses such glue will need to have a C compiler installed.  A non issue
under GNU/Linux, but a point of friction under W32.


        Stefan

Re: Why not zlib-compress-region?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 27 Jun 2014 18:07:38 -0400
> Cc: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>,
> 	Leo Liu <sdl.web@gmail.com>,
> 	Emacs development discussions <emacs-devel@gnu.org>
> 
> Of course, writing the glue in C means it needs to be passed to
> a C compiler, which means that the user who wants to use a package that
> uses such glue will need to have a C compiler installed.  A non issue
> under GNU/Linux, but a point of friction under W32.

Isn't it true that lately most Posix systems, including GNU/Linux, are
by default packaged without a development environment?

Re: Why not zlib-compress-region?

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Isn't it true that lately most Posix systems, including GNU/Linux, are
 > by default packaged without a development environment?

Yes, but GNU/Linux package managers (among other free software package
managers) generally include development dependencies in their
databases, so anybody who is not in a high-security situation (where
management has a short list of libraries you can use) can fairly
easily acquire the necessary dependencies, usually without going
through DLL hell (still building your own everything, right, Eli?)

You occasionally run into annoying CADT bigotry (such as the common
omission of the bitmap-dev package from X11 dev environments), but
at least our C dev environments are generally fairly interoperable
(unlike Windows where there are three mutually mostly incompatible
GCCs).

There are minor annoyances, but not so much as to deter people who
aren't afraid to type "./configure; make; make install".  I've given
up more times than I can count (temporarily of course, when I've got a
wekk of evenings free I get back to it) on putting together Windows
dev environments, though.

Re: Why not zlib-compress-region?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Sat, 28 Jun 2014 21:51:46 +0900
> Cc: aurelien.aptel+emacs@gmail.com, dmantipov@yandex.ru, emacs-devel@gnu.org,
> 	Stefan Monnier <monnier@iro.umontreal.ca>, sdl.web@gmail.com
> 
> Eli Zaretskii writes:
> 
>  > Isn't it true that lately most Posix systems, including GNU/Linux, are
>  > by default packaged without a development environment?
> 
> Yes, but GNU/Linux package managers (among other free software package
> managers) generally include development dependencies in their
> databases, so anybody who is not in a high-security situation (where
> management has a short list of libraries you can use) can fairly
> easily acquire the necessary dependencies

I know about the package managers, I just wanted to make a point that
AFAIU end-user Posix systems might well lack a compiler.  AFAIR, on
non-free Posix systems, installing a compiler actually costs money.
So having an FFI implementation that requires a compiler might be a
disadvantage on Posix systems as well.

> usually without going through DLL hell

(This is unrelated.)  I don't believe in package managers as a means
to avoid the "DLL hell".  Dependencies are written by people, which
are prone to errors, and having several libFOO.so versions on the same
system, even if their names don't conflict, is not fun.

> (still building your own everything, right, Eli?)

Not everything, just things no one built, or built in a broken way.
The compiler and Binutils are definitely not my build.

> There are minor annoyances, but not so much as to deter people who
> aren't afraid to type "./configure; make; make install".

We are talking about the compiler and Binutils.  Building those,
especially the former, is not for the faint at heart, not even on a
Posix host.

> I've given up more times than I can count (temporarily of course,
> when I've got a wekk of evenings free I get back to it) on putting
> together Windows dev environments, though.

It's not that hard, actually, especially if there's no need to run the
configure scripts (as I'd imagine will happen when Emacs supports some
kind of FFI).

Re: Why not zlib-compress-region?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    Isn't it true that lately most Posix systems, including GNU/Linux, are
    by default packaged without a development environment?

As far as I know, most GNU/Linux distros include C development tools.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: Why not zlib-compress-region?

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > I know about the package managers, I just wanted to make a point that
 > AFAIU end-user Posix systems might well lack a compiler.  AFAIR, on
 > non-free Posix systems, installing a compiler actually costs money.

Not on Mac OS X.  And how many important non-free POSIX systems aren't
supported by GCC (not to mention The-Compiler-Suite-That-Shall-Not-Be-
Mentioned-On-GNU-Channels)?

The real problem for those is more likely whether there is an
enterprise requirement for vetting all installed software.

 > So having an FFI implementation that requires a compiler might be a
 > disadvantage on Posix systems as well.

Anything's possible.  I don't think it's worth worrying about it.
XEmacs/SXEmacs have *never* received complaints about availability of
compilers for our version of "Stefan-style FFI".  It's always "ooh,
yuck, no GUI installer??!!  I can't do that!" or "why can't you
distribute binaries in the prebuilt packages?", not a "I have to
install a compiler" problem.

FFI implementations that *don't* require compilers have their
problems, too, since Lisp types sometimes map to different C types for
different libraries, which requires a lot of fiddly low-level
knowledge on the part of the Lisp programmers that (if they stick to
pure Lisp) they just don't need at all.  FFI == crashable Lisp.

 > > usually without going through DLL hell
 > 
 > (This is unrelated.)  I don't believe in package managers as a
 > means to avoid the "DLL hell".  Dependencies are written by people,
 > which are prone to errors, and having several libFOO.so versions on
 > the same system, even if their names don't conflict, is not fun.

I have a bunch of them (three versions of GTK, two of libpng for
example).  I don't notice it at all, the PMS installs the right
dependencies for everything.  (Not to mention simultaneous installs of
32- and 64-bit versions of many libraries.)  OTOH, I only have one
instance of each version level (modulo the 32-bit issue), because the
distros provide extremely good coverage and generally do a good job of
sorting incompatibilities (occasionally in the opposite way from my
preference, but them's the breaks).  So the real point is not the
PMS's dependency database, but rather than fact that each distro has a
(more or less) canonical compiler suite, and you just install that and
it pulls everything else in.

 > We are talking about the compiler and Binutils.  Building those,
 > especially the former, is not for the faint at heart, not even on a
 > Posix host.

Nonsense.  I do it about once a month, sometimes twice a week
(automatically via Gentoo's Portage PMS, which always builds from
source).  Unlike, say, Boost or KDE's nepomuk (which are always
holding up the upgrade), I can't remember well the last time a GCC (or
TCSTSNBMOGC, see above) build failed on me (except for out of space
errors), and binutils failure is even more rare.

It might be harder for non-free POSIX systems, and then it might not.
Many have open-source repositories with PMSes of some sort.

 > > I've given up more times than I can count (temporarily of course,
 > > when I've got a wekk of evenings free I get back to it) on putting
 > > together Windows dev environments, though.
 > 
 > It's not that hard, actually, especially if there's no need to run the
 > configure scripts (as I'd imagine will happen when Emacs supports some
 > kind of FFI).

No, it probably isn't, but since I do it less than once a year, and
once done, I use it once then forget it, it induces fear and
loathing. :-)

Re: Why not zlib-compress-region?

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: aurelien.aptel+emacs@gmail.com,
>     sdl.web@gmail.com,
>     dmantipov@yandex.ru,
>     monnier@iro.umontreal.ca,
>     emacs-devel@gnu.org
> Date: Sun, 29 Jun 2014 01:30:51 +0900
> 
> Eli Zaretskii writes:
> 
>  > I know about the package managers, I just wanted to make a point that
>  > AFAIU end-user Posix systems might well lack a compiler.  AFAIR, on
>  > non-free Posix systems, installing a compiler actually costs money.
> 
> Not on Mac OS X.  And how many important non-free POSIX systems aren't
> supported by GCC (not to mention The-Compiler-Suite-That-Shall-Not-Be-
> Mentioned-On-GNU-Channels)?

Again, building GCC is not something an end user would easily consider
when all she needs is to be able to use some plugin.

> The real problem for those is more likely whether there is an
> enterprise requirement for vetting all installed software.

Yes, that too.

> FFI implementations that *don't* require compilers have their
> problems, too, since Lisp types sometimes map to different C types for
> different libraries, which requires a lot of fiddly low-level
> knowledge on the part of the Lisp programmers that (if they stick to
> pure Lisp) they just don't need at all.  FFI == crashable Lisp.

Something to consider, I'm sure.  But that doesn't make the problem of
having to have a working compiler installation in any way.

>  > > usually without going through DLL hell
>  > 
>  > (This is unrelated.)  I don't believe in package managers as a
>  > means to avoid the "DLL hell".  Dependencies are written by people,
>  > which are prone to errors, and having several libFOO.so versions on
>  > the same system, even if their names don't conflict, is not fun.
> 
> I have a bunch of them (three versions of GTK, two of libpng for
> example).  I don't notice it at all

I have more than "a bunch" of them, too.  This is not about you or me,
you know.

>  > We are talking about the compiler and Binutils.  Building those,
>  > especially the former, is not for the faint at heart, not even on a
>  > Posix host.
> 
> Nonsense.  I do it about once a month, sometimes twice a week
> (automatically via Gentoo's Portage PMS, which always builds from
> source).

Again, this is not about you or me.

Re: Why not zlib-compress-region?

[Eli Zaretskii]

> Date: Sat, 28 Jun 2014 20:00:34 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: aurelien.aptel+emacs@gmail.com, dmantipov@yandex.ru, emacs-devel@gnu.org,
> 	sdl.web@gmail.com, monnier@iro.umontreal.ca
> 
> > FFI implementations that *don't* require compilers have their
> > problems, too, since Lisp types sometimes map to different C types for
> > different libraries, which requires a lot of fiddly low-level
> > knowledge on the part of the Lisp programmers that (if they stick to
> > pure Lisp) they just don't need at all.  FFI == crashable Lisp.
> 
> Something to consider, I'm sure.  But that doesn't make the problem of
> having to have a working compiler installation in any way.

The last sentence was meant to say

  But that doesn't make the problem of having to have a working
  compiler installation go away in any way.

Sorry.

Re: Why not zlib-compress-region?

[Paul Eggert]

Stephen J. Turnbull wrote:
>   > So having an FFI implementation that requires a compiler might be a
>   > disadvantage on Posix systems as well.
>
> Anything's possible.  I don't think it's worth worrying about it.
> XEmacs/SXEmacs have*never*  received complaints about availability of
> compilers for our version of "Stefan-style FFI".

My experience, such as it is, mirrors yours.  Availability of compilers 
is not a significant problem on GNUish and POSIXish systems.

It'd be reasonable to implement "Stefan-style FFI" assuming the presence 
of a compiler, and then worry about systems lacking compilers later.  We 
wouldn't be any worse off than we are now.

Re: Why not zlib-compress-region?

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Again, building GCC is not something an end user would easily consider
 > when all she needs is to be able to use some plugin.

For MacPorts and all end-user-oriented GNU/Linux distros I know of
there are binary packages for the C toolchain at least.  I know such
used to be available for AIX, HP/UX, SunOS and Solaris (which is now
OpenSolaris).

I wonder how many systems that don't distribute a C compiler in the
base distribution distribute Emacs (which last I looked was not in the
same space as Adobe Acrobat, much less *Office)?  I suspect on most
such systems to use Emacs, you have to build it yourself.

 > > I have a bunch of them (three versions of GTK, two of libpng for
 > > example).  I don't notice it at all
 > 
 > I have more than "a bunch" of them, too.  This is not about you or me,
 > you know.

No, it's not.  My point is that to the extent I have those, I *don't*
deal with them, I let the PMS do it.

True, I'm not a typical end-user, but people on systems with a PMS are
surely as used to using the PMS to install "stuff" as Windows users
are used to downloading MSIs.

 > > Nonsense.  I do it about once a month, sometimes twice a week
 > > (automatically via Gentoo's Portage PMS, which always builds from
 > > source).
 > 
 > Again, this is not about you or me.

Again, from my point of view it's all hidden by the PMS.

It's just more *stuff* to install, and since in XEmacs's system it's
all done by a specialized compiler driver (distributed and built with
XEmacs) which knows about XEmacs's configuration, Emacs could use the
same technology and it's probably as easy as installing the C
toolchain with the PMS, firing up Emacs and running

(defun ffi-install-package (package)
  (interactive "sWhat package would you like to install? ")
  (ffi-install-download-source package)
  (ffi-install-check-dependencies package)
  (shell-command (format "cd %s; ./configure %s; make; make install"
                         package
                         (ffi-install-configure-options package))))

where `ffi-install-download-source', `ffi-install-check-dependencies',
and `ffi-install-configure-options' consult a database distributed
with Emacs and/or online at ELPA.  (XEmacs doesn't actually do this
because all loadable modules currently available for XEmacs are
distributed with XEmacs.  So if you configure --with-modules they get
built as part of the default make target, and installed as part of the
"install" target.)

Sure, a few people will be left out, but AFAICT it should be possible
for Emacs to build a binary distribution for Windows users, and
pretty much everybody else is able to fend for themselves.

I'm not arguing that this is *better* than something like a libffi
binding in LISP, just that it's clearly feasible, and unlikely to
leave even a large minority of users without support.

Steve

Re: Why not zlib-compress-region?

[Stefan Monnier]

> Again, building GCC is not something an end user would easily consider
> when all she needs is to be able to use some plugin.

Indeed.   Luckily, there's no such need: the end user only needs to
*install* a compiler.  Also, for Windows we could consider binary
distributions, like we do for the emacs binary itself.

In any case, the fac that the dl-loaded library needs to include
a special "gpl-compatible" tag means that in 99% of the cases, it
wouldn't be one of the pre-existing libraries already installed on the
system.  So one way or another we'll end up having to tell people to
install a C compiler, or we'll have to distribute pre-compiled binaries.

Furthermore, a "plain FFI" like what I suggest (with bindings written in
C) is very useful even if we also have a "modern FFI" (with bindings
written in some DSL).  Rather than argue whether it's useful or not,
I wish someone could get the ball rolling (and if she wants to do it
with a "modern FFI", then it's fine by me as well, I'm only advocating
the "plain FFI" because it's much less work, so it seems like a much
more realistic goal, and some of its work would be useful for
a subsequent "modern FFI" anyway).


        Stefan