persistent lisp objects

persistent lisp objects

[Thierry Volpiatto]

Hi all,
it seem using `setf' combined with `eval-when-compile' is a convenient
way to save lisp object like hash-table.
It is not documented.
What do you think?

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France

Re: persistent lisp objects

[Thierry Volpiatto]

Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:

> Hi all,
> it seem using `setf' combined with `eval-when-compile' is a convenient
> way to save lisp object like hash-table.
> It is not documented.
> What do you think?

Maybe i was not clear enough.
We can actually save any lisp object in a compiled file:

,----
| (defun tv-dump-object-to-file (obj file)
|   "Save symbol object `obj' to the byte compiled version of `file'.
| `obj' can be any lisp object, list, hash-table, etc...
| `file' is an elisp file with ext *.el.
| Loading the *.elc file will restitute object."
|   (require 'cl) ; Be sure we use the CL version of `eval-when-compile'.
|   (if (file-exists-p file)
|       (error "File already exists.")
|       (with-temp-file file
|         (erase-buffer)
|         (let* ((str-obj (symbol-name obj))
|                (fmt-obj (format "(setq %s (eval-when-compile %s))" str-obj str-obj)))
|           (insert fmt-obj)))
|       (byte-compile-file file) (delete-file file)
|       (message "`%s' dumped to %sc" obj file)))
`----

This is a feature of `eval-when-compile' that is not documented.
It can be interesting as AFAIK there is no way to save object.

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France

Re: persistent lisp objects

[MON KEY]

Thierry Volpiatto <address@hidden> writes:
> This is a feature of `eval-when-compile' that is not documented.
> It can be interesting as AFAIK there is no way to save object.

What would are consequence for this type of output vis a vis GPL? e.g.:

,----
| In what cases is the output of a GPL program covered by the GPL too?
|
|     Only when the program copies part of itself into the output.
`----
See: (URL `http://www.gnu.org/licenses/gpl-faq.html#WhatCaseIsOutputGPL')

Likewise, for the GFDL?

This technique's ability to provide compiled persistence of symbols
would imply an ability to provide compiled persistence of the symbol's
value as well. Couldn't that have implications when those symbols
carry values for 'code writing code'?

For example, doesn't GNU Bison have "special" licensing consideration
to accommodate similar quandary?

s_P

s_P

Re: persistent lisp objects

[Stefan Monnier]

> For example, doesn't GNU Bison have "special" licensing consideration
> to accommodate similar quandary?

For Elisp, the situation is fairly different: there is pretty much no
way to write useful Elisp code that doesn't fall under the GPL, because
it essentially needs to be linked with Emacs (or XEmacs).  So it's all
GPL in any case.


        Stefan

Re: persistent lisp objects

[MON KEY]

On Fri, Oct 2, 2009 at 11:36 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> it essentially needs to be linked with Emacs (or XEmacs).  So it's all
> GPL in any case.

And w/re to the GFDL?

For example, wouldn't it be possible using this technique to assemble
byte-compiled component symbols each which of itself contains only a portion of
some documentation or documentation generating routine but which is only made
manifest at compile time by virtue of this technique?

If such is possible which license would best govern the assembly of and
subsequent aggregated output from implementation of such a technique - GPL/GFDL
or both?

I ask because these types of persistence issues appear to show up in the GFDL
faq w/re Java. I'm wondering if the pending integration of CEDET eieio might
pose similar questions in the future. For example, eieio already provides some
facility for persistent documentation. Combine that with CL style `#.' symbol
dumping, this technique could pose myriad potential mechanisms for extending the
self-documenting aspects of Emacs lisp.

I've no idea if the technique Thierry presents is possible in other modern
languages (braced or curly) but certainly this one does benefit from lisp's
syntax and I suspect that historically similar approaches were made possible
leveraging CL's CLOS. That said, CLOS doesn't (in recent times) do much double
duty as "The one true editor".  Likewise where CL and CLOS have existed in such
roles it is doubtful that the GFDL (or equivalent) was a factor of any
concern...

Apropos of this, it is worth considering that the first lines of the Emacs
manual begin:
,----
| You are reading about GNU Emacs, the GNU incarnation of the advanced,
| self-documenting, customizable, extensible editor Emacs.
`----(info "(emacs)Intro")

Which is immediately preceded by the section which begins:
,----
| The purpose of this License is to make a manual, textbook, or other
| functional and useful document "free" in the sense of freedom:
`----(info "(emacs)GNU Free Documentation License")

>        Stefan

s_P

Re: persistent lisp objects

[Richard Stallman]

The copyright on compiler output follows the copyright on the input
that was compiled.  In other words, the copyright holder of the input
also has the copyright on the output.  So if the output this produces
is a translation of some code, the copyright goes with the code that
was translated.  Otherwise I suppose it belongs to whoever decided to
produce it.

The Bison case is different and not relevant here.

Re: persistent lisp objects

[MON KEY]

On Sat, Oct 3, 2009 at 4:13 PM, Richard Stallman <rms@gnu.org> wrote:
> The copyright on compiler output follows the copyright on the input
> that was compiled.  In other words, the copyright holder of the input
> also has the copyright on the output.

So, if the output produced of the compiler originates by a method
dispatched on a
subclassed object it would be under GPL?

,----
| Subclassing is creating a derivative work. Therefore, the terms of the GPL
| affect the whole program where you create a subclass of a GPL'ed class.
`----(URL `http://www.gnu.org/licenses/gpl-faq.html#OOPLang')

What of cases where one or more methods so specialized act in concert to snarf
data from some other source(s) where these sources are extraneous to the local
handlers - e.g. the GPL'd objects/methods?

data -> object -> method -> compiled-output

If the compiled output so produced is a derivative produced under the
guise of the GPL
would this imply that that compiled output is GPL'd independent of
pre-existing licenses
which might otherwise govern that of the inputs originator(s)?

IOW, what of a gaggle of sub-classed eieio objects with methods defined to snarf
various third-party sources binding said snarfage to various instances each of
which are in turn directed to spit out portions therof on a per symbol basis to
compiled files. In the aggregate these individually snarfed and subsequently
compiled symbolic components may well represent thousands of seperate instances
of copywritten/licensed material.

Under such a regime would it be reasonable for one to claim that as
each symbolic
component could only have resulted of a method specialized on some particular
eieio sublcass that therefor this compiled output falls under the GPL?

Would it not be true that were one to investigate the compiled content of a
particular component - being merely a component of a whole compiled at the
direction of the lisp environment - that of itself these compiled contents being
otherwise divorced of their originating context would present as essentially
meaningless to other than the algorithmic environment producing the component?

By passing aggregate data through an objects methods on the way to
compilation the
individual symbolic data so compiled becomes GPL'd?

> is a translation of some code, the copyright goes with the code that
> was translated.  Otherwise I suppose it belongs to whoever decided to
> produce it.

Cool... I think.

,----
| Some templates make calls into Javascript functions. Since Javascript is often
| non-trivial, it is worth copylefting. Because the templates will be combined
| with user data, it's possible that template+user data+Javascript would be
| considered one work under copyright law. A line needs to be drawn between the
| Javascript (copylefted), and the user code (usually under incompatible terms).
`----(URL `http://www.gnu.org/licenses/gpl-faq.html#WMS')

>
> The Bison case is different and not relevant here.

OK. Thank you for clarifying.

s_P

Re: persistent lisp objects

[Richard Stallman]

    > The copyright on compiler output follows the copyright on the input
    > that was compiled.  In other words, the copyright holder of the input
    > also has the copyright on the output.

    So, if the output produced of the compiler originates by a method
    dispatched on a
    subclassed object it would be under GPL?

I do not follow you.
I am not sure what scenario you mean.

There are two questions here (and in any such case): who holds the
copyright, and whether there is any constraint on the license that the
copyright holder can release under.  What I spoke about is who holds
the copyright.

    What of cases where one or more methods so specialized act in concert to snarf
    data from some other source(s) where these sources are extraneous to the local
    handlers - e.g. the GPL'd objects/methods?

    data -> object -> method -> compiled-output

That is very abstract.  It is hard to consider these issues in the abstract.
Only a concrete scenario has all the details that affect the answer.

RE: persistent lisp objects

[Drew Adams]

> We can actually save any lisp object in a compiled file:
> 
> This is a feature of `eval-when-compile' that is not documented.
> It can be interesting as AFAIK there is no way to save object.

Now that the detour about licensing has died down, perhaps we can return to
this? ;-)

From what I understand, Thierry says that Emacs now has a general
serialization/deserialization feature, in the form of byte-compiling to a file
and then reading that file. I haven't tested it, but if true, this is great
news, and I'm surprised it isn't mentioned as such (that I can tell) in NEWS.

How about adding explicit Lisp functions that capture this feature, as such?
IOW, instead of a few people knowing about this as a nice hack that happens to
work with the (recent) byte-compiler, how about defining `(de)serialize'
functions that make use of this feature, and advertize those functions?

If the functions were standalone (even if they borrowed some of the
byte-compiler code), then they could perhaps even be used in older Emacs
versions.

IOW, how about breaking this part of the byte-compiler code out, as separate
functions (which would also be used by the byte-compiler, of course). (Dunno if
this last part makes sense - maybe it's the whole byte-compiler that would be
needed.)

Anyway, it sounds to me like Thierry's discovery, if general, could be exploited
and made more directly noticeable/usable by users.

And at the very least, how about documenting this existing feature, as Thierry
requested?

Re: persistent lisp objects

[Stefan Monnier]

> From what I understand, Thierry says that Emacs now has a general
> serialization/deserialization feature, in the form of byte-compiling
> to a file and then reading that file. I haven't tested it, but if

I have no idea what he thinks as novel, but such functions have been
available for ages.  Called print and read.


        Stefan

RE: persistent lisp objects

[Drew Adams]

> > From what I understand, Thierry says that Emacs now has a general
> > serialization/deserialization feature, in the form of byte-compiling
> > to a file and then reading that file. I haven't tested it, but if
> 
> I have no idea what he thinks as novel, but such functions have been
> available for ages.  Called print and read.

print and read are no good for objects whose print form is not readable by the
reader. That's the general problem (lack) in this regard.

I assumed he was saying that byte-compiling now did something more than that, so
that objects whose print form is not readable are neverthless handled. I didn't
try it myself.

RE: persistent lisp objects

[Stephen J. Turnbull]

Drew Adams writes:

 > print and read are no good for objects whose print form is not
 > readable by the reader. That's the general problem (lack) in this
 > regard.

Some objects (frames, for example) *cannot* have a readable print form
because they refer to objects that Emacs in principle cannot print
(GUI windows, in the case of frames).  For objects that can but don't,
the right thing to do is to file a bug against `print' and `read', not
request that an obscure hack be sanctified.

Cf. the recent creation of a read'able print representation for hash
tables.

Re: persistent lisp objects

[Thierry Volpiatto]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I have no idea what he thinks as novel, but such functions have been
> available for ages.  Called print and read.

There is nothing new.
I just show a feature of the CL version of `eval-when-compile'.
Thus since we have a new representation of hash-table in emacs-cvs, this
allow to save hash-tables to a compiled file.
But other lisp objects (e.g lists) can be saved in the same way in
inferior emacs versions.
I just provide a basic function to demonstrate this feature.(see
precedent posts)
The emacs manual say that about eval-when-compile:

,----
|  -- Special Form: eval-when-compile forms...
|      The FORMS are evaluated at compile-time; at execution time, this
|      form acts like a quoted constant of the resulting value.  Used at
|      top-level, `eval-when-compile' is just like `eval-when (compile
|      eval)'.  In other contexts, `eval-when-compile' allows code to be
|      evaluated once at compile-time for efficiency or other reasons.
| 
|      This form is similar to the `#.' syntax of true Common Lisp.
`----

Please read that about the `#.' feature of CL:

http://www.cs.cmu.edu/Groups/AI/html/faqs/lang/lisp/part5/faq-doc-5.html

That is surely not the best way to serialize objects, but that is better
than nothing until somebody write something to save hash-table, lists,
etc... in a data base.

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France

Re: persistent lisp objects

[Thierry Volpiatto]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Drew Adams writes:
>
>  > print and read are no good for objects whose print form is not
>  > readable by the reader. That's the general problem (lack) in this
>  > regard.
>
> Some objects (frames, for example) *cannot* have a readable print form
> because they refer to objects that Emacs in principle cannot print
> (GUI windows, in the case of frames).  For objects that can but don't,
> the right thing to do is to file a bug against `print' and `read', not
> request that an obscure hack be sanctified.

I never request an obscure hack to be sanctified.
I just show a feature of an emacs lisp function that is part of emacs
and referenced in emacs manual.
We don't speak here about frames etc...
We speak of saving data (e.g hash-tables, list etc...)

And notice that an obscure hack can become a feature if people are more
positive.
  
> Cf. the recent creation of a read'able print representation for hash
> tables.
>
>
>
>

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France

Re: persistent lisp objects

[Stefan Monnier]

>> I have no idea what he thinks as novel, but such functions have been
>> available for ages.  Called print and read.
[...]
> That is surely not the best way to serialize objects, but that is better
> than nothing until somebody write something to save hash-table, lists,
> etc... in a data base.

Again, what's wrong with print&read?
That's what your eval-when-compile uses.  All the eval-when-compile does
is to let the byte-compiler serialize an object into a .elc file.
If you're not limited to a .elc file, then it seems a lot simpler to
just use print&read directly.


        Stefan

Re: persistent lisp objects

[Ted Zlatanov]

On Sun, 25 Oct 2009 11:26:50 +0100 Thierry Volpiatto <thierry.volpiatto@gmail.com> wrote: 

TV> somebody write something to save hash-table, lists, etc... in a data
TV> base.

I think this is worth pursuing, and I've brought it up before.  What
persistence layer would be good so external databases can be "plugged"
in?  Ideally it should be able to handle everything from flat files and
SQLite to cloud storage (e.g. Amazon S3), user-defined code, and
traditional RDBMSs.

It should also be able to cache the data intelligently and be able to
force a cache update, like Memcached.  Does something like this exist?

I think hash tables are a good single choice for the data structures.  A
hash table tied to the persistence layer is easier to cache, iterate,
and control than generic lists.  Supporting both hash tables and lists
is extra work for little gain IMO.

Ted