entry separator for printing hashtables

entry separator for printing hashtables

[Ted Zlatanov]

Since hashtables can get pretty big, it would be nice to have a
separator between entries when they are printed.  Could that be provided
at the top level, e.g.

(let ((hash-table-print-separator "\\\n"))
   (format "%S" my-hash-table))

or in some other way?  It would make large hashtable dumps like the ones
from the new registry.el and gnus-registry.el much easier to read and
debug.  In the old gnus-registry.el I had special code to break up
alists like that but it was slow so I'd rather ask Emacs to do it for
me at the C level.

Thanks
Ted

list-print-separator (was: entry separator for printing hashtables)

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

On Wed, 06 Apr 2011 05:59:48 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> Since hashtables can get pretty big, it would be nice to have a
TZ> separator between entries when they are printed.  Could that be provided
TZ> at the top level, e.g.

TZ> (let ((hash-table-print-separator "\\\n"))
TZ>    (format "%S" my-hash-table))

TZ> or in some other way?  It would make large hashtable dumps like the ones
TZ> from the new registry.el and gnus-registry.el much easier to read and
TZ> debug.  In the old gnus-registry.el I had special code to break up
TZ> alists like that but it was slow so I'd rather ask Emacs to do it for
TZ> me at the C level.

The attached patch introduces `list-print-separator' to be used for list
and hashtable entries.  It's printed unescaped.  It could be a function
too, to be called with the print level.  For me it's sufficient as a
simple object to be printed.

Let me know what you think...

Thanks
Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: list-print-separator.patch --]
[-- Type: text/x-diff, Size: 1378 bytes --]

=== modified file 'src/print.c'
--- src/print.c	2011-04-14 07:09:45 +0000
+++ src/print.c	2011-04-21 23:12:46 +0000
@@ -1658,6 +1658,10 @@
 
 		print_object (XCAR (obj), printcharfun, escapeflag);
 
+                /* Separate entries with list-print-separator.  */
+                if (!NILP (Vlist_print_separator))
+                  print_object (Vlist_print_separator, printcharfun, 0);
+
 		obj = XCDR (obj);
 		if (!(i & 1))
 		  halftail = XCDR (halftail);
@@ -1849,6 +1853,9 @@
 		print_object (HASH_KEY (h, i), printcharfun, escapeflag);
 		PRINTCHAR (' ');
 		print_object (HASH_VALUE (h, i), printcharfun, escapeflag);
+                /* Separate entries with list-print-separator.  */
+                if (!NILP (Vlist_print_separator))
+                  print_object (Vlist_print_separator, printcharfun, 0);
 	      }
 
 	  if (size < real_size)
@@ -2108,6 +2115,11 @@
 A value of nil means no limit.  See also `eval-expression-print-level'.  */);
   Vprint_level = Qnil;
 
+  DEFVAR_LISP ("list-print-separator", Vlist_print_separator,
+	       doc: /* When non-nil, printed unescaped to separate list entries.
+Used for hashtable entries as well.  */);
+  Vlist_print_separator = Qnil;
+
   DEFVAR_BOOL ("print-escape-newlines", print_escape_newlines,
 	       doc: /* Non-nil means print newlines in strings as `\\n'.
 Also print formfeeds as `\\f'.  */);

Re: list-print-separator

[Stefan Monnier]

TZ> Since hashtables can get pretty big, it would be nice to have a
TZ> separator between entries when they are printed.  Could that be provided
TZ> at the top level, e.g.

TZ> (let ((hash-table-print-separator "\\\n"))
TZ> (format "%S" my-hash-table))

How 'bout changing `pp' instead?


        Stefan

Re: list-print-separator

[Ted Zlatanov]

On Sun, 24 Apr 2011 01:47:36 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

TZ> Since hashtables can get pretty big, it would be nice to have a
TZ> separator between entries when they are printed.  Could that be provided
TZ> at the top level, e.g.

TZ> (let ((hash-table-print-separator "\\\n"))
TZ> (format "%S" my-hash-table))

SM> How 'bout changing `pp' instead?

I tried that (thanks for the pointer, I didn't know about it) and it was
MUCH slower than my patch.

Tested against a hashtable with 40K elements.  My patch through `format'
took less than 3 seconds on a modern CPU with enough memory.  `pp' took
a minute before I interrupted it.

I intend to use it for large objects so the performance worries me.

Ted

Re: list-print-separator

[Stefan Monnier]

TZ> Since hashtables can get pretty big, it would be nice to have a
TZ> separator between entries when they are printed.  Could that be provided
TZ> at the top level, e.g.

TZ> (let ((hash-table-print-separator "\\\n"))
TZ> (format "%S" my-hash-table))

SM> How 'bout changing `pp' instead?

> I tried that (thanks for the pointer, I didn't know about it) and it was
> MUCH slower than my patch.

> Tested against a hashtable with 40K elements.  My patch through `format'
> took less than 3 seconds on a modern CPU with enough memory.  `pp' took
> a minute before I interrupted it.

Hmm... I wouldn't try to compare the speed of `print' with that of `pp'
since they do very different amounts of work and are implemented in
languages whose performance is very different, but there's clearly room
for improvement in pp's speed.

> I intend to use it for large objects so the performance worries me.

You could use an ad-hoc pp-like function, maybe?  The way I see it, you
typically either need "fast" or "pretty" but very rarely both.


        Stefan

Re: list-print-separator

[Ted Zlatanov]

On Mon, 25 Apr 2011 09:41:39 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Hmm... I wouldn't try to compare the speed of `print' with that of `pp'
SM> since they do very different amounts of work and are implemented in
SM> languages whose performance is very different, but there's clearly room
SM> for improvement in pp's speed.

I want to print out a large ELisp data structure prettily.  The Gnus
newsrc.eld, for example, or the Gnus registry hashtable.  I'll use
whatever makes it happen as fast as possible.

SM> You could use an ad-hoc pp-like function, maybe?  The way I see it, you
SM> typically either need "fast" or "pretty" but very rarely both.

With my list-print-separator patch you get fast and OK-looking.  Are you
concerned it's only useful sometimes?  The speed penalty is minor if
it's not used.

Perhaps people will use it more if it's available.  Maybe it could be a
function too, to be called with the print-level, so the indentation can
vary and so on.  But for my purposes it's enough to have just a string.

Ted

Re: list-print-separator

[Stefan Monnier]

SM> Hmm... I wouldn't try to compare the speed of `print' with that of `pp'
SM> since they do very different amounts of work and are implemented in
SM> languages whose performance is very different, but there's clearly room
SM> for improvement in pp's speed.
> I want to print out a large ELisp data structure prettily.  The Gnus
> newsrc.eld, for example, or the Gnus registry hashtable.  I'll use
> whatever makes it happen as fast as possible.

The newsrc.eld has never been pretty and it's never been a problem, AFAIK.
Not sure what's the "Gnus registry", so I can't comment on that.

SM> You could use an ad-hoc pp-like function, maybe?  The way I see it, you
SM> typically either need "fast" or "pretty" but very rarely both.
> With my list-print-separator patch you get fast and OK-looking.
> Are you concerned it's only useful sometimes?  The speed penalty is
> minor if it's not used.

It's just added complexity for a very unconvincing result.

I'd rather make `pp' faster, or if it's too difficult split it into
a "print-really-pretty" and "print-quickly-and-not-too-ugly".


        Stefan

Re: list-print-separator

[Ted Zlatanov]

On Mon, 25 Apr 2011 11:51:46 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Hmm... I wouldn't try to compare the speed of `print' with that of `pp'
SM> since they do very different amounts of work and are implemented in
SM> languages whose performance is very different, but there's clearly room
SM> for improvement in pp's speed.
>> I want to print out a large ELisp data structure prettily.  The Gnus
>> newsrc.eld, for example, or the Gnus registry hashtable.  I'll use
>> whatever makes it happen as fast as possible.

SM> The newsrc.eld has never been pretty and it's never been a problem,
SM> AFAIK.

It's unreadable without newlines so you need manual fixups to look at it
when things go wrong.

SM> Not sure what's the "Gnus registry", so I can't comment on that.

It's a large hashtable with lots of alist members.

>> With my list-print-separator patch you get fast and OK-looking.
>> Are you concerned it's only useful sometimes?  The speed penalty is
>> minor if it's not used.

SM> I'd rather make `pp' faster, or if it's too difficult split it into
SM> a "print-really-pretty" and "print-quickly-and-not-too-ugly".

The users most likely to benefit, at least for Gnus where I have
experience supporting them, are those who can't read ELisp well.  They
are also less likely to complain or roll their own solution.  So I hope
you see that this is not a "print pretty" request but a real need for
any package that uses ELisp's build-in serialization.  I'm concerned
that your "good enough" solution will not be adopted because it will be
much slower than the native `print' serialization, and optimizing it is
much more work than my proposed patch of 4 lines.

Ted

Re: list-print-separator

[Stefan Monnier]

>>> With my list-print-separator patch you get fast and OK-looking.
>>> Are you concerned it's only useful sometimes?  The speed penalty is
>>> minor if it's not used.
SM> I'd rather make `pp' faster, or if it's too difficult split it into
SM> a "print-really-pretty" and "print-quickly-and-not-too-ugly".
> The users most likely to benefit, at least for Gnus where I have
> experience supporting them, are those who can't read ELisp well.  They
> are also less likely to complain or roll their own solution.

I'm not sure how that relates: what I propose is for Gnus to call
print-quickly-and-not-too-ugly and that would be done by people who know
Elisp fairly well.  The users who don't know Elisp well would not need
to know that the file was printed via this function rather than via
`print'.

> So I hope you see that this is not a "print pretty" request but a real
> need for any package that uses ELisp's build-in serialization.
> I'm concerned that your "good enough" solution will not be adopted
> because it will be much slower than the native `print' serialization,
> and optimizing it is much more work than my proposed patch of 4 lines.

I'm not convinced it's difficult to write
a print-quickly-and-not-too-ugly that's reasonably fast.  Admittedly,
reasonably fast may still be too slow, but in that case I'd rather just
use an ugly print out than try to make `print's output prettier or
more customizable.


        Stefan


PS: you could also add a -*- mode: ugly-sexp -*- cookie in the
newsrc.eld file so that when a user opens it for reading it can easily
(automatically?) be prettified.

Re: list-print-separator

[Ted Zlatanov]

On Mon, 25 Apr 2011 12:37:07 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> With my list-print-separator patch you get fast and OK-looking.
>>>> Are you concerned it's only useful sometimes?  The speed penalty is
>>>> minor if it's not used.
SM> I'd rather make `pp' faster, or if it's too difficult split it into
SM> a "print-really-pretty" and "print-quickly-and-not-too-ugly".
>> The users most likely to benefit, at least for Gnus where I have
>> experience supporting them, are those who can't read ELisp well.  They
>> are also less likely to complain or roll their own solution.

SM> I'm not sure how that relates: what I propose is for Gnus to call
SM> print-quickly-and-not-too-ugly and that would be done by people who know
SM> Elisp fairly well.  The users who don't know Elisp well would not need
SM> to know that the file was printed via this function rather than via
SM> `print'.

Assuming Gnus is the only package to need this, you're right.  But the
reason for my proposal is that I think other packages can use it too.
EIOIO serialization, for instance, could use it.

>> So I hope you see that this is not a "print pretty" request but a real
>> need for any package that uses ELisp's build-in serialization.
>> I'm concerned that your "good enough" solution will not be adopted
>> because it will be much slower than the native `print' serialization,
>> and optimizing it is much more work than my proposed patch of 4 lines.

SM> I'm not convinced it's difficult to write
SM> a print-quickly-and-not-too-ugly that's reasonably fast.  Admittedly,
SM> reasonably fast may still be too slow, but in that case I'd rather just
SM> use an ugly print out than try to make `print's output prettier or
SM> more customizable.

OK.  I'll look at `pp' in my spare time :)

SM> PS: you could also add a -*- mode: ugly-sexp -*- cookie in the
SM> newsrc.eld file so that when a user opens it for reading it can easily
SM> (automatically?) be prettified.

That's a great idea, but I'm concerned that users may need to open the
file outside Emacs if it's causing startup problems or if the data is
triggering a bug (e.g. encoding cookie is wrong).  I've run into that.

Ted

Re: list-print-separator

[Thierry Volpiatto]

Ted Zlatanov <tzz@lifelogs.com> writes:

> OK.  I'll look at `pp' in my spare time :)

I have noticed in the past that printing a whole object with pp is
slower (much) than looping in this object and printing element by
element.
See comment in `bookmark-write-file'.

-- 
A+ Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: list-print-separator

[Stefan Monnier]

>> OK.  I'll look at `pp' in my spare time :)
> I have noticed in the past that printing a whole object with pp is
> slower (much) than looping in this object and printing element by
> element.

Yes, `pp's slowness (as is usually the case for most/all software
slowness problems) is algorithmic.


        Stefan

Re: list-print-separator

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 25 Apr 2011 14:28:38 -0300
> Cc: emacs-devel@gnu.org
> 
> >> OK.  I'll look at `pp' in my spare time :)
> > I have noticed in the past that printing a whole object with pp is
> > slower (much) than looping in this object and printing element by
> > element.
> 
> Yes, `pp's slowness (as is usually the case for most/all software
> slowness problems) is algorithmic.

Perhaps someone could try rewriting `pp' in Python, for the latest GDB
releases.  Maybe that could be faster.  More generally, it would be
nice to have Python versions of the .gdbinit commands we use.

Re: list-print-separator

[Stefan Monnier]

>> >> OK.  I'll look at `pp' in my spare time :)
>> > I have noticed in the past that printing a whole object with pp is
>> > slower (much) than looping in this object and printing element by
>> > element.
>> Yes, `pp's slowness (as is usually the case for most/all software
>> slowness problems) is algorithmic.
> Perhaps someone could try rewriting `pp' in Python, for the latest GDB
> releases.  Maybe that could be faster.  More generally, it would be
> nice to have Python versions of the .gdbinit commands we use.

;-)


        Stefan

Re: list-print-separator

[Andy Moreton]

On Mon 25 Apr 2011, Stefan Monnier wrote:

> SM> Hmm... I wouldn't try to compare the speed of `print' with that of `pp'
> SM> since they do very different amounts of work and are implemented in
> SM> languages whose performance is very different, but there's clearly room
> SM> for improvement in pp's speed.
>> I want to print out a large ELisp data structure prettily.  The Gnus
>> newsrc.eld, for example, or the Gnus registry hashtable.  I'll use
>> whatever makes it happen as fast as possible.
>
> The newsrc.eld has never been pretty and it's never been a problem, AFAIK.
> Not sure what's the "Gnus registry", so I can't comment on that.

I use somthing like this for a prettier newsrc.eld:

(add-hook 'gnus-save-quick-newsrc-hook
          (lambda ()
            (save-excursion
              (goto-char (point-min))
              (save-match-data
                (while (re-search-forward "(\\\"\\| ((\\| (nn" nil t)
                  (replace-match "\n  \\&" t)))
              (delete-trailing-whitespace))))

It may not be perfect, but it does make it a great deal more readable.

    AndyM

Re: list-print-separator

[Ted Zlatanov]

On Mon, 25 Apr 2011 22:05:43 +0100 Andy Moreton <andrewjmoreton@gmail.com> wrote: 

AM> On Mon 25 Apr 2011, Stefan Monnier wrote:
SM> Hmm... I wouldn't try to compare the speed of `print' with that of `pp'
SM> since they do very different amounts of work and are implemented in
SM> languages whose performance is very different, but there's clearly room
SM> for improvement in pp's speed.
>>> I want to print out a large ELisp data structure prettily.  The Gnus
>>> newsrc.eld, for example, or the Gnus registry hashtable.  I'll use
>>> whatever makes it happen as fast as possible.
>> 
>> The newsrc.eld has never been pretty and it's never been a problem, AFAIK.
>> Not sure what's the "Gnus registry", so I can't comment on that.

AM> I use somthing like this for a prettier newsrc.eld:

AM> (add-hook 'gnus-save-quick-newsrc-hook
AM>           (lambda ()
AM>             (save-excursion
AM>               (goto-char (point-min))
AM>               (save-match-data
AM>                 (while (re-search-forward "(\\\"\\| ((\\| (nn" nil t)
AM>                   (replace-match "\n  \\&" t)))
AM>               (delete-trailing-whitespace))))

AM> It may not be perfect, but it does make it a great deal more readable.

Me too, but that doesn't work for new users, it relies on regular
expressions, and it's slow.  My suggestion for the above was basically
to make `list-print-separator' "\n \\" but I will instead work on `pp'
as Stefan has requested.

Ted

Re: list-print-separator

[Andy Moreton]

On Mon 25 Apr 2011, Ted Zlatanov wrote:

> On Mon, 25 Apr 2011 22:05:43 +0100 Andy Moreton <andrewjmoreton@gmail.com> wrote: 
>
> AM> On Mon 25 Apr 2011, Stefan Monnier wrote:
> SM> Hmm... I wouldn't try to compare the speed of `print' with that of `pp'
> SM> since they do very different amounts of work and are implemented in
> SM> languages whose performance is very different, but there's clearly room
> SM> for improvement in pp's speed.
>>>> I want to print out a large ELisp data structure prettily.  The Gnus
>>>> newsrc.eld, for example, or the Gnus registry hashtable.  I'll use
>>>> whatever makes it happen as fast as possible.
>>> 
>>> The newsrc.eld has never been pretty and it's never been a problem, AFAIK.
>>> Not sure what's the "Gnus registry", so I can't comment on that.
>
> AM> I use somthing like this for a prettier newsrc.eld:
>
> AM> (add-hook 'gnus-save-quick-newsrc-hook
> AM>           (lambda ()
> AM>             (save-excursion
> AM>               (goto-char (point-min))
> AM>               (save-match-data
> AM>                 (while (re-search-forward "(\\\"\\| ((\\| (nn" nil t)
> AM>                   (replace-match "\n  \\&" t)))
> AM>               (delete-trailing-whitespace))))
>
> AM> It may not be perfect, but it does make it a great deal more readable.
>
> Me too, but that doesn't work for new users, it relies on regular
> expressions, and it's slow.  My suggestion for the above was basically
> to make `list-print-separator' "\n \\" but I will instead work on `pp'
> as Stefan has requested.

New users may have difficulty with regexps, so why not just do this
inside gnus anyway ? The regexp loop is pretty simple , and I've never
noticed it being slow. Tweaking `pp' is useful too though.

    AndyM

Re: list-print-separator

[Stefan Monnier]

AM> (add-hook 'gnus-save-quick-newsrc-hook
AM> (lambda ()
AM> (save-excursion
AM> (goto-char (point-min))
AM> (save-match-data
AM> (while (re-search-forward "(\\\"\\| ((\\| (nn" nil t)
AM> (replace-match "\n  \\&" t)))
AM> (delete-trailing-whitespace))))

AM> It may not be perfect, but it does make it a great deal more readable.

> Me too, but that doesn't work for new users, it relies on regular
> expressions, and it's slow.

The above shouldn't be slow (at least, not algorithmically so).


        Stefan

Re: list-print-separator

[Ted Zlatanov]

On Tue, 26 Apr 2011 09:55:07 -0300 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

AM> (add-hook 'gnus-save-quick-newsrc-hook
AM> (lambda ()
AM> (save-excursion
AM> (goto-char (point-min))
AM> (save-match-data
AM> (while (re-search-forward "(\\\"\\| ((\\| (nn" nil t)
AM> (replace-match "\n  \\&" t)))
AM> (delete-trailing-whitespace))))

AM> It may not be perfect, but it does make it a great deal more readable.

>> Me too, but that doesn't work for new users, it relies on regular
>> expressions, and it's slow.

SM> The above shouldn't be slow (at least, not algorithmically so).

I should have said "slower than C code," sorry.  It's not terribly slow
but (for me) stutters when saving my newsrc.eld, which is over 2 MB.  In
any case it's the least of the problems; the big issue is that it uses
regular expressions.

Ted