A question about read syntax and compiled backreferences

A question about read syntax and compiled backreferences

[Juanma Barranquero]

Why can I evaluate this:

(defun test ()
  (let ((z '#1=(a b . #1#)))
    (safe-length z)))

but trying to loading it from a byte-compiled file brings

  Invalid read syntax: "#"

?

(I'm sure there's a simple explanation and I'm overlooking something...)

                    /L/e/k/t/u

Re: A question about read syntax and compiled backreferences

[Eli Zaretskii]

> Date: Mon, 22 Jan 2007 17:13:07 +0100
> From: "Juanma Barranquero" <lekktu@gmail.com>
> 
> Why can I evaluate this:
> 
> (defun test ()
>   (let ((z '#1=(a b . #1#)))
>     (safe-length z)))
> 
> but trying to loading it from a byte-compiled file brings
> 
>   Invalid read syntax: "#"
> 
> ?

Maybe because the byte compiler emits constructs that use # ?

Re: A question about read syntax and compiled backreferences

[Juanma Barranquero]

On 1/22/07, Eli Zaretskii <eliz@gnu.org> wrote:

> Maybe because the byte compiler emits constructs that use # ?

And that should affect because...?  (I'm honestly trying to understand)

All in all, isn't a bug that I can write a function (not a macro), run
it interpreted or byte-compiled, but when I byte-compile it in a file
I'm unable to load it?

                    /L/e/k/t/u

Re: A question about read syntax and compiled backreferences

[Miles Bader]

"Juanma Barranquero" <lekktu@gmail.com> writes:
> (defun test ()
>  (let ((z '#1=(a b . #1#)))
>    (safe-length z)))
>
> but trying to loading it from a byte-compiled file brings
>
>  Invalid read syntax: "#"

The following patch seems to fix this bug; I'm not sure if it's OK to
apply or not (is print-circle significantly slower?).

-Miles


2007-01-23  Miles Bader  <miles@gnu.org>

	* emacs-lisp/bytecomp.el (byte-compile-output-file-form)
	(byte-compile-output-docform): Bind `print-circle' to t.

--- orig/lisp/emacs-lisp/bytecomp.el
+++ mod/lisp/emacs-lisp/bytecomp.el
@@ -1,7 +1,7 @@
 ;;; bytecomp.el --- compilation of Lisp code into byte code
 
 ;; Copyright (C) 1985, 1986, 1987, 1992, 1994, 1998, 2000, 2001, 2002,
-;;   2003, 2004, 2005, 2006 Free Software Foundation, Inc.
+;;   2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc.
 
 ;; Author: Jamie Zawinski <jwz@lucid.com>
 ;;	Hallvard Furuseth <hbf@ulrik.uio.no>
@@ -2082,7 +2082,8 @@
 	  (print-length nil)
 	  (print-level nil)
 	  (print-quoted t)
-	  (print-gensym t))
+	  (print-gensym t)
+	  (print-circle t))	       ; handle circular data structures
       (princ "\n" outbuffer)
       (prin1 form outbuffer)
       nil)))
@@ -2139,6 +2140,7 @@
 	       ;; print-gensym-alist not to be cleared
 	       ;; between calls to print functions.
 	       (print-gensym '(t))
+	       (print-circle t)	       ; handle circular data structures
 	       print-gensym-alist    ; was used before print-circle existed.
 	       (print-continuous-numbering t)
 	       print-number-table


-- 
"Whatever you do will be insignificant, but it is very important that
 you do it."  Mahatma Gandhi

Re: A question about read syntax and compiled backreferences

[Juanma Barranquero]

On 1/23/07, Miles Bader <miles.bader@necel.com> wrote:

> The following patch seems to fix this bug;

Yes.

> I'm not sure if it's OK to
> apply or not (is print-circle significantly slower?).

Well, even if it is slower is The Right Thing To Do, isn't? Or else we
should forbid circular structures while byte-compiling files.

                    /L/e/k/t/u

Re: A question about read syntax and compiled backreferences

[Miles Bader]

"Juanma Barranquero" <lekktu@gmail.com> writes:
>> I'm not sure if it's OK to
>> apply or not (is print-circle significantly slower?).
>
> Well, even if it is slower is The Right Thing To Do, isn't?

Yes, I think so; I'm just nervous about applying a patch to the
byte-compiler just before a release.

Since there have apparently been no previous bug reports about this
issue, it doesn't seem to be critical, and probably could wait until
after the release.

[I don't actually _know_ that it's slower -- Emacs does seem to track
object references even when print-circle is nil, so it can avoid an
infinite loop in the printer.  It just seems to be a possibility.]

-Miles

-- 
`Cars give people wonderful freedom and increase their opportunities.
 But they also destroy the environment, to an extent so drastic that
 they kill all social life' (from _A Pattern Language_)

Re: A question about read syntax and compiled backreferences

[Juanma Barranquero]

On 1/24/07, Miles Bader <miles@gnu.org> wrote:

> Since there have apparently been no previous bug reports about this
> issue

FWIW, my bug report wasn't an artificial case: I have a circular list
in my .emacs, and I was very surprised when I discovered that I could
byte-compile it but was unable to load it afterwards.

> it doesn't seem to be critical, and probably could wait until
> after the release.

No objections here.

                    /L/e/k/t/u

Re: A question about read syntax and compiled backreferences

[Richard Stallman]

It is more important for compilation to work right than for it to run
fast, so please install your patch.

Re: A question about read syntax and compiled backreferences

[Miles Bader]

Richard Stallman <rms@gnu.org> writes:
> It is more important for compilation to work right than for it to run
> fast, so please install your patch.

Ok, I've done so.

-miles

-- 
(\(\
(^.^)
(")")
*This is the cute bunny virus, please copy this into your sig so it can spread.

Re: A question about read syntax and compiled backreferences

[Andreas Schwab]

Miles Bader <miles.bader@necel.com> writes:

> "Juanma Barranquero" <lekktu@gmail.com> writes:
>> (defun test ()
>>  (let ((z '#1=(a b . #1#)))
>>    (safe-length z)))
>>
>> but trying to loading it from a byte-compiled file brings
>>
>>  Invalid read syntax: "#"
>
> The following patch seems to fix this bug; I'm not sure if it's OK to
> apply or not (is print-circle significantly slower?).

Yes, it is.  Compiling leim/ja-dic/ja-dic.el takes ages.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: A question about read syntax and compiled backreferences

[Andreas Schwab]

Andreas Schwab <schwab@suse.de> writes:

> Miles Bader <miles.bader@necel.com> writes:
>
>> "Juanma Barranquero" <lekktu@gmail.com> writes:
>>> (defun test ()
>>>  (let ((z '#1=(a b . #1#)))
>>>    (safe-length z)))
>>>
>>> but trying to loading it from a byte-compiled file brings
>>>
>>>  Invalid read syntax: "#"
>>
>> The following patch seems to fix this bug; I'm not sure if it's OK to
>> apply or not (is print-circle significantly slower?).
>
> Yes, it is.  Compiling leim/ja-dic/ja-dic.el takes ages.

16 minutes, to be precise.  Without print-circle, it takes just a couple
of seconds.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: A question about read syntax and compiled backreferences

[Juanma Barranquero]

On 1/27/07, Andreas Schwab <schwab@suse.de> wrote:

> 16 minutes, to be precise.  Without print-circle, it takes just a couple
> of seconds.

Take a look at this thread:

http://lists.gnu.org/archive/html/emacs-pretest-bug/2007-01/msg00470.html

Kenichi has a patch to turn print-circle checking off via local
variables. It is not yet installed.

                    /L/e/k/t/u

Re: A question about read syntax and compiled backreferences

[David Kastrup]

"Juanma Barranquero" <lekktu@gmail.com> writes:

> On 1/27/07, Andreas Schwab <schwab@suse.de> wrote:
>
>> 16 minutes, to be precise.  Without print-circle, it takes just a couple
>> of seconds.
>
> Take a look at this thread:
>
> http://lists.gnu.org/archive/html/emacs-pretest-bug/2007-01/msg00470.html
>
> Kenichi has a patch to turn print-circle checking off via local
> variables. It is not yet installed.

In my book, this is just dealing with symptoms.  I don't think that
the ability to byte-compile code with circular lists is really
important enough to warrant this sort of inconvenience.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: A question about read syntax and compiled backreferences

[Romain Francoise]

David Kastrup <dak@gnu.org> writes:

> I don't think that the ability to byte-compile code with circular
> lists is really important enough to warrant this sort of
> inconvenience.

I think we should revert this change for now, and add it back after
the release.

-- 
Romain Francoise <romain@orebokech.com> | The sea! the sea! the open
it's a miracle -- http://orebokech.com/ | sea! The blue, the fresh, the
                                        | ever free! --Bryan W. Procter

Re: A question about read syntax and compiled backreferences

[Stefan Monnier]

>> 16 minutes, to be precise.  Without print-circle, it takes just a couple
>> of seconds.

> Take a look at this thread:

> http://lists.gnu.org/archive/html/emacs-pretest-bug/2007-01/msg00470.html

> Kenichi has a patch to turn print-circle checking off via local
> variables. It is not yet installed.

BTW, maybe it would be worth it to try and investigate why it gets so much
slower: we may be able to fix the slowdown rather than circumvent it.


        Stefan

Re: A question about read syntax and compiled backreferences

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> BTW, maybe it would be worth it to try and investigate why it gets so much
> slower: we may be able to fix the slowdown rather than circumvent it.

It makes print_preprocess quadratic in the number of subobjects in the
object being printed.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: A question about read syntax and compiled backreferences

[Juanma Barranquero]

On 1/27/07, David Kastrup <dak@gnu.org> wrote:

> I don't think that
> the ability to byte-compile code with circular lists is really
> important enough to warrant this sort of inconvenience.

I don't think having a local variable to deactivate it is
inconvenient; after all, there's only a few elisp packages in Emacs
where the slowdown is significant.

OTOH, perhaps we could to the reverse: default print-circle to nil
during byte-compilation, and add a variable to activate it for the few
files that really need it.

But anyway, some sensible behavior should be used: either we do
compile correctly, or we detect circular lists (I mean, we detect uses
of backreferences, like #1=...) and refuse to byte-compile it. What is
not reasonable is byte-compiling it and generating wrong byte-code
that can not be loaded.

                    /L/e/k/t/u

Re: A question about read syntax and compiled backreferences

[Miles Bader]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> BTW, maybe it would be worth it to try and investigate why it gets so much
> slower: we may be able to fix the slowdown rather than circumvent it.

I think for each lisp object printed, it does a linear search on a list
of all lisp objects previously printed (within the same "top-level"
print).

It would presumably help a lot to use an eq hash table instead.

-miles

-- 
I'm beginning to think that life is just one long Yoko Ono album; no rhyme
or reason, just a lot of incoherent shrieks and then it's over.  --Ian Wolff

Re: A question about read syntax and compiled backreferences

[Kim F. Storm]

Andreas Schwab <schwab@suse.de> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> BTW, maybe it would be worth it to try and investigate why it gets so much
>> slower: we may be able to fix the slowdown rather than circumvent it.
>
> It makes print_preprocess quadratic in the number of subobjects in the
> object being printed.

So using a hash table would improve performance significantly ...
But now is not the time to make that change.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: A question about read syntax and compiled backreferences

[Richard Stallman]

    Kenichi has a patch to turn print-circle checking off via local
    variables. It is not yet installed.

Would someone please install this?

Re: A question about read syntax and compiled backreferences

[David Kastrup]

"Juanma Barranquero" <lekktu@gmail.com> writes:

> On 1/27/07, David Kastrup <dak@gnu.org> wrote:
>
>> I don't think that
>> the ability to byte-compile code with circular lists is really
>> important enough to warrant this sort of inconvenience.
>
> I don't think having a local variable to deactivate it is
> inconvenient; after all, there's only a few elisp packages in Emacs
> where the slowdown is significant.
>
> OTOH, perhaps we could to the reverse: default print-circle to nil
> during byte-compilation, and add a variable to activate it for the few
> files that really need it.

Sounds more sensible to me.  But actually, I see

    print-circle is a variable defined in `src/print.c'.
    Its value is nil


    Documentation:
    *Non-nil means print recursive structures using #N= and #N# syntax.
    If nil, printing proceeds recursively and may lead to
    `max-lisp-eval-depth' being exceeded or an error may occur:
    "Apparently circular structure being printed."  Also see
    `print-length' and `print-level'.
    If non-nil, shared substructures anywhere in the structure are printed
    with `#N=' before the first occurrence (in the order of the print
    representation) and `#N#' in place of each subsequent occurrence,
    where N is a positive decimal integer.

Can't we just catch the error, remove what has been printed so far,
and retry with print-circle set?

That would give us no performance hit whatsoever during normal
operation and would allow circular structures without announcing them.
And the performance hit occurs not on the whole file, but just on the
structure in question.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: A question about read syntax and compiled backreferences

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> Can't we just catch the error, remove what has been printed so far,
> and retry with print-circle set?

See the start of this thread.  When printing without print-circle there is
a simplistic circle check that can avoid the error, but creates output
that cannot be read back in.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: A question about read syntax and compiled backreferences

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Can't we just catch the error, remove what has been printed so far,
>> and retry with print-circle set?
>
> See the start of this thread.  When printing without print-circle there is
> a simplistic circle check that can avoid the error, but creates output
> that cannot be read back in.

If it avoids the circle, it presumably also detects it.  Why can't it
throw an error then instead of producing output that can't be read in
again?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: A question about read syntax and compiled backreferences

[Richard Stallman]

This issue is NOT about circular lists!

    OTOH, perhaps we could to the reverse: default print-circle to nil
    during byte-compilation, and add a variable to activate it for the few
    files that really need it.

No thanks.

The fix is needed to handle the code generated by legitimate macros
that put uninterned symbols into it.  People should not have to set a
flag to make calls to such macros work.

Re: A question about read syntax and compiled backreferences

[Richard Stallman]

    So using a hash table would improve performance significantly ...
    But now is not the time to make that change.

I agree.

Re: A question about read syntax and compiled backreferences

[Kenichi Handa]

In article <E1HB4fZ-00064U-Au@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     Kenichi has a patch to turn print-circle checking off via local
>     variables. It is not yet installed.

> Would someone please install this?

I've just installed this:

2007-01-29  Kenichi Handa  <handa@m17n.org>

	* emacs-lisp/bytecomp.el (byte-compile-disable-print-circle): New
	variable.
	(byte-compile-output-file-form): Bing print-circle to nil if
	byte-compile-output-file-form is not nil.
	(byte-compile-output-docform): Likewise.

I'm now working on the codes that generate leim/*.el and
ja-dic/ja-dic.el to make them add
"byte-compile-disable-print-circle:t".

---
Kenichi Handa
handa@m17n.org

Re: A question about read syntax and compiled backreferences

[Miles Bader]

Kenichi Handa <handa@m17n.org> writes:
> 	* emacs-lisp/bytecomp.el (byte-compile-disable-print-circle): New
> 	variable.

I think it's better to use a "positive" name, like
`byte-compile-print-circle' (defaulting to t).

Then it's more natural regardless of the default value (I can imagine
that we might decide to change that later), whereas names like
"...-disable-..." really only sound right if their default value is nil.

-Miles

-- 
`The suburb is an obsolete and contradictory form of human settlement'

Re: A question about read syntax and compiled backreferences

[Richard Stallman]

    Can't we just catch the error, remove what has been printed so far,
    and retry with print-circle set?

That would work for cyclic structure,
but it would not work for the case we really care about here:
shared structure such as uninterned symbols.

Re: A question about read syntax and compiled backreferences

[Stuart D. Herring]

> If it avoids the circle, it presumably also detects it.  Why can't it
> throw an error then instead of producing output that can't be read in
> again?

Sometimes it neither avoids the circle nor fails, yet produces wrong output:

(let* ((x (vector)) (y (cons x x))) (princ y))

just prints ([] . []), and the reader will not generate two `eq' vectors
from reading this.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: A question about read syntax and compiled backreferences

[Richard Stallman]

    > 	* emacs-lisp/bytecomp.el (byte-compile-disable-print-circle): New
    > 	variable.

    I think it's better to use a "positive" name, like
    `byte-compile-print-circle' (defaulting to t).

    Then it's more natural regardless of the default value (I can imagine
    that we might decide to change that later), whereas names like
    "...-disable-..." really only sound right if their default value is nil.

I think it is better if the default is nil, so let's stay with the
current name.

Re: A question about read syntax and compiled backreferences

[Romain Francoise]

It seems that the print-circle change also triggers a problem in VM,
which now takes an insanely long time to compile (I interrupted it
after 30 minutes).

Is there any hope that the speed issues introduced by this change
will be fixed before the release?  Otherwise I'm going to ask the VM
maintainer to use `byte-compile-disable-print-circle'...

(See http://bugs.debian.org/410492 for more info.)

Thanks,

-- 
Romain Francoise <romain@orebokech.com> | The sea! the sea! the open
it's a miracle -- http://orebokech.com/ | sea! The blue, the fresh, the
                                        | ever free! --Bryan W. Procter