Invalid read syntax for compiled bool vector

Invalid read syntax for compiled bool vector

[Lars Brinkhoff]

I run these forms (in the *scratch* buffer or with M-C-x) to create a
source file and byte-compile it:

  (with-temp-file "test.el"
    (princ `(defun foo ()
             ,(let ((vec (make-bool-vector 20 t)))
                (dolist (i '(11 13 16 18) vec)
                  (aset vec i nil))))
           (current-buffer)))
  (byte-compile-file "test.el")

When I load the compiled file:

  (load-file "test.elc")

I get this error:

  Invalid read syntax: "#&..."

The pattern in the last 12 elements of the bool vector is what
triggers the error.  The contents of the first 8 elements doesn't
seem to affect the result.

This happens in both GNU Emacs 21.3 and 20.7, running in Debian
GNU/Linux on a PowerPC.

-- 
Lars Brinkhoff,         Services for Unix, Linux, GCC, HTTP
Brinkhoff Consulting    http://www.brinkhoff.se/

Re: Invalid read syntax for compiled bool vector

[Lars Brinkhoff]

Richard Stallman <rms@gnu.org> writes:
>       (with-temp-file "test.el"
> 	(princ `(defun foo ()
> 		 ,(let ((vec (make-bool-vector 20 t)))
> 		    (dolist (i '(11 13 16 18) vec)
> 		      (aset vec i nil))))
> 	       (current-buffer)))
> 
> When I run that, it asks me what coding system to use when saving it.
> 
> When I add ;; -*-coding: no-conversion; -*- as the first line,
> there is no problem.

Apparently, you have to bind coding-system-for-write before writing a
source file with a literal bool-vector constant in it, or else Emacs
will either ask the user for the coding system, or write the file
using some default coding system which may not do the right thing.

Also, you have to bind coding-system-for-read before byte-compiling
such a file:

    (let ((coding-system-for-write 'no-conversion))
      (with-temp-file "test.el"
        (prin1 `(defun foo ()
                  ,(let ((vec (make-bool-vector 20 t)))
                     (dolist (i '(11 13 16 18) vec)
                       (aset vec i nil))))
               (current-buffer))))
    (let ((coding-system-for-read 'no-conversion))
      (byte-compile-file "test.el"))
    (load-file "test.elc")
    
In contrast, you don't have to bind coding-system-for-read when byte-
compiling a source file with a string constant:

    (let ((coding-system-for-write 'no-conversion))
      (with-temp-file "test.el"
        (insert (format "(defun foo () \"%s\")" "\377\327\n"))))
    (byte-compile-file "test.el")
    (load-file "test.elc")

Here, the string has the same binary data as the bool vector, but the
compiler handles it more gracefully, in my opinion.

I guess this isn't exactly a bug, but perhaps the Emacs Lisp manual
should say something about binding coding-system-for-read and
coding-system-for-write when reading and writing source files.

-- 
Lars Brinkhoff,         Services for Unix, Linux, GCC, HTTP
Brinkhoff Consulting    http://www.brinkhoff.se/

Re: Invalid read syntax for compiled bool vector

[Richard Stallman]

    Apparently, you have to bind coding-system-for-write before writing a
    source file with a literal bool-vector constant in it, or else Emacs
    will either ask the user for the coding system, or write the file
    using some default coding system which may not do the right thing.

I guess we should change the syntax for bool-vectors
so as to put just 4 bits into each character.
The question is how to do that in a somewhat compatible way.

Re: Invalid read syntax for compiled bool vector

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>     Apparently, you have to bind coding-system-for-write before writing a
>     source file with a literal bool-vector constant in it, or else Emacs
>     will either ask the user for the coding system, or write the file
>     using some default coding system which may not do the right thing.
>
> I guess we should change the syntax for bool-vectors
> so as to put just 4 bits into each character.
> The question is how to do that in a somewhat compatible way.

The print syntax could use octal or hexadecimal escapes in the bit string.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Invalid read syntax for compiled bool vector

[Lars Brinkhoff]

Andreas Schwab <schwab@suse.de> writes:
> Richard Stallman <rms@gnu.org> writes:
> >     Apparently, you have to bind coding-system-for-write before
> >     writing a source file with a literal bool-vector constant in
> >     it, or else Emacs will either ask the user for the coding
> >     system, or write the file using some default coding system
> >     which may not do the right thing.
> > I guess we should change the syntax for bool-vectors so as to put
> > just 4 bits into each character.  The question is how to do that
> > in a somewhat compatible way.
> The print syntax could use octal or hexadecimal escapes in the bit string.

Yes.  Since the print syntax for bool-vectors looks like strings, I
would sugggest doing whatever the print syntax for strings does.

-- 
Lars Brinkhoff,         Services for Unix, Linux, GCC, HTTP
Brinkhoff Consulting    http://www.brinkhoff.se/

Re: Invalid read syntax for compiled bool vector

[Andreas Schwab]

Lars Brinkhoff <lars@nocrew.org> writes:

> Andreas Schwab <schwab@suse.de> writes:
>> Richard Stallman <rms@gnu.org> writes:
>> >     Apparently, you have to bind coding-system-for-write before
>> >     writing a source file with a literal bool-vector constant in
>> >     it, or else Emacs will either ask the user for the coding
>> >     system, or write the file using some default coding system
>> >     which may not do the right thing.
>> > I guess we should change the syntax for bool-vectors so as to put
>> > just 4 bits into each character.  The question is how to do that
>> > in a somewhat compatible way.
>> The print syntax could use octal or hexadecimal escapes in the bit string.
>
> Yes.  Since the print syntax for bool-vectors looks like strings, I
> would sugggest doing whatever the print syntax for strings does.

I have now changed the print syntax to always use octal escapes for
non-ascii characters in the bool-vector string.  This way the string will
always be read as unibyte string, avoiding all coding issues.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Invalid read syntax for compiled bool vector

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> Lars Brinkhoff <lars@nocrew.org> writes:
>
> > Yes.  Since the print syntax for bool-vectors looks like strings,
> > I would sugggest doing whatever the print syntax for strings does.
> 
> I have now changed the print syntax to always use octal escapes for
> non-ascii characters in the bool-vector string.  This way the string
> will always be read as unibyte string, avoiding all coding issues.

I think that hexadecimal notation would be quite more compact, without
a loss of generality and (probably unimportant) readability.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Invalid read syntax for compiled bool vector

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> I think that hexadecimal notation would be quite more compact, without
> a loss of generality and (probably unimportant) readability.

If you use hexadecimal notation then the reader will force the string to
multibyte, with octal notation it is forced to unibyte.  The process of
converting a unibyte string to multibyte will change characters in the
range 0x80..0x9f.  Maybe the bool vector reader should just force the
string back to unibyte.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Invalid read syntax for compiled bool vector

[David Kastrup]

Andreas Schwab <schwab@suse.de> writes:

> David Kastrup <dak@gnu.org> writes:
> 
> > I think that hexadecimal notation would be quite more compact, without
> > a loss of generality and (probably unimportant) readability.
> 
> If you use hexadecimal notation then the reader will force the string to
> multibyte, with octal notation it is forced to unibyte.  The process of
> converting a unibyte string to multibyte will change characters in the
> range 0x80..0x9f.  Maybe the bool vector reader should just force the
> string back to unibyte.

Ah, uh, ok.  Just forget it, then.  Looks like I did not know what I
was talking about.  Not that this happens too rarely...

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Invalid read syntax for compiled bool vector

[Richard Stallman]

    The print syntax could use octal or hexadecimal escapes in the bit string.

That would be incompatible for some bool-vector values, wouldn't it?
The \ character could appear in a bool-vector with the current syntax.

To avoid misinterpreting some constants, I think we need to change the
syntax in a bigger way, to use a new syntax that would not be valid at
all under the old rules.

Re: Invalid read syntax for compiled bool vector

[Richard Stallman]

    I have now changed the print syntax to always use octal escapes for
    non-ascii characters in the bool-vector string.  This way the string will
    always be read as unibyte string, avoiding all coding issues.

That was too hasty.  As I mentioned earlier, this solution would cause
some previously-written bool-vectors to be read wrong.

If nobody has any previously-written bool-vectors, that incompatibility
does not matter, but are we confident of that?  And is there a better way?

Re: Invalid read syntax for compiled bool vector

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>     The print syntax could use octal or hexadecimal escapes in the bit string.
>
> That would be incompatible for some bool-vector values, wouldn't it?
> The \ character could appear in a bool-vector with the current syntax.

No, the print syntax already used backslash as escape within the string,
and the reader just uses the normal string parser.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Invalid read syntax for compiled bool vector

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>     I have now changed the print syntax to always use octal escapes for
>     non-ascii characters in the bool-vector string.  This way the string will
>     always be read as unibyte string, avoiding all coding issues.
>
> That was too hasty.  As I mentioned earlier, this solution would cause
> some previously-written bool-vectors to be read wrong.

The new syntax variant is completely backward and forward compatible.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux AG, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Invalid read syntax for compiled bool vector

[Richard Stallman]

    No, the print syntax already used backslash as escape within the string,
    and the reader just uses the normal string parser.

Ok, in that case I see no problem in your solution.
Thanks for taking care of the problem.