bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Michael Heerdegen]

Hello,

when updating packages with M-x package-list-packages, and using
async-bytecomp.el to compile, some users get an error like this one:

error in process sentinel: async-when-done: Invalid read syntax: "#<"
error in process sentinel: Invalid read syntax: "#<"

This seems to be related to the recent addition of symbols with
positions.

async uses `async-inject-variables' to collect variable bindings of
interest and deliver them to the other Emacs instance.  The return value
contains symbols with positions in some scenarios.

The root of the trouble seems to be the value of
`byte-optimize--dynamic-vars'.  As soon as I byte compile any file, I
see lots of symbols with positions in `byte-optimize--dynamic-vars'.

Is that expected?  What has to be done?

TIA,

Michael.


In GNU Emacs 29.0.50 (build 55, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2022-03-17 built on drachen
Repository revision: 97e687006498490cca1e6dabc1ea7e692271c5ed
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure --with-native-compilation --with-xwidgets'

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Alan Mackenzie]

Hello, Michael.

Thanks for the bug report.

On Thu, Mar 17, 2022 at 03:55:27 +0100, Michael Heerdegen wrote:

> Hello,

> when updating packages with M-x package-list-packages, and using
> async-bytecomp.el to compile, some users get an error like this one:

> error in process sentinel: async-when-done: Invalid read syntax: "#<"
> error in process sentinel: Invalid read syntax: "#<"

Could you please give a recipe for this situation, preferably starting
from emacs -Q?  I don't know how to use async-bytecomp.el, for example.

> This seems to be related to the recent addition of symbols with
> positions.

It sounds like something is storing a compiler intermediate form, and
reading it again with the reader.

> async uses `async-inject-variables' to collect variable bindings of
> interest and deliver them to the other Emacs instance.  The return value
> contains symbols with positions in some scenarios.

> The root of the trouble seems to be the value of
> `byte-optimize--dynamic-vars'.  As soon as I byte compile any file, I
> see lots of symbols with positions in `byte-optimize--dynamic-vars'.

Thanks, that should be useful information.

> Is that expected?  What has to be done?

No, it is not expected.  As for what needs doing, let's just say I
wasn't aware of the existence of async-bytecomp.el a day or two ago.
;-(

> TIA,

> Michael.


> In GNU Emacs 29.0.50 (build 55, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
>  of 2022-03-17 built on drachen
> Repository revision: 97e687006498490cca1e6dabc1ea7e692271c5ed
> Repository branch: master
> Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
> System Description: Debian GNU/Linux 11 (bullseye)

> Configured using:
>  'configure --with-native-compilation --with-xwidgets'

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Michael Heerdegen]

Alan Mackenzie <acm@muc.de> writes:

> Could you please give a recipe for this situation, preferably starting
> from emacs -Q?  I don't know how to use async-bytecomp.el, for
> example.

Here is the issue:

https://github.com/jwiegley/emacs-async/issues/153

But I don't think async or async-bytecomp is related to this issue at
all.  It only hits an issue with vanilla Emacs.  I think it's not
necessary for you to have a look at async, unless you are interested,
also because I have a recipe for emacs -Q:

Byte compile lisp/calendar/appt.el (I did that with B from dired); after
that:

byte-optimize--dynamic-vars ->
(#<symbol original-date at 24141> #<symbol number at 24125>)

The problem for async-bytecomp is then only that a result like this is
not `read'able after being printed.

Thierry has removed that variable from the list of handled variables for
now which circumvents the problem.

Michael.

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Michael Heerdegen]

Michael Heerdegen <michael_heerdegen@web.de> writes:

> Here is the issue:
>
> https://github.com/jwiegley/emacs-async/issues/153

For now I gave the advice to print symbols with `print-symbols-bare'
bound to t...

> byte-optimize--dynamic-vars ->
> (#<symbol original-date at 24141> #<symbol number at 24125>)

... and leave it to you to decide if this should be fixed.  It's not a
big surprise to find symbols with positions in an internal variable of
the byte compiler.  If the position information is of no relevance in
that variable, I guess you can strip the positions?

BTW, async-bytecomp just compiles files in another Emacs instance for
better speed.  The concept is really simple.  It only informs the other
instance of some variable bindings it needs to know
(e.g. default-directory and load-path) - and
`byte-optimize--dynamic-vars' is one of them.  The values are being
printed in the original session and `read' and assigned in the slave
instance.

Michael.

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Alan Mackenzie]

Hello, Michael.

On Fri, Mar 18, 2022 at 00:50:17 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Could you please give a recipe for this situation, preferably starting
> > from emacs -Q?  I don't know how to use async-bytecomp.el, for
> > example.

> Here is the issue:

> https://github.com/jwiegley/emacs-async/issues/153

Thanks.  I've read it, now.

> But I don't think async or async-bytecomp is related to this issue at
> all.  It only hits an issue with vanilla Emacs.  I think it's not
> necessary for you to have a look at async, unless you are interested,
> also because I have a recipe for emacs -Q:

> Byte compile lisp/calendar/appt.el (I did that with B from dired); after
> that:

> byte-optimize--dynamic-vars ->
> (#<symbol original-date at 24141> #<symbol number at 24125>)

> The problem for async-bytecomp is then only that a result like this is
> not `read'able after being printed.

I made a mistake in my last post.  There's actually no reason
byte-optimize--dynamic-vars shouldn't contain symbols with position.
That's what it accumulates.  It is also meant to be a private variable
in the byte compilation (hence the "--" in its name).

Why is async-bytecomp copying byte-optimize--dynamic-vars?  There seems
to be no reason.  Maybe any variables with "--" in their symbols should
be filtered out of the copying operation.

Also, is printing followed by reading really the best way to get a copy
of a Lisp object?  There are several Lisp types, possibly most of them,
whose printed representation can't be read; things like buffers,
windows, processes, .....  I think only the really common objects, like
lists, vectors, strings, numbers have readable printed representations.

> Thierry has removed that variable from the list of handled variables for
> now which circumvents the problem.

I've had at least two Cc:s to Thierry at <thierry.volpiatto@gmail.com>
bounce with "address does not exist".  Do you know if Thierry has
changed his address, or could it just be Google playing more silly
games?

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Thierry Volpiatto]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Michael.
>
> On Fri, Mar 18, 2022 at 00:50:17 +0100, Michael Heerdegen wrote:
>> Alan Mackenzie <acm@muc.de> writes:
>
>> > Could you please give a recipe for this situation, preferably starting
>> > from emacs -Q?  I don't know how to use async-bytecomp.el, for
>> > example.
>
>> Here is the issue:
>
>> https://github.com/jwiegley/emacs-async/issues/153
>
> Thanks.  I've read it, now.
>
>> But I don't think async or async-bytecomp is related to this issue at
>> all.  It only hits an issue with vanilla Emacs.  I think it's not
>> necessary for you to have a look at async, unless you are interested,
>> also because I have a recipe for emacs -Q:
>
>> Byte compile lisp/calendar/appt.el (I did that with B from dired); after
>> that:
>
>> byte-optimize--dynamic-vars ->
>> (#<symbol original-date at 24141> #<symbol number at 24125>)
>
>> The problem for async-bytecomp is then only that a result like this is
>> not `read'able after being printed.
>
> I made a mistake in my last post.  There's actually no reason
> byte-optimize--dynamic-vars shouldn't contain symbols with position.
> That's what it accumulates.  It is also meant to be a private variable
> in the byte compilation (hence the "--" in its name).
>
> Why is async-bytecomp copying byte-optimize--dynamic-vars?  There seems
> to be no reason.

You are right, there is no reason for this particular one and also the
private ones, however we used in the past to load all byte-* variables
for a reason I don't remember, IMO it is not needed anymore to load byte-* vars
private or not, so I removed this.

> Maybe any variables with "--" in their symbols should be filtered out
> of the copying operation.

Name space problem, byte-* vars and functions have no fixed prefix (not
always).
 
>
>
> Also, is printing followed by reading really the best way to get a copy
> of a Lisp object?  There are several Lisp types, possibly most of them,
> whose printed representation can't be read; things like buffers,
> windows, processes, .....  I think only the really common objects, like
> lists, vectors, strings, numbers have readable printed representations.
>
>> Thierry has removed that variable from the list of handled variables for
>> now which circumvents the problem.
>
> I've had at least two Cc:s to Thierry at <thierry.volpiatto@gmail.com>
> bounce with "address does not exist".  Do you know if Thierry has
> changed his address, or could it just be Google playing more silly
> games?

My fault I didn't update my mail address in async source files, will do
soon, sorry for this.

Thanks.

-- 
Thierry

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Alan Mackenzie]

Hello, Thierry.

On Fri, Mar 18, 2022 at 13:37:43 +0000, Thierry Volpiatto wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Michael.

> > On Fri, Mar 18, 2022 at 00:50:17 +0100, Michael Heerdegen wrote:
> >> Alan Mackenzie <acm@muc.de> writes:

> >> > Could you please give a recipe for this situation, preferably starting
> >> > from emacs -Q?  I don't know how to use async-bytecomp.el, for
> >> > example.

> >> Here is the issue:

> >> https://github.com/jwiegley/emacs-async/issues/153

> > Thanks.  I've read it, now.

> >> But I don't think async or async-bytecomp is related to this issue at
> >> all.  It only hits an issue with vanilla Emacs.  I think it's not
> >> necessary for you to have a look at async, unless you are interested,
> >> also because I have a recipe for emacs -Q:

> >> Byte compile lisp/calendar/appt.el (I did that with B from dired); after
> >> that:

> >> byte-optimize--dynamic-vars ->
> >> (#<symbol original-date at 24141> #<symbol number at 24125>)

> >> The problem for async-bytecomp is then only that a result like this is
> >> not `read'able after being printed.

> > I made a mistake in my last post.  There's actually no reason
> > byte-optimize--dynamic-vars shouldn't contain symbols with position.
> > That's what it accumulates.  It is also meant to be a private variable
> > in the byte compilation (hence the "--" in its name).

> > Why is async-bytecomp copying byte-optimize--dynamic-vars?  There seems
> > to be no reason.

> You are right, there is no reason for this particular one and also the
> private ones, however we used in the past to load all byte-* variables
> for a reason I don't remember, IMO it is not needed anymore to load byte-* vars
> private or not, so I removed this.

I think that all the byte-* variables are connected with byte
compilation.  But I also think that they are (?nearly?) all internal
variables, so it shouldn't be needed to copy them to an aynchronous
thread (?or process?).

> > Maybe any variables with "--" in their symbols should be filtered out
> > of the copying operation.

> Name space problem, byte-* vars and functions have no fixed prefix (not
> always).
 
OK.  I think you're saying that there are other variables with "--" that
you _do_ need to copy.

> > Also, is printing followed by reading really the best way to get a copy
> > of a Lisp object?  There are several Lisp types, possibly most of them,
> > whose printed representation can't be read; things like buffers,
> > windows, processes, .....  I think only the really common objects, like
> > lists, vectors, strings, numbers have readable printed representations.

> >> Thierry has removed that variable from the list of handled variables for
> >> now which circumvents the problem.

> > I've had at least two Cc:s to Thierry at <thierry.volpiatto@gmail.com>
> > bounce with "address does not exist".  Do you know if Thierry has
> > changed his address, or could it just be Google playing more silly
> > games?

> My fault I didn't update my mail address in async source files, will do
> soon, sorry for this.

Well, we're back in communication again.  :-)  I've heard good things
about posteo.

> Thanks.

> -- 
> Thierry

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Thierry Volpiatto]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]


Hello Alan,

Alan Mackenzie <acm@muc.de> writes:

> I think that all the byte-* variables are connected with byte
> compilation.  But I also think that they are (?nearly?) all internal
> variables, so it shouldn't be needed to copy them to an aynchronous
> thread (?or process?).

Thanks to confirm this.

>> > Maybe any variables with "--" in their symbols should be filtered out
>> > of the copying operation.
>
>> Name space problem, byte-* vars and functions have no fixed prefix (not
>> always).
>  
> OK.  I think you're saying that there are other variables with "--" that
> you _do_ need to copy.

No, not anymore. Wanted to say that if needed it is harder to catch foo--,
foo-this--, foo-else-this-- etc... instead of just foo--.

> Well, we're back in communication again.  :-)  I've heard good things
> about posteo.

Yes, works great :-)

Thanks.

-- 
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 686 bytes --]

bug#54433: 29.0.50; Invalid read syntax: "#<" with async-bytecomp

[Michael Heerdegen]

Thierry Volpiatto <thievol@posteo.net> writes:

> > I think that all the byte-* variables are connected with byte
> > compilation.  But I also think that they are (?nearly?) all internal
> > variables, so it shouldn't be needed to copy them to an aynchronous
> > thread (?or process?).
>
> Thanks to confirm this.

I can confirm that the problem has been fixed well in the "async"
package, and we have found that everything in Emacs is working as
expected.  So I'm closing this report.  Thanks to everybody involved!

Michael.