Introducing 'safety' compilation parameter

Introducing 'safety' compilation parameter

[Andrea Corallo]

Hello,

I've put in scratch/comp-safety a branch wich introduces 'safety' as
compilation parameter.

'safety' can be used similarly to 'native-comp-speed' both as a global
variable to influence compilation both as a function declaration.

'safety' justification of existence is ATM being able to control the
undefined behaviour being created when function type declaration added
by the user is not correct.

ATM we can have two values:

1 Emitted code is generated in a safe matter even if function types are
miss-declared.
0 Emitted code can misbehave or crash Emacs if function declarations are
not correct and the function is native compiled (@pxref{Native
Compilation}).

1 is ATM the default.

I didn't want to give safety a prefix (byte- or comp-) as I believe we
should extend safety in the future with a value to have the byte
compiler generates runtime type checks to verify the declred types.
OTOH this is creating a warning for a missing prefix, not sure what's
the best way to fix this (give it a prefix or silence the compiler if
possible).

Also I added some doc for the declaration, but didn't kwnow where in the
manual the documention for the variable should go as now it has effect
only for the native compiler.  Should I document it under
"Native-Compilation Variables" for now?

Thanks

  Andrea

Re: Introducing 'safety' compilation parameter

[Alan Mackenzie]

Hello, Andrea.

On Tue, May 07, 2024 at 06:37:50 -0400, Andrea Corallo wrote:
> Hello,

> I've put in scratch/comp-safety a branch which introduces 'safety' as
> compilation parameter.

I'm against this; see below.

> 'safety' can be used similarly to 'native-comp-speed' both as a global
> variable to influence compilation both as a function declaration.

> 'safety' justification of existence is ATM being able to control the
> undefined behaviour being created when function type declaration added
> by the user is not correct.

How is this undefined behaviour?  Lisp is a dynamically typed language.
If a function type declaration is added, this can only be advisory.  If
it is incorrect, then by the specification of (Emacs) Lisp, the code
generated must still correctly handle its arguments and run.

> ATM we can have two values:

> 1 Emitted code is generated in a safe matter even if function types are
> miss-declared.

I'm in favour of this remaining.

> 0 Emitted code can misbehave or crash Emacs if function declarations are
> not correct and the function is native compiled (@pxref{Native
> Compilation}).

I'm in favour of this not being merged.  Crashing Emacs for valid (or
even invalid) Lisp should not be an option.

> 1 is ATM the default.

I see this change as one more boil-the-frog-slowly step towards turning
Emacs Lisp into a statically typed language.  If this change gets merged
into master, how long will it be until these function declarations
become first common, then usual, then all but universal?  It that were
to happen, it would be, in my opinion, a net loss to Emacs.

Why do we need this change at all?  It may generate somewhat faster (or
smaller) code, but isn't native compiled Lisp already fast enough,
particularly on a modern machine?

> I didn't want to give safety a prefix (byte- or comp-) as I believe we
> should extend safety in the future with a value to have the byte
> compiler generates runtime type checks to verify the declred types.
> OTOH this is creating a warning for a missing prefix, not sure what's
> the best way to fix this (give it a prefix or silence the compiler if
> possible).

> Also I added some doc for the declaration, but didn't kwnow where in the
> manual the documention for the variable should go as now it has effect
> only for the native compiler.  Should I document it under
> "Native-Compilation Variables" for now?

> Thanks

>   Andrea

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Introducing 'safety' compilation parameter

[Eli Zaretskii]

> Date: Tue, 7 May 2024 12:01:33 +0000
> Cc: emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>,
>   monnier@iro.umontreal.ca, mattias.engdegard@gmail.com,
>   stefankangas@gmail.com
> From: Alan Mackenzie <acm@muc.de>
> 
> I see this change as one more boil-the-frog-slowly step towards turning
> Emacs Lisp into a statically typed language.

Alan, please be kinder, even if you dislike very much suggestions of
others.  The above could have been easily rephrased as

  Emacs Lisp should not be turned into a statically typed language.

without losing any useful content, including your strenuous objection
to the change.

Re: Introducing 'safety' compilation parameter

[Stefan Monnier]

> I didn't want to give safety a prefix (byte- or comp-) as I believe we
> should extend safety in the future with a value to have the byte
> compiler generates runtime type checks to verify the declred types.

I don't see a strong need for a namespace prefix, but I think "safety"
is too general a term.  It could be understood to apply also to
unrelated circumstances such as process confinement or disclosure of
sensitive info over a network connection.
Maybe `elisp-safety` or `compilation-safety`?

If I read your patch correctly, we get safety in a roundabout way:
rather than test the safety setting at the spot where we decide to apply
(or not) a dangerous optimization, we use the safety setting to
"mollify" the type annotations (presumably because this will in turn
make the risky optimizations inapplicable).

What is the reason for that?


        Stefan

Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Alan Mackenzie]

Hello, Eli.

On Tue, May 07, 2024 at 16:15:47 +0300, Eli Zaretskii wrote:
> > Date: Tue, 7 May 2024 12:01:33 +0000
> > Cc: emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>,
> >   monnier@iro.umontreal.ca, mattias.engdegard@gmail.com,
> >   stefankangas@gmail.com
> > From: Alan Mackenzie <acm@muc.de>

> > I see this change as one more boil-the-frog-slowly step towards turning
> > Emacs Lisp into a statically typed language.

> Alan, please be kinder, even if you dislike very much suggestions of
> others.

No offence was intended.

> The above could have been easily rephrased as

>   Emacs Lisp should not be turned into a statically typed language.

> without losing any useful content, ....

Not really - what would have been lost is the equivalent  of ".... and I
see this process happening at the moment.".  The frog metaphor was an
economical way of phrasing this.  Again, I'm sorry it caused offence.

> .... including your strenuous objection to the change.

I see Emacs Lisp steadily drifting towards being statically typed, and I
don't think that's a good thing.  As far as I'm aware, there has been no
general agreement amongst Emacs developers for this (unless it's
happened as a side-thread in some thread without having an accurate
Subject:).

We currently have the prospect of lots of functions being cluttered up
with "type" declarations.  We already have meaningless (to a Lisp
programmer) things like:

    Inferred type: (function (&optional t t) t)                                                                            

appearing in prominent positions in doc strings.  Why?

If this is the way Emacs Lisp is to develop, can't we at least have an
open discussion about it and a positive decision taken, rather than
letting it "just happen"?  As is already clear, I see static typing in
Emacs Lisp, except, perhaps, on a very limited scale, as a Bad Thing.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I didn't want to give safety a prefix (byte- or comp-) as I believe we
>> should extend safety in the future with a value to have the byte
>> compiler generates runtime type checks to verify the declred types.
>
> I don't see a strong need for a namespace prefix, but I think "safety"
> is too general a term.  It could be understood to apply also to
> unrelated circumstances such as process confinement or disclosure of
> sensitive info over a network connection.
> Maybe `elisp-safety` or `compilation-safety`?

Agree, boths SGTM, maaaybe I slightly prefer 'compilation-safety'.

> If I read your patch correctly, we get safety in a roundabout way:
> rather than test the safety setting at the spot where we decide to apply
> (or not) a dangerous optimization, we use the safety setting to
> "mollify" the type annotations (presumably because this will in turn
> make the risky optimizations inapplicable).
>
> What is the reason for that?

No specific reason, I think I thought was safer to write the patch with
way gating optimization at this (higher) level.  Thinking about now that
you raised the point, is probably easy to gate the single optimizations
as well.  Why not, I think I'll update the patch this way.

Thanks

  Andrea

Re: Introducing 'safety' compilation parameter

[Tomas Hlavaty]

On Tue 07 May 2024 at 12:01, Alan Mackenzie <acm@muc.de> wrote:
> turning
> Emacs Lisp into a statically typed language.  If this change gets merged
> into master, how long will it be until these function declarations
> become first common, then usual, then all but universal?  It that were
> to happen, it would be, in my opinion, a net loss to Emacs.

this looks like the plan:

     id:86sf0q5l0v.fsf@gnu.org
     Eli Zaretskii <eliz@gnu.org> (March 16)
     Subject: Re: Declaring Lisp function types
     To: Tomas Hlavaty <tom@logand.com>
     Cc: acorallo@gnu.org, emacs-devel@gnu.org, stefankangas@gmail.com
     Date: Sat, 16 Mar 2024 16:06:08 +0200

     [...]
     By contrast, _every_ function will have some type and some
     signature,
     so you will have a gazillion of hits if you search for those,
     no
     matter what its name.
     [...]

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Andrea.
>
> On Tue, May 07, 2024 at 06:37:50 -0400, Andrea Corallo wrote:
>> Hello,
>
>> I've put in scratch/comp-safety a branch which introduces 'safety' as
>> compilation parameter.
>
> I'm against this; see below.
>
>> 'safety' can be used similarly to 'native-comp-speed' both as a global
>> variable to influence compilation both as a function declaration.
>
>> 'safety' justification of existence is ATM being able to control the
>> undefined behaviour being created when function type declaration added
>> by the user is not correct.
>
> How is this undefined behaviour?  Lisp is a dynamically typed language.
> If a function type declaration is added, this can only be advisory.  If
> it is incorrect, then by the specification of (Emacs) Lisp, the code
> generated must still correctly handle its arguments and run.
>
>> ATM we can have two values:
>
>> 1 Emitted code is generated in a safe matter even if function types are
>> miss-declared.
>
> I'm in favour of this remaining.
>
>> 0 Emitted code can misbehave or crash Emacs if function declarations are
>> not correct and the function is native compiled (@pxref{Native
>> Compilation}).
>
> I'm in favour of this not being merged.  Crashing Emacs for valid (or
> even invalid) Lisp should not be an option.
>
>> 1 is ATM the default.
>
> I see this change as one more boil-the-frog-slowly step towards turning
> Emacs Lisp into a statically typed language.  If this change gets merged
> into master, how long will it be until these function declarations
> become first common, then usual, then all but universal?  It that were
> to happen, it would be, in my opinion, a net loss to Emacs.

Hi Alan,

I'm sorry I was not clear presenting this patch.  The scope of this is
not to introduce UB but actually to *remove* it.

Anyway your concern of having in few years code full of type declaration
declaration is IMO just not realistic.  Common Lisp has function type
declarations since forever and type declared functions are very rare in
CL codebases.

This is a tool that can be useful for specific usages and not for the
general case.

Thanks

  Andrea

Re: Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Andrea Corallo]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Eli.
>
> On Tue, May 07, 2024 at 16:15:47 +0300, Eli Zaretskii wrote:
>> > Date: Tue, 7 May 2024 12:01:33 +0000
>> > Cc: emacs-devel@gnu.org, Eli Zaretskii <eliz@gnu.org>,
>> >   monnier@iro.umontreal.ca, mattias.engdegard@gmail.com,
>> >   stefankangas@gmail.com
>> > From: Alan Mackenzie <acm@muc.de>
>
>> > I see this change as one more boil-the-frog-slowly step towards turning
>> > Emacs Lisp into a statically typed language.
>
>> Alan, please be kinder, even if you dislike very much suggestions of
>> others.
>
> No offence was intended.
>
>> The above could have been easily rephrased as
>
>>   Emacs Lisp should not be turned into a statically typed language.
>
>> without losing any useful content, ....
>
> Not really - what would have been lost is the equivalent  of ".... and I
> see this process happening at the moment.".  The frog metaphor was an
> economical way of phrasing this.  Again, I'm sorry it caused offence.
>
>> .... including your strenuous objection to the change.
>
> I see Emacs Lisp steadily drifting towards being statically typed, and I
> don't think that's a good thing.  As far as I'm aware, there has been no
> general agreement amongst Emacs developers for this (unless it's
> happened as a side-thread in some thread without having an accurate
> Subject:).
>
> We currently have the prospect of lots of functions being cluttered up
> with "type" declarations.  We already have meaningless (to a Lisp
> programmer) things like:
>
>     Inferred type: (function (&optional t t) t)                                                                            
>
> appearing in prominent positions in doc strings.  Why?

Because it can be informative (when there are types other than t),
anyway you can control it with 'help-display-function-type' if you don't
want to have it in the *Help* buffer.

> If this is the way Emacs Lisp is to develop, can't we at least have an
> open discussion about it and a positive decision taken, rather than
> letting it "just happen"?  As is already clear, I see static typing in
> Emacs Lisp, except, perhaps, on a very limited scale, as a Bad Thing.

Again, there's no plan to make elisp statically typed.  I agree with you
this is a tool to be used on a limited scale, if this is not clear
enough from the doc now I guess we can just improve it.

  Andrea

Re: Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Eli Zaretskii]

> Date: Tue, 7 May 2024 16:07:58 +0000
> Cc: acorallo@gnu.org, emacs-devel@gnu.org, monnier@iro.umontreal.ca,
>   mattias.engdegard@gmail.com, stefankangas@gmail.com
> From: Alan Mackenzie <acm@muc.de>
> 
> > > I see this change as one more boil-the-frog-slowly step towards turning
> > > Emacs Lisp into a statically typed language.
> 
> > Alan, please be kinder, even if you dislike very much suggestions of
> > others.
> 
> No offence was intended.

Suggest to re-read what you are going to send, with an eye towards
stuff that doesn't help you make your point, only to vent anger, and
delete any of that.

> If this is the way Emacs Lisp is to develop, can't we at least have an
> open discussion about it and a positive decision taken, rather than
> letting it "just happen"?

We are discussing each of these changes, so the opportunity for
raising a broader issues already exists, and you and others are
welcome to voice their opinions.  It isn't like Andrea is doing this
without any discussions or without incorporating all the comments and
suggestions that people come up with.

Re: Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Emanuel Berg]

Alan Mackenzie wrote:

> We currently have the prospect of lots of functions being
> cluttered up with "type" declarations. We already have
> meaningless (to a Lisp programmer) things like:
>
>     Inferred type: (function (&optional t t) t)
>
> appearing in prominent positions in doc strings. Why?

In the help, not doc strings, but you know that.

Anyway I have this to reduce the verbosity of the help, maybe
the same can be set up/used for the inferred type note for
people who don't like it to show up?

(setq help-fns-describe-variable-functions
  (cl-set-difference help-fns-describe-variable-functions
    '(help-fns--customize-variable
      help-fns--customize-variable-version
      help-fns--mention-first-release
      help-fns--var-file-local
      help-fns--var-ignored-local
      help-fns--var-risky
      help-fns--var-safe-local) ))

> If this is the way Emacs Lisp is to develop, can't we at
> least have an open discussion about it and a positive
> decision taken, rather than letting it "just happen"? As is
> already clear, I see static typing in Emacs Lisp, except,
> perhaps, on a very limited scale, as a Bad Thing.

Maybe we can have both and then everyone could use it if and
where they wanted, and we would see if and to what extent it
would be used and what implications it would have, while still
being optional.

Maybe in some areas it would be used more often?

-- 
underground experts united
https://dataswamp.org/~incal

Re: Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Emanuel Berg]

Andrea Corallo wrote:

> Because it can be informative (when there are types other
> than t), anyway you can control it with
> 'help-display-function-type' if you don't want to have it in
> the *Help* buffer.

Ah, so you already added what I asked for/suggested in my
other post. Okay good, all the better then.

>> If this is the way Emacs Lisp is to develop, can't we at
>> least have an open discussion about it and a positive
>> decision taken, rather than letting it "just happen"? As is
>> already clear, I see static typing in Emacs Lisp, except,
>> perhaps, on a very limited scale, as a Bad Thing.
>
> Again, there's no plan to make elisp statically typed.
> I agree with you this is a tool to be used on a limited
> scale, if this is not clear enough from the doc now I guess
> we can just improve it.

I think it is cool with the inferred type system, as well as
the possibility to have explicit types for functions,
arguments, etc, in source.

Those types could be used in many different ways at compile as
well as run time, that doesn't mean we must "make Elisp
statically typed".

-- 
underground experts united
https://dataswamp.org/~incal

Re: Introducing 'safety' compilation parameter

[Emanuel Berg]

Andrea Corallo wrote:

> Anyway your concern of having in few years code full of type
> declaration declaration is IMO just not realistic.
> Common Lisp has function type declarations since forever and
> type declared functions are very rare in CL codebases.
>
> This is a tool that can be useful for specific usages and
> not for the general case.

Very much so, it will also be interesting to see what specific
usages those will be!

-- 
underground experts united
https://dataswamp.org/~incal

Re: Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Emanuel Berg]

Eli Zaretskii wrote:

>>> Alan, please be kinder, even if you dislike very much
>>> suggestions of others.
>> 
>> No offence was intended.
>
> Suggest to re-read what you are going to send, with an eye
> towards stuff that doesn't help you make your point, only to
> vent anger, and delete any of that.

No one has been angry in this thread so far, but if we keep up
the meta discussion that can change for that very reason.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Introducing 'safety' compilation parameter

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
>  mattias.engdegard@gmail.com, stefankangas@gmail.com
> Date: Tue, 07 May 2024 06:37:50 -0400
> 
> I've put in scratch/comp-safety a branch wich introduces 'safety' as
> compilation parameter.
> 
> 'safety' can be used similarly to 'native-comp-speed' both as a global
> variable to influence compilation both as a function declaration.
> 
> 'safety' justification of existence is ATM being able to control the
> undefined behaviour being created when function type declaration added
> by the user is not correct.
> 
> ATM we can have two values:
> 
> 1 Emitted code is generated in a safe matter even if function types are
> miss-declared.
> 0 Emitted code can misbehave or crash Emacs if function declarations are
> not correct and the function is native compiled (@pxref{Native
> Compilation}).
> 
> 1 is ATM the default.

Is this for Emacs 30 or Emacs 31?

If the former, we should conclude this discussion soon enough.

I don't have an opinion about the utility of this for Emacs Lisp, but
this part of the documentation:

> +Possible values are:
> +
> +0 Emitted code can misbehave or crash Emacs if function declarations are
> +not correct and the function is native compiled (@pxref{Native
> +Compilation}).
> +
> +1 Emitted code is generated in a safe matter even if function are
> +miss-declared."

should be formatted as a @table.

> Also I added some doc for the declaration, but didn't kwnow where in the
> manual the documention for the variable should go as now it has effect
> only for the native compiler.  Should I document it under
> "Native-Compilation Variables" for now?

Yes, I think so.

Re: Introducing 'safety' compilation parameter

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

> I've put in scratch/comp-safety a branch wich introduces 'safety' as
> compilation parameter.
>
> 'safety' can be used similarly to 'native-comp-speed' both as a global
> variable to influence compilation both as a function declaration.
>
> 'safety' justification of existence is ATM being able to control the
> undefined behaviour being created when function type declaration added
> by the user is not correct.
>
> ATM we can have two values:
>
> 1 Emitted code is generated in a safe matter even if function types are
> miss-declared.
> 0 Emitted code can misbehave or crash Emacs if function declarations are
> not correct and the function is native compiled (@pxref{Native
> Compilation}).
>
> 1 is ATM the default.
>
> I didn't want to give safety a prefix (byte- or comp-) as I believe we
> should extend safety in the future with a value to have the byte
> compiler generates runtime type checks to verify the declred types.
> OTOH this is creating a warning for a missing prefix, not sure what's
> the best way to fix this (give it a prefix or silence the compiler if
> possible).
>
> Also I added some doc for the declaration, but didn't kwnow where in the
> manual the documention for the variable should go as now it has effect
> only for the native compiler.  Should I document it under
> "Native-Compilation Variables" for now?

Hi Andrea,

I don't know if something like this has been discussed before... CMUCL
and SBCL treat declarations differently depending on the speed settings.
If speed is below some threshold, declarations work like assertions,
with high speed settings declarations are treated as promises.

Re: Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   Emacs Lisp should not be turned into a statically typed language.

  > > without losing any useful content, ....

  > Not really - what would have been lost is the equivalent  of ".... and I
  > see this process happening at the moment.".  The frog metaphor was an
  > economical way of phrasing this.  Again, I'm sorry it caused offence.

I worry about this too.

This is not the first time that Lisp compilation has inspired adding
features to declare types of various values that apear inside Lisp
functions.  For instance, that happened with Maclisp in the 1970s.

That pressure is not necessarily overwhelming; it does not necessarily
pressure programmers to declare types in all their code.  When I wrote
Lisp code in the 1970s, I did not declare types.  I was not required
to do so.

But what worries e is that users who want the highest speed
will put pressure on all Emacs developers to add type declaratons
everywhere to give those users the highest possible speed.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Drifting towards a statically typed Emacs Lisp. [Was: Introducing 'safety' compilation parameter]

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 09 May 2024 19:52:12 -0400
> 
> But what worries me is that users who want the highest speed
> will put pressure on all Emacs developers to add type declaratons
> everywhere to give those users the highest possible speed.

The onus will be on the Emacs maintainers to resist that pressure.  I
don't see how that pressure is any different from any other pressure
put on us about any other kind of changes.  It's part of our job to
deal with that, and deal correctly.

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
>>  mattias.engdegard@gmail.com, stefankangas@gmail.com
>> Date: Tue, 07 May 2024 06:37:50 -0400
>> 
>> I've put in scratch/comp-safety a branch wich introduces 'safety' as
>> compilation parameter.
>> 
>> 'safety' can be used similarly to 'native-comp-speed' both as a global
>> variable to influence compilation both as a function declaration.
>> 
>> 'safety' justification of existence is ATM being able to control the
>> undefined behaviour being created when function type declaration added
>> by the user is not correct.
>> 
>> ATM we can have two values:
>> 
>> 1 Emitted code is generated in a safe matter even if function types are
>> miss-declared.
>> 0 Emitted code can misbehave or crash Emacs if function declarations are
>> not correct and the function is native compiled (@pxref{Native
>> Compilation}).
>> 
>> 1 is ATM the default.
>
> Is this for Emacs 30 or Emacs 31?

Hi Eli,

the question is related to the patch or the defaul value?

> If the former, we should conclude this discussion soon enough.
>
> I don't have an opinion about the utility of this for Emacs Lisp, but
> this part of the documentation:
>
>> +Possible values are:
>> +
>> +0 Emitted code can misbehave or crash Emacs if function declarations are
>> +not correct and the function is native compiled (@pxref{Native
>> +Compilation}).
>> +
>> +1 Emitted code is generated in a safe matter even if function are
>> +miss-declared."
>
> should be formatted as a @table.

Thx will do.

  Andrea

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>> I've put in scratch/comp-safety a branch wich introduces 'safety' as
>> compilation parameter.
>>
>> 'safety' can be used similarly to 'native-comp-speed' both as a global
>> variable to influence compilation both as a function declaration.
>>
>> 'safety' justification of existence is ATM being able to control the
>> undefined behaviour being created when function type declaration added
>> by the user is not correct.
>>
>> ATM we can have two values:
>>
>> 1 Emitted code is generated in a safe matter even if function types are
>> miss-declared.
>> 0 Emitted code can misbehave or crash Emacs if function declarations are
>> not correct and the function is native compiled (@pxref{Native
>> Compilation}).
>>
>> 1 is ATM the default.
>>
>> I didn't want to give safety a prefix (byte- or comp-) as I believe we
>> should extend safety in the future with a value to have the byte
>> compiler generates runtime type checks to verify the declred types.
>> OTOH this is creating a warning for a missing prefix, not sure what's
>> the best way to fix this (give it a prefix or silence the compiler if
>> possible).
>>
>> Also I added some doc for the declaration, but didn't kwnow where in the
>> manual the documention for the variable should go as now it has effect
>> only for the native compiler.  Should I document it under
>> "Native-Compilation Variables" for now?
>
> Hi Andrea,
>
> I don't know if something like this has been discussed before... CMUCL
> and SBCL treat declarations differently depending on the speed settings.
> If speed is below some threshold, declarations work like assertions,
> with high speed settings declarations are treated as promises.

I though looking at [1] only 'safety' was involved in declarations
working like assertions.

After your hint I see now SBCL does that when:

(or (>=safety 2) (>= safety speed 1)) [2]

That's complex/confusing 🤯!  I think would be better to control that
only with 'safety' no?

Thanks

  Andrea


[1] <https://www.lispworks.com/documentation/lw61/CLHS/Body/d_optimi.htm>
[2] <https://www.sbcl.org/manual/#Declarations-as-Assertions>

Re: Introducing 'safety' compilation parameter

[Gerd Möllmann]

Andrea Corallo <acorallo@gnu.org> writes:

>> I don't know if something like this has been discussed before... CMUCL
>> and SBCL treat declarations differently depending on the speed settings.
>> If speed is below some threshold, declarations work like assertions,
>> with high speed settings declarations are treated as promises.
>
> I though looking at [1] only 'safety' was involved in declarations
> working like assertions.
>
> After your hint I see now SBCL does that when:
>
> (or (>=safety 2) (>= safety speed 1)) [2]
>
> That's complex/confusing 🤯!  I think would be better to control that
> only with 'safety' no?

You could read it as "either I the programmer value safety very much, or
I value safety, and I value safety at least as much as speed".

Anyway, I just wanted to hint at the "declarations as
assertions/promises", and since you know about it, mission accomplished
:-).

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Andrea Corallo <acorallo@gnu.org> writes:
>
>>> I don't know if something like this has been discussed before... CMUCL
>>> and SBCL treat declarations differently depending on the speed settings.
>>> If speed is below some threshold, declarations work like assertions,
>>> with high speed settings declarations are treated as promises.
>>
>> I though looking at [1] only 'safety' was involved in declarations
>> working like assertions.
>>
>> After your hint I see now SBCL does that when:
>>
>> (or (>=safety 2) (>= safety speed 1)) [2]
>>
>> That's complex/confusing 🤯!  I think would be better to control that
>> only with 'safety' no?
>
> You could read it as "either I the programmer value safety very much, or
> I value safety, and I value safety at least as much as speed".

Right, sounds much better in words actually! :)

> Anyway, I just wanted to hint at the "declarations as
> assertions/promises", and since you know about it, mission accomplished
> :-).

Thanks

Re: Introducing 'safety' compilation parameter

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: emacs-devel@gnu.org,  monnier@iro.umontreal.ca,
>   mattias.engdegard@gmail.com,  stefankangas@gmail.com
> Date: Fri, 10 May 2024 03:47:53 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andrea Corallo <acorallo@gnu.org>
> >> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
> >>  mattias.engdegard@gmail.com, stefankangas@gmail.com
> >> Date: Tue, 07 May 2024 06:37:50 -0400
> >> 
> >> I've put in scratch/comp-safety a branch wich introduces 'safety' as
> >> compilation parameter.
> >> 
> >> 'safety' can be used similarly to 'native-comp-speed' both as a global
> >> variable to influence compilation both as a function declaration.
> >> 
> >> 'safety' justification of existence is ATM being able to control the
> >> undefined behaviour being created when function type declaration added
> >> by the user is not correct.
> >> 
> >> ATM we can have two values:
> >> 
> >> 1 Emitted code is generated in a safe matter even if function types are
> >> miss-declared.
> >> 0 Emitted code can misbehave or crash Emacs if function declarations are
> >> not correct and the function is native compiled (@pxref{Native
> >> Compilation}).
> >> 
> >> 1 is ATM the default.
> >
> > Is this for Emacs 30 or Emacs 31?
> 
> Hi Eli,
> 
> the question is related to the patch or the defaul value?

Both.

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: emacs-devel@gnu.org,  monnier@iro.umontreal.ca,
>>   mattias.engdegard@gmail.com,  stefankangas@gmail.com
>> Date: Fri, 10 May 2024 03:47:53 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Andrea Corallo <acorallo@gnu.org>
>> >> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
>> >>  mattias.engdegard@gmail.com, stefankangas@gmail.com
>> >> Date: Tue, 07 May 2024 06:37:50 -0400
>> >> 
>> >> I've put in scratch/comp-safety a branch wich introduces 'safety' as
>> >> compilation parameter.
>> >> 
>> >> 'safety' can be used similarly to 'native-comp-speed' both as a global
>> >> variable to influence compilation both as a function declaration.
>> >> 
>> >> 'safety' justification of existence is ATM being able to control the
>> >> undefined behaviour being created when function type declaration added
>> >> by the user is not correct.
>> >> 
>> >> ATM we can have two values:
>> >> 
>> >> 1 Emitted code is generated in a safe matter even if function types are
>> >> miss-declared.
>> >> 0 Emitted code can misbehave or crash Emacs if function declarations are
>> >> not correct and the function is native compiled (@pxref{Native
>> >> Compilation}).
>> >> 
>> >> 1 is ATM the default.
>> >
>> > Is this for Emacs 30 or Emacs 31?
>> 
>> Hi Eli,
>> 
>> the question is related to the patch or the defaul value?
>
> Both.

ATM the default is the equivalent to the proposed 0 (and so it was since
the introduction of native compilation).

The proposal is for 30 but if it's too late it's for 31 :)

Thanks

  Andrea

Re: Introducing 'safety' compilation parameter

[Stefan Monnier]

> After your hint I see now SBCL does that when:
>
> (or (>=safety 2) (>= safety speed 1)) [2]

FWIW, I always thought that having both `safety` and `speed` was a form
of overengineering which doesn't serve the users.


        Stefan

Re: Introducing 'safety' compilation parameter

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: emacs-devel@gnu.org,  monnier@iro.umontreal.ca,
>   mattias.engdegard@gmail.com,  stefankangas@gmail.com
> Date: Fri, 10 May 2024 07:42:16 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Is this for Emacs 30 or Emacs 31?
> >> 
> >> Hi Eli,
> >> 
> >> the question is related to the patch or the defaul value?
> >
> > Both.
> 
> ATM the default is the equivalent to the proposed 0 (and so it was since
> the introduction of native compilation).
> 
> The proposal is for 30 but if it's too late it's for 31 :)

I didn't (yet) say it was too late, I just said that if we want this
in Emacs 30, we need top conclude this discussion soon enough.

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> I didn't want to give safety a prefix (byte- or comp-) as I believe we
>>> should extend safety in the future with a value to have the byte
>>> compiler generates runtime type checks to verify the declred types.
>>
>> I don't see a strong need for a namespace prefix, but I think "safety"
>> is too general a term.  It could be understood to apply also to
>> unrelated circumstances such as process confinement or disclosure of
>> sensitive info over a network connection.
>> Maybe `elisp-safety` or `compilation-safety`?
>
> Agree, boths SGTM, maaaybe I slightly prefer 'compilation-safety'.
>
>> If I read your patch correctly, we get safety in a roundabout way:
>> rather than test the safety setting at the spot where we decide to apply
>> (or not) a dangerous optimization, we use the safety setting to
>> "mollify" the type annotations (presumably because this will in turn
>> make the risky optimizations inapplicable).
>>
>> What is the reason for that?
>
> No specific reason, I think I thought was safer to write the patch with
> way gating optimization at this (higher) level.  Thinking about now that
> you raised the point, is probably easy to gate the single optimizations
> as well.  Why not, I think I'll update the patch this way.

Okay I pushed scratch/comp-safety2 changing the knob name to
'compilation-safety' and gating the dangerous optimizations as
discussed.

Thanks

  Andrea

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
>>  mattias.engdegard@gmail.com, stefankangas@gmail.com
>> Date: Tue, 07 May 2024 06:37:50 -0400
>> 
>> I've put in scratch/comp-safety a branch wich introduces 'safety' as
>> compilation parameter.
>> 
>> 'safety' can be used similarly to 'native-comp-speed' both as a global
>> variable to influence compilation both as a function declaration.
>> 
>> 'safety' justification of existence is ATM being able to control the
>> undefined behaviour being created when function type declaration added
>> by the user is not correct.
>> 
>> ATM we can have two values:
>> 
>> 1 Emitted code is generated in a safe matter even if function types are
>> miss-declared.
>> 0 Emitted code can misbehave or crash Emacs if function declarations are
>> not correct and the function is native compiled (@pxref{Native
>> Compilation}).
>> 
>> 1 is ATM the default.
>
> Is this for Emacs 30 or Emacs 31?
>
> If the former, we should conclude this discussion soon enough.
>
> I don't have an opinion about the utility of this for Emacs Lisp, but
> this part of the documentation:
>
>> +Possible values are:
>> +
>> +0 Emitted code can misbehave or crash Emacs if function declarations are
>> +not correct and the function is native compiled (@pxref{Native
>> +Compilation}).
>> +
>> +1 Emitted code is generated in a safe matter even if function are
>> +miss-declared."
>
> should be formatted as a @table.

Okay, I moved that part to doc/lispref/compile.texi to document in
'compilation-safety' and added the @table for the possible values.
Still in functions.texi we keep a reference to it.

Thanks

  Andrea

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Andrea Corallo <acorallo@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <acorallo@gnu.org>
>>> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
>>>  mattias.engdegard@gmail.com, stefankangas@gmail.com
>>> Date: Tue, 07 May 2024 06:37:50 -0400
>>> 
>>> I've put in scratch/comp-safety a branch wich introduces 'safety' as
>>> compilation parameter.
>>> 
>>> 'safety' can be used similarly to 'native-comp-speed' both as a global
>>> variable to influence compilation both as a function declaration.
>>> 
>>> 'safety' justification of existence is ATM being able to control the
>>> undefined behaviour being created when function type declaration added
>>> by the user is not correct.
>>> 
>>> ATM we can have two values:
>>> 
>>> 1 Emitted code is generated in a safe matter even if function types are
>>> miss-declared.
>>> 0 Emitted code can misbehave or crash Emacs if function declarations are
>>> not correct and the function is native compiled (@pxref{Native
>>> Compilation}).
>>> 
>>> 1 is ATM the default.
>>
>> Is this for Emacs 30 or Emacs 31?
>>
>> If the former, we should conclude this discussion soon enough.
>>
>> I don't have an opinion about the utility of this for Emacs Lisp, but
>> this part of the documentation:
>>
>>> +Possible values are:
>>> +
>>> +0 Emitted code can misbehave or crash Emacs if function declarations are
>>> +not correct and the function is native compiled (@pxref{Native
>>> +Compilation}).
>>> +
>>> +1 Emitted code is generated in a safe matter even if function are
>>> +miss-declared."
>>
>> should be formatted as a @table.
>
> Okay, I moved that part to doc/lispref/compile.texi to document in
> 'compilation-safety' and added the @table for the possible values.
> Still in functions.texi we keep a reference to it.

Sorry forgot to mention that the new code is in scratch/comp-safety2
(I'll do some GC at the end but wanted to keep both approaches for now).

Thanks

  Andrea

Re: Introducing 'safety' compilation parameter

[Stefan Monnier]

> Okay I pushed scratch/comp-safety2 changing the knob name to
> 'compilation-safety' and gating the dangerous optimizations as
> discussed.

Thanks.  LGTM now (I'm still not sure we need both "safety" and "speed"
controls, but we may as well follow CL's lead).


        Stefan

Re: Introducing 'safety' compilation parameter

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org,  Eli Zaretskii <eliz@gnu.org>,
>  mattias.engdegard@gmail.com,  stefankangas@gmail.com
> Date: Sun, 12 May 2024 22:29:08 -0400
> 
> > Okay I pushed scratch/comp-safety2 changing the knob name to
> > 'compilation-safety' and gating the dangerous optimizations as
> > discussed.
> 
> Thanks.  LGTM now (I'm still not sure we need both "safety" and "speed"
> controls, but we may as well follow CL's lead).

Agreed.  Andrea, feel free to land this on master, and thanks.

Re: Introducing 'safety' compilation parameter

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org,  Eli Zaretskii <eliz@gnu.org>,
>>  mattias.engdegard@gmail.com,  stefankangas@gmail.com
>> Date: Sun, 12 May 2024 22:29:08 -0400
>> 
>> > Okay I pushed scratch/comp-safety2 changing the knob name to
>> > 'compilation-safety' and gating the dangerous optimizations as
>> > discussed.
>> 
>> Thanks.  LGTM now (I'm still not sure we need both "safety" and "speed"
>> controls, but we may as well follow CL's lead).
>
> Agreed.  Andrea, feel free to land this on master, and thanks.

Done now.  As usual feel free to round any edge if you spot it 🙏

Thanks

  Andrea

Re: Introducing 'safety' compilation parameter

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org,
>   mattias.engdegard@gmail.com,  stefankangas@gmail.com
> Date: Mon, 13 May 2024 17:12:15 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > Okay I pushed scratch/comp-safety2 changing the knob name to
> >> > 'compilation-safety' and gating the dangerous optimizations as
> >> > discussed.
> >> 
> >> Thanks.  LGTM now (I'm still not sure we need both "safety" and "speed"
> >> controls, but we may as well follow CL's lead).
> >
> > Agreed.  Andrea, feel free to land this on master, and thanks.
> 
> Done now.  As usual feel free to round any edge if you spot it 🙏

Done.