emacs master + org Wrong type argument: number-or-marker-p

emacs master + org Wrong type argument: number-or-marker-p

[hx]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

hi,
I just pulled emacs master, click link in org buffer not working anymore:


File mode specification error: (wrong-type-argument integer-or-marker-p nil)

org-open-at-point: Wrong type argument: number-or-marker-p, nil [4 times]
Org mode version 9.5.4 (release_9.5.4-17-g6e991f...)


Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  org-element-context()
  org-open-at-point()
  org-open-at-mouse((mouse-2 (#<window 3 on migration.org> 812 (180 . 398)
2091273234 nil 812 (20 . 18) nil (0 . 7) (9 . 19))))
  funcall-interactively(org-open-at-mouse (mouse-2 (#<window 3 on test.org>
812 (180 . 398) 2091273234 nil 812 (20 . 18) nil (0 . 7) (9 . 19))))
  command-execute(org-open-at-mouse)

[-- Attachment #2: Type: text/html, Size: 1013 bytes --]

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: hx <silent2600@gmail.com>
> Date: Mon, 1 Aug 2022 21:13:51 +0800
> 
> I just pulled emacs master, click link in org buffer not working anymore:
> 
> File mode specification error: (wrong-type-argument integer-or-marker-p nil)
> 
> org-open-at-point: Wrong type argument: number-or-marker-p, nil [4 times]
> Org mode version 9.5.4 (release_9.5.4-17-g6e991f...)

Byte-compile all the Org files anew, and the problem should go away.

emacs master + org Wrong type argument: number-or-marker-p

[Gerd Möllmann]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 194 bytes --]

Don't know if this is also your problem, but I needed to M-x 
package-recompile-all because of a changed byte code recently.

BTW, M-x report-emacs-bug is better than mail to emacs-devel.


[-- Attachment #1.1.1.2: Type: text/html, Size: 437 bytes --]

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3211 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: emacs master + org Wrong type argument: number-or-marker-p

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hx <silent2600@gmail.com>
>> Date: Mon, 1 Aug 2022 21:13:51 +0800
>> 
>> I just pulled emacs master, click link in org buffer not working anymore:
>> 
>> File mode specification error: (wrong-type-argument integer-or-marker-p nil)
>> 
>> org-open-at-point: Wrong type argument: number-or-marker-p, nil [4 times]
>> Org mode version 9.5.4 (release_9.5.4-17-g6e991f...)
>
> Byte-compile all the Org files anew, and the problem should go away.

What caused the issue?  Will if affect users after 29.1 is released?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: hx <silent2600@gmail.com>,  emacs-devel@gnu.org
> Date: Mon, 01 Aug 2022 15:11:48 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: hx <silent2600@gmail.com>
> >> Date: Mon, 1 Aug 2022 21:13:51 +0800
> >> 
> >> I just pulled emacs master, click link in org buffer not working anymore:
> >> 
> >> File mode specification error: (wrong-type-argument integer-or-marker-p nil)
> >> 
> >> org-open-at-point: Wrong type argument: number-or-marker-p, nil [4 times]
> >> Org mode version 9.5.4 (release_9.5.4-17-g6e991f...)
> >
> > Byte-compile all the Org files anew, and the problem should go away.
> 
> What caused the issue?

I don't know.

> Will if affect users after 29.1 is released?

If someone explains why this happened, we may then see if it can do
that.  Note that somehow only Org was affected.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: hx <silent2600@gmail.com>,  emacs-devel@gnu.org
>> Date: Mon, 01 Aug 2022 15:11:48 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: hx <silent2600@gmail.com>
>> >> Date: Mon, 1 Aug 2022 21:13:51 +0800
>> >> 
>> >> I just pulled emacs master, click link in org buffer not working anymore:
>> >> 
>> >> File mode specification error: (wrong-type-argument integer-or-marker-p nil)
>> >> 
>> >> org-open-at-point: Wrong type argument: number-or-marker-p, nil [4 times]
>> >> Org mode version 9.5.4 (release_9.5.4-17-g6e991f...)
>> >
>> > Byte-compile all the Org files anew, and the problem should go away.
>> 
>> What caused the issue?
>
> I don't know.
>
>> Will if affect users after 29.1 is released?
>
> If someone explains why this happened, we may then see if it can do
> that.  Note that somehow only Org was affected.

Not only Org.  It affected (at least) Gnus and TRAMP on my system as
well, but I couldn't figure out what was responsible.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>> What caused the issue?
>
> I don't know.
>

I think it's the new optional argument to narrow-to-region, which adds an 
argument to the narrow-to-region opcode in byte-compiled code.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 16:11:33 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Philip Kaludercic <philipk@posteo.net>, silent2600@gmail.com, 
>     emacs-devel@gnu.org
> 
> 
> >> What caused the issue?
> >
> > I don't know.
> >
> 
> I think it's the new optional argument to narrow-to-region, which adds an 
> argument to the narrow-to-region opcode in byte-compiled code.

If that's the reason, why didn't it affect more files?
narrow-to-region is called in much more places.

And if it's indeed due to narrow-to-region, we may have a problem: it
means old bytecode will not work with Emacs 29 and later.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Visuwesh]

[திங்கள் ஆகஸ்ட் 01, 2022] Philip Kaludercic wrote:

>> If someone explains why this happened, we may then see if it can do
>> that.  Note that somehow only Org was affected.
>
> Not only Org.  It affected (at least) Gnus and TRAMP on my system as
> well, but I couldn't figure out what was responsible.

I had problems with show-paren-mode and eval-last-sexp too.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 01 Aug 2022 16:11:33 +0000
>> From: Gregory Heytings <gregory@heytings.org>
>> cc: Philip Kaludercic <philipk@posteo.net>, silent2600@gmail.com, 
>>     emacs-devel@gnu.org
>> 
>> 
>> >> What caused the issue?
>> >
>> > I don't know.
>> >
>> 
>> I think it's the new optional argument to narrow-to-region, which adds an 
>> argument to the narrow-to-region opcode in byte-compiled code.
>
> If that's the reason, why didn't it affect more files?
> narrow-to-region is called in much more places.
>
> And if it's indeed due to narrow-to-region, we may have a problem: it
> means old bytecode will not work with Emacs 29 and later.

Should the functionality be moved out into a separate command
(e.g. `narrow-to-region-lock')?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>> I think it's the new optional argument to narrow-to-region, which adds 
>> an argument to the narrow-to-region opcode in byte-compiled code.
>
> If that's the reason, why didn't it affect more files? narrow-to-region 
> is called in much more places.
>

I don't know, and I'm not even sure that the actual cause.

>
> And if it's indeed due to narrow-to-region, we may have a problem: it 
> means old bytecode will not work with Emacs 29 and later.
>

If that's a problem (what kind of compatibility does Emacs promise about 
byte-compiled files?), we could pass that optional argument with a let 
binding.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Gregory Heytings <gregory@heytings.org>,  silent2600@gmail.com,
>   emacs-devel@gnu.org
> Date: Mon, 01 Aug 2022 16:34:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I think it's the new optional argument to narrow-to-region, which adds an 
> >> argument to the narrow-to-region opcode in byte-compiled code.
> >
> > If that's the reason, why didn't it affect more files?
> > narrow-to-region is called in much more places.
> >
> > And if it's indeed due to narrow-to-region, we may have a problem: it
> > means old bytecode will not work with Emacs 29 and later.
> 
> Should the functionality be moved out into a separate command
> (e.g. `narrow-to-region-lock')?

It isn't a command.

Anyway, please hold your horses.  Mattias, can you please look into
this?  If indeed the change in narrow-to-region caused the problem,
can we make the change in a way that doesn't break backward
compatibility of the byte code, or is that impossible with primitives
that have their own operation bytecode?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Andreas Schwab]

On Aug 01 2022, Gregory Heytings wrote:

> I think it's the new optional argument to narrow-to-region, which adds an
> argument to the narrow-to-region opcode in byte-compiled code.

That's bad.  Instead, the byte compiler should be changed to not use the
narrow-to-region byteop when the optional argument is present.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: emacs master + org Wrong type argument: number-or-marker-p

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> And if it's indeed due to narrow-to-region, we may have a problem: it
> means old bytecode will not work with Emacs 29 and later.

I had a quick look at some things in byte-defop-compiler that have
changed arity (or made things optional) to see whether that's happened
before, and it has.  So I'm not sure that's the explanation.

Other bug reports about this has mentioned `file-name-sans-extension',
which doesn't use `narrow-to-region' as far as I can see.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Mattias Engdegård]

[-- Attachment #1: Type: text/plain, Size: 245 bytes --]

1 aug. 2022 kl. 18.39 skrev Eli Zaretskii <eliz@gnu.org>:

> If indeed the change in narrow-to-region caused the problem,
> can we make the change in a way that doesn't break backward
> compatibility of the byte code

Does this fix it for you?


[-- Attachment #2: narrow-to-region.diff --]
[-- Type: application/octet-stream, Size: 3373 bytes --]

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index 1ecd77f751..0f2dbe4efb 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -767,7 +767,7 @@ 121
 (byte-defop 122  0 byte-char-syntax)
 (byte-defop 123 -1 byte-buffer-substring)
 (byte-defop 124 -1 byte-delete-region)
-(byte-defop 125 -2 byte-narrow-to-region)
+(byte-defop 125 -1 byte-narrow-to-region)
 (byte-defop 126  1 byte-widen)
 (byte-defop 127  0 byte-end-of-line)
 
@@ -3711,7 +3711,7 @@ byte-defop-compiler
 is the function and the second element is the bytecode-symbol.
 The second element may be nil, meaning there is no opcode.
 COMPILE-HANDLER is the function to use to compile this byte-op, or
-may be the abbreviations 0, 1, 2, 2-and, 3, 0-1, 1-2, 1-3, or 2-3.
+may be the abbreviations 0, 1, 2, 2-and, 3, 0-1, 1-2, 1-3, 2-3, or 2-call.
 If it is nil, then the handler is \"byte-compile-SYMBOL.\""
   (let (opcode)
     (if (symbolp function)
@@ -3731,6 +3731,7 @@ byte-defop-compiler
 					(1-2 . byte-compile-one-or-two-args)
 					(2-3 . byte-compile-two-or-three-args)
 					(1-3 . byte-compile-one-to-three-args)
+                                        (2-call . byte-compile-two-args-or-call)
 					)))
 			   compile-handler
 			   (intern (concat "byte-compile-"
@@ -3833,7 +3834,7 @@ setcar
 (byte-defop-compiler setcdr		2)
 (byte-defop-compiler buffer-substring	2)
 (byte-defop-compiler delete-region	2)
-(byte-defop-compiler narrow-to-region	2-3)
+(byte-defop-compiler narrow-to-region	2-call)
 (byte-defop-compiler (% byte-rem)	2)
 (byte-defop-compiler aset		3)
 
@@ -3911,6 +3912,12 @@ byte-compile-two-or-three-args
 	  ((= len 4) (byte-compile-three-args form))
 	  (t (byte-compile-subr-wrong-args form "2-3")))))
 
+(defun byte-compile-two-args-or-call (form)
+  "Compile using a byte-op for 2 args, plain call for anything else."
+  (if (= (length form) 3)
+      (byte-compile-two-args form)
+    (byte-compile-normal-call form)))
+
 (defun byte-compile-one-to-three-args (form)
   (let ((len (length form)))
     (cond ((= len 2) (byte-compile-three-args (append form '(nil nil))))
diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 4354ea03a4..6e9132e430 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -1915,7 +1915,11 @@ comp-limplify-lap-inst
       (byte-char-syntax auto)
       (byte-buffer-substring auto)
       (byte-delete-region auto)
-      (byte-narrow-to-region auto)
+      (byte-narrow-to-region
+       (comp-emit-set-call (comp-call 'narrow-to-region
+                                      (comp-slot)
+                                      (comp-slot+1)
+                                      (make-comp-mvar :constant nil))))
       (byte-widen
        (comp-emit-set-call (comp-call 'widen)))
       (byte-end-of-line auto)
diff --git a/src/bytecode.c b/src/bytecode.c
index 2b1eccdc51..310bc065ef 100644
--- a/src/bytecode.c
+++ b/src/bytecode.c
@@ -1480,8 +1480,10 @@ #define DEFINE(name, value) [name] = &&insn_ ## name,
 
 	CASE (Bnarrow_to_region):
 	  {
-	    Lisp_Object v2 = POP, v1 = POP;
-	    TOP = Fnarrow_to_region (TOP, v1, v2);
+	    /* To preserve bytecode compatibility, this opcode does not
+	       accept the optional third argument added in Emacs 29.  */
+	    Lisp_Object v1 = POP;
+	    TOP = Fnarrow_to_region (TOP, v1, Qnil);
 	    NEXT;
 	  }
 

[-- Attachment #3: Type: text/plain, Size: 2 bytes --]

Re: emacs master + org Wrong type argument: number-or-marker-p

[Andreas Schwab]

On Aug 01 2022, Lars Ingebrigtsen wrote:

> I had a quick look at some things in byte-defop-compiler that have
> changed arity (or made things optional) to see whether that's happened
> before, and it has.

This is about the change in the bytecode interpreter.  The same opcode
now pops 3 values from the stack instead of just two, making it
unbalanced.

You will probably see aborts if you compile bytecomp.c with
-DBYTE_CODE_SAFE.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Mon, 1 Aug 2022 19:15:40 +0200
> Cc: Philip Kaludercic <philipk@posteo.net>, gregory@heytings.org,
>         silent2600@gmail.com, emacs-devel@gnu.org
> 
> > If indeed the change in narrow-to-region caused the problem,
> > can we make the change in a way that doesn't break backward
> > compatibility of the byte code
> 
> Does this fix it for you?

LGTM, but I don't know how to test it to verify the problem is fixed.

Thanks.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gerd Möllmann]

Hi Lars!

> I had a quick look at some things in byte-defop-compiler that have
> changed arity (or made things optional) to see whether that's happened
> before, and it has.  So I'm not sure that's the explanation.

I can confirm that narrow-to-region in its new form is not compatible
with old byte code.  The reason: old byte code puts 2 values on the byte
stack, the new byte code implementation expects 3 values on the stack.
What the third value will be is whatever happenes to be there.  And what
happens when it finds an integer or marker is also less desirable.

> Other bug reports about this has mentioned `file-name-sans-extension',
> which doesn't use `narrow-to-region' as far as I can see.

That's unrelated: It was an error in nextstep/Makefile.in by which Emacs
didn't have a .pdmp, and hence didn't have a file-name-sans-extension
(bug#56853).

Re: emacs master + org Wrong type argument: number-or-marker-p

[Mattias Engdegård]

1 aug. 2022 kl. 19.24 skrev Eli Zaretskii <eliz@gnu.org>:

> I don't know how to test it to verify the problem is fixed.

I'm not sure I do either, because the change to narrow-to-region was made without adding any tests at all.

Frankly I'd recommend that the whole change be reverted because it performs unbalanced specbinds which we expect functions not to do. I don't think it can be salvaged in its current form; better back it out and let the author submit a new proposal for how to handle the problem.

Sorry about not noticing this earlier.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Lars Ingebrigtsen]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I can confirm that narrow-to-region in its new form is not compatible
> with old byte code.  The reason: old byte code puts 2 values on the byte
> stack, the new byte code implementation expects 3 values on the stack.
> What the third value will be is whatever happenes to be there.  And what
> happens when it finds an integer or marker is also less desirable.

Right.

>> Other bug reports about this has mentioned `file-name-sans-extension',
>> which doesn't use `narrow-to-region' as far as I can see.
>
> That's unrelated: It was an error in nextstep/Makefile.in by which Emacs
> didn't have a .pdmp, and hence didn't have a file-name-sans-extension
> (bug#56853).

Ah, I see.  Thanks for the explanation.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Mon, 1 Aug 2022 19:36:56 +0200
> Cc: philipk@posteo.net, gregory@heytings.org, silent2600@gmail.com,
>         emacs-devel@gnu.org
> 
> Frankly I'd recommend that the whole change be reverted because it performs unbalanced specbinds which we expect functions not to do. I don't think it can be salvaged in its current form; better back it out and let the author submit a new proposal for how to handle the problem.

OK, thanks.

Gregory, we don't really need to be able to make such "locked"
narrowing from Lisp, do we?  If we don't, my suggestion is to make a
3-arg C function whose guts is the current code of Fnarrow_to_region,
and then have Fnarrow_to_region call that with the last argument nil.
Then the changes in the byte compiler, byte-interpreter, and
native-compiler can be reverted.

If we do need to have a 3-arg version exposed to Lisp, a separate
DEFUN is probably the best alternative.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>> Frankly I'd recommend that the whole change be reverted because it 
>> performs unbalanced specbinds which we expect functions not to do. I 
>> don't think it can be salvaged in its current form; better back it out 
>> and let the author submit a new proposal for how to handle the problem.
>
> Gregory, we don't really need to be able to make such "locked" narrowing 
> from Lisp, do we?
>

We don't, indeed, at least not now.

>
> If we don't, my suggestion is to make a 3-arg C function whose guts is 
> the current code of Fnarrow_to_region, and then have Fnarrow_to_region 
> call that with the last argument nil. Then the changes in the byte 
> compiler, byte-interpreter, and native-compiler can be reverted.
>

I'll do that.  But that wouldn't solve the "unbalanced specbinds" Mattias 
mentions.  Is it not allowed to specbind in a function and to unbind in 
its caller?  I don't see why it wouldn't, but I may be missing something.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Mon, 01 Aug 2022 18:06:50 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Mattias Engdegård <mattiase@acm.org>, philipk@posteo.net, 
>     silent2600@gmail.com, emacs-devel@gnu.org
> 
> > If we don't, my suggestion is to make a 3-arg C function whose guts is 
> > the current code of Fnarrow_to_region, and then have Fnarrow_to_region 
> > call that with the last argument nil. Then the changes in the byte 
> > compiler, byte-interpreter, and native-compiler can be reverted.
> 
> I'll do that.  But that wouldn't solve the "unbalanced specbinds" Mattias 
> mentions.  Is it not allowed to specbind in a function and to unbind in 
> its caller?  I don't see why it wouldn't, but I may be missing something.

It's a bit...unorthodox.

Can't we instead specbind in the caller and unbind after the call
returns?  If this isn't required from Lisp, I see no problem with
that.  Do you?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Mattias Engdegård]

1 aug. 2022 kl. 20.06 skrev Gregory Heytings <gregory@heytings.org>:

> Is it not allowed to specbind in a function and to unbind in its caller?

No, functions must be balanced with respect to the specbind stack. The bytecode machinery assumes this to be the case.

If you want a special behaviour for narrow-to-region over a dynamic extent, just bind a dynamic variable. Or, if that is insufficient, write a new function for it that takes a body function as argument which is called with the necessary clean-up done afterwards. Or implement it as a Lisp macro.

So please revert all changes relating to the new narrow-to-region argument and submit a new patch. It's better than somehow trying to patch up the current unworkable approach.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>
> Can't we instead specbind in the caller and unbind after the call 
> returns?  If this isn't required from Lisp, I see no problem with that. 
> Do you?
>

Indeed.  Now done, thanks for your help!

Another 'make bootstrap' is necessary.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>
> No, functions must be balanced with respect to the specbind stack. The 
> bytecode machinery assumes this to be the case.
>

Thanks for the clarification, I'll keep that in mind in the future.

>
> So please revert all changes relating to the new narrow-to-region 
> argument and submit a new patch. It's better than somehow trying to 
> patch up the current unworkable approach.
>

That won't be necessary, it was actually easy to fix.  Now done on master, 
and we are back to the statu quo ante with respect to byte compilation. 
Of course another 'make boostrap' is necessary.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Alan Mackenzie]

On Mon, Aug 01, 2022 at 19:16:44 +0000, Gregory Heytings wrote:


> > No, functions must be balanced with respect to the specbind stack. The 
> > bytecode machinery assumes this to be the case.


> Thanks for the clarification, I'll keep that in mind in the future.


> > So please revert all changes relating to the new narrow-to-region 
> > argument and submit a new patch. It's better than somehow trying to 
> > patch up the current unworkable approach.


> That won't be necessary, it was actually easy to fix.  Now done on master, 
> and we are back to the statu quo ante with respect to byte compilation. 
> Of course another 'make boostrap' is necessary.

I see in

    commit 9d8a6c82838f2f24e76a67379b02956aa668d7cf
    Author: Gregory Heytings <gregory@heytings.org>
    Date:   Mon Aug 1 19:11:01 2022 +0000

the following text:

    +Note that, in rare circumstances, Emacs may decide to leave, for
    +performance reasons, the accessible portion of the buffer unchanged
    +after a call to @code{narrow-to-region}.

You cannot do this.  narrow-to-region and widen aren't nice-to-have
optional extras, they are essential parts of program functionality.  If
you stop them working properly, programs will break.  It is a bit like
the Lisp machine randomly failing to perform some car operations.

If you must do something like this, why don't you throw an error when an
"invalid" narrow-to-region or widen is attempted?  This would at least
allow the programmer to do some debugging.

Would it also be possible to add in some commentry motivating this
unusual action?  All I can see from looking at the source (xdisp.c) is
that it probably has something to do with long lines.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs master + org Wrong type argument: number-or-marker-p

[Mattias Engdegård]

1 aug. 2022 kl. 21.16 skrev Gregory Heytings <gregory@heytings.org>:

> Now done on master, and we are back to the statu quo ante with respect to byte compilation.

Thank you -- no remaining objections pertaining to interaction with Lisp (since there is none).

Re: emacs master + org Wrong type argument: number-or-marker-p

[Stefan Monnier]

Mattias Engdegård [2022-08-01 20:47:04] wrote:
> 1 aug. 2022 kl. 20.06 skrev Gregory Heytings <gregory@heytings.org>:
>> Is it not allowed to specbind in a function and to unbind in its caller?
> No, functions must be balanced with respect to the specbind stack. The
> bytecode machinery assumes this to be the case.

With some exceptions, of course, such as the `specbind` function itself,
or the `record_unwind_protect_*`.  I.e. functions which do nothing else
than push something onto the specpdl.


        Stefan

Re: emacs master + org Wrong type argument: number-or-marker-p

[Mattias Engdegård]

2 aug. 2022 kl. 10.28 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> With some exceptions, of course, such as the `specbind` function itself,
> or the `record_unwind_protect_*`.  I.e. functions which do nothing else
> than push something onto the specpdl.

All fine as long as the specpdl level remains the same at entry and return from a function called from Lisp.
Maybe we should add an assertion for this when checking is enabled.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>> Now done on master, and we are back to the statu quo ante with respect 
>> to byte compilation.
>
> Thank you -- no remaining objections pertaining to interaction with Lisp 
> (since there is none).
>

Note that it would make sense to add some checks in the test suite for the 
problems you mentioned.

When make bootstrap and make check passes (which they did), it seems 
reasonable for a contributor to conclude that their changeset does not 
introduce a regression.

And perhaps also some clear note in bytecode.c and bytecomp.el that 
changes to the bytecode are in general not allowed (perhaps there is one 
already, but I did not see it).

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: hx <silent2600@gmail.com>,  emacs-devel@gnu.org
>> Date: Mon, 01 Aug 2022 15:11:48 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: hx <silent2600@gmail.com>
>> >> Date: Mon, 1 Aug 2022 21:13:51 +0800
>> >> 
>> >> I just pulled emacs master, click link in org buffer not working anymore:
>> >> 
>> >> File mode specification error: (wrong-type-argument integer-or-marker-p nil)
>> >> 
>> >> org-open-at-point: Wrong type argument: number-or-marker-p, nil [4 times]
>> >> Org mode version 9.5.4 (release_9.5.4-17-g6e991f...)
>> >
>> > Byte-compile all the Org files anew, and the problem should go away.
>> 
>> What caused the issue?
>
> I don't know.
>
>> Will if affect users after 29.1 is released?
>
> If someone explains why this happened, we may then see if it can do
> that.  Note that somehow only Org was affected.

It is because if recent changes to bytecode.c that were reverted.  Org
is only affected because it loads something that uses `narrow-to-region'
and was compiled between d3c4833d1350e26a2ae35e00eaf2d6bef1724679 and
a5adcbdf28eb8ad376a1004f4a6c9eda1f1447fb.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Mon, 1 Aug 2022 20:05:44 +0000
> Cc: Mattias Engdegård <mattiase@acm.org>,
>  Eli Zaretskii <eliz@gnu.org>, philipk@posteo.net, silent2600@gmail.com,
>  emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I see in
> 
>     commit 9d8a6c82838f2f24e76a67379b02956aa668d7cf
>     Author: Gregory Heytings <gregory@heytings.org>
>     Date:   Mon Aug 1 19:11:01 2022 +0000
> 
> the following text:
> 
>     +Note that, in rare circumstances, Emacs may decide to leave, for
>     +performance reasons, the accessible portion of the buffer unchanged
>     +after a call to @code{narrow-to-region}.
> 
> You cannot do this.  narrow-to-region and widen aren't nice-to-have
> optional extras, they are essential parts of program functionality.  If
> you stop them working properly, programs will break.  It is a bit like
> the Lisp machine randomly failing to perform some car operations.

This happens only in buffers with very long lines, where we want to
prevent Lisp programs called from low-level facilities, like
redisplay, to scan the entire buffer.  "Normal" invocations from Lisp
will not meet with this.

> Would it also be possible to add in some commentry motivating this
> unusual action?

I added some verbiage.

Opcode Versioning Was: emacs master + org Wrong type argument: number-or-marker-p

[Sam Steingold]

> * Gregory Heytings <tertbel@urlgvatf.bet> [2022-08-01 16:11:33 +0000]:
>
> I think it's the new optional argument to narrow-to-region, which adds an
> argument to the narrow-to-region opcode in byte-compiled code.

Do Emacs compiled files have a version marker that is updated whenever
opcodes are changed?

This is the usual approach to this issue: whenever an opcode is changed,
this internal version is modified and this version is saved into every
compiled file, so that Emacs can reject loading a file with an incorrect
version.

One can also use a hash of all opcode specs as the version...

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.2113
http://childpsy.net http://calmchildstories.com http://steingoldpsychology.com
https://jij.org https://camera.org https://mideasttruth.com
Bug free software merely has random features.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2022 at 16:46:24 +0300, Eli Zaretskii wrote:
> > Date: Mon, 1 Aug 2022 20:05:44 +0000
> > Cc: Mattias Engdegård <mattiase@acm.org>,
> >  Eli Zaretskii <eliz@gnu.org>, philipk@posteo.net, silent2600@gmail.com,
> >  emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I see in

> >     commit 9d8a6c82838f2f24e76a67379b02956aa668d7cf
> >     Author: Gregory Heytings <gregory@heytings.org>
> >     Date:   Mon Aug 1 19:11:01 2022 +0000

> > the following text:

> >     +Note that, in rare circumstances, Emacs may decide to leave, for
> >     +performance reasons, the accessible portion of the buffer unchanged
> >     +after a call to @code{narrow-to-region}.

> > You cannot do this.  narrow-to-region and widen aren't nice-to-have
> > optional extras, they are essential parts of program functionality.  If
> > you stop them working properly, programs will break.  It is a bit like
> > the Lisp machine randomly failing to perform some car operations.

> This happens only in buffers with very long lines, where we want to
> prevent Lisp programs called from low-level facilities, like
> redisplay, to scan the entire buffer.

So Lisp programs will "only" fail to work in buffers with long lines.  I
protest at this.  There surely could have been a solution to whatever
the problem was that respected the integrity of the Lisp machine.  There
is not even a return code to say that a byte-code instruction has failed
to work.  Surely there should be an error signalled if such happens,
since the program is broken after ignoring an instruction.

Ignoring what a programmer programmed cannot be a good strategy.

> "Normal" invocations from Lisp will not meet with this.

A great deal of pain is fore-ordained.

I protest also that this wasn't discussed openly on emacs-devel.

> > Would it also be possible to add in some commentry motivating this
> > unusual action?

> I added some verbiage.

Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 18:59:07 +0000
> Cc: gregory@heytings.org, mattiase@acm.org, philipk@posteo.net,
>   silent2600@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > This happens only in buffers with very long lines, where we want to
> > prevent Lisp programs called from low-level facilities, like
> > redisplay, to scan the entire buffer.
> 
> So Lisp programs will "only" fail to work in buffers with long lines.  I
> protest at this.

Not any Lisp programs, only those invoked from those hooks.

And they won't necessarily fail.  In fact, we have yet to see a single
serious failure due to these measures.  In general, the restriction is
large enough to satisfy any reasonable processing, so it shouldn't
matter unless the Lisp program misbehaves.

> There surely could have been a solution to whatever
> the problem was that respected the integrity of the Lisp machine.

Theoretically, yes.  But in practice, Emacs had this problem since 22
years ago, and no solution presented itself.

> There is not even a return code to say that a byte-code instruction
> has failed to work.

A program can always test point-min and point-max.

> Surely there should be an error signalled if such happens, since the
> program is broken after ignoring an instruction.

It isn't broken, though.

> Ignoring what a programmer programmed cannot be a good strategy.

It isn't ignored, just restricted: we don't let such programs run amok
high and low in these extra-long lines.

> I protest also that this wasn't discussed openly on emacs-devel.

It is being discussed, here and on the bug tracker, for about a month
now.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Alan Mackenzie]

Hello, Eli.

On Tue, Aug 02, 2022 at 22:15:37 +0300, Eli Zaretskii wrote:
> > Date: Tue, 2 Aug 2022 18:59:07 +0000
> > Cc: gregory@heytings.org, mattiase@acm.org, philipk@posteo.net,
> >   silent2600@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > This happens only in buffers with very long lines, where we want to
> > > prevent Lisp programs called from low-level facilities, like
> > > redisplay, to scan the entire buffer.

> > So Lisp programs will "only" fail to work in buffers with long lines.  I
> > protest at this.

> Not any Lisp programs, only those invoked from those hooks.

> And they won't necessarily fail.

No, they'll continue in a state different from that conceived by their
creator.  Is that not a failure?

> In fact, we have yet to see a single serious failure due to these
> measures.  In general, the restriction is large enough to satisfy any
> reasonable processing, so it shouldn't matter unless the Lisp program
> misbehaves.

> > There surely could have been a solution to whatever
> > the problem was that respected the integrity of the Lisp machine.

> Theoretically, yes.  But in practice, Emacs had this problem since 22
> years ago, and no solution presented itself.

> > There is not even a return code to say that a byte-code instruction
> > has failed to work.

> A program can always test point-min and point-max.

A rigorous program MUST now test these.  This is such a horrible
artifact that it won't get done most of the time.

> > Surely there should be an error signalled if such happens, since the
> > program is broken after ignoring an instruction.

> It isn't broken, though.

I disagree fundamentally.  CC Mode, for example, uses widen and
narrow-to-region all over the place, and surely other modes will too.
When the opcodes break, so will these modes.

> > Ignoring what a programmer programmed cannot be a good strategy.

> It isn't ignored, just restricted:

There are no safety fences of any sort.  The meaning of a program using
w and n-t-r is no longer determinate.

> we don't let such programs run amok high and low in these extra-long
> lines.

Using widen and narrow-to-region is hardly running amok.

> > I protest also that this wasn't discussed openly on emacs-devel.

> It is being discussed, here and on the bug tracker, for about a month
> now.

That's a month in which the discussion was open only to somebody who
reads every post in the bug list, such as yourself, or who came upon the
discussion by chance.  Others, such as me, were unaware of it.  It looks
like the decision to change the byte code interpreter has already been
taken, and I had no chance to participate in that process.

Is it really right that fundamental changes to Emacs get discussed only
on the bug list, or in secret[*] on emacs-devel?  [*]I.e. when the
subject line is not explicit.

Anyhow, I wanted to protest and I have done so.  The arguments between
the two of us here about widen and n-t-r are clearly not going anywhere,
so I don't intend to continue them.  I won't feel put out if you don't
want to answer them.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Opcode Versioning Was: emacs master + org Wrong type argument: number-or-marker-p

[Stefan Monnier]

> compiled file, so that Emacs can reject loading a file with an incorrect
> version.

Emacs instead aims to be able to keep loading old byte-compiled files so
we try hard to change the byte-codes only in ways which are
backward compatible.


        Stefan

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Alan Mackenzie <acm@muc.de> writes:

> A rigorous program MUST now test these.  This is such a horrible
> artifact that it won't get done most of the time.

[...]

> There are no safety fences of any sort.  The meaning of a program using
> w and n-t-r is no longer determinate.

[...]

> That's a month in which the discussion was open only to somebody who
> reads every post in the bug list, such as yourself, or who came upon the
> discussion by chance.  Others, such as me, were unaware of it.  It looks
> like the decision to change the byte code interpreter has already been
> taken, and I had no chance to participate in that process.
>
> Is it really right that fundamental changes to Emacs get discussed only
> on the bug list, or in secret[*] on emacs-devel?  [*]I.e. when the
> subject line is not explicit.
>
> Anyhow, I wanted to protest and I have done so.  The arguments between
> the two of us here about widen and n-t-r are clearly not going anywhere,
> so I don't intend to continue them.  I won't feel put out if you don't
> want to answer them.

I agree with Alan.

Is there really a slowdown associated with allowing code to widen
outside the bounds specified by redisplay long-line "optimizations"?  Or
are we speculating about a hypothetical problem that doesn't exist
again?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  gregory@heytings.org,  mattiase@acm.org,
>   philipk@posteo.net,  silent2600@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 03 Aug 2022 09:21:00 +0800
> 
> I agree with Alan.
> 
> Is there really a slowdown associated with allowing code to widen
> outside the bounds specified by redisplay long-line "optimizations"?  Or
> are we speculating about a hypothetical problem that doesn't exist
> again?

It isn't a hypothetical problem.  You can try the examples posted here
yourself, after disabling the feature.  Try it once with and then
without font-lock, and draw your own conclusions.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> It isn't a hypothetical problem.  You can try the examples posted here
> yourself, after disabling the feature.  Try it once with and then
> without font-lock, and draw your own conclusions.

Then wouldn't it be better to fix the font-locking code to not widen
outside bounds explictly specified by redisplay, instead of making
`widen' and `narrow-to-region' in effect inoperable in those
circumstances?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>
> Is there really a slowdown associated with allowing code to widen 
> outside the bounds specified by redisplay long-line "optimizations"? 
> Or are we speculating about a hypothetical problem that doesn't exist 
> again?
>

How kindly this is said!  "Optimizations" between quotation marks, as if 
they weren't.  Speculating.  Hypothetical problem.  That doesn't exist. 
Again.

Thank you!

Re: emacs master + org Wrong type argument: number-or-marker-p

[Stefan Monnier]

> Is there really a slowdown associated with allowing code to widen
> outside the bounds specified by redisplay long-line "optimizations"?  Or
> are we speculating about a hypothetical problem that doesn't exist
> again?

There is a real problem here: currently font-lock will widen right back.
There might be a few more cases where widening will happen currently,
but I think font-lock is the main one.  IMO we should specifically fix
those few cases we encounter, rather than add to the narrowing
complexity (which tends to encourage major modes to widen since they
basically can never know why a narrowing is in effect) by imposing a
"locked narrowing".


        Stefan

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> IMO we should specifically fix those few cases we encounter, rather
> than add to the narrowing complexity (which tends to encourage major
> modes to widen since they basically can never know why a narrowing is
> in effect) by imposing a "locked narrowing".

Exactly.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Gregory Heytings <gregory@heytings.org> writes:

> "Optimizations" between quotation marks, as if they weren't.

The way I see it, "locked narrowing" is a very questionably designed
optimization, yes.

> Speculating.  Hypothetical problem.  That doesn't exist. Again.

I didn't know whether or not that was the case, which is why I asked.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Tue, 2 Aug 2022 20:28:42 +0000
> Cc: gregory@heytings.org, mattiase@acm.org, philipk@posteo.net,
>   silent2600@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Not any Lisp programs, only those invoked from those hooks.
> 
> > And they won't necessarily fail.
> 
> No, they'll continue in a state different from that conceived by their
> creator.

Not necessarily.  Only if previously they forcefully widened the
buffer and looked beyond the restriction.

> Is that not a failure?

See above: not necessarily.

> > > There is not even a return code to say that a byte-code instruction
> > > has failed to work.
> 
> > A program can always test point-min and point-max.
> 
> A rigorous program MUST now test these.

Only if it MUST misbehave.

> > > Surely there should be an error signalled if such happens, since the
> > > program is broken after ignoring an instruction.
> 
> > It isn't broken, though.
> 
> I disagree fundamentally.  CC Mode, for example, uses widen and
> narrow-to-region all over the place, and surely other modes will too.
> When the opcodes break, so will these modes.

CC Mode is extremely unlikely to be in effect in files with such long
lines.  So you, as the CC Mode developer, can relax: these changes are
not relevant to CC Mode.

> > we don't let such programs run amok high and low in these extra-long
> > lines.
> 
> Using widen and narrow-to-region is hardly running amok.

It is, if the Lisp program then goes on to examine all the characters
from BOB to point.  It also is if the Lisp program insists to start
from the beginning of the line, no matter how far back that is.

> > > I protest also that this wasn't discussed openly on emacs-devel.
> 
> > It is being discussed, here and on the bug tracker, for about a month
> > now.
> 
> That's a month in which the discussion was open only to somebody who
> reads every post in the bug list, such as yourself, or who came upon the
> discussion by chance.  Others, such as me, were unaware of it.  It looks
> like the decision to change the byte code interpreter has already been
> taken, and I had no chance to participate in that process.

The byte code interpreter was yesterday changed back, so there's no
change there.

> Is it really right that fundamental changes to Emacs get discussed only
> on the bug list, or in secret[*] on emacs-devel?  [*]I.e. when the
> subject line is not explicit.

The Subject lines are not always explicit enough, that is true, but
there's nothing here specific to the discussion at hand, as it happens
all the time here.

And yes, if you don't want to miss important changes, you need wither
(a) read everything on the lists, or (b) track the committed changes
(e.g., via emacs-diffs@gnu.org), or both.  I see no way around that,
unless someone volunteers to set up some kind of notification service.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  gregory@heytings.org,  mattiase@acm.org,
>   philipk@posteo.net,  silent2600@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 03 Aug 2022 12:34:07 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It isn't a hypothetical problem.  You can try the examples posted here
> > yourself, after disabling the feature.  Try it once with and then
> > without font-lock, and draw your own conclusions.
> 
> Then wouldn't it be better to fix the font-locking code to not widen
> outside bounds explictly specified by redisplay, instead of making
> `widen' and `narrow-to-region' in effect inoperable in those
> circumstances?

How would that work in practice?  Font-locking code uses functions and
regexps provided by the major modes, so it cannot by itself prevent
widening.  Many major modes want their fontification to start from the
beginning of the line, which can be very far in a buffer with long
lines; and quite a few major modes use code that widens to BOB.

What can font-lock do to prevent that?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> How would that work in practice?  Font-locking code uses functions and
> regexps provided by the major modes, so it cannot by itself prevent
> widening.  Many major modes want their fontification to start from the
> beginning of the line, which can be very far in a buffer with long
> lines; and quite a few major modes use code that widens to BOB.
>
> What can font-lock do to prevent that?

But not every piece of code widens and scans from BOB to EOL.  The idea
is to change all problematic code in Emacs to only widen to limits
placed by redisplay, instead of changing widen and narrow-to-region
themselves.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 03 Aug 2022 19:07:37 +0800
> 
> Gregory Heytings <gregory@heytings.org> writes:
> 
> > "Optimizations" between quotation marks, as if they weren't.
> 
> The way I see it, "locked narrowing" is a very questionably designed
> optimization, yes.

It isn't.  If it were, I would never have agreed to including it in
Emacs.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  gregory@heytings.org,  mattiase@acm.org,
>   philipk@posteo.net,  silent2600@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 03 Aug 2022 20:07:16 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How would that work in practice?  Font-locking code uses functions and
> > regexps provided by the major modes, so it cannot by itself prevent
> > widening.  Many major modes want their fontification to start from the
> > beginning of the line, which can be very far in a buffer with long
> > lines; and quite a few major modes use code that widens to BOB.
> >
> > What can font-lock do to prevent that?
> 
> But not every piece of code widens and scans from BOB to EOL.

Many do.

> The idea is to change all problematic code in Emacs to only widen to
> limits placed by redisplay, instead of changing widen and
> narrow-to-region themselves.

There's no reason to "widen to limits placed by redisplay", they should
just not change the existing restriction.

And when (and if!) your idea is implemented for the relevant modes in
Emacs, we can flip the default value of long-line-threshold to nil.
So dazzle me!

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> There's no reason to "widen to limits placed by redisplay", they should
> just not change the existing restriction.

That's the restriction I meant by "limits placed by redisplay."
Doing so would avoid introducing non-deterministic behavior to
narrow-to-region.

> And when (and if!) your idea is implemented for the relevant modes in
> Emacs, we can flip the default value of long-line-threshold to nil.
> So dazzle me!

Why would that be so?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: acm@muc.de,  gregory@heytings.org,  mattiase@acm.org,
>   philipk@posteo.net,  silent2600@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 03 Aug 2022 21:10:18 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And when (and if!) your idea is implemented for the relevant modes in
> > Emacs, we can flip the default value of long-line-threshold to nil.
> > So dazzle me!
> 
> Why would that be so?

Because we've waited long enough for such changes to emerge, and they
didn't.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>> Speculating.  Hypothetical problem.  That doesn't exist.  Again.
>
> I didn't know whether or not that was the case, which is why I asked.
>

A sequence of negatively loaded words followed by a question mark is not a 
question, no.  Especially when the one who writes these words has 
personally been given all the elements to answer that supposed question 
less than two weeks ago, here: 
https://lists.gnu.org/archive/html/emacs-devel/2022-07/msg00597.html .

Re: emacs master + org Wrong type argument: number-or-marker-p

[Stefan Monnier]

>> Then wouldn't it be better to fix the font-locking code to not widen
>> outside bounds explictly specified by redisplay, instead of making
>> `widen' and `narrow-to-region' in effect inoperable in those
>> circumstances?
> How would that work in practice?  Font-locking code uses functions and
> regexps provided by the major modes, so it cannot by itself prevent
> widening.

I don't understand what you're talking about.

AFAIK in 99% of the cases, font-lock.el itself widens, then uses the
regexps (which can't widen) and the functions provided by the major mode
almost none of which (with rare exceptions, of course, most of them
historical) will widen since font-lock already did it for them (and
since widening will lead to bugs when used within something like
mmm-mode or mhtml-mode).

I don't see much need to "prevent widening".


        Stefan

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Gregory Heytings <gregory@heytings.org> writes:

> A sequence of negatively loaded words followed by a question mark is
> not a question, no.

Whether or not I asked a question is not up to you to decide.

> Especially when the one who writes these words has personally been
> given all the elements to answer that supposed question less than two
> weeks ago, here:
> https://lists.gnu.org/archive/html/emacs-devel/2022-07/msg00597.html .

How was that related to font locking or locked narrowing at all?  The
latter did not even exist on the 21st of July.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Because we've waited long enough for such changes to emerge, and they
> didn't.

Does removing "locked narrowing" and replacing Qrestrictions_locked
with:

  specbind (Qfont_lock_dont_widen, Qt);

in handle_fontified_prop solve the problem?

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>> A sequence of negatively loaded words followed by a question mark is 
>> not a question, no.
>
> Whether or not I asked a question is not up to you to decide.
>

See https://www.gnu.org/philosophy/kind-communication.en.html .

Re: emacs master + org Wrong type argument: number-or-marker-p

[Gregory Heytings]

>> Because we've waited long enough for such changes to emerge, and they 
>> didn't.
>
> Does removing "locked narrowing" and replacing Qrestrictions_locked 
> with:
>
>  specbind (Qfont_lock_dont_widen, Qt);
>
> in handle_fontified_prop solve the problem?
>

It doesn't.  Otherwise that's what I would have done, obviously.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Gregory Heytings <gregory@heytings.org> writes:

> It doesn't.  Otherwise that's what I would have done, obviously.

Which part of font-lock then proceeds to widen too much?
It should probably be made to respect `font-lock-dont-widen'.

Alternatively, could you please tell me where in bug#56682 to look for
this information? The thread is very long and difficult to follow.

Thanks.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Po Lu <luangruo@yahoo.com>,  acm@muc.de,  gregory@heytings.org,
>   mattiase@acm.org,  philipk@posteo.net,  silent2600@gmail.com,
>   emacs-devel@gnu.org
> Date: Wed, 03 Aug 2022 16:47:17 -0400
> 
> >> Then wouldn't it be better to fix the font-locking code to not widen
> >> outside bounds explictly specified by redisplay, instead of making
> >> `widen' and `narrow-to-region' in effect inoperable in those
> >> circumstances?
> > How would that work in practice?  Font-locking code uses functions and
> > regexps provided by the major modes, so it cannot by itself prevent
> > widening.
> 
> I don't understand what you're talking about.
> 
> AFAIK in 99% of the cases, font-lock.el itself widens, then uses the
> regexps (which can't widen) and the functions provided by the major mode
> almost none of which (with rare exceptions, of course, most of them
> historical) will widen since font-lock already did it for them (and
> since widening will lead to bugs when used within something like
> mmm-mode or mhtml-mode).

What will we do with the "rare" exceptions, such as CC Mode, for
example?

And what will we do with (I expect) the much larger group which
becomes broken when font-lock stops widening, and their assumptions
about being always able to access any buffer position become false?

IOW, I'd love to see the brave new world where font-lock doesn't widen
and none of the major modes we care about do, and fontifications still
work reasonably well.  Then we can revisit the strategy of handling
long lines, and perhaps change the defaults as result.  But for now,
this is purely academic, with too many "ifs" and "whens" and too
little actual development in that direction.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> What will we do with the "rare" exceptions, such as CC Mode, for
> example?

Fix them, individually?

> And what will we do with (I expect) the much larger group which
> becomes broken when font-lock stops widening, and their assumptions
> about being always able to access any buffer position become false?

We don't have make font lock avoid widening -- what I was proposing was
to ensure that font lock, CC Mode, and any other exceptions that might
come up only widen to the "locked" restriction, without changing the
behavior of narrow-to-region and friends.

The end result is the same as the current code, but the behavior of
narrow-to-region is not changed.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Thu, 04 Aug 2022 01:09:21 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Eli Zaretskii <eliz@gnu.org>, acm@muc.de, mattiase@acm.org, 
>     philipk@posteo.net, silent2600@gmail.com, emacs-devel@gnu.org
> 
> > Does removing "locked narrowing" and replacing Qrestrictions_locked 
> > with:
> >
> >  specbind (Qfont_lock_dont_widen, Qt);
> >
> > in handle_fontified_prop solve the problem?
> 
> It doesn't.

Which isn't surprising, given the intent of that variable, as provided
in the doc string:

  (defvar font-lock-dont-widen nil
    "If non-nil, font-lock will work on the non-widened buffer.
  Useful for things like RMAIL and Info where the whole buffer is not
  a very meaningful entity to highlight.")

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  acm@muc.de,
>   gregory@heytings.org,  mattiase@acm.org,  philipk@posteo.net,
>   silent2600@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 04 Aug 2022 14:19:20 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What will we do with the "rare" exceptions, such as CC Mode, for
> > example?
> 
> Fix them, individually?

Fine by me.  People are encouraged to work on that ASAP.  But we'd
like to have a reasonably performant Emacs 29 when editing long lines
without waiting for that pipe dream to come true.  If and when it does
come true, we can reset the long-line-threshold to nil by default,
even if it happens before Emacs 29 is released.

> > And what will we do with (I expect) the much larger group which
> > becomes broken when font-lock stops widening, and their assumptions
> > about being always able to access any buffer position become false?
> 
> We don't have make font lock avoid widening -- what I was proposing was
> to ensure that font lock, CC Mode, and any other exceptions that might
> come up only widen to the "locked" restriction, without changing the
> behavior of narrow-to-region and friends.

We don't know how to do that, as was already explained and
demonstrated here.  Modes do what they want, and in some cases we have
hard time even convincing the mode developers that they should try to
avoid doing that.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> We don't know how to do that, as was already explained and
> demonstrated here.

I will give that a look after a while.

> Modes do what they want, and in some cases we have hard time even
> convincing the mode developers that they should try to avoid doing
> that.

Then maybe we should not impose our opinion of what narrowing is best on
major mode developers?  Forcing restrictions on user code never works.
Sooner or later, developers will start performing fontification in a
timer, in order to widen past the "locked narrowing".

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: monnier@iro.umontreal.ca,  acm@muc.de,  gregory@heytings.org,
>   mattiase@acm.org,  philipk@posteo.net,  silent2600@gmail.com,
>   emacs-devel@gnu.org
> Date: Thu, 04 Aug 2022 15:31:21 +0800
> 
> > Modes do what they want, and in some cases we have hard time even
> > convincing the mode developers that they should try to avoid doing
> > that.
> 
> Then maybe we should not impose our opinion of what narrowing is best on
> major mode developers?

We decided that we do want to impose our opinion, because not doing so
results in Emacs being unusable, which is a long-standing gripe of our
users.

> Forcing restrictions on user code never works.

fontification-functions are not user code.

> Sooner or later, developers will start performing fontification in a
> timer, in order to widen past the "locked narrowing".

Long-running timer functions, if they are not interruptible, are a
clear bug in the package that does such things, so any such timers
will come back as a boomerang to those developers.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> We decided that we do want to impose our opinion, because not doing so
> results in Emacs being unusable, which is a long-standing gripe of our
> users.

Since this problem wasn't bad enough for major mode developers to care
about in the past, what makes you think they will fix it now?

IMHO, it is much better to make what can be fixed right now (by being
present in the Emacs repository) explictly use the "new" narrowing, and
not interfere with user code that does not concern us.

And what if someone wants to write a fontification function whose
results depend, for example, on a single character at a known position
outside of the accessible region? That cannot result in a freeze, but is
impossible outside of a timer function when "locked narrowing" is in
effect.

> fontification-functions are not user code.

Anything not under our control is user code that we are forcing
restrictions onto.

> Long-running timer functions, if they are not interruptible, are a
> clear bug in the package that does such things, so any such timers
> will come back as a boomerang to those developers.

I'm just trying to demonstrate what will happen once the forced
narrowing starts to interfere with the operation of various pieces of
third party code, which might possibly be discovered some time after
Emacs 29 is released.  Long-running fontification was not a significant
source of complaints for their developers in the past, and things are
likely to remain that way.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: monnier@iro.umontreal.ca,  acm@muc.de,  gregory@heytings.org,
>   mattiase@acm.org,  philipk@posteo.net,  silent2600@gmail.com,
>   emacs-devel@gnu.org
> Date: Thu, 04 Aug 2022 16:42:33 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > We decided that we do want to impose our opinion, because not doing so
> > results in Emacs being unusable, which is a long-standing gripe of our
> > users.
> 
> Since this problem wasn't bad enough for major mode developers to care
> about in the past, what makes you think they will fix it now?

Nothing.  I actually don't really expect them to do anything in this
regard, which is why solving that in display code sounds like a much
better alternative.

> IMHO, it is much better to make what can be fixed right now (by being
> present in the Emacs repository) explictly use the "new" narrowing, and
> not interfere with user code that does not concern us.

I cannot parse this, sorry.  What do you mean by "being present in the
Emacs repository"? who or what should be present there?  And what do
you mean by "the new narrowing"?  And what does "user code", whatever
that is, have to do with this issue?

> And what if someone wants to write a fontification function whose
> results depend, for example, on a single character at a known position
> outside of the accessible region? That cannot result in a freeze

I cannot reason about something as abstract and theoretical.

One thing you need to keep in mind: fontifications never have only a
local effect.  If some part of the buffer is re-fontified, it can
potentially affect all the rest of the buffer, because display layout
calculations are affected.  So "cannot result in a freeze" is
extremely optimistic, IME.  But again, unless you show some code and
explain how is that a real-life use case that should be of interest,
this discussion is not useful.

> > fontification-functions are not user code.
> 
> Anything not under our control is user code that we are forcing
> restrictions onto.

Not in my book.  At least the major modes that come with Emacs are not
"user code" and are under our control.

> > Long-running timer functions, if they are not interruptible, are a
> > clear bug in the package that does such things, so any such timers
> > will come back as a boomerang to those developers.
> 
> I'm just trying to demonstrate what will happen once the forced
> narrowing starts to interfere with the operation of various pieces of
> third party code, which might possibly be discovered some time after
> Emacs 29 is released.  Long-running fontification was not a significant
> source of complaints for their developers in the past, and things are
> likely to remain that way.

Users did and do complain about locked-up Emacs when they try to edit
files with long lines.  They just didn't realize one of the reasons
was font-lock (or syntax-ppss), so they didn't complain specifically
about that.  But every complaint you see about these situations is
basically a complaint about font-lock and its syntax-parsing parts.

Anyway, this discussion doesn't lead anywhere.  We have decided to try
this way of solving a long-standing problem in Emacs, and no amount of
talking will cause us to change that decision.  Only code that makes
font-lock and syntax.c significantly faster, and/or reports about
specific issues (as opposed to general semi-philosophical concerns)
caused by these changes, can affect both the implementation of these
changes and our future decisions about its defaults.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Stefan Monnier]

Po Lu [2022-08-04 09:02:02] wrote:
> Gregory Heytings <gregory@heytings.org> writes:
>> A sequence of negatively loaded words followed by a question mark is
>> not a question, no.
> Whether or not I asked a question is not up to you to decide.

Please, let's try and stick to the technical issues, and remember that
we're all just trying to have fun while improving our favorite editor.


        Stefan

Re: emacs master + org Wrong type argument: number-or-marker-p

[Alan Mackenzie]

Hello, Eli.

On Thu, Aug 04, 2022 at 12:06:45 +0300, Eli Zaretskii wrote:
> > From: Po Lu <luangruo@yahoo.com>
> > Cc: monnier@iro.umontreal.ca,  acm@muc.de,  gregory@heytings.org,
> >   mattiase@acm.org,  philipk@posteo.net,  silent2600@gmail.com,
> >   emacs-devel@gnu.org
> > Date: Thu, 04 Aug 2022 16:42:33 +0800

> > Eli Zaretskii <eliz@gnu.org> writes:

> > And what if someone wants to write a fontification function whose
> > results depend, for example, on a single character at a known position
> > outside of the accessible region? That cannot result in a freeze

> I cannot reason about something as abstract and theoretical.

CC Mode is such a mode, though it depends on more than just a single
character outside the accessible region.

[ .... ]

> > I'm just trying to demonstrate what will happen once the forced
> > narrowing starts to interfere with the operation of various pieces of
> > third party code, which might possibly be discovered some time after
> > Emacs 29 is released.  Long-running fontification was not a
> > significant source of complaints for their developers in the past,
> > and things are likely to remain that way.

> Users did and do complain about locked-up Emacs when they try to edit
> files with long lines.  They just didn't realize one of the reasons
> was font-lock (or syntax-ppss), so they didn't complain specifically
> about that.  But every complaint you see about these situations is
> basically a complaint about font-lock and its syntax-parsing parts.

Has anybody asked the question why is font-lock so slow on these long
lines?  The answer is surely not the use of widen and narrow-to-region.
A piece of text takes exactly as long to fontify when its buffer is
narrowed as when it is not.

I think (though I have not analysed it any further) the cause of this
slowness is font-lock fontifying from BOL to EOL.  That's an awful lot of
fontification if we have long lines.  If this is the case, a better
solution to the problem would be to restrict the font-locked region to,
say, the visible line.  Or to the window.  Or something like that.

The current "solution" breaks things, because it doesn't fix what is
broken and fixes what isn't broken.  In particular, it breaks CC Mode
when there are long lines in a buffer.

> Anyway, this discussion doesn't lead anywhere.  We have decided to try
> this way of solving a long-standing problem in Emacs, and no amount of
> talking will cause us to change that decision.  Only code that makes
> font-lock and syntax.c significantly faster, and/or reports about
> specific issues (as opposed to general semi-philosophical concerns)
> caused by these changes, can affect both the implementation of these
> changes and our future decisions about its defaults.

I don't want to be provocative, but having opcodes that only work most of
the time rather than all of the time isn't a mere semi-philosophical
concern.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs master + org Wrong type argument: number-or-marker-p

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> and/or reports about specific issues

Didn't Alan come up with one such report? (I admit to not having read it
very carefully, C++ raw strings sound like a mis-feature to me.)

Re: emacs master + org Wrong type argument: number-or-marker-p

[Werner LEMBERG]

> [...] C++ raw strings sound like a mis-feature to me.

It helps a lot if you have very long strings, like texinfo
documentation for Scheme functions implemented in C++.  In LilyPond,
we have recently introduced them for exactly that purpose, and IMHO it
makes the source code much more readable.


    Werner

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: monnier@iro.umontreal.ca,  acm@muc.de,  gregory@heytings.org,
>   mattiase@acm.org,  philipk@posteo.net,  silent2600@gmail.com,
>   emacs-devel@gnu.org
> Date: Thu, 04 Aug 2022 18:26:58 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > and/or reports about specific issues
> 
> Didn't Alan come up with one such report? (I admit to not having read it
> very carefully, C++ raw strings sound like a mis-feature to me.)

Not really, no.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 10:18:21 +0000
> Cc: Po Lu <luangruo@yahoo.com>, monnier@iro.umontreal.ca,
>   gregory@heytings.org, mattiase@acm.org, philipk@posteo.net,
>   silent2600@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > And what if someone wants to write a fontification function whose
> > > results depend, for example, on a single character at a known position
> > > outside of the accessible region? That cannot result in a freeze
> 
> > I cannot reason about something as abstract and theoretical.
> 
> CC Mode is such a mode, though it depends on more than just a single
> character outside the accessible region.

And what catastrophic results does the current master produce with CC
Mode, when you edit files with very long lines, due to this narrowing?

> > Users did and do complain about locked-up Emacs when they try to edit
> > files with long lines.  They just didn't realize one of the reasons
> > was font-lock (or syntax-ppss), so they didn't complain specifically
> > about that.  But every complaint you see about these situations is
> > basically a complaint about font-lock and its syntax-parsing parts.
> 
> Has anybody asked the question why is font-lock so slow on these long
> lines?  The answer is surely not the use of widen and narrow-to-region.
> A piece of text takes exactly as long to fontify when its buffer is
> narrowed as when it is not.

Of course.  But the problem is that font-lock was found to examine
much larger portions of the buffer than the ones to which we now
restrict it in those cases.  And the speedup is evident.  What does
that tell you?

> I think (though I have not analysed it any further) the cause of this
> slowness is font-lock fontifying from BOL to EOL.  That's an awful lot of
> fontification if we have long lines.  If this is the case, a better
> solution to the problem would be to restrict the font-locked region to,
> say, the visible line.  Or to the window.  Or something like that.

That's what we do: we restrict it to somewhat more than fits in the
window.  But the files in questions have lines that are much longer
than that.

> The current "solution" breaks things, because it doesn't fix what is
> broken and fixes what isn't broken.  In particular, it breaks CC Mode
> when there are long lines in a buffer.

No, the current solution sacrifices some of the font-lock correctness
to give us a usable Emacs.  I say it's a justified tradeoff.

> > Anyway, this discussion doesn't lead anywhere.  We have decided to try
> > this way of solving a long-standing problem in Emacs, and no amount of
> > talking will cause us to change that decision.  Only code that makes
> > font-lock and syntax.c significantly faster, and/or reports about
> > specific issues (as opposed to general semi-philosophical concerns)
> > caused by these changes, can affect both the implementation of these
> > changes and our future decisions about its defaults.
> 
> I don't want to be provocative, but having opcodes that only work most of
> the time rather than all of the time isn't a mere semi-philosophical
> concern.

They work all the time when there are no long lines.  When there are
long lines, they work slightly less correctly, and the lack of
correctness is only visible in the fontifications.  Hardly a disaster,
if you keep in mind that one of the previously recommended solutions
was to turn on so-long-mode, which turns off font-lock (to tell
nothing of the not-so-distant past, when Emacs didn't have font-lock
at all).

IOW, it's a tradeoff: we give up some of the font-lock correctness,
and in return gain a usable Emacs.

Re: Opcode Versioning Was: emacs master + org Wrong type argument: number-or-marker-p

[Sam Steingold]

> * Stefan Monnier <zbaavre@veb.hzbagerny.pn> [2022-08-02 18:10:02 -0400]:
>
>> compiled file, so that Emacs can reject loading a file with an incorrect
>> version.
>
> Emacs instead aims to be able to keep loading old byte-compiled files
> so we try hard to change the byte-codes only in ways which are
> backward compatible.

This is a very noble aim.
Does it ever happen that Emacs breaks byte-codes as a design decision?
Or is Emacs _supposed_ to be able to load ELC from 1990?

Thank you for your reply.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.2113
http://childpsy.net http://calmchildstories.com http://steingoldpsychology.com
https://fairforall.org https://www.dhimmitude.org https://jij.org
You think Oedipus had a problem -- Adam was Eve's mother.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Alan Mackenzie]

Hello, Eli.

On Thu, Aug 04, 2022 at 16:18:40 +0300, Eli Zaretskii wrote:
> > Date: Thu, 4 Aug 2022 10:18:21 +0000
> > Cc: Po Lu <luangruo@yahoo.com>, monnier@iro.umontreal.ca,
> >   gregory@heytings.org, mattiase@acm.org, philipk@posteo.net,
> >   silent2600@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > And what if someone wants to write a fontification function whose
> > > > results depend, for example, on a single character at a known position
> > > > outside of the accessible region? That cannot result in a freeze

> > > I cannot reason about something as abstract and theoretical.

> > CC Mode is such a mode, though it depends on more than just a single
> > character outside the accessible region.

> And what catastrophic results does the current master produce with CC
> Mode, when you edit files with very long lines, due to this narrowing?

This isn't yet known.  There is no evidence, no argument even, that such
"catastrophic results" won't occur.  Wrong fontification is already
evident.

[ .... ]

> > Has anybody asked the question why is font-lock so slow on these long
> > lines?  The answer is surely not the use of widen and
> > narrow-to-region.  A piece of text takes exactly as long to fontify
> > when its buffer is narrowed as when it is not.

> Of course.  But the problem is that font-lock was found to examine
> much larger portions of the buffer than the ones to which we now
> restrict it in those cases.  And the speedup is evident.  What does
> that tell you?

That you can speed things up by omitting part of the work.  Possibly that
font-lock is insufficiently flexible in what it offers major modes.

> > I think (though I have not analysed it any further) the cause of this
> > slowness is font-lock fontifying from BOL to EOL.  That's an awful lot of
> > fontification if we have long lines.  If this is the case, a better
> > solution to the problem would be to restrict the font-locked region to,
> > say, the visible line.  Or to the window.  Or something like that.

> That's what we do: we restrict it to somewhat more than fits in the
> window.

I meant doing this in a way which doesn't introduce bugs.  Something like
amending/replacing font-lock-extend-region-wholelines, for example.

> But the files in questions have lines that are much longer than that.

> > The current "solution" breaks things, because it doesn't fix what is
> > broken and fixes what isn't broken.  In particular, it breaks CC Mode
> > when there are long lines in a buffer.

> No, the current solution sacrifices some of the font-lock correctness
> ....

"Sacrificing correctness" sounds like a euphemism for tolerating bugs.

> .... to give us a usable Emacs.  I say it's a justified tradeoff.

Better would be to get a usable Emacs without introducing bugs.  CC Mode
(amongst other packages) is a victim of this tradeoff, yet I wasn't
alerted to the decision being taken, I just found out about it by chance.
I CARE about the correctness of CC Mode, and have spent a lot of time and
energy over the last 20 years tending to it.  Now it's being sacrificed,
even if only in an edge case, with barely a second thought - along with
an unknown number of other packages.

> > > Anyway, this discussion doesn't lead anywhere.  We have decided to
> > > try this way of solving a long-standing problem in Emacs, and no
> > > amount of talking will cause us to change that decision.  Only code
> > > that makes font-lock and syntax.c significantly faster, and/or
> > > reports about specific issues (as opposed to general
> > > semi-philosophical concerns) caused by these changes, can affect
> > > both the implementation of these changes and our future decisions
> > > about its defaults.

> > I don't want to be provocative, but having opcodes that only work
> > most of the time rather than all of the time isn't a mere
> > semi-philosophical concern.

> They work all the time when there are no long lines.

They don't work all the time, since sometimes there are long lines.

> When there are long lines, they work slightly less correctly, ....

Correctness doesn't admit of degrees.  Something is either correct, or
it's incorrect.

`widen' doesn't work at all in the pertinent circumstances.  It is as
though somebody unconnected with a project steals into the source code
and secretly removes `(widen)' and `(narrow-to-region ...)' without
telling the maintainer, who'll likely have to debug things later.
 
> .... and the lack of correctness is only visible in the fontifications.

That hasn't been shown.  Arbitrary Lisp can be executed through
fontification-functions and all the hooks in font-lock.el and
jit-lock.el.

> Hardly a disaster, if you keep in mind that one of the previously
> recommended solutions was to turn on so-long-mode, which turns off
> font-lock (to tell nothing of the not-so-distant past, when Emacs
> didn't have font-lock at all).

Do these long lines occur in modes where fontification is important?  I
don't know as I haven't seen any test files.

> IOW, it's a tradeoff: we give up some of the font-lock correctness,
> and in return gain a usable Emacs.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs master + org Wrong type argument: number-or-marker-p

[Eli Zaretskii]

> Date: Thu, 4 Aug 2022 16:07:48 +0000
> Cc: luangruo@yahoo.com, monnier@iro.umontreal.ca, gregory@heytings.org,
>   mattiase@acm.org, philipk@posteo.net, silent2600@gmail.com,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Of course.  But the problem is that font-lock was found to examine
> > much larger portions of the buffer than the ones to which we now
> > restrict it in those cases.  And the speedup is evident.  What does
> > that tell you?
> 
> That you can speed things up by omitting part of the work.  Possibly that
> font-lock is insufficiently flexible in what it offers major modes.

And how would you propose to solve that, given that it remains
unsolved for at least two decades?

> I meant doing this in a way which doesn't introduce bugs.  Something like
> amending/replacing font-lock-extend-region-wholelines, for example.

Feel free to work on this.  If the results are better, we will
certainly prefer using it, and will then have to revisit the changes
we are discussing here, and decide whether they are still needed.

But we are not there yet, and it sounds wrong to me to prevent our
users from having a usable Emacs when editing these files just because
our ideal solutions are not yet available, when less ideal solutions
yield satisfactory results.

> > No, the current solution sacrifices some of the font-lock correctness
> > ....
> 
> "Sacrificing correctness" sounds like a euphemism for tolerating bugs.

Minor bugs.  Very minor bugs.

> > .... to give us a usable Emacs.  I say it's a justified tradeoff.
> 
> Better would be to get a usable Emacs without introducing bugs.

Yes, we all love cheap and perfect solutions, don't we?  But there are
no such solutions in this case, so we need to choose among
not-so-cheap and not-so-perfect ones instead.

> Now it's being sacrificed, even if only in an edge case, with barely
> a second thought - along with an unknown number of other packages.

That's unfair, to say the least.  A lot of thought and discussions
went into these changes, even if you didn't notice them.  To say
nothing about the development and testing efforts.

> > > I don't want to be provocative, but having opcodes that only work
> > > most of the time rather than all of the time isn't a mere
> > > semi-philosophical concern.
> 
> > They work all the time when there are no long lines.
> 
> They don't work all the time, since sometimes there are long lines.

That's the same thing.  You just see the empty 1/100th of the glass.

> > When there are long lines, they work slightly less correctly, ....
> 
> Correctness doesn't admit of degrees.  Something is either correct, or
> it's incorrect.

Then CC Mode is incorrect and broken, because it at times produces
wrong fontification and wrong indentation.  Let's throw it out.

> `widen' doesn't work at all in the pertinent circumstances.

Then make your mode avoid widening in those circumstances.

> > .... and the lack of correctness is only visible in the fontifications.
> 
> That hasn't been shown.

Show us any other problems, and then let's talk.

> Arbitrary Lisp can be executed through fontification-functions and
> all the hooks in font-lock.el and jit-lock.el.

Arbitrary Lisp can still be executed, it should just avoid going far
away from the window's display area when the buffer has very long
lines.  In particular, it should not begin from the beginning of a
line, nor insist on examining the entire line from BOL to EOL.

> > Hardly a disaster, if you keep in mind that one of the previously
> > recommended solutions was to turn on so-long-mode, which turns off
> > font-lock (to tell nothing of the not-so-distant past, when Emacs
> > didn't have font-lock at all).
> 
> Do these long lines occur in modes where fontification is important?  I
> don't know as I haven't seen any test files.

The Emacs project decided long ago that fontifications are important
in all modes.  That's why we have font-lock turned on by default; it
wasn't like that once upon a time.  But if you or someone prefers no
fontifications to fontifications that are sometimes, rarely,
incorrect, you can always turn off font-lock; the changes we are
discussing didn't take away that possibility.

Re: emacs master + org Wrong type argument: number-or-marker-p

[Stefan Monnier]

> We decided that we do want to impose our opinion, because not doing so
> results in Emacs being unusable, which is a long-standing gripe of our
> users.

FWIW, I don't think the fact that Emacs suffered from "the long lines
problem" is due to package maintainers not being able to clean up
their act.

I never bothered to try something like `synax-wholeline-max` in the past
because even in fundamental-mode long lines were a problem, so clearly
the motivation to fix this bottleneck wasn't very high.

The situation now is completely different.
We can now announce "Emacs is now finally fast even with long lines,
except in those few major modes that still misbehave", so the onus will
now be on the package maintainers to clean up their act, and we can have
a clear guideline for how to do that.


        Stefan

Re: Opcode Versioning Was: emacs master + org Wrong type argument: number-or-marker-p

[Stefan Monnier]

Sam Steingold [2022-08-04 10:59:46] wrote:
>> Emacs instead aims to be able to keep loading old byte-compiled files
>> so we try hard to change the byte-codes only in ways which are
>> backward compatible.
> This is a very noble aim.
> Does it ever happen that Emacs breaks byte-codes as a design decision?

IIRC we did drop some byte-codes, yes.  Most of them are byte-codes
which had never been generated by our byte-compiler, but I think there
were a few other cases where we judged that it had been enough years.

> Or is Emacs _supposed_ to be able to load ELC from 1990?

More or less but not really: even if the byte code itself should
probably work, it's likely to bump into incompatible changes in
the functions we define (or fail to).


        Stefan

Re: emacs master + org Wrong type argument: number-or-marker-p

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Especially when the one who writes these words has 
  > personally been given all the elements to answer that supposed question 
  > less than two weeks ago, here: 

I don't know who you have in mind -- you didn't explicitly say -- and
I don't know what pieces of information that person received.

But I know that different things are obvious to different people.  It
is not at all surprising that person A sees several pieces of
information and perceives them as pointing to one obvious conclusion,
while person B sees the same pieces of information and doesn't see
that they imply any conclusion.
 
We're all working for the same goal: to make Emacs better.  Let's be kind
to each other as we do it.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs master + org Wrong type argument: number-or-marker-p

[Julien Cubizolles]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 01 Aug 2022 16:11:33 +0000
>> From: Gregory Heytings <gregory@heytings.org>
>> cc: Philip Kaludercic <philipk@posteo.net>, silent2600@gmail.com, 
>>     emacs-devel@gnu.org
>> 
>> 
>> >> What caused the issue?
>> >
>> > I don't know.
>> >
>> 
>> I think it's the new optional argument to narrow-to-region, which adds an 
>> argument to the narrow-to-region opcode in byte-compiled code.
>
> If that's the reason, why didn't it affect more files?
> narrow-to-region is called in much more places.

I had the same error messages with several other packages (bbdb,
yas…). Force compiling all my packages fixed it.

-- 
Julien Cubizolles