* Difficulties byte-compiling very large .el file @ 2009-08-19 21:49 Aemon Cannon 2009-08-20 8:27 ` Andreas Schwab 2009-08-25 10:16 ` Aidan Kehoe 0 siblings, 2 replies; 12+ messages in thread From: Aemon Cannon @ 2009-08-19 21:49 UTC (permalink / raw) To: emacs-devel This is a re-post of a question I asked of gnu.emacs.help. Xah Lee suggested I try this mailing list. ------------------------------------------------------------------------------------------------------- I've been working on a project to create an elisp target for the ANTLR parser-generator. Here's the project page: http://github.com/aemoncannon/antlr-elisp/tree/experimental The generated parser for ActionScript 3 is over 30k lines long and fails to byte-compile with the message in Emacs 22: as3_elispParser.el:31515:33:Error: Invalid character: 256, #o400, #x100 ..and this message in Emacs 23: ../test/grammars/as3_elispParser.el:31515:33:Error: Args out of range: 256, 0, 255 Here's the source in question: http://aemon.com/file_dump/as3_elispParser.el If you're interested in trying to compile, you'll need: http://github.com/aemoncannon/antlr-elisp/raw/c411f29743e3182523e0ad27a9384b6c4210d55e/src/runtime/ELisp/a3el-runtime.el I don't think the error is actually related to the content of the file, as there are no non-ascii characters in the file, and deleting seemingly arbitrary chunks makes the error go away. There are lots of "Warning: reference to free variable ...." warnings, but I understand where those are coming from and I'm not worried about them. Bear in mind there are *lots* of macros being expanded. Most of the forms with the prefix 'a3el' are macros. Any ideas? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-19 21:49 Difficulties byte-compiling very large .el file Aemon Cannon @ 2009-08-20 8:27 ` Andreas Schwab 2009-08-20 13:07 ` Aemon Cannon 2009-08-25 10:16 ` Aidan Kehoe 1 sibling, 1 reply; 12+ messages in thread From: Andreas Schwab @ 2009-08-20 8:27 UTC (permalink / raw) To: aemoncannon; +Cc: emacs-devel Aemon Cannon <aemoncannon@gmail.com> writes: > The generated parser for ActionScript 3 is over 30k lines long and > fails to byte-compile with the message in Emacs 22: > as3_elispParser.el:31515:33:Error: Invalid character: 256, #o400, > #x100 > > ..and this message in Emacs 23: > ../test/grammars/as3_elispParser.el:31515:33:Error: Args out of range: > 256, 0, 255 Probably the bytecode string is becoming too large so that a byte-goto operand overflows. Try this patch to verify: --- lisp/emacs-lisp/bytecomp.el.~2.248.~ 2009-08-06 13:15:39.000000000 +0200 +++ lisp/emacs-lisp/bytecomp.el 2009-08-20 10:26:05.000000000 +0200 @@ -853,7 +853,8 @@ otherwise pop it") (t ; Absolute jump (setq pc (car (cdr (car bytes)))) ; Pick PC from tag (setcar (cdr bytes) (logand pc 255)) - (setcar bytes (lsh pc -8)))) + (setcar bytes (lsh pc -8)) + (if (> (car bytes) 255) (error "Bytecode overflow")))) (setq patchlist (cdr patchlist)))) (apply 'unibyte-string (nreverse bytes)))) Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-20 8:27 ` Andreas Schwab @ 2009-08-20 13:07 ` Aemon Cannon 2009-08-20 13:23 ` Andreas Schwab 2009-08-20 18:30 ` Eli Zaretskii 0 siblings, 2 replies; 12+ messages in thread From: Aemon Cannon @ 2009-08-20 13:07 UTC (permalink / raw) To: Andreas Schwab; +Cc: emacs-devel Bingo! ../test/grammars/as3_elispParser.el:31515:33:Error: Bytecode overflow I'm not super familiar with emacs byte-compiling, but my guess is that increasing the size of the operands is not an option. Should I look for a conditional that spans a lot of code, and try to fix it that way? Thoughts? On Thu, Aug 20, 2009 at 4:27 AM, Andreas Schwab<schwab@linux-m68k.org> wrote: > Aemon Cannon <aemoncannon@gmail.com> writes: > >> The generated parser for ActionScript 3 is over 30k lines long and >> fails to byte-compile with the message in Emacs 22: >> as3_elispParser.el:31515:33:Error: Invalid character: 256, #o400, >> #x100 >> >> ..and this message in Emacs 23: >> ../test/grammars/as3_elispParser.el:31515:33:Error: Args out of range: >> 256, 0, 255 > > Probably the bytecode string is becoming too large so that a byte-goto > operand overflows. Try this patch to verify: > > --- lisp/emacs-lisp/bytecomp.el.~2.248.~ 2009-08-06 13:15:39.000000000 +0200 > +++ lisp/emacs-lisp/bytecomp.el 2009-08-20 10:26:05.000000000 +0200 > @@ -853,7 +853,8 @@ otherwise pop it") > (t ; Absolute jump > (setq pc (car (cdr (car bytes)))) ; Pick PC from tag > (setcar (cdr bytes) (logand pc 255)) > - (setcar bytes (lsh pc -8)))) > + (setcar bytes (lsh pc -8)) > + (if (> (car bytes) 255) (error "Bytecode overflow")))) > (setq patchlist (cdr patchlist)))) > (apply 'unibyte-string (nreverse bytes)))) > > > Andreas. > > -- > Andreas Schwab, schwab@linux-m68k.org > GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 > "And now for something completely different." > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-20 13:07 ` Aemon Cannon @ 2009-08-20 13:23 ` Andreas Schwab 2009-08-20 14:29 ` Aemon Cannon 2009-08-20 18:30 ` Eli Zaretskii 1 sibling, 1 reply; 12+ messages in thread From: Andreas Schwab @ 2009-08-20 13:23 UTC (permalink / raw) To: aemoncannon; +Cc: emacs-devel Aemon Cannon <aemoncannon@gmail.com> writes: > Should I look for a conditional that spans a lot of code, and try to > fix it that way? Use less macros. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-20 13:23 ` Andreas Schwab @ 2009-08-20 14:29 ` Aemon Cannon 0 siblings, 0 replies; 12+ messages in thread From: Aemon Cannon @ 2009-08-20 14:29 UTC (permalink / raw) To: Andreas Schwab; +Cc: emacs-devel I can't really do that. The idea is for these parsers to be used interactively in language modes, so they need to be fast. Macro use has been crucial in this area. Also, ActionScript 3 is not the most complicated grammar in the world. What happens when a bigger parser overflows my guess for the value of 'less' ? On Thu, Aug 20, 2009 at 9:23 AM, Andreas Schwab<schwab@linux-m68k.org> wrote: > Aemon Cannon <aemoncannon@gmail.com> writes: > >> Should I look for a conditional that spans a lot of code, and try to >> fix it that way? > > Use less macros. > > Andreas. > > -- > Andreas Schwab, schwab@linux-m68k.org > GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 > "And now for something completely different." > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-20 13:07 ` Aemon Cannon 2009-08-20 13:23 ` Andreas Schwab @ 2009-08-20 18:30 ` Eli Zaretskii 1 sibling, 0 replies; 12+ messages in thread From: Eli Zaretskii @ 2009-08-20 18:30 UTC (permalink / raw) To: aemoncannon; +Cc: emacs-devel > Date: Thu, 20 Aug 2009 09:07:01 -0400 > From: Aemon Cannon <aemoncannon@gmail.com> > Cc: emacs-devel@gnu.org > > Bingo! > ../test/grammars/as3_elispParser.el:31515:33:Error: Bytecode overflow > > I'm not super familiar with emacs byte-compiling, but my guess is that > increasing > the size of the operands is not an option. Should I look for a > conditional that spans > a lot of code, and try to fix it that way? > > Thoughts? Submit a bug report ("M-x report-emacs-bug RET") and someone will take care of it. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-19 21:49 Difficulties byte-compiling very large .el file Aemon Cannon 2009-08-20 8:27 ` Andreas Schwab @ 2009-08-25 10:16 ` Aidan Kehoe 2009-08-25 11:07 ` Miles Bader 2009-08-29 0:37 ` Stefan Monnier 1 sibling, 2 replies; 12+ messages in thread From: Aidan Kehoe @ 2009-08-25 10:16 UTC (permalink / raw) To: aemoncannon; +Cc: emacs-devel Ar an naoú lá déag de mí Lúnasa, scríobh Aemon Cannon: > [...] Here's the source in question: > http://aemon.com/file_dump/as3_elispParser.el > > If you're interested in trying to compile, you'll need: > http://github.com/aemoncannon/antlr-elisp/raw/c411f29743e3182523e0ad27a9384b6c4210d55e/src/runtime/ELisp/a3el-runtime.el > > I don't think the error is actually related to the content of the > file, as there are no non-ascii characters in the file, and deleting > seemingly arbitrary chunks makes the error go away. Tangentially to this, you should be aware that several of your integer constants are silently overflowing in that file. With an XEmacs with 31-bit integers and no bignum support, the Lisp reader throws an overflow error twice for me; GNU Emacs has (IIRC) 27-bit integers on 32-bit platforms and doesn’t error on reading integers it can’t represent, so more than two of your integer constants may be corrupt. The constants that overflow for me are 2147483647 on line 10006 and again on line 2291. I can work around this (though there’s still lots of thrashing when compiling the file) by using most-positive-fixnum instead, modifying #'a3el-lookahead-let-bindings to call #'symbol-value on max-k if it doesn’t satisfy #'numberp. -- ¿Dónde estará ahora mi sobrino Yoghurtu Nghe, que tuvo que huir precipitadamente de la aldea por culpa de la escasez de rinocerontes? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-25 10:16 ` Aidan Kehoe @ 2009-08-25 11:07 ` Miles Bader 2009-08-25 11:50 ` Aidan Kehoe 2009-08-29 0:37 ` Stefan Monnier 1 sibling, 1 reply; 12+ messages in thread From: Miles Bader @ 2009-08-25 11:07 UTC (permalink / raw) To: Aidan Kehoe; +Cc: aemoncannon, emacs-devel Aidan Kehoe <kehoea@parhasard.net> writes: > GNU Emacs has (IIRC) 27-bit integers on 32-bit platforms ``The range of values for integers in Emacs Lisp is -268435456 to 268435455 (29 bits; i.e., -2**28 to 2**28 - 1) on most machines.'' -Miles -- Bride, n. A woman with a fine prospect of happiness behind her. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-25 11:07 ` Miles Bader @ 2009-08-25 11:50 ` Aidan Kehoe 2009-08-25 15:04 ` Aemon Cannon 0 siblings, 1 reply; 12+ messages in thread From: Aidan Kehoe @ 2009-08-25 11:50 UTC (permalink / raw) To: Miles Bader; +Cc: aemoncannon, emacs-devel Ar an cúigiú lá is fiche de mí Lúnasa, scríobh Miles Bader: > Aidan Kehoe <kehoea@parhasard.net> writes: > > GNU Emacs has (IIRC) 27-bit integers on 32-bit platforms > > ``The range of values for integers in Emacs Lisp is -268435456 to > 268435455 (29 bits; i.e., -2**28 to 2**28 - 1) on most machines.'' Thanks for the correction. Aemon’s problem remains, though, independent of that. Is there a good reason you don’t error on encountering integers the emacs binary can’t represent? Portable code needs to be prepared for that anyway. -- ¿Dónde estará ahora mi sobrino Yoghurtu Nghe, que tuvo que huir precipitadamente de la aldea por culpa de la escasez de rinocerontes? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-25 11:50 ` Aidan Kehoe @ 2009-08-25 15:04 ` Aemon Cannon 2009-09-01 22:41 ` Aemon Cannon 0 siblings, 1 reply; 12+ messages in thread From: Aemon Cannon @ 2009-08-25 15:04 UTC (permalink / raw) To: Aidan Kehoe; +Cc: emacs-devel, Miles Bader [-- Attachment #1: Type: text/plain, Size: 2840 bytes --] Aidan, Thanks for pointing out the overflow issue. As you may have guessed, those constants are both max-value for a signed 32 bit integer, which is what ANTLR spits out in LL* situations (unbounded values for max-k). I may be able to specify the max value for elisp in the ANTLR template.... RE: My original problem, I narrowed it down a bit and submitted a report at: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=4251 I've since sent a followup with the following code, that also fails: ;;---------------- (defmacro many-forms () (let ((body '())) (dotimes (i 20000) (setq body (cons '(message "more") body))) `(progn ,@body))) (many-forms) (if (eq 1 a) (message "dude") (message "else")) ;;-------------- I had assumed that the goto opcode used a 16 bit relative offset, but this snippet seems to indicate that it's not relative to the conditional itself. I also noticed in bytecode.c that the goto opcodes all use: ..... stack.pc = stack.byte_string_start + op; ..... where 'op' is a 16 bit int. This indicates that the jump is measured in relation to the top of the current byte_string, so if the conditional is in its own byte_string, it should work: ;;-------------------- (defmacro many-forms () (let ((body '())) (dotimes (i 20000) (setq body (cons '(message "more") body))) `(progn ,@body))) (many-forms) (defun bar () (if (eq 1 a) (message "dude") (message "else"))) ;; ---------------------------- Which does work. Wrapping in lambda also works. So now I'm thinking that as3_elispParser.el must have a single function body whose bytecode overflows the 16bit limit. As removing chunks of code from the top-level seems to alleviate the error (the lines with (a3el-parser-bitset ..) for instance), I'm guessing that the problematic byte_string corresponds to the top-level of the file. I may be able to wrap this top-level setup code into smaller functions.... On Tue, Aug 25, 2009 at 7:50 AM, Aidan Kehoe <kehoea@parhasard.net> wrote: > > Ar an cúigiú lá is fiche de mí Lúnasa, scríobh Miles Bader: > > > Aidan Kehoe <kehoea@parhasard.net> writes: > > > GNU Emacs has (IIRC) 27-bit integers on 32-bit platforms > > > > ``The range of values for integers in Emacs Lisp is -268435456 to > > 268435455 (29 bits; i.e., -2**28 to 2**28 - 1) on most machines.'' > > Thanks for the correction. Aemon’s problem remains, though, independent of > that. Is there a good reason you don’t error on encountering integers the > emacs binary can’t represent? Portable code needs to be prepared for that > anyway. > > -- > ¿Dónde estará ahora mi sobrino Yoghurtu Nghe, que tuvo que huir > precipitadamente de la aldea por culpa de la escasez de rinocerontes? > [-- Attachment #2: Type: text/html, Size: 3580 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-25 15:04 ` Aemon Cannon @ 2009-09-01 22:41 ` Aemon Cannon 0 siblings, 0 replies; 12+ messages in thread From: Aemon Cannon @ 2009-09-01 22:41 UTC (permalink / raw) To: Aidan Kehoe; +Cc: emacs-devel, Miles Bader [-- Attachment #1: Type: text/plain, Size: 3427 bytes --] To followup: I'm now working around the original issue by wrapping parts of the generated parser in in lambdas thusly: (defmacro a3el-do-in-thunk (&rest body) (let ((thunk-name (gensym "thunk"))) `(progn (setq ,thunk-name (lambda () ,@body)) (funcall ,thunk-name)))) Works like a charm. Thanks to everyone for the help. On Tue, Aug 25, 2009 at 11:04 AM, Aemon Cannon <aemoncannon@gmail.com>wrote: > Aidan, > Thanks for pointing out the overflow issue. As you may have guessed, those > constants are both max-value for a signed 32 bit integer, which is what > ANTLR spits out in LL* situations (unbounded values for max-k). I may be > able to specify the max value for elisp in the ANTLR template.... > > RE: My original problem, > I narrowed it down a bit and submitted a report at: > http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=4251 > I've since sent a followup with the following code, that also fails: > ;;---------------- > > (defmacro many-forms () > (let ((body '())) > (dotimes (i 20000) > (setq body (cons '(message "more") body))) > `(progn ,@body))) > > (many-forms) > > (if (eq 1 a) > (message "dude") > (message "else")) > > ;;-------------- > > I had assumed that the goto opcode used a 16 bit relative offset, but this > snippet seems to indicate that it's not relative to the conditional itself. > I also noticed in bytecode.c that the goto opcodes all use: > ..... > stack.pc = stack.byte_string_start + op; > ..... > where 'op' is a 16 bit int. This indicates that the jump is measured in > relation to the top of the current byte_string, so if the conditional is in > its own byte_string, it should work: > ;;-------------------- > (defmacro many-forms () > (let ((body '())) > (dotimes (i 20000) > (setq body (cons '(message "more") body))) > `(progn ,@body))) > > (many-forms) > > (defun bar () > (if (eq 1 a) > (message "dude") > (message "else"))) > > ;; ---------------------------- > > Which does work. Wrapping in lambda also works. > > So now I'm thinking that as3_elispParser.el must have a single function > body whose bytecode overflows the 16bit limit. As removing chunks of code > from the top-level seems to alleviate the error (the lines with > (a3el-parser-bitset ..) for instance), I'm guessing that the problematic > byte_string corresponds to the top-level of the file. > > I may be able to wrap this top-level setup code into smaller functions.... > > > > > > > > > > > > > On Tue, Aug 25, 2009 at 7:50 AM, Aidan Kehoe <kehoea@parhasard.net> wrote: > >> >> Ar an cúigiú lá is fiche de mí Lúnasa, scríobh Miles Bader: >> >> > Aidan Kehoe <kehoea@parhasard.net> writes: >> > > GNU Emacs has (IIRC) 27-bit integers on 32-bit platforms >> > >> > ``The range of values for integers in Emacs Lisp is -268435456 to >> > 268435455 (29 bits; i.e., -2**28 to 2**28 - 1) on most machines.'' >> >> Thanks for the correction. Aemon’s problem remains, though, independent of >> that. Is there a good reason you don’t error on encountering integers the >> emacs binary can’t represent? Portable code needs to be prepared for that >> anyway. >> >> -- >> ¿Dónde estará ahora mi sobrino Yoghurtu Nghe, que tuvo que huir >> precipitadamente de la aldea por culpa de la escasez de rinocerontes? >> > > [-- Attachment #2: Type: text/html, Size: 4394 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Difficulties byte-compiling very large .el file 2009-08-25 10:16 ` Aidan Kehoe 2009-08-25 11:07 ` Miles Bader @ 2009-08-29 0:37 ` Stefan Monnier 1 sibling, 0 replies; 12+ messages in thread From: Stefan Monnier @ 2009-08-29 0:37 UTC (permalink / raw) To: Aidan Kehoe; +Cc: aemoncannon, emacs-devel > Tangentially to this, you should be aware that several of your integer > constants are silently overflowing in that file. With an XEmacs with 31-bit Good point. I've installed a few changes which should now handle this a bit better (partly by using floats to represent larger integers, as we already did in string-to-number, partly by detecting overflows). Stefan ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2009-09-01 22:41 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-08-19 21:49 Difficulties byte-compiling very large .el file Aemon Cannon 2009-08-20 8:27 ` Andreas Schwab 2009-08-20 13:07 ` Aemon Cannon 2009-08-20 13:23 ` Andreas Schwab 2009-08-20 14:29 ` Aemon Cannon 2009-08-20 18:30 ` Eli Zaretskii 2009-08-25 10:16 ` Aidan Kehoe 2009-08-25 11:07 ` Miles Bader 2009-08-25 11:50 ` Aidan Kehoe 2009-08-25 15:04 ` Aemon Cannon 2009-09-01 22:41 ` Aemon Cannon 2009-08-29 0:37 ` Stefan Monnier
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).