* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly @ 2014-03-01 13:31 Bozhidar Batsov 2014-03-01 22:19 ` Dmitry Gutov 0 siblings, 1 reply; 10+ messages in thread From: Bozhidar Batsov @ 2014-03-01 13:31 UTC (permalink / raw) To: 16915 In most editors/IDEs code like this regexp = / start # some text \s # white space char (group) # first group (?:alt1|alt2) # some alternation end /x will have the comments font-locked as comments, because comments are allowed in the extended regexp literal syntax (/x). It would be nice if this was taken into account in ruby-mode as well. In GNU Emacs 24.3.50.1 (x86_64-apple-darwin13.0.0, NS apple-appkit-1265.00) of 2014-01-27 on Bozhidars-MacBook-Pro.local Windowing system distributor `Apple', version 10.3.1265 Configured using: `configure --prefix=/usr/local/Cellar/emacs/HEAD --without-dbus --enable-locallisppath=/usr/local/share/emacs/site-lisp --infodir=/usr/local/Cellar/emacs/HEAD/share/info/emacs --without-gnutls --with-ns --disable-ns-self-contained' Important settings: locale-coding-system: utf-8-unix Major mode: Ruby Minor modes in effect: ruby-tools-mode: t inf-ruby-minor-mode: t diff-auto-refine-mode: t subword-mode: t guru-mode: t erc-truncate-mode: t erc-spelling-mode: t erc-list-mode: t erc-menu-mode: t erc-autojoin-mode: t erc-ring-mode: t erc-networks-mode: t erc-pcomplete-mode: t erc-track-mode: t erc-match-mode: t erc-button-mode: t erc-fill-mode: t erc-stamp-mode: t erc-netsplit-mode: t erc-irccontrols-mode: t erc-noncommands-mode: t erc-move-to-prompt-mode: t erc-readonly-mode: t global-flycheck-mode: t flycheck-mode: t which-function-mode: t flx-ido-mode: t ido-ubiquitous-mode: t winner-mode: t global-undo-tree-mode: t undo-tree-mode: t whitespace-mode: t global-anzu-mode: t anzu-mode: t projectile-global-mode: t projectile-mode: t flyspell-mode: t volatile-highlights-mode: t global-hl-line-mode: t shell-dirtrack-mode: t recentf-mode: t savehist-mode: t show-smartparens-global-mode: t show-smartparens-mode: t smartparens-mode: t global-auto-revert-mode: t delete-selection-mode: t prelude-global-mode: t prelude-mode: t tooltip-mode: t electric-indent-mode: t mouse-wheel-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t size-indication-mode: t column-number-mode: t line-number-mode: t transient-mark-mode: t Recent input: ' d SPC s u g g e s t SPC t h e SPC u s e SPC o f SPC t h e SPC ` <backspace> <backspace> <backspace> <backspace> <backspace> <backspace> <backspace> f SPC a SPC d i f f e r e n t SPC f a c e SPC t h a n SPC t h e SPC s t r i n g SPC f a c e SPC u s e d SPC o f <backspace> <backspace> f o r SPC t h e SPC l i t e r a i l <backspace> <backspace> l SPC i t s e l f , SPC b u t SPC e v e n SPC i t SPC w o u l d SPC b e SPC a n SPC i m p r o v e m e n t . SPC I SPC a s o <backspace> <backspace> l s o SPC t h i n k <return> t h a t SPC o n l y SPC v a l i d SPC m o d i f i e r s SPC s h o u l d SPC b e SPC c <backspace> f o n t SPC l o c k e d . SPC M-b M-b M-f C-d - C-x b <return> <return> % r / t e s t / i <help-echo> C-x b <return> C-p C-p C-p C-p C-p C-p C-p C-n C-p C-p C-a C-f C-d n C-n C-n C-e <return> $ <backspace> % r { t e s t } i C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-p C-p C-p C-p C-p C-p C-p C-n C-n C-n C-n C-e C-c C-c y M-x r e p o r t <return> Recent messages: Wrote /Users/bozhidar/*message*-20140301-152412 Saving file /Users/bozhidar/projects/test.rb... Wrote /Users/bozhidar/projects/test.rb Auto-saving...done Mark set Send this bug report to the Emacs maintainers? (y or n) y Sending... Mark set [2 times] Sending via mail... Sending...done Load-path shadows: /Users/bozhidar/.emacs.d/elpa/tabulated-list-20120406.2251/tabulated-list hides /usr/local/Cellar/emacs/HEAD/share/emacs/24.3.50/lisp/emacs-lisp/tabulated-list Features: (shadow sort mail-extr emacsbug vc-annotate vc vc-dispatcher nxml-uchnm rng-xsd xsd-regexp rng-cmpct edebug crm ielm hippie-exp markdown-mode noutline outline gitignore-mode conf-mode yaml-mode ace-jump-mode ffap url-parse url-vars ruby-tools inf-ruby ruby-mode-expansions smartparens-ruby ruby-mode smie misearch multi-isearch jka-compr eieio-opt speedbar sb-image ezimage dframe mule-util find-dired magit-key-mode magit view epa derived epg epg-config diff-mode git-rebase-mode git-commit-mode server log-edit message sendmail rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev mail-utils gmm-utils mailheader pcvs-util add-log executable vc-git lisp-mnt network-stream starttls tls cider cider-mode cider-repl cider-eldoc clojure-test-mode cider-interaction arc-mode archive-mode cider-client nrepl-client cider-util ewoc superword subword clojure-mode-expansions clojure-mode inf-lisp rainbow-mode color rainbow-delimiters elisp-slime-nav guru-mode prelude-key-chord key-chord prelude-xml nxml-mode-expansions html-mode-expansions sgml-mode smartparens-html rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok prelude-web prelude-scss prelude-scheme prelude-ruby prelude-perl prelude-org prelude-js prelude-erc erc-truncate erc-autoaway erc-spelling erc-notify erc-log erc-list erc-menu erc-join erc-ring erc-networks erc-pcomplete erc-track erc-match erc-button erc-fill erc-stamp erc-netsplit erc-goodies erc erc-backend erc-compat prelude-emacs-lisp prelude-css prelude-common-lisp slime-autoloads prelude-clojure prelude-lisp prelude-c prelude-programming flycheck help-mode rx f which-func imenu prelude-ido smex flx-ido flx ido-ubiquitous warnings ido prelude-osx exec-path-from-shell prelude-global-keybindings prelude-editor winner undo-tree diff esh-var esh-io esh-cmd esh-opt esh-ext esh-proc esh-arg eldoc esh-groups eshell esh-module esh-mode esh-util re-builder whitespace browse-kill-ring midnight ediff-merg ediff-wind ediff-diff ediff-mult ediff-help ediff-init ediff-util ediff dired-x dired anzu projectile pkg-info find-func grep compile s bookmark pp expand-region text-mode-expansions er-basic-expansions expand-region-core expand-region-custom flyspell ispell etags volatile-highlights hl-line windmove tramp-cache tramp-sh tramp tramp-compat auth-source gnus-util mm-util mail-prsvr password-cache tramp-loaddefs trampver shell pcomplete comint ansi-color ring format-spec recentf tree-widget wid-edit savehist saveplace diminish smartparens-config smartparens autorevert filenotify delsel prelude-mode easy-mmode edmacro kmacro prelude-core epl advice help-fns dash thingatpt prelude-ui zenburn-theme prelude-packages finder-inf ack-and-a-half-autoloads diminish-autoloads erlang-autoloads flx-ido-autoloads eieio byte-opt bytecomp byte-compile cconv eieio-core grizzl-autoloads key-chord-autoloads logito-autoloads info easymenu cl-macs gv move-text-autoloads pkg-info-autoloads puppet-mode-autoloads rainbow-delimiters-autoloads request-autoloads rubocop-autoloads volatile-highlights-autoloads yaml-mode-autoloads package cl cl-loaddefs cl-lib time-date tooltip electric uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode register page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote make-network-process cocoa ns multi-tty emacs) ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-01 13:31 bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly Bozhidar Batsov @ 2014-03-01 22:19 ` Dmitry Gutov 2014-03-02 11:03 ` Bozhidar Batsov 2014-03-07 21:04 ` Stefan Monnier 0 siblings, 2 replies; 10+ messages in thread From: Dmitry Gutov @ 2014-03-01 22:19 UTC (permalink / raw) To: Bozhidar Batsov; +Cc: 16915 Bozhidar Batsov <bozhidar@batsov.com> writes: > In most editors/IDEs code like this > > regexp = / > start # some text > \s # white space char > (group) # first group > (?:alt1|alt2) # some alternation > end > /x > > will have the comments font-locked as comments, because comments are > allowed in the extended regexp literal syntax (/x). It would be nice > if this was taken into account in ruby-mode as well. Not sure how to implement it best. Ideally, we'd have a new kind of syntax instead of strings (native regexp support?), which would make font-lock fontify comments inside. Or maybe a modifier on the string syntax? Of course, we could just scan the contents of every regexp, look for any hash character that doesn't look like it starts interpolation, and forcibly fontify the text from it till the end of line. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-01 22:19 ` Dmitry Gutov @ 2014-03-02 11:03 ` Bozhidar Batsov 2014-03-02 15:03 ` Dmitry Gutov 2014-03-07 21:04 ` Stefan Monnier 1 sibling, 1 reply; 10+ messages in thread From: Bozhidar Batsov @ 2014-03-02 11:03 UTC (permalink / raw) To: Dmitry Gutov; +Cc: 16915 [-- Attachment #1: Type: text/plain, Size: 1308 bytes --] On Sunday, March 2, 2014 at 12:19 AM, Dmitry Gutov wrote: > Bozhidar Batsov <bozhidar@batsov.com (mailto:bozhidar@batsov.com)> writes: > > > In most editors/IDEs code like this > > > > regexp = / > > start # some text > > \s # white space char > > (group) # first group > > (?:alt1|alt2) # some alternation > > end > > /x > > > > will have the comments font-locked as comments, because comments are > > allowed in the extended regexp literal syntax (/x). It would be nice > > if this was taken into account in ruby-mode as well. > > > > > Not sure how to implement it best. > > Ideally, we'd have a new kind of syntax instead of strings (native > regexp support?), which would make font-lock fontify comments inside. > > Or maybe a modifier on the string syntax? Native regexp support is preferable IMO. After all - regexps are not strings. If they were treated differently we’d also be able to have extra highlighting for things like named groups, quantifiers, regexp classes, etc. I guess, however, that this would require a lot of work. > > Of course, we could just scan the contents of every regexp, look for any > hash character that doesn't look like it starts interpolation, and > forcibly fontify the text from it till the end of line. > > [-- Attachment #2: Type: text/html, Size: 2196 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-02 11:03 ` Bozhidar Batsov @ 2014-03-02 15:03 ` Dmitry Gutov 2014-03-04 10:02 ` Bozhidar Batsov 0 siblings, 1 reply; 10+ messages in thread From: Dmitry Gutov @ 2014-03-02 15:03 UTC (permalink / raw) To: Bozhidar Batsov; +Cc: 16915 On 02.03.2014 13:03, Bozhidar Batsov wrote: > Native regexp support is preferable IMO. After all - regexps are not > strings. If they were treated differently we’d also be able to have > extra highlighting for things like named groups, quantifiers, regexp > classes, etc. I guess, however, that this would require a lot of work. Since the regexp syntax can be different between languages, we probably won't get all of that automatically. The direct benefits from the native support I can see is highlighting comments (for regexps with appropriate modifiers, in languages that support that), new faces for regexp itself, and maybe for groups, quantifiers, etc. The highlighting of the elements inside regexp would probably have to be done the old-fashioned way, in font-lock-keywords (although that code could be shared between many languages). We could implement something like it right now, the main difference is just the lack of standard faces. ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-02 15:03 ` Dmitry Gutov @ 2014-03-04 10:02 ` Bozhidar Batsov 0 siblings, 0 replies; 10+ messages in thread From: Bozhidar Batsov @ 2014-03-04 10:02 UTC (permalink / raw) To: Dmitry Gutov; +Cc: 16915 [-- Attachment #1: Type: text/plain, Size: 1484 bytes --] On Sunday, March 2, 2014 at 5:03 PM, Dmitry Gutov wrote: > On 02.03.2014 13:03, Bozhidar Batsov wrote: > > Native regexp support is preferable IMO. After all - regexps are not > > strings. If they were treated differently we’d also be able to have > > extra highlighting for things like named groups, quantifiers, regexp > > classes, etc. I guess, however, that this would require a lot of work. > > > > > Since the regexp syntax can be different between languages, we probably > won't get all of that automatically. The direct benefits from the native > support I can see is highlighting comments (for regexps with appropriate > modifiers, in languages that support that), new faces for regexp itself, > and maybe for groups, quantifiers, etc. > > That would be totally sufficient, but I have no idea how much work it would require. > > The highlighting of the elements inside regexp would probably have to be > done the old-fashioned way, in font-lock-keywords (although that code > could be shared between many languages). We could implement something > like it right now, the main difference is just the lack of standard faces. > > Many modes introduce "non-standard” faces, so the lack of non-standard faces is not a significant problem I think (though I’d love to see faces for regexps, command execution (%x, `` as in sh-mode) and symbols/keywords (as they are present in many languages outside Ruby). [-- Attachment #2: Type: text/html, Size: 2202 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-01 22:19 ` Dmitry Gutov 2014-03-02 11:03 ` Bozhidar Batsov @ 2014-03-07 21:04 ` Stefan Monnier 2014-03-10 7:21 ` Dmitry Gutov 1 sibling, 1 reply; 10+ messages in thread From: Stefan Monnier @ 2014-03-07 21:04 UTC (permalink / raw) To: Dmitry Gutov; +Cc: 16915, Bozhidar Batsov > Ideally, we'd have a new kind of syntax instead of strings (native > regexp support?), which would make font-lock fontify comments inside. This is not completely new. We have a similar problem for example with TeX's math $...$ and with PostScript's strings (...(...)..). > Or maybe a modifier on the string syntax? My preference would be to think about it as a "multi-mode" case, and hence make it possible to specify a different syntax-table to use within the regexp. I think of it along the lines of a new syntax-class, applied to the "/" char, which would change the syntax-table for the subsequent text. Stefan ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-07 21:04 ` Stefan Monnier @ 2014-03-10 7:21 ` Dmitry Gutov 2014-03-10 14:40 ` Stefan Monnier 0 siblings, 1 reply; 10+ messages in thread From: Dmitry Gutov @ 2014-03-10 7:21 UTC (permalink / raw) To: Stefan Monnier; +Cc: 16915, Bozhidar Batsov On 07.03.2014 23:04, Stefan Monnier wrote: > My preference would be to think about it as a "multi-mode" case, and > hence make it possible to specify a different syntax-table to use within > the regexp. I remember this idea, but have a hard time viewing it in the context of our latest discussion on the subject of multi-modes. First, why only syntax-table? For this specific case, a syntax table change is not required, we only need to be able to view the text between /'s as a separate context (but - and this is a change from certain other multi-mode uses - still fontify uncommented text inside them with the regexp face). But in the general case, we would at least want to be able to change font-lock-keywords, too. > I think of it along the lines of a new syntax-class, applied to the "/" > char, which would change the syntax-table for the subsequent text. How would this interact with a new hook that would `syntax-ppss' would run on the cached entries? Would its default value look for the chars bearing the new syntax class? ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-10 7:21 ` Dmitry Gutov @ 2014-03-10 14:40 ` Stefan Monnier 2014-03-12 7:48 ` Dmitry Gutov 0 siblings, 1 reply; 10+ messages in thread From: Stefan Monnier @ 2014-03-10 14:40 UTC (permalink / raw) To: Dmitry Gutov; +Cc: 16915, Bozhidar Batsov >> My preference would be to think about it as a "multi-mode" case, and >> hence make it possible to specify a different syntax-table to use within >> the regexp. > I remember this idea, but have a hard time viewing it in the context of our > latest discussion on the subject of multi-modes. > First, why only syntax-table? It was not meant as excluding other data. > For this specific case, a syntax table change is not required, we only > need to be able to view the text between /'s as a separate context Not necessarily required in all cases, but in order to handle the various existing cases in various languages, we need some way to indicate, for instance: - Do double quotes introduce strings when used within this new context? - Do comment chars introduce comments when used within this new context? - Does \ still escape chars in this new context? We usually use syntax-tables for that. They do provide more flexibility than needed (so far), such as making it possible to allow different commenting conventions within the new context. But it seems like a "simple" way to handle the problem, so the extra generality is a bonus. > (but - and this is a change from certain other multi-mode uses - still > fontify uncommented text inside them with the regexp face). But in the > general case, we would at least want to be able to change > font-lock-keywords, too. `font-lock-keywords' can use `syntax-ppss' to decide which rules to use (since syntax-ppss would have to include the "context state" somewhere in its output). It's not terribly convenient to do currently, so we may want to provide a new replacement for font-lock-keywords which makes it easier (and avoids relying on "eval" while we're at it). >> I think of it along the lines of a new syntax-class, applied to the "/" >> char, which would change the syntax-table for the subsequent text. > How would this interact with a new hook that would `syntax-ppss' would run > on the cached entries? The way I imagine it, it would work at the parse-partial-sexp level (extending the returned PPSS with a new field indicating the current syntax-table or something similar). So syntax.el would stay basically unchanged. How that would interact with a new hook? Haven't thought about it yet. > Would its default value look for the chars bearing the new syntax class? I don't really understand the question. But I'd guess that I don't know the answer. Stefan ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-10 14:40 ` Stefan Monnier @ 2014-03-12 7:48 ` Dmitry Gutov 2014-03-12 14:25 ` Stefan Monnier 0 siblings, 1 reply; 10+ messages in thread From: Dmitry Gutov @ 2014-03-12 7:48 UTC (permalink / raw) To: Stefan Monnier; +Cc: 16915, Bozhidar Batsov On 10.03.2014 16:40, Stefan Monnier wrote: > We usually use syntax-tables for that. They do provide more flexibility > than needed (so far), such as making it possible to allow different > commenting conventions within the new context. But it seems like > a "simple" way to handle the problem, so the extra generality is a bonus. To handle this part of the problem (literals within literals), yes, certainly. >> (but - and this is a change from certain other multi-mode uses - still >> fontify uncommented text inside them with the regexp face). But in the >> general case, we would at least want to be able to change >> font-lock-keywords, too. > > `font-lock-keywords' can use `syntax-ppss' to decide which rules to use > (since syntax-ppss would have to include the "context state" somewhere > in its output). It's not terribly convenient to do currently, so we may > want to provide a new replacement for font-lock-keywords which makes it > easier (and avoids relying on "eval" while we're at it). If we're speaking of multiple modes in general, syntax-ppss might not be enough to determine which keywords to highlight. Even if it includes information about new syntax classes. Suppose we have an HTML file with <script> and <style> tags inside. Would points inside of these tags have different states? What might be the difference between them? Would some font-lock code compare syntax table reference attached to a syntax class, with a syntax table it wants? ^ permalink raw reply [flat|nested] 10+ messages in thread
* bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly 2014-03-12 7:48 ` Dmitry Gutov @ 2014-03-12 14:25 ` Stefan Monnier 0 siblings, 0 replies; 10+ messages in thread From: Stefan Monnier @ 2014-03-12 14:25 UTC (permalink / raw) To: Dmitry Gutov; +Cc: 16915, Bozhidar Batsov > If we're speaking of multiple modes in general, syntax-ppss might not be > enough to determine which keywords to highlight. Even if it includes > information about new syntax classes. It wouldn't really include info *about* new syntax classes, but the new syntax class I suggest (which specifies a syntax-table to use) would require changing the "parse-partial-sexp state" by keeping track of the "current syntax-table". "Syntax classes" are "events" which cause changes in the parse-partial-sexp state machine, and syntax-ppss is a way to get the state of that machine at a given position. > Suppose we have an HTML file with <script> and <style> tags inside. Would > points inside of these tags have different states? Yes. E.g. (nth 10 (syntax-ppss)) would return a new "context" info (which could just be "the syntax table to use in that area"). > Would some font-lock code compare syntax table reference attached > to a syntax class, with a syntax table it wants? Yes, quite possibly. Stefan ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2014-03-12 14:25 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-03-01 13:31 bug#16915: 24.3.50; [ruby-mode] Comments in regexps using the extended syntax are not font-locked properly Bozhidar Batsov 2014-03-01 22:19 ` Dmitry Gutov 2014-03-02 11:03 ` Bozhidar Batsov 2014-03-02 15:03 ` Dmitry Gutov 2014-03-04 10:02 ` Bozhidar Batsov 2014-03-07 21:04 ` Stefan Monnier 2014-03-10 7:21 ` Dmitry Gutov 2014-03-10 14:40 ` Stefan Monnier 2014-03-12 7:48 ` Dmitry Gutov 2014-03-12 14:25 ` Stefan Monnier
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/emacs.git https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.