bug#21840: 24.5; semantic analysis of python files is broken by strings that end in backslash

bug#21840: 24.5; semantic analysis of python files is broken by strings that end in backslash

[Glyph Lefkowitz]

Python string literals that end in a backslash cause Semantic's parser
to halt and not recognize anything further in the buffer.  I personally
ran across this because I frequently use helm-semantic-or-imenu, but can
be demonstrated equally well by semantic-complete-jump-local or anything
else that makes use of the buffer's symbol list.

The trivial way to reproduce this is to put the string literal "\\" at
the top of a Python buffer and then invoke semantic in one of the ways
just mentioned and notice that nothing is picked up.  You can move the
backslash literal down in the file and see every symbol up to the point
where it is placed.



In GNU Emacs 24.5.1 (x86_64-apple-darwin13.4.0, NS apple-appkit-1265.21)
of 2015-04-10 on builder10-9.porkrind.org
Windowing system distributor `Apple', version 10.3.1404
Configured using:
`configure --with-ns '--enable-locallisppath=/Library/Application
Support/Emacs/${version}/site-lisp:/Library/Application
Support/Emacs/site-lisp''

Important settings:
  locale-coding-system: utf-8-unix

Major mode: Python

Minor modes in effect:
  jedi-mode: t
  diff-auto-refine-mode: t
  global-git-commit-mode: t
  flymake-mode: t
  quick-hack-python-mode: t
  ecb-minor-mode: t
  python-docstring-mode: t
  server-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  rainbow-identifiers-mode: t
  rainbow-delimiters-mode: t
  global-auto-complete-mode: t
  auto-complete-mode: t
  global-quiet-mousewheel-mode: t
  quiet-mousewheel-mode: t
  obb-mode: t
  adaptive-wrap-prefix-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  global-semanticdb-minor-mode: t
  global-semantic-idle-scheduler-mode: t
  semantic-idle-scheduler-mode: t
  which-function-mode: t
  show-paren-mode: t
  semantic-mode: t
  icomplete-mode: t
  global-auto-revert-mode: t
  electric-pair-mode: t
  delete-selection-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Mark set [2 times]
Saving file /Users/glyph/Projects/Twisted/twisted/internet/test/test_endpoints.py...
Wrote /Users/glyph/Projects/Twisted/twisted/internet/test/test_endpoints.py
Mark set [3 times]
Saving file /Users/glyph/Projects/Twisted/twisted/internet/test/test_endpoints.py...
Wrote /Users/glyph/Projects/Twisted/twisted/internet/test/test_endpoints.py
Mark set
"Beep."
Quit
"Beep."
Quit

Load-path shadows:
/Users/glyph/.emacs.d/elpa/helm-20151028.327/helm-multi-match hides /Users/glyph/.emacs.d/elpa/helm-core-20151024.2233/helm-multi-match

Features:
(shadow sort mail-extr semantic/complete eieio-opt find-func emacsbug
sendmail ido noutline outline mm-archive url-http url-gw url-cache
url-auth url-handlers epg finder-inf inversion semantic/tag-write tabify
misearch multi-isearch semantic/edit network-stream starttls tls
semantic/tag-file helm-semantic semantic/imenu semantic/sb vc-git
semantic/db-file data-debug cedet-files semantic/wisent/python
semantic/decorate/include semantic/decorate/mode semantic/decorate pulse
semantic/dep semantic/wisent/python-wy semantic/wisent
semantic/wisent/wisent flyflakes jedi jedi-core python-environment epc
ctable concurrent deferred ropemacs pymacs column-marker magit-svn linum
magit-blame magit-stash magit-bisect magit-remote magit-commit
magit-sequence magit magit-apply magit-wip magit-log magit-diff
smerge-mode diff-mode magit-core magit-process magit-popup magit-mode
help-mode magit-git crm magit-section magit-utils git-commit log-edit
message rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev mail-utils gmm-utils
mailheader pcvs-util add-log with-editor tramp-sh dash winner mule-util
flymake python-patches python json quickhack ecb-layout-defs cus-edit
warnings ecb ecb-symboldef ecb-analyse ecb-compatibility
ecb-winman-support ecb-autogen autoload lisp-mnt ecb-tod ecb-cycle
ecb-eshell ecb-help ecb-jde ecb-method-browser hideshow
ecb-semantic-wrapper ecb-semantic ecb-file-browser ecb-speedbar
ecb-layout ecb-create-layout ecb-compilation ecb-common-browser speedbar
sb-image dframe ecb-navigate ecb-mode-line ecb-face tree-buffer
ecb-upgrade ecb-cedet-wrapper semantic/db-find semantic/db-ref
semantic/analyze semantic/sort semantic/scope semantic/analyze/fcn
wid-edit ecb-util python-docstring server undo-tree diff pelican-mode
rainbow-identifiers color rainbow-delimiters disp-table
auto-complete-config auto-complete popup quiet-mousewheel-mode
backandforth obb-mode combinator goto-definition adaptive-wrap
helm-C-x-b helm-imenu helm-command helm-elisp helm-eval edebug eldoc
helm-mode helm-cmd-t helm-files rx image-dired dired-x dired-aux ffap
thingatpt helm-buffers helm-elscreen helm-tags helm-bookmark
helm-adaptive helm-info bookmark pp helm-locate helm-grep helm-regexp
helm-plugin helm-external helm-net browse-url xml url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util url-parse url-vars mailcap helm-utils compile helm-help
helm-types helm easy-mmode helm-source helm-multi-match helm-lib dired
helm-config helm-easymenu edmacro kmacro async-bytecomp async
helm-aliases tramp tramp-compat auth-source gnus-util mm-util mail-prsvr
password-cache tramp-loaddefs trampver shell pcomplete comint ansi-color
ring format-spec semantic/db-mode semantic/db eieio-base semantic/idle
semantic/format ezimage semantic/tag-ls semantic/find semantic/ctxt
jka-compr vale-theme which-func imenu paren semantic/util-modes
semantic/util semantic semantic/tag semantic/lex semantic/fw eieio
byte-opt bytecomp byte-compile cl-extra cconv eieio-core mode-local
cedet icomplete autorevert filenotify elec-pair delsel cus-start
cus-load info easymenu package epg-config glyph-setup advice help-fns
cl-macs cl cl-loaddefs cl-lib gv time-date tooltip electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel ns-win tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode
register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
cocoa ns multi-tty emacs)

Memory information:
((conses 16 1026645 103760)
(symbols 48 50999 14)
(miscs 40 2212 1016)
(strings 32 133794 14262)
(string-bytes 1 4110532)
(vectors 16 54622)
(vector-slots 8 1559457 187147)
(floats 8 753 1789)
(intervals 56 121346 2669)
(buffers 960 34))

bug#21840: 24.5; semantic analysis of python files is broken by strings that end in backslash

[Lars Ingebrigtsen]

Glyph Lefkowitz <glyph@twistedmatrix.com> writes:

> Python string literals that end in a backslash cause Semantic's parser
> to halt and not recognize anything further in the buffer.  I personally
> ran across this because I frequently use helm-semantic-or-imenu, but can
> be demonstrated equally well by semantic-complete-jump-local or anything
> else that makes use of the buffer's symbol list.
>
> The trivial way to reproduce this is to put the string literal "\\" at
> the top of a Python buffer and then invoke semantic in one of the ways
> just mentioned and notice that nothing is picked up.  You can move the
> backslash literal down in the file and see every symbol up to the point
> where it is placed.

(This bug report unfortunately got no response at the time.)

Do you have a step-by-step recipe to reproduce this bug, starting from
"emacs -Q"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21840: 24.5; semantic analysis of python files is broken by strings that end in backslash

[Glyph]

I don't know how to force semantic to fully reparse a file, I just observe its behavior in response to idle times.

Here's a test file you can use though, if you know how to do that:


def a(v):
    "test"

def x():
    "test"
    a(b"\\")

def y():
    "test"

def z():
    "test"


Comment and uncomment the 'a(b"\\")' line and let semantic get around to reparsing the buffer, however it decides to do that, and (mapcar 'car (semantic-fetch-tags)) will  return ("a" "x") rather than the expected ("a" "x" "y" "z").  However, it seems like there's tons of cache state somewhere that I can't easily clear out because it randomly seems to fluctuate between doing what I expect, returning "nil", and returning a stale copy of the symbols (including stuff added and then removed entirely from the bottom of the buffer).

Sadly, given that I don't use semantic any more (specifically because of this type of inscrutable unreliability) I don't have any more time for debugging this.  Thanks for following up and good luck!

-g

> On Dec 3, 2020, at 2:46 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> Glyph Lefkowitz <glyph@twistedmatrix.com> writes:
> 
>> Python string literals that end in a backslash cause Semantic's parser
>> to halt and not recognize anything further in the buffer.  I personally
>> ran across this because I frequently use helm-semantic-or-imenu, but can
>> be demonstrated equally well by semantic-complete-jump-local or anything
>> else that makes use of the buffer's symbol list.
>> 
>> The trivial way to reproduce this is to put the string literal "\\" at
>> the top of a Python buffer and then invoke semantic in one of the ways
>> just mentioned and notice that nothing is picked up.  You can move the
>> backslash literal down in the file and see every symbol up to the point
>> where it is placed.
> 
> (This bug report unfortunately got no response at the time.)
> 
> Do you have a step-by-step recipe to reproduce this bug, starting from
> "emacs -Q"?
> 
> -- 
> (domestic pets only, the antidote for overdose, milk.)
>   bloggy blog: http://lars.ingebrigtsen.no

bug#21840: 24.5; semantic analysis of python files is broken by strings that end in backslash

[Lars Ingebrigtsen]

Glyph <glyph@twistedmatrix.com> writes:

> I don't know how to force semantic to fully reparse a file, I just
> observe its behavior in response to idle times.

A complete recipe, starting from "emacs -Q", would be helpful.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no