bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Jose A. Ortega Ruiz]

Unfortunately, i cannot reproduce the bug reliably, but these are the
symptoms:

  - sometimes, when composing a reply to an email in gnus (major mode is
    message-mode there, auto-fill-mode is enabled), when a line is
    auto-filled and broken in two, do-auto-fill incorrectly thinks that
    it is inside a comment and adds a `> ' prefix to it.

    For instance, i'm writing

        > This is a citation of the message i'm replying to

        This is my response and now i am approaching fill-colum so it

    which causes an auto-fill, which should be:

        > This is a citation of the message i'm replying to

        This is my response and now i am approaching fill-colum
        so it

    but instead i get:

        > This is a citation of the message i'm replying to

        This is my response and now i am approaching fill-colum
        > so it

   - i have no reliable way of reproducing the problem.  even when it's
     happening, if i kill and yank the buffer's content into a new
     buffer and put it in message-mode, the problem goes away.

   - stepping over do-auto-fill (called by message-do-auto-fill), i
     finally found out that the reason a > is being inserted is because
     `comment-beginning` is misbehaving, and thinks point is inside a
     comment.

   - and stepping into comment-beginning i've discovered that
     comment-beginning is wrong beacause someone sets the local variable
     `comment-end-skip` to a bogus value in the process of
     auto-filling.  that variable starts life as nil in message-mode,
     but, sometimes, after some calls to auto-fill it ends up having a
     bogus value.  that value is,  i think, set by
     newcomment.el's `comment-normalize-vars`, at the end of its body.

   - i don't know if setting the value is wrong in the context it's
     being set, but i've got the suspicion that the fact that that value
     is not returned to nil once do-auto-fill has ended its business
     might be causing havoc.  the concrete value is "[ 	]*\\(\\s>\\|\\n\\)"

   - at any rate, i know for sure that is that value of comment-end-skip
     the responsible for the bad behaviour of `comment-beginning` doing
     the wrong thing in some cases at least.

Cheers,
jao


In GNU Emacs 26.0.90 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw scroll bars)
 of 2017-12-18 built on imladris
Repository revision: c51e797bbace83181a3c778ba610560e236ee62b
Windowing system distributor 'The X.Org Foundation', version 11.0.11905000
System Description:	Debian GNU/Linux unstable (sid)

Recent messages:
Sending email 
Sending email done
Mark set
Wrote /home/jao/.emacs.d/gnus/Mail/bigml/48389
Sending...done
Deleting article /home/jao/.emacs.d/gnus/drafts/drafts/2 in drafts...
Mark set [4 times]
nil
Quit
Message modified; kill anyway? (y or n) y

Configured using:
 'configure --without-pop --with-x-toolkit=lucid'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
GNUTLS LIBXML2 FREETYPE LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11
LIBSYSTEMD LCMS2

Important settings:
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Summary

Minor modes in effect:
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  winner-mode: t
  savehist-mode: t
  recentf-mode: t
  save-place-mode: t
  network-watch-mode: t
  display-time-mode: t
  intero-global-mode: t
  global-edit-server-edit-mode: t
  flx-ido-mode: t
  ido-everywhere: t
  circe-lagmon-mode: t
  diff-auto-refine-mode: t
  tracking-mode: t
  persistent-scratch-autosave-mode: t
  show-paren-mode: t
  global-auto-revert-mode: t
  pdf-occur-global-minor-mode: t
  override-global-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t

Load-path shadows:
/home/jao/etc/emacs/site/custom/jao-themes hides /home/jao/lib/elisp/jao/themes/jao-themes
/home/jao/etc/emacs/site/custom/jao-fold hides /home/jao/etc/emacs/custom/jao-fold
/home/jao/etc/emacs/site/custom/jao-elisp hides /home/jao/etc/emacs/custom/jao-elisp
/home/jao/etc/emacs/site/custom/jao-muse hides /home/jao/etc/emacs/custom/jao-muse
/home/jao/etc/emacs/site/custom/jao-browse-url hides /home/jao/etc/emacs/custom/jao-browse-url
/home/jao/etc/emacs/site/custom/jao-vc hides /home/jao/etc/emacs/custom/jao-vc
/home/jao/etc/emacs/site/custom/jao-w3m hides /home/jao/etc/emacs/custom/jao-w3m
/home/jao/etc/emacs/site/custom/jao-gnus hides /home/jao/etc/emacs/custom/jao-gnus
/home/jao/etc/emacs/site/custom/jao-afio hides /home/jao/etc/emacs/custom/jao-afio
/home/jao/etc/emacs/site/custom/jao-clojure hides /home/jao/etc/emacs/custom/jao-clojure
/home/jao/etc/emacs/site/custom/jao-lisp hides /home/jao/etc/emacs/custom/jao-lisp
/home/jao/etc/emacs/site/custom/jao-skels hides /home/jao/etc/emacs/custom/jao-skels
/home/jao/etc/emacs/site/custom/jao-fci hides /home/jao/etc/emacs/custom/jao-fci
/home/jao/etc/emacs/site/custom/jao-gnuplot hides /home/jao/etc/emacs/custom/jao-gnuplot
/home/jao/etc/emacs/site/custom/jao-completion hides /home/jao/etc/emacs/custom/jao-completion
/home/jao/etc/emacs/site/custom/jao-python hides /home/jao/etc/emacs/custom/jao-python
/home/jao/etc/emacs/site/custom/jao-oz hides /home/jao/etc/emacs/custom/jao-oz
/home/jao/etc/emacs/site/custom/jao-conkeror hides /home/jao/etc/emacs/custom/jao-conkeror
/home/jao/etc/emacs/site/custom/jao-slime hides /home/jao/etc/emacs/custom/jao-slime
/home/jao/etc/emacs/site/custom/jao-markdown hides /home/jao/etc/emacs/custom/jao-markdown
/home/jao/etc/emacs/site/custom/jao-snippets hides /home/jao/etc/emacs/custom/jao-snippets
/home/jao/etc/emacs/site/custom/jao-edit hides /home/jao/etc/emacs/custom/jao-edit
/home/jao/etc/emacs/site/custom/jao-translate hides /home/jao/etc/emacs/custom/jao-translate
/home/jao/etc/emacs/site/custom/jao-sawfish hides /home/jao/etc/emacs/custom/jao-sawfish
/home/jao/etc/emacs/site/custom/jao-kbd hides /home/jao/etc/emacs/custom/jao-kbd
/home/jao/etc/emacs/site/custom/jao-spotify hides /home/jao/etc/emacs/custom/jao-spotify
/home/jao/etc/emacs/site/custom/jao-tex hides /home/jao/etc/emacs/custom/jao-tex
/home/jao/etc/emacs/site/custom/jao-compile hides /home/jao/etc/emacs/custom/jao-compile
/home/jao/etc/emacs/site/custom/jao-ruby hides /home/jao/etc/emacs/custom/jao-ruby
/home/jao/etc/emacs/site/custom/jao-network hides /home/jao/etc/emacs/custom/jao-network
/home/jao/etc/emacs/site/custom/jao-dict hides /home/jao/etc/emacs/custom/jao-dict
/home/jao/etc/emacs/site/custom/jao-custom hides /home/jao/etc/emacs/custom/jao-custom
/home/jao/etc/emacs/site/custom/jao-diminish hides /home/jao/etc/emacs/custom/jao-diminish
/home/jao/etc/emacs/site/custom/jao-goto-chg hides /home/jao/etc/emacs/custom/jao-goto-chg
/home/jao/etc/emacs/site/custom/jao-erlang hides /home/jao/etc/emacs/custom/jao-erlang
/home/jao/etc/emacs/site/custom/jao-prolog hides /home/jao/etc/emacs/custom/jao-prolog
/home/jao/etc/emacs/site/custom/jao-maxima hides /home/jao/etc/emacs/custom/jao-maxima
/home/jao/etc/emacs/site/custom/jao-deft hides /home/jao/etc/emacs/custom/jao-deft
/home/jao/etc/emacs/site/custom/jao-pdf hides /home/jao/etc/emacs/custom/jao-pdf
/home/jao/etc/emacs/site/custom/jao-time hides /home/jao/etc/emacs/custom/jao-time
/home/jao/etc/emacs/site/custom/jao-session hides /home/jao/etc/emacs/custom/jao-session
/home/jao/etc/emacs/site/custom/jao-c hides /home/jao/etc/emacs/custom/jao-c
/home/jao/etc/emacs/site/custom/jao-project-root hides /home/jao/etc/emacs/custom/jao-project-root
/home/jao/etc/emacs/site/custom/jao-circe hides /home/jao/etc/emacs/custom/jao-circe
/home/jao/etc/emacs/site/custom/jao-fonts hides /home/jao/etc/emacs/custom/jao-fonts
/home/jao/etc/emacs/site/custom/jao-eshell hides /home/jao/etc/emacs/custom/jao-eshell
/home/jao/etc/emacs/site/custom/jao-emms-config hides /home/jao/etc/emacs/custom/jao-emms-config
/home/jao/etc/emacs/site/custom/jao-frames hides /home/jao/etc/emacs/custom/jao-frames
/home/jao/etc/emacs/site/custom/jao-buffers hides /home/jao/etc/emacs/custom/jao-buffers
/home/jao/etc/emacs/site/custom/jao-dired hides /home/jao/etc/emacs/custom/jao-dired
/home/jao/etc/emacs/site/custom/jao-diary hides /home/jao/etc/emacs/custom/jao-diary
/home/jao/etc/emacs/site/custom/jao-colors hides /home/jao/etc/emacs/custom/jao-colors
/home/jao/etc/emacs/site/custom/jao-epg hides /home/jao/etc/emacs/custom/jao-epg
/home/jao/etc/emacs/site/custom/jao-mode-line hides /home/jao/etc/emacs/custom/jao-mode-line
/home/jao/etc/emacs/site/custom/jao-mail hides /home/jao/etc/emacs/custom/jao-mail
/home/jao/etc/emacs/site/custom/jao-utils hides /home/jao/etc/emacs/custom/jao-utils
/home/jao/etc/emacs/site/custom/jao-haskell hides /home/jao/etc/emacs/custom/jao-haskell
/home/jao/etc/emacs/site/custom/jao-auto hides /home/jao/etc/emacs/custom/jao-auto
/home/jao/etc/emacs/site/custom/jao-org hides /home/jao/etc/emacs/custom/jao-org
/home/jao/etc/emacs/site/custom/jao-undo-tree hides /home/jao/etc/emacs/custom/jao-undo-tree
/home/jao/etc/emacs/site/custom/jao-factor hides /home/jao/etc/emacs/custom/jao-factor

Features:
(shadow emacsbug edebug debug trace pulse eieio-opt speedbar sb-image
ezimage dframe cl-print help-fns radix-tree iso-transl gnus-fun gnus-dup
bbdb-message mailalias quail flow-fill counsel swiper w3m-cookie
w3m-filter w3m-bookmark w3m-tabmenu w3m-session url-http url-gw
url-cache cal-iso org-w3m org-info org-id org-gnus org-crypt org-bibtex
org-bbdb org-agenda misearch multi-isearch copyright magit-bookmark
magit-obsolete magit-blame magit-stash magit-bisect magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-branch
magit-collab ghub url-auth magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log magit-diff smerge-mode
magit-core magit-autorevert magit-process magit-margin magit-mode
git-commit magit-git magit-section magit-utils magit-popup log-edit
pcvs-util add-log with-editor async-bytecomp async colir ivy delsel
ivy-overlay ffap w3m-form w3m-symbol mm-archive gnus-cite qp gnus-async
gnus-bcklg gnus-ml gnus-topic mail-extr utf-7 nnml bbdb-gnus bbdb-mua
network-stream nsm starttls gnus-registry registry eieio-base nnir
gnus-delay gnus-draft gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-cache gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime
smime dig gnus-sum nndraft nnmh gnus-demon nntp gnus-group gnus-undo
gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo
gnus-spec gnus-int gnus-range gnus-win cursor-sensor windmove winner
ido-completing-read+ memoize minibuf-eldef server whizzml-skeletons
whizzml-mode bml-logs bml bml-misc bml-clojure bml-clj-tests bml-python
bml-skels bml-utils geiser jao-w3m w3m w3m-hist w3m-fb bookmark-w3m
w3m-ems w3m-ccl ccl w3m-favicon w3m-image w3m-proc w3m-util jao-vc
git-timemachine diff-hl vc-dir vc vc-dispatcher jao-utils battery
jao-undo-tree jao-translate google-translate google-translate-default-ui
google-translate-core-ui google-translate-core google-translate-tk
jao-tex biblio biblio-download biblio-dissemin biblio-hal biblio-dblp
biblio-crossref biblio-arxiv biblio-doi biblio-core hl-line ebib
ebib-reading-list ebib-notes ebib-filters ebib-keywords ebib-utils
ebib-db parsebib bibtex jao-spotify jao-snippets jao-slime slime-banner
slime-asdf grep slime-fancy slime-trace-dialog slime-fontifying-fu
slime-package-fu slime-references slime-compiler-notes-tree
slime-scratch slime-presentations bridge slime-macrostep macrostep
slime-mdot-fu slime-enclosing-context slime-fuzzy slime-fancy-trace
slime-fancy-inspector slime-c-p-c slime-editing-commands slime-autodoc
slime-repl slime-parse slime hyperspec jao-skels texinfo-skel lisp-skel
muse-skel haskell-skel jao-dominating-file scsh-skel init-skel
common-skel skeleton autoinsert jao-session savehist recentf tree-widget
saveplace jao-sawfish jao-ruby ruby-mode smie jao-python
virtualenvwrapper gud jao-prolog jao-project-root jao-pdf jao-doc-view
jao-oz jao-org jao-org-links jao-devon jao-applescript jao-org-utils
jao-network network-watch jao-muse muse-journal muse-book muse-latex
muse-wiki muse-colors muse-html muse-xml-common muse-publish
muse-project muse-mode muse-protocols muse-regexps muse muse-nested-tags
jao-mode-line spaceline-config spaceline-segments s spaceline powerline
powerline-separators powerline-themes jao-time time jao-maxima
jao-markdown json-mode json-reformat json-snatcher js sgml-mode cc-mode
cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs gh-md eww mm-url gnus nnheader url-queue shr svg dom
markdown-mode color jao-mail randomsig bbdb-anniv bbdb-com crm bbdb
bbdb-site timezone smtpmail sendmail message rmc puny rfc822 mml mml-sec
gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev mail-utils gmm-utils
mailheader jao-lisp info-look jao-kbd jao-haskell intero company pcase
warnings flycheck json map dash jao-goto-chg jao-gnus jao-gnuplot
jao-frames jao-fonts mm-util mail-prsvr jao-fold jao-factor jao-eshell
fish-completion em-cmpl esh-var esh-io esh-cmd esh-opt esh-ext esh-proc
esh-arg esh-groups eshell esh-module esh-util esh-mode bash-completion
eshell-up pcmpl-args pcmpl-gnu pcmpl-linux pcmpl-unix esh-toggle
git-ps1-mode jao-erlang erlang jao-emms-config jao-emms-info-track
jao-emms jao-osd jao-emms-lyrics jao-emms-random-album
emms-info-metaflac emms-librefm-stream xml emms-librefm-scrobbler
emms-playlist-limit emms-volume emms-volume-amixer emms-i18n
emms-history emms-score emms-stream-info emms-metaplaylist-mode
emms-bookmarks emms-cue emms-mode-line-icon emms-browser sort
emms-playlist-sort emms-last-played emms-player-xine emms-player-mpd
emms-playing-time emms-lyrics emms-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap emms-streams emms-show-all emms-tag-editor emms-mark
emms-mode-line emms-cache emms-info-ogginfo emms-info-mp3info emms-info
later-do emms-playlist-mode emms-player-vlc emms-player-mplayer
emms-player-simple emms-source-playlist emms-source-file locate
emms-setup emms emms-compat jao-elisp edit-list jao-edit edit-server
jao-dired dired+ image-dired image-file dired-x dired-aux jao-diminish
jao-dict jao-diary view cal-china lunar solar cal-dst cal-bahai
cal-islam cal-hebrew holidays hol-loaddefs vc-git appt diary-lib
diary-loaddefs jao-deft jao-custom jao-conkeror jao-completion
jao-recoll flx-ido flx ido jao-compile jao-colors jao-mono-dark-theme
jao-themes jao-clojure cider-macroexpansion cider-mode cider-interaction
etags xref project arc-mode archive-mode cider-repl cider-resolve
cider-eldoc cider-test cider-overlays cider-stacktrace cider-doc
org-table org-element avl-tree generator org org-macro org-footnote
org-pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp ob ob-tangle org-src ob-ref ob-lob ob-table
ob-keys ob-exp ob-comint ob-core ob-eval org-compat org-macs
org-loaddefs cal-menu calendar cal-loaddefs cider-grimoire cider-popup
cider-client cider-common cider-util nrepl-client queue nrepl-dict
cider-compat ewoc spinner clojure-mode subr-x align diminish paredit
jao-fci fill-column-indicator jao-circe circe-lagmon jao-epg epa-file
epa derived epg circe diff-mode lui-irc-colors irc make-tls-process tls
gnutls lcs lui-format lui tracking shorten thingatpt flyspell ispell
circe-compat jao-c jao-buffers elec-pair persistent-scratch time-date
paren autorevert filenotify jao-browse-url edmacro kmacro jao-docview
doc-view pdf-occur ibuf-ext ibuffer ibuffer-loaddefs tablist
tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw mode-local find-func cedet dired
dired-loaddefs pdf-isearch let-alist pdf-misc imenu pdf-tools compile
cus-edit cus-start cus-load wid-edit pdf-view bookmark pp jka-compr
pdf-cache pdf-info tq pdf-util image-mode cl-extra help-mode term
disp-table ehelp browse-url jao-auto jao-afio use-package
use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
tex-site rx cl slime-autoloads tramp tramp-compat tramp-loaddefs
trampver ucs-normalize shell pcomplete comint ansi-color ring parse-time
format-spec advice info jao-elpa package-x package easymenu epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 3257313 343804)
 (symbols 48 86674 1)
 (miscs 40 10668 5632)
 (strings 32 669521 101418)
 (string-bytes 1 24294796)
 (vectors 16 135917)
 (vector-slots 8 3387304 187382)
 (floats 8 2401 2048)
 (intervals 56 78147 19074)
 (buffers 992 318))

-- 
To be well informed, one must read quickly a great number of merely
instructive books. To be cultivated, one must read slowly and with a
lingering appreciation the comparatively few books that have been written
by men who lived, thought, and felt with style. -Aldous Huxley

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Michael Heerdegen]

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

>   - sometimes, when composing a reply to an email in gnus (major mode is
>     message-mode there, auto-fill-mode is enabled), when a line is
>     auto-filled and broken in two, do-auto-fill incorrectly thinks that
>     it is inside a comment and adds a `> ' prefix to it.

This has bugged me for a long time.  I also don't have a recipe, the
issue has a Heisenbug-like behavior.  But it happens quite often - it
>     even happens right now, I didn't comment this line.


Michael.

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Tue, 19 Dec 2017 12:52:30 +0100
> Cc: 29767@debbugs.gnu.org
> 
> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:
> 
> >   - sometimes, when composing a reply to an email in gnus (major mode is
> >     message-mode there, auto-fill-mode is enabled), when a line is
> >     auto-filled and broken in two, do-auto-fill incorrectly thinks that
> >     it is inside a comment and adds a `> ' prefix to it.
> 
> This has bugged me for a long time.  I also don't have a recipe, the
> issue has a Heisenbug-like behavior.  But it happens quite often - it
> >     even happens right now, I didn't comment this line.

What does "C-h l" show immediately after this happens?

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Katsumi Yamaoka]

On Tue, 19 Dec 2017 18:16:23 +0200, Eli Zaretskii wrote:
>> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:

>>>   - sometimes, when composing a reply to an email in gnus (major mode is
>>>     message-mode there, auto-fill-mode is enabled), when a line is
>>>     auto-filled and broken in two, do-auto-fill incorrectly thinks that
>>>     it is inside a comment and adds a `> ' prefix to it.

>> This has bugged me for a long time.  I also don't have a recipe, the
>> issue has a Heisenbug-like behavior.  But it happens quite often - it
>>>     even happens right now, I didn't comment this line.

> What does "C-h l" show immediately after this happens?

I found a way to reproduce this bug.  It requires there are two
or more quoted lines just above the unquoted line to be filled,
and the last quoted line has two or more >'s.  Interesting.

> The quick brown fox jumps over the lazy dog.
>> The quick brown fox jumps over the lazy dog.
The quick brown fox jumps over the lazy dog.  The quick brown fox jumps over the lazy dog.

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Jose Antonio Ortega Ruiz]

On Wed, Dec 20 2017, Katsumi Yamaoka wrote:

> On Tue, 19 Dec 2017 18:16:23 +0200, Eli Zaretskii wrote:
>>> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:
>
>>>>   - sometimes, when composing a reply to an email in gnus (major mode is
>>>>     message-mode there, auto-fill-mode is enabled), when a line is
>>>>     auto-filled and broken in two, do-auto-fill incorrectly thinks that
>>>>     it is inside a comment and adds a `> ' prefix to it.
>
>>> This has bugged me for a long time.  I also don't have a recipe, the
>>> issue has a Heisenbug-like behavior.  But it happens quite often - it
>>>>     even happens right now, I didn't comment this line.
>
>> What does "C-h l" show immediately after this happens?
>
> I found a way to reproduce this bug.  It requires there are two
> or more quoted lines just above the unquoted line to be filled,
> and the last quoted line has two or more >'s.  Interesting.
>
>
> The quick brown fox jumps over the lazy dog.
>> The quick brown fox jumps over the lazy dog.

not for me. i am answering here with a long line, and the two quoted
lines above don't trigger the bug on my side.  here you see the lines
that was auto-filled without any extra >.

i've also seen the bug triggered many times below cited lines with a
single >.


jao
-- 
Superficially there are right views and wrong views. When we look
deeply we find that all views are wrong.
  -Thich Nhat Hanh

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Katsumi Yamaoka]

On Wed, 20 Dec 2017 01:39:35 +0100, Jose Antonio Ortega Ruiz wrote:
> not for me. i am answering here with a long line, and the two quoted
> lines above don't trigger the bug on my side.  here you see the lines
> that was auto-filled without any extra >.

> i've also seen the bug triggered many times below cited lines with a
> single >.

Confirmed.  This causes the bug for me (note that the cite prefix
should be ">", not "> ").

>The quick brown fox jumps over the lazy dog.
>The quick brown fox jumps over the lazy dog.
A  l o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o n g  line.

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Jose A. Ortega Ruiz]

On Wed, Dec 20 2017, Katsumi Yamaoka wrote:

> On Wed, 20 Dec 2017 01:39:35 +0100, Jose Antonio Ortega Ruiz wrote:
>> not for me. i am answering here with a long line, and the two quoted
>> lines above don't trigger the bug on my side.  here you see the lines
>> that was auto-filled without any extra >.
>
>> i've also seen the bug triggered many times below cited lines with a
>> single >.
>
> Confirmed.  This causes the bug for me (note that the cite prefix
> should be ">", not "> ").
>
>>The quick brown fox jumps over the lazy dog.
>>The quick brown fox jumps over the lazy dog.
> A  l o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o n g  line.

Let me try: i'm copying below the two lines quoted above with a single >
and no spaces:

>The quick brown fox jumps over the lazy dog.
>The quick brown fox jumps over the lazy dog.

and now here's a l o o o o o o o o o o o o o o o o o o o o o o o o o o o
o o o o o ng line... which i am afraid got auto-filled without
trigerring the bug... so i guess we must be doing something different,
but i don't see what.

as i mentioned in my original report, this bug is annoying in that i can
paste the exact content of a buffer where it's happening into a second
one in message mode, and the bug won't be triggered in the second
buffer, so it seems to very sensitive to context.

anyway, Katsumi, since you have a couple of cases where it's happening
for you, perhaps you could answer Eli's question and tell him what's the
output of C-h l when you manage to reproduce it?

thanks!
jao
-- 
A lot of people have my books on their bookshelves.
That's the problem, they need to read them. -- Don Knuth

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

Hi Jose,

On Wed, 20 Dec 2017 02:20:41 +0100, Jose A. Ortega Ruiz wrote:
> so i guess we must be doing something different,

Well, does evaluating this snippet in the message-mode buffer
fix your problem?

(setq paragraph-separate (default-value 'paragraph-separate))
...[1]

As for me, I found the cause is that message-mode modifies its
value in a message-mode buffer locally so as to have the value
of `message-cite-prefix-regexp' (containing ">", etc.).  It makes
the paragraph commands, `M-{', `M-}', etc. work conveniently in
the message-mode.  However, it also seems to affect the auto-fill
command undesirably.  A patch is below.

But if [1] does not fix your problem, try evaluating the next
one before composing a message-mode buffer:

(fset 'message-setup-fill-variables 'ignore)
...[2]

If it does the trick, a cause should be something done in the
`message-setup-fill-variables' function.  It will probably be
easy to find the culprit by the brute force method (actually I
did it -- commenting forms one by one in the function).

[1] To restore its value, eval
(setq paragraph-separate paragraph-start)
in the message-mode buffer.

[2] To restore it, do: M-x load-library RET message RET


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 503 bytes --]

--- message.el~	2017-12-17 22:00:18.943355400 +0000
+++ message.el	2017-12-20 07:21:00.211760400 +0000
@@ -3441,7 +3441,8 @@
 (defun message-do-auto-fill ()
   "Like `do-auto-fill', but don't fill in message header."
   (unless (message-point-in-header-p)
-    (do-auto-fill)))
+    (let ((paragraph-separate (default-value 'paragraph-separate)))
+      (do-auto-fill))))
 
 (defun message-insert-signature (&optional force)
   "Insert a signature.  See documentation for variable `message-signature'."

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Jose A. Ortega Ruiz]

On Wed, Dec 20 2017, Katsumi Yamaoka wrote:

> Hi Jose,
>
> On Wed, 20 Dec 2017 02:20:41 +0100, Jose A. Ortega Ruiz wrote:
>> so i guess we must be doing something different,
>
> Well, does evaluating this snippet in the message-mode buffer
> fix your problem?
>
> (setq paragraph-separate (default-value 'paragraph-separate))
> ...[1]

okay, i will absolutely try the next time the bug is triggered by some
mail... so far, this one is working fine :)

> As for me, I found the cause is that message-mode modifies its
> value in a message-mode buffer locally so as to have the value
> of `message-cite-prefix-regexp' (containing ">", etc.).  It makes
> the paragraph commands, `M-{', `M-}', etc. work conveniently in
> the message-mode.  However, it also seems to affect the auto-fill
> command undesirably.  A patch is below.

in my case, the value of message-cite-prefix-regexp does not coincide
with that of paragraph-separate (the latter is much longer, and includes
also things like header and multipart separators) ... but then, i'm
checking these variables in *this* buffer, in which the problem is not
happening...).

> But if [1] does not fix your problem, try evaluating the next
> one before composing a message-mode buffer:
>
> (fset 'message-setup-fill-variables 'ignore)
> ...[2]
>
> If it does the trick, a cause should be something done in the
> `message-setup-fill-variables' function.

yes, that was also the function that i was suspecting, although i got
> the impression ...

ah! here we have the problem happening.  i've set paragraph-separate to
> the suggested value, but that isn't fixing the problem either, as you
> can see...

jao
-- 
If all else fails read the instructions. - Donald Knuth

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Jose A. Ortega Ruiz]

I think i've found a reliable way to reproduce the problem.

If i write my in-line response by opening a line in the middle of the
citacion text with a plain RET or C-o, things work as expected.

But, i i go to the middle of cited text and press M-j (i.e.,
indent-new-comment-line, which is an alias for comment-indent-new-line),
the bug is always triggered (or at least, it's been triggered for me all
the times i've tried).

Possibly telling: any line written above the point where i pressed M-j
is auto-filled correctly, and any line written below that point gets the
wrong behaviour.


Of course, the first thing comment-indent-new-line does is calling
comment-normalize-vars, so it seems Katsumi is on the right track.

BTW, after i trigger the bug in this way, setting paragraph-separate to
its previous value doesn't seem to have any effect: th problem's still
there.

Cheers,
jao
-- 
A man is like a fraction whose numerator is what he is and whose
denominator is what he thinks of himself. The larger the denominator,
the smaller the fraction. -Leo Tolstoy (1828-1910)

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Tomas Nordin]

Jose Antonio Ortega Ruiz <jao@gnu.org> writes:

> not for me. i am answering here with a long line, and the two quoted
> lines above don't trigger the bug on my side.  here you see the lines
> that was auto-filled without any extra >.

I have been experimenting with this, debugging with debugger. An early
function called is do-auto-fill. In the process at some point later, the
function comment-beginning is called to determine if we are in a comment
or not. If it returns nil, there will be no problems, when it returns a
value there will be problems. In that case point is also moved to the
place above the text where the comment syntax is picked (or prefix if
you want).

I have been in reply buffers where there is no problem and then in reply
buffers where there is problem. But also in a buffer where there was
this problem in the bottom of the buffer but not in the mid of the
buffer.

Anyhow, I claim that if you eval (comment-beginning) with point close to
the break point and it returns nil, there will be no problem. Else, a
value is returned and point will move back in the buffer and there will
be problems when filling.

No fix here but just adding in the hope it could be of any use.

Regards
--
Tomas

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Tomas Nordin]

Please ignore this my message (was ignored I think already). I feel a
bit dumb, because the same idea but with better detail was given in the
OP of this thread already.

Tomas Nordin <tomasn@posteo.net> writes:

> Jose Antonio Ortega Ruiz <jao@gnu.org> writes:
>
>> not for me. i am answering here with a long line, and the two quoted
>> lines above don't trigger the bug on my side.  here you see the lines
>> that was auto-filled without any extra >.
>
> I have been experimenting with this, debugging with debugger. An early
> function called is do-auto-fill. In the process at some point later, the
> function comment-beginning is called to determine if we are in a comment
> or not. If it returns nil, there will be no problems, when it returns a
> value there will be problems. In that case point is also moved to the
> place above the text where the comment syntax is picked (or prefix if
> you want).
>
> I have been in reply buffers where there is no problem and then in reply
> buffers where there is problem. But also in a buffer where there was
> this problem in the bottom of the buffer but not in the mid of the
> buffer.
>
> Anyhow, I claim that if you eval (comment-beginning) with point close to
> the break point and it returns nil, there will be no problem. Else, a
> value is returned and point will move back in the buffer and there will
> be problems when filling.
>
> No fix here but just adding in the hope it could be of any use.
>
> Regards
> --
> Tomas

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> What does "C-h l" show immediately after this happens?

Sorry about the late answer - it didn't happen for a while here.  But
> there's nothing unusual - oh it just happened again in this moment:

 B [self-insert-command]
 u u [self-insert-command]
 t t [self-insert-command]
 SPC SPC [self-insert-command]
 t [self-insert-command]
 h h [self-insert-command]
 e e [self-insert-command]
 r r [self-insert-command]
 e e [self-insert-command]
 ' ' [self-insert-command]
 s s [self-insert-command]
 SPC SPC [self-insert-command]
 n [self-insert-command]
 o o [self-insert-command]
 t t [self-insert-command]
 h h [self-insert-command]
 i i [self-insert-command]
 n n [self-insert-command]
 g g [self-insert-command]
 C-h C-h l [view-lossage]

it only shows self-insert-command.


Michael.

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Stefan Monnier]

> Unfortunately, i cannot reproduce the bug reliably, but these are the
> symptoms:

I think I found the culprit: syntax-propertize relies incorrectly on
syntax-ppss to add syntax-ppss-flush-cache to before-change-functions.
This works fine as long as syntax-ppss is called, but in message-mode
buffers, syntax-propertize is always called but not syntax-ppss (it may
get called depending on the operations you perform, but it's not called
initially, e.g. because font-lock is told not to use the syntax-table to
highlight strings and comments).

The patch below seems to fix it for me.  Is it OK to push it to emacs-26?


        Stefan


diff --git a/lisp/emacs-lisp/syntax.el b/lisp/emacs-lisp/syntax.el
index a1b70b1869..a0493cc1ba 100644
--- a/lisp/emacs-lisp/syntax.el
+++ b/lisp/emacs-lisp/syntax.el
@@ -291,6 +291,9 @@ syntax-propertize
       ;; (message "Needs to syntax-propertize from %s to %s"
       ;;          syntax-propertize--done pos)
       (set (make-local-variable 'parse-sexp-lookup-properties) t)
+      (when (< syntax-propertize--done (point-min))
+        (add-hook 'before-change-functions
+	          #'syntax-ppss-flush-cache t t))
       (save-excursion
         (with-silent-modifications
           (make-local-variable 'syntax-propertize--done) ;Just in case!

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Katsumi Yamaoka]

On Wed, 03 Jan 2018 23:59:39 -0500, Stefan Monnier wrote:
> The patch below seems to fix it for me.  Is it OK to push it to emacs-26?

Is it also OK to push my patch that I posted as
<b4m4lol274e.fsf@jpl.org>, i.e.,
<http://lists.gnu.org/archive/html/bug-gnu-emacs/2017-12/msg00714.html>
?
This prevents `message-auto-fill' from adding a useless citation in
the following case:

> foo
>> bar
A  l o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o n g  line.

(Try RET at the end of the long line in a messege-mode buffer.)

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Jose A. Ortega Ruiz]

On Wed, Jan 10 2018, Katsumi Yamaoka wrote:

> On Wed, 03 Jan 2018 23:59:39 -0500, Stefan Monnier wrote:
>> The patch below seems to fix it for me.  Is it OK to push it to emacs-26?

I've been using your patch since you sent it and it seems to be working
for me too, except for Katsumi's case below..

> Is it also OK to push my patch that I posted as
> <b4m4lol274e.fsf@jpl.org>, i.e.,
> <http://lists.gnu.org/archive/html/bug-gnu-emacs/2017-12/msg00714.html>
> ?
> This prevents `message-auto-fill' from adding a useless citation in
> the following case:
>
>> foo
>>> bar
> A  l o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o n g  line.
>
> (Try RET at the end of the long line in a messege-mode buffer.)

Yes, i can reproduce the problem too, even with Stefan's patch applied.

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Stefan Monnier]

>> The patch below seems to fix it for me.  Is it OK to push it to emacs-26?
>
> Is it also OK to push my patch that I posted as
> <b4m4lol274e.fsf@jpl.org>, i.e.,
> <http://lists.gnu.org/archive/html/bug-gnu-emacs/2017-12/msg00714.html>
> ?
> This prevents `message-auto-fill' from adding a useless citation in
> the following case:

I don't have an opinion on this, except that despite appearances this is
a completely different issue: bug#29767 is about a new bug where Emacs
"loses track" of the \n that closes a quoted line, whereas your problem
is about the definition of what is a paragraph and that problem is
not new, AFAIK.


        Stefan

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Katsumi Yamaoka]

On Thu, 11 Jan 2018 08:41:06 -0500, Stefan Monnier wrote:
>>> The patch below seems to fix it for me.  Is it OK to push it to emacs-26?

>> Is it also OK to push my patch that I posted as
>> <b4m4lol274e.fsf@jpl.org>, i.e.,
>> <http://lists.gnu.org/archive/html/bug-gnu-emacs/2017-12/msg00714.html>
>> ?
>> This prevents `message-auto-fill' from adding a useless citation in
>> the following case:

> I don't have an opinion on this, except that despite appearances this is
> a completely different issue: bug#29767 is about a new bug where Emacs
> "loses track" of the \n that closes a quoted line, whereas your problem
> is about the definition of what is a paragraph and that problem is
> not new, AFAIK.

Thanks.  I've installed my patch with a short comment.

bug#29767: 26.0.90; Failing auto-fill in message-mode because of bad state (comment-skip-end)

[Stefan Monnier]

> The patch below seems to fix it for me.  Is it OK to push it to emacs-26?

Installed,


        Stefan