bug#21742: 25.0.50; Document that ert-deftest expands all macros

bug#21742: 25.0.50; Document that ert-deftest expands all macros

[immerrr again]

I wrote a library ample-regexps [1] that does some macro magic to extend
rx and ran into an issue with ert when tests started failing without any
explanation.  I started digging and here's what I found.

The library provides a macro called `define-arx` which roughly expands
to ([2])

   (eval-and-compile
     (defvar SOME-VAR ...)
     (defun SOME-FUN (...) ...)
     (defmacro SOME-MACRO (...)
       (SOME-FUN ...))
     ...)

The whole expansion is wrapped into eval-and-compile so that it can be
used in a library during byte-compilation.

I wrote a macro to use in tests that expands to roughly

  (progn
    (define-arx ...)
    (unwind-protect
        ,@body
      (fmakunbound SOME-MACRO)
      (fmakunbound SOME-FUN)
      (makunbound SOME-VAR)))

The macro worked when executed manually, but not during tests.

I started grepping and xref-ing through the code till I found that
ert-deftest includes cl-macrolet which in turn runs macroexpand-all on
the test definition.  Quite expectedly, it expands the define-arx macro
that results in eval-and-compile which is also expanded and all that's
left in the resulting test definition is a symbol.  More importantly,
all side effects, i.e. function/macro definitions happen during test
definition while they should really happen during test execution.

I ended up using (eval (quote (define-arx ...))) to prevent early
expansion, but it would really be nice if this was documented (would
have saved me an hour-something of what-is-going-on and couple hours of
how-do-i-stop-that).  Or if that expansion didn't happen in the first
place, but I guess that's not easy since it does that to expand all the
(should ...) forms and test helpers to improve test run time.

1. https://github.com/immerrr/ample-regexps.el
2. https://github.com/immerrr/ample-regexps.el/blob/8368cc16e724cea1f60755fe40e25fe374b24418/ample-regexps.el#L339-L371



In GNU Emacs 25.0.50.10 (x86_64-unknown-linux-gnu, GTK+ Version 3.4.2)
 of 2015-10-12
Repository revision: 149066dbfc7f11c40f6dbed70a3279153bf3ab7a
Windowing system distributor 'The X.Org Foundation', version 11.0.11103000
System Description:    Ubuntu 12.04.5 LTS

Configured using:
 'configure --prefix=/home/immerrr/.local'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: el

Minor modes in effect:
  global-git-commit-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  diff-auto-refine-mode: t
  nameless-mode: t
  whitespace-mode: t
  flycheck-mode: t
  guide-key-mode: t
  global-company-mode: t
  company-mode: t
  pyvenv-mode: t
  paredit-mode: t
  auto-compile-on-save-mode: t
  auto-compile-mode: t
  recentf-mode: t
  shell-dirtrack-mode: t
  async-bytecomp-package-mode: t
  projectile-global-mode: t
  projectile-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  save-place-mode: t
  show-paren-mode: t
  override-global-mode: t
  global-linum-mode: t
  linum-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Making completion list... [2 times]

mouse-2, RET: find function's definition
uncompressing bytecomp.el.gz...done
Mark set
Mark saved where search started

Mark set

Making completion list...

Load-path shadows:
/home/immerrr/.emacs.d/elpa/helm-20150921.1209/helm-multi-match hides
/home/immerrr/.emacs.d/elpa/helm-core-20150921.1209/helm-multi-match

Features:
(shadow sort mail-extr emacsbug sendmail yaml-mode gravatar url-cache
git-rebase log-view vc-annotate macrostep ielm goto-chg novice
network-stream nsm starttls warnings company-jedi jedi-core epc ctable
concurrent python bs ag vc-svn find-dired mc-mark-more
multiple-cursors-core rect make-mode magit-blame magit-stash
magit-bisect magit-remote magit-commit magit-sequence magit magit-apply
magit-wip magit-log magit-diff smerge-mode magit-core magit-process
magit-popup magit-mode magit-git crm magit-section magit-utils
git-commit log-edit message rfc822 mml mml-sec mailabbrev gmm-utils
mailheader pcvs-util add-log with-editor server ert ewoc debug align
tabify two-column iso-transl face-remap cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs ido seq
eieio-opt speedbar sb-image ezimage dframe trace undo-tree diff misearch
multi-isearch pulse jka-compr executable edebug test-helper
ample-regexps vc vc-dispatcher vc-git diff-mode company-elisp nameless
disp-table whitespace flycheck find-func subr-x guide-key s
ucs-normalize popwin image-file winner wgrep traad request-deferred
request url-http tls url-auth url-gw mail-utils popup python-environment
deferred company-files company-oddmuse company-keywords company-etags
etags xref project company-gtags company-dabbrev-code company-dabbrev
company-capf company-cmake company-xcode company-clang company-semantic
company-eclim company-template company-css company-nxml company-bbdb
company pyvenv paredit auto-compile packed tramp-cache tramp-sh recentf
tree-widget wid-edit helm-projectile helm-files rx image-dired tramp
tramp-compat tramp-loaddefs trampver shell pcomplete format-spec dired-x
dired-aux ffap helm-tags helm-bookmark helm-adaptive helm-info bookmark
pp helm-external helm-net browse-url xml url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse auth-source gnus-util password-cache url-vars mailcap
helm-buffers helm-grep helm-regexp helm-plugin helm-elscreen helm-utils
helm-locate helm-help helm-types helm-config helm-easymenu
async-bytecomp async helm-aliases helm helm-source eieio-compat eieio
eieio-core cl-macs helm-multi-match helm-lib dired projectile grep
compile comint ansi-color ibuf-ext ibuffer thingatpt yasnippet cl gv
saveplace hydra ring lv paren solarized-dark-theme solarized-theme
solarized dash byte-opt edmacro kmacro mmr-bootstrap quelpa-use-package
cl-seq quelpa package-build mm-decode mm-bodies mm-encode mail-parse
rfc2231 rfc2047 rfc2045 ietf-drums mm-util help-fns mail-prsvr json
lisp-mnt use-package bytecomp byte-compile cl-extra help-mode cconv
bind-key easy-mmode cl-loaddefs pcase cl-lib diminish linum-ex
bar-cursor advice mmr-minimal mmr-ediff windmove finder-inf info package
easymenu epg-config time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 1017131 97896)
 (symbols 48 52825 1)
 (miscs 40 11688 6032)
 (strings 32 126665 12804)
 (string-bytes 1 3827188)
 (vectors 16 85648)
 (vector-slots 8 1983427 52910)
 (floats 8 1126 2620)
 (intervals 56 75316 4827)
 (buffers 976 104)
 (heap 1024 95787 17598))

bug#21742: 25.0.50; Document that ert-deftest expands all macros

[Lars Ingebrigtsen]

immerrr again <immerrr@gmail.com> writes:

> I started grepping and xref-ing through the code till I found that
> ert-deftest includes cl-macrolet which in turn runs macroexpand-all on
> the test definition.  Quite expectedly, it expands the define-arx macro
> that results in eval-and-compile which is also expanded and all that's
> left in the resulting test definition is a symbol.  More importantly,
> all side effects, i.e. function/macro definitions happen during test
> definition while they should really happen during test execution.

I've now added this to the doc string.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no