bug#30635: No compiler warning if code forgets to require cl-lib

bug#30635: No compiler warning if code forgets to require cl-lib

[Glenn Morris]

Package: emacs
Version: 26.0.91

There's no compiler warning if a library uses cl-lib without requiring it.
I assume this is because bytecomp.el requires cl-lib.
Perhaps bytecomp can track cl-lib specially to work around this.

(I was wondering how the problem shown in 6288c3d went unspotted.)

Example:

foo.el:
(defun foo ()
  (cl-member-if 'cl-evenp '(1 2 3)))

emacs -Q -batch -f batch-byte-compile foo.el
  -> silently succeeds

emacs -Q -l foo.elc
M-: (foo)
 -> void-function cl-member-if

bug#30635: No compiler warning if code forgets to require cl-lib

[Stefan Monnier]

> There's no compiler warning if a library uses cl-lib without requiring it.
> I assume this is because bytecomp.el requires cl-lib.

Should we apply the patch below?


        Stefan


diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index b3ea9300b0..e75403d80d 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -124,17 +124,11 @@
 (require 'backquote)
 (require 'macroexp)
 (require 'cconv)
-(require 'cl-lib)
-
-;; During bootstrap, cl-loaddefs.el is not created yet, so loading cl-lib
-;; doesn't setup autoloads for things like cl-every, which is why we have to
-;; require cl-extra as well (bug#18804).
-(or (fboundp 'cl-every)
-    (require 'cl-extra))
-
-(or (fboundp 'defsubst)
-    ;; This really ought to be loaded already!
-    (load "byte-run"))
+(eval-when-compile
+  ;; We should refrain from loading cl-lib at run-time within the compiler
+  ;; code, otherwise we can't detect if a file forgets to (require 'cl-lib),
+  ;; as mentioned in bug#30635.
+  (require 'cl-lib))
 
 ;; The feature of compiling in a specific target Emacs version
 ;; has been turned off because compile time options are a bad idea.
@@ -3582,7 +3576,10 @@ byte-compile-and-folded
     (cond
      ((< l 3) (byte-compile-form `(progn ,(nth 1 form) t)))
      ((= l 3) (byte-compile-two-args form))
-     ((cl-every #'macroexp-copyable-p (nthcdr 2 form))
+     ;; This used to use `cl-every' but we need to avoid cl-lib at run-time as
+     ;; mentioned at the beginning of this file.
+     ((null (delq nil (mapcar (lambda (e) (not (macroexp-copyable-p e)))
+                              (nthcdr 2 form))))
       (byte-compile-form `(and (,(car form) ,(nth 1 form) ,(nth 2 form))
 			       (,(car form) ,@(nthcdr 2 form)))))
      (t (byte-compile-normal-call form)))))

bug#30635: No compiler warning if code forgets to require cl-lib

[Glenn Morris]

Stefan Monnier wrote:

>> There's no compiler warning if a library uses cl-lib without requiring it.
>> I assume this is because bytecomp.el requires cl-lib.
>
> Should we apply the patch below?

Works for me, thanks.

It reveals a problem in isearch.el since 6bc78d5:

emacs -Q --eval "(setq search-exit-option 'append)"
C-s
 -> Error in pre-command-hook (isearch-pre-command-hook): (void-function cl-every)

bug#30635: No compiler warning if code forgets to require cl-lib

[Stefan Monnier]

>>> There's no compiler warning if a library uses cl-lib without requiring it.
>>> I assume this is because bytecomp.el requires cl-lib.
>> Should we apply the patch below?
> Works for me, thanks.

Doesn't work for me, OTOH:

    % src/emacs --batch --eval "(eval-after-load 'cl-lib '(debug t))"  -f batch-byte-compile lisp/emacs-lisp/bytecomp.el
    Debugger entered--beginning evaluation of function call form:
      (lambda nil (debug t))()
      eval-after-load-helper("/home/monnier/src/emacs/trunk/lisp/emacs-lisp/cl-lib.elc")
      run-hook-with-args(eval-after-load-helper "/home/monnier/src/emacs/trunk/lisp/emacs-lisp/cl-lib.elc")
      do-after-load-evaluation("/home/monnier/src/emacs/trunk/lisp/emacs-lisp/cl-lib.elc")
      require(cl-lib)
      require(seq)
      require(map)
      dir-locals-read-from-dir("/home/monnier/src/emacs/trunk/")
      hack-dir-local-variables()
      hack-local-variables(no-mode)
      normal-mode(t)
      byte-compile-file("lisp/emacs-lisp/bytecomp.el")
      batch-byte-compile-file("lisp/emacs-lisp/bytecomp.el")
      batch-byte-compile()
      command-line-1(("--eval" "(eval-after-load 'cl-lib '(debug t))" "-f" "batch-byte-compile" "lisp/emacs-lisp/bytecomp.el"))
      command-line()
      normal-top-level()

and if you look at `dir-locals-read-from-dir`, you'll see that it does
(require 'map) and then uses its `map-merge(-with)`.


        Stefan "the pressure to preload cl-lib is definitely mounting"

bug#30635: No compiler warning if code forgets to require cl-lib

[Stefan Monnier]

>>>> There's no compiler warning if a library uses cl-lib without requiring it.
>>>> I assume this is because bytecomp.el requires cl-lib.
>>> Should we apply the patch below?
>> Works for me, thanks.
> Doesn't work for me, OTOH:
[...]
> and if you look at `dir-locals-read-from-dir`, you'll see that it does
> (require 'map) and then uses its `map-merge(-with)`.

And if I tweak dir-locals-read-from-dir so it doesn't require `map`, and
then try to compile a file like gnus-cloud.el where I removed all the
`require`s, I still get:

    % src/emacs --batch --eval "(eval-after-load 'cl-lib '(debug t))"  -f batch-byte-compile lisp/gnus/gnus-cloud.el  
    Debugger entered--beginning evaluation of function call form:
      (lambda nil (debug t))()
      [...]
      require(cl-lib)
      byte-code("\300\301!\210\300\302!\210\300\303!\207" [require cl-lib macroexp gv] 2)
      cl--defsubst-expand((cl-x) (cl-block epg-context-armor (or (cl--struct-epg-context-p cl-x) (signal 'wrong-type-argument (list 'epg-context cl-x))) (aref cl-x 4)) nil nil nil context)
      epg-context-armor--cmacro((epg-context-armor context) context)
      apply(epg-context-armor--cmacro (epg-context-armor context) context)
      gv-get((epg-context-armor context) #f(compiled-function (getter setter) #<bytecode 0x57f35d>))
      [...]
      macroexpand-all((defalias 'gnus-cloud-encode-data ...) ...)
      [...]
      byte-compile-recurse-toplevel((defun gnus-cloud-encode-data ...) ...)

The end result is better tho; if I then add a call to `cl-every` into
that file, I do get a warning of the form:

    gnus/gnus-cloud.el:512:1:Warning: the function ‘cl-every’ might not be
        defined at runtime.

So I pushed the previous two patches to master, since I think they at
least partly fix this bug.


        Stefan

bug#30635: No compiler warning if code forgets to require cl-lib

[Glenn Morris]

Stefan Monnier wrote:

> And if I tweak dir-locals-read-from-dir so it doesn't require `map`, and

Nice; this provides a measurable build speed-up by finally getting rid
of some of the slowdown introduced by support for multiple dir-locals.

(bug#22307)

bug#30635: No compiler warning if code forgets to require cl-lib

[Basil L. Contovounesios]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> So I pushed the previous two patches to master, since I think they at
> least partly fix this bug.

I've bisected the following build error to your commit 97b7e58c4d "Try
and fix the more obvious sources of bug#30635" of 2018-03-22:

    [...]
    Dumping under the name emacs
    13102432 of 33554432 static heap bytes used
    99255 pure bytes used
    mv -f emacs bootstrap-emacs
    make -C ../lisp compile-first EMACS="../src/bootstrap-emacs"
    make[2]: Entering directory '/home/blc/.local/src/emacs/lisp'
      ELC      emacs-lisp/bytecomp.elc
    Error reading dir-locals: (invalid-read-syntax "#")
    
    In toplevel form:
    emacs-lisp/bytecomp.el:124:1:Error: Symbol’s value as variable is void: =
    Makefile:301: recipe for target 'emacs-lisp/bytecomp.elc' failed
    make[2]: *** [emacs-lisp/bytecomp.elc] Error 1
    make[2]: Leaving directory '/home/blc/.local/src/emacs/lisp'
    Makefile:745: recipe for target 'bootstrap-emacs' failed
    make[1]: *** [bootstrap-emacs] Error 2
    make[1]: Leaving directory '/home/blc/.local/src/emacs/src'
    Makefile:418: recipe for target 'src' failed
    make: *** [src] Error 2

The immediately preceding incantation on my part is:

    make clean && ./configure [...] && make

where the configuration used follows my signature.

Subsequently invoking 'make boostrap' similarly barfs with:

    Dumping under the name emacs
    11966048 of 33554432 static heap bytes used
    2450001 pure bytes used
    Adding name emacs-27.0.50.1
    ln -f emacs bootstrap-emacs
    make[2]: Leaving directory '/home/blc/.local/src/emacs/src'
    make -C lisp all
    make[2]: Entering directory '/home/blc/.local/src/emacs/lisp'
    make -C ../leim all EMACS="../src/emacs"
    make[3]: Entering directory '/home/blc/.local/src/emacs/leim'
    make[3]: Nothing to be done for 'all'.
    make[3]: Leaving directory '/home/blc/.local/src/emacs/leim'
    make -C ../admin/grammars all EMACS="../../src/emacs"
    make[3]: Entering directory '/home/blc/.local/src/emacs/admin/grammars'
    make[3]: Nothing to be done for 'all'.
    make[3]: Leaving directory '/home/blc/.local/src/emacs/admin/grammars'
    make[3]: Entering directory '/home/blc/.local/src/emacs/lisp'
      ELC      emacs-lisp/eieio.elc
    
    In toplevel form:
    emacs-lisp/eieio.el:Error: Invalid read syntax: "#"
    Error reading dir-locals: (invalid-function "
    In toplevel form:
    emacs-lisp/eieio.el:Error: Invalid read syntax: \"#\"")
    emacs-lisp/eieio.el:52:4:Error: Symbol’s value as variable is void: =
    Makefile:301: recipe for target 'emacs-lisp/eieio.elc' failed
    make[3]: *** [emacs-lisp/eieio.elc] Error 1
    make[3]: Leaving directory '/home/blc/.local/src/emacs/lisp'
    Makefile:324: recipe for target 'compile-main' failed
    make[2]: *** [compile-main] Error 2
    make[2]: Leaving directory '/home/blc/.local/src/emacs/lisp'
    Makefile:405: recipe for target 'lisp' failed
    make[1]: *** [lisp] Error 2
    make[1]: Leaving directory '/home/blc/.local/src/emacs'
    Makefile:1099: recipe for target 'bootstrap' failed
    make: *** [bootstrap] Error 2

Any ideas on where I am or what is going wrong and how I can further
troubleshoot this?

TIA,

-- 
Basil

Configured using:
 'configure --prefix=/home/blc/.local --with-mailutils
 --with-x-toolkit=lucid --with-modules --with-file-notification=yes
 --with-x 'CFLAGS=-march=native -O2 -pipe'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS LUCID X11 MODULES THREADS LIBSYSTEMD LCMS2

bug#30635: No compiler warning if code forgets to require cl-lib

[Basil L. Contovounesios]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Any ideas on where I am or what is going wrong and how I can further
> troubleshoot this?

I ultimately nuked everything via 'git clean -fdx', as per INSTALL.REPO,
and the errors no longer appear.  Sorry about the noise.

-- 
Basil