builds are getting slower?

builds are getting slower?

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]


Subjectively it felt to me lately like it was taking longer to build
Emacs than it used to. When I looked at my daily build logs, that seems
to be confirmed - see attached figure (time for a full bootstrap).
In particular, things seem to have gotten about 30% slower around 10th
November. Maybe (?) this is borne out by
http://hydra.nixos.org/job/gnu/emacs-trunk/tarball#tabs-charts
or maybe it isn't.

Anyway, I just mention it in case anyone feels like investigating.


[-- Attachment #2: bootstrap_time.png --]
[-- Type: image/png, Size: 11155 bytes --]

Re: builds are getting slower?

[Paul Eggert]

Eyeballing it, it looks like there was one slowdown in Emacs master around Oct 
19 and another around Nov 17.  You have the raw data; can you give the commit 
IDs both before and after these observed slowdowns?

Re: builds are getting slower?

[Glenn Morris]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

Paul Eggert wrote:

> Eyeballing it, it looks like there was one slowdown in Emacs master
> around Oct 19 and another around Nov 17.  You have the raw data; can
> you give the commit IDs both before and after these observed
> slowdowns?

I don't have that level of granularity. These builds were (normally)
started at 7.30am Pacific time, so correspond to whatever was in the
repo at that time. I didn't track the commit id. The raw data are
attached.


[-- Attachment #2: buildsecs.txt --]
[-- Type: text/plain, Size: 4189 bytes --]

20150101 830.799
20150103 843.938
20150104 817.782
20150106 881.91
20150108 876.898
20150109 864.786
20150110 870.198
20150111 932.539
20150112 935.13
20150113 938.132
20150114 942.447
20150117 952.583
20150120 952.581
20150121 966.927
20150122 974.282
20150123 975.117
20150127 1006.1
20150130 1026.03
20150131 1031.14
20150201 1026.69
20150202 1029.29
20150203 1029.92
20150204 1029.65
20150206 1038.55
20150207 1111.3
20150208 1031
20150209 1024.58
20150210 1032.88
20150211 1039.69
20150212 1036.35
20150213 1040.59
20150214 1050.08
20150215 1050.29
20150216 1049.97
20150217 1094.29
20150218 1057.52
20150219 1049.77
20150220 1050.2
20150221 1045.97
20150222 1045.69
20150223 1048.9
20150225 1057.69
20150227 1056.93
20150228 1066.25
20150301 1053.63
20150302 1052.7
20150304 1079.16
20150305 1065.26
20150306 1062.64
20150307 1083.73
20150311 1059.98
20150312 1061.7
20150313 1062.82
20150314 1060.92
20150319 1075.09
20150320 1073.91
20150321 1074.9
20150322 1078.4
20150323 1077.47
20150325 1080.25
20150326 1081.28
20150327 1081.46
20150328 1031.63
20150329 1031
20150330 1026.48
20150331 1036.17
20150401 1034.82
20150402 1035.76
20150403 1035.5
20150404 1031.84
20150405 1033.95
20150406 1034.55
20150407 1033.37
20150408 1034.11
20150409 1033.95
20150410 1037.97
20150411 1037.5
20150412 1039.62
20150413 1040.7
20150414 1088.47
20150417 1116.09
20150418 1086.6
20150425 1094.13
20150426 1096.18
20150502 1090.73
20150503 1095.99
20150509 1089.2
20150510 1098.74
20150514 1049.84
20150517 1053.94
20150524 1058.22
20150528 1096.38
20150529 1077.04
20150530 1135.07
20150531 1123.61
20150601 1073.57
20150603 1103.79
20150606 984.518
20150607 1033.87
20150613 955.557
20150617 960.781
20150619 973.94
20150620 962.837
20150621 959.901
20150626 960.811
20150701 959.047
20150702 952.712
20150703 953.747
20150704 970.472
20150705 957.688
20150706 958.987
20150708 980.255
20150709 982.61
20150710 982.913
20150711 996.511
20150712 1007.04
20150713 1004.33
20150714 993.642
20150715 993.568
20150717 975.349
20150718 983.545
20150719 984.618
20150720 994.289
20150722 952.925
20150723 976.664
20150724 979.348
20150725 968.588
20150726 975.447
20150727 974.083
20150728 988.994
20150729 991.112
20150730 997.056
20150731 991.928
20150801 996.333
20150802 1007.47
20150803 990.978
20150804 992.024
20150805 965.451
20150806 970.614
20150807 993.034
20150808 1026.95
20150809 971.832
20150810 970.767
20150811 969.963
20150813 971.245
20150814 976.036
20150815 981.821
20150816 982.394
20150817 985.848
20150818 974.948
20150819 978.103
20150820 983.572
20150821 989.445
20150822 995.02
20150824 995.744
20150825 984.628
20150826 965.071
20150828 974.074
20150829 970.529
20150830 963.299
20150831 966.725
20150902 980.007
20150903 974.879
20150904 972.26
20150905 976.507
20150906 978.598
20150907 979.484
20150908 996.367
20150910 989.794
20150911 993.415
20150912 961.313
20150913 961.133
20150914 961.876
20150915 964.06
20150916 958.888
20150917 971.086
20150918 972.313
20150919 966.132
20150920 967.138
20150921 968.586
20150922 966.193
20150923 959.525
20150924 958.269
20150925 965.446
20150926 964.584
20150927 965.49
20150928 958.607
20150929 964.871
20150930 968.464
20151001 971.856
20151002 975.958
20151003 983.583
20151004 970.008
20151005 978.057
20151006 977.279
20151008 979.724
20151009 988.965
20151010 964.139
20151012 970.663
20151013 963.012
20151015 1030.35
20151016 1030.81
20151017 1027.37
20151018 1024.63
20151019 1027.26
20151020 1033.55
20151021 1095.59
20151023 1034.97
20151024 1040.52
20151025 1037.96
20151026 1044.14
20151027 1042.13
20151028 1041.36
20151029 1045.51
20151030 1045.21
20151031 1034.06
20151101 1027.41
20151102 1032.37
20151103 1032.79
20151104 1035.93
20151105 1040.39
20151106 1044.94
20151107 1069.06
20151108 1047.21
20151109 1040.85
20151110 1288.94
20151111 1292.89
20151113 1297.49
20151114 1303.17
20151115 1302.23
20151116 1315.56
20151117 1309.62
20151118 1309.82
20151119 1310.88
20151120 1311.57
20151121 1312.69
20151122 1312.26
20151123 1316.19
20151124 1317.39
20151125 1317.65
20151126 1306.24
20151128 1302.35
20151129 1302.26
20151130 1298.47
20151202 1310.04
20151207 1322.69
20151208 1315.94
20151209 1315.32

Re: builds are getting slower?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 911 bytes --]

Glenn Morris wrote:
> I don't have that level of granularity. These builds were (normally)
> started at 7.30am Pacific time, so correspond to whatever was in the
> repo at that time. I didn't track the commit id.

 From that raw data, there's a smaller performance hit between 20151013 and 
20151015.  The log for that time period (cutoff at 0730 Pacific time) is 
attached as log1.txt.  The commit I'd have the most suspicion of is 
2cc412cdc2635ecb99129271abe94bdd744742c2.  You might try reversing that and 
seeing whether it boosts performance for you.  (What flags are you passing to 
'configure' and to 'make'?  What is your platform?)

There's a larger performance hit between 20151109 and 20151110.  The log for 
that time period (0730 Pacific cutoff) is attached as log2.txt.  Nothing jumps 
out, I'm afraid.

Maybe next time you could record the commit ID as well; that might help track 
things down.

[-- Attachment #2: log1.txt --]
[-- Type: text/plain, Size: 8795 bytes --]

commit 59def59158f9c5081664253e30362d0061a2a64f
Author:     Dmitry Gutov <dgutov@yandex.ru>
AuthorDate: Thu Oct 15 02:18:16 2015
Commit:     Dmitry Gutov <dgutov@yandex.ru>
CommitDate: Thu Oct 15 02:18:16 2015

    Refer to `(elisp)Basic Completion' in completing-read docstring

    * src/minibuf.c (Fcompleting_read): Refer to `(elisp)Basic
    Completion' in the docstring (bug#21644).

commit 453af81f16a7eca0bc246c9278d19ba995641dfa
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Wed Oct 14 16:00:31 2015
Commit:     Mark Oteiza <mvoteiza@udel.edu>
CommitDate: Wed Oct 14 16:00:31 2015

    * lisp/mpc.el (mpc-format): Always push form to pred

commit 943f7f902ed3d7a5ce00bbb5a9cc60e01607b661
Author:     Paul Eggert <eggert@cs.ucla.edu>
AuthorDate: Wed Oct 14 14:46:22 2015
Commit:     Paul Eggert <eggert@cs.ucla.edu>
CommitDate: Wed Oct 14 14:46:50 2015

    Spelling fixes

    * configure.ac (bitmapdir): Fix misspelling of bmd_acc.
    * test/automated/coding-tests.el (ert-test-coding-bogus-coding-systems):
    Fix misspelling of nonexistent file name.

commit 100a96c92ba9cdf49f695edcf74975321eafae2e
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Wed Oct 14 14:32:47 2015
Commit:     Mark Oteiza <mvoteiza@udel.edu>
CommitDate: Wed Oct 14 14:32:47 2015

    * lisp/mpc.el (mpc-mode-menu, mpc-toggle-play): Fix docstrings

commit e646242e13cd41742a3ff2acb7a0c3eec4e874e9
Author:     Michael Albinus <michael.albinus@gmx.de>
AuthorDate: Wed Oct 14 11:16:14 2015
Commit:     Michael Albinus <michael.albinus@gmx.de>
CommitDate: Wed Oct 14 11:16:14 2015

    Some editing fixes in Tramp

    * lisp/net/tramp-gvfs.el:
    * doc/misc/tramp.texi: "customer option" -> "custom option".

    * lisp/net/tramp.el (tramp-completion-function-alist): Fix docstring.

commit 8318dec8e167a3c688534e3f9faff862b7e374ac
Author:     Michael Albinus <michael.albinus@gmx.de>
AuthorDate: Wed Oct 14 11:10:14 2015
Commit:     Michael Albinus <michael.albinus@gmx.de>
CommitDate: Wed Oct 14 11:10:14 2015

    ; Submit changes promised last commit already

commit 0cced99164baab5b56ee57e6b1dccea21164c85e
Author:     Jürgen Hötzel <juergen@archlinux.org>
AuthorDate: Wed Oct 14 11:09:03 2015
Commit:     Michael Albinus <michael.albinus@gmx.de>
CommitDate: Wed Oct 14 11:09:03 2015

    Use proper localization in tramp-gvfs.el

    * lisp/net/tramp-gvfs.el (tramp-gvfs-handle-file-attributes):
    Suppress localized settings in order to proper parse gfvs output.

commit 96764333c438566b20fd381960fef77fa12eb586
Author:     Warren Lynn <wrn.lynn@gmail.com>
AuthorDate: Wed Oct 14 11:03:50 2015
Commit:     Michael Albinus <michael.albinus@gmx.de>
CommitDate: Wed Oct 14 11:03:50 2015

    Fix Bug#21562

    * lisp/net/tramp-sh.el (tramp-do-copy-or-rename-file-out-of-band):
    Quote argument in proper order.  (Bug#21562)

    Copyright-paperwork-exempt: yes

commit 17d4f60b55dd7998a4afffa9413c9e1731791f1a
Author:     Nicolas Petton <nicolas@petton.fr>
AuthorDate: Wed Oct 14 08:55:53 2015
Commit:     Nicolas Petton <nicolas@petton.fr>
CommitDate: Wed Oct 14 08:57:32 2015

    Fix typos in docstrings

    * lisp/emacs-lisp/map.el:
    * lisp/emacs-lisp/seq.el: Fix typos in the docstrings of the pcase
      macros.

commit aebf282aec46e3e8246a3dce535ecccf173c391b
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Wed Oct 14 05:52:44 2015
Commit:     Mark Oteiza <mvoteiza@udel.edu>
CommitDate: Wed Oct 14 05:52:44 2015

    * lisp/mpc.el (mpc-volume-refresh): Check if buffer is live.

commit c6cbf6c06cceaf1f8d855e070df3c3d5a0bdf8e6
Author:     Glenn Morris <rgm@gnu.org>
AuthorDate: Wed Oct 14 03:22:34 2015
Commit:     Glenn Morris <rgm@gnu.org>
CommitDate: Wed Oct 14 03:22:34 2015

    ; Auto-commit of loaddefs files.

commit 0e38e94b1c4125951b198b213a67e010cafcd724
Author:     Oleh Krehel <ohwoeowho@gmail.com>
AuthorDate: Wed Oct 14 02:19:21 2015
Commit:     Oleh Krehel <ohwoeowho@gmail.com>
CommitDate: Wed Oct 14 02:24:08 2015

    Make dired-jump work with tar-subfile-mode

    * lisp/dired-x.el (dired-jump): When in `tar-subfile-mode', instead of
      emitting an error, switch to `tar-superior-buffer'.

commit f147c0f3a5f32343b381e8412907235ade27a81d
Author:     Juanma Barranquero <lekktu@gmail.com>
AuthorDate: Wed Oct 14 00:44:40 2015
Commit:     Juanma Barranquero <lekktu@gmail.com>
CommitDate: Wed Oct 14 00:46:57 2015

    * .gitignore: Add build-aux/ar-lib.

commit b5e2d7495017e0d87de331f41838810b72730942
Author:     Nicolas Petton <nicolas@petton.fr>
AuthorDate: Wed Oct 14 00:37:59 2015
Commit:     Nicolas Petton <nicolas@petton.fr>
CommitDate: Wed Oct 14 00:40:10 2015

    Better docstrings in seq.el and map.el

    * lisp/emacs-lisp/map.el:
    * lisp/emacs-lisp/seq.el: Improve the docstring for the pcase patterns.

commit e668176e7d89e885902287da18c69297bf04fed3
Author:     Paul Eggert <eggert@cs.ucla.edu>
AuthorDate: Tue Oct 13 23:34:16 2015
Commit:     Paul Eggert <eggert@cs.ucla.edu>
CommitDate: Tue Oct 13 23:34:47 2015

    Merge from gnulib

    This incorporates:
    2015-10-13 binary-io, u64, unistd: port to strict C
    2015-09-26 c-ctype: do not worry about EBCDIC + char signed
    2015-09-25 c-ctype: port better to z/OS EBCDIC
    2015-09-25 gnulib-common.m4: fix gl_PROG_AR_RANLIB/AM_PROG_AR clash
    * doc/misc/texinfo.tex, lib/binary-io.c, lib/c-ctype.h, lib/u64.c:
    * lib/unistd.c, m4/gnulib-common.m4, m4/gnulib-comp.m4:
    Copy from gnulib.

commit 2cc412cdc2635ecb99129271abe94bdd744742c2
Author:     Paul Eggert <eggert@cs.ucla.edu>
AuthorDate: Tue Oct 13 23:09:43 2015
Commit:     Paul Eggert <eggert@cs.ucla.edu>
CommitDate: Tue Oct 13 23:10:14 2015

    Take XPNTR private

    * src/alloc.c (PURE_POINTER_P): Remove.
    All uses replaced with PURE_P.
    (XPNTR_OR_SYMBOL_OFFSET): New function.
    (XPNTR): Move here from lisp.h.
    Reimplement in terms of XPNTR_OR_SYMBOL_OFFSET.
    (mark_maybe_object, valid_lisp_object_p, survives_gc_p):
    Remove unnecessary cast.
    (purecopy): Use XPNTR_OR_SYMBOL_OFFSET instead of XPNTR,
    to avoid an unnecessary runtime test for symbols.
    * src/lisp.h (lisp_h_XPNTR, XPNTR): Remove, moving XPNTR to alloc.c.
    Only alloc.c needs XPNTR now.

commit 3fa424ca48f4f48a40a514153726758b6ed47892
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Tue Oct 13 19:49:58 2015
Commit:     Mark Oteiza <mvoteiza@udel.edu>
CommitDate: Tue Oct 13 19:49:58 2015

    Add MPC play/pause command

    * lisp/mpc.el (mpc-toggle-play): New command.
    (mpc-mode-map): Bind it to "s".
    (mpc-mode-menu): Add corresponding menu item.

commit a7e6637162827a09c13e72a49412b3ab915cf473
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Tue Oct 13 19:08:48 2015
Commit:     Mark Oteiza <mvoteiza@udel.edu>
CommitDate: Tue Oct 13 19:27:57 2015

    Add bindings and menu items for prev and next tracks

    * lisp/mpc.el (mpc-mode-map): Bind ">" to mpc-next,
    "<" to mpc-prev.
    (mpc-mode-menu): Add corresponding menu items

commit 9fa9c26e42ddb3f67133bc18112147926bae8a50
Author:     Ken Raeburn <raeburn@raeburn.org>
AuthorDate: Tue Oct 13 19:06:01 2015
Commit:     Ken Raeburn <raeburn@raeburn.org>
CommitDate: Tue Oct 13 19:12:55 2015

    Reduce face-related consing during frame creation.

    * faces.el (face--attributes-unspecified): Compute the "unspecified"
    attribute list once.
    (face-spec-reset-face): Use it instead of building the list.

commit 85c12310ff9a6721fb1ecbfdf6d89e59a34fb882
Author:     Ken Raeburn <raeburn@permabit.com>
AuthorDate: Tue Oct 13 16:33:15 2015
Commit:     Ken Raeburn <raeburn@raeburn.org>
CommitDate: Tue Oct 13 19:12:48 2015

    Do process ConfigureNotify events indicating size changes.

    * src/xterm.c (handle_one_xevent): If consecutive ConfigureNotify
    events don't have the same size, process each one.

commit e90de8276fb8c8365be8b8d0f696b3c93c4b7c4f
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Tue Oct 13 15:14:49 2015
Commit:     Mark Oteiza <mvoteiza@udel.edu>
CommitDate: Tue Oct 13 15:14:49 2015

    Derive mpc-mode from special-mode

    lisp/mpc.el (mpc-mode-map): Make from sparse keymap. Unbind g.
    (mpc-mode): Derive from special mode.
    (mpc-songs-mode-map): Don't set parent keymap.

commit 18b0eb7f1cc16fe33f89c74d2497a6fcb4b863fd
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Tue Oct 13 11:19:18 2015
Commit:     Mark Oteiza <mvoteiza@udel.edu>
CommitDate: Tue Oct 13 11:25:35 2015

    Fix error messages for when covers are not found.

    The last change to mpc-format let the binding to file call
    mpc-file-local-copy with nil argument. Instead, employ if-let here so
    nil bindings don't result in needless computation and errors.
    * lisp/mpc.el: Require 'subr-x at compile time.
    * lisp/mpc.el (mpc-format): Use if-let.

[-- Attachment #3: log2.txt --]
[-- Type: text/plain, Size: 8260 bytes --]

commit d149ca81c33ab95900306c3c71f6eff62a3ec1a1
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Tue Nov 10 05:47:42 2015
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 06:45:50 2015

    * doc/lispref/variables.texi (Directory Local Variables):

    Document dir-locals wildcards

    * lisp/files.el (dir-locals-file): Point to Info node.

    * doc/emacs/custom.texi (Directory Variables):
    Document dir-locals wildcards.

    * etc/NEWS: Document new functionality.

commit 9145e79dc2042fb477959ddda59c3e2ff5fa3914
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Tue Nov 10 05:26:00 2015
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 05:26:00 2015

    * lisp/files.el: Don't allow customization of dir-locals sorting

    In retrospect, this is not a good idea for the same reason that
    `dir-locals-file' is a defconst, because it is important that this
    behaviour be "uniform across different environments and users".
    Sure, the user can still change the sorting with a hack, but we
    shouldn't encourage them to change it.

    (dir-locals--all-files): Return list in the order returned by
    `file-expand-wildcards'.
    (file-expand-wildcards): Document the sorting predicate used.
    (dir-locals-sort-predicate): Delete variable.

commit 77cebbc1e77edf23bc2c23a218b56d9d6ad68e74
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Tue Nov 10 05:14:49 2015
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 05:14:49 2015

    * lisp/files.el (dir-locals-read-from-file): Better handle errors

commit 4d82aa3abdad1871b99f0a9e56fe54ec497bd290
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Tue Nov 10 05:04:02 2015
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 05:04:31 2015

    * lisp/isearch.el (search-default-regexp-mode): change default value

commit 1e98f041acae7cee012b1b157d4aa3f80b226123
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Mon Nov 9 20:13:25 2015
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 05:04:31 2015

    * lisp/files.el (dir-locals-find-file): Don't stop at unreadable files

    `locate-dominating-file' will now keep looking if the files it finds in
    a given directory are unreadable (or not files).

commit 2e8488858c7b8df40610c1cd3038348fdc9bf3ed
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Sat Nov 7 05:54:56 2015
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 05:04:31 2015

    * lisp/files.el (dir-locals-file): Allow wildcards

    (dir-locals-find-file, dir-locals-collect-variables)
    (dir-locals-read-from-file): Update accordingly.
    (hack-dir-local-variables): Rename a local variable.

    * lisp/files-x.el (modify-dir-local-variable): Update accordingly

    * lisp/help-fns.el (describe-variable): Update accordingly

    * .gitignore: Add .dir-locals?.el

commit cbaa04014e0c9efdfc6393bccde0e6579b5d7051
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Sat Nov 7 04:45:18 2015
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 05:04:30 2015

    * lisp/emacs-lisp/map.el (map-merge-with): New function

    * test/automated/map-tests.el (test-map-merge-with): New test

commit cbc51211f9e4f8f3d4b8a1feaa6cbfd2fd4ac1ca
Author:     Dmitry Gutov <dgutov@yandex.ru>
AuthorDate: Mon Nov 9 18:39:32 2015
Commit:     Dmitry Gutov <dgutov@yandex.ru>
CommitDate: Tue Nov 10 03:31:20 2015

    ; project-library-roots-function: Update the FIXME

commit 0f50e5163cf747fcf18124039a82b5156a48316b
Author:     Karl Fogel <kfogel@red-bean.com>
AuthorDate: Mon Nov 9 20:14:49 2015
Commit:     Karl Fogel <kfogel@red-bean.com>
CommitDate: Mon Nov 9 20:14:54 2015

    Fix some recently-perturbed bookmark autoloads

    * lisp/bookmark.el (bookmark-set-internal): Remove unnecessary autoload.
      (bookmark-set): Restore autoload.
      (bookmark-set-no-overwrite): Add autoload.

    Thanks to Juanma Barranquero for noticing the autoload problems
    introduced by my recent commit adding/changing the above functions
    (Sun Nov 8 14:16:43 2015 -0500, git commit 3812e17978).

commit f5eac7baefacd8944a1851596a944fd29dec98fa
Author:     Noah Friedman <friedman@splode.com>
AuthorDate: Mon Nov 9 17:34:40 2015
Commit:     Noah Friedman <friedman@splode.com>
CommitDate: Mon Nov 9 17:34:40 2015

    (ydump-buffer): Handle case where gap is at the start of buffer.
    I don't recall if older versions of gdb were less strict but you
    cannot dump a 0-length range in gdb 7.9.1.

commit 82d59f1b3ba6d7ad9d9cd0af15e237f97bb5906b
Author:     Dmitry Gutov <dgutov@yandex.ru>
AuthorDate: Mon Nov 9 16:56:55 2015
Commit:     Dmitry Gutov <dgutov@yandex.ru>
CommitDate: Mon Nov 9 16:56:55 2015

    * lisp/progmodes/project.el: Update Commentary.

commit 0be6fb8e17f708fe03430d0b1e701810ae51b5e3
Merge: 3c3aad7 1c72afb
Author:     Dmitry Gutov <dgutov@yandex.ru>
AuthorDate: Mon Nov 9 16:47:46 2015
Commit:     Dmitry Gutov <dgutov@yandex.ru>
CommitDate: Mon Nov 9 16:47:46 2015

    Merge branch 'project-next'

commit 1c72afb7aa48c2ea06103113ef70ccea0c1c961d
Author:     Dmitry Gutov <dgutov@yandex.ru>
AuthorDate: Mon Nov 9 16:41:06 2015
Commit:     Dmitry Gutov <dgutov@yandex.ru>
CommitDate: Mon Nov 9 16:41:06 2015

    Fold `project-ask-user' into `project-current'

    * lisp/progmodes/project.el (project-find-functions): Remove
    `project-ask-user'.
    (project-ask-user): Remove function and the corresponding
    `project-roots' implementation.
    (project-current): Add a new argument, MAYBE-PROMPT.  Prompt the
    user in case there's no project in the current directory.  Update
    all callers.

commit 3c3aad733522365a8fe729d7c92e64e98bc4ce92
Author:     Karl Fogel <kfogel@red-bean.com>
AuthorDate: Mon Nov 9 13:57:23 2015
Commit:     Karl Fogel <kfogel@red-bean.com>
CommitDate: Mon Nov 9 13:57:29 2015

    When VC detects a conflict, specify which file

    * lisp/vc/vc.el (vc-message-unresolved-conflicts): New function.
    * lisp/vc/vc-svn.el (vc-svn-find-file-hook):
    * lisp/vc/vc-hg.el (vc-hg-find-file-hook):
    * lisp/vc/vc-bzr.el (vc-bzr-find-file-hook):
    * lisp/vc/vc-git.el (vc-git-find-file-hook): Use above new function
      to display a standard message that specifies the conflicted file.

    Before this change, the message VC used for indicating a conflicted
    file was just "There are unresolved conflicts in this file" without
    naming the file (and this language was duplicated in several places).
    After this change, it's "There are unresolved conflicts in file FOO"
    (and this language is now centralized in one function in vc.el).

    Justification: It's important for the message to name the conflicted
    file because the moment when VC realizes a file is conflicted does not
    always come interactively.  For example, some people automatically
    find a set of Org Mode files on startup, and may keep those .org files
    under version control.  If any of the files are conflicted, the user
    just sees some messages fly by, and might later check the "*Messages*"
    buffer to find out what files were conflicted.  I'm not saying this
    happened to me or anything; it's a purely hypothetical example.

commit 86c19714b097aa477d339ed99ffb5136c755a046
Author:     Eli Zaretskii <eliz@gnu.org>
AuthorDate: Mon Nov 9 10:31:45 2015
Commit:     Eli Zaretskii <eliz@gnu.org>
CommitDate: Mon Nov 9 10:31:45 2015

    Fix assertion violation in define-key

    * src/keymap.c (store_in_keymap): Don't use XFASTINT on non-character
    objects.  Reported by Drew Adams <drew.adams@oracle.com>
    and Juanma Barranquero <lekktu@gmail.com>.

commit c6c16fb3f8fe5909baafd53c6b26153dec021064
Author:     Dima Kogan <dima@secretsauce.net>
AuthorDate: Mon Nov 9 08:36:05 2015
Commit:     Eli Zaretskii <eliz@gnu.org>
CommitDate: Mon Nov 9 08:36:05 2015

    Fix a memory leak in GC of font cache

    * src/alloc.c (compact_font_cache_entry): Don't GC unmarked font
    entities if some of the fonts it references are marked.  This
    plugs a memory leak.  (Bug#21556)

Re: builds are getting slower?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

On 10 Dec 2015 6:53 am, "Paul Eggert" <eggert@cs.ucla.edu> wrote:
> There's a larger performance hit between 20151109 and 20151110.  The log
for that time period (0730 Pacific cutoff) is attached as log2.txt.
Nothing jumps out, I'm afraid.
>

In one of my commis there I think I added (require 'map) to files.el, but I
don't think the impact of that would be so noticeable.

There's another commit there which could affect performance (it talks about
not using XFASTINT, in a function that is used a bajillion times throughout
core). It's this one by Eli.

commit 86c19714b097aa477d339ed99ffb5136c755a046
Author:     Eli Zaretskii <eliz@gnu.org>
AuthorDate: Mon Nov 9 10:31:45 2015
Commit:     Eli Zaretskii <eliz@gnu.org>
CommitDate: Mon Nov 9 10:31:45 2015

    Fix assertion violation in define-key

    * src/keymap.c (store_in_keymap): Don't use XFASTINT on non-character
    objects.  Reported by Drew Adams <drew.adams@oracle.com>
    and Juanma Barranquero <lekktu@gmail.com>.

[-- Attachment #2: Type: text/html, Size: 1429 bytes --]

Re: builds are getting slower?

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 10 Dec 2015 6:53 am, "Paul Eggert" <eggert@cs.ucla.edu> wrote:
>> There's a larger performance hit between 20151109 and 20151110.  The log
> for that time period (0730 Pacific cutoff) is attached as log2.txt.
> Nothing jumps out, I'm afraid.
>>
>
> In one of my commis there I think I added (require 'map) to files.el, but I
> don't think the impact of that would be so noticeable.

If that _replaces_ some builtins by more generic stuff, that could
explain some of it.

> There's another commit there which could affect performance (it talks about
> not using XFASTINT, in a function that is used a bajillion times throughout
> core). It's this one by Eli.
>
> commit 86c19714b097aa477d339ed99ffb5136c755a046
> Author:     Eli Zaretskii <eliz@gnu.org>
> AuthorDate: Mon Nov 9 10:31:45 2015
> Commit:     Eli Zaretskii <eliz@gnu.org>
> CommitDate: Mon Nov 9 10:31:45 2015
>
>     Fix assertion violation in define-key
>
>     * src/keymap.c (store_in_keymap): Don't use XFASTINT on non-character
>     objects.  Reported by Drew Adams <drew.adams@oracle.com>
>     and Juanma Barranquero <lekktu@gmail.com>.

I doubt it is this one.  It may be used "bajillion times" throughout the
core, but as a rule not in a loop.  So XFASTINT may be replaced by XINT
(or whatever) a few ten thousands of times.  That should not be enough
to cause a significant difference.

-- 
David Kastrup

Re: builds are getting slower?

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Wed, 09 Dec 2015 21:15:38 -0500
> 
> Subjectively it felt to me lately like it was taking longer to build
> Emacs than it used to. When I looked at my daily build logs, that seems
> to be confirmed - see attached figure (time for a full bootstrap).
> In particular, things seem to have gotten about 30% slower around 10th
> November. Maybe (?) this is borne out by
> http://hydra.nixos.org/job/gnu/emacs-trunk/tarball#tabs-charts
> or maybe it isn't.

Strange: the plot at Hydra doesn't seem to show any perceptible
tendency towards longer times.  If I draw an imaginary line to show
the long-term average, it stays at about the same value.  Or maybe I
misunderstand the graphs there.

I have a few questions:

  . Are these are builds from master or from emacs-25?
  . Is this reproducible? i.e., if you checkout what was HEAD on
    Nov 5, do you get bootstrap time around the values recorded
    for that date?
  . Did anything change in Hydra system configuration since the
    beginning of the year?  I mean stuff like OS upgrades, changes in
    hardware (including disk drives), new versions of GCC and Binutils
    installed.
  . (Paul already asked this.) What are the configure-time and
    make-time options? are they identical in all these builds?

I tried byte compiling large files, and I don't see more than 10%
slowdown between some build near October end and a build a few days
ago.  I measured the same 10% in optimized and non-optimized builds
(of course, absolute times are different).  So this cannot explain a
30% increase in bootstrap time, at least not on my system.  Can you
try the same experiment on Hydra?

Re: builds are getting slower?

[Glenn Morris]

Eli Zaretskii wrote:

> Strange: the plot at Hydra doesn't seem to show any perceptible
> tendency towards longer times.  If I draw an imaginary line to show
> the long-term average, it stays at about the same value.

I think so too.

>   . Are these are builds from master or from emacs-25?

master (emacs-25 didn't exist for most of the year).

>   . Is this reproducible? i.e., if you checkout what was HEAD on
>     Nov 5, do you get bootstrap time around the values recorded
>     for that date?

I haven't checked.
That would be a first step before spending any time on this.

>   . Did anything change in Hydra system configuration since the
>     beginning of the year?  I mean stuff like OS upgrades, changes in
>     hardware (including disk drives), new versions of GCC and Binutils
>     installed.

Don't know about the hardware, but the software changes all the time.
It's all recorded in the "build dependencies" tab.

>   . (Paul already asked this.) What are the configure-time and
>     make-time options? are they identical in all these builds?

These are non-parallel debug (CFLAGS=-O0 -g3) builds on x86_64 RHEL 7.1,
using the Lucid toolkit and almost all of the optional libraries
(ImageMagick etc) that Emacs supports. The build script was unchanged.

These builds were not done under scientific conditions. Sometimes the
system might have been doing something else. It has been getting the
normal round of OS updates during the year (although these are not
large for an "enterprise" distribution). 

> Can you try the same experiment on Hydra?

No; I have no more access to Hydra than you do.
It isn't set up for that kind of investigation.
I suggest forgetting about Hydra for the purposes of this discussion.

I'm not really interested in doing the detective work on this myself.
I just presented the data I had in case it piqued anyone else's interest.

Re: builds are getting slower?

[Paul Eggert]

Glenn Morris wrote:
> I'm not really interested in doing the detective work on this myself.
> I just presented the data I had in case it piqued anyone else's interest.

In emacs-25 commit 09663d9b91803882508bd805581b95f8990eb441 I installed 
something that fixes the first performance glitch you observed (the smaller 
glitch). Whenever we get around to merging emacs-25 into master, that should fix 
things there too.

I couldn't reproduce the larger performance glitch, though. Is it possible that 
your build platform switched from Texinfo 4 to Texinfo 5 or 6? That could 
explain things, as the new 'makeinfo' is way slower.

Re: builds are getting slower?

[Glenn Morris]

Paul Eggert wrote:

> Is it possible that your build platform switched from Texinfo 4 to
> Texinfo 5 or 6? That could explain things, as the new 'makeinfo' is
> way slower.

No. (I'm a /usr/local/bin/makeinfo-4 holdout for this very reason.)
At some point I'll try to check if I can reproduce that November change
building before and after versions on the same day.

Re: builds are getting slower?

[Glenn Morris]

So I just checked. Times as measured today:

rev 536f6fc826b  (Dec 13th): 24m23.556s
rev 95a5c23f741  (Dec 12th): 25m39.348s
rev b9acb9502fd  (Nov 8th) : 21m33.631s

So for me the effect is real and not due to a system change.

Re: builds are getting slower?

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Sun, 13 Dec 2015 18:53:56 -0500
> Cc: emacs-devel@gnu.org
> 
> 
> So I just checked. Times as measured today:
> 
> rev 536f6fc826b  (Dec 13th): 24m23.556s
> rev 95a5c23f741  (Dec 12th): 25m39.348s
> rev b9acb9502fd  (Nov 8th) : 21m33.631s
> 
> So for me the effect is real and not due to a system change.

Thanks.  The next step, IMO, is to see which part of the bootstrap is
25% (or 4 min) slower.  Is that the part that byte-compiles Lisp
files, or is that some other part?

Re: builds are getting slower?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> So I just checked. Times as measured today:
>> 
>> rev 536f6fc826b  (Dec 13th): 24m23.556s

Which part of this is "make", and which "make check"?  Are you doing a
bootstrap build, or a "clean" build?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: builds are getting slower?

[Paul Eggert]

John Wiegley wrote:
> Which part of this is "make", and which "make check"?  Are you doing a
> bootstrap build, or a "clean" build?

When I reproduced the small part of the performance regression on Fedora x86-64 
(the part that's been fixed since then, on emacs-25), I did this:

./autogen.sh
./configure CFLAGS='-g3 -O0'
make -j5
time make bootstrap

So there's no 'make check', just a 'make bootstrap'.

'make check' is so slow that I normally don't run it these days. We should do 
what coreutils does, and run the expensive tests only if RUN_EXPENSIVE_TESTS=yes 
is in the environment.  coreutils has three categories of expensiveness, but 
Emacs could probably get by with two for now.

Re: builds are getting slower?

[John Wiegley]

>>>>> Paul Eggert <eggert@cs.ucla.edu> writes:

> 'make check' is so slow that I normally don't run it these days. We should
> do what coreutils does, and run the expensive tests only if
> RUN_EXPENSIVE_TESTS=yes is in the environment. coreutils has three
> categories of expensiveness, but Emacs could probably get by with two for
> now.

That sounds like a great idea.  I usually define three too: sanity tests,
regular tests, extensive tests.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: builds are getting slower?

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Paul Eggert <eggert@cs.ucla.edu> writes:
>
>> 'make check' is so slow that I normally don't run it these days. We should
>> do what coreutils does, and run the expensive tests only if
>> RUN_EXPENSIVE_TESTS=yes is in the environment. coreutils has three
>> categories of expensiveness, but Emacs could probably get by with two for
>> now.
>
> That sounds like a great idea.  I usually define three too: sanity tests,
> regular tests, extensive tests.

I'll would declare the following tests as extensive: autorevert-tests.el,
dbus-tests.el, filenotify-tests.el, tramp-tests.el.

All of them use events and timers, and they take their time. And all of
them written by me :-(

Another idea: those tests could check environment variable
RUN_EXPENSIVE_TESTS. If existing, they would run all tests, otherwise
they would run just a short version of them.

Best regards, Michael.

mark expensive tests (was: builds are getting slower?)

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Paul Eggert <eggert@cs.ucla.edu> writes:
>
>> 'make check' is so slow that I normally don't run it these days. We should
>> do what coreutils does, and run the expensive tests only if
>> RUN_EXPENSIVE_TESTS=yes is in the environment. coreutils has three
>> categories of expensiveness, but Emacs could probably get by with two for
>> now.
>
> That sounds like a great idea.  I usually define three too: sanity tests,
> regular tests, extensive tests.

Do we have an agreement how to handle this? There was no further
reaction for 3 weeks ...

Best regards, Michael.

Re: mark expensive tests

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

>>>>> Michael Albinus <michael.albinus@gmx.de> writes:

>> That sounds like a great idea.  I usually define three too: sanity tests,
>> regular tests, extensive tests.

> Do we have an agreement how to handle this? There was no further reaction
> for 3 weeks ...

Yes, we will have three test intensity levels:

  1. Sanity tests just make sure that the environment works, and the whole of
     them should run in <30 seconds on typical hardware. Some shops call these
     'smoke tests', as they are just there to make sure that Emacs can
     function in the most basic ways.

  2. Regular tests do not incur intensive CPU or memory costs, and so can be
     run on any hardware. These should finish on a scale of minutes, likely
     <10 minutes at the most.

  3. Extensive tests are given free reign, and may not even be able to
     complete on systems with CPU or memory constraints. These should feel
     free to take up to an hour, maybe even beyond.

  4. Selective tests are never run automatically, but exist to stress test
     some particular area of the system. These could take days, it doesn't
     really matter what their requirements are.

Each test should be marked, and the test runner should take these into account
and provide options for picking which grade of test (or lower) to be run. This
way I can tell Nix to build and run #2 tests. 'make check' should run #2,
'make fullcheck' should run #3, and there should be individual check targets
for the tests in #4.

I'd especially like the file-notify tests to move to #3, since these are what
consistently bog down my testing environment.

Michael, are these changes you would be interested in making?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: mark expensive tests

[Paul Eggert]

John Wiegley wrote:

>   1. Sanity tests just make sure that the environment works, and the whole of
>      them should run in <30 seconds on typical hardware. Some shops call these
>      'smoke tests'...
>   2. Regular tests ..., likely <10 minutes at the most.
>   3. Extensive tests ...
>   4. Selective tests ...

> ...  'make check' should run #2,

Although that all sounds fine, if 'make check' takes 10 minutes then I likely 
won't run it routinely. I'm just not that patient. I suggest an easy-to-remember 
name for #1, 'make smoke' say, which people like me can try to run more often.

Re: mark expensive tests

[John Wiegley]

>>>>> Paul Eggert <eggert@cs.ucla.edu> writes:

> Although that all sounds fine, if 'make check' takes 10 minutes then I
> likely won't run it routinely. I'm just not that patient. I suggest an
> easy-to-remember name for #1, 'make smoke' say, which people like me can try
> to run more often.

How about this:

    make check        sanity tests
    make morecheck    sanity tests + regular tests
    make fullcheck    sanity tests + regular tests + extensive tests

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: mark expensive tests

[Paul Eggert]

John Wiegley wrote:
> How about this:
>
>      make check        sanity tests
>      make morecheck    sanity tests + regular tests
>      make fullcheck    sanity tests + regular tests + extensive tests

Sure, that's fine.  Or we can do what coreutils does:

make check
make check-expensive
make check-very-expensive

Re: mark expensive tests

[John Wiegley]

>>>>> Paul Eggert <eggert@cs.ucla.edu> writes:

> Sure, that's fine.  Or we can do what coreutils does:

> make check
> make check-expensive
> make check-very-expensive

Prior art from an existing GNU project is even better. Now, who wants to make
it so?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: mark expensive tests

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > make check
  > > make check-expensive
  > > make check-very-expensive

  > Prior art from an existing GNU project is even better. Now, who wants to make
  > it so?

Would it be useful to add this to the GNU Coding Standards?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: mark expensive tests

[Paul Eggert]

Richard Stallman wrote:
>    > > make check
>    > > make check-expensive
>    > > make check-very-expensive
>
>    > Prior art from an existing GNU project is even better. Now, who wants to make
>    > it so?
>
> Would it be useful to add this to the GNU Coding Standards?

I expect so, yes.

Re: mark expensive tests

[Michael Albinus]

John Wiegley <jwiegley@gmail.com> writes:

>> make check
>> make check-expensive
>> make check-very-expensive
>
> Prior art from an existing GNU project is even better. Now, who wants to make
> it so?

I could start with it, in the emacs-25 branch. Those targets do cover #2,
#3 and #4 tests. Which make target shall we use for #1? "make check-smoke"?

Also, we need to decide how to distinguish such test categories. I don't
believe we shall do it on test file basis, except maybe for the smoke
tests. For the other tests, we could request something like this in the
test functions:

#3 tests: (skip-unless (or (getenv "RUN_EXPENSIVE_TESTS")
                           (getenv "RUN_VERY_EXPENSIVE_TESTS")))

#4 tests: (skip-unless (getenv "RUN_VERY_EXPENSIVE_TESTS"))

Alternatively, we could extend ert.el by a mechanism, marking tests
optionally as "smoke", "normal", "expensive", "very-expensive". "normal"
would be the default category. This shall be in accordance to the GNU
Coding Standards, as Richard proposes.

Best regards, Michael.

Re: mark expensive tests

[Paul Eggert]

Michael Albinus wrote:
> I could start with it, in the emacs-25 branch. Those targets do cover #2,
> #3 and #4 tests. Which make target shall we use for #1? "make check-smoke"?

I was thinking that we use just 3 categories for cost, so #1 would be 'make 
check'. If there are some special tests that don't fall under the expensiveness 
axis, we can invent a new name for them.

Re: mark expensive tests

[Michael Albinus]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Michael Albinus wrote:
>> I could start with it, in the emacs-25 branch. Those targets do cover #2,
>> #3 and #4 tests. Which make target shall we use for #1? "make check-smoke"?
>
> I was thinking that we use just 3 categories for cost, so #1 would be
> 'make check'. If there are some special tests that don't fall under
> the expensiveness axis, we can invent a new name for them.

Quoting John:

  Each test should be marked, and the test runner should take these into account
  and provide options for picking which grade of test (or lower) to be run. This
  way I can tell Nix to build and run #2 tests. 'make check' should run #2,
  'make fullcheck' should run #3, and there should be individual check targets
  for the tests in #4.

#1 are sanity (or smoke) tests, something which shall run <30 seconds,
and maybe recommended to run prior any commit (or push) to emacs repo.

Best regards, Michael.

Re: mark expensive tests

[Dmitry Gutov]

On 01/04/2016 10:16 AM, Michael Albinus wrote:

> #3 tests: (skip-unless (or (getenv "RUN_EXPENSIVE_TESTS")
>                             (getenv "RUN_VERY_EXPENSIVE_TESTS")))
>
> #4 tests: (skip-unless (getenv "RUN_VERY_EXPENSIVE_TESTS"))

ert has the notion of tags already. We should use them.

Re: mark expensive tests

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 01/04/2016 10:16 AM, Michael Albinus wrote:
>
>> #3 tests: (skip-unless (or (getenv "RUN_EXPENSIVE_TESTS")
>>                             (getenv "RUN_VERY_EXPENSIVE_TESTS")))
>>
>> #4 tests: (skip-unless (getenv "RUN_VERY_EXPENSIVE_TESTS"))
>
> ert has the notion of tags already. We should use them.

Ahh, yes. I was unaware of them (sometimes it's useful to check the doc :-)

So we might use the tags :sanity-tests, :expensive-tests and
:very-expensive-tests. No tag means regular tests.

I'm not sure whether we need the tag :smoke-tests, let's see.

Best regards, Michael.

Re: mark expensive tests

[Phillip Lord]

Paul Eggert <eggert@cs.ucla.edu> writes:

> John Wiegley wrote:
>> How about this:
>>
>>      make check        sanity tests
>>      make morecheck    sanity tests + regular tests
>>      make fullcheck    sanity tests + regular tests + extensive tests
>
> Sure, that's fine.  Or we can do what coreutils does:
>
> make check
> make check-expensive
> make check-very-expensive

It's rather inconsistent with "clean" which puts the qualifier first.

It's worth mentioning that I've already added

make check-maybe

to master, and it is consistent with that.

check-maybe is often much faster than make check. The heuristic it
uses for identify which tests should run will break, of course, but it's
probably quick enough to run on a pre-commit hook most of the time.

Phil

Re: mark expensive tests

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Thu, 07 Jan 2016 22:14:19 +0000
> Cc: emacs-devel@gnu.org
> 
> >>      make check        sanity tests
> >>      make morecheck    sanity tests + regular tests
> >>      make fullcheck    sanity tests + regular tests + extensive tests
> >
> > Sure, that's fine.  Or we can do what coreutils does:
> >
> > make check
> > make check-expensive
> > make check-very-expensive
> 
> It's rather inconsistent with "clean" which puts the qualifier first.
> 
> It's worth mentioning that I've already added
> 
> make check-maybe
> 
> to master, and it is consistent with that.
> 
> check-maybe is often much faster than make check. The heuristic it
> uses for identify which tests should run will break, of course, but it's
> probably quick enough to run on a pre-commit hook most of the time.

Whatever you guys agree to call those targets, please document them in
README.  They are currently semi-documented in Makefile.in, which is
not a good place for this stuff (the new targets don't seem to be
documented at all, btw).  Likewise with the way to run a test under
debugger, which is hinted upon somewhere in the comments to
Makefile.in (and also doesn't really work without tinkering, AFAIR).

We really need to have a slightly better documentation of the test
suite, IMO.  Rather minor efforts can bring large benefits in this
area.

TIA

Re: mark expensive tests

[Achim Gratz]

Eli Zaretskii writes:
> Whatever you guys agree to call those targets, please document them in
> README.  They are currently semi-documented in Makefile.in, which is
> not a good place for this stuff (the new targets don't seem to be
> documented at all, btw).

Org has "make help" and "make helpall" and it's all placed directly and
easily readable in the main Makefile (which then provides the actual
make machinery via includes).  Also a variable to select which tests to
run via a regex, which is handy when bisecting.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: mark expensive tests

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> Org has "make help" and "make helpall" and it's all placed directly and
> easily readable in the main Makefile (which then provides the actual make
> machinery via includes).

Except that no one would know to use those as targets (am I building help, or
getting help?). The README is a better place.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: mark expensive tests

[Achim Gratz]

John Wiegley writes:
>>>>>> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> Org has "make help" and "make helpall" and it's all placed directly and
>> easily readable in the main Makefile (which then provides the actual make
>> machinery via includes).
>
> Except that no one would know to use those as targets (am I building help, or
> getting help?). The README is a better place.

One doesn't preclude the other, of course.  The point was that Makefile
could be way more self-documenting if better documentation of its
targets was the goal.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: mark expensive tests

[Simon Michael]

On 1/9/16 4:09 AM, Achim Gratz wrote:
> John Wiegley writes:
>>>>>>> Achim Gratz <Stromeko@nexgo.de> writes:
>>> Org has "make help" and "make helpall" and it's all placed directly and
>>> easily readable in the main Makefile (which then provides the actual make
>>> machinery via includes).
>>
>> Except that no one would know to use those as targets (am I building help, or
>> getting help?). The README is a better place.
> 
> One doesn't preclude the other, of course.  The point was that Makefile
> could be way more self-documenting if better documentation of its
> targets was the goal.

FWIW,
http://www.cmcrossroads.com/print/article/self-documenting-makefiles is
a nice system.
https://github.com/simonmichael/hledger/blob/master/help-system.mk is a
tweaked version, used by
https://github.com/simonmichael/hledger/blob/master/Makefile:

$ make
Makefile:39: -------------------- hledger make rules --------------------
Makefile:41: make [help] -- list documented rules in this makefile. make
-n RULE shows more detail.
Makefile:217: (INSTALLING:)
Makefile:219: make install -- download dependencies and install hledger
executables to ~/.local/bin or equivalent (with stack)
Makefile:244: (BUILDING:)
Makefile:248: make build -- download dependencies and build hledger
executables (with stack)
etc.

Re: mark expensive tests

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> We really need to have a slightly better documentation of the test suite,
> IMO. Rather minor efforts can bring large benefits in this area.

+1

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: mark expensive tests

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Sun, 03 Jan 2016 13:08:50 -0800
> Cc: emacs-devel@gnu.org
> 
> How about this:
> 
>     make check        sanity tests
>     make morecheck    sanity tests + regular tests
>     make fullcheck    sanity tests + regular tests + extensive tests

Fine by me, but I think having sanity run up to, say, 4 or 5 min
(based on what we have now) should be okay.  IOW, we should have the
entire current test suite (with perhaps the single exception of remote
file-notify-tests) in the 1st category, IMO.

Re: mark expensive tests

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 3 Jan 2016 12:40:09 -0800
> 
> Although that all sounds fine, if 'make check' takes 10 minutes then I likely won't run it routinely.

How long do the tests take now on your machine(s)?  Do they really
take a lot of time?

Re: mark expensive tests

[Paul Eggert]

On 01/04/2016 07:48 AM, Eli Zaretskii wrote:
> How long do the tests take now on your machine(s)?  Do they really
> take a lot of time?

Yes, it takes 13 minutes with 'make', 11 minutes with 'make -j4'. This 
is on my roughly-6-year-old desktop, a 4-core AMD Phenom II X4 910e, 
Fedora 23 x86-64, a local ext4 file system mounted 
rw,relatime,seclabel,data=ordered on a hard disk. It's been this slow 
for ages (I've mentioned it here), and is why I typically don't bother 
running 'make check'.

I prefer a 'make check' (or whatever) that takes less than a minute. 
This is so that I can test the change I just installed in my private 
copy of Emacs, before publishing it. I don't want to test in the 
background because then I lose my train of thought. I prefer testing 
while writing the ChangeLog entry.

PS. I am getting due for a faster machine but not yet. I want an Intel 
Skylake so that I can verify that Emacs works with the new x86-64 
hardware-based subscript checking implemented via GCC 5's 
-fcheck-pointer-bounds option. I also want ECC memory for reliability's 
sake. This combination isn't available yet for desktops, at least not in 
configurations I can afford. All this being said, I expect that Emacs 
'make check' is slow for reasons unrelated to my 6-year-old CPU.

Re: mark expensive tests

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 4 Jan 2016 08:50:13 -0800
> 
> On 01/04/2016 07:48 AM, Eli Zaretskii wrote:
> > How long do the tests take now on your machine(s)?  Do they really
> > take a lot of time?
> 
> Yes, it takes 13 minutes with 'make', 11 minutes with 'make -j4'.

Is this on master or on emacs-25?  The former is much slower.  (I was
timing emacs-25.)

Re: mark expensive tests

[Paul Eggert]

On 01/04/2016 09:18 AM, Eli Zaretskii wrote:
>> it takes 13 minutes with 'make', 11 minutes with 'make -j4'.
> Is this on master or on emacs-25?  The former is much slower.  (I was
> timing emacs-25.)
>

Ah, sorry, my times were for master. For emacs-25, 'make check' takes 
2.7 minutes, 'make -j4 check' takes 1.5 minutes. These times assume the 
test .el files are already built. For me this is merely annoyingly slow, 
not "let's skip the tests" slow like on master.

Re: mark expensive tests

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Sun, 03 Jan 2016 12:09:12 -0800
> Cc: emacs-devel@gnu.org
> 
> Yes, we will have three test intensity levels:
> 
>   1. Sanity tests just make sure that the environment works, and the whole of
>      them should run in <30 seconds on typical hardware. Some shops call these
>      'smoke tests', as they are just there to make sure that Emacs can
>      function in the most basic ways.
> 
>   2. Regular tests do not incur intensive CPU or memory costs, and so can be
>      run on any hardware. These should finish on a scale of minutes, likely
>      <10 minutes at the most.
> 
>   3. Extensive tests are given free reign, and may not even be able to
>      complete on systems with CPU or memory constraints. These should feel
>      free to take up to an hour, maybe even beyond.
> 
>   4. Selective tests are never run automatically, but exist to stress test
>      some particular area of the system. These could take days, it doesn't
>      really matter what their requirements are.

I'm fine with this gradation, but here's some reality check:

  . The current test suite takes 3.5 min for a full run, including
    compilation of all the *.el files, 2.25 min if the *.el files are
    already compiled

  . For comparison, the test suite of the latest version of Texinfo I
    just built takes 4.5 min on the same machine, even though a lot of
    tests are skipped (due to some required infrastructure not being
    installed)

So it sounds like we have no candidates for #3 and #4, and the
division between #1 and #2 is questionable, since 2 min is not such a
long time to wait, IMO.

> I'd especially like the file-notify tests to move to #3, since these are what
> consistently bog down my testing environment.

file-notify-tests takes about 25 sec here, so I'm unsure why it should
be in #3.  Maybe that's because of remote tests (which are skipped on
my system), in which case perhaps the remote tests should be separated
into a separate test file and run on demand only.

Re: mark expensive tests

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> I'd especially like the file-notify tests to move to #3, since these are what
>> consistently bog down my testing environment.
>
> file-notify-tests takes about 25 sec here, so I'm unsure why it should
> be in #3.  Maybe that's because of remote tests (which are skipped on
> my system), in which case perhaps the remote tests should be separated
> into a separate test file and run on demand only.

The most time consuming test is file-notify-test06-many-events. It takes
more than 4 minutes on my laptop:

Started at:   2016-01-04 17:08:45+0100
Finished.
Finished at:  2016-01-04 17:13:05+0100

I'm impressed that your supercomputer is that fast. Or do you test under
cygwin? Then this test is skipped.

Remote tests in filenotify-tests.el could go into #3, indeed. Like some
of the tests of tramp-test.el.

Best regards, Michael.

Re: mark expensive tests

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: John Wiegley <jwiegley@gmail.com>,  emacs-devel@gnu.org
> Date: Mon, 04 Jan 2016 17:19:01 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I'd especially like the file-notify tests to move to #3, since these are what
> >> consistently bog down my testing environment.
> >
> > file-notify-tests takes about 25 sec here, so I'm unsure why it should
> > be in #3.  Maybe that's because of remote tests (which are skipped on
> > my system), in which case perhaps the remote tests should be separated
> > into a separate test file and run on demand only.
> 
> The most time consuming test is file-notify-test06-many-events. It takes
> more than 4 minutes on my laptop:
> 
> Started at:   2016-01-04 17:08:45+0100
> Finished.
> Finished at:  2016-01-04 17:13:05+0100
> 
> I'm impressed that your supercomputer is that fast. Or do you test under
> cygwin? Then this test is skipped.

You are talking about the master branch.  I was talking about
emacs-25, where that test doesn't exist.

So this long test should also go into #3.

My main point was that other than a couple of very specific tests, the
rest of the suite is quite fast, as test suites go.

Re: mark expensive tests

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> My main point was that other than a couple of very specific tests, the
> rest of the suite is quite fast, as test suites go.

If we want to recommend to run the smoke tests at least before
committing, then 2 minutes might be long enough to let people refuse
that. That's why John has proposed a 30 seconds upper limit for the smoke
tests.

Anyway, we can introduce smoke tests and very expansive tests later as
needed. As minimum, we shall introduce and document the :expansive-test
ert tag. And I will add it to some Tramp tests and also to
file-notify-test06-many-events on master.

OK?

Best regards, Michael.

Re: mark expensive tests

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: jwiegley@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 04 Jan 2016 18:01:50 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > My main point was that other than a couple of very specific tests, the
> > rest of the suite is quite fast, as test suites go.
> 
> If we want to recommend to run the smoke tests at least before
> committing, then 2 minutes might be long enough to let people refuse
> that. That's why John has proposed a 30 seconds upper limit for the smoke
> tests.

I'm afraid that arbitrarily selecting 25% of the current test suite
will leave too of the functionality much untested.  Emacs is a
hodge-podge of mostly unrelated functionalities, so I think it won't
be easy to come up with a meaningful sanity test.

But maybe I'm just missing something.  Can you tell how will you
select the 30-sec worth of tests?

Thanks.

Re: mark expensive tests

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> If we want to recommend to run the smoke tests at least before
>> committing, then 2 minutes might be long enough to let people refuse
>> that. That's why John has proposed a 30 seconds upper limit for the smoke
>> tests.
>
> I'm afraid that arbitrarily selecting 25% of the current test suite
> will leave too of the functionality much untested.  Emacs is a
> hodge-podge of mostly unrelated functionalities, so I think it won't
> be easy to come up with a meaningful sanity test.
>
> But maybe I'm just missing something.  Can you tell how will you
> select the 30-sec worth of tests?

Not yet. As said, I would start to puzzle out the expensive
tests. Reducing make check from 10+ minutes to 2 minutes seems to be a
good start. Whether we need to define smoke tests I don't know.

> Thanks.

Best regards, Michael.

Re: mark expensive tests

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

> Anyway, we can introduce smoke tests and very expansive tests later as
> needed. As minimum, we shall introduce and document the :expansive-test
> ert tag. And I will add it to some Tramp tests and also to
> file-notify-test06-many-events on master.

I gave it a try in the emacs-25 branch. The following tests have been
tagged :expensive-test:

auto-revert-test01-auto-revert-several-files
file-notify-*-remote
tramp-test26-process-file
tramp-test27-start-file-process
tramp-test28-shell-command
tramp-test29-vc-registered
tramp-test31-special-characters-with-*
tramp-test32-utf8-with-*
tramp-test33-asynchronous-requests
tramp-test35-unload

"make check" needs now 1:39 min.

--8<---------------cut here---------------start------------->8---
SUMMARY OF TEST RESULTS
-----------------------
Files examined: 92
Ran 1635 tests, 1634 results as expected, 1 skipped
make[2]: Leaving directory '/usr/local/src/emacs-25/test/automated'
make[1]: Leaving directory '/usr/local/src/emacs-25/test/automated'
32.796u 4.836s 1:38.51 38.1%	0+0k 8+21192io 0pf+0w
--8<---------------cut here---------------end--------------->8---

"make check-expensive needs" 2:34 min.

--8<---------------cut here---------------start------------->8---
SUMMARY OF TEST RESULTS
-----------------------
Files examined: 92
Ran 1654 tests, 1653 results as expected, 1 skipped
make[3]: Leaving directory '/usr/local/src/emacs-25/test/automated'
make[2]: Leaving directory '/usr/local/src/emacs-25/test/automated'
make[1]: Leaving directory '/usr/local/src/emacs-25/test/automated'
48.312u 6.252s 2:33.95 35.4%	0+0k 0+34176io 0pf+0w
--8<---------------cut here---------------end--------------->8---

About one minute gain already. In the master branch, the differences
will be more noteworthy.

Best regards, Michael.

Re: builds are getting slower?

[Phillip Lord]

Paul Eggert <eggert@cs.ucla.edu> writes:

> John Wiegley wrote:
>> Which part of this is "make", and which "make check"?  Are you doing a
>> bootstrap build, or a "clean" build?
>
> When I reproduced the small part of the performance regression on Fedora
> x86-64 (the part that's been fixed since then, on emacs-25), I did this:
>
> ./autogen.sh
> ./configure CFLAGS='-g3 -O0'
> make -j5
> time make bootstrap
>
> So there's no 'make check', just a 'make bootstrap'.
>
> 'make check' is so slow that I normally don't run it these days. We should do
> what coreutils does, and run the expensive tests only if
> RUN_EXPENSIVE_TESTS=yes is in the environment.  coreutils has three categories
> of expensiveness, but Emacs could probably get by with two for now.


When you get to master, make check-maybe should be pretty quick
(although not from bootstrap) now.

In the ideal world, it would be nice to include this in the pre-commit
hook, although that might be impractical in the short term.

Phil

Re: builds are getting slower?

[Glenn Morris]

On investigation, I find 2e84888 caused a 20% increase in build time.

Re: builds are getting slower?

[Artur Malabarba]

2015-12-15 8:18 GMT+00:00 Glenn Morris <rgm@gnu.org>:
>
> On investigation, I find 2e84888 caused a 20% increase in build time.
>

Looks like that's my commit. I'm attaching the diff of this commit
below so that people can have a look and help pinpoint something.
I only see 2 things that could be to blame:

1. dir-locals-find-file changed quite a bit. I suspect it is called
once for each .el file we compile? In this situation, the local-vars
cache should kick in and avoid the slowness, but maybe there's a bug
where the cache is not being used.
2. There's a new (require 'map) inside one of the functions and a
(require 'seq) inside another. (I put these inside the functions so
that they would only be loaded when necessary, but if it's getting
loaded at boot, perhaps these should just go at the top level). Anyway
(require 'seq) could be a bit heavy because it uses cl-generic and
cl-extra, but I'd be really surprised if this amounted to 20% of the
build time.

--------------------------------------------------------------------------------
@ 2e8488858c7b8df40610c1cd3038348fdc9bf3ed
Author:     Artur Malabarba <bruce.connor.am@gmail.com>
AuthorDate: Sat Nov 7 13:54:56 2015 +0000
Commit:     Artur Malabarba <bruce.connor.am@gmail.com>
CommitDate: Tue Nov 10 13:04:31 2015 +0000

Parent:     cbaa040 * lisp/emacs-lisp/map.el (map-merge-with): New function
Merged:     (HEAD detached at 2e84888) debugging-package
Containing: (HEAD detached at 2e84888) emacs-25 master scratch/rest.el
Follows:    emacs-24.5-rc3-fixed (6527)

* lisp/files.el (dir-locals-file): Allow wildcards

(dir-locals-find-file, dir-locals-collect-variables)
(dir-locals-read-from-file): Update accordingly.
(hack-dir-local-variables): Rename a local variable.

* lisp/files-x.el (modify-dir-local-variable): Update accordingly

* lisp/help-fns.el (describe-variable): Update accordingly

* .gitignore: Add .dir-locals?.el

4 files changed, 154 insertions(+), 91 deletions(-)
.gitignore       |   1 +
lisp/files-x.el  |  29 ++++++----
lisp/files.el    | 172 +++++++++++++++++++++++++++++++++++--------------------
lisp/help-fns.el |  43 ++++++++------

modified   .gitignore
@@ -255,6 +255,7 @@ gnustmp*
 ChangeLog
 [0-9]*.patch
 [0-9]*.txt
+.dir-locals?.el
 /vc-dwim-log-*

 # Built by 'make install'.
modified   lisp/files-x.el
@@ -429,18 +429,25 @@ modify-dir-local-variable
   (catch 'exit
     (unless enable-local-variables
       (throw 'exit (message "Directory-local variables are disabled")))
-    (let ((variables-file (or (and (buffer-file-name)
-   (not (file-remote-p (buffer-file-name)))
-   (dir-locals-find-file (buffer-file-name)))
-      dir-locals-file))
+    (let ((variables-file (and (buffer-file-name)
+                               (not (file-remote-p (buffer-file-name)))
+                               (dir-locals-find-file (buffer-file-name))))
   variables)
-      (if (consp variables-file) ; result from cache
-  ;; If cache element has an mtime, assume it came from a file.
-  ;; Otherwise, assume it was set directly.
-  (setq variables-file (if (nth 2 variables-file)
-   (expand-file-name dir-locals-file
-     (car variables-file))
- (cadr variables-file))))
+      (setq variables-file
+            ;; If there are several .dir-locals, the user probably
+            ;; wants to edit the last one (the highest priority).
+            (cond ((stringp variables-file)
+                   (car (last (dir-locals--all-files variables-file))))
+                  ((consp variables-file) ; result from cache
+                   ;; If cache element has an mtime, assume it came
from a file.
+                   ;; Otherwise, assume it was set directly.
+                   (if (nth 2 variables-file)
+                       (car (last (dir-locals--all-files (car
variables-file))))
+                     (cadr variables-file)))
+                  ;; Try to make a proper file-name.  This doesn't cover all
+                  ;; wildcards, but it covers the default value of
`dir-locals-file'.
+                  (t (replace-regexp-in-string
+                      "\\*" "" (replace-regexp-in-string  "\\?" "-"
dir-locals-file)))))
       ;; I can't be bothered to handle this case right now.
       ;; Dir locals were set directly from a class.  You need to
       ;; directly modify the class in dir-locals-class-alist.
modified   lisp/files.el
@@ -3648,7 +3648,7 @@ dir-locals-collect-variables
       (error
        ;; The file's content might be invalid (e.g. have a merge conflict), but
        ;; that shouldn't prevent the user from opening the file.
-       (message ".dir-locals error: %s" (error-message-string err))
+       (message "%s error: %s" dir-locals-file (error-message-string err))
        nil))))

 (defun dir-locals-set-directory-class (directory class &optional mtime)
@@ -3698,11 +3698,38 @@ dir-locals-set-class-variables
   applied by recursively following these rules."
   (setf (alist-get class dir-locals-class-alist) variables))

-(defconst dir-locals-file ".dir-locals.el"
+(defconst dir-locals-file ".dir-locals*.el"
   "File that contains directory-local variables.
 It has to be constant to enforce uniform values
 across different environments and users.")

+(defcustom dir-locals-sort-predicate #'string<
+  "Predicate used to sort dir-locals files before loading them.
+The function should take two arguments (file names) and return
+non-nil if the first argument should be loaded first (which means
+the values in the second file will override those in the first)."
+  :group 'files
+  :type 'function)
+
+(defun dir-locals--all-files (file-or-dir)
+  "Return a list of all readable dir-locals files matching FILE-OR-DIR.
+If FILE-OR-DIR is a file pattern, expand wildcards in it and
+return a sorted list of the results.  If it is a directory name,
+return a sorted list of all files matching `dir-locals-file' in
+this directory."
+  (require 'seq)
+  (let ((default-directory (if (file-directory-p file-or-dir)
+                               file-or-dir
+                             default-directory)))
+    (sort (seq-filter (lambda (f) (and (file-readable-p f)
+                                  (file-regular-p f)))
+                      (file-expand-wildcards
+                       (cond ((not (file-directory-p file-or-dir)) file-or-dir)
+                             ((eq system-type 'ms-dos)
(dosified-file-name dir-locals-file))
+                             (t dir-locals-file))
+                       'full))
+          dir-locals-sort-predicate)))
+
 (defun dir-locals-find-file (file)
   "Find the directory-local variables for FILE.
 This searches upward in the directory tree from FILE.
@@ -3719,75 +3746,96 @@ dir-locals-find-file
 This function returns either nil (no directory local variables found),
 or the matching entry from `dir-locals-directory-cache' (a list),
 or the full path to the `dir-locals-file' (a string) in the case
-of no valid cache entry."
+of no valid cache entry.  If `dir-locals-file' contains
+wildcards, then the return value is not a proper filename, it is
+an absolute version of `dir-locals-file' which is guaranteed to
+expand to at least one file."
   (setq file (expand-file-name file))
-  (let* ((dir-locals-file-name
-  (if (eq system-type 'ms-dos)
-      (dosified-file-name dir-locals-file)
-    dir-locals-file))
- (locals-file (locate-dominating-file file dir-locals-file-name))
- (dir-elt nil))
+  (let* ((dir-locals-file-name (if (eq system-type 'ms-dos)
+                                   (dosified-file-name dir-locals-file)
+                                 dir-locals-file))
+         (locals-dir (locate-dominating-file
+                      (file-name-directory file)
+                      (lambda (dir)
+                        (let ((default-directory dir))
+                          (file-expand-wildcards dir-locals-file-name
'full)))))
+         locals-file dir-elt)
     ;; `locate-dominating-file' may have abbreviated the name.
-    (and locals-file
- (setq locals-file (expand-file-name dir-locals-file-name locals-file)))
- ;; Let dir-locals-read-from-file inform us via demoted-errors
- ;; about unreadable files, etc.
- ;; Maybe we'd want to keep searching though - that is
- ;; a locate-dominating-file issue.
+    (when locals-dir
+      (setq locals-dir (expand-file-name locals-dir))
+      (setq locals-file (expand-file-name dir-locals-file-name locals-dir)))
+    ;; Let dir-locals-read-from-file inform us via demoted-errors
+    ;; about unreadable files, etc.
+    ;; Maybe we'd want to keep searching though - that is
+    ;; a locate-dominating-file issue.
 ;;; (or (not (file-readable-p locals-file))
 ;;;     (not (file-regular-p locals-file)))
 ;;; (setq locals-file nil))
     ;; Find the best cached value in `dir-locals-directory-cache'.
     (dolist (elt dir-locals-directory-cache)
       (when (and (string-prefix-p (car elt) file
-  (memq system-type
- '(windows-nt cygwin ms-dos)))
- (> (length (car elt)) (length (car dir-elt))))
- (setq dir-elt elt)))
+                                  (memq system-type
+                                        '(windows-nt cygwin ms-dos)))
+                 (> (length (car elt)) (length (car dir-elt))))
+        (setq dir-elt elt)))
     (if (and dir-elt
-     (or (null locals-file)
- (<= (length (file-name-directory locals-file))
-     (length (car dir-elt)))))
- ;; Found a potential cache entry.  Check validity.
- ;; A cache entry with no MTIME is assumed to always be valid
- ;; (ie, set directly, not from a dir-locals file).
- ;; Note, we don't bother to check that there is a matching class
- ;; element in dir-locals-class-alist, since that's done by
- ;; dir-locals-set-directory-class.
- (if (or (null (nth 2 dir-elt))
- (let ((cached-file (expand-file-name dir-locals-file-name
-     (car dir-elt))))
-  (and (file-readable-p cached-file)
-       (equal (nth 2 dir-elt)
-      (nth 5 (file-attributes cached-file))))))
-    ;; This cache entry is OK.
-    dir-elt
-  ;; This cache entry is invalid; clear it.
-  (setq dir-locals-directory-cache
- (delq dir-elt dir-locals-directory-cache))
-  ;; Return the first existing dir-locals file.  Might be the same
-  ;; as dir-elt's, might not (eg latter might have been deleted).
-  locals-file)
+             (or (null locals-dir)
+                 (<= (length locals-dir)
+                     (length (car dir-elt)))))
+        ;; Found a potential cache entry.  Check validity.
+        ;; A cache entry with no MTIME is assumed to always be valid
+        ;; (ie, set directly, not from a dir-locals file).
+        ;; Note, we don't bother to check that there is a matching class
+        ;; element in dir-locals-class-alist, since that's done by
+        ;; dir-locals-set-directory-class.
+        (if (or (null (nth 2 dir-elt))
+                (let ((cached-files (dir-locals--all-files (car dir-elt))))
+                  ;; The entry MTIME should match the most recent
+                  ;; MTIME among matching files.
+                  (and cached-files
+                       (= (time-to-seconds (nth 2 dir-elt))
+                          (apply #'max (mapcar (lambda (f)
(time-to-seconds (nth 5 (file-attributes f))))
+                                               cached-files))))))
+            ;; This cache entry is OK.
+            dir-elt
+          ;; This cache entry is invalid; clear it.
+          (setq dir-locals-directory-cache
+                (delq dir-elt dir-locals-directory-cache))
+          ;; Return the first existing dir-locals file.  Might be the same
+          ;; as dir-elt's, might not (eg latter might have been deleted).
+          locals-file)
       ;; No cache entry.
       locals-file)))

 (defun dir-locals-read-from-file (file)
   "Load a variables FILE and register a new class and instance.
-FILE is the name of the file holding the variables to apply.
+FILE is the absolute name of the file holding the variables to
+apply.  It may contain wildcards.
 The new class name is the same as the directory in which FILE
 is found.  Returns the new class name."
-  (with-temp-buffer
+  (require 'map)
+  (let* ((dir-name (file-name-directory file))
+         (class-name (intern dir-name))
+         (files (dir-locals--all-files file))
+         (read-circle nil)
+         (variables))
     (with-demoted-errors "Error reading dir-locals: %S"
-      (insert-file-contents file)
-      (unless (zerop (buffer-size))
-        (let* ((dir-name (file-name-directory file))
-               (class-name (intern dir-name))
-               (variables (let ((read-circle nil))
-                            (read (current-buffer)))))
-          (dir-locals-set-class-variables class-name variables)
-          (dir-locals-set-directory-class dir-name class-name
-                                          (nth 5 (file-attributes file)))
-          class-name)))))
+      (dolist (file files)
+        (with-temp-buffer
+          (insert-file-contents file)
+          (condition-case-unless-debug nil
+              (setq variables
+                    (map-merge-with 'list (lambda (a b) (map-merge 'list a b))
+                                    variables
+                                    (read (current-buffer))))
+            (end-of-file nil)))))
+    (dir-locals-set-class-variables class-name variables)
+    (dir-locals-set-directory-class
+     dir-name class-name
+     (seconds-to-time (apply #'max (mapcar (lambda (file)
+                                             (time-to-seconds (nth 5
(file-attributes file))))
+                                           files))))
+    class-name))

 (defcustom enable-remote-dir-locals nil
   "Non-nil means dir-local variables will be applied to remote files."
@@ -3810,17 +3858,17 @@ hack-dir-local-variables
  (not (file-remote-p (or (buffer-file-name)
  default-directory)))))
     ;; Find the variables file.
-    (let ((variables-file (dir-locals-find-file
-                           (or (buffer-file-name) default-directory)))
+    (let ((file-pattern-or-cache (dir-locals-find-file
+                                  (or (buffer-file-name) default-directory)))
   (class nil)
   (dir-name nil))
       (cond
-       ((stringp variables-file)
- (setq dir-name (file-name-directory variables-file)
-      class (dir-locals-read-from-file variables-file)))
-       ((consp variables-file)
- (setq dir-name (nth 0 variables-file))
- (setq class (nth 1 variables-file))))
+       ((stringp file-pattern-or-cache)
+ (setq dir-name (file-name-directory file-pattern-or-cache)
+      class (dir-locals-read-from-file file-pattern-or-cache)))
+       ((consp file-pattern-or-cache)
+ (setq dir-name (nth 0 file-pattern-or-cache))
+ (setq class (nth 1 file-pattern-or-cache))))
       (when class
  (let ((variables
        (dir-locals-collect-variables
modified   lisp/help-fns.el
@@ -907,29 +907,36 @@ describe-variable
                                             (buffer-file-name buffer)))
                                       (dir-locals-find-file
                                        (buffer-file-name buffer))))
-                          (dir-file t))
+                          (is-directory nil))
       (princ (substitute-command-keys
       "  This variable's value is directory-local"))
-      (if (null file)
-  (princ ".\n")
- (princ ", set ")
-                        (if (consp file) ; result from cache
-                            ;; If the cache element has an mtime, we
-                            ;; assume it came from a file.
-                            (if (nth 2 file)
-                                (setq file (expand-file-name
-                                            dir-locals-file (car file)))
-                              ;; Otherwise, assume it was set directly.
-                              (setq file (car file)
-                                    dir-file nil)))
- (princ (substitute-command-keys
-                                (if dir-file
-                                    "by the file\n  `"
-                                  "for the directory\n  `")))
+                      (when (consp file) ; result from cache
+                        ;; If the cache element has an mtime, we
+                        ;; assume it came from a file.
+                        (if (nth 2 file)
+                            (setq file (expand-file-name
+                                        dir-locals-file (car file)))
+                          ;; Otherwise, assume it was set directly.
+                          (setq file (car file)
+                                is-directory t)))
+                      (if (null file)
+                          (princ ".\n")
+                        (princ ", set ")
+                        (let ((files (file-expand-wildcards file)))
+                          (princ (substitute-command-keys
+                                  (cond
+                                   (is-directory "for the directory\n  `")
+                                   ;; Many files matched.
+                                   ((cdr files)
+                                    (setq file (file-name-directory
(car files)))
+                                    (format "by a file\n  matching
`%s' in the directory\n  `"
+                                            dir-locals-file))
+                                   (t (setq file (car files))
+                                      "by the file\n  `"))))
  (with-current-buffer standard-output
   (insert-text-button
    file 'type 'help-dir-local-var-def
-   'help-args (list variable file)))
+                             'help-args (list variable file))))
  (princ (substitute-command-keys "'.\n"))))
   (princ (substitute-command-keys
   "  This variable's value is file-local.\n"))))

Re: builds are getting slower?

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-12-15 8:18 GMT+00:00 Glenn Morris <rgm@gnu.org>:
>>
>> On investigation, I find 2e84888 caused a 20% increase in build time.
>>
>
> Looks like that's my commit. I'm attaching the diff of this commit
> below so that people can have a look and help pinpoint something.
> I only see 2 things that could be to blame:

This is a very, very large commit.  It should have been split into
multiple commits addressing separate issues.  However, in the
description one thing stands out:

> * lisp/files.el (dir-locals-file): Allow wildcards

Depending on how the wildcard search is organized, this may make the
decisive difference.

Can people split the timing tests into user and system time?  Because if
the directory search is at fault, it may be the system time that hits
here.

-- 
David Kastrup

Re: builds are getting slower?

[Eli Zaretskii]

> Date: Tue, 15 Dec 2015 12:38:07 +0000
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>, emacs-devel <emacs-devel@gnu.org>
> 
> > On investigation, I find 2e84888 caused a 20% increase in build time.
> >
> 
> Looks like that's my commit. I'm attaching the diff of this commit
> below so that people can have a look and help pinpoint something.
> I only see 2 things that could be to blame:
> 
> 1. dir-locals-find-file changed quite a bit. I suspect it is called
> once for each .el file we compile? In this situation, the local-vars
> cache should kick in and avoid the slowness, but maybe there's a bug
> where the cache is not being used.
> 2. There's a new (require 'map) inside one of the functions and a
> (require 'seq) inside another. (I put these inside the functions so
> that they would only be loaded when necessary, but if it's getting
> loaded at boot, perhaps these should just go at the top level). Anyway
> (require 'seq) could be a bit heavy because it uses cl-generic and
> cl-extra, but I'd be really surprised if this amounted to 20% of the
> build time.

Maybe I'm reading the code wrongly, but it looks like even before the
cache could kick in, dir-locals-find-file calls
locate-dominating-file, passing it dir-locals--all-files as its
predicate.  This has 2 effects: (a) it loads "seq", and (b) it calls
file-expand-wildcards, which needs to read the entire directory and
then match each file against a regexp.

Could these be the cause of slowdown?

Re: builds are getting slower?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On 15 Dec 2015 4:15 pm, "Eli Zaretskii" <eliz@gnu.org> wrote:
>
> Maybe I'm reading the code wrongly, but it looks like even before the
> cache could kick in, dir-locals-find-file calls
> locate-dominating-file, passing it dir-locals--all-files as its
> predicate.  This has 2 effects: (a) it loads "seq", and (b) it calls
> file-expand-wildcards, which needs to read the entire directory and
> then match each file against a regexp.
>
> Could these be the cause of slowdown?

Probably. And no, You're not reading it wrong.
We need to locate the dir locals file(s) in order to compare its timestamp
to the cache. Do we have a faster way of expanding wildcards?

Maybe it would be faster to not allow wildcards at all, and just have a
fixed regexp that we use in directory-files.

If that doesn't help, then we could also forgo of the regexp, and just have
two allowed filenames (like .dir-locals.el and .dir-locals-2.el).

[-- Attachment #2: Type: text/html, Size: 1143 bytes --]

Re: builds are getting slower?

[Eli Zaretskii]

> Date: Wed, 16 Dec 2015 11:50:11 +0000
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: emacs-devel <emacs-devel@gnu.org>, Glenn Morris <rgm@gnu.org>, eggert@cs.ucla.edu
> 
> We need to locate the dir locals file(s) in order to compare its timestamp to
> the cache. Do we have a faster way of expanding wildcards? 

Does it have to be wildcards?  Will file-name-all-completions not do?

And why exactly this code needs to run when we need to byte-compile
Lisp files, btw?

Also, since we know only very well which dir-local files are present
in the Emacs tree, we could make some trick (like binding a special
variable) for this case.  After all, 20% is only clearly visible when
the total build time is long, and that doesn't happen a lot outside of
bootstrapping Emacs.

In any case, I suggest to profile this code, before we decide that
this is indeed the culprit.  It could be something utterly unrelated.

Re: builds are getting slower?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On 16 Dec 2015 11:58 am, "Eli Zaretskii" <eliz@gnu.org> wrote:
> > We need to locate the dir locals file(s) in order to compare its
timestamp to
> > the cache. Do we have a faster way of expanding wildcards?
>
> Does it have to be wildcards?  Will file-name-all-completions not do?

Yes, I think we can do that.

> And why exactly this code needs to run when we need to byte-compile
> Lisp files, btw?

I suspect it's because the byte-compiler is affected by local variables, so
it needs to know them. We could certainly add some shortcut or let-bind a
variable for the purpose of slipping this during emacs bootstrap.

> In any case, I suggest to profile this code, before we decide that
> this is indeed the culprit.  It could be something utterly unrelated.

Indeed. I'll be glad to do it myself once things settle down here (which
might be around January).

[-- Attachment #2: Type: text/html, Size: 1087 bytes --]

Re: builds are getting slower?

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On 16 Dec 2015 11:58 am, "Eli Zaretskii" <eliz@gnu.org> wrote:
>> > We need to locate the dir locals file(s) in order to compare its
> timestamp to
>> > the cache. Do we have a faster way of expanding wildcards?
>>
>> Does it have to be wildcards?  Will file-name-all-completions not do?
>
> Yes, I think we can do that.
>
>> And why exactly this code needs to run when we need to byte-compile
>> Lisp files, btw?
>
> I suspect it's because the byte-compiler is affected by local variables, so
> it needs to know them.

Not really.  It loads the files, and loading the files sets the
directory-local variables.  I don't think it is a good idea to use them
for byte compilation even though I have to admit that a directory-wide
setting of `lexical-binding' would seem less ugly than putting it in
every file manually.  But it sounds like a recipe for trouble.

-- 
David Kastrup

Re: builds are getting slower?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Glenn Morris <rgm@gnu.org>,  eggert@cs.ucla.edu,  emacs-devel <emacs-devel@gnu.org>
> Date: Wed, 16 Dec 2015 14:25:56 +0100
> 
> >> And why exactly this code needs to run when we need to byte-compile
> >> Lisp files, btw?
> >
> > I suspect it's because the byte-compiler is affected by local variables, so
> > it needs to know them.
> 
> Not really.  It loads the files, and loading the files sets the
> directory-local variables.  I don't think it is a good idea to use them
> for byte compilation even though I have to admit that a directory-wide
> setting of `lexical-binding' would seem less ugly than putting it in
> every file manually.  But it sounds like a recipe for trouble.

We can disable that.

Re: builds are getting slower?

[Glenn Morris]

>> >> And why exactly this code needs to run when we need to byte-compile
>> >> Lisp files, btw?

It's run whenever Emacs visits a file, thus making code efficiency very
important.

>> Not really.  It loads the files, and loading the files sets the
>> directory-local variables.  I don't think it is a good idea to use them
>> for byte compilation even though I have to admit that a directory-wide
>> setting of `lexical-binding' would seem less ugly than putting it in
>> every file manually.  But it sounds like a recipe for trouble.
>
> We can disable that.

IIRC, Stefan was against this kind of thing; ie, making tweaks to the
build process to work around an issue that would otherwise be present
in normal Emacs usage. The feeling was that we should fix issues
properly, for everyone, rather than hiding them. The context in which
it came up before was VC, which, like dir-locals, is another operation
that runs every time Emacs visits a file, so also needs careful
treatment. I think I agree with him more now then I did back then.

(Outside of that general principle, I can actually see dir-locals
possibly being wanted during byte-compilation in future.)

Irrespective of the details, personally I'd like to see this code
rewritten to avoid requiring seq (which it does whenever Emacs visits a
file) and map (which it does whenever a dir-locals is present); ie avoid
non-preloaded libraries.

But you are right that actually profiling things is the place to start.

Re: builds are getting slower?

[John Wiegley]

>>>>> Glenn Morris <rgm@gnu.org> writes:

> IIRC, Stefan was against this kind of thing; ie, making tweaks to the build
> process to work around an issue that would otherwise be present in normal
> Emacs usage. The feeling was that we should fix issues properly, for
> everyone, rather than hiding them. The context in which it came up before
> was VC, which, like dir-locals, is another operation that runs every time
> Emacs visits a file, so also needs careful treatment. I think I agree with
> him more now then I did back then.

I also agree.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: builds are getting slower?

[Artur Malabarba]

Eli Zaretskii <eliz@gnu.org> writes:

> Does it have to be wildcards?  Will file-name-all-completions not do?

Ok. I've pushed a change using completion instead of wildcards (the
documentation still needs to be updated).

Could someone check whether build speeds have improved?

Re: builds are getting slower?

[Glenn Morris]

Artur Malabarba wrote:

> Ok. I've pushed a change using completion instead of wildcards (the
> documentation still needs to be updated).
>
> Could someone check whether build speeds have improved?

Somewhat faster, but still much slower than it was when there could be
only one dir-locals. Rough, one-off timings: wildcards was ~ 22% slower,
completion seems to be ~ 18% slower.

Re: builds are getting slower?

[Artur Malabarba]

Glenn Morris <rgm@gnu.org> writes:

> Somewhat faster, but still much slower than it was when there could be
> only one dir-locals. Rough, one-off timings: wildcards was ~ 22% slower,
> completion seems to be ~ 18% slower.

Thanks Glenn.

Then I guess I'll change it to use a fixed filename, probably
`.dir-locals-2.el' or `.dir-locals-user.el'.

This shouldn't slow the build noticeably, unless the old dir-locals
behaviour (using a fixed filename) already took up a large fraction of
time without us knowing.

Re: builds are getting slower?

[Artur Malabarba]

> Then I guess I'll change it to use a fixed filename, probably
> .dir-locals-2.el' or.dir-locals-user.el'.

Done.

Re: builds are getting slower?

[Glenn Morris]

Artur Malabarba wrote:

>> Then I guess I'll change it to use a fixed filename, probably
>> .dir-locals-2.el' or.dir-locals-user.el'.
>
> Done.

I'm afraid this has made no difference to the times I measure.
I checked that if I apply a diff to revert back to a single dir-locals
file, the times do indeed fall back to what they were originally.

If no-one else other than me notices this, perhaps you should forget
about it. (Again, these are the times for non-parallel debug bootstrap
builds.)

As a final comment, if the aim is just to override an existing
dir-locals files without modifying it, this can be done by direct
application of dir-locals-set-class-variables and
dir-locals-set-directory-class. The issue there is that the UI isn't as
convenient, and it completely replaces any dir-locals file rather than
appending+overriding it. But those things could presumably be improved.

Re: builds are getting slower?

[Stefan Monnier]

> As a final comment, if the aim is just to override an existing
> dir-locals files without modifying it, this can be done by direct
> application of dir-locals-set-class-variables and
> dir-locals-set-directory-class. The issue there is that the UI isn't as
> convenient, and it completely replaces any dir-locals file rather than
> appending+overriding it. But those things could presumably be improved.

FWIW, I also dislike the .dir-locals2.el thingy and would rather we
improve the support for doing it in ~/.emacs (like
dir-locals-set-directory-class).


        Stefan

Re: builds are getting slower?

[John Wiegley]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> FWIW, I also dislike the .dir-locals2.el thingy and would rather we improve
> the support for doing it in ~/.emacs (like dir-locals-set-directory-class).

The advantage to .dir-locals<N>.el is that it's dead simple, and the semantics
are pretty obvious. We didn't realize it would have such unintended
consequences, though. I'd like to keep the simplicity, if we can.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: builds are getting slower?

[Artur Malabarba]

On 26 January 2016 at 18:52, Glenn Morris <rgm@gnu.org> wrote:
> Artur Malabarba wrote:
>
>>> Then I guess I'll change it to use a fixed filename, probably
>>> .dir-locals-2.el' or.dir-locals-user.el'.
>>
>> Done.
>
> I'm afraid this has made no difference to the times I measure.
> I checked that if I apply a diff to revert back to a single dir-locals
> file, the times do indeed fall back to what they were originally.

I'm very surprised at this. The emacs sources don't even have a
.dir-locals-2.el. So we're not visiting an extra file or reading extra
lisp, we're just checking for the existence of this file (exactly once
for each time we would check the existence of .dir-locals.el anyway).
Maybe the slowness is not really where I expected (filesystem
interaction), but is perhaps somewhere else.

> If no-one else other than me notices this, perhaps you should forget
> about it. (Again, these are the times for non-parallel debug bootstrap
> builds.)

I don't notice it because I build infrequently and never pay attention
while building, so I really don't count.

> As a final comment, if the aim is just to override an existing
> dir-locals files without modifying it, this can be done by direct
> application of dir-locals-set-class-variables and
> dir-locals-set-directory-class. The issue there is that the UI isn't as
> convenient, and it completely replaces any dir-locals file rather than
> appending+overriding it. But those things could presumably be improved.

I can revert this feature if people prefer. But I quite like the
convenience (and ease-of-use) of writing a second .dir-locals, and the
merge (appending+overriding) helps as well.
We can also just revert it from the release branch for the moment, and
postpone it until emacs 26 while we settle on a preferred
interface/implementation.

Re: builds are getting slower?

[Paul Eggert]

On 01/26/2016 12:15 PM, Artur Malabarba wrote:
> We can also just revert it from the release branch for the moment, and
> postpone it until emacs 26 while we settle on a preferred
> interface/implementation.

I like this suggestion.

Re: builds are getting slower?

[John Wiegley]

>>>>> Paul Eggert <eggert@cs.ucla.edu> writes:

> On 01/26/2016 12:15 PM, Artur Malabarba wrote:
>> We can also just revert it from the release branch for the moment, and
>> postpone it until emacs 26 while we settle on a preferred
>> interface/implementation.

> I like this suggestion.

Me too.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: builds are getting slower?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

I make the change on Friday the 5th. Please don't release before then. ;-)

On 28 Jan 2016 11:29 pm, "John Wiegley" <jwiegley@gmail.com> wrote:
>
> >>>>> Paul Eggert <eggert@cs.ucla.edu> writes:
>
> > On 01/26/2016 12:15 PM, Artur Malabarba wrote:
> >> We can also just revert it from the release branch for the moment, and
> >> postpone it until emacs 26 while we settle on a preferred
> >> interface/implementation.
>
> > I like this suggestion.
>
> Me too.
>
> --
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: Type: text/html, Size: 982 bytes --]

Re: builds are getting slower?

[Glenn Morris]

Artur Malabarba wrote:

> If that doesn't help, then we could also forgo of the regexp, and just have
> two allowed filenames (like .dir-locals.el and .dir-locals-2.el).

BTW, if the aim (?) is to allow overriding of dir-locals.el, can't
direct application of dir-locals-set-directory-class etc do that,
without the need to allow multiple dir-locals files to exist?

Re: builds are getting slower?

[Glenn Morris]

PS actual (one-off) numbers:

2e84888:
real    25m28.352s
user    24m22.170s
sys     1m7.141s

cbaa040:
real    21m10.912s
user    20m8.635s
sys     1m4.202s

Re: builds are getting slower?

[David Kastrup]

Glenn Morris <rgm@gnu.org> writes:

> PS actual (one-off) numbers:
>
> 2e84888:
> real    25m28.352s
> user    24m22.170s
> sys     1m7.141s
>
> cbaa040:
> real    21m10.912s
> user    20m8.635s
> sys     1m4.202s

Ok, so it's not significantly the system calls themselves but the time
Emacs spends working with their results.

-- 
David Kastrup

Re: builds are getting slower?

[Pádraig Brady]

I noticed you mentioned inotify tests were slow on emacs.
tail(1) tests in coreutils had the same issue,
and when fixed it identified harder to hit race bugs within tail
as well as greatly speeding up the test runs as shown below:

  $ time make -j8 check TESTS="..."
  real 0m4.886s
  user 0m5.375s
  sys 0m4.565s
  =======================================================
  Testsuite summary for GNU coreutils 8.24.107-e369f
  =======================================================
  # TOTAL: 29
  # PASS:  28
  # SKIP:  1

The main technique used in those tests was a truncated exponential back-off
mechanism, which allows for faster operation in the normal case, but
will add extra delays where required. See the use of retry_delay_():
git.sv.gnu.org/gitweb/?p=coreutils.git;a=blob;f=init.cfg;h=e4309ae5;hb=HEAD#l628

cheers,
Pádraig.