Script for compiling more quietly

Script for compiling more quietly

[Lars Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

For some time now, I've been using the simple shell script below to do
non-development Emacs compilation.  That is, when I say "git pull" I
always say "emake" afterwards, and I also run it when I apply patches
from people and stuff.  It basically suppresses all the informational
messages from the build process, leaving me with just the bear
necessities, so that you can forget about your worries and your strife,
I mean, the wall of text.

I find it handy, because it really makes things obvious when warnings do
happen -- I don't have to actually pay attention.

(It can obviously not be used when actually developing, because it makes
it impossible to jump to errors from the *Compilation* buffer.)

I wonder whether anybody would find this useful, and if so, whether it
would make sense to stash it in, say, admin/make-silent or something.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

[-- Attachment #2: emake --]
[-- Type: application/octet-stream, Size: 466 bytes --]

#!/bin/bash

make -j`nproc` "$@" 2>&1 | \
    egrep -v "^make|\
^Loading|\
SCRAPE|\
INFO.*Scraping.*[.]\$|\
^Waiting for git|\
^Finding pointers|\
^Using load-path|\
^Adding name|\
^Dump mode|\
^Dumping finger|\
^Byte counts|\
^Reloc counts|\
^Pure-hashed|\
^cp -f temacs|\
^rm -f bootstrap|\
^Dump complete|\
^rm -f emacs|\
mkdir -p ../etc|\
^LC_ALL.*pdump|\
^cp -f emacs.p|\
GEN.*loaddefs|\
^Reloading stale|\
^Source file.*newer than|\
^Directories for loaddefs"

Re: Script for compiling more quietly

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]


Hi Lars,

>
> For some time now, I've been using the simple shell script below to do 
> non-development Emacs compilation.  That is, when I say "git pull" I 
> always say "emake" afterwards, and I also run it when I apply patches 
> from people and stuff.  It basically suppresses all the informational 
> messages from the build process, leaving me with just the bear 
> necessities, so that you can forget about your worries and your strife, 
> I mean, the wall of text.
>

Thanks for your script, which is indeed handy.  I attach a slighly 
improved version, which also "cleans" the autogen and configure steps, and 
which displays error/warning messages in bold red.  I also replaced your 
"nproc" by "expr $(nproc) / 2 + 1", IME using nproc makes the computer 
unuseable, and is not much faster than nproc/2+1.

[-- Attachment #2: Type: text/plain, Size: 1191 bytes --]

#!/bin/bash

make -j`expr $(nproc) / 2 + 1` "$@" 2>&1 | \
sed -u 's# \.\./\.\./# #
s# \.\./# #
s#^Running # Running #
s#^Configured for # Configured for #
s#^./temacs # ./temacs #
s#^Dumping under the name# Dumping under the name#
' | \
egrep --line-buffered -v "^make|\
^Loading|\
SCRAPE|\
INFO.*Scraping.*[.]\$|\
^Waiting for git|\
^Finding pointers|\
^Using load-path|\
^Adding name|\
^Dump mode|\
^Dumping finger|\
^Byte counts|\
^Reloc counts|\
^Pure-hashed|\
^cp -f temacs|\
^rm -f bootstrap|\
^Dump complete|\
^rm -f emacs|\
mkdir -p etc|\
mkdir -p info|\
mkdir -p lisp|\
^LC_ALL.*pdump|\
^cp -f emacs.p|\
GEN.*loaddefs|\
^Reloading stale|\
^Source file.*newer than|\
^Directories for loaddefs|\
^./autogen.sh|\
^[Cc]hecking |\
^.Read INSTALL.REPO for more|\
^Your system has the required tools.|\
^Building aclocal.m4|\
^ Running 'autoreconf|\
^You can now run './configure'|\
^./configure|\
^configure: creating|\
^\"configure\" file built.|\
^There seems to be no|\
^config.status:|\
^ *$|\
^Makefile built\
" | \
while read
do
  [[ "X${REPLY:0:1}" == "X " ]] && C="" || C="\033[1;31m"
  printf "$C%s\033[0m\n" "$REPLY"
done

Re: Script for compiling more quietly

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 38 bytes --]


(With yet another small improvement.)

[-- Attachment #2: Type: text/plain, Size: 1244 bytes --]

#!/bin/bash

make -j`expr $(nproc) / 2 + 1` "$@" 2>&1 | \
sed -u 's# \.\./\.\./# #
s# \.\./# #
s#^Running # Running #
s#^Configured for # Configured for #
s#^./temacs # ./temacs #
s#^Dumping under the name# Dumping under the name#
' | \
egrep --line-buffered -v "^make|\
^Loading|\
SCRAPE|\
INFO.*Scraping.*[.]\$|\
^Waiting for git|\
^Finding pointers|\
^Using load-path|\
^Adding name|\
^Dump mode|\
^Dumping finger|\
^Byte counts|\
^Reloc counts|\
^Pure-hashed|\
^cp -f temacs|\
^rm -f bootstrap|\
^Dump complete|\
^rm -f emacs|\
mkdir -p etc|\
mkdir -p info|\
mkdir -p lisp|\
^LC_ALL.*pdump|\
^cp -f emacs.p|\
GEN.*loaddefs|\
^Reloading stale|\
^Source file.*newer than|\
^Directories for loaddefs|\
^./autogen.sh|\
^[Cc]hecking |\
^.Read INSTALL.REPO for more|\
^Your system has the required tools.|\
^Building aclocal.m4|\
^ Running 'autoreconf|\
^You can now run './configure'|\
^./configure|\
^configure: creating|\
^\"configure\" file built.|\
^There seems to be no|\
^config.status:|\
^ *$|\
^Makefile built\
" | \
while read
do
  C=""
  [[ "X${REPLY:0:1}" != "X " ]] && C="\033[1;31m"
  [[ "X${REPLY:0:3}" == "X   " ]] && C="\033[1;31m"
  printf "$C%s\033[0m\n" "$REPLY"
done

Re: Script for compiling more quietly

[Lars Ingebrigtsen]

Gregory Heytings <ghe@sdf.org> writes:

>  I also replaced your "nproc" by "expr $(nproc) / 2 + 1", IME using
> nproc makes the computer unuseable, and is not much faster than
> nproc/2+1.

It depends on whether you have Hypertheading, I guess -- I don't (on
this laptop), so -j`nproc` is twice as fast as nproc / 2.

Absurdly enough, nproc doesn't have an option for "just say how many
cores there are, OK?"  But I guess one could parse /proc/cpuinfo to try
to figure that out.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Script for compiling more quietly

[Lars Ingebrigtsen]

Gregory Heytings <ghe@sdf.org> writes:

> Thanks for your script, which is indeed handy.  I attach a slighly
> improved version, which also "cleans" the autogen and configure steps,
> and which displays error/warning messages in bold red.

Looks nice, but it made a whole lot of stuff red when doing "bootstrap"?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Script for compiling more quietly

[Stefan Monnier]

> by "expr $(nproc) / 2 + 1", IME using nproc makes the computer unuseable,
> and is not much faster than nproc/2+1.

My crystal ball tells me you have hyperthreading, so `nproc` returns
twice as many CPUs as you have "cores".  If you don't have
hyperthreading, I'd expect the performance difference between using
`nproc` or only half to be more significant.


        Stefan

Re: Script for compiling more quietly

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Absurdly enough, nproc doesn't have an option for "just say how many
> cores there are, OK?"  But I guess one could parse /proc/cpuinfo to try
> to figure that out.

But this is now easier than it was in The Olden Days, when you had to
look at sibling IDs and the like.  The following will tell you the
number of cores:

grep "^cpu cores" /proc/cpuinfo | tail -n 1 |  awk '{print $4}' 
8

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Script for compiling more quietly

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]


>> Thanks for your script, which is indeed handy.  I attach a slighly 
>> improved version, which also "cleans" the autogen and configure steps, 
>> and which displays error/warning messages in bold red.
>
> Looks nice, but it made a whole lot of stuff red when doing "bootstrap"?
>

Gosh, I forgot to filter out the initial Git repository setup!  Fixed in 
the attached script.  Now I only see three warnings in red with the 
current master: the famous https://gitlab.gnome.org/GNOME/gtk/issues/221 , 
two obsolete warnings in cedet/semantic/idle.el , and the (sporadic) 
Python shift/reduce conflict.

>> I also replaced your "nproc" by "expr $(nproc) / 2 + 1", IME using 
>> nproc makes the computer unuseable, and is not much faster than 
>> nproc/2+1.
>
> It depends on whether you have Hypertheading, I guess -- I don't (on 
> this laptop), so -j`nproc` is twice as fast as nproc / 2.
>
> Absurdly enough, nproc doesn't have an option for "just say how many 
> cores there are, OK?"  But I guess one could parse /proc/cpuinfo to try 
> to figure that out.
>

Yes indeed, that's it, somehow all computers I see/use have 
hyperthreading, and I concluded a bit too fast that this was always the 
case.  Also fixed in the attached script, which uses /proc/cpuinfo.

[-- Attachment #2: Type: text/plain, Size: 1536 bytes --]

#!/bin/bash

C=$(grep '^cpu cores' /proc/cpuinfo | cut -d: -f2 | head -1)
make -j$C "$@" 2>&1 | \
sed -u 's# \.\./\.\./# #
s# \.\./# #
s#^Configuring local git # Configuring local git #
s#^Installing git hooks...# Installing git hooks...#
s#^Running # Running #
s#^Configured for # Configured for #
s#^./temacs # ./temacs #
s#^Dumping under the name# Dumping under the name#
' | \
egrep --line-buffered -v "^make|\
^Loading|\
SCRAPE|\
INFO.*Scraping.*[.]\$|\
^Waiting for git|\
^Finding pointers|\
^Using load-path|\
^Adding name|\
^Dump mode|\
^Dumping finger|\
^Byte counts|\
^Reloc counts|\
^Pure-hashed|\
^cp -f temacs|\
^rm -f bootstrap|\
^Dump complete|\
^rm -f emacs|\
mkdir -p etc|\
mkdir -p info|\
mkdir -p lisp|\
^LC_ALL.*pdump|\
^cp -f emacs.p|\
GEN.*loaddefs|\
^Reloading stale|\
^Source file.*newer than|\
^Directories for loaddefs|\
^./autogen.sh|\
^[Cc]hecking |\
^.Read INSTALL.REPO for more|\
^Your system has the required tools.|\
^Building aclocal.m4|\
^ Running 'autoreconf|\
^You can now run './configure'|\
^./configure|\
^configure: creating|\
^\"configure\" file built.|\
^There seems to be no|\
^config.status:|\
^ *$|\
^Makefile built|\
The GNU allocators don't work|\
^git config |\
^'\.git/|\
^\^\(\(|\
^'build-aux/git-hooks\
" | \
while read
do
  C=""
  [[ "X${REPLY:0:1}" != "X " ]] && C="\033[1;31m"
  [[ "X${REPLY:0:3}" == "X   " ]] && C="\033[1;31m"
  [[ "X$C" == "X" ]] && printf "%s\n" "$REPLY" || printf "$C%s\033[0m\n" "$REPLY"
done

Re: Script for compiling more quietly

[Gregory Heytings via Emacs development discussions.]

>> by "expr $(nproc) / 2 + 1", IME using nproc makes the computer 
>> unuseable, and is not much faster than nproc/2+1.
>
> My crystal ball tells me you have hyperthreading, so `nproc` returns 
> twice as many CPUs as you have "cores".  If you don't have 
> hyperthreading, I'd expect the performance difference between using 
> `nproc` or only half to be more significant.
>

Yes, that's it.

BTW, thanks for the redisplay-adhoc-scroll-in-resize-mini-windows!

Re: Script for compiling more quietly

[Stefan Monnier]

> C=$(grep '^cpu cores' /proc/cpuinfo | cut -d: -f2 | head -1)

sed -ne '/^cpu cores/{s/.*: *//;p;q}' /proc/cpuinfo


        Stefan

Re: Script for compiling more quietly

[Gregory Heytings via Emacs development discussions.]

>> C=$(grep '^cpu cores' /proc/cpuinfo | cut -d: -f2 | head -1)
>
> sed -ne '/^cpu cores/{s/.*: *//;p;q}' /proc/cpuinfo
>

awk '/^cpu cores/{print $4;exit}' /proc/cpuinfo

;-)

Re: Script for compiling more quietly

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 14 Dec 2020 10:23:15 -0500
> Cc: Gregory Heytings <ghe@sdf.org>, Lars Ingebrigtsen <larsi@gnus.org>
> 
> > C=$(grep '^cpu cores' /proc/cpuinfo | cut -d: -f2 | head -1)
> 
> sed -ne '/^cpu cores/{s/.*: *//;p;q}' /proc/cpuinfo

FWIW, I think you should leave the N part of "make -jN" to the end
user.  No one said they have nothing else to do at the same time as
they build Emacs; they could be building some other large program at
the same time, or need the CPU for something else.

Re: Script for compiling more quietly

[Gregory Heytings via Emacs development discussions.]

>>> C=$(grep '^cpu cores' /proc/cpuinfo | cut -d: -f2 | head -1)
>>
>> sed -ne '/^cpu cores/{s/.*: *//;p;q}' /proc/cpuinfo
>
> FWIW, I think you should leave the N part of "make -jN" to the end user. 
> No one said they have nothing else to do at the same time as they build 
> Emacs; they could be building some other large program at the same time, 
> or need the CPU for something else.
>

It is, the number of cores is just a default value, that can be overridden 
by "emake -j1" for instance.

Re: Script for compiling more quietly

[Eli Zaretskii]

> Date: Mon, 14 Dec 2020 16:07:04 +0000
> Cc: larsi@gnus.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org
> From: Gregory Heytings via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> > FWIW, I think you should leave the N part of "make -jN" to the end user. 
> > No one said they have nothing else to do at the same time as they build 
> > Emacs; they could be building some other large program at the same time, 
> > or need the CPU for something else.
> 
> It is, the number of cores is just a default value, that can be overridden 
> by "emake -j1" for instance.

In which case I don't understand why it is important to be accurate
with the actual number of cores.  You could just use 4 as the default.

Re: Script for compiling more quietly

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> In which case I don't understand why it is important to be accurate
> with the actual number of cores.  You could just use 4 as the default.

The default should be sensible, and the sensible default is "all the
cores".  -j4 on a system with two cores would be slower than -j2.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Script for compiling more quietly

[Andrea Corallo via Emacs development discussions.]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> In which case I don't understand why it is important to be accurate
>> with the actual number of cores.  You could just use 4 as the default.
>
> The default should be sensible, and the sensible default is "all the
> cores".  -j4 on a system with two cores would be slower than -j2.

I propose to run a machine learning algo to determine the best value :) :)

Re: Script for compiling more quietly

[Gregory Heytings via Emacs development discussions.]

>> In which case I don't understand why it is important to be accurate 
>> with the actual number of cores.  You could just use 4 as the default.
>
> The default should be sensible, and the sensible default is "all the 
> cores".  -j4 on a system with two cores would be slower than -j2.
>

BTW, I'm not sure the current formula is correct for a computer with 
multiple physical CPU's, each with multiple cores... but I don't have a 
computer with that configuration to check.

Re: Script for compiling more quietly

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Gregory Heytings <ghe@sdf.org>,  monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> Date: Mon, 14 Dec 2020 17:53:42 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > In which case I don't understand why it is important to be accurate
> > with the actual number of cores.  You could just use 4 as the default.
> 
> The default should be sensible, and the sensible default is "all the
> cores".  -j4 on a system with two cores would be slower than -j2.

How many people who regularly build Emacs have 2 cores?

And I don't think it's true that -j4 on such a system will be slower,
certainly not significantly slower.  I build my Emacs with -j8 on a
system with 4 cores all the time, and never saw any slowdown.

Anyway, it isn't worth an argument, so I will now bow out.

Re: Script for compiling more quietly

[Eli Zaretskii]

> Date: Mon, 14 Dec 2020 17:24:56 +0000
> Cc: Gregory Heytings <ghe@sdf.org>, Eli Zaretskii <eliz@gnu.org>,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Andrea Corallo via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> I propose to run a machine learning algo to determine the best value :) :)

Algo away, by all means!

Re: Script for compiling more quietly

[Lars Ingebrigtsen]

Gregory Heytings <ghe@sdf.org> writes:

> BTW, I'm not sure the current formula is correct for a computer with
> multiple physical CPU's, each with multiple cores... but I don't have
> a computer with that configuration to check.

No, it'll give the wrong results on multi-CPU machines.

I've now committed all of this to admin/emake, and further bikeshedding
can take place there.  :-)

(And if it turns out that this isn't really generally useful for people
doing Emacs stuff, then we can consider removing it again.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Script for compiling more quietly

[Stefan Monnier]

> How many people who regularly build Emacs have 2 cores?

Most of my computers are 2 cores (some of them hyperthreaded).


        Stefan