Multi-OS Emacs buildbot?

Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Over the past few months, I've installed a whole bunch of VMs with
different operating systems to help debug various problems.  Now that
I've got them, I'm wondering whether it would be helpful to do a
buildbot kind of thing across them all?

This can either be a lot of work if done right, or it can be done very
simply.  Doing it right would mean automatically building on all
commits, and notifying the committer about breakages etc...  but that's,
like, real work, and I'm not volunteering to do that at all.

A much simpler solution would be to just write a script that ssh-es to
all the VMs once a day, "git pull; make bootstrap; make check" and then
make a summary report on all the tests that fail to some new Emacs
mailing list.  I can do that; it's trivial.

It'd be...  somewhat useful?  But not really that useful?  Opinions?

(My current VM collection is FreeBSD, NetBSD, OpenBSD, Windows 10
w/Cygwin, Macos-es catalina, mojave and high sierra (with HomeBrew),
Macos catalina (with Macports), and Fedora.  Big Sur and Windows 10
w/MinGW soon to follow.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[dick.r.chiang]

I suspect you're aware commercial entities (called CI tools) build across
multiple platforms for no financial costs (only political ones).  But I can
never tell with the graybeards at emacs-devel.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

dick.r.chiang@gmail.com writes:

> I suspect you're aware commercial entities (called CI tools) build across
> multiple platforms for no financial costs (only political ones).  But I can
> never tell with the graybeards at emacs-devel.

That was a convoluted way of writing "I hereby offer to set up CI for
Emacs", I think.  Or did I misread you?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> This can either be a lot of work if done right, or it can be done very
> simply.  Doing it right would mean automatically building on all
> commits, and notifying the committer about breakages etc...  but that's,
> like, real work, and I'm not volunteering to do that at all.

So it goes like this:

=> git pull on driver system once in a day
=> make a list of commits
=> on each vm (concurrent)
=>   pick commit = last
=>   make bootstrap; make check
=>   if fail
=>     for commit in [(last-1)..0]
=>       make bootstrap; make check
=>       if pass
=>         make report for commit+1
=>         email-to author of commit+1
=>         break
=>   else
=>     OK

Or something more ambitious?

> A much simpler solution would be to just write a script that ssh-es to
> all the VMs once a day, "git pull; make bootstrap; make check" and then
> make a summary report on all the tests that fail to some new Emacs
> mailing list.  I can do that; it's trivial.

I guess this is sufficient for us.

The realtime stuff, like build on each commit, will be useful only to
the authors who are directly committing to the repository. They will get
to know within 15mins if there is a problem in commit. But those who are
sending patches there will be some delay.

> It'd be...  somewhat useful?  But not really that useful?  Opinions?

I guess such a system will be more useful during the period when we are
about to release. Otherwise we’ll be spending too much energy. People
are anyway build on their systems.

Or, we can use this system to find out the faulty commit, once someone
reports an issue.

> (My current VM collection is FreeBSD, NetBSD, OpenBSD, Windows 10
> w/Cygwin, Macos-es catalina, mojave and high sierra (with HomeBrew),
> Macos catalina (with Macports), and Fedora.  Big Sur and Windows 10
> w/MinGW soon to follow.)

I heard aws is also offering macos now. Does it actually run on VM? I
thought Apple hardware is must for macos. Or is it something that runs
on real Mac[(book)] only?

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Pankaj Jangid <pankaj@codeisgreat.org> writes:

> Or something more ambitious?

Way less ambitious.  Almost literally "for i in machines; ssh $i; git
pull; make; make check | tail -n 20; done | mail".

With some timeouts and additional filtering.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Yuri Khan]

On Sun, 20 Dec 2020 at 01:49, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Way less ambitious.  Almost literally "for i in machines; ssh $i; git
> pull; make; make check | tail -n 20; done | mail".

GNU Parallel[1] might be of some use.

[1]: https://www.gnu.org/software/parallel/

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Pankaj Jangid <pankaj@codeisgreat.org> writes:
>
>> Or something more ambitious?
>
> Way less ambitious.  Almost literally "for i in machines; ssh $i; git
> pull; make; make check | tail -n 20; done | mail".

We are not check if it pass or fail.


> With some timeouts and additional filtering.  :-)
            ^^^^^^^^                ^^^^^^^^^

Here is the real work I guess. Which filters are you thinking? and
timeouts I didn’t get. Is this about build time? That could be machine
specific.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Pankaj Jangid <pankaj@codeisgreat.org> writes:

> Here is the real work I guess. Which filters are you thinking?

The email should be pretty, but since I'm not sure this has any value at
all, I'm not thinking much of anything.

> and timeouts I didn’t get. Is this about build time? That could be
> machine specific.

Some of the checks will invariably hang sometimes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Some of the checks will invariably hang sometimes.

Or variably hang all the time.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Some of the checks will invariably hang sometimes.
>
> Or variably hang all the time.

Okay. Let me write and come up with something. I take this up.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Pankaj Jangid <pankaj@codeisgreat.org> writes:

> Okay. Let me write and come up with something. I take this up.

Hey, hang on a minute.  :-)  I appreciate the enthusiasm, but we should
first decide whether we want this or not.  And remember, Dick promised
to set up a real multi-platform CI thing at some buildbot service or
other, and we won't need this super-simple one then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Okay. Let me write and come up with something. I take this up.
>
> Hey, hang on a minute.  :-)  I appreciate the enthusiasm, but we should
> first decide whether we want this or not.  And remember, Dick promised
> to set up a real multi-platform CI thing at some buildbot service or
> other, and we won't need this super-simple one then.

Definitely I was going to hold till today. I was very late. :-) Read the
whole thread now. Looks like we already have most parts.

Even if that infrastruction is ready, I would certainly want a script to
find out the commit that broke the build on my hardware. That reverse
choronological loop.

If someone has that ready then please share. Or may be that could be
part of the repository.

Re: Multi-OS Emacs buildbot?

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]


() Pankaj Jangid <pankaj@codeisgreat.org>
() Sun, 20 Dec 2020 08:05:03 +0530

   Even if that infrastruction is ready, I would certainly want
   a script to find out the commit that broke the build on my
   hardware. That reverse choronological loop.

How would that loop be helpful?  If the late{st,r} commit works,
i don't see the point looking for trouble.  What am i missing?

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)               ; (2020) Software Libero
   (pcase (context query)               ;       = Dissenso Etico
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 219 bytes --]

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> () Pankaj Jangid <pankaj@codeisgreat.org>
> () Sun, 20 Dec 2020 08:05:03 +0530
>
>    Even if that infrastruction is ready, I would certainly want
>    a script to find out the commit that broke the build on my
>    hardware. That reverse choronological loop.
>
> How would that loop be helpful?  If the late{st,r} commit works,
> i don't see the point looking for trouble.  What am i missing?

That script will not be useful only if the last commit works. It is useful
to file a bug if the last build fails. Usually, I git-pull once in a day
and that comes with many commits. If HEAD works then no worries. Else
try HEAD~1 and so on...

Re: Multi-OS Emacs buildbot?

[Tim Landscheidt]

Pankaj Jangid <pankaj@codeisgreat.org> wrote:

>>> Okay. Let me write and come up with something. I take this up.

>> Hey, hang on a minute.  :-)  I appreciate the enthusiasm, but we should
>> first decide whether we want this or not.  And remember, Dick promised
>> to set up a real multi-platform CI thing at some buildbot service or
>> other, and we won't need this super-simple one then.

> Definitely I was going to hold till today. I was very late. :-) Read the
> whole thread now. Looks like we already have most parts.

> Even if that infrastruction is ready, I would certainly want a script to
> find out the commit that broke the build on my hardware. That reverse
> choronological loop.

> If someone has that ready then please share. Or may be that could be
> part of the repository.

I'm not sure what you mean exactly.  If you want to find out
which commit broke a build (in any Git repository), you can
use "git bisect" to tell Git which commit worked and which
commit failed, and Git will then run an arbitrary command
(e. g. "make && make check") to see which commit broke the
build.  (NB: Strictly speaking, this only works when the
failure is, eh, monotonous (?), i. e. the build worked for
all commits from last-good to the-one-before-first-failure
and failed for all commits from first-failure to now.)

In the context of build farms, you can make the command more
complex, e. g. set up a container/copy a VM template, ssh
into the instance and run "git checkout $commit && make &&
make check" there.

Tim

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

>> Even if that infrastruction is ready, I would certainly want a script to
>> find out the commit that broke the build on my hardware. That reverse
>> choronological loop.
>
>> If someone has that ready then please share. Or may be that could be
>> part of the repository.
>
> I'm not sure what you mean exactly.  If you want to find out
> which commit broke a build (in any Git repository), you can
> use "git bisect" to tell Git which commit worked and which
> commit failed, and Git will then run an arbitrary command
> (e. g. "make && make check") to see which commit broke the
> build.

That might be useful if Git already has the information about good and
bad commits. It helps in narrowing down problems.

> (NB: Strictly speaking, this only works when the
> failure is, eh, monotonous (?), i. e. the build worked for
> all commits from last-good to the-one-before-first-failure
> and failed for all commits from first-failure to now.)

My use-case is very simple. I pull and make bootstrap daily. And on some
odd day if the build breaks, I want to file a bug. If I have the
information which commit broke it then it helps the maintainers. Even if
the failure is monotonous (not sure what to call linear may be), that
may help in narrowing down the issue. It could be inaccurate but still.

> In the context of build farms, you can make the command more
> complex, e. g. set up a container/copy a VM template, ssh
> into the instance and run "git checkout $commit && make &&
> make check" there.

As Lars asked/said, not sure if that is of any help to us.

Re: Multi-OS Emacs buildbot?

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]


() Pankaj Jangid <pankaj@codeisgreat.org>
() Sun, 20 Dec 2020 13:47:46 +0530

   My use-case is very simple. I pull and make bootstrap
   daily. And on some odd day if the build breaks, I want to
   file a bug. If I have the information which commit broke it
   then it helps the maintainers. Even if the failure is
   monotonous (not sure what to call linear may be), that may
   help in narrowing down the issue. It could be inaccurate but
   still.

It sounds like you're talking about "continuous" vs "transient".

What you seek is the first in a series of failures, no matter
the subsequent outcomes, in an attempt to reduce the variables.

No matter the subsequent outcomes, because if they are:

(a) success (transient state) we still might be legitimately
    worried about that failure mode re-appearing;

(b) failure (continuous state) we want to debug w/ as few
    changes as possibile (reducing the variability).

Bottom line: first failure is easier to debug (usually :-D).

Did i miss anything?

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)               ; (2020) Software Libero
   (pcase (context query)               ;       = Dissenso Etico
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 219 bytes --]

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> It sounds like you're talking about "continuous" vs "transient".
>
> What you seek is the first in a series of failures, no matter
> the subsequent outcomes, in an attempt to reduce the variables.
>
> No matter the subsequent outcomes, because if they are:
>
> (a) success (transient state) we still might be legitimately
>     worried about that failure mode re-appearing;
>
> (b) failure (continuous state) we want to debug w/ as few
>     changes as possibile (reducing the variability).
>
> Bottom line: first failure is easier to debug (usually :-D).
>
> Did i miss anything?

You have covered more. :-) For my simple case, I was interested in (b)
only. (a) may be too complex. If a subsequent build is ‘success’ then it
could mean two things - (1) the issue is fixed (2) another commit has
caused it to behave. If we have a good test case coverage then nothing
to worry in case of (a).

Re: Multi-OS Emacs buildbot?

[Tim Landscheidt]

Pankaj Jangid <pankaj@codeisgreat.org> wrote:

> […]

>> (NB: Strictly speaking, this only works when the
>> failure is, eh, monotonous (?), i. e. the build worked for
>> all commits from last-good to the-one-before-first-failure
>> and failed for all commits from first-failure to now.)

> My use-case is very simple. I pull and make bootstrap daily. And on some
> odd day if the build breaks, I want to file a bug. If I have the
> information which commit broke it then it helps the maintainers. Even if
> the failure is monotonous (not sure what to call linear may be), that
> may help in narrowing down the issue. It could be inaccurate but still.

> […]

Git records the changes to the local repository in a refer-
ence log.  One can access that with "git reflog" and/or "git
rev-parse".  (For example:

| [tim@passepartout ~/src/emacs]$ git rev-parse 'HEAD'
| 2d790c6c57b244447390c023679752243e0049c9
| [tim@passepartout ~/src/emacs]$ git rev-parse 'HEAD@{one.week.ago}'
| 2d790c6c57b244447390c023679752243e0049c9
| [tim@passepartout ~/src/emacs]$ git rev-parse 'HEAD@{two.week.ago}'
| 67a8bdb90c9b5865b7f17290c7135b1a5458c36d
| [tim@passepartout ~/src/emacs]$

suggests that I last fetched the Emacs repository some time
between one and two weeks ago.)

So, untested, if you encounter a bug in HEAD, you could
start the bisection, mark that commit as "bad", then iterate
over HEAD@{1}, HEAD@{2}, etc. marking failures as "bad" com-
mits as you go along, until you find a working commit, mark
that as "good", and then start "git bisect run".

(I hope it is clear that I'm not advocating against using CI
systems; I just want to point out how one can narrow down a
bug if either a repository does not have a CI system attach-
ed or the CI system was not configured to track a particular
bug.)

Tim

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

> Git records the changes to the local repository in a refer-
> ence log.  One can access that with "git reflog" and/or "git
> rev-parse".  (For example:
>
> | [tim@passepartout ~/src/emacs]$ git rev-parse 'HEAD'
> | 2d790c6c57b244447390c023679752243e0049c9
> | [tim@passepartout ~/src/emacs]$ git rev-parse 'HEAD@{one.week.ago}'
> | 2d790c6c57b244447390c023679752243e0049c9
> | [tim@passepartout ~/src/emacs]$ git rev-parse 'HEAD@{two.week.ago}'
> | 67a8bdb90c9b5865b7f17290c7135b1a5458c36d
> | [tim@passepartout ~/src/emacs]$
>
> suggests that I last fetched the Emacs repository some time
> between one and two weeks ago.)
>
> So, untested, if you encounter a bug in HEAD, you could
> start the bisection, mark that commit as "bad", then iterate
> over HEAD@{1}, HEAD@{2}, etc. marking failures as "bad" com-
> mits as you go along, until you find a working commit, mark
> that as "good", and then start "git bisect run".

I didn’t know git has a builtin system like this. We were probably on
the same page. I was doing it outside git. Building previous commits in
reverse order and recording somewhere. Thanks for sharing this.

> (I hope it is clear that I'm not advocating against using CI
> systems; I just want to point out how one can narrow down a
> bug if either a repository does not have a CI system attach-
> ed or the CI system was not configured to track a particular
> bug.)

The two approaches have different context. CI is a centralized approach
where commits are comming and the system points out a faulty one and
informs the maintainers. The other approach is distributed. So many
people are already pulling code and building on their systems. Probably
a script can ask them, “*this* particular commit has issues, do you want
to report a bug?”. Of course, the later one covers only the platforms
which are most used and leave out the others. Lars has discovered that
so many *BSD builds are failing. Only a centralized system will be
effective in such a case.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Pankaj Jangid <pankaj@codeisgreat.org> writes:

> Lars has discovered that so many *BSD builds are failing.

Well, not failing, exactly -- Emacs builds and works on the BSDs, but
there's a varying list of test failures.  Here's the FreeBSD one, for
instance:

SUMMARY OF TEST RESULTS
-----------------------
Files examined: 312
Ran 4271 tests, 4192 results as expected, 14 unexpected, 65 skipped
8 files contained unexpected results:
  src/emacs-module-tests.log
  lisp/wdired-tests.log
  lisp/progmodes/flymake-tests.log
  lisp/net/dbus-tests.log
  lisp/epg-tests.log
  lisp/dired-aux-tests.log
  lisp/cedet/semantic-utest-ia.log
  lisp/net/tramp-tests.log


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Lars has discovered that so many *BSD builds are failing.
>
> Well, not failing, exactly -- Emacs builds and works on the BSDs, but
> there's a varying list of test failures.  Here's the FreeBSD one, for
> instance:

Oh! sorry about that. I meant that in more generic sense - all kinds of
failures.

Re: Multi-OS Emacs buildbot?

[Dmitry Gutov]

On 19.12.2020 21:18, Lars Ingebrigtsen wrote:
> And remember, Dick promised
> to set up a real multi-platform CI thing at some buildbot service or
> other, and we won't need this super-simple one then.

In case you're not being snarky here:

There are free-ish CI solutions which we could use, one example is 
CircleCI (https://circleci.com/open-source/).

But they usually support only a limited number of code hosting 
platforms. E.g. CircleCI only supports GitHub and BitBucket.

So to use it we would have to rely on a GitHub mirror. Can/should we do 
that?

There is one at https://github.com/emacs-mirror/emacs/, but we don't 
know who controls it, so we can't ask them to set up the integration 
from that side.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> And remember, Dick promised
>> to set up a real multi-platform CI thing at some buildbot service or
>> other, and we won't need this super-simple one then.
>
> In case you're not being snarky here:
>
> There are free-ish CI solutions which we could use, one example is
> CircleCI (https://circleci.com/open-source/).

I am totally being snarky, because (in my experience) setting up CI for
a project takes a bit of work to get to a useful state, especially when
doing CI over a wide range of systems.  If somebody is willing to do the
work, that'd be wonderful, but apparently nobody is?

So instead we have the current state, where hydra builds on Debian, and
that's it.  (Plus Emba, and I don't know what systems it's building on,
really.)

> So to use it we would have to rely on a GitHub mirror. Can/should we
> do that?
>
> There is one at https://github.com/emacs-mirror/emacs/, but we don't
> know who controls it, so we can't ask them to set up the integration
> from that side.

I think anybody's that interested can set this up in whatever way
they want to.  The CI doesn't have to be a GNU/FSF-approved thing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Dmitry Gutov]

On 20.12.2020 19:40, Lars Ingebrigtsen wrote:
>> So to use it we would have to rely on a GitHub mirror. Can/should we
>> do that?
>>
>> There is one athttps://github.com/emacs-mirror/emacs/, but we don't
>> know who controls it, so we can't ask them to set up the integration
>> from that side.
> I think anybody's that interested can set this up in whatever way
> they want to.  The CI doesn't have to be a GNU/FSF-approved thing.

True, but such a CI shouldn't (?) be allowed to send emails to Emacs 
committers, I think.

And if CircleCI was, say, integrated with a GitHub clone of the Emacs 
repository, I'm guessing it would only send emails to the GH account 
that does the "cloning" and pushes to that repository. So some 
forwarding would be necessary, and at that junction the information 
about individual pushes (to Savannah's Git) is lost.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> True, but such a CI shouldn't (?) be allowed to send emails to Emacs
> committers, I think.

It could send mails to a mailing list, though...  and perhaps people
could opt in to having mails sent to themselves?

> And if CircleCI was, say, integrated with a GitHub clone of the Emacs
> repository, I'm guessing it would only send emails to the GH account
> that does the "cloning" and pushes to that repository. So some
> forwarding would be necessary, and at that junction the information
> about individual pushes (to Savannah's Git) is lost.

Hm.  I don't quite see how that pipeline would lose anything.  I mean, a
commit is a commit.  (I'm assuming the CI works on individual commits,
and not on pushes.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Dmitry Gutov]

On 20.12.2020 20:08, Lars Ingebrigtsen wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> True, but such a CI shouldn't (?) be allowed to send emails to Emacs
>> committers, I think.
> It could send mails to a mailing list, though...  and perhaps people
> could opt in to having mails sent to themselves?

Either would imply some organizational approval, I think.

Or else I would be re-sending build results emails with URLs pointing to 
CircleCI from my personal email to some official Emacs mailing list, 
and... well, either we as a project approve of that kind of integration, 
or we don't. I certainly don't need anybody coming around and blaming me 
as supporting proprietary platforms this way.

> I'm assuming the CI works on individual commits,
> and not on pushes.

On pushes. It might be not apparent from the web interface, but the 
Gitlab CI skips intermediate commits for, say, efficiency reasons.

It's common practice, IME.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> I'm assuming the CI works on individual commits,
>> and not on pushes.
>
> On pushes. It might be not apparent from the web interface, but the
> Gitlab CI skips intermediate commits for, say, efficiency reasons.
>
> It's common practice, IME.

Right.  But when there's a failure, I thought they bisected back to the
first failing commit in that batch?

Anyway, I see that I was wrong about hydra -- it does builds on Macos,
too?  At least it talks about "darwin"...  but the output from that
build doesn't seem to contain compilation output from the .m files that
I'm used to.  And it doesn't run "make check" there, for some reason?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Dmitry Gutov]

On 22.12.2020 08:27, Lars Ingebrigtsen wrote:
>> On pushes. It might be not apparent from the web interface, but the
>> Gitlab CI skips intermediate commits for, say, efficiency reasons.
>>
>> It's common practice, IME.
> Right.  But when there's a failure, I thought they bisected back to the
> first failing commit in that batch?

I don't remember any of them doing that.

Mising posts in thread from gwene Was: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Recently I have started to read the mailing list via gwene. I see that
some posts are coming very late. Context is missing in a thread in such
a case.

Is this known or the issue has appeared recently?

Re: Mising posts in thread from gwene Was: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Pankaj Jangid <pankaj@codeisgreat.org> writes:

> Recently I have started to read the mailing list via gwene. I see that
> some posts are coming very late. Context is missing in a thread in such
> a case.
>
> Is this known or the issue has appeared recently?

The FSF mailing lists are sometimes slow (because of the thorough spam
handling).  So if somebody Cc's you directly, you'll often get the
responses much sooner than if you get the responses via the mailing
list.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Mising posts in thread from gwene Was: Multi-OS Emacs buildbot?

[Pankaj Jangid]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Recently I have started to read the mailing list via gwene. I see that
>> some posts are coming very late. Context is missing in a thread in such
>> a case.
>>
>> Is this known or the issue has appeared recently?
>
> The FSF mailing lists are sometimes slow (because of the thorough spam
> handling).  So if somebody Cc's you directly, you'll often get the
> responses much sooner than if you get the responses via the mailing
> list.

Okay. So switching off MFT for important groups is a good idea. Or may
be use C-c C-f C-a for specifc threads.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

I did a run-through of a bunch of the VMs, and we apparently have "make
check" failures on absolutely all of them currently...  except Debian.
Of course, the question is: Do we care sufficiently that something is
failing on NetBSD?  I mean, to put in some work on that, given our
limited developer resources.  Perhaps not?

On the other hand, it would be nice to know that Emacs builds at all on
Fedora, say -- it might give us greater confidence when making changes
that we're not breaking something fundamental on a bunch of systems.  Go
fast and don't break things.

If we want to do this, but don't want to prioritise fixing test failures
on systems we don't actually use ourselves, we could just disable test
more aggressively (and "FIXME" them). 

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

'make check' errors on Debian (was: Multi-OS Emacs buildbot?)

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I did a run-through of a bunch of the VMs, and we apparently have "make
> check" failures on absolutely all of them currently...  except Debian.

Really?  I've been getting the errors that follow my signature for quite
a while now on Debian.

-- 
Basil

(current-time-zone) ;; => (0 "GMT")

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw3d scroll bars)
 of 2020-12-19 built on thunk
Repository revision: b9edbaed01a91d5fc6235fc679d8e0cd827f6fa9
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12010000
System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O2 -march=native' --config-cache
 --prefix=/home/blc/.local --with-x-toolkit=lucid
 --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB
NOTIFY INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
LIBOTF ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS
LIBSYSTEMD JSON PDUMPER LCMS2

Important settings:
  value of $LANG: en_IE.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Test solar-sunrise-sunset backtrace:
  signal(ert-test-failed (((should (< (abs (- sunrise 7.27)) epsilon))
  ert-fail(((should (< (abs (- sunrise 7.27)) epsilon)) :form (< 1.003
Test solar-sunrise-sunset condition:
    (ert-test-failed
     ((should
       (<
	(abs ...)
	epsilon))
      :form
      (< 1.0033333324640985 0.016666666666666666)
      :value nil))
   FAILED  1/1  solar-sunrise-sunset (0.003660 sec)

Test nsm-check-local-subnet-ipv4 condition:
    (ert-test-failed
     ((should
       (eq t
	   (nsm-should-check "google.com")))
      :form
      (eq t nil)
      :value nil))
   FAILED  1/2  nsm-check-local-subnet-ipv4 (0.085710 sec)

Test time-stamp-format-am-pm condition:
    (ert-test-failed
     ((should
       (equal
	(time-stamp-string "%#p" ref-time1)
	"pm"))
      :form
      (equal "" "pm")
      :value nil :explanation
      (arrays-of-different-length 0 2 "" "pm" first-mismatch-at 0)))
   FAILED   7/27  time-stamp-format-am-pm (0.000140 sec)

Test time-stamp-format-ignored-modifiers condition:
    (ert-test-failed
     ((should
       (equal
	(time-stamp-string "%.,@-+_ ^(stuff)P" ref-time3)
	"AM"))
      :form
      (equal "  " "AM")
      :value nil :explanation
      (array-elt 0
		 (different-atoms
		  (32 "#x20" "? ")
		  (65 "#x41" "?A")))))
   FAILED  13/27  time-stamp-format-ignored-modifiers (0.000121 sec)

Test time-stamp-format-string-width condition:
    (ert-test-failed
     ((should
       (equal
	(time-stamp-string "%1P" ref-time3)
	"A"))
      :form
      (equal " " "A")
      :value nil :explanation
      (array-elt 0
		 (different-atoms
		  (32 "#x20" "? ")
		  (65 "#x41" "?A")))))
   FAILED  20/27  time-stamp-format-string-width (0.000126 sec)

Re: 'make check' errors on Debian

[Lars Ingebrigtsen]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> I did a run-through of a bunch of the VMs, and we apparently have "make
>> check" failures on absolutely all of them currently...  except Debian.
>
> Really?  I've been getting the errors that follow my signature for quite
> a while now on Debian.

I don't get any of those, and neither does hydra nor emba (which are
both Debian, I guess?)  You should file a bug report.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Of course, the question is: Do we care sufficiently that something is
  > failing on NetBSD?  I mean, to put in some work on that, given our
  > limited developer resources.  Perhaps not?

That is presented as one question but really it is a range of
questions.

Do we care about NetBSD enough to debug Emacs failures on NetBSD
on our own initiative?  The Emacs maintainers should not.

Do we care about NetBSD enough to arrange to inform people
who might want to work on that debugging?  I think we should,
unless informing them is harder than I'd expect.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> Do we care about NetBSD enough to arrange to inform people
> who might want to work on that debugging?  I think we should,
> unless informing them is harder than I'd expect.

Yeah, I guess that has some value...  It might possibly entice some
people (who uses Emacs on these systems) to have a whack at fixing some
of the problems?  Perhaps worth a try.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Alan Third]

On Sat, Dec 19, 2020 at 04:58:38PM +0100, Lars Ingebrigtsen wrote:
> Over the past few months, I've installed a whole bunch of VMs with
> different operating systems to help debug various problems.  Now that
> I've got them, I'm wondering whether it would be helpful to do a
> buildbot kind of thing across them all?
> 
> This can either be a lot of work if done right, or it can be done very
> simply.  Doing it right would mean automatically building on all
> commits, and notifying the committer about breakages etc...  but that's,
> like, real work, and I'm not volunteering to do that at all.
> 
> A much simpler solution would be to just write a script that ssh-es to
> all the VMs once a day, "git pull; make bootstrap; make check" and then
> make a summary report on all the tests that fail to some new Emacs
> mailing list.  I can do that; it's trivial.
> 
> It'd be...  somewhat useful?  But not really that useful?  Opinions?
> 
> (My current VM collection is FreeBSD, NetBSD, OpenBSD, Windows 10
> w/Cygwin, Macos-es catalina, mojave and high sierra (with HomeBrew),
> Macos catalina (with Macports), and Fedora.  Big Sur and Windows 10
> w/MinGW soon to follow.)

You should perhaps have a look at the gitlab CI/CD stuff that's
running on emba.gnu.org.

In theory you could install a gitlab runner on each of your VMs and
emba could push the compile jobs out to them.
-- 
Alan Third

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Alan Third <alan@idiocy.org> writes:

> In theory you could install a gitlab runner on each of your VMs and
> emba could push the compile jobs out to them.

Yeah, that's more work than I'm willing to do.

This reminds me, though -- why isn't EMBA sending out "you broke the
build" messages to people?  That'd be quite useful, wouldn't it?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> In theory you could install a gitlab runner on each of your VMs and
>> emba could push the compile jobs out to them.
>
> Yeah, that's more work than I'm willing to do.

Maybe somebody will help you? I, for example, am already thinking
about GitLab runners. But I didn't start to implement.

> This reminds me, though -- why isn't EMBA sending out "you broke the
> build" messages to people?  That'd be quite useful, wouldn't it?

Because nobody has implemented it? And it isn't trivial. Since every
commit triggers the whole machinery on Emba, only the *first* commit
which has broken something should send the message.

Imagine, I have broken Tramp (not unlikely, you know). So I will get a
message, and everybody who commits afterwards, would also get such
message, until I have fixed this. Not so nice.

AFAIK, GitLab has no memory. It doesn't remember, whether the previous
job has failed.

And think about several breakages, for example you have broken Gnus in
parallel (this *is* unlikely). GitLab sees only that the job returns
with error, it doesn't know the reason.

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Dmitry Gutov]

On 20.12.2020 10:50, Michael Albinus wrote:
> AFAIK, GitLab has no memory. It doesn't remember, whether the previous
> job has failed.

I think it does. At least, at my place of employment I do receive emails 
with titles like "Pipeline #xx has failed" or "Pipeline #yy has been 
fixed", with, naturally, no emails for pipelines that simply succeed.

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

Hi Dmitry,

> I think it does. At least, at my place of employment I do receive
> emails with titles like "Pipeline #xx has failed" or "Pipeline #yy has
> been fixed", with, naturally, no emails for pipelines that simply
> succeed.

This feature shall work also in our GitLab instance; messages about
failed pipelines shall be sent to <emacs-buildstatus@gnu.org>. This
worked sometimes, but it fails again after a while. No idea what's
wrong.

My statement about missing GitLab memory was, that a failed pipeline
doesn't know, whether the previous pipeline has succeed or failed. With
that knowledge, one could decrease the numbers of messages to be sent.

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Dmitry Gutov]

On 20.12.2020 16:11, Michael Albinus wrote:

> This feature shall work also in our GitLab instance; messages about
> failed pipelines shall be sent to <emacs-buildstatus@gnu.org>. This
> worked sometimes, but it fails again after a while. No idea what's
> wrong.

Yeah, we should look into it.

> My statement about missing GitLab memory was, that a failed pipeline
> doesn't know, whether the previous pipeline has succeed or failed. With
> that knowledge, one could decrease the numbers of messages to be sent.

But it does. Or else it couldn't provide the email subjects I gave 
example of. "has failed" and "has been fixed" are for the firs failure 
and the first fixed build, correspondingly.

Re: Multi-OS Emacs buildbot?

[Toon Claes]

On Sun, Dec 20 2020, Dmitry Gutov wrote:

> On 20.12.2020 16:11, Michael Albinus wrote:
>
>> This feature shall work also in our GitLab instance; messages about
>> failed pipelines shall be sent to <emacs-buildstatus@gnu.org>. This
>> worked sometimes, but it fails again after a while. No idea what's
>> wrong.
>
> Yeah, we should look into it.

If I'm not mistaken we disabled it on purpose. EMBA was set up as a
try-out, it not the official CI, as far as I understood. I've been asked
to disable mails from GitLab CI to emacs-buildstatus@gnu.org because not
everybody following that address would like it.

If people here agree to enable it again, I don't mind doing that.


--
Toon

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

Hi Toon,

>>> This feature shall work also in our GitLab instance; messages about
>>> failed pipelines shall be sent to <emacs-buildstatus@gnu.org>. This
>>> worked sometimes, but it fails again after a while. No idea what's
>>> wrong.
>>
>> Yeah, we should look into it.
>
> If I'm not mistaken we disabled it on purpose. EMBA was set up as a
> try-out, it not the official CI, as far as I understood. I've been asked
> to disable mails from GitLab CI to emacs-buildstatus@gnu.org because not
> everybody following that address would like it.
>
> If people here agree to enable it again, I don't mind doing that.

I've just checked my mail archive, I couldn't see such a deactivation
message. But this doesn't mean anything but there's no such a message in
my archive ...

However, the mailing list is empty. It shouldn't hurt anybody if
buildstatus messages arrive there.

> Toon

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> However, the mailing list is empty. It shouldn't hurt anybody if
> buildstatus messages arrive there.

Yeah, the hydra build messages, at least, should go there, I think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

>> However, the mailing list is empty. It shouldn't hurt anybody if
>> buildstatus messages arrive there.
>
> Yeah, the hydra build messages, at least, should go there, I think?

They did. Then it was broken somehow, and as far as I understand Glenn,
there are problems to reanimate.

Yes, of course, it would be very helpful.

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Toon Claes]

On Tue, Dec 22 2020, Michael Albinus wrote:

> However, the mailing list is empty. It shouldn't hurt anybody if
> buildstatus messages arrive there.

I've had some issues, but they should be working again.
https://lists.gnu.org/archive/html/emacs-buildstatus/2020-12/msg00001.html


-- 
Toon

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

Hi Toon,

> I've had some issues, but they should be working again.
> https://lists.gnu.org/archive/html/emacs-buildstatus/2020-12/msg00001.html

Thanks! I've got also the info from Fermin, that the authentication
email has found its way.

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Toon Claes <toon@iotcl.com> writes:

> On Tue, Dec 22 2020, Michael Albinus wrote:
>
>> However, the mailing list is empty. It shouldn't hurt anybody if
>> buildstatus messages arrive there.
>
> I've had some issues, but they should be working again.
> https://lists.gnu.org/archive/html/emacs-buildstatus/2020-12/msg00001.html

Yup.  The messages aren't really that informative, though?  They just
say that something has failed and then there's a link you have to click
to see what.  Would it be possible to include the "make check" summary,
at least?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> Toon Claes <toon@iotcl.com> writes:
>
>> On Tue, Dec 22 2020, Michael Albinus wrote:
>>
>>> However, the mailing list is empty. It shouldn't hurt anybody if
>>> buildstatus messages arrive there.
>>
>> I've had some issues, but they should be working again.
>> https://lists.gnu.org/archive/html/emacs-buildstatus/2020-12/msg00001.html
>
> Yup.  The messages aren't really that informative, though?  They just
> say that something has failed and then there's a link you have to click
> to see what.  Would it be possible to include the "make check" summary,
> at least?

You might follow the job link, like
<https://emba.gnu.org/emacs/emacs/-/jobs/9532/raw>. At the bottom line
you'll see the summary.

Btw, I have a private package emba.el, which shows you all (or the last
ones) jobs, and their status. Plus following a link to the log page, in
Emacs.

Needs some polishing, 'tho.

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

>> Yup.  The messages aren't really that informative, though?  They just
>> say that something has failed and then there's a link you have to click
>> to see what.

[...]

> You might follow the job link, like
> <https://emba.gnu.org/emacs/emacs/-/jobs/9532/raw>. At the bottom line
> you'll see the summary.

I think that's what I said.  :-)

But the request was:

>>  Would it be possible to include the "make check" summary,
>> at least?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> But the request was:
>
>>>  Would it be possible to include the "make check" summary,
>>> at least?

That's not so simple. Jobs fail for different reasons, and there isn't
always a make check summary.

So you would need to implement kind of log file parser.

And if you start to implement different kind of jobs (for example jobs
for build, jobs for test), it will be even more complicate.

The best approximation I can think of is to add the last N lines of the
log file to the notification message.

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> That's not so simple. Jobs fail for different reasons, and there isn't
> always a make check summary.
>
> So you would need to implement kind of log file parser.
>
> And if you start to implement different kind of jobs (for example jobs
> for build, jobs for test), it will be even more complicate.

Well, the "make check" output is pretty regular, so it shouldn't be very
difficult to recognise.

> The best approximation I can think of is to add the last N lines of the
> log file to the notification message.

That's better than nothing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> And if you start to implement different kind of jobs (for example jobs
>> for build, jobs for test), it will be even more complicate.
>
> Well, the "make check" output is pretty regular, so it shouldn't be very
> difficult to recognise.

Again, we have already jobs which don't call "make check". And this type
of jobs will increase. See for example
<https://emba.gnu.org/emacs/emacs/-/jobs/9526/raw>.

>> The best approximation I can think of is to add the last N lines of the
>> log file to the notification message.
>
> That's better than nothing.

Yes. Toon, I have no idea whether this would be possible in the mail
template?

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Toon Claes]

On Sat, Dec 26 2020, Michael Albinus wrote:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Well, the "make check" output is pretty regular, so it shouldn't be very
>> difficult to recognise.

GitLab is able to parse JUnit [1] formatted test output. It's an XML
format specified by IBM [2].

But as far as I know, it's currently not possible to include these
results in the pipeline emails. So that won't help us much at this
point.

>>> The best approximation I can think of is to add the last N lines of the
>>> log file to the notification message.
>>
>> That's better than nothing.
>
> Yes. Toon, I have no idea whether this would be possible in the mail
> template?

Until a few months ago the last 30 lines of the log were included in the
emails. But unfortunately they were removed [3] for security and
performance reasons. There is a follow-up issue [4] to discuss an
alternative.

We can apply a patch to our installation to revert the above, we did
something like this before. But there's an automatic update set up on
the machine, so we need a mechanism to reapply the patch after updating.


--
Toon

[1] https://docs.gitlab.com/ee/ci/unit_test_reports.html
[2] https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html
[3] https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42395.patch
[4] https://gitlab.com/gitlab-org/gitlab/-/issues/290306

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

>>>> The best approximation I can think of is to add the last N lines of the
>>>> log file to the notification message.
>>>
>>> That's better than nothing.
>>
>> Yes. Toon, I have no idea whether this would be possible in the mail
>> template?
>
> Until a few months ago the last 30 lines of the log were included in the
> emails. But unfortunately they were removed [3] for security and
> performance reasons. There is a follow-up issue [4] to discuss an
> alternative.
>
> We can apply a patch to our installation to revert the above, we did
> something like this before. But there's an automatic update set up on
> the machine, so we need a mechanism to reapply the patch after updating.

It appended something like this

--8<---------------cut here---------------start------------->8---
Stage: test
Name: test-all
Trace: Makefile:319: recipe for target 'check-doit' failed
make[2]: Leaving directory '/builds/emacs/emacs/test'
make[2]: *** [check-doit] Error 1
Makefile:294: recipe for target 'check-expensive' failed
make[1]: *** [check-expensive] Error 2
make[1]: Leaving directory '/builds/emacs/emacs/test'
make: *** [check-expensive] Error 2
Makefile:959: recipe for target 'check-expensive' failed
section_end:1567624331:build_script
section_start:1567624331:after_script
section_end:1567624333:after_script
section_start:1567624333:upload_artifacts_on_failure
section_end:1567624335:upload_artifacts_on_failure
ERROR: Job failed: exit code 1
--8<---------------cut here---------------end--------------->8---

So it is some information that stage test has failed, but it shows only
the Makefile output, and not the more informative test summary. That
doesn't help.

> Toon

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Toon Claes]

On Sun, Dec 27 2020, Michael Albinus wrote:

> It appended something like this
>
> ...
>
> So it is some information that stage test has failed, but it shows only
> the Makefile output, and not the more informative test summary. That
> doesn't help.

I'm not familiar with the Emacs test suite, but where else can we get
info about the test results? Is there another log file?


--
Toon

Re: Multi-OS Emacs buildbot?

[Alan Third]

On Wed, Dec 30, 2020 at 02:43:32PM +0100, Toon Claes wrote:
> On Sun, Dec 27 2020, Michael Albinus wrote:
> 
> > It appended something like this
> >
> > ...
> >
> > So it is some information that stage test has failed, but it shows only
> > the Makefile output, and not the more informative test summary. That
> > doesn't help.
> 
> I'm not familiar with the Emacs test suite, but where else can we get
> info about the test results? Is there another log file?

As I recall there's one log file per test .el file produced in src,
lib-src and lisp under test.

I wouldn't imagine it would be much work to save them as artifacts?
-- 
Alan Third

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

>> So it is some information that stage test has failed, but it shows only
>> the Makefile output, and not the more informative test summary. That
>> doesn't help.
>
> I'm not familiar with the Emacs test suite, but where else can we get
> info about the test results? Is there another log file?

As Alan wrote already, there is one log file per test file. If a test run
(per test file) fails, the log file is shown. See definition of Makefile
variable WRITE_LOG:

--8<---------------cut here---------------start------------->8---
## Save logs, and show logs for failed tests.
WRITE_LOG = > $@ 2>&1 || { STAT=$$?; cat $@; exit $$STAT; }
--8<---------------cut here---------------end--------------->8---

However, since we ran into a job timeout, the job has stopped before we
could see the log file. This is fixed now for process-tests.el, because
every single test is wrapped now by a timeout of 60 seconds, which let
the tests fail in time.

The Makefile knows also special handling when running on hydra, showing
the test progress unconditionally in the Makefile output. Currently, it
has the value:

--8<---------------cut here---------------start------------->8---
ifdef EMACS_HYDRA_CI
## On Hydra, always show logs for certain problematic tests.
lisp/net/tramp-tests.log \
: WRITE_LOG = 2>&1 | tee $@
endif
--8<---------------cut here---------------end--------------->8---

That is, running Tramp tests will always show the progress in the
Makefile output. We could use the same feature for emba if we want.

> Toon

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Maybe somebody will help you? I, for example, am already thinking
> about GitLab runners. But I didn't start to implement.

The VMs are running on my machine on the desk over there *points*, so
I'm a bit leery of hooking them up to some external system that can
issue commands.

>> This reminds me, though -- why isn't EMBA sending out "you broke the
>> build" messages to people?  That'd be quite useful, wouldn't it?
>
> Because nobody has implemented it? And it isn't trivial. Since every
> commit triggers the whole machinery on Emba, only the *first* commit
> which has broken something should send the message.
>
> Imagine, I have broken Tramp (not unlikely, you know). So I will get a
> message, and everybody who commits afterwards, would also get such
> message, until I have fixed this. Not so nice.
>
> AFAIK, GitLab has no memory. It doesn't remember, whether the previous
> job has failed.

When I've worked with CI systems, they usually are pretty good at
assigning blame for a breakage?  I'd be surprised if GitLab didn't have
such a basic feature.  But I know nothing about GitLab, and the
dashboard at emba isn't very encouraging -- it seems to say that most of
the builds failed because of git locking issues:

----
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
----

So it looks like it needs some work?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

>> Maybe somebody will help you? I, for example, am already thinking
>> about GitLab runners. But I didn't start to implement.
>
> The VMs are running on my machine on the desk over there *points*, so
> I'm a bit leery of hooking them up to some external system that can
> issue commands.

Yes, it is also my concern. Well, I have a local OpenBSD VM, I also plan
to do first tests installing a GitLab runner there. Just to get a feeling.

Where to put such VMs afterwards is another story. Or you install a
local GitLab instance in your environment, and let the CI jobs run
there?

>> Imagine, I have broken Tramp (not unlikely, you know). So I will get a
>> message, and everybody who commits afterwards, would also get such
>> message, until I have fixed this. Not so nice.
>>
>> AFAIK, GitLab has no memory. It doesn't remember, whether the previous
>> job has failed.
>
> When I've worked with CI systems, they usually are pretty good at
> assigning blame for a breakage?  I'd be surprised if GitLab didn't have
> such a basic feature.

Maybe it has, and I don't know. I know just the trigger "pipeline is
broken" and "pipeline is working"; GitLab sends the messages exactly
when the trigger changes.

For blaming^W informing *people*, we need a more fine-grained trigger.

> But I know nothing about GitLab, and the
> dashboard at emba isn't very encouraging -- it seems to say that most of
> the builds failed because of git locking issues:
>
> ----
> Another git process seems to be running in this repository, e.g.
> an editor opened by 'git commit'. Please make sure all processes
> are terminated then try again. If it still fails, a git process
> may have crashed in this repository earlier:
> ----
>
> So it looks like it needs some work?

Yes, it needs some love. Likely this happens when there are too many
commits in fast sequence; the resulting pipelines block each other. It
is on my TODO to check what's up; perhaps I should give it more
priority.

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> Where to put such VMs afterwards is another story. Or you install a
> local GitLab instance in your environment, and let the CI jobs run
> there?

Sounds like work to me.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi Lars,

> Over the past few months, I've installed a whole bunch of VMs with
> different operating systems to help debug various problems.  Now that
> I've got them, I'm wondering whether it would be helpful to do a
> buildbot kind of thing across them all?
>
> This can either be a lot of work if done right, or it can be done very
> simply.  Doing it right would mean automatically building on all
> commits, and notifying the committer about breakages etc...  but that's,
> like, real work, and I'm not volunteering to do that at all.
>
> A much simpler solution would be to just write a script that ssh-es to
> all the VMs once a day, "git pull; make bootstrap; make check" and then
> make a summary report on all the tests that fail to some new Emacs
> mailing list.  I can do that; it's trivial.
>
> It'd be...  somewhat useful?  But not really that useful?  Opinions?
>
> (My current VM collection is FreeBSD, NetBSD, OpenBSD, Windows 10
> w/Cygwin, Macos-es catalina, mojave and high sierra (with HomeBrew),
> Macos catalina (with Macports), and Fedora.  Big Sur and Windows 10
> w/MinGW soon to follow.)

We have the CI infrastructure on emba.gnu.org. Looks like your VMs are
perfect candidates for GitLab runners from that.

<https://emba.gnu.org/emacs/emacs/pipelines>

<https://docs.gitlab.com/runner/>

Best regards, Michael.

Re: Multi-OS Emacs buildbot?

[Daniel Martín]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> It'd be...  somewhat useful?  But not really that useful?  Opinions?

I think it's a good idea.  As other have said, I think we could somehow
add your machines to https://emba.gnu.org/emacs/emacs as GitLab Runners.

I'd be careful with the macOS VMs, though: Apple does not allow
virtualizing macOS in non-Apple hardware.

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Daniel Martín <mardani29@yahoo.es> writes:

> I'd be careful with the macOS VMs, though: Apple does not allow
> virtualizing macOS in non-Apple hardware.

I've signed no contract with Apple, and I'm fortunately not in the US,
so I'm free to run whatever I want on my private machine.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Nikolay Kudryavtsev]

Hi Lars.

A few years ago I wrote a builder script for Windows that installs all 
of the required dependencies for building Emacs:

https://github.com/sg2002/ms-windows-builder.el

This may be of some use to you in your project, if you decide to pursue it.

I had thoughts of refactoring this into a more generic cross-platform 
tool, with the end goal of abstracting away the OS and the OS's package 
manager, but it's a lot of work... Then again, the benefit is pretty clear.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Multi-OS Emacs buildbot?

[Lars Ingebrigtsen]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

> A few years ago I wrote a builder script for Windows that installs all
> of the required dependencies for building Emacs:
>
> https://github.com/sg2002/ms-windows-builder.el
>
> This may be of some use to you in your project, if you decide to pursue it.

Thanks -- it seems like a lot of code to build Emacs, though.  But I
haven't yet attempted doing that under MinGW, so perhaps it's really
that complicated...

> I had thoughts of refactoring this into a more generic cross-platform
> tool, with the end goal of abstracting away the OS and the OS's
> package manager, but it's a lot of work... Then again, the benefit is
> pretty clear.

I've written up instructions on building Emacs on just about all the
operating systems, and it turns out to be pretty easy on most of
them...  if you know which magical incantations to use.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Multi-OS Emacs buildbot?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 23 Dec 2020 08:10:32 +0100
> Cc: emacs-devel@gnu.org
> 
> > https://github.com/sg2002/ms-windows-builder.el
> >
> > This may be of some use to you in your project, if you decide to pursue it.
> 
> Thanks -- it seems like a lot of code to build Emacs, though.  But I
> haven't yet attempted doing that under MinGW, so perhaps it's really
> that complicated...

You should only install the necessary tools, which is a one-time job.
GCC and Binutils for compiling C programs; Bash, Autoconf, Make, and
Coreutils for running the configury and the build, Texinfo for
producing the Info manuals, Gawk, Grep, gzip...  Thereafter, Emacs
configures and builds on Windows as on any other system, using the
same scripts and Makefiles.