Alternatives for reliable build environments for emacs?

Alternatives for reliable build environments for emacs?

[joakim]

Hello,

I'm finding it increasingly difficult to build emacs in my distro,
Fedora(Ok, I havent managed to build emacs using distro dependencies for
a long time)

What are the alternatives?

One can use for example:
- Guix
- Docker
- Flatpak

I have tried guix and docker but not yet flatpak.

These are nice because you can pin the dependency versions, to make sure
you have a reproducible build environment.

These all have the drawback that they are inconvenient when making local
modifications to the emacs source, at least for me. Way back in the days
when my original build environment worked, I automatically merged some
branches and applied local patchs with stgit, and its these kind of
things I've found awkward with the above mentioned containerized build
systems.

How are the emacs developers managing this?


Regards,

-- 
Joakim Verona
joakim@verona.se

Re: Alternatives for reliable build environments for emacs?

[Eli Zaretskii]

> From: joakim@verona.se
> Date: Thu, 28 Oct 2021 14:55:27 +0200
> 
> I'm finding it increasingly difficult to build emacs in my distro,
> Fedora(Ok, I havent managed to build emacs using distro dependencies for
> a long time)

Why? what are the difficulties?

Re: Alternatives for reliable build environments for emacs?

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: joakim@verona.se
>> Date: Thu, 28 Oct 2021 14:55:27 +0200
>> 
>> I'm finding it increasingly difficult to build emacs in my distro,
>> Fedora(Ok, I havent managed to build emacs using distro dependencies for
>> a long time)
>
> Why? what are the difficulties?

In my case one of the problems that started the downward spiral was the versions shipped of gnutls in
fedora which  didnt work with emacs.

The separate aproach of building all the dependencies emacs need in a
separate build environment does work, like the Guix aproach.

I could probably convince guix to do what I want, but guix is also
fiddly. I was just wondering if somebody else had a solution for a
reproducible development environment for emacs.

Maybe I'm the only one facing these difficulties, that's fine I guess.


-- 
Joakim Verona
joakim@verona.se

Re: Alternatives for reliable build environments for emacs?

[Robert Pluim]

>>>>> On Thu, 28 Oct 2021 16:38:54 +0200, joakim@verona.se said:

    joakim> Eli Zaretskii <eliz@gnu.org> writes:
    >>> From: joakim@verona.se
    >>> Date: Thu, 28 Oct 2021 14:55:27 +0200
    >>> 
    >>> I'm finding it increasingly difficult to build emacs in my distro,
    >>> Fedora(Ok, I havent managed to build emacs using distro dependencies for
    >>> a long time)
    >> 
    >> Why? what are the difficulties?

    joakim> In my case one of the problems that started the downward spiral was the versions shipped of gnutls in
    joakim> fedora which  didnt work with emacs.

Thatʼs the first Iʼve heard of it. GnuTLS too old, too new, something else?

    joakim> The separate aproach of building all the dependencies emacs need in a
    joakim> separate build environment does work, like the Guix aproach.

    joakim> I could probably convince guix to do what I want, but guix is also
    joakim> fiddly. I was just wondering if somebody else had a solution for a
    joakim> reproducible development environment for emacs.

    joakim> Maybe I'm the only one facing these difficulties, that's fine I guess.

Debian has always worked ok for me. Although I donʼt target
reproducible builds.

Robert
-- 

Re: Alternatives for reliable build environments for emacs?

[joakim]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Thu, 28 Oct 2021 16:38:54 +0200, joakim@verona.se said:
>
>     joakim> Eli Zaretskii <eliz@gnu.org> writes:
>     >>> From: joakim@verona.se
>     >>> Date: Thu, 28 Oct 2021 14:55:27 +0200
>     >>> 
>     >>> I'm finding it increasingly difficult to build emacs in my distro,
>     >>> Fedora(Ok, I havent managed to build emacs using distro dependencies for
>     >>> a long time)
>     >> 
>     >> Why? what are the difficulties?
>
>     joakim> In my case one of the problems that started the downward spiral was the versions shipped of gnutls in
>     joakim> fedora which  didnt work with emacs.
>
> Thatʼs the first Iʼve heard of it. GnuTLS too old, too new, something else?

Too new in Fedora(at the time before I threw in the towel at least).

>
>     joakim> The separate aproach of building all the dependencies emacs need in a
>     joakim> separate build environment does work, like the Guix aproach.
>
>     joakim> I could probably convince guix to do what I want, but guix is also
>     joakim> fiddly. I was just wondering if somebody else had a solution for a
>     joakim> reproducible development environment for emacs.
>
>     joakim> Maybe I'm the only one facing these difficulties, that's fine I guess.
>
> Debian has always worked ok for me. Although I donʼt target
> reproducible builds.

Well, I could likely build emacs just fine in a docker container using
debian as the base, and run the container in my host fedora os. I know
some people do this. This approach is also brittle with how to manage
the src tree for development.


>
> Robert
-- 
Joakim Verona
joakim@verona.se

Re: Alternatives for reliable build environments for emacs?

[Robert Pluim]

>>>>> On Thu, 28 Oct 2021 17:22:39 +0200, joakim@verona.se said:
    >> Thatʼs the first Iʼve heard of it. GnuTLS too old, too new, something else?

    joakim> Too new in Fedora(at the time before I threw in the towel at least).

OK. Iʼll guess we'll start seeing reports at some point then.

    >> Debian has always worked ok for me. Although I donʼt target
    >> reproducible builds.

    joakim> Well, I could likely build emacs just fine in a docker container using
    joakim> debian as the base, and run the container in my host fedora os. I know
    joakim> some people do this. This approach is also brittle with how to manage
    joakim> the src tree for development.

I just point my dockers at a local directory, then you donʼt need to worry
about volume persistence.

Robert
-- 

Re: Alternatives for reliable build environments for emacs?

[Yuri Khan]

On Thu, 28 Oct 2021 at 19:57, <joakim@verona.se> wrote:

> I'm finding it increasingly difficult to build emacs in my distro,
> Fedora(Ok, I havent managed to build emacs using distro dependencies for
> a long time)
>
> What are the alternatives?
> I have tried guix and docker but not yet flatpak.
>
> These are nice because you can pin the dependency versions, to make sure
> you have a reproducible build environment.
>
> These all have the drawback that they are inconvenient when making local
> modifications to the emacs source, at least for me. Way back in the days
> when my original build environment worked, I automatically merged some
> branches and applied local patchs with stgit, and its these kind of
> things I've found awkward with the above mentioned containerized build
> systems.

I’m not an Emacs developer but the issue is familiar. I use Docker for
similar cases: (1) building software targeting an older version of the
distribution than is installed on my host system, or (2) building
software using newer versions of the toolchain than I’m comfortable
installing on my host machine.

The trick is that you keep the toolchain and build dependencies in the
container, but bind-mount the source and build trees so they are not
parts of the container image. This way, you work with the source tree
using your local tools; the container is completely disposable.

I’d use the following workspace layout:

…/emacs-workspace
  build/     ← out-of-tree build directory
  docker/    ← separate directory so nothing gets into Docker build context
    Dockerfile
  emacs/     ← source tree
  .env       ← see below
  docker-compose.yml  ← convenience wrapper
  Makefile   ← convenience wrapper over docker-compose

Sample configuration files below are completely untested and provided
only to give an idea.

docker/Dockerfile:
```
FROM ubuntu:20.04  # whatever distro is closest to yours

RUN apt-get update && \
    apt-get install --yes \
        build-essential gdb \
        …more toolchain and dependencies…
RUN adduser ubuntu  # non-privileged build user if not already provided by image
```

docker-compose.yaml:
```
version: "3.7"
services:
  builder:
    build:
      context: docker/
    user: ${DOCKER_UID:-1000}:${DOCKER_GID:-1000}
    volumes:
      - ./build:/home/ubuntu/build
      - ./emacs:/home/ubuntu/emacs
      - /tmp/.X11-unix:/tmp/.X11-unix
```

.env:
```
# If your local user ID or group ID is different from those in the container,
# specify them here
# DOCKER_UID=1001
# DOCKER_GID=1001
```

Makefile:
```
DOCKER_RUN = docker-compose run \
    --rm \
    --user=ubuntu \
    --workdir=/home/ubuntu/emacs \
    builder
CONFIGURE_FLAGS = …

configure:
    $(DOCKER_RUN) ./configure $(CONFIGURE_FLAGS)

build:
    $(DOCKER_RUN) make

debug:
    $(DOCKER_RUN) gdb --args ../build/emacs $(ARGS)
```

Re: Alternatives for reliable build environments for emacs?

[Eli Zaretskii]

> From: joakim@verona.se
> Cc: emacs-devel@gnu.org
> Date: Thu, 28 Oct 2021 16:38:54 +0200
> 
> >> I'm finding it increasingly difficult to build emacs in my distro,
> >> Fedora(Ok, I havent managed to build emacs using distro dependencies for
> >> a long time)
> >
> > Why? what are the difficulties?
> 
> In my case one of the problems that started the downward spiral was the versions shipped of gnutls in
> fedora which  didnt work with emacs.

Which version of GnuTLS was that?

And what was the rest of the spiral?

> The separate aproach of building all the dependencies emacs need in a
> separate build environment does work, like the Guix aproach.

I'd say build all the dependencies that aren't already available, yes.

Re: Alternatives for reliable build environments for emacs?

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: joakim@verona.se
>> Cc: emacs-devel@gnu.org
>> Date: Thu, 28 Oct 2021 16:38:54 +0200
>> 
>> >> I'm finding it increasingly difficult to build emacs in my distro,
>> >> Fedora(Ok, I havent managed to build emacs using distro dependencies for
>> >> a long time)
>> >
>> > Why? what are the difficulties?
>> 
>> In my case one of the problems that started the downward spiral was the versions shipped of gnutls in
>> fedora which  didnt work with emacs.
>
> Which version of GnuTLS was that?

From my notes at the time:
------------------------------------
working emacs:
libgnutls-version is a variable defined in ‘C source code’.

Its value is 30615


Broken emacs:
libgnutls-version is a variable defined in ‘src/gnutls.c’.

Its value is 30702
------------------------------------



>
> And what was the rest of the spiral?

Mostly tripping over my own feet, until I got tired of it and ejected my
entire build system and switched to Guix, but the limitation with guix
is I find it difficult to use as a build system for an emacs source tree
kludged together with duckt-tape.

To be clear, Emacs is high quality software, I'm astonished it builds in
as many circumstances as it does. 


>> The separate aproach of building all the dependencies emacs need in a
>> separate build environment does work, like the Guix aproach.
>
> I'd say build all the dependencies that aren't already available, yes.

Thanks

-- 
Joakim Verona
joakim@verona.se

Re: Alternatives for reliable build environments for emacs?

[Stefan Monnier]

> I'm finding it increasingly difficult to build emacs in my distro,
> Fedora(Ok, I havent managed to build emacs using distro dependencies for
> a long time)

That's a serious problem for Emacs, so please report these as bugs.
I use the kind of "stgit" setup you describe on Debian testing machines
and have none of the problems you describe, so my guess is that Fedora
is a bit more "bleeding edge" and the Emacs code gets adjusted for the
new libs before they reach Debian testing.

Maybe we should have a "bleeding edge Fedora" build VM in EMBA to catch
those problems?


        Stefan

Re: Alternatives for reliable build environments for emacs?

[Tassilo Horn]

Robert Pluim <rpluim@gmail.com> writes:

>     >> Thatʼs the first Iʼve heard of it. GnuTLS too old, too new, something else?
>
>     joakim> Too new in Fedora(at the time before I threw in the towel at least).
>
> OK. Iʼll guess we'll start seeing reports at some point then.

FWIW, Arch ships with gnutls 3.7.2 which is the most current "next"
release and emacs builds and works just fine with it.  libgnutls-version
reports 30702.

What concretely didn't work with that version?

Bye,
Tassilo

Re: Alternatives for reliable build environments for emacs?

[Tim Cross]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'm finding it increasingly difficult to build emacs in my distro,
>> Fedora(Ok, I havent managed to build emacs using distro dependencies for
>> a long time)
>
> That's a serious problem for Emacs, so please report these as bugs.
> I use the kind of "stgit" setup you describe on Debian testing machines
> and have none of the problems you describe, so my guess is that Fedora
> is a bit more "bleeding edge" and the Emacs code gets adjusted for the
> new libs before they reach Debian testing.
>
> Maybe we should have a "bleeding edge Fedora" build VM in EMBA to catch
> those problems?
>

Yes, some more details would be useful. I built emacs on fedora a few
months back without issue. Could it possibly be the issue is not that
Fedora is too bleeding endge or is it that the OP's version is too old
and so they have to install later lib versions than are available? It
would be good to know what version of fedora they are trying to build on.

I've also built emacs on recent arch without issue and arch typically
has pretty bleeding edge libs.

Re: Alternatives for reliable build environments for emacs?

[Daniel Martín]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Thu, 28 Oct 2021 at 19:57, <joakim@verona.se> wrote:
>
>> I'm finding it increasingly difficult to build emacs in my distro,
>> Fedora(Ok, I havent managed to build emacs using distro dependencies for
>> a long time)
>>
>> What are the alternatives?
>> I have tried guix and docker but not yet flatpak.
>>
>> These are nice because you can pin the dependency versions, to make sure
>> you have a reproducible build environment.
>>
>> These all have the drawback that they are inconvenient when making local
>> modifications to the emacs source, at least for me. Way back in the days
>> when my original build environment worked, I automatically merged some
>> branches and applied local patchs with stgit, and its these kind of
>> things I've found awkward with the above mentioned containerized build
>> systems.
>
> I’m not an Emacs developer but the issue is familiar. I use Docker for
> similar cases: (1) building software targeting an older version of the
> distribution than is installed on my host system, or (2) building
> software using newer versions of the toolchain than I’m comfortable
> installing on my host machine.

Emacs is a stable software with few dependencies (by 2021 standards), so
I wouldn't expect Docker or similar tools to be necessary at all, except
in very rare situations.

If there are problems building Emacs in some particular platform, I
think it's better to report the errors instead of sweeping them under
the carpet by using Docker or a similar container technology.  I'd say
they are probably bugs in the build files or the toolchains.

Re: Alternatives for reliable build environments for emacs?

[Pierre Téchoueyres]

Le jeudi 28 oct. 2021 à 16:38, joakim@verona.se a écrit :

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: joakim@verona.se
>>> Date: Thu, 28 Oct 2021 14:55:27 +0200
>>> 
>>> I'm finding it increasingly difficult to build emacs in my distro,
>>> Fedora(Ok, I havent managed to build emacs using distro dependencies for
>>> a long time)
>>
>> Why? what are the difficulties?
>
> In my case one of the problems that started the downward spiral was the versions shipped of gnutls in
> fedora which  didnt work with emacs.
>
Could you elaborate on what's didn't work when you build Emacs with the
library provided by the Fedora package manager, please ?

For what it worth :
Here I'm on Fedora 34, building Emacs regularly from GIT (branches
27/28/master) and everything is working as expected. I could update
packages, use gnus with gmane, use elfeed etc.

"Here it's work" ™ © Me :-)

> The separate aproach of building all the dependencies emacs need in a
> separate build environment does work, like the Guix aproach.
>
> I could probably convince guix to do what I want, but guix is also
> fiddly. I was just wondering if somebody else had a solution for a
> reproducible development environment for emacs.
>
> Maybe I'm the only one facing these difficulties, that's fine I guess.

Re: Alternatives for reliable build environments for emacs?

[joakim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'm finding it increasingly difficult to build emacs in my distro,
>> Fedora(Ok, I havent managed to build emacs using distro dependencies for
>> a long time)
>
> That's a serious problem for Emacs, so please report these as bugs.
> I use the kind of "stgit" setup you describe on Debian testing machines
> and have none of the problems you describe, so my guess is that Fedora
> is a bit more "bleeding edge" and the Emacs code gets adjusted for the
> new libs before they reach Debian testing.
>
> Maybe we should have a "bleeding edge Fedora" build VM in EMBA to catch
> those problems?

It seems my original mail somehow came out wrong.

I neglected to add this important piece of information:

My local build system is a complete insane mess, and I like it that way,
however, is there some other way to make other interesting insane build
systems for emacs, that also actually work and are productive?

Without this information it sounds like everything is broken except my
build, which isnt true.

With that said, I still think its reasonable to isolate builds to have
different dependency versions, in docker, chroots, guix, etc.

Sometimes you want to have newer, or older, versions of dependencies to
try things out.

>
>
>         Stefan
>
>
-- 
Joakim Verona
joakim@verona.se

Re: Alternatives for reliable build environments for emacs?

[Yuchen Pei]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]


joakim@verona.se writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: joakim@verona.se
>>> Date: Thu, 28 Oct 2021 14:55:27 +0200
>>> 
>>> I'm finding it increasingly difficult to build emacs in my 
>>> distro,
>>> Fedora(Ok, I havent managed to build emacs using distro 
>>> dependencies for
>>> a long time)
>>
>> Why? what are the difficulties?
>
> In my case one of the problems that started the downward spiral 
> was the versions shipped of gnutls in
> fedora which  didnt work with emacs.
>
> The separate aproach of building all the dependencies emacs need 
> in a
> separate build environment does work, like the Guix aproach.
>
> I could probably convince guix to do what I want, but guix is 
> also
> fiddly. I was just wondering if somebody else had a solution for 
> a
> reproducible development environment for emacs.
>
> Maybe I'm the only one facing these difficulties, that's fine I 
> guess.

I was trying to build emacs on a debian + guix system today.  The 
debian is bare minimum, and guix is used as the main package 
manager.

Without `guix shell`, I got stuck at problem with gnutls:

> configure: error: The following required libraries were not 
> found:
>      gnutls
> Maybe some development libraries/packages are missing?

The version of gnutls installed by guix in the system is 3.6.15.

With `guix shell --development emacs`, gnutls wasn't a problem, 
but I got stuck at segfault at bootstrap-emacs.pdmp:

> make[1]: *** [Makefile:858: bootstrap-emacs.pdmp] Segmentation 
> fault
> make[1]: Leaving directory '/home/foo/source/emacs/src'
> make: *** [Makefile:450: src] Error 2

On an Arch system where I used to be able to build emacs, it 
doesn't build either, and the error is about Recursive load in 
international/mule-cmds.el.  I haven't looked into this error 
because it is a slow system and I'd rather build emacs on the 
debian + guix system.

-- 
Best,
Yuchen

PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
           <https://ypei.me/assets/ypei-pubkey.txt>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 243 bytes --]

Re: Alternatives for reliable build environments for emacs?

[Eli Zaretskii]

> From: Yuchen Pei <hi@ypei.me>
> Date: Fri, 29 Oct 2021 11:29:39 +1100
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> I was trying to build emacs on a debian + guix system today.  The 
> debian is bare minimum, and guix is used as the main package 
> manager.
> 
> Without `guix shell`, I got stuck at problem with gnutls:
> 
> > configure: error: The following required libraries were not 
> > found:
> >      gnutls
> > Maybe some development libraries/packages are missing?
> 
> The version of gnutls installed by guix in the system is 3.6.15.

It tells you to make sure you have all the development packages
installed, and I think it's the accurate advice.  People build Emacs
with that version of GnuTLS without any problems.

> With `guix shell --development emacs`, gnutls wasn't a problem, 
> but I got stuck at segfault at bootstrap-emacs.pdmp:
> 
> > make[1]: *** [Makefile:858: bootstrap-emacs.pdmp] Segmentation 
> > fault
> > make[1]: Leaving directory '/home/foo/source/emacs/src'
> > make: *** [Makefile:450: src] Error 2

If you want help in solving this problem, please submit a bug report
with all the details, including at least the configure command and the
full log of the build from the configure step till the segfault.
config.log would also help.

Thanks.

Re: Alternatives for reliable build environments for emacs?

[Po Lu]

joakim@verona.se writes:

> Too new in Fedora(at the time before I threw in the towel at least).

FWIW, Emacs builds fine for me on Fedora 34, with the latest version of
gnutls it includes (3.7.2).

Re: Alternatives for reliable build environments for emacs?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm finding it increasingly difficult to build emacs in my distro,
  > Fedora(Ok, I havent managed to build emacs using distro dependencies for
  > a long time)

  > In my case one of the problems that started the downward spiral
  > was the versions shipped of gnutls in fedora which didnt work with
  > emacs.

  > > Thatʼs the first Iʼve heard of it. GnuTLS too old, too new, something else?

  > Too new in Fedora(at the time before I threw in the towel at least).

We want to keep Emacs's build procedure working properly on all the systems
it is supposed to support.  Would you please help us do that, by sending
detailed bug reports about each problem that you encounter?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Alternatives for reliable build environments for emacs?

[Yuchen Pei]

[-- Attachment #1: Type: text/plain, Size: 1755 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yuchen Pei <hi@ypei.me>
>> Date: Fri, 29 Oct 2021 11:29:39 +1100
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> 
>> I was trying to build emacs on a debian + guix system today. 
>> The 
>> debian is bare minimum, and guix is used as the main package 
>> manager.
>> 
>> Without `guix shell`, I got stuck at problem with gnutls:
>> 
>> > configure: error: The following required libraries were not 
>> > found:
>> >      gnutls
>> > Maybe some development libraries/packages are missing?
>> 
>> The version of gnutls installed by guix in the system is 
>> 3.6.15.
>
> It tells you to make sure you have all the development packages
> installed, and I think it's the accurate advice.  People build 
> Emacs
> with that version of GnuTLS without any problems.

I managed to resolve this by installing gnutls-dane.

>
>> With `guix shell --development emacs`, gnutls wasn't a problem, 
>> but I got stuck at segfault at bootstrap-emacs.pdmp:
>> 
>> > make[1]: *** [Makefile:858: bootstrap-emacs.pdmp] 
>> > Segmentation 
>> > fault
>> > make[1]: Leaving directory '/home/foo/source/emacs/src'
>> > make: *** [Makefile:450: src] Error 2
>
> If you want help in solving this problem, please submit a bug 
> report
> with all the details, including at least the configure command 
> and the
> full log of the build from the configure step till the segfault.
> config.log would also help.

This was resolved as in
<https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg02216.html>
<https://lists.gnu.org/archive/html/emacs-devel/2021-10/msg02247.html>.

>
> Thanks.


-- 
Best,
Yuchen

PGP Key: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0
           <https://ypei.me/assets/ypei-pubkey.txt>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 243 bytes --]