MPS: Win64 testers?

MPS: Win64 testers?

[Pip Cet]

Is anyone building the scratch/igc branch for/on win64 systems? I'm trying to do that on wine/msys2/mingw64 (not an actual Windows installation), and there are a number of issues that need to be fixed, and I'm wondering whether I'm the first person to run into them. Quite possibly, they're specific to wine.

- mps doesn't compile with GCC (and, when it does, needs to be told that a machine word is unsigned long long, not unsigned long)
- _setjmp_ex assumes 16-byte alignment of jmp_buf to store the XMM registers, but we allocate handlers with 8-byte alignment in MPS builds.

Is this a wine bug, or is the MPS build currently non-functional on win64?

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Wed, 24 Jul 2024 15:45:18 +0000
> From: Pip Cet <pipcet@protonmail.com>
> 
> Is anyone building the scratch/igc branch for/on win64 systems? I'm trying to do that on wine/msys2/mingw64 (not an actual Windows installation), and there are a number of issues that need to be fixed, and I'm wondering whether I'm the first person to run into them. Quite possibly, they're specific to wine.

I think you are indeed the first one.  I use 32-bit MinGW, and for
that I have a working build of MPS (and can share the diffs if someone
wants them), but a 64-bit MinGW build of MPS will need other changes,
at least in some cases.

> - mps doesn't compile with GCC (and, when it does, needs to be told that a machine word is unsigned long long, not unsigned long)

Right.  I wonder why they didn't use intprtr_t or somesuch to begin
with.

> - _setjmp_ex assumes 16-byte alignment of jmp_buf to store the XMM registers, but we allocate handlers with 8-byte alignment in MPS builds.

Where and how do you see that?

Re: MPS: Win64 testers?

[Pip Cet]

On Thursday, July 25th, 2024 at 07:16, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Wed, 24 Jul 2024 15:45:18 +0000
> 
> > From: Pip Cet pipcet@protonmail.com
> >
> > Is anyone building the scratch/igc branch for/on win64 systems? I'm trying to do that on wine/msys2/mingw64 (not an actual Windows installation), and there are a number of issues that need to be fixed, and I'm wondering whether I'm the first person to run into them. Quite possibly, they're specific to wine.
> 
> I think you are indeed the first one. I use 32-bit MinGW, and for
> that I have a working build of MPS (and can share the diffs if someone
> wants them), but a 64-bit MinGW build of MPS will need other changes,
> at least in some cases.

It'd be great if you could share the diffs, so I could see where I've gone wrong! I've got the 64-bit build with mps to start by now, but haven't really tested it for stability much. It survives a few garbage collection cycles, at least.

As a general question, would there be interest in CI for such systems? I'm setting up something locally, but I've been trying to be careful so that the Dockerfiles I'm using should work on x86_64 GNU/Linux machines generally.

> > - _setjmp_ex assumes 16-byte alignment of jmp_buf to store the XMM registers, but we allocate handlers with 8-byte alignment in MPS builds.
> 
> Where and how do you see that?

I'm not quite sure what you're asking. I saw this trying to run things in wine in a docker image on an x86_64 GNU/Linux machine, ran it in gdb.exe, found the instruction causing the segfault, and fixed things so it wouldn't.

https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/winecrt0/setjmp.c#L123 is the instruction that segfaulted here. I'm not sure whether that's what you were asking.

The jump buffer is declared with 16-byte alignment here: https://gitlab.winehq.org/wine/wine/-/blob/master/include/msvcrt/setjmp.h#L50

A side effect of this is that LISP_ALIGNMENT is 16, which breaks MPS because vectorlike_nbytes rounds up to a multiple of LISP_ALIGNMENT, not just GCALIGNMENT.

Looking over my diff, here's what I had to fix to get things to the current stage on msys and msys2 (using MSYSTEM=MINGW64).

- disable the sanity check in the Makefile, because backticks/pipes don't work properly
- make WINDRES include the --use-temp-file flag, for the same reason
- wrap the add-name-to-file calls in loadup.el in with-demoted-errors (link problems?)
- force LISP_ALIGNMENT to be 8, not 16
- change my local generation size settings to be much more conservative for the msys build. I think scratch/igc is fine, though.
- make sys_setjmp and sys_longjmp use a larger buffer and memmove() the data in it so it's 16-byte aligned
- disable the failure exit on close_stream failure in sysdep.c, and silently ignore such errors
- trace the extra 'ztrillion' bignum in timefns.c
- for the msys build, run autogen.sh outside of wine, since there are virtual memory issues inflooping the process otherwise

I must confess I know little about the UCRT64 default MSYSTEM value. Maybe it doesn't have the same problems.

If anyone wants to study or reuse any of my efforts, let me know and I'll provide a link to a public repository that can be used. Note that my Wine build environment, as far as I know, uses only free software, so it's in the "technically free OS" category. (The Android build uses the SDK, which claims to be decidedly non-free but makes an exception for those components which are indeed free; that means it's possible to build Android APKs without paying for it, using non-free software, and run them on technically free Android versions if you can find those. AFAIK, the macOS build requires non-free and expensive software both to compile and to run the executable?)

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Thu, 25 Jul 2024 15:39:51 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Thursday, July 25th, 2024 at 07:16, Eli Zaretskii <eliz@gnu.org> wrote:
> > I think you are indeed the first one. I use 32-bit MinGW, and for
> > that I have a working build of MPS (and can share the diffs if someone
> > wants them), but a 64-bit MinGW build of MPS will need other changes,
> > at least in some cases.
> 
> It'd be great if you could share the diffs, so I could see where I've gone wrong! I've got the 64-bit build with mps to start by now, but haven't really tested it for stability much. It survives a few garbage collection cycles, at least.

My diffs are attached below.  Note that they are for the 32-bit MinGW
build, so not all of them are relevant for a 64-bit MinGW build, and
some stuff for a 64-bit build is missing and you will have to add it.
For example, the code/w3i3gc.gmk file is only for the 32-bit build;
you will need to create code/w3i6gc.gmk instead.  You'll also need to
add the 64-bit MinGW fragment to code/mpstd.h (where I added the stuff
for the 32-bit MinGW).

To build, I did the following from the MSYS Bash prompt:

  $ cd code
  $ make -f w3i3gc.gmk

To run the test suite, I did the following from the cmd window:

  d:\usr\...\mps-release-1.118.0\code>..\tool\testrun.bat w3i3gc cool testrun

(where "..." stands for the specific place I have the sources, which
is immaterial for the purposes of this discussion).

To install, say from the MSYS Bash:

  $ make install

> As a general question, would there be interest in CI for such systems? I'm setting up something locally, but I've been trying to be careful so that the Dockerfiles I'm using should work on x86_64 GNU/Linux machines generally.

I think it will be useful, but I don't know how much effort it will
need from you, so I cannot say if it's justified at this stage.

> > > - _setjmp_ex assumes 16-byte alignment of jmp_buf to store the XMM registers, but we allocate handlers with 8-byte alignment in MPS builds.
> > 
> > Where and how do you see that?
> 
> I'm not quite sure what you're asking. I saw this trying to run things in wine in a docker image on an x86_64 GNU/Linux machine, ran it in gdb.exe, found the instruction causing the segfault, and fixed things so it wouldn't.
> 
> https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/winecrt0/setjmp.c#L123 is the instruction that segfaulted here. I'm not sure whether that's what you were asking.
> 
> The jump buffer is declared with 16-byte alignment here: https://gitlab.winehq.org/wine/wine/-/blob/master/include/msvcrt/setjmp.h#L50
> 
> A side effect of this is that LISP_ALIGNMENT is 16, which breaks MPS because vectorlike_nbytes rounds up to a multiple of LISP_ALIGNMENT, not just GCALIGNMENT.

If the "_setjmp_ex assumes 16-byte alignment of jmp_buf to store the
XMM registers" part is about what Wine does, then my question is where
did you see that "we allocate handlers with 8-byte alignment in MPS
builds"?

> Looking over my diff, here's what I had to fix to get things to the current stage on msys and msys2 (using MSYSTEM=MINGW64).
> 
> - disable the sanity check in the Makefile, because backticks/pipes don't work properly

How so? what doesn't work?  By "makefile", which Makefile do you mean?
the one in Emacs or the one in MPS?

> - make WINDRES include the --use-temp-file flag, for the same reason

Is this in Emacs Makefiles?  I guess Wine is not a good emulation of
Windows, then.

> - wrap the add-name-to-file calls in loadup.el in with-demoted-errors (link problems?)

I have no idea.  Maybe hard links are not properly emulated by Wine?

> - force LISP_ALIGNMENT to be 8, not 16

Does this mean 'struct Lisp_*' structures are 16-byte aligned in the
64-bit MinGW build?  If not, what forces LISP_ALIGNMENT to be 16?

> - change my local generation size settings to be much more conservative for the msys build. I think scratch/igc is fine, though.

Don't understand what this means, sorry.  What are your "local
generation size settings"?

> - make sys_setjmp and sys_longjmp use a larger buffer and memmove() the data in it so it's 16-byte aligned

Why would this be any different from a non-MPS build for 64-bit MinGW?

> - disable the failure exit on close_stream failure in sysdep.c, and silently ignore such errors

I guess you are using UCRT?  These problems with UCRT are known, but
no one came up with an explanation for them yet.  What happens if you
link against MSVCRT instead?

> - trace the extra 'ztrillion' bignum in timefns.c

What happens if you don't?

> - for the msys build, run autogen.sh outside of wine, since there are virtual memory issues inflooping the process otherwise

Sounds like Wine is not a good idea at all...

Here are the diffs I promised.

--- ./code/comm.gmk~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/comm.gmk	2024-04-23 15:07:15.288562200 +0300
@@ -72,7 +72,7 @@ endif
 
 # TELEMETRY TARGETS
 
-EVENT_TARGETS = mpseventcnv mpseventpy mpseventsql mpseventtxt
+EVENT_TARGETS = mpseventcnv$(EXEEXT) mpseventpy$(EXEEXT) mpseventsql$(EXEEXT) mpseventtxt$(EXEEXT)
 
 
 # EXTRA TARGETS
@@ -80,7 +80,7 @@ EVENT_TARGETS = mpseventcnv mpseventpy m
 # Don't build mpseventsql by default (might not have sqlite3 installed),
 # but do build the other event target.
 
-EXTRA_TARGETS ?= $(filter-out mpseventsql,$(EVENT_TARGETS))
+EXTRA_TARGETS ?= $(filter-out mpseventsql$(EXEEXT),$(EVENT_TARGETS))
 
 
 #
@@ -161,7 +161,11 @@ POOLN = pooln.c
 MV2 = poolmv2.c
 MVFF = poolmvff.c
 TESTLIB = testlib.c
+ifneq ($(PFM),w3i3gc)
 TESTTHR = testthrix.c
+else
+TESTTHR = testthrw3.c
+endif
 FMTDY = fmtdy.c fmtno.c
 FMTDYTST = fmtdy.c fmtno.c fmtdytst.c
 FMTHETST = fmthe.c fmtdy.c fmtno.c fmtdytst.c
@@ -253,57 +257,61 @@ LIB_TARGETS=mps.a mpsplan.a
 # Test executables go in TEST_TARGETS.
 
 TEST_TARGETS=\
-    abqtest \
-    addrobj \
-    airtest \
-    amcss \
-    amcsshe \
-    amcssth \
-    amsss \
-    amssshe \
-    apss \
-    arenacv \
-    awlut \
-    awluthe \
-    awlutth \
-    btcv \
-    bttest \
-    djbench \
-    extcon \
-    finalcv \
-    finaltest \
-    forktest \
-    fotest \
-    gcbench \
-    landtest \
-    locbwcss \
-    lockcov \
-    lockut \
-    locusss \
-    locv \
-    messtest \
-    mpmss \
-    mpsicv \
-    mv2test \
-    nailboardtest \
-    poolncv \
-    qs \
-    sacss \
-    segsmss \
-    sncss \
-    steptest \
-    tagtest \
-    teletest \
-    walkt0 \
-    zcoll \
-    zmess \
-    ztfm
+    abqtest$(EXEEXT) \
+    addrobj$(EXEEXT) \
+    airtest$(EXEEXT) \
+    amcss$(EXEEXT) \
+    amcsshe$(EXEEXT) \
+    amcssth$(EXEEXT) \
+    amsss$(EXEEXT) \
+    amssshe$(EXEEXT) \
+    apss$(EXEEXT) \
+    arenacv$(EXEEXT) \
+    awlut$(EXEEXT) \
+    awluthe$(EXEEXT) \
+    awlutth$(EXEEXT) \
+    btcv$(EXEEXT) \
+    bttest$(EXEEXT) \
+    djbench$(EXEEXT) \
+    extcon$(EXEEXT) \
+    finalcv$(EXEEXT) \
+    finaltest$(EXEEXT) \
+    fotest$(EXEEXT) \
+    gcbench$(EXEEXT) \
+    landtest$(EXEEXT) \
+    locbwcss$(EXEEXT) \
+    lockcov$(EXEEXT) \
+    lockut$(EXEEXT) \
+    locusss$(EXEEXT) \
+    locv$(EXEEXT) \
+    messtest$(EXEEXT) \
+    mpmss$(EXEEXT) \
+    mpsicv$(EXEEXT) \
+    mv2test$(EXEEXT) \
+    nailboardtest$(EXEEXT) \
+    poolncv$(EXEEXT) \
+    qs$(EXEEXT) \
+    sacss$(EXEEXT) \
+    segsmss$(EXEEXT) \
+    sncss$(EXEEXT) \
+    steptest$(EXEEXT) \
+    tagtest$(EXEEXT) \
+    teletest$(EXEEXT) \
+    walkt0$(EXEEXT) \
+    zcoll$(EXEEXT) \
+    zmess$(EXEEXT) \
+    ztfm$(EXEEXT)
+
+ifneq ($(PFM),w3i3gc)
+TEST_TARGETS = $(TEST_TARGETS) forktest$(EXEEXT) 
+
+endif
 
 # This target records programs that we were once able to build but
 # can't at the moment:
 
 UNBUILDABLE_TARGETS=\
-    replay # depends on the EPVM pool
+    replay$(EXEEXT) # depends on the EPVM pool
 
 ALL_TARGETS=$(LIB_TARGETS) $(TEST_TARGETS) $(EXTRA_TARGETS)
 
@@ -446,154 +454,154 @@ $(PFM)/cool/mps.a: $(MPMOBJ)
 
 ifdef VARIETY
 
-$(PFM)/$(VARIETY)/abqtest: $(PFM)/$(VARIETY)/abqtest.o \
+$(PFM)/$(VARIETY)/abqtest$(EXEEXT): $(PFM)/$(VARIETY)/abqtest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/addrobj: $(PFM)/$(VARIETY)/addrobj.o \
+$(PFM)/$(VARIETY)/addrobj$(EXEEXT): $(PFM)/$(VARIETY)/addrobj.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/airtest: $(PFM)/$(VARIETY)/airtest.o \
+$(PFM)/$(VARIETY)/airtest$(EXEEXT): $(PFM)/$(VARIETY)/airtest.o \
 	$(FMTSCMOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/amcss: $(PFM)/$(VARIETY)/amcss.o \
+$(PFM)/$(VARIETY)/amcss$(EXEEXT): $(PFM)/$(VARIETY)/amcss.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/amcsshe: $(PFM)/$(VARIETY)/amcsshe.o \
+$(PFM)/$(VARIETY)/amcsshe$(EXEEXT): $(PFM)/$(VARIETY)/amcsshe.o \
 	$(FMTHETSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/amcssth: $(PFM)/$(VARIETY)/amcssth.o \
+$(PFM)/$(VARIETY)/amcssth$(EXEEXT): $(PFM)/$(VARIETY)/amcssth.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(TESTTHROBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/amsss: $(PFM)/$(VARIETY)/amsss.o \
+$(PFM)/$(VARIETY)/amsss$(EXEEXT): $(PFM)/$(VARIETY)/amsss.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/amssshe: $(PFM)/$(VARIETY)/amssshe.o \
+$(PFM)/$(VARIETY)/amssshe$(EXEEXT): $(PFM)/$(VARIETY)/amssshe.o \
 	$(FMTHETSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/apss: $(PFM)/$(VARIETY)/apss.o \
+$(PFM)/$(VARIETY)/apss$(EXEEXT): $(PFM)/$(VARIETY)/apss.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/arenacv: $(PFM)/$(VARIETY)/arenacv.o \
+$(PFM)/$(VARIETY)/arenacv$(EXEEXT): $(PFM)/$(VARIETY)/arenacv.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/awlut: $(PFM)/$(VARIETY)/awlut.o \
+$(PFM)/$(VARIETY)/awlut$(EXEEXT): $(PFM)/$(VARIETY)/awlut.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(TESTTHROBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/awluthe: $(PFM)/$(VARIETY)/awluthe.o \
+$(PFM)/$(VARIETY)/awluthe$(EXEEXT): $(PFM)/$(VARIETY)/awluthe.o \
         $(FMTHETSTOBJ) $(TESTLIBOBJ) $(TESTTHROBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/awlutth: $(PFM)/$(VARIETY)/awlutth.o \
+$(PFM)/$(VARIETY)/awlutth$(EXEEXT): $(PFM)/$(VARIETY)/awlutth.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(TESTTHROBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/btcv: $(PFM)/$(VARIETY)/btcv.o \
+$(PFM)/$(VARIETY)/btcv$(EXEEXT): $(PFM)/$(VARIETY)/btcv.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/bttest: $(PFM)/$(VARIETY)/bttest.o \
+$(PFM)/$(VARIETY)/bttest$(EXEEXT): $(PFM)/$(VARIETY)/bttest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/djbench: $(PFM)/$(VARIETY)/djbench.o \
+$(PFM)/$(VARIETY)/djbench$(EXEEXT): $(PFM)/$(VARIETY)/djbench.o \
 	$(TESTLIBOBJ) $(TESTTHROBJ)
 
-$(PFM)/$(VARIETY)/extcon: $(PFM)/$(VARIETY)/extcon.o \
+$(PFM)/$(VARIETY)/extcon$(EXEEXT): $(PFM)/$(VARIETY)/extcon.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/finalcv: $(PFM)/$(VARIETY)/finalcv.o \
+$(PFM)/$(VARIETY)/finalcv$(EXEEXT): $(PFM)/$(VARIETY)/finalcv.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/finaltest: $(PFM)/$(VARIETY)/finaltest.o \
+$(PFM)/$(VARIETY)/finaltest$(EXEEXT): $(PFM)/$(VARIETY)/finaltest.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/forktest: $(PFM)/$(VARIETY)/forktest.o \
+$(PFM)/$(VARIETY)/forktest$(EXEEXT): $(PFM)/$(VARIETY)/forktest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/fotest: $(PFM)/$(VARIETY)/fotest.o \
+$(PFM)/$(VARIETY)/fotest$(EXEEXT): $(PFM)/$(VARIETY)/fotest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/gcbench: $(PFM)/$(VARIETY)/gcbench.o \
+$(PFM)/$(VARIETY)/gcbench$(EXEEXT): $(PFM)/$(VARIETY)/gcbench.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(TESTTHROBJ)
 
-$(PFM)/$(VARIETY)/landtest: $(PFM)/$(VARIETY)/landtest.o \
+$(PFM)/$(VARIETY)/landtest$(EXEEXT): $(PFM)/$(VARIETY)/landtest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/locbwcss: $(PFM)/$(VARIETY)/locbwcss.o \
+$(PFM)/$(VARIETY)/locbwcss$(EXEEXT): $(PFM)/$(VARIETY)/locbwcss.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/lockcov: $(PFM)/$(VARIETY)/lockcov.o \
+$(PFM)/$(VARIETY)/lockcov$(EXEEXT): $(PFM)/$(VARIETY)/lockcov.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/lockut: $(PFM)/$(VARIETY)/lockut.o \
+$(PFM)/$(VARIETY)/lockut$(EXEEXT): $(PFM)/$(VARIETY)/lockut.o \
 	$(TESTLIBOBJ) $(TESTTHROBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/locusss: $(PFM)/$(VARIETY)/locusss.o \
+$(PFM)/$(VARIETY)/locusss$(EXEEXT): $(PFM)/$(VARIETY)/locusss.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/locv: $(PFM)/$(VARIETY)/locv.o \
+$(PFM)/$(VARIETY)/locv$(EXEEXT): $(PFM)/$(VARIETY)/locv.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/messtest: $(PFM)/$(VARIETY)/messtest.o \
+$(PFM)/$(VARIETY)/messtest$(EXEEXT): $(PFM)/$(VARIETY)/messtest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/mpmss: $(PFM)/$(VARIETY)/mpmss.o \
+$(PFM)/$(VARIETY)/mpmss$(EXEEXT): $(PFM)/$(VARIETY)/mpmss.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/mpsicv: $(PFM)/$(VARIETY)/mpsicv.o \
+$(PFM)/$(VARIETY)/mpsicv$(EXEEXT): $(PFM)/$(VARIETY)/mpsicv.o \
 	$(FMTDYTSTOBJ) $(FMTHETSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/mv2test: $(PFM)/$(VARIETY)/mv2test.o \
+$(PFM)/$(VARIETY)/mv2test$(EXEEXT): $(PFM)/$(VARIETY)/mv2test.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/nailboardtest: $(PFM)/$(VARIETY)/nailboardtest.o \
+$(PFM)/$(VARIETY)/nailboardtest$(EXEEXT): $(PFM)/$(VARIETY)/nailboardtest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/poolncv: $(PFM)/$(VARIETY)/poolncv.o \
+$(PFM)/$(VARIETY)/poolncv$(EXEEXT): $(PFM)/$(VARIETY)/poolncv.o \
 	$(POOLNOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/qs: $(PFM)/$(VARIETY)/qs.o \
+$(PFM)/$(VARIETY)/qs$(EXEEXT): $(PFM)/$(VARIETY)/qs.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/sacss: $(PFM)/$(VARIETY)/sacss.o \
+$(PFM)/$(VARIETY)/sacss$(EXEEXT): $(PFM)/$(VARIETY)/sacss.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/segsmss: $(PFM)/$(VARIETY)/segsmss.o \
+$(PFM)/$(VARIETY)/segsmss$(EXEEXT): $(PFM)/$(VARIETY)/segsmss.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/sncss: $(PFM)/$(VARIETY)/sncss.o \
+$(PFM)/$(VARIETY)/sncss$(EXEEXT): $(PFM)/$(VARIETY)/sncss.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/steptest: $(PFM)/$(VARIETY)/steptest.o \
+$(PFM)/$(VARIETY)/steptest$(EXEEXT): $(PFM)/$(VARIETY)/steptest.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/tagtest: $(PFM)/$(VARIETY)/tagtest.o \
+$(PFM)/$(VARIETY)/tagtest$(EXEEXT): $(PFM)/$(VARIETY)/tagtest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/teletest: $(PFM)/$(VARIETY)/teletest.o \
+$(PFM)/$(VARIETY)/teletest$(EXEEXT): $(PFM)/$(VARIETY)/teletest.o \
 	$(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/walkt0: $(PFM)/$(VARIETY)/walkt0.o \
+$(PFM)/$(VARIETY)/walkt0$(EXEEXT): $(PFM)/$(VARIETY)/walkt0.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/zcoll: $(PFM)/$(VARIETY)/zcoll.o \
+$(PFM)/$(VARIETY)/zcoll$(EXEEXT): $(PFM)/$(VARIETY)/zcoll.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/zmess: $(PFM)/$(VARIETY)/zmess.o \
+$(PFM)/$(VARIETY)/zmess$(EXEEXT): $(PFM)/$(VARIETY)/zmess.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/ztfm: $(PFM)/$(VARIETY)/ztfm.o \
+$(PFM)/$(VARIETY)/ztfm$(EXEEXT): $(PFM)/$(VARIETY)/ztfm.o \
 	$(FMTDYTSTOBJ) $(TESTLIBOBJ) $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/mpseventcnv: $(PFM)/$(VARIETY)/eventcnv.o \
+$(PFM)/$(VARIETY)/mpseventcnv$(EXEEXT): $(PFM)/$(VARIETY)/eventcnv.o \
   $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/mpseventpy: $(PFM)/$(VARIETY)/eventpy.o \
+$(PFM)/$(VARIETY)/mpseventpy$(EXEEXT): $(PFM)/$(VARIETY)/eventpy.o \
   $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/mpseventtxt: $(PFM)/$(VARIETY)/eventtxt.o \
+$(PFM)/$(VARIETY)/mpseventtxt$(EXEEXT): $(PFM)/$(VARIETY)/eventtxt.o \
   $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/mpseventsql: $(PFM)/$(VARIETY)/eventsql.o \
+$(PFM)/$(VARIETY)/mpseventsql$(EXEEXT): $(PFM)/$(VARIETY)/eventsql.o \
   $(PFM)/$(VARIETY)/mps.a
 
-$(PFM)/$(VARIETY)/replay: $(PFM)/$(VARIETY)/replay.o \
+$(PFM)/$(VARIETY)/replay$(EXEEXT): $(PFM)/$(VARIETY)/replay.o \
   $(PFM)/$(VARIETY)/eventrep.o \
   $(PFM)/$(VARIETY)/table.o \
   $(PFM)/$(VARIETY)/mps.a
@@ -656,18 +664,18 @@ ifdef TARGET
 # %%VARIETY: When adding a new variety, add the dependencies files for it
 # here.
 ifeq ($(VARIETY),rash)
-include $(PFM)/$(VARIETY)/mps.d
+-include $(PFM)/$(VARIETY)/mps.d
 else
 ifeq ($(VARIETY),hot)
-include $(PFM)/$(VARIETY)/mps.d
+-include $(PFM)/$(VARIETY)/mps.d
 else
-include $(MPM:%.c=$(PFM)/$(VARIETY)/%.d)
+-include $(MPM:%.c=$(PFM)/$(VARIETY)/%.d)
 endif # VARIETY != hot
 endif # VARIETY != rash
 
 # %%PART: When adding a new part, add the dependencies file for the
 # new part here.
-include \
+-include \
     $(FMTDY:%.c=$(PFM)/$(VARIETY)/%.d) \
     $(FMTDYTST:%.c=$(PFM)/$(VARIETY)/%.d) \
     $(FMTHETST:%.c=$(PFM)/$(VARIETY)/%.d) \
@@ -698,11 +706,11 @@ $(PFM)/$(VARIETY)/%.a:
 
 # Executable
 
-$(PFM)/$(VARIETY)/%:
+$(PFM)/$(VARIETY)/%$(EXEEXT):
 	$(ECHO) "$(PFM): $@"
 	$(CC) $(CFLAGSSTRICT) $(LINKFLAGS) -o $@ $^ $(LIBS)
 
-$(PFM)/$(VARIETY)/mpseventsql:
+$(PFM)/$(VARIETY)/mpseventsql$(EXEEXT):
 	$(ECHO) "$(PFM): $@"
 	$(CC) $(CFLAGSLAX) $(LINKFLAGS) -o $@ $^ $(LIBS) -lsqlite3
 
--- ./code/lockw3.c~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/lockw3.c	2024-04-23 13:11:22.497139900 +0300
@@ -33,6 +33,25 @@ SRCID(lockw3, "$Id$");
 
 #if defined(LOCK)
 
+#ifdef __MINGW32__
+# if defined __MINGW32_VERSION && __MINGW32_VERSION >= 5000000L
+
+   /* mingw.org's MinGW doesn't have this stuff in its headers.  */
+   typedef struct _RTL_RUN_ONCE { PVOID Ptr; } RTL_RUN_ONCE, *PRTL_RUN_ONCE;
+   typedef DWORD (WINAPI *PRTL_RUN_ONCE_INIT_FN)(PRTL_RUN_ONCE, PVOID, PVOID *);
+
+#  define RTL_RUN_ONCE_INIT {0}
+#  define INIT_ONCE_STATIC_INIT RTL_RUN_ONCE_INIT
+
+   typedef RTL_RUN_ONCE INIT_ONCE;
+   typedef PRTL_RUN_ONCE PINIT_ONCE;
+   typedef WINBOOL (WINAPI *PINIT_ONCE_FN) (PINIT_ONCE InitOnce, PVOID Parameter, PVOID *Context);
+
+   WINBASEAPI WINBOOL WINAPI InitOnceExecuteOnce (PINIT_ONCE InitOnce, PINIT_ONCE_FN InitFn, PVOID Parameter, LPVOID *Context);
+
+# endif	/* __MINGW32_VERSION >= 5000000L */
+#endif	/* __MINGW32__ */
+
 /* .lock.win32: Win32 lock structure; uses CRITICAL_SECTION */
 typedef struct LockStruct {
   Sig sig;                      /* design.mps.sig.field */
--- ./code/mps.c~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/mps.c	2024-04-23 13:56:08.221252500 +0300
@@ -228,9 +228,9 @@
 #include "prmclii6.c"   /* x86-64 for Linux mutator context */
 #include "span.c"       /* generic stack probe */
 
-/* Windows on IA-32 with Microsoft Visual Studio or Pelles C */
+/* Windows on IA-32 with Microsoft Visual Studio or Pelles C or GCC */
 
-#elif defined(MPS_PF_W3I3MV) || defined(MPS_PF_W3I3PC)
+#elif defined(MPS_PF_W3I3MV) || defined(MPS_PF_W3I3PC) || defined(MPS_PF_W3I3GC)
 
 #include "lockw3.c"     /* Windows locks */
 #include "thw3.c"       /* Windows threading */
--- ./code/mpsiw3.c~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/mpsiw3.c	2024-04-23 13:45:23.212499000 +0300
@@ -20,6 +20,11 @@ SRCID(mpsiw3, "$Id$");
 /* This is defined in protw3.c */
 extern LONG WINAPI ProtSEHfilter(LPEXCEPTION_POINTERS info);
 
+/* These seem to be unused, but MinGW GCC complains about lack of
+   previous prototype.  */
+LONG mps_SEH_filter(LPEXCEPTION_POINTERS, void **, size_t *);
+void mps_SEH_handler(void *, size_t);
+
 LONG mps_SEH_filter(LPEXCEPTION_POINTERS info,
                     void **hp_o, size_t *hs_o)
 {
--- ./code/mpstd.h~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/mpstd.h	2024-04-23 17:07:42.606079400 +0300
@@ -392,6 +392,22 @@
 #define MPS_PF_ALIGN    8
 
 
+#elif defined(__MINGW32__) && defined(_X86_) && defined(__GNUC__)
+#if defined(CONFIG_PF_STRING) && ! defined(CONFIG_PF_W3I3GC)
+#error "specified CONFIG_PF_... inconsistent with detected w3i3gc"
+#endif
+#define MPS_PF_W3I3GC
+#define MPS_PF_STRING   "w3i3gc"
+#define MPS_OS_W3
+#define MPS_ARCH_I3
+#define MPS_BUILD_GC
+#define MPS_T_WORD      unsigned long
+#define MPS_T_ULONGEST  unsigned long
+#define MPS_WORD_WIDTH  32
+#define MPS_WORD_SHIFT  5
+#define MPS_PF_ALIGN    4
+
+
 #else
 #error "The MPS Kit does not have a configuration for this platform out of the box; see manual/build.txt"
 #endif
--- ./code/protw3.c~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/protw3.c	2024-04-23 13:47:36.490634000 +0300
@@ -36,6 +36,7 @@ void ProtSet(Addr base, Addr limit, Acce
     NOTREACHED;
 }
 
+LONG WINAPI ProtSEHfilter(LPEXCEPTION_POINTERS);
 
 LONG WINAPI ProtSEHfilter(LPEXCEPTION_POINTERS info)
 {
--- ./code/spw3i3.c~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/spw3i3.c	2024-04-23 09:22:39.390462300 +0300
@@ -27,11 +27,20 @@
 
 void StackProbe(Size depth)
 {
+#ifdef __GNUC__
+  __asm__ volatile ("mov %0, %%eax\n\t"
+		    "neg %%eax\n\t"
+		    "mov (%%esp,%%eax,4), %%eax" /* do the actual probe */
+		    :	/* no outputs */
+		    : "r" (depth)
+  		    : "eax");
+#else	/* MSVC */
   __asm {
     mov  eax, depth
     neg  eax
     mov  eax, [esp+eax*4] /* do the actual probe */
   }
+#endif
 }
 
 
--- ./code/testlib.c~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/testlib.c	2024-04-23 17:02:34.566980300 +0300
@@ -221,7 +221,14 @@ double rnd_double(void)
 
 static unsigned sizelog2(size_t size)
 {
+#ifdef __MINGW32__
+  /* For some reason, MinGW sometimes produces a value slightly
+     smaller than the expected one, so we round up to the next FP
+     value.  */
+  return (unsigned)(log((double)size) / log(2.0) * (1.0 + __DBL_EPSILON__));
+#else
   return (unsigned)(log((double)size) / log(2.0));
+#endif
 }
 
 size_t rnd_grain(size_t arena_size)
@@ -232,17 +239,17 @@ size_t rnd_grain(size_t arena_size)
   return rnd_align(sizeof(void *), (size_t)1 << sizelog2(arena_size >> MPS_WORD_SHIFT));
 }
 
-size_t rnd_align(size_t min, size_t max)
+size_t rnd_align(size_t minv, size_t maxv)
 {
-  unsigned log2min = sizelog2(min);
-  unsigned log2max = sizelog2(max);
-  Insist(min <= max);
-  Insist((size_t)1 << log2min == min);
-  Insist((size_t)1 << log2max == max);
+  unsigned log2min = sizelog2(minv);
+  unsigned log2max = sizelog2(maxv);
+  Insist(minv <= maxv);
+  Insist((size_t)1 << log2min == minv);
+  Insist((size_t)1 << log2max == maxv);
   if (log2min < log2max)
-    return min << (rnd() % (log2max - log2min + 1));
+    return minv << (rnd() % (log2max - log2min + 1));
   else
-    return min;
+    return minv;
 }
 
 double rnd_pause_time(void)
--- ./code/testlib.h~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/testlib.h	2024-04-23 14:31:30.269692800 +0300
@@ -71,7 +71,7 @@
  * <https://docs.microsoft.com/en-gb/cpp/c-runtime-library/reference/alloca>
  */
 
-#if defined(MPS_OS_W3)
+#if defined(MPS_OS_W3) && !defined(__GNUC__)
 
 #define alloca _alloca
 
@@ -86,7 +86,7 @@
  * This macro version may evaluate the name argument twice.
  */
 
-#if defined(MPS_OS_W3)
+#if defined(MPS_OS_W3) && !defined(__GNUC__)
 
 #define setenv(name, value, overwrite) \
     (((overwrite) || !getenv(name)) ? _putenv_s(name, value) : 0)
--- ./code/thw3.c~0	2023-07-11 13:01:52.000000000 +0300
+++ ./code/thw3.c	2024-04-23 13:50:59.803414400 +0300
@@ -259,7 +259,7 @@ Res ThreadDescribe(Thread thread, mps_li
 }
 
 
-Res ThreadScan(ScanState ss, Thread thread, Word *stackCold,
+Res ThreadScan(ScanState ss, Thread thread, void *stackCold,
                mps_area_scan_t scan_area, void *closure)
 {
   DWORD id;
@@ -285,7 +285,7 @@ Res ThreadScan(ScanState ss, Thread thre
     stackPtr = MutatorContextSP(&context);
     /* .stack.align */
     stackBase  = (Word *)AddrAlignUp(stackPtr, sizeof(Word));
-    stackLimit = stackCold;
+    stackLimit = (Word *)stackCold;
     if (stackBase >= stackLimit)
       return ResOK;    /* .stack.below-bottom */
 
--- code/mingw.gmk~0	1970-01-01 02:00:00.000000000 +0200
+++ code/mingw.gmk	2024-04-23 14:29:54.439945000 +0300
@@ -0,0 +1,80 @@
+# -*- makefile -*-
+#
+# mingw.gmk: GNUMAKEFILE FRAGMENT FOR MinGW GCC
+#
+# $Id$
+# Copyright (c) 2001-2020 Ravenbrook Limited.  See end of file for license.
+#
+# This file is included by platform makefiles that use the GNU CC
+# compiler.  It defines the compiler-specific variables that the
+# common makefile fragment (<code/comm.gmk>) requires.
+
+CC = gcc
+CFLAGSDEBUG = -Og -gdwarf-4 -g3
+CFLAGSOPT = -O2 -gdwarf-4 -g3
+EXEEXT = .exe
+
+# Warnings that might be enabled by clients <design/config/#.warning.impl>.
+CFLAGSCOMPILER := \
+-Waggregate-return \
+-Wall \
+-Wcast-qual \
+-Werror \
+-Wextra \
+-Winline \
+-Wmissing-prototypes \
+-Wnested-externs \
+-Wpointer-arith \
+-Wshadow \
+-Wstrict-aliasing=2 \
+-Wstrict-prototypes \
+-Wswitch-default \
+-Wwrite-strings
+CFLAGSCOMPILERSTRICT := -std=gnu89 -pedantic
+
+# A different set of compiler flags for less strict compilation, for
+# instance when we need to #include a third-party header file that
+# won't fly with -ansi -pedantic.  Use sparingly!
+CFLAGSCOMPILERLAX :=
+
+# gcc -MM generates a dependency line of the form:
+#   thing.o : thing.c ...
+# The sed line converts this into:
+#   <pfm>/<variety>/thing.o <pfm>/<variety>/thing.d : thing.c ...
+# If interrupted, this is liable to leave a zero-length file behind.
+
+define gendep
+$(SHELL) -ec "$(CC) $(CFLAGSSTRICT) -MM $< | \
+	sed '/:/s!$*.o!$(@D)/& $(@D)/$*.d!' > $@"
+[ -s $@ ] || rm -f $@
+endef
+
+
+# C. COPYRIGHT AND LICENSE
+#
+# Copyright (C) 2001-2020 Ravenbrook Limited <https://www.ravenbrook.com/>.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the
+#    distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
--- code/w3i3gc.gmk~0	1970-01-01 02:00:00.000000000 +0200
+++ code/w3i3gc.gmk	2024-04-23 14:40:10.709906400 +0300
@@ -0,0 +1,74 @@
+# -*- makefile -*-
+#
+# w3i3gc.gmk: BUILD FOR Windows/x86/MinGW PLATFORM
+#
+# $Id$
+# Copyright (c) 2001-2020 Ravenbrook Limited.  See end of file for license.
+
+PFM = w3i3gc
+
+MPMPF = \
+    lockw3.c \
+    mpsiw3.c \
+    prmci3.c \
+    prmcw3.c \
+    prmcw3i3.c \
+    protw3.c \
+    spw3i3.c \
+    thw3.c \
+    vmw3.c
+
+include mingw.gmk
+include comm.gmk
+
+# Installation stuff copied from ../Makefile.in
+
+prefix=d:/usr
+INSTALL_PROGRAM = /bin/install -c
+INSTALL_DATA = /bin/install -c -m 644
+MPS_TARGET_NAME = w3i3gc
+
+build-via-make:
+	$(MAKE) -f $(MPS_TARGET_NAME).gmk EXTRA_TARGETS="$(EXTRA_TARGETS)"
+
+install-make-build: make-install-dirs build-via-make
+	$(INSTALL_DATA) mps*.h $(prefix)/include/
+	$(INSTALL_DATA) $(MPS_TARGET_NAME)/cool/mps.a $(prefix)/lib/libmps-debug.a
+	$(INSTALL_DATA) $(MPS_TARGET_NAME)/hot/mps.a $(prefix)/lib/libmps.a
+	for PROGRAM in $(EXTRA_TARGETS); do $(INSTALL_PROGRAM) $(MPS_TARGET_NAME)/hot/$$PROGRAM $(prefix)/bin/$$PROGRAM; done
+
+make-install-dirs:
+	mkdir -p $(prefix)/bin
+	mkdir -p $(prefix)/lib
+	mkdir -p $(prefix)/include
+
+install: install-make-build
+
+# C. COPYRIGHT AND LICENSE
+#
+# Copyright (C) 2001-2020 Ravenbrook Limited <https://www.ravenbrook.com/>.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+#
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the
+#    distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

Re: MPS: Win64 testers?

[Pip Cet]

On Saturday, July 27th, 2024 at 08:02, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Thu, 25 Jul 2024 15:39:51 +0000
> 
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org
> > 
> > On Thursday, July 25th, 2024 at 07:16, Eli Zaretskii eliz@gnu.org wrote:
> > 
> > > I think you are indeed the first one. I use 32-bit MinGW, and for
> > > that I have a working build of MPS (and can share the diffs if someone
> > > wants them), but a 64-bit MinGW build of MPS will need other changes,
> > > at least in some cases.
> > 
> > It'd be great if you could share the diffs, so I could see where I've gone wrong! I've got the 64-bit build with mps to start by now, but haven't really tested it for stability much. It survives a few garbage collection cycles, at least.
> 
> 
> My diffs are attached below.

Thanks!

> Note that they are for the 32-bit MinGW
> build, so not all of them are relevant for a 64-bit MinGW build, and
> some stuff for a 64-bit build is missing and you will have to add it.

Understood.

> For example, the code/w3i3gc.gmk file is only for the 32-bit build;
> you will need to create code/w3i6gc.gmk instead. You'll also need to
> add the 64-bit MinGW fragment to code/mpstd.h (where I added the stuff
> for the 32-bit MinGW).
> 
> To build, I did the following from the MSYS Bash prompt:
> 
> $ cd code
> $ make -f w3i3gc.gmk
> 
> To run the test suite, I did the following from the cmd window:
> 
> d:\usr\...\mps-release-1.118.0\code>..\tool\testrun.bat w3i3gc cool testrun
> 
> 
> (where "..." stands for the specific place I have the sources, which
> is immaterial for the purposes of this discussion).
> 
> To install, say from the MSYS Bash:
> 
> $ make install
> 
> > As a general question, would there be interest in CI for such systems? I'm setting up something locally, but I've been trying to be careful so that the Dockerfiles I'm using should work on x86_64 GNU/Linux machines generally.
> 
> 
> I think it will be useful, but I don't know how much effort it will
> need from you, so I cannot say if it's justified at this stage.

What is the state of Emacs CI generally? emba.gnu.org doesn't even appear to be reachable at this point...

> > > > - _setjmp_ex assumes 16-byte alignment of jmp_buf to store the XMM registers, but we allocate handlers with 8-byte alignment in MPS builds.
> > > 
> > > Where and how do you see that?
> > 
> > I'm not quite sure what you're asking. I saw this trying to run things in wine in a docker image on an x86_64 GNU/Linux machine, ran it in gdb.exe, found the instruction causing the segfault, and fixed things so it wouldn't.
> > 
> > https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/winecrt0/setjmp.c#L123 is the instruction that segfaulted here. I'm not sure whether that's what you were asking.
> > 
> > The jump buffer is declared with 16-byte alignment here: https://gitlab.winehq.org/wine/wine/-/blob/master/include/msvcrt/setjmp.h#L50
> > 
> > A side effect of this is that LISP_ALIGNMENT is 16, which breaks MPS because vectorlike_nbytes rounds up to a multiple of LISP_ALIGNMENT, not just GCALIGNMENT.
> 
> 
> If the "_setjmp_ex assumes 16-byte alignment of jmp_buf to store the
> XMM registers" part is about what Wine does, then my question is where
> did you see that "we allocate handlers with 8-byte alignment in MPS
> builds"?

All our pools are aligned to IGC_ALIGN, which is GCALIGNMENT, which is 8 bytes on this system.

> > Looking over my diff, here's what I had to fix to get things to the current stage on msys and msys2 (using MSYSTEM=MINGW64).
> > 
> > - disable the sanity check in the Makefile, because backticks/pipes don't work properly
> How so? what doesn't work? By "makefile", which Makefile do you mean?
> the one in Emacs or the one in MPS?

It's the Emacs Makefile. I haven't investigated properly, but it seems output to stdout and stderr is sometimes dropped.

> > - make WINDRES include the --use-temp-file flag, for the same reason

> Is this in Emacs Makefiles? I guess Wine is not a good emulation of
> Windows, then.

Yes.

> > - wrap the add-name-to-file calls in loadup.el in with-demoted-errors (link problems?)
> 
> I have no idea. Maybe hard links are not properly emulated by Wine?
> 
> > - force LISP_ALIGNMENT to be 8, not 16
> 
> 
> Does this mean 'struct Lisp_*' structures are 16-byte aligned in the
> 64-bit MinGW build? If not, what forces LISP_ALIGNMENT to be 16?

emacs_align_type contains struct thread_state, which contains a jmpbuf, which is 16-byte aligned.

> > - change my local generation size settings to be much more conservative for the msys build. I think scratch/igc is fine, though.
> 
> Don't understand what this means, sorry. What are your "local
> generation size settings"?

Ignore that, it's not in scratch/igc and not required for it. Sorry.

> > - make sys_setjmp and sys_longjmp use a larger buffer and memmove() the data in it so it's 16-byte aligned
> 
> Why would this be any different from a non-MPS build for 64-bit MinGW?

Because non-MPS allocates handlers with xmalloc(), which is 16-byte aligned, and the jmpbuf in it is, too. I expect the change is required on MPS builds on native Windows, too, though, because that also specifies 16-byte alignment for jmpbuf, IIUC.

> > - disable the failure exit on close_stream failure in sysdep.c, and silently ignore such errors
> I guess you are using UCRT? These problems with UCRT are known, but
> no one came up with an explanation for them yet. What happens if you
> link against MSVCRT instead?

I'm using the msys2-docker-experimental image with MSYSTEM=MINGW64. I don't know whether that means I'm using UCRT, sorry. If I use MSYSTEM=UCRT64, I get many more and different problems...

> > - trace the extra 'ztrillion' bignum in timefns.c
> What happens if you don't?

Oops, sorry, bignum tracing isn't in scratch/igc. So ignore that one, too.

> > - for the msys build, run autogen.sh outside of wine, since there are virtual memory issues inflooping the process otherwise
> 
> Sounds like Wine is not a good idea at all...

Specifically, I'm seeing messages such as this one:

      0 [main] perl 508 fork_copy: linked dll data/bss pass 0 failed, 0x56CDE000..0x56CFE794, done 0, windows pid 1352, Win32 error 5

The library in question is msys-perl5_8.dll, and using rebase.exe (from cygwin?) to rebase it to 0x7000000 appears to help.

I tend to agree that using wine for 32-bit mingw is problematic. Using it for 64-bit systems seems fine.

> Here are the diffs I promised.

Thanks again!

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sat, 27 Jul 2024 10:42:12 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> > If the "_setjmp_ex assumes 16-byte alignment of jmp_buf to store the
> > XMM registers" part is about what Wine does, then my question is where
> > did you see that "we allocate handlers with 8-byte alignment in MPS
> > builds"?
> 
> All our pools are aligned to IGC_ALIGN, which is GCALIGNMENT, which is 8 bytes on this system.
> [...]
> > > - force LISP_ALIGNMENT to be 8, not 16
> > 
> > 
> > Does this mean 'struct Lisp_*' structures are 16-byte aligned in the
> > 64-bit MinGW build? If not, what forces LISP_ALIGNMENT to be 16?
> 
> emacs_align_type contains struct thread_state, which contains a jmpbuf, which is 16-byte aligned.

And that is specific to MS-Windows?  Don't Posix systems store XMM
registers in jmpbuf as well?

> > > - make sys_setjmp and sys_longjmp use a larger buffer and memmove() the data in it so it's 16-byte aligned
> > 
> > Why would this be any different from a non-MPS build for 64-bit MinGW?
> 
> Because non-MPS allocates handlers with xmalloc(), which is 16-byte aligned, and the jmpbuf in it is, too. I expect the change is required on MPS builds on native Windows, too, though, because that also specifies 16-byte alignment for jmpbuf, IIUC.

Maybe the way you made MPS compile with MinGW64 is incorrect, because
it ought to return the same alignment as malloc on the target
platform?  What is the value of MPS_PF_ALIGN you used for building
MPS?  Also, did the library pass the test suite?  I had some failures
until I set MPS_PF_ALIGN to the right value (8 for the 32-bit build).

> > > - disable the failure exit on close_stream failure in sysdep.c, and silently ignore such errors
> > I guess you are using UCRT? These problems with UCRT are known, but
> > no one came up with an explanation for them yet. What happens if you
> > link against MSVCRT instead?
> 
> I'm using the msys2-docker-experimental image with MSYSTEM=MINGW64. I don't know whether that means I'm using UCRT, sorry. If I use MSYSTEM=UCRT64, I get many more and different problems...

There are no problems with close_stream when linking against MSVCRT,
AFAIK.  Problems such as what you report were only reported in UCRT
build.

Re: MPS: Win64 testers?

[Pip Cet]

On Saturday, July 27th, 2024 at 12:14, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sat, 27 Jul 2024 10:42:12 +0000
> 
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org
> > 
> > > If the "_setjmp_ex assumes 16-byte alignment of jmp_buf to store the
> > > XMM registers" part is about what Wine does, then my question is where
> > > did you see that "we allocate handlers with 8-byte alignment in MPS
> > > builds"?
> > 
> > All our pools are aligned to IGC_ALIGN, which is GCALIGNMENT, which is 8 bytes on this system.
> > [...]
> > 
> > > > - force LISP_ALIGNMENT to be 8, not 16
> > > 
> > > Does this mean 'struct Lisp_*' structures are 16-byte aligned in the
> > > 64-bit MinGW build? If not, what forces LISP_ALIGNMENT to be 16?
> > 
> > emacs_align_type contains struct thread_state, which contains a jmpbuf, which is 16-byte aligned.
> 
> 
> And that is specific to MS-Windows? Don't Posix systems store XMM
> registers in jmpbuf as well?

The SYSV ABI declares all XMM registers to be caller-saved, which means on such systems we don't store the XMM registers in a jmpbuf.

> > > > - make sys_setjmp and sys_longjmp use a larger buffer and memmove() the data in it so it's 16-byte aligned
> > > 
> > > Why would this be any different from a non-MPS build for 64-bit MinGW?
> > 
> > Because non-MPS allocates handlers with xmalloc(), which is 16-byte aligned, and the jmpbuf in it is, too. I expect the change is required on MPS builds on native Windows, too, though, because that also specifies 16-byte alignment for jmpbuf, IIUC.
> 
> Maybe the way you made MPS compile with MinGW64 is incorrect,

Very possible.

> because
> it ought to return the same alignment as malloc on the target
> platform?

I'm not sure that's correct, I think it assumes you pass the right value for MPS_KEY_FMT_ALIGN. Currently the code passes 8, which is necessary to keep conses at a mere 24 bytes.

> What is the value of MPS_PF_ALIGN you used for building
> MPS? Also, did the library pass the test suite? I had some failures
> until I set MPS_PF_ALIGN to the right value (8 for the 32-bit build).

The patch you sent has it at 4, though?

> > > > - disable the failure exit on close_stream failure in sysdep.c, and silently ignore such errors
> > > > I guess you are using UCRT? These problems with UCRT are known, but
> > > > no one came up with an explanation for them yet. What happens if you
> > > > link against MSVCRT instead?
> > 
> > I'm using the msys2-docker-experimental image with MSYSTEM=MINGW64. I don't know whether that means I'm using UCRT, sorry. If I use MSYSTEM=UCRT64, I get many more and different problems...
> 
> There are no problems with close_stream when linking against MSVCRT,
> AFAIK. Problems such as what you report were only reported in UCRT
> build.

I think the problem is init_ntproc in w32.c, which closes stdin, stdout, stderr, then calls _fdopen and assumes the result will live in stdin, stdout, stderr again. If I disable that code the backtick problem goes away:

    fclose (stdin);
    fclose (stdout);
    fclose (stderr);

    if (stdin_save != INVALID_HANDLE_VALUE)
      _open_osfhandle ((intptr_t) stdin_save, O_TEXT);
    else
      _open ("nul", O_TEXT | O_NOINHERIT | O_RDONLY);
    _fdopen (0, "r");

I believe that last line should, possibly, on some systems, be

    stdin = _fdopen (0, "r");

but I'm not sure the variable (or macro) stdin is an lvalue. That's probably why close_stream(stdout) and close_stream(stderr) fail, too: the streams they refer to are already closed, and the new streams we should be using instead were discarded.

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sat, 27 Jul 2024 15:49:59 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> > > > > - force LISP_ALIGNMENT to be 8, not 16
> > > > 
> > > > Does this mean 'struct Lisp_*' structures are 16-byte aligned in the
> > > > 64-bit MinGW build? If not, what forces LISP_ALIGNMENT to be 16?
> > > 
> > > emacs_align_type contains struct thread_state, which contains a jmpbuf, which is 16-byte aligned.
> > 
> > 
> > And that is specific to MS-Windows? Don't Posix systems store XMM
> > registers in jmpbuf as well?
> 
> The SYSV ABI declares all XMM registers to be caller-saved, which means on such systems we don't store the XMM registers in a jmpbuf.
> 
> > > > > - make sys_setjmp and sys_longjmp use a larger buffer and memmove() the data in it so it's 16-byte aligned
> > > > 
> > > > Why would this be any different from a non-MPS build for 64-bit MinGW?
> > > 
> > > Because non-MPS allocates handlers with xmalloc(), which is 16-byte aligned, and the jmpbuf in it is, too. I expect the change is required on MPS builds on native Windows, too, though, because that also specifies 16-byte alignment for jmpbuf, IIUC.
> > 
> > Maybe the way you made MPS compile with MinGW64 is incorrect,
> 
> Very possible.
> 
> > because
> > it ought to return the same alignment as malloc on the target
> > platform?
> 
> I'm not sure that's correct, I think it assumes you pass the right value for MPS_KEY_FMT_ALIGN. Currently the code passes 8, which is necessary to keep conses at a mere 24 bytes.
> 
> > What is the value of MPS_PF_ALIGN you used for building
> > MPS? Also, did the library pass the test suite? I had some failures
> > until I set MPS_PF_ALIGN to the right value (8 for the 32-bit build).
> 
> The patch you sent has it at 4, though?

Yes, because some tests failed with 8.

But note that the MSVC 64-bit build uses 16.

> > > > > - disable the failure exit on close_stream failure in sysdep.c, and silently ignore such errors
> > > > > I guess you are using UCRT? These problems with UCRT are known, but
> > > > > no one came up with an explanation for them yet. What happens if you
> > > > > link against MSVCRT instead?
> > > 
> > > I'm using the msys2-docker-experimental image with MSYSTEM=MINGW64. I don't know whether that means I'm using UCRT, sorry. If I use MSYSTEM=UCRT64, I get many more and different problems...
> > 
> > There are no problems with close_stream when linking against MSVCRT,
> > AFAIK. Problems such as what you report were only reported in UCRT
> > build.
> 
> I think the problem is init_ntproc in w32.c, which closes stdin, stdout, stderr, then calls _fdopen and assumes the result will live in stdin, stdout, stderr again. If I disable that code the backtick problem goes away:
> 
>     fclose (stdin);
>     fclose (stdout);
>     fclose (stderr);
> 
>     if (stdin_save != INVALID_HANDLE_VALUE)
>       _open_osfhandle ((intptr_t) stdin_save, O_TEXT);
>     else
>       _open ("nul", O_TEXT | O_NOINHERIT | O_RDONLY);
>     _fdopen (0, "r");

Why do you think it's a problem?  And why do you think the assumption
of fdopen is incorrect?  It is confirmed by the source code of MSVCRT.

> I believe that last line should, possibly, on some systems, be
> 
>     stdin = _fdopen (0, "r");
> 
> but I'm not sure the variable (or macro) stdin is an lvalue. That's probably why close_stream(stdout) and close_stream(stderr) fail, too: the streams they refer to are already closed, and the new streams we should be using instead were discarded.

But this never fails in MinGW builds on Windows.

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sat, 27 Jul 2024 19:10:50 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > I think the problem is init_ntproc in w32.c, which closes stdin, stdout, stderr, then calls _fdopen and assumes the result will live in stdin, stdout, stderr again. If I disable that code the backtick problem goes away:
> > 
> >     fclose (stdin);
> >     fclose (stdout);
> >     fclose (stderr);
> > 
> >     if (stdin_save != INVALID_HANDLE_VALUE)
> >       _open_osfhandle ((intptr_t) stdin_save, O_TEXT);
> >     else
> >       _open ("nul", O_TEXT | O_NOINHERIT | O_RDONLY);
> >     _fdopen (0, "r");
> 
> Why do you think it's a problem?  And why do you think the assumption
> of fdopen is incorrect?  It is confirmed by the source code of MSVCRT.
> 
> > I believe that last line should, possibly, on some systems, be
> > 
> >     stdin = _fdopen (0, "r");
> > 
> > but I'm not sure the variable (or macro) stdin is an lvalue. That's probably why close_stream(stdout) and close_stream(stderr) fail, too: the streams they refer to are already closed, and the new streams we should be using instead were discarded.
> 
> But this never fails in MinGW builds on Windows.

And if I change w32.c like below:

diff --git a/src/w32.c b/src/w32.c
index 31ffa30..c28a234 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -10517,20 +10517,21 @@ init_ntproc (int dumping)
     fclose (stdout);
     fclose (stderr);
 
+    int fd;
     if (stdin_save != INVALID_HANDLE_VALUE)
-      _open_osfhandle ((intptr_t) stdin_save, O_TEXT);
+      fd = _open_osfhandle ((intptr_t) stdin_save, O_TEXT);
     else
       _open ("nul", O_TEXT | O_NOINHERIT | O_RDONLY);
     _fdopen (0, "r");
 
     if (stdout_save != INVALID_HANDLE_VALUE)
-      _open_osfhandle ((intptr_t) stdout_save, O_TEXT);
+      fd = _open_osfhandle ((intptr_t) stdout_save, O_TEXT);
     else
       _open ("nul", O_TEXT | O_NOINHERIT | O_WRONLY);
     _fdopen (1, "w");
 
     if (stderr_save != INVALID_HANDLE_VALUE)
-      _open_osfhandle ((intptr_t) stderr_save, O_TEXT);
+      fd = _open_osfhandle ((intptr_t) stderr_save, O_TEXT);
     else
       _open ("nul", O_TEXT | O_NOINHERIT | O_WRONLY);
     _fdopen (2, "w");

then stepping through this code I see that fd is first assigned zero,
then 1, then 2.  As expected, because any decent emulation of Posix
file descriptors must keep this semantics: file descriptors are reused
starting from the lowest available slot.

Do you see something different in your build?

Re: MPS: Win64 testers?

[Pip Cet]

On Saturday, July 27th, 2024 at 16:56, Eli Zaretskii <eliz@gnu.org> wrote:
> then stepping through this code I see that fd is first assigned zero,
> then 1, then 2. As expected, because any decent emulation of Posix
> file descriptors must keep this semantics: file descriptors are reused
> starting from the lowest available slot.
> 
> Do you see something different in your build?

_open_osfhandle returns 0 for the first FD, but _fdopen returns 0x7b281160, while stdin is 0x7b281100.

I believe that _fdopen (0, "r") returns stdin if it has previously been closed is an implementation detail of Windows that Emacs relies on, and that wine fails to properly emulate. IMHO, this is a wine bug, but Emacs also shouldn't rely on it unless it's documented as part of the API; I don't know whether it is.

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sat, 27 Jul 2024 18:27:43 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Saturday, July 27th, 2024 at 16:56, Eli Zaretskii <eliz@gnu.org> wrote:
> > then stepping through this code I see that fd is first assigned zero,
> > then 1, then 2. As expected, because any decent emulation of Posix
> > file descriptors must keep this semantics: file descriptors are reused
> > starting from the lowest available slot.
> > 
> > Do you see something different in your build?
> 
> _open_osfhandle returns 0 for the first FD, but _fdopen returns 0x7b281160, while stdin is 0x7b281100.

Not on my system.  Here _fdopen returns a pointer to the same FILE
object as the original std* streams.  And it is also expected, since
the source code of MSVCRT explicitly shows that _fdopen calls a
routine that looks up a vacant stream slot, and that routine examines
the _iob[] array in the ascending order of indices, from zero to max,
until it finds an unused slot.

> I believe that _fdopen (0, "r") returns stdin if it has previously been closed is an implementation detail of Windows that Emacs relies on, and that wine fails to properly emulate. IMHO, this is a wine bug, but Emacs also shouldn't rely on it unless it's documented as part of the API; I don't know whether it is.

I believe this is how every Posix-like allocation of FILE objects
should behave.  I'm astonished that Wine doesn't work like that, since
it's the easiest way.

Moreover, the MS documentation says about the standard streams:

  These pointers are constants, and can't be assigned new values.

So you cannot do "stdin = _fdopen (...);" and hope for all the
references to stdin in the rest of the program to use the new value of
the stream pointer.  IOW, what Emacs does is the only way of doing
that.

Re: MPS: Win64 testers?

[Pip Cet]

On Saturday, July 27th, 2024 at 18:50, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sat, 27 Jul 2024 18:27:43 +0000
> 
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org
> > 
> > On Saturday, July 27th, 2024 at 16:56, Eli Zaretskii eliz@gnu.org wrote:
> > 
> > > then stepping through this code I see that fd is first assigned zero,
> > > then 1, then 2. As expected, because any decent emulation of Posix
> > > file descriptors must keep this semantics: file descriptors are reused
> > > starting from the lowest available slot.
> > > 
> > > Do you see something different in your build?
> > 
> > _open_osfhandle returns 0 for the first FD, but _fdopen returns 0x7b281160, while stdin is 0x7b281100.
> 
> 
> Not on my system. Here _fdopen returns a pointer to the same FILE
> object as the original std* streams. And it is also expected, since
> the source code of MSVCRT explicitly shows that _fdopen calls a
> routine that looks up a vacant stream slot, and that routine examines
> the _iob[] array in the ascending order of indices, from zero to max,
> until it finds an unused slot.

Wine does the same thing, except it starts at index 3:

  for (i = 3; i < MSVCRT_max_streams; i++)
  {
    file = msvcrt_get_file(i);
    if (!file)
      return NULL;

    if (file->_flag == 0)
    {
      if (i == MSVCRT_stream_idx)
      {
          CRITICAL_SECTION *cs = file_get_cs(file);
          if (cs)
          {
              InitializeCriticalSectionEx(cs, 0, RTL_CRITICAL_SECTION_FLAG_FORCE_DEBUG_INFO);
              cs->DebugInfo->Spare[0] = (DWORD_PTR)(__FILE__ ": file_crit.crit");
          }
          MSVCRT_stream_idx++;
      }
      return file;
    }
  }

It's been that way since 2004, if the git history is to be believed.

Fixing wine (single-character fixes are my favorite) to start at index 0 makes things work.

> > I believe that _fdopen (0, "r") returns stdin if it has previously been closed is an implementation detail of Windows that Emacs relies on, and that wine fails to properly emulate. IMHO, this is a wine bug, but Emacs also shouldn't rely on it unless it's documented as part of the API; I don't know whether it is.
> 
> 
> I believe this is how every Posix-like allocation of FILE objects
> should behave. I'm astonished that Wine doesn't work like that, since
> it's the easiest way.
> 
> Moreover, the MS documentation says about the standard streams:
> 
> These pointers are constants, and can't be assigned new values.
> 
> So you cannot do "stdin = _fdopen (...);" and hope for all the
> references to stdin in the rest of the program to use the new value of
> the stream pointer. IOW, what Emacs does is the only way of doing
> that.

Well, there's freopen, but no fdreopen, so you're probably right...

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sat, 27 Jul 2024 20:22:34 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Saturday, July 27th, 2024 at 18:50, Eli Zaretskii <eliz@gnu.org> wrote:
> > > Date: Sat, 27 Jul 2024 18:27:43 +0000
> > 
> > > From: Pip Cet pipcet@protonmail.com
> > > Cc: emacs-devel@gnu.org
> > > 
> > > On Saturday, July 27th, 2024 at 16:56, Eli Zaretskii eliz@gnu.org wrote:
> > > 
> > > > then stepping through this code I see that fd is first assigned zero,
> > > > then 1, then 2. As expected, because any decent emulation of Posix
> > > > file descriptors must keep this semantics: file descriptors are reused
> > > > starting from the lowest available slot.
> > > > 
> > > > Do you see something different in your build?
> > > 
> > > _open_osfhandle returns 0 for the first FD, but _fdopen returns 0x7b281160, while stdin is 0x7b281100.
> > 
> > 
> > Not on my system. Here _fdopen returns a pointer to the same FILE
> > object as the original std* streams. And it is also expected, since
> > the source code of MSVCRT explicitly shows that _fdopen calls a
> > routine that looks up a vacant stream slot, and that routine examines
> > the _iob[] array in the ascending order of indices, from zero to max,
> > until it finds an unused slot.
> 
> Wine does the same thing, except it starts at index 3:
> 
>   for (i = 3; i < MSVCRT_max_streams; i++)
>   {
>     file = msvcrt_get_file(i);
>     if (!file)
>       return NULL;
> 
>     if (file->_flag == 0)
>     {
>       if (i == MSVCRT_stream_idx)
>       {
>           CRITICAL_SECTION *cs = file_get_cs(file);
>           if (cs)
>           {
>               InitializeCriticalSectionEx(cs, 0, RTL_CRITICAL_SECTION_FLAG_FORCE_DEBUG_INFO);
>               cs->DebugInfo->Spare[0] = (DWORD_PTR)(__FILE__ ": file_crit.crit");
>           }
>           MSVCRT_stream_idx++;
>       }
>       return file;
>     }
>   }
> 
> It's been that way since 2004, if the git history is to be believed.

I guess they have some unsolved issues with the standard streams,
which is why they don't dare reusing their slots.  Which means closing
these standard streams is a no-no on Wine.

But if this is what Wine does, then what do you get from the various
fprintf debugging code in Emacs?  For example, if you build with the
"--enable-checking='yes,glyphs'" option, you get the trace-redisplay
command which outputs debugging information about redisplay to stderr,
and also the dump-glyph-matrix command which prints the glyph matrix
to stderr.  Do these print-outs still work although stderr was closed
in init_ntproc?

Or maybe under Wine closing a standard stream has no effect, and the
stream remains open?  But then why does close_output_streams reports
errors?

Re: MPS: Win64 testers?

[Pip Cet]

On Sunday, July 28th, 2024 at 05:25, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sat, 27 Jul 2024 20:22:34 +0000
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org
> > It's been that way since 2004, if the git history is to be believed.
> 
> I guess they have some unsolved issues with the standard streams,
> which is why they don't dare reusing their slots. Which means closing
> these standard streams is a no-no on Wine.

I don't know. I suspect they just never thought (20 years ago!) that anyone would want to close a standard stream. Any idea what UCRT does?

> But if this is what Wine does, then what do you get from the various
> fprintf debugging code in Emacs? For example, if you build with the
> "--enable-checking='yes,glyphs'" option, you get the trace-redisplay
> command which outputs debugging information about redisplay to stderr,
> and also the dump-glyph-matrix command which prints the glyph matrix
> to stderr. Do these print-outs still work although stderr was closed
> in init_ntproc?

I tried with GDB, and fprintf(stderr, ...) stops working in init_ntproc.

> Or maybe under Wine closing a standard stream has no effect, and the
> stream remains open? But then why does close_output_streams reports
> errors?

The FILE object remains assigned to its FD but loses its writable/readable flag, so fwrite returns -1/EBADF.  fclose() returns -1 (though it's spelled EOF in the source code) and doesn't set errno.

As for the code in question, if I understand it correctly, the current Windows idiom would be simply (I assume _open_osfhandle clears the flag?):

SetHandleInformation (GetStdHandle (STD_INPUT_HANDLE), HANDLE_FLAG_INHERIT, 0);

The version information for availability of "SetHandleInformation" and "DuplicateHandle" is the same, so any idea why we're not just doing that, or is it just lost in history?

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sun, 28 Jul 2024 13:00:20 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org
> 
> On Sunday, July 28th, 2024 at 05:25, Eli Zaretskii <eliz@gnu.org> wrote:
> > > Date: Sat, 27 Jul 2024 20:22:34 +0000
> > > From: Pip Cet pipcet@protonmail.com
> > > Cc: emacs-devel@gnu.org
> > > It's been that way since 2004, if the git history is to be believed.
> > 
> > I guess they have some unsolved issues with the standard streams,
> > which is why they don't dare reusing their slots. Which means closing
> > these standard streams is a no-no on Wine.
> 
> I don't know. I suspect they just never thought (20 years ago!) that anyone would want to close a standard stream. Any idea what UCRT does?

It seems to be doing the same as MSVCRT, at least if the UCRT sources
available out there are to be believed.

> > But if this is what Wine does, then what do you get from the various
> > fprintf debugging code in Emacs? For example, if you build with the
> > "--enable-checking='yes,glyphs'" option, you get the trace-redisplay
> > command which outputs debugging information about redisplay to stderr,
> > and also the dump-glyph-matrix command which prints the glyph matrix
> > to stderr. Do these print-outs still work although stderr was closed
> > in init_ntproc?
> 
> I tried with GDB, and fprintf(stderr, ...) stops working in init_ntproc.

So they do fclose the stream, just don't provide a way of reopening a
standard stream.

Is there any way for a program running on Wine to detect that this is
what happens? like some special version number or something?  If so,
I'd just avoid all that code in init_ntproc on Wine, since leaving the
standard streams inheritable is a much smaller trouble than losing the
standard streams.

> As for the code in question, if I understand it correctly, the current Windows idiom would be simply (I assume _open_osfhandle clears the flag?):
> 
> SetHandleInformation (GetStdHandle (STD_INPUT_HANDLE), HANDLE_FLAG_INHERIT, 0);

I'm not sure you can do this without closing the stream, since the
program has already started by that time.  And SetHandleInformation is
only available since Windows 2K, so we'd need to call it through a
function pointer.  Not sure if that is justified just to support Wine
(which probably has quite a few other incompatibilities).

> The version information for availability of "SetHandleInformation" and "DuplicateHandle" is the same, so any idea why we're not just doing that, or is it just lost in history?

See above: we still strive to support Windows 9X and NT.

Re: MPS: Win64 testers?

[Pip Cet]

On Sunday, July 28th, 2024 at 14:20, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sun, 28 Jul 2024 13:00:20 +0000
> 
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org
> > 
> > On Sunday, July 28th, 2024 at 05:25, Eli Zaretskii eliz@gnu.org wrote:
> > 
> > > > Date: Sat, 27 Jul 2024 20:22:34 +0000
> > > > From: Pip Cet pipcet@protonmail.com
> > > > Cc: emacs-devel@gnu.org
> > > > It's been that way since 2004, if the git history is to be believed.
> > > 
> > > I guess they have some unsolved issues with the standard streams,
> > > which is why they don't dare reusing their slots. Which means closing
> > > these standard streams is a no-no on Wine.
> > 
> > I don't know. I suspect they just never thought (20 years ago!) that anyone would want to close a standard stream. Any idea what UCRT does?
> 
> It seems to be doing the same as MSVCRT, at least if the UCRT sources
> available out there are to be believed.

By my reading of the source code I found, it's doing the same as Wine, not as MSVCRT.

> > > But if this is what Wine does, then what do you get from the various
> > > fprintf debugging code in Emacs? For example, if you build with the
> > > "--enable-checking='yes,glyphs'" option, you get the trace-redisplay
> > > command which outputs debugging information about redisplay to stderr,
> > > and also the dump-glyph-matrix command which prints the glyph matrix
> > > to stderr. Do these print-outs still work although stderr was closed
> > > in init_ntproc?
> > 
> > I tried with GDB, and fprintf(stderr, ...) stops working in init_ntproc.
> 
> So they do fclose the stream, just don't provide a way of reopening a
> standard stream.

Not in the way we want, no, unless I missed it.

> Is there any way for a program running on Wine to detect that this is
> what happens? like some special version number or something? If so,
> I'd just avoid all that code in init_ntproc on Wine, since leaving the
> standard streams inheritable is a much smaller trouble than losing the
> standard streams.

I don't think we should have Wine-specific workarounds, particularly not for such questionable code. IIUC, Wine discourages attempts of programs to detect whether they're running on Wine or Microsoft Windows.
 
> > As for the code in question, if I understand it correctly, the current Windows idiom would be simply (I assume _open_osfhandle clears the flag?):
> > 
> > SetHandleInformation (GetStdHandle (STD_INPUT_HANDLE), HANDLE_FLAG_INHERIT, 0);
> 
> I'm not sure you can do this without closing the stream, since the
> program has already started by that time. And SetHandleInformation is
> only available since Windows 2K, so we'd need to call it through a
> function pointer.

Thanks. Microsoft doesn't appear to provide version information pre-Windows-2000...

> Not sure if that is justified just to support Wine

And UCRT, if and when the time comes. And any other implementation of the Windows API that doesn't provide bug-for-bug compatibility with MSVCRT.

Wine is the only significant free implementation of the Windows API, so I disagree with the "just". If I understand RMS correctly, whether there is a free implementation of an OS is an important factor in deciding to redirect resources to supporting it in GNU software.

> (which probably has quite a few other incompatibilities).

I'm aware of the windres problem, but I suspect there are a few others. However, we can't discover them unless we fix the build first, and I am successfully running both mingw/msys/i386 builds and mingw64/msys2/x86_64 builds locally.

> > The version information for availability of "SetHandleInformation" and "DuplicateHandle" is the same, so any idea why we're not just doing that, or is it just lost in history?
> 
> See above: we still strive to support Windows 9X and NT.

You're right, thanks. Do we have people testing such systems?

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sun, 28 Jul 2024 15:00:52 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org, "rms@gnu.org" <rms@gnu.org>
> 
> > It seems to be doing the same as MSVCRT, at least if the UCRT sources
> > available out there are to be believed.
> 
> By my reading of the source code I found, it's doing the same as Wine, not as MSVCRT.

If someone steps through that code in the UCRT build and reports that
fdopen returns a different FILE pointer, I will believe you.

> > Is there any way for a program running on Wine to detect that this is
> > what happens? like some special version number or something? If so,
> > I'd just avoid all that code in init_ntproc on Wine, since leaving the
> > standard streams inheritable is a much smaller trouble than losing the
> > standard streams.
> 
> I don't think we should have Wine-specific workarounds, particularly not for such questionable code. IIUC, Wine discourages attempts of programs to detect whether they're running on Wine or Microsoft Windows.

We don't have to behave like Wine intends us to behave.  We have our
own considerations.  So I don't really care whether Wine endorses
detection of Wine, if we decide we need that.

> > > As for the code in question, if I understand it correctly, the current Windows idiom would be simply (I assume _open_osfhandle clears the flag?):
> > > 
> > > SetHandleInformation (GetStdHandle (STD_INPUT_HANDLE), HANDLE_FLAG_INHERIT, 0);
> > 
> > I'm not sure you can do this without closing the stream, since the
> > program has already started by that time. And SetHandleInformation is
> > only available since Windows 2K, so we'd need to call it through a
> > function pointer.
> 
> Thanks. Microsoft doesn't appear to provide version information pre-Windows-2000...

I use this resource to overcome the deficiencies of the official docs:

  http://winapi.freetechsecrets.com/win32/

> > Not sure if that is justified just to support Wine
> 
> And UCRT, if and when the time comes.

If that's the same problem, which we are not yet sure.  And, as I
said, support for UCRT will only become official if someone volunteers
to come on board and handle that build.

> And any other implementation of the Windows API that doesn't provide bug-for-bug compatibility with MSVCRT.

For now, only the MSVCRT support is official.

> Wine is the only significant free implementation of the Windows API, so I disagree with the "just". If I understand RMS correctly, whether there is a free implementation of an OS is an important factor in deciding to redirect resources to supporting it in GNU software.

Such decisions can only be relevant if there are resources to allocate
and redirect.  Since we don't have them, this is entirely academic.
For now, the only person who actively works on the Windows port is
myself, and I'm unable to support UCRT, for more than one reason.

> > > The version information for availability of "SetHandleInformation" and "DuplicateHandle" is the same, so any idea why we're not just doing that, or is it just lost in history?
> > 
> > See above: we still strive to support Windows 9X and NT.
> 
> You're right, thanks. Do we have people testing such systems?

We do, although not as much as I'd like to.

Re: MPS: Win64 testers?

[Pip Cet]

On Sunday, July 28th, 2024 at 15:20, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sun, 28 Jul 2024 15:00:52 +0000
> 
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org, "rms@gnu.org" rms@gnu.org
> > 
> > > It seems to be doing the same as MSVCRT, at least if the UCRT sources
> > > available out there are to be believed.
> > 
> > By my reading of the source code I found, it's doing the same as Wine, not as MSVCRT.
> 
> If someone steps through that code in the UCRT build and reports that
> fdopen returns a different FILE pointer, I will believe you.

I've verified this by running a GitHub action (I won't provide a link because it's probably impossible to use without running proprietary GitHub JavaScript), though I couldn't work out how to make clang link against msvcrt rather than its own libraries.

The result of 

  printf("%p\n", stdin);
  fclose(stdin);
  int fd = _open ("nul", _O_RDWR);
  printf("%d = 0\n", fd);
  FILE *f = _fdopen (fd, "r");
  printf("%p = %p\n", stdin, f);
  return 0;

is, for a MINGW32-built msvcrt binary on windows-latest:

  761f7600
  0 = 0
  761f7600 = 761f7600

but, for a UCRT64-built ucrt binary on windows-latest:

  00007FF8E7E9E4A0
  0 = 0
  00007FF8E7E9E4A0 = 000001F74AFF5660

> > > Is there any way for a program running on Wine to detect that this is
> > > what happens? like some special version number or something? If so,
> > > I'd just avoid all that code in init_ntproc on Wine, since leaving the
> > > standard streams inheritable is a much smaller trouble than losing the
> > > standard streams.
> > 
> > I don't think we should have Wine-specific workarounds, particularly not for such questionable code. IIUC, Wine discourages attempts of programs to detect whether they're running on Wine or Microsoft Windows.
> 
> 
> We don't have to behave like Wine intends us to behave. We have our
> own considerations. So I don't really care whether Wine endorses
> detection of Wine, if we decide we need that.

Let's see whether the Wine people want to provide bug-for-bug compatibility in this case first?

> > > > As for the code in question, if I understand it correctly, the current Windows idiom would be simply (I assume _open_osfhandle clears the flag?):
> > > > 
> > > > SetHandleInformation (GetStdHandle (STD_INPUT_HANDLE), HANDLE_FLAG_INHERIT, 0);
> > > 
> > > I'm not sure you can do this without closing the stream, since the
> > > program has already started by that time. And SetHandleInformation is
> > > only available since Windows 2K, so we'd need to call it through a
> > > function pointer.
> > 
> > Thanks. Microsoft doesn't appear to provide version information pre-Windows-2000...
> 
> I use this resource to overcome the deficiencies of the official docs:
> 
> http://winapi.freetechsecrets.com/win32/

Thanks for the link!

> > > Not sure if that is justified just to support Wine
> > 
> > And UCRT, if and when the time comes.
> 
> If that's the same problem, which we are not yet sure.

Of course.

> And, as I
> said, support for UCRT will only become official if someone volunteers
> to come on board and handle that build.

That's what I meant by "if and when the time comes".

> > And any other implementation of the Windows API that doesn't provide bug-for-bug compatibility with MSVCRT.
>
> For now, only the MSVCRT support is official.

Of course, and I think it's more likely it will cease being official than other builds becoming official.

> > Wine is the only significant free implementation of the Windows API, so I disagree with the "just". If I understand RMS correctly, whether there is a free implementation of an OS is an important factor in deciding to redirect resources to supporting it in GNU software.
>  
> Such decisions can only be relevant if there are resources to allocate
> and redirect.

I think the community of Emacs developers decides by consensus (or bickering) what to work on. My understanding was that we're considering whether to stop working on at least some variants of the Windows port.

> Since we don't have them, this is entirely academic.

Well, I disagree. Official guidance by the GNU Emacs project does have some influence on what volunteers will work on. I consider that a resource.

> For now, the only person who actively works on the Windows port is
> myself, and I'm unable to support UCRT, for more than one reason.

Are you working on MINGW64 systems at all, or is it mingw exclusively?

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sun, 28 Jul 2024 20:22:40 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org, rms@gnu.org
> 
> On Sunday, July 28th, 2024 at 15:20, Eli Zaretskii <eliz@gnu.org> wrote:
> > > By my reading of the source code I found, it's doing the same as Wine, not as MSVCRT.
> > 
> > If someone steps through that code in the UCRT build and reports that
> > fdopen returns a different FILE pointer, I will believe you.
> 
> I've verified this by running a GitHub action (I won't provide a link because it's probably impossible to use without running proprietary GitHub JavaScript), though I couldn't work out how to make clang link against msvcrt rather than its own libraries.

Thanks, but this is just a first small step in the right direction.
We need this verified with Emacs, not a small separate test program,
and we need then some serious testing of whatever solution we decide
to implement.  All of that requires a volunteer who will take upon
him/herself to implement and test the possible solutions on the actual
platform, and then maintaining that and the rest of UCRT-specific
stuff for some reasonably observable future.  IOW, this needs a
dedicated support person.

> > And, as I
> > said, support for UCRT will only become official if someone volunteers
> > to come on board and handle that build.
> 
> That's what I meant by "if and when the time comes".
> 
> > > And any other implementation of the Windows API that doesn't provide bug-for-bug compatibility with MSVCRT.
> >
> > For now, only the MSVCRT support is official.
> 
> Of course, and I think it's more likely it will cease being official than other builds becoming official.
> 
> > > Wine is the only significant free implementation of the Windows API, so I disagree with the "just". If I understand RMS correctly, whether there is a free implementation of an OS is an important factor in deciding to redirect resources to supporting it in GNU software.
> >  
> > Such decisions can only be relevant if there are resources to allocate
> > and redirect.
> 
> I think the community of Emacs developers decides by consensus (or bickering) what to work on. My understanding was that we're considering whether to stop working on at least some variants of the Windows port.

The community of Emacs developers cannot allocate resources that don't
exist.  And even when they do exist, the ability of the community to
redirect those resources is limited by the opinions and limitations of
the people whose resources need to be redirected.

> > Since we don't have them, this is entirely academic.
> 
> Well, I disagree. Official guidance by the GNU Emacs project does have some influence on what volunteers will work on. I consider that a resource.

I'm around for enough time to know that it is almost never a resource
we can control.  And the effectiveness of such guidance is severely
limited even in easier cases, see the recent arguments about naming
packages and similar stuff.

> > For now, the only person who actively works on the Windows port is
> > myself, and I'm unable to support UCRT, for more than one reason.
> 
> Are you working on MINGW64 systems at all, or is it mingw exclusively?

I'm using MinGW and don't intend to install MinGW64 any time soon.
And, as I said earlier on the list, even supporting the MSVCRT build
is becoming too much if I have to do that alone, together with my main
responsibilities here.

Re: MPS: Win64 testers?

[Pip Cet]

On Monday, July 29th, 2024 at 11:25, Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Sun, 28 Jul 2024 20:22:40 +0000
>
> > From: Pip Cet pipcet@protonmail.com
> > Cc: emacs-devel@gnu.org, rms@gnu.org
> >
> > On Sunday, July 28th, 2024 at 15:20, Eli Zaretskii eliz@gnu.org wrote:
> >
> > > > By my reading of the source code I found, it's doing the same as Wine, not as MSVCRT.
> > >
> > > If someone steps through that code in the UCRT build and reports that
> > > fdopen returns a different FILE pointer, I will believe you.
> >
> > I've verified this by running a GitHub action (I won't provide a link because it's probably impossible to use without running proprietary GitHub JavaScript), though I couldn't work out how to make clang link against msvcrt rather than its own libraries.
>
> Thanks, but this is just a first small step in the right direction.
> We need this verified with Emacs, not a small separate test program,
> and we need then some serious testing of whatever solution we decide
> to implement.

I think the bar is slightly lower than that: the code in Emacs is clearly buggy, because it relies on strange and peculiar implementation details that go far beyond anything guaranteed by the API (and that may break at any point on new systems). Replacing it is necessary.

> All of that requires a volunteer who will take upon
> him/herself to implement and test the possible solutions on the actual
> platform, and then maintaining that and the rest of UCRT-specific
> stuff for some reasonably observable future.

This isn't UCRT-specific, and, as far as I'm concerned, "the actual platform" is Wine, not Microsoft Windows. If a UCRT build works on wine, it's quite likely to work on Microsoft Windows machines, too.

> IOW, this needs a
> dedicated support person.

That's desirable, yes, but hardly necessary. There's a difference between fixing obvious bugs that will prevent builds from ever working and deciding a platform is "officially" supported.

FWIW, I've tested my UCRT build on Microsoft's proprietary Windows, briefly, and it starts up and appears to work. However, I cannot do so on a regular basis.

> > > And, as I
> > > said, support for UCRT will only become official if someone volunteers
> > > to come on board and handle that build.
> >
> > That's what I meant by "if and when the time comes".
> >
> > > > And any other implementation of the Windows API that doesn't provide bug-for-bug compatibility with MSVCRT.
> > >
> > > For now, only the MSVCRT support is official.
> >
> > Of course, and I think it's more likely it will cease being official than other builds becoming official.
> >
> > > > Wine is the only significant free implementation of the Windows API, so I disagree with the "just". If I understand RMS correctly, whether there is a free implementation of an OS is an important factor in deciding to redirect resources to supporting it in GNU software.
> > >
> > > Such decisions can only be relevant if there are resources to allocate
> > > and redirect.
> >
> > I think the community of Emacs developers decides by consensus (or bickering) what to work on. My understanding was that we're considering whether to stop working on at least some variants of the Windows port.
>
> The community of Emacs developers cannot allocate resources that don't
> exist. And even when they do exist, the ability of the community to
> redirect those resources is limited by the opinions and limitations of
> the people whose resources need to be redirected.

One thing we can certainly stop doing is to discourage people from even looking at stuff. Closing actual bugs as "wontfix" without a sensible explanation, for example, seems counterproductive to me.

> > > Since we don't have them, this is entirely academic.
> >
> > Well, I disagree. Official guidance by the GNU Emacs project does have some influence on what volunteers will work on. I consider that a resource.
>
> I'm around for enough time to know that it is almost never a resource
> we can control. And the effectiveness of such guidance is severely
> limited even in easier cases, see the recent arguments about naming
> packages and similar stuff.
>
> > > For now, the only person who actively works on the Windows port is
> > > myself, and I'm unable to support UCRT, for more than one reason.
> >
> > Are you working on MINGW64 systems at all, or is it mingw exclusively?
>
>
> I'm using MinGW and don't intend to install MinGW64 any time soon.

Maybe it's time to make that port unofficial, or at least to stop directing people to it rather than the MinGW64 port.

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Wed, 31 Jul 2024 06:18:16 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: emacs-devel@gnu.org, rms@gnu.org
> 
> > Thanks, but this is just a first small step in the right direction.
> > We need this verified with Emacs, not a small separate test program,
> > and we need then some serious testing of whatever solution we decide
> > to implement.
> 
> I think the bar is slightly lower than that: the code in Emacs is clearly buggy, because it relies on strange and peculiar implementation details that go far beyond anything guaranteed by the API (and that may break at any point on new systems). Replacing it is necessary.

I disagree.

> > All of that requires a volunteer who will take upon
> > him/herself to implement and test the possible solutions on the actual
> > platform, and then maintaining that and the rest of UCRT-specific
> > stuff for some reasonably observable future.
> 
> This isn't UCRT-specific, and, as far as I'm concerned, "the actual platform" is Wine, not Microsoft Windows. If a UCRT build works on wine, it's quite likely to work on Microsoft Windows machines, too.

Not relevant to the requirement of having a dedicated support person.

> > IOW, this needs a
> > dedicated support person.
> 
> That's desirable, yes, but hardly necessary.

I disagree, and that's not for you to decide, with all due respect.

> There's a difference between fixing obvious bugs that will prevent builds from ever working and deciding a platform is "officially" supported.

This bug is far from "obvious", and finding a fix for it that will do
what the original code does requires non-trivial testing.  Until that
is done, I don't trust the proposed solution (and it was not a
complete proposal anyway).

> FWIW, I've tested my UCRT build on Microsoft's proprietary Windows, briefly, and it starts up and appears to work. However, I cannot do so on a regular basis.

I know.  Neither can I.  Which is why the basic requirement to have a
dedicated support person on board still stands.

> > The community of Emacs developers cannot allocate resources that don't
> > exist. And even when they do exist, the ability of the community to
> > redirect those resources is limited by the opinions and limitations of
> > the people whose resources need to be redirected.
> 
> One thing we can certainly stop doing is to discourage people from even looking at stuff. Closing actual bugs as "wontfix" without a sensible explanation, for example, seems counterproductive to me.

Which bugs where "closed as wontfix without a sensible explanation"?

> > I'm using MinGW and don't intend to install MinGW64 any time soon.
> 
> Maybe it's time to make that port unofficial, or at least to stop directing people to it rather than the MinGW64 port.

We have been advertising MinGW64 (with MSVCRT) for a long time, see
nt/INSTALL.W64.  But since it doesn't support Windows versions older
than Vista (or maybe even that is not supported anymore), we also
advertise MinGW, which does.

Re: MPS: Win64 testers?

[Sebastián Monía]

>> Date: Wed, 31 Jul 2024 06:18:16 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>> Cc: emacs-devel@gnu.org, rms@gnu.org

> We have been advertising MinGW64 (with MSVCRT) for a long time, see
> nt/INSTALL.W64.  But since it doesn't support Windows versions older
> than Vista (or maybe even that is not supported anymore), we also
> advertise MinGW, which does.

Is there any advantage to using UCRT (just had to look up what it is :))
instead of the current approach of MinGW/MinGW64 builds?

I've been using MinGW64 to build Emacs from scratch whenever I am forced
to work in Windows, with no hiccups.

Thanks,
Seb

Re: MPS: Win64 testers?

[Eli Zaretskii]

> From: Sebastián Monía <sebastian@sebasmonia.com>
> Cc: Pip Cet <pipcet@protonmail.com>,  emacs-devel@gnu.org
> Date: Wed, 31 Jul 2024 15:04:12 -0400
> 
> >> Date: Wed, 31 Jul 2024 06:18:16 +0000
> >> From: Pip Cet <pipcet@protonmail.com>
> >> Cc: emacs-devel@gnu.org, rms@gnu.org
> 
> > We have been advertising MinGW64 (with MSVCRT) for a long time, see
> > nt/INSTALL.W64.  But since it doesn't support Windows versions older
> > than Vista (or maybe even that is not supported anymore), we also
> > advertise MinGW, which does.
> 
> Is there any advantage to using UCRT (just had to look up what it is :))
> instead of the current approach of MinGW/MinGW64 builds?

TBH, I don't know enough about the differences.  UCRT is supposed to
be more modern, and so more compatible with ANSI C and perhaps also
with Posix.  (However, MinGW has for a very long time a support
library which provides ANSI C compliant implementations of various
functions, so this problem is largely solved even without UCRT.)

The fact that we don't have experts who can answer these questions is
one serious indication that dedicated support persons are needed if we
want to be able to use whatever good features UCRT provides to Emacs's
benefit.

Re: MPS: Win64 testers?

[pipcet]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Wed, 31 Jul 2024 06:18:16 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>> Cc: emacs-devel@gnu.org, rms@gnu.org
>>
>> > Thanks, but this is just a first small step in the right direction.
>> > We need this verified with Emacs, not a small separate test program,
>> > and we need then some serious testing of whatever solution we decide
>> > to implement.
>>
>> I think the bar is slightly lower than that: the code in Emacs is
>> clearly buggy, because it relies on strange and peculiar
>> implementation details that go far beyond anything guaranteed by the
>> API (and that may break at any point on new systems). Replacing it
>> is necessary.
>
> I disagree.

If you disagree that it relies on details that aren't guaranteed by the
API, can you provide an API reference that backs you up?

>> > All of that requires a volunteer who will take upon
>> > him/herself to implement and test the possible solutions on the actual
>> > platform, and then maintaining that and the rest of UCRT-specific
>> > stuff for some reasonably observable future.
>>
>> This isn't UCRT-specific, and, as far as I'm concerned, "the actual
>> platform" is Wine, not Microsoft Windows. If a UCRT build works on
>> wine, it's quite likely to work on Microsoft Windows machines, too.
>
> Not relevant to the requirement of having a dedicated support person.

We don't need a dedicated UCRT support person to unbreak Emacs on
Wine. That's, again, an opinion.

>> > IOW, this needs a
>> > dedicated support person.
>>
>> That's desirable, yes, but hardly necessary.
>
> I disagree, and that's not for you to decide, with all due respect.

I was trying to express an opinion (that fixing a bug shouldn't have to
wait for a test person we haven't driven away yet), not make a
decision. That part is up to you.

>
>> There's a difference between fixing obvious bugs that will prevent builds from ever working and deciding a platform is "officially" supported.
>
> This bug is far from "obvious", and finding a fix for it that will do
> what the original code does requires non-trivial testing.  Until that
> is done, I don't trust the proposed solution (and it was not a
> complete proposal anyway).

I agree that there is a proposed solution we can use, so far. But, yes,
the bug itself is obvious. Throwing away the return value of _fdopen is
simply not something the API allows you to do.

>> FWIW, I've tested my UCRT build on Microsoft's proprietary Windows, briefly, and it starts up and appears to work. However, I cannot do so on a regular basis.
>
> I know.  Neither can I.  Which is why the basic requirement to have a
> dedicated support person on board still stands.

I think it's the other way around: if we fix what UCRT bugs we can, we
can actually look for testers in earnest. But, again, that's an opinion,
and the decision is yours.

>> > The community of Emacs developers cannot allocate resources that don't
>> > exist. And even when they do exist, the ability of the community to
>> > redirect those resources is limited by the opinions and limitations of
>> > the people whose resources need to be redirected.
>>
>> One thing we can certainly stop doing is to discourage people from
>> even looking at stuff. Closing actual bugs as "wontfix" without a
>> sensible explanation, for example, seems counterproductive to me.
>
> Which bugs where "closed as wontfix without a sensible explanation"?

72335

>> > I'm using MinGW and don't intend to install MinGW64 any time soon.
>>
>> Maybe it's time to make that port unofficial, or at least to stop directing people to it rather than the MinGW64 port.
>
> We have been advertising MinGW64 (with MSVCRT) for a long time, see
> nt/INSTALL.W64.  But since it doesn't support Windows versions older
> than Vista (or maybe even that is not supported anymore), we also
> advertise MinGW, which does.

Indeed, but people look at nt/INSTALL first, usually, and that's about
an entirely different port which hardly anyone considers usable at this
point, as far as I can tell.

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sat, 03 Aug 2024 07:50:21 +0000
> From: pipcet@protonmail.com
> Cc: emacs-devel@gnu.org, rms@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> >> Date: Wed, 31 Jul 2024 06:18:16 +0000
> >> From: Pip Cet <pipcet@protonmail.com>
> >> Cc: emacs-devel@gnu.org, rms@gnu.org
> >>
> >> > Thanks, but this is just a first small step in the right direction.
> >> > We need this verified with Emacs, not a small separate test program,
> >> > and we need then some serious testing of whatever solution we decide
> >> > to implement.
> >>
> >> I think the bar is slightly lower than that: the code in Emacs is
> >> clearly buggy, because it relies on strange and peculiar
> >> implementation details that go far beyond anything guaranteed by the
> >> API (and that may break at any point on new systems). Replacing it
> >> is necessary.
> >
> > I disagree.
> 
> If you disagree that it relies on details that aren't guaranteed by the
> API, can you provide an API reference that backs you up?

I disagree that the bar is lower than what I described.  The existing
code relies on undocumented features, yes, but we have quite a lot of
that in the Windows and MSDOS ports, and there's nothing in particular
wrong with that.  When the undocumented features stop working in some
situations, we need to find a solution for those situations, and those
solutions must be tested within Emacs, not a toy test program, and
they must be tested thoroughly, including (in this case) the
verification that the handle is not inherited by Emacs sub-processes.

> >> One thing we can certainly stop doing is to discourage people from
> >> even looking at stuff. Closing actual bugs as "wontfix" without a
> >> sensible explanation, for example, seems counterproductive to me.
> >
> > Which bugs where "closed as wontfix without a sensible explanation"?
> 
> 72335

That bug is not closed.

> >> > I'm using MinGW and don't intend to install MinGW64 any time soon.
> >>
> >> Maybe it's time to make that port unofficial, or at least to stop directing people to it rather than the MinGW64 port.
> >
> > We have been advertising MinGW64 (with MSVCRT) for a long time, see
> > nt/INSTALL.W64.  But since it doesn't support Windows versions older
> > than Vista (or maybe even that is not supported anymore), we also
> > advertise MinGW, which does.
> 
> Indeed, but people look at nt/INSTALL first, usually

If they read it, it tells them that those instructions are for older
versions of Windows.

> and that's about an entirely different port which hardly anyone
> considers usable at this point, as far as I can tell.

It isn't an entirely different port, no.  The code commonality between
the two is close to 100%, and so is the functionality, the only
difference is that one is 32-bit, the other 64-bit.

Re: MPS: Win64 testers?

[pipcet]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Sat, 03 Aug 2024 07:50:21 +0000
>> From: pipcet@protonmail.com
>> Cc: emacs-devel@gnu.org, rms@gnu.org
>>
>> "Eli Zaretskii" <eliz@gnu.org> writes:
>>
>> >> Date: Wed, 31 Jul 2024 06:18:16 +0000
>> >> From: Pip Cet <pipcet@protonmail.com>
>> >> Cc: emacs-devel@gnu.org, rms@gnu.org
>> >>
>> >> > Thanks, but this is just a first small step in the right direction.
>> >> > We need this verified with Emacs, not a small separate test program,
>> >> > and we need then some serious testing of whatever solution we decide
>> >> > to implement.
>> >>
>> >> I think the bar is slightly lower than that: the code in Emacs is
>> >> clearly buggy, because it relies on strange and peculiar
>> >> implementation details that go far beyond anything guaranteed by the
>> >> API (and that may break at any point on new systems). Replacing it
>> >> is necessary.
>> >
>> > I disagree.
>>
>> If you disagree that it relies on details that aren't guaranteed by the
>> API, can you provide an API reference that backs you up?
>
> I disagree that the bar is lower than what I described.  The existing
> code relies on undocumented features, yes, but we have quite a lot of
> that in the Windows and MSDOS ports, and there's nothing in particular
> wrong with that.  When the undocumented features stop working in some
> situations, we need to find a solution for those situations, and those
> solutions must be tested within Emacs, not a toy test program, and
> they must be tested thoroughly, including (in this case) the
> verification that the handle is not inherited by Emacs sub-processes.

Thank you. I think I finally understand your position.

>> >> One thing we can certainly stop doing is to discourage people from
>> >> even looking at stuff. Closing actual bugs as "wontfix" without a
>> >> sensible explanation, for example, seems counterproductive to me.
>> >
>> > Which bugs where "closed as wontfix without a sensible explanation"?
>>
>> 72335
>
> That bug is not closed.

Oh, I see. You're right, I should have said "marked as wontfix". Thanks
for correcting me.

>> >> > I'm using MinGW and don't intend to install MinGW64 any time soon.
>> >>
>> >> Maybe it's time to make that port unofficial, or at least to stop directing people to it rather than the MinGW64 port.
>> >
>> > We have been advertising MinGW64 (with MSVCRT) for a long time, see
>> > nt/INSTALL.W64.  But since it doesn't support Windows versions older
>> > than Vista (or maybe even that is not supported anymore), we also
>> > advertise MinGW, which does.
>>
>> Indeed, but people look at nt/INSTALL first, usually
>
> If they read it, it tells them that those instructions are for older
> versions of Windows.

I think a typical user will look at INSTALL first, which directs them to
nt/INSTALL, which finally directs them to the file they were looking for
in the first place, if they know enough to realize that MinGW and
MinGW64 are different things. I think skipping one of those files would
be a good thing, and increase the probability of successful builds.

>
>> and that's about an entirely different port which hardly anyone
>> considers usable at this point, as far as I can tell.
>
> It isn't an entirely different port, no.  The code commonality between
> the two is close to 100%, and so is the functionality, the only
> difference is that one is 32-bit, the other 64-bit.

What I meant was that building according to the nt/INSTALL instructions
will produce something people won't be happy with, with very rare
exceptions, because it uses an entirely different toolchain, produces a
32-bit binary, and doesn't support as many features for a naive build
(such as native compilation).

As for the test person issue, would it be possible to start a new thread
on emacs-devel with a detailed call for volunteers for testing Emacs on
64-bit versions of Microsoft Windows? I think it would be great if we
could make it clear to people that support for such systems is dependent
on someone volunteering to test a provided ZIP file once in a while
(maybe we can limit ourselves to one test per week or so), running a few
lines of elisp but mostly reporting back on whether bin/emacs.exe
starts up at all.

Maybe it would also help to offer people the chance to respond by
private email rather than on the list?

What do you think?

Pip

Re: MPS: Win64 testers?

[Eli Zaretskii]

> Date: Sat, 03 Aug 2024 15:17:31 +0000
> From: pipcet@protonmail.com
> Cc: emacs-devel@gnu.org, rms@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> > It isn't an entirely different port, no.  The code commonality between
> > the two is close to 100%, and so is the functionality, the only
> > difference is that one is 32-bit, the other 64-bit.
> 
> What I meant was that building according to the nt/INSTALL instructions
> will produce something people won't be happy with, with very rare
> exceptions, because it uses an entirely different toolchain, produces a
> 32-bit binary, and doesn't support as many features for a naive build
> (such as native compilation).

No, the MinGW build supports all the features when it runs on the same
system as the MinGW64 build.  That includes native compilation (I have
it enabled in all my production builds, including the one where I'm
typing this, which is a 32-bit MinGW build with large-ints).

> As for the test person issue, would it be possible to start a new thread
> on emacs-devel with a detailed call for volunteers for testing Emacs on
> 64-bit versions of Microsoft Windows?

I already did that, see

  https://lists.gnu.org/archive/html/emacs-devel/2024-07/msg01072.html

Feel free to post there with the additional bits about UCRT.

This is not the first time I post about the need for UCRT support,
either, there were a couple of discussions in the past.

IOW, it isn't like the issue isn't known.

> Maybe it would also help to offer people the chance to respond by
> private email rather than on the list?

Why? the job is definitely a public one, it cannot be done in private
email.

Re: MPS: Lose64 testers?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

It is useful for those interested in running Emacs on Losedows to test
the MPS code on that system, but I request that we all join in
refusing to refer to Microsoft Losedows as a "win".

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: MPS: Win64 testers?

[Michael Albinus]

Pip Cet <pipcet@protonmail.com> writes:

Hi,

> What is the state of Emacs CI generally? emba.gnu.org doesn't even appear to be reachable at this point...

emba.gnu.org is in a bad state I cannot fix myself. I've asked several
times for help from people who are knowing GitLab better than I do. No
response, unfortunately.

> Pip

Best regards, Michael.

Re: MPS: Win64 testers?

[Pip Cet]

On Saturday, July 27th, 2024 at 16:09, Michael Albinus <michael.albinus@gmx.de> wrote:
> Pip Cet pipcet@protonmail.com writes:
> > What is the state of Emacs CI generally? emba.gnu.org doesn't even appear to be reachable at this point...
> 
> emba.gnu.org is in a bad state I cannot fix myself. I've asked several
> times for help from people who are knowing GitLab better than I do. No
> response, unfortunately.

Oh. That's a pity. Were those requests public? If so, do you have a link?

Would switching to forgejo be an option, maybe? That _claims_ to be easy to set up, and it's also the platform I'm playing with right now; and there seems to be a strong focus on keeping the project free in more than the technical sense, which is what irritates me about gitlab.com.

(The SourceHut software is also free, but doesn't claim to be, and isn't, easy to set up.)

No idea whether resources for such experiments are available though, and I don't know about the use of GNU resources for CI for free reimplementations of non-free operating systems, either with a wholly free toolchain (Wine) or a decidedly non-free SDK (Android-based); my understanding is that any significant development work for macOS requires non-free Apple software and a developer's license that you have to pay for, so I assume that one's out of the question.

I can volunteer to look at things as a one-off effort, but can't take over any regular or recurring maintenance work at this point.

Pip

Emba (was: MPS: Win64 testers?)

[Michael Albinus]

Pip Cet <pipcet@protonmail.com> writes:

Hi Pip,

> Oh. That's a pity. Were those requests public? If so, do you have a link?

Not at hand. IIRC, I've asked mainly on the
<emacs-build-automation@gnu.org> ML.

> Would switching to forgejo be an option, maybe? That _claims_ to be
> easy to set up, and it's also the platform I'm playing with right now;
> and there seems to be a strong focus on keeping the project free in
> more than the technical sense, which is what irritates me about
> gitlab.com.

We don't have a problem with GitLab a priori. It has served for several
years. But recently, we see many authorization failures I have no
explanation for, and we run out of space permanently, although there are
daily prune cron jobs. Somebody with solid GitLab and/or Docker
experience whould be able to fix this.

> No idea whether resources for such experiments are available though,
> and I don't know about the use of GNU resources for CI for free
> reimplementations of non-free operating systems, either with a wholly
> free toolchain (Wine) or a decidedly non-free SDK (Android-based); my
> understanding is that any significant development work for macOS
> requires non-free Apple software and a developer's license that you
> have to pay for, so I assume that one's out of the question.

ATM, we run CI only for GNU/Linix. There are no plans yetto extend this
for other terget systems.

> I can volunteer to look at things as a one-off effort, but can't take
> over any regular or recurring maintenance work at this point.

Thanks! I'm occupied the next days, but later this week, I'll come back
with this issue.

> Pip

Best regards, Michael.