bug#45821: 28.0.50; Add UDP support for Emacs on Windows

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Lars Ingebrigtsen]

This patch from Robert Pluim adds that support, but needs somebody that
actually uses Windows to test it.

As it stands you need to arrange for WORKING_SELECT_EMULATION to be
defined.

diff --git i/nt/inc/ms-w32.h w/nt/inc/ms-w32.h
index 1cce2c3062..ea6ba38dea 100644
--- i/nt/inc/ms-w32.h
+++ w/nt/inc/ms-w32.h
@@ -63,10 +63,11 @@ #define _CALLBACK_ __cdecl
    Look in <sys/time.h> for a timeval structure.  */
 #define HAVE_TIMEVAL 1

+#ifndef WORKING_SELECT_EMULATION
 /* Our select emulation does 1-byte read-ahead waiting for received
    packets, so datagrams are broken.  */
 #define BROKEN_DATAGRAM_SOCKETS 1
-
+#endif
 #define MAIL_USE_SYSTEM_LOCK 1

 /* Define to 1 if GCC-style __attribute__ ((__aligned__ (expr))) works. */
diff --git i/src/w32.c w/src/w32.c
index 698e10e234..c0457ff00f 100644
--- i/src/w32.c
+++ w/src/w32.c
@@ -8798,6 +8798,45 @@ _sys_wait_accept (int fd)
   return cp->status;
 }

+#ifdef WORKING_SELECT_EMULATION
+int
+_sys_wait_readable (int fd)
+{
+  HANDLE hEv;
+  child_process * cp;
+  int rc;
+
+  if (fd < 0 || fd >= MAXDESC)
+    return STATUS_READ_ERROR;
+
+  cp = fd_info[fd].cp;
+
+  if (cp == NULL || cp->fd != fd || cp->status != STATUS_READ_READY)
+    return STATUS_READ_ERROR;
+
+  cp->status = STATUS_READ_FAILED;
+
+  hEv = pfn_WSACreateEvent ();
+  rc = pfn_WSAEventSelect (SOCK_HANDLE (fd), hEv, FD_READ);
+  if (rc != SOCKET_ERROR)
+    {
+      do
+        {
+          rc = WaitForSingleObject (hEv, 500);
+          Sleep (5);
+        } while (rc == WAIT_TIMEOUT
+                 && cp->status != STATUS_READ_ERROR
+                 && cp->char_avail);
+      pfn_WSAEventSelect (SOCK_HANDLE (fd), NULL, 0);
+      if (rc == WAIT_OBJECT_0)
+        cp->status = STATUS_READ_SUCCEEDED;
+    }
+  pfn_WSACloseEvent (hEv);
+
+  return cp->status;
+}
+#endif
+
 int
 _sys_wait_connect (int fd)
 {
@@ -8923,10 +8962,16 @@ sys_read (int fd, char * buffer, unsigned int count)
 	      return -1;

 	    case STATUS_READ_SUCCEEDED:
-	      /* consume read-ahead char */
-	      *buffer++ = cp->chr;
-	      count--;
-	      nchars++;
+#ifdef WORKING_SELECT_EMULATION
+              /* select on sockets no longer requires a 1-byte read.  */
+              if (fd_info[fd].flags & FILE_SOCKET == 0)
+#endif
+		{
+		  /* consume read-ahead char */
+		  *buffer++ = cp->chr;
+		  count--;
+		  nchars++;
+		}
 	      cp->status = STATUS_READ_ACKNOWLEDGED;
 	      ResetEvent (cp->char_avail);

diff --git i/src/w32.h w/src/w32.h
index cf1dadf64c..cabe39fb6d 100644
--- i/src/w32.h
+++ w/src/w32.h
@@ -175,6 +175,9 @@ #define FILE_SERIAL             0x0800

 extern int _sys_read_ahead (int fd);
 extern int _sys_wait_accept (int fd);
+#ifdef WORKING_SELECT_EMULATION
+extern int _sys_wait_readable (int fd);
+#endif
 extern int _sys_wait_connect (int fd);

 extern HMODULE w32_delayed_load (Lisp_Object);
diff --git i/src/w32proc.c w/src/w32proc.c
index de33726905..376e49d13d 100644
--- i/src/w32proc.c
+++ w/src/w32proc.c
@@ -1225,7 +1225,12 @@ reader_thread (void *arg)
       else if (cp->fd >= 0 && (fd_info[cp->fd].flags & FILE_LISTEN) != 0)
 	rc = _sys_wait_accept (cp->fd);
       else
-	rc = _sys_read_ahead (cp->fd);
+#ifdef WORKING_SELECT_EMULATION
+        if (fd_info[cp->fd].flags & FILE_SOCKET)
+          rc = _sys_wait_readable (cp->fd);
+        else
+#endif
+          rc = _sys_read_ahead (cp->fd);

       /* Don't bother waiting for the event if we already have been
 	 told to exit by delete_child.  */



In GNU Emacs 28.0.50 (build 20, x86_64-pc-linux-gnu, GTK+ Version 3.24.23, cairo version 1.16.0)
 of 2021-01-12 built on xo
Repository revision: 78ef0a72fa57c05c4be1401b2304c106a02c257d
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Debian GNU/Linux bullseye/sid


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 12 Jan 2021 19:08:12 +0100
> 
> 
> This patch from Robert Pluim adds that support, but needs somebody that
> actually uses Windows to test it.
> 
> As it stands you need to arrange for WORKING_SELECT_EMULATION to be
> defined.

Why do we need that #ifdef?  Leaving it there means this is dead code
that will never be tested.

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 12 Jan 2021 19:08:12 +0100
>> 
>> 
>> This patch from Robert Pluim adds that support, but needs somebody that
>> actually uses Windows to test it.
>> 
>> As it stands you need to arrange for WORKING_SELECT_EMULATION to be
>> defined.
>
> Why do we need that #ifdef?  Leaving it there means this is dead code
> that will never be tested.

Itʼs a WIP testing patch. If we applied it weʼd remove the ifdef (and
maybe add a lisp-level control var to turn it off)

Robert

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>>> This patch from Robert Pluim adds that support, but needs somebody that
>>> actually uses Windows to test it.
>>> 
>>> As it stands you need to arrange for WORKING_SELECT_EMULATION to be
>>> defined.
>>
>> Why do we need that #ifdef?  Leaving it there means this is dead code
>> that will never be tested.
>
> Itʼs a WIP testing patch. If we applied it weʼd remove the ifdef (and
> maybe add a lisp-level control var to turn it off)

I tried applying the patch and building on Windows and...  er...
nothing happened?  That is, touching w32.c didn't lead to anything being
compiled.  So I guess I don't understand how this works under Windows.
(A mingw/non-mingw thing?)

I still think it would be very nice if we had UDP support under Windows,
so if somebody else could poke at this, that'd be cool.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Robert Pluim]

>>>>> On Tue, 20 Jul 2021 17:41:05 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    >>>> This patch from Robert Pluim adds that support, but needs somebody that
    >>>> actually uses Windows to test it.
    >>>> 
    >>>> As it stands you need to arrange for WORKING_SELECT_EMULATION to be
    >>>> defined.
    >>> 
    >>> Why do we need that #ifdef?  Leaving it there means this is dead code
    >>> that will never be tested.
    >> 
    >> Itʼs a WIP testing patch. If we applied it weʼd remove the ifdef (and
    >> maybe add a lisp-level control var to turn it off)

    Lars> I tried applying the patch and building on Windows and...  er...
    Lars> nothing happened?  That is, touching w32.c didn't lead to anything being
    Lars> compiled.  So I guess I don't understand how this works under Windows.
    Lars> (A mingw/non-mingw thing?)

    Lars> I still think it would be very nice if we had UDP support under Windows,
    Lars> so if somebody else could poke at this, that'd be cool.

Iʼve resurrected this patch, it seems to be working nicely. Adding a
lisp-level control variable seems more trouble than itʼs worth, I
think the risk of breaking stuff on all platforms is higher than
that of breaking stuff on Windows.

Are we too late for emacs-28?

Robert
-- 

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Robert Pluim]

>>>>> On Mon, 27 Sep 2021 18:47:47 +0200, Robert Pluim <rpluim@gmail.com> said:
    Robert> Iʼve resurrected this patch, it seems to be working nicely. Adding a
    Robert> lisp-level control variable seems more trouble than itʼs worth, I
    Robert> think the risk of breaking stuff on all platforms is higher than
    Robert> that of breaking stuff on Windows.

    Robert> Are we too late for emacs-28?

Although Iʼve broken TLS connections in the process (but not http), so
maybe weʼre too late.

Robert
-- 

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  45821@debbugs.gnu.org
> Date: Mon, 27 Sep 2021 18:47:47 +0200
> 
> Iʼve resurrected this patch, it seems to be working nicely. Adding a
> lisp-level control variable seems more trouble than itʼs worth, I
> think the risk of breaking stuff on all platforms is higher than
> that of breaking stuff on Windows.
> 
> Are we too late for emacs-28?

Yes, please wait until the release branch is cut.

Thanks.

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Robert Pluim]

>>>>> On Mon, 27 Sep 2021 19:45:22 +0200, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Mon, 27 Sep 2021 18:47:47 +0200, Robert Pluim <rpluim@gmail.com> said:
    Robert> Iʼve resurrected this patch, it seems to be working nicely. Adding a
    Robert> lisp-level control variable seems more trouble than itʼs worth, I
    Robert> think the risk of breaking stuff on all platforms is higher than
    Robert> that of breaking stuff on Windows.

    Robert> Are we too late for emacs-28?

    Robert> Although Iʼve broken TLS connections in the process (but not http), so
    Robert> maybe weʼre too late.

I did some more debugging on this. What's happening is that in
'wait_reading_process_output', when calling gnutls_try_handshake,
weʼre exceeding GNUTLS_EMACS_HANDSHAKES_LIMIT. That has a value of
6000, so my suspicion is that because the reader_thread no longer
actually reads, but just calls pfn_WSAEventSelect,
'wait_reading_process_output' just spins. With the original code,
weʼre calling gnutls_try_handshake perhaps 2 or 3 times, so I donʼt
understand why the new code behaves so differently (and this
definitely won't make emacs-28).

Interestingly: only 'M-x eww RET https://www.gnu.org' causes
this. 'M-x eww RET https://www.google.com' works fine with both
implementations.

Robert
-- 

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  45821@debbugs.gnu.org
> Date: Mon, 04 Oct 2021 19:47:43 +0200
> 
> I did some more debugging on this. What's happening is that in
> 'wait_reading_process_output', when calling gnutls_try_handshake,
> weʼre exceeding GNUTLS_EMACS_HANDSHAKES_LIMIT. That has a value of
> 6000, so my suspicion is that because the reader_thread no longer
> actually reads, but just calls pfn_WSAEventSelect,
> 'wait_reading_process_output' just spins.

It spins why? because sys_select doesn't wait at all? or doesn't wait
on a handle of the reader thread?

IOW, that would mean the simple state machine between the reader
thread and sys_select somehow became broken, and the question is how?

Or looking at it from a different aspect: how is calling
pfn_WSAEventSelect different from calling _sys_read_ahead, for the
purposes of the dance between the reader thread and sys_select?

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  45821@debbugs.gnu.org
>> Date: Mon, 27 Sep 2021 18:47:47 +0200
>>
>> Iʼve resurrected this patch, it seems to be working nicely. Adding a
>> lisp-level control variable seems more trouble than itʼs worth, I
>> think the risk of breaking stuff on all platforms is higher than
>> that of breaking stuff on Windows.
>>
>> Are we too late for emacs-28?
>
> Yes, please wait until the release branch is cut.

I guess it's time to push this to master now?

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Robert Pluim]

>>>>> On Mon, 11 Oct 2021 04:56:43 -0700, Stefan Kangas <stefan@marxist.se> said:

    Stefan> Eli Zaretskii <eliz@gnu.org> writes:
    >>> From: Robert Pluim <rpluim@gmail.com>
    >>> Cc: Eli Zaretskii <eliz@gnu.org>,  45821@debbugs.gnu.org
    >>> Date: Mon, 27 Sep 2021 18:47:47 +0200
    >>> 
    >>> Iʼve resurrected this patch, it seems to be working nicely. Adding a
    >>> lisp-level control variable seems more trouble than itʼs worth, I
    >>> think the risk of breaking stuff on all platforms is higher than
    >>> that of breaking stuff on Windows.
    >>> 
    >>> Are we too late for emacs-28?
    >> 
    >> Yes, please wait until the release branch is cut.

    Stefan> I guess it's time to push this to master now?

No, because it breaks TLS connections.

I am in fact amazed that the existing code works at all on MS-Windows,
since we end up doing multiple 1-byte reads whilst also doing
gnutls_try_handshake, but work it does. And this patch doesnʼt.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 133 bytes --]

Hi,

Wanted to ask what is the status on this change?
If you need selfhosting on Windows, I can help with that!

Best,
Alex


[-- Attachment #2: Type: text/html, Size: 1335 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Tue, 3 May 2022 18:56:47 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Hi,
    Alex> Wanted to ask what is the status on this change?
    Alex> If you need selfhosting on Windows, I can help with that!

It needs a developer with time and motivation to figure out why the
latest patch breaks TLS sometimes.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 916 bytes --]

Thanks Robert!

I would like to give it a shot! How can I get started and have the patch applied?
I imagine I would need to build Emacs myself and manually apply the patch? Or is it a branch / commit that I can use?

Best,
Alex

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Wednesday, May 4, 2022 12:35 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Tue, 3 May 2022 18:56:47 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Hi,
    Alex> Wanted to ask what is the status on this change?
    Alex> If you need selfhosting on Windows, I can help with that!

It needs a developer with time and motivation to figure out why the
latest patch breaks TLS sometimes.

Robert
--


[-- Attachment #2: Type: text/html, Size: 2847 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Wed, 4 May 2022 11:50:02 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Thanks Robert!
    Alex> I would like to give it a shot! How can I get started and have the patch applied?
    Alex> I imagine I would need to build Emacs myself and manually apply the patch? Or is it a branch / commit that I can use?

Yes, you'd need a git checkout of Emacs, to apply the patch, and then
build and test the result. There are build instructions for MS-Windows
in nt/INSTALL and nt/INSTALL.W64 in the Emacs source tree.

`dns-query' can be used to test UDP, and M-x eww can test TCP and TLS
(and thereʼs the test suite as well).

Once itʼs all working, post the updated patch here in the bug and it
will get reviewed.

Good luck!

Robert
-- 

bug#45821: 28.0.50; Add UDP support for Emacs on Windows

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> `dns-query' can be used to test UDP, and M-x eww can test TCP and TLS
> (and thereʼs the test suite as well).
>
> Once itʼs all working, post the updated patch here in the bug and it
> will get reviewed.

Alex, did you make any progress here?

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 2843 bytes --]

Wanted to bring an update:

  *   ✅I have managed to build emacs for Windows and run it, and so far I wasn’t able to see any issues with Eww or `dns-query`
     *   Can you recommend some extensive selfhosting options so I can try to repro any TLS issues?
     *   What tests are there to run? How can I easily do this?

Wanted to share some thoughts as someone coming for the first time to Emacs development:

  *   Can we make the documentation for developing for Emacs more discoverable? Maybe with a markdown / .org README?
     *   Are we allowed to update the README with more up-to date information, on the series of MYSYS packages needed to be downloaded, and potentially with links to blog posts that describe the process in
  *   Can we add sections about creating/ applying patches to the README?
     *   I had a bit of work to discover how  `git am` works , and the whole business of email patches, etc.
        *   It would be great if all of this will be part of the original README
  *   Where can I find more information about logging from C code?
     *   Ideally I would like to compile Emacs with some new C functions, and then easily observe the behavior of these functions
        *   Logging is the most useful thing to have, before you can think of advanced debugging, and having an easy way (with examples) for you to send logs somewhere (preferably Emacs) would be great
        *   Sure, debugger is great but that has the overhead of you getting used to GDB, etc.. (saw some documentation on EmacsWiki)
        *   I am curious what people do? Do they write to a stream and then pipe it to a file, and monitor it, or how do they get feedback from the C code they wrote?

Best,
Alex




Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Wednesday, May 4, 2022 5:32 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Wed, 4 May 2022 11:50:02 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Thanks Robert!
    Alex> I would like to give it a shot! How can I get started and have the patch applied?
    Alex> I imagine I would need to build Emacs myself and manually apply the patch? Or is it a branch / commit that I can use?

Yes, you'd need a git checkout of Emacs, to apply the patch, and then
build and test the result. There are build instructions for MS-Windows
in nt/INSTALL and nt/INSTALL.W64 in the Emacs source tree.

`dns-query' can be used to test UDP, and M-x eww can test TCP and TLS
(and thereʼs the test suite as well).

Once itʼs all working, post the updated patch here in the bug and it
will get reviewed.

Good luck!

Robert
--


[-- Attachment #2: Type: text/html, Size: 8657 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1: Type: text/plain, Size: 4406 bytes --]

Updates on the UDP patch behavior:

  *   I got confused during my validation by 2 things:
     *   ‘dns-query’ works even without your patch changes => it is not a good test ❌
        *   One test that worked for me was using ‘(make-network-process :name "udp-socket1" :remote '[127 0 0 1 1500] :type 'datagram)’
           *   If returns ‘(error "Unsupported connection type")’ if UDP is disabled ❌ , or the process if UDP is allowed✅


     *   The original patch doesn’t include the #define for ‘WORKING_SELECT_EMULATION’, and I had to manually add this define to my emacs build (via 'CFLAGS= -O0 -g3 -DWORKING_SELECT_EMULATION' )
        *   That’s why I got confused about the patch working initially , since my assumption was that if ‘dns-query’ works => UDP works as expected😉and I didn’t even consider that I was required to set up some more flags..

❌ Indeed, TLS is broken -> Eww to https://www.gnu.org<https://www.gnu.org/> fails to load the page ( see attached image – Emacs instance on the left, compiled with UDP patch, didn’t load gnu.org while on the right side- default Emacs build for 28.1 opens it without any issues)




[cid:image002.png@01D91E00.B0A3FA30]
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Sunday, January 1, 2023 3:01 PM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

Wanted to bring an update:

  *   ✅I have managed to build emacs for Windows and run it, and so far I wasn’t able to see any issues with Eww or `dns-query`
     *   Can you recommend some extensive selfhosting options so I can try to repro any TLS issues?
     *   What tests are there to run? How can I easily do this?

Wanted to share some thoughts as someone coming for the first time to Emacs development:

  *   Can we make the documentation for developing for Emacs more discoverable? Maybe with a markdown / .org README?
     *   Are we allowed to update the README with more up-to date information, on the series of MYSYS packages needed to be downloaded, and potentially with links to blog posts that describe the process in
  *   Can we add sections about creating/ applying patches to the README?
     *   I had a bit of work to discover how  `git am` works , and the whole business of email patches, etc.
        *   It would be great if all of this will be part of the original README
  *   Where can I find more information about logging from C code?
     *   Ideally I would like to compile Emacs with some new C functions, and then easily observe the behavior of these functions
        *   Logging is the most useful thing to have, before you can think of advanced debugging, and having an easy way (with examples) for you to send logs somewhere (preferably Emacs) would be great
        *   Sure, debugger is great but that has the overhead of you getting used to GDB, etc.. (saw some documentation on EmacsWiki)
        *   I am curious what people do? Do they write to a stream and then pipe it to a file, and monitor it, or how do they get feedback from the C code they wrote?

Best,
Alex




Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Wednesday, May 4, 2022 5:32 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Wed, 4 May 2022 11:50:02 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Thanks Robert!
    Alex> I would like to give it a shot! How can I get started and have the patch applied?
    Alex> I imagine I would need to build Emacs myself and manually apply the patch? Or is it a branch / commit that I can use?

Yes, you'd need a git checkout of Emacs, to apply the patch, and then
build and test the result. There are build instructions for MS-Windows
in nt/INSTALL and nt/INSTALL.W64 in the Emacs source tree.

`dns-query' can be used to test UDP, and M-x eww can test TCP and TLS
(and thereʼs the test suite as well).

Once itʼs all working, post the updated patch here in the bug and it
will get reviewed.

Good luck!

Robert
--



[-- Attachment #1.2: Type: text/html, Size: 14970 bytes --]

[-- Attachment #2: 019BB84B0D7044EDBD941565E16AED26.png --]
[-- Type: image/png, Size: 383278 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Sun, 1 Jan 2023 23:01:04 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex>   *   Can we make the documentation for developing for Emacs more discoverable? Maybe with a markdown / .org README?
    Alex>      *   Are we allowed to update the README with more up-to
    Alex>      date information, on the series of MYSYS packages
    Alex>      needed to be downloaded, and potentially with links to
    Alex>      blog posts that describe the process in

That would go in INSTALL (I suspect a bunch of it is there already)

    Alex>   *   Can we add sections about creating/ applying patches to the README?
    Alex>      *   I had a bit of work to discover how  `git am` works , and the whole business of email patches, etc.
    Alex>         *   It would be great if all of this will be part of
    Alex>      the original README

Stuff about git, patches, etc, is documented in CONTRIBUTE

    Alex>   *   Where can I find more information about logging from C code?
    Alex>      *   Ideally I would like to compile Emacs with some new C functions, and then easily observe the behavior of these functions
    Alex>         *   Logging is the most useful thing to have, before you can think of advanced debugging, and having an easy way (with examples) for you to send logs somewhere (preferably Emacs) would be great
    Alex>         *   Sure, debugger is great but that has the overhead of you getting used to GDB, etc.. (saw some documentation on EmacsWiki)
    Alex>         *   I am curious what people do? Do they write to a stream and then pipe it to a file, and monitor it, or how do they get feedback from the C code they wrote?

Thereʼs nothing wrong with `fprintf' 😀. If you want stuff to stay at
the lisp level you can use `Fmessage' (or even `message1') which will
put things in the *Messages* buffer.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Mon, 2 Jan 2023 00:47:12 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Updates on the UDP patch behavior:
    Alex>   *   I got confused during my validation by 2 things:
    Alex>      *   ‘dns-query’ works even without your patch changes
    Alex>   => it is not a good test ❌

`dns-query' by default uses TCP on MS-Windows when thereʼs no UDP
support. You can use it, but youʼd have to either instrument it or
check your network traffic using wireshark or similar.

    Alex>         *   One test that worked for me was using ‘(make-network-process :name "udp-socket1" :remote '[127 0 0 1 1500] :type 'datagram)’
    Alex>            *   If returns ‘(error "Unsupported connection type")’ if UDP is disabled ❌ , or the process if UDP is allowed✅

(featurep 'make-network-process '(:type datagram)) is the canonical
way to check that. And it only proves you can create UDP sockets, not
that they work 😉

    Alex>      *   The original patch doesn’t include the #define for ‘WORKING_SELECT_EMULATION’, and I had to manually add this define to my emacs build (via 'CFLAGS= -O0 -g3 -DWORKING_SELECT_EMULATION' )
    Alex>         *   That’s why I got confused about the patch working initially , since my assumption was that if ‘dns-query’ works => UDP works as expected😉and I didn’t even consider that I was required to set up some more flags..

    Alex> ❌ Indeed, TLS is broken -> Eww to https://www.gnu.org<https://www.gnu.org/> fails to load the page ( see attached image – Emacs instance on the left, compiled with UDP patch, didn’t load gnu.org while on the right side- default Emacs build for 28.1 opens it without any issues)

Yep. Last time I looked at this, the TLS handshaking fails to complete
(see src/process.c around line 5329 and the checking against
GNUTLS_EMACS_HANDSHAKES_LIMIT) which means weʼre continually retrying
the handshake without giving the remote end a chance to send us
anything. Which I think means that our state machine for TLS
negotiation is subtly incorrect, but only on MS-Windows.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> Cc: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
> From: Alex Matei <matei.alexandru@live.com>
> Date: Sun, 1 Jan 2023 23:01:04 +0000
> 
> * Can we make the documentation for developing for Emacs more discoverable? Maybe with a markdown
>  / .org README?

Which development documentation did you find hard to discover?  Please
be more specific.

> * Are we allowed to update the README with more up-to date information, on the series of MYSYS
>  packages needed to be downloaded, and potentially with links to blog posts that describe the process in 

That was already done a few days ago, please look at the latest Git
repository, in the nt/INSTALL.W64 file.

> * Can we add sections about creating/ applying patches to the README?

This stuff is in CONTRIBUTE.

> * I had a bit of work to discover how  `git am` works , and the whole business of email patches, etc.

This is not specific to Emacs, so I don't think it's our job to teach
this.  You can always ask on emacs-devel if you have problems
figuring this stuff out.

> * It would be great if all of this will be part of the original README

The purpose of README is different, and it's already quite large.  We
describe this in CONTRIBUTE.

> * Where can I find more information about logging from C code?

Use the function 'message'.

> * Ideally I would like to compile Emacs with some new C functions, and then easily observe the behavior of
>  these functions 

I use the debugger when I need this.  It is much more flexible and
dfoesn't require me to decide up front what exactly I want to log.

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> Cc: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
> From: Robert Pluim <rpluim@gmail.com>
> Date: Mon, 02 Jan 2023 11:22:03 +0100
> 
>     Alex> ❌ Indeed, TLS is broken -> Eww to https://www.gnu.org<https://www.gnu.org/> fails to load the page ( see attached image – Emacs instance on the left, compiled with UDP patch, didn’t load gnu.org while on the right side- default Emacs build for 28.1 opens it without any issues)
> 
> Yep. Last time I looked at this, the TLS handshaking fails to complete
> (see src/process.c around line 5329 and the checking against
> GNUTLS_EMACS_HANDSHAKES_LIMIT) which means weʼre continually retrying
> the handshake without giving the remote end a chance to send us
> anything. Which I think means that our state machine for TLS
> negotiation is subtly incorrect, but only on MS-Windows.

On MS-Windows, there's another state machine involved, the one
vis-a-vis the reader thread we start to read the stuff from the
network connection.  See reader_thread and sys_select in w32proc.c and
sys_write, sys_read, _sys_read_ahead, _sys_wait_accept, and
_sys_wait_connect in w32.c.

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Mon, 02 Jan 2023 14:41:00 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> Cc: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
    >> From: Robert Pluim <rpluim@gmail.com>
    >> Date: Mon, 02 Jan 2023 11:22:03 +0100
    >> 
    Alex> ❌ Indeed, TLS is broken -> Eww to
    >> https://www.gnu.org<https://www.gnu.org/> fails to load the page (
    >> see attached image – Emacs instance on the left, compiled with UDP
    >> patch, didn’t load gnu.org while on the right side- default Emacs
    >> build for 28.1 opens it without any issues)
    >> 
    >> Yep. Last time I looked at this, the TLS handshaking fails to complete
    >> (see src/process.c around line 5329 and the checking against
    >> GNUTLS_EMACS_HANDSHAKES_LIMIT) which means weʼre continually retrying
    >> the handshake without giving the remote end a chance to send us
    >> anything. Which I think means that our state machine for TLS
    >> negotiation is subtly incorrect, but only on MS-Windows.

    Eli> On MS-Windows, there's another state machine involved, the one
    Eli> vis-a-vis the reader thread we start to read the stuff from the
    Eli> network connection.  See reader_thread and sys_select in w32proc.c and
    Eli> sys_write, sys_read, _sys_read_ahead, _sys_wait_accept, and
    Eli> _sys_wait_connect in w32.c.

Hmm, in that case I then suspect that `sys_select' is indicating that
the socket it connected even when it isnʼt. I took a look, but nothing
looks obviously wrong (and itʼs not something I can currently test).

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
> Date: Mon, 02 Jan 2023 14:29:50 +0100
> 
> >>>>> On Mon, 02 Jan 2023 14:41:00 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     >> Cc: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
>     >> From: Robert Pluim <rpluim@gmail.com>
>     >> Date: Mon, 02 Jan 2023 11:22:03 +0100
>     >> 
>     Alex> ❌ Indeed, TLS is broken -> Eww to
>     >> https://www.gnu.org<https://www.gnu.org/> fails to load the page (
>     >> see attached image – Emacs instance on the left, compiled with UDP
>     >> patch, didn’t load gnu.org while on the right side- default Emacs
>     >> build for 28.1 opens it without any issues)
>     >> 
>     >> Yep. Last time I looked at this, the TLS handshaking fails to complete
>     >> (see src/process.c around line 5329 and the checking against
>     >> GNUTLS_EMACS_HANDSHAKES_LIMIT) which means weʼre continually retrying
>     >> the handshake without giving the remote end a chance to send us
>     >> anything. Which I think means that our state machine for TLS
>     >> negotiation is subtly incorrect, but only on MS-Windows.
> 
>     Eli> On MS-Windows, there's another state machine involved, the one
>     Eli> vis-a-vis the reader thread we start to read the stuff from the
>     Eli> network connection.  See reader_thread and sys_select in w32proc.c and
>     Eli> sys_write, sys_read, _sys_read_ahead, _sys_wait_accept, and
>     Eli> _sys_wait_connect in w32.c.
> 
> Hmm, in that case I then suspect that `sys_select' is indicating that
> the socket it connected even when it isnʼt.

If that is the case, maybe we lack some flag(s) in the filedesc
structure.

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 3904 bytes --]

I like the nt/INSTALL.W64 file<https://github.com/emacs-mirror/emacs/blob/master/nt/INSTALL.W64> but somehow I found Step-by-step instructions to build Emacs for Windows 64 bit with MSYS2 and MinGW-w64. Now `native-comp` supported. (github.com)<https://gist.github.com/nauhygon/f3b44f51b34e89bc54f8> more useful. I was wondering if we are allowed to add links to such gists / blogposts? (I understand that they can become obsolete at some point, but a section like useful references would be nice)

  *   If you look at the gist, it gives a more detailed step-by-step approach around setting MYSYS
  *   I am not 100% that the packages listed in nt/INSTALL.W64 are sufficient, on a fresh installation of MYSYS -> I will have to check if there are any gaps

My problem with patching:

  *   I didn’t know how to consume @Robert Pluim<mailto:rpluim@gmail.com>’s change ☹ I had to read a blog post (Working with Git and patches in Emacs (ane.iki.fi)<https://ane.iki.fi/emacs/patches.html>) on how git am works and that I can manually save the text from his change to a file and then use that file to apply changes to my Emacs branch…
  *   What made it weird in the beginning was that I only had the information from #45821 - 28.0.50; Add UDP support for Emacs on Windows - GNU bug report logs<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45821> and most of the apply patch examples refer to you having received an email with the patches as an attachment and then performing actions on it (in my case, I didn’t have any email)

For me, having a good walkthrough with examples is really useful and that’s why I am asking if we can include links to such blogposts:

  *   I benefited from a walkthrough on MYSYS installation + emacs building
  *   I benefited from a walkthrough on email patches
I am hoping others could find this useful too when they first take a look at building Emacs themselves.

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

________________________________
From: Eli Zaretskii <eliz@gnu.org>
Sent: Monday, January 2, 2023 4:07:50 AM
To: Alex Matei <matei.alexandru@live.com>
Cc: rpluim@gmail.com <rpluim@gmail.com>; 45821@debbugs.gnu.org <45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> Cc: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
> From: Alex Matei <matei.alexandru@live.com>
> Date: Sun, 1 Jan 2023 23:01:04 +0000
>
> * Can we make the documentation for developing for Emacs more discoverable? Maybe with a markdown
>  / .org README?

Which development documentation did you find hard to discover?  Please
be more specific.

> * Are we allowed to update the README with more up-to date information, on the series of MYSYS
>  packages needed to be downloaded, and potentially with links to blog posts that describe the process in

That was already done a few days ago, please look at the latest Git
repository, in the nt/INSTALL.W64 file.

> * Can we add sections about creating/ applying patches to the README?

This stuff is in CONTRIBUTE.

> * I had a bit of work to discover how  `git am` works , and the whole business of email patches, etc.

This is not specific to Emacs, so I don't think it's our job to teach
this.  You can always ask on emacs-devel if you have problems
figuring this stuff out.

> * It would be great if all of this will be part of the original README

The purpose of README is different, and it's already quite large.  We
describe this in CONTRIBUTE.

> * Where can I find more information about logging from C code?

Use the function 'message'.

> * Ideally I would like to compile Emacs with some new C functions, and then easily observe the behavior of
>  these functions

I use the debugger when I need this.  It is much more flexible and
dfoesn't require me to decide up front what exactly I want to log.

[-- Attachment #2: Type: text/html, Size: 9131 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 2335 bytes --]

Thank you!
Yes I was looking for some LISP level logging options so planning on giving `Fmessage` / `message1` a try!


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Monday, January 2, 2023 2:10 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Sun, 1 Jan 2023 23:01:04 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex>   *   Can we make the documentation for developing for Emacs more discoverable? Maybe with a markdown / .org README?
    Alex>      *   Are we allowed to update the README with more up-to
    Alex>      date information, on the series of MYSYS packages
    Alex>      needed to be downloaded, and potentially with links to
    Alex>      blog posts that describe the process in

That would go in INSTALL (I suspect a bunch of it is there already)

    Alex>   *   Can we add sections about creating/ applying patches to the README?
    Alex>      *   I had a bit of work to discover how  `git am` works , and the whole business of email patches, etc.
    Alex>         *   It would be great if all of this will be part of
    Alex>      the original README

Stuff about git, patches, etc, is documented in CONTRIBUTE

    Alex>   *   Where can I find more information about logging from C code?
    Alex>      *   Ideally I would like to compile Emacs with some new C functions, and then easily observe the behavior of these functions
    Alex>         *   Logging is the most useful thing to have, before you can think of advanced debugging, and having an easy way (with examples) for you to send logs somewhere (preferably Emacs) would be great
    Alex>         *   Sure, debugger is great but that has the overhead of you getting used to GDB, etc.. (saw some documentation on EmacsWiki)
    Alex>         *   I am curious what people do? Do they write to a stream and then pipe it to a file, and monitor it, or how do they get feedback from the C code they wrote?

Thereʼs nothing wrong with `fprintf' 😀. If you want stuff to stay at
the lisp level you can use `Fmessage' (or even `message1') which will
put things in the *Messages* buffer.

Robert
--


[-- Attachment #2: Type: text/html, Size: 5043 bytes --]

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Alex Matei <matei.alexandru@live.com>
> CC: "rpluim@gmail.com" <rpluim@gmail.com>, "45821@debbugs.gnu.org"
> 	<45821@debbugs.gnu.org>
> Date: Mon, 2 Jan 2023 15:59:23 +0000
> 
> I like the nt/INSTALL.W64 file but somehow I found Step-by-step instructions to build Emacs for Windows 64
> bit with MSYS2 and MinGW-w64. Now `native-comp` supported. (github.com) more useful.

I don't see there anything that we don't have.  I do see a lot of
stuff unrelated to building Emacs per se, something that just sounds
like personal preferences of whoever wrote that post.

> I was wondering
> if we are allowed to add links to such gists / blogposts? (I understand that they can become obsolete at
> some point, but a section like useful references would be nice)

I don't think we should be responsible for someone else's posts.  It
is better to maintain our instructions to be accurate and with all the
necessary details.

> * If you look at the gist, it gives a more detailed step-by-step approach around setting MYSYS
> * I am not 100% that the packages listed in nt/INSTALL.W64 are sufficient, on a fresh installation of MYSYS
>  -> I will have to check if there are any gaps 

Please do tell if you find anything missing or incorrect, and we will
fix those deficiencies.

> My problem with patching: 
> 
> * I didn’t know how to consume @Robert Pluim’s change ☹ I had to read a blog post (Working with Git and
>  patches in Emacs (ane.iki.fi)) on how git am works and that I can manually save the text from his change
>  to a file and then use that file to apply changes to my Emacs branch… 
> * What made it weird in the beginning was that I only had the information from #45821 - 28.0.50; Add UDP
>  support for Emacs on Windows - GNU bug report logs and most of the apply patch examples refer to you
>  having received an email with the patches as an attachment and then performing actions on it (in my
>  case, I didn’t have any email)
> 
>  
> 
> For me, having a good walkthrough with examples is really useful and that’s why I am asking if we can
> include links to such blogposts:
> 
> * I benefited from a walkthrough on MYSYS installation + emacs building  
> * I benefited from a walkthrough on email patches
> 
> I am hoping others could find this useful too when they first take a look at building Emacs themselves.

This is just the steep learning curve, something that each one of us
has to go through just once.  It is impossible to have all this
information in our installation instructions.

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 3182 bytes --]

Thank you! I will take a look at the instructions that we have and try to reproduce from scratch and see if I run into any issues  😊

I get what you are saying, around the fact that you have to pay the price once, and then you are all set!

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Monday, January 2, 2023 9:01 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Alex Matei <matei.alexandru@live.com>
> CC: "rpluim@gmail.com" <rpluim@gmail.com>, "45821@debbugs.gnu.org"
>        <45821@debbugs.gnu.org>
> Date: Mon, 2 Jan 2023 15:59:23 +0000
>
> I like the nt/INSTALL.W64 file but somehow I found Step-by-step instructions to build Emacs for Windows 64
> bit with MSYS2 and MinGW-w64. Now `native-comp` supported. (github.com) more useful.

I don't see there anything that we don't have.  I do see a lot of
stuff unrelated to building Emacs per se, something that just sounds
like personal preferences of whoever wrote that post.

> I was wondering
> if we are allowed to add links to such gists / blogposts? (I understand that they can become obsolete at
> some point, but a section like useful references would be nice)

I don't think we should be responsible for someone else's posts.  It
is better to maintain our instructions to be accurate and with all the
necessary details.

> * If you look at the gist, it gives a more detailed step-by-step approach around setting MYSYS
> * I am not 100% that the packages listed in nt/INSTALL.W64 are sufficient, on a fresh installation of MYSYS
>  -> I will have to check if there are any gaps

Please do tell if you find anything missing or incorrect, and we will
fix those deficiencies.

> My problem with patching:
>
> * I didn’t know how to consume @Robert Pluim’s change ☹ I had to read a blog post (Working with Git and
>  patches in Emacs (ane.iki.fi)) on how git am works and that I can manually save the text from his change
>  to a file and then use that file to apply changes to my Emacs branch…
> * What made it weird in the beginning was that I only had the information from #45821 - 28.0.50; Add UDP
>  support for Emacs on Windows - GNU bug report logs and most of the apply patch examples refer to you
>  having received an email with the patches as an attachment and then performing actions on it (in my
>  case, I didn’t have any email)
>
>
>
> For me, having a good walkthrough with examples is really useful and that’s why I am asking if we can
> include links to such blogposts:
>
> * I benefited from a walkthrough on MYSYS installation + emacs building
> * I benefited from a walkthrough on email patches
>
> I am hoping others could find this useful too when they first take a look at building Emacs themselves.

This is just the steep learning curve, something that each one of us
has to go through just once.  It is impossible to have all this
information in our installation instructions.


[-- Attachment #2: Type: text/html, Size: 5572 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 3672 bytes --]

Is there anything we can do to speed up make install? Smth like just copying `emacs.exe` for small, incremental changes?

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Monday, January 2, 2023 9:57 AM
To: Eli Zaretskii<mailto:eliz@gnu.org>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

Thank you! I will take a look at the instructions that we have and try to reproduce from scratch and see if I run into any issues  😊

I get what you are saying, around the fact that you have to pay the price once, and then you are all set!

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Monday, January 2, 2023 9:01 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Alex Matei <matei.alexandru@live.com>
> CC: "rpluim@gmail.com" <rpluim@gmail.com>, "45821@debbugs.gnu.org"
>        <45821@debbugs.gnu.org>
> Date: Mon, 2 Jan 2023 15:59:23 +0000
>
> I like the nt/INSTALL.W64 file but somehow I found Step-by-step instructions to build Emacs for Windows 64
> bit with MSYS2 and MinGW-w64. Now `native-comp` supported. (github.com) more useful.

I don't see there anything that we don't have.  I do see a lot of
stuff unrelated to building Emacs per se, something that just sounds
like personal preferences of whoever wrote that post.

> I was wondering
> if we are allowed to add links to such gists / blogposts? (I understand that they can become obsolete at
> some point, but a section like useful references would be nice)

I don't think we should be responsible for someone else's posts.  It
is better to maintain our instructions to be accurate and with all the
necessary details.

> * If you look at the gist, it gives a more detailed step-by-step approach around setting MYSYS
> * I am not 100% that the packages listed in nt/INSTALL.W64 are sufficient, on a fresh installation of MYSYS
>  -> I will have to check if there are any gaps

Please do tell if you find anything missing or incorrect, and we will
fix those deficiencies.

> My problem with patching:
>
> * I didn’t know how to consume @Robert Pluim’s change ☹ I had to read a blog post (Working with Git and
>  patches in Emacs (ane.iki.fi)) on how git am works and that I can manually save the text from his change
>  to a file and then use that file to apply changes to my Emacs branch…
> * What made it weird in the beginning was that I only had the information from #45821 - 28.0.50; Add UDP
>  support for Emacs on Windows - GNU bug report logs and most of the apply patch examples refer to you
>  having received an email with the patches as an attachment and then performing actions on it (in my
>  case, I didn’t have any email)
>
>
>
> For me, having a good walkthrough with examples is really useful and that’s why I am asking if we can
> include links to such blogposts:
>
> * I benefited from a walkthrough on MYSYS installation + emacs building
> * I benefited from a walkthrough on email patches
>
> I am hoping others could find this useful too when they first take a look at building Emacs themselves.

This is just the steep learning curve, something that each one of us
has to go through just once.  It is impossible to have all this
information in our installation instructions.



[-- Attachment #2: Type: text/html, Size: 6550 bytes --]

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Alex Matei <matei.alexandru@live.com>
> CC: "rpluim@gmail.com" <rpluim@gmail.com>, "45821@debbugs.gnu.org"
> 	<45821@debbugs.gnu.org>
> Date: Mon, 2 Jan 2023 19:07:33 +0000
> 
> Is there anything we can do to speed up make install? Smth like just copying `emacs.exe` for small,
> incremental changes?

Why do you need to invoke "make install"?  You can run Emacs from its
build tree, and it should work just fine from there.

When developing Emacs, it is never a good idea to install the binary
you've built.

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

OMG🤯

I totally missed this, expected to have issues due to DLLs not being present, etc… launching directly the binary makes life easier! Thank you!

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Monday, January 2, 2023 11:22 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Alex Matei <matei.alexandru@live.com>
> CC: "rpluim@gmail.com" <rpluim@gmail.com>, "45821@debbugs.gnu.org"
>        <45821@debbugs.gnu.org>
> Date: Mon, 2 Jan 2023 19:07:33 +0000
>
> Is there anything we can do to speed up make install? Smth like just copying `emacs.exe` for small,
> incremental changes?

Why do you need to invoke "make install"?  You can run Emacs from its
build tree, and it should work just fine from there.

When developing Emacs, it is never a good idea to install the binary
you've built.


[-- Attachment #2: Type: text/html, Size: 3007 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1.1: Type: text/plain, Size: 5910 bytes --]

Thanks for the tips with logging!

I think I found the fix (at least for one of the issues?)
Please see the attached patch/diff for the fix.

The code will always  ignore (aka ‘eat’) one character if the file descriptor had the least significant bit set (0x1, so this would mean for any file descriptor with READ flag set)
[cid:image004.png@01D91EBA.5498F140]

Based on operator precedence rules, FILE_SOCKET == 0 will be evaluated first (before &) , resulting in 0, which would effectively cause the condition to always be false

This behavior can be observed when executing a call to  ‘(async-shell-command)’ (-> file descriptor flags 0x191) on a build with the patch vs one without the patch applied (see below, attached image)

  *   My suspicion is that this behavior is similar to what TLS was experiencing but just easier to reproduce (note: there might other issues with TLS , but so far, after applying this fix I have not run into any other issues)

[cid:image005.png@01D91EBA.5498F140]


@Robert Pluim<mailto:rpluim@gmail.com> I don’t know if this was the only missing piece, but from all my tests I couldn’t see any issues with TLS anymore, and with navigating https://gnu.org in EWW..
I wanted to thank you for creating this patch 🙏, and giving me pointers on how to apply and debug it.

In the meantime, I will continue to selfhost EWW and see if there are still other issues lurking around beyond this fix.


Funny enough, I ended up with good old ‘printf’, since it was nicely integrated with my build workflow: I would build and then just run emacs.exe from the shell => all my logging was nicely displayed there. ( message1 is also super nice, especially for useful user messages).


p.s.
- The patch also adds STATUS_READ_IN_PROGRESS state for the new ‘_sys_wait_readable’ function , based on what the read ahead logic was doing already (idk if it is needed, but it is a nice mirror, and a good status code to reflect, although I am not convinced anyone cares about this state transition..)
- I checked in other places of the code if we have issues with operator precedence for comparing socket flags, etc. and we are safe, most probably because we don’t bother with explicit equality conditions (like == 0) and instead we use the short form `if ( flag & socket_flag)`)

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Monday, January 2, 2023 5:38 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: matei.alexandru@live.com<mailto:matei.alexandru@live.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Robert Pluim <rpluim@gmail.com>
> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
> Date: Mon, 02 Jan 2023 14:29:50 +0100
>
> >>>>> On Mon, 02 Jan 2023 14:41:00 +0200, Eli Zaretskii <eliz@gnu.org> said:
>
>     >> Cc: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
>     >> From: Robert Pluim <rpluim@gmail.com>
>     >> Date: Mon, 02 Jan 2023 11:22:03 +0100
>     >>
>     Alex> ❌ Indeed, TLS is broken -> Eww to
>     >> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gnu.org%2F&data=05%7C01%7C%7Cb554d88b24e74c4839a108daecc69c01%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638082635001245924%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FqTmb2XqqC%2B9k7jFDvImdYlbUMXMJKCWJ3sNSV7hJVE%3D&reserved=0<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gnu.org%2F&data=05%7C01%7C%7Cb554d88b24e74c4839a108daecc69c01%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638082635001245924%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FqTmb2XqqC%2B9k7jFDvImdYlbUMXMJKCWJ3sNSV7hJVE%3D&reserved=0><https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gnu.org%2F&data=05%7C01%7C%7Cb554d88b24e74c4839a108daecc69c01%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638082635001245924%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FqTmb2XqqC%2B9k7jFDvImdYlbUMXMJKCWJ3sNSV7hJVE%3D&reserved=0%3chttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gnu.org%2F&data=05%7C01%7C%7Cb554d88b24e74c4839a108daecc69c01%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638082635001245924%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FqTmb2XqqC%2B9k7jFDvImdYlbUMXMJKCWJ3sNSV7hJVE%3D&reserved=0%3e> fails to load the page (
>     >> see attached image – Emacs instance on the left, compiled with UDP
>     >> patch, didn’t load gnu.org while on the right side- default Emacs
>     >> build for 28.1 opens it without any issues)
>     >>
>     >> Yep. Last time I looked at this, the TLS handshaking fails to complete
>     >> (see src/process.c around line 5329 and the checking against
>     >> GNUTLS_EMACS_HANDSHAKES_LIMIT) which means weʼre continually retrying
>     >> the handshake without giving the remote end a chance to send us
>     >> anything. Which I think means that our state machine for TLS
>     >> negotiation is subtly incorrect, but only on MS-Windows.
>
>     Eli> On MS-Windows, there's another state machine involved, the one
>     Eli> vis-a-vis the reader thread we start to read the stuff from the
>     Eli> network connection.  See reader_thread and sys_select in w32proc.c and
>     Eli> sys_write, sys_read, _sys_read_ahead, _sys_wait_accept, and
>     Eli> _sys_wait_connect in w32.c.
>
> Hmm, in that case I then suspect that `sys_select' is indicating that
> the socket it connected even when it isnʼt.

If that is the case, maybe we lack some flag(s) in the filedesc
structure.


[-- Attachment #1.1.2: Type: text/html, Size: 14553 bytes --]

[-- Attachment #1.2: 14D9BBA19389403AAE7B04F6D09528C2.png --]
[-- Type: image/png, Size: 32135 bytes --]

[-- Attachment #1.3: 22F9767EA55B4BA2BDE6B3027D69828E.png --]
[-- Type: image/png, Size: 86913 bytes --]

[-- Attachment #2: 45821-fix.txt --]
[-- Type: text/plain, Size: 1105 bytes --]

diff --git a/src/w32.c b/src/w32.c
index e11fa805f2..01a8616839 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -8928,7 +8928,7 @@ _sys_wait_readable (int fd)
   if (cp == NULL || cp->fd != fd || cp->status != STATUS_READ_READY)
     return STATUS_READ_ERROR;
 
-  cp->status = STATUS_READ_FAILED;
+  cp->status = STATUS_READ_IN_PROGRESS;
 
   hEv = pfn_WSACreateEvent ();
   rc = pfn_WSAEventSelect (SOCK_HANDLE (fd), hEv, FD_READ);
@@ -8944,6 +8944,8 @@ _sys_wait_readable (int fd)
       pfn_WSAEventSelect (SOCK_HANDLE (fd), NULL, 0);
       if (rc == WAIT_OBJECT_0)
         cp->status = STATUS_READ_SUCCEEDED;
+      else
+	cp->status = STATUS_READ_FAILED;
     }
   pfn_WSACloseEvent (hEv);
 
@@ -9078,7 +9080,7 @@ sys_read (int fd, char * buffer, unsigned int count)
 	    case STATUS_READ_SUCCEEDED:
 #ifdef WORKING_SELECT_EMULATION
               /* select on sockets no longer requires a 1-byte read.  */
-              if (fd_info[fd].flags & FILE_SOCKET == 0)
+              if ((fd_info[fd].flags & FILE_SOCKET) == 0)
 #endif
 		{
 		  /* consume read-ahead char */

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 2707 bytes --]

>>>>> On Mon, 2 Jan 2023 22:56:04 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Thanks for the tips with logging!
    Alex> I think I found the fix (at least for one of the issues?)
    Alex> Please see the attached patch/diff for the fix.

    Alex> The code will always ignore (aka ‘eat’) one character if the file
    Alex> descriptor had the least significant bit set (0x1, so this would mean
    Alex> for any file descriptor with READ flag set)
    Alex> [cid:image004.png@01D91EBA.5498F140]

Hmm, my local version already had that fixed, so I guess I posted the
wrong patch to the bug. But I still think you found a fix (see below)

    Alex> Based on operator precedence rules, FILE_SOCKET == 0 will be evaluated
    Alex> first (before &) , resulting in 0, which would effectively cause the
    Alex> condition to always be false

    Alex> This behavior can be observed when executing a call to
    Alex> ‘(async-shell-command)’ (-> file descriptor flags 0x191) on a build
    Alex> with the patch vs one without the patch applied (see below, attached
    Alex> image)

    Alex>   * My suspicion is that this behavior is similar to what TLS was
    Alex> experiencing but just easier to reproduce (note: there might other
    Alex> issues with TLS , but so far, after applying this fix I have not run
    Alex> into any other issues)

    Alex> [cid:image005.png@01D91EBA.5498F140]


    Alex> @Robert Pluim<mailto:rpluim@gmail.com> I don’t know if this was the
    Alex> only missing piece, but from all my tests I couldn’t see any issues
    Alex> with TLS anymore, and with navigating https://gnu.org in EWW..
    Alex> I wanted to thank you for creating this patch 🙏, and giving me pointers on how to apply and debug it.

Iʼd run the networking portion of our test suite to see if everything
works as intended (you might need to install the GnuTLS cli tools). eg:

    cd test
    make network-stream-tests
    
    Alex> p.s.
    Alex> - The patch also adds STATUS_READ_IN_PROGRESS state for the new
    Alex> ‘_sys_wait_readable’ function , based on what the read ahead logic was
    Alex> doing already (idk if it is needed, but it is a nice mirror, and a
    Alex> good status code to reflect, although I am not convinced anyone cares
    Alex> about this state transition..)

I think this is actually the fix, since I added it locally and TLS
started working, but I donʼt understand the code enough to be definite
about that. More testing required 😀

Can you post the full patch just to ensure weʼre all talking about the
same changes? Iʼve attached what Iʼm working from

Robert
-- 


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-UDP-sockets-work-on-Windows.patch --]
[-- Type: text/x-diff, Size: 5046 bytes --]

From 2ae91ed495f3972ddac383bd5f63c47946d5cdb5 Mon Sep 17 00:00:00 2001
From: Robert Pluim <rpluim@gmail.com>
Date: Mon, 27 Sep 2021 17:25:27 +0200
Subject: [PATCH] Make UDP sockets work on Windows
To: emacs-devel@gnu.org

* admin/CPP-DEFINES: remove BROKEN_DATAGRAM_SOCKETS
* nt/inc/ms-w32.h: remove BROKEN_DATAGRAM_SOCKETS.
* src/process.c (DATAGRAM_SOCKETS): remove dependency on
!BROKEN_DATAGRAM_SOCKETS.
* src/w32.c (_sys_wait_readable): New function.  Calls
pfn_WSAEventSelect for sockets to see if data can be read, so the
1-byte readahead is no longer necessary.
(sys_read): Only do 1-byte readahead for non-sockets.
* src/w32.h: Add prototype for _sys_wait_readable.
* src/w32proc.c (reader_thread): Call _sys_wait_readable for socket
handles.
---
 admin/CPP-DEFINES |  1 -
 nt/inc/ms-w32.h   |  4 ----
 src/process.c     |  8 +++-----
 src/w32.c         | 49 +++++++++++++++++++++++++++++++++++++++++++----
 src/w32.h         |  1 +
 src/w32proc.c     |  2 ++
 6 files changed, 51 insertions(+), 14 deletions(-)

diff --git a/admin/CPP-DEFINES b/admin/CPP-DEFINES
index 06986ec8f48..2704bc57675 100644
--- a/admin/CPP-DEFINES
+++ b/admin/CPP-DEFINES
@@ -103,7 +103,6 @@ getting at the full user name.  Only MSDOS overrides the default.
 anymore, so they can be removed.
 
 AMPERSAND_FULL_NAME
-BROKEN_DATAGRAM_SOCKETS
 BROKEN_GET_CURRENT_DIR_NAME
 BROKEN_PTY_READ_AFTER_EAGAIN
 DEFAULT_SOUND_DEVICE
diff --git a/nt/inc/ms-w32.h b/nt/inc/ms-w32.h
index 58be1199345..535d24cd57b 100644
--- a/nt/inc/ms-w32.h
+++ b/nt/inc/ms-w32.h
@@ -89,10 +89,6 @@ #define _CALLBACK_ __cdecl
    Look in <sys/time.h> for a timeval structure.  */
 #define HAVE_TIMEVAL 1
 
-/* Our select emulation does 1-byte read-ahead waiting for received
-   packets, so datagrams are broken.  */
-#define BROKEN_DATAGRAM_SOCKETS 1
-
 #define MAIL_USE_SYSTEM_LOCK 1
 
 /* Define to 1 if GCC-style __attribute__ ((__aligned__ (expr))) works. */
diff --git a/src/process.c b/src/process.c
index 67d1d3e425f..8f7fccfe538 100644
--- a/src/process.c
+++ b/src/process.c
@@ -234,11 +234,9 @@ #define PIPECONN1_P(p) (EQ (p->type, Qpipe))
    "non-destructive" select.  So we require either native select,
    or emulation of select using FIONREAD.  */
 
-#ifndef BROKEN_DATAGRAM_SOCKETS
-# if defined HAVE_SELECT || defined USABLE_FIONREAD
-#  if defined HAVE_SENDTO && defined HAVE_RECVFROM && defined EMSGSIZE
-#   define DATAGRAM_SOCKETS
-#  endif
+#if defined HAVE_SELECT || defined USABLE_FIONREAD
+# if defined HAVE_SENDTO && defined HAVE_RECVFROM && defined EMSGSIZE
+#  define DATAGRAM_SOCKETS
 # endif
 #endif
 
diff --git a/src/w32.c b/src/w32.c
index 47d79abc5b0..ef7d3de861b 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -8940,6 +8940,43 @@ _sys_wait_accept (int fd)
   return cp->status;
 }
 
+int
+_sys_wait_readable (int fd)
+{
+  HANDLE hEv;
+  child_process * cp;
+  int rc;
+
+  if (fd < 0 || fd >= MAXDESC)
+    return STATUS_READ_ERROR;
+
+  cp = fd_info[fd].cp;
+
+  if (cp == NULL || cp->fd != fd || cp->status != STATUS_READ_READY)
+    return STATUS_READ_ERROR;
+
+  cp->status = STATUS_READ_FAILED;
+
+  hEv = pfn_WSACreateEvent ();
+  rc = pfn_WSAEventSelect (SOCK_HANDLE (fd), hEv, FD_READ);
+  if (rc != SOCKET_ERROR)
+    {
+      do
+        {
+          rc = WaitForSingleObject (hEv, 500);
+          Sleep (5);
+        } while (rc == WAIT_TIMEOUT
+                 && cp->status != STATUS_READ_ERROR
+                 && cp->char_avail);
+      pfn_WSAEventSelect (SOCK_HANDLE (fd), NULL, 0);
+      if (rc == WAIT_OBJECT_0)
+        cp->status = STATUS_READ_SUCCEEDED;
+    }
+  pfn_WSACloseEvent (hEv);
+
+  return cp->status;
+}
+
 int
 _sys_wait_connect (int fd)
 {
@@ -9065,10 +9102,14 @@ sys_read (int fd, char * buffer, unsigned int count)
 	      return -1;
 
 	    case STATUS_READ_SUCCEEDED:
-	      /* consume read-ahead char */
-	      *buffer++ = cp->chr;
-	      count--;
-	      nchars++;
+              /* select on sockets no longer requires a 1-byte read.  */
+              if ((fd_info[fd].flags & FILE_SOCKET) == 0)
+		{
+		  /* consume read-ahead char */
+		  *buffer++ = cp->chr;
+		  count--;
+		  nchars++;
+		}
 	      cp->status = STATUS_READ_ACKNOWLEDGED;
 	      ResetEvent (cp->char_avail);
 
diff --git a/src/w32.h b/src/w32.h
index a3d0b75359a..4eba0fabe94 100644
--- a/src/w32.h
+++ b/src/w32.h
@@ -177,6 +177,7 @@ #define FILE_DONT_CLOSE         0x1000
 
 extern int _sys_read_ahead (int fd);
 extern int _sys_wait_accept (int fd);
+extern int _sys_wait_readable (int fd);
 extern int _sys_wait_connect (int fd);
 
 extern HMODULE w32_delayed_load (Lisp_Object);
diff --git a/src/w32proc.c b/src/w32proc.c
index 77a4ac1ff7e..19d9fc7d0e7 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -1243,6 +1243,8 @@ reader_thread (void *arg)
 	rc = _sys_wait_connect (fd);
       else if (fd >= 0 && (fd_info[fd].flags & FILE_LISTEN) != 0)
 	rc = _sys_wait_accept (fd);
+      else if (fd_info[fd].flags & FILE_SOCKET)
+	rc = _sys_wait_readable (fd);
       else
 	rc = _sys_read_ahead (fd);
 
-- 
2.38.1.420.g319605f8f0

bug#45821: Emacs UDP support on Windows

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alex Matei <matei.alexandru@live.com> writes:

> Thank you!
>
> Yes I was looking for some LISP level logging options so planning on giving `Fmessage` / `message1` a try!

If you want to print an object to the standard error stream, then:

  Fprint (foo, Qexternal_debugging_output);

where `foo' is the object you want to print.  Thanks for working on
Emacs.

bug#45821: Emacs UDP support on Windows

[Alex Matei]

Thabk you! Will give it a try!

@robert I will send a full patch as soon as I get back to my office.

Best
Alex 

Sent from my iPhone

> On Jan 3, 2023, at 5:30 AM, Po Lu <luangruo@yahoo.com> wrote:
> 
> Alex Matei <matei.alexandru@live.com> writes:
> 
>> Thank you!
>> 
>> Yes I was looking for some LISP level logging options so planning on giving `Fmessage` / `message1` a try!
> 
> If you want to print an object to the standard error stream, then:
> 
>  Fprint (foo, Qexternal_debugging_output);
> 
> where `foo' is the object you want to print.  Thanks for working on
> Emacs.

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1.1: Type: text/plain, Size: 4678 bytes --]

Here are the 2 patches that can be applied, in order:

  *   0001 -> the original patch from the email thread, that I used
  *   0002 -> the additions that I made locally

Note: TLS mostly works, but I’ve observed some weird behavior in the process list for EWW

  *   It almost feels like sometimes we hit snags in navigating to some HTTPS endpoints because the existing connections never get closes
  *   I think we need to make some investigations/ changes to the socket lifetime / close behavior to fix the class of issues we are seeing in EWW, where sometimes the page load hangs, and yes, you can load it after a couple of refreshes (‘g’ key binding) but then you see something really interesting if you take a look at the process list
     *   You will notice that some of the connection never close (properly) and hang around!
     *   If anyone has any good pointers of what are good functions to look at, please let me know; I will take a look at this later in the day, and navigate the reader thread and everything around it..

[cid:image003.png@01D91F6E.1C5C1280]

p.s. I didn’t find an obvious way of merging the two patches in magit, hence the 2 individual patches 😊 – I guess I can just rebase the commits into one..



Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Tuesday, January 3, 2023 12:51 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Mon, 2 Jan 2023 22:56:04 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Thanks for the tips with logging!
    Alex> I think I found the fix (at least for one of the issues?)
    Alex> Please see the attached patch/diff for the fix.

    Alex> The code will always ignore (aka ‘eat’) one character if the file
    Alex> descriptor had the least significant bit set (0x1, so this would mean
    Alex> for any file descriptor with READ flag set)
    Alex> [cid:image004.png@01D91EBA.5498F140]

Hmm, my local version already had that fixed, so I guess I posted the
wrong patch to the bug. But I still think you found a fix (see below)

    Alex> Based on operator precedence rules, FILE_SOCKET == 0 will be evaluated
    Alex> first (before &) , resulting in 0, which would effectively cause the
    Alex> condition to always be false

    Alex> This behavior can be observed when executing a call to
    Alex> ‘(async-shell-command)’ (-> file descriptor flags 0x191) on a build
    Alex> with the patch vs one without the patch applied (see below, attached
    Alex> image)

    Alex>   * My suspicion is that this behavior is similar to what TLS was
    Alex> experiencing but just easier to reproduce (note: there might other
    Alex> issues with TLS , but so far, after applying this fix I have not run
    Alex> into any other issues)

    Alex> [cid:image005.png@01D91EBA.5498F140]


    Alex> @Robert Pluim<mailto:rpluim@gmail.com> I don’t know if this was the
    Alex> only missing piece, but from all my tests I couldn’t see any issues
    Alex> with TLS anymore, and with navigating https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgnu.org%2F&data=05%7C01%7C%7C4f957b24a1204b5c8e5608daed67ab41%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638083326770863958%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oe0n96z7hwrRvkyKmiiG6Bpjn21g5OUcgkeV6pklPs4%3D&reserved=0 in EWW..
    Alex> I wanted to thank you for creating this patch 🙏, and giving me pointers on how to apply and debug it.

Iʼd run the networking portion of our test suite to see if everything
works as intended (you might need to install the GnuTLS cli tools). eg:

    cd test
    make network-stream-tests

    Alex> p.s.
    Alex> - The patch also adds STATUS_READ_IN_PROGRESS state for the new
    Alex> ‘_sys_wait_readable’ function , based on what the read ahead logic was
    Alex> doing already (idk if it is needed, but it is a nice mirror, and a
    Alex> good status code to reflect, although I am not convinced anyone cares
    Alex> about this state transition..)

I think this is actually the fix, since I added it locally and TLS
started working, but I donʼt understand the code enough to be definite
about that. More testing required 😀

Can you post the full patch just to ensure weʼre all talking about the
same changes? Iʼve attached what Iʼm working from

Robert
--


[-- Attachment #1.1.2: Type: text/html, Size: 11475 bytes --]

[-- Attachment #1.2: 51F8D5BC072047BE9A0B359605491906.png --]
[-- Type: image/png, Size: 348894 bytes --]

[-- Attachment #2: 0001-28.0.50-Add-UDP-support-for-Emacs-on-Windows.patch --]
[-- Type: application/octet-stream, Size: 3927 bytes --]

From 93fc2aad5383a43abd5476cda39fb30e24176d28 Mon Sep 17 00:00:00 2001
From: Lars Ingebrigtsen <larsi at>
Date: Tue, 12 Jan 2021 19:08:12 +0100
Subject: [PATCH 1/2] 28.0.50; Add UDP support for Emacs on Windows

This patch from Robert Pluim adds that support, but needs somebody that
actually uses Windows to test it.

As it stands you need to arrange for WORKING_SELECT_EMULATION to be
defined.
---
 nt/inc/ms-w32.h |  3 ++-
 src/w32.c       | 53 +++++++++++++++++++++++++++++++++++++++++++++----
 src/w32.h       |  3 +++
 src/w32proc.c   |  7 ++++++-
 4 files changed, 60 insertions(+), 6 deletions(-)

diff --git a/nt/inc/ms-w32.h b/nt/inc/ms-w32.h
index 3f4b2f3489..522d2d9ee5 100644
--- a/nt/inc/ms-w32.h
+++ b/nt/inc/ms-w32.h
@@ -89,10 +89,11 @@ #define _CALLBACK_ __cdecl
    Look in <sys/time.h> for a timeval structure.  */
 #define HAVE_TIMEVAL 1
 
+#ifndef WORKING_SELECT_EMULATION
 /* Our select emulation does 1-byte read-ahead waiting for received
    packets, so datagrams are broken.  */
 #define BROKEN_DATAGRAM_SOCKETS 1
-
+#endif
 #define MAIL_USE_SYSTEM_LOCK 1
 
 /* Define to 1 if GCC-style __attribute__ ((__aligned__ (expr))) works. */
diff --git a/src/w32.c b/src/w32.c
index 00b8c2515d..e11fa805f2 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -8912,6 +8912,45 @@ _sys_wait_accept (int fd)
   return cp->status;
 }
 
+#ifdef WORKING_SELECT_EMULATION
+int
+_sys_wait_readable (int fd)
+{
+  HANDLE hEv;
+  child_process * cp;
+  int rc;
+
+  if (fd < 0 || fd >= MAXDESC)
+    return STATUS_READ_ERROR;
+
+  cp = fd_info[fd].cp;
+
+  if (cp == NULL || cp->fd != fd || cp->status != STATUS_READ_READY)
+    return STATUS_READ_ERROR;
+
+  cp->status = STATUS_READ_FAILED;
+
+  hEv = pfn_WSACreateEvent ();
+  rc = pfn_WSAEventSelect (SOCK_HANDLE (fd), hEv, FD_READ);
+  if (rc != SOCKET_ERROR)
+    {
+      do
+        {
+          rc = WaitForSingleObject (hEv, 500);
+          Sleep (5);
+        } while (rc == WAIT_TIMEOUT
+                 && cp->status != STATUS_READ_ERROR
+                 && cp->char_avail);
+      pfn_WSAEventSelect (SOCK_HANDLE (fd), NULL, 0);
+      if (rc == WAIT_OBJECT_0)
+        cp->status = STATUS_READ_SUCCEEDED;
+    }
+  pfn_WSACloseEvent (hEv);
+
+  return cp->status;
+}
+#endif
+
 int
 _sys_wait_connect (int fd)
 {
@@ -9037,10 +9076,16 @@ sys_read (int fd, char * buffer, unsigned int count)
 	      return -1;
 
 	    case STATUS_READ_SUCCEEDED:
-	      /* consume read-ahead char */
-	      *buffer++ = cp->chr;
-	      count--;
-	      nchars++;
+#ifdef WORKING_SELECT_EMULATION
+              /* select on sockets no longer requires a 1-byte read.  */
+              if (fd_info[fd].flags & FILE_SOCKET == 0)
+#endif
+		{
+		  /* consume read-ahead char */
+		  *buffer++ = cp->chr;
+		  count--;
+		  nchars++;
+		}
 	      cp->status = STATUS_READ_ACKNOWLEDGED;
 	      ResetEvent (cp->char_avail);
 
diff --git a/src/w32.h b/src/w32.h
index 8a5c4ecbc7..50b03ae078 100644
--- a/src/w32.h
+++ b/src/w32.h
@@ -175,6 +175,9 @@ #define FILE_SERIAL             0x0800
 
 extern int _sys_read_ahead (int fd);
 extern int _sys_wait_accept (int fd);
+#ifdef WORKING_SELECT_EMULATION
+extern int _sys_wait_readable (int fd);
+#endif
 extern int _sys_wait_connect (int fd);
 
 extern HMODULE w32_delayed_load (Lisp_Object);
diff --git a/src/w32proc.c b/src/w32proc.c
index 3a6504c925..71337bed4d 100644
--- a/src/w32proc.c
+++ b/src/w32proc.c
@@ -1225,7 +1225,12 @@ reader_thread (void *arg)
       else if (cp->fd >= 0 && (fd_info[cp->fd].flags & FILE_LISTEN) != 0)
 	rc = _sys_wait_accept (cp->fd);
       else
-	rc = _sys_read_ahead (cp->fd);
+#ifdef WORKING_SELECT_EMULATION
+        if (fd_info[cp->fd].flags & FILE_SOCKET)
+          rc = _sys_wait_readable (cp->fd);
+        else
+#endif
+          rc = _sys_read_ahead (cp->fd);
 
       /* Don't bother waiting for the event if we already have been
 	 told to exit by delete_child.  */
-- 
2.37.2.windows.2


[-- Attachment #3: 0002-Fix-missing-characters-in-shell-and-potentially-TLS.patch --]
[-- Type: application/octet-stream, Size: 1393 bytes --]

From 17992fcf180451a6fb9508e33ad5c5573f7bdc9b Mon Sep 17 00:00:00 2001
From: Alex Matei <almat@microsoft.com>
Date: Tue, 3 Jan 2023 11:59:18 -0800
Subject: [PATCH 2/2] Fix missing characters in shell and potentially TLS

---
 src/w32.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/w32.c b/src/w32.c
index e11fa805f2..01a8616839 100644
--- a/src/w32.c
+++ b/src/w32.c
@@ -8928,7 +8928,7 @@ _sys_wait_readable (int fd)
   if (cp == NULL || cp->fd != fd || cp->status != STATUS_READ_READY)
     return STATUS_READ_ERROR;
 
-  cp->status = STATUS_READ_FAILED;
+  cp->status = STATUS_READ_IN_PROGRESS;
 
   hEv = pfn_WSACreateEvent ();
   rc = pfn_WSAEventSelect (SOCK_HANDLE (fd), hEv, FD_READ);
@@ -8944,6 +8944,8 @@ _sys_wait_readable (int fd)
       pfn_WSAEventSelect (SOCK_HANDLE (fd), NULL, 0);
       if (rc == WAIT_OBJECT_0)
         cp->status = STATUS_READ_SUCCEEDED;
+      else
+	cp->status = STATUS_READ_FAILED;
     }
   pfn_WSACloseEvent (hEv);
 
@@ -9078,7 +9080,7 @@ sys_read (int fd, char * buffer, unsigned int count)
 	    case STATUS_READ_SUCCEEDED:
 #ifdef WORKING_SELECT_EMULATION
               /* select on sockets no longer requires a 1-byte read.  */
-              if (fd_info[fd].flags & FILE_SOCKET == 0)
+              if ((fd_info[fd].flags & FILE_SOCKET) == 0)
 #endif
 		{
 		  /* consume read-ahead char */
-- 
2.37.2.windows.2

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Tue, 3 Jan 2023 20:22:58 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Here are the 2 patches that can be applied, in order:
    Alex>   *   0001 -> the original patch from the email thread, that I used
    Alex>   *   0002 -> the additions that I made locally

    Alex> Note: TLS mostly works, but I’ve observed some weird behavior in the process list for EWW

    Alex>   *   It almost feels like sometimes we hit snags in navigating to some HTTPS endpoints because the existing connections never get closes
    Alex>   *   I think we need to make some investigations/ changes to the socket lifetime / close behavior to fix the class of issues we are seeing in EWW, where sometimes the page load hangs, and yes, you can load it after a couple of refreshes (‘g’ key binding) but then you see something really interesting if you take a look at the process list
    Alex>      *   You will notice that some of the connection never
    Alex>   close (properly) and hang around!

Hmm, Iʼve not noticed anything like that, but I donʼt use Emacs on
windows regularly. Is this emacs-28 or emacs-29 or master? (I ask,
since emacs-29 did some reworking of some of the TLS code).

    Alex>      * If anyone has any good pointers of what are good
    Alex>      functions to look at, please let me know; I will take a
    Alex>      look at this later in the day, and navigate the reader
    Alex>      thread and everything around it..

    Alex> [cid:image003.png@01D91F6E.1C5C1280]

    Alex> p.s. I didn’t find an obvious way of merging the two patches in magit, hence the 2 individual patches 😊 – I guess I can just rebase the commits into one..

Either rebase, or if youʼve 'git apply' the changes from the second
patch you can add them in to the first patch using 'c e' (commit
extend). Of course you then need to to 'c w' to fix the commit
message, so rebase is the way to do it in one step.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

I’ve used the Emacs-28 branch :)

> On Jan 4, 2023, at 1:32 AM, Robert Pluim <rpluim@gmail.com> wrote:
> 
> 
>> 
>>>>>> On Tue, 3 Jan 2023 20:22:58 +0000, Alex Matei <matei.alexandru@live.com> said:
> 
>    Alex> Here are the 2 patches that can be applied, in order:
>    Alex>   *   0001 -> the original patch from the email thread, that I used
>    Alex>   *   0002 -> the additions that I made locally
> 
>    Alex> Note: TLS mostly works, but I’ve observed some weird behavior in the process list for EWW
> 
>    Alex>   *   It almost feels like sometimes we hit snags in navigating to some HTTPS endpoints because the existing connections never get closes
>    Alex>   *   I think we need to make some investigations/ changes to the socket lifetime / close behavior to fix the class of issues we are seeing in EWW, where sometimes the page load hangs, and yes, you can load it after a couple of refreshes (‘g’ key binding) but then you see something really interesting if you take a look at the process list
>    Alex>      *   You will notice that some of the connection never
>    Alex>   close (properly) and hang around!
> 
> Hmm, Iʼve not noticed anything like that, but I donʼt use Emacs on
> windows regularly. Is this emacs-28 or emacs-29 or master? (I ask,
> since emacs-29 did some reworking of some of the TLS code).
> 
>    Alex>      * If anyone has any good pointers of what are good
>    Alex>      functions to look at, please let me know; I will take a
>    Alex>      look at this later in the day, and navigate the reader
>    Alex>      thread and everything around it..
> 
>    Alex> [cid:image003.png@01D91F6E.1C5C1280]
> 
>    Alex> p.s. I didn’t find an obvious way of merging the two patches in magit, hence the 2 individual patches 😊 – I guess I can just rebase the commits into one..
> 
> Either rebase, or if youʼve 'git apply' the changes from the second
> patch you can add them in to the first patch using 'c e' (commit
> extend). Of course you then need to to 'c w' to fix the commit
> message, so rebase is the way to do it in one step.
> 
> Robert
> -- 

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Wed, 4 Jan 2023 10:15:22 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> I’ve used the Emacs-28 branch :)

Then I suggest you switch to master. Thereʼs no way Iʼm getting Eli to
put this in Emacs 29 :-)

Thanks

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

Cool, I cherry-picked the patch on the latest master and my eww (gnu.org - TLS) selfhosting so far didn’t show any issues ✅ (compared to what I was seeing in the old, 28.2 branch..)

  *   In terms of UDP, I was able to check that I was able to send UDP packets to a nodejs sever, without any loss or any kind of issues
  *   I am considering also running a udp server in emacs, and test that side of the house, not just client connections

I will continue selfhosting and let you know if I run into any issues with TLS.



Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Wednesday, January 4, 2023 2:50 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Wed, 4 Jan 2023 10:15:22 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> I’ve used the Emacs-28 branch :)

Then I suggest you switch to master. Thereʼs no way Iʼm getting Eli to
put this in Emacs 29 :-)

Thanks

Robert
--


[-- Attachment #2: Type: text/html, Size: 5663 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1: Type: text/plain, Size: 2081 bytes --]

Also, for those of you who use EWW , is it expected for one instance of EWW, after you navigated across couple of pages, to have multiple network connections opened?

[cid:image003.png@01D92104.A9B4BB40]

This behavior makes me think that we might have missed something in the new state machine based on WSAEventSelect …

  *   An inconsistency around reading / socket closing

Please confirm that this is not the expected behavior with 1 instance of eww, navigating through multiple nodes (e.g. https://gnu.org -> news -> shop -> etc…)


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Thursday, January 5, 2023 11:06 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

Cool, I cherry-picked the patch on the latest master and my eww (gnu.org - TLS) selfhosting so far didn’t show any issues ✅ (compared to what I was seeing in the old, 28.2 branch..)

  *   In terms of UDP, I was able to check that I was able to send UDP packets to a nodejs sever, without any loss or any kind of issues
  *   I am considering also running a udp server in emacs, and test that side of the house, not just client connections

I will continue selfhosting and let you know if I run into any issues with TLS.



Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Wednesday, January 4, 2023 2:50 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Wed, 4 Jan 2023 10:15:22 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> I’ve used the Emacs-28 branch :)

Then I suggest you switch to master. Thereʼs no way Iʼm getting Eli to
put this in Emacs 29 :-)

Thanks

Robert
--



[-- Attachment #1.2: Type: text/html, Size: 9748 bytes --]

[-- Attachment #2: 8E06BEB125AE46B19B744067623FACE0.png --]
[-- Type: image/png, Size: 104022 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1: Type: text/plain, Size: 2718 bytes --]

🤷‍♂️Nevermind, I tested with the official build too (28.1)  and seeing the same behavior -> I guess it shouldn’t be a regression..

[cid:image001.png@01D92105.DE036DA0]

✅Again, I haven’t run yet into an issue where EWW would fail to load a page..


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Thursday, January 5, 2023 12:53 PM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

Also, for those of you who use EWW , is it expected for one instance of EWW, after you navigated across couple of pages, to have multiple network connections opened?

[cid:image006.png@01D92105.DE036DA0]

This behavior makes me think that we might have missed something in the new state machine based on WSAEventSelect …

  *   An inconsistency around reading / socket closing

Please confirm that this is not the expected behavior with 1 instance of eww, navigating through multiple nodes (e.g. https://gnu.org -> news -> shop -> etc…)


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Thursday, January 5, 2023 11:06 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

Cool, I cherry-picked the patch on the latest master and my eww (gnu.org - TLS) selfhosting so far didn’t show any issues ✅ (compared to what I was seeing in the old, 28.2 branch..)

  *   In terms of UDP, I was able to check that I was able to send UDP packets to a nodejs sever, without any loss or any kind of issues
  *   I am considering also running a udp server in emacs, and test that side of the house, not just client connections

I will continue selfhosting and let you know if I run into any issues with TLS.



Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Wednesday, January 4, 2023 2:50 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Wed, 4 Jan 2023 10:15:22 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> I’ve used the Emacs-28 branch :)

Then I suggest you switch to master. Thereʼs no way Iʼm getting Eli to
put this in Emacs 29 :-)

Thanks

Robert
--




[-- Attachment #1.2: Type: text/html, Size: 11429 bytes --]

[-- Attachment #2: 044AA55213024A768CFB7C42C59A48CD.png --]
[-- Type: image/png, Size: 308867 bytes --]

[-- Attachment #3: 8B88BA654B274EAAADB1AE940B069F54.png --]
[-- Type: image/png, Size: 107138 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Thu, 5 Jan 2023 21:01:48 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> 🤷‍♂️Nevermind, I tested with the official build too (28.1)  and seeing the same behavior -> I guess it shouldn’t be a regression..

    Alex> Also, for those of you who use EWW , is it expected for one instance
    Alex> of EWW, after you navigated across couple of pages, to have multiple
    Alex> network connections opened?

Yes, I think thatʼs normal. Iʼve found that either eww or the
underlying url library will tend to open a new connection for each
resource on a web page, such as an image.

As long as those connections end up closing then things are as
expected.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Fri, 06 Jan 2023 08:56:05 +0100, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Thu, 5 Jan 2023 21:01:48 +0000, Alex Matei <matei.alexandru@live.com> said:
    Alex> 🤷‍♂️Nevermind, I tested with the official build too (28.1)
    Alex> and seeing the same behavior -> I guess it shouldn’t be a
    Alex> regression..

    Alex> Also, for those of you who use EWW , is it expected for one instance
    Alex> of EWW, after you navigated across couple of pages, to have multiple
    Alex> network connections opened?

    Robert> Yes, I think thatʼs normal. Iʼve found that either eww or the
    Robert> underlying url library will tend to open a new connection for each
    Robert> resource on a web page, such as an image.

    Robert> As long as those connections end up closing then things are as
    Robert> expected.

BTW, have you tried running the test-suite to see if all the TLS
related cases still pass? I canʼt run it here, since gnutls-serv
segfaults during the tests, and I donʼt have the energy to figure out
how to build a debuggable version on MS-Windows

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 1953 bytes --]

So far I only tried running `make check` and I can see what appear to be random failures in all kinds of places (including vc code)

  *   Now this may be because most of the tests rely on `process-read` etc. that might have been caused by this patch .. but it feels a bit far fetched..

Can you please tell me what would be the best way to run the tests for the scope of this change? Is there a specific subset of tests that we can call?

Best
Alex

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Saturday, January 7, 2023 3:24 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Fri, 06 Jan 2023 08:56:05 +0100, Robert Pluim <rpluim@gmail.com> said:

>>>>> On Thu, 5 Jan 2023 21:01:48 +0000, Alex Matei <matei.alexandru@live.com> said:
    Alex> 🤷‍♂️Nevermind, I tested with the official build too (28.1)
    Alex> and seeing the same behavior -> I guess it shouldn’t be a
    Alex> regression..

    Alex> Also, for those of you who use EWW , is it expected for one instance
    Alex> of EWW, after you navigated across couple of pages, to have multiple
    Alex> network connections opened?

    Robert> Yes, I think thatʼs normal. Iʼve found that either eww or the
    Robert> underlying url library will tend to open a new connection for each
    Robert> resource on a web page, such as an image.

    Robert> As long as those connections end up closing then things are as
    Robert> expected.

BTW, have you tried running the test-suite to see if all the TLS
related cases still pass? I canʼt run it here, since gnutls-serv
segfaults during the tests, and I donʼt have the energy to figure out
how to build a debuggable version on MS-Windows

Robert
--


[-- Attachment #2: Type: text/html, Size: 6651 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

On Sun, Jan 8, 2023, 16:31 Alex Matei <matei.alexandru@live.com> wrote:

> So far I only tried running `make check` and I can see what appear to be
> random failures in all kinds of places (including vc code)
>
>    - Now this may be because most of the tests rely on `process-read`
>    etc. that might have been caused by this patch .. but it feels a bit far
>    fetched..
>
>
>
> Can you please tell me what would be the best way to run the tests for the
> scope of this change? Is there a specific subset of tests that we can call?
>
>

cd test && make network-stream-tests

will run the UDP,TCP, and TLS tests (although most of the rest of the rest
suite should pass as well)

Robert

[-- Attachment #2: Type: text/html, Size: 1498 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

Thank you! Giving it a try 😊


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Sunday, January 8, 2023 7:42 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

On Sun, Jan 8, 2023, 16:31 Alex Matei <matei.alexandru@live.com<mailto:matei.alexandru@live.com>> wrote:
So far I only tried running `make check` and I can see what appear to be random failures in all kinds of places (including vc code)

  *   Now this may be because most of the tests rely on `process-read` etc. that might have been caused by this patch .. but it feels a bit far fetched..

Can you please tell me what would be the best way to run the tests for the scope of this change? Is there a specific subset of tests that we can call?


cd test && make network-stream-tests

will run the UDP,TCP, and TLS tests (although most of the rest of the rest suite should pass as well)

Robert


[-- Attachment #2: Type: text/html, Size: 6240 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]

Okay, this was helpful, I was able to hit an access violation error on the 4th test 😊 -> only issue that now I need to get up to speed with GDB since it looks like WinDBG (which is my goto debugger, and also setup as postmortem for any process that fails on my machine, including Emacs) cannot read the symbols generated by the GCC  compiler ☹

  *   Soo, any tips on how to easily run the tests under debugger?
  *   Or is there a way to set gdb to automatically launch whenever  a crash happens in Emacs?  -> this is what I typically do with WinDBG and it’s a pretty convenient feature

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Sunday, January 8, 2023 7:43 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

Thank you! Giving it a try 😊


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Sunday, January 8, 2023 7:42 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

On Sun, Jan 8, 2023, 16:31 Alex Matei <matei.alexandru@live.com<mailto:matei.alexandru@live.com>> wrote:
So far I only tried running `make check` and I can see what appear to be random failures in all kinds of places (including vc code)

  *   Now this may be because most of the tests rely on `process-read` etc. that might have been caused by this patch .. but it feels a bit far fetched..

Can you please tell me what would be the best way to run the tests for the scope of this change? Is there a specific subset of tests that we can call?


cd test && make network-stream-tests

will run the UDP,TCP, and TLS tests (although most of the rest of the rest suite should pass as well)

Robert



[-- Attachment #2: Type: text/html, Size: 10482 bytes --]

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Alex Matei <matei.alexandru@live.com>
> CC: Eli Zaretskii <eliz@gnu.org>, "45821@debbugs.gnu.org"
> 	<45821@debbugs.gnu.org>
> Date: Sun, 8 Jan 2023 16:00:27 +0000
> 
> Okay, this was helpful, I was able to hit an access violation error on the 4th test 😊 -> only issue that now I
> need to get up to speed with GDB since it looks like WinDBG (which is my goto debugger, and also setup as
> postmortem for any process that fails on my machine, including Emacs) cannot read the symbols generated
> by the GCC  compiler ☹
> 
> * Soo, any tips on how to easily run the tests under debugger?

Easy:

          cd /path/to/emacs/src
          gdb ./emacs.exe
in GDB:
          (gdb) run -Q
in Emacs:
          M-x load-file RET ../test/lisp/net/network-stream-tests.el RET
          M-x ert-run-tests-interactively RET THE-FAILING-TEST RET

where THE-FAILING-TEST is the name of the test which crashes.  Then
wait for the crash, at which point GDB will kick in.

> * Or is there a way to set gdb to automatically launch whenever  a crash happens in Emacs?  -> this is
>  what I typically do with WinDBG and it’s a pretty convenient feature

That is possible, but I don't recommend it: by the time the unhandled
exception will be thrown which starts GDB, it's too late: the original
crash was already converted into an unhandled exception, and you will
see in GDB a backtrace that has no traces of the code which actually
crashed.

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

This is amazing! 🙏Thank you

Let me give this a try!


From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Sunday, January 8, 2023 8:08 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Alex Matei <matei.alexandru@live.com>
> CC: Eli Zaretskii <eliz@gnu.org>, "45821@debbugs.gnu.org"
>        <45821@debbugs.gnu.org>
> Date: Sun, 8 Jan 2023 16:00:27 +0000
>
> Okay, this was helpful, I was able to hit an access violation error on the 4th test 😊 -> only issue that now I
> need to get up to speed with GDB since it looks like WinDBG (which is my goto debugger, and also setup as
> postmortem for any process that fails on my machine, including Emacs) cannot read the symbols generated
> by the GCC  compiler ☹
>
> * Soo, any tips on how to easily run the tests under debugger?

Easy:

          cd /path/to/emacs/src
          gdb ./emacs.exe
in GDB:
          (gdb) run -Q
in Emacs:
          M-x load-file RET ../test/lisp/net/network-stream-tests.el RET
          M-x ert-run-tests-interactively RET THE-FAILING-TEST RET

where THE-FAILING-TEST is the name of the test which crashes.  Then
wait for the crash, at which point GDB will kick in.

> * Or is there a way to set gdb to automatically launch whenever  a crash happens in Emacs?  -> this is
>  what I typically do with WinDBG and it’s a pretty convenient feature

That is possible, but I don't recommend it: by the time the unhandled
exception will be thrown which starts GDB, it's too late: the original
crash was already converted into an unhandled exception, and you will
see in GDB a backtrace that has no traces of the code which actually
crashed.


[-- Attachment #2: Type: text/html, Size: 4291 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 2526 bytes --]

Okay, it looks like the ‘connect-to-tls-ipv4-wait’ test just hangs (together with Emacs instance) so gdb still doesn’t capture any exceptions.

  *   It looks like this will require debugging and potentially some more tracing on my end 😊

It is great though that we have a good repro, that we can dive into! 🤞



Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Sunday, January 8, 2023 8:10 AM
To: Eli Zaretskii<mailto:eliz@gnu.org>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

This is amazing! 🙏Thank you

Let me give this a try!


From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Sunday, January 8, 2023 8:08 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Alex Matei <matei.alexandru@live.com>
> CC: Eli Zaretskii <eliz@gnu.org>, "45821@debbugs.gnu.org"
>        <45821@debbugs.gnu.org>
> Date: Sun, 8 Jan 2023 16:00:27 +0000
>
> Okay, this was helpful, I was able to hit an access violation error on the 4th test 😊 -> only issue that now I
> need to get up to speed with GDB since it looks like WinDBG (which is my goto debugger, and also setup as
> postmortem for any process that fails on my machine, including Emacs) cannot read the symbols generated
> by the GCC  compiler ☹
>
> * Soo, any tips on how to easily run the tests under debugger?

Easy:

          cd /path/to/emacs/src
          gdb ./emacs.exe
in GDB:
          (gdb) run -Q
in Emacs:
          M-x load-file RET ../test/lisp/net/network-stream-tests.el RET
          M-x ert-run-tests-interactively RET THE-FAILING-TEST RET

where THE-FAILING-TEST is the name of the test which crashes.  Then
wait for the crash, at which point GDB will kick in.

> * Or is there a way to set gdb to automatically launch whenever  a crash happens in Emacs?  -> this is
>  what I typically do with WinDBG and it’s a pretty convenient feature

That is possible, but I don't recommend it: by the time the unhandled
exception will be thrown which starts GDB, it's too late: the original
crash was already converted into an unhandled exception, and you will
see in GDB a backtrace that has no traces of the code which actually
crashed.



[-- Attachment #2: Type: text/html, Size: 8185 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Tue, 10 Jan 2023 12:41:12 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Okay, it looks like the ‘connect-to-tls-ipv4-wait’ test just hangs
    Alex> (together with Emacs instance) so gdb still doesn’t capture any
    Alex> exceptions.

    Alex>   *   It looks like this will require debugging and potentially some more tracing on my end 😊

    Alex> It is great though that we have a good repro, that we can dive into! 🤞

I see the same, but in my case itʼs because gnutls-serv has crashed,
so it never responds to emacs.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1: Type: text/plain, Size: 1717 bytes --]

I might be crazy 🤷‍♂️: from what I can tell, even the official build 28.1 of Emacs has the same issue:

  *   The network tests are not passing even for the official build 28.1! 🤯 ; I have manually downloaded the source code from emacs-mirror/emacs at emacs-28 (github.com)<https://github.com/emacs-mirror/emacs/tree/emacs-28>, and ran the networking tests using my existing 28.1 Emacs installation that uses the official Windows binaries
  *   As you can see in the attached screenshot: the network tests are failing in the same place, ‘connect-to-tls-ipv4-wait/nowait’

p.s. this doesn’t mean that we might not have a regression
p.p.s. this also doesn’t mean that we won’t fix these tests😃

[cid:image003.png@01D92570.FCF2DD80]

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Tuesday, January 10, 2023 5:56 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Tue, 10 Jan 2023 12:41:12 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Okay, it looks like the ‘connect-to-tls-ipv4-wait’ test just hangs
    Alex> (together with Emacs instance) so gdb still doesn’t capture any
    Alex> exceptions.

    Alex>   *   It looks like this will require debugging and potentially some more tracing on my end 😊

    Alex> It is great though that we have a good repro, that we can dive into! 🤞

I see the same, but in my case itʼs because gnutls-serv has crashed,
so it never responds to emacs.

Robert
--


[-- Attachment #1.2: Type: text/html, Size: 7070 bytes --]

[-- Attachment #2: 9261CDF71D904A868C9D36FFB275699E.png --]
[-- Type: image/png, Size: 326222 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1: Type: text/plain, Size: 2237 bytes --]

✅Network tests are passing in WSL Emacs.. => Windows has issues, and might have always had them? 🤷‍♂️
[cid:image004.png@01D9257C.C36EC810]

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Wednesday, January 11, 2023 5:09 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

I might be crazy 🤷‍♂️: from what I can tell, even the official build 28.1 of Emacs has the same issue:

  *   The network tests are not passing even for the official build 28.1! 🤯 ; I have manually downloaded the source code from emacs-mirror/emacs at emacs-28 (github.com)<https://github.com/emacs-mirror/emacs/tree/emacs-28>, and ran the networking tests using my existing 28.1 Emacs installation that uses the official Windows binaries
  *   As you can see in the attached screenshot: the network tests are failing in the same place, ‘connect-to-tls-ipv4-wait/nowait’

p.s. this doesn’t mean that we might not have a regression
p.p.s. this also doesn’t mean that we won’t fix these tests😃

[cid:image005.png@01D9257C.C36EC810]

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Tuesday, January 10, 2023 5:56 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Tue, 10 Jan 2023 12:41:12 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Okay, it looks like the ‘connect-to-tls-ipv4-wait’ test just hangs
    Alex> (together with Emacs instance) so gdb still doesn’t capture any
    Alex> exceptions.

    Alex>   *   It looks like this will require debugging and potentially some more tracing on my end 😊

    Alex> It is great though that we have a good repro, that we can dive into! 🤞

I see the same, but in my case itʼs because gnutls-serv has crashed,
so it never responds to emacs.

Robert
--



[-- Attachment #1.2: Type: text/html, Size: 8472 bytes --]

[-- Attachment #2: FB77C4BBBECE418EA4BACF8BEDE1D1DD.png --]
[-- Type: image/png, Size: 139870 bytes --]

[-- Attachment #3: 051B19ED630948D092DEBCD6BA4A83BD.png --]
[-- Type: image/png, Size: 350834 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Wed, 11 Jan 2023 13:23:03 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> ✅Network tests are passing in WSL Emacs.. => Windows has issues, and might have always had them? 🤷‍♂️

Thatʼs without a doubt true. Iʼm using MSYS2 to build Emacs and run
the tests, Iʼve not tried WSL at all. Eli, what environment should we
be using to build and test Emacs on MS-Windows? I know there are
legion, and I always forget the differences.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "45821@debbugs.gnu.org"
>  <45821@debbugs.gnu.org>
> Date: Thu, 12 Jan 2023 10:22:17 +0100
> 
> >>>>> On Wed, 11 Jan 2023 13:23:03 +0000, Alex Matei <matei.alexandru@live.com> said:
> 
>     Alex> ✅Network tests are passing in WSL Emacs.. => Windows has issues, and might have always had them? 🤷‍♂️
> 
> Thatʼs without a doubt true. Iʼm using MSYS2 to build Emacs and run
> the tests, Iʼve not tried WSL at all. Eli, what environment should we
> be using to build and test Emacs on MS-Windows? I know there are
> legion, and I always forget the differences.

The environment should be native Windows, not WSL.  WSL is Unix
running in a VM.

As for "using MSYS2", if that boils down to invoking Emacs from the
MSYS2 Bash prompt, it is still native Windows, and should not affect
the tests.

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

Thanks, that was my expectation.

To be clear, I ran WSL just to assert that the networking tests are correct & that they pass on Linux ✅.

❌When it comes to Windows, unfortunately I wasn’t able to find a baseline build where the network tests are passing ☹🤷‍♂️

  *   Running the tests using the official Windows binaries for Emacs 28.1 fail ❌
  *   Same applies when running the tests from my dev build of Emacs, using the latest master branch ❌
  *   And of course, same applies to my patched Emacs that has UDP support ❌

So my conclusion thus far is that the network tests didn’t pass on Windows for a while now, and I have no idea if our UDP patch is introducing regressions since I don’t have a good baseline.



Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Thursday, January 12, 2023 2:08 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: matei.alexandru@live.com<mailto:matei.alexandru@live.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "45821@debbugs.gnu.org"
>  <45821@debbugs.gnu.org>
> Date: Thu, 12 Jan 2023 10:22:17 +0100
>
> >>>>> On Wed, 11 Jan 2023 13:23:03 +0000, Alex Matei <matei.alexandru@live.com> said:
>
>     Alex> ✅Network tests are passing in WSL Emacs.. => Windows has issues, and might have always had them? 🤷‍♂️
>
> Thatʼs without a doubt true. Iʼm using MSYS2 to build Emacs and run
> the tests, Iʼve not tried WSL at all. Eli, what environment should we
> be using to build and test Emacs on MS-Windows? I know there are
> legion, and I always forget the differences.

The environment should be native Windows, not WSL.  WSL is Unix
running in a VM.

As for "using MSYS2", if that boils down to invoking Emacs from the
MSYS2 Bash prompt, it is still native Windows, and should not affect
the tests.


[-- Attachment #2: Type: text/html, Size: 7456 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Thu, 12 Jan 2023 12:08:54 +0200, Eli Zaretskii <eliz@gnu.org> said:

    Eli> The environment should be native Windows, not WSL.  WSL is Unix
    Eli> running in a VM.

    Eli> As for "using MSYS2", if that boils down to invoking Emacs from the
    Eli> MSYS2 Bash prompt, it is still native Windows, and should not affect
    Eli> the tests.

Yes, thatʼs what I do. And the tests hang for me because gnutls-serv
crashes, which is nothing to do with Emacs.

Is it worth me trying to get emacs and the tests to run from cmd.exe?
At the moment that has dll problems, but Iʼm sure theyʼre fixable.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
> Date: Thu, 12 Jan 2023 11:24:45 +0100
> 
> Is it worth me trying to get emacs and the tests to run from cmd.exe?

I don't see how would that be different?  And what exactly are the
problems that make this sound like it doesn't work OOTB?  I regularly
run tests in an interactive session I start from cmd.exe or from GDB,
and it always works.

> At the moment that has dll problems, but Iʼm sure theyʼre fixable.

What "dll problems"?

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

I don’t know – at least wasn’t able to pinpoint something given that even if I run the networking tests under GDB, my Emacs just hangs while running the tests, and I have to press C-g to unblock the test run..

Easy repro:

     *   Start Emacs (official build 28.1): typically via runemacs.exe
     *   Load network tests (load-file "../emacs-sourcecode/test/lisp/net/network-stream-tests.el")
     *   (ert-run-tests-interactively)

Expected:

  *   All tests pass

Actual:

  *   Test suit executes, and around test #4 you hit a snag, it hangs, and you need to press C-g to unblock it


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Thursday, January 12, 2023 2:46 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: matei.alexandru@live.com<mailto:matei.alexandru@live.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Robert Pluim <rpluim@gmail.com>
> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
> Date: Thu, 12 Jan 2023 11:24:45 +0100
>
> Is it worth me trying to get emacs and the tests to run from cmd.exe?

I don't see how would that be different?  And what exactly are the
problems that make this sound like it doesn't work OOTB?  I regularly
run tests in an interactive session I start from cmd.exe or from GDB,
and it always works.

> At the moment that has dll problems, but Iʼm sure theyʼre fixable.

What "dll problems"?


[-- Attachment #2: Type: text/html, Size: 6373 bytes --]

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1: Type: text/plain, Size: 2106 bytes --]

The only other thing I can think of that might be different from your setup @Eli Zaretskii<mailto:eliz@gnu.org> is the OS version…
I am running W11 21H2..

[cid:image002.png@01D92631.E8561970]


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Alex Matei<mailto:matei.alexandru@live.com>
Sent: Thursday, January 12, 2023 2:57 AM
To: Eli Zaretskii<mailto:eliz@gnu.org>; Robert Pluim<mailto:rpluim@gmail.com>
Cc: 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: RE: bug#45821: Emacs UDP support on Windows

I don’t know – at least wasn’t able to pinpoint something given that even if I run the networking tests under GDB, my Emacs just hangs while running the tests, and I have to press C-g to unblock the test run..

Easy repro:

     *   Start Emacs (official build 28.1): typically via runemacs.exe
     *   Load network tests (load-file "../emacs-sourcecode/test/lisp/net/network-stream-tests.el")
     *   (ert-run-tests-interactively)

Expected:

  *   All tests pass

Actual:

  *   Test suit executes, and around test #4 you hit a snag, it hangs, and you need to press C-g to unblock it


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Thursday, January 12, 2023 2:46 AM
To: Robert Pluim<mailto:rpluim@gmail.com>
Cc: matei.alexandru@live.com<mailto:matei.alexandru@live.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Robert Pluim <rpluim@gmail.com>
> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
> Date: Thu, 12 Jan 2023 11:24:45 +0100
>
> Is it worth me trying to get emacs and the tests to run from cmd.exe?

I don't see how would that be different?  And what exactly are the
problems that make this sound like it doesn't work OOTB?  I regularly
run tests in an interactive session I start from cmd.exe or from GDB,
and it always works.

> At the moment that has dll problems, but Iʼm sure theyʼre fixable.

What "dll problems"?



[-- Attachment #1.2: Type: text/html, Size: 8126 bytes --]

[-- Attachment #2: 70D7ADE8328E4AB1A520F14DE33737E1.png --]
[-- Type: image/png, Size: 14427 bytes --]

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Alex Matei <matei.alexandru@live.com>
> CC: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
> Date: Thu, 12 Jan 2023 10:57:31 +0000
> 
> I don’t know – at least wasn’t able to pinpoint something given that even if I run the networking tests
> under GDB, my Emacs just hangs while running the tests, and I have to press C-g to unblock the test run..

I think Robert explained that?  gnutls-serv crashes, so Emacs is
waiting forever for something that doesn't happen.  Or am I missing
something?

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

Yeah – I guess I wasn’t able to asses that myself but yeah, that might be the case 😊

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Eli Zaretskii<mailto:eliz@gnu.org>
Sent: Thursday, January 12, 2023 3:04 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: rpluim@gmail.com<mailto:rpluim@gmail.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

> From: Alex Matei <matei.alexandru@live.com>
> CC: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
> Date: Thu, 12 Jan 2023 10:57:31 +0000
>
> I don’t know – at least wasn’t able to pinpoint something given that even if I run the networking tests
> under GDB, my Emacs just hangs while running the tests, and I have to press C-g to unblock the test run..

I think Robert explained that?  gnutls-serv crashes, so Emacs is
waiting forever for something that doesn't happen.  Or am I missing
something?


[-- Attachment #2: Type: text/html, Size: 2789 bytes --]

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Thu, 12 Jan 2023 12:46:23 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
    >> Date: Thu, 12 Jan 2023 11:24:45 +0100
    >> 
    >> Is it worth me trying to get emacs and the tests to run from cmd.exe?

    Eli> I don't see how would that be different?  And what exactly are the
    Eli> problems that make this sound like it doesn't work OOTB?  I regularly
    Eli> run tests in an interactive session I start from cmd.exe or from GDB,
    Eli> and it always works.

Itʼs Windows: I always feel like thereʼs some voodoo lurking that mere
mortals such as myself canʼt ever understand 😀

    >> At the moment that has dll problems, but Iʼm sure theyʼre fixable.

    Eli> What "dll problems"?

Emacs fails to find the libgmp and libwinpthread dlls, presumably
because when run from cmd.exe the path isnʼt set up correctly (theyʼre
somewhere under C:\msys64\mingw64). Anyway, after fixing that, the
result is the same: gnutls-serv crashes and Emacs waits forever.

Perhaps I should build a 32-bit emacs with MSYS instead of a 64bit one
with MSYS2

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Thu, 12 Jan 2023 11:12:36 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Yeah – I guess I wasn’t able to asses that myself but yeah, that might be the case 😊

Itʼs easy enough to check. run gnutls-serv manually from a shell with
the same parameters as the test, comment out the server creation code
in the relevant test, and run it manually.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1.1: Type: text/plain, Size: 1355 bytes --]

Thanks!

For me, the process starts as expected -> I guess later messages might cause it to hang / crash

[cid:image003.png@01D92635.3522DE20]

Also, on the topic of voodoo magic:


I would be interested if anyone gets the networking tests to run successfully  on an official build for Windows (GNU Emacs download - GNU Project<https://www.gnu.org/software/emacs/download.html>).

  *   I think we should start here, and have this as a baseline

     *   Unless we are saying that the tests require some Voodoo magic that the official builds don’t have readily available …


Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Thursday, January 12, 2023 3:21 AM
To: Alex Matei<mailto:matei.alexandru@live.com>
Cc: Eli Zaretskii<mailto:eliz@gnu.org>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Thu, 12 Jan 2023 11:12:36 +0000, Alex Matei <matei.alexandru@live.com> said:

    Alex> Yeah – I guess I wasn’t able to asses that myself but yeah, that might be the case 😊

Itʼs easy enough to check. run gnutls-serv manually from a shell with
the same parameters as the test, comment out the server creation code
in the relevant test, and run it manually.

Robert
--


[-- Attachment #1.2: Type: text/html, Size: 6453 bytes --]

[-- Attachment #2: 03D0E96EFFAE4CCC88C548424A20C482.png --]
[-- Type: image/png, Size: 42569 bytes --]

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
> Date: Thu, 12 Jan 2023 12:18:51 +0100
> 
> Perhaps I should build a 32-bit emacs with MSYS instead of a 64bit one
> with MSYS2

Why would that change anything?  Are you saying that the 32-bit build
of gnutls-serv doesn't crash?

bug#45821: Emacs UDP support on Windows

[Robert Pluim]

>>>>> On Thu, 12 Jan 2023 14:25:18 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
    >> Date: Thu, 12 Jan 2023 12:18:51 +0100
    >> 
    >> Perhaps I should build a 32-bit emacs with MSYS instead of a 64bit one
    >> with MSYS2

    Eli> Why would that change anything?  Are you saying that the 32-bit build
    Eli> of gnutls-serv doesn't crash?

I have no idea, itʼs wild speculation.

Robert
-- 

bug#45821: Emacs UDP support on Windows

[Alex Matei]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

Okay,

So the best avenue we have right now is trying to build emacs for x86 and see if that can run the tests successfully?

To achieve this, I imagine I just need to replace w64 to w32 in the build process?

  ./configure \
      --host=x86_64-w64-mingw32 \
      --target=x86_64-w64-mingw32 \
      --build=x86_64-w64-mingw32 \

Thanks,
Alex
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows

From: Robert Pluim<mailto:rpluim@gmail.com>
Sent: Thursday, January 12, 2023 5:28 AM
To: Eli Zaretskii<mailto:eliz@gnu.org>
Cc: matei.alexandru@live.com<mailto:matei.alexandru@live.com>; 45821@debbugs.gnu.org<mailto:45821@debbugs.gnu.org>
Subject: Re: bug#45821: Emacs UDP support on Windows

>>>>> On Thu, 12 Jan 2023 14:25:18 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: matei.alexandru@live.com,  45821@debbugs.gnu.org
    >> Date: Thu, 12 Jan 2023 12:18:51 +0100
    >>
    >> Perhaps I should build a 32-bit emacs with MSYS instead of a 64bit one
    >> with MSYS2

    Eli> Why would that change anything?  Are you saying that the 32-bit build
    Eli> of gnutls-serv doesn't crash?

I have no idea, it’s wild speculation.

Robert
--


[-- Attachment #2: Type: text/html, Size: 3546 bytes --]

bug#45821: Emacs UDP support on Windows

[Eli Zaretskii]

> From: Alex Matei <matei.alexandru@live.com>
> CC: "45821@debbugs.gnu.org" <45821@debbugs.gnu.org>
> Date: Sat, 28 Jan 2023 21:17:55 +0000
> 
> So the best avenue we have right now is trying to build emacs for x86 and see if that can run the tests
> successfully?
> 
> To achieve this, I imagine I just need to replace w64 to w32 in the build process? 
> 
>   ./configure \
>       --host=x86_64-w64-mingw32 \
>       --target=x86_64-w64-mingw32 \
>       --build=x86_64-w64-mingw32 \

No, the 32-bit host names use i686, not x86_64.