[feature request] SOCK_SEQPACKET

[feature request] SOCK_SEQPACKET

[Daniel Hackney]

I'm currently working on Ezbl [1], an Emacs interface for the Uzbl [2]
web browser, and have hit a bit of a hitch. Uzbl communicates over a
socket connection using the SOCK_SEQPACKET socket type. I think they did
this because it preserves record boundaries, as opposed to
SOCK_STREAM. Regardless, I noticed that Emacs can't use the
SOCK_SEQPACKET type.

It doesn't seem like SOCK_SEQPACKET is used much, but it would be nice
to have, if only for completeness sake. I don't really know any C-level
Emacs programming (I'm pretty comfortable with Elisp), but I would be
willing to help out however I can if given some guidance and an overview
of what would need to be done.

Thanks!

[1] http://github.com/dhax/ezbl
[2] http://www.uzbl.org/

--
Daniel M. Hackney

Re: [feature request] SOCK_SEQPACKET

[Stefan Monnier]

> I'm currently working on Ezbl [1], an Emacs interface for the Uzbl [2]
> web browser, and have hit a bit of a hitch. Uzbl communicates over a
> socket connection using the SOCK_SEQPACKET socket type. I think they did
> this because it preserves record boundaries, as opposed to
> SOCK_STREAM. Regardless, I noticed that Emacs can't use the
> SOCK_SEQPACKET type.

> It doesn't seem like SOCK_SEQPACKET is used much, but it would be nice
> to have, if only for completeness sake. I don't really know any C-level
> Emacs programming (I'm pretty comfortable with Elisp), but I would be
> willing to help out however I can if given some guidance and an overview
> of what would need to be done.

You can start with the patch below (guaranteed 100% untested).  I know
nothing about SOCK_SEQPACKET, other than what I just read in the manpage
of `socket', so this code probably doesn't do the right thing.  The main
problem I'd except would have to do with the fact that Emacs will
typically read and write in blocks of fixed size (like 1KB or 4KB maybe,
IIRC), so for packets smaller than that size it might work fine, but for
larger packets you may end up having to make more substantial changes.


        Stefan


=== modified file 'src/process.c'
--- src/process.c	2009-11-24 15:30:54 +0000
+++ src/process.c	2009-11-24 20:30:20 +0000
@@ -127,7 +127,7 @@
 Lisp_Object Qprocessp;
 Lisp_Object Qrun, Qstop, Qsignal;
 Lisp_Object Qopen, Qclosed, Qconnect, Qfailed, Qlisten;
-Lisp_Object Qlocal, Qipv4, Qdatagram;
+Lisp_Object Qlocal, Qipv4, Qdatagram, Qseqpacket;
 Lisp_Object Qreal, Qnetwork, Qserial;
 #ifdef AF_INET6
 Lisp_Object Qipv6;
@@ -253,6 +253,10 @@
 #endif /* DATAGRAM_SOCKETS */
 #endif /* BROKEN_DATAGRAM_SOCKETS */
 
+#if defined HAVE_LOCAL_SOCKETS && defined DATAGRAM_SOCKETS
+# define HAVE_SEQPACKET
+#endif
+
 #if !defined (ADAPTIVE_READ_BUFFERING) && !defined (NO_ADAPTIVE_READ_BUFFERING)
 #ifdef EMACS_HAS_USECS
 #define ADAPTIVE_READ_BUFFERING
@@ -3299,6 +3303,10 @@
   else if (EQ (tem, Qdatagram))
     socktype = SOCK_DGRAM;
 #endif
+#ifdef HAVE_SEQPACKET
+  else if (EQ (tem, Qseqpacket))
+    socktype = SOCK_SEQPACKET;
+#endif
   else
     error ("Unsupported connection type");
 
@@ -7330,10 +7338,12 @@
 #ifdef DATAGRAM_SOCKETS
    ADD_SUBFEATURE (QCtype, Qdatagram);
 #endif
+#ifdef HAVE_SEQPACKET
+   ADD_SUBFEATURE (QCtype, Qseqpacket);
+#endif
 #ifdef HAVE_LOCAL_SOCKETS
    ADD_SUBFEATURE (QCfamily, Qlocal);
 #endif
-   ADD_SUBFEATURE (QCfamily, Qipv4);
 #ifdef AF_INET6
    ADD_SUBFEATURE (QCfamily, Qipv6);
 #endif
@@ -7403,6 +7413,8 @@
 #endif
   Qdatagram = intern_c_string ("datagram");
   staticpro (&Qdatagram);
+  Qseqpacket = intern_c_string ("seqpacket");
+  staticpro (&Qseqpacket);
 
   QCport = intern_c_string (":port");
   staticpro (&QCport);

Re: [feature request] SOCK_SEQPACKET

[Daniel Hackney]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> [...] I noticed that Emacs can't use the SOCK_SEQPACKET type.
>
>> It doesn't seem like SOCK_SEQPACKET is used much, but it would be
>> nice to have, if only for completeness sake. I don't really know any
>> C-level Emacs programming (I'm pretty comfortable with Elisp), but I
>> would be willing to help out however I can if given some guidance and
>> an overview of what would need to be done.
>
> You can start with the patch below (guaranteed 100% untested). I know
> nothing about SOCK_SEQPACKET, other than what I just read in the
> manpage of `socket', so this code probably doesn't do the right thing.

I'm in pretty much the same boat, as I had never even heard of
SOCK_SEQPACKET until I saw that Uzbl uses it. I'll hack on it a bit to
test it and make sure it works, but you'll probably want someone with
legitimate socket knowledge (i.e. not me ;) to review it before
committing anything I turn out.

> The main problem I'd except would have to do with the fact that Emacs
> will typically read and write in blocks of fixed size (like 1KB or 4KB
> maybe, IIRC), so for packets smaller than that size it might work
> fine, but for larger packets you may end up having to make more
> substantial changes.

I'll check it out. I'll definitely have to spend some more time wrapping
my head around the semantics of SOCK_SEQPACKET, but I'll try to turn out
something that works reliably. Thanks for the start, though!

--
Daniel M. Hackney

Re: [feature request] SOCK_SEQPACKET

[Daniel Hackney]

[-- Attachment #1: Type: text/plain, Size: 2749 bytes --]

Simon Leinen <simon.leinen@gmail.com> wrote:
>> I'm in pretty much the same boat, as I had never even heard of
>> SOCK_SEQPACKET until I saw that Uzbl uses it. I'll hack on it a bit
>> to test it and make sure it works, but you'll probably want someone
>> with legitimate socket knowledge (i.e. not me ;) to review it before
>> committing anything I turn out.
>
> I probably won't have the time or expertise to review this.

No worries; it seems like SOCK_SEQPACKET is pretty obscure, so not many
people have used it much.

> SOCK_SEQPACKET will select SCTP, which is an alternative transport
> protocol to TCP. It preserves message boundaries and has many other
> features over TCP. [...]

Yikes, I'd hate to have to deal with that! The one nice thing is that it
looks like for Unix sockets, SCTP isn't used, and I was able to
successfully send some data back and forth between Emacs and Python over
a local SOCK_SEQPACKET.

Another complication is that the Linux socket(2) manpage says:

  Some socket types may not be implemented by all protocol families; for
  example, SOCK_SEQPACKET is not implemented for AF_INET.

So we may be kind of out of luck on that front. I'll look into it a bit
more, but it looks like there isn't going to be much we can do about
that without hacking on the kernel or glibc (which I have no intention
of doing :).

> It would probably be a win if Emacs had SCTP support for those who
> need that, if that set of people has a non-empty intersection with the
> set of Emacs users.  The sockets API for SCTP seems to be
> unfinished/in flux - the Internet-Draft
> (http://tools.ietf.org/html/draft-ietf-tsvwg-sctpsocket) seems to have
> expired right now.

From what I hear, SCTP has some nice semantics, but requires new calls
and fancy stuff which is way beyond my ability.

> It is also 85 pages long.

Ahhhh!!!

> We probably wouldn't want to support all of SCTP's features.  The
> difficulty may be to find out what are the basic useful features that
> make people want to use SCTP rather than TCP...

Not really anything I can do to help there, as I'm pretty much a network
newbie.

At any rate, it looks like supporting SOCK_SEQPACKET over AF_UNIX
wouldn't be that bad (I've basically got it working), so it might be
worth cleaning that up for inclusion and waiting to see if anyone with
the proper knowledge wants to come along and add SCTP support to Emacs.

By the way, there were a few changes that needed to be made in order to
allow creating AF_UNIX SOCK_SEQPACKET servers, basically replacing
"socktype == SOCK_STREAM" with "socktype == SOCK_STREAM || socktype ==
SOCK_SEQPACKET". I've attached a patch (to be applied after Stefan
Monnier's previous patch) which takes care of this.

--
Daniel M. Hackney

[-- Attachment #2: commit-373bef0 --]
[-- Type: application/octet-stream, Size: 2005 bytes --]

commit 373bef0db55c5055a3a1086585c4cb17a73e56e0
Author: Dan Hackney <dan@haxney.org>
Date:   Thu Dec 3 02:29:20 2009 -0500

    Allow listening on sockets with SOCK_SEQPACKET.
    
    Does not take into account any of the semantic differences between
    SOCK_STREAM and SOCK_SEQPACKET, merely changes checks for "socktype ==
    SOCK_STREAM" to checks for SOCK_STREAM or SOCK_SEQPACKET.
    
    Signed-off-by: Dan Hackney <dan@haxney.org>

diff --git a/src/process.c b/src/process.c
index 2cce741..8186af7 100644
--- a/src/process.c
+++ b/src/process.c
@@ -3328,7 +3328,7 @@ usage: (make-network-process &rest ARGS)  */)
   QCaddress = is_server ? QClocal : QCremote;
 
   /* :nowait BOOL */
-  if (!is_server && socktype == SOCK_STREAM
+  if (!is_server && (socktype == SOCK_STREAM || socktype == SOCK_SEQPACKET)
       && (tem = Fplist_get (contact, QCnowait), !NILP (tem)))
     {
 #ifndef NON_BLOCKING_CONNECT
@@ -3423,7 +3423,7 @@ usage: (make-network-process &rest ARGS)  */)
      Some kernels have a bug which causes retrying connect to fail
      after a connect.  Polling can interfere with gethostbyname too.  */
 #ifdef POLL_FOR_INPUT
-  if (socktype == SOCK_STREAM)
+  if (socktype == SOCK_STREAM || socktype == SOCK_SEQPACKET)
     {
       record_unwind_protect (unwind_stop_other_atimers, Qnil);
       bind_polling_period (10);
@@ -3626,7 +3626,7 @@ usage: (make-network-process &rest ARGS)  */)
 	    }
 #endif
 
-	  if (socktype == SOCK_STREAM && listen (s, backlog))
+	  if ((socktype == SOCK_STREAM || socktype == SOCK_SEQPACKET) && listen (s, backlog))
 	    report_file_error ("Cannot listen on server socket", Qnil);
 
 	  break;
@@ -3789,7 +3789,7 @@ usage: (make-network-process &rest ARGS)  */)
   p->pid = 0;
   p->infd  = inch;
   p->outfd = outch;
-  if (is_server && socktype == SOCK_STREAM)
+  if (is_server && (socktype == SOCK_STREAM || socktype == SOCK_SEQPACKET))
     p->status = Qlisten;
 
   /* Make the process marker point into the process buffer (if any).  */

Re: [feature request] SOCK_SEQPACKET

[Stefan Monnier]

> At any rate, it looks like supporting SOCK_SEQPACKET over AF_UNIX
> wouldn't be that bad (I've basically got it working), so it might be
> worth cleaning that up for inclusion and waiting to see if anyone with
> the proper knowledge wants to come along and add SCTP support to Emacs.

Emacs hasn't been at the forefront of the networking technology, and
I don't see why that should change, so SEQPACKET with AF_UNIX is
perfectly sufficient.  If/when someone needs SEQPACKET over AF_INET, we
may then consider whether and how to add support for it.

> By the way, there were a few changes that needed to be made in order to
> allow creating AF_UNIX SOCK_SEQPACKET servers, basically replacing
> "socktype == SOCK_STREAM" with "socktype == SOCK_STREAM || socktype ==
> SOCK_SEQPACKET". I've attached a patch (to be applied after Stefan
> Monnier's previous patch) which takes care of this.

That's all?  With my patch plus yours, it works?
The mind boggles (I never wrote any networking code, so my patch was
really just an "educated guess" based on the existing code and the
manpage).


        Stefan

Re: [feature request] SOCK_SEQPACKET

[Daniel Hackney]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> At any rate, it looks like supporting SOCK_SEQPACKET over AF_UNIX
>> wouldn't be that bad (I've basically got it working), so it might be
>> worth cleaning that up for inclusion and waiting to see if anyone
>> with the proper knowledge wants to come along and add SCTP support to
>> Emacs.
>
> Emacs hasn't been at the forefront of the networking technology, and I
> don't see why that should change, so SEQPACKET with AF_UNIX is
> perfectly sufficient.

Well, from an "I want ponies" perspective, it would be great if Emacs
supported every network type :)

> If/when someone needs SEQPACKET over AF_INET, we may then consider
> whether and how to add support for it.

Plus, it looks like there are issues with underlying OS support, so it's
out of our hands for now.

>> By the way, there were a few changes that needed to be made in order
>> to allow creating AF_UNIX SOCK_SEQPACKET servers, basically replacing
>> "socktype == SOCK_STREAM" with "socktype == SOCK_STREAM || socktype
>> == SOCK_SEQPACKET". I've attached a patch (to be applied after Stefan
>> Monnier's previous patch) which takes care of this.
>
> That's all?  With my patch plus yours, it works?

Yup. I have cookies flowing brilliantly between Emacs and Uzbl, and
simple scripts connect just like they should.

> The mind boggles (I never wrote any networking code, so my patch was
> really just an "educated guess" based on the existing code and the
> manpage).

Same here. The nice thing is that it seems like SOCK_SEQPACKET can be
used basically the same as SOCK_STREAM, but with some additional funky
stuff (as well as using SCTP when done of AF_INET).

The only thing to consider now would be to try to take advantage of
SOCK_SEQPACKET's record delimiters, which are exposed through recvmsg(2)
and sendmsg(2) and the struct msghdr. This would allow for record
delimiting at the socket layer, potentially making certain kinds of
operations easier.

Without having looked at it, it seems like emacsclient could benefit
from record delimiters, as each command sent over the socket could be
delimited by a MSG_EOR, reducing the need for dealing with delimiting
records at the application layer.

I'll start to take a look at this, but it may be more effort than it's
worth, because of the limited AF support for SOCK_SEQPACKET. It looks
like, at a minimum, we would have to replace the calls to emacs_read/write()
with sendmsg()/recvmsg(). Also we would need to think about how to deal
with MSG_EOR in the context of reading into a buffer. There isn't any
change to the stream, but certain calls to recvmsg(2) would include a
MSG_EOR with them. Off the top of my head, it seems like we could call
the process sentinel when MSG_EOR is encountered.

Looking at the info page for sentinels, it looks like there could be a
chance that the sentinel might miss two MSG_EORs if they are encountered
in quick succession, though I could be wrong about this.

Anyway, I'm mainly thinking out loud here, let me know if I'm entirely
crazy :)

--
Daniel M. Hackney

Re: [feature request] SOCK_SEQPACKET

[Stefan Monnier]

>> That's all?  With my patch plus yours, it works?
> Yup. I have cookies flowing brilliantly between Emacs and Uzbl, and
> simple scripts connect just like they should.

Wonderful.  It's installed and will hence be included in Emacs-23.2.

> Without having looked at it, it seems like emacsclient could benefit
> from record delimiters, as each command sent over the socket could be
> delimited by a MSG_EOR, reducing the need for dealing with delimiting
> records at the application layer.

Not worth the trouble: we also want it to work over IP, so we're better
off using SOCK_STREAM for both local and remote cases.


        Stefan