nntp.el: netcat (was: Changes to emacs/lisp/gnus/nntp.el,v)

nntp.el: netcat (was: Changes to emacs/lisp/gnus/nntp.el,v)

[Reiner Steib]

On Wed, Mar 12 2008, Stefan Monnier on emacs-diffs:

| 2008-03-12  Stefan Monnier  <monnier@iro.umontreal.ca>
| 
| 	* nntp.el: Use with-current-buffer.
|         [...]
| 	(nntp-open-telnet-stream, nntp-open-via-rlogin-and-telnet): Recommend
| 	the use of the netcat alternatives.

Could you add this recommendation to the manual, too?
(info "(gnus)Direct Functions"), (info "(gnus)Indirect Functions")

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: nntp.el: netcat

[Stefan Monnier]

> | 2008-03-12  Stefan Monnier  <monnier@iro.umontreal.ca>
> | 
> | 	* nntp.el: Use with-current-buffer.
> |         [...]
> | 	(nntp-open-telnet-stream, nntp-open-via-rlogin-and-telnet): Recommend
> | 	the use of the netcat alternatives.

> Could you add this recommendation to the manual, too?
> (info "(gnus)Direct Functions"), (info "(gnus)Indirect Functions")

Done,


        Stefan

Re: nntp.el: netcat

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]

Hi,

>>>>> Stefan Monnier wrote:

>>| 2008-03-12  Stefan Monnier  <monnier@iro.umontreal.ca>
>>|
>>| 	* nntp.el: Use with-current-buffer.
>>|         [...]
>>| 	(nntp-open-telnet-stream, nntp-open-via-rlogin-and-telnet): Recommend
>>| 	the use of the netcat alternatives.

>> Could you add this recommendation to the manual, too?
>> (info "(gnus)Direct Functions"), (info "(gnus)Indirect Functions")

The function name `nntp-open-via-netcat' found in the doc string
of `nntp-open-telnet-stream' is not proper for it.  Because the
word `via' has been being used for the function names and the
variable names that are used for the nntp connection established
via the intemediate host.  Moreover, there is no function
definition for it in nntp.el.  I'd suggest the function name
`nntp-open-netcat-stream'.

In relation to this, some inconsistent descriptions are found in
the Gnus Info manual.

,---- (info "(gnus)Indirect Functions") ----
| `nntp-open-via-rlogin-and-telnet'
| [...]
|      `nntp-open-via-rlogin-and-telnet'-specific variables:
|
|     `nntp-telnet-command'
|           Command used to connect to the real NNTP server from the
|           intermediate host.  The default is `nc'.  You can also use
|           other programs like connect
|           (http://www.imasy.or.jp/~gotoh/ssh/connect.html) instead.
`----

Though the default value of `nntp-telnet-command' has not actually
been changed from "telnet", please note that changing the default
values of such defvoo'd variables will trouble users who haven't
set it in the select method.  Changing those names or removing
them will harm users who have customized those values, too.

The web site for `connect' is now:

http://www.meadowy.org/~gotoh/projects/connect

But it is an alternate of netcat, not telnet.

And, netcat doesn't seem to work with a socks wrapper.

,---- (info "(gnus)Example Methods") ----
|    If you're behind a firewall, but have direct access to the outside
| world through a wrapper command like "runsocks", you could open a
| socksified netcat connection to the news server as follows:
|
|      (nntp "outside"
|            (nntp-pre-command "runsocks")
|            (nntp-open-connection-function nntp-open-via-netcat)
|            (nntp-address "the.news.server"))
`----

Have you verified it?  Isn't specifying of some arguments passed
to netcat the proper way of letting netcat pass through the socks
server?  For example:

(nntp "outside"
      (nntp-open-connection-function nntp-open-netcat-stream)
      (nntp-netcat-switches ("-x" "the.socks.server:port" "-X" "5"))
      (nntp-address "the.news.server"))

Though I have chance of success in neither of the ways, because
the use of the 119 port is not permitted in the socks server.

In addition, the arguments ("-e" "none") passed to ssh seems to
be necessary:

,---- SSH(1) ----
| -e escape_char
|         Sets the escape character for sessions with a pty (default: ‘~’).
|         The escape character is only recognized at the beginning of a
|         line.  The escape character followed by a dot (‘.’) closes the
|         connection; followed by control-Z suspends the connection; and
|         followed by itself sends the escape character once.  Setting the
|         character to “none” disables any escapes and makes the session
|         fully transparent.
`----

Here is a patch for the Emacs trunk:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 7984 bytes --]

--- lisp/gnus/nntp.el~	2008-03-13 21:51:08 +0000
+++ lisp/gnus/nntp.el	2008-03-27 07:47:42 +0000
@@ -118,6 +118,13 @@
 (defvoo nntp-telnet-switches '("-8")
   "*Switches given to the telnet command `nntp-telnet-command'.")
 
+(defvoo nntp-netcat-command "nc"
+  "*Netcat command used to connect to the nntp server.
+This command is used by the `nntp-open-netcat-stream' method.")
+
+(defvoo nntp-netcat-switches nil
+  "*Switches given to the netcat command `nntp-netcat-command'.")
+
 (defvoo nntp-end-of-line "\r\n"
   "*String to use on the end of lines when talking to the NNTP server.
 This is \"\\r\\n\" by default, but should be \"\\n\" when using an indirect
@@ -1873,7 +1880,7 @@
 
 (defun nntp-open-telnet-stream (buffer)
   "Open a nntp connection by telnet'ing the news server.
-`nntp-open-via-netcat' is recommended in place of this function
+`nntp-open-netcat-stream' is recommended in place of this function
 because it is more reliable.
 
 Please refer to the following variables to customize the connection:
@@ -1896,6 +1903,26 @@
       (delete-region (point-min) (point))
       proc)))
 
+(defun nntp-open-netcat-stream (buffer)
+  "Open a nntp connection to the news server using the netcat command.
+
+Please refer to the following variables to customize the connection:
+- `nntp-pre-command',
+- `nntp-netcat-command',
+- `nntp-netcat-switches',
+- `nntp-address',
+- `nntp-port-number',
+- `nntp-end-of-line'."
+  (let ((command `(,@(when nntp-pre-command
+		       (list nntp-pre-command))
+		   ,nntp-netcat-command
+		   ,@nntp-via-netcat-switches
+		   ,nntp-address
+		   ,(if (numberp nntp-port-number)
+			(number-to-string nntp-port-number)
+		      nntp-port-number))))
+    (apply 'start-process "nntpd" buffer command)))
+
 (defun nntp-open-via-rlogin-and-telnet (buffer)
   "Open a connection to an nntp server through an intermediate host.
 First rlogin to the remote host, and then telnet the real news server
--- doc/misc/gnus.texi~	2008-03-16 21:49:43 +0000
+++ doc/misc/gnus.texi	2008-03-27 07:47:42 +0000
@@ -13057,26 +13057,26 @@
              (nntp-via-user-name "intermediate_user_name")
              (nntp-via-address "intermediate.host.example")
              (nntp-via-rlogin-command "ssh")
-             (nntp-via-rlogin-command-switches ("-C"))
+             (nntp-via-rlogin-command-switches ("-C" "-e" "none"))
              (nntp-open-connection-function nntp-open-via-rlogin-and-netcat)))
 @end lisp
 
+This means that you have to have set up @code{ssh-agent} correctly to
+provide automatic authorization, of course.  And to get a compressed
+connection, you have to have the @samp{Compression} option in the
+@code{ssh} @file{config} file.
+
 If you're behind a firewall, but have direct access to the outside world
-through a wrapper command like "runsocks", you could open a socksified
-netcat connection to the news server as follows:
+through the socks server, you could open a netcat connection to the news
+server as follows:
 
 @lisp
 (nntp "outside"
-      (nntp-pre-command "runsocks")
-      (nntp-open-connection-function nntp-open-via-netcat)
+      (nntp-open-connection-function nntp-open-netcat-stream)
+      (nntp-netcat-switches ("-x" "the.socks.server:port"))
       (nntp-address "the.news.server"))
 @end lisp
 
-This means that you have to have set up @code{ssh-agent} correctly to
-provide automatic authorization, of course.  And to get a compressed
-connection, you have to have the @samp{Compression} option in the
-@code{ssh} @file{config} file.
-
 
 @node Creating a Virtual Server
 @subsection Creating a Virtual Server
@@ -13543,28 +13543,42 @@
       (nntp-address "snews.bar.com"))
 @end lisp
 
-@findex nntp-open-via-netcat
-@item nntp-open-via-netcat
+@findex nntp-open-netcat-stream
+@item nntp-open-netcat-stream
 Opens a connection to an @acronym{NNTP} server using the @code{netcat}
 program.  You might wonder why this function exists, since we have
 the default @code{nntp-open-network-stream} which would do the job.  (One
 of) the reason(s) is that if you are behind a firewall but have direct
-connections to the outside world thanks to a command wrapper like
-@code{runsocks}, you can use it like this:
+connections to the outside world through the socks server, you could
+open a netcat connection to the news server as follows:
 
 @lisp
-(nntp "socksified"
-      (nntp-pre-command "runsocks")
-      (nntp-open-connection-function nntp-open-via-netcat)
+(nntp "outside"
+      (nntp-open-connection-function nntp-open-netcat-stream)
+      (nntp-netcat-switches ("-x" "the.socks.server:port"))
       (nntp-address "the.news.server"))
 @end lisp
 
-With the default method, you would need to wrap your whole Emacs
-session, which is not a good idea.
+@code{nntp-open-netcat-stream}-specific variables:
+
+@table @code
+@item nntp-netcat-command
+@vindex nntp-netcat-command
+Command to use when connecting to the @acronym{NNTP} server through
+@samp{netcat}.  This is @emph{not} for an intermediate host.  This is
+just for the real @acronym{NNTP} server.  The default is @samp{nc}.  You
+can also use other programs like
+@uref{http://www.meadowy.org/~gotoh/projects/connect, connect} instead.
+
+@item nntp-netcat-switches
+@vindex nntp-netcat-switches
+A list of switches to pass to @code{nntp-netcat-command}.  The default
+is @samp{()}.
+@end table
 
 @findex nntp-open-telnet-stream
 @item nntp-open-telnet-stream
-Like @code{nntp-open-via-netcat}, but uses @code{telnet} rather than
+Like @code{nntp-open-netcat-stream}, but uses @code{telnet} rather than
 @code{netcat}.  @code{telnet} is a bit less robust because of things
 like line-end-conversion, but sometimes netcat is simply
 not available.  The previous example would turn into:
@@ -13576,6 +13590,9 @@
       (nntp-address "the.news.server")
       (nntp-end-of-line "\n"))
 @end lisp
+
+With the default method, you would need to wrap your whole Emacs
+session, which is not a good idea.
 @end table
 
 
@@ -13610,7 +13627,20 @@
 List of strings to be used as the switches to
 @code{nntp-via-rlogin-command}.  The default is @code{nil}.  If you use
 @samp{ssh} for @code{nntp-via-rlogin-command}, you may set this to
-@samp{("-C")} in order to compress all data connections.
+@samp{("-C" "-e" "none")} in order to compress all data connections and
+to disable the escape character.
+
+@item nntp-via-netcat-command
+@vindex nntp-via-netcat-command
+Command that the intermediate host uses when connecting to the
+@acronym{NNTP} server through @samp{netcat}.  The default is @samp{nc}.
+You can also use other programs like
+@uref{http://www.meadowy.org/~gotoh/projects/connect, connect} instead.
+
+@item nntp-via-netcat-switches
+@vindex nntp-via-netcat-switches
+A list of switches to pass to @code{nntp-via-netcat-command}.  The
+default is @samp{()}.
 @end table
 
 @item nntp-open-via-rlogin-and-telnet
@@ -13626,9 +13656,7 @@
 @item nntp-telnet-command
 @vindex nntp-telnet-command
 Command used to connect to the real @acronym{NNTP} server from the
-intermediate host.  The default is @samp{nc}.  You can also use other
-programs like @uref{http://www.imasy.or.jp/~gotoh/ssh/connect.html,
-connect} instead.
+intermediate host.  The default is @samp{telnet}.
 
 @item nntp-telnet-switches
 @vindex nntp-telnet-switches
@@ -13744,19 +13772,6 @@
 String to use as end-of-line marker when talking to the @acronym{NNTP}
 server.  This is @samp{\r\n} by default, but should be @samp{\n} when
 using a non native telnet connection function.
-
-@item nntp-via-netcat-command
-@vindex nntp-via-netcat-command
-Command to use when connecting to the @acronym{NNTP} server through
-@samp{netcat}.  This is @emph{not} for an intermediate host.  This is
-just for the real @acronym{NNTP} server.  The default is
-@samp{nc}.
-
-@item nntp-via-netcat-switches
-@vindex nntp-via-netcat-switches
-A list of switches to pass to @code{nntp-via-netcat-command}.  The default
-is @samp{()}.
-
 @end table
 
 @node NNTP marks

Re: nntp.el: netcat

[Stefan Monnier]

> The function name `nntp-open-via-netcat' found in the doc string
> of `nntp-open-telnet-stream' is not proper for it.  Because the
> word `via' has been being used for the function names and the
> variable names that are used for the nntp connection established
> via the intemediate host.  Moreover, there is no function
> definition for it in nntp.el.  I'd suggest the function name
> `nntp-open-netcat-stream'.

I'll trust your judgment on that.  All I wanted to do was to make sure
we propose to use netcat before telnet since I've had several problems
with telnet.

> ,---- (info "(gnus)Example Methods") ----
> |    If you're behind a firewall, but have direct access to the outside
> | world through a wrapper command like "runsocks", you could open a
> | socksified netcat connection to the news server as follows:
> |
> |      (nntp "outside"
> |            (nntp-pre-command "runsocks")
> |            (nntp-open-connection-function nntp-open-via-netcat)
> |            (nntp-address "the.news.server"))
> `----

> Have you verified it?

No, indeed.

> In addition, the arguments ("-e" "none") passed to ssh seems to
> be necessary:

> ,---- SSH(1) ----
> | -e escape_char
> |         Sets the escape character for sessions with a pty (default: ‘~’).
> |         The escape character is only recognized at the beginning of a
> |         line.  The escape character followed by a dot (‘.’) closes the
> |         connection; followed by control-Z suspends the connection; and
> |         followed by itself sends the escape character once.  Setting the
> |         character to “none” disables any escapes and makes the session
> |         fully transparent.
> `----

No, because the ssh connection will not use a pty (that was necessary
(and problematic) for telnet, but luckily it's not needed for netcat).


        Stefan

Re: nntp.el: netcat

[Katsumi Yamaoka]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> The function name `nntp-open-via-netcat' found in the doc string
>> of `nntp-open-telnet-stream' is not proper for it.  Because...

> I'll trust your judgment on that.  All I wanted to do was to make sure
> we propose to use netcat before telnet since I've had several problems
> with telnet.

I see.  And thanks for improving nntp.el and gnus.texi.

>>|      (nntp "outside"
>>|            (nntp-pre-command "runsocks")
>>|            (nntp-open-connection-function nntp-open-via-netcat)
>>|            (nntp-address "the.news.server"))

>> Have you verified it?

> No, indeed.

Google shows the examples for "runsocks nc", so it will probably
be valid.  Though I'm not sure that it is applicable to various
versions of netcat (the typical ones are the BSD version and the
GNU version).

>> In addition, the arguments ("-e" "none") passed to ssh seems to
>> be necessary:

>> ,---- SSH(1) ----
>>| -e escape_char
>>|         Sets the escape character for sessions with a pty (default: ‘~’).
                                                          ===
I noticed I've overlooked it just after sending the last message.

> No, because the ssh connection will not use a pty (that was necessary
> (and problematic) for telnet, but luckily it's not needed for netcat).

Thanks for clarifying it.

Regards,

Re: nntp.el: netcat

[Stefan Monnier]

>> No, indeed.

> Google shows the examples for "runsocks nc", so it will probably
> be valid.  Though I'm not sure that it is applicable to various
> versions of netcat (the typical ones are the BSD version and the
> GNU version).

I thought the whole point of runsocks is that it tricks *any* subsequent
program into being socksified (probably using some kind of LD_PRELOAD hack).


        Stefan