UDP/DNS in Emacs

UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

I'm pondering adding various spam filtering stuff to Gnus.  Many of
the black-lists use DNS to say whether something comes from an open
relay, and I can't seem to find a real interface to DNS in Emacs,
which makes me believe I'm just not looking in the right place.

There are functions that call nslookup and dig externally, which can
be used.  However, calling an external program (typically) 20 times
per message when splitting would be rather inefficient.

Is there a native DNS interface in Emacs?  Or even a UDP interface?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen



_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Newsgroups: gmane.emacs.devel
> Date: Sat, 30 Mar 2002 18:21:40 +0100
> 
> I'm pondering adding various spam filtering stuff to Gnus.  Many of
> the black-lists use DNS to say whether something comes from an open
> relay, and I can't seem to find a real interface to DNS in Emacs,
> which makes me believe I'm just not looking in the right place.

I realize that this doesn't help you a bit, but: please, _please_
consider leaving Gnus free of the ORBS nuisance.  ORBS (and all its
work-alikes) simply SUCK!  There should be no reason for Gnus to use
any of their blacklists, as Gnus has enough power in its filters to do
the same without blindly trusting someone's blacklist.

TIA

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

"Eli Zaretskii" <eliz@is.elta.co.il> writes:

> I realize that this doesn't help you a bit, but: please, _please_
> consider leaving Gnus free of the ORBS nuisance.  ORBS (and all its
> work-alikes) simply SUCK!  There should be no reason for Gnus to use
> any of their blacklists, as Gnus has enough power in its filters to do
> the same without blindly trusting someone's blacklist.

I think that should be up to the user.

ORDB (and friends) block based on open relays.  Others block based on
reported spammers (SpamCop), and things like Vipul's Razor block based
on a sha1 digest of the message body, and some do filtering based on
certain tell-tales.

Some of these require DNS lookups.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen



_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Some of these require DNS lookups.

After a tip from Kai and some grepping around, I've found
`make-network-process' ("Datagram", not "UDP") in CVS Emacs.

I haven't yet started playing with it, but am I right in that nobody
has written a DNS layer for this yet?  If so, my plans are for writing
a dns.el to use `make-network-process' to implement a DNS lookup
thing.  (With several concurrent requests, time outs, etc.)

Please let me know if somebody has already done this.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen



_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I haven't yet started playing with it, but am I right in that nobody
> has written a DNS layer for this yet?  If so, my plans are for writing
> a dns.el to use `make-network-process' to implement a DNS lookup
> thing.  (With several concurrent requests, time outs, etc.)

I wrote a simple DNS client for TCP some time ago, but I'm sure that
seeing the code won't help you. :-)

Oh, and DNS can use both UDP and TCP, so I guess a dns.el should use
both, with the appropriate fallback UDP->TCP.

There was a library posted to gnu.emacs.sources recently for dealing
with binary structures, it might be useful for unpacking DNS packets.


_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> I wrote a simple DNS client for TCP some time ago, but I'm sure that
> seeing the code won't help you. :-)

:-)

> Oh, and DNS can use both UDP and TCP, so I guess a dns.el should use
> both, with the appropriate fallback UDP->TCP.

I'm concentrating on UDP for the time being...

> There was a library posted to gnu.emacs.sources recently for dealing
> with binary structures, it might be useful for unpacking DNS packets.

I've basically gotten dns.el to work, only I'm having the usual
problems with having bytes in the buffer remain bytes and not turning
into characters.  :-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen



_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've basically gotten dns.el to work, only I'm having the usual
> problems with having bytes in the buffer remain bytes and not turning
> into characters.  :-/

And the solution was also the usual.

Well, I've now got the DNS thing implemented, so I can go idea
implementing the actual spam handling stuff.

(query-dns "www.gnus.org")
=>
((id 4)
 (response-p t)
 (opcode query)
 (authoritative-p t)
 (truncated-p t)
 (recursion-desired-p t)
 (response-code no-error)
 (queries
  (("www.gnus.org"
    (type A)
    (class IN))))
 (answers
  ((".www.gnus.org"
    (type A)
    (class IN)
    (ttl 86400)
    (data "209.196.61.71"))))
 (authorities
  ((".gnus.org"
    (type NS)
    (class IN)
    (ttl 86400)
    (data "ns1.psychetect.com"))
   (".gnus.org"
    (type NS)
    (class IN)
    (ttl 86400)
    (data "ns1.netfonds.no"))
   (".gnus.org"
    (type NS)
    (class IN)
    (ttl 259200)
    (data "dns.sclp.com"))))
 (additionals
  ((".dns.sclp.com"
    (type A)
    (class IN)
    (ttl 86400)
    (data "209.196.61.66")))))

Well, ok, that's probably to verbose output from this function.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen



_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Eli Zaretskii]

On Sat, 30 Mar 2002, Lars Magne Ingebrigtsen wrote:

> I've basically gotten dns.el to work, only I'm having the usual
> problems with having bytes in the buffer remain bytes and not turning
> into characters.  :-/

Decode the data with raw-text-unix, and you should be fine.

If that doesn't help, please tell the details.

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://mail.gnu.org/mailman/listinfo/emacs-devel

Re: UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@is.elta.co.il> writes:

> Decode the data with raw-text-unix, and you should be fine.
>
> If that doesn't help, please tell the details.

I had just forgotten how to do all that stuff.  After getting a
unibyte buffer for dealing with binary data, it all just works.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: UDP/DNS in Emacs

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Newsgroups: gmane.emacs.devel
> Date: Sun, 31 Mar 2002 16:55:35 +0200
> 
> Eli Zaretskii <eliz@is.elta.co.il> writes:
> 
> > Decode the data with raw-text-unix, and you should be fine.
> >
> > If that doesn't help, please tell the details.
> 
> I had just forgotten how to do all that stuff.  After getting a
> unibyte buffer for dealing with binary data, it all just works.

IMHO, using unibyte buffers is asking for trouble.  It's the main
reason for those \201 beauties popping up in user buffers.  Emacs 21
doesn't need that anymore (although they are still supported).

Re: UDP/DNS in Emacs

[Stefan Monnier]

> > From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> > Newsgroups: gmane.emacs.devel
> > Date: Sun, 31 Mar 2002 16:55:35 +0200
> > 
> > Eli Zaretskii <eliz@is.elta.co.il> writes:
> > 
> > > Decode the data with raw-text-unix, and you should be fine.
> > >
> > > If that doesn't help, please tell the details.
> > 
> > I had just forgotten how to do all that stuff.  After getting a
> > unibyte buffer for dealing with binary data, it all just works.
> 
> IMHO, using unibyte buffers is asking for trouble.  It's the main
> reason for those \201 beauties popping up in user buffers.  Emacs 21
> doesn't need that anymore (although they are still supported).

And I disagree (and I think RMS partly disagrees as well).
When dealing with bytes rather than chars, using unibyte buffers
is the right thing to do.
It's the use of bytes in multibyte buffers that creates the problems
where \201 can occur.


	Stefan

Re: UDP/DNS in Emacs

[Eli Zaretskii]

On Sun, 31 Mar 2002, Stefan Monnier wrote:

> It's the use of bytes in multibyte buffers that creates the problems
> where \201 can occur.

Sure, but in many situations Emacs converts to multibyte behind your 
back.

Re: UDP/DNS in Emacs

[Richard Stallman]

Unibyte buffers are a useful and legitimate feature.

Re: UDP/DNS in Emacs

[Richard Stallman]

    I haven't yet started playing with it, but am I right in that nobody
    has written a DNS layer for this yet?  If so, my plans are for writing
    a dns.el to use `make-network-process' to implement a DNS lookup
    thing.  (With several concurrent requests, time outs, etc.)

This is a useful facility.  Its usefulness is broader than GNUS, so
let's put it in the /net subdirectory and not treat it as part of
GNUS.

Re: UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> This is a useful facility.  Its usefulness is broader than GNUS, so
> let's put it in the /net subdirectory and not treat it as part of
> GNUS.

Sure.  Once it has been given a thorough testing, it should be moved
out of Gnus.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: UDP/DNS in Emacs

[Richard Stallman]

I agree with Eli that it is a bad idea to delete spam under the
control of ORBS.  There are better ways to handle spam.  gnu.org has
been on some of these lists occasionally.

Re: UDP/DNS in Emacs

[Lars Magne Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> I agree with Eli that it is a bad idea to delete spam under the
> control of ORBS.

Well, ORBS is dead now (and so is ORBZ), but there are similar
services going still.

But this isn't really about deleting spam in a Gnus context.  Gnus
splits incoming mail into different groups based on various rules
given by the user.  What I'm building now is a convenient way for
users to split mail into a "spam" group using various methods.  The
user will determine themselves which bits to use.  There's a
white-list (of addresses explicitly allowed through), a black-list
(the inverse), these RBL things, Vipul's Razor (sha1 digests on
message bodies), and so on.

The user will construct predicates that say how to use these things. 

(or blacklist-p
    (and (not whitelist-p)
         (or (> rbl 4)
             vipul-p)))

To say that something is to be split into the "spam" group if it's
on the blacklist, and it's not on the whitelist, and more than 4
rbl's have deemed the relays to be open, or vipul has marked the
message contents to be spam.

And so on.
             
-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: UDP/DNS in Emacs

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Newsgroups: gmane.emacs.devel
> Date: Sun, 31 Mar 2002 18:54:48 +0200
> 
> But this isn't really about deleting spam in a Gnus context.  Gnus
> splits incoming mail into different groups based on various rules
> given by the user.  What I'm building now is a convenient way for
> users to split mail into a "spam" group using various methods.

My point is that basing this on the sender's ISP is a bad idea, one
that unduly discriminates people who don't have much choice but to use
whatever ISPs are available to them.  What does my ISP have got to do
with the content of my messages?  Why should my mail get labeled as
spam just because my ISP's server happened to be caught red-handed by
ORBS and its ilk at some point?

Re: UDP/DNS in Emacs

[Per Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

> My point is that basing this on the sender's ISP is a bad idea, one
> that unduly discriminates people who don't have much choice but to use
> whatever ISPs are available to them.  

It is their problem.  Why should I suffer for it?

My problem is that mail from certain ISP's are much more likely to be
spam than mail from other ISP's.  In fact, 95% of the mail I get which
is not caught by one of my "whitelist" filters (i.e. mail from people
I know, mail to "secret" adresses, or mail to mailingslists) are spam.

The remaining mail go to either a mail.junk folder (if they matches my
rather poor blacklist rules) I skim the summary monthly for potential
non-spam, and maybe find one every third month.  The rest go to a
misc.misc folder I skim the summary daily for the occational non-spam
(maybe there are one every other day), and junk the rest.  Obviously a
few non-spam messages gets junked as well.

If Gnus can improve my blacklist rules, I will have to spend less time
skimming the misc.misc folder, and junk less messages by accident.

I do not see why "fairness" to people who believe they have no other
choise than using an incompetent ISP should force me to lose relevant
mail.

> What does my ISP have got to do with the content of my messages?
> Why should my mail get labeled as spam just because my ISP's server
> happened to be caught red-handed by ORBS and its ilk at some point?

ORBS is dead, and was in any case a database of current open relays.
To get out of the database, all you had to close it.  Open relayes was
(and probably still is) the most efficient way for spammers to
propagate their messages, and blocking based on open relays was (and
probably still is) the most efficient way to block spam.

Re: UDP/DNS in Emacs

[Eli Zaretskii]

> From: Per Abrahamsen <abraham@dina.kvl.dk>
> Date: Tue, 02 Apr 2002 14:34:51 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > My point is that basing this on the sender's ISP is a bad idea, one
> > that unduly discriminates people who don't have much choice but to use
> > whatever ISPs are available to them.  
> 
> It is their problem.  Why should I suffer for it?

Because it will become your problem when relevant mail is
automatically junked based on the domain from which it comes.

> If Gnus can improve my blacklist rules, I will have to spend less time
> skimming the misc.misc folder, and junk less messages by accident.

If I understand your habits, it also means that some non-junk mail
will be read once a month instead of once a day.

It strikes me that a second or two it takes to recognize spam and
press that DELETE button is a much more efficient means (both
time-wise and as far as danger of junking real email is considered) of
dealing with mail than your convoluted procedure.  IMHO, of course.

> I do not see why "fairness" to people who believe they have no other
> choise than using an incompetent ISP should force me to lose relevant
> mail.

I think your assumption about availablility of good ISPs needs some
reality check.  How well are you familiar with the situation outside
Western Europe and North America?

For that matter, it's possible that your mail handling needs to be
revised, and that some different procedure will run less risk of
junking relevant mail even without draconian measures.

> ORBS is dead, and was in any case a database of current open relays.
> To get out of the database, all you had to close it.

I know all about this; do you?  How many times, if at all, did you
need or try to deal with these problems?

The number of times I removed my ISP from the data base is greater
than the number of words in this message.  It doesn't help: a few days
after that, it's in the data base again.

> Open relayes was
> (and probably still is) the most efficient way for spammers to
> propagate their messages, and blocking based on open relays was (and
> probably still is) the most efficient way to block spam.

As usual, this is a question of striking the fine balance of stopping
the guilty without unduly punishing the innocent.  It might be worth
remembering that spam is defined based on its content, not on the
server from which it comes.

Re: UDP/DNS in Emacs

[Per Abrahamsen]

Eli Zaretskii <eliz@fencepost.gnu.org> writes:

>> From: Per Abrahamsen <abraham@dina.kvl.dk>
>> Date: Tue, 02 Apr 2002 14:34:51 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > My point is that basing this on the sender's ISP is a bad idea, one
>> > that unduly discriminates people who don't have much choice but to use
>> > whatever ISPs are available to them.  
>> 
>> It is their problem.  Why should I suffer for it?
>
> Because it will become your problem when relevant mail is
> automatically junked based on the domain from which it comes.

Not junked, but moved to a lower priority folder.  

And yes, _that_ is my problem.  So that should be my choice how to
deal with it.  

>> If Gnus can improve my blacklist rules, I will have to spend less time
>> skimming the misc.misc folder, and junk less messages by accident.
>
> If I understand your habits, it also means that some non-junk mail
> will be read once a month instead of once a day.

Yes.  It is all a matter of how to optimize on the two variables:  

1) How much time do I have to spend manually sorting spam.
2) How many non-spam messages may accidentially be junked (or delayed).

I am the one to judge how to prioritize between these.  I find it
insulting that you want to deny me that ability, especially with the
justification that you can prioritize my time better than I can.

> I think your assumption about availablility of good ISPs needs some
> reality check.  How well are you familiar with the situation outside
> Western Europe and North America?

I assume it it is similar to question in Denmark in the past.  If we
were unhappy with our sole national ISP, our alternative was to use an
UUCP connection to a foreign ISP for mail.  

In any case: Automatically junking everything from outside Western
Europe and Northern America is the most popular alternative to the
blaklists.  It is extremely tempting for me to blacklist SE Asia,
where the fraction of relevant mail (outside my whitelists) to spam is
near zero.  For most users, that is true.

>> To get out of the database, all you had to close it.
>
> I know all about this; do you?  How many times, if at all, did you
> need or try to deal with these problems?

KVL has often been blacklisted, when someone inside have installed a
new "NT Server" with too many defauls on.

> The number of times I removed my ISP from the data base is greater
> than the number of words in this message.  It doesn't help: a few days
> after that, it's in the data base again.

So you had an open relay again.  Incompetent ISP.  I also know the
kind of things competent ISP's have to do to keep out of the open
releay blacklists.  Basically, they have to test whether their
customer is running an open relay before relaying their customers
mail.

As a side-effect, supporting open relay blacklists helps motivate
ISP's to act competently.

>> Open relayes was (and probably still is) the most efficient way for
>> spammers to propagate their messages, and blocking based on open
>> relays was (and probably still is) the most efficient way to block
>> spam.
>
> As usual, this is a question of striking the fine balance of
> stopping the guilty without unduly punishing the innocent.

Nope.  It is not a question of punishment.  It is a question of
finding the balance where e-mail regain most of its usefulness.

> It might be worth remembering that spam is defined based on its
> content, not on the server from which it comes.

Nope, that would be totally irrelevant to the question.

Re: UDP/DNS in Emacs

[Alan Shutko]

Per Abrahamsen <abraham@dina.kvl.dk> writes:

> So you had an open relay again.  Incompetent ISP.  I also know the
> kind of things competent ISP's have to do to keep out of the open
> releay blacklists.  Basically, they have to test whether their
> customer is running an open relay before relaying their customers
> mail.

Or they just block incoming port 25 to all their customers, and tell
them they should have a colocation or buy mail services somewhere else
if they want to receive mail.

-- 
Alan Shutko <ats@acm.org> - In a variety of flavors!
There will be big changes for you but you will be happy.

Re: UDP/DNS in Emacs

[Eli Zaretskii]

> From: Per Abrahamsen <abraham@dina.kvl.dk>
> Date: Thu, 04 Apr 2002 17:45:03 +0200
> 
> I am the one to judge how to prioritize between these.  I find it
> insulting that you want to deny me that ability, especially with the
> justification that you can prioritize my time better than I can.

I think you read too much into what I said, and overreact as a result.

> > I think your assumption about availablility of good ISPs needs some
> > reality check.  How well are you familiar with the situation outside
> > Western Europe and North America?
> 
> I assume it it is similar to question in Denmark in the past.  If we
> were unhappy with our sole national ISP, our alternative was to use an
> UUCP connection to a foreign ISP for mail.  

This is impractical in many places.  I'm sure you can imagine the
problems, so I won't spell them out.

> > As usual, this is a question of striking the fine balance of
> > stopping the guilty without unduly punishing the innocent.
> 
> Nope.  It is not a question of punishment.  It is a question of
> finding the balance where e-mail regain most of its usefulness.

All I can say is that to me, this blacklisting sounds akin to racism:
people are discriminated based on something they didn't do, and
something upon which they don't have any real control.

Re: UDP/DNS in Emacs

[Paul Eggert]

> From: "Eli Zaretskii" <eliz@is.elta.co.il>
> Date: Fri, 05 Apr 2002 13:02:26 +0300
> 
> All I can say is that to me, this blacklisting sounds akin to racism:

Umm, let's not go overboard here.

I don't read most of the physical mail that comes in over my doorstep.
Most of the time I don't even read the return addresses on the
envelopes.  This policy does cause some problems for me, but all
things considered it's better for me than wasting time actually
reading the stuff.

I think it's a bit of a stretch to call my policy "akin to racism".
On the contrary, I wish I could use a semiautomated blacklist
procedure on my physical mail, so that I could read it more
efficiently, and miss fewer important pieces of mail.  Like most
people, I'm not wealthy enough to afford a personal secretary.

I don't think everybody should use my policy, but I think it's
appropriate for many non-wealthy people who get too much mail and have
too many other things to do.

(Now, if I could only do my _taxes_ the same way.  :-)

Re: UDP/DNS in Emacs

[Per Abrahamsen]

"Eli Zaretskii" <eliz@is.elta.co.il> writes:

> This is impractical in many places.  

Inconvenient and expensive, certainly.  Nonetheless, it was the
reality I grow up with.

> All I can say is that to me, this blacklisting sounds akin to racism:

I don't think I'm the one overreacting.  People who use incompetent
ISP's will have inferior connectivity.  The guilty part are the ISP's
and the spammers, not the people who try to keep spam down at a level
where they don't have to drop email entirely as a communication media.
You are blaming victims.