Coverity Open Source Defect Scan of Emacs

Coverity Open Source Defect Scan of Emacs

[Ben Chelf]

Hello Emacs Developers,

   As some of you may have heard, last month Coverity set up 
http://scan.coverity.com as a site dedicated to scanning open source 
projects for defects. In just 1 month, over 4500 defects have been 
examined by various open source developers, and from what we can tell, 
it seems that there have been over 2500 patches to the scanned code 
bases! Due to popular request, I’m happy to announce that we’ve added 
Emacs to the list of projects scanned on the site. For those of you not 
familiar with "scan" yet and by way of introduction ...

   I'm the CTO of Coverity, Inc., a company that has technology that 
performs static source code analysis to look for defects in code. You 
may have heard of us or of our technology from its days at Stanford (the 
"Stanford Checker"). The reason I'm writing is because we have set up a 
framework internally to continually scan open source projects and 
provide the results of our analysis back to the developers of those 
projects. To see the results of the project, check out:

http://scan.coverity.com

   My belief is that we (Coverity) must reach out to the developers of 
these packages (you) in order to make progress in actually fixing the 
defects that we happen to find, so this is my first step in that 
mission. Of course, I think Coverity technology is great, but I want to 
hear what you think and that's why I worked with folks at Coverity to 
put this infrastructure in place. The process is simple -- it checks out 
your code each night from your repository and scans it so you can always 
see the latest results.

   Right now, we're guarding access to the actual defects that we report 
for a couple of reasons: (1) We think that you, as developers of Emacs, 
should have the chance to look at the defects we find to patch them 
before random other folks get to see what we found and (2) From a 
support perspective, we want to make sure that we have the appropriate 
time to engage with those who want to use the results to fix the code. 
Because of this second point, I'd ask that if you are interested in 
really digging into the results a bit further for your project, please 
have a couple of core maintainers and/or developers reach out to us to 
request access. As this is a new process for us and still involves a 
small number of packages, I want to make sure that I personally can be 
involved with the activity that is generated from this effort.

   So I'm basically asking for people who want to play around with some 
cool new technology to help make source code better. If this interests 
you, please feel free to register on our site or email me directly. And 
of course, if there are other packages you care about that aren't 
currently on the list, I want to know about those too.

   If this is the wrong list, my sincerest apologies and please let me 
know where would be a more appropriate forum for this type of message.

Many thanks for reading this far...

-ben

  Ben Chelf
  Chief Technology Officer
  Coverity, Inc.

Re: Coverity Open Source Defect Scan of Emacs

[Alan Mackenzie]

Good morning, Ben!

Just to make things quite clear from the start, I'm a contributor to
Emacs, but have no other responsibilities on the project.  So what I'm
writing here is a purely personal point of view; I don't speak for the
project.

On Wed, 5 Apr 2006, Ben Chelf wrote:

>Hello Emacs Developers,

>   As some of you may have heard, last month Coverity set up 
>http://scan.coverity.com as a site dedicated to scanning open source 
>projects for defects.

May I suggest you write here "_free_ and open source projects"?  These
two terms describe two distinct philosophical stances, although they have
much in common.  Amongst the people to whom the differences matter is
Richard Stallman, the maintainer of Emacs.  ;-)

On a similar note, using "charset=windows-1252" rather than UTF (or even
plain old ASCII) might drop you the odd percentage point in approval
ratings.

>In just 1 month, over 4500 defects have been examined by various open
>source developers, and from what we can tell, it seems that there have
>been over 2500 patches to the scanned code bases! Due to popular
>request, I~Rm happy to announce that we~Rve added Emacs to the list of
>projects scanned on the site. For those of you not familiar with "scan"
>yet and by way of introduction ...

As a matter of interest, _who_ has been asking?  Are they, in general,
individual programmers, or project managers who perhaps have reservations
about Emacs, or possibly other people.

>   I'm the CTO of Coverity, Inc., a company that has technology that 
>performs static source code analysis to look for defects in code. You 
>may have heard of us or of our technology from its days at Stanford (the 
>"Stanford Checker"). The reason I'm writing is because we have set up a 
>framework internally to continually scan open source projects and 
>provide the results of our analysis back to the developers of those 
>projects. To see the results of the project, check out:

>http://scan.coverity.com

I've just had a look at it.  The page seems aimed more at managers than
hackers.  Have you got a page which goes into more technical detail?  For
example, what sort of bugs does your product find, what sort does it
_not_ find, what resources does it need (OS, amount of disk space,
recommended minimum processing power...), what source languages does it
scan?  The last is particularly important to Emacs since, as you will
surely know, the bulk of Emacs is written in Lisp, with only a
(relatively) small core in C.  A product which doesn't analyse Lisp would
be of limited benefit to Emacs.  And, critically important, is your
product free software?

You are also asking for people's telephone numbers to register, something
unusual indeed, but don't give any assurance that they won't be passed on
to evil people or used inappropriately.  Nobody likes being rung up
during American working hours if it happens to be 3 o'clock in the
morning and they're fast asleep.

>   My belief is that we (Coverity) must reach out to the developers of 
>these packages (you) in order to make progress in actually fixing the
>defects that we happen to find, so this is my first step in that
>mission. Of course, I think Coverity technology is great, but I want to
>hear what you think and that's why I worked with folks at Coverity to
>put this infrastructure in place. The process is simple -- it checks out
>your code each night from your repository and scans it so you can always
>see the latest results.

>   Right now, we're guarding access to the actual defects that we report 
>for a couple of reasons: (1) We think that you, as developers of Emacs, 
>should have the chance to look at the defects we find to patch them 
>before random other folks get to see what we found ....

I don't think making these bugs public would be a problem at all.  Emacs
is a program with very few security aspects (though there are some).  As
mature hackers, none of us would take it as a personal slight to have our
bugs pointed out.  This mailing list, emacs-devel@gnu.org, would be an
appropriate place to report these bugs.  bug-gnu-emacs@gnu.org would be
the other appropriate place.  Bugs are to be fixed, not hidden!

>... and (2) From a support perspective, we want to make sure that we
>have the appropriate time to engage with those who want to use the
>results to fix the code.   Because of this second point, I'd ask that if
>you are interested in really digging into the results a bit further for
>your project, please have a couple of core maintainers and/or developers
>reach out to us to request access.

Emacs is about to release a new version, and most of us are busy tidying
up the "last few bugs", so now isn't a good time to be learning new tools
and processes, though there may be people who'll have a look at them.

>As this is a new process for us and still involves a small number of
>packages, I want to make sure that I personally can be involved with the
>activity that is generated from this effort.

Probably the best way would be to discuss the bugs on this mailing list,
which would be much more efficient than exchanging emails with
individuals, since the details will have to be discussed here anyway.  No
request for any sort of confidentiality would be acceptable, since this
would utterly violate the spirit of free software.  Richard Stallman
doesn't let _any_ bugs slip by, and is well known (and appreciated) for
nagging people to get them fixed.  ;-)

>   So I'm basically asking for people who want to play around with some 
>cool new technology to help make source code better. If this interests 
>you, please feel free to register on our site or email me directly. And 
>of course, if there are other packages you care about that aren't 
>currently on the list, I want to know about those too.

OK.  Again, could you either give a link to a page describing your
product in technical terms, or create such a page if it doesn't already
exist.  Even if the product doesn't cost any money, it will certainly
cost a fair bit of time to download, install and evaluate, so it's only
reasonable to expect this description.

Is the product free software?  If not, it will meet with some antipathy
from free software projects such as Emacs.

>   If this is the wrong list, my sincerest apologies and please let me 
>know where would be a more appropriate forum for this type of message.

No, I think this the right place.

>Many thanks for reading this far...

No problem!

>-ben

>  Ben Chelf
>  Chief Technology Officer
>  Coverity, Inc.

-- 
Alan Mackenzie (Munich, Germany)

Re: Coverity Open Source Defect Scan of Emacs

[David Kastrup]

Alan Mackenzie <acm@muc.de> writes:

> OK.  Again, could you either give a link to a page describing your
> product in technical terms, or create such a page if it doesn't
> already exist.  Even if the product doesn't cost any money, it will
> certainly cost a fair bit of time to download, install and evaluate,
> so it's only reasonable to expect this description.
>
> Is the product free software?  If not, it will meet with some
> antipathy from free software projects such as Emacs.

It's not really a principal problem for fixing bugs: bug reports don't
introduce copyrightable material into the fixed product.  And if
someone else is running the tool and reporting its results, there is
not much more to be annoyed at than at not having access to the brains
of human bug reporters.

Of course we would not want to become reliant on proprietary
technology, but refusing additional input for which it has played a
role seems a bit pointless.  Like a vegetarian not buying at a grocer
that isn't vegetarian himself.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Coverity Open Source Defect Scan of Emacs

[Jason Rumney]

Alan Mackenzie wrote:
> For example, what sort of bugs does your product find,
IIRC, the most notable bugs found by the Stanford project that I assume 
this is based on were in the MS Windows port, where malformed system 
messages could theoretically be used to make Emacs do things it 
shouldn't. This was not a remote exploit, and would not give the 
attacker any privileges it did not already have as far as I could tell.

Re: Coverity Open Source Defect Scan of Emacs

[Richard Stallman]

If you have found any bugs in Emacs, please report them now rather
than later.  It would be nice to fix them in the coming release.

Have you been studying the current release of Emacs, or the
development sources from CVS on savannah.gnu.org?  The latter
is the way to make it useful.

Re: Coverity Open Source Defect Scan of Emacs

[Richard Stallman]

    > Is the product free software?  If not, it will meet with some
    > antipathy from free software projects such as Emacs.

    It's not really a principal problem for fixing bugs: bug reports don't
    introduce copyrightable material into the fixed product.  And if
    someone else is running the tool and reporting its results, there is
    not much more to be annoyed at than at not having access to the brains
    of human bug reporters.

We won't encourage anyone to use non-free software for any reason,
including the finding of bugs in Emacs.  However, we won't reject or
ignore a bug report merely because the bug was found through the use
of non-free software.

Re: Coverity Open Source Defect Scan of Emacs

[Ben Chelf]

Richard Stallman wrote:
> If you have found any bugs in Emacs, please report them now rather
> than later.  It would be nice to fix them in the coming release.
> 
> Have you been studying the current release of Emacs, or the
> development sources from CVS on savannah.gnu.org?  The latter
> is the way to make it useful.
> 
> 

This scan should be the development source -- the goal is to make it 
easy to see the latest chnages as the code changes and bugs are fixed.

-ben

Re: Coverity Open Source Defect Scan of Emacs

[Richard Stallman]

    This scan should be the development source -- the goal is to make it 
    easy to see the latest chnages as the code changes and bugs are fixed.

In that case, please send us your bug reports.
The sooner we get them, the sooner we can fix them.