Regression testing for org-mode

Regression testing for org-mode

[Robert Goldman]

There's a page on the EmacsWiki describing schemes for unit/regression
testing in emacs lisp:

http://www.emacswiki.org/cgi-bin/wiki/UnitTesting

Perhaps we could pick up one of these?  I haven't evaluated any of them.

In response to Ben Alexander's question, yes, sometimes the screen
output is what you care about, but a big advantage of emacs is that the
screen output is a big array of text, so we could check it (it's a lot
worse, if it's just a bit map...).  Sure, we can't always do that, but I
think in many cases we could just grab up a buffer's contents and
compare it against a "right answer."

best,
r

Re: Regression testing for org-mode

[Eric Schulte]

I've added a "testing tools" section and a simple example of an org-mode
test using ert to Sebastian's org testing page in worg.  I don't mean to
advocate for ert, it is simply the only elisp testing framework that I
have any experience with.

I'm pessimistic that the colossal amount of work involved in wrapping
all of org-modes extensive and varied functionality into a test
framework can/should actually be completed.  Would it be sufficient to
simply begin dealing with bug reports in a test-first manner?

Also, should we start tracking bug reports somewhere (worg), so that
they can be claimed, tested against, and repaired?

Thanks -- Eric

Robert Goldman <rpgoldman@sift.info> writes:

> There's a page on the EmacsWiki describing schemes for unit/regression
> testing in emacs lisp:
>
> http://www.emacswiki.org/cgi-bin/wiki/UnitTesting
>
> Perhaps we could pick up one of these?  I haven't evaluated any of them.
>
> In response to Ben Alexander's question, yes, sometimes the screen
> output is what you care about, but a big advantage of emacs is that the
> screen output is a big array of text, so we could check it (it's a lot
> worse, if it's just a bit map...).  Sure, we can't always do that, but I
> think in many cases we could just grab up a buffer's contents and
> compare it against a "right answer."
>
> best,
> r
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Regression testing for org-mode

[Avdi Grimm]

On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>
> I'm pessimistic that the colossal amount of work involved in wrapping
> all of org-modes extensive and varied functionality into a test
> framework can/should actually be completed.  Would it be sufficient to
> simply begin dealing with bug reports in a test-first manner?

Absolutely.  I do NOT recommend making an effort to get test coverage
over existing code just for the sake of coverage - in my experience
this usually leads a lot of work for brittle and not terribly useful
tests.  Put tests only around the areas you are about to change.  Over
time the coverage will grow.

> Also, should we start tracking bug reports somewhere (worg), so that
> they can be claimed, tested against, and repaired?

Not a bad idea.  Normally I'd recommend just going with an established
bug tracker like Trac or Lighthouse, but since Org is so great for
managing tasks it seems only right that the developers should "eat
their own dogfood" by using Org to track tickets :-)

-- 
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com

Re: Regression testing for org-mode

[Robert Goldman]

Avdi Grimm wrote:
> On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>> I'm pessimistic that the colossal amount of work involved in wrapping
>> all of org-modes extensive and varied functionality into a test
>> framework can/should actually be completed.  Would it be sufficient to
>> simply begin dealing with bug reports in a test-first manner?
> 
> Absolutely.  I do NOT recommend making an effort to get test coverage
> over existing code just for the sake of coverage - in my experience
> this usually leads a lot of work for brittle and not terribly useful
> tests.  Put tests only around the areas you are about to change.  Over
> time the coverage will grow.
> 
>> Also, should we start tracking bug reports somewhere (worg), so that
>> they can be claimed, tested against, and repaired?
> 
> Not a bad idea.  Normally I'd recommend just going with an established
> bug tracker like Trac or Lighthouse, but since Org is so great for
> managing tasks it seems only right that the developers should "eat
> their own dogfood" by using Org to track tickets :-)
> 

Actually, I'm not sure I necessarily agree with the notion of using Org
to track tickets.  The reason is not that org mightn't be up to the job,
but that the use of org with git won't be up to it.  systems like trac
and bugzilla are set up to allow outsiders to post bugs, but if we use
git, then we're really raising the bar for bug submission.  Instead of
filling out a form, a bug reporter would now have to figure out how to
use git, pull the org file, modify it, and then either push it (which
would require someone to authorize him or her) or submit it to someone
else who would push it.  That seems inappropriate to me --- when you're
developing software a good bug report is very valuable, and one
shouldn't turn them away.

Unless someone can figure out an easier way for people to submit bugs
with what worg has now, I'm inclined to say that trac or bugzilla would
be better.

Best,
r

Re: worg for bug reports and feature requests was: (Regression testing for org-mode)

[Eric Schulte]

Robert Goldman <rpgoldman@sift.info> writes:

> Avdi Grimm wrote:
>> On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>>> Also, should we start tracking bug reports somewhere (worg), so that
>>> they can be claimed, tested against, and repaired?
>> 
>> Not a bad idea.  Normally I'd recommend just going with an established
>> bug tracker like Trac or Lighthouse, but since Org is so great for
>> managing tasks it seems only right that the developers should "eat
>> their own dogfood" by using Org to track tickets :-)
>> 
>
> Actually, I'm not sure I necessarily agree with the notion of using Org
> to track tickets.  The reason is not that org mightn't be up to the job,
> but that the use of org with git won't be up to it.  systems like trac
> and bugzilla are set up to allow outsiders to post bugs, but if we use
> git, then we're really raising the bar for bug submission.  Instead of
> filling out a form, a bug reporter would now have to figure out how to
> use git, pull the org file, modify it, and then either push it (which
> would require someone to authorize him or her) or submit it to someone
> else who would push it.  That seems inappropriate to me --- when you're
> developing software a good bug report is very valuable, and one
> shouldn't turn them away.
>
> Unless someone can figure out an easier way for people to submit bugs
> with what worg has now, I'm inclined to say that trac or bugzilla would
> be better.
>

Yes, you're correct the current method of contributing to worg is
definitely too high of a bar for bug reports, or feature requests.  That
said once they were submitted, worg would be a good mechanism for
tracking reported bugs/features, and for publishing lists of said
reports on the web.  Worg/org has the added advantage that it is already
familiar to the entire org community.

As you mention, the question seem to be: can we implement a simple
interface for reporting bugs/feature-recs which will?

1) work well with worg as it's currently used, and 
2) which won't constantly be begging for enhancements until we're
   re-implementing a full bug tracking system from scratch

I don't think this would be too difficult, say...

  a simple web form, which could be embedded into one of the worg pages,
  and could drop bug/feature-recs into an org-mode file under git.
  Would anyone be up to trying to throw such a thing together?  If only
  I had some more time...

Thanks -- Eric

Re: worg for bug reports and feature requests was: (Regression testing for org-mode)

[Robert Goldman]

Eric Schulte wrote:
> Robert Goldman <rpgoldman@sift.info> writes:
> 
>> Avdi Grimm wrote:
>>> On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>>>> Also, should we start tracking bug reports somewhere (worg), so that
>>>> they can be claimed, tested against, and repaired?
>>> Not a bad idea.  Normally I'd recommend just going with an established
>>> bug tracker like Trac or Lighthouse, but since Org is so great for
>>> managing tasks it seems only right that the developers should "eat
>>> their own dogfood" by using Org to track tickets :-)
>>>
>> Actually, I'm not sure I necessarily agree with the notion of using Org
>> to track tickets.  The reason is not that org mightn't be up to the job,
>> but that the use of org with git won't be up to it.  systems like trac
>> and bugzilla are set up to allow outsiders to post bugs, but if we use
>> git, then we're really raising the bar for bug submission.  Instead of
>> filling out a form, a bug reporter would now have to figure out how to
>> use git, pull the org file, modify it, and then either push it (which
>> would require someone to authorize him or her) or submit it to someone
>> else who would push it.  That seems inappropriate to me --- when you're
>> developing software a good bug report is very valuable, and one
>> shouldn't turn them away.
>>
>> Unless someone can figure out an easier way for people to submit bugs
>> with what worg has now, I'm inclined to say that trac or bugzilla would
>> be better.
>>
> 
> Yes, you're correct the current method of contributing to worg is
> definitely too high of a bar for bug reports, or feature requests.  That
> said once they were submitted, worg would be a good mechanism for
> tracking reported bugs/features, and for publishing lists of said
> reports on the web.  Worg/org has the added advantage that it is already
> familiar to the entire org community.
> 
> As you mention, the question seem to be: can we implement a simple
> interface for reporting bugs/feature-recs which will?
> 
> 1) work well with worg as it's currently used, and 
> 2) which won't constantly be begging for enhancements until we're
>    re-implementing a full bug tracking system from scratch
> 
> I don't think this would be too difficult, say...
> 
>   a simple web form, which could be embedded into one of the worg pages,
>   and could drop bug/feature-recs into an org-mode file under git.
>   Would anyone be up to trying to throw such a thing together?  If only
>   I had some more time...
> 



I don't mean to seem like a party-pooper, but I would suggest that we
not do this.  Getting even the simple web form done right involves
thinking hard about dealing with spam bots, people trying to hack your
site, etc.  The people who built bugzilla and trac (and apache) have a
lot more experience with this kind of issue than we do (at least
speaking for myself), and have had a lot of time to shake the kinks out.
 If you want a real ticket system, I'd suggest using one of these rather
than rolling another one.

Best,
r

Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode)

[Sebastian Rose]

While full fletched bug tracker is a _very_ usefull tool for developers,
it's a drag for users.

As for the users, I'm not shure if we should urge them to use a bug
tracking system at all. The list works so fine and natural. It's
is quite low/medium traffic with those bug reports.

As Richard Riley statud lately, it often turns out, that the 'bug'
boiles down to some configuration problem. Lots of testing is done by 
the users, simply by using org. Since we still don't have a complete set
of automated tests, this will be the case for quite some time.

And, the list is, what everyone reads. Sometimes new people show up,
read about a bug and provide patches. Why 'hide' the bugs somewhere?



That said, I still believe a bug tracker as an 'internal' tool for the 
testers/developers would be of _great_ help.

IMO, it would definitively make sense to use bugzilla or track
for that. I don't want to implement all those features from scratch,
that make a bug tracker usefull.

Where will such a system live?
Who installs and maintains it?





Robert Goldman wrote:
> Eric Schulte wrote:
>> Robert Goldman <rpgoldman@sift.info> writes:
>>
>>> Avdi Grimm wrote:
>>>> On Thu, Oct 23, 2008 at 7:57 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
>>>>> Also, should we start tracking bug reports somewhere (worg), so that
>>>>> they can be claimed, tested against, and repaired?
>>>> Not a bad idea.  Normally I'd recommend just going with an established
>>>> bug tracker like Trac or Lighthouse, but since Org is so great for
>>>> managing tasks it seems only right that the developers should "eat
>>>> their own dogfood" by using Org to track tickets :-)
>>>>
>>> Actually, I'm not sure I necessarily agree with the notion of using Org
>>> to track tickets.  The reason is not that org mightn't be up to the job,
>>> but that the use of org with git won't be up to it.  systems like trac
>>> and bugzilla are set up to allow outsiders to post bugs, but if we use
>>> git, then we're really raising the bar for bug submission.  Instead of
>>> filling out a form, a bug reporter would now have to figure out how to
>>> use git, pull the org file, modify it, and then either push it (which
>>> would require someone to authorize him or her) or submit it to someone
>>> else who would push it.  That seems inappropriate to me --- when you're
>>> developing software a good bug report is very valuable, and one
>>> shouldn't turn them away.
>>>
>>> Unless someone can figure out an easier way for people to submit bugs
>>> with what worg has now, I'm inclined to say that trac or bugzilla would
>>> be better.
>>>
>> Yes, you're correct the current method of contributing to worg is
>> definitely too high of a bar for bug reports, or feature requests.  That
>> said once they were submitted, worg would be a good mechanism for
>> tracking reported bugs/features, and for publishing lists of said
>> reports on the web.  Worg/org has the added advantage that it is already
>> familiar to the entire org community.
>>
>> As you mention, the question seem to be: can we implement a simple
>> interface for reporting bugs/feature-recs which will?
>>
>> 1) work well with worg as it's currently used, and 
>> 2) which won't constantly be begging for enhancements until we're
>>    re-implementing a full bug tracking system from scratch
>>
>> I don't think this would be too difficult, say...
>>
>>   a simple web form, which could be embedded into one of the worg pages,
>>   and could drop bug/feature-recs into an org-mode file under git.
>>   Would anyone be up to trying to throw such a thing together?  If only
>>   I had some more time...
>>
> 
> 
> 
> I don't mean to seem like a party-pooper, but I would suggest that we
> not do this.  Getting even the simple web form done right involves
> thinking hard about dealing with spam bots, people trying to hack your
> site, etc.  The people who built bugzilla and trac (and apache) have a
> lot more experience with this kind of issue than we do (at least
> speaking for myself), and have had a lot of time to shake the kinks out.
>  If you want a real ticket system, I'd suggest using one of these rather
> than rolling another one.

Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode)

[Avdi Grimm]

On Fri, Oct 24, 2008 at 10:12 AM, Sebastian Rose <sebastian_rose@gmx.de> wrote:
> Where will such a system live?
> Who installs and maintains it?

If someone actually wants to install and host the service, I recommend
using Trac.  It's open source, mature, full-featured, and used on gobs
of projects.  My place of employment uses it and we are quite happy
with it.

If you want to go the hosted ruote, there are a few possibilities:
- A SourceForge[1] account includes a ticket tracker.
- Presumably Savannah[2] (a GNU SourceForge clone) provides the same.
- Lighthouse[3] provides free hosted ticket tracking for open-source
projects.  Personally I'm not a huge fan of Lighthouse, but they've
been gaining popularity recently.

[1] http://sourceforge.net/
[2] http://savannah.nongnu.org/
[3] http://lighthouseapp.com/

-- 
Avdi

Home: http://avdi.org
Developer Blog: http://avdi.org/devblog/
Twitter: http://twitter.com/avdi
Journal: http://avdi.livejournal.com

Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode)

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]

Sebastian Rose <sebastian_rose@gmx.de> writes:

> And, the list is, what everyone reads. Sometimes new people show up,
> read about a bug and provide patches. Why 'hide' the bugs somewhere?
>

Point taken, the list has worked very well, and there is probably no
need to change it.

That said I couldn't help myself... and implemented a very simple little
web-form which can accept error reports, and append them to a "bugs.org"
org-file.  This gives every new report it's own headline in the outline,
uses TODO keywords to track the status of the report, uses properties to
track information such as the type of the report, and who is responsible
for completion, and it comments out the users input using the "^: "
syntax, to thwart any malicious inputs.  I know, I know, we should use a
mature bug tracking system or none at all, but org seems so well suited,
and this is so small.  How much trouble could it cause? :)

> Where will such a system live?
> Who installs and maintains it?

also, this solves the above problems, because if such a page were to
live as part of worg, then the resulting bugs.org files could live in
the worg git repo, and be maintained by worgers...

Thanks -- Eric

The attached mini-web-application is a ruby "camping" application, it
will build and maintain a bug file named "bugs.org" in the same
directory the script is located in.


[-- Attachment #2: org_bugs.rb --]
[-- Type: application/octet-stream, Size: 3198 bytes --]

#!/usr/bin/env ruby
require 'rubygems'
require 'camping'
## Setup
Camping.goes :OrgBugs
ORG_FILE = File.join(File.expand_path(File.dirname(__FILE__)), "bugs.org")
DAYS_OF_THE_WEEK = %w[Sun Mon Tue Wed Thu Fri Sat] 

## Views
module OrgBugs::Views
  def index
    html do
      head do
        title "Org-Mode Bug Reports and Feature Requests"
        style "label {font-weight: bold;}"
      end
      body do
        h2 "Org-Mode Bug Reports and Feature Requests"
        form(:action => "submit", :method => "post", :enctype => "multipart/form-data") do
          p do
            label "Title (required) "
            input({:name => "title", :id => "title", :type => 'text_field', :size => 52})
            br
            label "Name (required)"
            input({:name => "name", :id => "name", :type => 'text_field', :size => 52})
            br
            label "Email (required)"
            input({:name => "email", :id => "email", :type => 'text_field', :size => 52})
            br
            label "Type (required) "
            tag! :select, :name => "report_type", :id => "report_type" do
              tag! :option, "bug"
              tag! :option, "feature request"
              tag! :option, "other"
            end
            br
            p do
              "Please see the <a href=\"http://orgmode.org/org.html#Feedback\">Feedback</a> "+
                "section of the org-mode manual for information on </br> how to submit a useful bug report. "+
                "Don't forget to mention..."
            end
            ol do
              li {"What exactly did you do? "}
              li {"What did you expect to happen? "}
              li {"What happened instead? "}
            end
            label "Description (required)"
            br
            textarea({:name => 'desc', :id => 'desc', :cols => 80, :rows => 16})
            br
            input(:type => "submit", :value => "Submit")
          end
        end
      end
    end
  end
  def thanks
    html do
      head do
        title "Thanks for the Report!"
        style "label {font-weight: bold;}"
      end
      body do
        h1 "Report Submitted"
        p {"Thanks <i>#{@input.name}</i> your report <i>#{@input.title}</i> "+
          "has been submitted, and will be reviewed."}
      end
    end
  end
end

## Controllers
module OrgBugs::Controllers
  class Index < R '/'
    def get; render :index end
  end
  class Submit < R '/submit'
    def post
      report_to_org(@input)
      render :thanks
    end
  end
end

def write_prop(prop, value) "  :#{prop}: #{value}\n" end

def report_to_org(input)
  File.open(ORG_FILE, "a") do |f|
    f << "** SUBMITTED #{input.title}\n"
    f << "  :PROPERTIES:\n"
    f << write_prop(:submitter, input.name)
    f << write_prop(:submitter_email, input.email)
    f << write_prop(:date, DateTime.now.strftime("<%Y-%m-%d -%w- %H:%M>").
                    sub(/-\d-/, DAYS_OF_THE_WEEK[Integer($1)]))
    f << write_prop(:type, input.report_type)
    f << write_prop(:assigned_to, "no-one")
    f << "  :END:\n"
    f << "\n"
    f << input.desc.split(/[\n\r]/).map{|line| ": #{line.chomp}\n" if line.size > 0}.compact.join("")
    f << "\n"
  end
end

[-- Attachment #3: Type: text/plain, Size: 204 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Re: worg for bug reports and feature requests was: (Regression testing for org-mode)

[Eric Schulte]

"Eric Schulte" <schulte.eric@gmail.com> writes:
> Sebastian Rose <sebastian_rose@gmx.de> writes:
>> And, the list is, what everyone reads. Sometimes new people show up,
>> read about a bug and provide patches. Why 'hide' the bugs somewhere?
>>
>
> Point taken, the list has worked very well, and there is probably no
> need to change it.
>
> That said I couldn't help myself... and implemented a very simple little
> web-form which can accept error reports, and append them to a "bugs.org"
> org-file.  This gives every new report it's own headline in the outline,
> uses TODO keywords to track the status of the report, uses properties to
> track information such as the type of the report, and who is responsible
> for completion, and it comments out the users input using the "^: "
> syntax, to thwart any malicious inputs.  I know, I know, we should use a
> mature bug tracking system or none at all, but org seems so well suited,
> and this is so small.  How much trouble could it cause? :)
>
>> Where will such a system live?
>> Who installs and maintains it?
>
> also, this solves the above problems, because if such a page were to
> live as part of worg, then the resulting bugs.org files could live in
> the worg git repo, and be maintained by worgers...
>
> Thanks -- Eric
>
> The attached mini-web-application is a ruby "camping" application, it
> will build and maintain a bug file named "bugs.org" in the same
> directory the script is located in.

figured I'd actually put this up, as a better demonstration than just
the source.  It's just running on a server in my study, so please if you
see any glaring security holes, don't kill my machine.

http://org-bug.suey.ath.cx/

Thanks -- Eric