Re: [Bitcoin-development] Bitcoin Testing Project

[Matt Corallo <bitcoin-list@bluematt.me>]

On Wed, 2012-09-26 at 13:28 +0100, steve wrote:
> Hi Matt,
> 
> Glad to have another ninja onboard :)
> 
> On 25/09/2012 21:41, Matt Corallo wrote:
> > Although Jenkins may not be the best system, we already have
> > jenkins and pull-tester (which is a dumb python script I wrote to
> > test all incoming pull requests from github).
> 
> I have never heard of jenkins before.  I need to do some more digging.
> is this the right thing?
> 
> https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin
For a mantis plugin, sure, I guess...
> 
> Mantis on the other hand, I know exceptionally well.  I hate
> duplication of work/data unless absolutely necessary.  I will check
> jenkins out (just out of interest what is it actually meant to do? the
> website implies framework, but not what its for)
Jenkins currently just runs the test script after each new commit to
bitcoin (and provides binaries to anyone who wants them), so its pretty
basic (though jenkins has way more features than we use).  The bitcoin
one lives at http://jenkins.bluematt.me/
> 
> > 
> > They both run the same set of scripts, namely those at 
> > https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> > now, but since it is on github, I was hoping someone would find
> > the inspiration to add to it).
> 
> I will check it out. I wrote a very basic script that wikified the
> changelog,
We currently keep a changelog at https://en.bitcoin.it/wiki/Changelog (I
went back and added tons of logs a while back and it got updated, though
0.7 seems to be missing...) anyway, automating that would be nice...
> and linked to the changes and created wiki pages for the
> testcases.  
Having more info on that changelog page would be nice.
> have you seen the stuff I put on bettermeans? bits keep
> vanishing then re appearing.
I have been meaning to catch up with the various attempts at better
bitcoin testing that have started up a few times, but I keep never
getting around to it...
> 
> This is the outline of the testing that I setup for 0.7
> 
> https://secure.bettermeans.com/projects/4256/wiki
> 
> > 
> > I dont really care if we keep using jenkins, but I figure we might
> > as well keep all the tests in one place?
> 
> Yes, I would love to unify all build testing and testcases into one
> place.  I am still on the fence as to including unit tests into this.
> However I do see 3 distinct type of testcases
Even if unit tests are considered separate, having it all run in one
huge test script makes it quite easy to implement new things (like
pull-tester) which test some arbitrary bitcoind commit in the same way
as every other tester.  
> 1 - requirements based testcases (requirements based off the current
> block chain rules - these are edge cases and known interoperability
> issues)
The BitcoinjBitcoindComparisonTool.jar file which is run as a part of
the test scripts tries to hit as many block acceptance edge cases as
possible (I'm sure I missed a ton, but it hits a lot too).  I've also
been pushing alternate implementation implementors to use it to test
their own implementations.
> 
> 2 - Acceptance based testcases - these are testcases that should be
> run for every build.  Check out the General Acceptance Tests in the
> wiki link for examples and testcases
> 
> 3 - Testcases for reference implementations of things (like multisig -
> i see these working like the /test folder when you install a new perl
> module)
> 
> These three things alone are a massive task. and they still wont cover
> everything.  I would like to get the workflow so that people can
> sponsor or donate to a specific campaign (eg a new feature is
> implemented, people want it tested so can donate just for that
> campaign [developing testcases, structure, requirements, etc])
> 
> Once this is done, I will get to do some exciting stuff (like writing
> fuzzers, automation, etc) unfortunately I do not know python, only perl.
As far as I'm concerned more test cases are more test cases, it may get
unwieldy to maintain, but at least we'd have more test cases :)

In terms of general testing strategies, I really prefer to script it
all, jenkins is quite nice in that it can have slave workers using a
different OS which run their own tests and then report back to the main
jenkins instance.  Getting a real Windows slave to run the installer and
test that thoroughly as well as basic Mac things (I know OSX uses a very
different build system...) would be nice (though I dont really have time
to write all those tests...)

re: GUI testing is hard: I've heard Qt's unit test framework is really
powerful and can even include things like click scripting and analysis
of the current views (though, I agree, its still no doubt hard).  

Matt