[Bitcoin-development] Integration testing for BitCoin

[Bitcoin-development] Integration testing for BitCoin

[Adam Ritter]

Hey guys,

I just bought some BitCoins after being lazy to do it for the last few
years, but also looked at the client code and the messages that are
going on this mailing list.
I saw that there are quite some unit tests, but I didn't find
integration test for BitCoin, and I believe that it's quite important
for the future of BitCoin (making the current code more stable,
testing attack scenarios, refactoring and extending code).

I have wrote some integration tests before at other projects, and they
usually turned out useful, but I have 0 experience with the BitCoin
development and codebase.
I wrote a short document of what I think would be the safest way to do
the testing (but not yet the tests themselves, as I don't have enough
experience..I'd like to have something like testing that the wallets
are empty, and after somebody mines she'll have more money..after she
sends money to the other person, the other person will see it...things
like this, just to get to know the code base).

What do you guys think?
The plan is here:
https://github.com/xiphias/bitcoin/blob/master/src/test/integration/README.md
Please feel free to comment/fork, I'll try to write all your replies
in the document as well.

Also here's the text to make it easier to comment:

Integration testing for bitcoin
================================

Tests that simulate multiple bitcoin users and can verify that the
whole network of bitcoin clients work together
to achieve the goals of Bitcoin. Also maybe [System
testing](http://en.wikipedia.org/wiki/System_testing)
would be a better name for the tests, but I'm not sure.

Goals
---------
- Make the bitcoin code easier to refactor while increasing the guarantee
 that it doesn't break the overall behaviour of the client.
- Make it easier to have multiple experimental coins (for example
LightCoin or PPCoin) in the codebase, while guaranteeing that the
original BitCoin protocol doesn't break
- Make it easy to test attack scenarios (like DOS,
 releasing an incompatible BitCoin client), monitoring
- Have tests without (or at least minimal number of) unreadable
constants and unreadable fake data to make them easier to verify.

Proposed implementation
------------------------------------
The first implementation should use the JSON-RPC interface and build
up as much verification
of BitCoin network as possible. It should be in C++, which makes it
easier to move to the second implementation.
The JSON-RPC calls should be hidden by a C++ interface.

The second implementation should use the same interface that was used
for JSON-RPC,
but using the BitCoin code directly (while not breaking the JSON-RPC tests).
For this the BitCoin client has to be refactored as a library,
getting rid of all global variables (and having them in a data structure),
so that multiple BitCoin clients can be run in the same process.

The improvement of the second implementation should have dependency injection
for the time and for finding/verifying a mined block,
so that the tests don't need to use real CPU power for mining,
and they can run faster and test more complex scenarios.

Re: [Bitcoin-development] Integration testing for BitCoin

[Matt Corallo]

These tests are run on each pull requests and on each new commit to
master.  They arent very complete, but they do test a lot of block
acceptance rules.

https://github.com/TheBlueMatt/test-scripts

Matt

On Fri, 2013-04-05 at 12:24 -0500, Adam Ritter wrote:
> Hey guys,
> 
> I just bought some BitCoins after being lazy to do it for the last few
> years, but also looked at the client code and the messages that are
> going on this mailing list.
> I saw that there are quite some unit tests, but I didn't find
> integration test for BitCoin, and I believe that it's quite important
> for the future of BitCoin (making the current code more stable,
> testing attack scenarios, refactoring and extending code).
> 
> I have wrote some integration tests before at other projects, and they
> usually turned out useful, but I have 0 experience with the BitCoin
> development and codebase.
> I wrote a short document of what I think would be the safest way to do
> the testing (but not yet the tests themselves, as I don't have enough
> experience..I'd like to have something like testing that the wallets
> are empty, and after somebody mines she'll have more money..after she
> sends money to the other person, the other person will see it...things
> like this, just to get to know the code base).
> 
> What do you guys think?
> The plan is here:
> https://github.com/xiphias/bitcoin/blob/master/src/test/integration/README.md
> Please feel free to comment/fork, I'll try to write all your replies
> in the document as well.
> 
> Also here's the text to make it easier to comment:
> 
> Integration testing for bitcoin
> ================================
> 
> Tests that simulate multiple bitcoin users and can verify that the
> whole network of bitcoin clients work together
> to achieve the goals of Bitcoin. Also maybe [System
> testing](http://en.wikipedia.org/wiki/System_testing)
> would be a better name for the tests, but I'm not sure.
> 
> Goals
> ---------
> - Make the bitcoin code easier to refactor while increasing the guarantee
>  that it doesn't break the overall behaviour of the client.
> - Make it easier to have multiple experimental coins (for example
> LightCoin or PPCoin) in the codebase, while guaranteeing that the
> original BitCoin protocol doesn't break
> - Make it easy to test attack scenarios (like DOS,
>  releasing an incompatible BitCoin client), monitoring
> - Have tests without (or at least minimal number of) unreadable
> constants and unreadable fake data to make them easier to verify.
> 
> Proposed implementation
> ------------------------------------
> The first implementation should use the JSON-RPC interface and build
> up as much verification
> of BitCoin network as possible. It should be in C++, which makes it
> easier to move to the second implementation.
> The JSON-RPC calls should be hidden by a C++ interface.
> 
> The second implementation should use the same interface that was used
> for JSON-RPC,
> but using the BitCoin code directly (while not breaking the JSON-RPC tests).
> For this the BitCoin client has to be refactored as a library,
> getting rid of all global variables (and having them in a data structure),
> so that multiple BitCoin clients can be run in the same process.
> 
> The improvement of the second implementation should have dependency injection
> for the time and for finding/verifying a mined block,
> so that the tests don't need to use real CPU power for mining,
> and they can run faster and test more complex scenarios.
> 
> ------------------------------------------------------------------------------
> Minimize network downtime and maximize team effectiveness.
> Reduce network management and security costs.Learn how to hire 
> the most talented Cisco Certified professionals. Visit the 
> Employer Resources Portal
> http://www.cisco.com/web/learning/employer_resources/index.html
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Integration testing for BitCoin

[Gregory Maxwell]

On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <aritter@gmail.com> wrote:
> Hey guys,
>
> I just bought some BitCoins after being lazy to do it for the last few
> years, but also looked at the client code and the messages that are
> going on this mailing list.
> I saw that there are quite some unit tests, but I didn't find
> integration test for BitCoin, and I believe that it's quite important
> for the future of BitCoin (making the current code more stable,
> testing attack scenarios, refactoring and extending code).
[...]
> Tests that simulate multiple bitcoin users and can verify that the
> whole network of bitcoin clients work together
> to achieve the goals of Bitcoin. Also maybe [System
> testing](http://en.wikipedia.org/wiki/System_testing)
> would be a better name for the tests, but I'm not sure.

I prefer to call them system tests.

We use a system called blocktester that Matt Corallo wrote,
https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc

It's based on BitcoinJ and works by simulating a peer against a
slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid
computationally expensive mining).  The tests simulates many
complicated network scenarios and tests the boundaries of many
(hopefully all) the particular rules of the blockchain validation
protocol.  We can use these tests to compare different versions of the
reference software to each other and to bitcoinj (or other full node
implementations) as well as comparing them to our abstract
understanding of what we believe the rules of the protocol to be.

These tests are run as part of the automated tests on every proposed
patch to the reference software. Via a robot called pulltester which
comments on github requests and produces logs like this:
http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/.
Pulltester also performs automatic code coverage measurements.

Additionally, we run a public secondary test bitcoin network called
'testnet', which can be accessed by anyone by starting the reference
software with testnet=1.  Testnet operates the same as the production
network except it allows mining low difficulty blocks to prevent it
going for long times without blocks, and some of the protective
relaying rules against "non standard" transaction types are disabled.

Most of this testing work has been centered around validating the
blockchain behavior because thats what has serious systemic risk.
Measuring the json rpc behavior is strictly less interesting, though
interesting too.

Re: [Bitcoin-development] Integration testing for BitCoin

[Adam Ritter]

Thanks guys, it sounds great.
Testing the JSON-RPC is/was not the main goal, just an interface for testing.
I didn't know that the bitcoinj implementation is getting close to a
full implementation..it sounds interesting, as it's much easier to
understand and work with. I'll look at the test cases.

Thanks very much,
Adam


On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <aritter@gmail.com> wrote:
>> Hey guys,
>>
>> I just bought some BitCoins after being lazy to do it for the last few
>> years, but also looked at the client code and the messages that are
>> going on this mailing list.
>> I saw that there are quite some unit tests, but I didn't find
>> integration test for BitCoin, and I believe that it's quite important
>> for the future of BitCoin (making the current code more stable,
>> testing attack scenarios, refactoring and extending code).
> [...]
>> Tests that simulate multiple bitcoin users and can verify that the
>> whole network of bitcoin clients work together
>> to achieve the goals of Bitcoin. Also maybe [System
>> testing](http://en.wikipedia.org/wiki/System_testing)
>> would be a better name for the tests, but I'm not sure.
>
> I prefer to call them system tests.
>
> We use a system called blocktester that Matt Corallo wrote,
> https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc
>
> It's based on BitcoinJ and works by simulating a peer against a
> slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid
> computationally expensive mining).  The tests simulates many
> complicated network scenarios and tests the boundaries of many
> (hopefully all) the particular rules of the blockchain validation
> protocol.  We can use these tests to compare different versions of the
> reference software to each other and to bitcoinj (or other full node
> implementations) as well as comparing them to our abstract
> understanding of what we believe the rules of the protocol to be.
>
> These tests are run as part of the automated tests on every proposed
> patch to the reference software. Via a robot called pulltester which
> comments on github requests and produces logs like this:
> http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/.
> Pulltester also performs automatic code coverage measurements.
>
> Additionally, we run a public secondary test bitcoin network called
> 'testnet', which can be accessed by anyone by starting the reference
> software with testnet=1.  Testnet operates the same as the production
> network except it allows mining low difficulty blocks to prevent it
> going for long times without blocks, and some of the protective
> relaying rules against "non standard" transaction types are disabled.
>
> Most of this testing work has been centered around validating the
> blockchain behavior because thats what has serious systemic risk.
> Measuring the json rpc behavior is strictly less interesting, though
> interesting too.

Re: [Bitcoin-development] Integration testing for BitCoin

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 4789 bytes --]

In bitcoinj we desperately need integration tests to exercise the wallet
code, and I think if it was done well the tests would be applicable to
bitcoind as well. There have been a series of bugs in bitcoinj that boiled
down to "the unit tests were not realistic enough", either because they
stopped simulating too early or they weren't combining multiple different
things together in the same ways as happens on the real network. Sometimes
timing was an issue too.

Examples of what I mean - ensure that re-orgs are handled correctly and
update the wallet properly in every case, etc.

Something else that would be really useful, a standalone tool that
stress-tests the system. If we had a tool that randomly generated chains of
transactions we might have caught the bdb lock limit bug earlier. You could
write such a tool using bitcoinj easily, or the raw transaction APIs on
bitcoind.



On Fri, Apr 5, 2013 at 8:29 PM, Adam Ritter <aritter@gmail.com> wrote:

> Thanks guys, it sounds great.
> Testing the JSON-RPC is/was not the main goal, just an interface for
> testing.
> I didn't know that the bitcoinj implementation is getting close to a
> full implementation..it sounds interesting, as it's much easier to
> understand and work with. I'll look at the test cases.
>
> Thanks very much,
> Adam
>
>
> On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell <gmaxwell@gmail.com>
> wrote:
> > On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <aritter@gmail.com> wrote:
> >> Hey guys,
> >>
> >> I just bought some BitCoins after being lazy to do it for the last few
> >> years, but also looked at the client code and the messages that are
> >> going on this mailing list.
> >> I saw that there are quite some unit tests, but I didn't find
> >> integration test for BitCoin, and I believe that it's quite important
> >> for the future of BitCoin (making the current code more stable,
> >> testing attack scenarios, refactoring and extending code).
> > [...]
> >> Tests that simulate multiple bitcoin users and can verify that the
> >> whole network of bitcoin clients work together
> >> to achieve the goals of Bitcoin. Also maybe [System
> >> testing](http://en.wikipedia.org/wiki/System_testing)
> >> would be a better name for the tests, but I'm not sure.
> >
> > I prefer to call them system tests.
> >
> > We use a system called blocktester that Matt Corallo wrote,
> >
> https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc
> >
> > It's based on BitcoinJ and works by simulating a peer against a
> > slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid
> > computationally expensive mining).  The tests simulates many
> > complicated network scenarios and tests the boundaries of many
> > (hopefully all) the particular rules of the blockchain validation
> > protocol.  We can use these tests to compare different versions of the
> > reference software to each other and to bitcoinj (or other full node
> > implementations) as well as comparing them to our abstract
> > understanding of what we believe the rules of the protocol to be.
> >
> > These tests are run as part of the automated tests on every proposed
> > patch to the reference software. Via a robot called pulltester which
> > comments on github requests and produces logs like this:
> >
> http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/
> .
> > Pulltester also performs automatic code coverage measurements.
> >
> > Additionally, we run a public secondary test bitcoin network called
> > 'testnet', which can be accessed by anyone by starting the reference
> > software with testnet=1.  Testnet operates the same as the production
> > network except it allows mining low difficulty blocks to prevent it
> > going for long times without blocks, and some of the protective
> > relaying rules against "non standard" transaction types are disabled.
> >
> > Most of this testing work has been centered around validating the
> > blockchain behavior because thats what has serious systemic risk.
> > Measuring the json rpc behavior is strictly less interesting, though
> > interesting too.
>
>
> ------------------------------------------------------------------------------
> Minimize network downtime and maximize team effectiveness.
> Reduce network management and security costs.Learn how to hire
> the most talented Cisco Certified professionals. Visit the
> Employer Resources Portal
> http://www.cisco.com/web/learning/employer_resources/index.html
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 6437 bytes --]

Re: [Bitcoin-development] Integration testing for BitCoin

[Adam Ritter]

Hey guys,
it sounds great. I read through the bitcoinj documentation and started
reading the code.
A few years ago it wasn't a full client, but now that I see that it's
almost there, it looks much more interesting :-)
Testing the reorg looks critical.

Thanks for the help everyone,
Adam

On Sat, Apr 6, 2013 at 7:21 AM, Mike Hearn <mike@plan99.net> wrote:
> In bitcoinj we desperately need integration tests to exercise the wallet
> code, and I think if it was done well the tests would be applicable to
> bitcoind as well. There have been a series of bugs in bitcoinj that boiled
> down to "the unit tests were not realistic enough", either because they
> stopped simulating too early or they weren't combining multiple different
> things together in the same ways as happens on the real network. Sometimes
> timing was an issue too.
>
> Examples of what I mean - ensure that re-orgs are handled correctly and
> update the wallet properly in every case, etc.
>
> Something else that would be really useful, a standalone tool that
> stress-tests the system. If we had a tool that randomly generated chains of
> transactions we might have caught the bdb lock limit bug earlier. You could
> write such a tool using bitcoinj easily, or the raw transaction APIs on
> bitcoind.
>
>
>
> On Fri, Apr 5, 2013 at 8:29 PM, Adam Ritter <aritter@gmail.com> wrote:
>>
>> Thanks guys, it sounds great.
>> Testing the JSON-RPC is/was not the main goal, just an interface for
>> testing.
>> I didn't know that the bitcoinj implementation is getting close to a
>> full implementation..it sounds interesting, as it's much easier to
>> understand and work with. I'll look at the test cases.
>>
>> Thanks very much,
>> Adam
>>
>>
>> On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell <gmaxwell@gmail.com>
>> wrote:
>> > On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <aritter@gmail.com> wrote:
>> >> Hey guys,
>> >>
>> >> I just bought some BitCoins after being lazy to do it for the last few
>> >> years, but also looked at the client code and the messages that are
>> >> going on this mailing list.
>> >> I saw that there are quite some unit tests, but I didn't find
>> >> integration test for BitCoin, and I believe that it's quite important
>> >> for the future of BitCoin (making the current code more stable,
>> >> testing attack scenarios, refactoring and extending code).
>> > [...]
>> >> Tests that simulate multiple bitcoin users and can verify that the
>> >> whole network of bitcoin clients work together
>> >> to achieve the goals of Bitcoin. Also maybe [System
>> >> testing](http://en.wikipedia.org/wiki/System_testing)
>> >> would be a better name for the tests, but I'm not sure.
>> >
>> > I prefer to call them system tests.
>> >
>> > We use a system called blocktester that Matt Corallo wrote,
>> >
>> > https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc
>> >
>> > It's based on BitcoinJ and works by simulating a peer against a
>> > slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid
>> > computationally expensive mining).  The tests simulates many
>> > complicated network scenarios and tests the boundaries of many
>> > (hopefully all) the particular rules of the blockchain validation
>> > protocol.  We can use these tests to compare different versions of the
>> > reference software to each other and to bitcoinj (or other full node
>> > implementations) as well as comparing them to our abstract
>> > understanding of what we believe the rules of the protocol to be.
>> >
>> > These tests are run as part of the automated tests on every proposed
>> > patch to the reference software. Via a robot called pulltester which
>> > comments on github requests and produces logs like this:
>> >
>> > http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/.
>> > Pulltester also performs automatic code coverage measurements.
>> >
>> > Additionally, we run a public secondary test bitcoin network called
>> > 'testnet', which can be accessed by anyone by starting the reference
>> > software with testnet=1.  Testnet operates the same as the production
>> > network except it allows mining low difficulty blocks to prevent it
>> > going for long times without blocks, and some of the protective
>> > relaying rules against "non standard" transaction types are disabled.
>> >
>> > Most of this testing work has been centered around validating the
>> > blockchain behavior because thats what has serious systemic risk.
>> > Measuring the json rpc behavior is strictly less interesting, though
>> > interesting too.
>>
>>
>> ------------------------------------------------------------------------------
>> Minimize network downtime and maximize team effectiveness.
>> Reduce network management and security costs.Learn how to hire
>> the most talented Cisco Certified professionals. Visit the
>> Employer Resources Portal
>> http://www.cisco.com/web/learning/employer_resources/index.html
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>