public inbox for bitcoindev@googlegroups.com
 help / color / mirror / Atom feed
From: Adam Ritter <aritter@gmail.com>
To: Mike Hearn <mike@plan99.net>
Cc: Bitcoin Development <bitcoin-development@lists.sourceforge.net>
Subject: Re: [Bitcoin-development] Integration testing for BitCoin
Date: Sun, 7 Apr 2013 08:50:01 -0500	[thread overview]
Message-ID: <CAKuKjyVAR=r164mYy_g_NjgXO-DSZi7BgzsbwyksuymB43DS1A@mail.gmail.com> (raw)
In-Reply-To: <CANEZrP35kFE0mkJjFWOkKMorqFg+p0BuhDpJs2Ne=MtnZwb=DA@mail.gmail.com>

Hey guys,
it sounds great. I read through the bitcoinj documentation and started
reading the code.
A few years ago it wasn't a full client, but now that I see that it's
almost there, it looks much more interesting :-)
Testing the reorg looks critical.

Thanks for the help everyone,
Adam

On Sat, Apr 6, 2013 at 7:21 AM, Mike Hearn <mike@plan99.net> wrote:
> In bitcoinj we desperately need integration tests to exercise the wallet
> code, and I think if it was done well the tests would be applicable to
> bitcoind as well. There have been a series of bugs in bitcoinj that boiled
> down to "the unit tests were not realistic enough", either because they
> stopped simulating too early or they weren't combining multiple different
> things together in the same ways as happens on the real network. Sometimes
> timing was an issue too.
>
> Examples of what I mean - ensure that re-orgs are handled correctly and
> update the wallet properly in every case, etc.
>
> Something else that would be really useful, a standalone tool that
> stress-tests the system. If we had a tool that randomly generated chains of
> transactions we might have caught the bdb lock limit bug earlier. You could
> write such a tool using bitcoinj easily, or the raw transaction APIs on
> bitcoind.
>
>
>
> On Fri, Apr 5, 2013 at 8:29 PM, Adam Ritter <aritter@gmail.com> wrote:
>>
>> Thanks guys, it sounds great.
>> Testing the JSON-RPC is/was not the main goal, just an interface for
>> testing.
>> I didn't know that the bitcoinj implementation is getting close to a
>> full implementation..it sounds interesting, as it's much easier to
>> understand and work with. I'll look at the test cases.
>>
>> Thanks very much,
>> Adam
>>
>>
>> On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell <gmaxwell@gmail.com>
>> wrote:
>> > On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter <aritter@gmail.com> wrote:
>> >> Hey guys,
>> >>
>> >> I just bought some BitCoins after being lazy to do it for the last few
>> >> years, but also looked at the client code and the messages that are
>> >> going on this mailing list.
>> >> I saw that there are quite some unit tests, but I didn't find
>> >> integration test for BitCoin, and I believe that it's quite important
>> >> for the future of BitCoin (making the current code more stable,
>> >> testing attack scenarios, refactoring and extending code).
>> > [...]
>> >> Tests that simulate multiple bitcoin users and can verify that the
>> >> whole network of bitcoin clients work together
>> >> to achieve the goals of Bitcoin. Also maybe [System
>> >> testing](http://en.wikipedia.org/wiki/System_testing)
>> >> would be a better name for the tests, but I'm not sure.
>> >
>> > I prefer to call them system tests.
>> >
>> > We use a system called blocktester that Matt Corallo wrote,
>> >
>> > https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc
>> >
>> > It's based on BitcoinJ and works by simulating a peer against a
>> > slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid
>> > computationally expensive mining).  The tests simulates many
>> > complicated network scenarios and tests the boundaries of many
>> > (hopefully all) the particular rules of the blockchain validation
>> > protocol.  We can use these tests to compare different versions of the
>> > reference software to each other and to bitcoinj (or other full node
>> > implementations) as well as comparing them to our abstract
>> > understanding of what we believe the rules of the protocol to be.
>> >
>> > These tests are run as part of the automated tests on every proposed
>> > patch to the reference software. Via a robot called pulltester which
>> > comments on github requests and produces logs like this:
>> >
>> > http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/.
>> > Pulltester also performs automatic code coverage measurements.
>> >
>> > Additionally, we run a public secondary test bitcoin network called
>> > 'testnet', which can be accessed by anyone by starting the reference
>> > software with testnet=1.  Testnet operates the same as the production
>> > network except it allows mining low difficulty blocks to prevent it
>> > going for long times without blocks, and some of the protective
>> > relaying rules against "non standard" transaction types are disabled.
>> >
>> > Most of this testing work has been centered around validating the
>> > blockchain behavior because thats what has serious systemic risk.
>> > Measuring the json rpc behavior is strictly less interesting, though
>> > interesting too.
>>
>>
>> ------------------------------------------------------------------------------
>> Minimize network downtime and maximize team effectiveness.
>> Reduce network management and security costs.Learn how to hire
>> the most talented Cisco Certified professionals. Visit the
>> Employer Resources Portal
>> http://www.cisco.com/web/learning/employer_resources/index.html
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>



      reply	other threads:[~2013-04-07 13:50 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-04-05 17:24 [Bitcoin-development] Integration testing for BitCoin Adam Ritter
2013-04-05 17:33 ` Matt Corallo
2013-04-05 17:42 ` Gregory Maxwell
2013-04-05 19:29   ` Adam Ritter
2013-04-06 12:21     ` Mike Hearn
2013-04-07 13:50       ` Adam Ritter [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAKuKjyVAR=r164mYy_g_NjgXO-DSZi7BgzsbwyksuymB43DS1A@mail.gmail.com' \
    --to=aritter@gmail.com \
    --cc=bitcoin-development@lists.sourceforge.net \
    --cc=mike@plan99.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox