From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-1.v43.ch3.sourceforge.com ([172.29.43.191] helo=mx.sourceforge.net) by sfs-ml-1.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1UOCKk-0001eX-DB for bitcoin-development@lists.sourceforge.net; Fri, 05 Apr 2013 19:29:58 +0000 Received-SPF: pass (sog-mx-1.v43.ch3.sourceforge.com: domain of gmail.com designates 209.85.216.175 as permitted sender) client-ip=209.85.216.175; envelope-from=aritter@gmail.com; helo=mail-qc0-f175.google.com; Received: from mail-qc0-f175.google.com ([209.85.216.175]) by sog-mx-1.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1UOCKi-0003d1-Vn for bitcoin-development@lists.sourceforge.net; Fri, 05 Apr 2013 19:29:58 +0000 Received: by mail-qc0-f175.google.com with SMTP id j3so1375847qcs.34 for ; Fri, 05 Apr 2013 12:29:51 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.229.14.130 with SMTP id g2mr3772175qca.90.1365190191405; Fri, 05 Apr 2013 12:29:51 -0700 (PDT) Received: by 10.49.26.41 with HTTP; Fri, 5 Apr 2013 12:29:51 -0700 (PDT) In-Reply-To: References: Date: Fri, 5 Apr 2013 14:29:51 -0500 Message-ID: From: Adam Ritter To: Gregory Maxwell , Matt Corallo Content-Type: text/plain; charset=ISO-8859-1 X-Spam-Score: -1.6 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (aritter[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1UOCKi-0003d1-Vn Cc: Bitcoin Development Subject: Re: [Bitcoin-development] Integration testing for BitCoin X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2013 19:29:58 -0000 Thanks guys, it sounds great. Testing the JSON-RPC is/was not the main goal, just an interface for testing. I didn't know that the bitcoinj implementation is getting close to a full implementation..it sounds interesting, as it's much easier to understand and work with. I'll look at the test cases. Thanks very much, Adam On Fri, Apr 5, 2013 at 12:42 PM, Gregory Maxwell wrote: > On Fri, Apr 5, 2013 at 10:24 AM, Adam Ritter wrote: >> Hey guys, >> >> I just bought some BitCoins after being lazy to do it for the last few >> years, but also looked at the client code and the messages that are >> going on this mailing list. >> I saw that there are quite some unit tests, but I didn't find >> integration test for BitCoin, and I believe that it's quite important >> for the future of BitCoin (making the current code more stable, >> testing attack scenarios, refactoring and extending code). > [...] >> Tests that simulate multiple bitcoin users and can verify that the >> whole network of bitcoin clients work together >> to achieve the goals of Bitcoin. Also maybe [System >> testing](http://en.wikipedia.org/wiki/System_testing) >> would be a better name for the tests, but I'm not sure. > > I prefer to call them system tests. > > We use a system called blocktester that Matt Corallo wrote, > https://code.google.com/r/bluemattme-bitcoinj/source/browse/core/src/test/java/com/google/bitcoin/core/FullBlockTestGenerator.java?name=fullverif&r=874c5904b12d1fcec5b556429cf208f63cd4e1bc > > It's based on BitcoinJ and works by simulating a peer against a > slightly instrumented copy of Bitcoin(d/-qt) (modified to avoid > computationally expensive mining). The tests simulates many > complicated network scenarios and tests the boundaries of many > (hopefully all) the particular rules of the blockchain validation > protocol. We can use these tests to compare different versions of the > reference software to each other and to bitcoinj (or other full node > implementations) as well as comparing them to our abstract > understanding of what we believe the rules of the protocol to be. > > These tests are run as part of the automated tests on every proposed > patch to the reference software. Via a robot called pulltester which > comments on github requests and produces logs like this: > http://jenkins.bluematt.me/pull-tester/92a129980fb9b506da6c7f876aa8adb405c88e17/. > Pulltester also performs automatic code coverage measurements. > > Additionally, we run a public secondary test bitcoin network called > 'testnet', which can be accessed by anyone by starting the reference > software with testnet=1. Testnet operates the same as the production > network except it allows mining low difficulty blocks to prevent it > going for long times without blocks, and some of the protective > relaying rules against "non standard" transaction types are disabled. > > Most of this testing work has been centered around validating the > blockchain behavior because thats what has serious systemic risk. > Measuring the json rpc behavior is strictly less interesting, though > interesting too.