From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1TGv8E-0005en-H6 for bitcoin-development@lists.sourceforge.net; Wed, 26 Sep 2012 17:10:42 +0000 X-ACL-Warn: Received: from mail-qc0-f175.google.com ([209.85.216.175]) by sog-mx-2.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1TGv89-0006Vc-0T for bitcoin-development@lists.sourceforge.net; Wed, 26 Sep 2012 17:10:42 +0000 Received: by qcad10 with SMTP id d10so757037qca.34 for ; Wed, 26 Sep 2012 10:10:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:cc:content-type:x-gm-message-state; bh=gEOGK3lLDrAzobHcfIGr6i7BVOv6XQW8g5/AGgOk3/c=; b=ACeYx3ounPO1wvtNMIb5TI7ytXrOrJ7hJy+z7qBUyZlNMB77EkcX9H1Y+w8tkLi1tP SIg46ihnlHZdoDi9DQ98Zs1EmImMfxh52+RMLIlKHq2ol0yXtlQu7A3TrWqL3uQWsdqu fCFtOYD8M6JF2taulP+5xr/8Vnq5degp1C+hBGNm5mZGVAxSw5ZJHl3WL+vXVTD7xBOZ isF+57dmCu1z/Uop4suByoR726Sna89A0SudZ+ViBBS3BF8UgjmXCoXiJb1iwsGQDT7L 5YeggOVxWN2Cf8lNubXttKfRZ4rCUtRwNH7f24p892BHVS6wjclT3JuO6wBi3ZzVVTZn Movw== MIME-Version: 1.0 Received: by 10.224.70.138 with SMTP id d10mr3538694qaj.12.1348679431459; Wed, 26 Sep 2012 10:10:31 -0700 (PDT) Received: by 10.49.97.6 with HTTP; Wed, 26 Sep 2012 10:10:31 -0700 (PDT) X-Originating-IP: [2001:4830:1603:2:21c:c0ff:fe79:c8c2] In-Reply-To: References: <5061F8CC.9070906@mistfpga.net> <1348605677.2284.2.camel@localhost.localdomain> <5062F4F8.6040504@mistfpga.net> <506301AC.90101@mistfpga.net> Date: Wed, 26 Sep 2012 13:10:31 -0400 Message-ID: From: Jeff Garzik To: Mark Friedenbach Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQlh3RoymHYHf9YQsU45+X0UJA1XGoaUjky6KQjIMlwNa6Ilq7+pmH9jYgCemomob4AeF29u X-Spam-Score: 0.0 (/) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. X-Headers-End: 1TGv89-0006Vc-0T Cc: Bitcoin Development List , Bill Hees Subject: Re: [Bitcoin-development] Bitcoin Testing Project X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2012 17:10:42 -0000 On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach wrote: > Certainly developers should be responsible for making sure that regression > tests for bugs they fix make it either into the unit tests or Matt's > functional test repository. QA should hold them accountable for that > (re-opening tickets for bugs that have been fixed but without regression > tests). As a goal or general principle, this is agreeable. But slavish attention to this will only get ignored. There is finite developer resources, and regression tests for certain types of bugs, like prickly P2P network interaction bugs or RPC API bugs, could potentially involve many days or weeks of coding, to sufficiently simulate the environment. The ability to easily, automatically and programmatically reproduce certain classes of bugs is simply out of reach right now, and nobody is going to shut down development to fix that problem. We should move towards this direction, yes, but bitcoin test cases are not always going to be as easy as writing (say) a compiler testcase. We can always use the help of a few good QA coders: simulating a P2P environment and checking the RPC API are two examples of very complicated problems that -can- be automated for testing... with a lot of work. -- Jeff Garzik exMULTI, Inc. jgarzik@exmulti.com