[Bitcoin-development] The Bitcoin Testing Project

[Bitcoin-development] The Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I know most of you have more important things to do, and not enough
time as it is, but I would really like your feedback on the testing
project.

Bettermeans seemed ideal, it allowed for leaderless leadership with
people being able to dip in and out of what they wanted to do.
however it seems that bettermeans is in no way a finished product and
is rotting fast.

I would like to move away from bettermeans - whilst retaining the
voting and sub workstream style of working.  I would like to set
something up around MantisBT (Mantis Bug Tracker) - this is an
exceptionally versatile bit of software with plugins and interaction
with other testing products (everything from test setups and plans to
test runs and results) I would go as far as saying it is industry
standard (yeah, i know about bugzilla, etc. mantis beats them all
hands down.) obviously we would need a wiki and other software - this
is not a problem.

I am happpy to pay for the VPS' to host this stuff and set it all up.

I have quite a bit of experience with mantis and other opensource
testing stuff.

I see the testing of bitcoin to be very similar to the testing process
in the games industry.  for those that do not know how this works it
is like this:-

A company comes up with a device (xbox/ps),
They then publish a set of dos and donts for this device (TRC for sony
and TCRs for microsoft.  - I wrote quite a few of the MS TCR's for the
original xbox.)
They (ms/sony) then test your game against these rules and depending
on how many you pass/fail your game can be released or not.

I see this as mapping to bitcoin very well, the device is the
blockchain, and the TRC/TCR _tests_ are published so third party
developers can see how they fair [look at gavins recent blockchain
edge cases for an example] (ms/sony do not publish their testcases
only the requirements)

I believe that this will allow the bitcoin testing project to be able
to cope with the stable builds, bleeding edge builds and 3rd party
implementations all at the same time.

It doesnt matter what the app is, it is its interaction with the
blockchain, the safe guarding of the blockchain and compatibility with
the previous/future versions that are tested.

[for the bitcoin dev list]
A little about me:-
The below is more or less a cut and paste of some of the stuff I sent
gavin in my initial email about wanting to be in on the project.

I can back all of this up with references. I can go into more detail if
needed.

I was heavily involved with setting up the microsoft xbox european cert
department
I set up qa department for europes largest independant games developer
(although they are no longer)

worked for microsoft secure science designing security automation tools

setup the internal pentest for thales e-security (now TITS [Thales
Information Technology Solutions (or Trotters Independant Traders ;),
on thier datacryptors (fpga, crypto and product) - have done full test
cycles on hsm 8000, payshield 9000, dc2k and thier latest line of
military spec comms equipment.

Setup and pentested Thales and nCiphers credit control software (to
fips level 4 standard - This shit bitcoin exchanges need!! I know bank
is a dirty word in the bitcoin world, but we should be at least as
secure as them)

I currently find exploits in stuff like office, quicktime, ie, ff, etc
and sell them to companies like ZDI (3com) iDefense (verisgn) and some
pentest companies that require zero day exploits.

however I would like to ditch this and get my bitcoin related stuff
off the ground - with the BTP being top of the list.

my spelling is rubbish, and sometimes i forget to spellcheck before i
hit send. sorry about that.

any feedback would be really appreciated. please! I feel that this has
stagnated enough and I want to get my work out there and I want it to
be useful. (I lost 60 or so testcases because Kev left himself logged
into bettermeans and I added them under his account... none appeared
on the wiki, none appeared anywhere...)

There is already lots of stuff on the wiki that outlines how i see
things holding together - but you cant see who posted what because of
a bug that has appeared recently [meta info is just displayed as its
meta tag, {name} on {date} wrote.]

as apposed to Mistfpga on 30/8/2012 wrote...

so the tl;dr
1 - I want to use something other than bettermeans
2 - I can admin opensource software to do the same
3 - I want to take the voting/hiearachy style from bettermeans and
apply it to mantis and test workflow.
4 - I want to get some testing done asap.
5 - I have a full msdn and technet licence
6 - I have a vast array of machines [nearly 100 cores] that I can use
to automate testing and to test different setups.
7 - been mining for 18 months or so.

cheers,

steve

I do not belive this to be related, but I am not ashamed and feel no
stigma. I have posted this on a public forum. I suffer from fast
cycling (withing a day) type two bipolar disorder.  This has been and
is being managed very well by my meds (300mg quetiapine fumerate and
30mg mirtrazipine at night) I have been well for over a year now. [and
yes, even with these mental health problems I still hold SCE clearance]

- -- 
my PGP public key is at pgp.mit.edu id: 0x5016FB50

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQGRU7AAoJEFvEB9dQFvtQY9oH/RiNBeWDs+tz0NQgkgtXmxcs
tDtlLWB91gRtmxvXv+yYHE0WXS/qQ5gwaor8Vek61Cvnhij4AQBf7JVYLj03/7fy
cMAFJ5k7ZQmH9URo8Jx8xzQ2tLgZ5BqLbP6fTzDstpExNGJlkZvsC6hTx9OFCubU
bzWuWmcIRzBiMhdz7DPoskOgie5sQRzzO0zrK7NvS1P2hyEhM3eFYFcYecpS8jq9
SbkH77qnc5pZD2JULq38htVwtI16vxmg/s4GIs3OpEPaMk5xtVc5XHf6gwBZQtYw
fErwZpNDEAmGidRCB4LAqdSx1Xi0ix7GOyvTxBLWTnfB7wIAQNQpBHB9zN8diiA=
=am8B
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] The Bitcoin Testing Project

[Gary Rowe]

[-- Attachment #1: Type: text/plain, Size: 7376 bytes --]

Hi Steve,

This looks like a good idea to me. The test suites could act similarly to
the 100% Pure Java approach that successfully fended off a lot of
corrupting influences to Java over the years.

Maybe it's worth putting together a small starter suite of tests and
showing them to the community then providing a suitable process, perhaps
through BIPs, to allow tests to be created, reviewed and updated before
getting incorporated into a reference. I imagine a BIP would cover an
aspect of the blockchain rather than a single test or test suite since
having that many BIPs would get onerous fast.

Kind regards,

Gary

On 1 August 2012 12:38, steve <steve@mistfpga.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> I know most of you have more important things to do, and not enough
> time as it is, but I would really like your feedback on the testing
> project.
>
> Bettermeans seemed ideal, it allowed for leaderless leadership with
> people being able to dip in and out of what they wanted to do.
> however it seems that bettermeans is in no way a finished product and
> is rotting fast.
>
> I would like to move away from bettermeans - whilst retaining the
> voting and sub workstream style of working.  I would like to set
> something up around MantisBT (Mantis Bug Tracker) - this is an
> exceptionally versatile bit of software with plugins and interaction
> with other testing products (everything from test setups and plans to
> test runs and results) I would go as far as saying it is industry
> standard (yeah, i know about bugzilla, etc. mantis beats them all
> hands down.) obviously we would need a wiki and other software - this
> is not a problem.
>
> I am happpy to pay for the VPS' to host this stuff and set it all up.
>
> I have quite a bit of experience with mantis and other opensource
> testing stuff.
>
> I see the testing of bitcoin to be very similar to the testing process
> in the games industry.  for those that do not know how this works it
> is like this:-
>
> A company comes up with a device (xbox/ps),
> They then publish a set of dos and donts for this device (TRC for sony
> and TCRs for microsoft.  - I wrote quite a few of the MS TCR's for the
> original xbox.)
> They (ms/sony) then test your game against these rules and depending
> on how many you pass/fail your game can be released or not.
>
> I see this as mapping to bitcoin very well, the device is the
> blockchain, and the TRC/TCR _tests_ are published so third party
> developers can see how they fair [look at gavins recent blockchain
> edge cases for an example] (ms/sony do not publish their testcases
> only the requirements)
>
> I believe that this will allow the bitcoin testing project to be able
> to cope with the stable builds, bleeding edge builds and 3rd party
> implementations all at the same time.
>
> It doesnt matter what the app is, it is its interaction with the
> blockchain, the safe guarding of the blockchain and compatibility with
> the previous/future versions that are tested.
>
> [for the bitcoin dev list]
> A little about me:-
> The below is more or less a cut and paste of some of the stuff I sent
> gavin in my initial email about wanting to be in on the project.
>
> I can back all of this up with references. I can go into more detail if
> needed.
>
> I was heavily involved with setting up the microsoft xbox european cert
> department
> I set up qa department for europes largest independant games developer
> (although they are no longer)
>
> worked for microsoft secure science designing security automation tools
>
> setup the internal pentest for thales e-security (now TITS [Thales
> Information Technology Solutions (or Trotters Independant Traders ;),
> on thier datacryptors (fpga, crypto and product) - have done full test
> cycles on hsm 8000, payshield 9000, dc2k and thier latest line of
> military spec comms equipment.
>
> Setup and pentested Thales and nCiphers credit control software (to
> fips level 4 standard - This shit bitcoin exchanges need!! I know bank
> is a dirty word in the bitcoin world, but we should be at least as
> secure as them)
>
> I currently find exploits in stuff like office, quicktime, ie, ff, etc
> and sell them to companies like ZDI (3com) iDefense (verisgn) and some
> pentest companies that require zero day exploits.
>
> however I would like to ditch this and get my bitcoin related stuff
> off the ground - with the BTP being top of the list.
>
> my spelling is rubbish, and sometimes i forget to spellcheck before i
> hit send. sorry about that.
>
> any feedback would be really appreciated. please! I feel that this has
> stagnated enough and I want to get my work out there and I want it to
> be useful. (I lost 60 or so testcases because Kev left himself logged
> into bettermeans and I added them under his account... none appeared
> on the wiki, none appeared anywhere...)
>
> There is already lots of stuff on the wiki that outlines how i see
> things holding together - but you cant see who posted what because of
> a bug that has appeared recently [meta info is just displayed as its
> meta tag, {name} on {date} wrote.]
>
> as apposed to Mistfpga on 30/8/2012 wrote...
>
> so the tl;dr
> 1 - I want to use something other than bettermeans
> 2 - I can admin opensource software to do the same
> 3 - I want to take the voting/hiearachy style from bettermeans and
> apply it to mantis and test workflow.
> 4 - I want to get some testing done asap.
> 5 - I have a full msdn and technet licence
> 6 - I have a vast array of machines [nearly 100 cores] that I can use
> to automate testing and to test different setups.
> 7 - been mining for 18 months or so.
>
> cheers,
>
> steve
>
> I do not belive this to be related, but I am not ashamed and feel no
> stigma. I have posted this on a public forum. I suffer from fast
> cycling (withing a day) type two bipolar disorder.  This has been and
> is being managed very well by my meds (300mg quetiapine fumerate and
> 30mg mirtrazipine at night) I have been well for over a year now. [and
> yes, even with these mental health problems I still hold SCE clearance]
>
> - --
> my PGP public key is at pgp.mit.edu id: 0x5016FB50
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJQGRU7AAoJEFvEB9dQFvtQY9oH/RiNBeWDs+tz0NQgkgtXmxcs
> tDtlLWB91gRtmxvXv+yYHE0WXS/qQ5gwaor8Vek61Cvnhij4AQBf7JVYLj03/7fy
> cMAFJ5k7ZQmH9URo8Jx8xzQ2tLgZ5BqLbP6fTzDstpExNGJlkZvsC6hTx9OFCubU
> bzWuWmcIRzBiMhdz7DPoskOgie5sQRzzO0zrK7NvS1P2hyEhM3eFYFcYecpS8jq9
> SbkH77qnc5pZD2JULq38htVwtI16vxmg/s4GIs3OpEPaMk5xtVc5XHf6gwBZQtYw
> fErwZpNDEAmGidRCB4LAqdSx1Xi0ix7GOyvTxBLWTnfB7wIAQNQpBHB9zN8diiA=
> =am8B
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 8502 bytes --]

Re: [Bitcoin-development] The Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/08/2012 01:07, Gary Rowe wrote:
> Hi Steve,
> 
> This looks like a good idea to me. The test suites could act
> similarly to the 100% Pure Java approach that successfully fended
> off a lot of corrupting influences to Java over the years.
> 
> Maybe it's worth putting together a small starter suite of tests
> and showing them to the community then providing a suitable
> process, perhaps through BIPs, to allow tests to be created,
> reviewed and updated before getting incorporated into a reference.
> I imagine a BIP would cover an aspect of the blockchain rather than
> a single test or test suite since having that many BIPs would get
> onerous fast.
> 
> Kind regards,
> 
> Gary
> 

Hi Gary,

Thanks for the response. :)

I have started all this in bettermeans, but lost a lot of work (which
I am working on how to redo - but I am on holiday at the moment, and
have restricted access to my test setups)

Here is the discussion thread I had with gavin about acceptance tests.

https://secure.bettermeans.com/boards/4316/topics/7261

Here is the work I have currently done (note, it was losing all the
General Acceptance Tests and getting no response from bettermeans that
has lead me to not want to use it)

also note that the terminology I have been using is a little wrong, I
refer to release 0.7 as testnet release 0.7 - I will tidy it up. - I
did this before and it looks like it got reverted somehow.

here is the main wiki space that I have been using.

https://secure.bettermeans.com/projects/4256/wiki

For the General Accceptance Tests, check

https://secure.bettermeans.com/projects/4256/wiki/Dev_general_acceptance_tests

These are the basic acceptance tests based off the changelog

https://secure.bettermeans.com/projects/4256/wiki/Dev_acceptance_tests

However, notice no tests are in there yet.

There is plenty more stuff on bettermeans so please go have a poke
around.  I will try to get at least a wiki setup on a vps I have
control of (in germany, and provided by CINFU and paid for with
bitcoins) and get the stuff moved over and put the tests back up.  I
can do a limited amount of testing if the release is to happen in the
next week or two, So I will focus my efforts on the installation tests
and wallet tests. Ideally I would like to get all the GAT's done.

Hopefully this fleshes things out a bit more.

Please feel free to add/edit but remember your stuff might magically
disappear, it might be better to wait for the wiki move.  I intend on
doing the wiki move today. but that might not happen.

(I have a feeling that it was me adding stuff from an account that is
not on the BTP that blew things up, just a feeling though)

I would like to keep tests informal if possible, but that kinda goes
against the purpose of the tests. this is a bit tricky to explain if
you have not had a look at bettermeans.  bettermeans kinda has its own
BIP and voting mechanism, which is quite nice and it is what I want to
emulate - this should reduce the need for vetting of tests... have a
look at what is there and see what you think.

:)

cheers,

steve
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJQGxkFAAoJEFvEB9dQFvtQeD0IAJe9BJz/mv+kZjhk7LH1d7HH
c46D7s2Y8a+2Yobve4KtRGMoQZQiqqXGIdZ2nHVO77s0zICixqdtcKlRvBZHybw9
pB8hFYmeBdXvMHj7TR4kMbMKqTJ2z/B6m1qEKFfCRIXQXnyD5qNYhFocyQMwz53A
dkwhpoiWNVqcgnz51XEnphyohu0TPsPbOOyCrT7ORdyAgLJAs5Ig1sKbTAdSxOux
flEYKOVk0gse2b8lO2ly+eLwcQgI7jrzy+qkSKmNajRKFdvHUODXo4RraR08qiaJ
SUpmN/43uQZ4atMdOCZxD5DWKjBO96sj6mkB/po5lzIEEtkhzyp/wmKdHtlvZ/Q=
=Fonn
-----END PGP SIGNATURE-----