[Bitcoin-development] Bitcoin Testing Project

[Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi All,

After the failure to get any real testing done for the 0.7 release (all
of which is my fault) I have decided to rejig things.

I am heavily into test driven development, and I have a strong
background in requirements management, and automation.

I want to leave bettermeans behind, maybe we might be able to keep the
voting aspect of it, and link it into mantis.

So, what I have been doing over the past few weeks is developing a
rudimentary requirements set, basic requirement tracking, tests to
prove/stress the requirements.

The next most important thing is to get release/acceptance tests done -
these primarily focus on new stuff doesnt break old (ie lose a wallet,
etc) and needs no special requirements.

To this end I have installed various opensource applications (mantis,
salomeTMF, bugzilla, etc) and am currently evaluating the best workflow
process.

This was a much longer post, but decided against it. :)

So, what I want to know is who wants to be a part helping me out with
all this? I am finalising the workflow flow diagrams and they should be
ready for inspection soon.

Anyone interested in helping out/reviewing processes? even if it is just
some encouragement, it is all greatly appreciated.

Drop me an email if you want access to the current setup and help me
review the different software for the bitcoin workflow process.

cheers,

steve

- -- 
my PGP public key is at pgp.mit.edu id: 0x5016FB50
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYfjMAAoJEFvEB9dQFvtQSmsH/R/FEdOQRB7ncTnHhaP8woLu
nIiGX2DgLOWLOF9launSuTCrtVm2G56B9Dgl/BqScFxeuJGbzje7+kp7LgjtA3uy
kS9DUZ1zhUfhslGP0UpVJJGX6Yfk8GbQ4nUcuL1VTv6nSZXWP2EvLMDPpRgKwyi5
z1FiyBg2A3Kg3Er+VmHPmpI0zZAGB5ytaenUp4xXGhL7Nk66i5X0twVr51xlEm0L
zKCDXHzWTvNNlT7TzMjIxShJ/EcgCI1r6tVD3T+2e9QeVm0QNw3xeNUkMxKn+ul8
d1v1OxJbHD1CsNqW+XgVvFE2SJReizaHNOFwrqcpVCp7bABnWAB5eyTzB9B9IX8=
=di5x
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Matt Corallo]

Although Jenkins may not be the best system, we already have jenkins and
pull-tester (which is a dumb python script I wrote to test all incoming
pull requests from github).  

They both run the same set of scripts, namely those at
https://github.com/TheBlueMatt/test-scripts (its pretty basic right now,
but since it is on github, I was hoping someone would find the
inspiration to add to it).

I dont really care if we keep using jenkins, but I figure we might as
well keep all the tests in one place?

Anyway, I'm all for more testing (I'm always complaining about how we
need more tests for stuff...).

Matt

On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
> Hi All,
> 
> After the failure to get any real testing done for the 0.7 release (all
> of which is my fault) I have decided to rejig things.
> 
> I am heavily into test driven development, and I have a strong
> background in requirements management, and automation.
> 
> I want to leave bettermeans behind, maybe we might be able to keep the
> voting aspect of it, and link it into mantis.
> 
> So, what I have been doing over the past few weeks is developing a
> rudimentary requirements set, basic requirement tracking, tests to
> prove/stress the requirements.
> 
> The next most important thing is to get release/acceptance tests done -
> these primarily focus on new stuff doesnt break old (ie lose a wallet,
> etc) and needs no special requirements.
> 
> To this end I have installed various opensource applications (mantis,
> salomeTMF, bugzilla, etc) and am currently evaluating the best workflow
> process.
> 
> This was a much longer post, but decided against it. :)
> 
> So, what I want to know is who wants to be a part helping me out with
> all this? I am finalising the workflow flow diagrams and they should be
> ready for inspection soon.
> 
> Anyone interested in helping out/reviewing processes? even if it is just
> some encouragement, it is all greatly appreciated.
> 
> Drop me an email if you want access to the current setup and help me
> review the different software for the bitcoin workflow process.
> 
> cheers,
> 
> steve

Re: [Bitcoin-development] Bitcoin Testing Project

[Wladimir]

I think it's a great initiative Steve,

On Tue, Sep 25, 2012 at 10:41 PM, Matt Corallo <bitcoin-list@bluematt.me> wrote:

> They both run the same set of scripts, namely those at
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right now,
> but since it is on github, I was hoping someone would find the
> inspiration to add to it).

The pulltester and Jenkins are really useful, I think it's best to
keep these and improve them if needed.

> I dont really care if we keep using jenkins, but I figure we might as
> well keep all the tests in one place?

Which brings me to another issue: we desperately need a page that
links all the developer resources for bitcoin dev:

- link to my doxygen docs (https://dev.visucore.com/bitcoin/doxygen/)
- bluematt's pulltester and jenkins
- any testing dashboards Steve is going to add

But also:

- how to check out the source code, how to find the build instructions
for your platform
- contribution guidelines
- where to ask w/ development problems (this mailing list, #bitcoin-dev)
- where to begin, which bugs to solve first
...

Maybe on bitcoin.org, but at least easy to find for people that want
to contribute, not just some deeply nested wiki page. Ie, something
like

https://www.mozilla.org/en-US/contribute/
http://www.ogre3d.org/developers
...

I'm willing to write this. But I know these kinds of proposals always
end in a big discussion about what should be and what should not be on
bitcoin.org, however we should be a bit pragmatic here.

Wladimir

Re: [Bitcoin-development] Bitcoin Testing Project

[Daniel F]

[-- Attachment #1: Type: text/plain, Size: 466 bytes --]

on 09/26/2012 01:49 AM Wladimir said the following:

> I'm willing to write this. But I know these kinds of proposals always
> end in a big discussion about what should be and what should not be on
> bitcoin.org, however we should be a bit pragmatic here.

May I suggest a page bitcoin.org/developers, that links to a wiki page
of developer resources?
That way there's an easy link from the main site, but the content is
readily editable and expandable.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [Bitcoin-development] Bitcoin Testing Project

[Luke-Jr]

On Wednesday, September 26, 2012 11:41:13 AM Daniel F wrote:
> on 09/26/2012 01:49 AM Wladimir said the following:
> > I'm willing to write this. But I know these kinds of proposals always
> > end in a big discussion about what should be and what should not be on
> > bitcoin.org, however we should be a bit pragmatic here.
> 
> May I suggest a page bitcoin.org/developers, that links to a wiki page
> of developer resources?
> That way there's an easy link from the main site, but the content is
> readily editable and expandable.

The front page already has wiki links. Adding a direct link to a developer 
resources page there would probably make sense.

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Matt,

Glad to have another ninja onboard :)

On 25/09/2012 21:41, Matt Corallo wrote:
> Although Jenkins may not be the best system, we already have
> jenkins and pull-tester (which is a dumb python script I wrote to
> test all incoming pull requests from github).

I have never heard of jenkins before.  I need to do some more digging.
is this the right thing?

https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin

Mantis on the other hand, I know exceptionally well.  I hate
duplication of work/data unless absolutely necessary.  I will check
jenkins out (just out of interest what is it actually meant to do? the
website implies framework, but not what its for)

So, currently there are 4 potential places for bugs to be reported
1 - jenkins (and unit tests)
2 - git
3 - mailing list
4 - forum (bitcointalk...)
5? - is there still the ability to add bugs via sourceforge?

Adding to this doesnt make sense.  Each one of these reporting methods
is for a different thing.  I am not seeking to replace these (or even
unify them) I am looking for software that will take testcases and bug
reports against them [and allow for test campaigns].  Mantis is so
flexible and industry standard and if the jenkins plugin works... then
we can keep things as they are until they fit into better places.

The reason I am so behind mantis as the backbone is it works with more
or less anything, and can easily modded to work with whatever people
are most comfortable with - however it is exceptionally powerful and
has had a constant stream of workflow improvements over the past few
years.

> 
> They both run the same set of scripts, namely those at 
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> now, but since it is on github, I was hoping someone would find
> the inspiration to add to it).

I will check it out. I wrote a very basic script that wikified the
changelog, and linked to the changes and created wiki pages for the
testcases.  have you seen the stuff I put on bettermeans? bits keep
vanishing then re appearing.

This is the outline of the testing that I setup for 0.7

https://secure.bettermeans.com/projects/4256/wiki

> 
> I dont really care if we keep using jenkins, but I figure we might
> as well keep all the tests in one place?

Yes, I would love to unify all build testing and testcases into one
place.  I am still on the fence as to including unit tests into this.
However I do see 3 distinct type of testcases
1 - requirements based testcases (requirements based off the current
block chain rules - these are edge cases and known interoperability
issues)

2 - Acceptance based testcases - these are testcases that should be
run for every build.  Check out the General Acceptance Tests in the
wiki link for examples and testcases

3 - Testcases for reference implementations of things (like multisig -
i see these working like the /test folder when you install a new perl
module)

These three things alone are a massive task. and they still wont cover
everything.  I would like to get the workflow so that people can
sponsor or donate to a specific campaign (eg a new feature is
implemented, people want it tested so can donate just for that
campaign [developing testcases, structure, requirements, etc])

Once this is done, I will get to do some exciting stuff (like writing
fuzzers, automation, etc) unfortunately I do not know python, only perl.

> 
> Anyway, I'm all for more testing (I'm always complaining about how
> we need more tests for stuff...).

Nice, I love testing.  I think we will get on :)

And I would rather go for interoperability between testing rather than
rewriting it all.

Cheers,

steve

> 
> Matt
> 
> On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
>> Hi All,
>> 
>> After the failure to get any real testing done for the 0.7
>> release (all of which is my fault) I have decided to rejig
>> things.
>> 
>> I am heavily into test driven development, and I have a strong 
>> background in requirements management, and automation.
>> 
>> I want to leave bettermeans behind, maybe we might be able to
>> keep the voting aspect of it, and link it into mantis.
>> 
>> So, what I have been doing over the past few weeks is developing
>> a rudimentary requirements set, basic requirement tracking, tests
>> to prove/stress the requirements.
>> 
>> The next most important thing is to get release/acceptance tests
>> done - these primarily focus on new stuff doesnt break old (ie
>> lose a wallet, etc) and needs no special requirements.
>> 
>> To this end I have installed various opensource applications
>> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating
>> the best workflow process.
>> 
>> This was a much longer post, but decided against it. :)
>> 
>> So, what I want to know is who wants to be a part helping me out
>> with all this? I am finalising the workflow flow diagrams and
>> they should be ready for inspection soon.
>> 
>> Anyone interested in helping out/reviewing processes? even if it
>> is just some encouragement, it is all greatly appreciated.
>> 
>> Drop me an email if you want access to the current setup and help
>> me review the different software for the bitcoin workflow
>> process.
>> 
>> cheers,
>> 
>> steve
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYvT4AAoJEFvEB9dQFvtQlgkIAJX7JYel5RGmCsbptGdQrCnT
BR42tUwTg1t/NRUJ6RA8/Ou8lzallztQquShpLn4mZdQpoalvETdtAwcPnQKnaZb
M5inZE/IEq8WJM1y4YkHt3BLou4BJbjwncCNy1/jqcm6f2Oonrg7isVbDwY/7JlP
y/epm7XELS7NU4vVubBwQCunwvtsuydXRzuI812LiLXNqpXFMHvG2m8a2RajXE0/
xW4lOMy/hUFzEgYRQWCTAru4Ts2x3Xt26NaEUh/uKvHLwBZJ4xbdu3gpupiPb4sI
bCHnVFOC7zoQKOAnfPkCMyvtyoqpzM9HW2+DWI51FoOz851Y2F36N3Fpk/2lii4=
=W5xI
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Wladimir]

Steve,

> So, currently there are 4 potential places for bugs to be reported
> 1 - jenkins (and unit tests)
> 2 - git
> 3 - mailing list
> 4 - forum (bitcointalk...)
> 5? - is there still the ability to add bugs via sourceforge?

Currently github is the authoritative place to report issues. When
someone reports a bug on the mailing list, IRC or forum, they are
generally asked to make a github issue (or, someone else makes the
issue for them). Failed tests are generally also reported on github,
by the pull tester.

We currently have 232 issues, mostly classified into categories such
as "Bug", "Improvement", "GUI", "Wallet", and so on.

Also it's easy to refer to github issues in commits with #123, with
automatic linking.

I'm not sure it is worth the effort to move to another system
(especially if you need a another login etc...). But I'm probably
misunderstanding what you're trying to do.

Wladimir

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 26/09/2012 13:49, Wladimir wrote:
> Steve,

Hi Wladimir,

> 
>> So, currently there are 4 potential places for bugs to be 
>> reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 
>> - forum (bitcointalk...) 5? - is there still the ability to add 
>> bugs via sourceforge?
> 
> Currently github is the authoritative place to report issues. When
>  someone reports a bug on the mailing list, IRC or forum, they are
>  generally asked to make a github issue (or, someone else makes the
>  issue for them). Failed tests are generally also reported on 
> github, by the pull tester.

excellent, that makes things much easier.

> 
> We currently have 232 issues, mostly classified into categories 
> such as "Bug", "Improvement", "GUI", "Wallet", and so on.
> 
> Also it's easy to refer to github issues in commits with #123, with
> automatic linking.
> 
> I'm not sure it is worth the effort to move to another system 
> (especially if you need a another login etc...). But I'm probably 
> misunderstanding what you're trying to do.

I think you might be misunderstanding a little. I am not trying to
replace the current system, I need to make sure that what I do will be
compatible with it (seamlessly so for the developer). I do not want this
to generate extra work for the development team.

However testing is a lot more than just bug reporting, dont get me
wrong bug reports are important, but so is running a testcase and that
testcase passing, especially if that testcase is linked to the proof
of a requirement. I am trying to develop a qa environment that is
conducive to testing and will allow the testers to shine in all their
glory :) and we need different tools and methodologies.

Git is too developer centric to be useful for organising testing. -
however there is a large amount of software that is compatible with
git, so the core development team only ever need to work with git.

The linking between a bug, the requirement, the fix, the retest, and
updating of regression testplan is vital. So is the ability to
organise testing campaigns and assigning tests, work units and test
relevant docs/scripts/ideas, etc.

I hope this clears things up a bit?

Cheers,

steve

> 
> Wladimir
> 

- -----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYwESAAoJEFvEB9dQFvtQ/GUH/jv2c5L0OcL/kHkX/z0Yqbl/
2IntPLdjXNKLuz0A7BMz7XfUyVmWlZrw44qxmi+Vyk5PKNBjYIidm763xHnTeJLN
ULQBckYexMvan9hAyYZUOt85IpesdNgqTIsqh8f49y4roHOy8GT4M/2fhzXpnsGg
G9d2m8jWGpj/kxl9qE7/WjVQC4APwBi/NiJsCrcHswgweN+zENc/Pot9YBLxAZu/
ACBUX/xFymRdaZN8P2LWBXuKx6E2WEcBdPCCWArX07wPiBlrashx9Gz6tiNzIiNq
x2c4ltLzRa45AmiDtQhwqyTprz/DbyeAYO1sIsfpUxDeu9e3xTb/Zd96jfKIWI0=
=iHI1
- -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYwGsAAoJEFvEB9dQFvtQxOsIAKgBBOKHNFtoV2cN+GVqzlip
yy0qiMvMTZKrraOhEw8QNNuOlB3LRchi+RDR/PvQkVfuwi/jHB2gUBzlapLoECBv
EH8pgT/MO281pXzARgRSVkRYqkb3ljhQz3mEQg9RhR9h5t9g2mL3Tvppt7249Bg8
oGXPj6xmMcrbClF5qDbwQUUDGJfOo4eti0jSVD3qp2NE7QpPVQwuN5buchpoKt3P
9aJnjeZdLmuAk2RPoDaLXUFc9unT8AcnW96juD0zoVA9wKvAa6/8IZQf0mzV4iZP
yiWGNOQtBZ+jyu2ixiEnvHqqG2ZmjtUVqWtjHkxYgrCyuuK2jOcTMNEWfn7SfKc=
=yP7N
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Mark Friedenbach]

[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]

Running a concurrent Mantis tracker would be confusing and fragment the
development pathway. We have an issue tracker; it's on github.

What's being talked about here are two separate things. Jenkins is a
continuous integration system. It can be configured to run the suite of
unit tests, regression tests, and any kind of automated functional tests
for every commit on github and every pull request.

Github is our issue tracker. Github, and only github, is where new issues
should be reported (unless it's security related, in which case an email
should be sent to the core devs directly).

Certainly developers should be responsible for making sure that regression
tests for bugs they fix make it either into the unit tests or Matt's
functional test repository. QA should hold them accountable for that
(re-opening tickets for bugs that have been fixed but without regression
tests).

The other thing we're talking about is coordinated release testing--getting
release candidates in the hands of actual users and making sure that issues
are reported. This is something that can't be automated as the point really
is to pick up on things that the testing suite missed. You sound more
qualified than me for coming up with a process, but in the end discovered
issues should be reported to github, the final repository of issues that
hold up Gavin from doing a release.

Just my 0.002BTC
Mark

On Wed, Sep 26, 2012 at 6:22 AM, steve <steve@mistfpga.net> wrote:
>
> I think you might be misunderstanding a little. I am not trying to
> replace the current system, I need to make sure that what I do will be
> compatible with it (seamlessly so for the developer). I do not want this
> to generate extra work for the development team.
>
> However testing is a lot more than just bug reporting, dont get me
> wrong bug reports are important, but so is running a testcase and that
> testcase passing, especially if that testcase is linked to the proof
> of a requirement. I am trying to develop a qa environment that is
> conducive to testing and will allow the testers to shine in all their
> glory :) and we need different tools and methodologies.
>
> Git is too developer centric to be useful for organising testing. -
> however there is a large amount of software that is compatible with
> git, so the core development team only ever need to work with git.
>
> The linking between a bug, the requirement, the fix, the retest, and
> updating of regression testplan is vital. So is the ability to
> organise testing campaigns and assigning tests, work units and test
> relevant docs/scripts/ideas, etc.
>
> I hope this clears things up a bit?
>
> Cheers,
>
> steve
>

[-- Attachment #2: Type: text/html, Size: 3149 bytes --]

Re: [Bitcoin-development] Bitcoin Testing Project

[Jeff Garzik]

On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach <mark@monetize.io> wrote:
> Certainly developers should be responsible for making sure that regression
> tests for bugs they fix make it either into the unit tests or Matt's
> functional test repository. QA should hold them accountable for that
> (re-opening tickets for bugs that have been fixed but without regression
> tests).

As a goal or general principle, this is agreeable.

But slavish attention to this will only get ignored.  There is finite
developer resources, and regression tests for certain types of bugs,
like prickly P2P network interaction bugs or RPC API bugs, could
potentially involve many days or weeks of coding, to sufficiently
simulate the environment.  The ability to easily, automatically and
programmatically reproduce certain classes of bugs is simply out of
reach right now, and nobody is going to shut down development to fix
that problem.

We should move towards this direction, yes, but bitcoin test cases are
not always going to be as easy as writing (say) a compiler testcase.

We can always use the help of a few good QA coders:  simulating a P2P
environment and checking the RPC API are two examples of very
complicated problems that -can- be automated for testing... with a lot
of work.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 26/09/2012 17:06, Mark Friedenbach wrote:
> Running a concurrent Mantis tracker would be confusing and fragment
> the development pathway. We have an issue tracker; it's on github.

I think you misunderstand what I am proposing.

QA needs more than just an issue tracker. i have yet to find any
opensource software that integrates testcases, nor any method for
generating testplans, nor any method for linking testcases and plans
to requirements, that work with git.

We will need software for this (as well as workflow software) and it
is much easier to integrate this into mantis/bugzilla. They are both
so much more functional than Git.

both mantis and bugzilla have full two way functionality with git.

> 
> What's being talked about here are two separate things. Jenkins is
> a continuous integration system. It can be configured to run the
> suite of unit tests, regression tests, and any kind of automated
> functional tests for every commit on github and every pull
> request.

well 3 but okay.  Jenkins integrates both with mantis (and therefore a
testsuite, etc) and with git. I do not see why anything should be any
different.  again I am not trying to change any current process, just
develop some new ones.

> 
> Github is our issue tracker. Github, and only github, is where new
> issues should be reported (unless it's security related, in which
> case an email should be sent to the core devs directly).

You will only ever receive bug reports via git. How they are entered
should not be of concern.

There will be no space in mantis/zilla for bugs that are not related
to testcases.

> 
> Certainly developers should be responsible for making sure that
> regression tests for bugs they fix make it either into the unit
> tests or Matt's functional test repository. QA should hold them
> accountable for that (re-opening tickets for bugs that have been
> fixed but without regression tests).

I feel very strongly that developers should not do regression testing
or any signoff testing on their own code. QA should do the testing. I
am 50/50 if they should write the testcases. the QA process should
make things easier for the dev team, not generate more work for them.

> 
> The other thing we're talking about is coordinated release
> testing--getting release candidates in the hands of actual users
> and making sure that issues are reported. This is something that
> can't be automated as the point really is to pick up on things that
> the testing suite missed. You sound more qualified than me for
> coming up with a process, but in the end discovered issues should
> be reported to github, the final repository of issues that hold up
> Gavin from doing a release.

All the core development team will still use git. the extra software
is needed by test.

(And the third point was coming up with a suite of tests for 3rd party
developers to test their interoperability - this will having nothing
to do with git, or mantis. But the solution should be compatible with
mantis/zilla)

> 
> Just my 0.002BTC Mark
> 
> On Wed, Sep 26, 2012 at 6:22 AM, steve <steve@mistfpga.net> wrote:
>> 
>> I think you might be misunderstanding a little. I am not trying
>> to replace the current system, I need to make sure that what I do
>> will be compatible with it (seamlessly so for the developer). I
>> do not want this to generate extra work for the development
>> team.
>> 
>> However testing is a lot more than just bug reporting, dont get
>> me wrong bug reports are important, but so is running a testcase
>> and that testcase passing, especially if that testcase is linked
>> to the proof of a requirement. I am trying to develop a qa
>> environment that is conducive to testing and will allow the
>> testers to shine in all their glory :) and we need different
>> tools and methodologies.
>> 
>> Git is too developer centric to be useful for organising testing.
>> - however there is a large amount of software that is compatible
>> with git, so the core development team only ever need to work
>> with git.
>> 
>> The linking between a bug, the requirement, the fix, the retest,
>> and updating of regression testplan is vital. So is the ability
>> to organise testing campaigns and assigning tests, work units and
>> test relevant docs/scripts/ideas, etc.
>> 
>> I hope this clears things up a bit?
>> 
>> Cheers,
>> 
>> steve
>> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYz8CAAoJEFvEB9dQFvtQjkMH/Apa95IRh21mfNIuyK8kOSdt
55tLoT9a6DFyF1IPTgjHQlPN/A0JCPy/p2rIEEL7XzWpCMu1zU8BzBNmJsxGAZJG
C0ue1eDEywKNFEMPTgQdebC2MbNSfUBA6lGJ5vijQlcXKoIuiV/LS7IMYh57T4u1
6Tc/SGypGe8kBLuFTihmIGH5uFS6arNGlcGgh+HRn+O4jKiAcw06lIoKh7S9Rj5e
bmkimvOfproCIZeNQfSJH1BfYZaVVsJ1ouVI7ch6ytFpKsZ622zYF0Iq3042kEEp
Fyqh9pDDNTJ/dwbyFpTx0WaxZySdZfZmQOCxFCAeLaCpop/nKeUnW5fy3i0sYno=
=rfHO
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Gavin Andresen]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

There are test cases that can be automated. That's Jenkins, and those will
be run automagically.

Then there are tests that cannot be automated; things like "Does the GUI
look OK on all of the platforms that we support (Windows XP/2000/Vista/7/8,
Ubuntu/Debian blah with window managers foo and bar, OSX 10.5/6/7/8)."

Thanks to Matt, we're doing great with automated functional test cases (can
always do better, of course).

We're failing on simple, boring stuff like making sure we actually run on
all of the platforms that we say we run on BEFORE final release. That is
where I think a QA team can add a lot of value.

Steve: I'm worried you're over-designing The Process. A release acceptance
test plan could be nothing more than a step-by-step checklist on a wiki
page, Google Doc, or Drobox shared folder...

-- 
--
Gavin Andresen

[-- Attachment #2: Type: text/html, Size: 1008 bytes --]

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Gavin,

Sorry for the delayed response, I wanted to take a couple of days to
reflect on your email.

On 26/09/2012 19:09, Gavin Andresen wrote:


And their are other methods too.



The GUI::Test package for perl will allow this to be greatly
automated. (I have done this before on the localisation of photoshop.)



this why we need detailed testscripts and plans.  so we know what has
and hasnt been done. The more boring the task the more work that needs
to go into testcase development.  This is the area I see as my
greatest failing last time.  I have a large number of virtual machines
and should have at least this work.  But we need very detailed
testcases.  with decent testplans just downloading the software,
syncing the block chain, syncing an existing wallet, rescanning the
blockchain and verifying the balance would cover a large number of
tests.  The idea behind having lots of very specific testcases is you
get to see what tests have not been run.



I understand your concern, however I have taken a couple of days to
reflect on this and I still strongly feel that in order to make sure
that this sticks, and is still useful in 1 years time we need to lay
proper foundations. Those foundations are not word documents,
spreadsheets, etc.  they are selecting the right tools for the job.

We can gain so much benefit from using 3rd party software.
(bettermeans would rock if it wasnt rotting)

I am sure you could do your coding work just using vi, but an sdk
makes it much easier and allows you to work in a more productive manner.

I have had a couple of off list emails with some testers and they also
feel that it is very important to make sure we have a sound foundation
(mantis is so much more than just a bug reporting tool, I see the bug
reporting functionality as secondary to the main test run
functionality - but it doesnt have to be mantis based, we do need
workflow and testcase software though - and proper software for this
is much better than just a massive google doc.) however I am checking
out some other software that has been recommended.  It will be very
hard to change 'the process' once we have something we are used too
(just look at the current resistance) I promise nothing will change
for the dev team.  But test does need other tools, and processes.

If you feel that strongly that I am going about this the wrong way, I
am happy to step back and let someone else sort it out (I will still
do all the testing I possibly can). I would feel that this would be a
real shame and we have the chance to setup requirements to
functionality to tests all with traceability. why not do it right from
the start?

I will open up my vps' somepoint over the next few days and you can
see what I mean. I will setup a fake git project, and sort out the
interactions.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQZz1pAAoJEFvEB9dQFvtQRLkIAJtPCkW1R9vmMPY9u4o+ET1t
w4pV/+W2PXo2p86HnljCIPLV/cua/1I/EJp7XR7s145Nj4KZUbzHGhvUUmwDOHW2
TGvJs+HO1bjsJfh4pWEb6PXcW3TguZxZSt5/rBAAI/5BeomSuRcZOdoV87D1xnK8
TSlgaseWrJcpKLO30/FQA3QnH/bjJ4OBmtHp8WaOtSnfww9Zbb5VYca37O15c2U4
2d0RUunDg1w2kRbkKjztxr3YasSOX+07Uvj4d5Lw7zgA0U93krNWVT1Ypo94dNJ7
6SyKi30UuqDdJ9XxZrMB/LBVNGOLlIBNWL++ocu5GFnOn9pnw57ZMBZM5g6YDpo=
=ekQ/
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Arklan Uth Oslin]

[-- Attachment #1: Type: text/plain, Size: 4787 bytes --]

Hi guys.

So, as I mentioned on the bitcointalk.org forums thread about the
foundation, I want to get involved in the QA side of bitcoin
development. I've done functional testing in the video game industry for
years. I've read all the messages in this thread, but I'm left unclear how
I can most effectively and quickly being helping out. Could I get a bit of
a directional nudge?

Arklan

----------
As long as there is light, the darkness holds no fear. And yet, even in the
deepest black, there is life. - Arklan Uth Oslin

I want to leave this world the same way I came into it: backwards and on
fire. - Arklan Uth Oslin



On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Gavin,
>
> Sorry for the delayed response, I wanted to take a couple of days to
> reflect on your email.
>
> On 26/09/2012 19:09, Gavin Andresen wrote:
>
>
> And their are other methods too.
>
>
>
> The GUI::Test package for perl will allow this to be greatly
> automated. (I have done this before on the localisation of photoshop.)
>
>
>
> this why we need detailed testscripts and plans.  so we know what has
> and hasnt been done. The more boring the task the more work that needs
> to go into testcase development.  This is the area I see as my
> greatest failing last time.  I have a large number of virtual machines
> and should have at least this work.  But we need very detailed
> testcases.  with decent testplans just downloading the software,
> syncing the block chain, syncing an existing wallet, rescanning the
> blockchain and verifying the balance would cover a large number of
> tests.  The idea behind having lots of very specific testcases is you
> get to see what tests have not been run.
>
>
>
> I understand your concern, however I have taken a couple of days to
> reflect on this and I still strongly feel that in order to make sure
> that this sticks, and is still useful in 1 years time we need to lay
> proper foundations. Those foundations are not word documents,
> spreadsheets, etc.  they are selecting the right tools for the job.
>
> We can gain so much benefit from using 3rd party software.
> (bettermeans would rock if it wasnt rotting)
>
> I am sure you could do your coding work just using vi, but an sdk
> makes it much easier and allows you to work in a more productive manner.
>
> I have had a couple of off list emails with some testers and they also
> feel that it is very important to make sure we have a sound foundation
> (mantis is so much more than just a bug reporting tool, I see the bug
> reporting functionality as secondary to the main test run
> functionality - but it doesnt have to be mantis based, we do need
> workflow and testcase software though - and proper software for this
> is much better than just a massive google doc.) however I am checking
> out some other software that has been recommended.  It will be very
> hard to change 'the process' once we have something we are used too
> (just look at the current resistance) I promise nothing will change
> for the dev team.  But test does need other tools, and processes.
>
> If you feel that strongly that I am going about this the wrong way, I
> am happy to step back and let someone else sort it out (I will still
> do all the testing I possibly can). I would feel that this would be a
> real shame and we have the chance to setup requirements to
> functionality to tests all with traceability. why not do it right from
> the start?
>
> I will open up my vps' somepoint over the next few days and you can
> see what I mean. I will setup a fake git project, and sort out the
> interactions.
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJQZz1pAAoJEFvEB9dQFvtQRLkIAJtPCkW1R9vmMPY9u4o+ET1t
> w4pV/+W2PXo2p86HnljCIPLV/cua/1I/EJp7XR7s145Nj4KZUbzHGhvUUmwDOHW2
> TGvJs+HO1bjsJfh4pWEb6PXcW3TguZxZSt5/rBAAI/5BeomSuRcZOdoV87D1xnK8
> TSlgaseWrJcpKLO30/FQA3QnH/bjJ4OBmtHp8WaOtSnfww9Zbb5VYca37O15c2U4
> 2d0RUunDg1w2kRbkKjztxr3YasSOX+07Uvj4d5Lw7zgA0U93krNWVT1Ypo94dNJ7
> 6SyKi30UuqDdJ9XxZrMB/LBVNGOLlIBNWL++ocu5GFnOn9pnw57ZMBZM5g6YDpo=
> =ekQ/
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> How fast is your code?
> 3 out of 4 devs don\\\'t know how their code performs in production.
> Find out how slow your code is with AppDynamics Lite.
> http://ad.doubleclick.net/clk;262219672;13503038;z?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 5896 bytes --]

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/10/2012 14:52, Arklan Uth Oslin wrote:
> Hi guys.
> 
> So, as I mentioned on the bitcointalk.org forums thread about the 
> foundation, I want to get involved in the QA side of bitcoin 
> development. I've done functional testing in the video game
> industry for years.

Nice one, I worked in games for quite a few years. (before getting
into finance then pentesting) there are about 6 keen testers now.
maybe we should get a bitcoin-test mailing list, where we can discuss
stuff without disturbing the dev team.

> I've read all the messages in this thread, but I'm left unclear
> how I can most effectively and quickly being helping out. Could I
> get a bit of a directional nudge?

Great question... for me I feel structure is the most important thing
to sort out first.  However we desperately need detailed testcases for
the release of a new version. - Not too much on the change log stuff,
more on the noddy stuff (as gavin points out below), downloading and
making sure it works on a non dev machine, make sure the wallet isnt
overwritten, etc.) doing games qa I imagine this would be an ideal
place for you to start.  I have a MSDN and TechNet licence so if you
need some reference ms virtual machines I can help you out.

However we need some testcase software.  Please check out what was
done on bettermeans for the stuff I was planning out...

It details everything from recompense and testcases. bettermeans kinda
died a death though...

check out:

Bitcoin over all-
https://secure.bettermeans.com/projects/4180/wiki/Page_index
discussion
https://secure.bettermeans.com/projects/4180/boards

0.7
https://secure.bettermeans.com/projects/4256/boards
and
https://secure.bettermeans.com/projects/4256/wiki

I still have the testcases, but until we get some proper testcase
software I am loathed to publish them in a half arsed format. (they
worked well on bettermeans, then just vanished one day...) what
testcase software are you familiar with?

apart from that, what do you feel you can do for the project? how long
have you been involved in bitcoin?  It may well be worth reading up
all the dev stuff on the wiki so you can get you head around how the
bitcoin protocol is different from the daemon and qt client. What do
you think you can and will enjoy doing? What is your skill set in
regard to networking, crypto and operating systems. (not that you need
any, in any we still want and need you. :)

there really is room for you to do whatever role you want, and as
little or as much as you want - however funding is now a very tricky
issue. so much so that I am not sure I want anything to do with
it(distribution of coin based on work.). - I just paid for some logo
spec work out of my own pocket (for example).  I have some testers i
know irl who are willing to work for coin.

NOTE: This response has nothing to do with the bitcoin foundation.  I
am not a member of the foundation. I do not speak for them or even
probably with them. I am still trying to work out how much qa the
foundation should be responsible for, and/how it is supposed to work.
I think the games cert process would be ideal for this.  This however
this a discussion that probably wont have my involvement.  (personally
I believe that the foundation should publish requirements with example
code and testcases for each aspect of the reference client. (on
reference platforms - I do not expect many to agree with this though)

As a side note, what happens to the donations to the bitcoin testing
project? do they get moved over to the foundation? this question is
bigger than this email. as far as I know they are all on an address
Gavin holds. Actually I would like to be involved in any discussions
that would impact QA, does this mean I need to join the foundation or
just go lone wolf?

tbh I dont really understand foundations.  I always thought they were
just a tax dodge.

Sorry for the long message. :)

> 
> Arklan
> 
> ---------- As long as there is light, the darkness holds no fear.
> And yet, even in the deepest black, there is life. - Arklan Uth
> Oslin
> 
> I want to leave this world the same way I came into it: backwards
> and on fire. - Arklan Uth Oslin
> 
> 
> 
> On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga.net>
> wrote:
> 
> Hi Gavin,
> 
> Sorry for the delayed response, I wanted to take a couple of days
> to reflect on your email.
> 
> On 26/09/2012 19:09, Gavin Andresen wrote:
> 
> 
> And their are other methods too.
> 
> 
> 
> The GUI::Test package for perl will allow this to be greatly 
> automated. (I have done this before on the localisation of
> photoshop.)
> 
> 
> 
> this why we need detailed testscripts and plans.  so we know what
> has and hasnt been done. The more boring the task the more work
> that needs to go into testcase development.  This is the area I see
> as my greatest failing last time.  I have a large number of virtual
> machines and should have at least this work.  But we need very
> detailed testcases.  with decent testplans just downloading the
> software, syncing the block chain, syncing an existing wallet,
> rescanning the blockchain and verifying the balance would cover a
> large number of tests.  The idea behind having lots of very
> specific testcases is you get to see what tests have not been run.
> 
> 
> 
> I understand your concern, however I have taken a couple of days
> to reflect on this and I still strongly feel that in order to make
> sure that this sticks, and is still useful in 1 years time we need
> to lay proper foundations. Those foundations are not word
> documents, spreadsheets, etc.  they are selecting the right tools
> for the job.
> 
> We can gain so much benefit from using 3rd party software. 
> (bettermeans would rock if it wasnt rotting)
> 
> I am sure you could do your coding work just using vi, but an sdk 
> makes it much easier and allows you to work in a more productive
> manner.
> 
> I have had a couple of off list emails with some testers and they
> also feel that it is very important to make sure we have a sound
> foundation (mantis is so much more than just a bug reporting tool,
> I see the bug reporting functionality as secondary to the main test
> run functionality - but it doesnt have to be mantis based, we do
> need workflow and testcase software though - and proper software
> for this is much better than just a massive google doc.) however I
> am checking out some other software that has been recommended.  It
> will be very hard to change 'the process' once we have something we
> are used too (just look at the current resistance) I promise
> nothing will change for the dev team.  But test does need other
> tools, and processes.
> 
> If you feel that strongly that I am going about this the wrong way,
> I am happy to step back and let someone else sort it out (I will
> still do all the testing I possibly can). I would feel that this
> would be a real shame and we have the chance to setup requirements
> to functionality to tests all with traceability. why not do it
> right from the start?
> 
> I will open up my vps' somepoint over the next few days and you
> can see what I mean. I will setup a fake git project, and sort out
> the interactions.
> 
> 
>> 
>> 
>> ------------------------------------------------------------------------------
>>
>> 
How fast is your code?
>> 3 out of 4 devs don\\\'t know how their code performs in
>> production. Find out how slow your code is with AppDynamics
>> Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? 
>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html 
>> _______________________________________________ 
>> Bitcoin-development mailing list 
>> Bitcoin-development@lists.sourceforge.net 
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>> 
> 
> 
> 
> ------------------------------------------------------------------------------
>
> 
Got visibility?
> Most devs has no idea what their production app looks like. Find
> out how fast your code is with AppDynamics Lite. 
> http://ad.doubleclick.net/clk;262219671;13503038;y? 
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> 
> 
> 
> _______________________________________________ Bitcoin-development
> mailing list Bitcoin-development@lists.sourceforge.net 
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQaaieAAoJEFvEB9dQFvtQUi0H/3Eh72DqxwBt6AeNos/hJNqQ
ZowMNFRupJQM301EJ7SPQmcnVuc3RF2Jw//ckpAqdpkqhHCgGO9HX/q+Ic2A9erQ
CfKbUOwQgqKuLQTZ8eT5UMh969Uw8PMazo711d2I23etU0VhSK/MvjB5IiYIq5fB
RJdCEluwvACaiftqfm4RA9HXXJGMnsZtlSWjVQCkg2FyYZKfpctEFdGCg6R8IaP7
ypyglR7TfygcjhUMHeWkQQCP2m2gBOHpx9H99uNBosF+6pLBhCiGUrwVkl8HUiiN
4YfhsvoRQ0BWNgvO9wH1H4EyyZtJsE7S926BNkhd4uewWGV/qe8gML+uJRstPDg=
=L66i
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Peter Vessenes]

[-- Attachment #1: Type: text/plain, Size: 10726 bytes --]

I'm a big proponent of a testing project.

I think if one could self organize that Gavin and team wanted to bless we
could put up some BTC as bounties or funding. We won't have our heads
around the foundation budget for a few more weeks, but self-organization is
often slower than budgeting. :)

This is just my opinion, but I would like very, very much to move the
current specification into unit tests so that anyone could validate their
alternate bitcoin implementation. This is a lot of work, some of which has
been done, much of which hasn't.

So, my two cents, plus an offer to bring it up at our next budgeting
meeting.

Peter


On Mon, Oct 1, 2012 at 7:28 AM, steve <steve@mistfpga.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 01/10/2012 14:52, Arklan Uth Oslin wrote:
> > Hi guys.
> >
> > So, as I mentioned on the bitcointalk.org forums thread about the
> > foundation, I want to get involved in the QA side of bitcoin
> > development. I've done functional testing in the video game
> > industry for years.
>
> Nice one, I worked in games for quite a few years. (before getting
> into finance then pentesting) there are about 6 keen testers now.
> maybe we should get a bitcoin-test mailing list, where we can discuss
> stuff without disturbing the dev team.
>
> > I've read all the messages in this thread, but I'm left unclear
> > how I can most effectively and quickly being helping out. Could I
> > get a bit of a directional nudge?
>
> Great question... for me I feel structure is the most important thing
> to sort out first.  However we desperately need detailed testcases for
> the release of a new version. - Not too much on the change log stuff,
> more on the noddy stuff (as gavin points out below), downloading and
> making sure it works on a non dev machine, make sure the wallet isnt
> overwritten, etc.) doing games qa I imagine this would be an ideal
> place for you to start.  I have a MSDN and TechNet licence so if you
> need some reference ms virtual machines I can help you out.
>
> However we need some testcase software.  Please check out what was
> done on bettermeans for the stuff I was planning out...
>
> It details everything from recompense and testcases. bettermeans kinda
> died a death though...
>
> check out:
>
> Bitcoin over all-
> https://secure.bettermeans.com/projects/4180/wiki/Page_index
> discussion
> https://secure.bettermeans.com/projects/4180/boards
>
> 0.7
> https://secure.bettermeans.com/projects/4256/boards
> and
> https://secure.bettermeans.com/projects/4256/wiki
>
> I still have the testcases, but until we get some proper testcase
> software I am loathed to publish them in a half arsed format. (they
> worked well on bettermeans, then just vanished one day...) what
> testcase software are you familiar with?
>
> apart from that, what do you feel you can do for the project? how long
> have you been involved in bitcoin?  It may well be worth reading up
> all the dev stuff on the wiki so you can get you head around how the
> bitcoin protocol is different from the daemon and qt client. What do
> you think you can and will enjoy doing? What is your skill set in
> regard to networking, crypto and operating systems. (not that you need
> any, in any we still want and need you. :)
>
> there really is room for you to do whatever role you want, and as
> little or as much as you want - however funding is now a very tricky
> issue. so much so that I am not sure I want anything to do with
> it(distribution of coin based on work.). - I just paid for some logo
> spec work out of my own pocket (for example).  I have some testers i
> know irl who are willing to work for coin.
>
> NOTE: This response has nothing to do with the bitcoin foundation.  I
> am not a member of the foundation. I do not speak for them or even
> probably with them. I am still trying to work out how much qa the
> foundation should be responsible for, and/how it is supposed to work.
> I think the games cert process would be ideal for this.  This however
> this a discussion that probably wont have my involvement.  (personally
> I believe that the foundation should publish requirements with example
> code and testcases for each aspect of the reference client. (on
> reference platforms - I do not expect many to agree with this though)
>
> As a side note, what happens to the donations to the bitcoin testing
> project? do they get moved over to the foundation? this question is
> bigger than this email. as far as I know they are all on an address
> Gavin holds. Actually I would like to be involved in any discussions
> that would impact QA, does this mean I need to join the foundation or
> just go lone wolf?
>
> tbh I dont really understand foundations.  I always thought they were
> just a tax dodge.
>
> Sorry for the long message. :)
>
> >
> > Arklan
> >
> > ---------- As long as there is light, the darkness holds no fear.
> > And yet, even in the deepest black, there is life. - Arklan Uth
> > Oslin
> >
> > I want to leave this world the same way I came into it: backwards
> > and on fire. - Arklan Uth Oslin
> >
> >
> >
> > On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga.net>
> > wrote:
> >
> > Hi Gavin,
> >
> > Sorry for the delayed response, I wanted to take a couple of days
> > to reflect on your email.
> >
> > On 26/09/2012 19:09, Gavin Andresen wrote:
> >
> >
> > And their are other methods too.
> >
> >
> >
> > The GUI::Test package for perl will allow this to be greatly
> > automated. (I have done this before on the localisation of
> > photoshop.)
> >
> >
> >
> > this why we need detailed testscripts and plans.  so we know what
> > has and hasnt been done. The more boring the task the more work
> > that needs to go into testcase development.  This is the area I see
> > as my greatest failing last time.  I have a large number of virtual
> > machines and should have at least this work.  But we need very
> > detailed testcases.  with decent testplans just downloading the
> > software, syncing the block chain, syncing an existing wallet,
> > rescanning the blockchain and verifying the balance would cover a
> > large number of tests.  The idea behind having lots of very
> > specific testcases is you get to see what tests have not been run.
> >
> >
> >
> > I understand your concern, however I have taken a couple of days
> > to reflect on this and I still strongly feel that in order to make
> > sure that this sticks, and is still useful in 1 years time we need
> > to lay proper foundations. Those foundations are not word
> > documents, spreadsheets, etc.  they are selecting the right tools
> > for the job.
> >
> > We can gain so much benefit from using 3rd party software.
> > (bettermeans would rock if it wasnt rotting)
> >
> > I am sure you could do your coding work just using vi, but an sdk
> > makes it much easier and allows you to work in a more productive
> > manner.
> >
> > I have had a couple of off list emails with some testers and they
> > also feel that it is very important to make sure we have a sound
> > foundation (mantis is so much more than just a bug reporting tool,
> > I see the bug reporting functionality as secondary to the main test
> > run functionality - but it doesnt have to be mantis based, we do
> > need workflow and testcase software though - and proper software
> > for this is much better than just a massive google doc.) however I
> > am checking out some other software that has been recommended.  It
> > will be very hard to change 'the process' once we have something we
> > are used too (just look at the current resistance) I promise
> > nothing will change for the dev team.  But test does need other
> > tools, and processes.
> >
> > If you feel that strongly that I am going about this the wrong way,
> > I am happy to step back and let someone else sort it out (I will
> > still do all the testing I possibly can). I would feel that this
> > would be a real shame and we have the chance to setup requirements
> > to functionality to tests all with traceability. why not do it
> > right from the start?
> >
> > I will open up my vps' somepoint over the next few days and you
> > can see what I mean. I will setup a fake git project, and sort out
> > the interactions.
> >
> >
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >>
> >>
> How fast is your code?
> >> 3 out of 4 devs don\\\'t know how their code performs in
> >> production. Find out how slow your code is with AppDynamics
> >> Lite. http://ad.doubleclick.net/clk;262219672;13503038;z?
> >> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> >> _______________________________________________
> >> Bitcoin-development mailing list
> >> Bitcoin-development@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> >>
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> >
> >
> Got visibility?
> > Most devs has no idea what their production app looks like. Find
> > out how fast your code is with AppDynamics Lite.
> > http://ad.doubleclick.net/clk;262219671;13503038;y?
> > http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> >
> >
> >
> > _______________________________________________ Bitcoin-development
> > mailing list Bitcoin-development@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJQaaieAAoJEFvEB9dQFvtQUi0H/3Eh72DqxwBt6AeNos/hJNqQ
> ZowMNFRupJQM301EJ7SPQmcnVuc3RF2Jw//ckpAqdpkqhHCgGO9HX/q+Ic2A9erQ
> CfKbUOwQgqKuLQTZ8eT5UMh969Uw8PMazo711d2I23etU0VhSK/MvjB5IiYIq5fB
> RJdCEluwvACaiftqfm4RA9HXXJGMnsZtlSWjVQCkg2FyYZKfpctEFdGCg6R8IaP7
> ypyglR7TfygcjhUMHeWkQQCP2m2gBOHpx9H99uNBosF+6pLBhCiGUrwVkl8HUiiN
> 4YfhsvoRQ0BWNgvO9wH1H4EyyZtJsE7S926BNkhd4uewWGV/qe8gML+uJRstPDg=
> =L66i
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> Got visibility?
> Most devs has no idea what their production app looks like.
> Find out how fast your code is with AppDynamics Lite.
> http://ad.doubleclick.net/clk;262219671;13503038;y?
> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>



-- 
------------------------------

[image: CoinLab Logo]PETER VESSENES
CEO

*peter@coinlab.com * /  206.486.6856  / SKYPE: vessenes
811 FIRST AVENUE  /  SUITE 480  /  SEATTLE, WA 98104

[-- Attachment #2: Type: text/html, Size: 14926 bytes --]

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/10/2012 17:52, Peter Vessenes wrote:
> I'm a big proponent of a testing project.

I am very happy to hear this, however, your actual words are slightly
evasive. I do not expect you to be up to speed on this. Gavin started
a project called 'the bitcoin testing project' This project solicited
donations, about 80 coins last time I checked.  However these 80 odd
coins were donated to 'the bitcoin testing project' This would seem to
be an official bitcoin (both protocol and client) testing project.  I
signed up to work on this, and organise as much as i could of this.
for various reasons I did not manage to do the testing I wanted to on
0.7 i over committed myself.

Are the donations solicited for the 'bitcoin testing project) funds
going to be given to 'the foundation'?

Does the foundation support 'the bitcoin testing project'? does the
foundation have any involvement with 'the bitcoin testing project'?

> 
> I think if one could self organize that Gavin and team wanted to
> bless we could put up some BTC as bounties or funding. We won't
> have our heads around the foundation budget for a few more weeks,
> but self-organization is often slower than budgeting. :)

Im ready to go, more or less.  Please check out the links in my
previous emails. I have over 400 testcases (8 platforms * 50 release
tests) - Also I am not sure what you mean by bless, I take it that is
a euphemism for pay?

I have tried my hardest to get bettermeans to work, but it doesnt.  It
does show quite a lot of work that I have done though. If you were to
say to me, 'steve, by monday we need end to end, requirements based
testing' It would be done. (I have already spent over 4 months on
this)  Leaderless leadership is something I am having a hard time
with, bettermeans is excellent at this.  But I have found very little
in regards to voting and polling that integrates with the project in
an effortless way like bettermeans.

I understand that the budget from the foundation is something that
needs to be worked on and organised.  I offer my services in this area
(qa only).  I would be happy to submit my cv and refs for this, if
required.

I am now feeling frustrated and useless.  has my last 4 months of work
been for nothing? it feels like it.  I know I bang on about processes
but they are sorted, you can only attract talent like Arklan if he has
a process to follow. i feel like a broken record.

tl;dr version
1 - Will donations to the 'bitcoin testing project' as started by
gavin going to be given to the foundation?
2 - Is the work bill hees and myself going to be binned?
3 - I feel like I have the knowledge and drive to push this, but I
cant do it on my own.
4 - Is bill or I entitled to any of the cash raised for 'the bitcoin
testing project'
5 - Do I have to join the foundation to have a say in how the project
(testing) is done?
6 - sorry for being so mercenary, but am I going to receive any coin
for work I have done?
7 - It really probably is the time for a bitcoin-test list to appear.
 Is there anything I can do to make this happen?

> 
> This is just my opinion, but I would like very, very much to move
> the current specification into unit tests so that anyone could
> validate their alternate bitcoin implementation. This is a lot of
> work, some of which has been done, much of which hasn't.

have a look at the stuff in bettermeans.  I personally think we can go
a step further and publish guidelines (similar to RFC's and all the
tests that we would do against a ref client)

But I dont want to waste any more time on stuff that is going to be
ignored, life is short.

> 
> So, my two cents, plus an offer to bring it up at our next
> budgeting meeting.

I accept that offer. and I really appreciate it.  I have some more
questions I would like you to ask in regards to QA. (Gavin and I
skyped about this a while ago and we didnt really come to a
resolution, weworked out the problems though ;) )

I have an exceptionally detailed qa process (based off the game
certification process) - but I have gone on about this at length in
previous messages.

I thank you for your email and your involvement with this, but do you
think we are closer to getting stuff tested? call my bluff... Not one
person has asked for login details to my proposals - and i even have a
bugzilla version now.

I need to sleep.  sorry if i rambled.

nite nite,

steve

> 
> Peter
> 
> 
> On Mon, Oct 1, 2012 at 7:28 AM, steve <steve@mistfpga.net> wrote:
> 
> On 01/10/2012 14:52, Arklan Uth Oslin wrote:
>>>> Hi guys.
>>>> 
>>>> So, as I mentioned on the bitcointalk.org forums thread about
>>>> the foundation, I want to get involved in the QA side of
>>>> bitcoin development. I've done functional testing in the
>>>> video game industry for years.
> 
> Nice one, I worked in games for quite a few years. (before getting 
> into finance then pentesting) there are about 6 keen testers now. 
> maybe we should get a bitcoin-test mailing list, where we can
> discuss stuff without disturbing the dev team.
> 
>>>> I've read all the messages in this thread, but I'm left
>>>> unclear how I can most effectively and quickly being helping
>>>> out. Could I get a bit of a directional nudge?
> 
> Great question... for me I feel structure is the most important
> thing to sort out first.  However we desperately need detailed
> testcases for the release of a new version. - Not too much on the
> change log stuff, more on the noddy stuff (as gavin points out
> below), downloading and making sure it works on a non dev machine,
> make sure the wallet isnt overwritten, etc.) doing games qa I
> imagine this would be an ideal place for you to start.  I have a
> MSDN and TechNet licence so if you need some reference ms virtual
> machines I can help you out.
> 
> However we need some testcase software.  Please check out what was 
> done on bettermeans for the stuff I was planning out...
> 
> It details everything from recompense and testcases. bettermeans
> kinda died a death though...
> 
> check out:
> 
> Bitcoin over all- 
> https://secure.bettermeans.com/projects/4180/wiki/Page_index 
> discussion https://secure.bettermeans.com/projects/4180/boards
> 
> 0.7 https://secure.bettermeans.com/projects/4256/boards and 
> https://secure.bettermeans.com/projects/4256/wiki
> 
> I still have the testcases, but until we get some proper testcase 
> software I am loathed to publish them in a half arsed format.
> (they worked well on bettermeans, then just vanished one day...)
> what testcase software are you familiar with?
> 
> apart from that, what do you feel you can do for the project? how
> long have you been involved in bitcoin?  It may well be worth
> reading up all the dev stuff on the wiki so you can get you head
> around how the bitcoin protocol is different from the daemon and qt
> client. What do you think you can and will enjoy doing? What is
> your skill set in regard to networking, crypto and operating
> systems. (not that you need any, in any we still want and need you.
> :)
> 
> there really is room for you to do whatever role you want, and as 
> little or as much as you want - however funding is now a very
> tricky issue. so much so that I am not sure I want anything to do
> with it(distribution of coin based on work.). - I just paid for
> some logo spec work out of my own pocket (for example).  I have
> some testers i know irl who are willing to work for coin.
> 
> NOTE: This response has nothing to do with the bitcoin foundation.
> I am not a member of the foundation. I do not speak for them or
> even probably with them. I am still trying to work out how much qa
> the foundation should be responsible for, and/how it is supposed to
> work. I think the games cert process would be ideal for this.  This
> however this a discussion that probably wont have my involvement.
> (personally I believe that the foundation should publish
> requirements with example code and testcases for each aspect of the
> reference client. (on reference platforms - I do not expect many to
> agree with this though)
> 
> As a side note, what happens to the donations to the bitcoin
> testing project? do they get moved over to the foundation? this
> question is bigger than this email. as far as I know they are all
> on an address Gavin holds. Actually I would like to be involved in
> any discussions that would impact QA, does this mean I need to join
> the foundation or just go lone wolf?
> 
> tbh I dont really understand foundations.  I always thought they
> were just a tax dodge.
> 
> Sorry for the long message. :)
> 
>>>> 
>>>> Arklan
>>>> 
>>>> ---------- As long as there is light, the darkness holds no
>>>> fear. And yet, even in the deepest black, there is life. -
>>>> Arklan Uth Oslin
>>>> 
>>>> I want to leave this world the same way I came into it:
>>>> backwards and on fire. - Arklan Uth Oslin
>>>> 
>>>> 
>>>> 
>>>> On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga.net> 
>>>> wrote:
>>>> 
>>>> Hi Gavin,
>>>> 
>>>> Sorry for the delayed response, I wanted to take a couple of
>>>> days to reflect on your email.
>>>> 
>>>> On 26/09/2012 19:09, Gavin Andresen wrote:
>>>> 
>>>> 
>>>> And their are other methods too.
>>>> 
>>>> 
>>>> 
>>>> The GUI::Test package for perl will allow this to be greatly 
>>>> automated. (I have done this before on the localisation of 
>>>> photoshop.)
>>>> 
>>>> 
>>>> 
>>>> this why we need detailed testscripts and plans.  so we know
>>>> what has and hasnt been done. The more boring the task the
>>>> more work that needs to go into testcase development.  This
>>>> is the area I see as my greatest failing last time.  I have a
>>>> large number of virtual machines and should have at least
>>>> this work.  But we need very detailed testcases.  with decent
>>>> testplans just downloading the software, syncing the block
>>>> chain, syncing an existing wallet, rescanning the blockchain
>>>> and verifying the balance would cover a large number of
>>>> tests.  The idea behind having lots of very specific
>>>> testcases is you get to see what tests have not been run.
>>>> 
>>>> 
>>>> 
>>>> I understand your concern, however I have taken a couple of
>>>> days to reflect on this and I still strongly feel that in
>>>> order to make sure that this sticks, and is still useful in 1
>>>> years time we need to lay proper foundations. Those
>>>> foundations are not word documents, spreadsheets, etc.  they
>>>> are selecting the right tools for the job.
>>>> 
>>>> We can gain so much benefit from using 3rd party software. 
>>>> (bettermeans would rock if it wasnt rotting)
>>>> 
>>>> I am sure you could do your coding work just using vi, but an
>>>> sdk makes it much easier and allows you to work in a more
>>>> productive manner.
>>>> 
>>>> I have had a couple of off list emails with some testers and
>>>> they also feel that it is very important to make sure we have
>>>> a sound foundation (mantis is so much more than just a bug
>>>> reporting tool, I see the bug reporting functionality as
>>>> secondary to the main test run functionality - but it doesnt
>>>> have to be mantis based, we do need workflow and testcase
>>>> software though - and proper software for this is much better
>>>> than just a massive google doc.) however I am checking out
>>>> some other software that has been recommended.  It will be
>>>> very hard to change 'the process' once we have something we 
>>>> are used too (just look at the current resistance) I promise 
>>>> nothing will change for the dev team.  But test does need
>>>> other tools, and processes.
>>>> 
>>>> If you feel that strongly that I am going about this the
>>>> wrong way, I am happy to step back and let someone else sort
>>>> it out (I will still do all the testing I possibly can). I
>>>> would feel that this would be a real shame and we have the
>>>> chance to setup requirements to functionality to tests all
>>>> with traceability. why not do it right from the start?
>>>> 
>>>> I will open up my vps' somepoint over the next few days and
>>>> you can see what I mean. I will setup a fake git project, and
>>>> sort out the interactions.
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> 
> ------------------------------------------------------------------------------
>>>>>
>>>>>
>
> 
How fast is your code?
>>>>> 3 out of 4 devs don\\\'t know how their code performs in 
>>>>> production. Find out how slow your code is with
>>>>> AppDynamics Lite.
>>>>> http://ad.doubleclick.net/clk;262219672;13503038;z? 
>>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>>>
>>>>> 
_______________________________________________
>>>>> Bitcoin-development mailing list 
>>>>> Bitcoin-development@lists.sourceforge.net 
>>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>>>
>>>>
>>>>
>>>>
>>>>
>
>>>>> 
-
------------------------------------------------------------------------------
>>>> 
>>>> 
> Got visibility?
>>>> Most devs has no idea what their production app looks like.
>>>> Find out how fast your code is with AppDynamics Lite. 
>>>> http://ad.doubleclick.net/clk;262219671;13503038;y? 
>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Bitcoin-development mailing list
>>>> Bitcoin-development@lists.sourceforge.net 
>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>>
>
>>>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>>
>> 
Got visibility?
>> Most devs has no idea what their production app looks like. Find
>> out how fast your code is with AppDynamics Lite. 
>> http://ad.doubleclick.net/clk;262219671;13503038;y? 
>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html 
>> _______________________________________________ 
>> Bitcoin-development mailing list 
>> Bitcoin-development@lists.sourceforge.net 
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>> 
> 
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQa5GXAAoJEFvEB9dQFvtQUR4H/3o0aOyvF6u4Wi5UUxMtGWXD
Bduuag9jNY/GACFshYs365wWqJcK96GXc/svOx0l/TTv10tw9BboaMpKY2x9gWri
7DJ+mym1AorgHBlZUgZJy3kQrb5yUCR7MiYU0O28QAqK5aVoklBa6cGQJqQVRa2j
vm9UrKCOeMbHiNMjagLP3s1tcQxYJUwm8hEuALWjyqsDQ+Rk7j2ndBnfonJLTCDc
rIy6HhmfAGSCaXFzhH35Ydfl2hV6ASaUHtBSKke3bB/oJ/bA5XspmPPhCkjum4Lo
4b28i4/O8BTisAsr8N3FYQ17dmD2fu8/gmQt1Rl9MBfWBHPJ9YVIIittk+WhcI0=
=nER6
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Gregory Maxwell]

On Tue, Oct 2, 2012 at 9:15 PM, steve <steve@mistfpga.net> wrote:
> Im ready to go, more or less.  Please check out the links in my
> previous emails. I have over 400 testcases (8 platforms * 50 release
> tests) - Also I am not sure what you mean by bless, I take it that is
> a euphemism for pay?

Perhaps a bit bluntly here— but since you seem to be rather boldly
insisting on getting paid:

With all of this testing where can I find the issues you've uncovered?
 Searching on your name/email in the issue tracker reports nothing,
likewise I can't find anything in my email (beyond abstract discussion
of testing).

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/10/2012 03:02, Gregory Maxwell wrote:
> On Tue, Oct 2, 2012 at 9:15 PM, steve <steve@mistfpga.net> wrote:
>> Im ready to go, more or less.  Please check out the links in my 
>> previous emails. I have over 400 testcases (8 platforms * 50
>> release tests) - Also I am not sure what you mean by bless, I
>> take it that is a euphemism for pay?
> 
> Perhaps a bit bluntly here— but since you seem to be rather boldly 
> insisting on getting paid:

that is not what i was getting at, nor what i meant at all.I am sorry
it came across this way. Im a bit drunk. (yeah that old excuse) I was
only trying to make an example of the quandary that exists, in what
happens to the current donations. There are far more deserving people
who have a claim on that coin rather than I.

I have stated on more than on occasion (written in public and verbal
with gavin) that I would be willing to work for free. But I would love
to be able to ditch the day job. (however i see that more as services
than mining/testing)

rereading what i wrote yeah, fair play i can see how you read my mail
like that. it wasnt meant to :/ i feel a bit of a turnip. the tone is
not what i meant. it does read pretty mercenary.

> 
> With all of this testing where can I find the issues you've
> uncovered? Searching on your name/email in the issue tracker
> reports nothing, likewise I can't find anything in my email (beyond
> abstract discussion of testing).
> 

This is a different issue. But I see why you have raised it and I
would like to address it. I personally believe you should earn
bitcoins for writing testcases, executing testcases, passing or
failing test cases. in that order (failing a testcase normally
generates a new one therefore encouraging indepth and recorded
testing.). QA should not be judged based solely on bug reports - this
is unfair and will result in race to report bugs.  I have worked on a
few projects that have tried it.

This is one reason we need workflow and testcase software, so we can
measure and compensate people for their work.

As I have stated in previous emails I should have, but for various
excuses did not manage to run a single testcase for 0.7. but I did
setup a frame work, and I know some people did do some good work on
0.7 I believe these people should be compensated from the (limited)
funds the project received... something needs to happen with those coins.

I am sorry you find my emails abstract (and therefore meaningless?)

this is a really confusing time. i dont know if i am doing right from
wrong. I am trying to lead, without leading... i have always preferred
benevolent dictator. I know a number of bitcoin businesses that are
really keen to work with the project, but I have no authority, noone
seems to. Someone should be able to say, nice one, thanks for the
vm's. maybe use $200 worth of coin to get a technet licence, that
gives 10 installs of each ms OS. slipstreaming in the service packs
means you can have whatever era pc you want (and the vps providers
will allow various configuration of vm's...) then even if you do it by
hand you can still do 6 or 7 at once...

as for bug reports, give me 1 week from now (168hrs), lets see what i
can do. (priority, protocol, daemon, qt ref?) but still process is
more important...

I still think that writing and doing testcases should get more coin
than bug reports. but from what I have read, the big bounties will be
paid by the foundation. im not sure if that is true or where i read
it. (protocol level bugs)

it feels like I am wasting peoples time,and I should get back in my
box. so I will.

contact me off list if you want to have a look at the various workflow
stuff i have done.

cheers,

steve
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQa6pAAAoJEFvEB9dQFvtQ0H8H/jUjvzmsp61w1bmDnHR+KmF4
LNu7WwVLrTvrT8AHSNh3mClWvMM3muJJA7NMb2WthAgVe3jtrGimfreAlstDsObL
XNEcGvU6WN1YosH0MkN7hyDl8jnrDFoiH1P5qsMecuZIxwq7Z0vCOHEJ9DPmZilW
R+G8OmoGcpaeWs9VqXR6zR7Uyz69KaDAQpMRE1GTu3zQP9HSSolciy3ESeJRR9Sd
yO7EcCGdQot80rOG/VIZ0wkOmzGGm1thzYzayD6Zn2eW4Hw+ME1en9ksIbXJFpSv
IdgThEm7p5UuBo0jFkbX4Awrf9hfusZSEGWfhZqdASqqkSBnYqLmWF1sLprDRF8=
=Iv87
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Peter Vessenes]

[-- Attachment #1: Type: text/plain, Size: 92 bytes --]

And, finally, when I say "Ditto to above" I mean "I have no idea", not
"nope". Double oops.

[-- Attachment #2: Type: text/html, Size: 128 bytes --]

Re: [Bitcoin-development] Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Peter,

Thank you for your in depth, forthright and professional response. You
have answered my questions.  And you have cleared up a lot of
confusion I had in my mind. (and I read ditto as i dont know) I
appreciate the extra information. :)

This is a top post and there is nothing inline so i snipped it.

cheers,

steve

On 03/10/2012 06:03, Peter Vessenes wrote:
> Oops, I accidentally didn't reply-all to Steve. I am ccing my
> detailed response to steve to the list since I think people are
> wondering about how the Foundation fits in; trash the e-mail if you
> don't really care. :)
> 
> On Tue, Oct 2, 2012 at 10:01 PM, Peter Vessenes <peter@coinlab.com>
> wrote:
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQbGKXAAoJEFvEB9dQFvtQ5QcIAJR0RlkjUb5Xm09eo7wjV2QV
IYXbyOBx6bZWox0wxmbygpPL23grEKxlehavf18Q1S6VjdtFs75K5GV83FZb9KPk
YeB0hz5ht48Ig7uQ3zu7MBCaNerTst+6fQ/k5Uu6l2WKCVwk18WykGrnXMnSTbpM
qONQCc4HQhOm7sgEsJD9KNp73eGZt/BG0hQAa8zLh2rHA0to8TER9nRUUx+mH0uf
Cj58EXKVz5WI5+G31M/4v4UgRRv3Z7efaVbczrvhvv6DZfVkCpuD6t0kt2ZlcFCn
oO1sKD05/VtC7DZxQvPEvvkikOp8onmue0xCJBvDpG269Xr55qNqGM039Yjy80c=
=1NTS
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Bitcoin Testing Project

[Arklan Uth Oslin]

[-- Attachment #1: Type: text/plain, Size: 2387 bytes --]

is it just me, or did peter not send his message to anyone but steve -
twice? all i see is his "and finally..." then this from steve.

Arklan

----------
As long as there is light, the darkness holds no fear. And yet, even in the
deepest black, there is life. - Arklan Uth Oslin

I want to leave this world the same way I came into it: backwards and on
fire. - Arklan Uth Oslin



On Wed, Oct 3, 2012 at 10:06 AM, steve <steve@mistfpga.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Peter,
>
> Thank you for your in depth, forthright and professional response. You
> have answered my questions.  And you have cleared up a lot of
> confusion I had in my mind. (and I read ditto as i dont know) I
> appreciate the extra information. :)
>
> This is a top post and there is nothing inline so i snipped it.
>
> cheers,
>
> steve
>
> On 03/10/2012 06:03, Peter Vessenes wrote:
> > Oops, I accidentally didn't reply-all to Steve. I am ccing my
> > detailed response to steve to the list since I think people are
> > wondering about how the Foundation fits in; trash the e-mail if you
> > don't really care. :)
> >
> > On Tue, Oct 2, 2012 at 10:01 PM, Peter Vessenes <peter@coinlab.com>
> > wrote:
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJQbGKXAAoJEFvEB9dQFvtQ5QcIAJR0RlkjUb5Xm09eo7wjV2QV
> IYXbyOBx6bZWox0wxmbygpPL23grEKxlehavf18Q1S6VjdtFs75K5GV83FZb9KPk
> YeB0hz5ht48Ig7uQ3zu7MBCaNerTst+6fQ/k5Uu6l2WKCVwk18WykGrnXMnSTbpM
> qONQCc4HQhOm7sgEsJD9KNp73eGZt/BG0hQAa8zLh2rHA0to8TER9nRUUx+mH0uf
> Cj58EXKVz5WI5+G31M/4v4UgRRv3Z7efaVbczrvhvv6DZfVkCpuD6t0kt2ZlcFCn
> oO1sKD05/VtC7DZxQvPEvvkikOp8onmue0xCJBvDpG269Xr55qNqGM039Yjy80c=
> =1NTS
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 3329 bytes --]

[Bitcoin-development] Fwd: Re: Bitcoin Testing Project

[steve]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I think he had a typo in the CC.  here is a forward of the email.
You will have to work out the indentations yourselves :)

- -------- Original Message --------
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
Date: Tue, 2 Oct 2012 22:01:19 -0700
From: Peter Vessenes <peter@coinlab.com>
To: steve <steve@mistfpga.net>

On Tue, Oct 2, 2012 at 6:15 PM, steve <steve@mistfpga.net> wrote:

> On 01/10/2012 17:52, Peter Vessenes wrote:
>> I'm a big proponent of a testing project.
> 
> I am very happy to hear this, however, your actual words are
> slightly evasive. I do not expect you to be up to speed on this.
> Gavin started a project called 'the bitcoin testing project' This
> project solicited donations, about 80 coins last time I checked.
> However these 80 odd coins were donated to 'the bitcoin testing
> project' This would seem to be an official bitcoin (both protocol
> and client) testing project.  I signed up to work on this, and
> organise as much as i could of this. for various reasons I did not
> manage to do the testing I wanted to on 0.7 i over committed
> myself.
> 
> Are the donations solicited for the 'bitcoin testing project)
> funds going to be given to 'the foundation'?
> 
> 
> Not as far as I know; sounds like they should go toward testing.
> 
> 
> Does the foundation support 'the bitcoin testing project'? does
> the foundation have any involvement with 'the bitcoin testing
> project'?
> 
> I personally support the idea of a testing project. I would like
> the Foundation to fund it if it can't crowdsource funding from the
> forums; sounds like so far the support hasn't been enough to get
> all the work done. The Foundation has no formal role with the
> bitcoin testing project that I'm aware of.
> 
> 
> 
> 
>> I think if one could self organize that Gavin and team wanted to 
>> bless we could put up some BTC as bounties or funding. We won't 
>> have our heads around the foundation budget for a few more
>> weeks, but self-organization is often slower than budgeting. :)
> 
> Im ready to go, more or less.  Please check out the links in my 
> previous emails. I have over 400 testcases (8 platforms * 50
> release tests) - Also I am not sure what you mean by bless, I take
> it that is a euphemism for pay?
> 
> Wow, that's awesome! I use bless to mean "Gavin saying that it
> sounds good."
> 
> 
> I have tried my hardest to get bettermeans to work, but it doesnt.
> It does show quite a lot of work that I have done though. If you
> were to say to me, 'steve, by monday we need end to end,
> requirements based testing' It would be done. (I have already spent
> over 4 months on this)  Leaderless leadership is something I am
> having a hard time with, bettermeans is excellent at this.  But I
> have found very little in regards to voting and polling that
> integrates with the project in an effortless way like bettermeans.
> 
> I understand that the budget from the foundation is something that 
> needs to be worked on and organised.  I offer my services in this
> area (qa only).  I would be happy to submit my cv and refs for
> this, if required.
> 
> I am now feeling frustrated and useless.  has my last 4 months of
> work been for nothing? it feels like it.  I know I bang on about
> processes but they are sorted, you can only attract talent like
> Arklan if he has a process to follow. i feel like a broken record.
> 
> I'm a little late to this conversation, so I don't know what to say
> in response. I will answer your questions below, though.
> 
> 
> tl;dr version 1 - Will donations to the 'bitcoin testing project'
> as started by gavin going to be given to the foundation?
> 
> 
> I don't expect so, although we'd take them if whoever is in charge
> of the testing project wants to do so. I'd expect that if the
> testing project is good and community approved and supported by the
> dev team the funding flow would go the other way, but we'll need to
> wait for budgets to get finished.
> 
> 
> 2 - Is the work bill hees and myself going to be binned?
> 
> 
> I have no idea whatsoever, I would guess that's up to you and bill
> hees and the dev team.
> 
> 
> 3 - I feel like I have the knowledge and drive to push this, but I 
> cant do it on my own.
> 
> 
> Totally understand the feeling!
> 
> 
> 4 - Is bill or I entitled to any of the cash raised for 'the
> bitcoin testing project'
> 
> 
> I have no idea what the bitcoin testing project finance situation
> is.
> 
> 
> 5 - Do I have to join the foundation to have a say in how the
> project (testing) is done?
> 
> 
> Nope
> 
> 
> 6 - sorry for being so mercenary, but am I going to receive any
> coin for work I have done?
> 
> 
> Ditto to above
> 
> 
> 7 - It really probably is the time for a bitcoin-test list to
> appear. Is there anything I can do to make this happen?
> 
> I don't have much to do with this.
> 
> 
>> This is just my opinion, but I would like very, very much to
>> move the current specification into unit tests so that anyone
>> could validate their alternate bitcoin implementation. This is a
>> lot of work, some of which has been done, much of which hasn't.
> 
> have a look at the stuff in bettermeans.  I personally think we can
> go a step further and publish guidelines (similar to RFC's and all
> the tests that we would do against a ref client)
> 
> 
> Cool, I will check it out. I agree with the RFC oriented approach,
> or perhaps a canonical test suite for validation and verification.
> 
> 
> But I dont want to waste any more time on stuff that is going to
> be ignored, life is short.
> 
> 
>> So, my two cents, plus an offer to bring it up at our next 
>> budgeting meeting.
> 
> I accept that offer. and I really appreciate it.  I have some more 
> questions I would like you to ask in regards to QA. (Gavin and I 
> skyped about this a while ago and we didnt really come to a 
> resolution, weworked out the problems though ;) )
> 
> I have an exceptionally detailed qa process (based off the game 
> certification process) - but I have gone on about this at length
> in previous messages.
> 
> I thank you for your email and your involvement with this, but do
> you think we are closer to getting stuff tested? call my bluff...
> Not one person has asked for login details to my proposals - and i
> even have a bugzilla version now.
> 
> I need to sleep.  sorry if i rambled.
> 
> nite nite,
> 
> steve
> 
> 
> Thanks for all the details. I'm trying to go into extra detail in
> my response to you because people are still figuring out the
> Foundation's situation with all things Bitcoin. I can't offer you
> any promises for your future, but I'm personally really psyched to
> hear about these tests. I'll let other more knowledgeable folks
> jump in on the technology side of things.
> 
> 
> 
>> Peter
> 
> 
>> On Mon, Oct 1, 2012 at 7:28 AM, steve <steve@mistfpga.net>
>> wrote:
> 
>> On 01/10/2012 14:52, Arklan Uth Oslin wrote:
>>>>> Hi guys.
>>>>> 
>>>>> So, as I mentioned on the bitcointalk.org forums thread
>>>>> about the foundation, I want to get involved in the QA side
>>>>> of bitcoin development. I've done functional testing in
>>>>> the video game industry for years.
> 
>> Nice one, I worked in games for quite a few years. (before
>> getting into finance then pentesting) there are about 6 keen
>> testers now. maybe we should get a bitcoin-test mailing list,
>> where we can discuss stuff without disturbing the dev team.
> 
>>>>> I've read all the messages in this thread, but I'm left 
>>>>> unclear how I can most effectively and quickly being
>>>>> helping out. Could I get a bit of a directional nudge?
> 
>> Great question... for me I feel structure is the most important 
>> thing to sort out first.  However we desperately need detailed 
>> testcases for the release of a new version. - Not too much on
>> the change log stuff, more on the noddy stuff (as gavin points
>> out below), downloading and making sure it works on a non dev
>> machine, make sure the wallet isnt overwritten, etc.) doing games
>> qa I imagine this would be an ideal place for you to start.  I
>> have a MSDN and TechNet licence so if you need some reference ms
>> virtual machines I can help you out.
> 
>> However we need some testcase software.  Please check out what
>> was done on bettermeans for the stuff I was planning out...
> 
>> It details everything from recompense and testcases. bettermeans 
>> kinda died a death though...
> 
>> check out:
> 
>> Bitcoin over all- 
>> https://secure.bettermeans.com/projects/4180/wiki/Page_index 
>> discussion https://secure.bettermeans.com/projects/4180/boards
> 
>> 0.7 https://secure.bettermeans.com/projects/4256/boards and 
>> https://secure.bettermeans.com/projects/4256/wiki
> 
>> I still have the testcases, but until we get some proper
>> testcase software I am loathed to publish them in a half arsed
>> format. (they worked well on bettermeans, then just vanished one
>> day...) what testcase software are you familiar with?
> 
>> apart from that, what do you feel you can do for the project?
>> how long have you been involved in bitcoin?  It may well be
>> worth reading up all the dev stuff on the wiki so you can get you
>> head around how the bitcoin protocol is different from the daemon
>> and qt client. What do you think you can and will enjoy doing?
>> What is your skill set in regard to networking, crypto and
>> operating systems. (not that you need any, in any we still want
>> and need you. :)
> 
>> there really is room for you to do whatever role you want, and
>> as little or as much as you want - however funding is now a very 
>> tricky issue. so much so that I am not sure I want anything to
>> do with it(distribution of coin based on work.). - I just paid
>> for some logo spec work out of my own pocket (for example).  I
>> have some testers i know irl who are willing to work for coin.
> 
>> NOTE: This response has nothing to do with the bitcoin
>> foundation. I am not a member of the foundation. I do not speak
>> for them or even probably with them. I am still trying to work
>> out how much qa the foundation should be responsible for, and/how
>> it is supposed to work. I think the games cert process would be
>> ideal for this.  This however this a discussion that probably
>> wont have my involvement. (personally I believe that the
>> foundation should publish requirements with example code and
>> testcases for each aspect of the reference client. (on reference
>> platforms - I do not expect many to agree with this though)
> 
>> As a side note, what happens to the donations to the bitcoin 
>> testing project? do they get moved over to the foundation? this 
>> question is bigger than this email. as far as I know they are
>> all on an address Gavin holds. Actually I would like to be
>> involved in any discussions that would impact QA, does this mean
>> I need to join the foundation or just go lone wolf?
> 
>> tbh I dont really understand foundations.  I always thought they 
>> were just a tax dodge.
> 
>> Sorry for the long message. :)
> 
>>>>> 
>>>>> Arklan
>>>>> 
>>>>> ---------- As long as there is light, the darkness holds
>>>>> no fear. And yet, even in the deepest black, there is life.
>>>>> - Arklan Uth Oslin
>>>>> 
>>>>> I want to leave this world the same way I came into it: 
>>>>> backwards and on fire. - Arklan Uth Oslin
>>>>> 
>>>>> 
>>>>> 
>>>>> On Sat, Sep 29, 2012 at 12:26 PM, steve
>>>>> <steve@mistfpga.net> wrote:
>>>>> 
>>>>> Hi Gavin,
>>>>> 
>>>>> Sorry for the delayed response, I wanted to take a couple
>>>>> of days to reflect on your email.
>>>>> 
>>>>> On 26/09/2012 19:09, Gavin Andresen wrote:
>>>>> 
>>>>> 
>>>>> And their are other methods too.
>>>>> 
>>>>> 
>>>>> 
>>>>> The GUI::Test package for perl will allow this to be
>>>>> greatly automated. (I have done this before on the
>>>>> localisation of photoshop.)
>>>>> 
>>>>> 
>>>>> 
>>>>> this why we need detailed testscripts and plans.  so we
>>>>> know what has and hasnt been done. The more boring the task
>>>>> the more work that needs to go into testcase development.
>>>>> This is the area I see as my greatest failing last time.  I
>>>>> have a large number of virtual machines and should have at
>>>>> least this work.  But we need very detailed testcases.
>>>>> with decent testplans just downloading the software,
>>>>> syncing the block chain, syncing an existing wallet,
>>>>> rescanning the blockchain and verifying the balance would
>>>>> cover a large number of tests.  The idea behind having lots
>>>>> of very specific testcases is you get to see what tests
>>>>> have not been run.
>>>>> 
>>>>> 
>>>>> 
>>>>> I understand your concern, however I have taken a couple
>>>>> of days to reflect on this and I still strongly feel that
>>>>> in order to make sure that this sticks, and is still useful
>>>>> in 1 years time we need to lay proper foundations. Those 
>>>>> foundations are not word documents, spreadsheets, etc.
>>>>> they are selecting the right tools for the job.
>>>>> 
>>>>> We can gain so much benefit from using 3rd party software. 
>>>>> (bettermeans would rock if it wasnt rotting)
>>>>> 
>>>>> I am sure you could do your coding work just using vi, but
>>>>> an sdk makes it much easier and allows you to work in a
>>>>> more productive manner.
>>>>> 
>>>>> I have had a couple of off list emails with some testers
>>>>> and they also feel that it is very important to make sure
>>>>> we have a sound foundation (mantis is so much more than
>>>>> just a bug reporting tool, I see the bug reporting
>>>>> functionality as secondary to the main test run
>>>>> functionality - but it doesnt have to be mantis based, we
>>>>> do need workflow and testcase software though - and proper
>>>>> software for this is much better than just a massive google
>>>>> doc.) however I am checking out some other software that
>>>>> has been recommended.  It will be very hard to change 'the
>>>>> process' once we have something we are used too (just look
>>>>> at the current resistance) I promise nothing will change
>>>>> for the dev team.  But test does need other tools, and
>>>>> processes.
>>>>> 
>>>>> If you feel that strongly that I am going about this the 
>>>>> wrong way, I am happy to step back and let someone else
>>>>> sort it out (I will still do all the testing I possibly
>>>>> can). I would feel that this would be a real shame and we
>>>>> have the chance to setup requirements to functionality to
>>>>> tests all with traceability. why not do it right from the
>>>>> start?
>>>>> 
>>>>> I will open up my vps' somepoint over the next few days
>>>>> and you can see what I mean. I will setup a fake git
>>>>> project, and sort out the interactions.
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
> 
> ------------------------------------------------------------------------------
>>>>>>
>>>>>>
>
> 
> 
> How fast is your code?
>>>>>> 3 out of 4 devs don\\\'t know how their code performs in 
>>>>>> production. Find out how slow your code is with 
>>>>>> AppDynamics Lite. 
>>>>>> http://ad.doubleclick.net/clk;262219672;13503038;z? 
>>>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>>>>
>>>>>>
>
>>>>>> 
_______________________________________________
>>>>>> Bitcoin-development mailing list 
>>>>>> Bitcoin-development@lists.sourceforge.net 
>>>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>
>>>>>> 
>>>>>> 
> -
> 
> ------------------------------------------------------------------------------
>>>>>
>>>>>
>>
> 
Got visibility?
>>>>> Most devs has no idea what their production app looks
>>>>> like. Find out how fast your code is with AppDynamics
>>>>> Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? 
>>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 
_______________________________________________
>>>>> Bitcoin-development mailing list 
>>>>> Bitcoin-development@lists.sourceforge.net 
>>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>>>
>
>>>>> 
>>>>> 
>>> 
>>> 
>>> 
> ------------------------------------------------------------------------------
>>>
>>>
>
> 
Got visibility?
>>> Most devs has no idea what their production app looks like.
>>> Find out how fast your code is with AppDynamics Lite. 
>>> http://ad.doubleclick.net/clk;262219671;13503038;y? 
>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html 
>>> _______________________________________________ 
>>> Bitcoin-development mailing list 
>>> Bitcoin-development@lists.sourceforge.net 
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>
>
>>> 
> 
> 
> 
> 

- -- 
- ------------------------------

[image: CoinLab Logo]PETER VESSENES
CEO

*peter@coinlab.com * /  206.486.6856  / SKYPE: vessenes
811 FIRST AVENUE  /  SUITE 480  /  SEATTLE, WA 98104



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQbGSIAAoJEFvEB9dQFvtQw1YH+wW3QDMHHV7kf7ZQXvAswv5T
S2htfdgp1DBYBatoJB9fUU3hgHTEz1UhENH509SjS1/dxHuIxguJkbgmuSdoeB/4
rDuuYzQVcmym/G1YXvmAkCDk5/fUY4RKlCDGLg9MVMpVFhLIr6Jiwxd0NiIqWm9K
aE2bXijiTIRp1sNFxJh5SZTh9/rz1zF8TU3LBWyCegISwyBH2XeIIbVgORoMDBlm
n8yKGwPSg5PaH/eWhXR17OUMPbX6xGlMqTmBo7BZDCwPpZU470rkDsPywsn55mAf
W6EQhFjxpSCnmZwRhrkdM/Le/0ySSUvM4spPu7MlQ6ekGP2gBzzUPbKnTGHzNgQ=
=r57G
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Fwd: Re: Bitcoin Testing Project

[Peter Vessenes]

[-- Attachment #1: Type: text/plain, Size: 104 bytes --]

My reply-all forward was blocked (over 40k), sigh. I figured I'd spammed
the list enough for one night.

[-- Attachment #2: Type: text/html, Size: 142 bytes --]

Re: [Bitcoin-development] Fwd: Re: Bitcoin Testing Project

[Gavin Andresen]

I had hope the Testing Project would be self-organizing, with somebody
taking on the QA lead role and figuring out the hard questions like:

+ How to do fundraising?
+ If/when bitcoins are available, how to decide who gets rewarded for what?
+ If somebody wants to help, how do they start?

Steve jumped in and started creating a gazillion tests cases, which is
great, but creating test cases isn't the hard part. Creating a
"community" of testing that gets things done is the hard part that I,
frankly, don't have time to do.

I hoped that the BetterMeans platform would help, but it sounds like
it was more of a hindrance than a help.  Ok:  live and learn.  Failed
experiment, lets move on...

So, RE: moving on:  I'd like to tag a 0.7.1rc1 release in the next few
days (I'll start another thread about that). How about a very
short-term goal of getting these QA deliverables:

1. A process for QA testers to sanity-test release builds, and
sign-off as "Tested/problems found" or "Tested/OK"
2. Some place online I can look to see if all of our supported
platforms have been tested before promoting a release candidate to
"final release"



PS: Thanks to Peter for responding to the "what's the relationship
between the Foundation and the Testing Project" (executive summary: no
relationship right now).

-- 
--
Gavin Andresen

Re: [Bitcoin-development] Bitcoin Testing Project

[Matt Corallo]

On Wed, 2012-09-26 at 13:28 +0100, steve wrote:
> Hi Matt,
> 
> Glad to have another ninja onboard :)
> 
> On 25/09/2012 21:41, Matt Corallo wrote:
> > Although Jenkins may not be the best system, we already have
> > jenkins and pull-tester (which is a dumb python script I wrote to
> > test all incoming pull requests from github).
> 
> I have never heard of jenkins before.  I need to do some more digging.
> is this the right thing?
> 
> https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin
For a mantis plugin, sure, I guess...
> 
> Mantis on the other hand, I know exceptionally well.  I hate
> duplication of work/data unless absolutely necessary.  I will check
> jenkins out (just out of interest what is it actually meant to do? the
> website implies framework, but not what its for)
Jenkins currently just runs the test script after each new commit to
bitcoin (and provides binaries to anyone who wants them), so its pretty
basic (though jenkins has way more features than we use).  The bitcoin
one lives at http://jenkins.bluematt.me/
> 
> > 
> > They both run the same set of scripts, namely those at 
> > https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> > now, but since it is on github, I was hoping someone would find
> > the inspiration to add to it).
> 
> I will check it out. I wrote a very basic script that wikified the
> changelog,
We currently keep a changelog at https://en.bitcoin.it/wiki/Changelog (I
went back and added tons of logs a while back and it got updated, though
0.7 seems to be missing...) anyway, automating that would be nice...
> and linked to the changes and created wiki pages for the
> testcases.  
Having more info on that changelog page would be nice.
> have you seen the stuff I put on bettermeans? bits keep
> vanishing then re appearing.
I have been meaning to catch up with the various attempts at better
bitcoin testing that have started up a few times, but I keep never
getting around to it...
> 
> This is the outline of the testing that I setup for 0.7
> 
> https://secure.bettermeans.com/projects/4256/wiki
> 
> > 
> > I dont really care if we keep using jenkins, but I figure we might
> > as well keep all the tests in one place?
> 
> Yes, I would love to unify all build testing and testcases into one
> place.  I am still on the fence as to including unit tests into this.
> However I do see 3 distinct type of testcases
Even if unit tests are considered separate, having it all run in one
huge test script makes it quite easy to implement new things (like
pull-tester) which test some arbitrary bitcoind commit in the same way
as every other tester.  
> 1 - requirements based testcases (requirements based off the current
> block chain rules - these are edge cases and known interoperability
> issues)
The BitcoinjBitcoindComparisonTool.jar file which is run as a part of
the test scripts tries to hit as many block acceptance edge cases as
possible (I'm sure I missed a ton, but it hits a lot too).  I've also
been pushing alternate implementation implementors to use it to test
their own implementations.
> 
> 2 - Acceptance based testcases - these are testcases that should be
> run for every build.  Check out the General Acceptance Tests in the
> wiki link for examples and testcases
> 
> 3 - Testcases for reference implementations of things (like multisig -
> i see these working like the /test folder when you install a new perl
> module)
> 
> These three things alone are a massive task. and they still wont cover
> everything.  I would like to get the workflow so that people can
> sponsor or donate to a specific campaign (eg a new feature is
> implemented, people want it tested so can donate just for that
> campaign [developing testcases, structure, requirements, etc])
> 
> Once this is done, I will get to do some exciting stuff (like writing
> fuzzers, automation, etc) unfortunately I do not know python, only perl.
As far as I'm concerned more test cases are more test cases, it may get
unwieldy to maintain, but at least we'd have more test cases :)

In terms of general testing strategies, I really prefer to script it
all, jenkins is quite nice in that it can have slave workers using a
different OS which run their own tests and then report back to the main
jenkins instance.  Getting a real Windows slave to run the installer and
test that thoroughly as well as basic Mac things (I know OSX uses a very
different build system...) would be nice (though I dont really have time
to write all those tests...)

re: GUI testing is hard: I've heard Qt's unit test framework is really
powerful and can even include things like click scripting and analysis
of the current views (though, I agree, its still no doubt hard).  

Matt

Re: [Bitcoin-development] Bitcoin Testing Project

[Gregory Maxwell]

On Wed, Sep 26, 2012 at 8:53 PM, Matt Corallo <bitcoin-list@bluematt.me> wrote:
> Jenkins currently just runs the test script after each new commit to
> bitcoin (and provides binaries to anyone who wants them), so its pretty
> basic (though jenkins has way more features than we use).  The bitcoin
> one lives at http://jenkins.bluematt.me/

Jenkins is excellent at cycling through tests,  while additional
external tools may bring some value they're not required.  It's also
essential to automate all tests that we really care are run— with our
small active development group and volunteer contributors the only
tests we can count on being run are the automated ones. Automated
tests included with the software— or at least the source— are also the
only way to have a good chance of catching gnarly platform
interactions.

I think more than talking about testing I think we need is actual
testing. Code coverage from the current tests (e.g. bitcoin-test and a
testnet sync) is very unimpressive, and while coverage isn't some
magical silver bullet and does not, by itself, mean the tests are good
flaws in uncovered code can't be detected by the tests.  We also lack
simple testing cycle documentation for people interested in testing
manually to walk through, etc. I think all the meta discussion is not
very useful until we actually have more substance to put into it.
Otherwise I fear we're just building an airport by painting stripes
and waiting for the planes to land...

If someone wants to help and would like a list of some of the testing
I think would be useful, ping me off-list and I can blast some
suggestions. But I assume that anyone who actually wants to work on
this isn't short of ideas, and at this point "work on what interests
you, report what interesting thing you accomplish or discover" is
probably a perfectly fine level of coordination.

Re: [Bitcoin-development] Bitcoin Testing Project

[Daniel F]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

on 09/25/2012 02:32 PM steve said the following:
> Anyone interested in helping out/reviewing processes? even if it is just
> some encouragement, it is all greatly appreciated.

not enough time in the day for me to seriously help out, but since you
asked, here's some encouragement. :) more testing == good.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [Bitcoin-development] Bitcoin Testing Project

[Gary Rowe]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

This is definitely worth doing and I wish you every encouragement.

For my part I'm working on a different area of the Bitcoin ecosystem and
that is taking up all my time so I can only cheer you on from the sidelines.

On 25 September 2012 21:49, Daniel F <nanotube@gmail.com> wrote:

> on 09/25/2012 02:32 PM steve said the following:
> > Anyone interested in helping out/reviewing processes? even if it is just
> > some encouragement, it is all greatly appreciated.
>
> not enough time in the day for me to seriously help out, but since you
> asked, here's some encouragement. :) more testing == good.
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 1827 bytes --]