public inbox for bitcoindev@googlegroups.com
 help / color / mirror / Atom feed
From: steve <steve@mistfpga.net>
To: Matt Corallo <bitcoin-list@bluematt.me>
Cc: Bitcoin Development List
	<bitcoin-development@lists.sourceforge.net>,
	Bill Hees <billhees@gmail.com>
Subject: Re: [Bitcoin-development] Bitcoin Testing Project
Date: Wed, 26 Sep 2012 13:28:40 +0100	[thread overview]
Message-ID: <5062F4F8.6040504@mistfpga.net> (raw)
In-Reply-To: <1348605677.2284.2.camel@localhost.localdomain>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Matt,

Glad to have another ninja onboard :)

On 25/09/2012 21:41, Matt Corallo wrote:
> Although Jenkins may not be the best system, we already have
> jenkins and pull-tester (which is a dumb python script I wrote to
> test all incoming pull requests from github).

I have never heard of jenkins before.  I need to do some more digging.
is this the right thing?

https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin

Mantis on the other hand, I know exceptionally well.  I hate
duplication of work/data unless absolutely necessary.  I will check
jenkins out (just out of interest what is it actually meant to do? the
website implies framework, but not what its for)

So, currently there are 4 potential places for bugs to be reported
1 - jenkins (and unit tests)
2 - git
3 - mailing list
4 - forum (bitcointalk...)
5? - is there still the ability to add bugs via sourceforge?

Adding to this doesnt make sense.  Each one of these reporting methods
is for a different thing.  I am not seeking to replace these (or even
unify them) I am looking for software that will take testcases and bug
reports against them [and allow for test campaigns].  Mantis is so
flexible and industry standard and if the jenkins plugin works... then
we can keep things as they are until they fit into better places.

The reason I am so behind mantis as the backbone is it works with more
or less anything, and can easily modded to work with whatever people
are most comfortable with - however it is exceptionally powerful and
has had a constant stream of workflow improvements over the past few
years.

> 
> They both run the same set of scripts, namely those at 
> https://github.com/TheBlueMatt/test-scripts (its pretty basic right
> now, but since it is on github, I was hoping someone would find
> the inspiration to add to it).

I will check it out. I wrote a very basic script that wikified the
changelog, and linked to the changes and created wiki pages for the
testcases.  have you seen the stuff I put on bettermeans? bits keep
vanishing then re appearing.

This is the outline of the testing that I setup for 0.7

https://secure.bettermeans.com/projects/4256/wiki

> 
> I dont really care if we keep using jenkins, but I figure we might
> as well keep all the tests in one place?

Yes, I would love to unify all build testing and testcases into one
place.  I am still on the fence as to including unit tests into this.
However I do see 3 distinct type of testcases
1 - requirements based testcases (requirements based off the current
block chain rules - these are edge cases and known interoperability
issues)

2 - Acceptance based testcases - these are testcases that should be
run for every build.  Check out the General Acceptance Tests in the
wiki link for examples and testcases

3 - Testcases for reference implementations of things (like multisig -
i see these working like the /test folder when you install a new perl
module)

These three things alone are a massive task. and they still wont cover
everything.  I would like to get the workflow so that people can
sponsor or donate to a specific campaign (eg a new feature is
implemented, people want it tested so can donate just for that
campaign [developing testcases, structure, requirements, etc])

Once this is done, I will get to do some exciting stuff (like writing
fuzzers, automation, etc) unfortunately I do not know python, only perl.

> 
> Anyway, I'm all for more testing (I'm always complaining about how
> we need more tests for stuff...).

Nice, I love testing.  I think we will get on :)

And I would rather go for interoperability between testing rather than
rewriting it all.

Cheers,

steve

> 
> Matt
> 
> On Tue, 2012-09-25 at 19:32 +0100, steve wrote:
>> Hi All,
>> 
>> After the failure to get any real testing done for the 0.7
>> release (all of which is my fault) I have decided to rejig
>> things.
>> 
>> I am heavily into test driven development, and I have a strong 
>> background in requirements management, and automation.
>> 
>> I want to leave bettermeans behind, maybe we might be able to
>> keep the voting aspect of it, and link it into mantis.
>> 
>> So, what I have been doing over the past few weeks is developing
>> a rudimentary requirements set, basic requirement tracking, tests
>> to prove/stress the requirements.
>> 
>> The next most important thing is to get release/acceptance tests
>> done - these primarily focus on new stuff doesnt break old (ie
>> lose a wallet, etc) and needs no special requirements.
>> 
>> To this end I have installed various opensource applications
>> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating
>> the best workflow process.
>> 
>> This was a much longer post, but decided against it. :)
>> 
>> So, what I want to know is who wants to be a part helping me out
>> with all this? I am finalising the workflow flow diagrams and
>> they should be ready for inspection soon.
>> 
>> Anyone interested in helping out/reviewing processes? even if it
>> is just some encouragement, it is all greatly appreciated.
>> 
>> Drop me an email if you want access to the current setup and help
>> me review the different software for the bitcoin workflow
>> process.
>> 
>> cheers,
>> 
>> steve
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQYvT4AAoJEFvEB9dQFvtQlgkIAJX7JYel5RGmCsbptGdQrCnT
BR42tUwTg1t/NRUJ6RA8/Ou8lzallztQquShpLn4mZdQpoalvETdtAwcPnQKnaZb
M5inZE/IEq8WJM1y4YkHt3BLou4BJbjwncCNy1/jqcm6f2Oonrg7isVbDwY/7JlP
y/epm7XELS7NU4vVubBwQCunwvtsuydXRzuI812LiLXNqpXFMHvG2m8a2RajXE0/
xW4lOMy/hUFzEgYRQWCTAru4Ts2x3Xt26NaEUh/uKvHLwBZJ4xbdu3gpupiPb4sI
bCHnVFOC7zoQKOAnfPkCMyvtyoqpzM9HW2+DWI51FoOz851Y2F36N3Fpk/2lii4=
=W5xI
-----END PGP SIGNATURE-----



  parent reply	other threads:[~2012-09-26 12:29 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-09-25 18:32 [Bitcoin-development] Bitcoin Testing Project steve
2012-09-25 20:41 ` Matt Corallo
2012-09-26  5:49   ` Wladimir
2012-09-26 11:41     ` Daniel F
2012-09-26 12:00       ` Luke-Jr
2012-09-26 12:28   ` steve [this message]
2012-09-26 12:49     ` Wladimir
2012-09-26 13:22       ` steve
2012-09-26 16:06         ` Mark Friedenbach
2012-09-26 17:10           ` Jeff Garzik
2012-09-26 17:44           ` steve
2012-09-26 18:09             ` Gavin Andresen
2012-09-29 18:26               ` steve
2012-10-01 13:52                 ` Arklan Uth Oslin
2012-10-01 14:28                   ` steve
2012-10-01 16:52                     ` Peter Vessenes
2012-10-03  1:15                       ` steve
2012-10-03  2:02                         ` Gregory Maxwell
2012-10-03  3:00                           ` steve
     [not found]                         ` <CAMGNxUu=LTZyAxKt3pAYSVxyhHBU9pyJPCiFs-tA_weYNNXbtw@mail.gmail.com>
     [not found]                           ` <CAMGNxUuHRBkE_MbmY=A0vQvq=gMfzCFG8Us7SdBn-14KiKMaNg@mail.gmail.com>
2012-10-03  5:04                             ` Peter Vessenes
2012-10-03 16:06                             ` steve
2012-10-03 16:11                               ` Arklan Uth Oslin
2012-10-03 16:15                           ` [Bitcoin-development] Fwd: " steve
2012-10-03 17:09                             ` Peter Vessenes
2012-10-03 17:30                               ` Gavin Andresen
2012-09-27  0:53     ` [Bitcoin-development] " Matt Corallo
2012-09-27  2:29       ` Gregory Maxwell
2012-09-25 20:49 ` Daniel F
2012-09-25 21:25   ` Gary Rowe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5062F4F8.6040504@mistfpga.net \
    --to=steve@mistfpga.net \
    --cc=billhees@gmail.com \
    --cc=bitcoin-development@lists.sourceforge.net \
    --cc=bitcoin-list@bluematt.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox