From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-3.v43.ch3.sourceforge.com ([172.29.43.193] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1VZQoS-0001dZ-Qc for bitcoin-development@lists.sourceforge.net; Thu, 24 Oct 2013 19:43:20 +0000 Received-SPF: pass (sog-mx-3.v43.ch3.sourceforge.com: domain of taplink.co designates 50.117.27.232 as permitted sender) client-ip=50.117.27.232; envelope-from=jeremy@taplink.co; helo=mail.taplink.co; Received: from mail.taplink.co ([50.117.27.232]) by sog-mx-3.v43.ch3.sourceforge.com with smtp (Exim 4.76) id 1VZQoQ-0003Ct-Ot for bitcoin-development@lists.sourceforge.net; Thu, 24 Oct 2013 19:43:20 +0000 Received: from laptop-air ([192.168.168.135]) by mail.taplink.co ; Thu, 24 Oct 2013 12:50:40 -0700 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: "Bitcoin Development" References: <791a727f-2188-4848-bd77-ea733c8c5c2c@me.com> <201310211947.59640.luke@dashjr.org> <52661DB7.7040805@250bpm.com> <52662AA1.5050509@250bpm.com> <52677CF7.9070609@250bpm.com> <20131023194039.GB31497@petertodd.org> <52682C24.30700@250bpm.com> <20131023202731.GA31783@petertodd.org> <5268C632.3030005@250bpm.com> Date: Thu, 24 Oct 2013 12:43:15 -0700 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Jeremy Spilman" Organization: TapLink Message-ID: In-Reply-To: User-Agent: Opera Mail/1.0 (Win32) oclient: 192.168.168.135#jeremy@taplink.co#465 X-Spam-Score: -2.0 (--) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain -0.0 SPF_PASS SPF: sender matches SPF record -0.4 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: blockexplorer.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1VZQoQ-0003Ct-Ot Subject: Re: [Bitcoin-development] Revisiting the BIPS process, a proposal X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Oct 2013 19:43:21 -0000 Thanks Christian, this is a really interesting bit of history. My own personal experience from when I wrote my own client and BCCAPI-ish server was that the protocol specification on the Wiki was hugely valuable, and rarely sent me astray. Supplement that with the occasional questions on #bitcoin-dev, and then just coding, coding, coding and getting unit tests to pass. Nothing compares (IMO) to stepping through your own code watching the unit tests run, scripts evaluate, calculating transaction hashes for the different SIGHASH modes, and finally getting your first transaction into the block chain. I really appreciated the .json files holding the unit test data, which were easy to load into my own test harness, the tables on the Wiki showing what the stack should look like at each point in a script execution, and the diagrams showing transaction signing. Bitcoin takes some time to "grok" when you first approach; more than a day, less than a month, and really no amount of reading documentation or specs will get you to that "ah ha" moment. When the fog lifts and the blockchain, scripting, signing, and wallet handling really click, suddenly the bitcoind code (and many other great public sources in just about any language you could want) actually does starts to feel fairly simple and obvious. But it certainly doesn't start out that way on day one. I think the majority of client code development is actually people writing 'agents' not end-user P2P wallets, and they tend to be written to connect to a single bitcoind acting as a proxy to the network. Even some end-user wallets work this way! As such, I spent very little time in my own client writing P2P protocol code, no peer discovery code, no anti-DoS, etc. Clients like this also don't pose much systemic risk, because they don't mine, they don't connect directly to external nodes, etc. They can certainly be used to "cause trouble" though, but so can 'sendrawtransaction'. I chose to speak the P2P protocol to bitcoind versus using some of the other options like ZeroMQ, but it still didn't take long to get headers, blocks, and transactions downloading. I remember getting stuck on the very first version message, because of missing the checksum and user-agent or something caused the latest bitcoind to just ignore me. A little wireshark capture of the exchange between two working bitcoind instances cleared it right up. I didn't mind the leg work, I don't think everything needs to be spoon fed, and it's certainly not purposefully obfuscated. Maybe one exception is the mix-matched endianness will throw you off, especially if you are developing on LE! Anyway, I have huge respect for how much effort it takes to keep even small bits of documentation up-to-date. For as "slow" as bitcoin moves, it's actually moving incredibly fast. Finally, the bitcoind console and debug logs, as well sites like blockchain.info and blockexplorer.com are hugely helpful for debugging raw and live transactions for when you get stuck. There's a surprisingly large tooling and support ecosystem out there. Moral of the story, I think, is everything you need is there. No, it's not all in one place. Yes, it takes time to find it and assimilate all that knowledge. It also really helps that the community is extremely willing to help and answer technical questions, and point you in the right direction, even when you're working on your own private client code. The IRC channel can certainly be intimidating because it seems like every time I hit enter to send a question, gmaxwell's respond 300ms later would invoke an immediate forehead slap and a groan of "shit, I knew/should have known that, now I feel dumb" ;-) but if you're working on bitcoin, you better get used to not being the smartest person in the room! The responses I got were never arrogant or disparaging, but they were straight to-the-point and surprisingly high quality. Ain't no slouches in that channel, yes you will have to bring your A-game and you are expected to have "tried first" before just asking. I have fairly limited experience working on open source projects, but I'm extremely happy with my experience with the Bitcoin community and found writing Bitcoin code hugely enjoyable. The flip side to helping people implement their own clients, agents, or even miners, is helping people to contribute pulls requests, or at the very highest echelon, a BIP. If you haven't written any significant Bitcoin code, you might want to consider investing in that first before submitting a BIP. :-) For a BIP to be valuable, often it requires widespread or even consensus adoption. BIPs are probably not the place to toss just any old 'good idea' because BIPs impose a cost on all active developers. I want to read and understand 'all the BIPs' because for the most part they are actually essential, like, how to handle duplicate transactions in BIP30 - if you don't read BIP30 you very likely totally miss that, until your code throws exceptions while processing block 91842. And perhaps the hardest kind of BIP of all is the "lets get wallets to add this user-facing feature" where it has no bearing on the blockchain or transaction processing, it doesn't make the network more resilient or add crucial functionality for increasing scalability. Kind of like JPK's HD wallet encryption proposal, which I love, and I tried to contribute to in the forums, but I can totally understand the headwinds for making progress on BIPs like that one and BIP39. No one is against it per-say, it's just much harder to articulate and justify the NEED for everyone to implement, test, and support this new not-yet-standard, nice-to-have feature. For those kinds of BIPs you probably have to go out and get some wallets to implement it, or implement it yourself, to prove the value and kick start critical mass before you will even get enough support for getting a BIP number assigned. IMO, it's not a Bad Thing. TL;DR; The current support systems worked very well for me. I was able to accomplish all my goals, and I would even say it was a pleasure. Keep a high bar for assigning BIP numbers. And I hope to be able to jump back in and do more with Bitcoin soon. Thanks all, sorry if I'm rambling, Jeremy Spilman On Thu, 24 Oct 2013 04:11:05 -0700, Christian Decker wrote: > I'd like to add some historical background about how the "protocol > specification" came to be in the first place. > > A bit over three years [1] ago I started an attempt to document the > network protocol, by reverse engineering it from the satoshi > client. My goal, back then, was to enable like-minded engineers to > create alternative clients and move away from the client-monoculture > that is still predominant today. It was clear from the beginning that > it would merely be a reverse engineering effort, and that it would > likely lag a bit behind the changes in the main client. It was meant > as a help for engineers that are not well versed in C/C++ to enable > them to contribute by creating new clients, but the satoshi client > would always be the de-facto standard. > > With the move from Google Code to the Bitcoin.it wiki somehow this > notion of it being a reverse engineering effort was lost and people > started assuming that if the behavior of the satoshi client did not > match the protocol description it was a bug on the client > side. Instead it is because the reverse engineering of the protocol is > incorrect or simply missing some details. Although the protocol > description is far more complete than it was back when we started, I > still don't feel comfortable giving it the name specification. > > I still believe that a client monoculture is bad for the system as a > whole, because a single bug might bring down the whole network. Giving > people the necessary tools to implement new clients brings > stability. I do understand the criticism that writing a specification > might hinder future development as it restricts the possible changes > to the protocol, but isn't this already the case as long as we have > legacy versions of the client participating in the network? I would > also argue that having a specification allows an application > independent review of the protocol to identify possible improvements > and bugs. > > I think the protocol description has an important place in the > development of Bitcoin, so much so that we pushed a long time ago to > separate protocol version from the client version. I would love to see > the protocol specification becoming official part of the bitcoin > github repository, which would ideally be maintained alongside the > satoshi client to keep it up to date. > > Regards, > Christian Decker > > [1] https://bitcointalk.org/index.php?topic=231 > -- > Christian Decker > >