[Bitcoin-development] BIP: Custom Services

[Bitcoin-development] BIP: Custom Services

[Stefan Thomas]

Hey everyone,

I was working on some custom protocol extensions for Bitcoin that I
wanted to experiment with and I noticed that in order to enable nodes to
announce these services the only mechanism the protocol currently
provides is to use one of the 64 bits of the services field. This is
obviously a resource that will run out quickly if we all just help
ourselves, so I set out to come up with a standardized way to announce
custom protocol extensions, without using up NODE_* flags.

Please kindly review my solution:

https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Services

Thanks to Amir Taaki, Mike Hearn and Pieter Wuille who provided
invaluable feedback while writing the draft.

Note: Earlier drafts of this BIP contained a description of a mechanism
for peer exchange for these custom services. However, since that part of
the BIP was (1) just a recommendation and (2) rather complex, Amir and I
agreed to split it off into a separate BIP [1] that will be refined some
more and submitted later.

Cheers,

Stefan

[1]
https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Service_Discovery

Re: [Bitcoin-development] BIP: Custom Services

[Mike Hearn]

I think it's pretty reasonable, although people will want to use node
flags to get into the addr broadcasts anyway.

That said, I suspect (based on previous discussions) that there would
be quite some pushback against putting extra functionality into the
core Bitcoin network. Most likely people will re-use the code with
different peer discovery seeds and bootstrap similar but unrelated P2P
networks for doing new applications.

For instance, what if we want to do the language translation app I've
talked about a few times before? You need a way to floodfill broadcast
invalid transactions to interested parties. The pubsub mechanism in
the Bitcoin protocol was an interesting way to do that, but I think it
got removed. To broadcast to interested nodes now, you'd have to find
them via addr broadcasts and then connect directly. And if you're
going to do that, you may as well just form an entirely independent
network.

More elaboration of the use cases might therefore be useful.

Re: [Bitcoin-development] BIP: Custom Services

[Jeff Garzik]

On Mon, Aug 13, 2012 at 3:41 AM, Stefan Thomas <moon@justmoon.de> wrote:
> I was working on some custom protocol extensions for Bitcoin that I
> wanted to experiment with and I noticed that in order to enable nodes to
> announce these services the only mechanism the protocol currently
> provides is to use one of the 64 bits of the services field. This is
> obviously a resource that will run out quickly if we all just help
> ourselves, so I set out to come up with a standardized way to announce
> custom protocol extensions, without using up NODE_* flags.
>
> Please kindly review my solution:
>
> https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Services

heh, this is not a new idea.  I even implemented a pull request for
service discovery myself, which simply consisted of querying the list
of supported commands:
https://github.com/bitcoin/bitcoin/pull/1471

On IRC, I proposed several alternatives including modifying 'version'
(which you did) and a new "getcaps" (get capabilities) command to be
added in protocol_version X.

gmaxwell seems continually unenthused, and made a valid point about
service advertisement:  these capabilities are not advertised with
CAddress, so how does one usefully discover and make use of them?
What are real world use cases, that cannot be solved with nService
bits?

My only response is a weak one:  inevitability.  It seems likely that
-somebody- will implement their own P2P commands for their own client
subset, even if only a simple "use 'getstatus' with strSubVer matching
/FooClient/"

Therefore, if it is inevitable, we might as well make some basic rules
about how to extended your P2P command set.

-- 
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti.com

Re: [Bitcoin-development] BIP: Custom Services

[Gregory Maxwell]

On Mon, Aug 13, 2012 at 10:24 AM, Jeff Garzik <jgarzik@exmulti.com> wrote:
> My only response is a weak one:  inevitability.  It seems likely that
> -somebody- will implement their own P2P commands for their own client
> subset, even if only a simple "use 'getstatus' with strSubVer matching
> /FooClient/"
>
> Therefore, if it is inevitable, we might as well make some basic rules
> about how to extended your P2P command set.

I'm not opposed to that logic.  But for cases where an introduction mechanism
will be needed... it would be awfully good to have one, and I do think that
there is harm in making people think that simple services negotiation will
actually work for their needs for cases where a separate p2p network is
needed.

Re: [Bitcoin-development] BIP: Custom Services

[Stefan Thomas]

> More elaboration of the use cases might therefore be useful.

I'm implementing a DHT, rather than storing the whole blockchain index
locally, a future version of BitcoinJS will store only a user defined
percentage (anywhere from 0-100%.) Any misses will be resolved by
querying the network.

Thanks to ultraprune, we no longer need a full index for verification.
And for all the other use cases like lightweight server queries and
block explorer queries a bit of latency is acceptable in exchange for
scalability and decentralization. This feature will give people to
option to run anywhere from a large BitcoinJS node (100% index) to a
small one (20% index) to a lightweight one (0% index.) All of them are
equally queryable, so if you're just trying out BitcoinJS you won't have
to download the block chain just to run the block explorer example. Only
when your block explorer's traffic grows will you need to contribute
some query services back to the network in order not to get rate-limited.


> these capabilities are not advertised with
> CAddress, so how does one usefully discover and make use of them?

Pieter brought up this very point when he reviewed an earlier draft.
This prompted the creation of the second BIP I mentioned:

https://en.bitcoin.it/wiki/User:Justmoon/BIP_Draft:_Custom_Service_Discovery

The basic principle is quite simple - prefix the standard addr message
with a service-specific message to mark off service support. It's easy
to implement and very efficient (without compression it's 1.288 bits per
node, with compression it's quite possibly more efficient than the
services field.) Because this stuff is a bit more complex and because it
requires no changes to the Bitcoin protocol, Amir and I chose to split
it off into a separate BIP and I want to work on it a little more. But
if you are wondering how peer exchange might work for these custom
services, please do take a look at it.


> you may as well just form an entirely independent network

When I designed the DHT, I did just that. Later I was working on a
concept for a decentralized pool and I noticed I was solving the same
problems again. And with all three services running a node might be
maintaining three separate TCP connections to the exact same peer.

So then I considered making the DHT network extensible, so that the
decentralized pool protocol could live in there. But, well if I'm doing
that, why not just make the regular Bitcoin protocol extensible and let
both extensions live in there.

For a custom service you DO need the following:

- service-specific DNS seeds
- service announcement (BIP: CS)
- service-specific messages (BIP: CS)
- service-specific peer exchange (BIP: CSD)

And those four things cover a lot of what Bitcoin does. But once you're
thinking about n custom services it starts to look easier to add
semantics for "some nodes support some things" in one network rather
than instantiating n networks.

On the opposite end of the spectrum there are very simple services.
Consider a WebSocket transport. Some nodes might offer
Bitcoin-over-WebSocket, for example to implement an SPV client in the
browser. But they don't connect to each other via WebSocket, since they
prefer plain TCP. So in this case you need peer exchange, but not much
else. You could create a new P2P network for the sole purpose of
exchanging peers, but again it seems much easier if there were ways to
do this on the Bitcoin network.

One final point: A major focus of this BIP is to make it easy to
canonicalize custom services if we choose to do so. The idea is that
custom services get to prove themselves in the wild - those that work
well may be added to the standard protocol. That's a good reason to 1)
encourage custom services to live in-band and 2) recommend compatibility
with Bitcoin's standard mechanisms (12 byte command names, 1 bit service
announcement, 1 bit peer exchange etc.)

On 8/13/2012 3:15 PM, Mike Hearn wrote:
> I think it's pretty reasonable, although people will want to use node
> flags to get into the addr broadcasts anyway.
>
> That said, I suspect (based on previous discussions) that there would
> be quite some pushback against putting extra functionality into the
> core Bitcoin network. Most likely people will re-use the code with
> different peer discovery seeds and bootstrap similar but unrelated P2P
> networks for doing new applications.
>
> For instance, what if we want to do the language translation app I've
> talked about a few times before? You need a way to floodfill broadcast
> invalid transactions to interested parties. The pubsub mechanism in
> the Bitcoin protocol was an interesting way to do that, but I think it
> got removed. To broadcast to interested nodes now, you'd have to find
> them via addr broadcasts and then connect directly. And if you're
> going to do that, you may as well just form an entirely independent
> network.
>
> More elaboration of the use cases might therefore be useful.
>