[Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Richard Brady]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

Hi Gavin, Mike and co

Is there a strong driver behind the choice of Google Protocol Buffers for
payment request encoding in BIP-0070?

Performance doesn't feel that relevant when you think that:
1. Payment requests are not broadcast, this is a request / response flow,
much more akin to a web request.
2. One would be cramming this data into a binary format just so you can
then attach it to a no-so-binary format such as HTTP.

Some great things about protocols/encodings such as HTTP/JSON/XML are:
1. They are human readable on-the-wire. No Wireshark plugin required,
tcpdump or ngrep will do.
2. There are tons of great open source libraries and API for parsing /
manipulating / generating.
3. It's really easy to hand-craft a test message for debugging.
4. The standards are much easier to read and write. They don't need to
contain code like BIP-0070 currently does and they can contain examples,
which BIP70 does not.
5. They are thoroughly specified by independent standards bodies such as
the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard.
6. They're a family ;-)

Keen to hear your thoughts on this and very keen to watch the payment
protocol grow regardless of encoding choice! My background is SIP / VoIP
and I think that could be a fascinating use case for this protocol which
I'm hoping to do some work on.

Best,
Richard

[-- Attachment #2: Type: text/html, Size: 1625 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 2363 bytes --]

Text formats such as XML or JSON are far less deterministic, are more
loosely specified, have wide variance in parsing, are not very hash-able,
the list goes on.


On Mon, Jan 19, 2015 at 2:07 PM, Richard Brady <rnbrady@gmail.com> wrote:

> Hi Gavin, Mike and co
>
> Is there a strong driver behind the choice of Google Protocol Buffers for
> payment request encoding in BIP-0070?
>
> Performance doesn't feel that relevant when you think that:
> 1. Payment requests are not broadcast, this is a request / response flow,
> much more akin to a web request.
> 2. One would be cramming this data into a binary format just so you can
> then attach it to a no-so-binary format such as HTTP.
>
> Some great things about protocols/encodings such as HTTP/JSON/XML are:
> 1. They are human readable on-the-wire. No Wireshark plugin required,
> tcpdump or ngrep will do.
> 2. There are tons of great open source libraries and API for parsing /
> manipulating / generating.
> 3. It's really easy to hand-craft a test message for debugging.
> 4. The standards are much easier to read and write. They don't need to
> contain code like BIP-0070 currently does and they can contain examples,
> which BIP70 does not.
> 5. They are thoroughly specified by independent standards bodies such as
> the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard.
> 6. They're a family ;-)
>
> Keen to hear your thoughts on this and very keen to watch the payment
> protocol grow regardless of encoding choice! My background is SIP / VoIP
> and I think that could be a fascinating use case for this protocol which
> I'm hoping to do some work on.
>
> Best,
> Richard
>
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 3238 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Richard Brady]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Fair points, although for me the line is blurred between which of those are
security considerations vs performance considerations.

Richard

On 19 January 2015 at 19:09, Jeff Garzik <jgarzik@bitpay.com> wrote:

> Text formats such as XML or JSON are far less deterministic, are more
> loosely specified, have wide variance in parsing, are not very hash-able,
> the list goes on.
>
>
> On Mon, Jan 19, 2015 at 2:07 PM, Richard Brady <rnbrady@gmail.com> wrote:
>
>> Hi Gavin, Mike and co
>>
>> Is there a strong driver behind the choice of Google Protocol Buffers for
>> payment request encoding in BIP-0070?
>>
>> Performance doesn't feel that relevant when you think that:
>> 1. Payment requests are not broadcast, this is a request / response flow,
>> much more akin to a web request.
>> 2. One would be cramming this data into a binary format just so you can
>> then attach it to a no-so-binary format such as HTTP.
>>
>> Some great things about protocols/encodings such as HTTP/JSON/XML are:
>> 1. They are human readable on-the-wire. No Wireshark plugin required,
>> tcpdump or ngrep will do.
>> 2. There are tons of great open source libraries and API for parsing /
>> manipulating / generating.
>> 3. It's really easy to hand-craft a test message for debugging.
>> 4. The standards are much easier to read and write. They don't need to
>> contain code like BIP-0070 currently does and they can contain examples,
>> which BIP70 does not.
>> 5. They are thoroughly specified by independent standards bodies such as
>> the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard.
>> 6. They're a family ;-)
>>
>> Keen to hear your thoughts on this and very keen to watch the payment
>> protocol grow regardless of encoding choice! My background is SIP / VoIP
>> and I think that could be a fascinating use case for this protocol which
>> I'm hoping to do some work on.
>>
>> Best,
>> Richard
>>
>>

[-- Attachment #2: Type: text/html, Size: 2794 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 2942 bytes --]

None of those listed were in the context of performance.  Parsing of binary
or text is quite fast these days, and is not really a consideration versus
other needs such as a predictable encoding for a single data
representation.  XML and JSON both can represent the same post-evaluation
user data a million different ways, which is awful for anything you are
signing and hashing.  Text formats also transit binary data very poorly,
leading to unnecessary wrapping and unwrappiing (a programmatic, visibility
& bug; again performance not a primary concern).

This is evident because both XML and JSON have standards efforts under way
to correct some of these problems and make them more deterministic.
However, such standards are not field deployed and widely supported by
parsers and generators alike.




On Mon, Jan 19, 2015 at 2:16 PM, Richard Brady <rnbrady@gmail.com> wrote:

> Fair points, although for me the line is blurred between which of those
> are security considerations vs performance considerations.
>
> Richard
>
> On 19 January 2015 at 19:09, Jeff Garzik <jgarzik@bitpay.com> wrote:
>
>> Text formats such as XML or JSON are far less deterministic, are more
>> loosely specified, have wide variance in parsing, are not very hash-able,
>> the list goes on.
>>
>>
>> On Mon, Jan 19, 2015 at 2:07 PM, Richard Brady <rnbrady@gmail.com> wrote:
>>
>>> Hi Gavin, Mike and co
>>>
>>> Is there a strong driver behind the choice of Google Protocol Buffers
>>> for payment request encoding in BIP-0070?
>>>
>>> Performance doesn't feel that relevant when you think that:
>>> 1. Payment requests are not broadcast, this is a request / response
>>> flow, much more akin to a web request.
>>> 2. One would be cramming this data into a binary format just so you can
>>> then attach it to a no-so-binary format such as HTTP.
>>>
>>> Some great things about protocols/encodings such as HTTP/JSON/XML are:
>>> 1. They are human readable on-the-wire. No Wireshark plugin required,
>>> tcpdump or ngrep will do.
>>> 2. There are tons of great open source libraries and API for parsing /
>>> manipulating / generating.
>>> 3. It's really easy to hand-craft a test message for debugging.
>>> 4. The standards are much easier to read and write. They don't need to
>>> contain code like BIP-0070 currently does and they can contain examples,
>>> which BIP70 does not.
>>> 5. They are thoroughly specified by independent standards bodies such as
>>> the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard.
>>> 6. They're a family ;-)
>>>
>>> Keen to hear your thoughts on this and very keen to watch the payment
>>> protocol grow regardless of encoding choice! My background is SIP / VoIP
>>> and I think that could be a fascinating use case for this protocol which
>>> I'm hoping to do some work on.
>>>
>>> Best,
>>> Richard
>>>
>>>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 4317 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256



On 19 January 2015 12:09:13 GMT-07:00, Jeff Garzik <jgarzik@bitpay.com> wrote:
>Text formats such as XML or JSON are far less deterministic, are more
>loosely specified, have wide variance in parsing, are not very
>hash-able,
>the list goes on.

Protocol buffers isn't any more hashable than XML or json - round trips aren't deterministic with standard protobuf libraries. To make it deterministic you end up creating a new standard.

I have this problem for an asset representation standard for one of my clients, and I've reluctantly had to roll my own format.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCAA6BQJUvV+KMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhRwcCACNkpkkjIh8Zv5I8bOy
BpM2Tc5hVpg4KY6eKRXYLYgxoEnekDXN1/LJ5bfl+xzJTMTdt4f7YF0EjFJSIJ0C
UpR9KbEVShmt7UsoNwwAFxtMQmZe84vANGG11NI/cb95GO2TOlxYtPMFizQrp80s
ULAelID3Pd8yPeadU/yrF+daz9I8UHqOyioL0piWUT+kshuzqQNclHQaPKWoOPbW
XF4w1SAJjb1tHmkHqCY1HRvwlv8fqxXgjtEyjkz/HK70ZzOI+8aR49aigx2njwyL
F8EJ1gO3XkivRidTRKfbSloeq96TRneXXXfmyB6p8jI3O3BRkrk9x465EWMnzYu7
uJqo
=N1G+
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Richard Brady]

[-- Attachment #1: Type: text/plain, Size: 151 bytes --]

Thanks guys, great answers.

The design choice certainly makes a lot more sense now regardless of
whether one agrees with it or not.

Regards,
Richard

[-- Attachment #2: Type: text/html, Size: 436 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Alan Reiner]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

I'm a bit confused.  It's been a long time since I looked at protobuf
(and will have to dig into it soon), but I seem to recall it doesn't
have any of the determinism properties you guys just said.  It is
intended to allow you to skip details of the on-the-wire representations
and just send a bunch of named fields between systems.  I thought there
was no guarantee that two identical protobuf structures will get
serialized identically...?




On 01/19/2015 02:57 PM, Richard Brady wrote:
> Thanks guys, great answers. 
>
> The design choice certainly makes a lot more sense now regardless of
> whether one agrees with it or not.
>
> Regards,
> Richard
>
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #2: Type: text/html, Size: 2607 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Peter Todd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

That's 100% true: BIP70 passes around serialized protobuf data that it signs directly for this reason; it could just as easily be a byte array with json in it. (not that json/XML/etc. doesn't have other flaws)


On 19 January 2015 13:03:32 GMT-07:00, Alan Reiner <etotheipi@gmail.com> wrote:
>I'm a bit confused.  It's been a long time since I looked at protobuf
>(and will have to dig into it soon), but I seem to recall it doesn't
>have any of the determinism properties you guys just said.  It is
>intended to allow you to skip details of the on-the-wire
>representations
>and just send a bunch of named fields between systems.  I thought there
>was no guarantee that two identical protobuf structures will get
>serialized identically...?
>
>
>
>
>On 01/19/2015 02:57 PM, Richard Brady wrote:
>> Thanks guys, great answers.
>>
>> The design choice certainly makes a lot more sense now regardless of
>> whether one agrees with it or not.
>>
>> Regards,
>> Richard
>>
>>
>>
>>
>------------------------------------------------------------------------------
>> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
>> GigeNET is offering a free month of service with a new server in
>Ashburn.
>> Choose from 2 high performing configs, both with 100TB of bandwidth.
>> Higher redundancy.Lower latency.Increased capacity.Completely
>compliant.
>> http://p.sf.net/sfu/gigenet
>>
>>
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>------------------------------------------------------------------------
>
>------------------------------------------------------------------------------
>New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
>GigeNET is offering a free month of service with a new server in
>Ashburn.
>Choose from 2 high performing configs, both with 100TB of bandwidth.
>Higher redundancy.Lower latency.Increased capacity.Completely
>compliant.
>http://p.sf.net/sfu/gigenet
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Bitcoin-development mailing list
>Bitcoin-development@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bitcoin-development
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCAA6BQJUvWPOMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhbq4CACAptz1s1rYdQtviI5Y
UrqoifRAldRlUaOphle5OKCK+Gus0zqhqm/kiaQtbCg39F1gZ43QrJ/+t9kjZ19l
QcrtnIEffAyBdzCXiZH7N9HwGalxBRyiUcKc5iZT2DOYw4rwdJvYXB/PdlPOvGkq
rx/q93r2dJSFMFonnmM9pzbvUKS6dLrAWUTJCWH9EVK54TxBjVkoPXuZJEGBHr88
6MMTiRu1+KblxZbOozGRIXmrfd/7+akM6jLMU9FO4BwosUitOc9OWznGS1rvumlQ
5ESTzaiM9TpBLIW3rkJZmqA51Z1lbgVxXFA4HbctyQeMKIZxNn8e+RpWGYdpCblc
wfd6
=GYj6
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

>
> I'm a bit confused.  It's been a long time since I looked at protobuf (and
> will have to dig into it soon), but I seem to recall it doesn't have any of
> the determinism properties you guys just said.
>

It's not guaranteed no, which is why we store signed sub-messages as byte
arrays instead of typed submessages. In practice though, most
implementations do seem to serialise things the same way. I recall Python
used to be an odd one out, unsure if it still is.

OK, I guess we can boil this down more simply. BIP 70 uses protocol buffers
because I designed it and implemented the original prototype (with lots of
input from Gavin and an earlier proposal by sipa). I used protocol buffers
because, beyond all their nice properties, I used to work at Google and so
was very familiar with them.

[-- Attachment #2: Type: text/html, Size: 1130 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Gavin Andresen]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

On Mon, Jan 19, 2015 at 3:40 PM, Mike Hearn <mike@plan99.net> wrote:

> OK, I guess we can boil this down more simply. BIP 70 uses protocol
>> buffers because I designed it and implemented the original prototype (with
>> lots of input from Gavin and an earlier proposal by sipa). I used protocol
>> buffers because, beyond all their nice properties, I used to work at Google
>> and so was very familiar with them.
>>
>
What Mike said. Runner-up for encoding was JSON.

XML+ASN.1 was Right Out, because lots of us hate XML and ASN.1 with a
burning passion. Complexity is the Enemy of Security, and both XML and
ASN.1 are too complex.


-- 
--
Gavin Andresen
Chief Scientist, Bitcoin Foundation
https://www.bitcoinfoundation.org/

[-- Attachment #2: Type: text/html, Size: 1739 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Brian Hoffman]

[-- Attachment #1: Type: text/plain, Size: 1575 bytes --]

Damn if SPKI had won out we would be parsing S-expressions instead of X.509 certificates. ASN.1 is not fun IMHO.


> On Jan 19, 2015, at 3:56 PM, Gavin Andresen <gavin@bitcoinfoundation.org> wrote:
> 
> On Mon, Jan 19, 2015 at 3:40 PM, Mike Hearn <mike@plan99.net> wrote:
>>> OK, I guess we can boil this down more simply. BIP 70 uses protocol buffers because I designed it and implemented the original prototype (with lots of input from Gavin and an earlier proposal by sipa). I used protocol buffers because, beyond all their nice properties, I used to work at Google and so was very familiar with them.
> 
> 
> What Mike said. Runner-up for encoding was JSON.
> 
> XML+ASN.1 was Right Out, because lots of us hate XML and ASN.1 with a burning passion. Complexity is the Enemy of Security, and both XML and ASN.1 are too complex.
> 
> 
> -- 
> --
> Gavin Andresen
> Chief Scientist, Bitcoin Foundation
> https://www.bitcoinfoundation.org/
> 
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development

[-- Attachment #2: Type: text/html, Size: 3210 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Ross Nicoll]

[-- Attachment #1: Type: text/plain, Size: 2414 bytes --]

For what it's worth, there was consideration of replacing protocol
buffers when modifying BIP70 to function with the altcoin I work on
(changes were required anyway in eliminate any risk that payment
requests could not be accidentally applied to the wrong blockchain). The
eventual conclusion was that while we might have used JSON or XML if we
were starting from scratch, there's no choice that's clearly better.
While deployed infrastructure for payment protocol is still quite
limited, it seems that the cost to replace at this point is higher than not.

If there's ever a major reworking of the standard, for example to handle
recurring payments, it's probably worth thinking about then, but
protocol buffers result in a compact data format which is supported by
most major languages (and size is a concern if dealing with Bluetooth or
NFC), and has no major drawbacks I am aware of.

Ross

On 19/01/2015 20:40, Mike Hearn wrote:
>> I'm a bit confused.  It's been a long time since I looked at protobuf (and
>> will have to dig into it soon), but I seem to recall it doesn't have any of
>> the determinism properties you guys just said.
>>
> It's not guaranteed no, which is why we store signed sub-messages as byte
> arrays instead of typed submessages. In practice though, most
> implementations do seem to serialise things the same way. I recall Python
> used to be an odd one out, unsure if it still is.
>
> OK, I guess we can boil this down more simply. BIP 70 uses protocol buffers
> because I designed it and implemented the original prototype (with lots of
> input from Gavin and an earlier proposal by sipa). I used protocol buffers
> because, beyond all their nice properties, I used to work at Google and so
> was very familiar with them.
>
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #2: Type: text/html, Size: 3425 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Isidor Zeuner]

> For what it's worth, there was consideration of replacing protocol
> buffers when modifying BIP70 to function with the altcoin I work on
> (changes were required anyway in eliminate any risk that payment
> requests could not be accidentally applied to the wrong blockchain).

Why not serialize some kind of blockchain identifier with the
messages? Arbitrarily deviating from a given design choice just for
the sake of doing it differently may serve the goal of creating more
overall code diversity, but would not necessarily serve the quality of
the blockchain network where it is done for.

Best regards,

Isidor

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Ross Nicoll]

That was essentially what we did in the end, we replaced the network
identifier ("main"/"test") with the genesis block hash. The result is
never going to accidentally work with Bitcoin Core (nor vice-versa), but
is readily extensible to any other altcoins that want to use the
specification without requiring any sort of central registry.

Ross

On 24/01/15 13:19, Isidor Zeuner wrote:
>> For what it's worth, there was consideration of replacing protocol
>> buffers when modifying BIP70 to function with the altcoin I work on
>> (changes were required anyway in eliminate any risk that payment
>> requests could not be accidentally applied to the wrong blockchain).
>
> Why not serialize some kind of blockchain identifier with the
> messages? Arbitrarily deviating from a given design choice just for
> the sake of doing it differently may serve the goal of creating more
> overall code diversity, but would not necessarily serve the quality of
> the blockchain network where it is done for.
>
> Best regards,
>
> Isidor

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Isidor Zeuner]

> That was essentially what we did in the end, we replaced the network
> identifier ("main"/"test") with the genesis block hash. The result is
> never going to accidentally work with Bitcoin Core (nor vice-versa), but
> is readily extensible to any other altcoins that want to use the
> specification without requiring any sort of central registry.
>

Interesting approach, and I also think that requiring a central
registry would be potentially harmful.

However, I think it might not be adequate to think of the network
identifier as being congruent with the genesis block hash. In the
theoretical case of the blockchain being continued on two forked
chains (with two communities which prefer one of the chains each),
clients would not be prevented from interpreting messages on the wrong
chain.

Best regards,

Isidor

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Jorge Timón]

[-- Attachment #1: Type: text/plain, Size: 2448 bytes --]

That case is very unlikely IMO, but still you can solve it while keeping
hash of the genesis block as the chain id. If a community decides to accept
a forking chain with new rules from block N (let's call it bitcoinB), the
original chain can maintain the original genesis block and the new
community can define N (which is not accepted by bitcoin due to the new
rules) as the genesis block for bitcoinB for the purposes of chain ID. As
said forking bitcoins and  bitcoinsB with the same owners doesn't make much
sense to me. If you're creating a new currency you can just as well define
a new chain. If you want to start an initial utxo giving the new coins to
bitcoin holders...I still don't see the point, but you can also do that in
a new chain.

In summary, your example is not a good reason not to adopt a hash of the
genesis block as chain ID.
On Mar 14, 2015 5:22 PM, "Isidor Zeuner" <cryptocurrencies@quidecco.de>
wrote:

> > That was essentially what we did in the end, we replaced the network
> > identifier ("main"/"test") with the genesis block hash. The result is
> > never going to accidentally work with Bitcoin Core (nor vice-versa), but
> > is readily extensible to any other altcoins that want to use the
> > specification without requiring any sort of central registry.
> >
>
> Interesting approach, and I also think that requiring a central
> registry would be potentially harmful.
>
> However, I think it might not be adequate to think of the network
> identifier as being congruent with the genesis block hash. In the
> theoretical case of the blockchain being continued on two forked
> chains (with two communities which prefer one of the chains each),
> clients would not be prevented from interpreting messages on the wrong
> chain.
>
> Best regards,
>
> Isidor
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for
> all
> things parallel software development, from weekly thought leadership blogs
> to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 3084 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 2305 bytes --]

Correct.  I should have said "more likely to be deterministic"  Bitcoin
Core does not *rely* on determinism in BIP70; I was referring to recent
upstream efforts to make protobufs usable in a deterministic fashion by
default.

On Mon, Jan 19, 2015 at 3:03 PM, Alan Reiner <etotheipi@gmail.com> wrote:

>  I'm a bit confused.  It's been a long time since I looked at protobuf
> (and will have to dig into it soon), but I seem to recall it doesn't have
> any of the determinism properties you guys just said.  It is intended to
> allow you to skip details of the on-the-wire representations and just send
> a bunch of named fields between systems.  I thought there was no guarantee
> that two identical protobuf structures will get serialized identically...?
>
>
>
>
>
> On 01/19/2015 02:57 PM, Richard Brady wrote:
>
>   Thanks guys, great answers.
>
>  The design choice certainly makes a lot more sense now regardless of
> whether one agrees with it or not.
>
>  Regards,
> Richard
>
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.http://p.sf.net/sfu/gigenet
>
>
>
> _______________________________________________
> Bitcoin-development mailing listBitcoin-development@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 3981 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Matt Whitlock]

Even if a compact binary encoding is a high priority, there are more "standard" choices than Google Protocol Buffers. For example, ASN.1 is a very rigorously defined standard that has been around for decades, and ASN.1 even has an XML encoding (XER) that is directly convertible to/from the binary encoding (BER/DER), given the schema. In practice, I'm mostly agnostic about what encoding is actually used in BIP70, and I wouldn't fault BIP70 for choosing Google Protocol Buffers, but the very existence of Protobuf perplexes me, as it apparently re-solves a problem that was solved 40 years ago by ASN.1. It's as though the engineers at Google weren't aware that ASN.1 existed.


On Monday, 19 January 2015, at 7:07 pm, Richard Brady wrote:
> Hi Gavin, Mike and co
> 
> Is there a strong driver behind the choice of Google Protocol Buffers for
> payment request encoding in BIP-0070?
> 
> Performance doesn't feel that relevant when you think that:
> 1. Payment requests are not broadcast, this is a request / response flow,
> much more akin to a web request.
> 2. One would be cramming this data into a binary format just so you can
> then attach it to a no-so-binary format such as HTTP.
> 
> Some great things about protocols/encodings such as HTTP/JSON/XML are:
> 1. They are human readable on-the-wire. No Wireshark plugin required,
> tcpdump or ngrep will do.
> 2. There are tons of great open source libraries and API for parsing /
> manipulating / generating.
> 3. It's really easy to hand-craft a test message for debugging.
> 4. The standards are much easier to read and write. They don't need to
> contain code like BIP-0070 currently does and they can contain examples,
> which BIP70 does not.
> 5. They are thoroughly specified by independent standards bodies such as
> the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard.
> 6. They're a family ;-)
> 
> Keen to hear your thoughts on this and very keen to watch the payment
> protocol grow regardless of encoding choice! My background is SIP / VoIP
> and I think that could be a fascinating use case for this protocol which
> I'm hoping to do some work on.
> 
> Best,
> Richard

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1914 bytes --]

The engineers at Google were well aware that ASN.1 existed. I can assure
you of that, because I was one of them.

The protobuf FAQ has a very polite take on the matter:

   https://developers.google.com/protocol-buffers/docs/faq

This email thread gives more enlightenment:

   https://groups.google.com/forum/#!topic/protobuf/eNAZlnPKVW4

Anyone who has actually had to work with both ASN.1 and protocol buffers
will be able to explain why ASN.1 should not be chosen for any modern
formats. A lot of it boils down to simplicty and quality of
implementations, especially open source implementations.

With respect to the specific concerns Richard raises:

Performance doesn't feel that relevant when you think that:
>

Performance wasn't a concern.


> 2. One would be cramming this data into a binary format just so you can
> then attach it to a no-so-binary format such as HTTP.
>

HTTP transmits files as binary on the wire. So it's binary-clean and,
moreover, HTTP/2 aka SPDY is fully binary and doesn't use text anywhere
except the gzip dictionary.


> 2. There are tons of great open source libraries and API for parsing /
> manipulating / generating.
>

Luckily, this is also true of protocol buffers. Language support is pretty
good these days.


> 4. The standards are much easier to read and write. They don't need to
> contain code like BIP-0070 currently does and they can contain examples,
> which BIP70 does not.
>

BIP 70 doesn't contain any code, as far as I know. The protobuf schema
might look like code, but it's not - it's just a description of what fields
a message can contain and their types. This is very relevant for a
specification!

JSON in particular is pretty awful and I don't like it much. It suffers
complexities with things as basic as encoding numbers and strings. It's
very much unsuited to applications where correctness matters and where
you're dealing with binary structures.

[-- Attachment #2: Type: text/html, Size: 3425 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 3668 bytes --]

ASN.1 is not nearly as flexible when it comes to well-supported libraries,
generators, and the ecosystem that surrounds the actual encoding.  You
don't see ASN.1 compilers + language support packages for [all popular
programming languages], as you do with protobufs.

Google engineers were well aware it existed I'm sure.  There are wider
considerations beyond the low-level specified format.

Protobufs have their problems and aren't perfect, but ASN.1 ecosystem is
far less developed in the programming ecosystem, far less approachable for
programmers.  BIP70 wouldn't have been as easily and widely adopted if
ASN.1 had been chosen.




On Mon, Jan 19, 2015 at 2:19 PM, Matt Whitlock <bip@mattwhitlock.name>
wrote:

> Even if a compact binary encoding is a high priority, there are more
> "standard" choices than Google Protocol Buffers. For example, ASN.1 is a
> very rigorously defined standard that has been around for decades, and
> ASN.1 even has an XML encoding (XER) that is directly convertible to/from
> the binary encoding (BER/DER), given the schema. In practice, I'm mostly
> agnostic about what encoding is actually used in BIP70, and I wouldn't
> fault BIP70 for choosing Google Protocol Buffers, but the very existence of
> Protobuf perplexes me, as it apparently re-solves a problem that was solved
> 40 years ago by ASN.1. It's as though the engineers at Google weren't aware
> that ASN.1 existed.
>
>
> On Monday, 19 January 2015, at 7:07 pm, Richard Brady wrote:
> > Hi Gavin, Mike and co
> >
> > Is there a strong driver behind the choice of Google Protocol Buffers for
> > payment request encoding in BIP-0070?
> >
> > Performance doesn't feel that relevant when you think that:
> > 1. Payment requests are not broadcast, this is a request / response flow,
> > much more akin to a web request.
> > 2. One would be cramming this data into a binary format just so you can
> > then attach it to a no-so-binary format such as HTTP.
> >
> > Some great things about protocols/encodings such as HTTP/JSON/XML are:
> > 1. They are human readable on-the-wire. No Wireshark plugin required,
> > tcpdump or ngrep will do.
> > 2. There are tons of great open source libraries and API for parsing /
> > manipulating / generating.
> > 3. It's really easy to hand-craft a test message for debugging.
> > 4. The standards are much easier to read and write. They don't need to
> > contain code like BIP-0070 currently does and they can contain examples,
> > which BIP70 does not.
> > 5. They are thoroughly specified by independent standards bodies such as
> > the IETF. Gotta love a bit of MUST / SHOULD / MAY in a standard.
> > 6. They're a family ;-)
> >
> > Keen to hear your thoughts on this and very keen to watch the payment
> > protocol grow regardless of encoding choice! My background is SIP / VoIP
> > and I think that could be a fascinating use case for this protocol which
> > I'm hoping to do some work on.
> >
> > Best,
> > Richard
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>



-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 4726 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Nicolas DORIER]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

I agree that the use protocol buffer and x509 by BIP70 is a poor choice.

The choice should have been done to maximize portability, not to maximize
efficiency and flexibility.

What I ended up doing for having a similar codebase on all plateform is to
parse a BIP70 messages with the help of a web service that convert it to
JSON.
I don't like this solution since it had a trust dependency, and the
certificate verification become handled by the web service, not the device.
But even if I solved google buffer problem, I would stumble upon having
headache to validate the x509 certificate chain on every plateforms.

A simple BIP70 using JSON + HTTPS would have make things more easy.
I agree that it requires that the merchant own the domain name of the BIP70
endpoint, but I don't consider such a big of a deal, since this is how
e-commerce works.

[-- Attachment #2: Type: text/html, Size: 985 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Wladimir]

On Wed, 28 Jan 2015, Nicolas DORIER wrote:

> I agree that the use protocol buffer and x509 by BIP70 is a poor choice.

Well x509 is an international standard in common use, you can't do much 
better with regard to portability. Your suggestion about HTTPS makes 
little sense, you do know what TLS uses x509 internally as well?

Re: protocol buffers, I don't know if it's the best possible one, but one 
serialization method had to be picked. If it weren't, we could still have 
still been discussing which one to use by now. Just like for JSON there 
are bindings for many languages.

Though JSON parsers are much more diverse, which people using Bitcoin 
Core's RPC have bumped into e.g. some have some problems 
handling large numbers. Something you wouldn't expect using a 
straightforward binary format. There's no obvious best choice.

Wladimir

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Nicolas DORIER]

[-- Attachment #1: Type: text/plain, Size: 1972 bytes --]

Sure I know that x509 is international standard. And that HTTPS uses TLS.
This is not my point, my point is that when we use HTTPS the developer
delegates certificates verification to the plateform he is running on, so
developer don't have to bother about it, making the implementation safer
and easier.

On the other hand, if you charge the developer (and not the plateform) to
check certificate validity, it means that you have to develop a different
codebase for all plateform you are targeting, because each plateform store
trusted root certificate in a different manner with different APIs, and
also have different types representing a X509 Certificate.

So, let's say I want to target IOS + WP + Android + WinRT + desktop win, I
need to develop 4 times chain verification and certificate parsing.
(Because I can't verify a certificate if it is not in the specific type of
the underlying plateform)

And since it would take too much time to do that, I end up delegating
parsing and trust verification to a third party service.

2015-01-28 14:32 GMT+01:00 Wladimir <laanwj@gmail.com>:

>
> On Wed, 28 Jan 2015, Nicolas DORIER wrote:
>
>  I agree that the use protocol buffer and x509 by BIP70 is a poor choice.
>>
>
> Well x509 is an international standard in common use, you can't do much
> better with regard to portability. Your suggestion about HTTPS makes little
> sense, you do know what TLS uses x509 internally as well?
>
> Re: protocol buffers, I don't know if it's the best possible one, but one
> serialization method had to be picked. If it weren't, we could still have
> still been discussing which one to use by now. Just like for JSON there are
> bindings for many languages.
>
> Though JSON parsers are much more diverse, which people using Bitcoin
> Core's RPC have bumped into e.g. some have some problems handling large
> numbers. Something you wouldn't expect using a straightforward binary
> format. There's no obvious best choice.
>
> Wladimir
>

[-- Attachment #2: Type: text/html, Size: 2627 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

>
> On the other hand, if you charge the developer (and not the plateform) to
> check certificate validity, it means that you have to develop a different
> codebase for all plateform you are targeting, because each plateform store
> trusted root certificate in a different manner with different APIs, and
> also have different types representing a X509 Certificate.
>

That's what cross-platform abstraction libraries are for. Both Java and Qt
provide a key store library that can load from either the OS root store or
a custom one. If your chosen app platform doesn't, OK, then you'll have to
make or find one yourself. Perhaps contribute it upstream or make it a
library. But that's not a limitation of BIP70.

Just as a reminder, there is no obligation to use the OS root store. You
can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT
etc stores and load it in your app. We do this in bitcoinj by default to
avoid cases where BIP70 requests work on some platforms and not others,
although the developer can easily override this and use the OS root store
instead.

Of all possible solutions, using a third party service to convert things to
JSON is one of the least obvious and highest effort. I don't know anyone
else who arrived at such a conclusion and respectfully disagree that this
is a problem with the design choices in BIP70. It sounds like a bizarre
hack around lack of features in whatever runtime you're using.

[-- Attachment #2: Type: text/html, Size: 1798 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 2896 bytes --]

Not to mention the tiresome and error-prone task of writing your own
JSON-to-schema marshalling code -- or something equivalent to the protobufs
compiler and libs for JSON.

protobufs -- and its modern competitors such as msgpack -- natively provide
type support in a way that must be hacked into JSON or XML.

The protobuf/msgpack design is engineered to avoid bugs routinely found in
JSON parsing code; due to the amount of code & effort involved in JSON
input sanity checking, bugs and inconsistencies inevitable arise.  We have
seen this in bitcoind with JSON-RPC.



On Wed, Jan 28, 2015 at 10:42 AM, Mike Hearn <mike@plan99.net> wrote:

> On the other hand, if you charge the developer (and not the plateform) to
>> check certificate validity, it means that you have to develop a different
>> codebase for all plateform you are targeting, because each plateform store
>> trusted root certificate in a different manner with different APIs, and
>> also have different types representing a X509 Certificate.
>>
>
> That's what cross-platform abstraction libraries are for. Both Java and Qt
> provide a key store library that can load from either the OS root store or
> a custom one. If your chosen app platform doesn't, OK, then you'll have to
> make or find one yourself. Perhaps contribute it upstream or make it a
> library. But that's not a limitation of BIP70.
>
> Just as a reminder, there is no obligation to use the OS root store. You
> can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT
> etc stores and load it in your app. We do this in bitcoinj by default to
> avoid cases where BIP70 requests work on some platforms and not others,
> although the developer can easily override this and use the OS root store
> instead.
>
> Of all possible solutions, using a third party service to convert things
> to JSON is one of the least obvious and highest effort. I don't know anyone
> else who arrived at such a conclusion and respectfully disagree that this
> is a problem with the design choices in BIP70. It sounds like a bizarre
> hack around lack of features in whatever runtime you're using.
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 3944 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Nicolas DORIER]

[-- Attachment #1: Type: text/plain, Size: 3784 bytes --]

For the number of field there is in the spec, I don't consider having a
JSON to schama really worthwhile.
If you fear it is error prone, then we should provide some testing data for
the BIP70. (Which I already did for protobuf, but was rejected, because
deemed no useful thanks to the code generator... But such code generator
gave me inconsistencies with gavin's implementation for example)

Why do you think type support is very useful in our case ? we have 3 types,
and dealing only with bytes, int, and string.
It cost me more time to find a suitable cross plateform lib for protobuf
(in c#, that works in ios and winrt) than I would by just coding the json
wrapper classes by hand. (JSON libs are more wildspread and supported than
protobuf)

2015-01-28 17:04 GMT+01:00 Jeff Garzik <jgarzik@bitpay.com>:

> Not to mention the tiresome and error-prone task of writing your own
> JSON-to-schema marshalling code -- or something equivalent to the protobufs
> compiler and libs for JSON.
>
> protobufs -- and its modern competitors such as msgpack -- natively
> provide type support in a way that must be hacked into JSON or XML.
>
> The protobuf/msgpack design is engineered to avoid bugs routinely found in
> JSON parsing code; due to the amount of code & effort involved in JSON
> input sanity checking, bugs and inconsistencies inevitable arise.  We have
> seen this in bitcoind with JSON-RPC.
>
>
>
> On Wed, Jan 28, 2015 at 10:42 AM, Mike Hearn <mike@plan99.net> wrote:
>
>> On the other hand, if you charge the developer (and not the plateform) to
>>> check certificate validity, it means that you have to develop a different
>>> codebase for all plateform you are targeting, because each plateform store
>>> trusted root certificate in a different manner with different APIs, and
>>> also have different types representing a X509 Certificate.
>>>
>>
>> That's what cross-platform abstraction libraries are for. Both Java and
>> Qt provide a key store library that can load from either the OS root store
>> or a custom one. If your chosen app platform doesn't, OK, then you'll have
>> to make or find one yourself. Perhaps contribute it upstream or make it a
>> library. But that's not a limitation of BIP70.
>>
>> Just as a reminder, there is no obligation to use the OS root store. You
>> can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT
>> etc stores and load it in your app. We do this in bitcoinj by default to
>> avoid cases where BIP70 requests work on some platforms and not others,
>> although the developer can easily override this and use the OS root store
>> instead.
>>
>> Of all possible solutions, using a third party service to convert things
>> to JSON is one of the least obvious and highest effort. I don't know anyone
>> else who arrived at such a conclusion and respectfully disagree that this
>> is a problem with the design choices in BIP70. It sounds like a bizarre
>> hack around lack of features in whatever runtime you're using.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is
>> your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>>
>
>
> --
> Jeff Garzik
> Bitcoin core developer and open source evangelist
> BitPay, Inc.      https://bitpay.com/
>

[-- Attachment #2: Type: text/html, Size: 5265 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Jeff Garzik]

[-- Attachment #1: Type: text/plain, Size: 4451 bytes --]

It is not "fear", it is field experience.

JSON has proven to be a bug generator for the reasons already stated.

JSON does not include type marshalling and input validation.
Protobufs/msgpack/etc. engineered those to occur automatically, because
that is an area shown by field experience to be a constant source of bugs
and inconsistent parsing/validation behavior.




On Wed, Jan 28, 2015 at 11:52 AM, Nicolas DORIER <nicolas.dorier@gmail.com>
wrote:

> For the number of field there is in the spec, I don't consider having a
> JSON to schama really worthwhile.
> If you fear it is error prone, then we should provide some testing data
> for the BIP70. (Which I already did for protobuf, but was rejected, because
> deemed no useful thanks to the code generator... But such code generator
> gave me inconsistencies with gavin's implementation for example)
>
> Why do you think type support is very useful in our case ? we have 3
> types, and dealing only with bytes, int, and string.
> It cost me more time to find a suitable cross plateform lib for protobuf
> (in c#, that works in ios and winrt) than I would by just coding the json
> wrapper classes by hand. (JSON libs are more wildspread and supported than
> protobuf)
>
> 2015-01-28 17:04 GMT+01:00 Jeff Garzik <jgarzik@bitpay.com>:
>
>> Not to mention the tiresome and error-prone task of writing your own
>> JSON-to-schema marshalling code -- or something equivalent to the protobufs
>> compiler and libs for JSON.
>>
>> protobufs -- and its modern competitors such as msgpack -- natively
>> provide type support in a way that must be hacked into JSON or XML.
>>
>> The protobuf/msgpack design is engineered to avoid bugs routinely found
>> in JSON parsing code; due to the amount of code & effort involved in JSON
>> input sanity checking, bugs and inconsistencies inevitable arise.  We have
>> seen this in bitcoind with JSON-RPC.
>>
>>
>>
>> On Wed, Jan 28, 2015 at 10:42 AM, Mike Hearn <mike@plan99.net> wrote:
>>
>>> On the other hand, if you charge the developer (and not the plateform)
>>>> to check certificate validity, it means that you have to develop a
>>>> different codebase for all plateform you are targeting, because each
>>>> plateform store trusted root certificate in a different manner with
>>>> different APIs, and also have different types representing a X509
>>>> Certificate.
>>>>
>>>
>>> That's what cross-platform abstraction libraries are for. Both Java and
>>> Qt provide a key store library that can load from either the OS root store
>>> or a custom one. If your chosen app platform doesn't, OK, then you'll have
>>> to make or find one yourself. Perhaps contribute it upstream or make it a
>>> library. But that's not a limitation of BIP70.
>>>
>>> Just as a reminder, there is no obligation to use the OS root store. You
>>> can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT
>>> etc stores and load it in your app. We do this in bitcoinj by default to
>>> avoid cases where BIP70 requests work on some platforms and not others,
>>> although the developer can easily override this and use the OS root store
>>> instead.
>>>
>>> Of all possible solutions, using a third party service to convert things
>>> to JSON is one of the least obvious and highest effort. I don't know anyone
>>> else who arrived at such a conclusion and respectfully disagree that this
>>> is a problem with the design choices in BIP70. It sounds like a bizarre
>>> hack around lack of features in whatever runtime you're using.
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming. The Go Parallel Website,
>>> sponsored by Intel and developed in partnership with Slashdot Media, is
>>> your
>>> hub for all things parallel software development, from weekly thought
>>> leadership blogs to news, videos, case studies, tutorials and more. Take
>>> a
>>> look and join the conversation now. http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Bitcoin-development mailing list
>>> Bitcoin-development@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>>
>>>
>>
>>
>> --
>> Jeff Garzik
>> Bitcoin core developer and open source evangelist
>> BitPay, Inc.      https://bitpay.com/
>>
>
>


-- 
Jeff Garzik
Bitcoin core developer and open source evangelist
BitPay, Inc.      https://bitpay.com/

[-- Attachment #2: Type: text/html, Size: 6296 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]

>
> It is not "fear", it is field experience.
>
> JSON has proven to be a bug generator for the reasons already stated.
>

To back Jeff up on this point, today we see this story:

http://www.theregister.co.uk/2015/01/27/trivial_hole_left_black_phones_open_to_plunder/

The maker of BlackPhone – a mobile marketed as offering unusually high
levels of security – has patched *a critical vulnerability that allows
hackers to run malicious code on the handsets*. Attackers need little more
than a phone number to send a message that can compromise the devices via
the Silent Text application.

"The SCIMP protocol encodes messages as JSON objects, which are then
transmitted to the remote party over XMPP," Dowd explained to *The Register*.
"*The flaw I discovered occurs during the deserialization of these JSON
objects*. It is *a type confusion vulnerability*, which when exploited
allows an attacker to overwrite a pointer in memory, either partially or in
full. This pointer is later manipulated by the program and also the system
allocator, allowing you to do things such as pass arbitrary pointers to
free()."

The C++/Java/Python protocol buffer implementations are used by Google for
all internal inter-server communication. Any similar exploit in them would
result in total bypass of their entire internal security and auditing
system by allowing you to run code as any user. The Google security team is
very good, the protobuf code is carefully reviewed and the format is
relatively constrained. The chances of there being any security problems in
the parsing code generated by the protobuf compilers is drastically
smaller. As BIP70 requests are parsed by security sensitive code, this
matters.

The vision for BIP70 has always been to be a foundation for many features.
We haven't really done much with it so far because there have always been
higher priorities. But I hope that if Bitcoin continues to be successful
and grows, one day payment requests will have many different features in
them and those will likely include many complex data structures.

[-- Attachment #2: Type: text/html, Size: 3885 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Giuseppe Mazzotta]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 28-01-15 16:42, Mike Hearn wrote:
> Just as a reminder, there is no obligation to use the OS root
> store. You can (and quite possibly should) take a snapshot of the 
> Mozilla/Apple/MSFT etc stores and load it in your app. We do this
> in bitcoinj by default to avoid cases where BIP70 requests work on
> some platforms and not others, although the developer can easily
> override this and use the OS root store instead.
> 
Except that Mozilla/Apple/MSFT will update these certificate stores -
second their policies - and your snapshot/collection might get
outdated at a different pace than the OS-provided certificates,
depending on how you (or the package maintainer) are rolling out updates.

I am not saying that OS-provided certificate stores are a holy grail,
as they comes with their quirks&headaches (for example try to install
your CA certificate on Mac), but generally I consider shipping your
own snapshot a bad practice, as it makes the system less secure by
default for the casual user and harder to audit for the (eventual) admins.

If you are a developer, that's a whole different story.

- -- 
  Giuseppe Mazzotta
    /- Bitonic _/
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJUyQwrAAoJEKWX1kB3NXekfLkH/3Bd2yPD0ccZRUzWJ47wQmFd
uRxpOjlxZMy3ww4PpxPxqJTrSwyH0gwbNCbuXkds9kkx9+AwxEkT8VZXZVA1KNeo
RaLNgqs4R5zK06shTjV+AXe5HwDCEeFuJIIRGM35D8kSKtbPuzKe5UtiIRaBRdZG
e+d6CQKklYK8yn24hUGg30Y0rEcrLicXmcJlrjElA+8pKYDIeP3SrMWjLj9QKFhJ
VBKxXlyviZ2LGf9wD6p+GLgjDu4LNEQdJPpTNzg7yWSegnKmxopefojkCrHtqJQ7
LC4Lr3K3hp5Nbct1YWRt09VQYic/xaGBCSUdKTEL1xTUv7oD/DY7l15zjKi7gtw=
=ZJVu
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Matt Whitlock]

On Wednesday, 28 January 2015, at 5:19 pm, Giuseppe Mazzotta wrote:
> On 28-01-15 16:42, Mike Hearn wrote:
> > Just as a reminder, there is no obligation to use the OS root
> > store. You can (and quite possibly should) take a snapshot of the
> > Mozilla/Apple/MSFT etc stores and load it in your app. We do this
> > in bitcoinj by default to avoid cases where BIP70 requests work on
> > some platforms and not others, although the developer can easily
> > override this and use the OS root store instead.
> >
> Except that Mozilla/Apple/MSFT will update these certificate stores -
> second their policies - and your snapshot/collection might get
> outdated at a different pace than the OS-provided certificates,
> depending on how you (or the package maintainer) are rolling out updates.

I'm frankly _horrified_ to learn that BitcoinJ ships its own root CA certificates bundle. This means that, if a root CA gets breached and a certificate gets revoked, all BitcoinJ-using software will be vulnerable until BitcoinJ ships an update *and* the software in question pulls in the new BitcoinJ update and releases its own update. That might never happen.

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

>
> I'm frankly _horrified_ to learn that BitcoinJ ships its own root CA
> certificates bundle. This means that, if a root CA gets breached and a
> certificate gets revoked, all BitcoinJ-using software will be vulnerable
> until BitcoinJ ships an update *and* the software in question pulls in the
> new BitcoinJ update and releases its own update. That might never happen.


If your wallet is unmaintained, you have other problems beyond (extremely
rare) root CA revocations.

As far as I know the only time a CA in wide usage has been revoked entirely
is DigiNotar.

One advantage of doing it this way is if, for example, a widely used piece
of community infrastructure (e.g. bitcointalk, reddit, whatever) decides to
become a CA, the Bitcoin community can decide to have different inclusion
rules vs the OS/browser root CA programs. For example we'd probably relax
the constraint to use an HSM and just ensure that the rendering of the
asserted identity isn't confusible with other kinds of more strongly
protected identities. For example no forum usernames like "foo.com" but
rendering it in the UI as "Reddit forum user foo.com" would be OK.

Also you don't get problems due to old operating systems not including new
certs.

Finally, Linux doesn't have any kind of standardised cert/keystore API.
There are a few places where popular distros put certs but AFAIK they
aren't standardised and there's no standard code to load them. So that's
another reason why there's a built in store.

But yes, this is a debatable topic on which reasonable people can disagree.
The API makes it easy to use the platform OS store for wallet devs that
want to do that, and I think using the platform store on Android is the
default. It's only on the desktop where we fall back to a different store.

[-- Attachment #2: Type: text/html, Size: 2235 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Nicolas DORIER]

[-- Attachment #1: Type: text/plain, Size: 3696 bytes --]

My point is not that there is a limitation in BIP70. My point is that you
put the burden of certificate verification on developer's shoulder when we
can just leverage built in HTTPS support of the platform.
This make cross plateform dev a nightmare.

Sure I can use a snapshot of moz/apple/msft store. I depends on
BouncyCastle, as bitcoinj, so I theorically can use that way.
However, if you want to use your plateform's store, then you are toasted,
and the code for converting from BC X509 Certificate to one of each
plateform is not obvious and is a headache. Thing that could be just left
to the HTTPS support of your plateform.

Have you tried to do that on windows RT and IOS ? I tried, and I quickly
stopped doing that since it is not worth the effort. (Frankly I am not even
sure you can on win rt, since the API is a stripped down version of windows)

Why have you not heard about the problem ? (until now, because I have this
problem because I need to have the same codebase on
winrt/win/android/ios/tablets)
Because bitcoinj just rely either java's own abstraction of certificate or
on BC one. But I highly doubt they are using the plateform store, and even
if you theorically can, dealing with X509 is very prone to error... for
something that the plateform should just do for you.
Also, you bundle mozilla's store in bitcoinj, what happen when the store
change and your customer have not intent to use bitcoinj new version ? by
leveraging the plateform you benefit from automatic updates.
Also, does java stores deals with certificate revocations ? sure you can
theorically code that too... or just let the plateform deals with it.

BIP70 does not limit to anything but it is a gigantic pain in the ass for
easy cross development because of protobuff and embedded certificates.
BIP70 is a client side technology, not a performance and storage critical
data structure.

The only valid point of having embedded certificates is to allow the owner
of the website to be different from the merchant. But since merchants often
have their own website, a protocol without having to reinvent x509 would
have been better suited to current needs.

2015-01-28 16:42 GMT+01:00 Mike Hearn <mike@plan99.net>:

> On the other hand, if you charge the developer (and not the plateform) to
>> check certificate validity, it means that you have to develop a different
>> codebase for all plateform you are targeting, because each plateform store
>> trusted root certificate in a different manner with different APIs, and
>> also have different types representing a X509 Certificate.
>>
>
> That's what cross-platform abstraction libraries are for. Both Java and Qt
> provide a key store library that can load from either the OS root store or
> a custom one. If your chosen app platform doesn't, OK, then you'll have to
> make or find one yourself. Perhaps contribute it upstream or make it a
> library. But that's not a limitation of BIP70.
>
> Just as a reminder, there is no obligation to use the OS root store. You
> can (and quite possibly should) take a snapshot of the Mozilla/Apple/MSFT
> etc stores and load it in your app. We do this in bitcoinj by default to
> avoid cases where BIP70 requests work on some platforms and not others,
> although the developer can easily override this and use the OS root store
> instead.
>
> Of all possible solutions, using a third party service to convert things
> to JSON is one of the least obvious and highest effort. I don't know anyone
> else who arrived at such a conclusion and respectfully disagree that this
> is a problem with the design choices in BIP70. It sounds like a bizarre
> hack around lack of features in whatever runtime you're using.
>
>

[-- Attachment #2: Type: text/html, Size: 4517 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 3692 bytes --]

>
> My point is not that there is a limitation in BIP70. My point is that you
> put the burden of certificate verification on developer's shoulder when we
> can just leverage built in HTTPS support of the platform.
>

Platforms that support HTTPS but not certificate handling are rare - I know
HTML5 is such a platform but such apps are inherently dependent on the
server anyway and the server can just do the parsing and validation work
itself. If WinRT is such a platform, OK, too bad.

The embedding of the certificates is not arbitrary or pointless, by the
way. It's there for a very good reason - it makes the signed payment
request verifiable by third parties. Effectively you can store the signed
message and present it later to someone else, it's undeniable. Combined
with the transactions and merkle branches linking them to the block chain,
what you have is a form of digital receipt ... a proof of purchase that can
be automatically verified as legitimate. This has all kinds of use cases.

Because of how HTTPS works, you can't easily prove to a third party that a
server gave you a piece of data. Doing so requires staggeringly complex
hacks (see tls notary) and when we designed BIP70, those hacks didn't even
exist. So we'd lose the benefit of having a digitally signed request.

Additionally, doing things this way means BIP70 requests can be signed by
things which are not HTTPS servers. For example you can sign with an email
address cert, an EV certificate i.e. a company, a certificate issued by
some user forum, whatever else we end up wanting. Not every payment
recipient can be identified by a domain name + dynamic session.


> However, if you want to use your plateform's store, then you are toasted
>

That's a bit melodramatic. BitcoinJ is able to use the Android, JRE,
Windows and Mac certificate stores all using the same code or very minor
variants on it (e.g. on Mac you have to specify you want the system store
but it's a one-liner).

Yes, that's not *every* platform. Some will require custom binding glue and
it depends what abstractions and languages you are using.


> Have you tried to do that on windows RT and IOS ? I tried, and I quickly
> stopped doing that since it is not worth the effort. (Frankly I am not even
> sure you can on win rt, since the API is a stripped down version of windows)
>

There is code to do iOS using the Apple APIs here:

https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391


> Why have you not heard about the problem ? (until now, because I have this
> problem because I need to have the same codebase on
> winrt/win/android/ios/tablets)
>

WinRT is a minority platform in the extreme, and all the other platforms
you mentioned have the necessary APIs. Java abstracts you from them. So I
think you are encountering this problem because you desire to target WinRT
and other platforms with a single codebase. That's an unusual constraint.

AFAIK the only other people who encountered this are BitPay, because they
want to do everything in Javascript which doesn't really provide any major
APIs.


> Also, you bundle mozilla's store in bitcoinj, what happen when the store
> change and your customer have not intent to use bitcoinj new version ? by
> leveraging the plateform you benefit from automatic updates.
>

Yes, there are pros and cons to bundling a custom root store.


> Also, does java stores deals with certificate revocations ? sure you can
> theorically code that too... or just let the plateform deals with it.
>

It can do OCSP checks, yes, although I believe no wallets currently do so.
A better solution would be to implement an OCSP stapling extension to BIP70
though.

[-- Attachment #2: Type: text/html, Size: 5748 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Nicolas Dorier]

[-- Attachment #1: Type: text/plain, Size: 4361 bytes --]

Mike, I am not denying it is impossible to do all of that.
Just that it is not a trivial stuff to do to make it works everywhere, and
I think that it is not a good thing for a client side technology.
BIP70 has its use, and I understand why there is case where it is good to
ship the certs in the message and not depends on the transport.

But a standard that just use JSON and HTTPS, even if less flexible that
BIP70, would make it easier and sufficient for today's use case.

On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99.net> wrote:

> My point is not that there is a limitation in BIP70. My point is that you
>> put the burden of certificate verification on developer's shoulder when we
>> can just leverage built in HTTPS support of the platform.
>>
>
> Platforms that support HTTPS but not certificate handling are rare - I
> know HTML5 is such a platform but such apps are inherently dependent on the
> server anyway and the server can just do the parsing and validation work
> itself. If WinRT is such a platform, OK, too bad.
>
> The embedding of the certificates is not arbitrary or pointless, by the
> way. It's there for a very good reason - it makes the signed payment
> request verifiable by third parties. Effectively you can store the signed
> message and present it later to someone else, it's undeniable. Combined
> with the transactions and merkle branches linking them to the block chain,
> what you have is a form of digital receipt ... a proof of purchase that can
> be automatically verified as legitimate. This has all kinds of use cases.
>
> Because of how HTTPS works, you can't easily prove to a third party that a
> server gave you a piece of data. Doing so requires staggeringly complex
> hacks (see tls notary) and when we designed BIP70, those hacks didn't even
> exist. So we'd lose the benefit of having a digitally signed request.
>
> Additionally, doing things this way means BIP70 requests can be signed by
> things which are not HTTPS servers. For example you can sign with an email
> address cert, an EV certificate i.e. a company, a certificate issued by
> some user forum, whatever else we end up wanting. Not every payment
> recipient can be identified by a domain name + dynamic session.
>
>
>> However, if you want to use your plateform's store, then you are toasted
>>
>
> That's a bit melodramatic. BitcoinJ is able to use the Android, JRE,
> Windows and Mac certificate stores all using the same code or very minor
> variants on it (e.g. on Mac you have to specify you want the system store
> but it's a one-liner).
>
> Yes, that's not *every* platform. Some will require custom binding glue
> and it depends what abstractions and languages you are using.
>
>
>> Have you tried to do that on windows RT and IOS ? I tried, and I quickly
>> stopped doing that since it is not worth the effort. (Frankly I am not even
>> sure you can on win rt, since the API is a stripped down version of windows)
>>
>
> There is code to do iOS using the Apple APIs here:
>
>
> https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391
>
>
>> Why have you not heard about the problem ? (until now, because I have
>> this problem because I need to have the same codebase on
>> winrt/win/android/ios/tablets)
>>
>
> WinRT is a minority platform in the extreme, and all the other platforms
> you mentioned have the necessary APIs. Java abstracts you from them. So I
> think you are encountering this problem because you desire to target WinRT
> and other platforms with a single codebase. That's an unusual constraint.
>
> AFAIK the only other people who encountered this are BitPay, because they
> want to do everything in Javascript which doesn't really provide any major
> APIs.
>
>
>> Also, you bundle mozilla's store in bitcoinj, what happen when the store
>> change and your customer have not intent to use bitcoinj new version ? by
>> leveraging the plateform you benefit from automatic updates.
>>
>
> Yes, there are pros and cons to bundling a custom root store.
>
>
>> Also, does java stores deals with certificate revocations ? sure you can
>> theorically code that too... or just let the plateform deals with it.
>>
>
> It can do OCSP checks, yes, although I believe no wallets currently do so.
> A better solution would be to implement an OCSP stapling extension to BIP70
> though.
>

[-- Attachment #2: Type: text/html, Size: 6654 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 5653 bytes --]

I think we'll just have to agree to disagree on this one. I've implemented
BIP70 a couple of times now and didn't find it to be difficult. I know you
had odd problems with the C# protobuf implementation you were using but
library bugs can happen for any kind of programming.

I forgot to mention the other reason it's done this way. One of the driving
goals of BIP70 was to support the TREZOR and similar devices. For hardware
wallets, it's critical to keep the amount of code they need to run as small
as possible. Any bugs in the code there can cause security holes and lead
to the device being hacked.

Doing it the way you suggest would mean the secure code would have to
contain complex and bug-prone text parsing logic as well as a full blown
HTTP and SSL stack, that requires not only X.509 handling but also lots of
other stuff on top. It'd increase cost, complexity and decrease security
quite a bit.

Whilst I appreciate if your platform provides a scripting-like API and
nothing low level it might seem easier to use JSON+HTTPS, that isn't the
case for one of the primary design targets.



On Wed, Jan 28, 2015 at 6:04 PM, Nicolas Dorier <nicolas.dorier@gmail.com>
wrote:

> Mike, I am not denying it is impossible to do all of that.
> Just that it is not a trivial stuff to do to make it works everywhere, and
> I think that it is not a good thing for a client side technology.
> BIP70 has its use, and I understand why there is case where it is good to
> ship the certs in the message and not depends on the transport.
>
> But a standard that just use JSON and HTTPS, even if less flexible that
> BIP70, would make it easier and sufficient for today's use case.
>
> On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99.net> wrote:
>
>> My point is not that there is a limitation in BIP70. My point is that you
>>> put the burden of certificate verification on developer's shoulder when we
>>> can just leverage built in HTTPS support of the platform.
>>>
>>
>> Platforms that support HTTPS but not certificate handling are rare - I
>> know HTML5 is such a platform but such apps are inherently dependent on the
>> server anyway and the server can just do the parsing and validation work
>> itself. If WinRT is such a platform, OK, too bad.
>>
>> The embedding of the certificates is not arbitrary or pointless, by the
>> way. It's there for a very good reason - it makes the signed payment
>> request verifiable by third parties. Effectively you can store the signed
>> message and present it later to someone else, it's undeniable. Combined
>> with the transactions and merkle branches linking them to the block chain,
>> what you have is a form of digital receipt ... a proof of purchase that can
>> be automatically verified as legitimate. This has all kinds of use cases.
>>
>> Because of how HTTPS works, you can't easily prove to a third party that
>> a server gave you a piece of data. Doing so requires staggeringly complex
>> hacks (see tls notary) and when we designed BIP70, those hacks didn't even
>> exist. So we'd lose the benefit of having a digitally signed request.
>>
>> Additionally, doing things this way means BIP70 requests can be signed by
>> things which are not HTTPS servers. For example you can sign with an email
>> address cert, an EV certificate i.e. a company, a certificate issued by
>> some user forum, whatever else we end up wanting. Not every payment
>> recipient can be identified by a domain name + dynamic session.
>>
>>
>>> However, if you want to use your plateform's store, then you are toasted
>>>
>>
>> That's a bit melodramatic. BitcoinJ is able to use the Android, JRE,
>> Windows and Mac certificate stores all using the same code or very minor
>> variants on it (e.g. on Mac you have to specify you want the system store
>> but it's a one-liner).
>>
>> Yes, that's not *every* platform. Some will require custom binding glue
>> and it depends what abstractions and languages you are using.
>>
>>
>>> Have you tried to do that on windows RT and IOS ? I tried, and I quickly
>>> stopped doing that since it is not worth the effort. (Frankly I am not even
>>> sure you can on win rt, since the API is a stripped down version of windows)
>>>
>>
>> There is code to do iOS using the Apple APIs here:
>>
>>
>> https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391
>>
>>
>>> Why have you not heard about the problem ? (until now, because I have
>>> this problem because I need to have the same codebase on
>>> winrt/win/android/ios/tablets)
>>>
>>
>> WinRT is a minority platform in the extreme, and all the other platforms
>> you mentioned have the necessary APIs. Java abstracts you from them. So I
>> think you are encountering this problem because you desire to target WinRT
>> and other platforms with a single codebase. That's an unusual constraint.
>>
>> AFAIK the only other people who encountered this are BitPay, because they
>> want to do everything in Javascript which doesn't really provide any major
>> APIs.
>>
>>
>>> Also, you bundle mozilla's store in bitcoinj, what happen when the store
>>> change and your customer have not intent to use bitcoinj new version ? by
>>> leveraging the plateform you benefit from automatic updates.
>>>
>>
>> Yes, there are pros and cons to bundling a custom root store.
>>
>>
>>> Also, does java stores deals with certificate revocations ? sure you can
>>> theorically code that too... or just let the plateform deals with it.
>>>
>>
>> It can do OCSP checks, yes, although I believe no wallets currently do
>> so. A better solution would be to implement an OCSP stapling extension to
>> BIP70 though.
>>
>
>

[-- Attachment #2: Type: text/html, Size: 8314 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Angel Leon]

[-- Attachment #1: Type: text/plain, Size: 6659 bytes --]

why not allow both serializations and keep serialization format a
parameter, keep everyone happy.

http://twitter.com/gubatron

On Wed, Jan 28, 2015 at 12:14 PM, Mike Hearn <mike@plan99.net> wrote:

> I think we'll just have to agree to disagree on this one. I've implemented
> BIP70 a couple of times now and didn't find it to be difficult. I know you
> had odd problems with the C# protobuf implementation you were using but
> library bugs can happen for any kind of programming.
>
> I forgot to mention the other reason it's done this way. One of the
> driving goals of BIP70 was to support the TREZOR and similar devices. For
> hardware wallets, it's critical to keep the amount of code they need to run
> as small as possible. Any bugs in the code there can cause security holes
> and lead to the device being hacked.
>
> Doing it the way you suggest would mean the secure code would have to
> contain complex and bug-prone text parsing logic as well as a full blown
> HTTP and SSL stack, that requires not only X.509 handling but also lots of
> other stuff on top. It'd increase cost, complexity and decrease security
> quite a bit.
>
> Whilst I appreciate if your platform provides a scripting-like API and
> nothing low level it might seem easier to use JSON+HTTPS, that isn't the
> case for one of the primary design targets.
>
>
>
> On Wed, Jan 28, 2015 at 6:04 PM, Nicolas Dorier <nicolas.dorier@gmail.com>
> wrote:
>
>> Mike, I am not denying it is impossible to do all of that.
>> Just that it is not a trivial stuff to do to make it works everywhere,
>> and I think that it is not a good thing for a client side technology.
>> BIP70 has its use, and I understand why there is case where it is good to
>> ship the certs in the message and not depends on the transport.
>>
>> But a standard that just use JSON and HTTPS, even if less flexible that
>> BIP70, would make it easier and sufficient for today's use case.
>>
>> On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99.net> wrote:
>>
>>> My point is not that there is a limitation in BIP70. My point is that
>>>> you put the burden of certificate verification on developer's shoulder when
>>>> we can just leverage built in HTTPS support of the platform.
>>>>
>>>
>>> Platforms that support HTTPS but not certificate handling are rare - I
>>> know HTML5 is such a platform but such apps are inherently dependent on the
>>> server anyway and the server can just do the parsing and validation work
>>> itself. If WinRT is such a platform, OK, too bad.
>>>
>>> The embedding of the certificates is not arbitrary or pointless, by the
>>> way. It's there for a very good reason - it makes the signed payment
>>> request verifiable by third parties. Effectively you can store the signed
>>> message and present it later to someone else, it's undeniable. Combined
>>> with the transactions and merkle branches linking them to the block chain,
>>> what you have is a form of digital receipt ... a proof of purchase that can
>>> be automatically verified as legitimate. This has all kinds of use cases.
>>>
>>> Because of how HTTPS works, you can't easily prove to a third party that
>>> a server gave you a piece of data. Doing so requires staggeringly complex
>>> hacks (see tls notary) and when we designed BIP70, those hacks didn't even
>>> exist. So we'd lose the benefit of having a digitally signed request.
>>>
>>> Additionally, doing things this way means BIP70 requests can be signed
>>> by things which are not HTTPS servers. For example you can sign with an
>>> email address cert, an EV certificate i.e. a company, a certificate issued
>>> by some user forum, whatever else we end up wanting. Not every payment
>>> recipient can be identified by a domain name + dynamic session.
>>>
>>>
>>>> However, if you want to use your plateform's store, then you are toasted
>>>>
>>>
>>> That's a bit melodramatic. BitcoinJ is able to use the Android, JRE,
>>> Windows and Mac certificate stores all using the same code or very minor
>>> variants on it (e.g. on Mac you have to specify you want the system store
>>> but it's a one-liner).
>>>
>>> Yes, that's not *every* platform. Some will require custom binding glue
>>> and it depends what abstractions and languages you are using.
>>>
>>>
>>>> Have you tried to do that on windows RT and IOS ? I tried, and I
>>>> quickly stopped doing that since it is not worth the effort. (Frankly I am
>>>> not even sure you can on win rt, since the API is a stripped down version
>>>> of windows)
>>>>
>>>
>>> There is code to do iOS using the Apple APIs here:
>>>
>>>
>>> https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391
>>>
>>>
>>>> Why have you not heard about the problem ? (until now, because I have
>>>> this problem because I need to have the same codebase on
>>>> winrt/win/android/ios/tablets)
>>>>
>>>
>>> WinRT is a minority platform in the extreme, and all the other platforms
>>> you mentioned have the necessary APIs. Java abstracts you from them. So I
>>> think you are encountering this problem because you desire to target WinRT
>>> and other platforms with a single codebase. That's an unusual constraint.
>>>
>>> AFAIK the only other people who encountered this are BitPay, because
>>> they want to do everything in Javascript which doesn't really provide any
>>> major APIs.
>>>
>>>
>>>> Also, you bundle mozilla's store in bitcoinj, what happen when the
>>>> store change and your customer have not intent to use bitcoinj new version
>>>> ? by leveraging the plateform you benefit from automatic updates.
>>>>
>>>
>>> Yes, there are pros and cons to bundling a custom root store.
>>>
>>>
>>>> Also, does java stores deals with certificate revocations ? sure you
>>>> can theorically code that too... or just let the plateform deals with it.
>>>>
>>>
>>> It can do OCSP checks, yes, although I believe no wallets currently do
>>> so. A better solution would be to implement an OCSP stapling extension to
>>> BIP70 though.
>>>
>>
>>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is
> your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>

[-- Attachment #2: Type: text/html, Size: 9858 bytes --]

Re: [Bitcoin-development] BIP70: why Google Protocol Buffers for encoding?

[Nicolas DORIER]

[-- Attachment #1: Type: text/plain, Size: 6684 bytes --]

Sure,

But the mobile targets, it is still easier to use Json + HTTPS, especially
when you want one code base for everything.
And as you said, developers need to think about fetching mozilla store time
to time, and check revocations themselves. This is not obvious thing to do,
and hard to test correctly.

If your use case was the primary utility of BIP70, then I'd say it fit the
bill. But for cross plateform client development an atlernative would be
easier.

> why not allow both serializations and keep serialization format a
parameter, keep everyone happy.

It would be another BIP, because if we use JSON with HTTPS, the difference
is also in the semantic (no embedded certificates)

I will likely provide this option for a product I am developing. I will
only use another Content Type. We'll see then how it goes.

2015-01-28 18:14 GMT+01:00 Mike Hearn <mike@plan99.net>:

> I think we'll just have to agree to disagree on this one. I've implemented
> BIP70 a couple of times now and didn't find it to be difficult. I know you
> had odd problems with the C# protobuf implementation you were using but
> library bugs can happen for any kind of programming.
>
> I forgot to mention the other reason it's done this way. One of the
> driving goals of BIP70 was to support the TREZOR and similar devices. For
> hardware wallets, it's critical to keep the amount of code they need to run
> as small as possible. Any bugs in the code there can cause security holes
> and lead to the device being hacked.
>
> Doing it the way you suggest would mean the secure code would have to
> contain complex and bug-prone text parsing logic as well as a full blown
> HTTP and SSL stack, that requires not only X.509 handling but also lots of
> other stuff on top. It'd increase cost, complexity and decrease security
> quite a bit.
>
> Whilst I appreciate if your platform provides a scripting-like API and
> nothing low level it might seem easier to use JSON+HTTPS, that isn't the
> case for one of the primary design targets.
>
>
>
> On Wed, Jan 28, 2015 at 6:04 PM, Nicolas Dorier <nicolas.dorier@gmail.com>
> wrote:
>
>> Mike, I am not denying it is impossible to do all of that.
>> Just that it is not a trivial stuff to do to make it works everywhere,
>> and I think that it is not a good thing for a client side technology.
>> BIP70 has its use, and I understand why there is case where it is good to
>> ship the certs in the message and not depends on the transport.
>>
>> But a standard that just use JSON and HTTPS, even if less flexible that
>> BIP70, would make it easier and sufficient for today's use case.
>>
>> On Wed, Jan 28, 2015 at 5:55 PM, Mike Hearn <mike@plan99.net> wrote:
>>
>>> My point is not that there is a limitation in BIP70. My point is that
>>>> you put the burden of certificate verification on developer's shoulder when
>>>> we can just leverage built in HTTPS support of the platform.
>>>>
>>>
>>> Platforms that support HTTPS but not certificate handling are rare - I
>>> know HTML5 is such a platform but such apps are inherently dependent on the
>>> server anyway and the server can just do the parsing and validation work
>>> itself. If WinRT is such a platform, OK, too bad.
>>>
>>> The embedding of the certificates is not arbitrary or pointless, by the
>>> way. It's there for a very good reason - it makes the signed payment
>>> request verifiable by third parties. Effectively you can store the signed
>>> message and present it later to someone else, it's undeniable. Combined
>>> with the transactions and merkle branches linking them to the block chain,
>>> what you have is a form of digital receipt ... a proof of purchase that can
>>> be automatically verified as legitimate. This has all kinds of use cases.
>>>
>>> Because of how HTTPS works, you can't easily prove to a third party that
>>> a server gave you a piece of data. Doing so requires staggeringly complex
>>> hacks (see tls notary) and when we designed BIP70, those hacks didn't even
>>> exist. So we'd lose the benefit of having a digitally signed request.
>>>
>>> Additionally, doing things this way means BIP70 requests can be signed
>>> by things which are not HTTPS servers. For example you can sign with an
>>> email address cert, an EV certificate i.e. a company, a certificate issued
>>> by some user forum, whatever else we end up wanting. Not every payment
>>> recipient can be identified by a domain name + dynamic session.
>>>
>>>
>>>> However, if you want to use your plateform's store, then you are toasted
>>>>
>>>
>>> That's a bit melodramatic. BitcoinJ is able to use the Android, JRE,
>>> Windows and Mac certificate stores all using the same code or very minor
>>> variants on it (e.g. on Mac you have to specify you want the system store
>>> but it's a one-liner).
>>>
>>> Yes, that's not *every* platform. Some will require custom binding glue
>>> and it depends what abstractions and languages you are using.
>>>
>>>
>>>> Have you tried to do that on windows RT and IOS ? I tried, and I
>>>> quickly stopped doing that since it is not worth the effort. (Frankly I am
>>>> not even sure you can on win rt, since the API is a stripped down version
>>>> of windows)
>>>>
>>>
>>> There is code to do iOS using the Apple APIs here:
>>>
>>>
>>> https://github.com/voisine/breadwallet/blob/master/BreadWallet/BRPaymentProtocol.m#L391
>>>
>>>
>>>> Why have you not heard about the problem ? (until now, because I have
>>>> this problem because I need to have the same codebase on
>>>> winrt/win/android/ios/tablets)
>>>>
>>>
>>> WinRT is a minority platform in the extreme, and all the other platforms
>>> you mentioned have the necessary APIs. Java abstracts you from them. So I
>>> think you are encountering this problem because you desire to target WinRT
>>> and other platforms with a single codebase. That's an unusual constraint.
>>>
>>> AFAIK the only other people who encountered this are BitPay, because
>>> they want to do everything in Javascript which doesn't really provide any
>>> major APIs.
>>>
>>>
>>>> Also, you bundle mozilla's store in bitcoinj, what happen when the
>>>> store change and your customer have not intent to use bitcoinj new version
>>>> ? by leveraging the plateform you benefit from automatic updates.
>>>>
>>>
>>> Yes, there are pros and cons to bundling a custom root store.
>>>
>>>
>>>> Also, does java stores deals with certificate revocations ? sure you
>>>> can theorically code that too... or just let the plateform deals with it.
>>>>
>>>
>>> It can do OCSP checks, yes, although I believe no wallets currently do
>>> so. A better solution would be to implement an OCSP stapling extension to
>>> BIP70 though.
>>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 9596 bytes --]