From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-2.v43.ch3.sourceforge.com ([172.29.43.192] helo=mx.sourceforge.net) by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1TdiBD-00065b-Gf for bitcoin-development@lists.sourceforge.net; Wed, 28 Nov 2012 13:59:59 +0000 Received-SPF: pass (sog-mx-2.v43.ch3.sourceforge.com: domain of petertodd.org designates 62.13.148.110 as permitted sender) client-ip=62.13.148.110; envelope-from=pete@petertodd.org; helo=outmail148110.authsmtp.com; Received: from outmail148110.authsmtp.com ([62.13.148.110]) by sog-mx-2.v43.ch3.sourceforge.com with esmtp (Exim 4.76) id 1TdiB7-00051X-Az for bitcoin-development@lists.sourceforge.net; Wed, 28 Nov 2012 13:59:59 +0000 Received: from mail-c233.authsmtp.com (mail-c233.authsmtp.com [62.13.128.233]) by punt7.authsmtp.com (8.14.2/8.14.2/Kp) with ESMTP id qASCvJl7065364; Wed, 28 Nov 2012 12:57:19 GMT Received: from savin (206-248-185-49.dsl.teksavvy.com [206.248.185.49]) (authenticated bits=128) by mail.authsmtp.com (8.14.2/8.14.2/) with ESMTP id qASCvBiv006778 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 28 Nov 2012 12:57:14 GMT Date: Wed, 28 Nov 2012 07:57:10 -0500 From: Peter Todd To: Mike Hearn Message-ID: <20121128125710.GA9893@savin> References: <201211271703.39282.andyparkins@gmail.com> <201211271726.56370.andyparkins@gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="h31gzZEtNLTqOjlF" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Server-Quench: 22801040-395b-11e2-a49c-0025907707a1 X-AuthReport-Spam: If SPAM / abuse - report it at: http://www.authsmtp.com/abuse X-AuthRoute: OCd2Yg0TA1ZNQRgX IjsJECJaVQIpKltL GxAVKBZePFsRUQkR aQdMdAoUFloCAgsB AmQbWlZeVVp7WGY7 aQpXcwdZalRPVwB0 VEFWR1pVCwQmQG1i dVhsC19ydwNBf3Y+ ZkZjVngVXkZydUd9 Rk5JFmQDN3phaTUd TRFcflBJcANIexZF bVd/UyIMLwdSbGoL NQ4vNDcwO3BTJTpY RgYVKF8UXXNDJjct Qh0EAX0XB0oZQC40 K1QeMFkEG10YNhdQ eXonR1UZOBIJTTVX DkRABjNCYGMMTCcq FR9BNQAA X-Authentic-SMTP: 61633532353630.1021:706 X-AuthFastPath: 0 (Was 255) X-AuthSMTP-Origin: 206.248.185.49/587 X-AuthVirus-Status: No virus detected - but ensure you scan with your own anti-virus system. X-Spam-Score: -1.5 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain -0.0 SPF_PASS SPF: sender matches SPF record X-Headers-End: 1TdiB7-00051X-Az Cc: Bitcoin Dev Subject: Re: [Bitcoin-development] Payment Protocol Proposal: Invoices/Payments/Receipts X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Nov 2012 13:59:59 -0000 --h31gzZEtNLTqOjlF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 28, 2012 at 11:43:19AM +0100, Mike Hearn wrote: > Peter is correct that there are a few degrees of freedom in protobuf > serialization, though far fewer than with JSON. FWIW I re-read the specs again and turns out my memory was wrong. (I last looked at this about four months ago) Duplicated fields are handled in a defined manner, with the last field seen in the serialization being the one whose value is used. Again, repeated fields are treated as elements of a list, preserving order. It does raise the interesting question do the implementations that don't preserve order of unknown fields, preserve the order of multiple unknown fields, either repeated or not? > I'd like to think upstream would be open to resolving these > ambiguities. I gotta admit, I suspect they won't be that open. Protocol buffers was designed because Google needed a fast serialization method suitable for many different internal projects. Needing round-trip idempotence seems like a rare requirement to me, especially for internal use. > Re-serialization of an Invoice message in the Payment message is a > potential source of mistakes. There's no need to ever concatenate > these messages and alternative implementations that don't order > serialized fields by tag number are missing an important optimization, > so they could be fixed. The main issue is treatment of unknown fields. > If/when the Invoice message is extended with other fields that are > round-tripped through an old client, the data may get lost. JSON > doesn't help resolve that either, of course. There are a few > solutions: Well, actually you can take advantage of the message concatination ability of protocol buffers to extend a message by simply appending the new fields to the existing thus either defining new fields, or overriding old values as required. If you want to de-duplicate though you run into the problem all over again. On the other hand JSON handles this case fine too provided that your JSON implementation supports dictionary objects with arbitrary fields. Just use the object as is and the unknown fields will be re-serialized properly at the other end. Some implementations will have to be careful to handle collisions with existing keys in the namespace. (consider in Python what would happen if you mapped your object to a class instance, and the serialization included the key "__init__") That said, JSON is quite problematic with numbers. For instance, you have to be careful to keep integers represented as pure integers below what Javascript can handle, the maximum integer exactly representable in a double float, or the JSON won't be parsable in Javascript even if many other languages handle it fine. Protocol buffers is at least pretty explicit about what size integers are. > 1) Change the type of the Invoice field in Payment to be "bytes" and > set it to be the hash of the originally received binary Invoice > message. Downside, requires merchants to track all outstanding > invoices. > 2) Ask protobufs upstream to modify the spec/implementations so > ordering of unknown fields is specified. The Python implementation > could be extended to support them so Python implementors don't end up > with accidental message downgrades. > 3) Language of the spec could be changed to explicitly state that the > received Invoice may not be binary-identical to the one that was sent, > in the case of a client that incorrectly downgrades the message. Thus > you'd be expected to check what the Invoice was using merchant_data > which is opaque and could just be, eg, a database key on your own end. > 4) Instead of submitting the entire Invoice back to the merchant, just > the merchant_data could be in the Payment message. >=20 > Of the four options I prefer the last. What is the use case for > resubmitting the entire invoice anyway? Even if protobufs are improved Note that I think the SignedInvoice message itself is broken, because protobuf implementations have no reason to guarantee that they can give you the serialized bytes of the Invoice sub-message. It's a quite specific use-case that isn't needed for pretty much anything but crypto. FWIW I took a quick look at the official API's, C++, Java and Python, and as far as I can tell none of them support accessing the binary serialization of a message field other than by re-serializing the message. Really the invoice field should be declared as bytes serialized_invoice, as inconvenient as that is to work with. > so handling of round-tripping new messages through old [Python] > clients is more rigorous, some implementors will probably convert the > protobuf objects into some internal forms for whatever reason (or > serialize them to a database, etc) and they're very likely to mess up > the handling of unknown fields when they do it. Since the Payment message includes an *untrusted* Invoice that the vendor needs to authenticate the whole invoice no matter what on Payment reception. In many cases that implies they have to keep some sort of database of "quotes" or similar anyway as the client can change anything they want otherwise. Again that leads back to the argument of why not just stick with the merchant_dat as you suggest, which will usually be some short invoice number attached to a database? A vendor that wants to operation a stateless invoicing system can just stuff a HMAC-protected serialized invoice into the merchant_data I guess you could use a mutable invoice field as a way of achieving some sort of negotiation protocol, but I think it's better to stick to the original concept of just ensuring that the user is really paying the right amount to the right address. --=20 'peter'[:-1]@petertodd.org --h31gzZEtNLTqOjlF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBAgAGBQJQtgomAAoJEH+rEUJn5PoECCMH/i1bYVfMnt1h6z56jxZ73c4C 0W2nY3iB2F+B6hnvp4oI1o8Cfu44mOxgylZkqS+/flb2Zj8uew0LFQbx+o9SXlJr CXy4qHaBaZbaBM1JvfkPTwYcC7nSCwzHeNvWaXGTg0qJ+su5nSK8lR7DeZFnKJMK c77HIkmgStihl8GiJYKaPZROcUTcNmo5jzQedmZ3AelUIvQRQZYD0Zmvyx7SEAms 9EW8nxw20RwQZ4zYFIyeDWsYfO2kVf8n4oqy6tJvcJanspWF7DAf/XfLLWfNLsLL 2khi0POpAEk7t8exGb5YUP+G/ANnFgnF+dNz81yss8iFyluN2kWVA/LFSd5AFXA= =XYYi -----END PGP SIGNATURE----- --h31gzZEtNLTqOjlF--