[bitcoin-dev] Currency/exchange rate information API

[bitcoin-dev] Currency/exchange rate information API

[Luke Dashjr]

Investigating what it would take to add fiat currency information to Bitcoin 
Knots, I noticed Electrum currently has many implementations, one for each 
exchange rate provider, due to lack of a common format for such data.

Therefore, I put together an initial draft of a BIP that could standardise 
this so wallets (or other software) and exchange rate providers can simply 
interoperate without a lot of overhead reimplementing the same thing many 
ways.

One thing I am unsure about, is that currently this draft requires using XBT 
(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those 
don't really have a pseudo-ISO currency code to fit in nicely...

Current draft here:
    https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki

Thoughts? Anything critical missing? Ways to make the interface better?

Luke

Re: [bitcoin-dev] Currency/exchange rate information API

[アルム カールヨハン]

I am surprised nothing like this exists already, so am all for the idea.

Maybe I am misunderstanding, but I'm not sure you want to have
thousand separators and other locale stuff in there. All currencies
including USD are often shown in Swedish using space or dot as
thousand separator and comma as decimal separator, e.g. "1.234,56 USD"
or "1 234,56 USD". I.e. this is something that the locale of the
user's environment defines, not something the server should have
opinions about. It is also not ideal to propose a given format based
on the user's locale, as some users have preferences for this (I
personally use US locale for numbers, and a US person who is visiting
Sweden wouldn't want this to suddenly change).

-Kalle.

On Sat, Mar 4, 2017 at 12:27 AM, Luke Dashjr via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Investigating what it would take to add fiat currency information to Bitcoin
> Knots, I noticed Electrum currently has many implementations, one for each
> exchange rate provider, due to lack of a common format for such data.
>
> Therefore, I put together an initial draft of a BIP that could standardise
> this so wallets (or other software) and exchange rate providers can simply
> interoperate without a lot of overhead reimplementing the same thing many
> ways.
>
> One thing I am unsure about, is that currently this draft requires using XBT
> (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those
> don't really have a pseudo-ISO currency code to fit in nicely...
>
> Current draft here:
>     https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki
>
> Thoughts? Anything critical missing? Ways to make the interface better?
>
> Luke
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Currency/exchange rate information API

[Tim Ruffing]

I'm not sure if a BIP is the right thing in that case, given that the
provided functionality is not special to Bitcoin and can be used in
other contexts as well. 

But ignoring this, the server should be authenticated at a
minimum. Otherwise manipulating exchange rates seems to be a nice
way for the attacker on the wire to make money...

Apart from that, my feeling is that it could be simplified. Is 
longpolling useful? And is the historical rate thing really necessary
for typical applications?

If yes, the client should be allowed to decide on which time scale the
data should be. (tick, min, hour, day, ...) That goes together with
clearly defining the type field (something like low, high, open, close,
but without flexibility). Think of a candle-stick chart basically.

Also, pushing may be more appropriate for "current" rates than polling.
Then no polling interval is necessary. On the other hand, this adds
complexity in other places, e.g., state.

Tim  

On Sat, 2017-03-04 at 08:27 +0000, Luke Dashjr via bitcoin-dev wrote:
> Investigating what it would take to add fiat currency information to
> Bitcoin 
> Knots, I noticed Electrum currently has many implementations, one for
> each 
> exchange rate provider, due to lack of a common format for such data.
> 
> Therefore, I put together an initial draft of a BIP that could
> standardise 
> this so wallets (or other software) and exchange rate providers can
> simply 
> interoperate without a lot of overhead reimplementing the same thing
> many 
> ways.
> 
> One thing I am unsure about, is that currently this draft requires
> using XBT 
> (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis,
> but those 
> don't really have a pseudo-ISO currency code to fit in nicely...
> 
> Current draft here:
>     https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.me
> diawiki
> 
> Thoughts? Anything critical missing? Ways to make the interface
> better?
> 
> Luke
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Currency/exchange rate information API

[Luke Dashjr]

On Monday, March 06, 2017 5:37:24 AM Tim Ruffing via bitcoin-dev wrote:
> But ignoring this, the server should be authenticated at a
> minimum. Otherwise manipulating exchange rates seems to be a nice
> way for the attacker on the wire to make money...

HTTPS would be used for that. It's not something that needs to be at a higher 
layer.

> Apart from that, my feeling is that it could be simplified. Is 
> longpolling useful?

I would think so, at least for Bitcoin since rates can change significantly in 
a short period of time (or can they anymore? I haven't really watched lately.)

> And is the historical rate thing really necessary for typical applications?

When displaying historical transactions, it doesn't really make sense to use 
the current market rate, but rather the market rate at the time the payment 
was made. While wallets might simply cache it with the transaction, it would 
be perhaps nicer if it could be automatically restored for seed-only 
recoveries. In any case, if a service/wallet doesn't want to provide/use 
historical information, it can simply not implement that part.

> If yes, the client should be allowed to decide on which time scale the
> data should be. (tick, min, hour, day, ...) That goes together with
> clearly defining the type field (something like low, high, open, close,
> but without flexibility). Think of a candle-stick chart basically.

How is the current draft insufficient for this?

> Also, pushing may be more appropriate for "current" rates than polling.
> Then no polling interval is necessary. On the other hand, this adds
> complexity in other places, e.g., state.

Pushing is what longpolling does.

Luke

Re: [bitcoin-dev] Currency/exchange rate information API

[Jonas Schnelli]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]


I like the BIP. It can reduce workload during implementation on both sides of the API and it allows to show the user more data without implementing tons of proprietary APIs.
It’s not directly Bitcoin specific (example: BIP32 is also not Bitcoin specific), but I think a BIP is the right way for this.


> 
>> Apart from that, my feeling is that it could be simplified. Is
>> longpolling useful?
> 
> I would think so, at least for Bitcoin since rates can change significantly in
> a short period of time (or can they anymore? I haven't really watched lately.)

Long polling is a simple push concept that works on most type of network configurations (NAT, proxy, etc.).
The only concern I see here is that an public API will quickly fill up the maximum allowed httpd connections.
But it’s solvable.

> 
>> And is the historical rate thing really necessary for typical applications?
> 
> When displaying historical transactions, it doesn't really make sense to use
> the current market rate, but rather the market rate at the time the payment
> was made. While wallets might simply cache it with the transaction, it would
> be perhaps nicer if it could be automatically restored for seed-only
> recoveries. In any case, if a service/wallet doesn't want to provide/use
> historical information, it can simply not implement that part.

I’m also not sure how useful historical datapoint are. I don’t think the use case where someone wants to restore from a seed and get all exchange rates during the time of the payment is something users are looking for.
However, It’s optional.

> 
>> If yes, the client should be allowed to decide on which time scale the
>> data should be. (tick, min, hour, day, ...) That goes together with
>> clearly defining the type field (something like low, high, open, close,
>> but without flexibility). Think of a candle-stick chart basically.
> 
> How is the current draft insufficient for this?
> 
>> Also, pushing may be more appropriate for "current" rates than polling.
>> Then no polling interval is necessary. On the other hand, this adds
>> complexity in other places, e.g., state.
> 
> Pushing is what longpolling does.

Agree with Luke. A „real“ push (though I’d say long-polling is the real push, AFAIK it’s also the technique behind Apple’s iOS push channel) would require a complex server setup that complicates many things like load-balancer, mem-caching, etc.


</jonas>


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [bitcoin-dev] Currency/exchange rate information API

[Tim Ruffing]

On Mon, 2017-03-06 at 07:09 +0000, Luke Dashjr wrote:
> On Monday, March 06, 2017 5:37:24 AM Tim Ruffing via bitcoin-dev
> wrote:
> > But ignoring this, the server should be authenticated at a
> > minimum. Otherwise manipulating exchange rates seems to be a nice
> > way for the attacker on the wire to make money...
> 
> HTTPS would be used for that. It's not something that needs to be at
> a higher 
> layer.
Sure, HTTPS is the way to go. But I think that should be required or at
least noted in the BIP, because people could miss easily, e.g., "I
don't need TLS, all the data is public anyway."

> When displaying historical transactions, it doesn't really make sense
> to use 
> the current market rate, but rather the market rate at the time the
> payment 
> was made. While wallets might simply cache it with the transaction,
> it would 
> be perhaps nicer if it could be automatically restored for seed-only 
> recoveries. In any case, if a service/wallet doesn't want to
> provide/use 
> historical information, it can simply not implement that part.
> 
Having the rate at the time of payment is indeed very useful, yes.
However that requires just a single value per payment, and there is no
query that tells the server "give me the value closest to timestamp t"
or similar.
Of course the client can download and keep a large part of history and
extract the information on its own but I can imagine that not every
clients wants to do that, and also the client does not know in advance
the bounds (from, to) that it must query.

> > If yes, the client should be allowed to decide on which time scale
> the
> > data should be. (tick, min, hour, day, ...) That goes together with
> > clearly defining the type field (something like low, high, open,
> close,
> > but without flexibility). Think of a candle-stick chart basically.
> 
> How is the current draft insufficient for this?
> 
In the current draft the client or the server cannot specify
granularity. If the clients only wants one value per day but for an
entire year, then it has to perform many requests or download and
process a very large response.
Also, I think it's okay that the type field allows for arbitrary user-
defined values, but it should also have some precisely defined values
(e.g. the mentioned low/high/open/close/typical).
For example, it's not clear currently what "low" means for a timestamp
(as opposed to a time span). Is it the low of the entire day or the low
since the previous record or something different?  

One has to be careful not to add too much complexity though. As soon as
one moves away from timestamps to something like hours or days, all
kind of issues with timezone, daylight saving time etc. appear. Maybe a
simple way to let the client ask "give me one value for every interval
of 3600 seconds" or similar. 


> Pushing is what longpolling does.
> 
That makes a lot of sense, yes.


Tim

Re: [bitcoin-dev] Currency/exchange rate information API

[Tim Ruffing]

On Mon, 2017-03-06 at 16:54 -0500, Tim Ruffing via bitcoin-dev wrote:
> 
> > Pushing is what longpolling does.
> > 
> 
> That makes a lot of sense, yes.
> 
Forgot one thing:
For longpolling, maybe we would like to have the possibility to request
some periodic message from the server. Otherwise clients cannot
distinguish between the situations 1. "value is still in the requested
bounds (minrate, maxrate)" and 2. "connection has dropped". So the user
may take a wrong decision because  he assumed that the value is still
in bounds holds but actually the server has died.

Tim 

Re: [bitcoin-dev] Currency/exchange rate information API

[Luke Dashjr]

On Monday, March 06, 2017 9:54:16 PM Tim Ruffing wrote:
> Having the rate at the time of payment is indeed very useful, yes.
> However that requires just a single value per payment, and there is no
> query that tells the server "give me the value closest to timestamp t"
> or similar.
> Of course the client can download and keep a large part of history and
> extract the information on its own but I can imagine that not every
> clients wants to do that, and also the client does not know in advance
> the bounds (from, to) that it must query.

It would be a privacy leak to request only the specific timestamps, but I 
suppose many wallets lack even basic privacy to begin with.

To address the bounds issue, I have specified that when from/to don't have an 
exact record, that the previous/next (respectively) is provided.

Hopefully this addresses both concerns?

> In the current draft the client or the server cannot specify
> granularity. If the clients only wants one value per day but for an
> entire year, then it has to perform many requests or download and
> process a very large response.

That's what the "timedelta" field solves, no?
If you want one value per day, you'd put 86400.

> Also, I think it's okay that the type field allows for arbitrary user-
> defined values, but it should also have some precisely defined values
> (e.g. the mentioned low/high/open/close/typical).
> For example, it's not clear currently what "low" means for a timestamp
> (as opposed to a time span). Is it the low of the entire day or the low
> since the previous record or something different?

Is it not sufficient for the server to specify this in the description of the 
given currency-pair feed?

Luke

Re: [bitcoin-dev] Currency/exchange rate information API

[Luke Dashjr]

On Monday, March 06, 2017 10:02:53 PM Tim Ruffing via bitcoin-dev wrote:
> For longpolling, maybe we would like to have the possibility to request
> some periodic message from the server. Otherwise clients cannot
> distinguish between the situations 1. "value is still in the requested
> bounds (minrate, maxrate)" and 2. "connection has dropped". So the user
> may take a wrong decision because  he assumed that the value is still
> in bounds holds but actually the server has died.

That's the job of TCP.
Do you think we need to explicitly specify a keepalive configuration?
Seems like that would vary based on client or network.

Re: [bitcoin-dev] Currency/exchange rate information API

[Tim Ruffing]

On Mon, 2017-03-06 at 22:14 +0000, Luke Dashjr wrote:
> It would be a privacy leak to request only the specific timestamps,
> but I 
> suppose many wallets lack even basic privacy to begin with.
> 
> To address the bounds issue, I have specified that when from/to don't
> have an 
> exact record, that the previous/next (respectively) is provided.
> 
> Hopefully this addresses both concerns?
Yes, that solves it. You're right with the privacy concern however.

> 
> > In the current draft the client or the server cannot specify
> > granularity. If the clients only wants one value per day but for an
> > entire year, then it has to perform many requests or download and
> > process a very large response.
> 
> That's what the "timedelta" field solves, no?
> If you want one value per day, you'd put 86400.
Oh sure, I had overlooked that.

> 
> > Also, I think it's okay that the type field allows for arbitrary
> user-
> > defined values, but it should also have some precisely defined
> values
> > (e.g. the mentioned low/high/open/close/typical).
> > For example, it's not clear currently what "low" means for a
> timestamp
> > (as opposed to a time span). Is it the low of the entire day or the
> low
> > since the previous record or something different?
> 
> Is it not sufficient for the server to specify this in the
> description of the 
> given currency-pair feed?
That works but a standardized way of indicating that piece of
information to the client is useful. Then the client can display a
"connection status" to the user, e.g., an "possible out-of-date"
warning like the warning sign in the Qt GUI when Bitcoin Core is
catching up the network.


Tim

Re: [bitcoin-dev] Currency/exchange rate information API

[Tim Ruffing]

On Mon, 2017-03-06 at 17:30 -0500, Tim Ruffing via bitcoin-dev wrote:
> 
> That works but a standardized way of indicating that piece of
> information to the client is useful. Then the client can display a
> "connection status" to the user, e.g., an "possible out-of-date"
> warning like the warning sign in the Qt GUI when Bitcoin Core is
> catching up the network.

Wait, forget this reply, I mixed up the two issues of keepalive and
definition of low, high etc... -.-

1. Keepalive for longpolling:
As I said, this can be useful for an out-of-date warning. I don't know
if this is better solved with TCP keepalive or on the higher layer.

2. Definition of low, high:
My feeling is that there is nothing wrong with providing exact
definitions in the BIP, i.e.., giving up the flexibility does not too
hurt much. However all of this is a minor issue after all.

Tim

Re: [bitcoin-dev] Currency/exchange rate information API

[Andreas Schildbach]

On 03/06/2017 06:37 AM, Tim Ruffing via bitcoin-dev wrote:

> And is the historical rate thing really necessary
> for typical applications?

Yes, it is. Basically all incoming transactions are historical, as your
wallet is likely not online when it happens.

Re: [bitcoin-dev] Currency/exchange rate information API

[Marcel Jamin]

> Why are multiple results separated by a line-feed rather than using a JSON Array?
> * Clients ought to cache historical data, and using a line-feed format allows them to simply append to a cache file.
> * Parsing JSON typically requires the entire data parsed together as a single memory object. Using simple lines to separate results, however, allows parsing a single result at a time.
> What if long descriptions require line and paragraph breaks?
> * Clients should word-wrap long lines, and JSON escapes newlines as "\n" which can be used doubly ("\n\n") for paragraph breaks.

I'd file this under premature optimization at the cost of
interoperability. If you use JSON, then please use it properly.

I'd also say it's the job of the parser to offer a way of doing that,
e.g. in .NET you can easily achieve that with Newtonsoft's JSON
parser: http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time.

On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> Investigating what it would take to add fiat currency information to Bitcoin
> Knots, I noticed Electrum currently has many implementations, one for each
> exchange rate provider, due to lack of a common format for such data.
>
> Therefore, I put together an initial draft of a BIP that could standardise
> this so wallets (or other software) and exchange rate providers can simply
> interoperate without a lot of overhead reimplementing the same thing many
> ways.
>
> One thing I am unsure about, is that currently this draft requires using XBT
> (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those
> don't really have a pseudo-ISO currency code to fit in nicely...
>
> Current draft here:
>     https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki
>
> Thoughts? Anything critical missing? Ways to make the interface better?
>
> Luke
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Currency/exchange rate information API

[Andrew LeCody]

[-- Attachment #1: Type: text/plain, Size: 2783 bytes --]

This formatting of JSON isn't unheard of though, it's typically called JSON
Streaming[1]. As long as exchanges implementing the API actually follow the
BIP and keep one JSON object per line, it shouldn't be a problem to decode.

1. https://en.wikipedia.org/wiki/JSON_Streaming

On Tue, Mar 7, 2017 at 8:07 AM Marcel Jamin via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> > Why are multiple results separated by a line-feed rather than using a
> JSON Array?
> > * Clients ought to cache historical data, and using a line-feed format
> allows them to simply append to a cache file.
> > * Parsing JSON typically requires the entire data parsed together as a
> single memory object. Using simple lines to separate results, however,
> allows parsing a single result at a time.
> > What if long descriptions require line and paragraph breaks?
> > * Clients should word-wrap long lines, and JSON escapes newlines as "\n"
> which can be used doubly ("\n\n") for paragraph breaks.
>
> I'd file this under premature optimization at the cost of
> interoperability. If you use JSON, then please use it properly.
>
> I'd also say it's the job of the parser to offer a way of doing that,
> e.g. in .NET you can easily achieve that with Newtonsoft's JSON
> parser:
> http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time
> .
>
> On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
> >
> > Investigating what it would take to add fiat currency information to
> Bitcoin
> > Knots, I noticed Electrum currently has many implementations, one for
> each
> > exchange rate provider, due to lack of a common format for such data.
> >
> > Therefore, I put together an initial draft of a BIP that could
> standardise
> > this so wallets (or other software) and exchange rate providers can
> simply
> > interoperate without a lot of overhead reimplementing the same thing many
> > ways.
> >
> > One thing I am unsure about, is that currently this draft requires using
> XBT
> > (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but
> those
> > don't really have a pseudo-ISO currency code to fit in nicely...
> >
> > Current draft here:
> >
> https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki
> >
> > Thoughts? Anything critical missing? Ways to make the interface better?
> >
> > Luke
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 4983 bytes --]