[Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Matt Smith]

[-- Attachment #1.1: Type: text/plain, Size: 1952 bytes --]

Hey guys,

The crew at Gem is considering a new HD wallet path structure for our
wallets, which are coin-agnostic, that separates the coin_type field
into two fields as such:

m / purpose' / network' / asset_type' / account' / change / index

where network refers to the blockchain (0 - bitcoin, 1 - testnet3, 2 -
litecoin, etc) and the new asset_type refers to the kind of asset to be
held in accounts below that path (Open Assets, Omni, Counterparty).

The intent is to allow us to validate the address format, select the
appropriate daemon to scan for tokenized assets, and choose multiple
blockchain data sources (that may not know anything about token systems
running on the blockchain they expose) relevant to an HDNode in the
wallet using only information in the HDNode's path -- without having to
maintain an explicit mapping of coin_type -> network.

For example, we already have the issue of mapping network identifiers
because of the lack of standardization across cryptocurrency libraries
which ends up being ugly and obnoxious to maintain, i.e.

netcode_map = {
  testnet: testnet3,
  bitcoin_testnet: testnet3,
  testnet3: testnet3,
  XTN: testnet3, ...
}
netcode_i_want = netcode_map[netcode_returned_by_libwhatever]

We want to avoid maintaining a similar asset_type_to_blockchain mapping.
Additionally, it would be helpful for utxo selection to exclude utxos
tied to assets based on path.

BIP43 seems to suggest that we request a BIP number and publish an
informational BIP specifying the new purpose. If that's not appropriate,
then maybe we just need to publish the information in a blog post to
allow any wallet developers who want to to implement
import_from_gem_structure.

I was also wondering if anyone had previously suggested something
similar that I missed when cruising the mailing list archives on the
subject.

Thanks,
–
Matt Smith | Gem
https://gem.co | GH: @thedoctor

[-- Attachment #1.2: 0x63331857.asc --]
[-- Type: application/pgp-keys, Size: 2201 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Matt @ Envrin Group]

[-- Attachment #1: Type: text/plain, Size: 2958 bytes --]


Hi Matt,

I think your best bet is probably just push it out privately via blog 
post / Github, and see if it gains any traction with other developers.

I'm a little uncertain as to the relevance though.  All those variables 
(purpose, network, asset_type, account, change, index) need to be stored 
internally within the wallet database, as there's no way to retrieve the 
path used from just the address, correct?  In that case, what's the 
meaning of that exact path structure when a) it can't be retrieved from 
just the address, and b) the values will be stored internally within the 
wallet when you lookup the address.

Matt



On 06/20/2015 03:42 AM, Matt Smith wrote:
> Hey guys,
>
> The crew at Gem is considering a new HD wallet path structure for our
> wallets, which are coin-agnostic, that separates the coin_type field
> into two fields as such:
>
> m / purpose' / network' / asset_type' / account' / change / index
>
> where network refers to the blockchain (0 - bitcoin, 1 - testnet3, 2 -
> litecoin, etc) and the new asset_type refers to the kind of asset to be
> held in accounts below that path (Open Assets, Omni, Counterparty).
>
> The intent is to allow us to validate the address format, select the
> appropriate daemon to scan for tokenized assets, and choose multiple
> blockchain data sources (that may not know anything about token systems
> running on the blockchain they expose) relevant to an HDNode in the
> wallet using only information in the HDNode's path -- without having to
> maintain an explicit mapping of coin_type -> network.
>
> For example, we already have the issue of mapping network identifiers
> because of the lack of standardization across cryptocurrency libraries
> which ends up being ugly and obnoxious to maintain, i.e.
>
> netcode_map = {
>    testnet: testnet3,
>    bitcoin_testnet: testnet3,
>    testnet3: testnet3,
>    XTN: testnet3, ...
> }
> netcode_i_want = netcode_map[netcode_returned_by_libwhatever]
>
> We want to avoid maintaining a similar asset_type_to_blockchain mapping.
> Additionally, it would be helpful for utxo selection to exclude utxos
> tied to assets based on path.
>
> BIP43 seems to suggest that we request a BIP number and publish an
> informational BIP specifying the new purpose. If that's not appropriate,
> then maybe we just need to publish the information in a blog post to
> allow any wallet developers who want to to implement
> import_from_gem_structure.
>
> I was also wondering if anyone had previously suggested something
> similar that I missed when cruising the mailing list archives on the
> subject.
>
> Thanks,
> --
> Matt Smith | Gem
> https://gem.co | GH: @thedoctor
>
>
> ------------------------------------------------------------------------------
>
>
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development


[-- Attachment #2: Type: text/html, Size: 3792 bytes --]

Re: [Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Matt Smith]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

I'm not sure I understand your question about the need to store paths in
the wallet database -- there's no way to infer the path of an address
inside an HD wallet from the address alone (short of an exhaustive
search), and HD wallets need to store either the paths, addresses, or
both that have been previously derived/used to monitor the blockchain
usefully, but those facts aren't new or specific to this path format.

The motivation for this path structure over standard bip44 is that it
separates the concept of network (or which blockchain I'm using) and
coin_type (or what kind of thing I'm sending over that blockchain).

This is useful, for example, if I want to import a wallet into my
application and I know that an account was in use at

m/##'/0'/99'/0'

where 99 is the identifier for, say, counterparty - I only need to check
the addresses derived below that path for balances against
counterpartyd. It may be worth pointing out that I expect multisig HD
wallet imports to require master keys and a list of account paths – not
a list of addresses, as it's very possible that a new address could be
derived between the time when the wallet data was exported and when it
will be imported.

This use case might be very specific to our model, but the reason I
figured we should request a BIP # for this is that to start using it, we
need to pick a number for the purpose field and don't want to do it
arbitrarily (and risk having to change it later) or overload 44 (which
would be misleading).

Did I either  a) answer  or  b) misunderstand  your questions?
--
Matt Smith | Gem
https://gem.co | GH: @thedoctor



On 6/19/15 2:25 PM, Matt @ Envrin Group wrote:
> 
> Hi Matt,
> 
> I think your best bet is probably just push it out privately via blog
> post / Github, and see if it gains any traction with other developers.
> 
> I'm a little uncertain as to the relevance though.  All those variables
> (purpose, network, asset_type, account, change, index) need to be stored
> internally within the wallet database, as there's no way to retrieve the
> path used from just the address, correct?  In that case, what's the
> meaning of that exact path structure when a) it can't be retrieved from
> just the address, and b) the values will be stored internally within the
> wallet when you lookup the address.
> 
> Matt


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Andreas Petersson]

>m/##'/0'/99'/0'
>
>where 99 is the identifier for, say, counterparty


What is stopping you from using m/44'/9'/a'/c/i as descibed here:
http://doc.satoshilabs.com/slips/slip-0044.html

to avoid having an internal mapping from 9'-> 0' to find out what
blockchain to query, this sounds like it should be trivial for any wallet.

Re: [Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Matt Smith]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

> to avoid having an internal mapping from 9'-> 0' to find out what
> blockchain to query, this sounds like it should be trivial for any wallet.

Trivial to implement, a headache to *maintain*

But if a new platform is released on an existing blockchain, my wallet
doesn't need to know about the new magic number it claims in order to
handle it correctly.

Say I make a new token layer, BobCoin, which runs on bitcoin and say I
use an HD wallet and always generate new BobCoin token addresses as
m/##'/0'/808'/*'/*/*. If I import that wallet into older HD wallet
software that doesn't know anything about BobCoin, it will still:

- understand what blockchain to query for utxos on the addresses below
that path
- be able to generate valid BobCoin addresses without any updates

I think this is particularly valuable if you're developing against a
platform where updates can't be forced on clients.

To be clear: I am not suggesting this as a general-purpose successor to
BIP44.

–
Matt Smith | Gem
https://gem.co | GH: @thedoctor


On 6/19/15 5:57 PM, Andreas Petersson wrote:
>> m/##'/0'/99'/0'
>>
>> where 99 is the identifier for, say, counterparty
> 
> 
> What is stopping you from using m/44'/9'/a'/c/i as descibed here:
> http://doc.satoshilabs.com/slips/slip-0044.html
> 
> to avoid having an internal mapping from 9'-> 0' to find out what
> blockchain to query, this sounds like it should be trivial for any wallet.
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Matt @ Envrin Group]

Say you generate a child key using the path m/6'/4'/7'/99'/0/196, which 
is what your proposed path structure would be, and it results in the 
address 1DpY7PtPVURvjrGsdAjbZAZ7cL9GD8tc5w.

When the wallet notices a transaction in the blockchain that has 
1DpY7PtPVURvjrGsdAjbZAZ7cL9GD8tc5w as an output, it's going to have to 
lookup the address within its database to get the values 6/4/7/99/0/196, 
as there's no way to retrieve them from just the address.  So 
technically, you might as well just use m/account'/change/index if using 
hardened child keys, or m/change/index if not, as recommended, because 
the wallet will still function the exact same way.

Matt



On 06/20/2015 06:31 AM, Matt Smith wrote:
> I'm not sure I understand your question about the need to store paths in
> the wallet database -- there's no way to infer the path of an address
> inside an HD wallet from the address alone (short of an exhaustive
> search), and HD wallets need to store either the paths, addresses, or
> both that have been previously derived/used to monitor the blockchain
> usefully, but those facts aren't new or specific to this path format.
>
> The motivation for this path structure over standard bip44 is that it
> separates the concept of network (or which blockchain I'm using) and
> coin_type (or what kind of thing I'm sending over that blockchain).
>
> This is useful, for example, if I want to import a wallet into my
> application and I know that an account was in use at
>
> m/##'/0'/99'/0'
>
> where 99 is the identifier for, say, counterparty - I only need to check
> the addresses derived below that path for balances against
> counterpartyd. It may be worth pointing out that I expect multisig HD
> wallet imports to require master keys and a list of account paths – not
> a list of addresses, as it's very possible that a new address could be
> derived between the time when the wallet data was exported and when it
> will be imported.
>
> This use case might be very specific to our model, but the reason I
> figured we should request a BIP # for this is that to start using it, we
> need to pick a number for the purpose field and don't want to do it
> arbitrarily (and risk having to change it later) or overload 44 (which
> would be misleading).
>
> Did I either  a) answer  or  b) misunderstand  your questions?
> --
> Matt Smith | Gem
> https://gem.co | GH: @thedoctor
>
>
>
> On 6/19/15 2:25 PM, Matt @ Envrin Group wrote:
>> Hi Matt,
>>
>> I think your best bet is probably just push it out privately via blog
>> post / Github, and see if it gains any traction with other developers.
>>
>> I'm a little uncertain as to the relevance though.  All those variables
>> (purpose, network, asset_type, account, change, index) need to be stored
>> internally within the wallet database, as there's no way to retrieve the
>> path used from just the address, correct?  In that case, what's the
>> meaning of that exact path structure when a) it can't be retrieved from
>> just the address, and b) the values will be stored internally within the
>> wallet when you lookup the address.
>>
>> Matt

Re: [Bitcoin-development] Alternate HD path structure: BIP, blog, or wat?

[Jonas Schnelli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

 > m / purpose' / network' / asset_type' / account' / change / index
> 
> ...
> 
> I was also wondering if anyone had previously suggested something 
> similar that I missed when cruising the mailing list archives on
> the subject.


Hi Matt

a)
In my opinion, HD chain standards like Bip43/44 are there to allow
users to use different wallets with the same seed/wordlist (switch
over or use in parallel).
In practice, wallet re-importing / re-creating is not trivial and has
risks of missing out spent or spendable coins.
Take the current landscape. Electrum doesn’t follow the Bip39 standard
[1]. Breadwallet does not support Bip44 [2].
If users start to think, as long as they keep the Bip39 wordlist save,
nothing can happen, is a material fallacy (and i see a trend into this
direction).
A hd seed will not recover your non-blockchain-metadata. HD seed
recovery is a disaster recovery method.

Because of that I am unsure if standards of hd chainstructurs does lead
users and developers into the right directions.
In my opinion, reimporting a hd seed is a experts feature that requires
manual configuration.


b)
Creating deep level chains like m/a/b/c/d/e/f/g/h requires more cpu
power to derive private key with a given seed or wordlist.
If you are using hardware wallets like Trezor or Digitalbitbox, you
probably feel the required time when signing/deriving.

[1] https://bitcointalk.org/index.php?topic=1082586.0
[2] https://github.com/voisine/breadwallet/issues/131

</jonas>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVhTxdAAoJECnUvLZBb1Ps7HIP/00wqG9uEDGE77S6QsSmWWWm
OHYSlWzSeFURnkWyH4B1pcyGL9JxkHJMqmiHr5aBGIt0WGM8S1xwFImg7aFeWe8B
s5tjT80QwH6/RrGnrIFwdJp4QdAriVq7G0lfexika8O8LxW45LzP7BTQ1SoFVZ5o
mQuEzLG3A+NaKREqUMXBiSWAAuM9m08GBcRGsqd9tpvUUxqiJFHAbKqn2yVtuR3h
f+wJeQyjUf6ZqkIDor73lyjay6YpDyYveHaIxP9WfKJGvEVCVisuOXx0C3C7u5ps
SF1p2xCQUEC1xRVHySBgr0VdTYQlMpSVg7XOWSPRPXjfyBBwGW9aLZyGuDDTe0ca
Wa+u3CNRg3wIwsp3gG15wWK/M2eFPMKNwlE0DqSP9Siin0QlLRcuRe39D5BSjcsQ
xfQXNEf2gA0C7rMGA47K5fmxnB4qukkM8UqrOSqtczUO2AP1vdhhwJRi+/Cq9l1r
i1isL4cncwtf3/z965eN9vvou+a1dgU9L2gONmHp5eHwOtPh83u8BXEUof+VMUrN
B+vLHJMRRMucyASlehmzGe8EJiJ3Hlo4NjVpTNg/V8bnHY59lpm6GylvCa7YjR/d
xWB6jmG/7nqyRrJ0byIdZRGLtr/LuleIoFCwPUAO1jR4x626h0iVo82wPBPQXAJ1
IhkX/FW90s9i8vErMjwf
=To1w
-----END PGP SIGNATURE-----