[Bitcoin-development] Plans to separate wallet from core

[Bitcoin-development] Plans to separate wallet from core

[Jorge Timón]

I know there are plans to separate the wallet from the core code and I
think it's a great idea that will result in cleaner and more modular
software.
But it seems like my assumptions on how this would be done may be incorrect.

I was assuming that the wallet would consume data from a trusted
bitcoind core node using rpc or a better interface like an http rest
api (see PR #2844).
So the core would take care of the hard consensus stuff, and the
wallet would maintain its own database with private keys, addresses,
balances, etc. and would consume some data contained in bitcoind's
database.
I also assumed that the interface between wallet and core would
include queries to the UTXO (see PR #4351) and maybe TXO (see PR
#3652) for getting the historic balances.

As said, I'm not sure these assumptions are true anymore so I ask.
Is this the plan?
Is the plan that the wallet will use the p2p directly and maintain its
own chain database?

Well, it's something that generally interests me so if anyone can
detail the steps for separating the wallet a little bit, maybe I can
help with some of the steps.

Maybe there's no roadway yet. In that case I would like to advance
that discussion in this thread.

-- 
Jorge Timón

Re: [Bitcoin-development] Plans to separate wallet from core

[Wladimir]

On Mon, Jun 23, 2014 at 11:50 AM, Jorge Timón <jtimon@monetize.io> wrote:
> I know there are plans to separate the wallet from the core code and I
> think it's a great idea that will result in cleaner and more modular
> software.
> But it seems like my assumptions on how this would be done may be incorrect.
>
> I was assuming that the wallet would consume data from a trusted
> bitcoind core node using rpc or a better interface like an http rest
> api (see PR #2844).

It's least surprising if the wallet works as a SPV client by default.
Then, users can use it without first setting up a core. Thus the idea
would be to use P2P primarily.

There could be a mode to use a trusted core by RPC for
mempool/conflicted transaction validation and such. But I'm not sure
about this - as we've seen, pure-SPV wallets work pretty well. If you
want it to act as an edge router you can point a SPV wallet at your
trusted core as well.

There are no plans for adding Electrum-like functionality to bitcoind.
There is already Electrum. Let's not reinvent any wheels.

> So the core would take care of the hard consensus stuff, and the
> wallet would maintain its own database with private keys, addresses,
> balances, etc. and would consume some data contained in bitcoind's
> database.

Right, the wallet would keep track of those.

> I also assumed that the interface between wallet and core would
> include queries to the UTXO (see PR #4351) and maybe TXO (see PR
> #3652) for getting the historic balances.
>
> As said, I'm not sure these assumptions are true anymore so I ask.
> Is this the plan?
> Is the plan that the wallet will use the p2p directly and maintain its
> own chain database?

It does not need to keep a full chain database. But it needs its own
record of the chain, headers-only + what concerns the keys in the
wallet.

Wladimir

Re: [Bitcoin-development] Plans to separate wallet from core

[Jorge Timón]

On 6/23/14, Wladimir <laanwj@gmail.com> wrote:
> It's least surprising if the wallet works as a SPV client by default.
> Then, users can use it without first setting up a core. Thus the idea
> would be to use P2P primarily.

So first bitcoind will support SPV mode then we separate the wallet?
Are the core and the wallet share any code (say, the p2p messages via
a sub-repo or something)?

> There could be a mode to use a trusted core by RPC for
> mempool/conflicted transaction validation and such. But I'm not sure
> about this - as we've seen, pure-SPV wallets work pretty well. If you
> want it to act as an edge router you can point a SPV wallet at your
> trusted core as well.

I thought we would first separate wallet from core (maintaining the
full-node wallet status) and then implement an optional SPV mode for
the core (and transitively for "qt-wallet", which would support both
full and SPV mode).

> There are no plans for adding Electrum-like functionality to bitcoind.
> There is already Electrum. Let's not reinvent any wheels.

I'm sorry, but I still don't know what Electrum has to do with all this.
Bitcoin companies often like to interface with the network via
bitcoind nodes, what's wrong with their custom wallets consuming some
optional indexes from a bitcoind node their run themselves?
I'm not proposing a "bitcoind-client" similar to a electrum-client. I
thought it was assumed you where still going to run both the core and
the wallet and we just wanted to separate the code for better
modularity. Seriously, Mike also said something about Electrum too and
I'm really lost about what you people mean here.

> It does not need to keep a full chain database. But it needs its own
> record of the chain, headers-only + what concerns the keys in the
> wallet.

Why cannot consume that data from a bitcoind node that always run alongside it?

I still don't get the plan but it feels like it won't look as DRY as I
was expecting...

Re: [Bitcoin-development] Plans to separate wallet from core

[Wladimir]

On Mon, Jun 23, 2014 at 10:15 PM, Jorge Timón <jtimon@monetize.io> wrote:
> On 6/23/14, Wladimir <laanwj@gmail.com> wrote:
>> It's least surprising if the wallet works as a SPV client by default.
>> Then, users can use it without first setting up a core. Thus the idea
>> would be to use P2P primarily.
>
> So first bitcoind will support SPV mode then we separate the wallet?
> Are the core and the wallet share any code (say, the p2p messages via
> a sub-repo or something)?

Yes, they will share code. At least the basic data structures and
serialization and deserialization. Probably also part of the network
code and utilities like argument parsing (although that's not a hard
requirement - it may be an opportunity to do things better).

So part of Bitcoin Core will have to be turned into a library or
libraries. Movement toward this is already in progress for a while.

>> There could be a mode to use a trusted core by RPC for
>> mempool/conflicted transaction validation and such. But I'm not sure
>> about this - as we've seen, pure-SPV wallets work pretty well. If you
>> want it to act as an edge router you can point a SPV wallet at your
>> trusted core as well.
>
> I thought we would first separate wallet from core (maintaining the
> full-node wallet status) and then implement an optional SPV mode for
> the core (and transitively for "qt-wallet", which would support both
> full and SPV mode).

We want to move away from "full node wallets". In the beginning it
made some sense to jump-start the network, but the more the chain
grows the less unwieldy they become.

My main argument for the split is that full nodes and wallets have
completely different usage scenarios:

- A wallet should be online as little as possible, ideally only when
you do transactions or want to check for them.

- A full node should be online 24/7 or it is virtually useless to the network.

>> There are no plans for adding Electrum-like functionality to bitcoind.
>> There is already Electrum. Let's not reinvent any wheels.
>
> I'm sorry, but I still don't know what Electrum has to do with all this.

I suggest you look at the interface for Electrum. It probably does
exactly what you expected the interface between the Bitcoin Core
wallet and Bitcoin Core to become. Electrum server keeps some extra
indices that can be queried by the wallets. It already exists!

But IMO this is a passed stage. SPV wallets w/ Bloom filtering can
work without any special servers, so why require a 'close binding' to
a trusted bitcoin core?

(As said - I'm fine with optional close binding to a core using RPC
with slight security benefits like utxo queries and conflicted
transaction checking, and to get the dynamic fee data, but it should
not be required)

>> It does not need to keep a full chain database. But it needs its own
>> record of the chain, headers-only + what concerns the keys in the
>> wallet.
>
> Why cannot consume that data from a bitcoind node that always run alongside it?

To not require having a bitcoind node always running alongside it.

Wladimir

Re: [Bitcoin-development] Plans to separate wallet from core

[Wladimir]

> But IMO this is a passed stage. SPV wallets w/ Bloom filtering can
> work without any special servers, so why require a 'close binding' to
> a trusted bitcoin core?

To clarify (and not piss off ThomasV :-): I do not think the idea of
having servers with a reputation of their own is a passed stage. There
are many things that cannot be done at SPV level security with just
the P2P protocol yet. So having fewer but more trusted Electrum
servers is a reasonable compromise.

But for basic wallet functionality it isn't necessary.

Wladimir

Re: [Bitcoin-development] Plans to separate wallet from core

[Thomas Voegtlin]

Le 24/06/2014 11:44, Wladimir a écrit :
>> But IMO this is a passed stage. SPV wallets w/ Bloom filtering can
>> work without any special servers, so why require a 'close binding' to
>> a trusted bitcoin core?
> 
> To clarify (and not piss off ThomasV :-): I do not think the idea of
> having servers with a reputation of their own is a passed stage. There
> are many things that cannot be done at SPV level security with just
> the P2P protocol yet. So having fewer but more trusted Electrum
> servers is a reasonable compromise.
> 


Thanks for that :)

Note that my goal is to make the Electrum servers as trustless as
possible, and not to rely on some sort of 'reputation'.

Thomas

Re: [Bitcoin-development] Plans to separate wallet from core

[Justus Ranvier]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/24/2014 09:07 AM, Wladimir wrote:
> My main argument for the split is that full nodes and wallets have 
> completely different usage scenarios:
> 
> - A wallet should be online as little as possible, ideally only
> when you do transactions or want to check for them.
> 
> - A full node should be online 24/7 or it is virtually useless to
> the network.

I think btcd has done this right.

A wallet is a daemon that runs constantly in the background, just like
the full node.

The GUI (which is distinct from the wallet) runs as little as
possible. Presumably there's no need for a 1:1 relationship between
wallets and GUIs.


- -- 
Support online privacy by using email encryption whenever possible.
Learn how here: http://www.youtube.com/watch?v=bakOKJFtB-k
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJTqZpVAAoJEMP3uyY4RQ21E48H/0XNYBzR8QZjfku/MeL5IbwL
A56jrlWe2EZTabwfKdGx12L5oeBXe3DOeLsTizqIu0vijcl7qQryU49AjrVe91Rx
OdEi4lmaiXdkM3lWeWUxLoLLHR1B+1f8T18Mrnzo+yasyrywPb+6H79VN5KERdA2
5yHYCZyHXdNoXpzyf38GC2PdYJmAZdrRfAGyC5+OXSwE3XLhpRBrSBh/mrx0ct5M
ghkCKtsmJCJJ6sR2TbFxaj71kPp0J0tp8JVvjVEqC2uB4Ih0NY+79kz8L81TaNmw
ol1o6p7TypIk7QRQ4ES3Fq0ALh2aQ/tX4rc0GC0ed+xLi+dHJ2wGBI37HoyGIyg=
=Nn9X
-----END PGP SIGNATURE-----

[-- Attachment #2: 0x38450DB5.asc --]
[-- Type: application/pgp-keys, Size: 12464 bytes --]

Re: [Bitcoin-development] Plans to separate wallet from core

[Jorge Timón]

On 6/24/14, Justus Ranvier <justusranvier@gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 06/24/2014 09:07 AM, Wladimir wrote:
>> My main argument for the split is that full nodes and wallets have
>> completely different usage scenarios:
>>
>> - A wallet should be online as little as possible, ideally only
>> when you do transactions or want to check for them.
>>
>> - A full node should be online 24/7 or it is virtually useless to
>> the network.
>
> I think btcd has done this right.
>
> A wallet is a daemon that runs constantly in the background, just like
> the full node.
>
> The GUI (which is distinct from the wallet) runs as little as
> possible. Presumably there's no need for a 1:1 relationship between
> wallets and GUIs.

I think he means that the wallet shouldn't be running as much as it is
currently doing.
But yes, I think you're right about wallets and GUIs not necessarily
mapping 1:1.

Re: [Bitcoin-development] Plans to separate wallet from core

[Wladimir]

On Tue, Jun 24, 2014 at 6:40 PM, Jorge Timón <jtimon@monetize.io> wrote:
>
> I think he means that the wallet shouldn't be running as much as it is
> currently doing.
> But yes, I think you're right about wallets and GUIs not necessarily
> mapping 1:1.

I haven't been talking about the GUI at all in this entire
conversation. When I say wallet, I mean the wallet code. There are
very few reasons for running a wallet 24/7. One could be if you're
running an exchange that has to send and receive transactions all the
time.

But that's an exception - in the usual merchant/end-user case you're
mostly receiving transactions and sometimes manually or batched-
sending them. No need to leave the wallet running, exposed to the
network outside those times.

How the wallet GUI (if any) relates to the wallet code is a completely
different topic, and a much less pressing matter. Anything from 'GUI
uses wallet as a library' (multibit, electrum, bitcoin core) to
elaborate client-server protocols (btcd, coinvault?) are acceptable
depending on the use case.

Wladimir

Re: [Bitcoin-development] Plans to separate wallet from core

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

>
> So first bitcoind will support SPV mode then we separate the wallet?
> Are the core and the wallet share any code (say, the p2p messages via
> a sub-repo or something)?
>

bitcoind already supports SPV mode, that's how bitcoinj clients work.
However the current wallet code doesn't use it, it integrates directly with
the full mode main loop and doesn't talk P2P internally. Which is the fine
and obvious way to implement the wallet feature. I'm not totally convinced
it should become an SPV wallet given the complexity of doing that. But if
you did want to separate the wallet code from the full node then that'd be
the way to do it.

The question is; what does this buy us, and is it worth the potentially
huge amount of time it could take? My gut feeling is we have bigger fish to
fry. There's plenty of work to do just on the core consensus code, making
Bitcoin Core into a competitive wallet as well would be an additional
burden.

However I may be quite biased, as I am the maintainer of what is primarily
a wallet library :)


> I'm sorry, but I still don't know what Electrum has to do with all this.
>

People use Electrum as shorthand to mean "something a bit like the P2P
network, but with trusted remote servers which build additional databases
and thus support additional commands".

[-- Attachment #2: Type: text/html, Size: 1798 bytes --]

Re: [Bitcoin-development] Plans to separate wallet from core

[Wladimir]

> The question is; what does this buy us, and is it worth the potentially huge
> amount of time it could take? My gut feeling is we have bigger fish to fry.
> There's plenty of work to do just on the core consensus code, making Bitcoin
> Core into a competitive wallet as well would be an additional burden.

I don't intend to work on that myself but that's up to the people that
want to contribute to that. Once it's a separate project it could
either be a big success, or it could slowly wither away. It can have a
release cycle separate from the node. Likely faster.

The organizational reason to split off the wallet is to get rid of
that responsibility (and code) from the bitcoind repo. Maintaining a
wallet should not be part of maintaining the core infrastructure. But
just deleting it would be unreasonable.

> However I may be quite biased, as I am the maintainer of what is primarily a wallet library :)

Hah. I've thought about that migration path as well.

From my experience the main thing people are missing with BitcoinJ is
a quick and easy way to set up a wallet as a daemon, to use the
functionality from non-java through RPC.

But there are other interesting upcoming wallet projects as well, for
example CoinVault.

Wladimir

Re: [Bitcoin-development] Plans to separate wallet from core

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

>
> From my experience the main thing people are missing with BitcoinJ is
> a quick and easy way to set up a wallet as a daemon, to use the
> functionality from non-java through RPC.


Yes. I'd love to have a mostly Core compatible JSON-RPC frontend. Most of
my current users are happy using it as a library though. A lot of popular
languages can run directly on the JVM these days. The big ones we miss are
C++ and PHP, I think. But you can use JavaScript, Python 2.7, Lisp, Ruby,
along with other less well known ones.

The other good reason to have JSON-RPC support would be to reuse the Core
regression tests.

Anyway, this is off topic :)

[-- Attachment #2: Type: text/html, Size: 926 bytes --]

Re: [Bitcoin-development] Plans to separate wallet from core

[Jorge Timón]

On 6/24/14, Mike Hearn <mike@plan99.net> wrote:
> bitcoind already supports SPV mode, that's how bitcoinj clients work.
> However the current wallet code doesn't use it, it integrates directly with
> the full mode main loop and doesn't talk P2P internally. Which is the fine
> and obvious way to implement the wallet feature. I'm not totally convinced
> it should become an SPV wallet given the complexity of doing that. But if
> you did want to separate the wallet code from the full node then that'd be
> the way to do it.
>
> The question is; what does this buy us, and is it worth the potentially
> huge amount of time it could take? My gut feeling is we have bigger fish to
> fry. There's plenty of work to do just on the core consensus code, making
> Bitcoin Core into a competitive wallet as well would be an additional
> burden.

Exactly, this is part of my point, the qt-wallet does not support SPV
operation at this point, and that complex work should be done after
the wallet is separated. Thus the first version of the separated
wallet should be functionally equivalent to today's wallet, that is, a
full node wallet (even though I understand Wladimir's arguments
against full-node wallets).

>> I'm sorry, but I still don't know what Electrum has to do with all this.
>>
>
> People use Electrum as shorthand to mean "something a bit like the P2P
> network, but with trusted remote servers which build additional databases
> and thus support additional commands".

Ok, thanks. So a bitcoin core node which maintains and serves
additional indexes (say, an utxo index via rpc or something else) is
referred to as "an electrum" even though it doesn't use
electrum-server. Strange, but now I get it.

So in summary:

1) I accept the library approach over the "core-wallet protocol".

2) That doesn't necessarily mean that optionally maintaining
additional indexes in the core is not interesting for some use cases
(interesting for bitcoind, I don't care much if electrum-server
currently does this and more [with more dependencies]). Although
Pieter thinks that should also be separated into an "index node" too,
but I think that's another discussion.

3) The wallet doesn't currently operate as SPV and that work should be
done (if there's enough interest) only after the wallet is separated
from the core.

Re: [Bitcoin-development] Plans to separate wallet from core

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 373 bytes --]

I think there are three typical uses:

1. Building consensus on the block chain. This is what the core is for.
2. Single user wallets. This is where SPV alone is good.
3. Services e.g. exchange, payment processor .... This is where core + indexing server talking SPV to core is the right choice

Regards,

Tamás Blummer
Founder, CEO

http://bitsofproof.com



[-- Attachment #1.2.1: Type: text/html, Size: 4899 bytes --]

[-- Attachment #1.2.2: email.png --]
[-- Type: image/png, Size: 7120 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [Bitcoin-development] Plans to separate wallet from core

[Jorge Timón]

On 6/24/14, Tamas Blummer <tamas@bitsofproof.com> wrote:
> 3. Services e.g. exchange, payment processor .... This is where core +
> indexing server talking SPV to core is the right choice

I think this is my main question, what's the advantage of having the
processes talking via the p2p protocol instead of something more
direct when you control both processes?

Wladimir, of course headers-first is generally good, and of course
nobody will be force to separate the code in a way he doesn't like, I
was just testing the waters to see what people had in mind, since I
realized the ideas could be more different than I had assumed.
I don't have any issues with electrum, I'm just not convinced by the
arguments that say that the indexes must be necessarily out of the
core, specially when some of them could be committed in the future.
So I'm all in favor of modularity and competing codebases, I'm just
not convinced that the "core full-node" must be necessarily restricted
to validation only (for example, I think it should maintain a minimal
and non-optimized mining functionality,even if it's only used for
testing purposes).

So far it is great that everybody seems to agree that the wallet code
needs to be separated.

Thanks everyone for sharing your views on the subject.

Re: [Bitcoin-development] Plans to separate wallet from core

[Pieter Wuille]

I'd like to point out that there is quite a difference between "what
core nodes should be like" and "what the codebase core nodes are built
from must support".

Given sufficiently modularized code (which I think everyone seems to
be in favor of, regardless of the goals), you can likely build a
binary that does full verification and maintains some indexes of some
sort.

I still believe that what we push for to run as the core nodes of the
network should aim for purely verification and relay, and nothing
else, but people can and will do things differently if the source code
allows it. And that's fine.

On Tue, Jun 24, 2014 at 3:26 PM, Jorge Timón <jtimon@monetize.io> wrote:
> I think this is my main question, what's the advantage of having the
> processes talking via the p2p protocol instead of something more
> direct when you control both processes?

IMHO, maintaining a correct view of the current state of the chain
(excluding blocks, just headers) is already sufficiently hard (I hope
that everyone who ever implemented an SPV wallet can agree). You
simplify things a bit by not needing to verify what the peer claims if
you trust them, but not much. You still need to support
reorganizations, counting confirmations, making sure you stay
up-to-date. These are functions the (SPV) P2P protocol has already
shown to do well, and there are several codebases out there that
implement it. No need to reinvent the wheel with a marginally more
efficient protocol, if it means starting over everything else.

-- 
Pieter

Re: [Bitcoin-development] Plans to separate wallet from core

[Wladimir]

On Tue, Jun 24, 2014 at 1:29 PM, Jorge Timón <jtimon@monetize.io> wrote:
> On 6/24/14, Mike Hearn <mike@plan99.net> wrote:
ou did want to separate the wallet code from the full node then that'd be
>> the way to do it.
>
> Exactly, this is part of my point, the qt-wallet does not support SPV
> operation at this point, and that complex work should be done after
> the wallet is separated. Thus the first version of the separated
> wallet should be functionally equivalent to today's wallet, that is, a
> full node wallet (even though I understand Wladimir's arguments
> against full-node wallets).

Do mind that some of the steps on the path of bitcoind towards SPV are
also useful in general. For example, headers-first allows parallel
block download, which would solve a lot of sync issues people are
having - a much higher priority than the wallet.

But anyhow I'm describing how I would do it. If you're going to do it,
you can do it in any order that you want. As we're talking about a
separate project here it's not even clear who will be maintainer.

> 2) That doesn't necessarily mean that optionally maintaining
> additional indexes in the core is not interesting for some use cases
> (interesting for bitcoind, I don't care much if electrum-server
> currently does this and more [with more dependencies]). Although
> Pieter thinks that should also be separated into an "index node" too,
> but I think that's another discussion.

I don't understand your argument against Electrum here. Dependencies?
Come on, that's a matter of software distribution. If that really
bothers you I suppose you could contribute to Electrum server so that
it has less deps. It doesn't make the protocol worth any less.

Although Pieter and I disagree with regard to issue #4351, we agree on
wanting to keep (or at least making) bitcoind as lean as possible.
Maintaining extra indices for others doesn't fit in there - that's
also why the address index patch was not merged. An 'index node' could
be a different animal.

Wladimir

Re: [Bitcoin-development] Plans to separate wallet from core

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

>
> Although Pieter and I disagree with regard to issue #4351, we agree on
> wanting to keep (or at least making) bitcoind as lean as possible.
> Maintaining extra indices for others doesn't fit in there - that's
> also why the address index patch was not merged. An 'index node' could
> be a different animal.


We definitely want to head in the direction of allowing a p2p node to be as
useful as possible within its resource constraints and optional advertising
of new (expensive) indexes is the way to go.

Sometimes I wonder if we should have an RPC or new socket based method
where additional programs could run along side Bitcoin Core and opt to
handle a subset of p2p commands. But then I think, that seems like a lot of
complexity for people who just want to help out the system, which I guess
is the bulk of our network now. Keeping their lives simple should have a
high priority. So a single unified program that just figures it out
automatically rather than expecting users to assemble a bag of parts seems
a goal worth striving for.

[-- Attachment #2: Type: text/html, Size: 1344 bytes --]

Re: [Bitcoin-development] Plans to separate wallet from core

[Wladimir]

On Tue, Jun 24, 2014 at 2:16 PM, Mike Hearn <mike@plan99.net> wrote:
> priority. So a single unified program that just figures it out automatically
> rather than expecting users to assemble a bag of parts seems a goal worth
> striving for.

As I've said before -- and I think we disagree here - I like moving
towards a bag of parts of specialized tools, maintained by people that
specialize in those tools, instead of a single project that aims to do
and know everything. This encourages experimentation  and makes
competition possible and I think that is healthy in this space.

Bitcoin has a strict need for consensus in the block chain format,
scripting system and validation. Outside of those, innovation should
be possible without any gatekeeper bottleneck or even widespread
agreement. Wallets, what data to store on disk, what indices to
maintain. But even P2P message extensions, as long as it doesn't
interfere with the rest of the network.

After an experiment is successful it could always be merged into
bitcoin core. But then the 'what-ifers' have less ammo, as it has been
tested in the real world.

For user convenience it's still possible to package pre-assembled
bags. But that doesn't need to figure into how things are developed.

Wladimir