[Bitcoin-development] Near-term scalability

[Bitcoin-development] Near-term scalability

[Mike Hearn]

I had to hit the sack last night as it was 2am CET, but I'd like to
sum up the discussion we had on IRC about scalability and SatoshiDice
in particular.

I think we all agreed on the following:

- Having senders/buyers pay no fees is psychologically desirable even
though we all understand that eventually, somebody, somewhere will be
paying fees to use Bitcoin

- In the ideal world Bitcoin would scale perfectly and there would be
no need for there to be some "winners" and some "losers" when it comes
to confirmation time.

There was discussion of some one-off changes to address the current
situation, namely de-ranking transactions that re-use addresses. Gavin
and myself were not keen on this idea, primarily because it just
avoids the real problem and Bitcoin already has a good way to
prioritize transactions via the fees mechanism itself. The real issue
is that SatoshiDice does indeed pay fees and generates a lot of
transactions, pushing more traditional traffic out due to artificial
throttles.

The following set of proposals were discussed:

(1) Change the mining code to group transactions together with their
mempool dependencies and then calculate all fees as a group. A tx with
a fee of 1 BTC that depends on 5 txns with zero fees would result in
all 6 transactions being considered to have a fee of 1BTC and
therefore become prioritized for inclusion. This allows a transition
to "receiver pays" model for fees. There are many advantages. One is
that it actually makes sense ... it's always the receiver who wants
confirmations because it's the receiver that fears double spends.
Senders never do. What's more, whilst Bitcoin is designed to operate
on a zero-trust model in the real world trust often exists and it can
be used to optimize by passing groups of transactions around with
their dependencies, until that group passes a trust boundary and gets
broadcast with a send-to-self tx to add fees. Another advantage is it
simplifies usage for end users who primarily buy rather than sell,
because it avoids the need to guess at fees, one of the most
problematic parts of Bitcoins design now.

The disadvantages are that it can result in extra transactions that
exist only for adding fees, and it requires a more modern payment
protocol than the direct-IP protocol Satoshi designed.

It would help address the current situation by avoiding angry users
who want to buy things, but don't know what fee to set and so their
transactions get stuck.

(2) SatoshiDice should use the same fee algorithms as Bitcoin-Qt to
avoid paying excessive fees and queue-jumping. Guess that's on my
plate.

(3) Scalability improvements seem like a no brainer to everyone, it's
just a case of how complicated they are.

(4) Making the block size limit float is better than picking a new
arbitrary threshold.

On the forums Matt stated that block chain pruning was a no-go because
"it makes bitcoin more centralized". I think we've thrashed this one
out sufficiently well by now that there should be a united opinion on
it. There are technical ways to implement it such that there is no
change of trust requirements. All the other issues (finding archival
nodes, etc) can be again addressed with sufficient programming.

For the case of huge blocks slowing down end user syncing and wasting
their resources, SPV clients like MultiBit and Android Wallet already
exist and will get better with time. If Jeff implements the bloom
filtering p2p commands I'll make bitcoinj use them and that'll knock
out excessive bandwidth usage and parse overheads from end users who
are on these clients. At some point Bitcoin-Qt can have a dual mode,
but who knows when that'll get implemented.

Does that all sound reasonable?

Re: [Bitcoin-development] Near-term scalability

[Matt Corallo]

On Fri, 2012-06-15 at 13:29 +0200, Mike Hearn wrote:
> I had to hit the sack last night as it was 2am CET, but I'd like to
> sum up the discussion we had on IRC about scalability and SatoshiDice
> in particular.
> 
> I think we all agreed on the following:
> 
> - Having senders/buyers pay no fees is psychologically desirable even
> though we all understand that eventually, somebody, somewhere will be
> paying fees to use Bitcoin
> 
> - In the ideal world Bitcoin would scale perfectly and there would be
> no need for there to be some "winners" and some "losers" when it comes
> to confirmation time.
> 
> There was discussion of some one-off changes to address the current
> situation, namely de-ranking transactions that re-use addresses. Gavin
> and myself were not keen on this idea, primarily because it just
> avoids the real problem and Bitcoin already has a good way to
> prioritize transactions via the fees mechanism itself. The real issue
> is that SatoshiDice does indeed pay fees and generates a lot of
> transactions, pushing more traditional traffic out due to artificial
> throttles.
The idea can be more generalized in that there are many cases where the
generator of a transaction doesn't care about confirmation times, and
would really be willing to make their transaction lower priority than
other 0-fee transactions.  This enables the first point with lower
confirmation times for a while longer.
As it turns out, we already have an indication that someone is willing
to wait longer for confirmations - rapid reuse of an address.  
1) Green Addresses: The whole point of a green address is that you are
trusted based on your address, not necessarily based on confirmations of
your transactions.  In this case, you are generally willing to wait a
bit longer for confirmations than the average user depositing coins into
their Mt. Gox account.  
2) Donation Addresses: If you are using a publicized donation address,
you probably aren't depending on getting your coins *now* to turn around
and ship a product and, again, you are a bit more willing to tolerate
longer confirmation times.
3) Lazy (or overworked) coders: If, for whatever reason, someone
designing a bitcoin site decides that it is simply easier to make users
pay to a single address for everything, such actions should generally be
discouraged.  Such a setup is worse for end-user privacy.  Also, such
laziness (or likely just overworked and not having time to fix the
issue) is likely also laziness across the board including ignoring
multisend for payouts.  If you discourage such address use forcing site
designers to implement more sane policies, hopefully they will do enough
research to also do multisend.  Note that though this is where one
addresses sites like SatoshiDice, its also the one where we are likely
to have the least impact...

One of the ways to implement such deprioritization of rapidly-reused
addresses is to limit the count of address re-uses by default in memory
pool.  By limiting relaying of such transactions, you a) give nodes
across the network some small say in the transactions which they have to
deal with relaying outside of blocks, instead of relying on miners to
make decisions which are good for the total network load, but which are
worse for them.  b) You allow sites which wish to re-use addresses to do
so initially to keep the time-to-launch the same as it is today, but
force them to re-think their design decisions as they grow to
(hopefully) decrease their impact on the average Bitcoin full-node
operator.  Sites which begin to see their transactions rate-limited have
several options:
1) Make a deal with a miner to feed them their list of now-non-relayed
transactions outside of the regular p2p network and have them manually
added to blocks.  Id argue that such setups are going to become more
common in the future and such out-of-band transaction relaying should be
encouraged.  This also shifts the delay for other transactions from a
constant delay getting into blocks until there is room for additional
0-fee transactions to a spike on each block from the given miner.  I
highly prefer this, as you would see usually only one or two block delay
getting your transaction confirmed at the worst case, instead of a very
fuzzy unknown delay that could stretch on for some time.
2) Use rotating addresses.  This is likely the simplest to implement,
and I would absolutely think this is what most sites would end up doing.
Though it doesn't result in a decreased load on the transaction-relaying
nodes, it does at least allow for a minor improvement in user privacy.  

In the end, it boils down to an optional transaction deprioritization.
> 
> The following set of proposals were discussed:
> 
> (1) Change the mining code to group transactions together with their
> mempool dependencies and then calculate all fees as a group. A tx with
> a fee of 1 BTC that depends on 5 txns with zero fees would result in
> all 6 transactions being considered to have a fee of 1BTC and
> therefore become prioritized for inclusion. This allows a transition
> to "receiver pays" model for fees. There are many advantages. One is
> that it actually makes sense ... it's always the receiver who wants
> confirmations because it's the receiver that fears double spends.
> Senders never do. What's more, whilst Bitcoin is designed to operate
> on a zero-trust model in the real world trust often exists and it can
> be used to optimize by passing groups of transactions around with
> their dependencies, until that group passes a trust boundary and gets
> broadcast with a send-to-self tx to add fees. Another advantage is it
> simplifies usage for end users who primarily buy rather than sell,
> because it avoids the need to guess at fees, one of the most
> problematic parts of Bitcoins design now.
> 
> The disadvantages are that it can result in extra transactions that
> exist only for adding fees, and it requires a more modern payment
> protocol than the direct-IP protocol Satoshi designed.
> 
> It would help address the current situation by avoiding angry users
> who want to buy things, but don't know what fee to set and so their
> transactions get stuck.
> 
> (2) SatoshiDice should use the same fee algorithms as Bitcoin-Qt to
> avoid paying excessive fees and queue-jumping. Guess that's on my
> plate.
> 
> (3) Scalability improvements seem like a no brainer to everyone, it's
> just a case of how complicated they are.
I think all of the above are largely no brianers to everyone.
> 
> (4) Making the block size limit float is better than picking a new
> arbitrary threshold.
Definitely something that is very appealing as we need to scale up.
> 
> On the forums Matt stated that block chain pruning was a no-go because
> "it makes bitcoin more centralized". I think we've thrashed this one
> out sufficiently well by now that there should be a united opinion on
> it. There are technical ways to implement it such that there is no
> change of trust requirements. All the other issues (finding archival
> nodes, etc) can be again addressed with sufficient programming.
My point was that the easiest way to do it would be to ship a pruned
snapshot with Bitcoin, and such a system, while verifiable, would
increase Bitocin's centralization.  Though it is quite possible to prune
the chain while downloading at checkpoints or when blocks are N deep, it
complicates the initial download if no one has the chain to begin with. 

Another point I made was that by doing chain pruning by default, we may
see a decrease in non-fClient nodes (for compatibility, I would assume
pruned nodes have to set fClient) which is what old clients look for to
connect to, possibly complicating using Bitcoin for clients that either
wish to run a full IBD or older clients which need a non-fClient node
before they are happy (which could be an issue when you look at the very
poor "upgrade-apathy" in the Bitcoin community with people running
long-outdated versions because they don't feel like upgrading).

All that said, I do believe pruning will eventually have to come to
encourage p2pool and other getmemorypool-based pool mining, but
(obviously) its something that needs careful consideration in its
overall effects across the network before its applied.
> 
> For the case of huge blocks slowing down end user syncing and wasting
> their resources, SPV clients like MultiBit and Android Wallet already
> exist and will get better with time. If Jeff implements the bloom
> filtering p2p commands I'll make bitcoinj use them and that'll knock
> out excessive bandwidth usage and parse overheads from end users who
> are on these clients. At some point Bitcoin-Qt can have a dual mode,
> but who knows when that'll get implemented.
> 
> Does that all sound reasonable?

Re: [Bitcoin-development] Near-term scalability

[Mike Hearn]

> The idea can be more generalized in that there are many cases where the
> generator of a transaction doesn't care about confirmation times, and
> would really be willing to make their transaction lower priority than
> other 0-fee transactions.

Just to be clear, I think this solution is a hack and don't support it
because it's yet another change of network rules. Some random people
will get whacked because of a heuristic "rule of thumb".

If it's implemented, SD could/would switch to fresh addresses and
nothing would have been achieved except making an already complex
system more complex.

I disagree with the notion that you need "less important than free".
If you care about the confirmation time of a transaction that was sent
to you and you need space in a limited resource, you can pay for it.
It's an auction like any other. Besides, the idea that transactions
are free today is just a psychological trick befitting governments but
not us - transactions are funded by aggressive hyperinflation. I would
never describe Bitcoin as a free system and I suggest nobody else does
either.

If grouped fee calculations are implemented, we can keep the nice
property that the person who cares about double spending risk pays the
fees, and if you assume most transactions are hub-and-spoke from
buyers to merchants, rather than a pure p2p graph, in practice it'll
work out to seeming free most of the time even if seen globally it
doesn't make much difference.

> My point was that the easiest way to do it would be to ship a pruned
> snapshot with Bitcoin, and such a system, while verifiable, would
> increase Bitocin's centralization.

I'm not sure why. If you want to audit everything from scratch, after
checking the code you could just blow away the included files and then
"-connect=archive.bitcoin.org" or something like that. After
rebuilding the chain from scratch, check the databases for consistency
with the included data.

It reduces the number of nodes with full copies of the block chain,
yes, but as long as there's at least one copy of the old data in an
accessible location new nodes can still bootstrap just fine.

I'm sure we can find organizations willing to host full chains for
people who want to rebuild their databases from scratch, given how
cheap disk space is.

> connect to, possibly complicating using Bitcoin for clients that either
> wish to run a full IBD or older clients which need a non-fClient node

Yes, but old nodes probably have a copy of the chain already, so it
wouldn't affect them. New blocks would still be fully distributed,
right?

The only case where it'd cause issues is if you install a fresh copy
of a very old node. Not a common occurrence, and those nodes will have
to wait until they find an archival node announcing itself. Those
nodes could be made to announce more frequently than normal, if need
be.

Re: [Bitcoin-development] Near-term scalability

[Matt Corallo]

On Fri, 2012-06-15 at 15:34 +0200, Mike Hearn wrote:
> > The idea can be more generalized in that there are many cases where the
> > generator of a transaction doesn't care about confirmation times, and
> > would really be willing to make their transaction lower priority than
> > other 0-fee transactions.
> 
> Just to be clear, I think this solution is a hack and don't support it
> because it's yet another change of network rules. Some random people
> will get whacked because of a heuristic "rule of thumb".
Its arguably not a change to network rules as its something that users
can already do today by patching their clients.  Obviously any
implementation would have sane defaults which allowed for a significant
number of transactions to/from a given address at a time, avoiding
whacking random people unless they are large enough that they should
really already be fully aware of how bitcoin works.
> 
> If it's implemented, SD could/would switch to fresh addresses and
> nothing would have been achieved except making an already complex
> system more complex.
I would think SD would switch to using fresh addresses for each bet.
But even that is a good thing, at least where user privacy is concerned.
However, I would hope that SD would see the rule tweak and, in order to
avoid having to generate a number of new addresses per second (or, if
they went the pool route, having a huge pool of many thousands of
addresses), they would consider implementing sendmulti support.
> 
> I disagree with the notion that you need "less important than free".
> If you care about the confirmation time of a transaction that was sent
> to you and you need space in a limited resource, you can pay for it.
> It's an auction like any other. Besides, the idea that transactions
> are free today is just a psychological trick befitting governments but
> not us - transactions are funded by aggressive hyperinflation. I would
> never describe Bitcoin as a free system and I suggest nobody else does
> either.
I agree, free transactions isnt something we should aggressively push as
a feature of Bitcoin, its simply not.  However, in the current system
free transactions are usually confirmed within a small number of blocks,
and for a number of users, that is an important feature that draws them
to get through the initial hurdles of converting money to Bitcoin and
understanding enough of the system to trust it.  I believe that if we
can incentive large transaction creators to avoid delaying free
transactions, we should and giving them the option to delay their own
transactions seems like a perfectly reasonable way to do so.  Even if
you drop all the per-address limit stuff, allowing transaction creators
to add a simple flag to transactions seems reasonable when they want to
encourage Bitcoin to continue to grow as it does today.  Obviously
keeping free transactions confirming won't be possible forever, but
hopefully that will be as a result of natural growth which can encourage
further growth without the need for free transactions and not as a
result of a few actors in the community creating a transaction volume
significantly greater than their user-base.
> 
> If grouped fee calculations are implemented, we can keep the nice
> property that the person who cares about double spending risk pays the
> fees, and if you assume most transactions are hub-and-spoke from
> buyers to merchants, rather than a pure p2p graph, in practice it'll
> work out to seeming free most of the time even if seen globally it
> doesn't make much difference.
ACK, thats an important thing to implement IMO, but I really dont see it
as something that replaces the option to deprioritize your own
transactions to below 0-fee transactions.  It could even allow users who
receive payouts which are below 0-fee transactions to place a fee on the
subsequent transactions to allow the payouts to confirm quicker (if done
right).
> 
> > My point was that the easiest way to do it would be to ship a pruned
> > snapshot with Bitcoin, and such a system, while verifiable, would
> > increase Bitocin's centralization.
> 
> I'm not sure why. If you want to audit everything from scratch, after
> checking the code you could just blow away the included files and then
> "-connect=archive.bitcoin.org" or something like that. After
> rebuilding the chain from scratch, check the databases for consistency
> with the included data.
I would be surprised if more than a handful of devs audit such a thing.
And I would say that does define an increase in centralization.
> 
> It reduces the number of nodes with full copies of the block chain,
> yes, but as long as there's at least one copy of the old data in an
> accessible location new nodes can still bootstrap just fine.
Sadly, old nodes do not know where to look for such data, and I'm fairly
certain people running old nodes don't read the forums enough to catch
when it is announced that old nodes should make sure to
-connect=archive.bitcoin.org in order to avoid initially having horrible
initial bootstrap times and eventually not being able to connect to
full-chain-serving nodes at all.
> 
> I'm sure we can find organizations willing to host full chains for
> people who want to rebuild their databases from scratch, given how
> cheap disk space is.
Sadly, disk space isnt the issue.  Each connection to bitcoind (not that
it cant be fixed, but currently) eats a nice chunk of memory.  An
organization that wants to provide nodes for old nodes to connect to
would need to have a significant number of open incoming connection
slots, have plenty of bandwidth for nodes that are in IBD and have
plenty of memory and CPU to manage all the connections.

> 
> > connect to, possibly complicating using Bitcoin for clients that either
> > wish to run a full IBD or older clients which need a non-fClient node
> 
> Yes, but old nodes probably have a copy of the chain already, so it
> wouldn't affect them. New blocks would still be fully distributed,
> right?
Sadly, BDB's infamous database corrupted messages appear all too often,
and the usual response is "delete the chain and resync."  I have a hard
time believing that old nodes will rarely be in IBD.  
> 
> The only case where it'd cause issues is if you install a fresh copy
> of a very old node. Not a common occurrence, and those nodes will have
> to wait until they find an archival node announcing itself. Those
> nodes could be made to announce more frequently than normal, if need
> be.
I agree that its very possible to have archival nodes available and to
make it work, but I have yet to see anyone doing any work to actually
get commitments to run archival nodes and I have yet to see any
discussion of what, exactly, that would entail.

Matt

[Bitcoin-development] Near-term scalability

[Gregory Maxwell]

[I originally sent an earlier version of this message to Mike off
list, but I figure it's worth adding to the public discussion]

On Fri, Jun 15, 2012 at 7:29 AM, Mike Hearn <mike@plan99.net> wrote:
> (4) Making the block size limit float is better than picking a new
> arbitrary threshold.
> On the forums Matt stated that block chain pruning was a no-go because
> "it makes bitcoin more centralized". I think we've thrashed this one
> out sufficiently well by now that there should be a united opinion on
> it.

By itself letting the size float has non-trivial existential risk.  A
Bitcoin with expensive transactions due to competition for space in
blocks can be front-ended with fast payment systems and still provide
the promised decentralized currency. Bitcoin with a very large
blockchain and blocks does not.  It would do the bitcoin users no good
to increase the transaction volume while concurrently making Bitcoin
more or less pointless over the alternatives.

Scalability must be improved, we can unite on that opinion.  But
scalability can't come at the expense of what made Bitcoin worth
having in the first place.

Fortunately it appear to be possible to greatly increase the
scalability without compromising on keeping the costs of operating a
fully validating node very low,  for example Pieter's experimentation
with txout+txid indexing (for the 'flip the chain' proposals)
indicates that the data required right now to validate further
transactions is only about 85MiB— and that would be somewhat smaller
with compression and with clients which intentionally try to reduce
the set of unspent transactions.   Commitments to these indexes in the
chain would allow almost-full validating nodes with fairly limited
resources.  (Almost-full meaning they would not validate the history
long before they started, they'd trusted header difficulty for that. They
could still mine and otherwise act as full nodes).

Achieving scalability improvements without breaking the radical
decentralization will be a lot harder than just improving scalability
but it's effort that is justified if the scalability is actually
needed.

How much decentralization is needed in the end?  That isn't clear— "As
much as possible" should generally be the goal.  Modern currencies
aren't controlled by single parties but by tens of thousands of
parties locked in economic, legal, and political compromise that
limits their control.  In Bitcoin the traditional controls that keep
parties honest are non-existent and if they were just directly applied
we'd potentially lose the properties that make Bitcoin distinct and
useful (e.g. make all miners mine only with FED permission and you
just have a really bandwidth inefficient interface to the dollar).
Instead we have aggressive decentralization and autonomous rule
enforcement.

Mike pointed out that  "Before he left Satoshi made a comment saying
he used to think Bitcoin would need millions of nodes if it became
really popular, but in the end he thought it could do fine with just
tens of thousands"    I'm not so sure— and I think the truth is in
between.  Tens of thousands of nodes— run by a self-selecting bunch of
people who reap the greatest rewards from controlling the validation
of Bitcoin, who by that criteria necessarily have a lot in common with
each other and perhaps not with the regular users— could easily be an
outcome where control is _less_ publicly vested than popular
government controlled currencies.   We probably don't need the raw
numbers of nodes, but we need a distribution of ownership and a
distribution of interest (e.g. not a system by bankers for bankers) of
those nodes which I think can only be achieved by making them cheap to
operate and having a lot more than we actually need. — though not so
much that it has to run on every laptop.

The core challenge is that the only obvious ways to justify the cost
of maintaining expensive validation infrastructure is because you
intend to manipulate the currency using it or because you intend to
prevent other people from manipulating the currency.  The latter
motivation is potentially subject to a tragedy of the commons— you
don't need to run a full validating node as long as 'enough' other
people do, and enough is a nice slippery slope to zero.   Right now
just the random computers I— some random geek— had at home prior to
Bitcoin could store over a hundred years of max size blocks and
process the maximum rate of transactions.   With the costs so low
there isn't any real question about a consolidation of validation
making Bitcoin pointless.  You could probably increase the scale 10x
without breaking that analysis  but beyond that unless the
cost-per-scale goes down a highly consolidated future seems likely.
40 years from now why would people use Bitcoin over centralized
private banknotes like paypal or democratic government controlled
currencies?

Perhaps Bitcoin transaction could transition to being more of the
same— controlled by a consortium of banks, exchanging gigabyte blocks
over terabit ethernet, but I think that would be sad.  An alternative
which was autonomous and decentralized even if the transactions were
somewhat slow or costly would be excellent competition for everything
else, and it's something I think man kind ought to have.

Re: [Bitcoin-development] Near-term scalability

[Stefan Thomas]

Thanks Mike for the writeup - I'm very sad to have missed the discussion
on IRC since fee economics are probably my favorite topic, but I'll try
to contribute to the email discussion instead.

> (4) Making the block size limit float is better than picking a new
> arbitrary threshold.

Fees are a product of both real and artificial limits to transaction
validation.

The artificial limits like the block size limit are essentially putting
a floor on prices by limiting supply beyond what it would otherwise be.
E.g. the network could confirm more transactions theoretically, but the
block size limit prevents it.

The real limits are the bandwidth, computing and memory resources of
participating nodes. For the sake of argument suppose a 1 TB block was
released into the network right now and we'll also assume there was no
block size limit of any kind. Many nodes would likely not be able to
successfully download this block in under 10-30 minutes, so there is a
very good chance that other miners will have generated two blocks before
this block makes its way to them.

What does this mean? The miner generating a 1 TB block knows this would
happen. So in terms of economic self interest he will generate the
largest possible block that he is still confident that other miners will
accept and process. A miner who receives a block will also consider
whether to build on it based on whether they think other miners will be
able to download it. In other words, if I receive a large block I may
decide not to mine on it, because I believe that the majority of mining
power will not mine on it - because it is either too large for them to
download or because their rules against large blocks reject it.

It's important to understand that in practice economic actors tend to
plan ahead. In other words, if there is no block size limit that doesn't
mean that there will be constant forks and total chaos. Rather, no miner
will ever want to have a block rejected due to size, there is plenty of
incentive to be conservative with your limits. Even if there are forks,
this simply means that miners have decided that they can make more money
by including more transactions at the cost of the occasional dud.

Therefore, from an economic perspective, we do not need a global block
size limit of any kind. As "guardians of the network" the only thing we
need to do is to let miners figure out what they wanna do.

HOWEVER, the existing economic incentives won't manifest unless somebody
translates them into code. We have to give our users (miners & endusers)
the tools to create a genuine fee-based verification market.

On the miner side: I would make the block size limit configurable with a
relatively high default. If the default is too low few people will
bother changing it, which means that it is not worth changing (because a
majority uses the default anyway), which means even fewer people will
change it and so on.

The block size limit should also be a soft rather than a hard limit -
here are some ideas for this:

- The default limit for accepting blocks from others should always be
significantly greater than the default limit for blocks that the client
itself will generate.

- There should be different size limits for side chains that are longer
than the currently active chain. In other words, I might reject a block
for being slightly too large, but if everyone else accepts it I should
eventually accept it too, and my client should also consider
automatically raising my size limit if this happens a lot.

The rationale for the soft limit is to allow for gradual upward
adjustment. It needs to be risky for individual miners to raise the size
of their blocks to new heights, but ideally there won't be one solid
wall for them to run into.

On the user side: I would display the fee on the Send Coins dialog and
allow users to choose a different fee per transaction. We also talked
about adding some UI feedback where the client tries to estimate how
long a transaction will take to confirm given a certain fee, based on
recent information about what it observed from the network. If the fee
can be changed on the Send Coins tab, then this could be a red, yellow,
green visual indication whether the fee is sufficient, adequate or
dangerously low.

A criticism one might raise is: "The block size limit is not to protect
miners, but to protect end users who may have less resources than miners
and can't download gigantic block chains." - That's a viewpoint that is
certainly valid. I believe that we will be able to do a lot just with
efficiency improvements, pruning, compression and whatnot. But when it
comes down to it, I'd prefer a large network with cheap
microtransactions even if that means that consumer hardware can't
operate as a standalone validating node anymore. Headers-only mode is
already a much-requested feature anyway and there are many ways of
improving the security of various header-only or lightweight protocols.

(I just saw Greg's message advocating the opposite viewpoint, I'll
respond to that as soon as I can.)


> (1) Change the mining code to group transactions together with their
> mempool dependencies and then calculate all fees as a group.

+1 Very good change. This would allow miners to maximize their revenue
and in doing so better represent the existing priorities that users
express through fees.


> There was discussion of some one-off changes to address the current
> situation, namely de-ranking transactions that re-use addresses.

Discouraging address reuse will not change the amount of transactions, I
think we all agree on that. As for whether it improves the
prioritization, I'm not sure. Use cases that we seek to discourage may
simply switch to random addresses and I don't agree in and of itself
this is a benefit (see item 4 below). Here are a few reasons one might
be against this proposal:

1) Certain use cases like green addresses will be forced to become more
complicated than they would otherwise need to be.

2) It will be harder to read information straight out of the block
chain, for example right now we can pretty easily see how much volume is
caused by Satoshi Dice, perhaps allowing us to make better decisions.

3) The address index that is used by block explorers and lightweight
client servers will grow unnecessarily (an address -> tx index will be
larger if the number of unique addresses increases given the same number
of txs), so for people like myself who work on that type of software
you're actually making our scalability equation slightly worse.

4) You're forcing people into privacy best practices which you think are
good, but others may not subscribe to. For example I have absolutely
zero interest in privacy, anyone who cares that I buy Bitcoins with my
salary and spend them on paragliding is welcome to know about it.
Frankly, if I cared about privacy I wouldn't be using Bitcoin. If other
people want to use mixing services and randomize their addresses and
communicate through Tor that's fine, but the client shouldn't force me
to do those things if I don't want to by "deprioritizing" my transactions.

5) We may not like firstbits, but the fact remains that for now they are
extremely popular, because they improve the user experience where we
failed to do so. If you deprioritize transactions to reused addresses
you'll for example deprioritize all/most of Girls Gone Bitcoin, which
(again, like it or not) is one of the few practical, sustainable niches
that Bitcoin has managed to carve out for itself so far.


> Having senders/buyers pay no fees is psychologically desirable even
> though we all understand that eventually, somebody, somewhere will be
> paying fees to use Bitcoin

Free is just an extreme form of cheap, so if we can make transactions
very cheap (through efficiency and very large blocks) then it will be
easier for charitable miners to include free transactions. In practice,
my prediction is that free transactions on the open network will simply
not be possible in the long run. Dirty hacks aside there is simply no
way of distinguishing a spam transaction from a charity-worthy
transaction. So the way I envision free transactions in the future is
that there may be miners in partnership with wallet providers like
BlockChain.info that let you submit feeless transactions straight to
them based on maybe a captcha or some ads. (For the purist, the captcha
challenge and response could be communicated across the bitcoin network,
but I think we agree that such things should ideally take place
out-of-band.)

That way, the available charity of miners who wish to include feeless
transactions would go to human users as opposed to the potentially
infinite demand of auto-generated feeless transactions.



On 6/15/2012 1:29 PM, Mike Hearn wrote:
> I had to hit the sack last night as it was 2am CET, but I'd like to
> sum up the discussion we had on IRC about scalability and SatoshiDice
> in particular.
>
> I think we all agreed on the following:
>
> - Having senders/buyers pay no fees is psychologically desirable even
> though we all understand that eventually, somebody, somewhere will be
> paying fees to use Bitcoin
>
> - In the ideal world Bitcoin would scale perfectly and there would be
> no need for there to be some "winners" and some "losers" when it comes
> to confirmation time.
>
> There was discussion of some one-off changes to address the current
> situation, namely de-ranking transactions that re-use addresses. Gavin
> and myself were not keen on this idea, primarily because it just
> avoids the real problem and Bitcoin already has a good way to
> prioritize transactions via the fees mechanism itself. The real issue
> is that SatoshiDice does indeed pay fees and generates a lot of
> transactions, pushing more traditional traffic out due to artificial
> throttles.
>
> The following set of proposals were discussed:
>
> (1) Change the mining code to group transactions together with their
> mempool dependencies and then calculate all fees as a group. A tx with
> a fee of 1 BTC that depends on 5 txns with zero fees would result in
> all 6 transactions being considered to have a fee of 1BTC and
> therefore become prioritized for inclusion. This allows a transition
> to "receiver pays" model for fees. There are many advantages. One is
> that it actually makes sense ... it's always the receiver who wants
> confirmations because it's the receiver that fears double spends.
> Senders never do. What's more, whilst Bitcoin is designed to operate
> on a zero-trust model in the real world trust often exists and it can
> be used to optimize by passing groups of transactions around with
> their dependencies, until that group passes a trust boundary and gets
> broadcast with a send-to-self tx to add fees. Another advantage is it
> simplifies usage for end users who primarily buy rather than sell,
> because it avoids the need to guess at fees, one of the most
> problematic parts of Bitcoins design now.
>
> The disadvantages are that it can result in extra transactions that
> exist only for adding fees, and it requires a more modern payment
> protocol than the direct-IP protocol Satoshi designed.
>
> It would help address the current situation by avoiding angry users
> who want to buy things, but don't know what fee to set and so their
> transactions get stuck.
>
> (2) SatoshiDice should use the same fee algorithms as Bitcoin-Qt to
> avoid paying excessive fees and queue-jumping. Guess that's on my
> plate.
>
> (3) Scalability improvements seem like a no brainer to everyone, it's
> just a case of how complicated they are.
>
> (4) Making the block size limit float is better than picking a new
> arbitrary threshold.
>
> On the forums Matt stated that block chain pruning was a no-go because
> "it makes bitcoin more centralized". I think we've thrashed this one
> out sufficiently well by now that there should be a united opinion on
> it. There are technical ways to implement it such that there is no
> change of trust requirements. All the other issues (finding archival
> nodes, etc) can be again addressed with sufficient programming.
>
> For the case of huge blocks slowing down end user syncing and wasting
> their resources, SPV clients like MultiBit and Android Wallet already
> exist and will get better with time. If Jeff implements the bloom
> filtering p2p commands I'll make bitcoinj use them and that'll knock
> out excessive bandwidth usage and parse overheads from end users who
> are on these clients. At some point Bitcoin-Qt can have a dual mode,
> but who knows when that'll get implemented.
>
> Does that all sound reasonable?
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

Re: [Bitcoin-development] Near-term scalability

[Mike Koss]

[-- Attachment #1: Type: text/plain, Size: 15169 bytes --]

Grouping mempool transactions based on fees of the group seems
an unnecessary complexity; it makes it harder to predict if an isolated
transaction has enough "juice" to be included in the next Block.

Given your point about economic actors adapting to conditions, would it not
be simpler to use a individual "fee per byte" priority algorithm and let
transaction generators distribute their fees accordingly (and more
predictably)?

This simpler algorithm will prune arbitrary transactions sub-optimally, but
has the benefit of being more understandable and predictable from the point
of view of transaction generators.

On Fri, Jun 15, 2012 at 9:56 AM, Stefan Thomas <moon@justmoon.de> wrote:

> Thanks Mike for the writeup - I'm very sad to have missed the discussion
> on IRC since fee economics are probably my favorite topic, but I'll try
> to contribute to the email discussion instead.
>
> > (4) Making the block size limit float is better than picking a new
> > arbitrary threshold.
>
> Fees are a product of both real and artificial limits to transaction
> validation.
>
> The artificial limits like the block size limit are essentially putting
> a floor on prices by limiting supply beyond what it would otherwise be.
> E.g. the network could confirm more transactions theoretically, but the
> block size limit prevents it.
>
> The real limits are the bandwidth, computing and memory resources of
> participating nodes. For the sake of argument suppose a 1 TB block was
> released into the network right now and we'll also assume there was no
> block size limit of any kind. Many nodes would likely not be able to
> successfully download this block in under 10-30 minutes, so there is a
> very good chance that other miners will have generated two blocks before
> this block makes its way to them.
>
> What does this mean? The miner generating a 1 TB block knows this would
> happen. So in terms of economic self interest he will generate the
> largest possible block that he is still confident that other miners will
> accept and process. A miner who receives a block will also consider
> whether to build on it based on whether they think other miners will be
> able to download it. In other words, if I receive a large block I may
> decide not to mine on it, because I believe that the majority of mining
> power will not mine on it - because it is either too large for them to
> download or because their rules against large blocks reject it.
>
> It's important to understand that in practice economic actors tend to
> plan ahead. In other words, if there is no block size limit that doesn't
> mean that there will be constant forks and total chaos. Rather, no miner
> will ever want to have a block rejected due to size, there is plenty of
> incentive to be conservative with your limits. Even if there are forks,
> this simply means that miners have decided that they can make more money
> by including more transactions at the cost of the occasional dud.
>
> Therefore, from an economic perspective, we do not need a global block
> size limit of any kind. As "guardians of the network" the only thing we
> need to do is to let miners figure out what they wanna do.
>
> HOWEVER, the existing economic incentives won't manifest unless somebody
> translates them into code. We have to give our users (miners & endusers)
> the tools to create a genuine fee-based verification market.
>
> On the miner side: I would make the block size limit configurable with a
> relatively high default. If the default is too low few people will
> bother changing it, which means that it is not worth changing (because a
> majority uses the default anyway), which means even fewer people will
> change it and so on.
>
> The block size limit should also be a soft rather than a hard limit -
> here are some ideas for this:
>
> - The default limit for accepting blocks from others should always be
> significantly greater than the default limit for blocks that the client
> itself will generate.
>
> - There should be different size limits for side chains that are longer
> than the currently active chain. In other words, I might reject a block
> for being slightly too large, but if everyone else accepts it I should
> eventually accept it too, and my client should also consider
> automatically raising my size limit if this happens a lot.
>
> The rationale for the soft limit is to allow for gradual upward
> adjustment. It needs to be risky for individual miners to raise the size
> of their blocks to new heights, but ideally there won't be one solid
> wall for them to run into.
>
> On the user side: I would display the fee on the Send Coins dialog and
> allow users to choose a different fee per transaction. We also talked
> about adding some UI feedback where the client tries to estimate how
> long a transaction will take to confirm given a certain fee, based on
> recent information about what it observed from the network. If the fee
> can be changed on the Send Coins tab, then this could be a red, yellow,
> green visual indication whether the fee is sufficient, adequate or
> dangerously low.
>
> A criticism one might raise is: "The block size limit is not to protect
> miners, but to protect end users who may have less resources than miners
> and can't download gigantic block chains." - That's a viewpoint that is
> certainly valid. I believe that we will be able to do a lot just with
> efficiency improvements, pruning, compression and whatnot. But when it
> comes down to it, I'd prefer a large network with cheap
> microtransactions even if that means that consumer hardware can't
> operate as a standalone validating node anymore. Headers-only mode is
> already a much-requested feature anyway and there are many ways of
> improving the security of various header-only or lightweight protocols.
>
> (I just saw Greg's message advocating the opposite viewpoint, I'll
> respond to that as soon as I can.)
>
>
> > (1) Change the mining code to group transactions together with their
> > mempool dependencies and then calculate all fees as a group.
>
> +1 Very good change. This would allow miners to maximize their revenue
> and in doing so better represent the existing priorities that users
> express through fees.
>
>
> > There was discussion of some one-off changes to address the current
> > situation, namely de-ranking transactions that re-use addresses.
>
> Discouraging address reuse will not change the amount of transactions, I
> think we all agree on that. As for whether it improves the
> prioritization, I'm not sure. Use cases that we seek to discourage may
> simply switch to random addresses and I don't agree in and of itself
> this is a benefit (see item 4 below). Here are a few reasons one might
> be against this proposal:
>
> 1) Certain use cases like green addresses will be forced to become more
> complicated than they would otherwise need to be.
>
> 2) It will be harder to read information straight out of the block
> chain, for example right now we can pretty easily see how much volume is
> caused by Satoshi Dice, perhaps allowing us to make better decisions.
>
> 3) The address index that is used by block explorers and lightweight
> client servers will grow unnecessarily (an address -> tx index will be
> larger if the number of unique addresses increases given the same number
> of txs), so for people like myself who work on that type of software
> you're actually making our scalability equation slightly worse.
>
> 4) You're forcing people into privacy best practices which you think are
> good, but others may not subscribe to. For example I have absolutely
> zero interest in privacy, anyone who cares that I buy Bitcoins with my
> salary and spend them on paragliding is welcome to know about it.
> Frankly, if I cared about privacy I wouldn't be using Bitcoin. If other
> people want to use mixing services and randomize their addresses and
> communicate through Tor that's fine, but the client shouldn't force me
> to do those things if I don't want to by "deprioritizing" my transactions.
>
> 5) We may not like firstbits, but the fact remains that for now they are
> extremely popular, because they improve the user experience where we
> failed to do so. If you deprioritize transactions to reused addresses
> you'll for example deprioritize all/most of Girls Gone Bitcoin, which
> (again, like it or not) is one of the few practical, sustainable niches
> that Bitcoin has managed to carve out for itself so far.
>
>
> > Having senders/buyers pay no fees is psychologically desirable even
> > though we all understand that eventually, somebody, somewhere will be
> > paying fees to use Bitcoin
>
> Free is just an extreme form of cheap, so if we can make transactions
> very cheap (through efficiency and very large blocks) then it will be
> easier for charitable miners to include free transactions. In practice,
> my prediction is that free transactions on the open network will simply
> not be possible in the long run. Dirty hacks aside there is simply no
> way of distinguishing a spam transaction from a charity-worthy
> transaction. So the way I envision free transactions in the future is
> that there may be miners in partnership with wallet providers like
> BlockChain.info that let you submit feeless transactions straight to
> them based on maybe a captcha or some ads. (For the purist, the captcha
> challenge and response could be communicated across the bitcoin network,
> but I think we agree that such things should ideally take place
> out-of-band.)
>
> That way, the available charity of miners who wish to include feeless
> transactions would go to human users as opposed to the potentially
> infinite demand of auto-generated feeless transactions.
>
>
>
> On 6/15/2012 1:29 PM, Mike Hearn wrote:
> > I had to hit the sack last night as it was 2am CET, but I'd like to
> > sum up the discussion we had on IRC about scalability and SatoshiDice
> > in particular.
> >
> > I think we all agreed on the following:
> >
> > - Having senders/buyers pay no fees is psychologically desirable even
> > though we all understand that eventually, somebody, somewhere will be
> > paying fees to use Bitcoin
> >
> > - In the ideal world Bitcoin would scale perfectly and there would be
> > no need for there to be some "winners" and some "losers" when it comes
> > to confirmation time.
> >
> > There was discussion of some one-off changes to address the current
> > situation, namely de-ranking transactions that re-use addresses. Gavin
> > and myself were not keen on this idea, primarily because it just
> > avoids the real problem and Bitcoin already has a good way to
> > prioritize transactions via the fees mechanism itself. The real issue
> > is that SatoshiDice does indeed pay fees and generates a lot of
> > transactions, pushing more traditional traffic out due to artificial
> > throttles.
> >
> > The following set of proposals were discussed:
> >
> > (1) Change the mining code to group transactions together with their
> > mempool dependencies and then calculate all fees as a group. A tx with
> > a fee of 1 BTC that depends on 5 txns with zero fees would result in
> > all 6 transactions being considered to have a fee of 1BTC and
> > therefore become prioritized for inclusion. This allows a transition
> > to "receiver pays" model for fees. There are many advantages. One is
> > that it actually makes sense ... it's always the receiver who wants
> > confirmations because it's the receiver that fears double spends.
> > Senders never do. What's more, whilst Bitcoin is designed to operate
> > on a zero-trust model in the real world trust often exists and it can
> > be used to optimize by passing groups of transactions around with
> > their dependencies, until that group passes a trust boundary and gets
> > broadcast with a send-to-self tx to add fees. Another advantage is it
> > simplifies usage for end users who primarily buy rather than sell,
> > because it avoids the need to guess at fees, one of the most
> > problematic parts of Bitcoins design now.
> >
> > The disadvantages are that it can result in extra transactions that
> > exist only for adding fees, and it requires a more modern payment
> > protocol than the direct-IP protocol Satoshi designed.
> >
> > It would help address the current situation by avoiding angry users
> > who want to buy things, but don't know what fee to set and so their
> > transactions get stuck.
> >
> > (2) SatoshiDice should use the same fee algorithms as Bitcoin-Qt to
> > avoid paying excessive fees and queue-jumping. Guess that's on my
> > plate.
> >
> > (3) Scalability improvements seem like a no brainer to everyone, it's
> > just a case of how complicated they are.
> >
> > (4) Making the block size limit float is better than picking a new
> > arbitrary threshold.
> >
> > On the forums Matt stated that block chain pruning was a no-go because
> > "it makes bitcoin more centralized". I think we've thrashed this one
> > out sufficiently well by now that there should be a united opinion on
> > it. There are technical ways to implement it such that there is no
> > change of trust requirements. All the other issues (finding archival
> > nodes, etc) can be again addressed with sufficient programming.
> >
> > For the case of huge blocks slowing down end user syncing and wasting
> > their resources, SPV clients like MultiBit and Android Wallet already
> > exist and will get better with time. If Jeff implements the bloom
> > filtering p2p commands I'll make bitcoinj use them and that'll knock
> > out excessive bandwidth usage and parse overheads from end users who
> > are on these clients. At some point Bitcoin-Qt can have a dual mode,
> > but who knows when that'll get implemented.
> >
> > Does that all sound reasonable?
> >
> >
> ------------------------------------------------------------------------------
> > Live Security Virtual Conference
> > Exclusive live event will cover all the ways today's security and
> > threat landscape has changed and how IT managers can respond. Discussions
> > will include endpoint security, mobile security and the latest in malware
> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> > _______________________________________________
> > Bitcoin-development mailing list
> > Bitcoin-development@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bitcoin-development
> >
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>



-- 
Mike Koss
CTO, CoinLab
(425) 246-7701 (m)

A Bitcoin Primer <http://coinlab.com/a-bitcoin-primer.pdf> - What you need
to know about Bitcoins.

[-- Attachment #2: Type: text/html, Size: 17585 bytes --]

Re: [Bitcoin-development] Near-term scalability

[Amir Taaki]

Forcing users to switch addresses per received payment to work around a bad fee system would be a braindead decision. You might love software and playing with web plugins, but not everyone does. Artists like Rap News can right now simply throw up an address and begin accepting donations. That's a hugely powerful and impactful selling point for Bitcoin.

I don't really see these problems as a concern. Stefan made an excellent post which touched on this, in that miners have an incentive to keep block sizes low so that their blocks propagate. The real problem here is not about block propagation but the user experience. The way I see it, Bitcoin is becoming more specialised over time and part of that process is abstraction. In the past we all used the Satoshi client for mining, merchant functions, validating blocks and personal uses. These are rapidly diverging, and managing the blockchain is not something that user clients should be doing.

Mike is right when he says the network only needs a few thousand nodes to function fairly. I am not worried about Bitcoin becoming corrupted because of it being a network "by bankers for bankers" because unlike the conventional finance industry, there are no artificial barriers to entry beyond the base cost. This network would always be competitive and strictly operate based on market dynamics.

Case in point: http://en.wikipedia.org/wiki/Coase_theorem

With strict property rights and zero (or low) transaction costs, the allocation of a system does not matter. The system will make efficient use of its resources. I don't see why a cabal would try to corrupt Bitcoin at expense to themselves when a new competitor can enter the market and undercut them. It's why we expect the ROI on mining to be 0 or negative.


I figured out that if you trust data from a blockchain service and only accept data with multiple confirms from each connected service, then you can trivially calculate the probability of being fed corrupt data (assuming a fixed chance per server). In this way, the model is a fault tolerant byzantine system. The chance of being manipulated falls expontentially as you add more servers. And these services can be made highly scalable if you see my BIP 33.

https://en.bitcoin.it/wiki/BIP_0033

________________________________
From: Mike Koss <mike@coinlab.com>
To: Stefan Thomas <moon@justmoon.de> 
Cc: bitcoin-development@lists.sourceforge.net 
Sent: Friday, June 15, 2012 7:37 PM
Subject: Re: [Bitcoin-development] Near-term scalability


Grouping mempool transactions based on fees of the group seems an unnecessary complexity; it makes it harder to predict if an isolated transaction has enough "juice" to be included in the next Block.

Given your point about economic actors adapting to conditions, would it not be simpler to use a individual "fee per byte" priority algorithm and let transaction generators distribute their fees accordingly (and more predictably)?

This simpler algorithm will prune arbitrary transactions sub-optimally, but has the benefit of being more understandable and predictable from the point of view of transaction generators.



On Fri, Jun 15, 2012 at 9:56 AM, Stefan Thomas <moon@justmoon.de> wrote:

Thanks Mike for the writeup - I'm very sad to have missed the discussion
>on IRC since fee economics are probably my favorite topic, but I'll try
>to contribute to the email discussion instead.
>
>
>> (4) Making the block size limit float is better than picking a new
>> arbitrary threshold.
>
>Fees are a product of both real and artificial limits to transaction
>validation.
>
>The artificial limits like the block size limit are essentially putting
>a floor on prices by limiting supply beyond what it would otherwise be.
>E.g. the network could confirm more transactions theoretically, but the
>block size limit prevents it.
>
>The real limits are the bandwidth, computing and memory resources of
>participating nodes. For the sake of argument suppose a 1 TB block was
>released into the network right now and we'll also assume there was no
>block size limit of any kind. Many nodes would likely not be able to
>successfully download this block in under 10-30 minutes, so there is a
>very good chance that other miners will have generated two blocks before
>this block makes its way to them.
>
>What does this mean? The miner generating a 1 TB block knows this would
>happen. So in terms of economic self interest he will generate the
>largest possible block that he is still confident that other miners will
>accept and process. A miner who receives a block will also consider
>whether to build on it based on whether they think other miners will be
>able to download it. In other words, if I receive a large block I may
>decide not to mine on it, because I believe that the majority of mining
>power will not mine on it - because it is either too large for them to
>download or because their rules against large blocks reject it.
>
>It's important to understand that in practice economic actors tend to
>plan ahead. In other words, if there is no block size limit that doesn't
>mean that there will be constant forks and total chaos. Rather, no miner
>will ever want to have a block rejected due to size, there is plenty of
>incentive to be conservative with your limits. Even if there are forks,
>this simply means that miners have decided that they can make more money
>by including more transactions at the cost of the occasional dud.
>
>Therefore, from an economic perspective, we do not need a global block
>size limit of any kind. As "guardians of the network" the only thing we
>need to do is to let miners figure out what they wanna do.
>
>HOWEVER, the existing economic incentives won't manifest unless somebody
>translates them into code. We have to give our users (miners & endusers)
>the tools to create a genuine fee-based verification market.
>
>On the miner side: I would make the block size limit configurable with a
>relatively high default. If the default is too low few people will
>bother changing it, which means that it is not worth changing (because a
>majority uses the default anyway), which means even fewer people will
>change it and so on.
>
>The block size limit should also be a soft rather than a hard limit -
>here are some ideas for this:
>
>- The default limit for accepting blocks from others should always be
>significantly greater than the default limit for blocks that the client
>itself will generate.
>
>- There should be different size limits for side chains that are longer
>than the currently active chain. In other words, I might reject a block
>for being slightly too large, but if everyone else accepts it I should
>eventually accept it too, and my client should also consider
>automatically raising my size limit if this happens a lot.
>
>The rationale for the soft limit is to allow for gradual upward
>adjustment. It needs to be risky for individual miners to raise the size
>of their blocks to new heights, but ideally there won't be one solid
>wall for them to run into.
>
>On the user side: I would display the fee on the Send Coins dialog and
>allow users to choose a different fee per transaction. We also talked
>about adding some UI feedback where the client tries to estimate how
>long a transaction will take to confirm given a certain fee, based on
>recent information about what it observed from the network. If the fee
>can be changed on the Send Coins tab, then this could be a red, yellow,
>green visual indication whether the fee is sufficient, adequate or
>dangerously low.
>
>A criticism one might raise is: "The block size limit is not to protect
>miners, but to protect end users who may have less resources than miners
>and can't download gigantic block chains." - That's a viewpoint that is
>certainly valid. I believe that we will be able to do a lot just with
>efficiency improvements, pruning, compression and whatnot. But when it
>comes down to it, I'd prefer a large network with cheap
>microtransactions even if that means that consumer hardware can't
>operate as a standalone validating node anymore. Headers-only mode is
>already a much-requested feature anyway and there are many ways of
>improving the security of various header-only or lightweight protocols.
>
>(I just saw Greg's message advocating the opposite viewpoint, I'll
>respond to that as soon as I can.)
>
>
>
>> (1) Change the mining code to group transactions together with their
>> mempool dependencies and then calculate all fees as a group.
>
>+1 Very good change. This would allow miners to maximize their revenue
>and in doing so better represent the existing priorities that users
>express through fees.
>
>
>
>> There was discussion of some one-off changes to address the current
>> situation, namely de-ranking transactions that re-use addresses.
>
>Discouraging address reuse will not change the amount of transactions, I
>think we all agree on that. As for whether it improves the
>prioritization, I'm not sure. Use cases that we seek to discourage may
>simply switch to random addresses and I don't agree in and of itself
>this is a benefit (see item 4 below). Here are a few reasons one might
>be against this proposal:
>
>1) Certain use cases like green addresses will be forced to become more
>complicated than they would otherwise need to be.
>
>2) It will be harder to read information straight out of the block
>chain, for example right now we can pretty easily see how much volume is
>caused by Satoshi Dice, perhaps allowing us to make better decisions.
>
>3) The address index that is used by block explorers and lightweight
>client servers will grow unnecessarily (an address -> tx index will be
>larger if the number of unique addresses increases given the same number
>of txs), so for people like myself who work on that type of software
>you're actually making our scalability equation slightly worse.
>
>4) You're forcing people into privacy best practices which you think are
>good, but others may not subscribe to. For example I have absolutely
>zero interest in privacy, anyone who cares that I buy Bitcoins with my
>salary and spend them on paragliding is welcome to know about it.
>Frankly, if I cared about privacy I wouldn't be using Bitcoin. If other
>people want to use mixing services and randomize their addresses and
>communicate through Tor that's fine, but the client shouldn't force me
>to do those things if I don't want to by "deprioritizing" my transactions.
>
>5) We may not like firstbits, but the fact remains that for now they are
>extremely popular, because they improve the user experience where we
>failed to do so. If you deprioritize transactions to reused addresses
>you'll for example deprioritize all/most of Girls Gone Bitcoin, which
>(again, like it or not) is one of the few practical, sustainable niches
>that Bitcoin has managed to carve out for itself so far.
>
>
>
>> Having senders/buyers pay no fees is psychologically desirable even
>> though we all understand that eventually, somebody, somewhere will be
>> paying fees to use Bitcoin
>
>Free is just an extreme form of cheap, so if we can make transactions
>very cheap (through efficiency and very large blocks) then it will be
>easier for charitable miners to include free transactions. In practice,
>my prediction is that free transactions on the open network will simply
>not be possible in the long run. Dirty hacks aside there is simply no
>way of distinguishing a spam transaction from a charity-worthy
>transaction. So the way I envision free transactions in the future is
>that there may be miners in partnership with wallet providers like
>BlockChain.info that let you submit feeless transactions straight to
>them based on maybe a captcha or some ads. (For the purist, the captcha
>challenge and response could be communicated across the bitcoin network,
>but I think we agree that such things should ideally take place
>out-of-band.)
>
>That way, the available charity of miners who wish to include feeless
>transactions would go to human users as opposed to the potentially
>infinite demand of auto-generated feeless transactions.
>
>
>
>
>On 6/15/2012 1:29 PM, Mike Hearn wrote:
>> I had to hit the sack last night as it was 2am CET, but I'd like to
>> sum up the discussion we had on IRC about scalability and SatoshiDice
>> in particular.
>>
>> I think we all agreed on the following:
>>
>> - Having senders/buyers pay no fees is psychologically desirable even
>> though we all understand that eventually, somebody, somewhere will be
>> paying fees to use Bitcoin
>>
>> - In the ideal world Bitcoin would scale perfectly and there would be
>> no need for there to be some "winners" and some "losers" when it comes
>> to confirmation time.
>>
>> There was discussion of some one-off changes to address the current
>> situation, namely de-ranking transactions that re-use addresses. Gavin
>> and myself were not keen on this idea, primarily because it just
>> avoids the real problem and Bitcoin already has a good way to
>> prioritize transactions via the fees mechanism itself. The real issue
>> is that SatoshiDice does indeed pay fees and generates a lot of
>> transactions, pushing more traditional traffic out due to artificial
>> throttles.
>>
>> The following set of proposals were discussed:
>>
>> (1) Change the mining code to group transactions together with their
>> mempool dependencies and then calculate all fees as a group. A tx with
>> a fee of 1 BTC that depends on 5 txns with zero fees would result in
>> all 6 transactions being considered to have a fee of 1BTC and
>> therefore become prioritized for inclusion. This allows a transition
>> to "receiver pays" model for fees. There are many advantages. One is
>> that it actually makes sense ... it's always the receiver who wants
>> confirmations because it's the receiver that fears double spends.
>> Senders never do. What's more, whilst Bitcoin is designed to operate
>> on a zero-trust model in the real world trust often exists and it can
>> be used to optimize by passing groups of transactions around with
>> their dependencies, until that group passes a trust boundary and gets
>> broadcast with a send-to-self tx to add fees. Another advantage is it
>> simplifies usage for end users who primarily buy rather than sell,
>> because it avoids the need to guess at fees, one of the most
>> problematic parts of Bitcoins design now.
>>
>> The disadvantages are that it can result in extra transactions that
>> exist only for adding fees, and it requires a more modern payment
>> protocol than the direct-IP protocol Satoshi designed.
>>
>> It would help address the current situation by avoiding angry users
>> who want to buy things, but don't know what fee to set and so their
>> transactions get stuck.
>>
>> (2) SatoshiDice should use the same fee algorithms as Bitcoin-Qt to
>> avoid paying excessive fees and queue-jumping. Guess that's on my
>> plate.
>>
>> (3) Scalability improvements seem like a no brainer to everyone, it's
>> just a case of how complicated they are.
>>
>> (4) Making the block size limit float is better than picking a new
>> arbitrary threshold.
>>
>> On the forums Matt stated that block chain pruning was a no-go because
>> "it makes bitcoin more centralized". I think we've thrashed this one
>> out sufficiently well by now that there should be a united opinion on
>> it. There are technical ways to implement it such that there is no
>> change of trust requirements. All the other issues (finding archival
>> nodes, etc) can be again addressed with sufficient programming.
>>
>> For the case of huge blocks slowing down end user syncing and wasting
>> their resources, SPV clients like MultiBit and Android Wallet already
>> exist and will get better with time. If Jeff implements the bloom
>> filtering p2p commands I'll make bitcoinj use them and that'll knock
>> out excessive bandwidth usage and parse overheads from end users who
>> are on these clients. At some point Bitcoin-Qt can have a dual mode,
>> but who knows when that'll get implemented.
>>
>> Does that all sound reasonable?
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
>------------------------------------------------------------------------------
>Live Security Virtual Conference
>Exclusive live event will cover all the ways today's security and
>threat landscape has changed and how IT managers can respond. Discussions
>will include endpoint security, mobile security and the latest in malware
>threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>_______________________________________________
>Bitcoin-development mailing list
>Bitcoin-development@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>


-- 
Mike Koss
CTO, CoinLab
(425) 246-7701 (m)

A Bitcoin Primer - What you need to know about Bitcoins.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Bitcoin-development mailing list
Bitcoin-development@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bitcoin-development

Re: [Bitcoin-development] Near-term scalability

[Amir Taaki]

> less expensive. This is no more "real" or less "artificial" then an
> imposed licensing fee or the like and it is not subject to market
> forces.

Sure, the market is not always efficient nor desirable. This seems more like a social question though about choice and information. I do strongly feel that users should have more control over their technology, and a say in how Bitcoin operates. It is our job to present the choices and inform them to make good decisions. If we think how to implement this with a social component of the users operating the network rather than hard and fast rules, I think that's the preferrable way.

Part of the problem is that Satoshi didn't totally anticipate the growth of the network. The block reward (the subsidy) is too high, which is why transactions can afford to be so cheap. What would happen if blocks required a cumulative fee of XN BTC for N transactions before being accepted?



----- Original Message -----
From: Gregory Maxwell <gmaxwell@gmail.com>
To: Amir Taaki <zgenjix@yahoo.com>
Cc: 
Sent: Friday, June 15, 2012 8:43 PM
Subject: Re: [Bitcoin-development] Near-term scalability

On Fri, Jun 15, 2012 at 2:38 PM, Amir Taaki <zgenjix@yahoo.com> wrote:
> Forcing users to switch addresses per received payment to work around a bad fee system would be a braindead decision. You might love software and playing with web plugins, but not everyone does. Artists like Rap News can right now simply throw up an address and begin accepting donations. That's a hugely powerful and impactful selling point for Bitcoin.

And that use case does not need fast confirmations!

This is making the point.

>there are no artificial barriers to entry beyond the base cost. This network would always be competitive and strictly operate based on market dynamics.

The users of bitcoin can collectively choose how expensive operating a
full node is by accepting validation rules that allow it to be more or
less expensive. This is no more "real" or less "artificial" then an
imposed licensing fee or the like and it is not subject to market
forces.

Re: [Bitcoin-development] Near-term scalability

[Gregory Maxwell]

On Fri, Jun 15, 2012 at 2:50 PM, Amir Taaki <zgenjix@yahoo.com> wrote:
> Part of the problem is that Satoshi didn't totally anticipate the growth of the network. The block reward (the subsidy) is too high, which is why transactions can afford to be so cheap. What would happen if blocks required a cumulative fee of XN BTC for N transactions before being accepted?

I would take the last block I solved and use it to write a transaction
to nowhere which which gave all 50 BTC out in fee.  This pays for as
many transactions in the block as I like for any value of X you want
to choose.

You should read the bitcointalk forums more often: variants on that
idea are frequently suggested and dismantled. There is a lot of noise
there but also a lot of ideas and knowing what doesn't work is good
too.

Re: [Bitcoin-development] Near-term scalability

[Gavin Andresen]

> (1) Change the mining code to group transactions together with their
> mempool dependencies and then calculate all fees as a group.

I think there is general consensus this is a good idea.

> (2) SatoshiDice should use the same fee algorithms as Bitcoin-Qt to
> avoid paying excessive fees and queue-jumping. Guess that's on my
> plate.

I don't think the problem is with SatoshiDice, but is with the current
fixed/inflexible fee rules:

----------------------------

I've been thinking about fees a lot the last couple of days, and I'm
close to making a formal proposal. Here are my thoughts so far:

It seems to me there are two typical use cases:

Case 1:  I want my transaction to be confirmed quickly, and am willing
to pay a fee to make that happen.

Case 2: I want my transaction to be confirmed eventually. I'd rather
not pay, unless I have to.

I don't think the current code handles those two cases as well as it
could; here's a proposal to fix that:

o Let miners decide on how many free transactions they'll support, by
letting them specify how much of each block to set aside for 'free'
transactions (bytes) and what the threshold for 'free' is
(bitcoins/kilobyte). I think a new RPC call to get/set the parameters
dynamically is the right way to go.

o Change the block creation code to calculate a
bitcoin-fee-per-kilobyte for each transaction, where the fee and size
are both calculated based on the transaction and it's dependent
descendants (so we get the receiver-can-opt-to-pay-the-fee behavior we
want). Order transactions so highest-paying get into the non-free
space first.

o Fill up the "free" space (if any) with the highest-priority
transactions, where priority is a function of transaction size, age of
inputs, number of bitcoins... and ratio of inputs to outputs (to
encourage combining inputs so more pruning is possible).

The fee-paying space in a block lets Use Case #1 users compete to make
their fees high enough to get into the block.

The free space should let non-spamming Use Case #2 users (who don't
send a lot of transactions, and so have well-aged, high-priority
inputs) send transactions for free, at least as long as there are
miners willing to accept free transactions.

The question is: how do clients suggest fees to users if miners might
have very different fee policies?

I think full, transaction-verifying clients can watch how long
transactions stay in the memory pool to figure it out. I'm gathering
statistics right now to test a couple of simple heuristic algorithms
for reasonable fee/priority policies.

But that won't work for newly started clients that haven't seen a lot
of transactions enter/exit the memory pool, or SPV clients that can't
lookup transaction inputs (so can't calculate what fees are being paid
-- and once we have bloom filters may not be paying attention anything
but their own transactions, anyway).

I'm still thinking about that.

Maybe a new p2p network command: you give me a list of block hashes, I
tell you average fees paid per kilobyte for fee-paying transactions in
those blocks, and minimum and average priority of free transactions in
those blocks.

Maybe the big mining pools all publish their fee policies and that
information somehow gets to clients (encoded in the coinbase? ... but
they have a strong incentive to lie to try put upward pressure on
fees... ).

Maybe each client developer runs a "fee policy server" and clients
periodically ask it for reasonable fee rules (HTTP fetch info from a
web page that is updated as often or infrequently as is convenient,
maybe). I think I like this solution the best, it should let clients
compete to have the smartest/bestest algorithms for saving their
user's money on transaction fees.

-- 
--
Gavin Andresen

Re: [Bitcoin-development] Near-term scalability

[Mike Hearn]

[resend, sorry gavin]

I think these ideas all make a ton of sense, some have been floating
around for a while in various forms but it's good to draw them
together coherently.

> o Fill up the "free" space (if any) with the highest-priority
> transactions, where priority is a function of transaction size, age of
> inputs, number of bitcoins... and ratio of inputs to outputs (to
> encourage combining inputs so more pruning is possible).

Is more incentive needed? If you have tons of tiny outputs you already
have incentives to merge them because otherwise your txns will become
large and the fees needed to overcome the DoS limits and gain priority
will rise.

The code to do it is a bit irritating as you really want to de-frag
wallets in the background when the user is not likely to need the
outputs quickly, and I suspect over time transaction volumes will
become diurnal so it'd be cheaper to do that at night time, but it's
all possible.

> But that won't work for newly started clients that haven't seen a lot
> of transactions enter/exit the memory pool

Peers could provide first-seen timestamps for transactions when
announced or when downloaded with Jeffs proposed command, but the
timestamps are not necessarily trustable. Not sure if that'd open up
new attacks.

> or SPV clients that can't lookup transaction inputs

SPV clients can do it by getdata-ing on the relevant inputs, but it's
very bandwidth intensive just to guesstimate fees.

> Maybe each client developer runs a "fee policy server"

That's reasonable. I don't believe this case is worth worrying about
right now. For the common cases of

a) Customer buys from merchant (runs full node)
b) Trusted person sends money to trusting person (does not need confirms)

it wouldn't matter after the changes to the block creation code. It's
only really an issue when a user running an SPV client wishes to
accept money from somebody they do not trust, and they want it to
confirm quick-ish (within an hour), but can tolerate delays up to
that. I think this is likely to be rare.

Much more common is that you want to accept the payment immediately,
which is an oft discussed but different problem.