[bitcoin-dev] Properties of an ideal PoW algorithm & implementation

[bitcoin-dev] Properties of an ideal PoW algorithm & implementation

[Natanael]

[-- Attachment #1: Type: text/plain, Size: 4777 bytes --]

To expand on this below;

Den 18 apr. 2017 00:34 skrev "Natanael" <natanael.l@gmail.com>:

IMHO the best option if we change PoW is an algorithm that's moderately
processing heavy (we still need reasonably fast verification) and which
resists partial state reuse (not fast or fully "linear" in processing like
SHA256) just for the sake of invalidating asicboost style attacks, and it
should also have an existing reference implementation for hardware that's
provably close in performance to the theoretical ideal implementation of
the algorithm (in other words, one where we know there's no hidden
optimizations).

[...] The competition would mostly be about packing similar gate designs
closely and energy efficiency. (Now that I think about it, the proof MAY
have to consider energy use too, as a larger and slower but more efficient
chip still is competitive in mining...)


What matters for miners in terms of cost is primarily (correctly computed)
hashes per joule (watt-seconds). The most direct proxy for this in terms of
algorithm execution is the number of transistor (gate) activations per
computed hash (PoW unit).

To prove that an implementation is near optimal, you would show there's a
minimum number of necessary transistor activations per computed hash, and
that your implementation is within a reasonable range of that number.

We also need to show that for a practical implementation you can't reuse
much internal state (easiest way is "whitening" the block header,
pre-hashing or having a slow hash with an initial whitening step of its
own). This is to kill any ASICBOOST type optimization. Performance should
be constant, not linear relative to input size.

The PoW step should always be the most expensive part of creating a
complete block candidate! Otherwise it loses part of its meaning. It should
however still also be reasonably easy to verify.

Given that there's already PoW ASIC optimizations since years back that use
deliberately lossy hash computations just because those circuits can run
faster (X% of hashes are computed wrong, but you get Y% more computed
hashes in return which exceeds the error rate), any proof of an
implementation being near optimal (for mining) must also consider the
possibility of implementations of a design that deliberately allows errors
just to reduce the total count of transistor activations per N amount of
computed hashes. Yes, that means the reference implementation is allowed to
be lossy.

So for a reasonably large N (number of computed hashes, to take batch
processing into consideration), the proof would show that there's a
specific ratio for a minimum number of average gate activations per
correctly computed hash, a smallest ratio = X number of gate activations /
(N * success rate) across all possible implementations of the algorithm.
And you'd show your implementation is close to that ratio.

It would also have to consider a reasonable range of time-memory tradeoffs
including the potential of precomputation. Hopefully we could implement an
algorithm that effectively makes such precomputation meaningless by making
the potential gain insignificant for any reasonable ASIC chip size and
amount of precomputation resources.

A summary of important mining PoW algorithm properties;

* Constant verification speed, reasonably fast even on slow hardware

* As explained above, still slow / expensive enough to dominate the costs
of block candidate creation

* Difficulty must be easy to adjust (no problem for simple hash-style
algorithms like today)

* Cryptographic strength, something like preimage resistance (the algorithm
can't allow forcing a particular output, the chance must not be better than
random within any achievable computational bounds)

* As explained above, no hidden shortcuts. Everybody has equal knowledge.

* Predictable and close to constant PoW computation performance, and not
linear in performance relative to input size the way SHA256 is (lossy
implementations will always make it not-quite-constant)

* As explained above, no significant reusable state or other reusable work
(killing ASICBOOST)

* As explained above, no meaningful precomputation possible. No unfair
headstarts.

* Should only rely on just transistors for implementation, shouldn't rely
on memory or other components due to unknowable future engineering results
and changes in cost

* Reasonably compact implementation, measured in memory use, CPU load and
similar metrics

* Reasonably small inputs and outputs (in line with regular hashes)

* All mining PoW should be "embarrassingly parallel" (highly
parallellizable) with minimal or no gain from batch computation,
performance scaling should be linear with increased chip size & cycle
speed.

What else is there? Did I miss anything important?

[-- Attachment #2: Type: text/html, Size: 6450 bytes --]

Re: [bitcoin-dev] Properties of an ideal PoW algorithm & implementation

[praxeology_guy]

[-- Attachment #1: Type: text/plain, Size: 2561 bytes --]

Natanael,

=== Metal Layers ===

One factor in chip cost other than transistor count is the number of layers required to route all the interconnects in the desired die area constraint. The need for fewer layers can result in less patent-able costs of layering technology. Fewer layers are quicker and easier to manufacture.

I'm not an expert in the field, and I can't vouch for the validity of the entirety of the paper, but this paper discusses various factors that impact chip cost design.
http://www.cse.psu.edu/~juz138/files/3d-cost-tcad10.pdf

=== Early nonce mixing, Variable Length Input with Near Constant Work ===

To minimize asicboost like optimizations... the entirety of the input should be mixed with the nonce data ASAP. For example with Bitcoin as it is now, the 80 byte block header doesn't fully fit in one 64 byte SHA256 input block. This results in a 2nd SHA256 block input that only has 4 bytes of nonce and the rest constant that are mixed much later than the rest of the input... which allows for unexpected optimizations.

Solution: A hash algorithm that could have more linear computation time vs input size would be a 2 stage algorithm:
1. 1st stage Merkle tree hash to pre-lossy-mix-compress the variable length input stream to the size of the 2nd stage state vector. Each bit of input should have about equal influence on each of the output bits. (Minimize information loss, maximize mixed-ness).
2. Multi-round mixing of the 2nd stage, where this stage is significantly more work than the 1st stage.

This is somewhat done already in Bitcoin by the PoW doing SHA256 twice in serial. The first time is pretty much the merkle tree hash (a node with two children), and then the second time is the mult-round mixing. If the Bitcoin PoW did SHA256 three or four times or more, then asicboost like optimizations would have less of an effect.

In actual hardware, assuming a particular input length for the design can result in a significantly more optimized design than creating hardware that can handle a variable length input. So your design goal of "not linear in performance relative to input size" to me seems to be a hard one to attain... in practical, to support very large input sizes in a constant work fashion requires a trade off between memory/parallelization and die space. I think it would be better to make an assumption about the block header size, such as that it is exactly 80 bytes, or, at least something reasonable like the hardware should be able to support a block header size <= 128 bytes.

Cheers,
Praxeology Guy

[-- Attachment #2: Type: text/html, Size: 3023 bytes --]

Re: [bitcoin-dev] Properties of an ideal PoW algorithm & implementation

[Tim Ruffing]

On Tue, 2017-04-18 at 12:34 +0200, Natanael via bitcoin-dev wrote:
> To prove that an implementation is near optimal, you would show
> there's a minimum number of necessary transistor activations per
> computed hash, and that your implementation is within a reasonable
> range of that number. 

I'm not an expert on lower bounds of algorithms but I think proving
such properties is basically out of reach for mankind currently.

> 
> We also need to show that for a practical implementation you can't
> reuse much internal state (easiest way is "whitening" the block
> header, pre-hashing or having a slow hash with an initial whitening
> step of its own). This is to kill any ASICBOOST type optimization.
> Performance should be constant, not linear relative to input size. 

Yes, a reasonable thing in practice seems to use a slower hash function
(or just iterating the hash function many times), see also this thread:
 https://twitter.com/Ethan_Heilman/status/850015029189644288 .

PoW verification will still be fast enough. That's not the bottleneck
of block verification anyway.

Also, I don't agree that a PoW function should not rely on memory.
Memory-hard functions are the best we have currently.


Tim

Re: [bitcoin-dev] Properties of an ideal PoW algorithm & implementation

[Bram Cohen]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

Repeatedly hashing to make it so that lossy implementations just fail
sounds like a great idea. Also relying on a single crypto primitive which
is as simple as possible is also a great idea, and specifically using
blake2b is conservative because not only is it simple but its block size is
larger than the amount of data being hashed so asicboost-style attacks
don't apply at all and the logic of multiple blocks doesn't have to be
built.

Memory hard functions are a valiant effort and are holding up better than
expected but the problem is that when they fail they fail catastrophically,
immediately going from running on completely commodity hardware to only
running on hardware from the one vendor who's pulled off the feat of making
it work. My guess is it's only a matter of time until that happens.

So the best PoW function we know of today, assuming that you're trying to
make mining hardware as commodity as possible, is to repeatedly hash using
blake2b ten or maybe a hundred times.

Mind you, I still think hard forking the PoW function is a very bad idea,
but if you were to do it, that would be the way to go.

[-- Attachment #2: Type: text/html, Size: 1391 bytes --]