[bitcoin-dev] Adjusted difficulty depending on relative blocksize

[bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Jakob Rönnbäck]

Greetings,

a thought occurred to me that I would love to hear what some bitcoin experts think about.

What if one were to adjust the difficulty (for individual blocks) depending on the relative size to the average block size of the previous difficulty period? (I apologize if i’m not using the correct terms, I’m not a real programmer, and I’ve only recently started to subscribe to the mailing list)


In practice:

1. calculate average block size for the previous difficulty period (is it 2016-blocks?)
2. when trying to find a new block adjust the difficulty by adding the relative size difference. For instance, if i’m trying to create a block half (or double) the size of the average block size for the previous difficulty period then my difficulty will be 2x the normal one… if I’m trying to make one that is 30% bigger (or smaller) then the difficulty is 1.3 times the normal one


Right now this would force miners to make blocks as close to 1mb as possible (since the block reward >> fees). But unless I’m mistaken sometime in the future the block size should be adjusted to maximize the fees…


Could the concept be useful somehow?

I apologize if it’s been discussed before or if it’s a stupid idea, I would have run it by some other people, but I’m afraid I don’t know anyone that have any interest in bitcoin.

Regards
/jakob

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Angel Leon]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

Like this?
https://gist.github.com/gubatron/143e431ee01158f27db4

http://twitter.com/gubatron

On Fri, Aug 14, 2015 at 5:59 AM, Jakob Rönnbäck <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Greetings,
>
> a thought occurred to me that I would love to hear what some bitcoin
> experts think about.
>
> What if one were to adjust the difficulty (for individual blocks)
> depending on the relative size to the average block size of the previous
> difficulty period? (I apologize if i’m not using the correct terms, I’m not
> a real programmer, and I’ve only recently started to subscribe to the
> mailing list)
>
>
> In practice:
>
> 1. calculate average block size for the previous difficulty period (is it
> 2016-blocks?)
> 2. when trying to find a new block adjust the difficulty by adding the
> relative size difference. For instance, if i’m trying to create a block
> half (or double) the size of the average block size for the previous
> difficulty period then my difficulty will be 2x the normal one… if I’m
> trying to make one that is 30% bigger (or smaller) then the difficulty is
> 1.3 times the normal one
>
>
> Right now this would force miners to make blocks as close to 1mb as
> possible (since the block reward >> fees). But unless I’m mistaken sometime
> in the future the block size should be adjusted to maximize the fees…
>
>
> Could the concept be useful somehow?
>
> I apologize if it’s been discussed before or if it’s a stupid idea, I
> would have run it by some other people, but I’m afraid I don’t know anyone
> that have any interest in bitcoin.
>
> Regards
> /jakob
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 2547 bytes --]

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Jakob Rönnbäck]

[-- Attachment #1: Type: text/plain, Size: 3746 bytes --]

Hmm… well, yes and no. Mostly no :)

The main idea i was trying to describe was that the actual difficulty for the block could be adjusted according to how much the size of the proposed block differ compared to the average size of blocks in the previous difficulty period. Unless I’m being very dense atm your gist is just about dynamically adjusting the blocksize?


 I’ll give a numeric example to clarify a bit.

Assume the current difficulty was calculated to be 1000, and the average size of the blocks in the period used to calculate the difficulty was 500kb.
Example 1:
I’m now attempting to find a new block with a size of 450 kb, or 450/500 = 10% smaller than average. The difficulty would then be 1000 * 110% = 1100
Example 2:
If I instead was trying to make a block sized 10000 kb, or 10000/500 = 2000% bigger than average the difficulty would be adjusted to 1000*20 = 20000


Why I find this interesting is in a possible future when the block reward is insignificant compared to the transactions fees miners would make bigger blocks as fees rise. A miner could include more transactions into blocks as long as the fees are high enough to offset the reduced chance of actually finding the block. However, I now realize that there wouldn’t be any downward pressure below the average size if the price shrinks (using the particular numbers i have in my examples) though. Maybe this method is only useful on the upside of the blocks, meaning blocks smaller than the average size doesn’t get adjusted difficulty. I need to go for a walk and think this through :)


> 14 aug 2015 kl. 15:32 skrev Angel Leon <gubatron@gmail.com>:
> 
> Like this?
> https://gist.github.com/gubatron/143e431ee01158f27db4 <https://gist.github.com/gubatron/143e431ee01158f27db4>
> 
> http://twitter.com/gubatron <http://twitter.com/gubatron>
> 
> On Fri, Aug 14, 2015 at 5:59 AM, Jakob Rönnbäck <bitcoin-dev@lists.linuxfoundation.org <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
> Greetings,
> 
> a thought occurred to me that I would love to hear what some bitcoin experts think about.
> 
> What if one were to adjust the difficulty (for individual blocks) depending on the relative size to the average block size of the previous difficulty period? (I apologize if i’m not using the correct terms, I’m not a real programmer, and I’ve only recently started to subscribe to the mailing list)
> 
> 
> In practice:
> 
> 1. calculate average block size for the previous difficulty period (is it 2016-blocks?)
> 2. when trying to find a new block adjust the difficulty by adding the relative size difference. For instance, if i’m trying to create a block half (or double) the size of the average block size for the previous difficulty period then my difficulty will be 2x the normal one… if I’m trying to make one that is 30% bigger (or smaller) then the difficulty is 1.3 times the normal one
> 
> 
> Right now this would force miners to make blocks as close to 1mb as possible (since the block reward >> fees). But unless I’m mistaken sometime in the future the block size should be adjusted to maximize the fees…
> 
> 
> Could the concept be useful somehow?
> 
> I apologize if it’s been discussed before or if it’s a stupid idea, I would have run it by some other people, but I’m afraid I don’t know anyone that have any interest in bitcoin.
> 
> Regards
> /jakob
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org <mailto:bitcoin-dev@lists.linuxfoundation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
> 


[-- Attachment #2: Type: text/html, Size: 5465 bytes --]

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Anthony Towns]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

On 14 August 2015 at 11:59, Jakob Rönnbäck <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> What if one were to adjust the difficulty (for individual blocks)
> depending on the relative size to the average block size of the previous
> difficulty period? (I apologize if i’m not using the correct terms, I’m not
> a real programmer, and I’ve only recently started to subscribe to the
> mailing list)
>

​That would mean that as usage grew, blocksize could increase, but
confirmation times would also increase (though presumably less than
linearly). That seems like a loss?

If you also let the increase in confirmation time (due to miners finding
harder blocks rather than a reduction in hashpower) then get reflected back
as decreased difficulty, it'd probably be simpler to just dynamically
adjust the max blocksize wouldn't it?

Cheers,
aj​

-- 
Anthony Towns <aj@erisian.com.au>

[-- Attachment #2: Type: text/html, Size: 1919 bytes --]

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Jakob Rönnbäck]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]


> 14 aug 2015 kl. 16:20 skrev Anthony Towns <aj@erisian.com.au <mailto:aj@erisian.com.au>>:
> 
> On 14 August 2015 at 11:59, Jakob Rönnbäck <bitcoin-dev@lists.linuxfoundation.org <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
> What if one were to adjust the difficulty (for individual blocks) depending on the relative size to the average block size of the previous difficulty period? (I apologize if i’m not using the correct terms, I’m not a real programmer, and I’ve only recently started to subscribe to the mailing list)
> 
> ​That would mean that as usage grew, blocksize could increase, but confirmation times would also increase (though presumably less than linearly). That seems like a loss?
> 

Would that really be the case though? If it takes 5% to find a block, but it contains 5% more transactions would that not mean it’s the same? That would argue against the change if not for the fact that the blocks will be bigger for the next difficulty period.

> If you also let the increase in confirmation time (due to miners finding harder blocks rather than a reduction in hashpower) then get reflected back as decreased difficulty, it'd probably be simpler to just dynamically adjust the max blocksize wouldn't it?
> 

I guess that could make the difficulty fluctuate a bit depending on the amount of transactions and the fees being paid. Would it really matter in the long run though? Since it’s the same amount of miners, doesn’t that just mean it’s just the number that is lower, not the actual investment needed to mine the blocks? Not sure if this would open up some forms of attacks on the system for someone willing to lose money though…


Very good feedback though, thanks a lot :)

/jakob

[-- Attachment #2: Type: text/html, Size: 3470 bytes --]

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Anthony Towns]

[-- Attachment #1: Type: text/plain, Size: 2404 bytes --]

On 14 August 2015 at 16:48, Jakob Rönnbäck <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> 14 aug 2015 kl. 16:20 skrev Anthony Towns <aj@erisian.com.au>:
>
> On 14 August 2015 at 11:59, Jakob Rönnbäck <
> bitcoin-dev@lists.linuxfoundation.org> wrote:
>
>> What if one were to adjust the difficulty (for individual blocks)
>> depending on the relative size to the average block size of the previous
>> difficulty period? (I apologize if i’m not using the correct terms, I’m not
>> a real programmer, and I’ve only recently started to subscribe to the
>> mailing list)
>>
>
> ​That would mean that as usage grew, blocksize could increase, but
> confirmation times would also increase (though presumably less than
> linearly). That seems like a loss?
>
> Would that really be the case though? If it takes 5% to find a block, but
> it contains 5% more transactions would that not mean it’s the same? That
> would argue against the change if not for the fact that the blocks will be
> bigger for the next difficulty period.
>

​If you're waiting for one confirmation, something like that works -- you
might from 95% chance of 10 minutes 5% chance of 20 minutes to 100% chance
of 10m30s. But if you want 144 confirmations (eg) you go from 95% chance of
1 day, 5% chance of 1 day 10 minutes; to 100% chance of 1 day 72 minutes.

> If you also let the increase in confirmation time (due to miners finding
> harder blocks rather than a reduction in hashpower) then get reflected back
> as decreased difficulty, it'd probably be simpler to just dynamically
> adjust the max blocksize wouldn't it?
>
> I guess that could make the difficulty fluctuate a bit depending on the
> amount of transactions and the fees being paid. Would it really matter in
> the long run though? Since it’s the same amount of miners, doesn’t that
> just mean it’s just the number that is lower, not the actual investment
> needed to mine the blocks? Not sure if this would open up some forms of
> attacks on the system for someone willing to lose money though…
>

Once blocksizes had normalised as much larger than 1MB with a corresponding
higher average hashrate, a bad actor could easily mine a raft of valid
empty/small blocks at the minimum hash rate and force a reorg (and do
doublespends, etc).

​Cheers,
aj​

-- 
Anthony Towns <aj@erisian.com.au>

[-- Attachment #2: Type: text/html, Size: 4445 bytes --]

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Adam Back]

There is a proposal that relates to this, see the flexcap proposal by
Greg Maxwell & Mark Friedenbach, it was discussed on the list back in
May:

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008017.html

and http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008038.html

Adam

On 14 August 2015 at 15:48, Jakob Rönnbäck
<bitcoin-dev@lists.linuxfoundation.org> wrote:
>
> 14 aug 2015 kl. 16:20 skrev Anthony Towns <aj@erisian.com.au>:
>
> On 14 August 2015 at 11:59, Jakob Rönnbäck
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> What if one were to adjust the difficulty (for individual blocks)
>> depending on the relative size to the average block size of the previous
>> difficulty period? (I apologize if i’m not using the correct terms, I’m not
>> a real programmer, and I’ve only recently started to subscribe to the
>> mailing list)
>
>
> That would mean that as usage grew, blocksize could increase, but
> confirmation times would also increase (though presumably less than
> linearly). That seems like a loss?
>
>
> Would that really be the case though? If it takes 5% to find a block, but it
> contains 5% more transactions would that not mean it’s the same? That would
> argue against the change if not for the fact that the blocks will be bigger
> for the next difficulty period.
>
> If you also let the increase in confirmation time (due to miners finding
> harder blocks rather than a reduction in hashpower) then get reflected back
> as decreased difficulty, it'd probably be simpler to just dynamically adjust
> the max blocksize wouldn't it?
>
>
> I guess that could make the difficulty fluctuate a bit depending on the
> amount of transactions and the fees being paid. Would it really matter in
> the long run though? Since it’s the same amount of miners, doesn’t that just
> mean it’s just the number that is lower, not the actual investment needed to
> mine the blocks? Not sure if this would open up some forms of attacks on the
> system for someone willing to lose money though…
>
>
> Very good feedback though, thanks a lot :)
>
> /jakob
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Jakob Rönnbäck]

Ah, there we go. I should have dug deeper into the mailing list

Thanks

/jakob

> 14 aug 2015 kl. 17:03 skrev Adam Back <adam@cypherspace.org>:
> 
> There is a proposal that relates to this, see the flexcap proposal by
> Greg Maxwell & Mark Friedenbach, it was discussed on the list back in
> May:
> 
> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008017.html
> 
> and http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008038.html
> 
> Adam

[bitcoin-dev] libconsensus assertion fails if used in multiple threads

[Tamas Blummer]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

We integrated libconsensus into bits of proof. It works well, in-line for all test cases with our Java engine and is about 50% faster on a single thread.

The performance advantage unfortunatelly reverses if libconsensus is executed on several threads simultaneously as we do with the Java engine, since an error:

	Assertion failed: (pkey != NULL), function CECKey, file ecwrapper.cpp, line 96.

arises under that stress.

I guess that the cause is that thread callbacks as advised for OpenSSL on https://www.openssl.org/docs/crypto/threads.html are not registered.
Registering those however would require access to OpenSSL functions, not exported from the lib.

I’d be thankful for a pointer to a workaround.

Tamas Blummer

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] libconsensus assertion fails if used in multiple threads

[Cory Fields]

Ugh, what an unfortunate oversight!

The good news is that this issue should be solved in future versions
when we switch to the new libsecp256k1 lib for validation.

For now, I've thrown together a quick hack to allow a user-specifiable
callback for libbitcoinconsensus. I think it's not worth messing with
the official API since it will be fixed soon, but rather hacked in as
a temporary work-around as needed. It _should_ be documented as an
issue with the current version, though.

Please see here for a work-around to try:
https://github.com/theuni/bitcoin/commits/openssl-consensus-threads
Unfortunately it's not pretty, but it works fine here. Note that you
should give this some _serious_ testing before deploying in any real
way. It should mimic the way we do it in Core, though.

That's on top of current master, but it should be trivial to apply to
release tags.

Please let me know how it works out.

Regards,
Cory

On Fri, Aug 14, 2015 at 12:37 PM, Tamas Blummer via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> We integrated libconsensus into bits of proof. It works well, in-line for all test cases with our Java engine and is about 50% faster on a single thread.
>
> The performance advantage unfortunatelly reverses if libconsensus is executed on several threads simultaneously as we do with the Java engine, since an error:
>
>         Assertion failed: (pkey != NULL), function CECKey, file ecwrapper.cpp, line 96.
>
> arises under that stress.
>
> I guess that the cause is that thread callbacks as advised for OpenSSL on https://www.openssl.org/docs/crypto/threads.html are not registered.
> Registering those however would require access to OpenSSL functions, not exported from the lib.
>
> I’d be thankful for a pointer to a workaround.
>
> Tamas Blummer
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Tom Harding]

Nobody mentioned exchange rates.  Those matter to miners too.

Does it make sense for George Soros and every other rich person /
institution to have the power to move difficulty, even pin it to min or
max, just by buying or selling piles of BTC to swing the exchange rate?


On 8/14/2015 8:03 AM, Adam Back via bitcoin-dev wrote:
> There is a proposal that relates to this, see the flexcap proposal by
> Greg Maxwell & Mark Friedenbach, it was discussed on the list back in
> May:
>
> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008017.html
>
> and http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008038.html
>
> Adam
>
> On 14 August 2015 at 15:48, Jakob Rönnbäck
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> 14 aug 2015 kl. 16:20 skrev Anthony Towns <aj@erisian.com.au>:
>>
>> On 14 August 2015 at 11:59, Jakob Rönnbäck
>> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>> What if one were to adjust the difficulty (for individual blocks)
>>> depending on the relative size to the average block size of the previous
>>> difficulty period? (I apologize if i’m not using the correct terms, I’m not
>>> a real programmer, and I’ve only recently started to subscribe to the
>>> mailing list)
>>
>> That would mean that as usage grew, blocksize could increase, but
>> confirmation times would also increase (though presumably less than
>> linearly). That seems like a loss?
>>
>>
>> Would that really be the case though? If it takes 5% to find a block, but it
>> contains 5% more transactions would that not mean it’s the same? That would
>> argue against the change if not for the fact that the blocks will be bigger
>> for the next difficulty period.
>>
>> If you also let the increase in confirmation time (due to miners finding
>> harder blocks rather than a reduction in hashpower) then get reflected back
>> as decreased difficulty, it'd probably be simpler to just dynamically adjust
>> the max blocksize wouldn't it?
>>
>>
>> I guess that could make the difficulty fluctuate a bit depending on the
>> amount of transactions and the fees being paid. Would it really matter in
>> the long run though? Since it’s the same amount of miners, doesn’t that just
>> mean it’s just the number that is lower, not the actual investment needed to
>> mine the blocks? Not sure if this would open up some forms of attacks on the
>> system for someone willing to lose money though…
>>
>>
>> Very good feedback though, thanks a lot :)
>>
>> /jakob
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] libconsensus assertion fails if used in multiple threads

[Cory Fields]

Back to the list (from github) in case anyone finds this via Google.

The patch that I posted here a few days ago did not fix the issue for Tamas.

I spent some time tracking down this edge-case because
libbitcoinconsensus needs to be as bullet-proof as possible. Thanks to
Tamas for creating a bare-bones test case after some discussion.

I finally managed to reproduce the issue on OSX. It's subtle and
likely rare in the real-world, though obviously not impossible given
the report here. For posterity, here's a rundown (braindump) of the
issue.

When calling EC_KEY_new_by_curve_name(), openssl internally checks to
see how to setup the curve's EC_METHOD (simple, montgomery, or nist).

Unfortunately, in all released OpenSSL versions (as far as I can tell
master is the only branch that has fixed this issue), it's tested like
so:

- Try a method. If it fails, set a global error and return.
- If the global error is set, try a different method.

Prior to OpenSSL 1.0.0, these were tested in the order:
EC_GFp_nist_method -> EC_GFp_mont_method. The secp256k1 curve fails
the ec_GFp_nist_group_set_curve test and sets the global error. That
error is then checked for failure, and EC_GFp_mont_method is tried
(and succeeds).

Obviously that global error usage is dangerous, especially since it
happens for _each_ transaction verification in libbitcoinconsensus. In
a multi-threaded environment, a crash is guaranteed within a few
seconds.

However, OpenSSL 1.0.1 reversed the order, trying EC_GFp_mont_method
first, so that the global error doesn't end up being used:
https://github.com/openssl/openssl/commit/17674bfdf75bffa4e225f8328b9d42cb74504005

This was backported from master back to 1.0.1, but not to 1.0.0 or 0.9.8.

So that change (accidentally) "solved" the problem. As you can see,
it's still possible to hit the reversed order in the
!defined(OPENSSL_BN_ASM_MONT) case. That's easily tested by building
OpenSSL with the -no-asm config option. It's probably also the case
for obscure architectures and OSs, but I haven't looked deeply into
that. In that case, it's reasonable to assume that this crash would
likely occur on such platforms.

Also, OSX, even the latest version (10.10 as of now), still ships with
OpenSSL 0.9.8. Which is how Tamas ran into it.

Since Bitcoin Core and libbitcoinconsensus are switching away from
OpenSSL for verification in the near future, I don't think this is
much of an issue. Especially since the problem manifests as a
controlled assertion failure/abort. However, I've prepared a patch for
anyone who may run into the issue in the short-term:
https://github.com/theuni/bitcoin/commit/adf0a691ee1c2f02e26828f976cfe5b78896b507

I'll open a pull-request for Bitcoin Core to discuss whether it's
worth merging or not.

Regards,
Cory

On Fri, Aug 14, 2015 at 5:10 PM, Cory Fields <lists@coryfields.com> wrote:
> Ugh, what an unfortunate oversight!
>
> The good news is that this issue should be solved in future versions
> when we switch to the new libsecp256k1 lib for validation.
>
> For now, I've thrown together a quick hack to allow a user-specifiable
> callback for libbitcoinconsensus. I think it's not worth messing with
> the official API since it will be fixed soon, but rather hacked in as
> a temporary work-around as needed. It _should_ be documented as an
> issue with the current version, though.
>
> Please see here for a work-around to try:
> https://github.com/theuni/bitcoin/commits/openssl-consensus-threads
> Unfortunately it's not pretty, but it works fine here. Note that you
> should give this some _serious_ testing before deploying in any real
> way. It should mimic the way we do it in Core, though.
>
> That's on top of current master, but it should be trivial to apply to
> release tags.
>
> Please let me know how it works out.
>
> Regards,
> Cory
>
> On Fri, Aug 14, 2015 at 12:37 PM, Tamas Blummer via bitcoin-dev
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>> We integrated libconsensus into bits of proof. It works well, in-line for all test cases with our Java engine and is about 50% faster on a single thread.
>>
>> The performance advantage unfortunatelly reverses if libconsensus is executed on several threads simultaneously as we do with the Java engine, since an error:
>>
>>         Assertion failed: (pkey != NULL), function CECKey, file ecwrapper.cpp, line 96.
>>
>> arises under that stress.
>>
>> I guess that the cause is that thread callbacks as advised for OpenSSL on https://www.openssl.org/docs/crypto/threads.html are not registered.
>> Registering those however would require access to OpenSSL functions, not exported from the lib.
>>
>> I’d be thankful for a pointer to a workaround.
>>
>> Tamas Blummer
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>

Re: [bitcoin-dev] libconsensus assertion fails if used in multiple threads

[Tamas Blummer]

[-- Attachment #1.1: Type: text/plain, Size: 6472 bytes --]

Thanks a lot Cory for following through the test case and producing a patch.

I confirm that libconsensus is now running stable within the Bits of Proof stack,
in-line with test cases we use to verify the java implementation of the script engine,
that are BTW borrowed from Bitcoin Core.

The performance of libconsensus is surprisingly close to the java one.
Validating a 2-of-2 a multi-sig  transaction runs at 1021 ops/sec with java and 1135 ops/sec
in libconsensus. This is on a 2.2GH i7 laptop (4 hyper threading cores used by 8 threads).
Another nice demonstration why one should not trade in advances
of languages for the last decades for a marginal gain of performance with C/C++,
I assume thereby that Bouncy Castle’ EC lib s not superior to OpenSSL's.

I disagree that the problem was rare in the real-world, it should affect any modern
implementation that validates transactions parallel in multiple threads.

Aborting also does not make the problem less severe in my opinion.
Therefore hope the pull will be included into Core with next release.

I can’t assign a timeline to “near future" secp256k1 integration. Can you?

Tamas Blummer


> On Aug 18, 2015, at 07:03, Cory Fields <lists@coryfields.com> wrote:
> 
> Back to the list (from github) in case anyone finds this via Google.
> 
> The patch that I posted here a few days ago did not fix the issue for Tamas.
> 
> I spent some time tracking down this edge-case because
> libbitcoinconsensus needs to be as bullet-proof as possible. Thanks to
> Tamas for creating a bare-bones test case after some discussion.
> 
> I finally managed to reproduce the issue on OSX. It's subtle and
> likely rare in the real-world, though obviously not impossible given
> the report here. For posterity, here's a rundown (braindump) of the
> issue.
> 
> When calling EC_KEY_new_by_curve_name(), openssl internally checks to
> see how to setup the curve's EC_METHOD (simple, montgomery, or nist).
> 
> Unfortunately, in all released OpenSSL versions (as far as I can tell
> master is the only branch that has fixed this issue), it's tested like
> so:
> 
> - Try a method. If it fails, set a global error and return.
> - If the global error is set, try a different method.
> 
> Prior to OpenSSL 1.0.0, these were tested in the order:
> EC_GFp_nist_method -> EC_GFp_mont_method. The secp256k1 curve fails
> the ec_GFp_nist_group_set_curve test and sets the global error. That
> error is then checked for failure, and EC_GFp_mont_method is tried
> (and succeeds).
> 
> Obviously that global error usage is dangerous, especially since it
> happens for _each_ transaction verification in libbitcoinconsensus. In
> a multi-threaded environment, a crash is guaranteed within a few
> seconds.
> 
> However, OpenSSL 1.0.1 reversed the order, trying EC_GFp_mont_method
> first, so that the global error doesn't end up being used:
> https://github.com/openssl/openssl/commit/17674bfdf75bffa4e225f8328b9d42cb74504005
> 
> This was backported from master back to 1.0.1, but not to 1.0.0 or 0.9.8.
> 
> So that change (accidentally) "solved" the problem. As you can see,
> it's still possible to hit the reversed order in the
> !defined(OPENSSL_BN_ASM_MONT) case. That's easily tested by building
> OpenSSL with the -no-asm config option. It's probably also the case
> for obscure architectures and OSs, but I haven't looked deeply into
> that. In that case, it's reasonable to assume that this crash would
> likely occur on such platforms.
> 
> Also, OSX, even the latest version (10.10 as of now), still ships with
> OpenSSL 0.9.8. Which is how Tamas ran into it.
> 
> Since Bitcoin Core and libbitcoinconsensus are switching away from
> OpenSSL for verification in the near future, I don't think this is
> much of an issue. Especially since the problem manifests as a
> controlled assertion failure/abort. However, I've prepared a patch for
> anyone who may run into the issue in the short-term:
> https://github.com/theuni/bitcoin/commit/adf0a691ee1c2f02e26828f976cfe5b78896b507
> 
> I'll open a pull-request for Bitcoin Core to discuss whether it's
> worth merging or not.
> 
> Regards,
> Cory
> 
> On Fri, Aug 14, 2015 at 5:10 PM, Cory Fields <lists@coryfields.com> wrote:
>> Ugh, what an unfortunate oversight!
>> 
>> The good news is that this issue should be solved in future versions
>> when we switch to the new libsecp256k1 lib for validation.
>> 
>> For now, I've thrown together a quick hack to allow a user-specifiable
>> callback for libbitcoinconsensus. I think it's not worth messing with
>> the official API since it will be fixed soon, but rather hacked in as
>> a temporary work-around as needed. It _should_ be documented as an
>> issue with the current version, though.
>> 
>> Please see here for a work-around to try:
>> https://github.com/theuni/bitcoin/commits/openssl-consensus-threads
>> Unfortunately it's not pretty, but it works fine here. Note that you
>> should give this some _serious_ testing before deploying in any real
>> way. It should mimic the way we do it in Core, though.
>> 
>> That's on top of current master, but it should be trivial to apply to
>> release tags.
>> 
>> Please let me know how it works out.
>> 
>> Regards,
>> Cory
>> 
>> On Fri, Aug 14, 2015 at 12:37 PM, Tamas Blummer via bitcoin-dev
>> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>> We integrated libconsensus into bits of proof. It works well, in-line for all test cases with our Java engine and is about 50% faster on a single thread.
>>> 
>>> The performance advantage unfortunatelly reverses if libconsensus is executed on several threads simultaneously as we do with the Java engine, since an error:
>>> 
>>>        Assertion failed: (pkey != NULL), function CECKey, file ecwrapper.cpp, line 96.
>>> 
>>> arises under that stress.
>>> 
>>> I guess that the cause is that thread callbacks as advised for OpenSSL on https://www.openssl.org/docs/crypto/threads.html are not registered.
>>> Registering those however would require access to OpenSSL functions, not exported from the lib.
>>> 
>>> I’d be thankful for a pointer to a workaround.
>>> 
>>> Tamas Blummer
>>> 
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev@lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>> 
> 


[-- Attachment #1.2: Type: text/html, Size: 9355 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: [bitcoin-dev] libconsensus assertion fails if used in multiple threads

[Cory Fields]

See responses inline.

On Tue, Aug 18, 2015 at 6:31 AM, Tamas Blummer <tamas@bitsofproof.com> wrote:
> Thanks a lot Cory for following through the test case and producing a patch.
>
> I confirm that libconsensus is now running stable within the Bits of Proof
> stack,
> in-line with test cases we use to verify the java implementation of the
> script engine,
> that are BTW borrowed from Bitcoin Core.
>
> The performance of libconsensus is surprisingly close to the java one.
> Validating a 2-of-2 a multi-sig  transaction runs at 1021 ops/sec with java
> and 1135 ops/sec
> in libconsensus. This is on a 2.2GH i7 laptop (4 hyper threading cores used
> by 8 threads).
> Another nice demonstration why one should not trade in advances
> of languages for the last decades for a marginal gain of performance with
> C/C++,
> I assume thereby that Bouncy Castle’ EC lib s not superior to OpenSSL's.

A few points there. First, Core is switching to libsecp256k1 for
several reasons, and one of them is speed. I seem to recall it being
up to 8x faster than OpenSSL.

Also, it can depend heavily on compiler switches and optimization
levels. For example, In playing with my test-case for hitting the
OpenSSL race issue, I managed to get a ~100% speedup by simply using
-O3 and lto.

>
> I disagree that the problem was rare in the real-world, it should affect any
> modern
> implementation that validates transactions parallel in multiple threads.
>

Well I'd say you're a bit biased in this case ;)

It's only those using ancient (0.98 or 1.00) versions of OpenSSL who
are affected, or those with OPENSSL_BN_ASM_MONT support disabled or
missing. Note that official releases of libbitcoinconsensus are
compiled against a much newer version and shouldn't have any issues.

The earlier patches for locking callbacks should be unnecessary.

> Aborting also does not make the problem less severe in my opinion.

Well it's not a good thing by any means, but it's certainly better
than incorrect results! In any undefined/error condition for the
consensus library, aborting is the right thing to do. If we can't
explain how we've reached a certain "unreachable" condition as is the
case here, the only reasonable recourse is to shut down. Otherwise we
risk network forks, DOS, etc.

> Therefore hope the pull will be included into Core with next release.
>

It will likely be unnecessary for the next release, but I do think
it's worth backporting to the 0.10 and 0.9 series.

> I can’t assign a timeline to “near future" secp256k1 integration. Can you?

I believe the libsecp256k1 guys are generally happy with the lib these
days, but I'll avoid guessing at a timeline. We can discuss that on
the PR for this fix, which I'll do today.

Regards,
Cory

Re: [bitcoin-dev] libconsensus assertion fails if used in multiple threads

[Cory Fields]

Pull request submitted: https://github.com/bitcoin/bitcoin/pull/6571

Regards,
Cory

On Tue, Aug 18, 2015 at 1:25 PM, Cory Fields <lists@coryfields.com> wrote:
> See responses inline.
>
> On Tue, Aug 18, 2015 at 6:31 AM, Tamas Blummer <tamas@bitsofproof.com> wrote:
>> Thanks a lot Cory for following through the test case and producing a patch.
>>
>> I confirm that libconsensus is now running stable within the Bits of Proof
>> stack,
>> in-line with test cases we use to verify the java implementation of the
>> script engine,
>> that are BTW borrowed from Bitcoin Core.
>>
>> The performance of libconsensus is surprisingly close to the java one.
>> Validating a 2-of-2 a multi-sig  transaction runs at 1021 ops/sec with java
>> and 1135 ops/sec
>> in libconsensus. This is on a 2.2GH i7 laptop (4 hyper threading cores used
>> by 8 threads).
>> Another nice demonstration why one should not trade in advances
>> of languages for the last decades for a marginal gain of performance with
>> C/C++,
>> I assume thereby that Bouncy Castle’ EC lib s not superior to OpenSSL's.
>
> A few points there. First, Core is switching to libsecp256k1 for
> several reasons, and one of them is speed. I seem to recall it being
> up to 8x faster than OpenSSL.
>
> Also, it can depend heavily on compiler switches and optimization
> levels. For example, In playing with my test-case for hitting the
> OpenSSL race issue, I managed to get a ~100% speedup by simply using
> -O3 and lto.
>
>>
>> I disagree that the problem was rare in the real-world, it should affect any
>> modern
>> implementation that validates transactions parallel in multiple threads.
>>
>
> Well I'd say you're a bit biased in this case ;)
>
> It's only those using ancient (0.98 or 1.00) versions of OpenSSL who
> are affected, or those with OPENSSL_BN_ASM_MONT support disabled or
> missing. Note that official releases of libbitcoinconsensus are
> compiled against a much newer version and shouldn't have any issues.
>
> The earlier patches for locking callbacks should be unnecessary.
>
>> Aborting also does not make the problem less severe in my opinion.
>
> Well it's not a good thing by any means, but it's certainly better
> than incorrect results! In any undefined/error condition for the
> consensus library, aborting is the right thing to do. If we can't
> explain how we've reached a certain "unreachable" condition as is the
> case here, the only reasonable recourse is to shut down. Otherwise we
> risk network forks, DOS, etc.
>
>> Therefore hope the pull will be included into Core with next release.
>>
>
> It will likely be unnecessary for the next release, but I do think
> it's worth backporting to the 0.10 and 0.9 series.
>
>> I can’t assign a timeline to “near future" secp256k1 integration. Can you?
>
> I believe the libsecp256k1 guys are generally happy with the lib these
> days, but I'll avoid guessing at a timeline. We can discuss that on
> the PR for this fix, which I'll do today.
>
> Regards,
> Cory

Re: [bitcoin-dev] libconsensus assertion fails if used in multiple threads

[Eric Voskuil]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

On 08/18/2015 03:31 AM, Tamas Blummer via bitcoin-dev wrote:
> The performance of libconsensus is surprisingly close to the java one. 
...
> Another nice demonstration why one should not trade in advances
> of languages for the last decades for a marginal gain of performance
> with C/C++...

If performance was the only trade-off with Java, and if C++ was frozen
in time, I would say you have a point.

Thanks for submitting, and Cory, thanks for working this out.

e


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Tom Harding]

There is another concern regarding "flexcap" that was not discussed.

A change to difficulty in response to anything BUT observed block
production rate unavoidably changes the money supply schedule, unless
you also change the reward, and in that case you've still changed the
timing even if not the average rate.


On 8/14/2015 8:14 AM, Jakob Rönnbäck via bitcoin-dev wrote:
> Ah, there we go. I should have dug deeper into the mailing list
>
> Thanks
>
> /jakob
>
>> 14 aug 2015 kl. 17:03 skrev Adam Back <adam@cypherspace.org>:
>>
>> There is a proposal that relates to this, see the flexcap proposal by
>> Greg Maxwell & Mark Friedenbach, it was discussed on the list back in
>> May:
>>
>> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008017.html
>>
>> and http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008038.html
>>
>> Adam
>

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Warren Togami Jr.]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

Does it really change the schedule when the next difficulty retarget
readjusts to an average of 10 minutes again?

On Tue, Sep 8, 2015 at 8:27 PM, Tom Harding via bitcoin-dev <
bitcoin-dev@lists.linuxfoundation.org> wrote:

>
> There is another concern regarding "flexcap" that was not discussed.
>
> A change to difficulty in response to anything BUT observed block
> production rate unavoidably changes the money supply schedule, unless
> you also change the reward, and in that case you've still changed the
> timing even if not the average rate.
>
>

[-- Attachment #2: Type: text/html, Size: 952 bytes --]

Re: [bitcoin-dev] Adjusted difficulty depending on relative blocksize

[Tom Harding]

[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]


Well let's see.  All else being equal, if everybody uses difficulty to 
buy big blocks during retarget interval 0, blocks and therefore money 
issuance is slower during that interval.  Then, the retargeting causes 
it to be faster during interval 1.  Subsidy got shifted from the 
calendar period corresponding to interval 0, to interval 1.

If you change the reward, you can lower the time-frame of the effect to 
the order of a single block interval, but there is still an effect.

These schemes do not avoid the need for a hard cap, and there are new 
rules for the size of the allowed adjustment, in addition to the main 
rule relating difficulty to block size.  So it seems they generally have 
more complexity than the other blocksize schemes being considered.


On 9/9/2015 11:59 AM, Warren Togami Jr. via bitcoin-dev wrote:
> Does it really change the schedule when the next difficulty retarget 
> readjusts to an average of 10 minutes again?
>
> On Tue, Sep 8, 2015 at 8:27 PM, Tom Harding via bitcoin-dev 
> <bitcoin-dev@lists.linuxfoundation.org 
> <mailto:bitcoin-dev@lists.linuxfoundation.org>> wrote:
>
>
>     There is another concern regarding "flexcap" that was not discussed.
>
>     A change to difficulty in response to anything BUT observed block
>     production rate unavoidably changes the money supply schedule, unless
>     you also change the reward, and in that case you've still changed the
>     timing even if not the average rate.
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev


[-- Attachment #2: Type: text/html, Size: 3204 bytes --]