[bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[Scott Roberts]

Bitcoin cash will hard fork on Nov 13 to implement a new difficulty
algorithm.  Bitcoin itself might need to hard fork to employ a similar
algorithm. It's about as good as they come because it followed the
"simplest is best" route. Their averaging window is probably
significantly too long (N=144). It's:

next_D = sum (past 144 D's) * T / sum(past 144 solvetimes)

They correctly did not use max(timestamp) - min(timestamp) in the
denominator like others do.

They've written the code and they're about to use it live, so Bitcoin
will have a clear, simple, and tested path if it suddenly needs to
hard fork due to having 20x delays for the next 2000 blocks (taking it
a year to get unstuck).

Details on it and the decision process:
https://www.bitcoinabc.org/november

It uses a nice median of 3 for the beginning and end of the window to
help alleviate bad timestamp problems. It's nice, helps a little, but
will also slow its response by 1 block.  They also have 2x and 1/2
limits on the adjustment per block, which is a lot more than they will
ever need.

I recommend bitcoin consider using it and making it N=50 instead of 144.

I have seen that any attempts to modify the above with things like a
low pass filter, starting the window at MTP, or preventing negative
timestamps will only reduce its effectiveness. Bitcoin's +12 and -6
limits on the timestamps are sufficient and well chosen, although
something a bit smaller than the +12 might have been better.

One of the contenders to the above is new and actually better, devised
by Degnr8 and they call it D622 or wt-144.It's a little better than
they realize. It's the only real improvement in difficulty algorithms
since the rolling average.  It gives a linearly higher weight to the
more recent timestamps. Otherwise it is the same. Others have probably
come across it, but there is too much noise in difficulty algorithms
to find the good ones.

# Degnr8's D622 difficulty algorithm
# T=TargetTime, S=Solvetime
# modified by zawy
for i = 1 to N  (from oldest to most recent block)
    t += T[i] / D[i] * i
    j += i
next i
next_D = j / t * T

I believe any modification to the above strict mathematical weighted
average will reduce it's effectiveness. It does not oscillate anymore
than regular algos and rises faster and drops faster, when needed.

Re: [bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[Gregory Maxwell]

On Thu, Nov 2, 2017 at 11:31 PM, Scott Roberts via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Bitcoin cash will hard fork on Nov 13 to implement a new difficulty
> algorithm.  Bitcoin itself might need to hard fork to employ a similar
> algorithm. It's about as good as they come because it followed the

This is the bitcoin development mailing list, not the "give free
review to the obviously defective proposals of adversarial competing
systems" mailing list. Your posting is off-topic.

Re: [bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[CryptAxe]

[-- Attachment #1: Type: text/plain, Size: 2881 bytes --]

Is there an issue with the current difficulty adjustment algorithm? It's
worked very well as far as I can tell. Introducing a new one seems pretty
risky, what would the benefit be?

On Nov 2, 2017 4:34 PM, "Scott Roberts via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> Bitcoin cash will hard fork on Nov 13 to implement a new difficulty
> algorithm.  Bitcoin itself might need to hard fork to employ a similar
> algorithm. It's about as good as they come because it followed the
> "simplest is best" route. Their averaging window is probably
> significantly too long (N=144). It's:
>
> next_D = sum (past 144 D's) * T / sum(past 144 solvetimes)
>
> They correctly did not use max(timestamp) - min(timestamp) in the
> denominator like others do.
>
> They've written the code and they're about to use it live, so Bitcoin
> will have a clear, simple, and tested path if it suddenly needs to
> hard fork due to having 20x delays for the next 2000 blocks (taking it
> a year to get unstuck).
>
> Details on it and the decision process:
> https://www.bitcoinabc.org/november
>
> It uses a nice median of 3 for the beginning and end of the window to
> help alleviate bad timestamp problems. It's nice, helps a little, but
> will also slow its response by 1 block.  They also have 2x and 1/2
> limits on the adjustment per block, which is a lot more than they will
> ever need.
>
> I recommend bitcoin consider using it and making it N=50 instead of 144.
>
> I have seen that any attempts to modify the above with things like a
> low pass filter, starting the window at MTP, or preventing negative
> timestamps will only reduce its effectiveness. Bitcoin's +12 and -6
> limits on the timestamps are sufficient and well chosen, although
> something a bit smaller than the +12 might have been better.
>
> One of the contenders to the above is new and actually better, devised
> by Degnr8 and they call it D622 or wt-144.It's a little better than
> they realize. It's the only real improvement in difficulty algorithms
> since the rolling average.  It gives a linearly higher weight to the
> more recent timestamps. Otherwise it is the same. Others have probably
> come across it, but there is too much noise in difficulty algorithms
> to find the good ones.
>
> # Degnr8's D622 difficulty algorithm
> # T=TargetTime, S=Solvetime
> # modified by zawy
> for i = 1 to N  (from oldest to most recent block)
>     t += T[i] / D[i] * i
>     j += i
> next i
> next_D = j / t * T
>
> I believe any modification to the above strict mathematical weighted
> average will reduce it's effectiveness. It does not oscillate anymore
> than regular algos and rises faster and drops faster, when needed.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 3745 bytes --]

Re: [bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[Scott Roberts]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

Whatever their failings from their previous code or their adversarial
nature, they got this code right and I'm only presenting it as a real and
excellent solution for the impending threat to bitcoin. As a big core fan,
I really wanted to delete the word Cash from my post because I was afraid
someone would turn this technical discussion into a political football.

On Nov 2, 2017 7:37 PM, "Gregory Maxwell" <greg@xiph.org> wrote:

On Thu, Nov 2, 2017 at 11:31 PM, Scott Roberts via bitcoin-dev
<bitcoin-dev@lists.linuxfoundation.org> wrote:
> Bitcoin cash will hard fork on Nov 13 to implement a new difficulty
> algorithm.  Bitcoin itself might need to hard fork to employ a similar
> algorithm. It's about as good as they come because it followed the

This is the bitcoin development mailing list, not the "give free
review to the obviously defective proposals of adversarial competing
systems" mailing list. Your posting is off-topic.

[-- Attachment #2: Type: text/html, Size: 1447 bytes --]

Re: [bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[Gregory Maxwell]

On Thu, Nov 2, 2017 at 11:53 PM, Scott Roberts <wordsgalore@gmail.com> wrote:
> Whatever their failings from their previous code or their adversarial
> nature, they got this code right and I'm only presenting it as a real and
> excellent solution for the impending threat to bitcoin. As a big core fan, I
> really wanted to delete the word Cash from my post because I was afraid
> someone would turn this technical discussion into a political football.

I urge my colleagues here to not fall for the obvious xkcd386 bait.

The competitive advantage of prudence and competence is diminished if
competitors are able to divert our efforts into reviewing their
proposals.

Re: [bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[gb]

You launched the political football by coming here with a verbose
'recommendation'. Without a code submission in form of pull request to
the core repo on github this was never a technical discussion.

On Thu, 2017-11-02 at 19:53 -0400, Scott Roberts via bitcoin-dev wrote:
> Whatever their failings from their previous code or their adversarial
> nature, they got this code right and I'm only presenting it as a real
> and excellent solution for the impending threat to bitcoin. As a big
> core fan, I really wanted to delete the word Cash from my post because
> I was afraid someone would turn this technical discussion into a
> political football.
> 
> On Nov 2, 2017 7:37 PM, "Gregory Maxwell" <greg@xiph.org> wrote:
>         On Thu, Nov 2, 2017 at 11:31 PM, Scott Roberts via bitcoin-dev
>         <bitcoin-dev@lists.linuxfoundation.org> wrote:
>         > Bitcoin cash will hard fork on Nov 13 to implement a new
>         difficulty
>         > algorithm.  Bitcoin itself might need to hard fork to employ
>         a similar
>         > algorithm. It's about as good as they come because it
>         followed the
>         
>         
>         This is the bitcoin development mailing list, not the "give
>         free
>         review to the obviously defective proposals of adversarial
>         competing
>         systems" mailing list. Your posting is off-topic.
> 
> 
> 
> 
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[Scott Roberts]

The current DA is only sufficient if the coin has the highest
hashpower. It's also just really slow.  If miners somehow stick with
SegWit2x despite the higher rewards in defecting back to bitcoin, then
bitcoin will have long block delays. High transaction fees will
probably help them defect back to us. But if SegWit2x manages to be
more comparable in price than BCH (despite the futures), hashpower
could very well oscillate back and forth between the two coins,
causing delays in both of them. The first one to hard fork to fix the
difficulty problem will have a large advantage, as evidenced by what
happens in alts.   In any event someday BTC may not be the biggest kid
on the block and will need a difficulty algorithm that alts would find
acceptable. Few alts use anything like BTC's because they are not able
to survive the resulting long delays.   I am recommending BTC
developers watch what happens as BCH goes live with a much better
algorithm, in case BTC needs to hard fork for the same reason and
needs a similar fix. Ignore the trolls.

On Thu, Nov 2, 2017 at 7:39 PM, CryptAxe <cryptaxe@gmail.com> wrote:
> Is there an issue with the current difficulty adjustment algorithm? It's
> worked very well as far as I can tell. Introducing a new one seems pretty
> risky, what would the benefit be?
>
> On Nov 2, 2017 4:34 PM, "Scott Roberts via bitcoin-dev"
> <bitcoin-dev@lists.linuxfoundation.org> wrote:
>>
>> Bitcoin cash will hard fork on Nov 13 to implement a new difficulty
>> algorithm.  Bitcoin itself might need to hard fork to employ a similar
>> algorithm. It's about as good as they come because it followed the
>> "simplest is best" route. Their averaging window is probably
>> significantly too long (N=144). It's:
>>
>> next_D = sum (past 144 D's) * T / sum(past 144 solvetimes)
>>
>> They correctly did not use max(timestamp) - min(timestamp) in the
>> denominator like others do.
>>
>> They've written the code and they're about to use it live, so Bitcoin
>> will have a clear, simple, and tested path if it suddenly needs to
>> hard fork due to having 20x delays for the next 2000 blocks (taking it
>> a year to get unstuck).
>>
>> Details on it and the decision process:
>> https://www.bitcoinabc.org/november
>>
>> It uses a nice median of 3 for the beginning and end of the window to
>> help alleviate bad timestamp problems. It's nice, helps a little, but
>> will also slow its response by 1 block.  They also have 2x and 1/2
>> limits on the adjustment per block, which is a lot more than they will
>> ever need.
>>
>> I recommend bitcoin consider using it and making it N=50 instead of 144.
>>
>> I have seen that any attempts to modify the above with things like a
>> low pass filter, starting the window at MTP, or preventing negative
>> timestamps will only reduce its effectiveness. Bitcoin's +12 and -6
>> limits on the timestamps are sufficient and well chosen, although
>> something a bit smaller than the +12 might have been better.
>>
>> One of the contenders to the above is new and actually better, devised
>> by Degnr8 and they call it D622 or wt-144.It's a little better than
>> they realize. It's the only real improvement in difficulty algorithms
>> since the rolling average.  It gives a linearly higher weight to the
>> more recent timestamps. Otherwise it is the same. Others have probably
>> come across it, but there is too much noise in difficulty algorithms
>> to find the good ones.
>>
>> # Degnr8's D622 difficulty algorithm
>> # T=TargetTime, S=Solvetime
>> # modified by zawy
>> for i = 1 to N  (from oldest to most recent block)
>>     t += T[i] / D[i] * i
>>     j += i
>> next i
>> next_D = j / t * T
>>
>> I believe any modification to the above strict mathematical weighted
>> average will reduce it's effectiveness. It does not oscillate anymore
>> than regular algos and rises faster and drops faster, when needed.
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Re: [bitcoin-dev] Bitcoin Cash's new difficulty algorithm

[Jacob Eliosoff]

[-- Attachment #1: Type: text/plain, Size: 5289 bytes --]

I'm no BCH fan, but I agree with Scott that changes to the DAA may be of
more than purely theoretical interest for BTC.  Anyway just for those
interested, below is an algo I've been playing with that adjusts difficulty
every block, based only on the previous block's time and difficulty.  I
tested it a bit and it seems to adapt to hashrate swings pretty well.

weight_n = 1 - e^-(blocktime_n / 1 hr)    # 1 hr = exp moving avg window -
too short?
adj_n = (10 min / blocktime_n) - 1
difficulty_(n+1) = difficulty_n * (1 + weight_n * adj_n)

It could also be tweaked to make the *historical* avg block time ~exactly
10 minutes, ie, to target > 10 min if past blocks were < 10 min.  This
would, eg, make mapping future block numbers to calendar times much more
exact.


On Nov 3, 2017 7:24 AM, "Scott Roberts via bitcoin-dev" <
bitcoin-dev@lists.linuxfoundation.org> wrote:

> The current DA is only sufficient if the coin has the highest
> hashpower. It's also just really slow.  If miners somehow stick with
> SegWit2x despite the higher rewards in defecting back to bitcoin, then
> bitcoin will have long block delays. High transaction fees will
> probably help them defect back to us. But if SegWit2x manages to be
> more comparable in price than BCH (despite the futures), hashpower
> could very well oscillate back and forth between the two coins,
> causing delays in both of them. The first one to hard fork to fix the
> difficulty problem will have a large advantage, as evidenced by what
> happens in alts.   In any event someday BTC may not be the biggest kid
> on the block and will need a difficulty algorithm that alts would find
> acceptable. Few alts use anything like BTC's because they are not able
> to survive the resulting long delays.   I am recommending BTC
> developers watch what happens as BCH goes live with a much better
> algorithm, in case BTC needs to hard fork for the same reason and
> needs a similar fix. Ignore the trolls.
>
> On Thu, Nov 2, 2017 at 7:39 PM, CryptAxe <cryptaxe@gmail.com> wrote:
> > Is there an issue with the current difficulty adjustment algorithm? It's
> > worked very well as far as I can tell. Introducing a new one seems pretty
> > risky, what would the benefit be?
> >
> > On Nov 2, 2017 4:34 PM, "Scott Roberts via bitcoin-dev"
> > <bitcoin-dev@lists.linuxfoundation.org> wrote:
> >>
> >> Bitcoin cash will hard fork on Nov 13 to implement a new difficulty
> >> algorithm.  Bitcoin itself might need to hard fork to employ a similar
> >> algorithm. It's about as good as they come because it followed the
> >> "simplest is best" route. Their averaging window is probably
> >> significantly too long (N=144). It's:
> >>
> >> next_D = sum (past 144 D's) * T / sum(past 144 solvetimes)
> >>
> >> They correctly did not use max(timestamp) - min(timestamp) in the
> >> denominator like others do.
> >>
> >> They've written the code and they're about to use it live, so Bitcoin
> >> will have a clear, simple, and tested path if it suddenly needs to
> >> hard fork due to having 20x delays for the next 2000 blocks (taking it
> >> a year to get unstuck).
> >>
> >> Details on it and the decision process:
> >> https://www.bitcoinabc.org/november
> >>
> >> It uses a nice median of 3 for the beginning and end of the window to
> >> help alleviate bad timestamp problems. It's nice, helps a little, but
> >> will also slow its response by 1 block.  They also have 2x and 1/2
> >> limits on the adjustment per block, which is a lot more than they will
> >> ever need.
> >>
> >> I recommend bitcoin consider using it and making it N=50 instead of 144.
> >>
> >> I have seen that any attempts to modify the above with things like a
> >> low pass filter, starting the window at MTP, or preventing negative
> >> timestamps will only reduce its effectiveness. Bitcoin's +12 and -6
> >> limits on the timestamps are sufficient and well chosen, although
> >> something a bit smaller than the +12 might have been better.
> >>
> >> One of the contenders to the above is new and actually better, devised
> >> by Degnr8 and they call it D622 or wt-144.It's a little better than
> >> they realize. It's the only real improvement in difficulty algorithms
> >> since the rolling average.  It gives a linearly higher weight to the
> >> more recent timestamps. Otherwise it is the same. Others have probably
> >> come across it, but there is too much noise in difficulty algorithms
> >> to find the good ones.
> >>
> >> # Degnr8's D622 difficulty algorithm
> >> # T=TargetTime, S=Solvetime
> >> # modified by zawy
> >> for i = 1 to N  (from oldest to most recent block)
> >>     t += T[i] / D[i] * i
> >>     j += i
> >> next i
> >> next_D = j / t * T
> >>
> >> I believe any modification to the above strict mathematical weighted
> >> average will reduce it's effectiveness. It does not oscillate anymore
> >> than regular algos and rises faster and drops faster, when needed.
> >> _______________________________________________
> >> bitcoin-dev mailing list
> >> bitcoin-dev@lists.linuxfoundation.org
> >> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 7272 bytes --]