From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 8D84AC19 for ; Thu, 7 Dec 2017 21:01:48 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from APC01-HK2-obe.outbound.protection.outlook.com (mail-oln040092255103.outbound.protection.outlook.com [40.92.255.103]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8410679 for ; Thu, 7 Dec 2017 21:01:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=ryNkjGXITZt4Nv+yN47XEFVl9APoPt06w8M5Dqu4iCw=; b=GMfz4lnABApUYl37kPHdnLDdxoCxljzU1A80xy9TROvJdd7Zrpwh4IZW9vK8zXe8G7h6DKcKmYthUmM9HZfCjbUIZyx3Gs9SaDo65BPxJyXZ1x58jHG1p0EOTJgGMLTvDyVHPhT0KsCG8pXl7xcx+2YgfLm+WLSkzY1Hnmwy1RH0ctrS/H6ZWNyXVc5tj9pkEnnpJVhsCDz/YbqbvZLust7PkbQb1q8Zdnb4CsDqv6WJYZPQ71DoMdNKiGRG98P1rf8MYCx2djkr/bDu25N7AKvzyQIM5CSiurHB1/lvG7TzjbV+b2yczOej0l3NtSlK6WHOAwir84Fc5+TGCYWbHw== Received: from HK2APC01FT024.eop-APC01.prod.protection.outlook.com (10.152.248.51) by HK2APC01HT017.eop-APC01.prod.protection.outlook.com (10.152.249.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.282.5; Thu, 7 Dec 2017 21:01:44 +0000 Received: from PS2P216MB0179.KORP216.PROD.OUTLOOK.COM (10.152.248.51) by HK2APC01FT024.mail.protection.outlook.com (10.152.248.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.282.5 via Frontend Transport; Thu, 7 Dec 2017 21:01:44 +0000 Received: from PS2P216MB0179.KORP216.PROD.OUTLOOK.COM ([10.171.225.19]) by PS2P216MB0179.KORP216.PROD.OUTLOOK.COM ([10.171.225.19]) with mapi id 15.20.0282.007; Thu, 7 Dec 2017 21:01:43 +0000 From: Damian Williamson To: "bitcoin-dev@lists.linuxfoundation.org" Thread-Topic: BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks Thread-Index: AQHTb5z9LKaNJrfjBkOvllhZtrqBGg== Date: Thu, 7 Dec 2017 21:01:43 +0000 Message-ID: Accept-Language: en-AU, en-US Content-Language: en-AU X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:35EF7BF8405596006A12970223C3214A21F8E913D6121BA2042DCFC2AA735666; UpperCasedChecksum:8855B4A8ED2EB0AA8BAA1727764A378EF9ABD96A79AAE8C70922482FB0BA68BF; SizeAsReceived:6966; Count:44 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [C9UP2Hm1bBDnjgpCIuU9wivrDXO2Tmsr] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HK2APC01HT017; 6:4GxWogGjd860LvEcwjEaAjAGpT4z4qNUzRI8Pp+nUq64GytvUE7z4BXtuUOj8gSr5zur63md9FuLWwhdDqiOgJK7/YL196kQuPB0qn8/EKHOsvVlBjjwhxs8ibU4+vEfvHdBD6pMqqoNx9PT8KZrurzjXejwpUaIH30EevZH2+Tzr05UZQy5ME9Hj2xMIrv8wW79Ju21JeQtEHwbg1GSBtrngKwzcAE3tBE7spUO8OhSgzsgcmLVXwUX4oTbF6HWTQm0Rp4XbkuezVrUWjuyNDViFDuJnXp0dBPItmwlZILsrthxXXnGEm13SbDwwX1JaIjvdpI0ouV7iOSiRXrrXBHyGVmurGXgrMWbnbsXUhs=; 5:lwi+6azTfP7W/8sV9b5u81XoYumFjE7Pf7BqajgX50TgImJl57njXhUq7qVhTSEDuMMv/8vug3KCHD39agVUwgqfqDqfP9ZUVyOS+18pu1CUO2dSN/ObfYfksoppdQjjJ0dd/t0rcktj1x+1bkubCgN+8YeH7hHeRKrqS4ASIZ8=; 24:yhe01ZuqdMJMzTj7C/vtvQQ9kAQtueZDahcqhNhGpomlB5mVJzle493xIedmcgO4TP3JyxZXiKA8NYu2nGE4hhcnmnsiJ4LpjLTLL0BDzU8=; 7:oDlB0P3eFGikIC6vLOIDYyLkS8jTWvH/gmr5sPPG5J0H06kg2LiDFam2ZcEPExByhk95Y5b0plH8jjb04ZaVYb4c9XdX+dVqXt7jyGVQO0B/+lQuG9/Dwjjw0ukfxjeSIQU23i6mRUAqpGpLvyRYWTTwHWzf08cARE92/nZXj/HUPSuwGBy3sf6TSwAEgfkAE0OC0I6jQZC63XIaHbyDliEBMQxYHam1f9Rug0e9crCwUV/Ilyrbh3kCcL1oJBJL x-incomingheadercount: 44 x-eopattributedmessage: 0 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1601125374)(1603101448)(1701031045); SRVR:HK2APC01HT017; x-ms-traffictypediagnostic: HK2APC01HT017: x-ms-office365-filtering-correlation-id: b12b5b10-4926-4a86-b28c-08d53db5b8ae x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(444000031); SRVR:HK2APC01HT017; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:HK2APC01HT017; x-forefront-prvs: 05143A8241 x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(98901004); DIR:OUT; SFP:1901; SCL:1; SRVR:HK2APC01HT017; H:PS2P216MB0179.KORP216.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_PS2P216MB01794ABD544248B27BF0DFD99D330PS2P216MB0179KORP_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: b12b5b10-4926-4a86-b28c-08d53db5b8ae X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Dec 2017 21:01:43.8687 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: HK2APC01HT017 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, HTML_MESSAGE, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Thu, 07 Dec 2017 21:02:44 +0000 Subject: [bitcoin-dev] BIP Proposal: Revised: UTPFOTIB - Use Transaction Priority For Ordering Transactions In Blocks X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Dec 2017 21:01:48 -0000 --_000_PS2P216MB01794ABD544248B27BF0DFD99D330PS2P216MB0179KORP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Good afternoon, The need for this proposal: We all must learn to admit that transaction bandwidth is still lurking as a= serious issue for the operation, reliability, safety, consumer acceptance,= uptake and, for the value of Bitcoin. I recently sent a payment which was not urgent so; I chose three-day target= confirmation from the fee recommendation. That transaction has still not c= onfirmed after now more than six days - even waiting twice as long seems qu= ite reasonable to me. That transaction is a valid transaction; it is not ru= bbish, junk or, spam. Under the current model with transaction bandwidth li= mitation, the longer a transaction waits, the less likely it is ever to con= firm due to rising transaction numbers and being pushed back by transaction= s with rising fees. I argue that no transactions are rubbish or junk, only some zero fee transa= ctions might be spam. Having an ever-increasing number of valid transaction= s that do not confirm as more new transactions with higher fees are created= is the opposite of operating a robust, reliable transaction system. Business cannot operate with a model where transactions may or may not conf= irm. Even a business choosing a modest fee has no guarantee that their vali= d transaction will not be shuffled down by new transactions to the realm of= never confirming after it is created. Consumers also will not accept this = model as Bitcoin expands. If Bitcoin cannot be a reliable payment system fo= r confirmed transactions then consumers, by and large, will simply not acce= pt the model once they understand. Bitcoin will be a dirty payment system, = and this will kill the value of Bitcoin. Under the current system, a minority of transactions will eventually be the= lucky few who have fees high enough to escape being pushed down the list. Once there are more than x transactions (transaction bandwidth limit) every= ten minutes, only those choosing twenty-minute confirmation (2 blocks) wil= l have initially at most a fifty percent chance of ever having their paymen= t confirm. Presently, not even using fee recommendations can ensure a suffi= ciently high fee is paid to ensure transaction confirmation. I also argue that the current auction model for limited transaction bandwid= th is wrong, is not suitable for a reliable transaction system and, is wron= g for Bitcoin. All transactions must confirm in due time. Currently, Bitcoi= n is not a safe way to send payments. I do not believe that consumers and business are against paying fees, even = high fees. What is required is operational reliability. This great issue needs to be resolved for the safety and reliability of Bit= coin. The time to resolve issues in commerce is before they become great bi= g issues. The time to resolve this issue is now. We must have the foresight= to identify and resolve problems before they trip us over. Simply doublin= g block sizes every so often is reactionary and is not a reliable permanent= solution. I have written a BIP proposal for a technical solution but, need= your help to write it up to an acceptable standard to be a full BIP. I have formatted the following with markdown which is human readable so, I = hope nobody minds. I have done as much with this proposal as I feel that I = am able so far but continue to take your feedback. # BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactio= ns In Blocks ## The problem: Everybody wants value. Miners want to maximize revenue from fees (and we pr= esume, to minimize block size). Consumers need transaction reliability and,= (we presume) want low fees. The current transaction bandwidth limit is a limiting factor for both. As t= he operational safety of transactions is limited, so is consumer confidence= as they realize the issue and, accordingly, uptake is limited. Fees are ar= tificially inflated due to bandwidth limitations while failing to provide a= full confirmation service for all transactions. Current fee recommendations provide no satisfaction for transaction reliabi= lity and, as Bitcoin scales, this will worsen. Bitcoin must be a fully scalable and reliable service, providing full trans= action confirmation for every valid transaction. The possibility to send a transaction with a fee lower than one that is acc= eptable to allow eventual transaction confirmation should be removed from t= he protocol and also from the user interface. ## Solution summary: Provide each transaction with an individual transaction priority each time = before choosing transactions to include in the current block, the priority = being a function of the fee paid (on a curve), and the time waiting in the = transaction pool (also on a curve) out to n days (n=3D60 ?). The transactio= n priority to serve as the likelihood of a transaction being included in th= e current block, and for determining the order in which transactions are tr= ied to see if they will be included. Use a target block size. Determine the target block size using; current tra= nsaction pool size x ( 1 / (144 x n days ) ) =3D number of transactions to = be included in the current block. Broadcast the next target block size with= the current block when it is solved so that nodes know the next target blo= ck size for the block that they are building on. The curves used for the priority of transactions would have to be appropria= te. Perhaps a mathematician with experience in probability can develop the = right formulae. My thinking is a steep curve. I suppose that the probabilit= y of all transactions should probably account for a sufficient number of in= clusions that the target block size is met although, it may not always be. = As a suggestion, consider including some zero fee transactions to pad, high= est BTC value first? **Explanation of the operation of priority:** > If transaction priority is, for example, a number between one (low) and o= ne-hundred (high) it can be directly understood as the percentage chance in= one-hundred of a transaction being included in the block. Using probabilit= y or likelihood infers that there is some function of random. If random (10= 0) < transaction priority then the transaction is included. >To break it down further, if both the fee on a curve value and the time wa= iting on a curve value are each a number between one and one-hundred, a rud= imentary method may be to simply multiply those two numbers, to find the pr= iority number. For example, a middle fee transaction waiting thirty days (i= f n =3D 60 days) may have a value of five for each part (yes, just five, t= he values are on a curve). When multiplied that will give a priority value = of twenty-five, or, a twenty-five percent chance at that moment of being i= ncluded in the block; it will likely be included in one of the next four bl= ocks, getting more likely each chance. If it is still not included then the= value of time waiting will be higher, making for more probability. A very = low fee transaction would have a value for the fee of one. It would not be = until near sixty-days that the particular low fee transaction has a high li= kelihood of being included in the block. I am not concerned with low (or high) transaction fees, the primary reason = for addressing the issue is to ensure transactional reliability and scalabi= lity while having each transaction confirm in due time. ## Pros: * Maximizes transaction reliability. * Fully scalable. * Maximizes possibility for consumer and business uptake. * Maximizes total fees paid per block without reducing reliability; because= of reliability, in time confidence and overall uptake are greater; therefo= re, more transactions. * Market determines fee paid for transaction priority. * Fee recommendations work all the way out to 30 days or greater. * Provides additional block entropy; greater security since there is less p= robability of predicting the next block. ## Cons: * Could initially lower total transaction fees per block. * Must be first be programmed. ## Solution operation: This is a simplistic view of the operation. The actual operation will need = to be determined in a spec for the programmer. 1. Determine the target block size for the current block. 2. Assign a transaction priority to each transaction in the pool. 3. Select transactions to include in the current block using probability in= transaction priority order until the target block size is met. 5. Solve block. 6. Broadcast the next target block size with the current block when it is s= olved. 7. Block is received. 8. Block verification process. 9. Accept/reject block based on verification result. 10. Repeat. ## Closing comments: It may be possible to verify blocks conform to the proposal by showing that= the probability for all transactions included in the block statistically c= onforms to a probability distribution curve, *if* the individual transactio= n priority can be recreated. I am not that deep into the mathematics; howev= er, it may also be possible to use a similar method to do this just based o= n the fee, that statistically, the blocks conform to a fee distribution. An= y zero fee transactions would have to be ignored. This solution needs a cle= ver mathematician. I implore, at the very least, that we use some method that validates full t= ransaction reliability and enables scalability of block sizes. If not this = proposal, an alternative. Regards, Damian Williamson --_000_PS2P216MB01794ABD544248B27BF0DFD99D330PS2P216MB0179KORP_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Good afternoon,

The need for this proposal:

We all must learn to admit that transaction bandwidth is still lurking as a= serious issue for the operation, reliability, safety, consumer acceptance,= uptake and, for the value of Bitcoin.

I recently sent a payment which was not urgent so; I chose three-day target= confirmation from the fee recommendation. That transaction has still not c= onfirmed after now more than six days - even waiting twice as long seems qu= ite reasonable to me. That transaction is a valid transaction; it is not rubbish, junk or, spam. Under the curren= t model with transaction bandwidth limitation, the longer a transaction wai= ts, the less likely it is ever to confirm due to rising transaction numbers= and being pushed back by transactions with rising fees.

I argue that no transactions are rubbish or junk, only some zero fee transa= ctions might be spam. Having an ever-increasing number of valid transaction= s that do not confirm as more new transactions with higher fees are created= is the opposite of operating a robust, reliable transaction system.

Business cannot operate with a model where transactions may or may not conf= irm. Even a business choosing a modest fee has no guarantee that their vali= d transaction will not be shuffled down by new transactions to the realm of= never confirming after it is created. Consumers also will not accept this model as Bitcoin expands. If Bitcoin c= annot be a reliable payment system for confirmed transactions then consumer= s, by and large, will simply not accept the model once they understand. Bit= coin will be a dirty payment system, and this will kill the value of Bitcoin.

Under the current system, a minority of transactions will eventually be the= lucky few who have fees high enough to escape being pushed down the list.<= br>
Once there are more than x transactions (transaction bandwidth limit) every= ten minutes, only those choosing twenty-minute confirmation (2 blocks) wil= l have initially at most a fifty percent chance of ever having their paymen= t confirm. Presently, not even using fee recommendations can ensure a sufficiently high fee is paid to ensure t= ransaction confirmation.

I also argue that the current auction model for limited transaction bandwid= th is wrong, is not suitable for a reliable transaction system and, is wron= g for Bitcoin. All transactions must confirm in due time. Currently, Bitcoi= n is not a safe way to send payments.

I do not believe that consumers and business are against paying fees, even = high fees. What is required is operational reliability.

This great issue needs to be resolved for the safety and reliability of Bit= coin. The time to resolve issues in commerce is before they become great bi= g issues. The time to resolve this issue is now. We must have the foresight= to identify and resolve problems before they trip us over.  Simply doubling block sizes every so often= is reactionary and is not a reliable permanent solution. I have written a = BIP proposal for a technical solution but, need your help to write it up to= an acceptable standard to be a full BIP.

I have formatted the following with markdown which is human readable so, I = hope nobody minds. I have done as much with this proposal as I feel that I = am able so far but continue to take your feedback.

# BIP Proposal: UTPFOTIB - Use Transaction Priority For Ordering Transactio= ns In Blocks

## The problem:
Everybody wants value. Miners want to maximize revenue from fees (and we pr= esume, to minimize block size). Consumers need transaction reliability and,= (we presume) want low fees.

The current transaction bandwidth limit is a limiting factor for both. As t= he operational safety of transactions is limited, so is consumer confidence= as they realize the issue and, accordingly, uptake is limited. Fees are ar= tificially inflated due to bandwidth limitations while failing to provide a full confirmation service for all t= ransactions.

Current fee recommendations provide no satisfaction for transaction reliabi= lity and, as Bitcoin scales, this will worsen.

Bitcoin must be a fully scalable and reliable service, providing full trans= action confirmation for every valid transaction.

The possibility to send a transaction with a fee lower than one that is acc= eptable to allow eventual transaction confirmation should be removed from t= he protocol and also from the user interface.

## Solution summary:
Provide each transaction with an individual transaction priority each time = before choosing transactions to include in the current block, the priority = being a function of the fee paid (on a curve), and the time waiting in the = transaction pool (also on a curve) out to n days (n=3D60 ?). The transaction priority to serve as the likelih= ood of a transaction being included in the current block, and for determini= ng the order in which transactions are tried to see if they will be include= d.

Use a target block size. Determine the target block size using; current tra= nsaction pool size x ( 1 / (144 x n days ) ) =3D number of transactions to = be included in the current block. Broadcast the next target block size with= the current block when it is solved so that nodes know the next target block size for the block that they are = building on.

The curves used for the priority of transactions would have to be appropria= te. Perhaps a mathematician with experience in probability can develop the = right formulae. My thinking is a steep curve. I suppose that the probabilit= y of all transactions should probably account for a sufficient number of inclusions that the target block size i= s met although, it may not always be. As a suggestion, consider including s= ome zero fee transactions to pad, highest BTC value first?

**Explanation of the operation of priority:**
> If transaction priority is, for example, a number between one (low) an= d one-hundred (high) it can be directly understood as the percentage chance= in one-hundred of a transaction being included in the block. Using probabi= lity or likelihood infers that there is some function of random. If random (100) < transaction priority then= the transaction is included.

>To break it down further, if both the fee on a curve value and the time= waiting on a curve value are each a number between one and one-hundred, a = rudimentary method may be to simply multiply those two numbers, to find the= priority number. For example, a middle fee transaction waiting thirty days (if n =3D 60 days) may have a value of= five for each part  (yes, just five, the values are on a curve). When= multiplied that will give a priority value of twenty-five, or,  a twe= nty-five percent chance at that moment of being included in the block; it will likely be included in one of the next four = blocks, getting more likely each chance. If it is still not included then t= he value of time waiting will be higher, making for more probability. A ver= y low fee transaction would have a value for the fee of one. It would not be until near sixty-days that the= particular low fee transaction has a high likelihood of being included in = the block.

I am not concerned with low (or high) transaction fees, the primary reason = for addressing the issue is to ensure transactional reliability and scalabi= lity while having each transaction confirm in due time.

## Pros:
* Maximizes transaction reliability.
* Fully scalable.
* Maximizes possibility for consumer and business uptake.
* Maximizes total fees paid per block without reducing reliability; because= of reliability, in time confidence and overall uptake are greater; therefo= re, more transactions.
* Market determines fee paid for transaction priority.
* Fee recommendations work all the way out to 30 days or greater.
* Provides additional block entropy; greater security since there is less p= robability of predicting the next block.

## Cons:
* Could initially lower total transaction fees per block.
* Must be first be programmed.

## Solution operation:
This is a simplistic view of the operation. The actual operation will need = to be determined in a spec for the programmer.

1. Determine the target block size for the current block.
2. Assign a transaction priority to each transaction in the pool.
3. Select transactions to include in the current block using probability in= transaction priority order until the target block size is met.
5. Solve block.
6. Broadcast the next target block size with the current block when it is s= olved.
7. Block is received.
8. Block verification process.
9. Accept/reject block based on verification result.
10. Repeat.

## Closing comments:
It may be possible to verify blocks conform to the proposal by showing that= the probability for all transactions included in the block statistically c= onforms to a probability distribution curve, *if* the individual transactio= n priority can be recreated. I am not that deep into the mathematics; however, it may also be possible to us= e a similar method to do this just based on the fee, that statistically, th= e blocks conform to a fee distribution. Any zero fee transactions would hav= e to be ignored. This solution needs a clever mathematician.

I implore, at the very least, that we use some method that validates full t= ransaction reliability and enables scalability of block sizes. If not this = proposal, an alternative.

Regards,
Damian Williamson

--_000_PS2P216MB01794ABD544248B27BF0DFD99D330PS2P216MB0179KORP_--