From mboxrd@z Thu Jan 1 00:00:00 1970 Delivery-date: Tue, 23 Jul 2024 09:19:39 -0700 Received: from mail-oo1-f56.google.com ([209.85.161.56]) by mail.fairlystable.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1sWIEk-0007mO-VW for bitcoindev@gnusha.org; Tue, 23 Jul 2024 09:19:39 -0700 Received: by mail-oo1-f56.google.com with SMTP id 006d021491bc7-5ccb04330ccsf4472556eaf.3 for ; Tue, 23 Jul 2024 09:19:38 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721751572; cv=pass; d=google.com; s=arc-20160816; b=IMhUiizp0Ycyrj+EkpzXMg/FSWYM0LXkpcnr+bk5qF7J7w/weRcod7qaNiMwT19msg AgLbXzPpMemE49nL1TS9h/bVYKhlpDAfqXZE0j9G/6bIBUi80TOhDFdOPdZoZ/WhfxAx x5cicgGHpV4YYbxPHeZ3CU7NXEuDRQ0XkZ4WrcZ5UgKViVmy31/+pdCZU6xYhZVDzkk9 LrPYQ+j/nTeTz3yRxoegl2JGxEJgJ3LlKajk61T3Mtfy00UOjOmFkj2DD8Uabvz5OLCy APIjToFyh2+vxZQdBfLhg4ruk93qq8ntFunwhUefOjOyO1TVO1iR/EPG3DkeGUVCc6FN 6zHQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:content-transfer-encoding :content-disposition:mime-version:message-id:subject:to:from:date :sender:dkim-signature; bh=JvkpqKYmLYDeWo7dP1niAa0SBJUBAGwAgtXadfwB29Q=; fh=uzBleHtgOhsRsUsYhJ3afpucRXBT6V27Jl6BurcxBFY=; b=hnKO5LE6e6ySQpWUcuuQvrR/r+tkZuTyIDVqmRFpctpv3fxw25Q8T1xuIECdQRgkB3 QCvidu0uFp71iAaXoAwIUsCy1GOAnhsHrSUNsO159PEPvt1R7h3OUH0yEcoN47S0s1yb bV1kne40Jso+EcACX79/aF1/eki0lBpfuMa4yAG0bZqvUp2UPZhHpJUleCmh4WnzjMFT ebkBgOKNSMYqNfnvA7aFIt8mhs4ITGdxCltfKqx028qFTNg1EEkBzImRvJXHiAm69IM1 2AZ/FKZdT8ZgxApxZY6bZWxrqVb65KUeDJyT2Pum1gxgZmXIel9LctYKCyHLscY4cYEt WOEw==; darn=gnusha.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of aj@erisian.com.au designates 172.104.61.193 as permitted sender) smtp.mailfrom=aj@erisian.com.au DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721751572; x=1722356372; darn=gnusha.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:content-transfer-encoding:content-disposition :mime-version:message-id:subject:to:from:date:sender:from:to:cc :subject:date:message-id:reply-to; bh=JvkpqKYmLYDeWo7dP1niAa0SBJUBAGwAgtXadfwB29Q=; b=sxE2g0JLkPHnD/dmrujPDrLkmtDYv4wLZOV5Ug+YYpN8YkhkV4jHhUrq0Xv8qHeiEL u9ja4EzZCOi61AMj4UYeGGNkbR58pvV5AcyrFl3O7ypMYOVp4h6wOWYUuBKkCCmeOERo vKZyGm1pjQ8/goUofTR5sLftMXuiRvNu91l41Z7Rpg0Xy2aLlDB6l9yGBqE4sARtU3Oe h22n0VmeXta+A9rwFBmQQXJ2HEGk1ioSKQ9cN0z/+/o2tGaUZzsj8CBj7GyV7VClVJ8i VE2JueNAKEcDsYp/bUzgaRpPZGURlBNNjz6+XvwMitBJHMu4VjeYsUAG3du+QYAG79mG wEiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721751572; x=1722356372; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:content-transfer-encoding:content-disposition :mime-version:message-id:subject:to:from:date:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=JvkpqKYmLYDeWo7dP1niAa0SBJUBAGwAgtXadfwB29Q=; b=ikwxRtY2eO1+5NkPUJ39eN1ZiO6Lh6lVlM5uP8Nwp5omT2vm+wYAabCuFJaz55X3f4 +vHrljCZmuiSxhAqMvHMEbqk7OBX4BDN0zzH8ubLNw/y21LjbHyVrBtXJ9Ja1z9SJvfo QKTSbhBVZQUyUOetUf5u4m/8zY1Zc8cbtAB74tkMu07+6J93t7D+BVfMNsuGCitDmNAL jWovEItVDa3rJPwdNqzD+6NvRx0s13vjMZmTnfcOypUnIpgdRR6O9qtIULVHJoriK2Q8 JVSNApwE3GjIve0wivnqHuu8eTvJGlbTksgiItYEnsb3LaEVxtEKAzW6gxMgaF5ipNKy nCSQ== Sender: bitcoindev@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUVChsDbXIhSj5T/fTVhTbUgiciqq4lptV7A0taQJXhakeTnyLEJp1OiBY4hWMwkSouXWX8dSusnJITut+DRtgOHJNW7MY= X-Gm-Message-State: AOJu0YyMpp+AMx+HkML67wiF1qQSOSR3c/Q+K4q0MXPiVLA/6oQ0Q+Fh BmOp8Dm/kJyO/uyCPgisR5+YcCRK0J8R9wFfjYC50xDLh3FBOs3K X-Google-Smtp-Source: AGHT+IHoDzY/wSF8USm9naHKdLk+ytpPjobycmlHUFFN4sOfIE9/P4gMOk8bdAnenHDUasDaK53HCg== X-Received: by 2002:a05:6820:1b15:b0:5c6:9320:53a3 with SMTP id 006d021491bc7-5d564fb672fmr14903515eaf.4.1721751572176; Tue, 23 Jul 2024 09:19:32 -0700 (PDT) X-BeenThere: bitcoindev@googlegroups.com Received: by 2002:a4a:106:0:b0:5c4:40e9:9160 with SMTP id 006d021491bc7-5d51a246fc5ls5434440eaf.0.-pod-prod-01-us; Tue, 23 Jul 2024 09:19:30 -0700 (PDT) X-Received: by 2002:a05:6820:1c87:b0:5cd:13ff:4fa6 with SMTP id 006d021491bc7-5d564eec276mr406400eaf.2.1721751570421; Tue, 23 Jul 2024 09:19:30 -0700 (PDT) Received: by 2002:a05:6808:984:b0:3d9:302f:bb85 with SMTP id 5614622812f47-3dadf171cc4msb6e; Tue, 23 Jul 2024 08:02:35 -0700 (PDT) X-Received: by 2002:a05:6870:1715:b0:254:d163:c3aa with SMTP id 586e51a60fabf-2638df302acmr9141858fac.16.1721746954815; Tue, 23 Jul 2024 08:02:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721746954; cv=none; d=google.com; s=arc-20160816; b=hEnLZ6mM3uEg9ObwzoFj5FHVc7QDuTaC4J7AfmHWd6sPys1UmaoPd1ut04v9AkZ/nK yyjsyeQXez8/bJVsnx8m3hwrwlMffF6lChNkT5QO91c3f1+eFEUk6XTkMi4Pmg8Ejshz LGWITd7kGZuqDVd7teKUVJZK44eDrJ7Xzp2SMHax02VX27hkghLMCCmBI8taCB3HJYZq xE0BHfy4wcn2KaGS8qjby/PdMh4ZWryvCX+DRHAuaK18klNCSnadUoj2KGWvU2ONZqgX +K9Se9NbTFV3rUgmSFumit2er38xUtjLg1uHIS0FVhiRStJ8S09e3QfpJzr7MVEAZdnR jfOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-disposition:mime-version :message-id:subject:to:from:date; bh=xaaPNedDkXs3mk58+HfoUJH9ql7lZIPGOJeuygSyFFY=; fh=VcGcg+Zjs9gw1uDcHbxsAILhBAcecnbJzZRdxgKVDIc=; b=u23u/WIhBFLiruW+nlJFDMycR0zf/AD5ux8cSQ5YltIA477LMyVJiRmTOHxHyYDzUR jlPb5WLQh5VakpqwVwDG6+t6tL+AviY5FVeXqOj5Wiwca2HPzhtLD7o+db/2gf5lKw4P +46oNR5UT3ChEHGz1RcNBaQVX0Gp8WyTnR89YbXcp89EToBfJeS+E/aZEMf9XtWhkOO+ dK0enpAKvHvOfnM9JW54dmEeeZddFPTKThX2m8ulZDqdj7mwnIdoQwBc2LUrtr7bzyE6 gkbnYq4YLOeon3FSSJnqDvFJBobw0oI/5RpvwEyFfMN1BiPhk+Wz5jsSnoP93EOMJ+GK rshQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of aj@erisian.com.au designates 172.104.61.193 as permitted sender) smtp.mailfrom=aj@erisian.com.au Received: from cerulean.erisian.com.au (azure.erisian.com.au. [172.104.61.193]) by gmr-mx.google.com with ESMTPS id 586e51a60fabf-2610cad2ef7si398776fac.3.2024.07.23.08.02.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jul 2024 08:02:34 -0700 (PDT) Received-SPF: pass (google.com: domain of aj@erisian.com.au designates 172.104.61.193 as permitted sender) client-ip=172.104.61.193; Received: from aj@azure.erisian.com.au by cerulean.erisian.com.au with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sWH25-0004bN-Fq for bitcoindev@googlegroups.com; Wed, 24 Jul 2024 01:02:32 +1000 Received: by email (sSMTP sendmail emulation); Wed, 24 Jul 2024 01:02:24 +1000 Date: Wed, 24 Jul 2024 01:02:24 +1000 From: Anthony Towns To: bitcoindev@googlegroups.com Subject: [bitcoindev] Mining pools, stratumv2 and oblivious shares Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable X-Spam_score: 0.0 X-Spam_bar: / X-Original-Sender: aj@erisian.com.au X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of aj@erisian.com.au designates 172.104.61.193 as permitted sender) smtp.mailfrom=aj@erisian.com.au Precedence: list Mailing-list: list bitcoindev@googlegroups.com; contact bitcoindev+owners@googlegroups.com List-ID: X-Google-Group-Id: 786775582512 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-Spam-Score: -0.8 (/) Hi *, 1. stratumv2 templates via client-push =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The stratumv2 design document suggests: > Use a separate communication channel for transaction selection so > that it does not have a performance impact on the main mining/share > communication, as well as can be run in three modes - disabled (i.e.pool > does not yet support client work selection, to provide an easier > transition from Stratum v1), client-push (to maximally utilize the > client=E2=80=99s potential block-receive-latency differences from the poo= l), > and client-negotiated (for pools worried about the potential of clients > generating invalid block templates). -- https://stratumprotocol.org/specification/02-Design-Goals/ To me, the "client-push" approach (vs the client-negotiated approach) seems somewhat unworkable (at least outside of solo mining). In particular, if you're running a pool and have many clients generating their own templates, and you aren't doing KYC on those clients, and aren't fully validating every proposed share, then it becomes very easy to do a "block withholding attack" [0] -- just locally create invalid blocks, submit shares as normal, receive payouts as normal for partial shares because the shares aren't being validated, and if you happen to find an actual block, the pool loses out because none of your blocks were actually valid. (You can trivially make your block invalid by simply increasing the pool payout by 1 sat above the correct value) Validating arbitrary attacker submitted templates seems likely to be expensive, as they can produce transactions which aren't already in your mempool, are relatively expensive to validate, and potentially conflict with transactions that other clients are selecting, causing you to have to churn txs in and out of your mempool. Particularly if an attacker could have an array of low hashpower workers all submitting different templates to a server, it seems like it would be relatively easy to overload any small array of template-validater nodes, given a pure client-push model. In which case client-push would only really make sense for pure solo-mining, where you're running your own stratumv2 server and your own miners and have full control (and thus trust) from end to end. Does that make sense, or am I missing something? I think a negotiated approach could look close to what Ocean's template selection looks like today: that is the pool operator runs some nodes with various relay policies that generate blocks, and perhaps also allows for externally submitted templates that it validates. Then clients request work according to their preferences, perhaps they specify that they prefer the "no-ordinals template", or perhaps "whatever template has the highest payout (after pool fees)". The pool tracks the various templates it's offered in order to validate shares and to broadcast valid blocks once one is found. A negotiated approach seems also more amenable to including out of band payments, such as mempool.space's accelerator, whether those payments are distributed to miners or kept by the pool. This could perhaps also be closer to the ethereum model of proposer/builder separation [7], which may provide a modest boost to MEV/MEVil resistance -- that is if there is MEV available via some clever template construction, specialist template builders can construct specialised templates paying slightly higher fees and submit those to mining pools, perhaps splitting any excess profits between the pool and template constructor. Providing a way for external parties to submit high fee templates to pools (rate-limited by a KYC-free bond payment over LN perhaps), seems like it would help limit the chances of that knowledge being kept as a trade secret to one pool or mining farm, which could then use its excess profits to become dominant, and then gain too much ability to censor txs. Having pools publish the full templates for auditing purposes would allow others to easily incrementally improve on clever templates by adding any high-fee censored transactions back in. 2. block withholding and oblivious shares =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Anyway, as suggested by the subject-line and the reference to [0], I'm still a bit concerned by block withholding attacks -- where an attacker finds decentralised pools and throws hashrate at them to make them unprofitable by only sending the 99.9% of shares that aren't valid blocks, while withholding/discarding the 0.1% of shares that would be a valid block. The result being that decentralised non-KYC pools are less profitable and are replaced by centralised KYC pools that can ban attackers, and we end up with most hashrate being in KYC pools, where it's easier for the pools to censor txs, or miners to be found and threatened or have their hardware confiscated. (See also [6]) If it were reasonable to mine blocks purely via the tx merkle root, with only the pool knowing the coinbase or transaction set at the time of mining, I think it could be plausible to change the PoW algorithm to enable oblivious shares: where miners can't tell if a given valid share corresponds to a valid block or not, but pools and nodes can still easily easily validate work form just the block header. In particular I think an approach like this could work: * take the top n-bits of the prev hash in the header, which are currently required to be all zeroes due to `powLimit` (n=3D0 for regtes= t, n<=3D8 in general) * stop requiring these bits to be zero, instead interpret them as `nBitsShareShift`, from 0 to up to 255 * when nBitsShareShift > 0, check that the share hash is less than or equal to (2**256 - 1) >> nBitsShareShift, where the share hash is calculated as sha256d( nVersion, hashPrevBlock, sha256d( hashMerkleRoot ), nTime, nBits, nNonce ) * check that the normal block hash is not greater than FromCompact(nBits) << nBitsShareShift Note that this would be a light-client visible hard fork -- any blocks that set nBitsShareShift to a non-zero value would be seen as invalid by existing software that checks header chains. It's also possible to take a light-client invisible approach by decreasing the value of nBits, but providing an `nBitsBump` that new nodes validate but existing nodes do not (see [1] eg). This has the drawback that it would become easy to fool old nodes/light clients into following the wrong chain, as they would not be fully validating proof-of-work, which already breaks the usual expectations of a soft fork (see [2] eg). Because of that, a hard fork approach seems safer here to me. YMMV, obviously. The above approach requires two additional sha256d operations when nodes or light clients are validating header work, but no additional data, which seems reasonable, and should require no changes to machines capable of header-only mining -- you just use sha256d(hashMerkleRoot) instead of hashMerkleRoot directly. In this scenario, pools are giving their client sha256d(hashMerkleRoot) rather than hashMerkleRoot or a transaction tree directly, and `nBitsShareShift` is set based on the share difficulty. Pools then check the share is valid, and additionally check whether the share has sufficient work to be a valid block, which they are able to do because unlike the miner, they can calculate the normal block hash. The above assumes that the pool has full control over the coinbase and transaction selection, and that the miner/client is not able to reconstruct all that data from its mining job, so this would be another reason why a pool would only support a client-negotiated approach for templates, not a client-push approach. Note that miners/clients could still *audit* the work they've been given if the pool makes the full transaction set (including coinbase) for a template available after each template expires. Some simple numbers: if a miner with control of 10% hashrate decided to attack a decentralised non-KYC pool with 30% hashrate, then they could apply 3.3% hashrate towards a blockwithholding attack, reducing the victim's income to 90% (30% hashrate finding blocks vs 33% hashrate getting payouts) while only reducing their own income to 96.7% (6.7% hashrate at 100% payout, 3.3% at 90%). If they decided to attack a miner with 50% hashrate, they would need to provide 5.55% of total hashrate to reduce the victim's income to 90%, which would reduce their own income to 94.45% (4.45% at 100%, 5.55% at 90%). I've seen suggestions that block withholding could be used as a way to attack pools that gain >50% hashpower, but as far as I can see it's only effective against decentralised, non-KYC pools, and more effective against small pools than large ones, which seems more or less exactly backwards to what we'd actually want... Some previous discussion of block withholding and KYC is at [3] [4] [5]. 3. extra header nonce space =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Given BIP320, you get 2**48 values to attempt per second of nTime, which is about 281 TH/s, or enough workspace to satisfy a single S21 XP at 270 TH/s. Maybe that's okay, but maybe it's not very much. Since the above would already be a (light-client visible) hard fork, it could make sense to also provide extra nonce space that doesn't require touching the coinbase transaction (since we're relying on miners not having access to the contents of the tx merkle tree). One approach would be to use the leading zeroes in hashPrevBlock as extra nonce space (eg, [8]). That's particularly appealing since it scales exactly with difficulty -- the higher the difficulty, the more nonce space you need, but the more required zeroes you have. However the idea above unfortuantely reduces the available number of zero bits in the previous block hash by that block's choice of nBitsShareShift, which may take a relatively large value in order to reduce traffic with the pool. So sadly I don't think that approach would really work. Another way to do that could work might be to add perhaps 20 bytes of extra nonce to the header, and calculate the block hashes as: normal hash -> sha256d( nVersion, hashPrevBlock, sha256d( merkleRoot, TagHash_BIPxxx(extraNonce) ), nTime, nBits, nNonce ) and share hash -> sha256d( nVersion, hashPrevBlock, sha256d( sha256d(merkleRoot), TagHash_BIPxxx(extraNonce) ), nTime, nBits, nNonce ) That should still be compatible with existing mining hardware. Though it would mean nodes are calculating 5x sha256d and 1x taghash to validate a block header's PoW, rather than just 1x sha256d (now) or 3x sha256d (above). This approach would also not require changes in how light clients verify merkle proofs of transaction inclusion in blocks, I believe. (Using a bip340-esque TagHash for the extraNonce instead of nothing or sha256d hopefully prevents hiding fake transactions in that portion) 4. plausibility =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D It's not clear to me how serious a problem block withholding is. It seems like it would explain why we have few pools, why they're all pretty centralised, and why major ones care about KYC, but there are plenty of other explanations for both those things. So even if this was an easy fix, it's not clear to me how much sense it makes to think about. And beyond that's it's a consensus change (ouch), a hard fork (ouch, ouch) and one that requires every light client to upgrade (ouch, ouch, ouch!). However, all of that is still just code, and none of those things are impossible to achieve, if they're worth the effort. I would expect a multiyear deployment timeline even once the code was written and it was widely accepted as a good idea, though. If this is a serious problem for the privacy and decentralisation of mining, and a potential threat to bitcoin's censorship resistance, it seems to me like it would be worth the effort. 5. conclusions? =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Anyway, I wanted to write my thoughts on this down somewhere they could be critiqued. Particularly the idea that everyone building their own blocks for public pools running stratumv2 doesn't actually make that much sense, as far as I can see. I think the share approach in section 2 and the extranonce approach in section 3 are slightly better than previous proposed approaches I've seen, so are worth having written down somewhere. Cheers, aj [0] https://bitcoil.co.il/pool_analysis.pdf [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/0= 12051.html [2] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-February/0= 12443.html [3] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/0= 12060.html [4] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/0= 12111.html [5] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/0= 12069.html [6] https://bitcoinops.org/en/topics/pooled-mining/#block-withholding-attac= ks [7] https://ethereum.org/en/roadmap/pbs/ [8] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-December/0= 15386.html --=20 You received this message because you are subscribed to the Google Groups "= Bitcoin Development Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to bitcoindev+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/= bitcoindev/Zp/GADXa8J146Qqn%40erisian.com.au.