From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 80339BAE for ; Fri, 7 Apr 2017 23:50:36 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pg0-f41.google.com (mail-pg0-f41.google.com [74.125.83.41]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 1AB06139 for ; Fri, 7 Apr 2017 23:50:36 +0000 (UTC) Received: by mail-pg0-f41.google.com with SMTP id x125so79427662pgb.0 for ; Fri, 07 Apr 2017 16:50:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=voskuil-org.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=CKuCErc5xcq5/wh2ePvs98RJWMzZS6dooXIEP6C5BqU=; b=BqKyMt1UQZf4u3Wnv0EYRdDXxB3a9c+7qw1Wv60BtaIw2iXIjcQBoo5oX8ZJ5aPKIN c4JZHW337mLHOpsJypyHQQlsf32RuKc1JQgSd3+aDFcowUSdeF8WyxAQi27CeWaJIy7o QPI2X2z3GjTgYqJbgHAtyBBndz217MouCRgqfGKtOUlssUyoKG2Fn7vXtwkxtWHfj/iV PYLZ23tL4/Y3+8t39YBm6HbJR4dZZwY2Og8p2vf1/ovnChZ9ILcTC9A5YKW8zlaDMn8k MljQwdmF8wc0xOcjMHoAqpWmMXeol7RBfJYJXSABHGJwXH5RIe+wO117YW5U8uJkj1GR lxlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=CKuCErc5xcq5/wh2ePvs98RJWMzZS6dooXIEP6C5BqU=; b=NWB7VvR4/8RVnK45FBjnPQ9DhpoRawCwhHCmC7fDSsWCiagj+RNXuYEMbI4r4cFIS7 NsKkJFFaw7uLdwOZhlAc2mgLxNsCwhAdmRB8xiFPZMCgIEKm9dTgqsn5W6E3CdnFf683 7Tjv4fWQIKgcIlhTKyFjfnyuzp3IzNDPvAIHm7AqeHuOrM4JyuJZAJl5wTc91t+26G5V z5vgXDry02gqaD3KMRKBTeVCTg1nt0EfubJjwFmWxpuDGOddpyGCS6rgIy9rIguzeLrZ wm2AFUDX1LjxADNWhMUtX+QjVC0qzReIcmP6QXJjST9sxSOBVpPFDn8yX2i9cbG7rotf VEtA== X-Gm-Message-State: AFeK/H0YCjLUrcehwY6ff0tJH/RWcpHjvgl2U8pYk3fpE584LqTynecMAARPe0RG0JYsMg== X-Received: by 10.84.224.12 with SMTP id r12mr13602568plj.69.1491609035670; Fri, 07 Apr 2017 16:50:35 -0700 (PDT) Received: from ?IPv6:2601:600:9000:d69e:7543:944:9329:6f9? ([2601:600:9000:d69e:7543:944:9329:6f9]) by smtp.gmail.com with ESMTPSA id b8sm11545798pgn.51.2017.04.07.16.50.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 07 Apr 2017 16:50:34 -0700 (PDT) To: Tomas , Bitcoin Protocol Discussion References: <1491516747.3791700.936828232.69F82904@webmail.messagingengine.com> <1491601483.1253274.937963928.1B1D9B41@webmail.messagingengine.com> From: Eric Voskuil X-Enigmail-Draft-Status: N1110 Message-ID: <03ab63f2-d4b0-33d4-db73-1cf5a94592ba@voskuil.org> Date: Fri, 7 Apr 2017 16:51:08 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <1491601483.1253274.937963928.1B1D9B41@webmail.messagingengine.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_NONE, RCVD_IN_SORBS_SPAM autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org X-Mailman-Approved-At: Sat, 08 Apr 2017 00:39:08 +0000 Subject: Re: [bitcoin-dev] Using a storage engine without UTXO-index X-BeenThere: bitcoin-dev@lists.linuxfoundation.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Bitcoin Protocol Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Apr 2017 23:50:36 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 04/07/2017 02:44 PM, Tomas via bitcoin-dev wrote: > Hi Eric, > > On Fri, Apr 7, 2017, at 21:55, Eric Voskuil via bitcoin-dev wrote: >> Optimization for lower memory platforms then becomes a process >> of reducing the need for paging. This is the purpose of a cache. >> The seam between disk and memory can be filled quite nicely by a >> small amount of cache. On high RAM systems any cache is actually >> a de-optimization but on low RAM systems it can prevent excessive >> paging. This is directly analogous to a CPU cache. > > > I am not entirely sure I agree with that, or understand it > correctly. > > If -for example - the data of some application is a set of > records which can be sorted from least frequently used to most > frequently used then doing just that sort will beat any > application-layer cache. Regardless of size of data and size of > RAM, you simply allow the OS to use disk caching or memory map > caching to work its magic . It's a reasonable assumption, and given that the no-explicit-cache implementation is a subset of the optionally-cached implementation, was of course the initial implementation. > In fact, I would argue that an application-layer cache *only* > makes sense if the data model shows a *hard* distinction between > often and not often used data. If usage-frequency is a continuous > line, caching is best left to the OS by focussing on proper spatial > and temporal locality of reference of your data, because the OS has > much more information to make the right decision. In practice this is not the case. The Bitcoin data model is neither continuous nor strictly segregated by usage. It is true that with sufficient RAM a cache is totally counterproductive. It is also my experience that an independent UTXO store is not a reasonable/necessary trade of disk space, memory scalability, and/or code complexity in exchange for speed. But on lower memory systems a explicit cache is beneficial. The difference is clearly measurable in production code by simply changing the cache limit and testing on various configurations. e -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBCAAGBQJY6CXnAAoJEDzYwH8LXOFOf0YH/2qk3hYC6iEDW/DWM2ffkdb9 QM7A29Pvbfw9Wjr5Xx+ugIQvlAr4T+nByOCT6AnrqNU5K3UUmbC0KIB1rEL94hsK QYVlLs0cOrjg8qKJpck+wcgiWw3VbEa/Y44hK7NLUxoy2HsLYaxPhqFH3GGgowqR syga626jf2YUyudZxj1gFuqn7grkwghnzdrEUJMcqQo8IvCqjftGXlKxBGyB/AIs Dx+5EWO9Q9IxrNpg/fsKKB6xkMxkmSx2hbD7dmEBvi/afbVF66rDTinjInG/LCju pV7kT/GAWqGQGku6sQyAOexsxVhWA8EA/QEjvbyyGb+3YnR0s6nPk+CxO+RkOgo= =e+Pr -----END PGP SIGNATURE-----