[Bitcoin-development] Linux packaging letter

[Bitcoin-development] Linux packaging letter

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

Hi,

Some of us have put together an open letter to the Linux packaging
community, outlining why Bitcoin is different to other programs and asking
them to not patch or modify the upstream sources.

Please consider signing it if you agree (I think the wording by now is
fine, so don't edit the contents - use the comment feature if you want to
though).

https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi

The trigger for this is the discovery that Debian bitcoind's got split out
of the consensus some time in April, for reasons that nobody yet figured
out but is presumably related to a patch (eg it uses system leveldb).

[-- Attachment #2: Type: text/html, Size: 940 bytes --]

Re: [Bitcoin-development] Linux packaging letter

[Gregory Maxwell]

On Tue, Jul 23, 2013 at 1:01 PM, Mike Hearn <mike@plan99.net> wrote:
> Hi,
>
> Some of us have put together an open letter to the Linux packaging
> community, outlining why Bitcoin is different to other programs and asking
> them to not patch or modify the upstream sources.
>
> Please consider signing it if you agree (I think the wording by now is fine,
> so don't edit the contents - use the comment feature if you want to though).
>
> https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi

Ah, this is not entirely in sync with the last (mostly minor)
copyedits that had been signed off by Gavin, Pieter, Jgarzik, and I...
but that appears to be a smaller issue than the fact that the whole
thing is now being rewritten by "anonymous beaver" and friends.

Re: [Bitcoin-development] Linux packaging letter

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

Yes. Someone decided to actually delete the people who had signed so far
and replace it with a request for PGP signing - no. Not everyone even uses
PGP, which is overkill for this anyway.

I'm going to roll the document back and lock it. Sorry, I had hoped people
would respect my request to not fiddle with the content, which they did not
do.

If you'd like to have your name on it, let me know or post here and I'll
add it.


On Tue, Jul 23, 2013 at 10:14 PM, Gregory Maxwell <gmaxwell@gmail.com>wrote:

> On Tue, Jul 23, 2013 at 1:01 PM, Mike Hearn <mike@plan99.net> wrote:
> > Hi,
> >
> > Some of us have put together an open letter to the Linux packaging
> > community, outlining why Bitcoin is different to other programs and
> asking
> > them to not patch or modify the upstream sources.
> >
> > Please consider signing it if you agree (I think the wording by now is
> fine,
> > so don't edit the contents - use the comment feature if you want to
> though).
> >
> >
> https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi
>
> Ah, this is not entirely in sync with the last (mostly minor)
> copyedits that had been signed off by Gavin, Pieter, Jgarzik, and I...
> but that appears to be a smaller issue than the fact that the whole
> thing is now being rewritten by "anonymous beaver" and friends.
>

[-- Attachment #2: Type: text/html, Size: 2013 bytes --]

Re: [Bitcoin-development] Linux packaging letter

[Gregory Maxwell]

On Tue, Jul 23, 2013 at 1:32 PM, Mike Hearn <mike@plan99.net> wrote:
> Yes. Someone decided to actually delete the people who had signed so far and

Some people/person went and actually started making substantive edits
to the text.
The text it's rolled back to is missing the last copyedits from last night too.

The text that had been ACKed last night was a3e52973,  available at
http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md

As far as the PGP goes—

I think using the PGP is good: it's making use of the right tools,
avoids issues like we just had where people go changing the content
after names had been affixed,  shows solidarity with people building
security infrastructure that our ecosystem depends on.  If you only
use it occasionally then its easy for someone to strip it when it _is_
needed and disguise that just as regular non-use.  It's my general
view that for people working in our domain basic competence and use of
these tools, even when they kinda stink, is a kind of civic hygiene.

At the same time it's not urgent. It's poorly used by people and will
be ignored by most but packagers are the most frequent users of it
that I've encountered.  Fortunately, it's harmless in any case.

If people are interested in offering PGP signatures of it:

wget http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md
gpg --clearsign 20130723-linux-distribution-packaging-and-bitcoin.md

and post the little signature asc. The result composes nicely:
http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md.asc

Re: [Bitcoin-development] Linux packaging letter

[John Dillon]

My signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Linux distribution packaging and Bitcoin
========================================
2013-07-23

This note summarises the dangers inherent in the Linux distribution
packaging model for Bitcoin, and forms a request from upstream
maintainers to not distribute Bitcoin node software as part of
distribution package repositories without understanding the special
requirements of Bitcoin.

Distributors typically unbundle internal libraries and apply other
patches for a variety of generally good reasons, including ensuring
that security-critical fixes can be applied once, rather than multiple
times for many different packages. In most cases, the common
distribution packaging policy has many advantages.

However, Bitcoin nodes are an unusual category of software: they
implement a complex group consensus in which every client verifies the
behaviour of every other exactly. Even an exceptionally subtle change -
including apparently harmless bugfixes - can cause a failure to reach
consensus. A consensus failure of one client is a security risk to the
user of that client. A significant number of nodes failing to reach
consensus - as happened in March 2013 due to a change in database
libraries[1] - is a critical problem that threatens the functionality
and security of the system for all users.

For this reason, it is _vital_ that as much of the network as possible
uses _unmodified_ implementations that have been carefully audited and
tested, including dependencies. For instance, if the included copy
of LevelDB in bitcoind is replaced by a system-wide shared library,
_any_ change to that shared library requires auditing and testing,
a requirement generally not met by standard distributor packaging
practices.

Because distributed global consensus is a new area of computer science
research, the undersigned request that distributors refrain from
packaging Bitcoin node software (including bitcoind and Bitcoin-Qt)
and direct users to the upstream-provided binaries instead _until they
understand the unique testing procedures and other requirements to
achieve consensus_. Beyond being globally consistent, upstream binaries
are produced using a reproducible build system[2], ensuring that they
can be audited for backdoors.

1. https://en.bitcoin.it/wiki/BIP_0050
2. http://gitian.org/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBCAAGBQJR9WC5AAoJEEWCsU4mNhiPg6UH/2oHzBWBPaQMhH/GCTHQEi5U
7GSRfqwihIs/M2ROHLNq0HhgWR7mPAh5TTI6+tG95FCGCGNZq0seqw9wW4ZyGCoc
VueY51q4hcn23405oLD/QGK2lDxxywWY8XtFYVPqAzXTq6zRzgpNJkjoRtOAUOP7
3PrRkimYYyj0KrqFg+cEvZDT27dkeX+5PXM6Ua0o7h/TlhR2RJPhej5DI8cNLXgA
f0t+mES4Apb6zLgnEYYlPp22FR9vuFcJO3z1akhVL4DLUMqr58NYHLVnH1FH0Jhn
hVuld159QtCjQ5Qyn19cn86akTQJVi+ikCXqaKriCc2jBFX7TCI8WTDc6aSZpsQ=
=oX5d
-----END PGP SIGNATURE-----

Re: [Bitcoin-development] Linux packaging letter

[Scott Howard]

On Tue, Jul 23, 2013 at 4:01 PM, Mike Hearn <mike@plan99.net> wrote:
> Hi,
>
> Some of us have put together an open letter to the Linux packaging
> community, outlining why Bitcoin is different to other programs and asking
> them to not patch or modify the upstream sources.
>
> Please consider signing it if you agree (I think the wording by now is fine,
> so don't edit the contents - use the comment feature if you want to though).
>
> https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi
>
> The trigger for this is the discovery that Debian bitcoind's got split out
> of the consensus some time in April, for reasons that nobody yet figured out
> but is presumably related to a patch (eg it uses system leveldb).

Hi Mike,
Debian's bitcoin is maintained on an open and archived mailing list
and git repo:
Debian Bitcoin Packaging Team <pkg-bitcoin-devel@lists.alioth.debian.org>

If there is a problem or question, getting an answer should be really
easy. It would be good to include them in the discussion there (I
CC:ed the list). If the upstream developers have a consensus that
distribution packaging is not in the best interest of the project,
then I personally would defer to their judgement and request removal.
I'm leaving this open for other opinions from the Debian side.

That said, let me summarize the arguments and see if we can figure out
a permanent solution:

1) It appears that the consensus of upstream developers is that any
distributed binary should only be linked against libraries that the
bitcoin developers have tested and audited since any compatibility bug
is a risk to both the user and the network.

Response: Is there a way to "certify" the Debian libraries? Debian
bitcoind/bitcoin-qt runs the compile test during all architectures.
MIPS has been failing recently, but no one has looked into it yet.
Perhaps it's not worth developers efforts yet, but at some point the
technology should reach a point it can be redistributed.


2) Bitcoin is new technology, so any patches have the ability of
harming both the network and user.

Response: I, and I'm sure everyone else working on it, totally agrees.
All patches are public [1], you can see that the patches are only to
the build system (except for one adding a debug message). Is there a
specific patch or bug that is problematic? This seems to be
reiterating (1) above: don't patch the build system to use libraries
that were not audited by the developers.



The two solutions are: (1) no one besides the upstream developers
compiles and distributes binaries, ever, or (2) upstream comes up with
a system where someone besides them can compile working binaries for
distribution. Most likely the best solution is to do (1) until
upstream sets up a system to allow (2).

I'm curious as to other's opinions.
Thanks,
Scott


[1] http://anonscm.debian.org/gitweb/?p=collab-maint/bitcoin.git;a=tree;f=debian/patches;h=ba576f9f3ddad47a2f85dcbfb7a0b3482834f19f;hb=HEAD

Re: [Bitcoin-development] Linux packaging letter

[Luke-Jr]

[-- Attachment #1: Type: Text/Plain, Size: 3667 bytes --]

On Tuesday, July 23, 2013 10:02:28 PM Scott Howard wrote:
> 1) It appears that the consensus of upstream developers is that any
> distributed binary should only be linked against libraries that the
> bitcoin developers have tested and audited since any compatibility bug
> is a risk to both the user and the network.
> 
> Response: Is there a way to "certify" the Debian libraries? Debian
> bitcoind/bitcoin-qt runs the compile test during all architectures.

It doesn't need to be audited by any given person or team, just someone who 
understands the issues and can dedicate the time to doing a competent audit.
Testing bitcoind/bitcoin-qt is not sufficient: you must guarantee that your 
libraries (especially LevelDB) are bug-for-bug compatible with the ones used 
by everyone else. And not only the current versions, but every future version 
to ever hit the repository. This means a lot of additional work for the 
maintainers of the library packages, and the security team; for example, the 
security team must understand that they *cannot* deploy a critical security 
bugfix to LevelDB until someone competent has reviewed that it is 
behaviourally (including bug behaviours!) compatible with the versions in use 
everywhere else/previously. I think it is likely all this additional 
work/delays will be considered unacceptable to your library/security teams, 
thus using the bundled/embedded LevelDB is probably the best solution.

> MIPS has been failing recently, but no one has looked into it yet.
> Perhaps it's not worth developers efforts yet, but at some point the
> technology should reach a point it can be redistributed.

MIPS (and any other big endian architecture) has *always* failed on the 
Satoshi codebase, and will not be supported until someone has time to dedicate 
to fixing the numerous little-endian assumptions in the code. I have an 
incomplete port, but it isn't very high on my priority list to figure out what 
else it's missing.

> 2) Bitcoin is new technology, so any patches have the ability of
> harming both the network and user.
> 
> Response: I, and I'm sure everyone else working on it, totally agrees.
> All patches are public [1], you can see that the patches are only to
> the build system (except for one adding a debug message). Is there a
> specific patch or bug that is problematic? This seems to be
> reiterating (1) above: don't patch the build system to use libraries
> that were not audited by the developers.
> 
> The two solutions are: (1) no one besides the upstream developers
> compiles and distributes binaries, ever, or (2) upstream comes up with
> a system where someone besides them can compile working binaries for
> distribution. Most likely the best solution is to do (1) until
> upstream sets up a system to allow (2).

Debian could probably get away with packaging Bitcoin-Qt and bitcoind as-is 
with no modifications, and not have any problems. It's only when you begin 
making modifications that it becomes a problem. There are also some concerns 
that it puts a much larger price on compromising Debian's build servers and/or 
repositories (suddenly the attacker can steal a LOT of money).

The official binaries are not simply built by upstream developers: using 
Gitian, *anyone* can produce bit-for-bit identical binaries. Official releases 
are only published after 3 or more people have done an independent compile and 
signed the output. It would be excellent if the whole of Debian could work 
toward achieving this level of security eventually, which would make 
distributing Bitcoin node software much safer as well.

Luke

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 1530 bytes --]

Re: [Bitcoin-development] Linux packaging letter

[Scott Howard]

On Tue, Jul 23, 2013 at 6:26 PM, Luke-Jr <luke@dashjr.org> wrote:
> This means a lot of additional work for the
> maintainers of the library packages, and the security team; for example, the
> security team must understand that they *cannot* deploy a critical security
> bugfix to LevelDB until someone competent has reviewed that it is
> behaviourally (including bug behaviours!) compatible with the versions in use
> everywhere else/previously. I think it is likely all this additional
> work/delays will be considered unacceptable to your library/security teams,
> thus using the bundled/embedded LevelDB is probably the best solution.

The above is a key point, lukejr addressed it well "I think it is
likely all this additional work/delays will be considered unacceptable
to your library/security teams, thus using the bundled/embedded
LevelDB is probably the best solution."

>> MIPS has been failing recently, but no one has looked into it yet.
>> Perhaps it's not worth developers efforts yet, but at some point the
>> technology should reach a point it can be redistributed.
>
> MIPS (and any other big endian architecture) has *always* failed on the
> Satoshi codebase, and will not be supported until someone has time to dedicate
> to fixing the numerous little-endian assumptions in the code. I have an
> incomplete port, but it isn't very high on my priority list to figure out what
> else it's missing.

To be clear, bitcoind/bitcoin-qt is only built on little endian machines
https://buildd.debian.org/status/package.php?p=bitcoin

> Debian could probably get away with packaging Bitcoin-Qt and bitcoind as-is
> with no modifications, and not have any problems. It's only when you begin
> making modifications that it becomes a problem. There are also some concerns
> that it puts a much larger price on compromising Debian's build servers and/or
> repositories (suddenly the attacker can steal a LOT of money).

The only current modification is to use system leveldb instead of the
packaged leveldb. (There is also a patch porting libmemenv.a to
several other architectures, but that is only used in test suites - so
it shouldn't pose a risk to users).

>
> The official binaries are not simply built by upstream developers: using
> Gitian, *anyone* can produce bit-for-bit identical binaries. Official releases
> are only published after 3 or more people have done an independent compile and
> signed the output. It would be excellent if the whole of Debian could work
> toward achieving this level of security eventually, which would make
> distributing Bitcoin node software much safer as well.

Ironically, debian (in general) doesn't trust upstream security
maintenance of third part libraries - that's why they typically get
dropped in favor of use system libraries.

In this case, upstream doesn't trust (rightfully) that some future
debian security team bug fix to a stable library won't be tested
properly against bitcoin, causing problems for users (since bitcoin
might expect buggy library behavior).


I'm not the original packager or maintainer - I just came across the
package in really bad shape and helped bring it to something
reasonable and have done the most recent uploads (since 0.8, I
believe). Since updated libraries could pose a security risk because
bitcoin may expect buggy behavior, I think that is a good argument for
debian to use the included library. However, I'm just a recent helper
- I still want to hear what people who have been doing this for longer
think.

~Scott

Re: [Bitcoin-development] Linux packaging letter

[Douglas Huff]

Honestly, until I read the quoted part of your response, I actually wasn't in favor of this whole thing since in general the types of issues being mentioned are, in large part, the types of issues that maintainers deal with all the time.

On Jul 23, 2013, at 3:02 PM, Scott Howard <showard314@gmail.com> wrote:

> Response: Is there a way to "certify" the Debian libraries? Debian
> bitcoind/bitcoin-qt runs the compile test during all architectures.
> MIPS has been failing recently, but no one has looked into it yet.
> Perhaps it's not worth developers efforts yet, but at some point the
> technology should reach a point it can be redistributed.


The fact that you're even trying to package and/or at some point have packaged and shipped big endian binaries is straight up *NEGLIGENT.*

Stop that. Now. It won't work.

Thanks for showing that this *is* necessary, I guess.

Re: [Bitcoin-development] Linux packaging letter

[Scott Howard]

On Tue, Jul 23, 2013 at 9:45 PM, Douglas Huff <dhuff@jrbobdobbs.org> wrote:
> Honestly, until I read the quoted part of your response, I actually wasn't in favor of this whole thing since in general the types of issues being mentioned are, in large part, the types of issues that maintainers deal with all the time.
>
> On Jul 23, 2013, at 3:02 PM, Scott Howard <showard314@gmail.com> wrote:
>
>> Response: Is there a way to "certify" the Debian libraries? Debian
>> bitcoind/bitcoin-qt runs the compile test during all architectures.
>> MIPS has been failing recently, but no one has looked into it yet.
>> Perhaps it's not worth developers efforts yet, but at some point the
>> technology should reach a point it can be redistributed.
>
>
> The fact that you're even trying to package and/or at some point have packaged and shipped big endian binaries is straight up *NEGLIGENT.*
>
> Stop that. Now. It won't work.
>
> Thanks for showing that this *is* necessary, I guess.

before people get too upset, I'm talking about little-endian MIPS
(debian-mipsel)
http://www.debian.org/ports/mips/

[Bitcoin-development] Endianness (was: Linux packaging letter)

[Wendell]

Forking for curiosity's sake:

Is there a substantial barrier to endian independence in the Bitcoin codebase?

-wendell

grabhive.com | twitter.com/grabhive

On Jul 24, 2013, at 3:45 AM, Douglas Huff wrote:

> The fact that you're even trying to package and/or at some point have packaged and shipped big endian binaries is straight up *NEGLIGENT.*

Re: [Bitcoin-development] Endianness (was: Linux packaging letter)

[Luke-Jr]

On Wednesday, July 24, 2013 3:54:25 AM Wendell wrote:
> Is there a substantial barrier to endian independence in the Bitcoin
> codebase?

I got the obvious stuff ('endian' branch in my repo), but it still didn't work 
when I moved on. I haven't had time to try to figure out why not yet.

Luke

Re: [Bitcoin-development] Endianness (was: Linux packaging letter)

[Gregory Maxwell]

On Tue, Jul 23, 2013 at 8:54 PM, Wendell <w@grabhive.com> wrote:
> Forking for curiosity's sake:
> Is there a substantial barrier to endian independence in the Bitcoin codebase?

Not really. The software was originally written to write out memory
order to and from the wire, which is part of why the protocol is LE
everywhere, so fixing that much is pretty typical endianness fixes.
There is an extra kink in that almost everything Bitcoin sends and
receives is an authenticated data structure— the stuff gets hashed for
authentication.  So that simply swizzling the byte order on
immediately on input isn't enough because sometimes you'll go on to
hash that data and it can't be in memory order for that.

Luke gave an initial crack at it a long time ago:
http://gitorious.org/~Luke-Jr/bitcoin/luke-jr-bitcoin/commits/endian
But it wasn't enough yet.

Seems like its just enough of an undertaking that absent a really good
reason to care about it no real progress in fixing it is happening.

Re: [Bitcoin-development] Endianness (was: Linux packaging letter)

[Gregory Maxwell]

On Tue, Jul 23, 2013 at 9:07 PM, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> order to and from the wire, which is part of why the protocol is LE
> everywhere,
*before someone corrects me, it's not LE everywhere (I meant
"manywhere" :P)— there is just enough BE to keep you on your toes. :P

Re: [Bitcoin-development] Linux packaging letter

[Pieter Wuille]

On Tue, Jul 23, 2013 at 10:01:55PM +0200, Mike Hearn wrote:
> The trigger for this is the discovery that Debian bitcoind's got split out
> of the consensus some time in April, for reasons that nobody yet figured
> out but is presumably related to a patch (eg it uses system leveldb).

Just to make sure there are no misunderstandings, as far as I know there is
no reason to assume the reported problem (comment on #2726) is:
1) a fork (it's an indeterministic and avoidable database corruption, it seems)
2) related to leveldb
3) reproducible by more than one person
4) debian's fault.

That said, I think reaching out to packagers to educate them about the risks
is a good idea - but let's not blame people before we understand our own
problems.

-- 
Pieter

Re: [Bitcoin-development] Linux packaging letter

[Greg Troxel]

I find it interesting that this is a "linux packaging letter".  How much
of this applies to pkgsrc, FreeBSD ports, OpenBSD ports, and other
non-Linux packaging systems (pkgsrc supports Linux as on of 20 operating
systems, but is not a "Linux packaging system")?

Is the repeatable build infrastructure portable (to any reasonable
mostly-POSIX-compliant system, with gcc or clang)?  I have the vague
impression it's Ubuntu only, but I am very unclear on this point.  How
does this repeatableness interact with building for multiple operating
systems and cpu types (say 20 OS, with typically 3 versions of the OS
for each, with 1-20 cpu types per OS, for a cross-product of perhaps 200
combinations)?

Requiring precise library depdendencies is quite awkward.  Certainly
requiring new enough to avoid known bugs is understandable, but that
should be caught at configure time and fail.  Synchronous updates of
multiple packages is difficult, and runs into A wants only n of C, while
B wants only m.  So if you are talking about running regression tests
with the set of versions of a dependency that are considered reasonable,
and there's therefore a solution to the multiple-package constraint
problem, that seems ok.

It seems like a bug that the package will build on BE systems and then
fail tests.   If it's known not to be ok, it seems that absent some
configure-time flag the build should fail.

Asking people not to patch should mean willingnesss to make accomodation
in the master sources for build issues for multiple packaging systems.
I haven't gotten around to packaging this for pkgsrc - so far I only
have the energy to lurk (due to too many things on the todo list).  But
I often find that some changes are needed.  If you're willing (in
theory) to add in configure flags to control build behavior (in a way
that you can audit and decide is safe), that's great, and of course we
can discuss an actual situation when one gets figured out.

Greg

Re: [Bitcoin-development] Linux packaging letter

[Luke-Jr]

[-- Attachment #1: Type: Text/Plain, Size: 2556 bytes --]

On Tuesday, July 23, 2013 11:23:27 PM Greg Troxel wrote:
> I find it interesting that this is a "linux packaging letter".  How much
> of this applies to pkgsrc, FreeBSD ports, OpenBSD ports, and other
> non-Linux packaging systems (pkgsrc supports Linux as on of 20 operating
> systems, but is not a "Linux packaging system")?

It was written with bitcoind/Bitcoin-Qt in mind, which don't work on BSD. :p

> Is the repeatable build infrastructure portable (to any reasonable
> mostly-POSIX-compliant system, with gcc or clang)?  I have the vague
> impression it's Ubuntu only, but I am very unclear on this point.  How
> does this repeatableness interact with building for multiple operating
> systems and cpu types (say 20 OS, with typically 3 versions of the OS
> for each, with 1-20 cpu types per OS, for a cross-product of perhaps 200
> combinations)?

It should be portable to other systems, though hasn't been done yet.
Would be nice if the concepts it uses could be integrated into the package-
building systems.

> Requiring precise library depdendencies is quite awkward.  Certainly
> requiring new enough to avoid known bugs is understandable, but that
> should be caught at configure time and fail.

The problem is that we require bugs. That is, if your library has those bugs 
fixed, you have introduced a security vulnerability.

> It seems like a bug that the package will build on BE systems and then
> fail tests.   If it's known not to be ok, it seems that absent some
> configure-time flag the build should fail.

There is no configure-time for this node software yet. The autoconf-based one 
in the works *does* make this check, however.

> Asking people not to patch should mean willingnesss to make accomodation
> in the master sources for build issues for multiple packaging systems.
> I haven't gotten around to packaging this for pkgsrc - so far I only
> have the energy to lurk (due to too many things on the todo list).  But
> I often find that some changes are needed.  If you're willing (in
> theory) to add in configure flags to control build behavior (in a way
> that you can audit and decide is safe), that's great, and of course we
> can discuss an actual situation when one gets figured out.

The review process is very slow and strict, but that is because of the same 
reasons it isn't safe to distribute patched versions in general. Merging your 
patches to mainline is not only a good idea, but it helps to ensure they get 
the necessary testing needed to be safe.

Luke

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 1530 bytes --]

Re: [Bitcoin-development] Linux packaging letter

[Gregory Maxwell]

On Tue, Jul 23, 2013 at 4:23 PM, Greg Troxel <gdt@work.lexort.com> wrote:
> Is the repeatable build infrastructure portable (to any reasonable
> mostly-POSIX-compliant system, with gcc or clang)?  I have the vague

It's "portable" to anything that can run the relevant VMs.  Uh
provided you don't mind cross compiling everything from an unbuntu VM.
 It certainly would be nice if the trusted-computing-base for gitian
were a bit smaller, thats an area for long term improvement for sure.

It may need some massaging. The tor project is beginning to use the
same infrastructure, so this could be usefully conserved work.

Likewise expanding the supported output targets would be great— though
in the case of Bitcoin this is bounded by resources to adequately QA
builds on alternative targets.

> Requiring precise library depdendencies is quite awkward.  Certainly
> requiring new enough to avoid known bugs is understandable, but that
> should be caught at configure time and fail.

In some cases packages solving bugs is problematic for Bitcoin.

This is something that it seems to take a whiteboard to explain, so I
apologize for the opacity of simple email here.

From a technical perspective Bitcoin's whole purpose is getting a
whole bunch of computers world wide to reach a bit identical agreement
on the content of a database, subject to a whole pile of rules, in the
face of potentially malicious input, without any trusted parties at
all (even the guy you got the software from, assuming you have the
resources to audit it).

I'll walk through a simple example:

Say Bitcoin used a backing database which had an unknown a bug where
any item with a key that begins with 0xDEADBEEF returns not found when
queried, even if its in the DB. Once discovered, any database library
would want to fix that quickly and they'd fix it in a point release
without reservation. They might not even release note that particular
fix it if went along with some others, it could even be fixed
accidentally.

Now say that we have a state where half the Bitcoin network is running
the old buggy version, and half is running the fixed version.  Someone
creates a transaction with ID 0xDEADBEEF...  and then subsequently
spends the output of that transaction. This could be by pure chance or
it could be a malicious act.

To half the network that spending transaction looks like someone
spending coin from nowhere, a violation of the rules.  The consensus
would then fork, effectively partitioning the network.  On each fork
any coin could be spent twice, and the fork will only be resolvable by
one side or the other abandoning their state (generally the more
permissive side would need to be abandoned because the permissive one
is tolerant of the restrictive one's behavior) by manually downgrading
or patching software.  As a result of this parties who believed some
of their transactions were safely settled would find them reversed by
people who exploited the inconsistent consensus.

To deploy such a fix in Bitcoin without creating a risk for
participants we need to make a staged revision of the network protocol
rules:  There would be a protocol update that fixed the database bug
_and_ explicitly rejected 0xDEADBEEF transactions until either some
far out future date or until triggered by quorum sensing (or both).
The users of Bitcoin would all be advised that they had to apply
fixes/workaround by the switchover point or be left out of service or
vulnerable. At designated time / quorum nodes would simultaneously
switch to the new behavior.  (Or in some cases, we'd just move the
'bug' into the Bitcoin code so that it can be fixed in the database,
and we'd then just keep it forever, depending on how harmful it was to
Bitcoin, a one if 4 billion chance of having to rewrite a transaction
wouldn't be a big deal)

We've done these organized updates to solve problems (as various flaws
in Bitcoin itself can present similar consensus risks) several times
with great success, typical time horizons spanning for months to
years.  But it cannot work if the behavior is changed out from under
the software.

Fortunately, if the number of users running with an uncontrolled
consensus relevant inconsistent behavior is small the danger is only
to themselves (and, perhaps, their customers). I'm not happy to see
anyone get harmed, but it's better if its few people than many. This
is part of the reason that it's a "linux packaging letter", since for
Bitcoin the combination of uncoordinated patching and non-trivial
userbases appears to be currently unique to GNU/Linux systems.  Though
indeed, the concerns do apply more broadly.

> multiple packages is difficult, and runs into A wants only n of C, while
> B wants only m.

My understanding is that gentoo is actually able to handle this (and
does, for Bitcoin)— and really I presume just about everything else
could with enough effort. I certainly wouldn't ask anyone else to do
that.  If you're really getting into the rathole of building separate
libraries just for Bitcoin the value of packaging it goes away.

> So if you are talking about running regression tests
> with the set of versions of a dependency that are considered reasonable,
> and there's therefore a solution to the multiple-package constraint
> problem, that seems ok.

Running a complete set of tests is a start— though the unit tests are
not and cannot be adequate. There is a full systems testing harnesses
which should be used on new platforms.  Even that though isn't really
adequate, as it is currently infeasible to even achieve complete test
coverage in things like cryptographic libraries and database
environments.

This is an area where both the Bitcoin software ecosystem and the
greater art of large scale software validation need to mature. You
won't hear anyone applauding the fact that harmless looking bugfixes
in leveldb, boost, or openssl could be major doom event makers

We're not crazy folks who insist on using formally undefined behavior
and argue that it should never be changed out from under us. When
there is a known risk we will boil the oceans to close it even if we
think that the world would be more 'proper' some other way,  but for
known-unknowns and unknown-unknowns we can only adopt a conservative
approach and try to do our best.

One of the middle term things we did was internally integrated our
validation database library (leveldb).  Since we _know_ that its a
consistency critical component, and a part of our system that is
especially difficult to validate, integrating it meant removing a lot
of risk and allowed it to be upgraded with an eye on the
Bitcoin-specific consequences.  Unfortunately distributions have been
patching Bitcoin to unbundle it.  Checking versions isn't adequate
because, at least in other packages, some distributors frequently
backport fixes or apply novel fixes which may not even be shared with
upstream.

Other considerations may drive us out of external dependencies for
many of the consensus parts of Bitcoin. E.g. Pieter has writen an
ECDSA library for our specific ECC curve which does signature
validation >6x faster than OpenSSL (but it isn't obviously
upstreamable due to some differing requirements for constant time
operations), at some point we may need to adopt a backing database
that is able to produce authentication proofs, etc.  Certainly
additional clever tests will make undiscovered surprising behavior
less likely, though figuring out how to get the tests actually run if
they take two hours and use 20GB of disk space is a challenge.

... but today we need to work with what we have, which is fragile in
some atypical ways.  Part of that is making an effort to make sure
that anyone who might create a big footgun event has some idea of the
concern space.

> It seems like a bug that the package will build on BE systems and then
> fail tests.   If it's known not to be ok, it seems that absent some
> configure-time flag the build should fail.

Configure time?  At the moment Bitcoin is built with a straight
forward makefile (though there is a switch to autotools in the
pipeline).

BE isn't currently supported (and I believe this is well documented in
the package).  Fixing this would be nice, patches accepted.   There
was an amusing incident a while back where a distributor was refusing
to take an update that added unit tests because they revealed failures
on BE, nevermind that the application itself instantly failed on BE
and never worked there. I believe that has since been resolved.

> Asking people not to patch should mean willingnesss to make accomodation
> in the master sources for build issues for multiple packaging systems.
> I haven't gotten around to packaging this for pkgsrc - so far I only
> have the energy to lurk (due to too many things on the todo list).  But
> I often find that some changes are needed.  If you're willing (in
> theory) to add in configure flags to control build behavior (in a way
> that you can audit and decide is safe), that's great, and of course we
> can discuss an actual situation when one gets figured out.

I _believe_  (and hope) we've been very accommodating system specific
fixes, even for systems we don't formally support.

I personally believe that portable software is better software.
Portability forces you to dust out nasty cobwebs, reveals dependency
on dangerous undefined behavior, encourages intelligent abstractions
and appropriate testing, and it invites contributions from more hands
and eyes— I don't care if you use a weird OS: I just want you for your
code and your bug-reports.  So even if we don't consider a platform
worth supporting in any rigorous way, we should still be open to fixes
and build support.

... although we're typical very much resource bound on testing. Our
upstreaming pipeline is often somewhat slow. But it's slow because we
are serious about review, even of trivial changes. Being slow is no
reason to not submit, even if you make a decision to not block on it
(though, if you're doing that you should make the decision in full
knowledge of the potential implications). Like all things stepping up
and being willing to do the work goes a long way to getting things
done.

Re: [Bitcoin-development] Linux packaging letter

[zooko]

Folks:

With all due respect, I think the letter as I see it at
https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi
should be changed before being shown to package maintainers. I think some
package maintainers might perceive this version of the letter as high-handed --
telling someone else how to do their job -- and they might not notice the
actual facts included in the letter explaining why Bitcoin really *is*
different than a lot of software.

You should understand that without a careful read, this letter sounds much like
a cry that packagers have heard from hundreds of other authors who say things
to the effect that "my software is different and more important and packager
maintainers have to do things my way".

Why not solicit the cooperation of a few package maintainers and write a joint
letter with them signing on? Instead of it being a one-sided lecture from
Bitcoin devs to packagers, it would be a shared statement *and* packagers, and
it would be phrased in language that would make it instantly clear to other
packagers that this isn't just another whine from ignorant devs.

If you're interested in that, there are lots of packagers who would be happy to
help. Greg Troxel (pkgsrc) is one, who has already posted to this thread. I'd
be happy to invite the ones that I've worked with to package the software that
I am a dev on -- Tahoe-LAFS.

What I'm proposing is that we contact some packagers and say "Here's this rough
draft, and we'd like you to suggest edits that would make it into the kind of
letter that you'd sign your name to.". At the very least, we'd learn something
from the ensuing conversation.

Regards,

Zooko

Re: [Bitcoin-development] Linux packaging letter

[Gregory Maxwell]

On Tue, Jul 23, 2013 at 7:35 PM, zooko <zooko@zooko.com> wrote:
> I think some
> package maintainers might perceive this version of the letter as high-handed --
> telling someone else how to do their job -- and they might not notice the
> actual facts included in the letter explaining why Bitcoin really *is*
> different than a lot of software.

Bummer, because this was a explicit consideration while writing it and
a concern several people had with the initial draft Mike did.

We're very much aware that upstreams frequently cry (wolf) at the
mutilation of their unique and precious snowflake.

The intention was that second paragraph acknowledging the many good
motivations for the existing norms and the third paragraph talking
about consensus systems would address these concerns— showing that we
aren't totally clueless, and pointing out that we have an actually
unusual situation. In intermediate drafts they were longer and more
elaborate, but we were struggling against length and trying to avoid
delving into a highly technical discussion which would lose anyone who
wasn't already very interested.

We also compromised on an initial approach of "please don't package
this at all" to "please understand first", in part at the protest of
our gentoo package (which also bundles leveldb but hard locks it to an
exact version in the package system with exact build flags, which is a
sophisticated compromise which might not generalize to other
distributors) maintainer (uh, Luke-Jr, not exactly the most
representative sample).

As a first step it's at least important to know that there is a
concern here shared by a bunch of people. Helping talk people through
understanding it is part of the job here.  I certainly didn't expect
the discussion to stop with the letter but getting it out there is a
way to start the discussion and make it more likely that we have it
again with the next packager who comes around.

I guess the first priority though is avoiding gratuitously offending
people.  Can anyone point out any specific tweaks that would reduce
initial bristling?

On Tue, Jul 23, 2013 at 6:45 PM, Douglas Huff <dhuff@jrbobdobbs.org> wrote:
> Honestly, until I read the quoted part of your response,

Oh be nice. If any of this were easy it would all be _done_ already. :)

There is naturally some tension when people with different priorities
and backgrounds interact, ... I've seen a lot of upstreams run into
disagreements with packagers the result is usually better for
everyone.

Re: [Bitcoin-development] Linux packaging letter

[Mike Hearn]

[-- Attachment #1: Type: text/plain, Size: 4295 bytes --]

Yeah, if anyone wants to make the letter more digestable please do propose
an alternative, although by this point it's probably not worth it as people
have already signed.

FWIW, Gregory is right that my original draft was much more brusque. The
pain in the packaging relationship travels both ways. I have in the past
wasted a lot of time due to bogus packaging applied by non-expert packagers
that broke things. In fact the project I was a part of adopted a policy of
automatically closing bug reports from people who were using distributor
packages (any distro) because the quality was so inconsistent and so many
subtle bugs were introduced.

If packagers hear upstreams cry about packaging a lot, I think you should
keep an open mind that some of them probably know what they're talking
about. We really shouldn't have to beg and cajole here. Saying "we have our
reasons and we want you to stop" should be enough.




On Wed, Jul 24, 2013 at 5:19 AM, Gregory Maxwell <gmaxwell@gmail.com> wrote:

> On Tue, Jul 23, 2013 at 7:35 PM, zooko <zooko@zooko.com> wrote:
> > I think some
> > package maintainers might perceive this version of the letter as
> high-handed --
> > telling someone else how to do their job -- and they might not notice the
> > actual facts included in the letter explaining why Bitcoin really *is*
> > different than a lot of software.
>
> Bummer, because this was a explicit consideration while writing it and
> a concern several people had with the initial draft Mike did.
>
> We're very much aware that upstreams frequently cry (wolf) at the
> mutilation of their unique and precious snowflake.
>
> The intention was that second paragraph acknowledging the many good
> motivations for the existing norms and the third paragraph talking
> about consensus systems would address these concerns— showing that we
> aren't totally clueless, and pointing out that we have an actually
> unusual situation. In intermediate drafts they were longer and more
> elaborate, but we were struggling against length and trying to avoid
> delving into a highly technical discussion which would lose anyone who
> wasn't already very interested.
>
> We also compromised on an initial approach of "please don't package
> this at all" to "please understand first", in part at the protest of
> our gentoo package (which also bundles leveldb but hard locks it to an
> exact version in the package system with exact build flags, which is a
> sophisticated compromise which might not generalize to other
> distributors) maintainer (uh, Luke-Jr, not exactly the most
> representative sample).
>
> As a first step it's at least important to know that there is a
> concern here shared by a bunch of people. Helping talk people through
> understanding it is part of the job here.  I certainly didn't expect
> the discussion to stop with the letter but getting it out there is a
> way to start the discussion and make it more likely that we have it
> again with the next packager who comes around.
>
> I guess the first priority though is avoiding gratuitously offending
> people.  Can anyone point out any specific tweaks that would reduce
> initial bristling?
>
> On Tue, Jul 23, 2013 at 6:45 PM, Douglas Huff <dhuff@jrbobdobbs.org>
> wrote:
> > Honestly, until I read the quoted part of your response,
>
> Oh be nice. If any of this were easy it would all be _done_ already. :)
>
> There is naturally some tension when people with different priorities
> and backgrounds interact, ... I've seen a lot of upstreams run into
> disagreements with packagers the result is usually better for
> everyone.
>
>
> ------------------------------------------------------------------------------
> See everything from the browser to the database with AppDynamics
> Get end-to-end visibility with application monitoring from AppDynamics
> Isolate bottlenecks and diagnose root cause in seconds.
> Start your free trial of AppDynamics Pro today!
> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>

[-- Attachment #2: Type: text/html, Size: 5320 bytes --]

Re: [Bitcoin-development] Linux packaging letter

[Jeff Garzik]

On Wed, Jul 24, 2013 at 4:28 AM, Mike Hearn <mike@plan99.net> wrote:
> Yeah, if anyone wants to make the letter more digestable please do propose
> an alternative, although by this point it's probably not worth it as people
> have already signed.

I'm working on a more digestable alternative:
https://gist.github.com/jgarzik/6065679

Should be ready in another ~24 hours, as its obviously incomplete
right now.  Alas the publishing of the lame version (which yes, I did
ACK) didn't give me time to finish my version.

I worked on Fedora packaging while at Red Hat, so hopefully, I have a
bit of insight here.  Was also thinking about publishing this on
opensource.com.

-- 
Jeff Garzik
Senior Software Engineer and open source evangelist
BitPay, Inc.      https://bitpay.com/

Re: [Bitcoin-development] Linux packaging letter

[zooko]

On Wed, Jul 24, 2013 at 09:52:33AM -0400, Jeff Garzik wrote:
> 
> I'm working on a more digestable alternative:
> https://gist.github.com/jgarzik/6065679

Hi Jeff! Thanks for working on it. Even if that letter
(https://gist.github.com/jgarzik/6065679) doesn't supplant
https://docs.google.com/a/leastauthority.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#heading=h.i7tz3gqh65mi
as a message-to-packagers, it looks like it will still turn out to be a useful
text.

My first question about it is this part:

"""
 Make a mistake, lose $1 billion

The consequences of bitcoin consensus failure are very high, comparable to avionics or medical device software. As of this writing, over $1 billion of value depends on bitcoin software being able to reliably achieve consensus over the worldwide Internet. This is the digital equivalent of Fort Knox: consensus must be achieved, or bitcoin has no value.
"""

This makes it sound like if, for example, Debian were to link bitcoind to the
system leveldb, and then upgrade the system leveldb to fix a bug that affects
bitcoind, that this would spell the end of Bitcoin.

I hope that's not true!

I'd like to try to be more specific about two things:

1. What is the behavior that a dependency or a patch could cause that would be
problematic? I liked what Luke-Jr said earlier in this thread -- that in some
cases a bitcoin node (i.e. a bitcoind process) needs certain bugs or
limitations in order to maintain consensus with other bitcoin nodes. Maybe you
could use a statement like that, without attempting to explain in *what* cases
that applies.

2. What is the consequence if this goes wrong? This is something I don't
understand as well. I think the answer is:

   2.a. All bitcoin nodes which encounter one of these cases and are
        differently-buggy than the upstream bitcoind form their own consensus,
        causing a blockchain fork.

   2.b. There is a risk of double-spending attacks.

   2.c. The process for healing a blockchain fork is not very smooth or
        well-understood.

Regards,

Zooko

Re: [Bitcoin-development] Linux packaging letter

[Gregory Maxwell]

On Wed, Jul 24, 2013 at 8:32 AM, zooko <zooko@zooko.com> wrote:
> This makes it sound like if, for example, Debian were to link bitcoind to the
> system leveldb, and then upgrade the system leveldb to fix a bug that affects
> bitcoind, that this would spell the end of Bitcoin.

Maybe!  A widespread consensus failure causes people to lose money
even absent malice. How much depends on a bunch of details, including
the luck of attackers.

The total ramifications are as much social as they are technical so
it's hard to reason over the outcomes beyond "at a minimum, it's not
good".

A really bad splitting event could results in large amounts of Bitcoin
being stolen through reversals. Obviously the system itself would keep
on ticking once the issue was resolved... but if millions of dollars
at recent prices in coins were stolen,  would people want to keep
using it?

The most dire outcomes are (very?) unlikely, but they're not necessary
to recognize that risk mitigation is important.

It's good to be careful here just to avoid the bad outcomes we are
sure will happen (because we've experienced them before):   Hundreds
of dollars worth of coin income 'lost' per minute to miners on the
losing side of a 50/50 fork, hours long disruption of the lives of
dozens of people in the Bitcoin technical ecosystem (many of whom are
volunteer OSS developers), hours of disruption (no payments processed)
to Bitcoin users and businesses.  These are the best case outcomes in
a substantial non-transient hard forking event.

I think one of the challenges in talking about this stuff is correctly
framing these risks.  Bitcoin is a novel technology that lacks a lot
of the recourse that other systems have— No Bitcoin central bank to
create a bit of inflation to paper over a glitch,  eliminating those
kinds of centralized "fixes" is much of the point, after all—  so with
the idea of starry eyed people taking out second mortgages on their
kids kidneys to buy up coin clearly in my mind I do think it's
important to be clear about the full range of risk:  It's _possible_
that due to some amazing sequences of technical screwups that by next
week most everyone could consider Bitcoin worthless. I think it's
important to be frank about those risks.  ... but it's also not good
to be chicken little, calling doom on anyone who wants to change the
color of the GUI. :P   Navigating it is hard, and generally I'd prefer
that if there is any misunderstanding people overestimate the risks a
little— so long as things stay in the realm of the possible— rather
than underestimate them.

Re: [Bitcoin-development] Linux packaging letter

[zooko]

On Wed, Jul 24, 2013 at 10:28:16AM +0200, Mike Hearn wrote:
> Yeah, if anyone wants to make the letter more digestable please do propose
> an alternative, although by this point it's probably not worth it as people
> have already signed.

Okay, here's my attempt:

https://docs.google.com/document/d/1m3wyBIjqwPQ3wxVT7P_wJtdWt9a9RXvt9NV7rggLAOs/edit#

Please feel free to use any or all of it as you see fit.

> FWIW, Gregory is right that my original draft was much more brusque. The
> pain in the packaging relationship travels both ways. I have in the past
> wasted a lot of time due to bogus packaging applied by non-expert packagers
> that broke things. In fact the project I was a part of adopted a policy of
> automatically closing bug reports from people who were using distributor
> packages (any distro) because the quality was so inconsistent and so many
> subtle bugs were introduced.
> 
> If packagers hear upstreams cry about packaging a lot, I think you should
> keep an open mind that some of them probably know what they're talking
> about. We really shouldn't have to beg and cajole here. Saying "we have our
> reasons and we want you to stop" should be enough.

Yes, I know what you mean.

Regards,

Zooko

Re: [Bitcoin-development] Linux packaging letter

[Greg Troxel]

Mike Hearn <mike@plan99.net> writes:

> If packagers hear upstreams cry about packaging a lot, I think you should
> keep an open mind that some of them probably know what they're talking
> about. We really shouldn't have to beg and cajole here. Saying "we have our
> reasons and we want you to stop" should be enough.

Asserting without explaining isn't going to work; lots of people think
their code is more special than it is, and most of these claims are
unwarranted.  But, there is a good explanation for the bitcoin case.

Re: [Bitcoin-development] Linux packaging letter

[Greg Troxel]

Gregory Maxwell <gmaxwell@gmail.com> writes:

> It's "portable" to anything that can run the relevant VMs.  Uh
> provided you don't mind cross compiling everything from an unbuntu VM.
>  It certainly would be nice if the trusted-computing-base for gitian
> were a bit smaller, thats an area for long term improvement for sure.

Thanks - I'll look forward to this being portable someday.  Right now it
sounds similar to "a windows binary but you can use wine" with
substitution of variables :-) People may want to look at the NetBSD
build system, which I think achieves bit-identical builds from different
hosts (but I haven't really checked), by having the toolchain be part of
the source and building cross-compilers from host to target and then
using those to build the system.

> Say Bitcoin used a backing database which had an unknown a bug where
> any item with a key that begins with 0xDEADBEEF returns not found when
> queried, even if its in the DB. Once discovered, any database library
> would want to fix that quickly and they'd fix it in a point release
> without reservation. They might not even release note that particular
> fix it if went along with some others, it could even be fixed
> accidentally.
>
> Now say that we have a state where half the Bitcoin network is running
> the old buggy version, and half is running the fixed version.  Someone
> creates a transaction with ID 0xDEADBEEF...  and then subsequently
> spends the output of that transaction. This could be by pure chance or
> it could be a malicious act.
>
> To half the network that spending transaction looks like someone
> spending coin from nowhere, a violation of the rules.  The consensus
> would then fork, effectively partitioning the network.  On each fork
> any coin could be spent twice, and the fork will only be resolvable by
> one side or the other abandoning their state (generally the more
> permissive side would need to be abandoned because the permissive one
> is tolerant of the restrictive one's behavior) by manually downgrading
> or patching software.  As a result of this parties who believed some
> of their transactions were safely settled would find them reversed by
> people who exploited the inconsistent consensus.

Thanks for the explanation - that indeed makes sense.

>> multiple packages is difficult, and runs into A wants only n of C, while
>> B wants only m.
>
> My understanding is that gentoo is actually able to handle this (and
> does, for Bitcoin)— and really I presume just about everything else
> could with enough effort. I certainly wouldn't ask anyone else to do
> that.  If you're really getting into the rathole of building separate
> libraries just for Bitcoin the value of packaging it goes away.

Well, if you insist on not having updates and bugfixes, then either it's
the included version or there's a special package just for you.
Typically packaging systems don't like included versions because often a
package will have a security bug fixed long before there are updates of
packages that bundle that fixed version.    But given bitcoin's special
needs, that means you have to stay on top of these dependent included
packages and re-release if there are security fixes (that don't break
consensus).

> Running a complete set of tests is a start— though the unit tests are
> not and cannot be adequate. There is a full systems testing harnesses
> which should be used on new platforms.  Even that though isn't really
> adequate, as it is currently infeasible to even achieve complete test
> coverage in things like cryptographic libraries and database
> environments.

It would be nice if the regression tests were installed and it were
normal culturallly for end-users to run them.


Thanks again for the explanation; I understand where you are coming from
now.