From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-3.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1X74U0-0006TO-HX for bitcoin-development@lists.sourceforge.net; Tue, 15 Jul 2014 15:17:32 +0000 Received-SPF: pass (sog-mx-4.v43.ch3.sourceforge.com: domain of bitpay.com designates 74.125.82.174 as permitted sender) client-ip=74.125.82.174; envelope-from=jgarzik@bitpay.com; helo=mail-we0-f174.google.com; Received: from mail-we0-f174.google.com ([74.125.82.174]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:RC4-SHA:128) (Exim 4.76) id 1X74Tz-0002CK-9d for bitcoin-development@lists.sourceforge.net; Tue, 15 Jul 2014 15:17:32 +0000 Received: by mail-we0-f174.google.com with SMTP id x48so3956711wes.19 for ; Tue, 15 Jul 2014 08:17:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=Blm4Sbgd37HGHr+DD7sRVxSx995FEqZBJvIsNEDy+Tc=; b=I3ZgORA3li97gaxs+rdAxmxxJWIWGikzPTOvBNqcPcqkh/TVlme3BtO65LlFtYyZQo /FJJXdRPHr1Cmm5c436OWB8EA+KTwfr6WFD0fyPe7uIUrrIJMSdEC1rJd40lNp/YPtK3 37yTYkCs3kwplPZYKtno4RgbkPF9PQUiwVoB5dGkcax34FN5641+e68ySS4vcZP2FCpo qFY9qQVtDxYjbhicl5eljgF7EWO1pBdxF/qh/nA5fmp8rEbk2TimTmEB+YljJjKKxHrK B7gwq3dnRqaR3RLkvoOjWxFOsc7iET1TKq2MkKg9035JhsNlGaLDJBBF+Qu1okCW/lF7 ruAA== X-Gm-Message-State: ALoCoQlKpmytZNP8mb/eT9eSFfVPN2d/RNaKch4/OD3bya/4IGGbg103sLERAQTercv5ik5QNN/+ X-Received: by 10.194.186.178 with SMTP id fl18mr27574870wjc.83.1405437444960; Tue, 15 Jul 2014 08:17:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.5.67 with HTTP; Tue, 15 Jul 2014 08:17:04 -0700 (PDT) In-Reply-To: References: From: Jeff Garzik Date: Tue, 15 Jul 2014 11:17:04 -0400 Message-ID: To: Eric Winer Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.6 (-) X-Spam-Report: Spam Filtering performed by mx.sourceforge.net. See http://spamassassin.org/tag/ for more details. -1.5 SPF_CHECK_PASS SPF reports sender host as permitted sender for sender-domain -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-Headers-End: 1X74Tz-0002CK-9d Cc: Bitcoin Dev , Andreas Schildbach Subject: Re: [Bitcoin-development] BIP 38 NFC normalisation issue X-BeenThere: bitcoin-development@lists.sourceforge.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jul 2014 15:17:32 -0000 Unicode guarantees that null-terminated strings still work. U+0000 terminates a unicode (or C) string. strlen() gets the string byte count. mbstowcs() gets the character count. Whitespace can be problematic, but should be allowed. Control characters should be filtered. Emoticons probably cannot be filtered without substandard approaches such as character blacklists, a road you do not want to travel. (all this is simply standard practice) On Tue, Jul 15, 2014 at 9:07 AM, Eric Winer wrote: > I don't know for sure if the test vector is correct NFC form. But for wh= at > it's worth, the Pile of Poo character is pretty easily accessible on the > iPhone and Android keyboards, and in this string it's already in NFC form > (f09f92a9 in the test result). I've certainly seen it in usernames aroun= d > the internet, and wouldn't be surprised to see it in passphrases entered = on > smartphones, especially if the author of a BIP38-compatible app includes = a > (possibly ill-advised) suggestion to have your passphrase "include specia= l > characters". > > I haven't seen the NULL character on any smartphone keyboards, though - I > assume the iOS and Android developers had the foresight to know how much > havoc that would wreak on systems assuming null-terminated strings. It > seems unlikely that NULL would be in a real-world passphrase entered by a > sane user. > > > On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn wrote: >> >> [+cc aaron] >> >> We recently added an implementation of BIP 38 (password protected privat= e >> keys) to bitcoinj. It came to my attention that the third test vector ma= y be >> broken. It gives a hex version of what the NFC normalised version of the >> input string should be, but this does not match the results of the Java >> unicode normaliser, and in fact I can't even get Python to print the nam= es >> of the characters past the embedded null. I'm curious where this normali= sed >> version came from. >> >> Given that "pile of poo" is not a character I think any sane user would >> put into a passphrase, I question the value of this test vector. NFC for= m is >> intended to collapse things like umlaut control characters onto their pr= ior >> code point, but here we're feeding the algorithm what is basically garba= ge >> so I'm not totally surprised that different implementations appear to >> disagree on the outcome. >> >> Proposed action: we remove this test vector as it does not represent any >> real world usage of the spec, or if we desperately need to verify NFC >> normalisation I suggest using a different, more realistic test string, l= ike >> Z=C3=BCrich, or something written in Thai. >> >> >> >> Test 3: >> >> Passphrase =CF=92=CC=81=E2=90=80=F0=90=90=80=F0=9F=92=A9 (\u03D2\u0301\u= 0000\U00010400\U0001F4A9; GREEK UPSILON >> WITH HOOK, COMBINING ACUTE ACCENT, NULL, DESERET CAPITAL LETTER LONG I, = PILE >> OF POO) >> Encrypted key: 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQ= n >> Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF >> Unencrypted private key (WIF): >> 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4 >> Note: The non-standard UTF-8 characters in this passphrase should be NFC >> normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before >> further processing >> >> >> >> >> >> ------------------------------------------------------------------------= ------ >> Want fast and easy access to all the code in your enterprise? Index and >> search up to 200,000 lines of code with a free copy of Black Duck >> Code Sight - the same software that powers the world's largest code >> search on Ohloh, the Black Duck Open Hub! Try it now. >> http://p.sf.net/sfu/bds >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> > > > -------------------------------------------------------------------------= ----- > Want fast and easy access to all the code in your enterprise? Index and > search up to 200,000 lines of code with a free copy of Black Duck > Code Sight - the same software that powers the world's largest code > search on Ohloh, the Black Duck Open Hub! Try it now. > http://p.sf.net/sfu/bds > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > --=20 Jeff Garzik Bitcoin core developer and open source evangelist BitPay, Inc. https://bitpay.com/