[bitcoin-dev] Storing the Merkle Tree in a compact way

[bitcoin-dev] Storing the Merkle Tree in a compact way

[shymaa arafat]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

Allow me to introduce this simple idea that could be useful ...

-The Intuition was some discussion on Utreexo project about storage saving
and some traversing issues in handling the UTXOS Merkle Tree/ forest; that
is  N internal nodes need to be stored along with 2N pointers (left&right),
+ maybe 1 more pointer in the leaves special nodes to handle different
traversing options (insert, delete, & differently proof fetch that traverse
aunt or niece node according to your implementation
https://github.com/mit-dci/utreexo/discussions/316)
.
Then, I thought of a simple idea that gets rid of all the pointers;
specially appealing when we have all trees are full (complete) in the
forest, but can work for any Merkle Tree:

- 2D array with variable row size; R[j] is of length (N/2^j)
-For example when N=8 nodes
R[0]=0,1,2,...,7
R[1]=8,9,10,11
R[2]=12,13
R[3]=14
.
-We can see that total storage is just 2N-1 nodes,
no need for pointers, and traversing could be neat in any direction with
the right formula:

-Pseudo code to fetch proof[i] ...

//direction to know + or -
If ((i mod 2)==0) drct=1;
            else drct=-1;
// first, the sibling node
proof[i]=R[0,i+drct]

//add the rest thru loop
For(j=1; j≤logN; j++)
 { index= i/(2^j)+drct;
    proof[i]=Add(R[j,index]);
 }

-In fact it's just the simple primitive approach of transforming a
recursion to an iteration, and even if Utreexo team solved their problem
differently I thought it is worth telling as it can work for any Merkle Tree
.
Thanks for your time,
Shymaa M Arafat

[-- Attachment #2: Type: text/html, Size: 2424 bytes --]

Re: [bitcoin-dev] [Lightning-dev] Storing the Merkle Tree in a compact way

[shymaa arafat]

[-- Attachment #1.1: Type: text/plain, Size: 2953 bytes --]

It could be viewed as the simple complete tree to 1D array  with no
pointers described in lecture 8 here
https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-006-introduction-to-algorithms-spring-2020/lecture-notes/index.htm
starting from min 15 in this video
https://youtu.be/Xnpo1atN-Iw

Since all trees in Utreexo forest are full binary trees, this is perfect to
use, and we can save *76*10⁶*2*size of pointer(probably4bytes)*
*~600MB *with almost no effort.

However, I suggest to put it in a 2D array to make it more easy to handle
(the indexing math) as we, different than the lecture, traverse in many
ways ( normally to delete or insert, and the parent siblings for the proofs)

I wrote more details here
https://bitcointalk.org/index.php?topic=5360009.0

On Thu, Sep 16, 2021, 14:37 Vincent <vincent.palazzo@protonmail.com> wrote:

> Hi.
>
> Thanks for the reference, but I missed where you want save space with this
> compression on the Merkle Tree.
>
> Regards.
>
> Vincent.
> vincenzo.palazzo@protonmail.com
> https://github.com/vincenzopalazzo
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, September 16th, 2021 at 5:15 AM, shymaa arafat <
> shymaa.arafat@gmail.com> wrote:
>
> Allow me to introduce this simple idea that could be useful ...
>
> -The Intuition was some discussion on Utreexo project about storage saving
> and some traversing issues in handling the UTXOS Merkle Tree/ forest; that
> is  N internal nodes need to be stored along with 2N pointers (left&right),
> + maybe 1 more pointer in the leaves special nodes to handle different
> traversing options (insert, delete, & differently proof fetch that traverse
> aunt or niece node according to your implementation
> https://github.com/mit-dci/utreexo/discussions/316)
> .
> Then, I thought of a simple idea that gets rid of all the pointers;
> specially appealing when we have all trees are full (complete) in the
> forest, but can work for any Merkle Tree:
>
> - 2D array with variable row size; R[j] is of length (N/2^j)
> -For example when N=8 nodes
> R[0]=0,1,2,...,7
> R[1]=8,9,10,11
> R[2]=12,13
> R[3]=14
> .
> -We can see that total storage is just 2N-1 nodes,
> no need for pointers, and traversing could be neat in any direction with
> the right formula:
>
> -Pseudo code to fetch proof[i] ...
>
> //direction to know + or -
> If ((i mod 2)==0) drct=1;
>             else drct=-1;
> // first, the sibling node
> proof[i]=R[0,i+drct]
>
> //add the rest thru loop
> For(j=1; j≤logN; j++)
>  { index= i/(2^j)+drct;
>     proof[i]=Add(R[j,index]);
>  }
>
> -In fact it's just the simple primitive approach of transforming a
> recursion to an iteration, and even if Utreexo team solved their problem
> differently I thought it is worth telling as it can work for any Merkle Tree
> .
> Thanks for your time,
> Shymaa M Arafat
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 5377 bytes --]

[-- Attachment #2: IMG_20210913_194256.jpg --]
[-- Type: image/jpeg, Size: 279763 bytes --]

[-- Attachment #3: IMG_20210913_193322.jpg --]
[-- Type: image/jpeg, Size: 164592 bytes --]