Adds ground level view

docs/guide/guide.css

@@ -21,3 +21,8 @@ body {
   font-weight: 600;
   margin-bottom: 5px;
 }
+
+code {
+  font-family: monospace;
+  font-weight: bold;
+}

docs/guide/guide.html

@@ -8,7 +8,7 @@
     <meta name="description" content="a guide to the architecture of Veilid">
     <meta name="author" content="beka">
 
-    <!-- <link rel="stylesheet" href="guide.css"> -->
+    <link rel="stylesheet" href="guide.css">
 
   </head>
 
@@ -81,7 +81,7 @@
     <h3 id="peer-network-for-data-storage">Peer Network for Data Storage</h3>
 
     <p>
-      The bottom-most level of Veilid is a network of peers communicating to one another over the internet. Peers send each other messages (remote procedure calls) about the data being stored on the network, and also messages about the network itself. For instance, one peer might ask another for some file, or it might ask for info about what other nodes exist in the network.
+      The bottom-most level of Veilid is a network of peers communicating to one another over the internet. Peers send each other messages (remote procedure calls) about the data being stored on the network, and also messages about the network itself. For instance, one peer might ask another for some file, or it might ask for info about what other peers exist in the network.
     </p>
 
     <p>
@@ -105,7 +105,7 @@
     </p>
 
     <p>
-      KV store data is also versioned, so that updates to it can be made. Boone's bio, for instance, would not be fixed in time, but rather is likely to vary over time as he changes jobs, picks up new hobbies, etc. Versioning, together with arbitrary user-chosen identifiers instead of content hashes, means that we can talk about "Boone's Bio" as an abstract thing, and subscribe to updates to it.
+      KV store data is also stateful, so that updates to it can be made. Boone's bio, for instance, would not be fixed in time, but rather is likely to vary over time as he changes jobs, picks up new hobbies, etc. Statefulness, together with arbitrary user-chosen identifiers instead of content hashes, means that we can talk about "Boone's Bio" as an abstract thing, and subscribe to updates to it.
     </p>
 
     <h3 id="structuring-data">Structuring Data</h3>
@@ -129,39 +129,109 @@
     </p>
 
     <p>
-      User identity is a slightly richer notion. Users, that is to say, *people*, will want to access the Veilid network in a way that has a consistent identity across devices and apps. But since Veilid doesn't have servers in any traditional sense, we can't have a normal notion of "account". Doing so would also introduce points of centralization, which federated systems have shown to be a source of trouble. Many Mastodon users have found themselves in a tricky situation when their instance sysadmins burned out and suddenly shut down the instance without enough warning.
+      User identity is a slightly richer notion. Users, that is to say, <em>people</em>, will want to access the Veilid network in a way that has a consistent identity across devices and apps. But since Veilid doesn't have servers in any traditional sense, we can't have a normal notion of "account". Doing so would also introduce points of centralization, which federated systems have shown to be a source of trouble. Many Mastodon users have found themselves in a tricky situation when their instance sysadmins burned out and suddenly shut down the instance without enough warning.
     </p>
 
     <p>
       To avoid this re-centralization of identity, we use cryptographic identity for users as well. The user's key pair is used to sign and encrypt their content as needed for publication to the data store. A user is said to be "logged in" to a client app whenever that app has a copy of their private key. When logged in a client app act like any other of the user's client apps, able to decrypt and encrypt content, sign messages, and so forth. Keys can be added to new apps to sign in on them, allowing the user to have any number of clients they want, on any number of devices they want.
     </p>
 
-    <h3 id="peer-privacy">Peer Privacy</h3>
-    
-    <p>
-      In order to ensure that peers can participate in Veilid with some amount of privacy, we need to address the fact that being connected to Veilid entails communicating with other peers, and therefore sharing IP addresses.
-    </p>
-    
-    <p>
-      The approach that Veilid takes to privacy is two sided: privacy of the sender of a message, and privacy of the receiver of a message. Either or both sides can want privacy or opt out of privacy. To achieve sender privacy, we use something called a Safety Route: a sequence of any number of other peers, chosen by the sender, who will relay messages. The sequence of addresses is put into a nesting doll of encryption, so that each hop can see the previous and next hops, while no hop can see the whole route. This is similar to a Tor route, except only the addresses are hidden from view. Additionally, the route can be chosen at random for each send.
-    </p>
-    
-    <p>
-      Receiver privacy is similar, in that we have a nesting doll of encrypted peer address, except because it's for incoming messages, the various addresses have to be shared ahead of time. We call such things Private Routes, and they are published to the key-value store as part of a peer's public data. For full privacy on both ends, a Private Route will be used as the final destination of a Safety Route, so that a total of four intermediate hops are used to send a message so that neither the sender nor receiver knows the IP address of the other.
-    </p>
-    
     <hr/>
 
     <h2 id="on-the-ground">On The Ground</h2>
 
     <p>
-      The bird's eye view of things makes it possible to hold it all in mind at once, but leaves out lots of information about implementation choice. It's now time to come down to earth and get our hands dirty.
+      The bird's eye view of things makes it possible to hold it all in mind at once, but leaves out lots of information about implementation choice. It's now time to come down to earth and get our hands dirty. In principl, this should be enough information to implement a system very much like Veilid, with the exception perhaps of the specific details of the APIs and data formats. This section won't have code, it's not documentation of the codebase, but rather is intended to form the meat of a whitepaper.
+    </p>
+
+    <h3>Peer Network, Revisited</h3>
+
+    <p>
+      First, let's look at the peer network, since it's structure forms the basis for the remainder of the data storage approach. Veilid's peer network is similar to other peer-to-peer systems in that it's overlaid on top of other protocols. Veilid tries to be somewhat protocol-agnostic, however, and currently is designed to use TCP, UDP, WebSockets, and WebRTC, as well as various methods of traversing NATs so that Veilid peers can be smartphones, personal computers on hostile ISPs, etc. To facilitate this, peers are identified not by some network identity like an IP address, but instead by peer-chosen cryptographic key-pairs. Each peer also advertises a variety of options for how to communicate with it, called dial info, and when one peer wants to talk to another, it gets the dial info for that peer from the network and then uses it to communicate.
     </p>
 
     <p>
-      TODO
+      When a peer first connects to Veilid, it does so by contacting bootstrap peers, which have simple IP address dial info that is guaranteed to be stable by the maintainers of the network. These bootstrap peers are the first entries in the peer's routing table -- an address book of sorts, which it uses to figure out how to talk to a peer. The routing table consists of a mapping from peer public keys to prioritized choices for dial info. To populate the routing table, the peer asks other peers what its neighbors are in the network. The notion of neighbor here is defined by a similarity metric on peer IDs, in particular an XOR metric like many DHTs use. Over the course of interacting with the network, the peer will keep dial info up to date when it detects changes. It may also add dial info for peers it discovers along the way, depending on the peer ID.
     </p>
 
+    <p>
+      To talk to a specific peer, it's dial info is looked up in the routing table. If there is dial info present, then the options are attempted in order of the priority specified in the routing table. Otherwise, the peer has to request the dial info from the network, so it looks through its routing table to find the peer who's ID is nearest the target peer according to the XOR metric, and sends it an RPC call with a procedure named <code>find_node</code>. Given any particular peer ID, the receiver of a <code>find_node</code> call returns dial info for the peers in its routing table that are nearest the given ID. This gets the peer closer to its destination, at least in the direction of the other peer it asked. If the desired peer's information was in the result of the call, then it's done, otherwise it calls <code>find_node</code> again to get closer. It iterates in this way, possibly trying alternate peers, as necessary, in a nearest-first fashion until it either finds the desire'd peer's dial info, has exhausted the entire network, or gives up.
+    </p>
+
+    <h3 id="user-privacy">User Privacy</h3>
+
+    <p>
+      In order to ensure that users can participate in Veilid with some amount of privacy, we need to address the fact that being connected to Veilid entails communicating with other peers, and therefore sharing IP addresses.
+    </p>
+
+    <p>
+      The approach that Veilid takes to privacy is two sided: privacy of the sender of a message, and privacy of the receiver of a message. Either or both sides can want privacy or opt out of privacy. To achieve sender privacy, we use something called a Safety Route: a sequence of any number of peers, chosen by the sender, who will relay messages. The sequence of addresses is put into a nesting doll of encryption, so that each hop can see the previous and next hops, while no hop can see the whole route. This is similar to a Tor route, except only the addresses are hidden from view. Additionally, the route can be chosen at random for each message being sent.
+    </p>
+
+    <p>
+      Receiver privacy is similar, in that we have a nesting doll of encrypted peer addresses, except because it's for incoming messages, the various addresses have to be shared ahead of time. We call such things Private Routes, and they are published to the key-value store as part of a user's public data. For full privacy on both ends, a Private Route will be used as the final destination of a Safety Route, and the total route is the composition of the two, so that neither the sender nor receiver knows the IP address of the other.
+    </p>
+
+    <p>
+      Note that the routes are <em>user</em> oriented. They should be understood as a way to talk to a particular <em>user's</em> peer, wherever that may be. Each peer of course has to know about the actual IP addresses of the peers, otherwise it couldn't communicate, but safety and private routes make it hard to associate the <em>user's</em> identity with their <em>peer's</em> identity. You know that the user is somewhere on the network, but you don't know which IP address is their's, even if you do in fact have their peer's dial info stored in the routing table.
+    </p>
+
+    <h3>Block Store</h3>
+
+    <p>
+      As mentioned in the Bird's Eye View, the block store is intended to store content-addressed blocks of data. Like many other peer-to-peer systems for storing data, Veilid uses a distributed hash table as the core of the block store. The block store DHT has as keys BLAKE3 hashes of block content. For each key the DHT associates a list of peer IDs for peers that have declared to the network that they can supply the block.
+    </p>
+
+    <p>
+      If a peer wishes to supply the block, it makes a <code>supply_block</code> RPC call to the network with the id of the block. The receiver of the call can then store the information that the peer supplies the designated block if it wants, and also can return other peers nearer to the block's ID that should also store the information. Peers determine whether or not to store this information based on how close it is to the block's ID. It may also choose to cache the block, possibly also declaring itself to be a supplier as well.
+    </p>
+
+    <p>
+      Supplier records are potentially brittle because peers leave the network, making their information unavailable. Because of this, any peer that wishes to supply a block will periodically send <code>supply_block</code> messages to refresh the records. Peers that are caching blocks determine when to stop caching based on how popular a block is, how much space or bandwidth it can spare, etc.
+    </p>
+
+    <p>
+      To retrieve a block that has been stored in the blockstore, a peer makes a <code>find_block</code> RPC. The receiver will then either return the block, or possibly return a list of suppliers for the block that it knows about, or return a list of peers that are closer to the block.
+    </p>
+
+    <p>
+      Unlike BitTorrent, blocks are not inherently part of a larger file. A block can be just a single file, and often that will be the case for small files. Large files can be broken up into smaller blocks, however, and then an additional block with a list of those component blocks can be stored in the block store. Veilid itself, however, would treat this like any other block, and there are no built-in mechanisms for determining which blocks to download first, which to share first, etc. like there are in BitTorrent. These features would be dependent on the peer software's implementation and could vary. Different clients will also be able to decide how they want to download such "compound" blocks -- automatically, via a prompt to the user, or something else.
+    </p>
+
+    <p>
+      The mechanism of having blocks that refer to other blocks also enables IPFS-style DAGs of hierarchical data as one mode of use of the block store, allowing entire directory structures to be stored, not just files. However, as with sub-file blocks, this is not a built-in part of Veilid but rather a mode of use, and how they're downloaded and presented to the user is up to the client program.
+    </p>
+
+    <h3>Key-Value Store</h3>
+
+    <p>
+      The key-value store is a DHT similar to the block store. However, rather than using content hashes as keys, the KV store uses user IDs as keys (note: <em>not</em> peer IDs). At a given key, the KV store has a hierarchical key-value map that associates in-principle arbitrary strings with values, which themselves can be numbers, strings, datetimes, or other key-value maps. The specific value stored in at a user's ID is versioned, so that particular schemas of subkeys and values can be defined and handled appropriately by different versions of clients.
+    </p>
+
+    <p>
+      When a user wishes to store data under their key, they send a <code>set_value</code> RPC to the peer's whose IDs are closest by the XOR metric to their own user ID. The value provided to the RPC is a signed value, so that the network can ensure only the designated user is storing data at their key. Those peers may return other peer IDs, and so on, similar to how the block store handles <code>supply_block</code> calls. Eventually, some peers will store the data. The user's peer should periodically refresh the stored data, to ensure that it persists. It's also good practice for the user's own peer to cache the data, so that client programs can use the user's own peer as a canonical source of the most-up-to-date value.
+    </p>
+
+    <p>
+      Retrieval is similar to block store retrieval. The desired key is provided to a <code>get_value</code> call, which may return th value, or a list of other peers that are closer to the key. Eventually the signed data is returned, and the recipient can verify that it does indeed belong to the specified user by checking the signature.
+    </p>
+
+    <p>
+      When storing and retrieving, the key provided to the RPCs is not required to be only the user's ID. It can include a list of strings which act as a path into the data stored at the user's key, targetting it specifically for update or retrieval. This lets the network minimize data transfer, because only the relevant information has to move around.
+    </p>
+
+    <p>
+      The specific content of the user's keys is determined partially by the protocol and partially by the client software. Early versions of the protocol use a DHT schema version that defines a fairly simple social network oriented schema. Later versions will enable a more generic schema so that client plugins can store and display richer information.
+    </p>
+
+    <p>
+      TODO How to avoid replay updates?? maybe via a sequence number in the signed patch?
+    </p>
+
+    <h2>Appendix 1: Dial Info</h2>
+
+    <h2>Appendix 2: RPC Listing</h2
+
     </div>
   </body>
 </html>