5dd0a3793b
set_subkey corrupts record_data_size in a Record struct by calling set_record_data_size with a value that depends only on the length of the new subkey value. This leads to various undesirable outcomes, such as: applications can write more than MAX_RECORD_DATA_SIZE without encountering the intended "veilid.error.VeilidAPIErrorGeneric: label='Generic' message='dht record too large'" error message, and "panicked at 'attempt to subtract with overflow'" (i.e., an attempt to set a negative value of a size) if a subkey's new length is less than a subkey's old length. Typically, record_data_size in a Record struct will be incorrect if a value was set for more than one subkey. Some users might want to start over with a table_store that doesn't have any incorrect record_data_size values. The issue begins here:6f71c6a00a/veilid-core/src/storage_manager/record_store.rs (L583-L586)and is triggered here:6f71c6a00a/veilid-core/src/storage_manager/record_store.rs (L613-L615)It should be clear that new_record_data_size is only related to the subkey that is currently being set. The amount of data in the record, before set_subkey is called, is ignored. It appears that new_total_size, not new_record_data_size, was intended to be used for set_record_data_size, and this change succeeds for me in limited testing but I don't have a comprehensive test suite. One way to reproduce is by running the code from https://gitlab.com/vatueil/veilid-file on a greater than 1 MB file while watching variable values within veilid-core/src/storage_manager/record_store.rs. For example: "poetry run file put /usr/bin/tcpdump" (1.3 MB on Ubuntu 23.04). With the original Veilid code, each of the dozens of subkey writes is checking whether a roughly 32K value is greater than 1048576, it never is, and thus there is never a "dht record too large" error. With the patch in this MR, each of the dozens of subkey writes is checking whether an ever-increasing value is greater than 1048576, it eventually is, and the "dht record too large" error is printed. With the patch, one can work with smaller files, e.g., do "poetry run file put /usr/bin/ssh" (0.8 MB) followed by "poetry run file get VLD0:<_insert_key_here_> ssh-copy" and the retrieved file ssh-copy is identical to /usr/bin/ssh. The more detailed behavior is that the modified code has record.total_size of 350 on the first iteration, then 33596, 66842, 100088, etc. The original code also has record.total_size of 350 on the first iteration, but then stays at 33246 forever (33246 is the user-supplied subkey size of 32768, plus 128, plus the minimum record size of 350),
Welcome to Veilid
From Orbit
The first matter to address is the question "What is Veilid?" The highest-level description is that Veilid is a peer-to-peer network for easily sharing various kinds of data.
Veilid is designed with a social dimension in mind, so that each user can have their personal content stored on the network, but also can share that content with other people of their choosing, or with the entire world if they want.
The primary purpose of the Veilid network is to provide the infrastructure for a specific kind of shared data: social media in various forms. That includes light-weight content such as Twitter's tweets or Mastodon's toots, medium-weight content like images and songs, and heavy-weight content like videos. Meta-content such as personal feeds, replies, private messages, and so forth are also intended to run atop Veilid.
Run a Node
The easiest way to help grow the Veilid network is to run your own node. Every user of Veilid is a node, but some nodes help the network more than others. These network support nodes are heavier than the node a user would establish on their phone in the form of a chat or social media application. A cloud based virtual private server (VPS), such as Digital Ocean Droplets or AWS EC2, with high bandwidth, processing resources, and up time availability is crucial for building the fast, secure, and private routing that Veilid is built to provide.
To run such a node, establish a Debian or Fedora based VPS and install the veilid-server service. To make this process simple we are hosting package manager repositories for .deb and .rpm packages. See the installing guide for more information.
Development
If you're inclined to get involved in code and non-code development, please check out the contributing guide. We're striving for this project to be developed in the open and by people for people. Specific areas in which we are looking for help include:
- Rust
- Flutter/Dart
- Python
- Gitlab DevOps and CI/CD
- Documentation
- Security reviews
- Linux packaging