krenoten

7 ms·

1.
by krenoten 6y ago
In practice this is the source of a tremendous number of bugs when async tasks don't actually clean up shared state in their Drop impls.
2.
by krenoten 6y ago
I haven't mentioned async Rust on HN since 2015.
3.
by krenoten 7y ago
The best solution is to use a 0-copy format like flatbuffers, google's zerocopy library, or serde's borrow attributes https://serde.rs/lifetimes.html#borrowing-data-in-a-derived-... You're totally right that
4.
by krenoten 7y ago
only narcs use windows for database workloads :P
5.
by krenoten 7y ago
yeah, le-be conversions are not generally measurable above noise compared to other database-related work. sled stores arbitrary bytes. endianness is the concern of the person who wants to store higher-level types than bytes, like integers.
6.
by krenoten 7y ago
On sled 0.31 the memory growth does not run away as with the earlier version. Note that some in-memory metadata tracking does accrue as nodes split and sled tracks where to find them in the future when the actual data is paged in. But this
7.
by krenoten 7y ago
Just use a hashmap. Hashmaps in some cases can be like 8 orders of magnitude faster than Redis.
8.
by krenoten 7y ago
I make a lot of money fixing problems in storage layers that these kinds of ideas create. Thanks :]
9.
by krenoten 8y ago
There is zero reason behind this assumption.
10.
by krenoten 8y ago
It means pay more attention to reliability than pop infrastructure and internet companies (who can offset poor reliability with human attention or intentionally deprioritize it to sell more support contacts) tend to put into these things. S
11.
by krenoten 8y ago
This was my interpretation as well. I'm going to compare a disk-backed bwtree with a disk-backed ART, both backed by the same pagecache, and maybe end up with an ART that scatters partial pages on disk, bwtree style. But I need to meas
12.
by krenoten 8y ago
Users can rely on sequential recovery. At some point I'll probably write a partial recovery tool that gives you all versions of all keys that are at all present anywhere in the readable file though, which won't be much work. Typic
13.
by krenoten 8y ago
Indeed. This is why I aggressively checksum everything and pay particular attention to throwing away all data that was written after any detected corruption during recovery. This is easier with the log-only architecture. It's also tota
14.
by krenoten 8y ago
Yeah, I'm curious about using sled as a more ssd friendly storage engine for mentat. I'm just starting to experiment with datalog implementations, but I think by having harmony between the storage engine, query language, and hardw
15.
by krenoten 8y ago
It might not. But the critiques of bw trees in terms of performance that I've seen have not had compelling data in terms of things that matter outside of academia or benchmarking shootouts, like write or space amplification. The bw tre
16.
by krenoten 8y ago
This is honestly a use case I'm experimenting with using a mix of CRDTs and OT. Our systems are becoming more and more location agnostic and I don't feel that our current data infrastructure is adequate to serve the workloads we&#
17.
by krenoten 8y ago
Use it for large scale HTAP! It's great for its flexible use cases at high scales :)
18.
by krenoten 8y ago
I am a total devotee to their approach to building simulable systems, although I seek to push it even farther and integrate lineage driven fault injection from an early stage. I see a lot of cool things in what they have done, technically.
19.
by krenoten 8y ago
It's not needed for the single-key atomic record store, which is the sled bwtree index that is the current highest level module. MVCC is implemented in most popular embedded DBs because it is an effective way to manage mixed workloads
20.
by krenoten 8y ago
It's modular, and there is a paxos implementation, but it has been built totally in simulation so far and I haven't plugged it into an io layer yet. But this is trivial. That said, sled will always be a bwtree index, and the other
21.
by krenoten 8y ago
Currently it's even more basic. The current usable parts are a pagecache following the llama approach, some great testing utility libraries, and an index (that you can use as a kv) that follows the bwtree approach. Later it will have s
22.
by krenoten 8y ago
ALICE showed that's not always true with sqlite. Sled is being built with an extreme bias toward reliability over features, but as the readme says, it has some time to go before reaching maturity. The tests are quite good at finding ne
23.
by krenoten 9y ago
SPIN is a nice tool for modeling them. You can extract code from a coq model. You can go into the world of dynamic instrumentation to try to verify invariants in implementations.
24.
by krenoten 10y ago
Indeed. jemalloc uses a lock-free radix tree at its core.
25.
by krenoten 10y ago
Whoops, I meant to say "up to the largest minority failing", if a majority is lost then the gig is up! Too late to edit!
26.
by krenoten 10y ago
These consensus algorithms are awesome! They are the backbones of many highly available systems in modern DC's today. It's how people manage to get any sleep at all when taking care of systems that require very reliable databases.
27.
by krenoten 10y ago
It depends on the implementation. If you have a replication system that generates a monotonic sequence ID for each update, and you have all writes block on a majority of replicas acking an update, then you can just read from a majority and
28.
by krenoten 10y ago
Servers very frequently benefit from lower memory footprints, as it can also dramatically improve performance by improving cache efficiency.
29.
Simple consensus(rystsov.info)
1 points by krenoten 10y ago|0 comments
30.
by krenoten 10y ago
Yeah, when I think of Berlin apartments, I think of cramped-and-lovingly-dilapidated altbau rooms, resigned attempts to enliven a ubiquitous WWII reconstruction, and probably a balcony :) Damn, I can't wait to move back in a couple mon

More ›