7 ms·
TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services. I am one of the maintainers of this project you can
by royguo1988 6y ago
TerarkDB was acquired by Bytedance two years ago and is now using widely in Bytedance's database services.
I am one of the maintainers of this project you can ask any question here.
- nikhilsimha 6y agoAlways love it when I see a maintainer offer clarifications in a HN comment section! <3 What are the reasons for the perf improvements we see here?
- royguo1988 6y agoWe are working on our `all-in-one docs` right now, please watch our repo, thanks! I replied some of the reasons in previous comment.
- josephg 6y agoYour all-in-one docs[1] refuse to render at all in firefox? Seems like a strange restriction, and disappointing that it doesn't even let you read the document in non-webkit browsers. Feels like the IE days all over again. "An error occurred. This browser is not supported, click here to learn more." [1] https://bytedance.feishu.cn/docs/doccnZmYFqHBm06BbvYgjsHHcKc https://bytedance.feishu.cn/docs/doccnZmYFqHBm06BbvYgjsHHcKc
- oefrha 6y agoWorks perfectly for me in Firefox (macOS). Might be a misleadingly worded "something went wrong, we don't know what" message. Probably want to check your console, could be a network problem.
- josephg 6y agoYeah there's some unreadable minified error stack traces in the console. Weirdly it seems to load fine and I can scroll the content for a second or two while its still loading in. Then it puts up an error dialog and I can't access the content any more. Once thats happened if I reload the page the error dialog comes up immediately and the page doesn't bother loading behind it. I'm on firefox on MacOS too, and it happens with my ad blocker on or off. Chrome works fine. The support document linked in the error explicitly says only chrome and safari are supported on macos. I'm confused why my grandparent comment has been downvoted - this is a real bug report stopping me (and maybe others) from reading documentation that looks to have a lot of thought put into it. And given the content seemed to be loading fine before the error message came up, well, it feels forced.
- ddorian43 6y agoIt renders on ubuntu 18.04 Firefox but also displays "This browser not supported with https://www.feishu.cn/hc/en-us/articles/360038713913 https://www.feishu.cn/hc/en-us/articles/360038713913". Probably doesn't support linux.
- erk__ 6y agoI get the same on Windows 10 so it is probably the browser.
- royguo1988 6y agoThis is wired, I will call our internal Lark team to deal with it.
- yomly 6y agoDo you have a write up for why you got rid of RocksDB
- deleted 6y ago[deleted]
- royguo1988 6y agoWe are working on our `all-in-one docs` which will explain everything. I want to address that we are not meant to "get rid of" RocksDB (which lots of KV engine claimed). What we want to do is provide another solution for storage engine users with different road path (focusing on new hardware and heavy-write workloads). For simple use cases, there will be no difference no matter what engine you use. And for most cases, upgrade your hardware (e.g. SATA SSD to NVMe SSD) or tuning your RocksDB parameters would save you lots time, just make sure you understand what you are doing. There's no cue for every workloads, try TerarkDB if RocksDB happens not fit your scenario.
- royguo1988 6y agoThe reasons we did a better job(from our own perspective) than RocksDB are: 1. We moved lots of code out side db_mutex (db mutex is convenient but costs too much) 2. We introduced a new KV separation implementation that we believe is better than RocksDB’s implementation (we didn't hear any production user are using RocksDB's KV separation yet) 3. We introduced a lazy compaction strategy that can delay compaction task while online services are dealing with short-time heavy writing. 4. Other optimizations like time histogram based TTL, pipelined WAL sync.
- why_only_15 6y agoI think the performance images would be a lot more clear if they were on the same scale, as it stands it was unclear what was happening with, e.g. the disk write image until i zoomed into the axes.
- royguo1988 6y agoThanks for your suggestion, I will update the image soon
- sudeepj 6y agoWhy not merge the improvements into RocksDB itself?
- royguo1988 6y agoThere are mainly three reasons here: 1. We changed the source code too much that we are not able to merge it back to RocksDB easily (This project started at 2016 as an close-source project) 2. We have different road path with RocksDB (e.g. We will remove a lot of un-used code to make TerarkDB much more light-weight than current version in the future) 3. We have lots of third-party partners (e.g. Intel, on Opatane SSD/Memory and others with ZNS...) may participant in this project so we want to handle all commits ourself to make sure everything is under control.
- ssakamoto 6y ago3) is not in line with an open source philosophy. EDIT: Detrimental to the original. Eg. Amazon forking and selling MongoDB.
- rishav_sharan 6y agoHow so? Unless they are stopping normal users from committing code as well?
- tinco 6y agoEven if they stopped normal users from committing it would still be adhering to open source philosophy.
- setr 6y agoSure it is; it’s exactly equivalent to something like forking Linux with the reasoning “I want to be the BDFL now” — eg the nvim fork
- ssakamoto 6y ago
- polskibus 6y agoYou may want to update the TerarkDb entry on dbdb.io.
- royguo1988 6y agoThanks, tried to log-in & reset my password but didn't receive reset email.
- apavlo 6y agoEmail me (pavlo@cs.cmu.edu). I don't think you ever had an account.
- continuations 6y agoI remember reading about this a few years ago here. If I remember correctly back then the main selling point was that it used succinct data structure and it was only the compression algo that was not open source - everything else was. But now when I look at the new repo and the online doc there is no mention of succinct data struct anywhere. Also, the benchmarks back then claimed 10x or more faster than RocksDB. Now the performance claim is much more modest. Does that mean TerarkDB no longer uses succinct data struct? Or are you just open sourcing a lower-end version of the software without the secret sauce? Can you talk about what makes TerarkDB faster than RocksDB?
- royguo1988 6y agoThanks for your attention, glad someone here still remember our history, TerarkDB is now FULLY open source with `succinct data structures`. Here's the reasons: 1. Our `all-in-one` docs are still under writing, we will cover that part later. 2. For the performance part, we are now showing real-world cases, not a well-designed benchmark.(We selected the best result to show our work few years ago, don't want to do it anymore) 3. About why TerarkDB is faster than RocksDB will be explained in our `all-in-one docs` in one week, and most of the reasons are not magic, just engineering efforts Thanks again for your remembering us.
- continuations 6y agoGreat. Looking forward to learn more about this.
- e12e 6y ago> most of the reasons are not magic So... You're saying there is magic? :)
- meta2meta 6y agoI see "#include <terark/fsa/cspptrie.inl>" in the "memtable/terark_zip_entry_index.cc" but I can't find "cspptrie.inl" in the repo. Is the code auto-generated or not open source now?
- wanghenshui 6y agosubmodule, https://github.com/bytedance/terark-zip https://github.com/bytedance/terark-zip
- royguo1988 6y agoAll source code is open source now. You can find them in `third-party/terark-zip`, terark-zip is a standalone repo that contains only core algorithms.
- supergirl 6y agowhere is it used? what kind of data is stored in it?
- royguo1988 6y agoIn bytedance, a few database services are using TerarkDB.
- supergirl 6y agoyes, I got that :) but can you say more? what kind of database services? what data is stored, what is the scale, what are the requirements, etc.
- royguo1988 6y agoSorry for the unclear response. 1) We use TerarkDB under a distributed SQL database and TerarkDB helps to store its pages (16KB page), its one of the most widely used SQL database inside Bytedance. 2) We use TerarkDB under a Redis compatible distributed cache system to store raw key value pairs. Almost all kinds of workloads are here since TerarkDB runs under too many database clusters (each cluster only serves a single application)
- deleted 6y ago[deleted]
- amrx431 6y agoIs there a backdoor for Chinese?
- siggen 6y agoGiven that it is open source, you can answer this question yourself?
- gurkanoluc 6y agoIs TerarkDB used as store engine for MySQL like FB does?
- loeg 6y agoThat glue layer is called MyRocks (MySQL -> MyRocks -> RocksDB). It may be possible to slot this in to replace RocksDB in that stack. (I don't know.)