6 ms·

I am developing a web/JavaScript database for several years now [1]. What I do not understand is, how graph based databases, (like surrealDB or gundb) plan to h

I am developing a web/JavaScript database for several years now [1]. What I do not understand is, how graph based databases, (like surrealDB or gundb) plan to have great performance when they run in the browser. The thing is that you are always limited to store your data inside of the browsers IndexedDB. So as soon as you do not work with the same document-based model similar to the IndexedDB API, you will have big troubles when it comes to performance. For example how can you store and query graph based data in IndexedDB without heavily overfetching either on read or on writes? IndexedDB is already way too slow. [1] https://github.com/pubkey/rxdb https://github.com/pubkey/rxdb

edgyquant 4y ago
Forgive my ignorance but What about the file system store? Is that the same thing?
- typingmonkey 4y ago
  There are 2 things called "file system api" in the browser. One is the "File System Access API" to store files on a users OS. This cannot be used to store data of a database.[1] The other one is the "real" filesytem api, which is not supported by most browsers. [2] [1] https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API https://developer.mozilla.org/en-US/docs/Web/API/File_System... [2] https://caniuse.com/filesystem https://caniuse.com/filesystem
tobiemh 4y ago
Hi typingmonkey. So SurrealDB abstracts away the document aspect of IndexedDB so it can be used as a key-value store. We then use that key-value store (in a similar way to RocksDB) to store the data in SurrealDB. You can see our abstraction layer here: https://github.com/surrealdb/indxdb https://github.com/surrealdb/indxdb . It's actually pretty performant in our tests so far, and we'll be releasing this soon as a WebAssembly library, enabling SurrealDB to run in the browser locally. I know rxdb (am a stargazer) so I'm really interested to hear suggestions or thoughts around this area!
- typingmonkey 4y ago
  Thank you for these links, I will inspect that. As far as I know, accessing IndexedDB from WebAssembly only works by passing the data through the JavaScript layer (correct me if I am wrong). So from how I think WebAssembly works, it is likely not any faster then directly doing that in JavaScript. I mean, the storage layer does not have to do any heavy computations, so tunneling the data would be more expensive then what you get from plain js. When you store the data into an IndexedDB based key-value store, how does the querying work? Do you still use IndexedDBs secondary indexes when a query only matches a range of documents, or does the K-V store has any method to support querying a subset of the data without fetching the whole database state?
  - tobiemh 4y ago
    Yes you're right. Accessing IndexedDB from WebAssembly needs to go through JavaScript. The only benefit from our perspective is that building it in Rust enables us to use the same logic and code that powers our entire database platform. From a performance aspect, you're right, it still needs to go through JavaScript. So we don't use any features from IndexedDB at all really. We literally treat IndexedDB as a key-value store interface like RocksDB or something like that. So all of the indexing is done within SurrealDB, and is stored in IndexedDB as a set of binary keys->values (Uint8Array if I remember correctly). When we want to query a whole table, or a subset of a table, SurrealDB determines what individual keys to load or what key ranges need to be loaded to perform the query. Would love to chat further about this - it's a really interesting area of discussion for me!
    - typingmonkey 4y ago
      Ok now I understand. You can come to the RxDB discord for a chat if you want. I am really interessted in using WebAssembly for browser side storage. I look forward to the day where someone reimplements IndexedDB via Webassembly, with the exact same API, so that we have a faster IndexedDB which uses the old one as storage layer :)
      - 4y ago

edgyquant 4y ago

Forgive my ignorance but What about the file system store? Is that the same thing?

typingmonkey 4y ago
There are 2 things called "file system api" in the browser. One is the "File System Access API" to store files on a users OS. This cannot be used to store data of a database.[1] The other one is the "real" filesytem api, which is not supported by most browsers. [2] [1] https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API https://developer.mozilla.org/en-US/docs/Web/API/File_System... [2] https://caniuse.com/filesystem https://caniuse.com/filesystem

tobiemh 4y ago

Hi typingmonkey. So SurrealDB abstracts away the document aspect of IndexedDB so it can be used as a key-value store. We then use that key-value store (in a similar way to RocksDB) to store the data in SurrealDB. You can see our abstraction layer here: https://github.com/surrealdb/indxdb https://github.com/surrealdb/indxdb . It's actually pretty performant in our tests so far, and we'll be releasing this soon as a WebAssembly library, enabling SurrealDB to run in the browser locally. I know rxdb (am a stargazer) so I'm really interested to hear suggestions or thoughts around this area!

typingmonkey 4y ago
Thank you for these links, I will inspect that. As far as I know, accessing IndexedDB from WebAssembly only works by passing the data through the JavaScript layer (correct me if I am wrong). So from how I think WebAssembly works, it is likely not any faster then directly doing that in JavaScript. I mean, the storage layer does not have to do any heavy computations, so tunneling the data would be more expensive then what you get from plain js. When you store the data into an IndexedDB based key-value store, how does the querying work? Do you still use IndexedDBs secondary indexes when a query only matches a range of documents, or does the K-V store has any method to support querying a subset of the data without fetching the whole database state?
- tobiemh 4y ago
  Yes you're right. Accessing IndexedDB from WebAssembly needs to go through JavaScript. The only benefit from our perspective is that building it in Rust enables us to use the same logic and code that powers our entire database platform. From a performance aspect, you're right, it still needs to go through JavaScript. So we don't use any features from IndexedDB at all really. We literally treat IndexedDB as a key-value store interface like RocksDB or something like that. So all of the indexing is done within SurrealDB, and is stored in IndexedDB as a set of binary keys->values (Uint8Array if I remember correctly). When we want to query a whole table, or a subset of a table, SurrealDB determines what individual keys to load or what key ranges need to be loaded to perform the query. Would love to chat further about this - it's a really interesting area of discussion for me!
  - typingmonkey 4y ago
    Ok now I understand. You can come to the RxDB discord for a chat if you want. I am really interessted in using WebAssembly for browser side storage. I look forward to the day where someone reimplements IndexedDB via Webassembly, with the exact same API, so that we have a faster IndexedDB which uses the old one as storage layer :)
    - 4y ago