7 ms·

Soul, SQLite REST server is realtime now. WebSockets added

38 points by thevahidal 4y ago

thevahidal 4y ago
Hi Folks, It's been an amazing journey since I first published Soul on HN and now I added a really major feature that Soul lacked, Realtime changes via Websockets. For those who are not familiar with Soul, it basically takes a SQLite database file and run a CRUD API on it, so you can have a minimal backend with no code. Now thanks to this new feature, users can subscribe to changes in a table and whenever a Create, Update or Delete operation happens, Soul will send the realtime data to subscribers. If you need some examples on how to work with websockets in Soul, you can find a bunch of examples here: https://github.com/thevahidal/soul/blob/main/docs/ws-examples.md https://github.com/thevahidal/soul/blob/main/docs/ws-example... Please let me know what you think of this new feature and also submit any issues you faced so we can fix them as soon as possible. Also if you have ideas to make Soul a better tool, please send me your ideas, it'll help me a lot. Once again thanks for the support and I'll see you in the next one! Repo: https://github.com/thevahidal/soul https://github.com/thevahidal/soul
- franky47 4y ago
  Nice work! FYI, just noticed a small typo in the repo description (relatime).
  - thevahidal 4y ago
    Thanks for the heads-up, I'll fix it asap.
- avinassh 4y ago
  How does it work? I mean how do you figure out that something has changed?
bsaul 4y ago
i love sqlite like as much as anybody else, but i'm really curious : why would you want to build a web api on top it ?
- deafpolygon 4y ago
  Lets you dump/query data into it without spinning up a complete database server + web frontend? I could have used something like this in my sysadmin days.
- claytongulick 4y ago
  For many cases SQLite has better performance characteristics than a full blown db server. If you can lock to a single-writer multi-reader model, SQLite is capable of handling very impressive loads. This works especially well with horizontal scale apps where, for example, you can shard by client and want to have an isolated environment per-client. Then you can do pg FDWs with a partitioned table and aggregate all of the SQLite dbs back together in a single view for super simple and fast cross-shard fan queries. I think the hardest part for these sorts of projects is figuring out authorization. For the two best-in-class (IMHO) examples of how to deal with this check directus and supabase. They have very different approaches to the problem, both very good.
  - bsaul 4y ago
    thanks a lot ! do you have real world examples and whitepapers of systems designed around that architecture ?
    - CoolCold 4y ago
      It's new hot topic, everyone feels being architect and showing others brave new world:)) You can read more by googling for fly.io, sqlite https://fly.io/blog/all-in-on-sqlite-litestream/ https://fly.io/blog/all-in-on-sqlite-litestream/ and listen this episode https://podcasts.google.com/feed/aHR0cHM6Ly9jaGFuZ2Vsb2cuY29tL3NoaXBpdC9mZWVk/episode/Y2hhbmdlbG9nLmNvbS8xNC8xNjU0?ep=14 https://podcasts.google.com/feed/aHR0cHM6Ly9jaGFuZ2Vsb2cuY29...
- jokethrowaway 4y ago
  Why not? SQLite is more than enough for most apps, there is no infra to manage and no network calls overhead.
claytonjy 4y ago
Is it fair to think of Soul like "PostgREST for SQLite"? Can definitely see this being useful!
- thevahidal 4y ago
  Hi claytonjy, Yeah that's a fair consumption, though now that Soul is realtime it could be considered as PostgREST + supabase/realtime.
IceWreck 4y ago
This looks like pocketbase (written in Go) but more resource intensive because it requires the entire node runtime. But anyways, good job !
- thevahidal 4y ago
  Hi IceWreck, I took so much inspiration from pocketbase, but the reason that I started Soul, is mainly because I think the bigger community of Node.js / Javascript developers, could potentially help a larger a group of devs to use and mostly extend Soul however they like.
  - csdvrx 4y ago
    Is there anything like soul or pocketbase, but focused on serving a local sqlite database to a browser, where you can present the results of premade requests, or issue plain SQL requests and plot the results if numerical? I'm thinking about something like a modernized and simpler "LAMP" setup, which could be used to teach SQL. Bonus point if the server can synchronize with other similar servers, to ingest records absent from the local database, which could be helpful to teach about why UNIQUE matters, and how consistency is hard
    - thunderbong 4y ago
      Maybe Datasette? https://datasette.io/ https://datasette.io/

38 points by thevahidal 4y ago

Hi Folks, It's been an amazing journey since I first published Soul on HN and now I added a really major feature that Soul lacked, Realtime changes via Websockets. For those who are not familiar with Soul, it basically takes a SQLite database file and run a CRUD API on it, so you can have a minimal backend with no code. Now thanks to this new feature, users can subscribe to changes in a table and whenever a Create, Update or Delete operation happens, Soul will send the realtime data to subscribers. If you need some examples on how to work with websockets in Soul, you can find a bunch of examples here: https://github.com/thevahidal/soul/blob/main/docs/ws-examples.md https://github.com/thevahidal/soul/blob/main/docs/ws-example... Please let me know what you think of this new feature and also submit any issues you faced so we can fix them as soon as possible. Also if you have ideas to make Soul a better tool, please send me your ideas, it'll help me a lot. Once again thanks for the support and I'll see you in the next one! Repo: https://github.com/thevahidal/soul https://github.com/thevahidal/soul

franky47 4y ago
Nice work! FYI, just noticed a small typo in the repo description (relatime).
- thevahidal 4y ago
  Thanks for the heads-up, I'll fix it asap.
avinassh 4y ago
How does it work? I mean how do you figure out that something has changed?

bsaul 4y ago

i love sqlite like as much as anybody else, but i'm really curious : why would you want to build a web api on top it ?

deafpolygon 4y ago
Lets you dump/query data into it without spinning up a complete database server + web frontend? I could have used something like this in my sysadmin days.
claytongulick 4y ago
For many cases SQLite has better performance characteristics than a full blown db server. If you can lock to a single-writer multi-reader model, SQLite is capable of handling very impressive loads. This works especially well with horizontal scale apps where, for example, you can shard by client and want to have an isolated environment per-client. Then you can do pg FDWs with a partitioned table and aggregate all of the SQLite dbs back together in a single view for super simple and fast cross-shard fan queries. I think the hardest part for these sorts of projects is figuring out authorization. For the two best-in-class (IMHO) examples of how to deal with this check directus and supabase. They have very different approaches to the problem, both very good.
- bsaul 4y ago
  thanks a lot ! do you have real world examples and whitepapers of systems designed around that architecture ?
  - CoolCold 4y ago
    It's new hot topic, everyone feels being architect and showing others brave new world:)) You can read more by googling for fly.io, sqlite https://fly.io/blog/all-in-on-sqlite-litestream/ https://fly.io/blog/all-in-on-sqlite-litestream/ and listen this episode https://podcasts.google.com/feed/aHR0cHM6Ly9jaGFuZ2Vsb2cuY29tL3NoaXBpdC9mZWVk/episode/Y2hhbmdlbG9nLmNvbS8xNC8xNjU0?ep=14 https://podcasts.google.com/feed/aHR0cHM6Ly9jaGFuZ2Vsb2cuY29...
jokethrowaway 4y ago
Why not? SQLite is more than enough for most apps, there is no infra to manage and no network calls overhead.

claytonjy 4y ago

Is it fair to think of Soul like "PostgREST for SQLite"? Can definitely see this being useful!

thevahidal 4y ago
Hi claytonjy, Yeah that's a fair consumption, though now that Soul is realtime it could be considered as PostgREST + supabase/realtime.

IceWreck 4y ago

This looks like pocketbase (written in Go) but more resource intensive because it requires the entire node runtime. But anyways, good job !

thevahidal 4y ago
Hi IceWreck, I took so much inspiration from pocketbase, but the reason that I started Soul, is mainly because I think the bigger community of Node.js / Javascript developers, could potentially help a larger a group of devs to use and mostly extend Soul however they like.
- csdvrx 4y ago
  Is there anything like soul or pocketbase, but focused on serving a local sqlite database to a browser, where you can present the results of premade requests, or issue plain SQL requests and plot the results if numerical? I'm thinking about something like a modernized and simpler "LAMP" setup, which could be used to teach SQL. Bonus point if the server can synchronize with other similar servers, to ingest records absent from the local database, which could be helpful to teach about why UNIQUE matters, and how consistency is hard
  - thunderbong 4y ago
    Maybe Datasette? https://datasette.io/ https://datasette.io/