8 ms·

Show HN: PocketBase – Open Source realtime backend in one file

563 points by randomwebdev 4y ago

aalaap 4y ago
This is incredible.
creakingstairs 4y ago
This is cool. I think it would also be neat to have option to have Litestream [1] integration so that users can use this without any persistent volume or worry about back ups [1] https://litestream.io/ https://litestream.io/
- marrobins 4y ago
  PocketBase uses embedded SQLite in WAL mode, so it's already compatible with Litestream if you want to integrate it. Or do you want Litestream somehow embedded in the Pocketbase binary so it's part of the "one file" artefact? They're both written in Go, so I guess that could theoretically be possible.
asmutalix 4y ago
Congrats! What a lovely project. What a fantastic job! Question: I am currently evaluating Ory/Kratos as authentication backend for a side project. Ory Kratos is a mature and excellent project. However, it has very little documentation when it comes to self-hosting. Further, I also feel like maintaining code that uses it down the line could is going to be problematic given its shear size and complexity. How do you think Pocketbase compares with Ory Kratos?
moon700 4y ago
Looks very interesting! looks like document database, but how i can create indexes?
- randomwebdev 4y ago
  Currently you cannot create indexes through the admin UI, but it's in my todo. As an alternative, you can always create indexes through DB migrations or add the index directly via the `sqlite3` cli.
  - moon700 4y ago
    thanks for reply! is it schema free? or i need to define scheme via migrations?
    - randomwebdev 4y ago
      Its not schema free (although you can create a json field and put whatever you want in it). You can define schema via migrations or through the admin UI (see https://pocketbase.io/docs/manage-collections/ https://pocketbase.io/docs/manage-collections/).
randomwebdev 4y ago
PocketBase is a small open source project that I've been working for the last couple of months. You could think of PocketBase as a lightweight Firebase/Supabase alternative. In short, it is an open source Go backend (framework and app) consisting of: - embedded database (SQLite) with realtime subscriptions\ - backed-in files and users management - convenient Admin dashboard UI - simple REST-ish API And all of this compiles to a single portable binary. It's still a little rough around the edges, but you could give it a try and share your feedback.
- swyx 4y ago
  feedback: the "in one file" marketing confused me for a bit until i realized you meant "as a single binary". i would say that instead.
  - inevaexisted 4y ago
    I got tripped up on the same thing.
- Silica6149 4y ago
  Nice! It looks like you put a lot of work into this, what was the motivation behind this project? Did you not like the existing solutions available?
  - randomwebdev 4y ago
    Thanks! It was initially created to be the backend for the next version of my other open source project - Presentator v3 (https://github.com/presentator/presentator https://github.com/presentator/presentator). You could read more at https://github.com/presentator/presentator/issues/183 https://github.com/presentator/presentator/issues/183, but to summarize - I wanted something that could be self hosted with almost no additional setup and no dependency on 3rd party services.
- Pr0ject217 4y ago
  This looks cool.
dugmartin 4y ago
Looks very nice. One thing I noticed is you are using server sent events for subscriptions. SSE have a lot of nice properties but do have one nasty limitation (from https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent...): "Warning: When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100)." This can lead to some weird bug reports from users.
- randomwebdev 4y ago
  Yes, I'm aware of this limitation, but PocketBase uses HTTP2 when combined with the --https flag (this is by go's default net/http implementation) so it shouldn't be too much of a concern.
  - antholeole 4y ago
    First time I’ve heard about the 6 connection limit. I’m not fully understanding the response - are you saying that the limit is not imposed if you use https? Or am I reading wrong?
    - randomwebdev 4y ago
      Go's default `net/http` package will serve HTTP2 when a https configuration is provided (also most web browsers don't support h2c, aka. HTTP2 without tls). The default limit of ~100 connections should be more than enough for most applications (additionally, the JS SDK client maintains a single SSE connection for a page no matter to how many things the user has subscribed, so that also helps).
    - 4y ago
pbronez 4y ago
Sweet. How tightly is this coupled to SQLite? I’d like to embed Dolt [0] instead, specifically for the Dolthub-backed collaboration model. [0] https://github.com/dolthub/dolt https://github.com/dolthub/dolt
- randomwebdev 4y ago
  Other than some json functions, PocketBase doesn't use too much SQLite specific features, but I don't plan adding support for other databases at the moment. Please also note that PocketBase is designed to be deployed only on a single server, so it cannot scale horizontally out of the box. Supabase/Nhost are better suited for that.
oakesm9 4y ago
How are the real time subscriptions with SQLite achieved? Is this a feature of SQLite itself or something that PocketBase itself implements on top of it?
- randomwebdev 4y ago
  It's implemented in the application, it's not a feature of SQLite, but its very simple and robust - when a record is created/updated/deleted I'm broadcasting the change to all subscribers that are allowed to receive it (the access control is based on the colleciton's list and view rules). The realtime api implementation could be found at https://github.com/pocketbase/pocketbase/blob/master/apis/realtime.go https://github.com/pocketbase/pocketbase/blob/master/apis/re...
  - oakesm9 4y ago
    That makes a lot of sense. Thanks for the link. I've not worked with Go before so I couldn't find exactly where the implementation was. This looks like a very interesting project and great for some of the small client projects which I currently use Firebase. I'll be sure to take a proper look in the future!
  - Norgie93 4y ago
    Reading through the code/docs it looks like you can subscribe to individual records (database rows) or collections (tables), is this correct? In your Records.getList method, you have filter functionality that would be nice to subscribe to (ie. subscribe to changes in "demo" collection where totalComments > 10). Subscribing to a specific result set seems like it could scale better than notifying subscribers of all table changes, which then get filtered in the subscribe() callback func in the SDK. Very cool work!
    - randomwebdev 4y ago
      Yes, you can subscribe to individual records or collections. User defined filters for the subscriptions are not supported, but it's a good idea and I may consider it in the future. Currently the subscriptions are filtered through the collection's list and view rules - they also act as "admin level filters", aka. filters that are always applied and regular users cannot modify (getList user defined filters are only appended to the search query together with the admin defined filters).
Comevius 4y ago
This here is some quality work top to bottom, including the documentation and the website.
- krat0sprakhar 4y ago
  +1 .. Very amazing!
Xeoncross 4y ago
Very nice, I thought it was a basic wrapper around SQLite but then realized it actually had more features then that, such as it's own admin dashboard. Then I realized you built multiple auth designs into the system, then I realized there was a javascript SDK to interact with it, then I found your beautiful documentation site, then I realized there is an actual product inspiring it's development (https://presentator.io/ https://presentator.io/) Each of these is another encouragement to use pocketbase. I went from "eh, I'll just keep my own wrapper" to "pocketbase has a lot of promise, I should try it"
codegeek 4y ago
Wow, super impressed. I have always wanted to build something like this where everything is embedded in 1 single file and I knew Go could be a great way to do it but always wondered how to handle the database part. Along comes SQLlite. One feedback which will probably deviate from everything in 1 file concept but adding an option to upload files to an S3 URL will be awesome. Perhaps a pull request ? I am thinking of use cases like document management using this and if I can have the actual files in S3 (or S3 compatible), that would be a huge plus. EDIT: I spoke too soon. You already have an option for S3 as I tested the demo. Very cool. How much work went into this btw ? The github, documentation everything is well done. I envy work like this :)
- endofreach 4y ago
  Uploaded files must deviate from „everything in one file“ regardless of using S3, right? How (and why) would you store user uploads within the binary?
  - codegeek 4y ago
    yea it was a silly question. It does allow S3 anyway so we are good.
    - endofreach 4y ago
      My intent was not to point out a „silly question“. I was legitimately interested if there is a reasoning and what a use case for that would be. Thank you for responding nicely!
  - randomwebdev 4y ago
    Uploaded files and your database are stored outside of the binary.
- randomwebdev 4y ago
  By default PocketBase uses the local filesystem, but you could also configure a S3 compatible storage (AWS, DigitalOcean, Vultr, etc.) from "Settings > Files Storage > Use S3 storage". I've started working on the project ~ 3-4 months ago. The last 4 weeks I've taken a longer break from my day-to-day job so that I could finalize the first release.
gervwyk 4y ago
Well done! Really impressive, will give it a spin.
Pr0ject217 4y ago
Potentially silly question (I'll go out on a limb anyway). I would like to create a multi-platform app for non-technical users where their data is transparently self-contained/self-hosted. So, they open a single binary and it presents them with a UI, and the data created from within the app would be stored in their local PocketBase database. However, I wouldn't want the user to have to start a server directly (they wouldn't know what a server is). So, I think the binary would have to start both the PocketBase server and a front-end server, and then launch a browser (similar to Electron) pointed to their front-end server's port. Does this sound like a good approach? What could you recommend to orchestrate this seamlessly to make for a simple user experience when launching the app? Thank you.
- qbasic_forever 4y ago
  Honestly I'd look at using Electron because you can take a hard dependency on the browser capabilities and state. If you rely on the user bringing their own browser you're going to have headaches of people on ancient browsers or just supremely jacked up browsers (messed up browser plugins, proxies, turned off javascript, etc.). Node can easily launch processes and there are even fancy process management libraries like pm2 you can add as a dependency should you need it, so the logic in your electron layer can be very simple and just launch some other processes and ferry requests to the view layer. Alternatively a webview-based framework or something like Lorca might be another option if you don't want to use Electron: https://maori.geek.nz/golang-desktop-app-webview-vs-lorca-vs-electron-a5e6b2869391 https://maori.geek.nz/golang-desktop-app-webview-vs-lorca-vs...
  - Pr0ject217 4y ago
    Thank you!
- randomwebdev 4y ago
  Hm, there is nothing preventing you to start multiple services from a single binary. I'm not sure that PocketBase would be suitable for your use case, but it is distributed also as a Go package (see https://pocketbase.io/docs/use-as-framework/ https://pocketbase.io/docs/use-as-framework/) and you could combine it with any other GUI Go package (fyne, go-astilectron, etc.)
melony 4y ago
This looks amazing, congrats on shipping! Do you plan to add Firebase-style query optimizations too? https://news.ycombinator.com/item?id=31832266 https://news.ycombinator.com/item?id=31832266
roguas 4y ago
I would love it. But: postgres, +backups, +replica managment. Basically a managed database for postgres in a container.
- _query 4y ago
  Check out https://thin.dev/ https://thin.dev/ :) It's similar, supports self-hosting and uses postgres. Quick demo video here: https://www.youtube.com/watch?v=-jj19fpkd2c&t=3s https://www.youtube.com/watch?v=-jj19fpkd2c&t=3s (I'm founder of Thin)
- randomwebdev 4y ago
  You may want to check Supabase or Nhost in that case. PocketBase was specifically designed to be self-contained and running only on a single server.
ezekiel11 4y ago
why is this written in Go , are there specific features that shine over Python3? I've been looking to learn Go, it seems lot of network intensive software use Go.
- rekwah 4y ago
  If a selling point is run from "one file", that's quite difficult to do in python. There are things like pyinstaller but you end up shipping the entire interpreter in the bundle.
  - mypalmike 4y ago
    The pex tool lets you build single file python executables which are well suited for server deployments.
    - ezekiel11 4y ago
      oh wow thanks i didn't know about this. in that case it would be super simple for me to wrap up a FastAPI server with SQLite or even some in-memory database that runs elsewhere, deliver as a single file!
      - killingtime74 4y ago
        You could also just use Docker to wrap anything
      - mypalmike 4y ago
        Yeah this kind of thing was common when I worked at Twitter (where AFAIK pex/pants was developed) on an infrastructure team. It's a cool tool that few outside of Twitter seem to be aware of.
    - mhio 4y ago
      How big is a python3 runtime when pex has packaged it up? And do you happen to know what the output binary is linked to?
beardicus 4y ago
this looks really great! i had recently been thinking “boy it’d be really great if there were a ‘supabase but sqlite’ available” and here you are! now i just need a clever project as an excuse to use this. 1 question: are the docs in a repo anywhere? i don’t see them in the main repo (i might just be missing them somehow) or anywhere in the org. if they are public somewhere would you be open to a docs polishing PR?
- randomwebdev 4y ago
  The documentation is not auto generated and is part of the static landing site (built with SvelteKit). I didn't bother open source it because it is kinda messy and a little complicated to be edited by users not familiar with the codebase, but I'll try to find some time in the future and may publish it. In the meantime, if you find typos or think that some of the wording could be improved, feel free to open an issue or discussion in the main repo and I'll fix them.
1xdevloper 4y ago
This looks perfect for storing the internal data of my side project! Love the website design too.
vincentge 4y ago
This actually looks suuuupppppeeeerrrr cool!!!
deleted 4y ago
[deleted]
segphault 4y ago
This looks like a great solution for rapid prototyping scenarios where you want a basic CRUD backend with as little friction as possible. The API preview feature in the admin dashboard is a really nice touch.
catchmeifyoucan 4y ago
This looks great! Does this have support for offline usage on the client?
ptudan 4y ago
This looks awesome! I will definitely give it a try. Right now I use postgres in a docker compose but its definitely a bit heavier for single machine prototyping. One thing that I like about my setup though is that there's a clear migration path to horizontally scale. Do you have a recommended way to move past pocketbase should an app get to that point?
- randomwebdev 4y ago
  I guess there will be differences from use case to use case, but in general migrating from PocketBase to a more common horizontal supported stack is not different from rewriting your application. Internally, each Collection creates a standard SQLite table that holds the collection records, so migrating the data structure shouldn't be too troublesome. The only thing that may prove difficult to migrate could be reimplementing the access rules and filters. But in my opinion, when your application reaches that level of growth requiring multiple servers and services, your business use cases mostly likely will have changed several times already from your initial idea.
- felipeccastro 4y ago
  Perhaps sharding data by geographical location could be one path - e.g. deploy one server in the US and another in Europe, each with its own database. The whole concept of edge computing - deploying app servers closer to your users for minimal network latency - seems to go hand in hand with using an embedded database like SQLite. (I haven't tried it in production yet, but looking forward to.)
codeptualize 4y ago
Looks amazing! Love the idea of a Firebase/Supabase like thing I can easily run locally and deploy as well. Also props for the dashboard design, looks and works very nice!
jitl 4y ago
This looks really great. Have you thought about adding Operational Transform or CRDT primitives to your offering? Realtime subscriptions are nice - but they're even better with realtime collaboration. Because you're using SQLite directly, it's easy(er) for you to build in efficient CRDT mutations to your database rows because SQLite's BLOB column supports incremental IO: https://www.sqlite.org/c3ref/blob_open.html https://www.sqlite.org/c3ref/blob_open.html This could allow you to save a CRDT update operation without needing to read the entire CRDT blob into memory, parse it, and then save it all back to disk.
- randomwebdev 4y ago
  Currently any additional data transformation is left to the developers to extend via event hooks or custom client side handling. I've worked with CRDT in the past (yjs), but it may not be very useful in PocketBase considering that the application was designed to run on a single server and db writes are practically queued (you can have only one sqlite writer at a time). I'll investigate it further and may consider it for future release.
  - jitl 4y ago
    I think if you integrate Yjs or a OT library first-party, you'll have a really exceptional framework - open source firebase alternative top-to-bottom, ready for "local-first" collaborate own-your-own-data apps. Sounds like my dream stack for small-scale projects or friend/family apps.
abraxas 4y ago
Nice but my super anal quip is please don't call this thing "realtime". It doesn't make sense in this context. What is realtime about it? Are there timing guarantees or are timing constraints provided in API requests? Because that's what realtime systems are. Realtime does not mean "real fast". For example, a train track switching system might be getting run by a 1MHz CPU and still be real time. It's not about speed, it's about deadline guarantees.
- willy_k 4y ago
  It’s not a real-time operating system. In this case, real-time means it is built such that a client gets access to updated database values as soon as they change, as opposed to a non real-time database which doesn’t necessarily have a built-in mechanism for reliably on time updates.
  - abraxas 4y ago
    You mean like eventual consistency vs immediate consistency? Because a database that reflects its changes at the end of a transaction is called an ACID database not a real-time database.
euroderf 4y ago
Suitable as the basis for a CMS ?
- randomwebdev 4y ago
  The existing admin interface could be used as a very rudimentary content management, but it was really intended for quick internal data explorations and edits. Additionally, currently there are no roles for admins so permission controls within the admin area are not possible. You could always implement your own custom CMS SPA frontend that interacts with the PocketBase API (that is how I'm planning to use it with the next version of my other open source project - Presentator).
  - euroderf 4y ago
    Maybe I open an issue for discussion. Mainly I wonder about making architectures mesh.
999900000999 4y ago
Any client SDKs for Android, iOS, React, etc.
- randomwebdev 4y ago
  There is already a JavaScript SDK (https://github.com/pocketbase/js-sdk https://github.com/pocketbase/js-sdk) that works both in the browser and node. Dart SDK (targeting mainly Flutter) is also on the way. Unfortunately I'm not very uptodate with Android (Kotlin) or iOS (Swift) development and therefore community contributions are more than welcomed.
  - 999900000999 4y ago
    This is much more than I expected, thank you !
ilrwbwrkhv 4y ago
This is fantastic work. Supabase local development has always been a pain.
- wenbo 4y ago
  Member of the Supabase team! Would love to hear your pain points with local development. Always looking for feedback so that we may improve your experience.
  - ilrwbwrkhv 4y ago
    Well all the UI stuff. It is super easy to add a column and change something fast with the UI when doing rapid development. But that piece is missing when developing locally. I have gone on to create 2 projects (one called local and one prod) and manually applying the changes. Would be nice to apply changes from one project to another. PocketBase here for example provides the UI locally too. That is the real selling point IMO.
    - wenbo 4y ago
      Thanks for the feedback! Passed it on to our frontend team.
benkaiser 4y ago
For anyone curious, another popular open-source backend as a service is AppWrite: https://appwrite.io/ https://appwrite.io/ I've tried them out for some dummy projects and they seem pretty great, and they are continually iterating and pushing new features.
dinvlad 4y ago
So good to see a renaissance of "portable apps", now on the web. All of this extra complexity associated with multi-tiered multi-microservice arch is not needed for many categories of apps, and especially for small teams.
traviscj 4y ago
Really neat! I’ve been wanting to make something similar for _years_ to be a standardized data modeling testbed as I explore/experiment/prototype some new feature or problem space or even doing ad-hoc fixes. I think this is a much better starting place than spreadsheets/similar. Good luck with the project! Seems like you’ve got some good interest :-)
newusertoday 4y ago
This is great, if it was available a year ago it would have saved me lot of time in creating my home-grown solution ;)
lexx 4y ago
This is a really nice idea and I love the execution as well. Congrats and thanks for sharing.
Wronnay 4y ago
I really like this. All my current projects use Go as a backend and I definitely need my own routes and business logic, so PocketBase looks way easier to integrate into my existing code than the JavaScript based solutions. And it would also give me more time to focus on business logic instead of Auth - i will definitely try this in the future.
h3rald 4y ago
Nice project! It looks pretty similar to my own LiteStore: https://h3rald.com/litestore/ https://h3rald.com/litestore/ The feature set is pretty similar, but I focused mainly on the free form/nosql-ish aspect of it, and although I added support for server-side JavaScript middleware I didn't include authentication OOTB (although it could be implemented or you can integrate with an OAuth2 system - it can validate JWT tokens). And yes, LiteStore's web app hasn't been updated in years and overall your project looks much more polished than mine. Good job! Bookmarking ;)
- resoluteteeth 4y ago
  I notice that it has full text search support, too, which pocketbase doesn't seem to.

563 points by randomwebdev 4y ago

This is incredible.

This is cool. I think it would also be neat to have option to have Litestream [1] integration so that users can use this without any persistent volume or worry about back ups [1] https://litestream.io/ https://litestream.io/

marrobins 4y ago
PocketBase uses embedded SQLite in WAL mode, so it's already compatible with Litestream if you want to integrate it. Or do you want Litestream somehow embedded in the Pocketbase binary so it's part of the "one file" artefact? They're both written in Go, so I guess that could theoretically be possible.

asmutalix 4y ago

Congrats! What a lovely project. What a fantastic job! Question: I am currently evaluating Ory/Kratos as authentication backend for a side project. Ory Kratos is a mature and excellent project. However, it has very little documentation when it comes to self-hosting. Further, I also feel like maintaining code that uses it down the line could is going to be problematic given its shear size and complexity. How do you think Pocketbase compares with Ory Kratos?

moon700 4y ago

Looks very interesting! looks like document database, but how i can create indexes?

randomwebdev 4y ago
Currently you cannot create indexes through the admin UI, but it's in my todo. As an alternative, you can always create indexes through DB migrations or add the index directly via the `sqlite3` cli.
- moon700 4y ago
  thanks for reply! is it schema free? or i need to define scheme via migrations?
  - randomwebdev 4y ago
    Its not schema free (although you can create a json field and put whatever you want in it). You can define schema via migrations or through the admin UI (see https://pocketbase.io/docs/manage-collections/ https://pocketbase.io/docs/manage-collections/).

randomwebdev 4y ago

PocketBase is a small open source project that I've been working for the last couple of months. You could think of PocketBase as a lightweight Firebase/Supabase alternative. In short, it is an open source Go backend (framework and app) consisting of: - embedded database (SQLite) with realtime subscriptions\ - backed-in files and users management - convenient Admin dashboard UI - simple REST-ish API And all of this compiles to a single portable binary. It's still a little rough around the edges, but you could give it a try and share your feedback.

swyx 4y ago
feedback: the "in one file" marketing confused me for a bit until i realized you meant "as a single binary". i would say that instead.
- inevaexisted 4y ago
  I got tripped up on the same thing.
Silica6149 4y ago
Nice! It looks like you put a lot of work into this, what was the motivation behind this project? Did you not like the existing solutions available?
- randomwebdev 4y ago
  Thanks! It was initially created to be the backend for the next version of my other open source project - Presentator v3 (https://github.com/presentator/presentator https://github.com/presentator/presentator). You could read more at https://github.com/presentator/presentator/issues/183 https://github.com/presentator/presentator/issues/183, but to summarize - I wanted something that could be self hosted with almost no additional setup and no dependency on 3rd party services.
Pr0ject217 4y ago
This looks cool.

dugmartin 4y ago

Looks very nice. One thing I noticed is you are using server sent events for subscriptions. SSE have a lot of nice properties but do have one nasty limitation (from https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events https://developer.mozilla.org/en-US/docs/Web/API/Server-sent...): "Warning: When not used over HTTP/2, SSE suffers from a limitation to the maximum number of open connections, which can be especially painful when opening multiple tabs, as the limit is per browser and is set to a very low number (6). The issue has been marked as "Won't fix" in Chrome and Firefox. This limit is per browser + domain, which means that you can open 6 SSE connections across all of the tabs to www.example1.com and another 6 SSE connections to www.example2.com (per Stackoverflow). When using HTTP/2, the maximum number of simultaneous HTTP streams is negotiated between the server and the client (defaults to 100)." This can lead to some weird bug reports from users.

randomwebdev 4y ago
Yes, I'm aware of this limitation, but PocketBase uses HTTP2 when combined with the --https flag (this is by go's default net/http implementation) so it shouldn't be too much of a concern.
- antholeole 4y ago
  First time I’ve heard about the 6 connection limit. I’m not fully understanding the response - are you saying that the limit is not imposed if you use https? Or am I reading wrong?
  - randomwebdev 4y ago
    Go's default `net/http` package will serve HTTP2 when a https configuration is provided (also most web browsers don't support h2c, aka. HTTP2 without tls). The default limit of ~100 connections should be more than enough for most applications (additionally, the JS SDK client maintains a single SSE connection for a page no matter to how many things the user has subscribed, so that also helps).
  - 4y ago

pbronez 4y ago

Sweet. How tightly is this coupled to SQLite? I’d like to embed Dolt [0] instead, specifically for the Dolthub-backed collaboration model. [0] https://github.com/dolthub/dolt https://github.com/dolthub/dolt

randomwebdev 4y ago
Other than some json functions, PocketBase doesn't use too much SQLite specific features, but I don't plan adding support for other databases at the moment. Please also note that PocketBase is designed to be deployed only on a single server, so it cannot scale horizontally out of the box. Supabase/Nhost are better suited for that.

oakesm9 4y ago

How are the real time subscriptions with SQLite achieved? Is this a feature of SQLite itself or something that PocketBase itself implements on top of it?

randomwebdev 4y ago
It's implemented in the application, it's not a feature of SQLite, but its very simple and robust - when a record is created/updated/deleted I'm broadcasting the change to all subscribers that are allowed to receive it (the access control is based on the colleciton's list and view rules). The realtime api implementation could be found at https://github.com/pocketbase/pocketbase/blob/master/apis/realtime.go https://github.com/pocketbase/pocketbase/blob/master/apis/re...
- oakesm9 4y ago
  That makes a lot of sense. Thanks for the link. I've not worked with Go before so I couldn't find exactly where the implementation was. This looks like a very interesting project and great for some of the small client projects which I currently use Firebase. I'll be sure to take a proper look in the future!
- Norgie93 4y ago
  Reading through the code/docs it looks like you can subscribe to individual records (database rows) or collections (tables), is this correct? In your Records.getList method, you have filter functionality that would be nice to subscribe to (ie. subscribe to changes in "demo" collection where totalComments > 10). Subscribing to a specific result set seems like it could scale better than notifying subscribers of all table changes, which then get filtered in the subscribe() callback func in the SDK. Very cool work!
  - randomwebdev 4y ago
    Yes, you can subscribe to individual records or collections. User defined filters for the subscriptions are not supported, but it's a good idea and I may consider it in the future. Currently the subscriptions are filtered through the collection's list and view rules - they also act as "admin level filters", aka. filters that are always applied and regular users cannot modify (getList user defined filters are only appended to the search query together with the admin defined filters).

Comevius 4y ago

This here is some quality work top to bottom, including the documentation and the website.

krat0sprakhar 4y ago
+1 .. Very amazing!

Xeoncross 4y ago

Very nice, I thought it was a basic wrapper around SQLite but then realized it actually had more features then that, such as it's own admin dashboard. Then I realized you built multiple auth designs into the system, then I realized there was a javascript SDK to interact with it, then I found your beautiful documentation site, then I realized there is an actual product inspiring it's development (https://presentator.io/ https://presentator.io/) Each of these is another encouragement to use pocketbase. I went from "eh, I'll just keep my own wrapper" to "pocketbase has a lot of promise, I should try it"

codegeek 4y ago

Wow, super impressed. I have always wanted to build something like this where everything is embedded in 1 single file and I knew Go could be a great way to do it but always wondered how to handle the database part. Along comes SQLlite. One feedback which will probably deviate from everything in 1 file concept but adding an option to upload files to an S3 URL will be awesome. Perhaps a pull request ? I am thinking of use cases like document management using this and if I can have the actual files in S3 (or S3 compatible), that would be a huge plus. EDIT: I spoke too soon. You already have an option for S3 as I tested the demo. Very cool. How much work went into this btw ? The github, documentation everything is well done. I envy work like this :)

endofreach 4y ago
Uploaded files must deviate from „everything in one file“ regardless of using S3, right? How (and why) would you store user uploads within the binary?
- codegeek 4y ago
  yea it was a silly question. It does allow S3 anyway so we are good.
  - endofreach 4y ago
    My intent was not to point out a „silly question“. I was legitimately interested if there is a reasoning and what a use case for that would be. Thank you for responding nicely!
- randomwebdev 4y ago
  Uploaded files and your database are stored outside of the binary.
randomwebdev 4y ago
By default PocketBase uses the local filesystem, but you could also configure a S3 compatible storage (AWS, DigitalOcean, Vultr, etc.) from "Settings > Files Storage > Use S3 storage". I've started working on the project ~ 3-4 months ago. The last 4 weeks I've taken a longer break from my day-to-day job so that I could finalize the first release.

gervwyk 4y ago

Well done! Really impressive, will give it a spin.

Pr0ject217 4y ago

Potentially silly question (I'll go out on a limb anyway). I would like to create a multi-platform app for non-technical users where their data is transparently self-contained/self-hosted. So, they open a single binary and it presents them with a UI, and the data created from within the app would be stored in their local PocketBase database. However, I wouldn't want the user to have to start a server directly (they wouldn't know what a server is). So, I think the binary would have to start both the PocketBase server and a front-end server, and then launch a browser (similar to Electron) pointed to their front-end server's port. Does this sound like a good approach? What could you recommend to orchestrate this seamlessly to make for a simple user experience when launching the app? Thank you.

qbasic_forever 4y ago
Honestly I'd look at using Electron because you can take a hard dependency on the browser capabilities and state. If you rely on the user bringing their own browser you're going to have headaches of people on ancient browsers or just supremely jacked up browsers (messed up browser plugins, proxies, turned off javascript, etc.). Node can easily launch processes and there are even fancy process management libraries like pm2 you can add as a dependency should you need it, so the logic in your electron layer can be very simple and just launch some other processes and ferry requests to the view layer. Alternatively a webview-based framework or something like Lorca might be another option if you don't want to use Electron: https://maori.geek.nz/golang-desktop-app-webview-vs-lorca-vs-electron-a5e6b2869391 https://maori.geek.nz/golang-desktop-app-webview-vs-lorca-vs...
- Pr0ject217 4y ago
  Thank you!
randomwebdev 4y ago
Hm, there is nothing preventing you to start multiple services from a single binary. I'm not sure that PocketBase would be suitable for your use case, but it is distributed also as a Go package (see https://pocketbase.io/docs/use-as-framework/ https://pocketbase.io/docs/use-as-framework/) and you could combine it with any other GUI Go package (fyne, go-astilectron, etc.)

melony 4y ago

This looks amazing, congrats on shipping! Do you plan to add Firebase-style query optimizations too? https://news.ycombinator.com/item?id=31832266 https://news.ycombinator.com/item?id=31832266

roguas 4y ago

I would love it. But: postgres, +backups, +replica managment. Basically a managed database for postgres in a container.

_query 4y ago
Check out https://thin.dev/ https://thin.dev/ :) It's similar, supports self-hosting and uses postgres. Quick demo video here: https://www.youtube.com/watch?v=-jj19fpkd2c&t=3s https://www.youtube.com/watch?v=-jj19fpkd2c&t=3s (I'm founder of Thin)
randomwebdev 4y ago
You may want to check Supabase or Nhost in that case. PocketBase was specifically designed to be self-contained and running only on a single server.

ezekiel11 4y ago

why is this written in Go , are there specific features that shine over Python3? I've been looking to learn Go, it seems lot of network intensive software use Go.

rekwah 4y ago
If a selling point is run from "one file", that's quite difficult to do in python. There are things like pyinstaller but you end up shipping the entire interpreter in the bundle.
- mypalmike 4y ago
  The pex tool lets you build single file python executables which are well suited for server deployments.
  - ezekiel11 4y ago
    oh wow thanks i didn't know about this. in that case it would be super simple for me to wrap up a FastAPI server with SQLite or even some in-memory database that runs elsewhere, deliver as a single file!
    - killingtime74 4y ago
      You could also just use Docker to wrap anything
    - mypalmike 4y ago
      Yeah this kind of thing was common when I worked at Twitter (where AFAIK pex/pants was developed) on an infrastructure team. It's a cool tool that few outside of Twitter seem to be aware of.
  - mhio 4y ago
    How big is a python3 runtime when pex has packaged it up? And do you happen to know what the output binary is linked to?

beardicus 4y ago

this looks really great! i had recently been thinking “boy it’d be really great if there were a ‘supabase but sqlite’ available” and here you are! now i just need a clever project as an excuse to use this. 1 question: are the docs in a repo anywhere? i don’t see them in the main repo (i might just be missing them somehow) or anywhere in the org. if they are public somewhere would you be open to a docs polishing PR?

randomwebdev 4y ago
The documentation is not auto generated and is part of the static landing site (built with SvelteKit). I didn't bother open source it because it is kinda messy and a little complicated to be edited by users not familiar with the codebase, but I'll try to find some time in the future and may publish it. In the meantime, if you find typos or think that some of the wording could be improved, feel free to open an issue or discussion in the main repo and I'll fix them.

1xdevloper 4y ago

This looks perfect for storing the internal data of my side project! Love the website design too.

vincentge 4y ago

This actually looks suuuupppppeeeerrrr cool!!!

deleted 4y ago

[deleted]

segphault 4y ago

This looks like a great solution for rapid prototyping scenarios where you want a basic CRUD backend with as little friction as possible. The API preview feature in the admin dashboard is a really nice touch.

catchmeifyoucan 4y ago

This looks great! Does this have support for offline usage on the client?

ptudan 4y ago

This looks awesome! I will definitely give it a try. Right now I use postgres in a docker compose but its definitely a bit heavier for single machine prototyping. One thing that I like about my setup though is that there's a clear migration path to horizontally scale. Do you have a recommended way to move past pocketbase should an app get to that point?

randomwebdev 4y ago
I guess there will be differences from use case to use case, but in general migrating from PocketBase to a more common horizontal supported stack is not different from rewriting your application. Internally, each Collection creates a standard SQLite table that holds the collection records, so migrating the data structure shouldn't be too troublesome. The only thing that may prove difficult to migrate could be reimplementing the access rules and filters. But in my opinion, when your application reaches that level of growth requiring multiple servers and services, your business use cases mostly likely will have changed several times already from your initial idea.
felipeccastro 4y ago
Perhaps sharding data by geographical location could be one path - e.g. deploy one server in the US and another in Europe, each with its own database. The whole concept of edge computing - deploying app servers closer to your users for minimal network latency - seems to go hand in hand with using an embedded database like SQLite. (I haven't tried it in production yet, but looking forward to.)

codeptualize 4y ago

Looks amazing! Love the idea of a Firebase/Supabase like thing I can easily run locally and deploy as well. Also props for the dashboard design, looks and works very nice!

jitl 4y ago

This looks really great. Have you thought about adding Operational Transform or CRDT primitives to your offering? Realtime subscriptions are nice - but they're even better with realtime collaboration. Because you're using SQLite directly, it's easy(er) for you to build in efficient CRDT mutations to your database rows because SQLite's BLOB column supports incremental IO: https://www.sqlite.org/c3ref/blob_open.html https://www.sqlite.org/c3ref/blob_open.html This could allow you to save a CRDT update operation without needing to read the entire CRDT blob into memory, parse it, and then save it all back to disk.

randomwebdev 4y ago
Currently any additional data transformation is left to the developers to extend via event hooks or custom client side handling. I've worked with CRDT in the past (yjs), but it may not be very useful in PocketBase considering that the application was designed to run on a single server and db writes are practically queued (you can have only one sqlite writer at a time). I'll investigate it further and may consider it for future release.
- jitl 4y ago
  I think if you integrate Yjs or a OT library first-party, you'll have a really exceptional framework - open source firebase alternative top-to-bottom, ready for "local-first" collaborate own-your-own-data apps. Sounds like my dream stack for small-scale projects or friend/family apps.

abraxas 4y ago

Nice but my super anal quip is please don't call this thing "realtime". It doesn't make sense in this context. What is realtime about it? Are there timing guarantees or are timing constraints provided in API requests? Because that's what realtime systems are. Realtime does not mean "real fast". For example, a train track switching system might be getting run by a 1MHz CPU and still be real time. It's not about speed, it's about deadline guarantees.

willy_k 4y ago
It’s not a real-time operating system. In this case, real-time means it is built such that a client gets access to updated database values as soon as they change, as opposed to a non real-time database which doesn’t necessarily have a built-in mechanism for reliably on time updates.
- abraxas 4y ago
  You mean like eventual consistency vs immediate consistency? Because a database that reflects its changes at the end of a transaction is called an ACID database not a real-time database.

euroderf 4y ago

Suitable as the basis for a CMS ?

randomwebdev 4y ago
The existing admin interface could be used as a very rudimentary content management, but it was really intended for quick internal data explorations and edits. Additionally, currently there are no roles for admins so permission controls within the admin area are not possible. You could always implement your own custom CMS SPA frontend that interacts with the PocketBase API (that is how I'm planning to use it with the next version of my other open source project - Presentator).
- euroderf 4y ago
  Maybe I open an issue for discussion. Mainly I wonder about making architectures mesh.

999900000999 4y ago

Any client SDKs for Android, iOS, React, etc.

randomwebdev 4y ago
There is already a JavaScript SDK (https://github.com/pocketbase/js-sdk https://github.com/pocketbase/js-sdk) that works both in the browser and node. Dart SDK (targeting mainly Flutter) is also on the way. Unfortunately I'm not very uptodate with Android (Kotlin) or iOS (Swift) development and therefore community contributions are more than welcomed.
- 999900000999 4y ago
  This is much more than I expected, thank you !

ilrwbwrkhv 4y ago

This is fantastic work. Supabase local development has always been a pain.

wenbo 4y ago
Member of the Supabase team! Would love to hear your pain points with local development. Always looking for feedback so that we may improve your experience.
- ilrwbwrkhv 4y ago
  Well all the UI stuff. It is super easy to add a column and change something fast with the UI when doing rapid development. But that piece is missing when developing locally. I have gone on to create 2 projects (one called local and one prod) and manually applying the changes. Would be nice to apply changes from one project to another. PocketBase here for example provides the UI locally too. That is the real selling point IMO.
  - wenbo 4y ago
    Thanks for the feedback! Passed it on to our frontend team.

benkaiser 4y ago

For anyone curious, another popular open-source backend as a service is AppWrite: https://appwrite.io/ https://appwrite.io/ I've tried them out for some dummy projects and they seem pretty great, and they are continually iterating and pushing new features.

dinvlad 4y ago

So good to see a renaissance of "portable apps", now on the web. All of this extra complexity associated with multi-tiered multi-microservice arch is not needed for many categories of apps, and especially for small teams.

traviscj 4y ago

Really neat! I’ve been wanting to make something similar for _years_ to be a standardized data modeling testbed as I explore/experiment/prototype some new feature or problem space or even doing ad-hoc fixes. I think this is a much better starting place than spreadsheets/similar. Good luck with the project! Seems like you’ve got some good interest :-)

newusertoday 4y ago

This is great, if it was available a year ago it would have saved me lot of time in creating my home-grown solution ;)

lexx 4y ago

This is a really nice idea and I love the execution as well. Congrats and thanks for sharing.

Wronnay 4y ago

I really like this. All my current projects use Go as a backend and I definitely need my own routes and business logic, so PocketBase looks way easier to integrate into my existing code than the JavaScript based solutions. And it would also give me more time to focus on business logic instead of Auth - i will definitely try this in the future.

h3rald 4y ago

Nice project! It looks pretty similar to my own LiteStore: https://h3rald.com/litestore/ https://h3rald.com/litestore/ The feature set is pretty similar, but I focused mainly on the free form/nosql-ish aspect of it, and although I added support for server-side JavaScript middleware I didn't include authentication OOTB (although it could be implemented or you can integrate with an OAuth2 system - it can validate JWT tokens). And yes, LiteStore's web app hasn't been updated in years and overall your project looks much more polished than mine. Good job! Bookmarking ;)

resoluteteeth 4y ago
I notice that it has full text search support, too, which pocketbase doesn't seem to.