hruk

5 ms·

1.
by hruk 9mo ago
bruh
2.
by hruk 11mo ago
Roughly 80/20 read to write. On the instance's gp3 EBS volume (which is pretty slow), we've pushed ~700 write transactions per second without much problem.
3.
by hruk 11mo ago
This is just untrue - the naive implementation (make the API call, write a single row to the db) will work fine, as transactions are quite fast on modern hardware. What do you consider "serious" work? We've served a SaaS prod
4.
by hruk 1y ago
This is basically Breiman's "two cultures" at play. Do you care about optimizing y-hat, or do you care about doing inference on some parameters in your model? Depends on the business case, typically.
5.
by hruk 1y ago
You can do fairly well here with ridge regression as a poor man's hierarchical model. We've used this library's Bayesian ridge regression to support a geo-pricing strategy (and it contains the Dirichlet-Multinomial approach a
6.
by hruk 1y ago
All databases are painful in their own way. I've used all three at various times in my career, and I think SQLite behaves quite predictably, which has made it a lot easier for me personally to administrate. If I had to start something
7.
by hruk 1y ago
Agree on many things here, but SQLite does support WAL mode which supports 1 writer/N writer readers with snapshot isolation on reads. Writes are serialized but still quite fast. SQLite (actually SQL-ite, like a mineral) maybe be light
8.
by hruk 1y ago
We've used this Python package to do this: https://github.com/bayesianbandits/bayesianbandits
9.
by hruk 1y ago
Yep, we use Litestream. It's been very reliable.
10.
by hruk 1y ago
FWIW, I've been running a system with roughly 100K users, about 25 qps on average, with a single SQLite file for several years. No issues with data.
11.
by hruk 2y ago
We've used this library for Bayesian contextual bandits in production (we have a critical business use case supported by a ~200K feature sparse Linear UCB bandit). It's a small community, but it's also a small enough codebase
12.
by hruk 2y ago
We do zero-downtime deployments with a single Docker volume containing the db. Spin up a container running the new code, wait til it's healthy, then kill the old container.
13.
by hruk 2y ago
We have some analytics queries powered by Airflow running on a separate box. We set up Litestream to replicate to that box via SFTP, and then dump changes hourly into Snowflake.
14.
by hruk 2y ago
You are still correct for network filesystems, but there are no issues opening multiple connections on the same host (even across OCI container boundaries).
15.
by hruk 2y ago
You can have as many connections from as many processes as you want.
16.
by hruk 2y ago
Yes, we've used the Litestream + SQLite + Go webserver route in production for several years and while we've had to deal with all of these issues, on the whole I don't have a strong argument for going back and doing it differ
17.
by hruk 2y ago
No, you definitely have to pick your battles. Something that you want to continuously optimize over time makes a lot more sense than something where it's reasonable to test and the commit to a path forever.
18.
by hruk 2y ago
Indeed, we are well aware.
19.
by hruk 2y ago
I don't know, all of these are pretty surmountable. We've done dynamic pricing with contextual multi-armed bandits, in which each context gets a single decision per time block and gross profit is summed up at the end of each block
20.
by hruk 2y ago
We've been happy using Thompson sampling in production with this library https://github.com/bayesianbandits/bayesianbandits
21.
by hruk 2y ago
Nah, just SQLite. I had to learn a number of things about good SQLite defaults and habits, but that was years back before it really started to take off in this space. Now, there are tons of blogs you can refer to giving reasonable advice. T
22.
by hruk 2y ago
I use SQLite for this sort of thing (small e-commerce storefront, wife's niche popular blog, etc.). Actually, I use multiple - one for the DB and Diskcache for the cache. Caching is cheap, backup is trivial and cheap, and pulling down
23.
by hruk 2y ago
Indeed. I've found that as long as you store your database on a modern NVMe drive, you can easily push 5K+ write transactions per second with SQLite. I've worked at very large companies who received an order of magnitude fewer wri
24.
by hruk 2y ago
You can have multiple containers on the same host use the same db via Docker volumes - we have a number of production services set up this way.
25.
by hruk 2y ago
I've used HTMX quite frequently recently for internal frontends (admin pages, reporting, etc.). It's quite good. A pattern I've been using a lot is not changing the backend at all - each endpoint still returns the full page.
26.
by hruk 2y ago
S3, yes. Litestream doesn't store the backup as a single DB file - rather, as snapshots and WAL segments that are streaming in, so it's unfortunately not so simple.
27.
by hruk 2y ago
I have not, but I keep meaning to collate everything I've learned into a set of useful defaults just to remind myself what settings I should be enabling and why. Regarding Litestream, I learned pretty much all I know from their documen
28.
by hruk 2y ago
Interesting question. Our services currently contain "own" all of cron jobs, so the jobs run as background tasks on the host containing the db. I think we used Huey for the Django app, and just some CLI applications fired via syst
29.
by hruk 2y ago
Yes, we use it to back several services in production at an 8 figure ARR business, with Litestream as our streaming backup option. I'm quite happy with it because it's trivial to debug and test against a local copy of the producti
30.
by hruk 2y ago
p99.9 referring to latency. However, we also do a weekly test of how quickly we recover from a catastrophic crash, which is roughly about 6 minutes (which is the amount of time it takes for the autoscaling group to spin up a new host, Lites

More ›