james_cowling

7 ms·

1.
by james_cowling 2y ago
I'm giving the first talk of the session and while I'll be talking about ideas we've actually built and tested in Convex, the point I want to get across is that I think the industry has gone in the wrong direction with exposi
2.
by james_cowling 2y ago
Yes the readsets aren't based on lists of objects but rather the data ranges that a query touches, i.e., they don't suffer from phantom read anomalies. If you were to attempt to fetch all items from an empty shopping cart, that qu
3.
by james_cowling 2y ago
Convex is FSL licensed which means you can basically do whatever you want with it re running a service, other than directly competing with Convex, i.e., reselling the Convex platform. There are no limitations on you running a business on to
4.
by james_cowling 2y ago
(convex cofounder here) Convex is in a very comfortable financial position but it is also open source: https://github.com/get-convex/convex-backend
5.
by james_cowling 2y ago
Convex cofounder here. Yep the database transaction logic, timestamp assignment, concurrency control, etc is all custom. This is a pretty key requirement in our ability to perform dynamic subscriptions and automatic caching in real time. In
6.
by james_cowling 3y ago
Convex cofounder here. Initial focus has been low-latency OLTP database workloads, which is pretty tied to the user-facing end-to-end reactivity angle we're pushing. As a bonus feature we also have built-in strongly consistent full tex
7.
by james_cowling 3y ago
(kmod was the founding engineer on the project and the guy who came up with the name!)
8.
by james_cowling 3y ago
We should also clarify that Sujay and I are part of the old old team :) The current team have been doing an awesome job since then! (a bunch of the early MP folks work at Convex now)
9.
by james_cowling 3y ago
"It’s fairly easy to design a system with astronomically high durability numbers. 24 nines is a mean time to failure of 1,000,000,000,000,000,000,000,000 years. When your MTTF dwarfs the age of the universe then it might be time to ree
10.
by james_cowling 3y ago
We had a lot of requests for open sourcing MP but I don't think it would have been very useful. The code itself is interesting to look at but the system is designed for very large data sets, has a lot of moving parts to manage and co
11.
by james_cowling 3y ago
Yeah we have a few talks about this and a chapter about this very issue in https://www.oreilly.com/library/view/seeking-sre/97814919788... . Totally agree that in a well designed system the sources of data los
12.
by james_cowling 3y ago
We designed and built the system almost completely ground-up, all the way down to the disk scheduler, but yeah we probably took the term OSD from Ceph.
13.
by james_cowling 4y ago
Yep that’s me! I have my full name on here so wasn’t hiding but good point to be clear about disclosing.
14.
by james_cowling 4y ago
Firebase has been a huge influence but with the rise of serverless there are a new generation of platforms people should be checking out instead, such as Convex, Supabase, etc. Convex in particular is designed far more for end-to-end cons
15.
by james_cowling 4y ago
We track the readset for any active subscription. When a new write transaction commits we compare the writeset for the transaction against any active subscriptions. If there's an intersection then it's likely that query will have
16.
by james_cowling 4y ago
Thank you! We have a really talented team working with us but we're very actively growing. If anyone is looking for a job in San Francisco: https://www.convex.dev/jobs
17.
by james_cowling 4y ago
This is very nice feedback. The short answer is that we want to solve problems that matter for the people who need them solved.
18.
by james_cowling 4y ago
Thanks for the feedback. We think React is the right community to target right now but the end state for Convex is certainly not limited to React. The specific concern with regards to multiple round trips is scenarios where the client needs
19.
by james_cowling 4y ago
I'm not sure what your definition of smart polling is but there is no polling going on here - the query only reruns when the data dependencies change server-side due to a subsequent mutation. Another key distinction is that "query
20.
by james_cowling 5y ago
Obviously depends on the kind of projects you're working on but in our experience we've been more productive in Rust compared to when we've built large systems in Python, Go, C and C++. This is with regards to long-term produ
21.
by james_cowling 6y ago
I've been using Valora for a while now and even though I discovered it through the crypto world, I use it because it solves an actual problem for me. It's been a great solution for venmo-style payments with friends in different co
22.
by james_cowling 10y ago
Thanks for the feedback! Glad folks are reading this stuff :) > Would you be able to give more details as to how you use disks as raw block devices, bypassing the filesystem? For example, the kind of abstraction layers you use, their int
23.
by james_cowling 10y ago
Yup we do this, and yes, the verifier has been broken before (during early development, not at production scale). We do a number of things here, like taking down a sufficient number of storage nodes in a single region to make blocks appear
24.
by james_cowling 10y ago
For sure. I hope we're all agreeing here :) We've very strong believers that an effective replication strategy is just table stakes and that from there the real risks to durability are the "black swan" events that are mu
25.
by james_cowling 10y ago
From what I hear from other major storage providers Magic Pocket is definitely very high up on the paranoia/verification scale. There is of course a cost to all this verification traffic. Any sensible storage provider will be doing stu
26.
by james_cowling 10y ago
Yeah to be clear my point from the article is that even though you can model an incredibly high durability number in theory, there's a lot more work required to see that in practice. I'm sure the S3 team also have extremely high i
27.
by james_cowling 10y ago
Hi folks, James from Dropbox here. Happy to answer any questions that come up so feel free to send them by way.
28.
by james_cowling 10y ago
Thanks! It's always a bit of a tradeoff. A 512MB bucket size would double the size of the Replication Table and we want make sure we keep the entire working set in the buffer cache. 1GB also has the advantage of filling up slower, whic
29.
by james_cowling 10y ago
This is a really great question since the vast majority of the work was in ensuring correctness and reliability: everything from testing discipline to fault injection to auditing. This also included hardware testing, like pulling out circui
30.
by james_cowling 10y ago
To be clear, we do need those durability and availability guarantees :) We just benefit from our very-large scale and close alignment between hardware and software to tailor our infrastructure to our workloads.

More ›