29 ms·
I was responsible for Stripe's API abstractions, including webhooks and /events, for a number of years. Some interesting tidbits: Many large customers eventual
by bkrausz 5y ago
I was responsible for Stripe's API abstractions, including webhooks and /events, for a number of years. Some interesting tidbits:
Many large customers eventually had some issue with webhooks that required intervention. Stripe retries webhooks that fail for up to 3 days: I remember $large_customer coming back from a 3 day weekend and discovering that they had pushed bad code and failed to process some webhooks. We'd often get requests to retry all failed webhooks in a time period. The best customers would have infrastructure to do this themselves off of /v1/events, though this was unfortunately rare.
The biggest challenges with webhooks:
- Delivery: some customer timing out connections for 30s causing the queues to get backed up (Stripe was much smaller back then).
- Versioning: synchronous API requests can use a version specified in the request, but webhooks, by virtue of rendering the object and showing its changed values (there was a `previous_attributes` hash), need to be rendered to a specific version. This made upgrading API versions hard for customers.
There was constant discussion about building some non-webhook pathway for events, but they all have challenges and webhooks + /v1/events were both simple enough for smaller customers and workable for larger customers.
- alexgartrell 5y agoI think the Stripe API stuff you did was fine, but you really did your best work as a concepts of mathematics TA.
- AtNightWeCode 5y agoThis is a fake post? Never encountered this level of incompetence in real life.
- spullara 5y agoPretty easy for a customer to setup an SQS queue and a lambda for receiving them rather than rely on their infrastructure to do all the actual receiving. Way more reliable than coupling your code directly to the callback.
- jon-wood 5y agoThis is precisely what we do where I work. We have a service which has just one responsibility - receive webhooks, do very basic validation that their legitimate, then ship the payload off to an SQS queue for processing. Doing it this way means that whatever’s going on in the service that wants the data, the webhooks get delivered, and we don’t have to worry about how 3rd party X have configured retries.
- throwaway290232 5y agoI always laugh when people end up with designs like this. They could have just used SMTP! It's designed to reliably deliver messages to distributed queues using a loosely-coupled interface while still being extensible. It scales to massive amounts of traffic. It's highly failure-resistant and will retry operations in various scenarios. And it's bi-directional. But it's not "cool" technology or "web-based" so developers won't consider it. Watch me get downvoted like crazy by all Nodejs developers. Even though they could accomplish exactly what they want with much less code and far less complex systems to maintain.
- oalae5niMiel7qu 5y agoSMTP doesn't reliably deliver messages, implementations of it do. A webshit could easily create an SMTP server (with the help of a library written by someone with actual programming skills) that silently drops messages when any error occurs instead of implementing all that robustness.
- deleted 5y ago[deleted]
- aidenn0 5y agoSMTP no longer reliably delivers messages. Try setting up an MTA on a Hetzner VPS and see how many messages get through
- vidarh 5y agoThat is only relevant if you require delivery to arbitrary endpoints rather than to endpoints explicitly set up to process your messages.
- mrtesthah 5y agoThat's not an applicable criticism for SMTP running on a private network and/or dedicated set of "mail" submitting servers, as in the specific model outlined in the grandparent comment.
- vidarh 5y ago
- ctas 5y agoCan you share a bit about how these events are stored on Stripes backend e.g. Kafka, Postgres?
- bastawhiz 5y agoIt's all just kafka and mongo. The event can be stored in any simple k/v storage. There's no magic. Edit: not sure why I'm being downvoted. I work at stripe and this is literally how it works.
- bkrausz 5y agoHi Basta! Can confirm both that he works at Stripe and is right. Years ago there wasn't even a Kafka portion, that's newer.
- ctas 5y agoThanks for the input. We're currently working on a similar solution, so I was really curious to learn more. One thing I really admire is how Stripe makes it transparent which events were fired both in general through the Developer area, and on specific objects like customers, subscriptions, etc..
- alexbouchard 5y agoShameless plug but I've built https://hookdeck.com https://hookdeck.com precisely to tackle some of these problems. It generally falls onto the consumer to build the necessary tools to process webhooks reliably. I'm trying to give everyone the opportunity to be the "best customers" as you are describing them. Stripe is big inspiration for the work.
- Redsquare 5y agoDo you provide the ability to consume, translate then forward? I am after a ubiquitous endpoint i can point webhooks at and then translate to the schema of another service and send on. You could then share these 'recipes' and allow customers to reuse well known transforms.
- alexbouchard 5y agoTransformations is something we haven't built yet but we have our eyes on it as your are not the first one to bring that up. You can use Hookdeck in front of lamda and to the transformation there, you'd still get the benefit of async processing, retries, etc
- rkazokas 5y agoDo you have an idea of when hookdeck will have transformations. It's not something we need immediately but would be the win over something like: https://webhookrelay.com/ https://webhookrelay.com/ if it's something you have on your roadmap for sometime soon.
- stympy 5y agoMy app, hookrelay.dev, has transformations today. :)
- alexbouchard 5y agoCan you reach out to me, I'd love to talk about your use case and prioritize accordingly. Email is alex at hookdeck dot com
- Eiriksmal 5y agoAt Lawn Love, we naively coupled our listening code directly to the Stripe webhook... but it worked flawlessly for years. I wasn't a big fan of the product changes necessitating us switching from the Transfer API for sending money to the complicated--and very confusing for the lawn pros--Connect product, but its webhooks also ran without issue from the moment we first implemented them. So thanks for making my life somewhat easier, Mr. Krausz. Like many others, I now pattern my own APIs after Stripe's.
- bkrausz 5y agoDon’t fully thank me, I was also the architect of the Transfers API to Connect transition :). There’s a lot I would have done differently there were I doing it again, though much of the complexity (e.g. the async verification webhooks) were to satisfy compliance needs. Hard to say how much easier the v1 could’ve been given the constraints at the time, though I’m very impressed with the work Stripe has done since to make paying people easier (particularly Express).
- rattray 5y ago> We'd often get requests to retry all failed webhooks in a time period. (I worked on the same team as bkraus, non-concurrently). For teams that are building webhooks into your API, I'd recommend including UI to view webhook attempts and resend them individually or in bulk by date range. Your customers are guaranteed to have a bad deploy at some point.
- tasn 5y agoThese reasons are exactly why we started Svix[1] (we do webhooks as a service). I wish we existed to serve you guys back when you started working on it. :) [1] https://www.svix.com https://www.svix.com