5 ms·

Interesting use-case for it. Without prior knowledge of a solution like this I would have suggested you send the webhooks to a queue backed notification system

Interesting use-case for it. Without prior knowledge of a solution like this I would have suggested you send the webhooks to a queue backed notification system (e.g. SNS backed by SQS) and subscribe to the event topic, but sounds must easier to configure and manage the way you instrumented it. Might be a good use-case for me to try out!

cameroncooper 7y ago
This is something you can easily configure with our automatic retry function. We have an option to return a pre-configured response to the caller, and put the request in a queue to be retried until successful. This allows you to have a sustained outage while making sure all calls are eventually delivered.
- ignoramous 7y ago
  > This allows you to have a sustained outage while making sure... Re-driving queue backlogs at services recovering from sustained outages ends in tears almost always. Tread carefully. :)
  - jrockway 7y ago
    Typically people use two pools for circuit breaking, with the limit set lower on retries: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking#arch-overview-circuit-break-cluster-maximum-connection-pools https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...
capableweb 7y ago
Yeah, this is what I've seen most services who rely on webhooks from another service to do. Add in some monitoring of how many events are not yet processed (set a alarm when there is X amount of events in it) and you're done!
- disposedtrolley 7y ago
  We're currently building a GitHub integration which receives webhooks and kicks off a bunch of processing actions based on the event type. Your suggestion sounds like a great way to add some observability to the service -- thanks!