4 ms·

I don't think creating the pipeline is the difficult part at all. When doing this kind of work I think more about things like: - Designing usable APIs, which i

• fleitz 15y ago
  These are largely trees for the forrest type problems. The first problem is the same as the second. (eg. which data (errors ARE data) where) Make all your services unreliable and it solves 90% of your issues as then unreliable services aren't special. Performance issues in data generally stem from two places (underspec'd hardware) and latency. Which are usually solved best by buying better hardware (more drives not more CPU) and buying a network nightmare box and putting 400ms of latency in the network. People stop designing chatty APIs pretty quickly at 400ms of latency.
  • goldmab 15y ago
    I don't understand what you're saying. The first problem is similar to the second, yes. They both still need to be solved, so what is your point? Unreliable services are everywhere. So what's your universal solution for dealing with them? I'll give a concrete example: suppose I have a JSON service being consumed by a Javascript web UI, and my service needs to hit some kind of authentication backend. In the event that the authentication backend server is [pick one: down, giving 500 errors, being slow], what kind of response does my service give to the Javascript app, and what kind of message or visual cue does the app give to the user? If you think the answer is something other than "it depends on exactly what the application does", then I disagree. If you think latency is the problem, why are you talking about building in latency? It seems like you actually think chatty APIs are the problem. And, yes, chatty APIs can cause slowness. But chatty APIs often exist because they are the simplest possible design. Once you realize that there is too much back-and-forth you may have to sacrifice API usability and simplicity by adding caching and eager-loading code. Again, you think this is just something that solves itself?
    • fleitz 15y ago
      That particular problem is a trinary response, true,false,error, or in general a datatype of: type Response<d,e> = | Data of d | Error of e which would be handled by a statement like: match response with | Data d -> doSomething(d) | Error e -> doSomethingElse(e) Or perhaps match response with | Data d -> Some(d) | Error -> None Any errors coalesce to error on the client and the client responds: "We're sorry this doesn't work, we've been notified and are investigating, here's your ticket #" Each system in the chain should have a reference identifier to make tracking the requests across the system easy. Chatty APIs often exist because a lot of programmers think that everything happens at the same speed and they think that having a getter and accessor for every field makes their code "object oriented". Putting in 400ms latency gets programmers to stop thinking that, it gets them thinking about "How can I issue a bunch of requests simultaneously, go do something else (like issuing more requests for someone else), and then respond to the client when I have all the data I need". It gets them writing async code, or using MARS. Maybe, 400ms is really excessive, but 100ms should still let your code run on systems with reasonable geographic separation. Chatty APIs aren't simple, they're generally really annoying, because for decades the predominant idea in programming has been put as thin a veneer on top of the implementation as possible and lets call that an interface. It makes for a simple implementation at the expense of a horrible interface. APIs are about the interface.
      • deleted 15y ago
        [deleted]