7 ms·

When REST Gets Messy

74 points by brian_cooksey 12y ago

leeoniya 12y ago
i've been meaning to write something similar about the whole REST craze. REST breaks down pretty rapidly once you get out of the key-value-store paradigm (read: anything involving child objects). REST lacks the ability to relay full state-change semantics without hackery. As the article pointed out, it forces you to be extra chatty over http, which is far from free over the congested, global network that is the interwebs. For example, what if a single PUT request creates multiple sub-objects? How does my server reply with multiple location headers? Do i have to first re-get the created object's child locations and re-request each individually? How about just sending the full object state back as the response to my PUT request? Well, according to REST, the body of the response just needs to be a description of the error or success status. Basically, REST sucks for reducing round trips if you're going to follow it pedantically. The theory is sound, but it needs to be updated to dictate how the server can send back more detailed info in response to POST/PUT/PATCH requests. /rant
- rdtsc 12y ago
  > How about just sending the full object state back as the response to my PUT request? That is what I do. I don't see anything wrong with it. PUT effectively replaces the state of the resource so you get it back right away (get the latest state, which should be what was sent in the request, minus a conflict for example if multiple clients do it or you have some incrementing revision id thing) > For example, what if a single PUT request creates multiple sub-objects? I'd use POST for creating/adding objects. I think of PUT usually as idempotent and as I mentioned above, use it to replace the state of the resource. So if this one POST ends up creating multiple-sub-objects you can return a json object that encapsulates URIs do those new objects. Remember resources don't have to map to internal objects, db rows or other such things. You can have the objects or db rows represented or used in different resources. If it makes sense and if you have this transactional interface, maybe it makes sense to have an explicit transaction resource that is in charge of managing a transaction (where multiple things happen and then it kind of becomes very explicit).
- jessaustin 12y ago
  How about just sending the full object state back as the response to my PUT request? Well, according to REST, the body of the response just needs to be a description of the error or success status. This isn't strictly true: If the target resource does not have a current representation and the PUT successfully creates one, then the origin server MUST inform the user agent by sending a 201 (Created) response. If the target resource does have a current representation and that representation is successfully modified in accordance with the state of the enclosed representation, then the origin server MUST send either a 200 (OK) or a 204 (No Content) response to indicate successful completion of the request. [0] Both the 200 and 201 payloads may include anything you want, including a giant representation with many constituent parts (which parts would ideally each contain a link to its respective associated resource). [0] http://tools.ietf.org/html/rfc7231#page-27 http://tools.ietf.org/html/rfc7231#page-27
- dragonwriter 12y ago
  > For example, what if a single PUT request creates multiple sub-objects? If a PUT, POST, or PATCH does that, then it does that. So what? > How does my server reply with multiple location headers? It doesn't respond with multiple location headers. With PUT or POST, with an 201 Created status response with the Location: header containing the URI of the parent object (the resource directly created by the PUT/POST), and an entity body, in some appropriate format, containing the URIs of all the created objects. (If there is a representation of the main resource which would do this that is acceptable to the client, this would seem to be an ideal way of communicating that.) With PATCH, much the same, but with 200 OK. > Do i have to first re-get the created object's child locations and re-request each individually? There's no reason for a REST API to require that. > How about just sending the full object state back as the response to my PUT request? I see no reason why sending a full resource representation in a 201 status response, provided that the client has indicated that they are willing to accept the relevant media type, is problematic, especially if it does what the HTTP/1.1 spec says the 201 response entity should do. (The one thing that common resource representations might not do that they should is provide their own location and the direct locations of any embedded subentities that are separate addressable, but there is no reason why a resource representation couldn't do that, and I'd argue that in REST it would be desirable that resource representations do do that.) > Well, according to REST, the body of the response just needs to be a description of the error or success status. As, stated, that's accurate, but you seem to be acting as if that was "needs to be just" rather than "just needs to be" (e.g., misunderstanding it as a maximum allowed content, rather than a minimum required content.) > Basically, REST sucks for reducing round trips if you're going to follow it pedantically. Except that none of the problems you've pointed to in that regard have anything to do with REST.
  - T-R 12y ago
    > I'd argue that in REST it would be desirable that resource representations do do that (provide [...] the direct locations of any embedded subentities that are separate addressable). You could actually go a step further - if you're using "Hypertext as the engine of application state" (HATEOAS), URIs for the locations of direct subentities need to be there, if the client is expected to be able to make those state transitions, for the API to be "fully RESTful". (Though, I'd personally agree with the article that features like discoverability and content negotiation are secondary to those you get from idempotence/properly using HTTP methods, and feel they should be considered bonus features, rather than strict requirements)
je42 12y ago
Thinking about this: By merging children into the parent resource, he traded a reduction of requests with the loss of separation of concerns. How about adding a layer instead whose only concern is the reduction of the number of requests for the client. It would collect and merge the info from parent and children resources and return it in merged form to the client ?
- lumpypua 12y ago
  That would probably actually work pretty well. I've run into a similar problem a number of times and I may try this. "All problems in computer science can be solved by another level of indirection" - David Wheeler "...except for the problem of too many layers of indirection." - Kevlin Henney
- pbh101 12y ago
  Netflix has apparently called this idea an 'orchestration layer': http://thenextweb.com/dd/2013/12/17/future-api-design-orchestration-layer/ http://thenextweb.com/dd/2013/12/17/future-api-design-orches...
- T-R 12y ago
  I really do wish HTTP had a mechanism for responding to a single request with multiple combined response bodies as if requests were made for each individually (from the perspective of, e.g., a caching proxy) - the loss of separation of concerns from merging children into the parent isn't just a usability issue, it also means you lose your cache granularity - the cache for the parent object is now stale whenever a child is modified, and a request for a child after getting the parent won't hit the cache at the request level. There's keep-alive, but that still means that if you want to get a parent and its children, you have to wait for the parent request round-trip. To add - this is a major issue we've been working through at Slant.co - we've combined child objects into the requests for the parent objects (sometimes even two levels down), but it's significantly complicated caching efforts - we're in the process of building some namespacing into our server-side request cache, so we can transparently make cached Question and Option pages stale whenever, e.g., the title of a Pro/Con gets changed. If HTTP allowed multiple responses, we could treat everything transparently server-side as individual requests, and get much higher hit-rates for proxy and client-side caches.
psadauskas 12y ago
I don't think this is messy, this is exactly how REST is intended to work, and is beautiful. I think the most common misconception that REST is simply a way to do CRUD over HTTP, when in fact they are completely orthogonal. You can do CRUD over REST, sure, but REST is also capable of much more. That's the realization the author came to with his `/request` and `/audit` resources, and is the zen of REST. Understanding that will put HTTP to work for you, instead of fighting against it. The reason Level 3 HATEOAS REST is hard in Backbone and most other frameworks -- both frontend and backend -- is because they have yet to obtain the same enlightenment.
- dragonwriter 12y ago
  > I think the most common misconception that REST is simply a way to do CRUD over HTTP, when in fact they are completely orthogonal. I dunno, REST maps to CRUD pretty well -- but its pretty limited when you think of it as restricted to CRUD operations against the equivalent of base tables in whatever your datastore of choice is -- its more like CRUD operations against views with arbitrarily complex rules mapping operations on the views to operations against base tables...
  - psadauskas 12y ago
    You are correct, I could have clarified it as "CRUD of business objects in a database".
- deleted 12y ago
  [deleted]
- munro 12y ago
  It's hard for me to submit to a philosophy for reasons like that it's beautiful, and that you will reach zen. Level 3 enlightenment sounds very cultish to me. :) I've dropped the notion of REST and been very happy with simple RPC, instead of contorting my mental model into resources or to align with the HTTP spec. I personally have found zen in applying simpler concepts to software development. Such as composition over inheritance to my API design, mixing in certain aspects like content negotiation or caching, when those complexities become necessary. Or separation of concerns, making sure endpoints don't do too much, and the realization of concerns vs technology [1]. Really thinking about the notion of simplicity as describe by Rick Hickley in Simple Made Easy [2]. Or "There are only two hard problems in Computer Science: cache invalidation and naming things"--putting off caching until an endpoint becomes a problem--and not worrying if my URL structure is RESTful. Here's an example of an API that I find beautiful [3]. [1] https://www.youtube.com/watch?v=x7cQ3mrcKaY https://www.youtube.com/watch?v=x7cQ3mrcKaY [2] http://www.infoq.com/presentations/Simple-Made-Easy http://www.infoq.com/presentations/Simple-Made-Easy [3] https://mandrillapp.com/api/docs/ https://mandrillapp.com/api/docs/
jessaustin 12y ago
Why can't I subscribe to a feed for this blog? I was going to blame NewsBlur, but upon viewing source I don't see link rels for anything but the favicons and stylesheet. I thought for a moment that I might just give them my email address, but then they pulled a bizarre "Looks like you have cookies disabled" out of some orifice. A: I don't. B: why would you need cookies to post a form?
- bryanh 12y ago
  Zapier co-founder here, thanks for pointing this out! We do have a feed (https://zapier.com/engineering/feeds/latest/ https://zapier.com/engineering/feeds/latest/) but we I think we need to add a meta tag for NewsBlur/readers to pick up on it. We'll do this! We use CSRF protection across all POSTs/PUTs, so cookies are generally required. I'll look into removing this for certain forms (like blog subscribe, fairly safe me thinks!). Thanks again!
  - jessaustin 12y ago
    Thanks for being responsive! Now that you point it out I realize I could have just clicked on the shuttlecock icon, whoops. I'm not certain what could have caused the cookie problem, since I've just gotten that same error message on a different device, neither of which actually have cookies turned off. (For instance, my HN cookies seem to be working fine...)
    - bryanh 12y ago
      Yeah, I'm not sure either. I'll look into it.
jchrisa 12y ago
Here's some other thinking about the cost of APIs (discussed earlier on HN) http://writings.quilt.org/2014/05/12/distributed-systems-and-the-end-of-the-api/ http://writings.quilt.org/2014/05/12/distributed-systems-and...
itsuart 12y ago
I found using HTTP (a mere transport layer) status codes as part of an API very unnatural and wrong. It feels like bending TCP/UDP packets structure to implement FTP to me. And shoehorning your API into any kind of "blessed guidlines" just to earn you a badge? That's just a waste of time.
- icebraining 12y ago
  They're not using HTTP as a mere transport layer, since they're using different HTTP methods and URLs to indicate different semantics. If they were passing payloads through a single endpoint and method, like XML-RPC and SOAP do, that would be using using HTTP as a mere transport layer.
  - itsuart 12y ago
    Seems I phrased my previous comment too vague, sorry for that. I'm aware of what REST stands for. My question is what advantages utilizing "obscure" HTTP verbs and headers with spreading "endpoints" all over ones application provides compared to RPC over HTTP? Surely, using RPC with a single endpoint is much simpler and thus more maintainable/modifiable?
    - icebraining 12y ago
      Which obscure HTTP verbs? I didn't see any in the article. You should read Fielding's thesis, but the short version is: when you're using a single endpoint, you're not really simplifying the communication -after all, your application still needs to do different things-, you're just replacing parts of the standard HTTP methods and server-sent URLs (which the client doesn't need to know beforehand) with custom/proprietary method names which the client needs to be coded against specifically. Of course, this depends on whether you're actually following REST or not. If you're hardcoding endpoints all over your client applications, your architecture is not really RESTful. Still, even if you do hardcode them, you still gain some advantages: for example, you can add a layer of caching by just sticking Varnish in the middle, which is not possible with RPC without custom code. That's the advantage of following the Uniform Interface constraint.
jmakeig 12y ago
REST, as Fielding described it, is good for decoupled systems that can (and need to) evolve independently, like web browsers consuming HTML and JavaScript. If you can relax the decoupling or independent evolution constraints, RPC over HTTP is usually easier to understand and implement. This is where most HTTP APIs fall. (…and that’s OK)
joereggan190 12y ago
http://campmediterraneo.com/en/node/36446 http://campmediterraneo.com/en/node/36446

74 points by brian_cooksey 12y ago

i've been meaning to write something similar about the whole REST craze. REST breaks down pretty rapidly once you get out of the key-value-store paradigm (read: anything involving child objects). REST lacks the ability to relay full state-change semantics without hackery. As the article pointed out, it forces you to be extra chatty over http, which is far from free over the congested, global network that is the interwebs. For example, what if a single PUT request creates multiple sub-objects? How does my server reply with multiple location headers? Do i have to first re-get the created object's child locations and re-request each individually? How about just sending the full object state back as the response to my PUT request? Well, according to REST, the body of the response just needs to be a description of the error or success status. Basically, REST sucks for reducing round trips if you're going to follow it pedantically. The theory is sound, but it needs to be updated to dictate how the server can send back more detailed info in response to POST/PUT/PATCH requests. /rant

rdtsc 12y ago
> How about just sending the full object state back as the response to my PUT request? That is what I do. I don't see anything wrong with it. PUT effectively replaces the state of the resource so you get it back right away (get the latest state, which should be what was sent in the request, minus a conflict for example if multiple clients do it or you have some incrementing revision id thing) > For example, what if a single PUT request creates multiple sub-objects? I'd use POST for creating/adding objects. I think of PUT usually as idempotent and as I mentioned above, use it to replace the state of the resource. So if this one POST ends up creating multiple-sub-objects you can return a json object that encapsulates URIs do those new objects. Remember resources don't have to map to internal objects, db rows or other such things. You can have the objects or db rows represented or used in different resources. If it makes sense and if you have this transactional interface, maybe it makes sense to have an explicit transaction resource that is in charge of managing a transaction (where multiple things happen and then it kind of becomes very explicit).
jessaustin 12y ago
How about just sending the full object state back as the response to my PUT request? Well, according to REST, the body of the response just needs to be a description of the error or success status. This isn't strictly true: If the target resource does not have a current representation and the PUT successfully creates one, then the origin server MUST inform the user agent by sending a 201 (Created) response. If the target resource does have a current representation and that representation is successfully modified in accordance with the state of the enclosed representation, then the origin server MUST send either a 200 (OK) or a 204 (No Content) response to indicate successful completion of the request. [0] Both the 200 and 201 payloads may include anything you want, including a giant representation with many constituent parts (which parts would ideally each contain a link to its respective associated resource). [0] http://tools.ietf.org/html/rfc7231#page-27 http://tools.ietf.org/html/rfc7231#page-27
dragonwriter 12y ago
> For example, what if a single PUT request creates multiple sub-objects? If a PUT, POST, or PATCH does that, then it does that. So what? > How does my server reply with multiple location headers? It doesn't respond with multiple location headers. With PUT or POST, with an 201 Created status response with the Location: header containing the URI of the parent object (the resource directly created by the PUT/POST), and an entity body, in some appropriate format, containing the URIs of all the created objects. (If there is a representation of the main resource which would do this that is acceptable to the client, this would seem to be an ideal way of communicating that.) With PATCH, much the same, but with 200 OK. > Do i have to first re-get the created object's child locations and re-request each individually? There's no reason for a REST API to require that. > How about just sending the full object state back as the response to my PUT request? I see no reason why sending a full resource representation in a 201 status response, provided that the client has indicated that they are willing to accept the relevant media type, is problematic, especially if it does what the HTTP/1.1 spec says the 201 response entity should do. (The one thing that common resource representations might not do that they should is provide their own location and the direct locations of any embedded subentities that are separate addressable, but there is no reason why a resource representation couldn't do that, and I'd argue that in REST it would be desirable that resource representations do do that.) > Well, according to REST, the body of the response just needs to be a description of the error or success status. As, stated, that's accurate, but you seem to be acting as if that was "needs to be just" rather than "just needs to be" (e.g., misunderstanding it as a maximum allowed content, rather than a minimum required content.) > Basically, REST sucks for reducing round trips if you're going to follow it pedantically. Except that none of the problems you've pointed to in that regard have anything to do with REST.
- T-R 12y ago
  > I'd argue that in REST it would be desirable that resource representations do do that (provide [...] the direct locations of any embedded subentities that are separate addressable). You could actually go a step further - if you're using "Hypertext as the engine of application state" (HATEOAS), URIs for the locations of direct subentities need to be there, if the client is expected to be able to make those state transitions, for the API to be "fully RESTful". (Though, I'd personally agree with the article that features like discoverability and content negotiation are secondary to those you get from idempotence/properly using HTTP methods, and feel they should be considered bonus features, rather than strict requirements)

je42 12y ago

Thinking about this: By merging children into the parent resource, he traded a reduction of requests with the loss of separation of concerns. How about adding a layer instead whose only concern is the reduction of the number of requests for the client. It would collect and merge the info from parent and children resources and return it in merged form to the client ?

lumpypua 12y ago
That would probably actually work pretty well. I've run into a similar problem a number of times and I may try this. "All problems in computer science can be solved by another level of indirection" - David Wheeler "...except for the problem of too many layers of indirection." - Kevlin Henney
pbh101 12y ago
Netflix has apparently called this idea an 'orchestration layer': http://thenextweb.com/dd/2013/12/17/future-api-design-orchestration-layer/ http://thenextweb.com/dd/2013/12/17/future-api-design-orches...
T-R 12y ago
I really do wish HTTP had a mechanism for responding to a single request with multiple combined response bodies as if requests were made for each individually (from the perspective of, e.g., a caching proxy) - the loss of separation of concerns from merging children into the parent isn't just a usability issue, it also means you lose your cache granularity - the cache for the parent object is now stale whenever a child is modified, and a request for a child after getting the parent won't hit the cache at the request level. There's keep-alive, but that still means that if you want to get a parent and its children, you have to wait for the parent request round-trip. To add - this is a major issue we've been working through at Slant.co - we've combined child objects into the requests for the parent objects (sometimes even two levels down), but it's significantly complicated caching efforts - we're in the process of building some namespacing into our server-side request cache, so we can transparently make cached Question and Option pages stale whenever, e.g., the title of a Pro/Con gets changed. If HTTP allowed multiple responses, we could treat everything transparently server-side as individual requests, and get much higher hit-rates for proxy and client-side caches.

psadauskas 12y ago

I don't think this is messy, this is exactly how REST is intended to work, and is beautiful. I think the most common misconception that REST is simply a way to do CRUD over HTTP, when in fact they are completely orthogonal. You can do CRUD over REST, sure, but REST is also capable of much more. That's the realization the author came to with his `/request` and `/audit` resources, and is the zen of REST. Understanding that will put HTTP to work for you, instead of fighting against it. The reason Level 3 HATEOAS REST is hard in Backbone and most other frameworks -- both frontend and backend -- is because they have yet to obtain the same enlightenment.

dragonwriter 12y ago
> I think the most common misconception that REST is simply a way to do CRUD over HTTP, when in fact they are completely orthogonal. I dunno, REST maps to CRUD pretty well -- but its pretty limited when you think of it as restricted to CRUD operations against the equivalent of base tables in whatever your datastore of choice is -- its more like CRUD operations against views with arbitrarily complex rules mapping operations on the views to operations against base tables...
- psadauskas 12y ago
  You are correct, I could have clarified it as "CRUD of business objects in a database".
deleted 12y ago
[deleted]
munro 12y ago
It's hard for me to submit to a philosophy for reasons like that it's beautiful, and that you will reach zen. Level 3 enlightenment sounds very cultish to me. :) I've dropped the notion of REST and been very happy with simple RPC, instead of contorting my mental model into resources or to align with the HTTP spec. I personally have found zen in applying simpler concepts to software development. Such as composition over inheritance to my API design, mixing in certain aspects like content negotiation or caching, when those complexities become necessary. Or separation of concerns, making sure endpoints don't do too much, and the realization of concerns vs technology [1]. Really thinking about the notion of simplicity as describe by Rick Hickley in Simple Made Easy [2]. Or "There are only two hard problems in Computer Science: cache invalidation and naming things"--putting off caching until an endpoint becomes a problem--and not worrying if my URL structure is RESTful. Here's an example of an API that I find beautiful [3]. [1] https://www.youtube.com/watch?v=x7cQ3mrcKaY https://www.youtube.com/watch?v=x7cQ3mrcKaY [2] http://www.infoq.com/presentations/Simple-Made-Easy http://www.infoq.com/presentations/Simple-Made-Easy [3] https://mandrillapp.com/api/docs/ https://mandrillapp.com/api/docs/

jessaustin 12y ago

Why can't I subscribe to a feed for this blog? I was going to blame NewsBlur, but upon viewing source I don't see link rels for anything but the favicons and stylesheet. I thought for a moment that I might just give them my email address, but then they pulled a bizarre "Looks like you have cookies disabled" out of some orifice. A: I don't. B: why would you need cookies to post a form?

bryanh 12y ago
Zapier co-founder here, thanks for pointing this out! We do have a feed (https://zapier.com/engineering/feeds/latest/ https://zapier.com/engineering/feeds/latest/) but we I think we need to add a meta tag for NewsBlur/readers to pick up on it. We'll do this! We use CSRF protection across all POSTs/PUTs, so cookies are generally required. I'll look into removing this for certain forms (like blog subscribe, fairly safe me thinks!). Thanks again!
- jessaustin 12y ago
  Thanks for being responsive! Now that you point it out I realize I could have just clicked on the shuttlecock icon, whoops. I'm not certain what could have caused the cookie problem, since I've just gotten that same error message on a different device, neither of which actually have cookies turned off. (For instance, my HN cookies seem to be working fine...)
  - bryanh 12y ago
    Yeah, I'm not sure either. I'll look into it.

jchrisa 12y ago

Here's some other thinking about the cost of APIs (discussed earlier on HN) http://writings.quilt.org/2014/05/12/distributed-systems-and-the-end-of-the-api/ http://writings.quilt.org/2014/05/12/distributed-systems-and...

itsuart 12y ago

I found using HTTP (a mere transport layer) status codes as part of an API very unnatural and wrong. It feels like bending TCP/UDP packets structure to implement FTP to me. And shoehorning your API into any kind of "blessed guidlines" just to earn you a badge? That's just a waste of time.

icebraining 12y ago
They're not using HTTP as a mere transport layer, since they're using different HTTP methods and URLs to indicate different semantics. If they were passing payloads through a single endpoint and method, like XML-RPC and SOAP do, that would be using using HTTP as a mere transport layer.
- itsuart 12y ago
  Seems I phrased my previous comment too vague, sorry for that. I'm aware of what REST stands for. My question is what advantages utilizing "obscure" HTTP verbs and headers with spreading "endpoints" all over ones application provides compared to RPC over HTTP? Surely, using RPC with a single endpoint is much simpler and thus more maintainable/modifiable?
  - icebraining 12y ago
    Which obscure HTTP verbs? I didn't see any in the article. You should read Fielding's thesis, but the short version is: when you're using a single endpoint, you're not really simplifying the communication -after all, your application still needs to do different things-, you're just replacing parts of the standard HTTP methods and server-sent URLs (which the client doesn't need to know beforehand) with custom/proprietary method names which the client needs to be coded against specifically. Of course, this depends on whether you're actually following REST or not. If you're hardcoding endpoints all over your client applications, your architecture is not really RESTful. Still, even if you do hardcode them, you still gain some advantages: for example, you can add a layer of caching by just sticking Varnish in the middle, which is not possible with RPC without custom code. That's the advantage of following the Uniform Interface constraint.

jmakeig 12y ago

REST, as Fielding described it, is good for decoupled systems that can (and need to) evolve independently, like web browsers consuming HTML and JavaScript. If you can relax the decoupling or independent evolution constraints, RPC over HTTP is usually easier to understand and implement. This is where most HTTP APIs fall. (…and that’s OK)

joereggan190 12y ago

http://campmediterraneo.com/en/node/36446 http://campmediterraneo.com/en/node/36446