10 ms·

Why?

erikig 7y ago
JSON is pretty ubiquitous for transferring data between web resources, compressing the data with a novel method is always welcome.
anderspitman 7y ago
One use case where I've put (uri-encoded) JSON in the query string is for parameterizing GET requests with more structured data, since URL encoding is fairly limited and GET requests don't have a body like POSTS.
- zer0faith 7y ago
  Can't you just base64 url safe encode it and get the same result? Aren't URl strings limited in their length as well?
  - emj 7y ago
    If you don't care about IE and Edge you can fit 8KB-64KB in the URI, otherwise 2K is pretty safe.
  - KilledByAPixel 7y ago
    As far as I know base64 is not guaranteed to be uri safe, though most of the time you should be fine. However more importantly converting to base64 automatically increases the size by 33% https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding https://developer.mozilla.org/en-US/docs/Web/API/WindowBase6...
    - CodesInChaos 7y ago
      The parent poster was referring to the "base64url" variant of Base64 which uses `-` and `_` as the two special characters and leaves out the useless padding, making it url safe. (The 33% expansion still applies though) https://tools.ietf.org/html/rfc4648#page-7 https://tools.ietf.org/html/rfc4648#page-7
  - anderspitman 7y ago
    As others mentioned, size is the main concern here, which is also why JSONCrush could be useful. But I'd definitely use POSTs first if possible.
- blowski 7y ago
  It may not apply in your case, but in some cases, you can give a GET request a body - Elasticsearch relies on this.
  - strbean 7y ago
    From what I've seen, many intermediaries do nasty things if you have a body on anything that doesn't 'normally' have one. I've heard a fair amount about GCP's load balancers dropping unexpected bodies, and other tools giving a 5XX. It is really unfortunate, because there are tons of use cases and zero reason to interfere with these requests. The justification is also downright ridiculous. The argument is that "GET, DELETE, ... have no defined semantics for bodies". Meanwhile the 'defined semantics' for POST bodies is... whatever the application decides.
    - josephg 7y ago
      I made this exact argument in session last week in the HTTP working group at the IETF while discussing the semantics of DELETE. The working group chair is quite passionate defender of the position that GET should never have a body, and in the end I was outvoted by the room. I'm slowly coming around to the idea that he might be right. The problem is the (semantic) question of what resource is being discussed. The semantics of GET, HEAD and OPTIONS are that (unless it gets modified) the same resource should always be answered the same way. If the resource that we're asking about is identified by the URL + the body, then those requests (and DELETE) all need a body too. And then there's an open question for PUT and POST about what resource exactly is being modified by the request - although as you say, the semantics are whatever so thats less important. I think HTTP has a problem in that its hard to express a resource name that is complex. Like, "the records which match this specific elasticsearch query" is a hard thing to pack into a URL. If HTTP was different, I could imagine a second body-like part of each request called the "resource section" with its own Resource-Type: application/json header and so on. But instead we just have a URL, so we end up doing awful hacks like packing JSON into URLs and make them unreadable. The URL is long enough for this sort of thing - implementations have to allow at least 8k of space for them, and can allow more space. But they're hard to read and display, and there's no standard way to pack json in there. So I wonder it'd be worth having a formal, consistent way to encode stuff like this in the URL. I'm spitballing, but maybe we need a standard way to encode JSON into the URL (base64 or whatever) defined in an RFC. Then if you do that, you can add a "Query-Type: application/json" header that declares that the URL contains JSON encoded in the standard way. Then we could at least have consistency and nice tooling around this. So like, when you're making a request you could just pass JSON into your http library and it'd encode it into the URL automatically in the standard way. And when the URL is being displayed in the dev tools or whatever, it could write out the actual embedded JSON / GraphQL / whatever object that you've packed in there in an easy to read way.
- m712 7y ago
  You can probably use extended URL encoding for that, which is supported by at least body-parser [https://npmjs.com/package/body-parser https://npmjs.com/package/body-parser]. Not sure about other PLs.
JMTQp8lwXL 7y ago
Ingress and egress costs are expensive in the cloud. Using this with gzip could save you on cost.
PNWChris 7y ago
I have a personal project that saves state to the URL's hash (https://www.rebalancecalc.com https://www.rebalancecalc.com). I do that because I wanted users to be able to save their state without burdening those users with accounts or burdening myself with maintaining a DB for something so simple. I also somewhat abuse the history API and use my "read the URL and load state" logic to implement undo-redo via navigating back and forth, though that doesn't seem to work right now. I am working on a refactor that uses redux to implement undo-redo and just replace state, to keep the user's history clean. Storing encoded JSON in the URL hash is a nifty hack in my opinion. Users can save state in a bookmark or share it with others easily, and it's clear to the users that "where they are" in the URL bar maps to the current app state. Plus, bookmark syncing is taken care of by most browsers to make that state available elsewhere, etc. For the site owner, it means not needing a DB to make an app with some kind of state persistence. One risk: be sure the state you persist to the URL is in a schema you plan to retain compatibility with! Blind serialization and de-serialization is a recipe for bugs and misery the next time you add a feature.
- delaaxe 7y ago
  I do the same on a production app at work, but for a different reason. Storing confidential data in the hash assures my users that I (the developer) don’t have access to this data, since anything after the hash never gets sent to the server by the browser. It wouldn’t stop me from sending that information with a post request afterwards but the code is open source and it could be noticed in developer tools.
  - mkesper 7y ago
    Sorry I don't understand, you put confidential data into an URL so it's probably in server logs etc.?
    - squiggleblaz 7y ago
      If a user types in web browser: https://www.example.com/webpage?q=54#{"userdata":"its https://www.example.com/webpage?q=54#{"userdata":"its a secret"} Then the web browser sends a message to the server that looks something like this: GET /webpage?q=54 HTTP/1.1 Host: www.example.com Cookie: well maybe there's a cookie here Although it's encrypted with ssl and there's some hopefully irrelevant messages along with it (which aren't irrelevant so they can be used to fingerprint you). As you can see, the bit that comes after the hash isn't ever sent from the client to the server. It was originally meant so that you could link to a particular section of a longer web page, so it was quite irrelevant. Nowadays it's exposed to javascript. This means that the code can rely on it - it can read and set it. The javascript author could read it and use it in an entirely inbrowser javascript app. Or the javascript author could read it and send it to the server in a more secure channel, like the body of a POST request, to reduce the chances it gets stored in server logs. But what comes after the hash is never processed by any standards compliant web server not transmitted by any standards compliant web browser/client.

Why?

JSON is pretty ubiquitous for transferring data between web resources, compressing the data with a novel method is always welcome.

anderspitman 7y ago

One use case where I've put (uri-encoded) JSON in the query string is for parameterizing GET requests with more structured data, since URL encoding is fairly limited and GET requests don't have a body like POSTS.

zer0faith 7y ago
Can't you just base64 url safe encode it and get the same result? Aren't URl strings limited in their length as well?
- emj 7y ago
  If you don't care about IE and Edge you can fit 8KB-64KB in the URI, otherwise 2K is pretty safe.
- KilledByAPixel 7y ago
  As far as I know base64 is not guaranteed to be uri safe, though most of the time you should be fine. However more importantly converting to base64 automatically increases the size by 33% https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding https://developer.mozilla.org/en-US/docs/Web/API/WindowBase6...
  - CodesInChaos 7y ago
    The parent poster was referring to the "base64url" variant of Base64 which uses `-` and `_` as the two special characters and leaves out the useless padding, making it url safe. (The 33% expansion still applies though) https://tools.ietf.org/html/rfc4648#page-7 https://tools.ietf.org/html/rfc4648#page-7
- anderspitman 7y ago
  As others mentioned, size is the main concern here, which is also why JSONCrush could be useful. But I'd definitely use POSTs first if possible.
blowski 7y ago
It may not apply in your case, but in some cases, you can give a GET request a body - Elasticsearch relies on this.
- strbean 7y ago
  From what I've seen, many intermediaries do nasty things if you have a body on anything that doesn't 'normally' have one. I've heard a fair amount about GCP's load balancers dropping unexpected bodies, and other tools giving a 5XX. It is really unfortunate, because there are tons of use cases and zero reason to interfere with these requests. The justification is also downright ridiculous. The argument is that "GET, DELETE, ... have no defined semantics for bodies". Meanwhile the 'defined semantics' for POST bodies is... whatever the application decides.
  - josephg 7y ago
    I made this exact argument in session last week in the HTTP working group at the IETF while discussing the semantics of DELETE. The working group chair is quite passionate defender of the position that GET should never have a body, and in the end I was outvoted by the room. I'm slowly coming around to the idea that he might be right. The problem is the (semantic) question of what resource is being discussed. The semantics of GET, HEAD and OPTIONS are that (unless it gets modified) the same resource should always be answered the same way. If the resource that we're asking about is identified by the URL + the body, then those requests (and DELETE) all need a body too. And then there's an open question for PUT and POST about what resource exactly is being modified by the request - although as you say, the semantics are whatever so thats less important. I think HTTP has a problem in that its hard to express a resource name that is complex. Like, "the records which match this specific elasticsearch query" is a hard thing to pack into a URL. If HTTP was different, I could imagine a second body-like part of each request called the "resource section" with its own Resource-Type: application/json header and so on. But instead we just have a URL, so we end up doing awful hacks like packing JSON into URLs and make them unreadable. The URL is long enough for this sort of thing - implementations have to allow at least 8k of space for them, and can allow more space. But they're hard to read and display, and there's no standard way to pack json in there. So I wonder it'd be worth having a formal, consistent way to encode stuff like this in the URL. I'm spitballing, but maybe we need a standard way to encode JSON into the URL (base64 or whatever) defined in an RFC. Then if you do that, you can add a "Query-Type: application/json" header that declares that the URL contains JSON encoded in the standard way. Then we could at least have consistency and nice tooling around this. So like, when you're making a request you could just pass JSON into your http library and it'd encode it into the URL automatically in the standard way. And when the URL is being displayed in the dev tools or whatever, it could write out the actual embedded JSON / GraphQL / whatever object that you've packed in there in an easy to read way.
m712 7y ago
You can probably use extended URL encoding for that, which is supported by at least body-parser [https://npmjs.com/package/body-parser https://npmjs.com/package/body-parser]. Not sure about other PLs.

JMTQp8lwXL 7y ago

Ingress and egress costs are expensive in the cloud. Using this with gzip could save you on cost.

PNWChris 7y ago

I have a personal project that saves state to the URL's hash (https://www.rebalancecalc.com https://www.rebalancecalc.com). I do that because I wanted users to be able to save their state without burdening those users with accounts or burdening myself with maintaining a DB for something so simple. I also somewhat abuse the history API and use my "read the URL and load state" logic to implement undo-redo via navigating back and forth, though that doesn't seem to work right now. I am working on a refactor that uses redux to implement undo-redo and just replace state, to keep the user's history clean. Storing encoded JSON in the URL hash is a nifty hack in my opinion. Users can save state in a bookmark or share it with others easily, and it's clear to the users that "where they are" in the URL bar maps to the current app state. Plus, bookmark syncing is taken care of by most browsers to make that state available elsewhere, etc. For the site owner, it means not needing a DB to make an app with some kind of state persistence. One risk: be sure the state you persist to the URL is in a schema you plan to retain compatibility with! Blind serialization and de-serialization is a recipe for bugs and misery the next time you add a feature.

delaaxe 7y ago
I do the same on a production app at work, but for a different reason. Storing confidential data in the hash assures my users that I (the developer) don’t have access to this data, since anything after the hash never gets sent to the server by the browser. It wouldn’t stop me from sending that information with a post request afterwards but the code is open source and it could be noticed in developer tools.
- mkesper 7y ago
  Sorry I don't understand, you put confidential data into an URL so it's probably in server logs etc.?
  - squiggleblaz 7y ago
    If a user types in web browser: https://www.example.com/webpage?q=54#{"userdata":"its https://www.example.com/webpage?q=54#{"userdata":"its a secret"} Then the web browser sends a message to the server that looks something like this: GET /webpage?q=54 HTTP/1.1 Host: www.example.com Cookie: well maybe there's a cookie here Although it's encrypted with ssl and there's some hopefully irrelevant messages along with it (which aren't irrelevant so they can be used to fingerprint you). As you can see, the bit that comes after the hash isn't ever sent from the client to the server. It was originally meant so that you could link to a particular section of a longer web page, so it was quite irrelevant. Nowadays it's exposed to javascript. This means that the code can rely on it - it can read and set it. The javascript author could read it and use it in an entirely inbrowser javascript app. Or the javascript author could read it and send it to the server in a more secure channel, like the body of a POST request, to reduce the chances it gets stored in server logs. But what comes after the hash is never processed by any standards compliant web server not transmitted by any standards compliant web browser/client.