12 ms·

Post Mortem: A single whitespace character

333 points by goleksiak 12y ago

vvpan 12y ago
Way to abuse :first-letter.
deleted 12y ago
[deleted]
jim_lawless 12y ago
I experienced a similar problem with a POP3 utility that I had written years ago. I had been appending an extra space to the end of each text line (before the CRLF ). There were a few people using this utility with no problems until one day a particular POP3 server no longer tolerated my utility's malformed requests.
mml 12y ago
Cowboy is quite a well respected we server of the Erlang flavor. I'd guess heroku rejiggered something in their stack, perhaps adding cowboy as a reverse proxy or load balancer in front of their junk. Cowboy apparently shot yor no-good dirty sidewinding web requests in the face.
- mqsiuser 12y ago
  It is well known that you can't (should not) rely on bugs (or internal APIs)
deleted 12y ago
[deleted]
kirab 12y ago
It's technically correct, according to the HTTP spec there must be a single "SP" character between the elements in the Request-Line: Request-Line = Method SP Request-URI SP HTTP-Version CRLF Source: http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1 http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1
spydum 12y ago
The Server: cowboy tag is from an Erlang web server: https://github.com/ninenines/cowboy/blob/master/src/cowboy_protocol.erl#L177 https://github.com/ninenines/cowboy/blob/master/src/cowboy_p... I'm guessing around here would be interesting to add a test case to handle. As far as whose server this is? I'd guess Heroku or AWS, though it's plenty possible T-Mobile could have devised some proxy to inspect traffic, but seems unlikely they would do so with Cowboy?
- lostcolony 12y ago
  T-mobile is known to have, in the past at least, used Erlang. And Cowboy is one of the most popular web servers within that community at this point.
- mischanix 12y ago
  It's simple enough to single out Heroku: $ cat <<EOF | nc example.herokuapp.com 80 GET /test HTTP/1.1 EOF ---- HTTP/1.1 505 HTTP Version Not Supported
  - davidw 12y ago
    Heroku use Erlang: http://www.erlang-in-anger.com/ http://www.erlang-in-anger.com/
  - 3ds 12y ago
    Your example fails with or without the whitespace. These work though: Request printf 'GET / HTTP/1.1\r\nHost: example.herokuapp.com\r\n\r\n' | nc example.herokuapp.com 80 Response HTTP/1.1 200 OK Connection: keep-alive Server: SimpleHTTP/0.6 Python/2.7.6 Request printf 'GET / HTTP/1.1\r\nHost: example.herokuapp.com\r\n\r\n' | nc example.herokuapp.com 80 Response HTTP/1.1 505 HTTP Version Not Supported Connection: close Server: Cowboy
    - mischanix 12y ago
      Ah right, forgot about the newline specification. I guess, for reference, the smallest string I can come up with to get Cowboy to spit that error message is '\x20\x20\n'. Parsers are fun.
jrochkind1 12y ago
This very example -- requests were technically illegal all the time without devs realizing, but something in the stack changed to start rejecting them -- demonstrates the fallacy of the "be liberal in what you accept, strict in what you issue" principal. If all the web servers involved had been strict in rejecting the illegal request from the start, they would have noticed the bug in development before deploying to firmware in the field.
- ajanuary 12y ago
  Doesn't it just demonstrate that you shouldn't switch from being liberal to being strict? For it to hold up, you need to provide the further argument that you frequently need to switch from liberal to strict.
  - antirez 12y ago
    Or... here the problem is that the "be liberal in what you accept" design principle failed to be captured by the HTTP specification writer, that forced a single SP character. It looks like a specification issue to me to use a syntax which is very prone to errors, and is even not much visible (you can't easily inspect double spaces in protocol traces when checking just with your eyes), and then be strict about it. Even changing separator, if you want to be strict, already helps, like in "foo|bar|zap" compared to "foo bar zap". Humans are strage: many will spot "foo||bar|zap" as an error, but not "foo bar zap" as an error as serious as the previous one.
    - jdmichal 12y ago
      Especially when you have display technologies like HTML that will actively compress whitespace... As happened in your last example.
- mqsiuser 12y ago
  > the fallacy of the "be liberal in what you accept, strict in what you issue" principal The market (players) (can) manipulate it to create an (perceived) competitive advantage. It's also a source where "evil" in IT comes from.
kyberias 12y ago
What's the deal with all the scrollbars on this page?
- spindritf 12y ago
  Yeah, why is every image, heading, and paragraph on that page surrounded by scrollbars where most don't work and are not necessary?
  - goleksiak 12y ago
    What browser are you using? I'm not seeing that here on my devices...? We are using pretty vanilla Bootstrap.
    - anonymoushn 12y ago
      http://i.imgur.com/v0b1LKC.png http://i.imgur.com/v0b1LKC.png This is what I see in Chrome on OSX.
    - ultrafez 12y ago
      The issue is you've got the following CSS on lines 31-33 of blog.css: body.blog .blog-post p { overflow: scroll; }
    - dceddia 12y ago
      Are your devices Apple devices by any chance? We had a similar bug reported at work recently, and it turned out that Windows browsers will always show scrollbars but the ones running on OS X/iOS will hide them until you start scrolling. To turn them on in OS X, go to System Preferences > General and set Show scroll bars to "Always".
  - ultrafez 12y ago
    In their CSS, they have a rule for every <p> tag to have "overflow: scroll" for some reason. Not sure why they didn't use the default value for overflow, since there's nothing on that page that needs to be specifically told to scroll.
colinbartlett 12y ago
It scares me to think all of these requests run over unencrypted HTTP.
- davidrusu 12y ago
  Why? it's just pizza
  - DanBC 12y ago
    Pizza has been used as a tool of harassment in the past. People order lots of pizza from different places for the victim, who then has to deal with a bunch of angry pizza drivers and being black-listed from those pizza places. Pizza drivers are often the victims of crime. Not only for the small amounts of cash that they carry, but sometimes just for the pizzas. edit: I should say that my comment here was a kneejerk reaction to "it's just pizza", and has nothing to do with how eatabit.com deals with this kind of harassment. i agree with other commenters that blog posts like these are a great way to promote the company.
    - pliny 12y ago
      How are you going to exploit the fact that these pizza orders aren't encrypted to achieve either of those things?
      - DanBC 12y ago
        PIZZA IS SRS BSNS
      - pjc50 12y ago
        Encrypt the pizzas.
      - serf 12y ago
        that's the wrong question to ask. We're not (all) in the "think-of-things-to-do-with-stolen-information" business like so many others are; but many of us are we're in the "encrypt-all-the-things-so-that-information-isn't-stolen" business.
rcconf 12y ago
I've had the same issues when developing with Flask in Python. I forgot to URL encode some query parameters and it worked fine with the local HTTP server. But when I put nginx in front as a proxy, it denied all requests.
- stevekemp 12y ago
  The thttpd webserver doesn't handle requests with too many slashes either, which I only found out recently This is treated as an invalid request: http://example.com//robots.txt
  - zimpenfish 12y ago
    Unless I'm reading RFC 3986 incorrectly, that's valid because you can't have an empty segment in the path part of a URI.
    - ajanuary 12y ago
      I think you're reading it incorrectly. You can have an empty segment in the path. The BNF for a segment is: segment = *pchar Which according to RFC2234 section 3.6 means zero or more repetitions.
      - gpvos 12y ago
        But then the server may still decide that an empty segment is so meaningless that it will refuse it. In fact, it would not be a smart move to just treat double slashes the same as single ones, because of relative URLs: a ".." segment only removes one slash, so the hierarchy levels would get messed up. thttpd is doing the smart thing here. As one of my teachers at university would say: the empty segment is also a segment.
        lmm 12y ago
        The server can of course interpret the path as it wants, but it should allow an application running under the server to give 'foo//bar' a meaning if that application wants to, IMO.
ericcholis 12y ago
Slightly off-topic, but this is why dev posts like this are important. I didn't know eatabit.com was a thing, it it sounds like a great service.
- robogeek78 12y ago
  Dev #2 here thanks for the compliment. Where are you? Maybe we should expand to your area. :)
  - ericcholis 12y ago
    Buffalo, NY. We're quite proud of our local restaurant industry. So, yes, you should!
jlouis 12y ago
This proves a very important pet peeve of mine: Your modern application has a highly dynamic operating point. There is no way you can deploy a system and expect it to be static for eternity. Back in the day with low interconnectivity you could. But today it is impossible. When you build stacks on top of system for which you have no direct control, you must be able to adapt your system. This means you can't statically deploy code without an upgrade path in one way or the other.
- goleksiak 12y ago
  True but that doesn't bother me. Nothing is static on the web these days and everyone plays under the same rule set. Keeps things interesting...
  - jlouis 12y ago
    It shouldn't bother you. It is just how moderns systems are.
- quesera 12y ago
  You're combining two issues. Yes, if you let other people run your infrastructure, you are beholden to their operations decisions and schedules. It is not impossible to design around that (new) problem, but it is sometimes expensive. The trick is to know what external dependencies you have, and that is almost impossible to fully quantify in the XaaS and cloud model.
Danieru 12y ago
That series of strcat's caught my eye as bad practice. Fine in this case since the destination string is short but horrible in general. Every single one of those calls needs to iterate over the entire existing string to find the string size. The code could be much cleaner with a small macro hiding the incrementation and the casts.
- userbinator 12y ago
  Basically it's an O(n^2) algorithm... well-known story about that here: http://www.joelonsoftware.com/articles/fog0000000319.html http://www.joelonsoftware.com/articles/fog0000000319.html The design of strcat() itself is partially to blame for this - the return value could've been more useful, like the number of characters in the resulting string or a pointer to the end of the appended string so it could be used to chain concatenations, but instead they chose to return the exact same pointer that was passed in as the source.
- mikeash 12y ago
  A sufficiently smart compiler could optimize a string of strcat calls to remove the redundant length finding. I have no idea if real compilers actually would....
  - srtjstjsj 12y ago
    Java does, for the "+" operator.
    - mikeash 12y ago
      Oh yes, nice example! So there's precedent for C compilers doing something similar.
- krakensden 12y ago
  Yeah- implicit concatenation + snprintf seems like the way to go. Although you'd have to calculate a length, I suspect avoiding that is the primary virtue of this approach.
justinsb 12y ago
Tangentially, why didn't curl escape the trailing space to %20?
userbinator 12y ago
I saw it right away - "that HTTP/1.1 looks a bit farther away than it should be..." - and confirmed it by selecting the spaces. I thought it would be a bit more subtle than that... I remember working with a server that violated the HTTP spec by not accepting allowed extra spaces in headers. According to the new HTTP/1.1 RFC 7230, it should be a single space - the previous RFC didn't specify this clearly in the wording, although it is implied by the grammar (SP and not 1 * SP). https://tools.ietf.org/html/rfc7230#section-3.1.1 https://tools.ietf.org/html/rfc7230#section-3.1.1 "A request-line begins with a method token, followed by a single space (SP), the request-target, another single space (SP), the protocol version, and ends with CRLF." I'm surprised there doesn't seem to be any widely-used and easily available HTTP conformance checker - unlike the well-known HTML validators. This is also why monospace fonts are ideal for seeing small but significant differences like this.
- michaelmior 12y ago
  That's an interesting idea. It would be useful to have a Web server where the output is just a conformance check of the request. That might be a fun project for a rainy day :)
  - zimpenfish 12y ago
    Sounds like something that could be added to http://httpbin.org http://httpbin.org
    - userbinator 12y ago
      That runs on Python/Flask, which is already a layer of abstraction above where HTTP conformance testing would be; what you need is something that listens on a TCP socket and parses the requests itself.
      - zimpenfish 12y ago
        Actually, thinking about it, didn't Zed Shaw make a Ragel-based strict-conformance HTTP parser? > Simply being more explicit about what is valid HTTP means that most of the security attacks that worked on Apache were rejected outright when tried on Mongrel. Which I guess is a qualified "sounds like it, maybe?"
pilif 12y ago
Likely "Cowboy" is a transparent proxy added by your mobile service provider. I had a similar thing happening a year ago when the mobile provider used by most of our barcode scanners decided to add a transparent proxy into the loop (without telling anybody). The solution for this problem: Use SSL. I mean: There are already many good reasons to use SSL, but whenever you need to send any kind of mission critical data over the mobile network, you practically must use SSL if you want any kind of guarantees that the data you send to the server is what actually reaches the server (and reverse). Here's my war story from last year: http://pilif.github.io/2013/09/when-in-doubt-ssl/ http://pilif.github.io/2013/09/when-in-doubt-ssl/
- ctz 12y ago
  This would require a vast, vast upgrade of client power to achieve the same communications performance. If you could achieve it all, SSL would also likely decrease reliability over a spotty GSM link.
  - pjc50 12y ago
    Is client CPU actually a limiting factor? How does this affect reliability?
    - serf 12y ago
      as someone unfamiliar with their specific workload, but who has used such boards to do vpn/ssl stuff, no -- it's not a limiting factor unless trying to skim power requirements.
    - ctz 12y ago
      You cannot physically fit a whole SSL datagram (max size 16KB) into 8KB of RAM. SSL requires multiple passes over the data to (eg) decrypt and verify a datagram. At this point, you cannot use standard SSL at either the server side or client side. On the subject of reliablity: a 8 bit uC running at 16MHz needs a long time to do the public key crypto required to set up the connection. This means you need a GSM data link to be continuously available for a longer period.
KMag 12y ago
When learning OCaml, I decided to write a little web client that would bruit force the password on my own home router. I wrote a client, and my router wasn't responding, so I tried having my client fetch pages from Yahoo, and it worked fine. I fired up wireshark and saw that everything looked fine... except that all of my line terminators were shift-in-formfeed instead of carriage-return-newline. It turns out that OCaml uses decimal character escapes instead of octal. (This was back when I was under the impression that portable code avoided use of \n in string literals because someone who misunderstood text mode file handles had told me that Microsoft compilers expanded \n to \015\012.) Apparently someone at Yahoo had experienced enough terribly terribly written web clients that they wrote their HTTP server to accept any two non-space whitespace characters as a line ending.
cofcdylan 12y ago
i'm just glad my city made it to HN.
- goleksiak 12y ago
  Charleston represent!
jameshart 12y ago
"our cellular printing api has printed over 9300 food orders for our client restaurants, stadiums and golf courses" Am I the only one who read this as a system using 3D printing to print food? Disappointed to discover it's not that kind of cellular.
cleanCodeAtWork 12y ago
Are there any languages out there that handle scale and many connections like Erlang does, but with an easier to swallow syntax?
- hlieberman 12y ago
  Erlang. The syntax really isn't that bad, once you get over the initial shock. In all honesty, grasping that the variables are immutable and how you need to change your thinking is much more difficult than the syntax itself.
- angersock 12y ago
  http://elixir-lang.org/ http://elixir-lang.org/ It's the Erlang VM you love, but with the Ruby syntax we all enjoy!
- buster 12y ago
  http://elixir-lang.org/ http://elixir-lang.org/
- lmm 12y ago
  I've been very happy writing these things in Scala using Spray. Honestly there are plenty of event-driven I/O frameworks in many languages, and almost as many green-threading systems. The Erlang supervision system and the ability to replace code on the fly, not so much.
kstrauser 12y ago
If this were my team, I would be unsettled by the fact that we never caught it in testing. Did no one write tests to exercise this part of the app - the one where we're handcrafting HTTP requests? Objectively, you need to write more tests. At the minimum, this bug should have a regression test so that it can never accidentally happen again (say when a dev merges an old branch in for whatever reason).
- shortstuffsushi 12y ago
  What test would you have written to catch this? One that checks the exact contents of headers passed along? It's possibly they even had tests around this, but were expecting the same output that they were inputting (copy+pasta). Perhaps they had a more "integration"-ee test that actually hit the web with that bad header. At the point they wrote it, that test would have been passing. It wasn't until the parsing server changed (to Coyote, it seems) that the test would have started to fail.
  - kstrauser 12y ago
    Yes, I would have written a test to confirm that input_a generates output_b. The first half of that function is nothing but a string builder and easily testable. If they were copy-and-pasting the actual output to get the expected output, then yes: they screwed that part up. I'm far from a TDD purist, but it's clearly true that they're not sufficiently validating their code. If they had been, this would not have happened. I'm not saying this as an attack on their skills as programmers, but as caution to others reading the story: you have to - have to - test your stuff. It's one thing to lean on third-party libraries and expect them to mostly Do The Right Thing, especially if they're popular and come from a culture of valuing test coverage. If you're writing a Rails app, for instance, you might be forgiven for not writing your own independent validations of the Ruby methods you call. But writing string-building code to implement RFC-defined network protocols? You should have some confidence that your program is generating the output that the other party will be expected. Especially with something as commonly proxied as unencrypted HTTP; you just have to assume that your data will be traversing and analyzed by systems 100% outside of your control.
robomartin 12y ago
Kudos for sorting this out quickly. Problems like this one can be really difficult to debug. I remember one case where the coefficient table for a polyphase FIR filter we implemented in an FPGA caused huge instability problems in a design. The coefficient table, if I remember correctly, was 32 wide (32 multipliers) and 128 phases long. That's 4096 numbers. The design had about 40 of these tables that would be loaded from firmware into FPGA registers in real time as needed. We built a tool in Excel to be able to compute these tables of FIR coefficients. We got word from a customer that things were not behaving correctly under certain circumstances. We were able to reproduce the problem in the lab but could not find anything wrong with the FPGA, microcontroller or Excel code after about three weeks of work by three engineers. This quickly became a nightmare as it threatened several lucrative contracts and failed to service our existing customer base adequately. I had to put our other two hardware engineers back to work on their existing projects so I took on the debugging process. This was the most intense debugging I've had to do in thirty years of software and hardware development. Lots at stake. The very reputation and financial well being of my business was at stake. Enter 18 hour days, 7 days a week. FOUR MONTHS LATER, at 2:00 AM on a fine Sunday morning without having slept for three days looking at code the bug jumped out at me. We've all had that moment but his one was well "one of those". The problem? We used "ROUND()" in instead of "ROUNDUP()" in calculation that had nothing to do with the FIR filter coefficients but rather affected the programming of counters related to them. This caused timing errors in a state machine that drove the FIR filters. If this were software this would be exactly like having the wrong count in a loop counter. Yup. I re-calculated after making the change and everything worked as advertised. That was the best Monday I've had in years. And I took a long vacation after that. Over four months to find a bug. That's why sometimes it is impossible and even unreasonable to create budgets for software development. One little bug can set you back weeks, if not months.
asveikau 12y ago
strcpy( ( char * ) commsOrderBuffer, "GET /v1/printer/"); strcat( ( char * ) commsOrderBuffer, ( char * ) settings.getIMEI()); strcat( ( char * ) commsOrderBuffer, "/orders.txt HTTP/1.1\r\n"); strcat( ( char * ) commsOrderBuffer, "HOST: "); strcat( ( char * ) commsOrderBuffer, SERVER_NAME); strcat( ( char * ) commsOrderBuffer, "\r\n"); strcat( ( char * ) commsOrderBuffer, "Authorization: Basic "); What the.... O(n) string concatenations, unnecessary pointer casts, no bounds checking... I think extra whitespace in an HTTP request is not their only problem.
- ams6110 12y ago
  Those would be "safe" (assuming that settings.getIMEI() is completely under your control, everything else is string literals) but yeah snprintf seems way better here (though it's been well over 20 years since I wrote any significant C code.
  - asveikau 12y ago
    Possibly safe but definitely inefficient, since it has to find the end of the string to know where the destination pointer starts. The right way is to keep a pointer to the end. (Or since they are already using std::string in other places, maybe just do that everywhere, I'm sure it makes better choices than they did here.) The pointer cast thing is glaring. Why not simply declare the buffer as a char array and be done with it, instead of casting at every use? IMO over-use of pointer casts is a clear sign someone is lost in the language, your goal should be to reduce them.
    - ams6110 12y ago
      Yeah agree, to me casts like that are a smell that someone is trying to squash compiler complaints rather than understanding them. It also has every appearance of "copy/paste" code writing.
  - drblast 12y ago
    Since these are all string literals you really don't need any concatenation function at all except to concatenate with the output of getIMEI(). char *a = "Hello " "world!"; Works just fine. Edit to add: You can really see the difference in code between someone coming to C/C++ from a high level language and someone who learned assembly first, where a list of literals is a common idiom. The original style is not functionally wrong, but it does look like Java :-) Also: DON'T post your potentially insecure string handling code on the Internet; are you crazy?
peterwwillis 12y ago
Assuming the problem originates from something relating to eatabit's infrastructure, the important takeway (for me) would be: Depend as little on 3rd parties as possible. I know this is not a popular opinion among the HN crowd, mainly due to the entire web's love of linking to some other site's js/css to offload cost from their own site. But this makes no sense; you're not really reducing costs, you're just delaying them. People talk about how 3rd parties speed up development or (potentially) reduce costs. But if the success of your business depends on providing a service all the time that has to be reliable, the reliability of your product is directly proportional to the reliability of the 3rd party. And each 3rd party adds additional points of failure. If you don't control whatever service or product the 3rd party is giving you, you will be unable to even attempt to isolate and fix it yourself. Typically the answer to this problem is 'buy a better service contract'. But if the 3rd party doesn't provide 24/7 365 support along with multiple contact methods and harsh penalties for failing to supply you with timely service, you're wasting your money. You don't want to be the guy who has to tell the CIO "Sorry, I can't get a hold of our service provider or they aren't giving me timely updates, so I do not know when our product will be up again."
- stevewilhelm 12y ago
  > Depend as little on 3rd parties as possible. This attitude has many a startup reinventing and supporting commodity infrastructure instead of focusing on developing unique products and value for their customers.
weissadam 12y ago
I have some advice. Hire a real C programmer. This code is _awful_ and probably full of vulns.
mikeklaas 12y ago
q
Animats 12y ago
Another broken network device which takes it upon itself to mess with TCP connections passing through. I ran into this a few years ago with Coyote Point load balancers. It turns out that if you send HTTP headers to a Coyote Point load balancer, and the last header field is "User-agent", and that field ends with "m" but does not otherwise contain "m", the connection does not go through the load balancer. Complaining to Coyote Point produced typical clueless responses such as "Upgrade your software". (The problem wasn't at my end, but at sites with Coyote Point devices. Fortunately, I knew someone who had a Coyote Point unit, and we were able to force the situation there.) I had our system ("Sitetruth.com site rating system", note the "m") put an unnecessary "Accept" header field at the end of the header to work around the problem. Coyote Point's filtering software is regular-expression based, and I suspect that somewhere, there is a rule with a "\m" instead of "\n". A current issue: there are some sites where, if you make three HTTP requests for the same URL from the same IP address in a short period, further requests are ignored for about 15 seconds. You can make this happen with three "wget" requests. Try "wget http://bitcointalk.org" http://bitcointalk.org" three times in quick succession. Amusingly, this limiter only applies for HTTP sessions, not HTTPS.
goleksiak 12y ago
Heroku came back and said: Looking through the system, I see that you were sent two emails (in August and September) as several of your apps were migrated to the new routing stack (https://devcenter.heroku.com/articles/heroku-improved-router https://devcenter.heroku.com/articles/heroku-improved-router). As mentioned in the documentation, the new router follows stricter adherence to the RFC specification, including sensitivity to spaces. ...and sure enough, there is a line that says: The request line expects single spaces to separate between the verb, the path, and the HTTP version. So the lesson is: RTFM -G
- sixwing 12y ago
  The team at Heroku (where I currently PM) is constantly trying to improve our communication and documentation. We're definitely sorry that this caused problems, and we'll work even harder to make sure that our communication calls out any potential issues. Again - thanks for reaching out to us, and let us know if we can help.
  - goleksiak 12y ago
    Heroku did their best here. They reached out to us (twice) advising of changes and linking to a document that describe EXACTLY the bug that we discovered (later). Honestly, I don't feel bad about his bug because even if I would have read the alert to the letter, we would not have audited the entire codebase because we don't have that luxury of time. Yes, it took our whole operation down...but we found it, fixed it and now we're back up. ...it's all in the game. -@eatabit
    - jacquesm 12y ago
      That's all true but once you start accepting illegal input on a protocol for a long enough time you can't just suddenly go and break things without an automated alert to the customer when that particular thing starts acting up. After all it would not be that hard to scan for which customers are going to be bitten by that particular change when it actually happens rather than using some fire-and-forget email.

333 points by goleksiak 12y ago

vvpan 12y ago

Way to abuse :first-letter.

deleted 12y ago

[deleted]

jim_lawless 12y ago

I experienced a similar problem with a POP3 utility that I had written years ago. I had been appending an extra space to the end of each text line (before the CRLF ). There were a few people using this utility with no problems until one day a particular POP3 server no longer tolerated my utility's malformed requests.

mml 12y ago

Cowboy is quite a well respected we server of the Erlang flavor. I'd guess heroku rejiggered something in their stack, perhaps adding cowboy as a reverse proxy or load balancer in front of their junk. Cowboy apparently shot yor no-good dirty sidewinding web requests in the face.

mqsiuser 12y ago
It is well known that you can't (should not) rely on bugs (or internal APIs)

deleted 12y ago

[deleted]

kirab 12y ago

It's technically correct, according to the HTTP spec there must be a single "SP" character between the elements in the Request-Line: Request-Line = Method SP Request-URI SP HTTP-Version CRLF Source: http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1 http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1

spydum 12y ago

The Server: cowboy tag is from an Erlang web server: https://github.com/ninenines/cowboy/blob/master/src/cowboy_protocol.erl#L177 https://github.com/ninenines/cowboy/blob/master/src/cowboy_p... I'm guessing around here would be interesting to add a test case to handle. As far as whose server this is? I'd guess Heroku or AWS, though it's plenty possible T-Mobile could have devised some proxy to inspect traffic, but seems unlikely they would do so with Cowboy?

lostcolony 12y ago
T-mobile is known to have, in the past at least, used Erlang. And Cowboy is one of the most popular web servers within that community at this point.
mischanix 12y ago
It's simple enough to single out Heroku: $ cat <<EOF | nc example.herokuapp.com 80 GET /test HTTP/1.1 EOF ---- HTTP/1.1 505 HTTP Version Not Supported
- davidw 12y ago
  Heroku use Erlang: http://www.erlang-in-anger.com/ http://www.erlang-in-anger.com/
- 3ds 12y ago
  Your example fails with or without the whitespace. These work though: Request printf 'GET / HTTP/1.1\r\nHost: example.herokuapp.com\r\n\r\n' | nc example.herokuapp.com 80 Response HTTP/1.1 200 OK Connection: keep-alive Server: SimpleHTTP/0.6 Python/2.7.6 Request printf 'GET / HTTP/1.1\r\nHost: example.herokuapp.com\r\n\r\n' | nc example.herokuapp.com 80 Response HTTP/1.1 505 HTTP Version Not Supported Connection: close Server: Cowboy
  - mischanix 12y ago
    Ah right, forgot about the newline specification. I guess, for reference, the smallest string I can come up with to get Cowboy to spit that error message is '\x20\x20\n'. Parsers are fun.

jrochkind1 12y ago

This very example -- requests were technically illegal all the time without devs realizing, but something in the stack changed to start rejecting them -- demonstrates the fallacy of the "be liberal in what you accept, strict in what you issue" principal. If all the web servers involved had been strict in rejecting the illegal request from the start, they would have noticed the bug in development before deploying to firmware in the field.

ajanuary 12y ago
Doesn't it just demonstrate that you shouldn't switch from being liberal to being strict? For it to hold up, you need to provide the further argument that you frequently need to switch from liberal to strict.
- antirez 12y ago
  Or... here the problem is that the "be liberal in what you accept" design principle failed to be captured by the HTTP specification writer, that forced a single SP character. It looks like a specification issue to me to use a syntax which is very prone to errors, and is even not much visible (you can't easily inspect double spaces in protocol traces when checking just with your eyes), and then be strict about it. Even changing separator, if you want to be strict, already helps, like in "foo|bar|zap" compared to "foo bar zap". Humans are strage: many will spot "foo||bar|zap" as an error, but not "foo bar zap" as an error as serious as the previous one.
  - jdmichal 12y ago
    Especially when you have display technologies like HTML that will actively compress whitespace... As happened in your last example.
mqsiuser 12y ago
> the fallacy of the "be liberal in what you accept, strict in what you issue" principal The market (players) (can) manipulate it to create an (perceived) competitive advantage. It's also a source where "evil" in IT comes from.

kyberias 12y ago

What's the deal with all the scrollbars on this page?

spindritf 12y ago
Yeah, why is every image, heading, and paragraph on that page surrounded by scrollbars where most don't work and are not necessary?
- goleksiak 12y ago
  What browser are you using? I'm not seeing that here on my devices...? We are using pretty vanilla Bootstrap.
  - anonymoushn 12y ago
    http://i.imgur.com/v0b1LKC.png http://i.imgur.com/v0b1LKC.png This is what I see in Chrome on OSX.
  - ultrafez 12y ago
    The issue is you've got the following CSS on lines 31-33 of blog.css: body.blog .blog-post p { overflow: scroll; }
  - dceddia 12y ago
    Are your devices Apple devices by any chance? We had a similar bug reported at work recently, and it turned out that Windows browsers will always show scrollbars but the ones running on OS X/iOS will hide them until you start scrolling. To turn them on in OS X, go to System Preferences > General and set Show scroll bars to "Always".
- ultrafez 12y ago
  In their CSS, they have a rule for every <p> tag to have "overflow: scroll" for some reason. Not sure why they didn't use the default value for overflow, since there's nothing on that page that needs to be specifically told to scroll.

colinbartlett 12y ago

It scares me to think all of these requests run over unencrypted HTTP.

davidrusu 12y ago
Why? it's just pizza
- DanBC 12y ago
  Pizza has been used as a tool of harassment in the past. People order lots of pizza from different places for the victim, who then has to deal with a bunch of angry pizza drivers and being black-listed from those pizza places. Pizza drivers are often the victims of crime. Not only for the small amounts of cash that they carry, but sometimes just for the pizzas. edit: I should say that my comment here was a kneejerk reaction to "it's just pizza", and has nothing to do with how eatabit.com deals with this kind of harassment. i agree with other commenters that blog posts like these are a great way to promote the company.
  - pliny 12y ago
    How are you going to exploit the fact that these pizza orders aren't encrypted to achieve either of those things?
    - DanBC 12y ago
      PIZZA IS SRS BSNS
    - pjc50 12y ago
      Encrypt the pizzas.
    - serf 12y ago
      that's the wrong question to ask. We're not (all) in the "think-of-things-to-do-with-stolen-information" business like so many others are; but many of us are we're in the "encrypt-all-the-things-so-that-information-isn't-stolen" business.

rcconf 12y ago

I've had the same issues when developing with Flask in Python. I forgot to URL encode some query parameters and it worked fine with the local HTTP server. But when I put nginx in front as a proxy, it denied all requests.

stevekemp 12y ago
The thttpd webserver doesn't handle requests with too many slashes either, which I only found out recently This is treated as an invalid request: http://example.com//robots.txt
- zimpenfish 12y ago
  Unless I'm reading RFC 3986 incorrectly, that's valid because you can't have an empty segment in the path part of a URI.
  - ajanuary 12y ago
    I think you're reading it incorrectly. You can have an empty segment in the path. The BNF for a segment is: segment = *pchar Which according to RFC2234 section 3.6 means zero or more repetitions.
    - gpvos 12y ago
      But then the server may still decide that an empty segment is so meaningless that it will refuse it. In fact, it would not be a smart move to just treat double slashes the same as single ones, because of relative URLs: a ".." segment only removes one slash, so the hierarchy levels would get messed up. thttpd is doing the smart thing here. As one of my teachers at university would say: the empty segment is also a segment.
      - lmm 12y ago
        The server can of course interpret the path as it wants, but it should allow an application running under the server to give 'foo//bar' a meaning if that application wants to, IMO.

ericcholis 12y ago

Slightly off-topic, but this is why dev posts like this are important. I didn't know eatabit.com was a thing, it it sounds like a great service.

robogeek78 12y ago
Dev #2 here thanks for the compliment. Where are you? Maybe we should expand to your area. :)
- ericcholis 12y ago
  Buffalo, NY. We're quite proud of our local restaurant industry. So, yes, you should!

jlouis 12y ago

This proves a very important pet peeve of mine: Your modern application has a highly dynamic operating point. There is no way you can deploy a system and expect it to be static for eternity. Back in the day with low interconnectivity you could. But today it is impossible. When you build stacks on top of system for which you have no direct control, you must be able to adapt your system. This means you can't statically deploy code without an upgrade path in one way or the other.

goleksiak 12y ago
True but that doesn't bother me. Nothing is static on the web these days and everyone plays under the same rule set. Keeps things interesting...
- jlouis 12y ago
  It shouldn't bother you. It is just how moderns systems are.
quesera 12y ago
You're combining two issues. Yes, if you let other people run your infrastructure, you are beholden to their operations decisions and schedules. It is not impossible to design around that (new) problem, but it is sometimes expensive. The trick is to know what external dependencies you have, and that is almost impossible to fully quantify in the XaaS and cloud model.

Danieru 12y ago

That series of strcat's caught my eye as bad practice. Fine in this case since the destination string is short but horrible in general. Every single one of those calls needs to iterate over the entire existing string to find the string size. The code could be much cleaner with a small macro hiding the incrementation and the casts.

userbinator 12y ago
Basically it's an O(n^2) algorithm... well-known story about that here: http://www.joelonsoftware.com/articles/fog0000000319.html http://www.joelonsoftware.com/articles/fog0000000319.html The design of strcat() itself is partially to blame for this - the return value could've been more useful, like the number of characters in the resulting string or a pointer to the end of the appended string so it could be used to chain concatenations, but instead they chose to return the exact same pointer that was passed in as the source.
mikeash 12y ago
A sufficiently smart compiler could optimize a string of strcat calls to remove the redundant length finding. I have no idea if real compilers actually would....
- srtjstjsj 12y ago
  Java does, for the "+" operator.
  - mikeash 12y ago
    Oh yes, nice example! So there's precedent for C compilers doing something similar.
krakensden 12y ago
Yeah- implicit concatenation + snprintf seems like the way to go. Although you'd have to calculate a length, I suspect avoiding that is the primary virtue of this approach.

justinsb 12y ago

Tangentially, why didn't curl escape the trailing space to %20?

userbinator 12y ago

I saw it right away - "that HTTP/1.1 looks a bit farther away than it should be..." - and confirmed it by selecting the spaces. I thought it would be a bit more subtle than that... I remember working with a server that violated the HTTP spec by not accepting allowed extra spaces in headers. According to the new HTTP/1.1 RFC 7230, it should be a single space - the previous RFC didn't specify this clearly in the wording, although it is implied by the grammar (SP and not 1 * SP). https://tools.ietf.org/html/rfc7230#section-3.1.1 https://tools.ietf.org/html/rfc7230#section-3.1.1 "A request-line begins with a method token, followed by a single space (SP), the request-target, another single space (SP), the protocol version, and ends with CRLF." I'm surprised there doesn't seem to be any widely-used and easily available HTTP conformance checker - unlike the well-known HTML validators. This is also why monospace fonts are ideal for seeing small but significant differences like this.

michaelmior 12y ago
That's an interesting idea. It would be useful to have a Web server where the output is just a conformance check of the request. That might be a fun project for a rainy day :)
- zimpenfish 12y ago
  Sounds like something that could be added to http://httpbin.org http://httpbin.org
  - userbinator 12y ago
    That runs on Python/Flask, which is already a layer of abstraction above where HTTP conformance testing would be; what you need is something that listens on a TCP socket and parses the requests itself.
    - zimpenfish 12y ago
      Actually, thinking about it, didn't Zed Shaw make a Ragel-based strict-conformance HTTP parser? > Simply being more explicit about what is valid HTTP means that most of the security attacks that worked on Apache were rejected outright when tried on Mongrel. Which I guess is a qualified "sounds like it, maybe?"

pilif 12y ago

Likely "Cowboy" is a transparent proxy added by your mobile service provider. I had a similar thing happening a year ago when the mobile provider used by most of our barcode scanners decided to add a transparent proxy into the loop (without telling anybody). The solution for this problem: Use SSL. I mean: There are already many good reasons to use SSL, but whenever you need to send any kind of mission critical data over the mobile network, you practically must use SSL if you want any kind of guarantees that the data you send to the server is what actually reaches the server (and reverse). Here's my war story from last year: http://pilif.github.io/2013/09/when-in-doubt-ssl/ http://pilif.github.io/2013/09/when-in-doubt-ssl/

ctz 12y ago
This would require a vast, vast upgrade of client power to achieve the same communications performance. If you could achieve it all, SSL would also likely decrease reliability over a spotty GSM link.
- pjc50 12y ago
  Is client CPU actually a limiting factor? How does this affect reliability?
  - serf 12y ago
    as someone unfamiliar with their specific workload, but who has used such boards to do vpn/ssl stuff, no -- it's not a limiting factor unless trying to skim power requirements.
  - ctz 12y ago
    You cannot physically fit a whole SSL datagram (max size 16KB) into 8KB of RAM. SSL requires multiple passes over the data to (eg) decrypt and verify a datagram. At this point, you cannot use standard SSL at either the server side or client side. On the subject of reliablity: a 8 bit uC running at 16MHz needs a long time to do the public key crypto required to set up the connection. This means you need a GSM data link to be continuously available for a longer period.

KMag 12y ago

When learning OCaml, I decided to write a little web client that would bruit force the password on my own home router. I wrote a client, and my router wasn't responding, so I tried having my client fetch pages from Yahoo, and it worked fine. I fired up wireshark and saw that everything looked fine... except that all of my line terminators were shift-in-formfeed instead of carriage-return-newline. It turns out that OCaml uses decimal character escapes instead of octal. (This was back when I was under the impression that portable code avoided use of \n in string literals because someone who misunderstood text mode file handles had told me that Microsoft compilers expanded \n to \015\012.) Apparently someone at Yahoo had experienced enough terribly terribly written web clients that they wrote their HTTP server to accept any two non-space whitespace characters as a line ending.

cofcdylan 12y ago

i'm just glad my city made it to HN.

goleksiak 12y ago
Charleston represent!

jameshart 12y ago

"our cellular printing api has printed over 9300 food orders for our client restaurants, stadiums and golf courses" Am I the only one who read this as a system using 3D printing to print food? Disappointed to discover it's not that kind of cellular.

cleanCodeAtWork 12y ago

Are there any languages out there that handle scale and many connections like Erlang does, but with an easier to swallow syntax?

hlieberman 12y ago
Erlang. The syntax really isn't that bad, once you get over the initial shock. In all honesty, grasping that the variables are immutable and how you need to change your thinking is much more difficult than the syntax itself.
angersock 12y ago
http://elixir-lang.org/ http://elixir-lang.org/ It's the Erlang VM you love, but with the Ruby syntax we all enjoy!
buster 12y ago
http://elixir-lang.org/ http://elixir-lang.org/
lmm 12y ago
I've been very happy writing these things in Scala using Spray. Honestly there are plenty of event-driven I/O frameworks in many languages, and almost as many green-threading systems. The Erlang supervision system and the ability to replace code on the fly, not so much.

kstrauser 12y ago

If this were my team, I would be unsettled by the fact that we never caught it in testing. Did no one write tests to exercise this part of the app - the one where we're handcrafting HTTP requests? Objectively, you need to write more tests. At the minimum, this bug should have a regression test so that it can never accidentally happen again (say when a dev merges an old branch in for whatever reason).

shortstuffsushi 12y ago
What test would you have written to catch this? One that checks the exact contents of headers passed along? It's possibly they even had tests around this, but were expecting the same output that they were inputting (copy+pasta). Perhaps they had a more "integration"-ee test that actually hit the web with that bad header. At the point they wrote it, that test would have been passing. It wasn't until the parsing server changed (to Coyote, it seems) that the test would have started to fail.
- kstrauser 12y ago
  Yes, I would have written a test to confirm that input_a generates output_b. The first half of that function is nothing but a string builder and easily testable. If they were copy-and-pasting the actual output to get the expected output, then yes: they screwed that part up. I'm far from a TDD purist, but it's clearly true that they're not sufficiently validating their code. If they had been, this would not have happened. I'm not saying this as an attack on their skills as programmers, but as caution to others reading the story: you have to - have to - test your stuff. It's one thing to lean on third-party libraries and expect them to mostly Do The Right Thing, especially if they're popular and come from a culture of valuing test coverage. If you're writing a Rails app, for instance, you might be forgiven for not writing your own independent validations of the Ruby methods you call. But writing string-building code to implement RFC-defined network protocols? You should have some confidence that your program is generating the output that the other party will be expected. Especially with something as commonly proxied as unencrypted HTTP; you just have to assume that your data will be traversing and analyzed by systems 100% outside of your control.

robomartin 12y ago

Kudos for sorting this out quickly. Problems like this one can be really difficult to debug. I remember one case where the coefficient table for a polyphase FIR filter we implemented in an FPGA caused huge instability problems in a design. The coefficient table, if I remember correctly, was 32 wide (32 multipliers) and 128 phases long. That's 4096 numbers. The design had about 40 of these tables that would be loaded from firmware into FPGA registers in real time as needed. We built a tool in Excel to be able to compute these tables of FIR coefficients. We got word from a customer that things were not behaving correctly under certain circumstances. We were able to reproduce the problem in the lab but could not find anything wrong with the FPGA, microcontroller or Excel code after about three weeks of work by three engineers. This quickly became a nightmare as it threatened several lucrative contracts and failed to service our existing customer base adequately. I had to put our other two hardware engineers back to work on their existing projects so I took on the debugging process. This was the most intense debugging I've had to do in thirty years of software and hardware development. Lots at stake. The very reputation and financial well being of my business was at stake. Enter 18 hour days, 7 days a week. FOUR MONTHS LATER, at 2:00 AM on a fine Sunday morning without having slept for three days looking at code the bug jumped out at me. We've all had that moment but his one was well "one of those". The problem? We used "ROUND()" in instead of "ROUNDUP()" in calculation that had nothing to do with the FIR filter coefficients but rather affected the programming of counters related to them. This caused timing errors in a state machine that drove the FIR filters. If this were software this would be exactly like having the wrong count in a loop counter. Yup. I re-calculated after making the change and everything worked as advertised. That was the best Monday I've had in years. And I took a long vacation after that. Over four months to find a bug. That's why sometimes it is impossible and even unreasonable to create budgets for software development. One little bug can set you back weeks, if not months.

asveikau 12y ago

strcpy( ( char * ) commsOrderBuffer, "GET /v1/printer/"); strcat( ( char * ) commsOrderBuffer, ( char * ) settings.getIMEI()); strcat( ( char * ) commsOrderBuffer, "/orders.txt HTTP/1.1\r\n"); strcat( ( char * ) commsOrderBuffer, "HOST: "); strcat( ( char * ) commsOrderBuffer, SERVER_NAME); strcat( ( char * ) commsOrderBuffer, "\r\n"); strcat( ( char * ) commsOrderBuffer, "Authorization: Basic "); What the.... O(n) string concatenations, unnecessary pointer casts, no bounds checking... I think extra whitespace in an HTTP request is not their only problem.

ams6110 12y ago
Those would be "safe" (assuming that settings.getIMEI() is completely under your control, everything else is string literals) but yeah snprintf seems way better here (though it's been well over 20 years since I wrote any significant C code.
- asveikau 12y ago
  Possibly safe but definitely inefficient, since it has to find the end of the string to know where the destination pointer starts. The right way is to keep a pointer to the end. (Or since they are already using std::string in other places, maybe just do that everywhere, I'm sure it makes better choices than they did here.) The pointer cast thing is glaring. Why not simply declare the buffer as a char array and be done with it, instead of casting at every use? IMO over-use of pointer casts is a clear sign someone is lost in the language, your goal should be to reduce them.
  - ams6110 12y ago
    Yeah agree, to me casts like that are a smell that someone is trying to squash compiler complaints rather than understanding them. It also has every appearance of "copy/paste" code writing.
- drblast 12y ago
  Since these are all string literals you really don't need any concatenation function at all except to concatenate with the output of getIMEI(). char *a = "Hello " "world!"; Works just fine. Edit to add: You can really see the difference in code between someone coming to C/C++ from a high level language and someone who learned assembly first, where a list of literals is a common idiom. The original style is not functionally wrong, but it does look like Java :-) Also: DON'T post your potentially insecure string handling code on the Internet; are you crazy?

peterwwillis 12y ago

Assuming the problem originates from something relating to eatabit's infrastructure, the important takeway (for me) would be: Depend as little on 3rd parties as possible. I know this is not a popular opinion among the HN crowd, mainly due to the entire web's love of linking to some other site's js/css to offload cost from their own site. But this makes no sense; you're not really reducing costs, you're just delaying them. People talk about how 3rd parties speed up development or (potentially) reduce costs. But if the success of your business depends on providing a service all the time that has to be reliable, the reliability of your product is directly proportional to the reliability of the 3rd party. And each 3rd party adds additional points of failure. If you don't control whatever service or product the 3rd party is giving you, you will be unable to even attempt to isolate and fix it yourself. Typically the answer to this problem is 'buy a better service contract'. But if the 3rd party doesn't provide 24/7 365 support along with multiple contact methods and harsh penalties for failing to supply you with timely service, you're wasting your money. You don't want to be the guy who has to tell the CIO "Sorry, I can't get a hold of our service provider or they aren't giving me timely updates, so I do not know when our product will be up again."

stevewilhelm 12y ago
> Depend as little on 3rd parties as possible. This attitude has many a startup reinventing and supporting commodity infrastructure instead of focusing on developing unique products and value for their customers.

weissadam 12y ago

I have some advice. Hire a real C programmer. This code is _awful_ and probably full of vulns.

mikeklaas 12y ago

Animats 12y ago

Another broken network device which takes it upon itself to mess with TCP connections passing through. I ran into this a few years ago with Coyote Point load balancers. It turns out that if you send HTTP headers to a Coyote Point load balancer, and the last header field is "User-agent", and that field ends with "m" but does not otherwise contain "m", the connection does not go through the load balancer. Complaining to Coyote Point produced typical clueless responses such as "Upgrade your software". (The problem wasn't at my end, but at sites with Coyote Point devices. Fortunately, I knew someone who had a Coyote Point unit, and we were able to force the situation there.) I had our system ("Sitetruth.com site rating system", note the "m") put an unnecessary "Accept" header field at the end of the header to work around the problem. Coyote Point's filtering software is regular-expression based, and I suspect that somewhere, there is a rule with a "\m" instead of "\n". A current issue: there are some sites where, if you make three HTTP requests for the same URL from the same IP address in a short period, further requests are ignored for about 15 seconds. You can make this happen with three "wget" requests. Try "wget http://bitcointalk.org" http://bitcointalk.org" three times in quick succession. Amusingly, this limiter only applies for HTTP sessions, not HTTPS.

goleksiak 12y ago

Heroku came back and said: Looking through the system, I see that you were sent two emails (in August and September) as several of your apps were migrated to the new routing stack (https://devcenter.heroku.com/articles/heroku-improved-router https://devcenter.heroku.com/articles/heroku-improved-router). As mentioned in the documentation, the new router follows stricter adherence to the RFC specification, including sensitivity to spaces. ...and sure enough, there is a line that says: The request line expects single spaces to separate between the verb, the path, and the HTTP version. So the lesson is: RTFM -G

sixwing 12y ago
The team at Heroku (where I currently PM) is constantly trying to improve our communication and documentation. We're definitely sorry that this caused problems, and we'll work even harder to make sure that our communication calls out any potential issues. Again - thanks for reaching out to us, and let us know if we can help.
- goleksiak 12y ago
  Heroku did their best here. They reached out to us (twice) advising of changes and linking to a document that describe EXACTLY the bug that we discovered (later). Honestly, I don't feel bad about his bug because even if I would have read the alert to the letter, we would not have audited the entire codebase because we don't have that luxury of time. Yes, it took our whole operation down...but we found it, fixed it and now we're back up. ...it's all in the game. -@eatabit
  - jacquesm 12y ago
    That's all true but once you start accepting illegal input on a protocol for a long enough time you can't just suddenly go and break things without an automated alert to the customer when that particular thing starts acting up. After all it would not be that hard to scan for which customers are going to be bitten by that particular change when it actually happens rather than using some fire-and-forget email.