9 ms·

App Engine charges $6,500 to update a ListProperty on 14.1 million entities

221 points by branola 15y ago

dangrossman 15y ago
> it cost me a few thousand dollars to delete my millions enities from the datastore after a migration job (ikai never replied my post though...) and im still paying since the deletion is not completed yet (spending 100-300$ a day for the past 2 weeks now!!). I don't know much about GAE, but a datastore-as-a-service that takes 2 weeks to delete your data and charges $300 a day to do so just seems... absurd.
- ntkachov 15y ago
  Well, If you run a few million entries, each one needing to write an individual entry and each entry takes 200 ms to write, that's your 2 weeks right there. I don't know how they are using GAE and if there is an alternative to their methods, but I personally always saw GAE as a very specialized platform. You really only should use it if you need to tie into a google account.
  - ryandvm 15y ago
    Even then it's not necessary. It's straightforward enough to use OAuth on any platform. In my experience, App Engine is mainly useful for in-house infrastructure apps for companies using the Google Apps platform. That or cheapskate developers throwing together a proof of concept / toy app in their spare time.
    - MatthewPhillips 15y ago
      Even toy apps can be had for free on other platforms. Dotcloud, Heroku, and AppHarbor all don't charge for the first instance.
- diego 15y ago
  In my opinion Google App Engine is a non-starter for serious applications. I only know of experiments and toys running there. Look at their gallery of successful applications: http://code.google.com/intl/it-IT/appengine/casestudies.html http://code.google.com/intl/it-IT/appengine/casestudies.html Contrast with AWS: http://aws.amazon.com/solutions/case-studies/ http://aws.amazon.com/solutions/case-studies/
  - chucknthem 15y ago
    There are definitely serious applications and businesses built on appengine, mostly started before their price hike coming out of Beta last November. http://www.optimizely.com http://www.optimizely.com (YC W10) and http://shoesofprey.com http://shoesofprey.com are two notable examples.
    - taylorbuley 15y ago
      Rescuetime (YC W08) also uses it I believe
    - avree 15y ago
      We're built on appengine at Getaround.
    - dextorious 15y ago
      I wouldn't call those two "serious" examples -- especially since the competitors (Amazon, Heroku, Rackspace etc) have orders of magnitude more popular, complex and profitable services running on them. This is like saying, "Famous actors frequent my restaurant" and point to a picture of Ralph Macchio. OK, somewhat known, but De Niro, Brad Pitt, Clooney and co eat at the joint across the street.
      - jay_kyburz 15y ago
        Actually, I think its great for startups. If you're a small team trying to work out what your product is, how to sell it and who is going to buy it, you have more important things to do than learn how to set up and run a server. Once you've worked out you have a product that people want you can start thinking about managing your own servers. People complain about migration, but its really not that big a deal.
- bemmu 15y ago
  I love GAE, but deleting stuff is really the major WTF point. I am about to shut down one application that declined in popularity, because it costs me $20 / week to run it and revenue just dropped under $20 / week. The cost is not from instance hours, but purely from the stored data. Deleting the data from the data store would cost more than I could recoup, so that is not an option either. Also I really feel frustrated giving hours of thought to something that should be a really simple operation. Perhaps .delete() should be free? After all, when I shut down the app, Google does delete everything for free.
  - jdeibele 15y ago
    My suggestion would be to turn off billing. That's what I did when the price went up on an app I had running and I didn't want to deal with deleting the data.
seanp2k2 15y ago
Google has a great platform here with lots of potential if they opened it up more, but they're pricing themselves into a corner for this already-niche service. Sadly, this is kind of "typical Google" -- great product, decent execution, but a bad identity problem -- it really feels like they're not sure yet what they want to do with this.
- salimmadjd 15y ago
  This is all because of their new pricing model. Overnight our pricing went up by 5X and that was after 50% discount, which make it really a 10X hike. Here is a graph of it: https://plus.google.com/114790424055754975707/posts/eUMhYDVf6i5 https://plus.google.com/114790424055754975707/posts/eUMhYDVf...
j45 15y ago
I wonder what a managed or dedicated server would cost to perform the same calculations. Sometime's it's still cheaper to have your own managed / self-managed gear... and from the looks of this pricing, even hire someone fulltime/freelancing to manage it all for you.
- smilliken 15y ago
  >"I wonder what a managed or dedicated server would cost to perform the same calculations." It wouldn't cost you any money (unless you have metered electricity), but rather just opportunity cost of being able to do other work with your resources. Unless you only need a short-term lease on the equipment, cloud servers will be more expensive that dedicated/colocated servers.
  - j45 15y ago
    I have run my own servers in a datacenter for 7-8 years and going to a VM setup was the quantum leap. I can copy and paste a server in minutes and be up and running with a nearly identical setup that I have to setup for my dev environment anyways. Hardening the server is something that can be outsourced for a lot less than thousands. Services like linode seem to be a nice middle ground. While I don't see myself going back to my own hardware in a rack i run in a datacenter, I do still see the benefit of knowing your stack a bit beyond coding. Knowing how the stack works helps when building software quite often. Anyhow, those are just my experiences. VPS' with a very strong toolkit to take the edge off self-administering like Linode, etc, seem to be a very nice option. Heroku has caught my eye too but they have completely different measurements.
- charliesome 15y ago
  s/sometimes/all the time/; I honestly do not get why people are so fascinated with the cloud. It's a very expensive way to avoid having to know what you're doing.
  - Gotperl 15y ago
    Like any other tool "the cloud", as you say, can be the right fit for the right job.
ajross 15y ago
Reading the thread (I'm curious about GAE, not an expert), it seems like the details aren't clear at all. Neither the original poster nor the Google rep seem to have a clear idea of what I/O operations are being generated. But this bit stood out: $0.10 per 100k writes. That price seems to be far too high. The poster is doing (something like) a reindex of 10M entries (that kind of data is pretty small really: it's the kind of database you might use as a test set on your laptop interactively). Figure each modification is atomic, and that the b-tree height of the storage is ~4. So that's 40M writes to create an index, or $400! Seriously? Again, this is the kind of task you'd expect to do quickly and interactively on your development box, and it costs a price of the same order as your day's salary (!) to execute in the cloud? Looking at this from the perspective of the underlying I/O device: this index consumes just a tiny, tiny fraction of a hard disk drive's capacity. Yet creating it costs enough to buy the device several times over? Something is wrong. Is that a misquote or have I misunderstood?
- CurtHagenlocher 15y ago
  I believe that $0.10 per 100k writes is what both Windows Azure Storage and Amazon S3 cost. But honestly, the cloud is the future! Disclaimer: I work at Microsoft and am required by the terms of my employment to believe in "the cloud".
  - cperciva 15y ago
    I believe that $0.10 per 100k writes is what both Windows Azure Storage and Amazon S3 cost. Correct. Of course, those are object writes -- Elastic Block Store disk I/O is 10x cheaper.
    - ajross 15y ago
      Right. The granularity to be expected with an S3 object (~= "a file") is much higher than with a database index of tree nodes of a few kilobytes each. And of course EBS is subject to caching on the host. Rolling the index into a transaction can eliminate almost all the duplicate I/O to the tree nodes.
- 15y ago
salimmadjd 15y ago
They jacked up our price by 5X, here is a nice graph of it: https://plus.google.com/114790424055754975707/posts/eUMhYDVf6i5 https://plus.google.com/114790424055754975707/posts/eUMhYDVf...
- magicalist 15y ago
  I understand you're angry, but did you really just post that same thing twice in this thread and twice in the linked thread??
- sologoub 15y ago
  Hate to take it off topic, but your site (http://www.f8daily.com/ http://www.f8daily.com/) is throwing errors: ValueError: Values may not be more than 1000000 bytes in length; received 1053462 bytes
  - troygoode 15y ago
    The image he linked said he was forced to shut down his site, so I'm not sure that he cares that there is an error.
endlessvoid94 15y ago
GAE has become completely infeasible as a hosting solution for me (ThatHigh.com). My hosting cost increased by 90x, and I did not get nearly enough notice. I don't have the time or resources to move the site, so I'm forced to shut it down. It really, really sucks. Personally I'm more disappointed by the lack of notice (1 month is nowhere near enough time) than the actual increase. I totally understand the need to charge.
- yesbabyyes 15y ago
  As a fan I urge you not to shut it down. Please consider selling it or something!
- Travis 15y ago
  Hey I shot you a tweet, I'd be interested in helping migrate the site. Email is in my HN profile, shoot me a message if you're interested.
- StavrosK 15y ago
  Did you turn multithreading on? It's helped on my sites somewhat.
- yahelc 15y ago
  If you implement optimizations, you can really significantly curb the cost. I spent some time tuning SharedCount's API, which would have cost me $30-$50/day, and its now at about $1-$2/day. - Move to Python 2.7 and enable multithreading - Setup Cloudflare (this swallows about half of all my requests) - Increase minimum latency and reduce the maximum number of idle instances. (I have 5-8s and 1-2 set, respectively) - Setup the semi-undocumented Google edge cache (basically, just a Cache-Control: public, max-age=[seconds] header. - Take advantage of memcache. With this setup, I'm doing 3 million API calls per day at $2.
  - cr4zy 15y ago
    Python 2.7 is experimental and requires migrating to the new datastore (high replication) which is a lot more expensive. So you might not want to do this if your costs are coming from datastore writes. Also, high replication queries can return stale results unless you use ancestor queries. Ancestor queries require putting entities in groups by giving them all the same parent (which can never be changed). Basically it's a very inflexible semaphore and kind of sucks IMO. Your suggestions in general are very good though. Thanks, I'm switching my DNS to CloudFlare now.
stickfigure 15y ago
The details have finally been posted in that thread. And while $6500 is a lot of money, we have to realize that this is a lot of writing - the headline doesn't give the full sense of it. For those unfamiliar with GAE, a ListProperty is really a collection of properties. The author is using the property as a geohash with a significant number of values, plus he has additional multiproperty indexes defined, plus he's doing a rewrite (delete + write). All combined it appears to be ~460 writes per entity. So what we're talking about is $6500 for 6.5 billion writes... exactly what is printed on the sales brochure. Is that a lot? Most datastores don't charge by the operation so I don't have a lot to compare it to. It seems expensive but not crazy, especially considering that the data is replicated via PAXOS to 3+ datacenters with automatic loadbalancing and failover.
- vosper 15y ago
  "Geoboxing is a technique used to search for entities near a point on the earth in a database that can only perform equality queries (like App Engine)" So their implementation is a compromise on account of GAE's limitations, and they have to pay through the nose to use it. This is when I'd be looking at hosting some features outside of GAE, which is what we do with Full-text Search.
  - stickfigure 15y ago
    No single platform or environment encompasses every use case. GAE doesn't offer R-trees or other spatial indexes; for that matter neither does MongoDB or MySQL. Geohashing is a reasonable solution for some spatial problem domains; it's one solution along the spectrum of "precalculate a lot up front and make queries cheap" vs "write in a cheap & easy format but make queries more expensive". Pre-calculation strategies are usually more scalable when you have large query loads, but they suck bigtime if you need to fully recalculate a large body of data (as the original blog author is doing). Maybe the blogger would be better off using PostGIS; but then, scaling and synchronizing a large cluster of PostGIS systems is nontrivial. The issues here are too application-specific to draw any positive or negative conclusions about appengine.
    - 15y ago
dextorious 15y ago
With Amazon AWS you can handle the scaling yourself, when you need it, with components tailored to your use cases, and better latency. And with Heroku you can have it taken care for you, following a few simple rules. So, why exactly would one use the crippled GAE platform, that constantly breaks its promises (re: reliability), forces you to code with very little flexibility (and, no, not every app that needs to automatically and massively scale "has to be coded exactly like a GAE app anyway"), costs a fortune (and sometimes an unexpected fortune), and breaks for you as soon as you need a technology not on offer?
latchkey 15y ago
A lot of these AppEngine costs too much notices have been coming up recently, but upon further inspection, they all tend to boil down to operator error. Unfortunately, AppEngine isn't forgiving of that and there is a real monetary value associated with questionable engineering design. Or, design that wasn't thought through enough in the context of a service like AppEngine. This leads to a few people getting upset and making a lot of noise when the reality is that AppEngine is actually an amazing service. So, to boil down the operator error from a quote in the thread: "We're running a mapreduce to change the geobox sizes/precision for a large number of entities." That is the real source of the problem. Instead of using geoboxes, they should be using geohashes, which allow arbitrary precision. http://code.google.com/apis/maps/articles/geospatial.html http://code.google.com/apis/maps/articles/geospatial.html http://en.wikipedia.org/wiki/Geohash http://en.wikipedia.org/wiki/Geohash Instead of an indexed property that looks like this (what they currently have): [u'37.3411|-121.8940|37.3395|-121.8926', u'37.3411|-121.8929|37.3395|-121.8916', ...] They would have an indexed List<String> property that looks like this: [8, 8f, 8f1, 8f12, 8f12a, 8f12ac, 8f12ac6, 8f12ac60, 8f12ac605, 8f12ac605f, 8f12ac605fb, 8f12ac605fb3, 8f12ac605fb34] Finding if the location is in a box would be computing the hash from the lat/lng (there is free code out there to do that) and then doing an indexed 'in' query. The indexes would only need to be updated if the location of the entity changes, not when they want varying levels of precision.
- erichocean 15y ago
  We wrote some App Engine code to do geospatial lookups. Before anyone gets the idea that the links in the parent are worth trying, they're not: the performance is absolutely atrocious – on our data set, 13 seconds per lookup. We wrote our own approach on App Engine and now get stable performance on our datasets at ~300ms per lookup. (We're doing Foursquare-type lookups.)
  - latchkey 15y ago
    I can't imagine how an indexed query on a List<String> of ~10 items takes 13 seconds. 'We wrote our own approach on App Engine' Hmm... details?
tuhin 15y ago
For those questioning the use of Google App engine as a serious platform for applications, we at Pulse use Google App Engine: http://googleappengine.blogspot.com/2011/11/scaling-with-kindle-fire.html http://googleappengine.blogspot.com/2011/11/scaling-with-kin... So, yes you can build serious applications on GAE but like everything else it boils down to, it depends on what you really need.
6ren 15y ago
I like the spot price idea (in the comments http://groups.google.com/group/google-appengine/msg/fe9a05c6868e086d http://groups.google.com/group/google-appengine/msg/fe9a05c6...). It's similar to adwords' automatic auction for how close to the top your ad is, with the same benefits of getting the best market price (for buyers and sellers) of a limited resource. If no one else is using it, it could become close to free. It also casts the other users as the opponent, instead of google.
IgorPartola 15y ago
GAE is a mistake. By that I mean that it's got a big design flaw that's bound to cost Google money, which means it'll always be expensive than alternatives. Consider shared PHP hosting: a request comes in, apache finds which PHP file is responsible for it, then directs the request to the PHP interpreter. The PHP interpreter will parse the file (or more likely load the parsed bytecode from a cache) and return a response, which apache will then forward to the user agent. Notice that aside from the cache the PHP interpreter is stateless. As soon as it is done serving a request from site foo.com it can immediately jump on a request from bar.com and the context switch doesn't cost anything (once again disregarding finite cache size issues). Contrast this with running a stand-alone application server for each site, which is what GAE does. Here, even if your code is not serving any requests it's still waiting to get them. Now, GAE has powerful magic in it to retire request handlers which aren't frequently used. This way if site foo.com is getting 1 request/minute, it only really needs one process/thread/hander abstraction at a time. However, it is expensive to start/stop these "processes", so instead GAE is forced to keep this "process" around for a while after a request has been served hoping that the cost of keeping it alive would be justified by a second request. Thus these stateful, slow-to-start processes are always taking up resources that could be used to serve other requests. Disclaimer: all my knowledge of GAE has been from reading their docs/blog, not from deploying projects to it. Disclaimer 2: I am not saying that PHP is better/worse than GAE in any way. However, I am saying that the model that GAE uses is more costly for a typical application. This can be easily seen by comparing the cost of running a basic site on GAE vs $2/month shared hosting.
- ww520 15y ago
  I don't understand the downvotes. This is a on-topic related detail discussion.
  - jbarham 15y ago
    Speaking for myself, I down-voted because of the absurd false dichotomy of comparing Google App Engine to $2/month shared PHP hosting! It's like complaining that an oil tanker costs more than a rowboat. The value proposition of App Engine is that with no systems administration expertise you can rent an extremely reliable, massively scalable web platform that is managed around the clock by a world class devops teams. Unsurprisingly this costs money. If you don't need the reliability or scalability of App Engine, no one is forcing you to pay for it. But it's absurd to suggest that you can get anything remotely comparable in PHP for $2/month.
tszming 15y ago
Besides the cost, if your startup can survive without SSL support on your own domain, go for App Engine. See: http://code.google.com/p/googleappengine/issues/detail?id=792 http://code.google.com/p/googleappengine/issues/detail?id=79... People requested custom SSL support at 2008, and today is 2012, if you still believe in App Engine, good luck!
- latchkey 15y ago
  http://googleappengine.blogspot.com/2011/10/app-engine-ssl-for-custom-domains-in.html http://googleappengine.blogspot.com/2011/10/app-engine-ssl-f...
  - tszming 15y ago
    See the last comment of my link: >> the "trusted tester program" is a joke . They never respond so it's just a waste of time . Even they launch this feature TODAY, so 4 years for a basic requirement, what you can expect from them?
    - latchkey 15y ago
      I can think of TONS of services I've used over the years, both in and out of the software field that have talked about adding features and never did. Big deal, don't play the victim. If you really wanted onto the trusted tester program, you'd bring it up on the app engine mailing list or contact someone at google directly (their emails are all over the place, Ikai is a great guy, and they are very responsive). I'm sure they'd be happy to have enthusiastic beta testers. I find great irony in your quote on your G+ profile: "Do you create anything, or just criticize others work and belittle their motivations? -- Steve Jobs"
      - tszming 15y ago
        Thanks for your reminder, I shouldn't belittle the enthusiasm of GAE's engineers/supporters, my bad. But as SSL support is not public yet, my statement above is still valid: If your startup can survive without SSL support on your own domain, go for App Engine! Good luck!

221 points by branola 15y ago

> it cost me a few thousand dollars to delete my millions enities from the datastore after a migration job (ikai never replied my post though...) and im still paying since the deletion is not completed yet (spending 100-300$ a day for the past 2 weeks now!!). I don't know much about GAE, but a datastore-as-a-service that takes 2 weeks to delete your data and charges $300 a day to do so just seems... absurd.

ntkachov 15y ago
Well, If you run a few million entries, each one needing to write an individual entry and each entry takes 200 ms to write, that's your 2 weeks right there. I don't know how they are using GAE and if there is an alternative to their methods, but I personally always saw GAE as a very specialized platform. You really only should use it if you need to tie into a google account.
- ryandvm 15y ago
  Even then it's not necessary. It's straightforward enough to use OAuth on any platform. In my experience, App Engine is mainly useful for in-house infrastructure apps for companies using the Google Apps platform. That or cheapskate developers throwing together a proof of concept / toy app in their spare time.
  - MatthewPhillips 15y ago
    Even toy apps can be had for free on other platforms. Dotcloud, Heroku, and AppHarbor all don't charge for the first instance.
diego 15y ago
In my opinion Google App Engine is a non-starter for serious applications. I only know of experiments and toys running there. Look at their gallery of successful applications: http://code.google.com/intl/it-IT/appengine/casestudies.html http://code.google.com/intl/it-IT/appengine/casestudies.html Contrast with AWS: http://aws.amazon.com/solutions/case-studies/ http://aws.amazon.com/solutions/case-studies/
- chucknthem 15y ago
  There are definitely serious applications and businesses built on appengine, mostly started before their price hike coming out of Beta last November. http://www.optimizely.com http://www.optimizely.com (YC W10) and http://shoesofprey.com http://shoesofprey.com are two notable examples.
  - taylorbuley 15y ago
    Rescuetime (YC W08) also uses it I believe
  - avree 15y ago
    We're built on appengine at Getaround.
  - dextorious 15y ago
    I wouldn't call those two "serious" examples -- especially since the competitors (Amazon, Heroku, Rackspace etc) have orders of magnitude more popular, complex and profitable services running on them. This is like saying, "Famous actors frequent my restaurant" and point to a picture of Ralph Macchio. OK, somewhat known, but De Niro, Brad Pitt, Clooney and co eat at the joint across the street.
    - jay_kyburz 15y ago
      Actually, I think its great for startups. If you're a small team trying to work out what your product is, how to sell it and who is going to buy it, you have more important things to do than learn how to set up and run a server. Once you've worked out you have a product that people want you can start thinking about managing your own servers. People complain about migration, but its really not that big a deal.
bemmu 15y ago
I love GAE, but deleting stuff is really the major WTF point. I am about to shut down one application that declined in popularity, because it costs me $20 / week to run it and revenue just dropped under $20 / week. The cost is not from instance hours, but purely from the stored data. Deleting the data from the data store would cost more than I could recoup, so that is not an option either. Also I really feel frustrated giving hours of thought to something that should be a really simple operation. Perhaps .delete() should be free? After all, when I shut down the app, Google does delete everything for free.
- jdeibele 15y ago
  My suggestion would be to turn off billing. That's what I did when the price went up on an app I had running and I didn't want to deal with deleting the data.

seanp2k2 15y ago

Google has a great platform here with lots of potential if they opened it up more, but they're pricing themselves into a corner for this already-niche service. Sadly, this is kind of "typical Google" -- great product, decent execution, but a bad identity problem -- it really feels like they're not sure yet what they want to do with this.

salimmadjd 15y ago
This is all because of their new pricing model. Overnight our pricing went up by 5X and that was after 50% discount, which make it really a 10X hike. Here is a graph of it: https://plus.google.com/114790424055754975707/posts/eUMhYDVf6i5 https://plus.google.com/114790424055754975707/posts/eUMhYDVf...

j45 15y ago

I wonder what a managed or dedicated server would cost to perform the same calculations. Sometime's it's still cheaper to have your own managed / self-managed gear... and from the looks of this pricing, even hire someone fulltime/freelancing to manage it all for you.

smilliken 15y ago
>"I wonder what a managed or dedicated server would cost to perform the same calculations." It wouldn't cost you any money (unless you have metered electricity), but rather just opportunity cost of being able to do other work with your resources. Unless you only need a short-term lease on the equipment, cloud servers will be more expensive that dedicated/colocated servers.
- j45 15y ago
  I have run my own servers in a datacenter for 7-8 years and going to a VM setup was the quantum leap. I can copy and paste a server in minutes and be up and running with a nearly identical setup that I have to setup for my dev environment anyways. Hardening the server is something that can be outsourced for a lot less than thousands. Services like linode seem to be a nice middle ground. While I don't see myself going back to my own hardware in a rack i run in a datacenter, I do still see the benefit of knowing your stack a bit beyond coding. Knowing how the stack works helps when building software quite often. Anyhow, those are just my experiences. VPS' with a very strong toolkit to take the edge off self-administering like Linode, etc, seem to be a very nice option. Heroku has caught my eye too but they have completely different measurements.
charliesome 15y ago
s/sometimes/all the time/; I honestly do not get why people are so fascinated with the cloud. It's a very expensive way to avoid having to know what you're doing.
- Gotperl 15y ago
  Like any other tool "the cloud", as you say, can be the right fit for the right job.

ajross 15y ago

Reading the thread (I'm curious about GAE, not an expert), it seems like the details aren't clear at all. Neither the original poster nor the Google rep seem to have a clear idea of what I/O operations are being generated. But this bit stood out: $0.10 per 100k writes. That price seems to be far too high. The poster is doing (something like) a reindex of 10M entries (that kind of data is pretty small really: it's the kind of database you might use as a test set on your laptop interactively). Figure each modification is atomic, and that the b-tree height of the storage is ~4. So that's 40M writes to create an index, or $400! Seriously? Again, this is the kind of task you'd expect to do quickly and interactively on your development box, and it costs a price of the same order as your day's salary (!) to execute in the cloud? Looking at this from the perspective of the underlying I/O device: this index consumes just a tiny, tiny fraction of a hard disk drive's capacity. Yet creating it costs enough to buy the device several times over? Something is wrong. Is that a misquote or have I misunderstood?

CurtHagenlocher 15y ago
I believe that $0.10 per 100k writes is what both Windows Azure Storage and Amazon S3 cost. But honestly, the cloud is the future! Disclaimer: I work at Microsoft and am required by the terms of my employment to believe in "the cloud".
- cperciva 15y ago
  I believe that $0.10 per 100k writes is what both Windows Azure Storage and Amazon S3 cost. Correct. Of course, those are object writes -- Elastic Block Store disk I/O is 10x cheaper.
  - ajross 15y ago
    Right. The granularity to be expected with an S3 object (~= "a file") is much higher than with a database index of tree nodes of a few kilobytes each. And of course EBS is subject to caching on the host. Rolling the index into a transaction can eliminate almost all the duplicate I/O to the tree nodes.
15y ago

salimmadjd 15y ago

They jacked up our price by 5X, here is a nice graph of it: https://plus.google.com/114790424055754975707/posts/eUMhYDVf6i5 https://plus.google.com/114790424055754975707/posts/eUMhYDVf...

magicalist 15y ago
I understand you're angry, but did you really just post that same thing twice in this thread and twice in the linked thread??
sologoub 15y ago
Hate to take it off topic, but your site (http://www.f8daily.com/ http://www.f8daily.com/) is throwing errors: ValueError: Values may not be more than 1000000 bytes in length; received 1053462 bytes
- troygoode 15y ago
  The image he linked said he was forced to shut down his site, so I'm not sure that he cares that there is an error.

endlessvoid94 15y ago

GAE has become completely infeasible as a hosting solution for me (ThatHigh.com). My hosting cost increased by 90x, and I did not get nearly enough notice. I don't have the time or resources to move the site, so I'm forced to shut it down. It really, really sucks. Personally I'm more disappointed by the lack of notice (1 month is nowhere near enough time) than the actual increase. I totally understand the need to charge.

yesbabyyes 15y ago
As a fan I urge you not to shut it down. Please consider selling it or something!
Travis 15y ago
Hey I shot you a tweet, I'd be interested in helping migrate the site. Email is in my HN profile, shoot me a message if you're interested.
StavrosK 15y ago
Did you turn multithreading on? It's helped on my sites somewhat.
yahelc 15y ago
If you implement optimizations, you can really significantly curb the cost. I spent some time tuning SharedCount's API, which would have cost me $30-$50/day, and its now at about $1-$2/day. - Move to Python 2.7 and enable multithreading - Setup Cloudflare (this swallows about half of all my requests) - Increase minimum latency and reduce the maximum number of idle instances. (I have 5-8s and 1-2 set, respectively) - Setup the semi-undocumented Google edge cache (basically, just a Cache-Control: public, max-age=[seconds] header. - Take advantage of memcache. With this setup, I'm doing 3 million API calls per day at $2.
- cr4zy 15y ago
  Python 2.7 is experimental and requires migrating to the new datastore (high replication) which is a lot more expensive. So you might not want to do this if your costs are coming from datastore writes. Also, high replication queries can return stale results unless you use ancestor queries. Ancestor queries require putting entities in groups by giving them all the same parent (which can never be changed). Basically it's a very inflexible semaphore and kind of sucks IMO. Your suggestions in general are very good though. Thanks, I'm switching my DNS to CloudFlare now.

stickfigure 15y ago

The details have finally been posted in that thread. And while $6500 is a lot of money, we have to realize that this is a lot of writing - the headline doesn't give the full sense of it. For those unfamiliar with GAE, a ListProperty is really a collection of properties. The author is using the property as a geohash with a significant number of values, plus he has additional multiproperty indexes defined, plus he's doing a rewrite (delete + write). All combined it appears to be ~460 writes per entity. So what we're talking about is $6500 for 6.5 billion writes... exactly what is printed on the sales brochure. Is that a lot? Most datastores don't charge by the operation so I don't have a lot to compare it to. It seems expensive but not crazy, especially considering that the data is replicated via PAXOS to 3+ datacenters with automatic loadbalancing and failover.

vosper 15y ago
"Geoboxing is a technique used to search for entities near a point on the earth in a database that can only perform equality queries (like App Engine)" So their implementation is a compromise on account of GAE's limitations, and they have to pay through the nose to use it. This is when I'd be looking at hosting some features outside of GAE, which is what we do with Full-text Search.
- stickfigure 15y ago
  No single platform or environment encompasses every use case. GAE doesn't offer R-trees or other spatial indexes; for that matter neither does MongoDB or MySQL. Geohashing is a reasonable solution for some spatial problem domains; it's one solution along the spectrum of "precalculate a lot up front and make queries cheap" vs "write in a cheap & easy format but make queries more expensive". Pre-calculation strategies are usually more scalable when you have large query loads, but they suck bigtime if you need to fully recalculate a large body of data (as the original blog author is doing). Maybe the blogger would be better off using PostGIS; but then, scaling and synchronizing a large cluster of PostGIS systems is nontrivial. The issues here are too application-specific to draw any positive or negative conclusions about appengine.
  - 15y ago

dextorious 15y ago

With Amazon AWS you can handle the scaling yourself, when you need it, with components tailored to your use cases, and better latency. And with Heroku you can have it taken care for you, following a few simple rules. So, why exactly would one use the crippled GAE platform, that constantly breaks its promises (re: reliability), forces you to code with very little flexibility (and, no, not every app that needs to automatically and massively scale "has to be coded exactly like a GAE app anyway"), costs a fortune (and sometimes an unexpected fortune), and breaks for you as soon as you need a technology not on offer?

latchkey 15y ago

A lot of these AppEngine costs too much notices have been coming up recently, but upon further inspection, they all tend to boil down to operator error. Unfortunately, AppEngine isn't forgiving of that and there is a real monetary value associated with questionable engineering design. Or, design that wasn't thought through enough in the context of a service like AppEngine. This leads to a few people getting upset and making a lot of noise when the reality is that AppEngine is actually an amazing service. So, to boil down the operator error from a quote in the thread: "We're running a mapreduce to change the geobox sizes/precision for a large number of entities." That is the real source of the problem. Instead of using geoboxes, they should be using geohashes, which allow arbitrary precision. http://code.google.com/apis/maps/articles/geospatial.html http://code.google.com/apis/maps/articles/geospatial.html http://en.wikipedia.org/wiki/Geohash http://en.wikipedia.org/wiki/Geohash Instead of an indexed property that looks like this (what they currently have): [u'37.3411|-121.8940|37.3395|-121.8926', u'37.3411|-121.8929|37.3395|-121.8916', ...] They would have an indexed List<String> property that looks like this: [8, 8f, 8f1, 8f12, 8f12a, 8f12ac, 8f12ac6, 8f12ac60, 8f12ac605, 8f12ac605f, 8f12ac605fb, 8f12ac605fb3, 8f12ac605fb34] Finding if the location is in a box would be computing the hash from the lat/lng (there is free code out there to do that) and then doing an indexed 'in' query. The indexes would only need to be updated if the location of the entity changes, not when they want varying levels of precision.

erichocean 15y ago
We wrote some App Engine code to do geospatial lookups. Before anyone gets the idea that the links in the parent are worth trying, they're not: the performance is absolutely atrocious – on our data set, 13 seconds per lookup. We wrote our own approach on App Engine and now get stable performance on our datasets at ~300ms per lookup. (We're doing Foursquare-type lookups.)
- latchkey 15y ago
  I can't imagine how an indexed query on a List<String> of ~10 items takes 13 seconds. 'We wrote our own approach on App Engine' Hmm... details?

tuhin 15y ago

For those questioning the use of Google App engine as a serious platform for applications, we at Pulse use Google App Engine: http://googleappengine.blogspot.com/2011/11/scaling-with-kindle-fire.html http://googleappengine.blogspot.com/2011/11/scaling-with-kin... So, yes you can build serious applications on GAE but like everything else it boils down to, it depends on what you really need.

6ren 15y ago

I like the spot price idea (in the comments http://groups.google.com/group/google-appengine/msg/fe9a05c6868e086d http://groups.google.com/group/google-appengine/msg/fe9a05c6...). It's similar to adwords' automatic auction for how close to the top your ad is, with the same benefits of getting the best market price (for buyers and sellers) of a limited resource. If no one else is using it, it could become close to free. It also casts the other users as the opponent, instead of google.

IgorPartola 15y ago

GAE is a mistake. By that I mean that it's got a big design flaw that's bound to cost Google money, which means it'll always be expensive than alternatives. Consider shared PHP hosting: a request comes in, apache finds which PHP file is responsible for it, then directs the request to the PHP interpreter. The PHP interpreter will parse the file (or more likely load the parsed bytecode from a cache) and return a response, which apache will then forward to the user agent. Notice that aside from the cache the PHP interpreter is stateless. As soon as it is done serving a request from site foo.com it can immediately jump on a request from bar.com and the context switch doesn't cost anything (once again disregarding finite cache size issues). Contrast this with running a stand-alone application server for each site, which is what GAE does. Here, even if your code is not serving any requests it's still waiting to get them. Now, GAE has powerful magic in it to retire request handlers which aren't frequently used. This way if site foo.com is getting 1 request/minute, it only really needs one process/thread/hander abstraction at a time. However, it is expensive to start/stop these "processes", so instead GAE is forced to keep this "process" around for a while after a request has been served hoping that the cost of keeping it alive would be justified by a second request. Thus these stateful, slow-to-start processes are always taking up resources that could be used to serve other requests. Disclaimer: all my knowledge of GAE has been from reading their docs/blog, not from deploying projects to it. Disclaimer 2: I am not saying that PHP is better/worse than GAE in any way. However, I am saying that the model that GAE uses is more costly for a typical application. This can be easily seen by comparing the cost of running a basic site on GAE vs $2/month shared hosting.

ww520 15y ago
I don't understand the downvotes. This is a on-topic related detail discussion.
- jbarham 15y ago
  Speaking for myself, I down-voted because of the absurd false dichotomy of comparing Google App Engine to $2/month shared PHP hosting! It's like complaining that an oil tanker costs more than a rowboat. The value proposition of App Engine is that with no systems administration expertise you can rent an extremely reliable, massively scalable web platform that is managed around the clock by a world class devops teams. Unsurprisingly this costs money. If you don't need the reliability or scalability of App Engine, no one is forcing you to pay for it. But it's absurd to suggest that you can get anything remotely comparable in PHP for $2/month.

tszming 15y ago

Besides the cost, if your startup can survive without SSL support on your own domain, go for App Engine. See: http://code.google.com/p/googleappengine/issues/detail?id=792 http://code.google.com/p/googleappengine/issues/detail?id=79... People requested custom SSL support at 2008, and today is 2012, if you still believe in App Engine, good luck!

latchkey 15y ago
http://googleappengine.blogspot.com/2011/10/app-engine-ssl-for-custom-domains-in.html http://googleappengine.blogspot.com/2011/10/app-engine-ssl-f...
- tszming 15y ago
  See the last comment of my link: >> the "trusted tester program" is a joke . They never respond so it's just a waste of time . Even they launch this feature TODAY, so 4 years for a basic requirement, what you can expect from them?
  - latchkey 15y ago
    I can think of TONS of services I've used over the years, both in and out of the software field that have talked about adding features and never did. Big deal, don't play the victim. If you really wanted onto the trusted tester program, you'd bring it up on the app engine mailing list or contact someone at google directly (their emails are all over the place, Ikai is a great guy, and they are very responsive). I'm sure they'd be happy to have enthusiastic beta testers. I find great irony in your quote on your G+ profile: "Do you create anything, or just criticize others work and belittle their motivations? -- Steve Jobs"
    - tszming 15y ago
      Thanks for your reminder, I shouldn't belittle the enthusiasm of GAE's engineers/supporters, my bad. But as SSL support is not public yet, my statement above is still valid: If your startup can survive without SSL support on your own domain, go for App Engine! Good luck!