7 ms·

I've noticed that Tornado comes with its own MySQL wrapper, however I'm concerned it might have blocking issues. Does anyone have experience with using this ?

I've noticed that Tornado comes with its own MySQL wrapper, however I'm concerned it might have blocking issues. Does anyone have experience with using this ? Also, what do people use for form validation with Tornado ?

candl 15y ago
I am curious about what kind of libraries are compatible with Tornado as well. Is there a list that someone maintains? I am specifically interested in things that can bring a bit of the goodies that django provides.
- peterbe 15y ago
  I've recently been blogging about my tornado-utils library which, for example, brings a email sending library similar to Djangos http://www.peterbe.com/oc-Tornado http://www.peterbe.com/oc-Tornado
wisty 15y ago
The theory is, you shouldn't need async database access. If your database is a bottleneck, you are screwed, regardless of how your app layer is scaling. The main advantage of async is not blocking if you call an external web API.
- wulczer 15y ago
  That sounds wrong. You need async database access so that while you're waiting for a query to finish, you can still process other requests. Imagine an app that receives an HTTP request every second. One in five of these requests requires making a query that takes 3 seconds to complete. The rest is serviced immediately. You don't want the long request freezing your entire server and preventing the fast requests from being serviced.
  - dchest 15y ago
    From this thread https://groups.google.com/forum/#!topic/python-tornado/mgj18IzasQg https://groups.google.com/forum/#!topic/python-tornado/mgj18... (posting here because Google Groups sometimes require login): We experimented with different async DB approaches, but settled on synchronous at FriendFeed because generally if our DB queries were backlogging our requests, our backends couldn't scale to the load anyway. Things that were slow enough were abstracted to separate backend services which we fetched asynchronously via the async HTTP module. I may open source the async MySQL client I wrote, but I am still skeptical of the long term value given the code complexity it introduces. Bret
    - wulczer 15y ago
      Erm, so if things are fast it doesn't matter and if things are slow you should write a whole new service, put a HTTP interface into it and use that? Not sure I buy that :) Disclaimer: I wrote an async DB module for Twisted/PostgreSQL and it did not turn out to be all that complex.
      - wisty 15y ago
        There's async libraries here: https://github.com/facebook/tornado/wiki/Links https://github.com/facebook/tornado/wiki/Links - You can get just about any db except MySQL. There's ones for Couch, Mongo, HBase, a bunch of other No-SQLs and Psycopg (PostGres). Also, Tornado is now introducing some kind of interface to Twisted's event loop, which does have everything ported. Bret just doesn't seem to want async My-SQL in the main code base.
    - spokengent 15y ago
      Not sure friendfeed ever got enough users to know scaling issues... Personally I would go with async DB, or have DB access in its own thread(s) and async communicate with that.
      - wisty 15y ago
        From Wikipedia: [FriendFeed] had on average one million monthly visitors ... [and] was bought for $15 million in cash, and $32.5 million in Facebook stock. I figure after the first million users, your embarrassingly parellel app server layer is not your bottleneck, it's getting the database to scale.
  - bdarnell 15y ago
    The impact of an occasional slow query is mitigated by the fact that you're running multiple processes. You need multiple processes anyway because of the GIL, so just run a few more to make sure you're able to keep the CPU busy. An async DB interface would be nice, but as long as you keep your database performance under control it's not crucial, so it hasn't been written yet.

candl 15y ago

I am curious about what kind of libraries are compatible with Tornado as well. Is there a list that someone maintains? I am specifically interested in things that can bring a bit of the goodies that django provides.

peterbe 15y ago
I've recently been blogging about my tornado-utils library which, for example, brings a email sending library similar to Djangos http://www.peterbe.com/oc-Tornado http://www.peterbe.com/oc-Tornado

wisty 15y ago

The theory is, you shouldn't need async database access. If your database is a bottleneck, you are screwed, regardless of how your app layer is scaling. The main advantage of async is not blocking if you call an external web API.

wulczer 15y ago
That sounds wrong. You need async database access so that while you're waiting for a query to finish, you can still process other requests. Imagine an app that receives an HTTP request every second. One in five of these requests requires making a query that takes 3 seconds to complete. The rest is serviced immediately. You don't want the long request freezing your entire server and preventing the fast requests from being serviced.
- dchest 15y ago
  From this thread https://groups.google.com/forum/#!topic/python-tornado/mgj18IzasQg https://groups.google.com/forum/#!topic/python-tornado/mgj18... (posting here because Google Groups sometimes require login): We experimented with different async DB approaches, but settled on synchronous at FriendFeed because generally if our DB queries were backlogging our requests, our backends couldn't scale to the load anyway. Things that were slow enough were abstracted to separate backend services which we fetched asynchronously via the async HTTP module. I may open source the async MySQL client I wrote, but I am still skeptical of the long term value given the code complexity it introduces. Bret
  - wulczer 15y ago
    Erm, so if things are fast it doesn't matter and if things are slow you should write a whole new service, put a HTTP interface into it and use that? Not sure I buy that :) Disclaimer: I wrote an async DB module for Twisted/PostgreSQL and it did not turn out to be all that complex.
    - wisty 15y ago
      There's async libraries here: https://github.com/facebook/tornado/wiki/Links https://github.com/facebook/tornado/wiki/Links - You can get just about any db except MySQL. There's ones for Couch, Mongo, HBase, a bunch of other No-SQLs and Psycopg (PostGres). Also, Tornado is now introducing some kind of interface to Twisted's event loop, which does have everything ported. Bret just doesn't seem to want async My-SQL in the main code base.
  - spokengent 15y ago
    Not sure friendfeed ever got enough users to know scaling issues... Personally I would go with async DB, or have DB access in its own thread(s) and async communicate with that.
    - wisty 15y ago
      From Wikipedia: [FriendFeed] had on average one million monthly visitors ... [and] was bought for $15 million in cash, and $32.5 million in Facebook stock. I figure after the first million users, your embarrassingly parellel app server layer is not your bottleneck, it's getting the database to scale.
- bdarnell 15y ago
  The impact of an occasional slow query is mitigated by the fact that you're running multiple processes. You need multiple processes anyway because of the GIL, so just run a few more to make sure you're able to keep the CPU busy. An async DB interface would be nice, but as long as you keep your database performance under control it's not crucial, so it hasn't been written yet.