5 ms·

Show HN: Silk, a profiling tool for Django

113 points by mtford 12y ago

Kaedon 12y ago
Very cool, this seems much nicer than using hotshot to profile Django. I'm a little confused by this page though: http://mtford.co.uk/silk/request/1907/profiling/ http://mtford.co.uk/silk/request/1907/profiling/. Why is wrapped_target listed so many times for a single blog post query? Is there a hierarchy to the profile calls that's not shown? Oh, does it go deepest level to highest level calls?
- mtford 12y ago
  Thats a great question... that def shouldn't be there. render_to_response is actually dynamically profiled i.e. the decorator is applied at runtime via configuration in settings.py. This looks like its probably a bug with that. Cheers, will look into it
notdonspaulding 12y ago
After reading through this intro, it isn't clear to me if I can use the decorator or context manager to profile code outside the request/response cycle. Any ideas if that's the case?
- mtford 12y ago
  Not at the moment. Silk doesn't actually save anything down until right at the end of the cycle i.e. in process_response of the middleware. Had to do this to avoid issues with atomic transactions. I certainly don't mind adding this as a feature if people would find that useful - perhaps by detecting that no request is in process and then commiting profile data on the fly if that's the case.
spanasik 12y ago
absolutely great tool!
numlocked 12y ago
Looks very interesting! We currently use django debug toolbar for profiling, which seems to have a lot of overlap. I like how Silk stores the profiles of multiple requests - that seems to be a nice differentiator. What else? Thanks for the awesome contribution to the Django community!
- mtford 12y ago
  Yep django-toolbar is great! My aim with this was, like you said, to provide the history of multiple requests, but also to provide finer-grained profiling with the decorator and context manager. Not only do they capture execution time but they also capture the queries that were executed whilst they were active. The aim was extra visibility all round really.
- rmc 12y ago
  > I like how Silk stores the profiles of multiple requests You might like my Django SQL Inspector which aggregates many SQL queries across many requests. https://github.com/rory/django-sql-inspector https://github.com/rory/django-sql-inspector
timc3 12y ago
Looks really nice. Have you thought that instead of sending the results to elasticsearch? Instead of using something like newrelic this could take it's place..
- mtford 12y ago
  That's not a bad idea, it would probably be more natural than the rigid filters that are currently in place for the requests and profiles. As far as having this running in prod I've only really thought of siphoning off the results to celery or something similar to avoid impacting response time, but a spray and pray at Elasticsearch would work just as well. What's nice about the current setup is that it takes very little effort to get up and running but I suppose could make that configurable...
- ergo14 12y ago
  If you are looking for something like newrelic you might want to give https://appenlight.com https://appenlight.com a try. I've put a lot of love into it and it can give you tons of information about your application.
philipn 12y ago
Thanks for posting this! Could you add a download link somewhere?
- mtford 12y ago
  Sure just added this to the intro. You can download via https://github.com/mtford90/silk/releases https://github.com/mtford90/silk/releases or install through pip install django-silk
rmc 12y ago
I wrote https://github.com/rory/django-sql-inspector https://github.com/rory/django-sql-inspector a django app to profile SQL queries done in multiple pages, including showing stack trace of where it's being called. You can find out which function(s) are responsible for long SQL requests.
SEJeff 12y ago
What is the overhead (roughly) for this guy? I do absolutely love the idea of using elasticsearch for this instead of a traditional database. Using celery or django-rq to delay the processing also seems reasonable.
- mtford 12y ago
  Tbh, probably fairly bad atm. I haven't done anything empirical but atm Silk will create a record for every SQL query executed during the request/response cycle effectively doubling the number of queries. Not only that but it also saves down any non-binary HTTP body to a TextField, and this isn't yet configurable. I think the Elasticsearch/Celery option is a good shout if this were to ever be used in production. Celery would help with the issue of response time but it wouldn't solve the load impact on the database (although I guess could also configure a different SQL database in Django). I will certainly do some investigation into this at some point :)
  - opendais 12y ago
    StatsD might be better since you are just recording datapoints + text field I'm assuming. https://pypi.python.org/pypi/python-statsd https://pypi.python.org/pypi/python-statsd
    - mtford 12y ago
      Sounds interesting... i'll take a look. The fact that its UDP would be helpful in avoiding the need for a dependency on celery/message queues. What's it like in terms of getting the data back out?
      - opendais 12y ago
        You can write your plugin for the back end to dump it. It defaults to graphite which would be familiar enough I think.
ergo14 12y ago
For people interested in production monitoring I've created https://appenlight.com https://appenlight.com - and our python middleware will integrate and start timing django applications without any code changes required (except 3 lines to load middleware itself). @mtford great job!
severb 12y ago
Probably not a good idea to keep your profiler open to the world as long as you allow reading from arbitrary files. After a bit of fiddling around I was able to read your secret key from the settings file. It starts with 'es!b3'. And your soundcloud password. Sorry.
- mtford 12y ago
  Nice... I did make an attempt to filter out sensitive stuff but clearly did a bad job. Thanks for letting me know. I will eventually introduce a demo app rather than my own site.
  - severb 12y ago
    I've sent you an email with the precise steps to reproduce it.
iamwithnail 12y ago
This is a really great tool, massive fan, made some good use of it in the ~15 hours since I installed it yesterday evening. Are you aware it seems to bork the admin interface? If, after installing Silk, I use admin to update a record, I get "You cannot access body after reading from request's data stream" - this applies across all admin models and pages. It's not really that big a deal, as I rarely use the admin pages now, but it might have knock-on effects that I haven't seen yet. (Commenting out Silk in the installed apps list clears the problem up straight away.) I'll ping you an email with the stack trace.

113 points by mtford 12y ago

Very cool, this seems much nicer than using hotshot to profile Django. I'm a little confused by this page though: http://mtford.co.uk/silk/request/1907/profiling/ http://mtford.co.uk/silk/request/1907/profiling/. Why is wrapped_target listed so many times for a single blog post query? Is there a hierarchy to the profile calls that's not shown? Oh, does it go deepest level to highest level calls?

mtford 12y ago
Thats a great question... that def shouldn't be there. render_to_response is actually dynamically profiled i.e. the decorator is applied at runtime via configuration in settings.py. This looks like its probably a bug with that. Cheers, will look into it

notdonspaulding 12y ago

After reading through this intro, it isn't clear to me if I can use the decorator or context manager to profile code outside the request/response cycle. Any ideas if that's the case?

mtford 12y ago
Not at the moment. Silk doesn't actually save anything down until right at the end of the cycle i.e. in process_response of the middleware. Had to do this to avoid issues with atomic transactions. I certainly don't mind adding this as a feature if people would find that useful - perhaps by detecting that no request is in process and then commiting profile data on the fly if that's the case.

spanasik 12y ago

absolutely great tool!

numlocked 12y ago

Looks very interesting! We currently use django debug toolbar for profiling, which seems to have a lot of overlap. I like how Silk stores the profiles of multiple requests - that seems to be a nice differentiator. What else? Thanks for the awesome contribution to the Django community!

mtford 12y ago
Yep django-toolbar is great! My aim with this was, like you said, to provide the history of multiple requests, but also to provide finer-grained profiling with the decorator and context manager. Not only do they capture execution time but they also capture the queries that were executed whilst they were active. The aim was extra visibility all round really.
rmc 12y ago
> I like how Silk stores the profiles of multiple requests You might like my Django SQL Inspector which aggregates many SQL queries across many requests. https://github.com/rory/django-sql-inspector https://github.com/rory/django-sql-inspector

timc3 12y ago

Looks really nice. Have you thought that instead of sending the results to elasticsearch? Instead of using something like newrelic this could take it's place..

mtford 12y ago
That's not a bad idea, it would probably be more natural than the rigid filters that are currently in place for the requests and profiles. As far as having this running in prod I've only really thought of siphoning off the results to celery or something similar to avoid impacting response time, but a spray and pray at Elasticsearch would work just as well. What's nice about the current setup is that it takes very little effort to get up and running but I suppose could make that configurable...
ergo14 12y ago
If you are looking for something like newrelic you might want to give https://appenlight.com https://appenlight.com a try. I've put a lot of love into it and it can give you tons of information about your application.

philipn 12y ago

Thanks for posting this! Could you add a download link somewhere?

mtford 12y ago
Sure just added this to the intro. You can download via https://github.com/mtford90/silk/releases https://github.com/mtford90/silk/releases or install through pip install django-silk

rmc 12y ago

I wrote https://github.com/rory/django-sql-inspector https://github.com/rory/django-sql-inspector a django app to profile SQL queries done in multiple pages, including showing stack trace of where it's being called. You can find out which function(s) are responsible for long SQL requests.

SEJeff 12y ago

What is the overhead (roughly) for this guy? I do absolutely love the idea of using elasticsearch for this instead of a traditional database. Using celery or django-rq to delay the processing also seems reasonable.

mtford 12y ago
Tbh, probably fairly bad atm. I haven't done anything empirical but atm Silk will create a record for every SQL query executed during the request/response cycle effectively doubling the number of queries. Not only that but it also saves down any non-binary HTTP body to a TextField, and this isn't yet configurable. I think the Elasticsearch/Celery option is a good shout if this were to ever be used in production. Celery would help with the issue of response time but it wouldn't solve the load impact on the database (although I guess could also configure a different SQL database in Django). I will certainly do some investigation into this at some point :)
- opendais 12y ago
  StatsD might be better since you are just recording datapoints + text field I'm assuming. https://pypi.python.org/pypi/python-statsd https://pypi.python.org/pypi/python-statsd
  - mtford 12y ago
    Sounds interesting... i'll take a look. The fact that its UDP would be helpful in avoiding the need for a dependency on celery/message queues. What's it like in terms of getting the data back out?
    - opendais 12y ago
      You can write your plugin for the back end to dump it. It defaults to graphite which would be familiar enough I think.

ergo14 12y ago

For people interested in production monitoring I've created https://appenlight.com https://appenlight.com - and our python middleware will integrate and start timing django applications without any code changes required (except 3 lines to load middleware itself). @mtford great job!

severb 12y ago

Probably not a good idea to keep your profiler open to the world as long as you allow reading from arbitrary files. After a bit of fiddling around I was able to read your secret key from the settings file. It starts with 'es!b3'. And your soundcloud password. Sorry.

mtford 12y ago
Nice... I did make an attempt to filter out sensitive stuff but clearly did a bad job. Thanks for letting me know. I will eventually introduce a demo app rather than my own site.
- severb 12y ago
  I've sent you an email with the precise steps to reproduce it.

iamwithnail 12y ago

This is a really great tool, massive fan, made some good use of it in the ~15 hours since I installed it yesterday evening. Are you aware it seems to bork the admin interface? If, after installing Silk, I use admin to update a record, I get "You cannot access body after reading from request's data stream" - this applies across all admin models and pages. It's not really that big a deal, as I rarely use the admin pages now, but it might have knock-on effects that I haven't seen yet. (Commenting out Silk in the installed apps list clears the problem up straight away.) I'll ping you an email with the stack trace.