4 ms·

Django-medusa: Rendering Django sites as static HTML

45 points by mtigas 14y ago

bifrost 14y ago
Thats kinda neat, I suspect this will be used a lot. I did something similar to this 5-6 years ago with wget but this is a lot more elegant IMHO.
- mtigas 14y ago
  It’s very wget-like, due to the use of the Django HTTP test client — just slightly more elegant due to the programatic definition of what gets scraped/rendered and the addition of the "direct to S3" backend, which allows arbitrary mimetypes. Glad you like it.
natrius 14y ago
"The process that actually generates the output simply uses (or abuses) Django’s internal testclient to request each URL and store the resulting data" It warms my heart to see another example of test client abuse. We used the test client to implement a pure Python ESI processor[1]. We ran it in production for awhile, but no one should ever do that. Varnish is the answer. The code was still running on our development machines when I left. [1] https://github.com/armstrong/armstrong.esi/ https://github.com/armstrong/armstrong.esi/
megaman821 14y ago
Why all these static site generators? Just use Varnish, it is the static site generator for any and all frameworks.
- rglullis 14y ago
  Do you have access to Varnish in your standard run-of-the-mill shared host service?
  - nilved 14y ago
    Or GitHub pages/Amazon S3/etc.
- mtigas 14y ago
  What if you don’t have control over your hosting environment? (Or don’t have an application hosting environment to work with? Or don’t want to provision one?) What if you don’t want to use your own infrastructure? (See Ars Technica’s WWDC liveblog[1], which polls JSON files that are in the same directory and appeared to be periodically updated during the event, by some software that a reporter was using. Ostensibly because the feature is short-lived, super-high-traffic — likely thousands of concurrent users — and should be as low latency as possible due to the nature of the event.) Not to knock on Varnish, because I use it on plenty of larger things and love it. I just think that there are usecases where you can rationalize not even having an application server to cache in front of. [1] https://s3.amazonaws.com/liveblogs/wwdc-keynote-2012/index.html https://s3.amazonaws.com/liveblogs/wwdc-keynote-2012/index.h...
ashray 14y ago
I just don't see the point to this kind of stuff.. =/ For high traffic apps, Varnish is the answer as you don't hit the application layer. If you think that's too complicated, try nginx-memcached - also an excellent solution. If not that, try django's template caching with memcached - also extremely fast but will hit the application layer. If you're in some shared hosting environment (you probably are too small still to warrant this kind of aggressive caching on static assets - but hey, efficiency never hurt anybody :P) without access to memcached, use django's cache backend with a file based cache. It's almost 100% as what this does and you don't have any additional overhead. Beats me why people are re-inventing the wheel - or am I missing something ?
- yuvadam 14y ago
  Yeah, deployment scenarios where you have no application layer, and can only deploy static assets (S3, GitHub pages and the likes).
  - ashray 14y ago
    That's interesting. So basically it's for rendering pages and then pushing them on to S3/Other-static-storage and serving them from there ? I don't see any particular gains however since high traffic pages (static or otherwise..) served off S3 will end up costing way more than on a shared host/dedicated server. What would be an exact use case where this would really be important and help out ? I'm just trying to understand what need it solves and under what circumstances hosting static pages on S3 is beneficial in cost/performance.
    - Bockit 14y ago
      Our last 2 clients have asked for static solutions because they don't have the capacity to maintain django apps down the line and don't want to hire new staff or continue paying us for maintenance. We ended up developing both sites in django, which made it easy for them to add copy during development and see how it would actually look on the site. Once finished, we got the HTML output and sent it over for them to put up on their servers. We were pretty happy with how it turned out, and I think django-medusa being able to automate the whole rendering to html files would be nice if we have to do it again.
55pts 14y ago
Very cool project, thanks for open sourcing it. I was looking for this type of library this week and found medusa and aymcms. Does it handle images? How about multiple sites/subdomains?
- mtigas 14y ago
  For my blog, static files (anything stored in an app's "static" directory, basically[1]) the like are handled transparently through django-storages' S3 support [2]. (The STATICFILES_STORAGE option in settings.) If you’ve used Django's staticfiles framework before, it’s pretty much plug-and-play. For more dynamic file storage (say, using FileField or ImageField in a model), I believe django-storages would work, too. (Make sure you configure django-storages with the DEFAULT_FILE_STORAGE option set to S3 also.) Assuming you’re managing your site via a local dev server (or a server that "hosts" the "hot type" version of the site), any time you "upload" a file to your local server, it'll actually upload to S3 (and any calls to "field.url" will actually map to the S3 URL). Not sure how well it'll work in all use cases: I haven't actually used FileField or ImageField myself in the django-medusa+django-storages usecase, but I have used both separately so I’m fairly sure this is possible. This is a pretty darn good question though, so I’ll likely make a follow-up blogpost with a more comprehensive walkthrough regarding handling staticfiles and FileField/ImageField. Sometime in the near future. Multiple sites/subdomains is a bit more complicated. I’d say you should probably use separate Django instances for each and render them separately. (For S3, you’d need to use separate buckets, anyway.) If they need to share data, you can configure multiple Django settings.py configurations for each site but still use the same source tree and local database. (See the Django sites framework: [3]) [1]: https://docs.djangoproject.com/en/dev/howto/static-files/ https://docs.djangoproject.com/en/dev/howto/static-files/ [2]: http://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html http://django-storages.readthedocs.org/en/latest/backends/am... [3]: https://docs.djangoproject.com/en/1.4/ref/contrib/sites/ https://docs.djangoproject.com/en/1.4/ref/contrib/sites/

45 points by mtigas 14y ago

Thats kinda neat, I suspect this will be used a lot. I did something similar to this 5-6 years ago with wget but this is a lot more elegant IMHO.

mtigas 14y ago
It’s very wget-like, due to the use of the Django HTTP test client — just slightly more elegant due to the programatic definition of what gets scraped/rendered and the addition of the "direct to S3" backend, which allows arbitrary mimetypes. Glad you like it.

natrius 14y ago

"The process that actually generates the output simply uses (or abuses) Django’s internal testclient to request each URL and store the resulting data" It warms my heart to see another example of test client abuse. We used the test client to implement a pure Python ESI processor[1]. We ran it in production for awhile, but no one should ever do that. Varnish is the answer. The code was still running on our development machines when I left. [1] https://github.com/armstrong/armstrong.esi/ https://github.com/armstrong/armstrong.esi/

megaman821 14y ago

Why all these static site generators? Just use Varnish, it is the static site generator for any and all frameworks.

rglullis 14y ago
Do you have access to Varnish in your standard run-of-the-mill shared host service?
- nilved 14y ago
  Or GitHub pages/Amazon S3/etc.
mtigas 14y ago
What if you don’t have control over your hosting environment? (Or don’t have an application hosting environment to work with? Or don’t want to provision one?) What if you don’t want to use your own infrastructure? (See Ars Technica’s WWDC liveblog[1], which polls JSON files that are in the same directory and appeared to be periodically updated during the event, by some software that a reporter was using. Ostensibly because the feature is short-lived, super-high-traffic — likely thousands of concurrent users — and should be as low latency as possible due to the nature of the event.) Not to knock on Varnish, because I use it on plenty of larger things and love it. I just think that there are usecases where you can rationalize not even having an application server to cache in front of. [1] https://s3.amazonaws.com/liveblogs/wwdc-keynote-2012/index.html https://s3.amazonaws.com/liveblogs/wwdc-keynote-2012/index.h...

ashray 14y ago

I just don't see the point to this kind of stuff.. =/ For high traffic apps, Varnish is the answer as you don't hit the application layer. If you think that's too complicated, try nginx-memcached - also an excellent solution. If not that, try django's template caching with memcached - also extremely fast but will hit the application layer. If you're in some shared hosting environment (you probably are too small still to warrant this kind of aggressive caching on static assets - but hey, efficiency never hurt anybody :P) without access to memcached, use django's cache backend with a file based cache. It's almost 100% as what this does and you don't have any additional overhead. Beats me why people are re-inventing the wheel - or am I missing something ?

yuvadam 14y ago
Yeah, deployment scenarios where you have no application layer, and can only deploy static assets (S3, GitHub pages and the likes).
- ashray 14y ago
  That's interesting. So basically it's for rendering pages and then pushing them on to S3/Other-static-storage and serving them from there ? I don't see any particular gains however since high traffic pages (static or otherwise..) served off S3 will end up costing way more than on a shared host/dedicated server. What would be an exact use case where this would really be important and help out ? I'm just trying to understand what need it solves and under what circumstances hosting static pages on S3 is beneficial in cost/performance.
  - Bockit 14y ago
    Our last 2 clients have asked for static solutions because they don't have the capacity to maintain django apps down the line and don't want to hire new staff or continue paying us for maintenance. We ended up developing both sites in django, which made it easy for them to add copy during development and see how it would actually look on the site. Once finished, we got the HTML output and sent it over for them to put up on their servers. We were pretty happy with how it turned out, and I think django-medusa being able to automate the whole rendering to html files would be nice if we have to do it again.

55pts 14y ago

Very cool project, thanks for open sourcing it. I was looking for this type of library this week and found medusa and aymcms. Does it handle images? How about multiple sites/subdomains?

mtigas 14y ago
For my blog, static files (anything stored in an app's "static" directory, basically[1]) the like are handled transparently through django-storages' S3 support [2]. (The STATICFILES_STORAGE option in settings.) If you’ve used Django's staticfiles framework before, it’s pretty much plug-and-play. For more dynamic file storage (say, using FileField or ImageField in a model), I believe django-storages would work, too. (Make sure you configure django-storages with the DEFAULT_FILE_STORAGE option set to S3 also.) Assuming you’re managing your site via a local dev server (or a server that "hosts" the "hot type" version of the site), any time you "upload" a file to your local server, it'll actually upload to S3 (and any calls to "field.url" will actually map to the S3 URL). Not sure how well it'll work in all use cases: I haven't actually used FileField or ImageField myself in the django-medusa+django-storages usecase, but I have used both separately so I’m fairly sure this is possible. This is a pretty darn good question though, so I’ll likely make a follow-up blogpost with a more comprehensive walkthrough regarding handling staticfiles and FileField/ImageField. Sometime in the near future. Multiple sites/subdomains is a bit more complicated. I’d say you should probably use separate Django instances for each and render them separately. (For S3, you’d need to use separate buckets, anyway.) If they need to share data, you can configure multiple Django settings.py configurations for each site but still use the same source tree and local database. (See the Django sites framework: [3]) [1]: https://docs.djangoproject.com/en/dev/howto/static-files/ https://docs.djangoproject.com/en/dev/howto/static-files/ [2]: http://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html http://django-storages.readthedocs.org/en/latest/backends/am... [3]: https://docs.djangoproject.com/en/1.4/ref/contrib/sites/ https://docs.djangoproject.com/en/1.4/ref/contrib/sites/