5 ms·
Writing a chat application in Django 4.2 using async StreamingHttpResponse
- pmontra 3y agoThis implementation is probably a little different but we were using long poll in the late 90s and early 2000s. The problem was that you were committing one thread (or worse, one process) to each client. That obviously doesn't scale unless threads are extremely light on RAM and either the OS or the runtime support a large number of them. I remember that a way out was using continuations. Jetty was a Java application server that supported them (random link [1]) One thread -> many connections. I didn't investigate how Django is implementing this now but CPUs and RAM are still CPUs and RAM. [1] https://stackoverflow.com/questions/10587660/how-does-jetty-handle-multiple-requests https://stackoverflow.com/questions/10587660/how-does-jetty-...
- klabb3 3y ago> The problem was that you were committing one thread (or worse, one process) to each client. This is mostly true today as well, although we do have beefier machines and more efficient thread pooling. I did some personal research for infrequent real-time notifications. My conclusion was that many stateful connections are poorly memory-optimized by language runtimes and reverse proxies. Even with lightweight tech like Golang, gRPC, nginx, etc, it’s hard to push anything less than 30 kB for an idle conn, mostly from thread/goroutine stacks, user space buffers and (easy to overlook) a bunch of HTTP headers and TLS handshake state that often remain after establishing the conn. That’s without any middleware or business logic wants their share of the cake. The only mature project I found that really took this stuff seriously is uWebSockets. It’s extremely lightweight, around an OOM better than most alternatives. Highly recommend – they also have a socket library so you can implement other protocols if needed. Anyway, it’s important to be aware that massive amounts of long running connections is not just about adding a websocket/SSE library. Chances are you’ll need a mostly-separate serving stack for that purpose.
- Ralfp 3y agoJust a heads up that currently Django is not cleaning up open PostgreSQL connections when ran in ASGI mode, leading to too many open connections error: https://code.djangoproject.com/ticket/33497 https://code.djangoproject.com/ticket/33497
- whalesalad 3y agothat’s kind of a dealbreaker
- thefreeman 3y agocould this be addressed by something like pgbouncer or another connection pooler?
- whalesalad 3y agoPerhaps yes, would be worthwhile to benchmark an internal (to your runtime) conn pooler versus external only. I'm using internal pools + pgbouncer but as I write this out I am beginning to wonder if that is even necessary.
- verandaguy 3y agoGoing off the pooling mode feature map[0], it might be possible to mitigate that issue using transaction pooling, but not session pooling — which means a restricted feature set and cooperation from the application (which would just not call restricted features up). [0] http://www.pgbouncer.org/features.html#fn:2
- pdhborges 3y agoI'll have zero trust running async Django until all sync_to_async and async_to_sync calls are removed from the code base.
- deleted 3y ago[deleted]
- 3y ago
- samwillis 3y agoAll the new asyncIO stuff in Django is awesome, they are doing a phenomenal job retrofitting it all to an inherently sync designed framework. One thing to note, it's been possible to build these sort of things with Django by using Gevent and Gunicorn for over a decade, and it works well. In many ways I wish Gevent had been adopted by Python rather than AsyncIO.
- jononomo 3y agoWhere can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible. Also, what was the reason not to go with Gevent?
- lastofus 3y ago>Also, what was the reason not to go with Gevent? The biggest downside of Gevent IMO is that it enables the magic of turning sync code into async code via monkey patching things like the socket lib. This lack of explicitness can make things a bit difficult to reason about, without a good mental model of what Gevent is doing underneath the hood.
- bastawhiz 3y agoI'd strongly recommend django-channels if you want to change very little. It feels like Django and it integrates without a lot of headache (at least in my experience).
- btown 3y agoCan second the Django/Gevent/Gunicorn stack - we use it in production and IMO it's much easier to reason about and code in than asyncio. Just write synchronous Python, including long-running `requests` calls, and the system will yield control whenever you're blocked on network or disk I/O, no matter how deep that is in your stack. The entire ecosystem of Python libraries just works. I also wish more people knew about gevent - it's truly magical, especially if you need to work with APIs with unpredictable latency!
- kiraaa 3y agohttps://github.com/sysid/sse-starlette https://github.com/sysid/sse-starlette makes token streaming so much easier in python
- deleted 3y ago[deleted]
- dnadler 3y agoI’m using this for an internal ChatGPT UI clone and it’s working great. The actual biggest pain for me has been the front end handling of it with the fetch api. But that’s likely just due my inexperience with it.
- Thaxll 3y agoDo modern chat actually use websocket and the like? Discord / Slack on the web ( browser ) what do they use?
- vorticalbox 3y agoThe slack_bolt library has support for web sockets. https://api.slack.com/apis/connections/socket https://api.slack.com/apis/connections/socket
- holler 3y agoyes they use websockets
- jhgg 3y agoDiscord uses websockets on all clients/platforms.
- Waterluvian 3y ago“ The idea is that the client "subscribes" to an HTTP endpoint, and the server can then issue data to the client as long as the connection is open.” To those who have been around longer than me: isn’t this just long polling, that predates websocket?