7 ms·

Not another language because the performance issues are limited to the code being sync. Anyway, a gradual migration was a must as well. This is the first I hea

by raybb 9d ago

Not another language because the performance issues are limited to the code being sync. Anyway, a gradual migration was a must as well. This is the first I hear of Litestar and I think it does seem very values aligned. I fastapi was picked simply because it was already known by the team and had a very thriving ecosystem. That being said, the hard part is getting off the old sync web.py stuff. Once everything is in async land moving to another framework if needed is relatively trivial. Thanks for sharing that though I'll keep an eye on it!

zplizzi 9d ago
I’ve been using litestar for loads of projects recently and have been loving it. The ability to have full request/response typing (and openapi spec autogeneration) with your favorite dataclass framework is quite nice (I use attrs). I didn’t want to be forced to use pydantic like some other frameworks do.
- what 8d ago
  Glad I’m not the only one that prefers (c)attrs over pydantic.
anandology 8d ago
I was one of the initial developers of Open Library and wrote most of it's backend code. I doubt any of the performance issues could be because of code being sync. Open Library is IO bound and it is most likely spending time waiting for db to respond. Was there any performance analysis done that indicated that the "code being sync" the main bottleneck? Happy to help.
- raybb 8d ago
  The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests. This was handled from both sides: handling high-concurrency waiting better, while simultaneously optimizing Solr and removing archive.org from the critical path where possible.
  - anandology 7d ago
    I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this. 1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path. 2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.
    - mekarpeles 7d ago
      Wonderful to see @raybb and @anandology in this thread. Lots of operational challenges come up when running a service for 14M patrons. And Open Library in particular has a handful of challenges. 1. It's database has grown significantly (800+ GB) and Anand is right that IO (even on SSDs) is a challenge. The `thing` (infobase/infogami) triple-store design is well thought out and gets us a lot, and any system has to be tuned as it scales to hundreds of millions of rows. One strategy here is being smarter about cache and also shifting some of the load from psql to solr. Rishabh and others volunteers have been amazing assets as we've moved in this direction. Jim Champ on staff has been helping me tune psql, pgbouncer, and some of our high IO crons to improve raw db performance. 2. Limited hardware resources. We're trying to move some of our services within the Internet Archive's kubernetes cluster and we've done a great job migrating towards a world where everything is dockerized. It used to be a very painful process for our team of 3 to handle server ops, upgrades, and networking for nearly 15 manually orchestrated servers. One of the bare-metal racks running much of Open Library is significanly oversubscribed on vCPUs and so moving services off to free space and eliminate steal is critical for us right now. Our main web server (ol-www0) suffers from up to 20% steal and we're seeing a lot of congestion before requests even get to our web nodes (app servers). We have a plan and it takes time. 3. Open Library is still dependent on Archive.org for many lookups -- like book availability (which Ben Deitch has been helping me and Drini move into solr). When there are network issues and a network requests takes 5+ seconds, every web.py worker on that thread grinds to a halt and Ray's work moving us to FastAPI has made a significant impact 4. Solr. Drini has been heroic at restructuring our setup to use replicated solr in a way that has increased performance and relieved some of the pressure on our main cluster. This was a huge bottleneck for us this time last year and we've taken a lot of steps to ameliorate our situation. See: https://blog.openlibrary.org/2025/09/12/open-library-search-balancing-high-impact-with-high-demand/ https://blog.openlibrary.org/2025/09/12/open-library-search-... 5. Raw spikes in traffic. We are seeing massive amounts of traffic that slams our book pages, increasing the pain of all the above. It saturates our limited resources, puts more strain on our database, ties us web workers... It makes modsecurity even more expensive. Part of the solutions is being more clever about provisioning, part of the solution is using fail2ban to prevent bad traffic from subtracting from the experience of the patrons who depend on us. Part of the solution is caching and optimizing our database to scale with load. There isn't just one solution and the same 3 engineers on staff (and the support of a completely stellar community of dedicated volunteers fellows and leads) are doing our best to balance ops improvements with the necessary "product" and design improvements necessary that ensure we're useful to people to begin with. I hope this gives the world a bit more of a glimpse how we operate and what some of our challenges are. We're an open source project and our goal is to share as many learnings as we can and to build something useful, sustainable, and beneficial for the community at large. Thank you Ray, Anand, Drini, Jim, Lokesh, Lisa, Charles, and so many dozens more for your tremendous work (present and past) and thank you for being in our corner.

by raybb 9d ago

zplizzi 9d ago

I’ve been using litestar for loads of projects recently and have been loving it. The ability to have full request/response typing (and openapi spec autogeneration) with your favorite dataclass framework is quite nice (I use attrs). I didn’t want to be forced to use pydantic like some other frameworks do.

what 8d ago
Glad I’m not the only one that prefers (c)attrs over pydantic.

anandology 8d ago

I was one of the initial developers of Open Library and wrote most of it's backend code. I doubt any of the performance issues could be because of code being sync. Open Library is IO bound and it is most likely spending time waiting for db to respond. Was there any performance analysis done that indicated that the "code being sync" the main bottleneck? Happy to help.

raybb 8d ago
The issue wasn't the latency of a single DB query (though many were improved/cached), but sync worker saturation: When traffic spikes, slow network calls to archive.org (for availability) and Solr tie up all available sync workers. The queue backs up, and suddenly every page becomes slow because there are no free workers left to handle requests. This was handled from both sides: handling high-concurrency waiting better, while simultaneously optimizing Solr and removing archive.org from the critical path where possible.
- anandology 7d ago
  I see the problem. But I feel switching to async is too big a change and there could be simpler ways to address this. 1. Solr was used (I think still continue to be used) for many things it is not quite the right fit. It was done then because, it was simpler to do. For example, it is used to get list of editions when a work is loaded. Limit Solr only to search and figure out a better way to manage the other pieces, would be a good start. Search could be made a different service and the frontend could call it via API. Both of these together will remove Solr from the critical path. 2. The availability status of archive.org could come from a database and a backend task can continue to update it very frequently.
  - mekarpeles 7d ago
    Wonderful to see @raybb and @anandology in this thread. Lots of operational challenges come up when running a service for 14M patrons. And Open Library in particular has a handful of challenges. 1. It's database has grown significantly (800+ GB) and Anand is right that IO (even on SSDs) is a challenge. The `thing` (infobase/infogami) triple-store design is well thought out and gets us a lot, and any system has to be tuned as it scales to hundreds of millions of rows. One strategy here is being smarter about cache and also shifting some of the load from psql to solr. Rishabh and others volunteers have been amazing assets as we've moved in this direction. Jim Champ on staff has been helping me tune psql, pgbouncer, and some of our high IO crons to improve raw db performance. 2. Limited hardware resources. We're trying to move some of our services within the Internet Archive's kubernetes cluster and we've done a great job migrating towards a world where everything is dockerized. It used to be a very painful process for our team of 3 to handle server ops, upgrades, and networking for nearly 15 manually orchestrated servers. One of the bare-metal racks running much of Open Library is significanly oversubscribed on vCPUs and so moving services off to free space and eliminate steal is critical for us right now. Our main web server (ol-www0) suffers from up to 20% steal and we're seeing a lot of congestion before requests even get to our web nodes (app servers). We have a plan and it takes time. 3. Open Library is still dependent on Archive.org for many lookups -- like book availability (which Ben Deitch has been helping me and Drini move into solr). When there are network issues and a network requests takes 5+ seconds, every web.py worker on that thread grinds to a halt and Ray's work moving us to FastAPI has made a significant impact 4. Solr. Drini has been heroic at restructuring our setup to use replicated solr in a way that has increased performance and relieved some of the pressure on our main cluster. This was a huge bottleneck for us this time last year and we've taken a lot of steps to ameliorate our situation. See: https://blog.openlibrary.org/2025/09/12/open-library-search-balancing-high-impact-with-high-demand/ https://blog.openlibrary.org/2025/09/12/open-library-search-... 5. Raw spikes in traffic. We are seeing massive amounts of traffic that slams our book pages, increasing the pain of all the above. It saturates our limited resources, puts more strain on our database, ties us web workers... It makes modsecurity even more expensive. Part of the solutions is being more clever about provisioning, part of the solution is using fail2ban to prevent bad traffic from subtracting from the experience of the patrons who depend on us. Part of the solution is caching and optimizing our database to scale with load. There isn't just one solution and the same 3 engineers on staff (and the support of a completely stellar community of dedicated volunteers fellows and leads) are doing our best to balance ops improvements with the necessary "product" and design improvements necessary that ensure we're useful to people to begin with. I hope this gives the world a bit more of a glimpse how we operate and what some of our challenges are. We're an open source project and our goal is to share as many learnings as we can and to build something useful, sustainable, and beneficial for the community at large. Thank you Ray, Anand, Drini, Jim, Lokesh, Lisa, Charles, and so many dozens more for your tremendous work (present and past) and thank you for being in our corner.