26 ms·
Batch Mode for the Gemini API feels like Google’s way of asking, “What if we made AI more affordable and slower, but at massive scale?” Now you can process 10,0
by anupj 1y ago
Batch Mode for the Gemini API feels like Google’s way of asking, “What if we made AI more affordable and slower, but at massive scale?” Now you can process 10,000 prompts like “Summarize each customer review in one line” for half the cost, provided you’re willing to wait until tomorrow for the results.
- dist-epoch 1y agoMost LLM providers have batch mode. Not sure why you are calling them out.
- okdood64 1y agoI'll take it further. Regular cloud compute have batch workload capabilities at cheaper rates, as well since forever.
- diggan 1y ago> Now you can process 10,000 prompts like “Summarize each customer review in one line” for half the cost, provided you’re willing to wait until tomorrow for the results. Sounds like a great option to have available? Not every task I use LLMs for need immediate responses, and if I wasn't using local models for those things, getting a 50% discount and having to wait a day sounds like a fine tradeoff.
- XTXinverseXTY 1y agoThis is an extremely common use case. Reading your comment history: are you an LLM? https://news.ycombinator.com/item?id=44531907 https://news.ycombinator.com/item?id=44531907 https://news.ycombinator.com/item?id=44531868 https://news.ycombinator.com/item?id=44531868
- okdood64 1y agoI don't understand the point you're making. This has been a commonly used offering since cloud blew up. https://aws.amazon.com/ec2/spot/ https://aws.amazon.com/ec2/spot/