12 ms·
Yeah we want to do exactly this, benchmark and add more data from differnt gpus/cloud providers, will appreciate your help a lot! There are many inference engin
by alextttty 2y ago
Yeah we want to do exactly this, benchmark and add more data from differnt gpus/cloud providers, will appreciate your help a lot!
There are many inference engines which can be tested and updated to find best inference methods
- Whiteshadow12 2y agoGoodluck, companies would love that. Don't get depressed unlike my tool I think you should charge, that might keep you motivated to keep doing the work. It's a lot of work, your target users is companies that use Runpod and AWS/GCP/Azure, not Fireworks and Together, they are in the game of selling tokens, you are selling the cost of running seconds on GPUs.
- alextttty 2y agoThank you!
- agcat 2y agoThis is true especially if you are deploying custom or fine-tuned models. Infact, for my company i also ran benchmark tests where we tested cold-starts, performance consistency, scalability, and cost-effectiveness for models like Llama2 7Bn & Stable Diffusion across different providers - https://www.inferless.com/learn/the-state-of-serverless-gpus-part-2 https://www.inferless.com/learn/the-state-of-serverless-gpus... Can save months of evaluation time. Do give it a read. P.S: I am from Inferless.