7 ms·
Yes accuracy, latency & throughput are the 3 poles we try to achieve, c++ helps with latency & throughput and helps keep the cost low.
by viig99 6y ago
Yes accuracy, latency & throughput are the 3 poles we try to achieve, c++ helps with latency & throughput and helps keep the cost low.
- mlthoughts2018 6y agoWhy would c++ help with latency in comparison to say Python with numpy / numba / Cython? All the production critical “this needs to be as fast as possible stuff” I’ve ever worked on has been all Python, achieving complete speed parity with C, at a much faster development speed and with way way less boilerplate code.
- disgruntledphd2 6y agoIf you have hard constraints at inference time, then it can be much easier to tune to a time budget with C++. Like, it's normally not worth it, but when you need it, you really need it.
- mlthoughts2018 6y agoI definitely agree that could be a case where you want a statically compiled module that avoid any interpreted language overheads or high cost abstractions. But what would make C++ easier to write, tune, integrate or deploy in that case than using Cython to create the C++ extension for you?
- disgruntledphd2 6y agoI dunno man, I was always against running stuff in C++ if I didn't have to, but I got over-ruled. I guess that the high availability of C++ developers helped swing the decision.
- viig99 6y agoI personally find C++ + pybind11 vastly easier to work with, also transitioning completely to c++ from there was a pretty small leap.
- mlthoughts2018 6y agoInteresting, I’ve never heard anyone who frequently uses Python and C++ together express this preference, it’s always the other direction that Cython is easier.
- viig99 6y agopytorch is pybind11 + c++
- mlthoughts2018 6y agoTrue, but that one project is just a drop in the bucket of scientific computing and C++ interop in Python, even despite the success and popularity of PyTorch - so it doesn’t really say much in favor of pybind that this or that project got good mileage out of it, it’s still such a deep minority compared to Cython.