5 ms·
ML engineer here, team started as a research team, now that we have things in production and have a lot of devops, engineering work, we bifurcated into pods and
by viig99 6y ago
ML engineer here, team started as a research team, now that we have things in production and have a lot of devops, engineering work, we bifurcated into pods and work on specific bits and pieces, lot of constant fire-fighting though. Re-wrote entire stack from python to C++ threadpool async grpc (is thrift the only good threadpool server implementation available ?), deployed on openshift, used vector + influx + grafana for dashboards / internal model monitors, elastic search for loggings, lot of other tools for validation, filtering for potential training candidates etc. Right now working CI/CD for ml, during training if model finds a better model based on different validation sets, have one click deployment ready for approval etc
- dnautics 6y agoWow, thanks for the detail. > Re-wrote entire stack from python to C++ threadpool async grpc Incredible. Presumably this is for latency/performance on the inference side?
- viig99 6y agoYes accuracy, latency & throughput are the 3 poles we try to achieve, c++ helps with latency & throughput and helps keep the cost low.
- mlthoughts2018 6y agoWhy would c++ help with latency in comparison to say Python with numpy / numba / Cython? All the production critical “this needs to be as fast as possible stuff” I’ve ever worked on has been all Python, achieving complete speed parity with C, at a much faster development speed and with way way less boilerplate code.
- disgruntledphd2 6y agoIf you have hard constraints at inference time, then it can be much easier to tune to a time budget with C++. Like, it's normally not worth it, but when you need it, you really need it.
- mlthoughts2018 6y agoI definitely agree that could be a case where you want a statically compiled module that avoid any interpreted language overheads or high cost abstractions. But what would make C++ easier to write, tune, integrate or deploy in that case than using Cython to create the C++ extension for you?
- disgruntledphd2 6y agoI dunno man, I was always against running stuff in C++ if I didn't have to, but I got over-ruled. I guess that the high availability of C++ developers helped swing the decision.
- viig99 6y agoI personally find C++ + pybind11 vastly easier to work with, also transitioning completely to c++ from there was a pretty small leap.
- mlthoughts2018 6y agoInteresting, I’ve never heard anyone who frequently uses Python and C++ together express this preference, it’s always the other direction that Cython is easier.
- viig99 6y agopytorch is pybind11 + c++
- mlthoughts2018 6y agoTrue, but that one project is just a drop in the bucket of scientific computing and C++ interop in Python, even despite the success and popularity of PyTorch - so it doesn’t really say much in favor of pybind that this or that project got good mileage out of it, it’s still such a deep minority compared to Cython.