5 ms·
CometML wants to do for machine learning what GitHub did for code
- gidim 8y agoHi, I’m one of the founders of Comet.ml. We built comet.ml to allow machine learning teams to automatically track their machine learning code, experiments, hyperparameters and results. We think that reproducibility is really important so we’re also giving free access to students, academics and open source projects. Feedback is welcome. Ask me anything.
- deleted 8y ago[deleted]
- ah- 8y agoAre you planning to open source it? A lot of your competitors have, like http://pipeline.ai/ http://pipeline.ai/, https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm and recently https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon.
- mmq 8y ago@ah- thanks for mentioning Polyaxon and congrats to the CometMl team for building this nice tool, it's good to see that many projects are trying to solve problems related to reproducibility in ML/DL, many people had to build an internal tool for the companies they work for to solve this issue, and many got frustrated after joining a new team and were not able to reproduce any results. I would like to outline a couple of differences between CometML and Polyaxon, as mentioned before, we are also trying to solve issues related to technical debt in ML, but not only, Polyaxon tries also to simplify training and scheduling parallel and distributed learning. there are also a couple of differences, I see CometML as dashboard, Polyaxon does not have an extensive dashboard as CometML, but it leverages Tensorboard for most of the visualisations. We use the CLI or the API for programatic access to the platform. Most importantly, Polyaxon aims to be an open source and to be installed on premise or in the cloud, it solves the issue related to code tracking based on an internal git and a docker registry, and as someone else mentioned that resources for running an experiment could be an issue for future reproducibility, Polyaxon restarts the experiments with the same resources and dockerfiles, it also tracks hyper params as part of the configuration. For hyper params tuning and suggestion, Polyaxon can also do hyper params search based on a couple of algorithms, and for the next release, it will include also a service similar to vizier for suggesting more experiments/group of experiments based on a given search space. Disclaimer: I am the author of Polyaxon
- jchung 8y agoLooks awesome. I'd love to give it a try with my team. Would you be open to extending the free teams plan to high tech nonprofits in addition to the access you provide to students, academics, and open source projects? We tend to work on distributed projects frequently with industry experts doing pro bono work for us, and something like comet could simplify our collaboration. The size of these pro bono project teams tend to ebb and flow, much like an open source project, so effective collaboration tools are critical for ramping up new folks as well as retaining learning when folks cycle off.
- gidim 8y agoSure. Shoot us an email and we'll get you started! mail@comet.ml
- shackenberg 8y agoHi, looks great! Love the feature of tracking code changes. But how does this work? Does it upload the code to the servers every time I launch a training?
- gidim 8y agoIt depends if it's a git project but pretty much yes.
- shackenberg 8y agoSo you would considered the latest committed version as the 'code of the current experiment'? Maybe to rephrase: I start a training with my local code and then I change one variable in the code or comment some processing and start the next training. Would I need to do something for comet to know how the code has changed?
- gidim 8y agoNo. You'll be able to see both runs with the code diffs.
- ReverseCold 8y agoDoes "did what GitHub did for code" sound like a negative thing to anyone else? It turned a decentralized platform (git) into basically the only place individuals store code.
- erk__ 8y agoWhen I saw that name the first thing I thought about was some new SML compiler or like MoscowML.
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- Macuyiko 8y agoGreat to see ML "governance" work being done on the training-part of the pipeline. Seems like this provides a Domino Data Labs based dashboards but without the walled garden environment. I've yet to see similar great initiatives also tackling the deployment-part. E.g. something similar you can stick on top of your model's API (or scheduled batch predictive outputs), as well as incoming instances, to monitor usage patterns, population shifts through time, probability distributions, newly popping up missing values or categorical levels, logs, etc, in order to provide warning lights to indicate that a retraining might be in order, for instance. Google's "What's your ML test score" paper provides some great insights, but I hope someone will tackle this with a turnkey solution as well.
- gidim 8y agoThanks! We indeed solve a similar pain point as Domino but we unlike them we allow you to train your models on your own infra/laptop. As for monitoring production models that's something we're also working on. It was important to get the training part out first so we can measure those distributions changing.
- deleted 8y ago[deleted]
- yodon 8y agoCan you talk about how CometML fits into the real world state of ML training tracking, which is a pretty terrible Wild West of non-reproducible practices and processes? There have been articles and comments here on HN about the sorry state of ML trackability, with papers being published on models whose training is not reproducible because no one really knows how it was trained. One in particular (I apologize for not having retained the link) described researchers starting with partially trained models they had lying around (with undocumented and unknown prior training applied), manually changing hyper parameters mid training while watching the learning progress, swapping different training sets in and out, and etc. From what I see, the problems in ML reproducibility aren’t in the code, they are in the external human processes that are used to drive and train the models (essentially bad DevOps practices more than bad dev practices). Do you help with these kind of real-world trackability and reproducabilty scenarios?
- gidim 8y agoThat's a great point! Machine Learning reproducibility is a huge problem, both in Academia and within companies. Comet.ml tracks every run of your script, the hyper-params (pulled automatically from the ML library when possible) and your results. So in the example you gave we would be able to track the DS loading pre-trained weights, manually changing hyperparams and the datasets hash. In such complex cases there's still a little discipline required on the user behalf but overall it's just including a few calls to our Experiment object.
- laythea 8y agoIm no expert but reading that it sounds like a bunch of people poking at knobs and dials in order to see which dials work and which don't, and then not understanding why...?
- jdoliner 8y agoMy sense is that CometML is tackling some but not all of the problems you're talking about here. Specifically I think it's pretty focused on tracking ML models so you can keep track of how models are performing, how they're improving etc. But I think, and the other comment seems to suggest, that external systems and practices will be required to make it so these results can be reproduced. Making ML experiments reproducible is a different and fairly challenging problem. You need to have a way to essentially snapshot everything that goes into training a model, training data, code, tuning parameters maybe even specific hardware it needs to run on. As you mention, this is as much a DevOps problem as it is an ML problem, maybe even more so. I think you'd have good luck pairing CometML with a system like Pachyderm [0] (which, full disclosure, is a system I helped write). Pachyderm can handle the Devopsy reproducibility piece and be paired with CometML to get insight into what's actually going on with the models. [0] https://github.com/pachyderm https://github.com/pachyderm
- spraak 8y agoI get so excited when I see ___ML but then I'm let down when I realize the context is Machine Learning instead of Meta Language. Not that Machine Learning isn't cool, but I really like ML family languages.
- jdpigeon 8y agoI have been anxiously waiting for something like this! Very excited to try it out. How does it handle data storage? Could we use CometML to store our continuously growing set of labeled data or is there a smart way to link it to Google Cloud Platform?
- gidim 8y agoWe do not store your data as it's usually a very sensitive. You can host your data on GCloud or AWS and use Comet.ml to track where it was coming from and if it changed between experiments.
- ExcelSaga 8y agoThis is nothing, but good news, I hope it goes well!
- jorgemf 8y agoI would like to use it but I think the price doesn't justify the tool. For a team of 5 people github is $25 a month, you are $745 a month. I do understand a price a bit higher that github but not 30 times more expensive.
- gidim 8y agoThanks @jorgemf. Keep in mind that $745 also includes unlimited usage of our hyper-parameter optimization service.
- XnoiVeX 8y agoAre there any similarities or distinctions between this article (link below) and how your system works to tune hyper-parameters? https://blog.coast.ai/lets-evolve-a-neural-network-with-a-genetic-algorithm-code-included-8809bece164 https://blog.coast.ai/lets-evolve-a-neural-network-with-a-ge...
- gidim 8y agoThis article seems to discuss genetic algorithms which could be used for hyperparam optimization. We use another method called Bayesian (GP) hyperparam optimization. According to our internal benchmarks and academic research Bayesian methods outperform genetic algorithms. Another thing to keep in mind that we automate the entire process for you. You only need to provide a list of parameters you'd like to tune.
- maksimum 8y agoIs hyper-parameter optimization a cherry on top or one of the core value propositions? IME hyper-parameter optimization doesn't require much in terms of implementation effort (e.g. [1]), but requires compute. I would be surprised if a professional ML/DS user were to seriously consider paying for the implementation of the optimization. [1] https://people.eecs.berkeley.edu/~kjamieson/hyperband.html https://people.eecs.berkeley.edu/~kjamieson/hyperband.html
- jorgemf 8y ago
- XnoiVeX 8y agoCometML vs Tensorboard? Thanks in advance for your feedback.
- gidim 8y agoBoth CometML and Tensorboard help track metrics/weights during training in a similar way. CometML also tracks your hyperparams, code, dependencies. We also allow you to compare models, collaborate by sharing projects and experiments and much more. TB is an amazing tool but it doesn't help with reproducibility. If you're already using Tensorboard just throw in our one liner: comet_ml.experiment(api_key="your-key") and you'll get everything TB gives you + our added value.
- nicodjimenez 8y agoThat's cool! I also wrote my own service https://losswise.com https://losswise.com
- dspoka 8y agoDo you have any information on pricing?
- nicodjimenez 8y agoCurrent feature set is completely free as the service is in early beta stage. We plan to charge later for additional features and services. The project started as an internal project for Mathpix.com and therefore already pays for itself.
- henripal 8y agoCool. I'm guessing from trying it out that you're using Highcharts. I've run into really unpleasant memory leaks/slowness when streaming data (especially when the existing chart is already thousands of data points). Are you seeing something similar?
- nicodjimenez 8y agoYes using Highcharts. You've had issues with Highcharts? Yeah it's not designed to stream data extremely rapidly but it's a great "good enough" product, especially for something like Losswise where the differentiation is the overall design and architecture and developer experience, not the prettiest possible graphs.
- henripal 8y agoYes... For example if I'm running three experiments at the same time, auto-refreshing the chart every two seconds, it essentially freezes the app to a crawl after a thousand points or so. So we reverted to manual updates. If you know of any better alternatives for data streaming, I'm curious. I tried benchmarking a couple libs recently: https://github.com/henripal/ChartingLibBenchmark https://github.com/henripal/ChartingLibBenchmark
- deleted 8y ago[deleted]
- p1esk 8y agoHow do you handle dataset/checkpoint management and versioning? Ideally with powerful dataset filtering options. Right now we’re using excel and panda dataframes, but are interested if your tool does it well.
- gidim 8y agoSince we do not host your data we cannot provide filtering on the actual dataset content. We do allow you to track where the data was coming from and if it changed (by hash). Same for checkpoints, you can log their location (S3/local path) and hash.
- henripal 8y agoCool, thanks. For those of you who want to tinker, there's a much rougher, open source library based on Vuejs, postgres, and Flask with some momentum on GitHub right now, LabNotebook https://github.com/henripal/labnotebook https://github.com/henripal/labnotebook (Disclaimer: I'm one of the authors)
- sbraden 8y agoSince a few people commented on the cost of using CometML (or its competitors), I wanted to suggest an open source project (that's been around for a while) I found helpful for organizing ML experiments and tracking. It has two different frontends to choose from (I like SacredBoard). If you like open source this might be the ML experiment tracker for you! edit (forgot the link): https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred
- softawre 8y agoDid you forget to suggest the project? Is it "sacred"? https://github.com/IDSIA/sacred https://github.com/IDSIA/sacred
- andbberger 8y ago+1 for sacred. It's changed my life.
- ellisv 8y agoData science has 3 areas for "versioning": 1. code versioning 2. data versioning 3. model versioning Code versioning is primarily dominated by GitHub and is a fairly saturated space (Bitbucket, GitLab). Data versioning is either not happening, or being done through regular data pulls, database snapshots, etc. It is not well standardized or adopted. CometML is tackling model versioning. It would be really nice to have a single solution for all of these but that is unlikely. Hopefully new standards evolve from this.
- gidim 8y agoWe actually do code and model versioning (and simple data versioning). One thing to keep in mind is that code/results/hyperparams must be coupled. If you have a git branch with some training code and you do not know what the hyperparams/results are then it's not very valuable.
- sandGorgon 8y agoCould you talk more about a simple data versioning architecture. I have wondered about this coupling problem and would love to hear more.
- gidim 8y agoI was mostly referring to coupling code with results. For example you have a code that loads a dataset from S3 and then trains a neural network. If you only use git you're likely to lose the hyperparms info (which is often passed as command line arguments) and your metrics/results.
- alexilliamson 8y agoNice breakdown. I agree that data versioning is the one area with limited standardized options. I would add that in addition to versioning the data, there is also the related problem of integrating the 3 areas of versioning... tying the "data version" to the "model version" and the "code version". That seems to me like it might be a good place to start in tackling data versioning, or is that too trivial? Is there a product out there that already does this?
- tedivm 8y agoTheir pricing is ridiculous. $50 per user per month for their cheapest plan that works with private spaces, $149 per user per month for self hosted. If you're a company with 15 people working on a single model you're going to be paying more than a team with five people who have 20 different models they are working on. The actual load and cost for the service is completely detached from the price. I was strongly considering looking at this for a project, but not at that cost.
- suff 8y agoGreat idea if AWS had not already solved this with the release of SageMaker.
- tedivm 8y agoSageMaker is awesome but it's a very different product. There is nothing in SageMaker that lets you track performance over time and no fancy dashboard. That being said SageMaker's pricing is actually reasonable, and adding a dashboard on top of it shouldn't be that complicated.
- nightski 8y agoSorry if this is off topic but does anyone else just instantly close the page or spew profanity when those stupid chat bubbles pop up on a page? No I don't want to talk to your ridiculous sales chat.
- euler_ 8y agoIs there an explore option like on github?