vtuulos

6 ms·

1.
by vtuulos 1y ago
If you are ok with executing your SFN steps on AWS Batch, Metaflow should do the job well. It's pretty inhuman to interact with SFN directly. One feature that's in our roadmap is the ability to define DAG fully programmatically, m
2.
by vtuulos 1y ago
Stay tuned! We have some cool new features coming soon to support agentic workloads (teaser: https://github.com/Netflix/metaflow/pull/2473 ) If you are curious, join the Metaflow Slack at http://sla
3.
by vtuulos 1y ago
Metaflow tracks all artifacts and allows you to build dashboards with them, so there’s no need to use MLFlow per se. There’s a Metaflow integration in Weights and Biases, CometML etc, if you want pretty off-the-shelf dashboards
4.
by vtuulos 1y ago
Metaflow was started to address the needs of ML/AI projects whereas Airflow and Dagster started in data engineering. Consequently, a major part of Metaflow focuses on facilitating easy and efficient access to (large scale) compute - in
5.
by vtuulos 1y ago
I don't know if it's a coincidence but we just released a major new feature in Metaflow a few days ago - composing flows with custom decorators: https://docs.metaflow.org/metaflow/composing-flows/introduc
6.
by vtuulos 2y ago
if you want to see similar tricks applied in Python (with a JIT compiler for query-time optimization), take a look at this fun deck that I presented a long time ago: https://tuulos.github.io/sf-python-meetup-sep-2013 we wer
7.
Introducing Configurable Metaflow(netflixtechblog.com)
2 points by vtuulos 2y ago|0 comments
8.
by vtuulos 2y ago
yes, this. In case you are interested in seeing some numbers backing this claim, see here https://outerbounds.com/blog/metaflow-fast-data Source: I used to work at Netflix, building systems that pull TBs from S3 hourly
9.
by vtuulos 2y ago
I had the same concern. However, the structure of the output was surprisingly stable. We rejected badly formatted responses: https://github.com/outerbounds/hacker-news-sentiment/blob/ma... The semantics of th
10.
by vtuulos 2y ago
here's how the model ranks the discussion on this page after 40 comments: SENTIMENT 6 :D
11.
by vtuulos 2y ago
yep! And of course the new 3.1 model
12.
by vtuulos 2y ago
even simpler, you can just do it in SQL You can find all titles and dates since the beginning of HN in this public BigQuery dataset: https://console.cloud.google.com/marketplace/product/y-combi...
13.
by vtuulos 2y ago
That's an interesting hypothesis but the words we use to express agreement and disagreement haven't changed much. We don't try to retrieve articles/topics from the model, which would be affected by the cutoff, just askin
14.
by vtuulos 2y ago
yes please! The data is conveniently available as JSON blobs here https://github.com/outerbounds/hacker-news-sentiment/tree/ma...
15.
by vtuulos 2y ago
search "divisive" here: https://github.com/outerbounds/hacker-news-sentiment/blob/ma... I actually spent 10 minutes trying to see if there are obvious tests for U-shaped distributions. I'd love
16.
by vtuulos 2y ago
Exactly. Now we have a new UI paradigm and a new, incredibly powerful NLP tool in our toolkit. Is this AGI? Of course not. Is this useful and valuable? Obviously.
17.
by vtuulos 3y ago
here's how we solved engineering challenges related to RAG using open-source Metaflow: https://outerbounds.com/blog/retrieval-augmented-generation/
18.
by vtuulos 3y ago
Metaflow integrates with AWS Batch which many folks use for serious HPC. Internode scheduling happens through the multinode scheduling supported by AWS Batch. networking via EFA etc. We'll blog more about this soon but you can certain
19.
Fast Data: Loading Tables from S3 at Lightning Speed(outerbounds.com)
2 points by vtuulos 3y ago|0 comments
20.
Show HN: Train an LLM with Metaflow, Featuring Dolly(outerbounds.com)
3 points by vtuulos 3y ago|0 comments
21.
Show HN: Learn ML and AI infrastructure in the browser(outerbounds.com)
13 points by vtuulos 4y ago|3 comments
22.
by vtuulos 4y ago
Congrats for the launch Stefan and Elijah! :) Like Stefan mention in the OP, Hamilton works well with tools like Metaflow which can help with many other concerns you mentioned. How you define your data transformations for ML is an open ques
23.
by vtuulos 4y ago
Outerbounds | Systems / Full-Stack engineer | SF / Remote | Full-time Outerbounds commercializes Metaflow, a widely used open-source Python framework that makes infrastructure easily accessible for machine learning/data scien
24.
by vtuulos 4y ago
Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow
25.
Better Airflow with Metaflow(outerbounds.com)
8 points by vtuulos 4y ago|3 comments
26.
by vtuulos 4y ago
This is great! If you are interested in running ML workloads across clouds, Netflix's Metaflow will officially announce support for all clouds tomorrow: https://outerbounds.com/blog/metaflow-on-all-major-clouds&#x2
27.
MLOps for Foundation Models: Whisper and Metaflow(outerbounds.com)
1 points by vtuulos 4y ago|0 comments
28.
Orchestrating Data/ML Workflows at Scale with Netflix Maestro(netflixtechblog.com)
1 points by vtuulos 4y ago|0 comments
29.
by vtuulos 4y ago
If you want to experiment with something similar by yourself and you don't have the patience to wait for Stable Diffusion to crunch through thousands of images on your laptop or in a Colab notebook, here's how you can parallelize
30.
Metaflow Sandboxes: Free Data Science Infrastructure in the Browser(outerbounds.com)
4 points by vtuulos 4y ago|0 comments

More ›