5 ms·
Thanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow)
by aboytsov 14y ago
Thanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow): http://www.youtube.com/watch?feature=player_detailpage&v=BUgxmvpuKAs#t=1016s http://www.youtube.com/watch?feature=player_detailpage&v...
Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on. This branch depends on some other files in the workflow. Let's say, generate synonyms from the sentence dataset. Or, some complicated cleaning of some intermediate data. This is not a small task and you will spend a couple of days doing it, re-running your code dozens of times in the process.
You don't care about other parts of the workflow. You only care about what you're developing and how it propagates. Does it propagate? Does it break something down the road? What is the final output? Did all this synonym collection help? Did the changes you made in learning code improve the results?
When you're done, you may commit your code and somewhere else somebody will build a nice new dataset, but while you're working on it, you really need to be able to run any target individually, with dependencies or without, as well as forcibly rebuild all steps down the tree to see the final result.
Makes sense?
- tedunangst 14y agoThanks.