6 ms·

Closer to production quality Python notebooks with `marimo check`

31 points by dmadisetti 1y ago

wodenokoto 11mo ago
I’m very interested in moving beyond Jupyter notebooks so I have my eye on marimo. My understanding is it’s just python code with a bit of notebook hints and an idea that gives the notebook experience. So I didn’t quite catch from the article why ruff and pylons aren’t enough.
- akshayka 11mo ago
  Hi! Thanks for your interest. marimo is much more than that — unlike traditional notebooks, marimo is "reactive", meaning it models notebooks as dataflow graphs and keeps code an outputs in sync. Moreover, marimo notebooks are not "just notebooks". They can be seamlessly run as interactive web apps or as Python scripts. Here is our original ShowHN post that explains what marimo is all about: https://news.ycombinator.com/item?id=38971966 https://news.ycombinator.com/item?id=38971966 Blog that goes deeper: https://marimo.io/blog/lessons-learned https://marimo.io/blog/lessons-learned
  - daxfohl 11mo ago
    I was just thinking about something like this a couple hours ago. Jeremy Howard (fast.ai, etc) posted a few weeks ago about his new AI class [1] and I was pumped. I went through the fast.ai course a couple years ago and had been wondering if another would ever come out, and the overview of this sounded really interesting. But the first thing the announcement went into was Jupyter Notebooks, and it kept on and on about them and I rolled my eyes. I remember they were my least favorite part of the fast.ai class. They've always felt awkward to me, being used to a code editor and all the functionality it provides, and I'm sure I'm not alone. I'm sure Jupyter is great for fast iteration if that's what you do all day, but if you spend most of your life in a standard code editor, you really want something that starts from there and adds a bit of interactivity on top, rather than jumping to a completely different concept. I haven't tried marimo yet, but I might give it a shot for my next project. [1] https://news.ycombinator.com/item?id=45455810 https://news.ycombinator.com/item?id=45455810
    - spwa4 11mo ago
      Just use VSCode, which will give you full language server editing in ipynb files?
      - daxfohl 11mo ago
        Maybe. At least that way it doesn't feel like some separate half-baked proprietary code editor from the 90s, which is the vibe I get from Jupyter. But a regular .py file with a couple attributes that can generate notebooks still sounds far nicer, assuming it works fairly seamlessly.
- ravila4 11mo ago
  I haven’t used it, but from looking at Marimo’s examples and docs, I’m not convinced by some of its design choices. The idea that you can run notebook cells out of order is supposed to be a strength, but I actually see it as a weakness. The order of cells is what makes a notebook readable and self-documenting. The discipline of keeping cells in order may be painful, but it’s what makes the flow of analysis understandable to others. Also, I find the way Marimo uses decorators and functions for defining cells pretty awkward (Although it’s nicely abstracted away in the UI). It looks like normal Python, but the functions don’t behave like real functions, and decorators are a fairly advanced feature that most beginners don’t use. For me, Quarto notebooks strike a better balance when it comes to generating sharable documents, prototypes, and reports. They’re git-friendly, use simple markup for defining cells, and still keep the clear, linear structure. However, Marimo might be the best tool for replacing Streamlit apps and “production notebooks” (Although I’d also argue that notebooks should not be in production).
  - dmadisetti 11mo ago
    marimo has a quarto extension and a markdown fileformat [1] (marimo check works on this too!). The python fileformat was chosen such that "notebooks" are still valid python, but yes- the format itself is almost an implementation detail to most "notebook" users. Cells _are_ actually callable and importable functions though (you can give them a name), but the return signature is a bit different from what's serialized. > The discipline of keeping cells in order may be painful, but it’s what makes the flow of analysis understandable to others. We might have to agree to disagree here, you can still chose to have your notebook in order and something you can be disciplined about. The difference is that a marimo notebook can't become unreproducible the same way a jupyter notebook can, _because_ the order doesn't matter. But thanks for the feedback! [1]: https://github.com/marimo-team/quarto-marimo https://github.com/marimo-team/quarto-marimo
loveparade 11mo ago
I wish this didn't have AI in it. I've been looking for a Jupyter alternative that is pure python and can be modified from a regular text editor. Jupytext works okay, but I miss the advanced Jupyter features. But I really don't want to deal with yet another AI assistant, especially not a custom one when I'm already using Claude/etc from the CLI and I want those agents to help me edit the notebooks. Take out all the AI stuff and I'd give it a try. I use AI coding agents as my daily driver, but I really don't need this AI enshittification in every tool/library I'm using.
- 3eb7988a1663 11mo ago
  I hate how much I lean into VSCode, but the Python interactive mode gets you a really good live coding environment. Instead of Jupyter cells, you have a regular .py file with chunks of code prefixed with a `# %%`. VSCode gives you a similar experience to a notebook, with the same controls (Run Above Cells, Restart and Run All, etc). So something like # %% import polars as pl # %% df = pl.DataFrame() df.shape # %% def foobar(): return 1 Since it is a regular .py file all of your existing tooling will work with it. The one thing you lose vs a Jupyter notebook is saved output. I mostly use these .py files, but have a few .ipynb notebook files for when I want to commit the output from some important task.
- akshayka 11mo ago
  You can not use the AI features, nothing is enabled by default (you have to bring your own keys)
- daxfohl 11mo ago
  Reading the article, I don't think it has AI. They've just made the tools in a way that AI assistants can also use them, and so fix linting errors without anyone needing to fine-tune the LLM on the syntax. That's actually pretty slick. I've been wondering how we could avoid blocking innovation in programming languages because of the death cycle of "no training data on language -> LLM can't learn language -> Assistant can't code language -> nobody uses language -> no training data on language".
  - 11mo ago
ZeroCool2u 11mo ago
Any updates on the revamped VS Code extension? That's kind of a deal breaker for me.
- Haleshot 11mo ago
  v.07 is live now on both VS Code and Open VSX marketplaces. https://github.com/marimo-team/marimo-lsp/ https://github.com/marimo-team/marimo-lsp/
- dmadisetti 11mo ago
  Soft launched in pre-release, but a full version is coming!
dmadisetti 1y ago
marimo notebooks double as python Python programs, which means they can be lint-ed like any other code. `marimo check` catches notebook-specific issues (variable redefinition across cells, circular dependencies) with Rust-inspired (and uv; uv is amazing) error messages. It outputs JSON for AI agents to self-correct, integrates into CI pipelines, and includes `--fix` flags for automatic repairs. The team is already using it in their own CI and seeing Claude Code iterate on notebooks without human intervention.
- aitchnyu 11mo ago
  IME Agents work fine on human readable error messages.
- sixo 11mo ago
  Unfortunately the code in Marimo notebooks implements cross-variable references as untyped function arguments, meaning that nothing passed between cells can be type-safe. This makes it very hard to use tooling directly on the Python representation of notebooks.
  - dmadisetti 11mo ago
    Arguments are typed if explicitly specified in definition, e.g. ```python @app.cell def my_cell(): x : int = 1 return (x,) @app.cell def consumer(x : int): y = x return ``` We've talked about building out implicit typing into our serialization- but that may be a larger can of worms.
    - sixo 11mo ago
      That would be wonderful. A lot of the arguments are just imports, and these would have probably have the largest upside. It would be great if there was a construct like import numpy @app.cell def my_cell(np: types.ModuleType[numpy]): ... editor treats np like `import numpy as np` ... I do not use Python enough to know if something like this can be hacked together. But if not, I imagine could be PIPed into the standard library if a convincing case were made for it.

31 points by dmadisetti 1y ago

I’m very interested in moving beyond Jupyter notebooks so I have my eye on marimo. My understanding is it’s just python code with a bit of notebook hints and an idea that gives the notebook experience. So I didn’t quite catch from the article why ruff and pylons aren’t enough.

akshayka 11mo ago
Hi! Thanks for your interest. marimo is much more than that — unlike traditional notebooks, marimo is "reactive", meaning it models notebooks as dataflow graphs and keeps code an outputs in sync. Moreover, marimo notebooks are not "just notebooks". They can be seamlessly run as interactive web apps or as Python scripts. Here is our original ShowHN post that explains what marimo is all about: https://news.ycombinator.com/item?id=38971966 https://news.ycombinator.com/item?id=38971966 Blog that goes deeper: https://marimo.io/blog/lessons-learned https://marimo.io/blog/lessons-learned
- daxfohl 11mo ago
  I was just thinking about something like this a couple hours ago. Jeremy Howard (fast.ai, etc) posted a few weeks ago about his new AI class [1] and I was pumped. I went through the fast.ai course a couple years ago and had been wondering if another would ever come out, and the overview of this sounded really interesting. But the first thing the announcement went into was Jupyter Notebooks, and it kept on and on about them and I rolled my eyes. I remember they were my least favorite part of the fast.ai class. They've always felt awkward to me, being used to a code editor and all the functionality it provides, and I'm sure I'm not alone. I'm sure Jupyter is great for fast iteration if that's what you do all day, but if you spend most of your life in a standard code editor, you really want something that starts from there and adds a bit of interactivity on top, rather than jumping to a completely different concept. I haven't tried marimo yet, but I might give it a shot for my next project. [1] https://news.ycombinator.com/item?id=45455810 https://news.ycombinator.com/item?id=45455810
  - spwa4 11mo ago
    Just use VSCode, which will give you full language server editing in ipynb files?
    - daxfohl 11mo ago
      Maybe. At least that way it doesn't feel like some separate half-baked proprietary code editor from the 90s, which is the vibe I get from Jupyter. But a regular .py file with a couple attributes that can generate notebooks still sounds far nicer, assuming it works fairly seamlessly.
ravila4 11mo ago
I haven’t used it, but from looking at Marimo’s examples and docs, I’m not convinced by some of its design choices. The idea that you can run notebook cells out of order is supposed to be a strength, but I actually see it as a weakness. The order of cells is what makes a notebook readable and self-documenting. The discipline of keeping cells in order may be painful, but it’s what makes the flow of analysis understandable to others. Also, I find the way Marimo uses decorators and functions for defining cells pretty awkward (Although it’s nicely abstracted away in the UI). It looks like normal Python, but the functions don’t behave like real functions, and decorators are a fairly advanced feature that most beginners don’t use. For me, Quarto notebooks strike a better balance when it comes to generating sharable documents, prototypes, and reports. They’re git-friendly, use simple markup for defining cells, and still keep the clear, linear structure. However, Marimo might be the best tool for replacing Streamlit apps and “production notebooks” (Although I’d also argue that notebooks should not be in production).
- dmadisetti 11mo ago
  marimo has a quarto extension and a markdown fileformat [1] (marimo check works on this too!). The python fileformat was chosen such that "notebooks" are still valid python, but yes- the format itself is almost an implementation detail to most "notebook" users. Cells _are_ actually callable and importable functions though (you can give them a name), but the return signature is a bit different from what's serialized. > The discipline of keeping cells in order may be painful, but it’s what makes the flow of analysis understandable to others. We might have to agree to disagree here, you can still chose to have your notebook in order and something you can be disciplined about. The difference is that a marimo notebook can't become unreproducible the same way a jupyter notebook can, _because_ the order doesn't matter. But thanks for the feedback! [1]: https://github.com/marimo-team/quarto-marimo https://github.com/marimo-team/quarto-marimo

loveparade 11mo ago

I wish this didn't have AI in it. I've been looking for a Jupyter alternative that is pure python and can be modified from a regular text editor. Jupytext works okay, but I miss the advanced Jupyter features. But I really don't want to deal with yet another AI assistant, especially not a custom one when I'm already using Claude/etc from the CLI and I want those agents to help me edit the notebooks. Take out all the AI stuff and I'd give it a try. I use AI coding agents as my daily driver, but I really don't need this AI enshittification in every tool/library I'm using.

3eb7988a1663 11mo ago
I hate how much I lean into VSCode, but the Python interactive mode gets you a really good live coding environment. Instead of Jupyter cells, you have a regular .py file with chunks of code prefixed with a `# %%`. VSCode gives you a similar experience to a notebook, with the same controls (Run Above Cells, Restart and Run All, etc). So something like # %% import polars as pl # %% df = pl.DataFrame() df.shape # %% def foobar(): return 1 Since it is a regular .py file all of your existing tooling will work with it. The one thing you lose vs a Jupyter notebook is saved output. I mostly use these .py files, but have a few .ipynb notebook files for when I want to commit the output from some important task.
akshayka 11mo ago
You can not use the AI features, nothing is enabled by default (you have to bring your own keys)
daxfohl 11mo ago
Reading the article, I don't think it has AI. They've just made the tools in a way that AI assistants can also use them, and so fix linting errors without anyone needing to fine-tune the LLM on the syntax. That's actually pretty slick. I've been wondering how we could avoid blocking innovation in programming languages because of the death cycle of "no training data on language -> LLM can't learn language -> Assistant can't code language -> nobody uses language -> no training data on language".
- 11mo ago

ZeroCool2u 11mo ago

Any updates on the revamped VS Code extension? That's kind of a deal breaker for me.

Haleshot 11mo ago
v.07 is live now on both VS Code and Open VSX marketplaces. https://github.com/marimo-team/marimo-lsp/ https://github.com/marimo-team/marimo-lsp/
dmadisetti 11mo ago
Soft launched in pre-release, but a full version is coming!

dmadisetti 1y ago

marimo notebooks double as python Python programs, which means they can be lint-ed like any other code. `marimo check` catches notebook-specific issues (variable redefinition across cells, circular dependencies) with Rust-inspired (and uv; uv is amazing) error messages. It outputs JSON for AI agents to self-correct, integrates into CI pipelines, and includes `--fix` flags for automatic repairs. The team is already using it in their own CI and seeing Claude Code iterate on notebooks without human intervention.

aitchnyu 11mo ago
IME Agents work fine on human readable error messages.
sixo 11mo ago
Unfortunately the code in Marimo notebooks implements cross-variable references as untyped function arguments, meaning that nothing passed between cells can be type-safe. This makes it very hard to use tooling directly on the Python representation of notebooks.
- dmadisetti 11mo ago
  Arguments are typed if explicitly specified in definition, e.g. ```python @app.cell def my_cell(): x : int = 1 return (x,) @app.cell def consumer(x : int): y = x return ``` We've talked about building out implicit typing into our serialization- but that may be a larger can of worms.
  - sixo 11mo ago
    That would be wonderful. A lot of the arguments are just imports, and these would have probably have the largest upside. It would be great if there was a construct like import numpy @app.cell def my_cell(np: types.ModuleType[numpy]): ... editor treats np like `import numpy as np` ... I do not use Python enough to know if something like this can be hacked together. But if not, I imagine could be PIPed into the standard library if a convincing case were made for it.