5 ms·

Been wondering about this for a while... How does one "graduate" a notebook to an actual program on a server? Is there an actual standard procedure for that? I

by cfv 6y ago

Been wondering about this for a while... How does one "graduate" a notebook to an actual program on a server? Is there an actual standard procedure for that? I'm pretty sure that if your notebook is big enough to debug what's in it you'd be doing everyone a disservice keeping it contained like that.

realbarack 6y ago
The fast.ai folks wrote nbdev to help with that. https://github.com/fastai/nbdev https://github.com/fastai/nbdev
- lllmarclll 6y ago
  yfcuyrfuyoguyvuytfvty
  - lllmarclll 6y ago
    ycfyitcyvy
starpilot 6y ago
Run https://jupyter.org/hub https://jupyter.org/hub if you mean sharing the notebook with central administration. Deploying is a whole nother deal.
zomglings 6y ago
I generally use the `jupyter nbconvert` command when I have to do something like this. This can convert a Jupyter notebook into a Python script but unless the notebook has been written with some care (with the intention of converting it into a script later), a lot of manual twiddling will be required. What I generally do is use Jupyter notebooks when I'm exploring a problem or dataset. When I'm done with the exploration phase, I immediately mark the notebook as deprecated (trying to keep it in sync with code is a nightmare) and link the the relevant source code elsewhere in the project that it was deprecated for. Databricks does a decent job productionizing their notebooks, and have seen people be quite productive on that platform.
musingsole 6y ago
https://papermill.readthedocs.io/en/latest/ https://papermill.readthedocs.io/en/latest/ Papermill and some light container tooling can take you a long way. It gets bonus points because data science folks are quicker to jump into a notebook and debug their own equations than they are a Python module.
zmk_ 6y ago
Jupytext [1] has been a good option to keep scripts alongside notebooks. The advantage is that it also keeps both versions synced so you can edit whichever you want (just not at the same time, as the browser will not live update cell contents for you). [1] https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext
fermienrico 6y ago
Along those lines, what's wrong with using free and open source editors with great debug support such as VSCode? I don't understand the appeal of using Jupyter notebooks unless you're plotting or documenting.
jinpan 6y ago
I gradually graduate a notebook by extracting what I can into libraries, and importing those utilities into the notebook. Eventually the logic in the notebook becomes trivial enough to easily convert into a production program.
megamindbrian2 6y ago
Someone at IBM turned a notebook into an RPC callable micro-service. I've adapted that idea too, and put an http (gcloud/away compatible) wrapper around all my functions. It was here but that is apparently gone. http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-restful-microservices/ http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-re...
maartenbreddels 6y ago
For a visual output, voila can give you a dashboard look: https://github.com/voila-dashboards/voila/ https://github.com/voila-dashboards/voila/ see e.g. https://github.com/maartenbreddels/voila-demo https://github.com/maartenbreddels/voila-demo for a mobile-ready responsive rendering of a notebook.

by cfv 6y ago

realbarack 6y ago

The fast.ai folks wrote nbdev to help with that. https://github.com/fastai/nbdev https://github.com/fastai/nbdev

lllmarclll 6y ago
yfcuyrfuyoguyvuytfvty
- lllmarclll 6y ago
  ycfyitcyvy

starpilot 6y ago

Run https://jupyter.org/hub https://jupyter.org/hub if you mean sharing the notebook with central administration. Deploying is a whole nother deal.

zomglings 6y ago

I generally use the `jupyter nbconvert` command when I have to do something like this. This can convert a Jupyter notebook into a Python script but unless the notebook has been written with some care (with the intention of converting it into a script later), a lot of manual twiddling will be required. What I generally do is use Jupyter notebooks when I'm exploring a problem or dataset. When I'm done with the exploration phase, I immediately mark the notebook as deprecated (trying to keep it in sync with code is a nightmare) and link the the relevant source code elsewhere in the project that it was deprecated for. Databricks does a decent job productionizing their notebooks, and have seen people be quite productive on that platform.

musingsole 6y ago

https://papermill.readthedocs.io/en/latest/ https://papermill.readthedocs.io/en/latest/ Papermill and some light container tooling can take you a long way. It gets bonus points because data science folks are quicker to jump into a notebook and debug their own equations than they are a Python module.

zmk_ 6y ago

Jupytext [1] has been a good option to keep scripts alongside notebooks. The advantage is that it also keeps both versions synced so you can edit whichever you want (just not at the same time, as the browser will not live update cell contents for you). [1] https://github.com/mwouts/jupytext https://github.com/mwouts/jupytext

fermienrico 6y ago

Along those lines, what's wrong with using free and open source editors with great debug support such as VSCode? I don't understand the appeal of using Jupyter notebooks unless you're plotting or documenting.

jinpan 6y ago

I gradually graduate a notebook by extracting what I can into libraries, and importing those utilities into the notebook. Eventually the logic in the notebook becomes trivial enough to easily convert into a production program.

megamindbrian2 6y ago

Someone at IBM turned a notebook into an RPC callable micro-service. I've adapted that idea too, and put an http (gcloud/away compatible) wrapper around all my functions. It was here but that is apparently gone. http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-restful-microservices/ http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-re...

maartenbreddels 6y ago

For a visual output, voila can give you a dashboard look: https://github.com/voila-dashboards/voila/ https://github.com/voila-dashboards/voila/ see e.g. https://github.com/maartenbreddels/voila-demo https://github.com/maartenbreddels/voila-demo for a mobile-ready responsive rendering of a notebook.