10 ms·

Observable 2.0, a static site generator for data apps

655 points by tmcw 3y ago

fburnaby 3y ago
I can't see from the docs what this gives me over a Makefile, Asciidoctor (or pandoc, Jekyll etc), and D3?
terpimost 3y ago
Wow guys. That is so cool! Thank you for what you do. This is a great example of high quality product!
foolswisdom 3y ago
I really like this idea.
jeffbee 3y ago
I am totally psyched up to try this. Observable (1.0) has been a very effective outlet for my research. There is no other platform that could have hosted it and offered me the quick and easy tools to make persuasive visualizations. I was a little concerned when I heard some people got laid off from the company, but it seems like it is still going.
yodon 3y ago
Lots of good stuff here, but are you really naming your framework "Framework" (as in "With Framework, you can build...")? You do realize that naming only makes sense inside your own company, right? To everyone who uses it, it's "a" framework or "Observable's framework". No consumer of the framework is going to refer to it as "Framework" without ridiculous amounts of confusion resulting.
- renewiltord 3y ago
  It's called Observable Framework one sentence before and on the product page. It's normal to write like this in a blog post. For instance, the Kubernetes deployments page uses the word "Deployment" doesn't say Kubernetes Deployment everywhere. It just says Deployment. I think introducing it like that is fine. You don't have to say Observable® Framework™ every sentence. That would be strange.
- prakis 3y ago
  Yes, it is confusing. It took me sometime to realize their product name itself is framework.
- alexgarcia-xyz 3y ago
  Observable has other open source libraries with similar "generic" names: Plot, Runtime, Inputs. When speaking generically, most people say "Observable Plot" or "Observable Runtime." In projects where people already know about them, then I say "Plot" or "Inputs" without much fuss. I imagine most people will say "Observable Framework" when talking out in the open, and "Framework" on established projects.
0cf8612b2e1e 3y ago
Sounds a bit like the “baked data” pattern. Which I think is a really good idea. I have long been toying with how to use Datasette to make a deliverable dashboard, so this is interesting.
- rmnclmnt 3y ago
  Me too, and that lead to developing the « datasette-dashboards » plugin[0]. I use this for my company where all the data is gathered by connectors scheduled in CI, storing data in Git, and triggering a SQLite db build and Datasette deployment. « BI as Code » if you will [0] https://github.com/rclement/datasette-dashboards https://github.com/rclement/datasette-dashboards
  - pzmarzly 3y ago
    How often do you refresh the data (how often your CI runs)?
    - rmnclmnt 3y ago
      Depends on the nature of the data and the requirements, can be every hour or once a day. Most people and organizations have small/mid data and most of all slow data: you do not need fresh data every hour. If you do, you need a way heavier real time analytics data
- mbostock 3y ago
  Yes! It first felt counterintuitive and constraining to prepare data ahead of time, rather than just loading whatever you want on the fly. But we’ve found this to be a great discipline in practice because it forces you to think about what data you actually need to show, and to shift as much of the compute to build time to reduce file sizes (through aggregation and filtering) and accelerate page load. And once you become accustomed to instant dashboards it becomes unthinkable to go back to slow queries.
  - shiandow 3y ago
lf-non 3y ago
The new direction seems very similar to what evidence has been doing for a while https://evidence.dev https://evidence.dev
- mbostock 3y ago
  Yep, Evidence is doing good work. We were most directly inspired by VitePress; we spent months rewriting both D3’s docs (https://d3js.org https://d3js.org) and Observable Plot’s docs (https://observablehq.com/plot https://observablehq.com/plot) in VitePress, and absolutely loved the experience. But we wanted a tool focused on data apps, dashboards, reports — observability and business intelligence use cases rather than documentation. Compared to Evidence, I’d say we’re trying to target data app developers more than data analysts; we offer a lot of power and expressiveness, and emphasize custom visualizations and interaction (leaning on Observable Plot or D3), as well as polyglot programming with data loaders written in any language (Python, R, not just SQL).
  - amcaskill 3y ago
    One of the founders of Evidence here. Thanks the kind words Mike - that means a lot coming from you. I think that distinction is right -- we are focused on making a framework that is easy to use with a data analyst skill set, which generally means as little javascript as possible. As an example, the way you program client-side interactions in Evidence is by templating SQL which we run in duckDB web assembly, rather than by writing javascript. Evidence is also open source, for anyone who's interested. Repo: https://github.com/evidence-dev/evidence https://github.com/evidence-dev/evidence Previous discussions on HN: https://news.ycombinator.com/item?id=28304781 https://news.ycombinator.com/item?id=28304781 - 91 comments https://news.ycombinator.com/item?id=35645464 https://news.ycombinator.com/item?id=35645464 - 97 comments
    - anentropic 3y ago
      This looks very interesting to me, I'm building a BI reporting tool in my company at the moment, but browsing the docs I felt what I was missing was a clear overview of the architecture. e.g. you say above that Evidence takes templated SQL and runs it in DuckDB WASM and then in the docs there's various https://docs.evidence.dev/core-concepts/data-sources/#supported-data-sources https://docs.evidence.dev/core-concepts/data-sources/#suppor... like Snowflake, MySQL etc I guess I am wondering where and when the queries are happening If I set up a Snowflake data source is it doing a build-time import (like in the new Observable, from this thread) into DuckDB? or DuckDB is connecting to the sources via extensions? Where does the data live? My question is really just "how does it work?" and the "What is Evidence? > How does Evidence work?" section on the docs homepage doesn't really answer that at all, it's just a list of things that it does.
rogue7 3y ago
Interesting pivot from the Observable team. I loved observable and wrote a couple of notebooks, it worked great ! I'm gonna try Framework asap !
mbostock 3y ago
Hey, HN. We’re thrilled to release Observable Framework today — a new open-source tool for developing data apps. I highly recommend viewing this example report adapted from our internal dashboard analyzing web logs: https://observablehq.com/framework/examples/api/ https://observablehq.com/framework/examples/api/ This technique of “just plot everything” (7.6M requests as a scatterplot) has revealed surprising insights we’ve used to optimize our servers and better control traffic. We’re also sharing a more traditional dashboard that visualizes the adoption of our open-source visualization library (and in some ways the successor to D3), Observable Plot: https://observablehq.com/framework/examples/plot/ https://observablehq.com/framework/examples/plot/ In addition to releasing Observable Framework, we’ve also made Observable free again for individuals (including private notebooks and databases connectors). Let me know if you have any questions!
- ZeroCool2u 3y ago
  This seems nice and the plots look great, but I have a hard time imagining switching to Observable from Plotly since there doesn't seem to be a way to make any plots interactive. By which I mean Zoom and Pan. The nearest point high light feature is nice, but what if I want to zoom in? None of the examples here seem to be able to do that and quick google search doesn't make it seem like that's straight forward. That's not even additional code when I use Plotly, it's just built-in. There's also the issue of convincing staff to use JS instead of Python which is still just a tough sell. I think everyone on my team (of data scientists) would look at me like I've got two heads if I were to suggest that. Maybe we're not the target demographic though. I do like the idea of shipping the data straight to the client, but I don't have a lot of confidence in our corporate network doing well and not slowing stuff down. Perhaps the graphics all are sent pre-rendered over the wire though? I'm not sure, but would be cool if Observable figured out a way to side step that issue.
  - skybrian 3y ago
    Though it’s not designed for animation, Observable Plot is just a JavaScript library and it renders fast enough that you can do things like that just by re-rendering. Here are some old notebooks with experiments with audio data hooked up to UI controls: https://observablehq.com/collection/@skybrian/observable-plot-tricks https://observablehq.com/collection/@skybrian/observable-plo...
xrd 3y ago
I love Observable. And, this is a phenomenal approach, untethering Observable from observablehq.com. I'm so excited. It probably goes without saying that EVERYONE should have a blog, and this approach from Observable means journalists everywhere can now easily create a dynamic and information-driven blog. It isn't a coincidence that Observable came from a guy that did amazing data visualizations at the NYTimes. We are on the precipice of a major power shift back to journalists and away from dubious corporations, and tools like this enable that transition. (Shameless plug: Svekyll is going towards the same goal. Svekyll is a static site generator inspired by Jekyll. If you want to use Svelte in your blog, check it out. Svekyll bundles the incredible Apache ECharts so you can use echarts in your blog with a few lines of code (and no complicated build process). https://extrastatic.dev/svekyll/svekyll-cli https://extrastatic.dev/svekyll/svekyll-cli. These are ideas I've been thinking about too.)
mbostock 3y ago
Another tidbit buried in this announcement is that Observable Framework is 100% vanilla JavaScript syntax — so you get Observable’s reactive runtime without the quirky Observable JavaScript syntax (as in Observable notebooks). And you can use static ES imports from npm or local modules, declare multiple top-level variables in a code block (not just a single named variable per cell), call the built-in display(…) function to put things on the page, etc. It’s a huge relief to have vanilla syntax and greatly improves interoperability. And we’re figuring out how to port these improvements back to Observable notebooks in the near future.
- skybrian 3y ago
  With regard to code edits (rather than UI reactivity), this looks similar to how many web development environments watch the file system for changes and then rebuild and reload the page. Is there more to it? How are syntax errors reported? Is there support for TypeScript syntax and type-checking? Can a page partially run that has errors in some JavaScript snippets, like a notebook with errors in some cells? In the examples, there is a “view source” link that goes to GitHub. Understanding the code involves finding the Markdown file and then going back and forth between the published page and the Markdown file, which hopefully correspond to the same version. It seems like the thing that’s lost compared to notebooks is letting the user see and edit the code in the browser. But I suppose that’s a niche use case for coding tutorials. Not everything needs to be a notebook. Even so, better built-in “view source” support might be nice, even if it doesn’t allow editing. It doesn’t have to be as prominent as it is in a notebook to be useful.
  - mbostock 3y ago
    You can read about our reactive runtime here (it’s the same as Observable notebooks even though Framework uses vanilla JavaScript syntax): https://observablehq.com/@observablehq/how-observable-runs https://observablehq.com/@observablehq/how-observable-runs And the source is here: https://github.com/observablehq/runtime https://github.com/observablehq/runtime The “trick” is to structure all code as reactive variables (or nodes, defined as pure functions) within a dataflow graph. So if you replace one variable (by replacing a function with a new import), you then have to recompute any downstream variables that depend on the replaced variable, while cleaning up old variables and updating the display. Invalid syntax doesn’t prevent other code blocks from running (though if it means a variable is then undefined, that might cause downstream errors). Syntax errors are displayed in the page, and also in the console for the running preview server. We’d like to improve the error display in the console to show more context around where the error occurred, since unlike notebooks the code isn’t immediately adjacent to the output. We don’t support TypeScript yet, but there’s a PR (https://github.com/observablehq/framework/pull/129 https://github.com/observablehq/framework/pull/129) and we are interested in stronger validation at build time to catch more errors. And yes, we’re making different tradeoffs, optimizing for data apps and dashboards (more polished presentation) rather than ad hoc exploration in notebooks. So it’s more work to find and edit the code, but conversely it’s a more careful, deliberate process that allows code review, unit tests, continuous integration, etc. And we think that’s appropriate for data apps that are depended on by many people. But still, a view source link back to your source control would be nice, yes!
mrtimo 3y ago
It would be really cool if you guys supported Malloy. Maybe you already do? https://www.malloydata.dev/ https://www.malloydata.dev/
- mythmon_ 3y ago
  Observable engineer here. I haven't looked into Malloy much, but Framework's data loaders are very flexible. If you can write a script or binary that uses Malloy and writes to stdout, it can be a data loader. For example, although we use SQL a lot in our internal usage of Framework, Framework doesn't actually have any specific SQL support. We just use our database's normal bindings.
  - chrisjc 3y ago
    Can you link to the "data loader" API or perhaps even a "data loader" example implementation for something similar to malloy, duckdb, or any other DB/SQL data source/provider? Would love to see it, thanks in advance! edit: found it https://observablehq.com/framework/lib/duckdb https://observablehq.com/framework/lib/duckdb
    - mythmon_ 3y ago
      The docs for data loaders are here: https://observablehq.com/framework/loaders https://observablehq.com/framework/loaders. The simple version is they are simply programs that write their output to standard out. Very Unixey. When those programs are referenced in client side parts of the JS, they are reactively run when in development, and prebuilt for deployment. I don't think we have any full examples of using a database yet, but we have written a bit about using DuckDB via its Node bindings here: https://observablehq.com/framework/lib/duckdb https://observablehq.com/framework/lib/duckdb I imagine that either Malloy's CLI or its Python bindings would fit very well here.
ddanieltan 3y ago
I'm super excited to try this out! Couple of questions since I see @mbostock active in the comments. 1. Is the flexibility of languages used in data loaders/backend going to eventually come to the front end/ui? Or will the paradigm always be bring-your-own-language for the data loading but build your dashboard with observablejs/observable plot? 2. Considering ObservableJS is supported by Quarto, can we look forward to Observable Framework integrated with Quarto too? Or is the fact that the latest Quarto version also featured Dashboards more of a competitor to Framework? 3. Saw some comparison to Evidence.dev in the comments. I saw some shades of similarity with the markdown focused dev experience too but I recall Evidence chose Apache Echarts for their main charting library. Any thoughts of the pros/cons of Echarts vs ObservableJS/Plot?
- mbostock 3y ago
  1. We don’t have immediate plans to bring other languages to the front-end — maybe TypeScript, but that’s just stripping annotations; maybe some WebAssembly. Our idea is to have a clear serializable “membrane” separating your back-end (in any language, running on build on your servers) from your front-end (in JavaScript, running on load in the client). Data loaders produce data during build, which gets handed-off to the client to render. Trying to do data processing on the client is often a frustrating and poor user experience. Likewise trying to render great interactive charts without web technologies is quite limiting! 2. I can’t speak to Quarto’s plans. Observable Framework is open-source so they might pick up some of this stuff. I look at Framework more as an alternative to Quarto than a complement. 3. As the creator of Observable Plot (and D3 before that), I’m a huge fan of visualization grammars! Apache Echarts is a chart typology, and while it’s got a lot of chart types in it, it has no overarching conceptual model of how to represent a visualization. And so it’s not very interesting. But “the proof of the pudding is in the eating” as I say in the post, so I encourage you to look at Observable Plot and decide for yourself if you like both the syntax and the resulting plots. I certainly do! Leland Wilkinson said it best: “If we endeavor to develop a charting instead of a graphing program, we will accomplish two things. First, we inevitably will offer fewer charts than people want. Second, our package will have no deep structure. Our computer program will be unnecessarily complex, because we will fail to reuse objects or routines that function similarly in different charts. And we will have no way to add new charts to our system without generating complex new code. Elegant design requires us to think about a theory of graphics, not charts.”
nalgeon 3y ago
And if you want a simpler tool for creating interactive docs, maybe try Codapi: https://codapi.org/ https://codapi.org/
austinpena 3y ago
I'm curious where this fits in relative to streamlit which I use heavily
jwilber 3y ago
This is really a game changer for creating data apps. I loved observable, but convincing scientists with no js knowledge to try the platform was almost impossible. Markdown + language agnostic loaders seems like the perfect way to collaborate. Are there any plans to allow existing observable notebooks to be deployed? For example, a one-click “deploy” button?
- mbostock 3y ago
  The Observable Framework CLI supports a `convert` command for downloading an Observable notebook and converting it to Markdown. E.g., `observable convert @d3/bar-chart` will download the notebook and save a bar-chart.md and alphabet.csv to your current working directory. (We did it from the command line so it’s easy for you to automate or batch-convert or refresh and keep things in sync.) You may have to make some tweaks to the code due to Framework’s vanilla JavaScript syntax, but we’ll work on making that more seamless over time. And you can `observable deploy` to deploy your app to Observable for sharing — though most often you’ll want to setup continuous deployment to keep your app up-to-date automatically.
beefman 3y ago
No mention of math formatting but from the docs it looks like there is TeX support! https://observablehq.com/framework/lib/tex https://observablehq.com/framework/lib/tex
- mbostock 3y ago
  And Graphviz (dot) and Mermaid, too!
RobinL 3y ago
Observable is such an incredible, powerful and enjoyable tool. I use it heavily, including to power my blog. I love it, but I've always had a slight concern about needing to rely on the Observable notebook website. So this has really made my day. To give a sense of the kind of performant, statically hosted interactive content that has only really been within my reach since using Observable, here are some examples: Highly interactive vis: https://www.robinlinacre.com/visualising_fellegi_sunter/ https://www.robinlinacre.com/visualising_fellegi_sunter/ Editable computations that flow through the document https://www.robinlinacre.com/computing_fellegi_sunter/ https://www.robinlinacre.com/computing_fellegi_sunter/ Multiple synced representations of data: https://www.robinlinacre.com/prob_bf_mw/ https://www.robinlinacre.com/prob_bf_mw/ https://www.robinlinacre.com/partial_match_weights/ https://www.robinlinacre.com/partial_match_weights/ (half way down, under 'Understanding the partial match weight chart and waterfall chart) Of course, there are a huge number of additional examples: https://observablehq.com/trending https://observablehq.com/trending but i think the whole thing makes much more sense to the end-user when embedded (sadly at which point they don't even know it's observable!)
lloydatkinson 3y ago
There are simply too many tools and sites called Observable
greenie_beans 3y ago
this is exciting. love observable.
- deleted 3y ago
  [deleted]
simonw 3y ago
There's an almost bewildering amount of interesting ideas buried in this. Things like data loaders which are ANY script that can output data (as JSON or something else) to standard output. Markdown files with ```js blocks in that get executed. The reinvention of the core Observable notebook to avoid custom syntax. This is really big.
- tophtucker 3y ago
  Yeah data loaders are like a UNIX pipe to a reactive notebook cell (?). There needn’t be any question of “do the data loaders support this or that”; it doesn’t even have a concept of “supporting” beyond supporting stdout… Still thinking through how to understand it myself!!
  - skybrian 3y ago
    Data loaders seem like an interesting way to define a multi-language build system without having to write a makefile. Lots of build systems do this, but the boundaries between build steps often isn't as clean and uniform as having a single output per build step and relying on a file naming convention. It's not truly reactive if you have to do a build to make anything happen. But maybe that doesn't matter, as long as it's reactive during development?
    - tophtucker 3y ago
      Yeah there are some open issues about more granular rebuilds and chaining data loaders. https://github.com/observablehq/framework/issues/638 https://github.com/observablehq/framework/issues/638, https://github.com/observablehq/framework/issues/332 https://github.com/observablehq/framework/issues/332 It’s kinda cool to think about the shearing layers of reactivity. Reactivity is what originally drew me to Observable. But the way notebooks have to be recomputed live for every viewer every time makes them feel silly for, like, a BI dashboard that changes daily at most. Like they only have one pace layer. Like they’re trying so hard to be _live_ that they can’t be _fast_! Idk. I guess even a chalkboard is reactive on the timescale of “someone noticing some information and telling it to someone who writes it down” lol.
kepano 3y ago
I appreciate the nod to "File over app"[1] in the announcement. It's so cool that a Markdown file with code blocks can be the source for complex data visualizations and dashboards. Interoperability of this kind makes me giddy. I played around with editing an Observable site from Obsidian and it works great[2]. [1]: https://stephango.com/file-over-app https://stephango.com/file-over-app [2]: https://twitter.com/kepano/status/1758202572446581025 https://twitter.com/kepano/status/1758202572446581025
- tophtucker 3y ago
  We’ve talked about and shared your “File over app” manifesto so many times internally over the last few months. It’s one of those tweets that gets immortalized as the perfect crystallization of an ethos that we might otherwise have only been able to gesture at vaguely. It gives the ethos weight and clarity and credibility, and it’s such a relief to be able to point to it! I’m very grateful. —an Observable employee
  - deleted 3y ago
    [deleted]
- dleeftink 3y ago
  My mind immediately went to how these Dasboards could be integrated in Obsidian, and seeing the `import` dependency graph reflected in Obsidian's graph view.
CJefferson 3y ago
Can anyone with knowledge of both systems compare this to quarto for me?
- allisonhorst 3y ago
  Hi there! I'm Allison, I've spent lots of time working in R, R Markdown, and more recently Quarto for environmental data science work & teaching. Working in Quarto is actually what introduced me to Observable ("what is this 'OJS' business?")! Now I'm a Developer Marketing Manager at Observable and am really enjoying working in Observable Framework. I still love Quarto. It lets me tinker, explore & troubleshoot in R easily while building things (I still have a bit of a hard time troubleshooting data loaders). I like that I can use Quarto for dashboards, but also for a personal blog, a scientific article, a nicely formatted PDF, etc. -- it feels like I only need to know one system to be able to create a bunch of different products. There are options for people like me who are more data scientist, less developer to quickly customize some biggies like fonts, background colors, etc. right in the yaml. There are really nice helpers (e.g. for cross referencing and citations, among others) that are great for researchers. What I like about Observable Framework is that getting started is so fast - minutes from install, to a dashboard I can preview & update locally, to deploy - with really clear prompts to walk me through it. I do really like data loaders. It feels better to just prep my data in an R script (or Python, or whatever else), then access the data I need from the loader (rather than to pass outputs from R code to an OJS cell in Quarto). Using any JS libraries and components feels more streamlined in Observable Framework (not sure why - the syntax isn't that different, though Framework uses import rather than require now). And dashboard layouts are easier for me in Framework using our grid class. I like how focused Observable Framework feels. Quarto feels awesome with is breadth of possibilities (output to PDF, or Word, or slides, or blogs, or ebooks, or websites, or ...!), whereas Framework feels sleek in its focus and design for developers creating beautiful, fast data apps & dashboards.
janice1999 3y ago
Are there any similar projects that allow you to plot thousands or tens of thousands of datapoints and also allows the viewer zoom into time series plots? I'd love to have an in-browser matplotlib replacement. So far I haven't found one. Observable plots look static.
rad_gruchalski 3y ago
Getting started guide: https://observablehq.com/framework/getting-started https://observablehq.com/framework/getting-started. Looks very nice!
nrjames 3y ago
I love this and hope that a django-observable package comes along that makes it very easy to integrate and serve these static apps through a larger Django site.
j-pb 3y ago
If you play the history of Observable backwards you start with a company creating a static site generator for dashboards, which then struggles to find a market fit as it tries to bring datascience to middle management, to finally reach a focused, simple and elegant tool for exploratory programming, data visualisation, and interactive documentation in javascript.
- nextworddev 3y ago
  Ok but is anyone paying for it?
  - j-pb 3y ago
    We used to pay for 4 licenses, until they switched to the weird pricing schemes and the new editor targeted towards people with no programming experience. I really wish they would open source Observable 0.5, Pluto is currently the only other notebook left that has the flexible data-flow model at reasonable simplicity.
omneity 3y ago
This is super cool! I’ve been looking for ways to integrate Observable with my blog posts and make them more interactive and engaging. This might just be it. Thank you and congrats on the release!
kuatroka 3y ago
A couple of questions: 1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't it be huge and how long will it take to generates this static data or is there an actual call back to the server to the duckdb file (I assume it works with .duckdb file too?) 2. If Observable Framework provides the front-end, does it mean I can use any auth library if I want to create a web site with a log in and subscription options? 3. If it's a static web page, does it mean that at any time a user views a chart, they will also be able to go to the Dev Tools and download the file with data that's behind the viz? 4. When (if you can share of course) is the planned release of Plot's interactions: zoom, pan, interactive legend, brush? 5. Deployment - with big parquet, sqlite, csv files, it's impossible to do CI/CO through github or vercel and such. Will your hosting services offer an option to host those files and runtimes to generate them? Thanks
- chrisjc 3y ago
  Came here with similar questions and Cmd-F "DuckDB". See the comment about "data loaders". Seems like a "data loader" would provide most of what you're asking about. I'm also thinking that a "data loader" combined with duckdb-wasm and arrow would be a pretty nice combination. I imagine that it might not be too difficult to switch two between two implementations of the "data loader" as needed. Switch between reading from a remote system (in your case DuckDB on a server) and DuckDB running locally in the browser (that can interact with its own remote or local data sources). edit: welp https://observablehq.com/framework/lib/duckdb https://observablehq.com/framework/lib/duckdb
  - recifs 3y ago
    See the example at https://huggingface.co/spaces/observablehq/fpdn https://huggingface.co/spaces/observablehq/fpdn where DuckDB is used both as a data loader (to download and digest 200GB worth of source data into a small 8MB parquet file) and on the client-side to allow the user to do live search queries on the minimized data. Server-side, we're using duckdb-the-binary, and client-side we're using duckdb-wasm.
johnmorrison 3y ago
This is very cool to see! We've been building something very similar (in some regards, very different in others) in https://rysana.com/bundown https://rysana.com/bundown Somehow ideas like this emerge in waves seemingly with no coordination, kind of like what happened when Calculus was first invented (Of course there's a lot of prior art here but it's interesting to see a specific jump towards polyglot single-file Markdown 'apps' and so on happen around the same time)
coolca 3y ago
It is so beautiful
77ko 3y ago
This looks amazing! I really like the clear seperation of loading/prepping data, and presenting it. Some requests: Add simple examples and more clarity to the publish docs. I assume most ppl would prefer to deploy via github actions[1], which the docs currently just link to a complex deploy file - can you please add some more documentation on this, or add an example of the simplest possible deploy file? Suggestion: is it possible to use a interface (like in vercel) to connect to a github repo and build/publish on changes? [1]: https://observablehq.com/framework/getting-started#deploying-via-github-actions https://observablehq.com/framework/getting-started#deploying...
one_buggy_boi 3y ago
What would be a good design pattern to put these dashboards behind auth? I suppose since they're static files you could just serve them with something like FastAPI or Spring Boot and have your CI/CD refresh the static files throughout the day on shared storage?
- a-ve 3y ago
  If you're putting these behind a reverse proxy (nginx, etc.) you can just setup client certificate authentication by using your own locally generated CA or by using something like Vault for UI-based certificate generation. When you visit this site with a certificate installed on your device, it will authenticate successfully, and for those who do not have a correct certificate installed, a "No certificate presented" error will be shown. It's fairly easy to setup and there are multiple guides available for it. Here's one: https://fardog.io/blog/2017/12/30/client-side-certificate-authentication-with-nginx/ https://fardog.io/blog/2017/12/30/client-side-certificate-au...
- mbostock 3y ago
  For one, you can deploy them to Observable (with `observable deploy`) and we’ll provide access control.
- apitman 3y ago
  I'd recommend a reverse proxy and login server using "forward auth". I made a list of such login servers here: https://github.com/lastlogin-io/obligator?tab=readme-ov-file#comparison-is-the-thief-of-joy https://github.com/lastlogin-io/obligator?tab=readme-ov-file...
farhanhubble 3y ago
I love how much is possible with the Observable framework and support for libraries like d3.js. However many data apps cannot precompute their outputs. For example, a pipeline that extracts text from documents based on what a user queries, cannot precompute the results and any visualizations must be updated every time. The best hack to accomplish this seems to be rebuilding the app on each update. Or is there another solution?
- recifs 3y ago
  The code in a Framework can do whatever you want it to do—it can load data on demand, call an external API, etc. Precomputing data is only an option, not an obligation. But even when you want things to be very interactive it is a good idea to minimize the data. Expose only the "rows and columns" that you need, and compress it as much as possible. This can be done in a data loader. For example, see the data app we deployed yesterday on hugging-face: its data loaders ingest a large source database (320 files totaling 200GB), and digests it into a single 8MB parquet file that we can then use on the page to "live query" 3 million newspaper titles and dates. https://huggingface.co/spaces/observablehq/fpdn https://huggingface.co/spaces/observablehq/fpdn
- RyanHamilton 3y ago
  I took the query only what you need approach with PulseUI: https://www.timestored.com/pulse/ https://www.timestored.com/pulse/ It emphasises not sending all data to the client. For real-time and very large data sets this is the only option.
tetris11 3y ago
I miss source loading D3 into a simple HTML page, and having D3 tutorials that were up to spec with the latest D3 release. Yeah Observable data pages look cool, but it really feels like excessive JS bloat for the features. I think I miss throwing a D3 viz together without having to load an entire framework library.
- meowtimemania 3y ago
  @observablehq/plot is what you're looking for if you want to have the same charts loaded via a script tag. https://observablehq.com/plot/getting-started#plot-in-vanilla-html https://observablehq.com/plot/getting-started#plot-in-vanill...
- recifs 3y ago
  You can still use D3 the good old-fashioned way, as a standalone script. See https://d3js.org/getting-started#d3-in-vanilla-html https://d3js.org/getting-started#d3-in-vanilla-html for examples.
cheptsov 3y ago
Congrats on the launch! Excited to hear that you go fully open-source with this! There is a certain need for great visualization tools that enable building apps.
dleeftink 3y ago
I've setup a codespace starter template that pulls in the required dependencies to start building dashboards with Observable Framework, right inside your browser. You can access the starter template here: [0]: https://github.com/dleeftink/observable-codespace https://github.com/dleeftink/observable-codespace
kuatroka 3y ago
Are you planning to add new UI components like Data Table or other in the future or it’s purely Plotting and Data ingestion and the UI through Tailwind or CSS or would it be possible to add UI libraries like shadcn or DaisyUI to make it a full fledged web site? Thanks
- recifs 3y ago
  It’s only the first public release, rest assured we have plans to develop it beyond that point :-) Data tables are high on the list, but it’s going to be a lot of work and I can’t say when we’ll have something to share. In the meantime almost any library available on npm should work out of the box—not just the ones that we added explicit support for (even though of course some might need more work than others).
vwkd 3y ago
How does this compare to a traditional reactive framework like React, Vue, or Svelte with Observable Plot?
petre 3y ago
ObservableHQ is great but for me this crashed on install. Unfortunately I'm not aware of any npm mechanism to report errors back to the developers. $ npm init @observablehq node:internal/errors:477 ErrorCaptureStackTrace(err);

655 points by tmcw 3y ago

fburnaby 3y ago

I can't see from the docs what this gives me over a Makefile, Asciidoctor (or pandoc, Jekyll etc), and D3?

terpimost 3y ago

Wow guys. That is so cool! Thank you for what you do. This is a great example of high quality product!

foolswisdom 3y ago

I really like this idea.

jeffbee 3y ago

I am totally psyched up to try this. Observable (1.0) has been a very effective outlet for my research. There is no other platform that could have hosted it and offered me the quick and easy tools to make persuasive visualizations. I was a little concerned when I heard some people got laid off from the company, but it seems like it is still going.

yodon 3y ago

Lots of good stuff here, but are you really naming your framework "Framework" (as in "With Framework, you can build...")? You do realize that naming only makes sense inside your own company, right? To everyone who uses it, it's "a" framework or "Observable's framework". No consumer of the framework is going to refer to it as "Framework" without ridiculous amounts of confusion resulting.

renewiltord 3y ago
It's called Observable Framework one sentence before and on the product page. It's normal to write like this in a blog post. For instance, the Kubernetes deployments page uses the word "Deployment" doesn't say Kubernetes Deployment everywhere. It just says Deployment. I think introducing it like that is fine. You don't have to say Observable® Framework™ every sentence. That would be strange.
prakis 3y ago
Yes, it is confusing. It took me sometime to realize their product name itself is framework.
alexgarcia-xyz 3y ago
Observable has other open source libraries with similar "generic" names: Plot, Runtime, Inputs. When speaking generically, most people say "Observable Plot" or "Observable Runtime." In projects where people already know about them, then I say "Plot" or "Inputs" without much fuss. I imagine most people will say "Observable Framework" when talking out in the open, and "Framework" on established projects.

0cf8612b2e1e 3y ago

Sounds a bit like the “baked data” pattern. Which I think is a really good idea. I have long been toying with how to use Datasette to make a deliverable dashboard, so this is interesting.

rmnclmnt 3y ago
Me too, and that lead to developing the « datasette-dashboards » plugin[0]. I use this for my company where all the data is gathered by connectors scheduled in CI, storing data in Git, and triggering a SQLite db build and Datasette deployment. « BI as Code » if you will [0] https://github.com/rclement/datasette-dashboards https://github.com/rclement/datasette-dashboards
- pzmarzly 3y ago
  How often do you refresh the data (how often your CI runs)?
  - rmnclmnt 3y ago
    Depends on the nature of the data and the requirements, can be every hour or once a day. Most people and organizations have small/mid data and most of all slow data: you do not need fresh data every hour. If you do, you need a way heavier real time analytics data
mbostock 3y ago
Yes! It first felt counterintuitive and constraining to prepare data ahead of time, rather than just loading whatever you want on the fly. But we’ve found this to be a great discipline in practice because it forces you to think about what data you actually need to show, and to shift as much of the compute to build time to reduce file sizes (through aggregation and filtering) and accelerate page load. And once you become accustomed to instant dashboards it becomes unthinkable to go back to slow queries.
- shiandow 3y ago

lf-non 3y ago

The new direction seems very similar to what evidence has been doing for a while https://evidence.dev https://evidence.dev

mbostock 3y ago
Yep, Evidence is doing good work. We were most directly inspired by VitePress; we spent months rewriting both D3’s docs (https://d3js.org https://d3js.org) and Observable Plot’s docs (https://observablehq.com/plot https://observablehq.com/plot) in VitePress, and absolutely loved the experience. But we wanted a tool focused on data apps, dashboards, reports — observability and business intelligence use cases rather than documentation. Compared to Evidence, I’d say we’re trying to target data app developers more than data analysts; we offer a lot of power and expressiveness, and emphasize custom visualizations and interaction (leaning on Observable Plot or D3), as well as polyglot programming with data loaders written in any language (Python, R, not just SQL).
- amcaskill 3y ago
  One of the founders of Evidence here. Thanks the kind words Mike - that means a lot coming from you. I think that distinction is right -- we are focused on making a framework that is easy to use with a data analyst skill set, which generally means as little javascript as possible. As an example, the way you program client-side interactions in Evidence is by templating SQL which we run in duckDB web assembly, rather than by writing javascript. Evidence is also open source, for anyone who's interested. Repo: https://github.com/evidence-dev/evidence https://github.com/evidence-dev/evidence Previous discussions on HN: https://news.ycombinator.com/item?id=28304781 https://news.ycombinator.com/item?id=28304781 - 91 comments https://news.ycombinator.com/item?id=35645464 https://news.ycombinator.com/item?id=35645464 - 97 comments
  - anentropic 3y ago
    This looks very interesting to me, I'm building a BI reporting tool in my company at the moment, but browsing the docs I felt what I was missing was a clear overview of the architecture. e.g. you say above that Evidence takes templated SQL and runs it in DuckDB WASM and then in the docs there's various https://docs.evidence.dev/core-concepts/data-sources/#supported-data-sources https://docs.evidence.dev/core-concepts/data-sources/#suppor... like Snowflake, MySQL etc I guess I am wondering where and when the queries are happening If I set up a Snowflake data source is it doing a build-time import (like in the new Observable, from this thread) into DuckDB? or DuckDB is connecting to the sources via extensions? Where does the data live? My question is really just "how does it work?" and the "What is Evidence? > How does Evidence work?" section on the docs homepage doesn't really answer that at all, it's just a list of things that it does.

rogue7 3y ago

Interesting pivot from the Observable team. I loved observable and wrote a couple of notebooks, it worked great ! I'm gonna try Framework asap !

mbostock 3y ago

Hey, HN. We’re thrilled to release Observable Framework today — a new open-source tool for developing data apps. I highly recommend viewing this example report adapted from our internal dashboard analyzing web logs: https://observablehq.com/framework/examples/api/ https://observablehq.com/framework/examples/api/ This technique of “just plot everything” (7.6M requests as a scatterplot) has revealed surprising insights we’ve used to optimize our servers and better control traffic. We’re also sharing a more traditional dashboard that visualizes the adoption of our open-source visualization library (and in some ways the successor to D3), Observable Plot: https://observablehq.com/framework/examples/plot/ https://observablehq.com/framework/examples/plot/ In addition to releasing Observable Framework, we’ve also made Observable free again for individuals (including private notebooks and databases connectors). Let me know if you have any questions!

ZeroCool2u 3y ago
This seems nice and the plots look great, but I have a hard time imagining switching to Observable from Plotly since there doesn't seem to be a way to make any plots interactive. By which I mean Zoom and Pan. The nearest point high light feature is nice, but what if I want to zoom in? None of the examples here seem to be able to do that and quick google search doesn't make it seem like that's straight forward. That's not even additional code when I use Plotly, it's just built-in. There's also the issue of convincing staff to use JS instead of Python which is still just a tough sell. I think everyone on my team (of data scientists) would look at me like I've got two heads if I were to suggest that. Maybe we're not the target demographic though. I do like the idea of shipping the data straight to the client, but I don't have a lot of confidence in our corporate network doing well and not slowing stuff down. Perhaps the graphics all are sent pre-rendered over the wire though? I'm not sure, but would be cool if Observable figured out a way to side step that issue.
- skybrian 3y ago
  Though it’s not designed for animation, Observable Plot is just a JavaScript library and it renders fast enough that you can do things like that just by re-rendering. Here are some old notebooks with experiments with audio data hooked up to UI controls: https://observablehq.com/collection/@skybrian/observable-plot-tricks https://observablehq.com/collection/@skybrian/observable-plo...

xrd 3y ago

I love Observable. And, this is a phenomenal approach, untethering Observable from observablehq.com. I'm so excited. It probably goes without saying that EVERYONE should have a blog, and this approach from Observable means journalists everywhere can now easily create a dynamic and information-driven blog. It isn't a coincidence that Observable came from a guy that did amazing data visualizations at the NYTimes. We are on the precipice of a major power shift back to journalists and away from dubious corporations, and tools like this enable that transition. (Shameless plug: Svekyll is going towards the same goal. Svekyll is a static site generator inspired by Jekyll. If you want to use Svelte in your blog, check it out. Svekyll bundles the incredible Apache ECharts so you can use echarts in your blog with a few lines of code (and no complicated build process). https://extrastatic.dev/svekyll/svekyll-cli https://extrastatic.dev/svekyll/svekyll-cli. These are ideas I've been thinking about too.)

mbostock 3y ago

Another tidbit buried in this announcement is that Observable Framework is 100% vanilla JavaScript syntax — so you get Observable’s reactive runtime without the quirky Observable JavaScript syntax (as in Observable notebooks). And you can use static ES imports from npm or local modules, declare multiple top-level variables in a code block (not just a single named variable per cell), call the built-in display(…) function to put things on the page, etc. It’s a huge relief to have vanilla syntax and greatly improves interoperability. And we’re figuring out how to port these improvements back to Observable notebooks in the near future.

skybrian 3y ago
With regard to code edits (rather than UI reactivity), this looks similar to how many web development environments watch the file system for changes and then rebuild and reload the page. Is there more to it? How are syntax errors reported? Is there support for TypeScript syntax and type-checking? Can a page partially run that has errors in some JavaScript snippets, like a notebook with errors in some cells? In the examples, there is a “view source” link that goes to GitHub. Understanding the code involves finding the Markdown file and then going back and forth between the published page and the Markdown file, which hopefully correspond to the same version. It seems like the thing that’s lost compared to notebooks is letting the user see and edit the code in the browser. But I suppose that’s a niche use case for coding tutorials. Not everything needs to be a notebook. Even so, better built-in “view source” support might be nice, even if it doesn’t allow editing. It doesn’t have to be as prominent as it is in a notebook to be useful.
- mbostock 3y ago
  You can read about our reactive runtime here (it’s the same as Observable notebooks even though Framework uses vanilla JavaScript syntax): https://observablehq.com/@observablehq/how-observable-runs https://observablehq.com/@observablehq/how-observable-runs And the source is here: https://github.com/observablehq/runtime https://github.com/observablehq/runtime The “trick” is to structure all code as reactive variables (or nodes, defined as pure functions) within a dataflow graph. So if you replace one variable (by replacing a function with a new import), you then have to recompute any downstream variables that depend on the replaced variable, while cleaning up old variables and updating the display. Invalid syntax doesn’t prevent other code blocks from running (though if it means a variable is then undefined, that might cause downstream errors). Syntax errors are displayed in the page, and also in the console for the running preview server. We’d like to improve the error display in the console to show more context around where the error occurred, since unlike notebooks the code isn’t immediately adjacent to the output. We don’t support TypeScript yet, but there’s a PR (https://github.com/observablehq/framework/pull/129 https://github.com/observablehq/framework/pull/129) and we are interested in stronger validation at build time to catch more errors. And yes, we’re making different tradeoffs, optimizing for data apps and dashboards (more polished presentation) rather than ad hoc exploration in notebooks. So it’s more work to find and edit the code, but conversely it’s a more careful, deliberate process that allows code review, unit tests, continuous integration, etc. And we think that’s appropriate for data apps that are depended on by many people. But still, a view source link back to your source control would be nice, yes!

mrtimo 3y ago

It would be really cool if you guys supported Malloy. Maybe you already do? https://www.malloydata.dev/ https://www.malloydata.dev/

mythmon_ 3y ago
Observable engineer here. I haven't looked into Malloy much, but Framework's data loaders are very flexible. If you can write a script or binary that uses Malloy and writes to stdout, it can be a data loader. For example, although we use SQL a lot in our internal usage of Framework, Framework doesn't actually have any specific SQL support. We just use our database's normal bindings.
- chrisjc 3y ago
  Can you link to the "data loader" API or perhaps even a "data loader" example implementation for something similar to malloy, duckdb, or any other DB/SQL data source/provider? Would love to see it, thanks in advance! edit: found it https://observablehq.com/framework/lib/duckdb https://observablehq.com/framework/lib/duckdb
  - mythmon_ 3y ago
    The docs for data loaders are here: https://observablehq.com/framework/loaders https://observablehq.com/framework/loaders. The simple version is they are simply programs that write their output to standard out. Very Unixey. When those programs are referenced in client side parts of the JS, they are reactively run when in development, and prebuilt for deployment. I don't think we have any full examples of using a database yet, but we have written a bit about using DuckDB via its Node bindings here: https://observablehq.com/framework/lib/duckdb https://observablehq.com/framework/lib/duckdb I imagine that either Malloy's CLI or its Python bindings would fit very well here.

ddanieltan 3y ago

I'm super excited to try this out! Couple of questions since I see @mbostock active in the comments. 1. Is the flexibility of languages used in data loaders/backend going to eventually come to the front end/ui? Or will the paradigm always be bring-your-own-language for the data loading but build your dashboard with observablejs/observable plot? 2. Considering ObservableJS is supported by Quarto, can we look forward to Observable Framework integrated with Quarto too? Or is the fact that the latest Quarto version also featured Dashboards more of a competitor to Framework? 3. Saw some comparison to Evidence.dev in the comments. I saw some shades of similarity with the markdown focused dev experience too but I recall Evidence chose Apache Echarts for their main charting library. Any thoughts of the pros/cons of Echarts vs ObservableJS/Plot?

mbostock 3y ago
1. We don’t have immediate plans to bring other languages to the front-end — maybe TypeScript, but that’s just stripping annotations; maybe some WebAssembly. Our idea is to have a clear serializable “membrane” separating your back-end (in any language, running on build on your servers) from your front-end (in JavaScript, running on load in the client). Data loaders produce data during build, which gets handed-off to the client to render. Trying to do data processing on the client is often a frustrating and poor user experience. Likewise trying to render great interactive charts without web technologies is quite limiting! 2. I can’t speak to Quarto’s plans. Observable Framework is open-source so they might pick up some of this stuff. I look at Framework more as an alternative to Quarto than a complement. 3. As the creator of Observable Plot (and D3 before that), I’m a huge fan of visualization grammars! Apache Echarts is a chart typology, and while it’s got a lot of chart types in it, it has no overarching conceptual model of how to represent a visualization. And so it’s not very interesting. But “the proof of the pudding is in the eating” as I say in the post, so I encourage you to look at Observable Plot and decide for yourself if you like both the syntax and the resulting plots. I certainly do! Leland Wilkinson said it best: “If we endeavor to develop a charting instead of a graphing program, we will accomplish two things. First, we inevitably will offer fewer charts than people want. Second, our package will have no deep structure. Our computer program will be unnecessarily complex, because we will fail to reuse objects or routines that function similarly in different charts. And we will have no way to add new charts to our system without generating complex new code. Elegant design requires us to think about a theory of graphics, not charts.”

nalgeon 3y ago

And if you want a simpler tool for creating interactive docs, maybe try Codapi: https://codapi.org/ https://codapi.org/

austinpena 3y ago

I'm curious where this fits in relative to streamlit which I use heavily

jwilber 3y ago

This is really a game changer for creating data apps. I loved observable, but convincing scientists with no js knowledge to try the platform was almost impossible. Markdown + language agnostic loaders seems like the perfect way to collaborate. Are there any plans to allow existing observable notebooks to be deployed? For example, a one-click “deploy” button?

mbostock 3y ago
The Observable Framework CLI supports a `convert` command for downloading an Observable notebook and converting it to Markdown. E.g., `observable convert @d3/bar-chart` will download the notebook and save a bar-chart.md and alphabet.csv to your current working directory. (We did it from the command line so it’s easy for you to automate or batch-convert or refresh and keep things in sync.) You may have to make some tweaks to the code due to Framework’s vanilla JavaScript syntax, but we’ll work on making that more seamless over time. And you can `observable deploy` to deploy your app to Observable for sharing — though most often you’ll want to setup continuous deployment to keep your app up-to-date automatically.

beefman 3y ago

No mention of math formatting but from the docs it looks like there is TeX support! https://observablehq.com/framework/lib/tex https://observablehq.com/framework/lib/tex

mbostock 3y ago
And Graphviz (dot) and Mermaid, too!

RobinL 3y ago

Observable is such an incredible, powerful and enjoyable tool. I use it heavily, including to power my blog. I love it, but I've always had a slight concern about needing to rely on the Observable notebook website. So this has really made my day. To give a sense of the kind of performant, statically hosted interactive content that has only really been within my reach since using Observable, here are some examples: Highly interactive vis: https://www.robinlinacre.com/visualising_fellegi_sunter/ https://www.robinlinacre.com/visualising_fellegi_sunter/ Editable computations that flow through the document https://www.robinlinacre.com/computing_fellegi_sunter/ https://www.robinlinacre.com/computing_fellegi_sunter/ Multiple synced representations of data: https://www.robinlinacre.com/prob_bf_mw/ https://www.robinlinacre.com/prob_bf_mw/ https://www.robinlinacre.com/partial_match_weights/ https://www.robinlinacre.com/partial_match_weights/ (half way down, under 'Understanding the partial match weight chart and waterfall chart) Of course, there are a huge number of additional examples: https://observablehq.com/trending https://observablehq.com/trending but i think the whole thing makes much more sense to the end-user when embedded (sadly at which point they don't even know it's observable!)

lloydatkinson 3y ago

There are simply too many tools and sites called Observable

greenie_beans 3y ago

this is exciting. love observable.

deleted 3y ago
[deleted]

simonw 3y ago

There's an almost bewildering amount of interesting ideas buried in this. Things like data loaders which are ANY script that can output data (as JSON or something else) to standard output. Markdown files with ```js blocks in that get executed. The reinvention of the core Observable notebook to avoid custom syntax. This is really big.

tophtucker 3y ago
Yeah data loaders are like a UNIX pipe to a reactive notebook cell (?). There needn’t be any question of “do the data loaders support this or that”; it doesn’t even have a concept of “supporting” beyond supporting stdout… Still thinking through how to understand it myself!!
- skybrian 3y ago
  Data loaders seem like an interesting way to define a multi-language build system without having to write a makefile. Lots of build systems do this, but the boundaries between build steps often isn't as clean and uniform as having a single output per build step and relying on a file naming convention. It's not truly reactive if you have to do a build to make anything happen. But maybe that doesn't matter, as long as it's reactive during development?
  - tophtucker 3y ago
    Yeah there are some open issues about more granular rebuilds and chaining data loaders. https://github.com/observablehq/framework/issues/638 https://github.com/observablehq/framework/issues/638, https://github.com/observablehq/framework/issues/332 https://github.com/observablehq/framework/issues/332 It’s kinda cool to think about the shearing layers of reactivity. Reactivity is what originally drew me to Observable. But the way notebooks have to be recomputed live for every viewer every time makes them feel silly for, like, a BI dashboard that changes daily at most. Like they only have one pace layer. Like they’re trying so hard to be _live_ that they can’t be _fast_! Idk. I guess even a chalkboard is reactive on the timescale of “someone noticing some information and telling it to someone who writes it down” lol.

kepano 3y ago

I appreciate the nod to "File over app"[1] in the announcement. It's so cool that a Markdown file with code blocks can be the source for complex data visualizations and dashboards. Interoperability of this kind makes me giddy. I played around with editing an Observable site from Obsidian and it works great[2]. [1]: https://stephango.com/file-over-app https://stephango.com/file-over-app [2]: https://twitter.com/kepano/status/1758202572446581025 https://twitter.com/kepano/status/1758202572446581025

tophtucker 3y ago
We’ve talked about and shared your “File over app” manifesto so many times internally over the last few months. It’s one of those tweets that gets immortalized as the perfect crystallization of an ethos that we might otherwise have only been able to gesture at vaguely. It gives the ethos weight and clarity and credibility, and it’s such a relief to be able to point to it! I’m very grateful. —an Observable employee
- deleted 3y ago
  [deleted]
dleeftink 3y ago
My mind immediately went to how these Dasboards could be integrated in Obsidian, and seeing the `import` dependency graph reflected in Obsidian's graph view.

CJefferson 3y ago

Can anyone with knowledge of both systems compare this to quarto for me?

allisonhorst 3y ago
Hi there! I'm Allison, I've spent lots of time working in R, R Markdown, and more recently Quarto for environmental data science work & teaching. Working in Quarto is actually what introduced me to Observable ("what is this 'OJS' business?")! Now I'm a Developer Marketing Manager at Observable and am really enjoying working in Observable Framework. I still love Quarto. It lets me tinker, explore & troubleshoot in R easily while building things (I still have a bit of a hard time troubleshooting data loaders). I like that I can use Quarto for dashboards, but also for a personal blog, a scientific article, a nicely formatted PDF, etc. -- it feels like I only need to know one system to be able to create a bunch of different products. There are options for people like me who are more data scientist, less developer to quickly customize some biggies like fonts, background colors, etc. right in the yaml. There are really nice helpers (e.g. for cross referencing and citations, among others) that are great for researchers. What I like about Observable Framework is that getting started is so fast - minutes from install, to a dashboard I can preview & update locally, to deploy - with really clear prompts to walk me through it. I do really like data loaders. It feels better to just prep my data in an R script (or Python, or whatever else), then access the data I need from the loader (rather than to pass outputs from R code to an OJS cell in Quarto). Using any JS libraries and components feels more streamlined in Observable Framework (not sure why - the syntax isn't that different, though Framework uses import rather than require now). And dashboard layouts are easier for me in Framework using our grid class. I like how focused Observable Framework feels. Quarto feels awesome with is breadth of possibilities (output to PDF, or Word, or slides, or blogs, or ebooks, or websites, or ...!), whereas Framework feels sleek in its focus and design for developers creating beautiful, fast data apps & dashboards.

janice1999 3y ago

Are there any similar projects that allow you to plot thousands or tens of thousands of datapoints and also allows the viewer zoom into time series plots? I'd love to have an in-browser matplotlib replacement. So far I haven't found one. Observable plots look static.

rad_gruchalski 3y ago

Getting started guide: https://observablehq.com/framework/getting-started https://observablehq.com/framework/getting-started. Looks very nice!

nrjames 3y ago

I love this and hope that a django-observable package comes along that makes it very easy to integrate and serve these static apps through a larger Django site.

j-pb 3y ago

If you play the history of Observable backwards you start with a company creating a static site generator for dashboards, which then struggles to find a market fit as it tries to bring datascience to middle management, to finally reach a focused, simple and elegant tool for exploratory programming, data visualisation, and interactive documentation in javascript.

nextworddev 3y ago
Ok but is anyone paying for it?
- j-pb 3y ago
  We used to pay for 4 licenses, until they switched to the weird pricing schemes and the new editor targeted towards people with no programming experience. I really wish they would open source Observable 0.5, Pluto is currently the only other notebook left that has the flexible data-flow model at reasonable simplicity.

omneity 3y ago

This is super cool! I’ve been looking for ways to integrate Observable with my blog posts and make them more interactive and engaging. This might just be it. Thank you and congrats on the release!

kuatroka 3y ago

A couple of questions: 1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't it be huge and how long will it take to generates this static data or is there an actual call back to the server to the duckdb file (I assume it works with .duckdb file too?) 2. If Observable Framework provides the front-end, does it mean I can use any auth library if I want to create a web site with a log in and subscription options? 3. If it's a static web page, does it mean that at any time a user views a chart, they will also be able to go to the Dev Tools and download the file with data that's behind the viz? 4. When (if you can share of course) is the planned release of Plot's interactions: zoom, pan, interactive legend, brush? 5. Deployment - with big parquet, sqlite, csv files, it's impossible to do CI/CO through github or vercel and such. Will your hosting services offer an option to host those files and runtimes to generate them? Thanks

chrisjc 3y ago
Came here with similar questions and Cmd-F "DuckDB". See the comment about "data loaders". Seems like a "data loader" would provide most of what you're asking about. I'm also thinking that a "data loader" combined with duckdb-wasm and arrow would be a pretty nice combination. I imagine that it might not be too difficult to switch two between two implementations of the "data loader" as needed. Switch between reading from a remote system (in your case DuckDB on a server) and DuckDB running locally in the browser (that can interact with its own remote or local data sources). edit: welp https://observablehq.com/framework/lib/duckdb https://observablehq.com/framework/lib/duckdb
- recifs 3y ago
  See the example at https://huggingface.co/spaces/observablehq/fpdn https://huggingface.co/spaces/observablehq/fpdn where DuckDB is used both as a data loader (to download and digest 200GB worth of source data into a small 8MB parquet file) and on the client-side to allow the user to do live search queries on the minimized data. Server-side, we're using duckdb-the-binary, and client-side we're using duckdb-wasm.

johnmorrison 3y ago

This is very cool to see! We've been building something very similar (in some regards, very different in others) in https://rysana.com/bundown https://rysana.com/bundown Somehow ideas like this emerge in waves seemingly with no coordination, kind of like what happened when Calculus was first invented (Of course there's a lot of prior art here but it's interesting to see a specific jump towards polyglot single-file Markdown 'apps' and so on happen around the same time)

coolca 3y ago

It is so beautiful

77ko 3y ago

This looks amazing! I really like the clear seperation of loading/prepping data, and presenting it. Some requests: Add simple examples and more clarity to the publish docs. I assume most ppl would prefer to deploy via github actions[1], which the docs currently just link to a complex deploy file - can you please add some more documentation on this, or add an example of the simplest possible deploy file? Suggestion: is it possible to use a interface (like in vercel) to connect to a github repo and build/publish on changes? [1]: https://observablehq.com/framework/getting-started#deploying-via-github-actions https://observablehq.com/framework/getting-started#deploying...

one_buggy_boi 3y ago

What would be a good design pattern to put these dashboards behind auth? I suppose since they're static files you could just serve them with something like FastAPI or Spring Boot and have your CI/CD refresh the static files throughout the day on shared storage?

a-ve 3y ago
If you're putting these behind a reverse proxy (nginx, etc.) you can just setup client certificate authentication by using your own locally generated CA or by using something like Vault for UI-based certificate generation. When you visit this site with a certificate installed on your device, it will authenticate successfully, and for those who do not have a correct certificate installed, a "No certificate presented" error will be shown. It's fairly easy to setup and there are multiple guides available for it. Here's one: https://fardog.io/blog/2017/12/30/client-side-certificate-authentication-with-nginx/ https://fardog.io/blog/2017/12/30/client-side-certificate-au...
mbostock 3y ago
For one, you can deploy them to Observable (with `observable deploy`) and we’ll provide access control.
apitman 3y ago
I'd recommend a reverse proxy and login server using "forward auth". I made a list of such login servers here: https://github.com/lastlogin-io/obligator?tab=readme-ov-file#comparison-is-the-thief-of-joy https://github.com/lastlogin-io/obligator?tab=readme-ov-file...

farhanhubble 3y ago

I love how much is possible with the Observable framework and support for libraries like d3.js. However many data apps cannot precompute their outputs. For example, a pipeline that extracts text from documents based on what a user queries, cannot precompute the results and any visualizations must be updated every time. The best hack to accomplish this seems to be rebuilding the app on each update. Or is there another solution?

recifs 3y ago
The code in a Framework can do whatever you want it to do—it can load data on demand, call an external API, etc. Precomputing data is only an option, not an obligation. But even when you want things to be very interactive it is a good idea to minimize the data. Expose only the "rows and columns" that you need, and compress it as much as possible. This can be done in a data loader. For example, see the data app we deployed yesterday on hugging-face: its data loaders ingest a large source database (320 files totaling 200GB), and digests it into a single 8MB parquet file that we can then use on the page to "live query" 3 million newspaper titles and dates. https://huggingface.co/spaces/observablehq/fpdn https://huggingface.co/spaces/observablehq/fpdn
RyanHamilton 3y ago
I took the query only what you need approach with PulseUI: https://www.timestored.com/pulse/ https://www.timestored.com/pulse/ It emphasises not sending all data to the client. For real-time and very large data sets this is the only option.

tetris11 3y ago

I miss source loading D3 into a simple HTML page, and having D3 tutorials that were up to spec with the latest D3 release. Yeah Observable data pages look cool, but it really feels like excessive JS bloat for the features. I think I miss throwing a D3 viz together without having to load an entire framework library.

meowtimemania 3y ago
@observablehq/plot is what you're looking for if you want to have the same charts loaded via a script tag. https://observablehq.com/plot/getting-started#plot-in-vanilla-html https://observablehq.com/plot/getting-started#plot-in-vanill...
recifs 3y ago
You can still use D3 the good old-fashioned way, as a standalone script. See https://d3js.org/getting-started#d3-in-vanilla-html https://d3js.org/getting-started#d3-in-vanilla-html for examples.

cheptsov 3y ago

Congrats on the launch! Excited to hear that you go fully open-source with this! There is a certain need for great visualization tools that enable building apps.

dleeftink 3y ago

I've setup a codespace starter template that pulls in the required dependencies to start building dashboards with Observable Framework, right inside your browser. You can access the starter template here: [0]: https://github.com/dleeftink/observable-codespace https://github.com/dleeftink/observable-codespace

kuatroka 3y ago

Are you planning to add new UI components like Data Table or other in the future or it’s purely Plotting and Data ingestion and the UI through Tailwind or CSS or would it be possible to add UI libraries like shadcn or DaisyUI to make it a full fledged web site? Thanks

recifs 3y ago
It’s only the first public release, rest assured we have plans to develop it beyond that point :-) Data tables are high on the list, but it’s going to be a lot of work and I can’t say when we’ll have something to share. In the meantime almost any library available on npm should work out of the box—not just the ones that we added explicit support for (even though of course some might need more work than others).

vwkd 3y ago

How does this compare to a traditional reactive framework like React, Vue, or Svelte with Observable Plot?

petre 3y ago

ObservableHQ is great but for me this crashed on install. Unfortunately I'm not aware of any npm mechanism to report errors back to the developers. $ npm init @observablehq node:internal/errors:477 ErrorCaptureStackTrace(err);