17 ms·

My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

126 points by jjp 1y ago

__mp 1y ago
I’ve been toying with the idea of implementing a distributed analytics engine on top of Cloudflare workers and DuckDB. I’m not sure if this goes against the CloudFlare TOS tough (last time I checked they had some provisons against processing images).
- judge2020 1y ago
  In the past few years there was this blog post[0] that clarified this. It moved the restriction on serving a "disproportionate percentage of pictures, audio files, or other large files" to another part of the TOS dedicated specifically to the CDN part[1] and clarified that, if you're using Cloudflare add-on services Stream, R2 (their S3), or Cloudflare Images, then you won't be at risk of termination. 0: https://blog.cloudflare.com/updated-tos/ https://blog.cloudflare.com/updated-tos/ 1: The restriction still exists at https://www.cloudflare.com/service-specific-terms-application-services/ https://www.cloudflare.com/service-specific-terms-applicatio... under "Content Delivery Network (Free, Pro, or Business)".
- httgp 1y ago
  BoilingData does something very similar, except with Lambdas on AWS.
- bobnamob 1y ago
  They've already got one: https://developers.cloudflare.com/analytics/analytics-engine/ https://developers.cloudflare.com/analytics/analytics-engine...
- tobilg 1y ago
  IMO that’s not really possible because of the size limits of Cloudflare Workers. Neither the WASM nor the Node version are small enough. I‘m running it on AWS Lambda functions with some success.
canadiantim 1y ago
How persistent can you make this data?
mcraiha 1y ago
Tip for all the blog authors, do NOT post code as image. Specially do not add fake editor UI and drop shadow to the image. In this case 25 lines of code is 50 kB of image binary. Also it cannot be searched via search engine. Nor can it be read with screen reader.
- goda90 1y ago
  This guide can help if you still want the code to be pretty: https://www.taniarascia.com/adding-syntax-highlighting-to-code-snippets/ https://www.taniarascia.com/adding-syntax-highlighting-to-co...
  - ciupicri 1y ago
    Or use vim to convert it to HTML https://vi.stackexchange.com/questions/792/how-to-convert-a-source-code-file-into-html https://vi.stackexchange.com/questions/792/how-to-convert-a-...
- doubled112 1y ago
  Pro tip for everybody: do not post any text as images Never should I receive a Java exception hundreds of lines long as a cut off JPEG file. Or a screenshot of a Google Sheet missing the information you’re talking to me about.
  - shepmaster 1y ago
    We made this to be used as a reply when pictures are misused where text would be better: https://fewer.pics/ https://fewer.pics/
    - stingraycharles 1y ago
      I’m disappointed the rendered picture isn’t advanced CSS. I honestly expected that to be the case.
      - 1y ago
curtisszmania 1y ago
[dead]
jasmcole 1y ago
We use the WASM build of DuckDB quite extensively at Count (https://count.co https://count.co - 2-3m queries per month). There are a couple of bugs we've noticed, but given that it's pretty much maintained by a single person seems impressively reliable!
- jillyboel 1y ago
  Looking at your insane pricing page I have to assume that you are sponsoring that single person?
  - Spivak 1y ago
    I'm confused, nothing about their pricing looks that weird. Businesses don't typically have large BI teams so you can ride that $199/mo $2400/year for a long time which is so small most SMBs can probably expense it without approval.
    - jillyboel 1y ago
      You're focussing on the wrong part
  - jillyboel 1y ago
    Gotta love being downvoted for daring to suggest a company should sponsor the sole open source dev making their whole product possible.
tobilg 1y ago
Not sure why the comparisons were made with pretty outdated versions to be honest. I‘m using a (older) v1.29.1 dev version with https://sql-workbench.com https://sql-workbench.com w/o any bigger issues.
azakai 1y ago
> [wasm] is executed in a stack-based virtual machine rather than as a native library code. Wasm's binary format is indeed a stack-based virtual machine, but that is not how it is executed. Optimizing VMs convert it to SSA form, basic blocks, and finally machine code, much the same as clang or gcc compile native library code. It is true that wasm has some overhead, but that is due to portability and sandboxing, not the stack-based binary format. > On top of the above, memory available to WASM is limited by the browser (in case of Chrome, the limit is currently set at 4GB per tab). wasm64 solves this, by allowing 64-bit pointers and a lot more than 4GB of memory. The feature is already supported in Chrome and Firefox, but not everywhere else yet.
- geokon 1y ago
  The more I read about WASM the more it sounds like the JVM I'm still not clear what at its core it's done differently (in a way that couldn't be bolted on to a subset of the JVM)
  - azakai 1y ago
    The JVM is designed around Java. That's really the main difference, and it brings some downsides for the goals of wasm, which include running native code - think C++ or Rust. The JVM is great at Java, which relies on runtime inlining etc., but not at C++, which assumes ahead-of-time inlining and other optimizations.
    - geokon 1y ago
      I don't understand how the virtual machine would preclude you from in-lining ahead of time...? That's done when you're compiling. What is WASM doing to facilitate recompiling native code that isn't practical to do on the JVM
pmm 1y ago
Author here. Thank you all for the comments. I take full responsibility for stupidly using an image for posting the code snippet. Sorry for that! Also, the article was originally posted almost 2 years ago (and "resurrected" with the recent migration to Medium). This is why a fairly old DuckDB version is referenced there. Some of the issues I observed are now gone too. Obviously, many things have changed since then. We've experimented extensively and moved back and forth with using DuckDB for our internal cloud processing architecture. We eventually settled on just using it for reading the data and then handling everything else in custom workers. Even using TypeScript, we achieved close to 1M events/s per worker overall with very high scalability. However, our use-case is quite distinct. We use a custom query engine (for sequence processing), which has driven many design decisions. Overall, I think DuckDB (both vanilla and WASM version) is absolutely phenomenal. It also matured since my original blog post. I believe we'll only see more and more projects using it as their backbone. For example, MotherDuck is doing some amazing things with it (e.g., https://duckdb.org/2023/03/12/duckdb-ui https://duckdb.org/2023/03/12/duckdb-ui) but there are also many more exciting initiatives.

126 points by jjp 1y ago

__mp 1y ago

I’ve been toying with the idea of implementing a distributed analytics engine on top of Cloudflare workers and DuckDB. I’m not sure if this goes against the CloudFlare TOS tough (last time I checked they had some provisons against processing images).

judge2020 1y ago
In the past few years there was this blog post[0] that clarified this. It moved the restriction on serving a "disproportionate percentage of pictures, audio files, or other large files" to another part of the TOS dedicated specifically to the CDN part[1] and clarified that, if you're using Cloudflare add-on services Stream, R2 (their S3), or Cloudflare Images, then you won't be at risk of termination. 0: https://blog.cloudflare.com/updated-tos/ https://blog.cloudflare.com/updated-tos/ 1: The restriction still exists at https://www.cloudflare.com/service-specific-terms-application-services/ https://www.cloudflare.com/service-specific-terms-applicatio... under "Content Delivery Network (Free, Pro, or Business)".
httgp 1y ago
BoilingData does something very similar, except with Lambdas on AWS.
bobnamob 1y ago
They've already got one: https://developers.cloudflare.com/analytics/analytics-engine/ https://developers.cloudflare.com/analytics/analytics-engine...
tobilg 1y ago
IMO that’s not really possible because of the size limits of Cloudflare Workers. Neither the WASM nor the Node version are small enough. I‘m running it on AWS Lambda functions with some success.

canadiantim 1y ago

How persistent can you make this data?

mcraiha 1y ago

Tip for all the blog authors, do NOT post code as image. Specially do not add fake editor UI and drop shadow to the image. In this case 25 lines of code is 50 kB of image binary. Also it cannot be searched via search engine. Nor can it be read with screen reader.

goda90 1y ago
This guide can help if you still want the code to be pretty: https://www.taniarascia.com/adding-syntax-highlighting-to-code-snippets/ https://www.taniarascia.com/adding-syntax-highlighting-to-co...
- ciupicri 1y ago
  Or use vim to convert it to HTML https://vi.stackexchange.com/questions/792/how-to-convert-a-source-code-file-into-html https://vi.stackexchange.com/questions/792/how-to-convert-a-...
doubled112 1y ago
Pro tip for everybody: do not post any text as images Never should I receive a Java exception hundreds of lines long as a cut off JPEG file. Or a screenshot of a Google Sheet missing the information you’re talking to me about.
- shepmaster 1y ago
  We made this to be used as a reply when pictures are misused where text would be better: https://fewer.pics/ https://fewer.pics/
  - stingraycharles 1y ago
    I’m disappointed the rendered picture isn’t advanced CSS. I honestly expected that to be the case.
    - 1y ago

curtisszmania 1y ago

[dead]

jasmcole 1y ago

We use the WASM build of DuckDB quite extensively at Count (https://count.co https://count.co - 2-3m queries per month). There are a couple of bugs we've noticed, but given that it's pretty much maintained by a single person seems impressively reliable!

jillyboel 1y ago
Looking at your insane pricing page I have to assume that you are sponsoring that single person?
- Spivak 1y ago
  I'm confused, nothing about their pricing looks that weird. Businesses don't typically have large BI teams so you can ride that $199/mo $2400/year for a long time which is so small most SMBs can probably expense it without approval.
  - jillyboel 1y ago
    You're focussing on the wrong part
- jillyboel 1y ago
  Gotta love being downvoted for daring to suggest a company should sponsor the sole open source dev making their whole product possible.

tobilg 1y ago

Not sure why the comparisons were made with pretty outdated versions to be honest. I‘m using a (older) v1.29.1 dev version with https://sql-workbench.com https://sql-workbench.com w/o any bigger issues.

azakai 1y ago

> [wasm] is executed in a stack-based virtual machine rather than as a native library code. Wasm's binary format is indeed a stack-based virtual machine, but that is not how it is executed. Optimizing VMs convert it to SSA form, basic blocks, and finally machine code, much the same as clang or gcc compile native library code. It is true that wasm has some overhead, but that is due to portability and sandboxing, not the stack-based binary format. > On top of the above, memory available to WASM is limited by the browser (in case of Chrome, the limit is currently set at 4GB per tab). wasm64 solves this, by allowing 64-bit pointers and a lot more than 4GB of memory. The feature is already supported in Chrome and Firefox, but not everywhere else yet.

geokon 1y ago
The more I read about WASM the more it sounds like the JVM I'm still not clear what at its core it's done differently (in a way that couldn't be bolted on to a subset of the JVM)
- azakai 1y ago
  The JVM is designed around Java. That's really the main difference, and it brings some downsides for the goals of wasm, which include running native code - think C++ or Rust. The JVM is great at Java, which relies on runtime inlining etc., but not at C++, which assumes ahead-of-time inlining and other optimizations.
  - geokon 1y ago
    I don't understand how the virtual machine would preclude you from in-lining ahead of time...? That's done when you're compiling. What is WASM doing to facilitate recompiling native code that isn't practical to do on the JVM

pmm 1y ago

Author here. Thank you all for the comments. I take full responsibility for stupidly using an image for posting the code snippet. Sorry for that! Also, the article was originally posted almost 2 years ago (and "resurrected" with the recent migration to Medium). This is why a fairly old DuckDB version is referenced there. Some of the issues I observed are now gone too. Obviously, many things have changed since then. We've experimented extensively and moved back and forth with using DuckDB for our internal cloud processing architecture. We eventually settled on just using it for reading the data and then handling everything else in custom workers. Even using TypeScript, we achieved close to 1M events/s per worker overall with very high scalability. However, our use-case is quite distinct. We use a custom query engine (for sequence processing), which has driven many design decisions. Overall, I think DuckDB (both vanilla and WASM version) is absolutely phenomenal. It also matured since my original blog post. I believe we'll only see more and more projects using it as their backbone. For example, MotherDuck is doing some amazing things with it (e.g., https://duckdb.org/2023/03/12/duckdb-ui https://duckdb.org/2023/03/12/duckdb-ui) but there are also many more exciting initiatives.