4 ms·

Recommendations when publishing a WASM library

40 points by comagoosie 4y ago

kylebarron 4y ago
Looks to be a great resource. I've been working on a WASM implementation of reading and writing Apache Parquet [0] and it's been difficult being new to WASM to find the best way of distributing the WASM that works on Node and through bundlers like Webpack. [0]: https://github.com/kylebarron/parquet-wasm https://github.com/kylebarron/parquet-wasm
Shadonototra 4y ago
Security wise, it is not a good idea to consume WASM libraries "as is", ask for the source, read it, and compile it yourself You don't want to be in a position to ship code to production with binary code that could potentially be harmful Off topic: Please don't mess up the way my browser scroll pages, it is infuriating
- dgb23 4y ago
  Can you elaborate on that? From what I know, WASM is a safe, rather abstract bytecode format and has far less API capabilites as JS has (which is why you need to call it from JS to affect the browser).
  - johannes1234321 4y ago
    The concern is the same as with any dependency: The dependency runs under your privileges with access to your data. A malice vendor could do "anything" at least within the scope of your application. For instance if you create a web mail application the code probably has access to all mails, can delete them, can send mail under the user's identity, ... How relevant those scenarios are you have to evaluate. If you compile yourself, you can verify the source to increase trust. If you just get the binary, you have to trust the vendor more.
    - dgb23 4y ago
      But again, it's not "just" a binary. You provide the interfaces to it that actually affect the world. I agree with the general implications of using a dependency though.
      - gwbas1c 4y ago
        At least in in-browser Blazor (C# compiled to WASM) I have full access to the page's Javascript environment: I can call most Javascript methods available, and I can even call eval. I'm still new to WASM, but I assume this functionality is part of the WASM runtime? (As opposed to a hook that's part of the Javascript part of Blazor?)
        jitl 4y ago
        You are assuming wrong, this is Blazor’s JS support library. By “default” WebAssembly has nothing outside the WASM bytecode, not even memory/allocator. It’s up to the host environment to provide the WASM module a Memory and whatever “imports” it needs.
        klibertp 4y ago
        Memory can also be statically reserved in a binary, like the .data section in x86 asm. IIRC (tell me if I'm very wrong) Emscripten malloc works by reserving a large buffer up front and then managing pointers to (parts of) it. Otherwise, you can allocate memory buffer dynamically from JS[1] and pass the pointer to it to the WASM. Obviously, you can do this in response to the request coming from WASM, but you don't have to. [1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly/Memory https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
      - johannes1234321 4y ago
        It is true, that you have to expose things first. However as soon as any non-trivial object is shared from JS to WASM it is likely that some reference to something global, to the DOM, ... is nested in there. And that makes all other attempts to limit access void. (DOM give networking etc )
- deleted 4y ago
  [deleted]
- aseipp 4y ago
  In the case of NPM consumers of wasm libraries, it often isn't realistic option "as-is", since they won't have the toolchain needed to build the code. Rust is a lot more well oiled than C/C++ in this regard but it's a bit of a hassle to line things up and keep things reliable and reproducible. (Good luck if the blob is not only C/C++ but uses 3rd party dependencies on top of that, which often require even more hurdles). If you don't use the pre-built blob you'll often have to 'insert' it otherwise into the library, somehow, which is its own chore. So it's all a bit chicken and egg at some level. Now, it's not like a lot of these packages ever followed best practices in this regard (I'm reminded of many Ruby, Python, JS packages that love to ship random .so files, and often do it incorrectly) but it is what it is. That said I generally agree with the premise, and even with sandboxing you should vet dependencies like these where appropriate if you can. A good example of this is something like an image decoder versus a database library (both of these being real scenarios; e.g. using a pure-Rust implementation of some SQL protocol.) The first one I probably wouldn't worry too about much, you're just giving it pixels in and getting pixels out. But the second one is likely worth a bit of scrutiny since it interfaces directly with a sensitive component.
- user-the-name 4y ago
jitl 4y ago
Wow, this is a great resource! I’ve been dealing with a lot of these issues with a JS sandbox library I’m building quickjs-emecripten. I want to offer a ton of different build variants of my core WASM file. Here’s what I tried: https://github.com/justjake/quickjs-emscripten/blob/master/ts/variants.ts https://github.com/justjake/quickjs-emscripten/blob/master/t... This approach works great for NodeJS, but once I ran a test bundle I found that Webpack (and bundlephobia) included all the base64 “release” variants instead of lazy-loading the import statements. Bummer. I assumed this because Typescript on its own compiled import to Promise.resolve(require(…)), so it’s good to know that most bundlers will STILL get this wrong even if I’m emitting ES6 module import syntax. Yikes! I need to bite the bullet and start using Rollup to emit a slew of separate entry points. Oy veh. Anyways A+++ would read again. This will save me 4-5 days of work stubbing my toe on bundlers and build system which is the Least Fun part of JS work.
westurner 4y ago
conda: "Adding a WebAssembly platform" https://github.com/conda/conda/issues/7619 https://github.com/conda/conda/issues/7619 pyodide.loadPackage("numpy"); pyodide.loadPackage("https://foo/bar/numpy.js"); # import micropip micropip.install('https://example.com/files/snowballstemmer-2.0.0-py2.py3-none-any.whl') "Creating a Pyodide package" > "2. Creating the meta.yaml file" https://pyodide.org/en/stable/development/new-packages.html#creating-the-meta-yaml-file https://pyodide.org/en/stable/development/new-packages.html#... conda-forge: "WASM as a supported architecture" https://github.com/conda-forge/conda-forge.github.io/issues/1401 https://github.com/conda-forge/conda-forge.github.io/issues/...
modeless 4y ago
I really hate that base64 is the best option for embedding inline assets in HTML/JS files. Have there been any proposals to add real heredoc support to HTML/JS? With the ability to choose delimiters and support for unrestricted binary data?
kansface 4y ago
Does anyone have practical experience running WASM in long running processes on node, as compared to something like Neon? Writing C in TS doesn't look particularly appealing to me, but node extensions bring their own set of problems.
- whb07 4y ago
  Write it in Rust.

40 points by comagoosie 4y ago

Looks to be a great resource. I've been working on a WASM implementation of reading and writing Apache Parquet [0] and it's been difficult being new to WASM to find the best way of distributing the WASM that works on Node and through bundlers like Webpack. [0]: https://github.com/kylebarron/parquet-wasm https://github.com/kylebarron/parquet-wasm

Shadonototra 4y ago

Security wise, it is not a good idea to consume WASM libraries "as is", ask for the source, read it, and compile it yourself You don't want to be in a position to ship code to production with binary code that could potentially be harmful Off topic: Please don't mess up the way my browser scroll pages, it is infuriating

dgb23 4y ago
Can you elaborate on that? From what I know, WASM is a safe, rather abstract bytecode format and has far less API capabilites as JS has (which is why you need to call it from JS to affect the browser).
- johannes1234321 4y ago
  The concern is the same as with any dependency: The dependency runs under your privileges with access to your data. A malice vendor could do "anything" at least within the scope of your application. For instance if you create a web mail application the code probably has access to all mails, can delete them, can send mail under the user's identity, ... How relevant those scenarios are you have to evaluate. If you compile yourself, you can verify the source to increase trust. If you just get the binary, you have to trust the vendor more.
  - dgb23 4y ago
    But again, it's not "just" a binary. You provide the interfaces to it that actually affect the world. I agree with the general implications of using a dependency though.
    - gwbas1c 4y ago
      At least in in-browser Blazor (C# compiled to WASM) I have full access to the page's Javascript environment: I can call most Javascript methods available, and I can even call eval. I'm still new to WASM, but I assume this functionality is part of the WASM runtime? (As opposed to a hook that's part of the Javascript part of Blazor?)
      - jitl 4y ago
        You are assuming wrong, this is Blazor’s JS support library. By “default” WebAssembly has nothing outside the WASM bytecode, not even memory/allocator. It’s up to the host environment to provide the WASM module a Memory and whatever “imports” it needs.
        klibertp 4y ago
        Memory can also be statically reserved in a binary, like the .data section in x86 asm. IIRC (tell me if I'm very wrong) Emscripten malloc works by reserving a large buffer up front and then managing pointers to (parts of) it. Otherwise, you can allocate memory buffer dynamically from JS[1] and pass the pointer to it to the WASM. Obviously, you can do this in response to the request coming from WASM, but you don't have to. [1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly/Memory https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
    - johannes1234321 4y ago
      It is true, that you have to expose things first. However as soon as any non-trivial object is shared from JS to WASM it is likely that some reference to something global, to the DOM, ... is nested in there. And that makes all other attempts to limit access void. (DOM give networking etc )
deleted 4y ago
[deleted]
aseipp 4y ago
In the case of NPM consumers of wasm libraries, it often isn't realistic option "as-is", since they won't have the toolchain needed to build the code. Rust is a lot more well oiled than C/C++ in this regard but it's a bit of a hassle to line things up and keep things reliable and reproducible. (Good luck if the blob is not only C/C++ but uses 3rd party dependencies on top of that, which often require even more hurdles). If you don't use the pre-built blob you'll often have to 'insert' it otherwise into the library, somehow, which is its own chore. So it's all a bit chicken and egg at some level. Now, it's not like a lot of these packages ever followed best practices in this regard (I'm reminded of many Ruby, Python, JS packages that love to ship random .so files, and often do it incorrectly) but it is what it is. That said I generally agree with the premise, and even with sandboxing you should vet dependencies like these where appropriate if you can. A good example of this is something like an image decoder versus a database library (both of these being real scenarios; e.g. using a pure-Rust implementation of some SQL protocol.) The first one I probably wouldn't worry too about much, you're just giving it pixels in and getting pixels out. But the second one is likely worth a bit of scrutiny since it interfaces directly with a sensitive component.
user-the-name 4y ago

jitl 4y ago

Wow, this is a great resource! I’ve been dealing with a lot of these issues with a JS sandbox library I’m building quickjs-emecripten. I want to offer a ton of different build variants of my core WASM file. Here’s what I tried: https://github.com/justjake/quickjs-emscripten/blob/master/ts/variants.ts https://github.com/justjake/quickjs-emscripten/blob/master/t... This approach works great for NodeJS, but once I ran a test bundle I found that Webpack (and bundlephobia) included all the base64 “release” variants instead of lazy-loading the import statements. Bummer. I assumed this because Typescript on its own compiled import to Promise.resolve(require(…)), so it’s good to know that most bundlers will STILL get this wrong even if I’m emitting ES6 module import syntax. Yikes! I need to bite the bullet and start using Rollup to emit a slew of separate entry points. Oy veh. Anyways A+++ would read again. This will save me 4-5 days of work stubbing my toe on bundlers and build system which is the Least Fun part of JS work.

westurner 4y ago

conda: "Adding a WebAssembly platform" https://github.com/conda/conda/issues/7619 https://github.com/conda/conda/issues/7619 pyodide.loadPackage("numpy"); pyodide.loadPackage("https://foo/bar/numpy.js"); # import micropip micropip.install('https://example.com/files/snowballstemmer-2.0.0-py2.py3-none-any.whl') "Creating a Pyodide package" > "2. Creating the meta.yaml file" https://pyodide.org/en/stable/development/new-packages.html#creating-the-meta-yaml-file https://pyodide.org/en/stable/development/new-packages.html#... conda-forge: "WASM as a supported architecture" https://github.com/conda-forge/conda-forge.github.io/issues/1401 https://github.com/conda-forge/conda-forge.github.io/issues/...

modeless 4y ago

I really hate that base64 is the best option for embedding inline assets in HTML/JS files. Have there been any proposals to add real heredoc support to HTML/JS? With the ability to choose delimiters and support for unrestricted binary data?

kansface 4y ago

Does anyone have practical experience running WASM in long running processes on node, as compared to something like Neon? Writing C in TS doesn't look particularly appealing to me, but node extensions bring their own set of problems.

whb07 4y ago
Write it in Rust.