5 ms·

How would (e.g.) strings work in this FFI, considering that wasm doesn’t have a built-in string type or any way to pass variable-length data to a function witho

by panic 4y ago

How would (e.g.) strings work in this FFI, considering that wasm doesn’t have a built-in string type or any way to pass variable-length data to a function without allocating space for it in the instance’s memory, and languages have different native string representations (UTF-8 vs UTF-16 vs “UTF-16”)?

jayd16 4y ago
No one said it would be cheap or allocation free.
kevingadd 4y ago
Very poorly considering that it seems like all the people doing wasm FFI work have decided that UTF-16/WTF-16 based language maintainers can pound sand and unless your native string type is UTF-8 your string performance is gonna suck. It's sad since many widely used runtimes (JS, Java, .NET) and one widely used operating system (Windows) all use WTF-16. UTF-8 is great and all but reality needs to be taken into account when drafting specs...
- deleted 4y ago
  [deleted]
tempodox 4y ago
Here's an example of handling strings with the WASI SDK in the browser (no wasmer involved): https://codeberg.org/bit-fu/Browser-WASI https://codeberg.org/bit-fu/Browser-WASI
garganzol 4y ago
That's a cool question. There are several aspects to that. One is storage and another is encoding. Let's start with storage. C strings, for instance, are terminated by NUL (zero) characters. Are they good enough? You would never answer that question unless you are somehow versed in math. From the mathematical standpoint, it is a mistake to narrow down the space by giving its elements special meaning. Imagine an integer number from -100 to 100. Would you be happy if somebody said you that you cannot use zero because "it has a special meaning"? No, you would be outraged every time you need to use it! (even if it's a rare event). So, from the mathematical standpoint universal strings cannot give special meanings to its characters. Now to the encoding. Is UTF-8 good enough? Or maybe UTF-16? Or something else? How to decide? Let's use math. From the math standpoint, it's always preferable to use an existing concept if it fulfills the need. So, it means that we should use the existing encoding scheme to avoid reinventing the wheel. The next thing the math dictates is that: the most minimal and normalized thing is always preferable to more complex synthetic things provided that they all do the job. This puts UTF-8 encoding at a special place - it's minimal and universal as it can encode the endless number of Unicode symbols (if we, for whatever reason, will decide to colonize the space and consequently add some more billions of characters to Unicode to dial the new cultures and knowledge in). What to do we have in result? UTF-8 string is universal and should be used in a standard component model. But how to encode string length? We already decided that giving special meaning to the NUL characters is a no-go from the math standpoint because it is opinionated and thus informal. The formal approach says that we should store the length of a string alongside the string itself. The beginning of the string is the right place because we want to know string length beforehand. What should we use? Is 32-bit unsigned integer good enough? It surely will be enough for anyone for all times, right? But the math says that a 32-bit unsigned integer can only represent [0, 2^32) range of string lengths. That's not universal because, you know, there may be foreign cultures out there in the universe who have longer strings than that. So what should we use then? The answer is: an unsigned infinite integer number with variable length encoding. We already have a Variable-Length Quantity (VLQ) concept, so let's use just that because it's already invented and universal. What do we have in the final result? The answer may surprise you: UTF-8 encoded Pascal-like immutable strings, where the string lengths are located at the start of the string and encoded with VLQ encoding. They are formally universal and pass the high mark of a standard component model as we may call it some day. At the same time, this way of building strings is quite a departure from what we have now and will have performance implications. That's why we should cut corners to make such level of universality to be practical in the computer world. An obvious measure we can take is to use size_t instead of VLQ-encoded integer for string length representation. The computer memory cannot hold strings larger than that anyway, so this will do the job. The math behind this decision is formal and clear: max_string_size = min(∞, 2 ^ sizeof(size_t)) = 2 ^ sizeof(size_t). This also means that a standard component model should have a notion of size_t-like type which is tied to the memory size of a host architecture. In this way, we make our component model dependent on a host architecture. In return, we get considerable performance benefits while preserving the formal universality which is a win.

by panic 4y ago

jayd16 4y ago

No one said it would be cheap or allocation free.

kevingadd 4y ago

Very poorly considering that it seems like all the people doing wasm FFI work have decided that UTF-16/WTF-16 based language maintainers can pound sand and unless your native string type is UTF-8 your string performance is gonna suck. It's sad since many widely used runtimes (JS, Java, .NET) and one widely used operating system (Windows) all use WTF-16. UTF-8 is great and all but reality needs to be taken into account when drafting specs...

deleted 4y ago
[deleted]

tempodox 4y ago

Here's an example of handling strings with the WASI SDK in the browser (no wasmer involved): https://codeberg.org/bit-fu/Browser-WASI https://codeberg.org/bit-fu/Browser-WASI

garganzol 4y ago

That's a cool question. There are several aspects to that. One is storage and another is encoding. Let's start with storage. C strings, for instance, are terminated by NUL (zero) characters. Are they good enough? You would never answer that question unless you are somehow versed in math. From the mathematical standpoint, it is a mistake to narrow down the space by giving its elements special meaning. Imagine an integer number from -100 to 100. Would you be happy if somebody said you that you cannot use zero because "it has a special meaning"? No, you would be outraged every time you need to use it! (even if it's a rare event). So, from the mathematical standpoint universal strings cannot give special meanings to its characters. Now to the encoding. Is UTF-8 good enough? Or maybe UTF-16? Or something else? How to decide? Let's use math. From the math standpoint, it's always preferable to use an existing concept if it fulfills the need. So, it means that we should use the existing encoding scheme to avoid reinventing the wheel. The next thing the math dictates is that: the most minimal and normalized thing is always preferable to more complex synthetic things provided that they all do the job. This puts UTF-8 encoding at a special place - it's minimal and universal as it can encode the endless number of Unicode symbols (if we, for whatever reason, will decide to colonize the space and consequently add some more billions of characters to Unicode to dial the new cultures and knowledge in). What to do we have in result? UTF-8 string is universal and should be used in a standard component model. But how to encode string length? We already decided that giving special meaning to the NUL characters is a no-go from the math standpoint because it is opinionated and thus informal. The formal approach says that we should store the length of a string alongside the string itself. The beginning of the string is the right place because we want to know string length beforehand. What should we use? Is 32-bit unsigned integer good enough? It surely will be enough for anyone for all times, right? But the math says that a 32-bit unsigned integer can only represent [0, 2^32) range of string lengths. That's not universal because, you know, there may be foreign cultures out there in the universe who have longer strings than that. So what should we use then? The answer is: an unsigned infinite integer number with variable length encoding. We already have a Variable-Length Quantity (VLQ) concept, so let's use just that because it's already invented and universal. What do we have in the final result? The answer may surprise you: UTF-8 encoded Pascal-like immutable strings, where the string lengths are located at the start of the string and encoded with VLQ encoding. They are formally universal and pass the high mark of a standard component model as we may call it some day. At the same time, this way of building strings is quite a departure from what we have now and will have performance implications. That's why we should cut corners to make such level of universality to be practical in the computer world. An obvious measure we can take is to use size_t instead of VLQ-encoded integer for string length representation. The computer memory cannot hold strings larger than that anyway, so this will do the job. The math behind this decision is formal and clear: max_string_size = min(∞, 2 ^ sizeof(size_t)) = 2 ^ sizeof(size_t). This also means that a standard component model should have a notion of size_t-like type which is tied to the memory size of a host architecture. In this way, we make our component model dependent on a host architecture. In return, we get considerable performance benefits while preserving the formal universality which is a win.