8 ms·

Arthur Whitney is showing how to build an array language using C in array language form. AW is known for kdb+ which is often used in finance due to its extreme

by eismcc 3y ago

Arthur Whitney is showing how to build an array language using C in array language form. AW is known for kdb+ which is often used in finance due to its extreme performance properties and ability for quants to quickly explore ideas. Personally, to get a better handle on how array languages worked, I implemented KlongPy which is a python implementation of Klong, which descends from K (which AW wrote). You have to play with this stuff to understand it intuitively.

bwanab 3y ago
Ha! For those of us who were in the Morgan Stanley Fixed Income group for many years in the 1990s and 2000s AW is known for APLus which was used extensively for modeling and application work.
- CraigRo 3y ago
  There were a lot of different and kind of weird attempts at this in the industry. The basic problem is that recalculating DAGs and monte-carlo simulations using a ton of floating point math was extremely expensive. Excel was way too slow, C++ required expert programmers and much better organization of programming teams to figure how to coordinate cached intermediate values. MS went with APL, which reduced a lot of the boilerplate and allowed for relatively easy memoization. GS went with SecDB which treated computation as memoized DAGs. Not sure what the others did
sudosysgen 3y ago
FWIW, kdb+ is not that extremely performant - there's a lot of things that could be faster, and a lot of limitations that mean that you often would be better of not using a DB at all (or to use another DB and just pull everything you might need into memory). There is/was a tradeoff in that many things that would make it faster would require more code, and a cool thing about q/kdb+ is that it takes so little code you don't have I$ issues, but I think that's a tradeoff that doesn't make as much sense anymore in 2023. What it's really great for is that it's really neatly integrated into the q language, which is great for exploratory programming, and it's fast enough not to get in the way.
- eismcc 3y ago
  Thanks the insights. Not to over do self promotion, but aside from learning, the main reason I made KlongPy was to allow for optionality with the ecosystem. Use Klong for array operations and other libraries for standard stuff.
- mst 3y ago
  > a cool thing about q/kdb+ is that it takes so little code you don't have I$ issues I'm not sure what an I$ issue is - any chance of a bit more explanation? (also I'd love to hear about the tradeoff part in more detail but that's a bigger ask)
  - durumu 3y ago
    I$ is short for instruction cache ($ -> "cash" -> cache). Since kdb has so little code, more of it fits into the instruction cache at once.
    - omnicognate 3y ago
      I've encountered this idea that k's terseness somehow improves instruction cache use before. Can you explain further? It seems nonsensical, since instruction caching is about machine code, not source code. Why should it use the instruction cache better than any other JIT? Or is it interpreted, in which case "the terseness of the language improves cache use" might seem more of an admission than a boast... :-)
      - mlochbaum 3y ago
        I say it's nonsensical (and yes, Ks are bytecode interpreted). https://mlochbaum.github.io/BQN/implementation/kclaims.html#instruction-cache https://mlochbaum.github.io/BQN/implementation/kclaims.html#...
  - scrawl 3y ago
    they mean CPU instruction cache

by eismcc 3y ago

bwanab 3y ago

Ha! For those of us who were in the Morgan Stanley Fixed Income group for many years in the 1990s and 2000s AW is known for APLus which was used extensively for modeling and application work.

CraigRo 3y ago
There were a lot of different and kind of weird attempts at this in the industry. The basic problem is that recalculating DAGs and monte-carlo simulations using a ton of floating point math was extremely expensive. Excel was way too slow, C++ required expert programmers and much better organization of programming teams to figure how to coordinate cached intermediate values. MS went with APL, which reduced a lot of the boilerplate and allowed for relatively easy memoization. GS went with SecDB which treated computation as memoized DAGs. Not sure what the others did

sudosysgen 3y ago

FWIW, kdb+ is not that extremely performant - there's a lot of things that could be faster, and a lot of limitations that mean that you often would be better of not using a DB at all (or to use another DB and just pull everything you might need into memory). There is/was a tradeoff in that many things that would make it faster would require more code, and a cool thing about q/kdb+ is that it takes so little code you don't have I$ issues, but I think that's a tradeoff that doesn't make as much sense anymore in 2023. What it's really great for is that it's really neatly integrated into the q language, which is great for exploratory programming, and it's fast enough not to get in the way.

eismcc 3y ago
Thanks the insights. Not to over do self promotion, but aside from learning, the main reason I made KlongPy was to allow for optionality with the ecosystem. Use Klong for array operations and other libraries for standard stuff.
mst 3y ago
> a cool thing about q/kdb+ is that it takes so little code you don't have I$ issues I'm not sure what an I$ issue is - any chance of a bit more explanation? (also I'd love to hear about the tradeoff part in more detail but that's a bigger ask)
- durumu 3y ago
  I$ is short for instruction cache ($ -> "cash" -> cache). Since kdb has so little code, more of it fits into the instruction cache at once.
  - omnicognate 3y ago
    I've encountered this idea that k's terseness somehow improves instruction cache use before. Can you explain further? It seems nonsensical, since instruction caching is about machine code, not source code. Why should it use the instruction cache better than any other JIT? Or is it interpreted, in which case "the terseness of the language improves cache use" might seem more of an admission than a boast... :-)
    - mlochbaum 3y ago
      I say it's nonsensical (and yes, Ks are bytecode interpreted). https://mlochbaum.github.io/BQN/implementation/kclaims.html#instruction-cache https://mlochbaum.github.io/BQN/implementation/kclaims.html#...
- scrawl 3y ago
  they mean CPU instruction cache