5 ms·

32 bit version of KDB+ is now free for commercial use

68 points by deathflute 12y ago

gohwell 12y ago
For the unindoctrinated, KDB+ is an extremely fast, column oriented, in-memory database. It's based on a language called Q and has been used at many banks to store exchange related data.
- druidsbane 12y ago
  Syntax is hard to read and easy to make mistakes in considering how it overloads every letter of the alphabet as a command, but the extreme speed pays off I think.
  - DannoHung 12y ago
    You're incorrect. No individual letters of the alphabet are commands. Every symbol on the keyboard however is an operator (excluding semicolon, braces, brackets, and parens, which operate as line/expression terminators, function definitions, function invocation/array access, and list definitions, respectively).
  - klibertp 12y ago
    It's the same thing I said in the recent J discussion: J (and probably Q) is meant to be read with a help of computer. Reading and writing J consists of incrementally building/decomposing expressions in a REPL. You have wonderful tools to visualize expressions structure in the REPL and you are expected to use them and to experiment with the expressions. You're not supposed to read it as prose, don't even try.
  - fraserm 12y ago
    There is also an sql like interface in addition to the Q and K languages. This is probably easier to get started than diving into Q if that is too daunting.
  - tiredandgrumpy 12y ago
    K is only hard if you try to read it without first studying it. Looping is achieved through adverbs. The key to it is understanding what is a noun (data), verb (operator/function) and adverb (takes a verb, creates a new verb to be used infix). A verb with a noun to its right is a dyad if there is also a noun to its left, and is otherwise a monad. If it is needed, the monad can be specified by appending a colon to the right of the symbol. Fortunately, most kdb+ developers program in Q, which has a bunch of helper routines defined in k, and assigns monads to names such as neg x instead of -:x.
fidotron 12y ago
This has come up before here, and the recent GNU APL stuff reminded me, but in summary, if you have ever been curious about APL or mildly suspicious of more conventional database approaches you owe it to yourself to take a look at the concepts at work here, especially primacy to columns instead of rows. The & "where" operator in raw k has stayed with me over the years as a particularly inspired way to deal with column based data.
- profquail 12y ago
  For those of you curious about array-based / columnar programming languages, there's an APL/J/K reddit: http://www.reddit.com/r/apljk http://www.reddit.com/r/apljk
- tiredandgrumpy 12y ago
  it came up here before, but this time is different. It is now free for commercial use and is not restricted with timeouts or expiry.
tom_b 12y ago
Ck out Arthur Whitney's abridged manual for fun: http://kx.com/q/d/kdb+.htm http://kx.com/q/d/kdb+.htm For using the 32-bit version (from Limits): 22 Limits Each database runs in memory and/or disk map-on-demand -- possibly partitioned. There is no limit on the size of a partitioned database but on 32-bit systems the main memory OLTP portion of a database is limited to about 1GB of raw data, i.e. 1/4 of the address space. The raw data of a main memory 64bit process should be limited to about 1/2 of available RAM.
jmnicolas 12y ago
Except financial applications what is it good for ?
- deathflute 12y ago
  This would actually make a terrific replacement for something like redis when you need a more structured schema. The q language is very powerful, and expressive - interesting mix of lisp and APL. You can do really powerful analytics without writing tons of code for it. You really have to see how fast KDB is compared to most nosql products out there.
  - patrickxb 12y ago
    Are there any open source projects or blog posts with examples of this?
    - gohwell 12y ago
      They have a reference wiki with some example programs. Have a look at this page: http://code.kx.com/wiki/Cookbook/ProgrammingExamples http://code.kx.com/wiki/Cookbook/ProgrammingExamples
- oddthink 12y ago
  It's great for basic data-analysis tasks, where you just want to slurp in a few CSV files, join them together, filter out some rows, and spit out the results. Sure, you can do the same in R or python, but the whole process is very quick and easy in q.
- jibberia 12y ago
  Almost 10 years ago, I did an undergrad independent study at NYU contributing to some PhDs' Query by Humming music search engine. We used q to query a kdb full of catchy-melody time series data -- short sequences of "is this pitch higher, lower, or the same as the last?" and "is this note short, long, or medium?" (and, of course, gobs upon tons of variations as we iterated!). I barely did any q / kdb; only made a functional and usable UI, and did some prototyping of new ideas in other languages (Java, Max/MSP, Csound). I spent some time looking into q and was thoroughly baffled. Still am. It was really, really fast, though! As I vaguely understand and can explain it, the k/q system made it easy to do fuzzy searches and deal with missing pieces of data. If the user missed a note, or our pitch detection failed, or our source data was bad, we were still able to find matches. (Yes, I wish I'd been able to understand this more at the time. Bygones, now...)
miecio13 12y ago
I'm not sure about the Q language, but their C API reads like obfuscated C contest entry: http://kx.com/q/c/c/k.h http://kx.com/q/c/c/k.h
- tiredandgrumpy 12y ago
  If you look closely at it, there's not much there - it's actually easy to understand - defines a variant struct and an bunch of accessors to the different types within the embedded union. He prefers short names, and finally years later, java recommends short variable names for lambdas too!
  - rcxdude 12y ago
    Short names make sense if they are easily understandable locally: This means either something extremely common throughout the codebase (I think the most common example being localisation wrappers for string literals. They should ideally be linked to a more clear explanation easily, e.g. from renaming import statements), or defined (clearly) and used only within a very small area of code. This API is neither of those.
    - silentbicycle 12y ago
      Short names also make sense if the same conventions appear over and over throughout the code base. While they can be quite opaque at first, they're very consistent, and people with experience in other APLs will recognize many of them.
  - ryanobjc 12y ago
    So, I guess I need to look closer than the pixels then: typedef struct k0{signed char m,a,t;C u;I r;union{G g;H h;I i;J j;E e;F f;S s;struct k0k;struct{J n;G G0[1];};};}K; Sorry I guess I'm just not seeing the "not much there and actually easy to understand" Whatever a 'H' is
    - igorii 12y ago
      It definitely is strange to look at, but it's quite easy to understand if you know k/q. H is a short in k/q, so H refers to a short here as well. This naming scheme is true for all of the above letters.
nightTrevors 12y ago
For anyone trying this out for the first time, Jeff Borror's q for mortals is the best guide out there http://code.kx.com/wiki/JB:QforMortals2/contents http://code.kx.com/wiki/JB:QforMortals2/contents
noname123 12y ago
Currently using MongoDB for my historical quotes ticks database. Any peeps in trading use KDB+ in production or for fun think it's expressive enough to write queries directly to it for backtesting?
kthielen 12y ago
Careful with these guys. I once built an open source implementation of the q language, and these guys immediately threatened to sue me, my employer, and our clients. The language is not that interesting, it's easy to reproduce, and these guys will threaten you if you prove this.
- tiredandgrumpy 12y ago
  and now they'll sue you for libel ;-)
- beagle3 12y ago
  > The language is not that interesting I would say the language is very interesting. It is probably not interesting enough to get sued for, though .... I suspect times have changed - there are implementations that have been out there for years (https://github.com/kevinlawler/kona https://github.com/kevinlawler/kona implements k3 with sprinkles of k4, and http://althenia.net/kuc http://althenia.net/kuc implements an almost-k4 with a JIT and writable closures). IIRC, when you did your implementation it was when k4 was still a "technology preview" and not their main product (or was just released) - I remember understanding the panic in those action, even though I totally disagree with them. (I didn't know about the threats, but I do remember seeing it appear and disappear within a day, and assumed something was happening behind the scenes)

68 points by deathflute 12y ago

For the unindoctrinated, KDB+ is an extremely fast, column oriented, in-memory database. It's based on a language called Q and has been used at many banks to store exchange related data.

druidsbane 12y ago
Syntax is hard to read and easy to make mistakes in considering how it overloads every letter of the alphabet as a command, but the extreme speed pays off I think.
- DannoHung 12y ago
  You're incorrect. No individual letters of the alphabet are commands. Every symbol on the keyboard however is an operator (excluding semicolon, braces, brackets, and parens, which operate as line/expression terminators, function definitions, function invocation/array access, and list definitions, respectively).
- klibertp 12y ago
  It's the same thing I said in the recent J discussion: J (and probably Q) is meant to be read with a help of computer. Reading and writing J consists of incrementally building/decomposing expressions in a REPL. You have wonderful tools to visualize expressions structure in the REPL and you are expected to use them and to experiment with the expressions. You're not supposed to read it as prose, don't even try.
- fraserm 12y ago
  There is also an sql like interface in addition to the Q and K languages. This is probably easier to get started than diving into Q if that is too daunting.
- tiredandgrumpy 12y ago
  K is only hard if you try to read it without first studying it. Looping is achieved through adverbs. The key to it is understanding what is a noun (data), verb (operator/function) and adverb (takes a verb, creates a new verb to be used infix). A verb with a noun to its right is a dyad if there is also a noun to its left, and is otherwise a monad. If it is needed, the monad can be specified by appending a colon to the right of the symbol. Fortunately, most kdb+ developers program in Q, which has a bunch of helper routines defined in k, and assigns monads to names such as neg x instead of -:x.

fidotron 12y ago

This has come up before here, and the recent GNU APL stuff reminded me, but in summary, if you have ever been curious about APL or mildly suspicious of more conventional database approaches you owe it to yourself to take a look at the concepts at work here, especially primacy to columns instead of rows. The & "where" operator in raw k has stayed with me over the years as a particularly inspired way to deal with column based data.

profquail 12y ago
For those of you curious about array-based / columnar programming languages, there's an APL/J/K reddit: http://www.reddit.com/r/apljk http://www.reddit.com/r/apljk
tiredandgrumpy 12y ago
it came up here before, but this time is different. It is now free for commercial use and is not restricted with timeouts or expiry.

tom_b 12y ago

Ck out Arthur Whitney's abridged manual for fun: http://kx.com/q/d/kdb+.htm http://kx.com/q/d/kdb+.htm For using the 32-bit version (from Limits): 22 Limits Each database runs in memory and/or disk map-on-demand -- possibly partitioned. There is no limit on the size of a partitioned database but on 32-bit systems the main memory OLTP portion of a database is limited to about 1GB of raw data, i.e. 1/4 of the address space. The raw data of a main memory 64bit process should be limited to about 1/2 of available RAM.

jmnicolas 12y ago

Except financial applications what is it good for ?

deathflute 12y ago
This would actually make a terrific replacement for something like redis when you need a more structured schema. The q language is very powerful, and expressive - interesting mix of lisp and APL. You can do really powerful analytics without writing tons of code for it. You really have to see how fast KDB is compared to most nosql products out there.
- patrickxb 12y ago
  Are there any open source projects or blog posts with examples of this?
  - gohwell 12y ago
    They have a reference wiki with some example programs. Have a look at this page: http://code.kx.com/wiki/Cookbook/ProgrammingExamples http://code.kx.com/wiki/Cookbook/ProgrammingExamples
oddthink 12y ago
It's great for basic data-analysis tasks, where you just want to slurp in a few CSV files, join them together, filter out some rows, and spit out the results. Sure, you can do the same in R or python, but the whole process is very quick and easy in q.
jibberia 12y ago
Almost 10 years ago, I did an undergrad independent study at NYU contributing to some PhDs' Query by Humming music search engine. We used q to query a kdb full of catchy-melody time series data -- short sequences of "is this pitch higher, lower, or the same as the last?" and "is this note short, long, or medium?" (and, of course, gobs upon tons of variations as we iterated!). I barely did any q / kdb; only made a functional and usable UI, and did some prototyping of new ideas in other languages (Java, Max/MSP, Csound). I spent some time looking into q and was thoroughly baffled. Still am. It was really, really fast, though! As I vaguely understand and can explain it, the k/q system made it easy to do fuzzy searches and deal with missing pieces of data. If the user missed a note, or our pitch detection failed, or our source data was bad, we were still able to find matches. (Yes, I wish I'd been able to understand this more at the time. Bygones, now...)

miecio13 12y ago

I'm not sure about the Q language, but their C API reads like obfuscated C contest entry: http://kx.com/q/c/c/k.h http://kx.com/q/c/c/k.h

tiredandgrumpy 12y ago
If you look closely at it, there's not much there - it's actually easy to understand - defines a variant struct and an bunch of accessors to the different types within the embedded union. He prefers short names, and finally years later, java recommends short variable names for lambdas too!
- rcxdude 12y ago
  Short names make sense if they are easily understandable locally: This means either something extremely common throughout the codebase (I think the most common example being localisation wrappers for string literals. They should ideally be linked to a more clear explanation easily, e.g. from renaming import statements), or defined (clearly) and used only within a very small area of code. This API is neither of those.
  - silentbicycle 12y ago
    Short names also make sense if the same conventions appear over and over throughout the code base. While they can be quite opaque at first, they're very consistent, and people with experience in other APLs will recognize many of them.
- ryanobjc 12y ago
  So, I guess I need to look closer than the pixels then: typedef struct k0{signed char m,a,t;C u;I r;union{G g;H h;I i;J j;E e;F f;S s;struct k0k;struct{J n;G G0[1];};};}K; Sorry I guess I'm just not seeing the "not much there and actually easy to understand" Whatever a 'H' is
  - igorii 12y ago
    It definitely is strange to look at, but it's quite easy to understand if you know k/q. H is a short in k/q, so H refers to a short here as well. This naming scheme is true for all of the above letters.

nightTrevors 12y ago

For anyone trying this out for the first time, Jeff Borror's q for mortals is the best guide out there http://code.kx.com/wiki/JB:QforMortals2/contents http://code.kx.com/wiki/JB:QforMortals2/contents

noname123 12y ago

Currently using MongoDB for my historical quotes ticks database. Any peeps in trading use KDB+ in production or for fun think it's expressive enough to write queries directly to it for backtesting?

kthielen 12y ago

Careful with these guys. I once built an open source implementation of the q language, and these guys immediately threatened to sue me, my employer, and our clients. The language is not that interesting, it's easy to reproduce, and these guys will threaten you if you prove this.

tiredandgrumpy 12y ago
and now they'll sue you for libel ;-)
beagle3 12y ago
> The language is not that interesting I would say the language is very interesting. It is probably not interesting enough to get sued for, though .... I suspect times have changed - there are implementations that have been out there for years (https://github.com/kevinlawler/kona https://github.com/kevinlawler/kona implements k3 with sprinkles of k4, and http://althenia.net/kuc http://althenia.net/kuc implements an almost-k4 with a JIT and writable closures). IIRC, when you did your implementation it was when k4 was still a "technology preview" and not their main product (or was just released) - I remember understanding the panic in those action, even though I totally disagree with them. (I didn't know about the threats, but I do remember seeing it appear and disappear within a day, and assumed something was happening behind the scenes)