7 ms·

I'd like to see some real metrics. All the words are subjective and the technical ones are common to other databases as well. How many results can it return in

I'd like to see some real metrics. All the words are subjective and the technical ones are common to other databases as well. How many results can it return in how many seconds from a table with how many rows? Can it do full text indexing? Can it aggregate? It feels like something from wallstreet, the language is almost pump and dump "I have this really great stock tip from this guy who made millions once, he's a legend. He says buy, so you should." We need more tangible and quantifiable information about this database. Until then it's not even hype it's vaporware.

kick 7y ago
Calling anything from Arthur "vaporware" is really laughable. This article isn't great, but it's from a finance site, not a technical site: of course it's going to be low on technical details. You would think that people who want technical details would know how to use a search engine. Further, kdb/kdb+ (which Shakti is the successor of) is a columnar-based database. "How many results can it return [...] from a table with how many rows?" is completely irrelevant. There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled.
- deleted 7y ago
  [deleted]
- mhd 7y ago
  > Calling anything from Arthur "vaporware" is really laughable. kOS?
  - beagle3 7y ago
    It was a research project, and it actually booted on bare metal and ran what it was supposed to (milspec bare bones os with gui, filesysten, db, etc in a few hundred KB)
    - sansnomme 7y ago
      Do you have a link? This sounds fascinating.
      - chrispsn 7y ago
        https://gist.github.com/chrispsn/da00835bb122c42f429a084df83fd746 https://gist.github.com/chrispsn/da00835bb122c42f429a084df83...
  - kick 7y ago
    kOS exists, at least one HN user has used it ('geocar). As a research operating system, seeing it released into the public was never a likely goal, and as far as I know no one claimed it would be..
- iskander 7y ago
  >There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled. Eh, I worked on array languages in grad school and the speed of k was significantly over-hyped. It's trivial to beet with Python and no contest if you want to work in a compiled language. Wall Street develops weird religions around tech.
  - beagle3 7y ago
    In my experience it is mostly trivial to beat with C if you adopt columnar storage (which isn’t idiomatic), but even numexpr doesn’t beat k (though it seems like it should), and numpy is still way behind; pandas and plain python not even in the same ballpark. What approach did you use? The one thing nothing beats k on is functionality per time spent (if you are proficient). It is often 100 times shorter but only takes 10 times as long to write.
    - iskander 7y ago
      I'm going to sidestep arguments about its terseness since some people are into that and it's really an aesthetic choice. But for machine learning or numerical algorithms, I found it quite hard to make k nearly as fast as vanilla NumPy code. Overall there's really no good reason for it to be faster. It didn't have (as of ~2011) a JIT or any mechanism for operator fusion, so you end up allocating many intermediates, sometimes using quadratic space which gets immediately reduced along some axis. Yeah, I know "Arthur Whitney is really smart"...but that's not actually a technical reason why k would be faster. I don't remember the specifics but I think that many of the array operators weren't even multi-threaded and I remember only a few of the array operators getting FLOPs that would correspond to SIMD acceleration. So, for the most part, it seemed like a lot of vanilla single threaded C code behind the operators. It's possible that the implementation got smarter over the past decade, but when I was working in the space it seemed like the hagiography and actual performance were worlds apart.
      - chrispsn 7y ago
        At the least, 'range' (!3 -> 0 1 2) is now deferred (including with offsets eg 1+!3).
- anon1m0us 7y ago
  Shakti themselves used rows as the comparison metric in the original link: https://shakti.com/database-software-history/ https://shakti.com/database-software-history/ Again, saying the speed of k is unrivaled is hype speak. You're adding negative value to the discussion.
  - zerozerocool 7y ago
    To give you an idea, here are some benchmarks comparing Python (with pandas/numpy) vs k (aka q/kdb+). https://www.linkedin.com/pulse/data-analysis-example-python-q-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/data-analysis-example-python-... https://www.linkedin.com/pulse/lists-python-q-side-by-side-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/lists-python-q-side-by-side-f... I think that software history page is meant to convey general workloads achievable, rather than performance. -(not the author of the benchmarks nor an employee of shakti FWIW)
- typon 7y ago
  It really depends on what specific benchmarks you're talking about, but I would be really interested in seeing benchmarks in which k beats C with AVX

kick 7y ago

Calling anything from Arthur "vaporware" is really laughable. This article isn't great, but it's from a finance site, not a technical site: of course it's going to be low on technical details. You would think that people who want technical details would know how to use a search engine. Further, kdb/kdb+ (which Shakti is the successor of) is a columnar-based database. "How many results can it return [...] from a table with how many rows?" is completely irrelevant. There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled.

deleted 7y ago
[deleted]
mhd 7y ago
> Calling anything from Arthur "vaporware" is really laughable. kOS?
- beagle3 7y ago
  It was a research project, and it actually booted on bare metal and ran what it was supposed to (milspec bare bones os with gui, filesysten, db, etc in a few hundred KB)
  - sansnomme 7y ago
    Do you have a link? This sounds fascinating.
    - chrispsn 7y ago
      https://gist.github.com/chrispsn/da00835bb122c42f429a084df83fd746 https://gist.github.com/chrispsn/da00835bb122c42f429a084df83...
- kick 7y ago
  kOS exists, at least one HN user has used it ('geocar). As a research operating system, seeing it released into the public was never a likely goal, and as far as I know no one claimed it would be..
iskander 7y ago
>There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled. Eh, I worked on array languages in grad school and the speed of k was significantly over-hyped. It's trivial to beet with Python and no contest if you want to work in a compiled language. Wall Street develops weird religions around tech.
- beagle3 7y ago
  In my experience it is mostly trivial to beat with C if you adopt columnar storage (which isn’t idiomatic), but even numexpr doesn’t beat k (though it seems like it should), and numpy is still way behind; pandas and plain python not even in the same ballpark. What approach did you use? The one thing nothing beats k on is functionality per time spent (if you are proficient). It is often 100 times shorter but only takes 10 times as long to write.
  - iskander 7y ago
    I'm going to sidestep arguments about its terseness since some people are into that and it's really an aesthetic choice. But for machine learning or numerical algorithms, I found it quite hard to make k nearly as fast as vanilla NumPy code. Overall there's really no good reason for it to be faster. It didn't have (as of ~2011) a JIT or any mechanism for operator fusion, so you end up allocating many intermediates, sometimes using quadratic space which gets immediately reduced along some axis. Yeah, I know "Arthur Whitney is really smart"...but that's not actually a technical reason why k would be faster. I don't remember the specifics but I think that many of the array operators weren't even multi-threaded and I remember only a few of the array operators getting FLOPs that would correspond to SIMD acceleration. So, for the most part, it seemed like a lot of vanilla single threaded C code behind the operators. It's possible that the implementation got smarter over the past decade, but when I was working in the space it seemed like the hagiography and actual performance were worlds apart.
    - chrispsn 7y ago
      At the least, 'range' (!3 -> 0 1 2) is now deferred (including with offsets eg 1+!3).
anon1m0us 7y ago
Shakti themselves used rows as the comparison metric in the original link: https://shakti.com/database-software-history/ https://shakti.com/database-software-history/ Again, saying the speed of k is unrivaled is hype speak. You're adding negative value to the discussion.
- zerozerocool 7y ago
  To give you an idea, here are some benchmarks comparing Python (with pandas/numpy) vs k (aka q/kdb+). https://www.linkedin.com/pulse/data-analysis-example-python-q-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/data-analysis-example-python-... https://www.linkedin.com/pulse/lists-python-q-side-by-side-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/lists-python-q-side-by-side-f... I think that software history page is meant to convey general workloads achievable, rather than performance. -(not the author of the benchmarks nor an employee of shakti FWIW)
typon 7y ago
It really depends on what specific benchmarks you're talking about, but I would be really interested in seeing benchmarks in which k beats C with AVX