7 ms·
Thank you for your feedback! I understand your point of view, let me share mine. spyql is 100% Python code and it is not a thin layer over something else. Ever
by dmoura 4y ago
Thank you for your feedback! I understand your point of view, let me share mine.
spyql is 100% Python code and it is not a thin layer over something else. Every row of data goes through a query engine built in python that takes care of evaluating the query, filtering and aggregating data, among other stuff. The only part that is offloaded to standard or external modules is the decoding and encoding from/to specific data formats. In the case of this benchmark, spyql uses the orjson module to convert each input json object into a python dict, one at a time.
Due to the nature of Python as an interpreted language, it is natural that python modules leverage C (or Rust) to provide highly efficient implementations of core functionalities. For instance, the json module of the standard library is implemented in C. If we would use the json module in the benchmark instead of orjson, spyql would remain as one of the fastest and lightest tools for querying json data. Using orjson, gives an extra boost of performance.
If you think it is worthwhile, I can add another benchmark entry where spyql uses the standard json lib. The queries would be exactly the same, I just need to use in the query `FROM json` instead of `FROM orjson`.
- davidatbu 4y agoI honestly totally agree with your POV now. I don't think another benchmark entry would be worthwhile either. Thank you for making this tool again!