9 ms·
JMESPath – A query language for JSON
- zero_intp 9y agoThanks for taking the time to write this tool. Can you explain how it is distinguished from jq?
- maxsavin 9y agoThis looks interesting - but doesn't MongoDB basically achieve the same effect? I kind of prefer MongoDB because you query JSON with JSON - but I'm open to changing my mind :)
- skywhopper 9y agoIf you're using MongoDB already, then sure use MongoDB's query tools. But if you are just working with raw JSON from a potential variety of sources, or in a streaming context, then you need something more in-place and general-purpose, which this appears to be.
- eli 9y agoMay also be interested in the `jq` CLI, which on first glance appears to use a similar but not identical query language. https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- eddd 9y agoHow do this query language compares to jq?
- donpdonp 9y agoseconded. the jq language is surprisingly powerful and at first glance at jmespath, the syntax is similar.
- shabble 9y agoI think the equivalent jq query would be something like: [.locations[] | select(.state == "WA").name] | sort | join(", ") | { WashingtonCities: . }
- rnhmjoj 9y agoAlso this tool is pretty interesting: https://github.com/chrisdone/jl https://github.com/chrisdone/jl
- dozzie 9y agoI'd rather use App::RecordStream.
- crooked-v 9y agoThe example on the front page seems equivalent to (and only marginally less verbose than): locations .filter(l => l.state === 'WA') .map(l => l.name) .sort() .join(', ')
- chatmasta 9y agoThe benefit of a query language is that it can be described declaratively (i.e. in a non-executable text file, perhaps within JSON itself), and then programs written in any language can execute its query logic using a standard interpreter written in that specific programming language. So you get reusability of queries across the stack, in all languages that implement a parser against the spec. Your example only provides re-usability in JavaScript, and requires evaluating code at run-time so may not be suitable for queries based on user-submitted data in multi-tenant environments.
- deleted 9y ago[deleted]
- steve_adams_86 9y agoI really appreciate this comment. I was trying to figure out why I wouldn't use native data types and functions, but this makes it clear. In your opinion, where would someone be storing the json such that they'd benefit from a tool like this? The only time I use json outside of pulling it from an API (where I can convert it to a native object) is probably storing it in postgres, where I've already got json querying tools.
- chatmasta 9y agoSome cases off the top of my head: - Infrastructure configuration stored in JSON. Query could reference other JSON files, or the JSON file itself (loops would need to be considered). - Declarative reactive programming, e.g. platforms like IFTT. You might want to take certain actions based on data in a JSON post. The IFTT GUI would create JSON config files that its server side parsers can safely use without eval'ing code to decide which action to take. - Adding conditional logic to jsonschema form generation. Recently I've built a questionnaire renderer in react that renders forms based on jsonschema. The user creates forms with a GUI, which compiles them to JSON, and then the renderer knows how to render. Conditional logic (e.g. question B is required if question A === true) can be quite limited when constrained to pure JSON. Something like this could help with that. The nice thing about declarative syntax is you can build a GUI to generate it, so users never use the JSON itself, but you can store it in a database, safely execute rules based on it, etc. without requiring programming from the user. That said, there are usually better ways to accomplish this, like in pure JSON for example. Mongo syntax achieved this, with declarative operators like $or{}, ${sum}, etc., but it can be quite cumbersome.
- avoidwork 9y agowhy choose this over jsonpath (like xmlpath) or jq?
- deleted 9y ago[deleted]
- nimish 9y agojq exists, is fast, and works well. Is this compatible?
- deleted 9y ago[deleted]
- fiddlerwoaroof 9y agoNo, and I generally like jq’s syntactic choices better.
- xchaotic 9y agoThe reinvention of XML in JSON is almost complete - JMESPath vs XPath, JSON Schema vs XML Schema etc. If you need semi structured data to that level, consider using XML instead - you can validate it, there's plenty of tools, it's very stable and mature etc
- btown 9y agoBut can you store XML in a database that is Web Scale?? /s
- edraferi 9y agoMarklogic.... sort of.
- dmitriid 9y agoYes. It's one of the many abilities a database like PostgreSQL has. The "Web scale" is what MongoDB tried to sell (unfortunately, successfully), and it's not even close to being web scale.
- lrem 9y agoSQLite will happily store whatever you throw at it and scales perfectly to my Web Site ;)
- bufferoverflow 9y agoWe still need XSLT for JSON to complete the circle. Though it seems like XSLT should support JSON: https://www.w3.org/TR/xslt-30/#json https://www.w3.org/TR/xslt-30/#json
- blattimwind 9y agoI'm rather disappointed its not called JPath.
- Ardren 9y agoUnsurprisingly there are already multiple projects called JPath
- medleybron 9y agothis, combined with GraphQL would be awesome
- ex3ndr 9y agoDoes someone knows nice human-friendly search language like the one in Jira, Slack, etc?
- dmoreno 9y agoThere is also JSON Pointer (https://tools.ietf.org/html/rfc6901 https://tools.ietf.org/html/rfc6901) from IETF. Very simple standard and easy to implement, but not as powerful as jmespath nor jq.
- bringtheaction 9y agoTo say that it’s “from IETF” is sort of a misnomer I think. RFCs are submitted to IETF by others, not by IETF itself.
- xonix 9y agoMy tiny lib with very similar functionality: [1]. The query syntax is slightly different though. Also I decided to re-use JS for evaluation of sub-expressions instead of implementing own full-fledged parser. [1] https://github.com/xonixx/jsqry https://github.com/xonixx/jsqry
- lioeters 9y agoI love libraries like this, which is small enough to be read in one sitting. I can scan through and get a general understanding of everything that it does. The "evaluation of sub-expressions" made me curious. This line: token.func = Function('_,i,args', 'return ' + token.val); ..could be a potential security issue with user-submitted expressions?
- xonix 9y agoThanks. I doubt this could be a security issue. Typical usage like so var name = one(users, '[_.id==?].name', 123) uses parameterized queries, same idea as with SQL to eliminate injections.
- lioeters 9y agoI see, I should have dug deeper before commenting. Wow, parameterized queries, there's been a lot of thought put into this compact library!
- gingerlime 9y agoI love JMESPath. I first discovered it when using the AWS CLI `--query` option[0]. I then realized that using it in my code would make things much more declarative and easy to grok than a bunch of maps, filters etc. Here's a real example which I think illustrates it[1]. It has libraries for lots of languages with a clear specification/compliance test[2]. The cherry on the top is the interactive query on the website. You can tweak any of the examples (both queries and data) and get results instantly. Extremely useful for playing around, building queries to work with JSON data (webhooks, API responses etc) [0] https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html https://docs.aws.amazon.com/cli/latest/userguide/controlling... [1] https://gist.github.com/gingerlime/757c7b4778c1ab68605dfce66ceb8378 https://gist.github.com/gingerlime/757c7b4778c1ab68605dfce66... [2] http://jmespath.org/libraries.html http://jmespath.org/libraries.html
- danbruc 9y agoDoes this have any mathematical foundation like the relational algebra for SQL? Or more generally, does a mathematical framework exist to treat this or similar constructs and that goes beyond what relational algebra provides and that, for example, also handles aggregate functions? The reason I am asking is that I am currently trying to build a tool to analyze a kind of time series data, think log file entries, in order to look for anomalies and visualize them. I could of course just build all the transformations I am interested in in an ad hoc fashion but it would be nice to have a mathematical framework in order to start out with a small set of basic operations and then compose those while having some guarantees about the expressiveness of that the basic operations and ideally also a rigorous foundation for transforming them, for example for performance optimizations. But so far I was unable to find something that seems fitting, everything I am aware of is either to limited like relational algebra or way to general like general functions. It feels like what I am looking for should exist but I am unable to find it.
- mjburgess 9y agoload the data into a database (sqlite is very good). use the database. delete the database.
- krapp 9y agoMight as well just keep everything in a relational database to begin with, in that case, and generate JSON from the query results.
- spraak 9y agoWhat reasons did you not go with SQL itself? I may not fully understand what you're trying to do, but in any case it sounds really interesting.
- danbruc 9y agoI never tried it but I am expecting the performance to be not good enough, it takes already several minutes with code specifically written to perform the calculations I am interested in. And because I don't know what exactly I am looking for I need more or less interactive speed so that I can try out many different ways to look at the data. But maybe I could use [materialized] views to convey enough information to the query planner how to efficiently carry out the calculations or maybe I am even underestimating how good query planners are. I just have the gut feeling that performing a lot of aggregation will make a database perform a lot of unnecessary work. But maybe I should and will try loading the data into SQL Server and see what happens. The other thing is that SQL seems not the best fit to me. Say you just want to know how many events occurred in the last three months in any hour, that is straight forward grouping and counting at first, but already rounding the timestamps to an hour is not as obvious as it should be. But if there was no event in a specific hour, your result will just have no row for that hour instead of a row saying there were zero events in that hour. This in turn will cause more trouble if you want to build a histogram showing in how many hours there were say 0 to 9, 10 to 19, 20 to 29, and so on events. Certainly still doable with SQL but we are already entering the territory where writing a single query will take most people several hours to get the desired result. I also couldn't easily tell how to express calculating the 99th percentile of the event size for every day of the week and hour of the day. I am pretty sure it is possible but I guess it would also be pretty unreadable unless you put in quite a bit of effort to create utility functions instead of hacking together one huge SQL statement. Then again I don't really know much about the more recent SQL features for partitioning and aggregating, maybe I should have a closer look at that first.
- thrownaway954 9y agothe second I looked at the example on the homepage and saw this: sort(@) I'm like nope! What is this "@" symbol? Why can't that be "name"? I'm already passing judgment that this library will be a nightmare to use which isn't good. Now I know I can read the docs and eventually what I can pass the sort expression and what it all means, however, this is an issue I come across more and more with new libraries in programming... show simple examples, not "smart" or complicated ones. I shouldn't have to read through docs to try to decipher an introductory example. There is a reason every programming language starts with "Hello World".
- Terretta 9y agoThis is not that new. And it happens to be the JSON query language built in to the AWS CLI tools. For last 3 - 4 years I see lots of AWS CLI examples piping through jq. That’s an extra dependency that’s not necessary when this is built in. Here’s the author’s idea of intro materials (posted Jan 2015): http://jamesls.com/how-to-easily-explore-jmespath-on-the-command-line.html http://jamesls.com/how-to-easily-explore-jmespath-on-the-com... I’ve found him very responsive on bugs and (provably useful) feature ideas.
- mjburgess 9y agoEach step is a transformation X | Y | Z The @ is the result from the prior transform. I would've preferred _ from scala, but meh.
- johnhenry 9y agoBesides's this projects cli, jp (https://github.com/jmespath/jp https://github.com/jmespath/jp), I see jl (https://github.com/chrisdone/jl https://github.com/chrisdone/jl) and jq (https://github.com/stedolan/jq/ https://github.com/stedolan/jq/) in the comments. I wonder if anyone has had experience with all three (or even just one) and can comment on their experiences?
- zero_intp 9y agoI use jq on simple projects and to prototype. It is lightweight, fast, and usually consistent in operation.
- vthriller 9y agoI only have used jq, but I noticed that even AWS CLI docs [0] (already mentioned in the comments a couple of times) suggest to use jq for "more advanced features that may not be possible with --query". [0] https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html#controlling-output-filter https://docs.aws.amazon.com/cli/latest/userguide/controlling...
- gregmac 9y agoI've been using jq quite a bit with AWS CLI inside various bash scripts. It allows manipulation and filtering the native api doesn't, which makes it pretty straight forward to get certain values - such as specific tags on the current instance (converted to key-value form, or to turn into bash variables), a list of other systems in the current auto scaling group, etc. I've also started using it just to get colorized/formatted output from curl or a local json file. cat file.json | jq
- billsmithaustin 9y agoI use jq frequently. It covers most of what I want to do.
- ak217 9y agojq is by far the best developed and has the most intuitive syntax, but it doesn't have a formal spec for its language. I have been maintaining https://github.com/kislyuk/yq https://github.com/kislyuk/yq, which wraps jq with a transcoder for YAML and XML.
- jnordwick 9y agoI remember when xml started down this road too. "We aren't going to be sgml, but just a lightweight markup. Xpath, xslt, etc. And now we have xml today, the modern day sgml. But this time it's different?
- rozzie 9y agoFor those interested in arbitrarily transforming JSON objects (for example, in a communications pipeline) I’d recommend JSONata. It’s quite useful and we’re well along in a Golang port with $function extensibility. http://jsonata.org http://jsonata.org
- cookiecaper 9y agoAgreed. I like JSONata a lot, even though it's the dark horse among JSON traversal languages. I've had a good experience parsing semi-unstable JSON with it.
- jjuhl 9y agoSee also https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- wooby 9y agoA worthwhile alternative to this approach (a JSON-specific query language) is a language for converting JSON structures to newline-delimited records. Then, standard shell tools can be used to query and join: https://github.com/micha/json-table https://github.com/micha/json-table
- octref 9y agoI built this VS Code plugin to convert JSON interactively using JMESPath: https://marketplace.visualstudio.com/items?itemName=octref.vscode-json-transform https://marketplace.visualstudio.com/items?itemName=octref.v... Might be useful if you are testing API or playing with JSON data.
- selrond 9y agoWell actually - your plugin has led me to find JMESPath and post the link to HN :D
- maltalex 9y agoI want to add one more into the mix - Couchbase has something called "N1QL" ('nickel'), which is actual SQL adapted for JSON: https://www.couchbase.com/products/n1ql https://www.couchbase.com/products/n1ql It's not standalone though, you need Couchbase to use it.
- justin_oaks 9y agoI like JMESPath, but it has some serious limitations which prevent it from being as general purpose as jq. JMESPath limitations: - No simple if/else, but it is possible using a hack, documented below. - The to_number function doesn't support boolean values, but it is possible using a hack, documented below. - can't reference parents when doing iteration. Why? All options for iteration, [* ] and map, all use the iterated item as the context for any expression. There's no opportunity to get any other values in. May be possible for a fixed set of lengths. Something akin to the following (except there is no syntax for switching or if statements): switch (length): case 1: [expression[0]] case 2: [expression[1], expression[1]] case 3: [expression[0], expression[1], expression[2]] ... - Key name can't come from an expression. Why? The ABNF for constructing key-value pairs is given as: keyval-expr = identifier ":" expression. The key is an identifier, which gives no possibility for making it an expression. No functions modify keys in such a way as to allow using an expression as a key.) - No basic math operations, add, multiply, divide, mod, etc. Why? Nobody added those operators/functions. - There's a join, but no split. - No array indexing based on expression. Why? Indexing is done based on a number or a slice expression, which also doesn't support expressions. Here's the ABNF: bracket-specifier = "[" (number / "* " / slice-expression) "]" / "[]" - No ability to group_by an expression. - No ability to get the index of an element in a list Hacks: Convert true/false to number: boolean_expression && `1` || `0` If/else: Option 1) [{q:CONDITIONAL_EXPRESSION, v:IF_RESULT_EXPRESSION},{q:!COND_EXPRESSION,v:ELSE_RESULT_EXPRESSION}][?q]|[0].v Option 2) {"if":CONDITIONAL_EXPRESSION, "ctx":@} | [{q:if ,v:ctx.IF_EXPRESSION},{q:!if,v:ctx.ELSE_EXPRESSION}][?q]|[0].v
- adamretter 9y agoSo no mention of JSONiq which came before JMESPath?
- hoppelhase 9y agoThere is also JsonPath. http://goessner.net/articles/JsonPath/ http://goessner.net/articles/JsonPath/
- granitosaurus 9y agoMan I hate that every json parser has the need to introduce their own language. We already have a bunch of well adopted and extremely powerful languages such as Xpath, why reinvent the wheel? I recently made a simple tool for enabling xpath parsing for json https://github.com/Granitosaurus/pq https://github.com/Granitosaurus/pq - it just converts json to xml and lets you go at it with xpath or css selectors. Easy, beautiful and convient, except it's really slow with bigger files :|
- nurettin 9y agoI already have a query language for json. I insert json into mssql 2017 community edition and query it there. https://docs.microsoft.com/en-us/sql/relational-databases/json/json-data-sql-server https://docs.microsoft.com/en-us/sql/relational-databases/js...
- danvk 9y agoI'm sad that JSONSelect (https://github.com/lloyd/JSONSelect https://github.com/lloyd/JSONSelect) never caught on. It uses CSS selectors to query JSON, which has the nice side effect that learning to use it improves your CSS as well!
- syats 9y agoOne more alternative to many listed here is SPARQL-Generate. A single query language that works for XML and JSON, and has syntax borrowed from SPARQL. https://ci.mines-stetienne.fr/sparql-generate/playground.html https://ci.mines-stetienne.fr/sparql-generate/playground.htm...