6 ms·

Show HN: PyEQS – query Elasticsearch like a Django Queryset

55 points by andrewgross 12y ago

jaegerpicker 12y ago
I'll have to play with it, currently we use the SearchQuerySet api provided by django_haystack to query Elasticsearch and it works pretty well for us.
- andrewgross 12y ago
  We used to use Haystack, but found it a bit too opinionated for us once we wanted to do some custom stuff. It is a bit more faithful to the Django queryset API, something we had to abandon to let us use more of the complex Elasticsearch query features.
coolsunglasses 12y ago
If you'd like something that tackles the challenge of making the Elasticsearch JSON API explicit: http://github.com/bitemyapp/bloodhound/ http://github.com/bitemyapp/bloodhound/ (Intended to act as a guide as well as a practical ES client)
- andrewgross 12y ago
  Thanks for sharing this, didn't know about it. Always nice to see how other people tackle the same issue.
  - coolsunglasses 12y ago
    Putting together and composing queries in ES is error prone and I have grown quite allergic to libraries that just write off the query/filter API ES provides as "json blob lol lol". You can lose massive amounts of time debugging a broken query in ES, due in no small part to a lack of an explicit spec. To respond to the mis-aimed comment: It's not about just beginning learning - it's a problem the moment you want to properly compose queries and not just have dumb templates. I am not new to ES, I've been using it for years. I understand wanting directness and the full breadth of the Elasticsearch API (thus moving away from Haystack), but not actually supporting anything in the API (which is the actual hard work, not wrapping an http-client) is problematic.
    - andrewgross 12y ago
      Are you referring to the drive to use more of a languages' built in operators to build queries, instead of my approach? I think the Mozilla library uses an approach like that. http://elasticutils.readthedocs.org/en/latest/ http://elasticutils.readthedocs.org/en/latest/
      - coolsunglasses 12y ago
        Yeah that looks to be closer to what I'd like, although it's still not type-safe.
  - andrewgross 12y ago
    Agreed. I had a rough time when I was first learning how to manually build queries as there are few examples of complex queries.
kevinastone 12y ago
Take a look at ElasticUtils from Mozilla. It also provides a queryset-inspired interface for elasticsearch. http://elasticutils.readthedocs.org/ http://elasticutils.readthedocs.org/
- andrewgross 12y ago
  Thanks, checking this out.
zmsmith 12y ago
I work with Andrew at Yipit. As he mentioned in a comment below, we're using features of Elasticsearch that aren't (and probably shouldn't be) exposed by haystack, and some of the query building started to get a little messy. I'm excited Andrew decided to work on this during our last hackathon and that now we're able to share it with the community. We've been really happy with Elasticsearch. Hopefully this library will help more people leverage Elasticsearch in their python projects.
lqdc13 12y ago
bad coding style in some places (e.g.: https://github.com/Yipit/pyeqs/blob/master/pyeqs/dsl/range.py https://github.com/Yipit/pyeqs/blob/master/pyeqs/dsl/range.p... where he redefines python range)
- andrewgross 12y ago
  Fixed. Completely missed that.
spencera 12y ago
We are currently working on an official QuerySet-like python library here at Elasticsearch: https://github.com/elasticsearch/elasticsearch-dsl-py https://github.com/elasticsearch/elasticsearch-dsl-py
- honzakral 12y ago
  Yes, the goal is similar - to provide a nice API without limiting any of the possibilities. It's much less restrictive (doesn't whitelist parameters) and tries not to force you to learn both Elasticsearch's query language and a DSL on top of by staying close to the original api. It's currently work in progress but you can already see in tests some of the features like chainable API, composing queries and filters and aggregations: https://github.com/elasticsearch/elasticsearch-dsl-py/blob/master/test_elasticsearch_dsl/test_search.py#L110-L118 https://github.com/elasticsearch/elasticsearch-dsl-py/blob/m... I try to work closely with the fine people at Mozilla currently maintaining ElasticUtils to make sure there is a nice migration path and I don't miss anything obvious. Any feedback would be more than welcome (github, twitter, email, irc #elasticsearch-py, carrier pigeon, ...)!
  - andrewgross 12y ago
    Thanks for sharing. Nice to see an alternative approach for defining these sorts of things.
    - honzakral 12y ago
      Happy to - note that the syntax in the search test is mostly just shortcut (though I fully expect that to be the main interface). You can also use the objects for individual query types: https://github.com/elasticsearch/elasticsearch-dsl-py/blob/master/test_elasticsearch_dsl/test_query.py#L52-L55 https://github.com/elasticsearch/elasticsearch-dsl-py/blob/m...
akoumjian 12y ago
Hi Yipit folks! It's Alec, from the Seattle Django group. We also developed something like this internally at Nimia due to the exact same use case: needing to access certain types of queries that didn't fit for haystack. Looks like we should have open sourced it so we could have collaborated.
andrewstuart 12y ago
Python 2 only?
- andrewgross 12y ago
  Good point, I will have to swap out some testing libs but should be possible.
jacquesc 12y ago
I use Ruby and am massively jealous of this. Anyone know of good Ruby alternatives? thanks!
- sprobertson 12y ago
  There's Plunk, a "human friendly" Elasticsearch DSL: https://github.com/elbii/plunk https://github.com/elbii/plunk
skkbits 12y ago
I am kinda newbie in search technology field. Can someone please explain what is difference between haystack, elastic search to me ? How does this compare to search engines like google , yahoo and others from application prospective ? Where to use haystack, elastic search and where not ?
sjaaktrekhaak 12y ago
Is the precommit hook[1] a custom one or is it an off-the-shelf Django recommended hook? Looks like it has some neat tricks and code style checking. [1]: https://github.com/Yipit/pyeqs/blob/master/pre-commit https://github.com/Yipit/pyeqs/blob/master/pre-commit
- andrewgross 12y ago
  It is a modified version of what we use at Yipit. I stripped out mostly things related to working with Django/Flask apps. It comes in very handy when making sure we maintain style and push fewer broken commits. Feel free to re-use.

55 points by andrewgross 12y ago

I'll have to play with it, currently we use the SearchQuerySet api provided by django_haystack to query Elasticsearch and it works pretty well for us.

andrewgross 12y ago
We used to use Haystack, but found it a bit too opinionated for us once we wanted to do some custom stuff. It is a bit more faithful to the Django queryset API, something we had to abandon to let us use more of the complex Elasticsearch query features.

coolsunglasses 12y ago

If you'd like something that tackles the challenge of making the Elasticsearch JSON API explicit: http://github.com/bitemyapp/bloodhound/ http://github.com/bitemyapp/bloodhound/ (Intended to act as a guide as well as a practical ES client)

andrewgross 12y ago
Thanks for sharing this, didn't know about it. Always nice to see how other people tackle the same issue.
- coolsunglasses 12y ago
  Putting together and composing queries in ES is error prone and I have grown quite allergic to libraries that just write off the query/filter API ES provides as "json blob lol lol". You can lose massive amounts of time debugging a broken query in ES, due in no small part to a lack of an explicit spec. To respond to the mis-aimed comment: It's not about just beginning learning - it's a problem the moment you want to properly compose queries and not just have dumb templates. I am not new to ES, I've been using it for years. I understand wanting directness and the full breadth of the Elasticsearch API (thus moving away from Haystack), but not actually supporting anything in the API (which is the actual hard work, not wrapping an http-client) is problematic.
  - andrewgross 12y ago
    Are you referring to the drive to use more of a languages' built in operators to build queries, instead of my approach? I think the Mozilla library uses an approach like that. http://elasticutils.readthedocs.org/en/latest/ http://elasticutils.readthedocs.org/en/latest/
    - coolsunglasses 12y ago
      Yeah that looks to be closer to what I'd like, although it's still not type-safe.
- andrewgross 12y ago
  Agreed. I had a rough time when I was first learning how to manually build queries as there are few examples of complex queries.

kevinastone 12y ago

Take a look at ElasticUtils from Mozilla. It also provides a queryset-inspired interface for elasticsearch. http://elasticutils.readthedocs.org/ http://elasticutils.readthedocs.org/

andrewgross 12y ago
Thanks, checking this out.

zmsmith 12y ago

I work with Andrew at Yipit. As he mentioned in a comment below, we're using features of Elasticsearch that aren't (and probably shouldn't be) exposed by haystack, and some of the query building started to get a little messy. I'm excited Andrew decided to work on this during our last hackathon and that now we're able to share it with the community. We've been really happy with Elasticsearch. Hopefully this library will help more people leverage Elasticsearch in their python projects.

lqdc13 12y ago

bad coding style in some places (e.g.: https://github.com/Yipit/pyeqs/blob/master/pyeqs/dsl/range.py https://github.com/Yipit/pyeqs/blob/master/pyeqs/dsl/range.p... where he redefines python range)

andrewgross 12y ago
Fixed. Completely missed that.

spencera 12y ago

We are currently working on an official QuerySet-like python library here at Elasticsearch: https://github.com/elasticsearch/elasticsearch-dsl-py https://github.com/elasticsearch/elasticsearch-dsl-py

honzakral 12y ago
Yes, the goal is similar - to provide a nice API without limiting any of the possibilities. It's much less restrictive (doesn't whitelist parameters) and tries not to force you to learn both Elasticsearch's query language and a DSL on top of by staying close to the original api. It's currently work in progress but you can already see in tests some of the features like chainable API, composing queries and filters and aggregations: https://github.com/elasticsearch/elasticsearch-dsl-py/blob/master/test_elasticsearch_dsl/test_search.py#L110-L118 https://github.com/elasticsearch/elasticsearch-dsl-py/blob/m... I try to work closely with the fine people at Mozilla currently maintaining ElasticUtils to make sure there is a nice migration path and I don't miss anything obvious. Any feedback would be more than welcome (github, twitter, email, irc #elasticsearch-py, carrier pigeon, ...)!
- andrewgross 12y ago
  Thanks for sharing. Nice to see an alternative approach for defining these sorts of things.
  - honzakral 12y ago
    Happy to - note that the syntax in the search test is mostly just shortcut (though I fully expect that to be the main interface). You can also use the objects for individual query types: https://github.com/elasticsearch/elasticsearch-dsl-py/blob/master/test_elasticsearch_dsl/test_query.py#L52-L55 https://github.com/elasticsearch/elasticsearch-dsl-py/blob/m...

akoumjian 12y ago

Hi Yipit folks! It's Alec, from the Seattle Django group. We also developed something like this internally at Nimia due to the exact same use case: needing to access certain types of queries that didn't fit for haystack. Looks like we should have open sourced it so we could have collaborated.

andrewstuart 12y ago

Python 2 only?

andrewgross 12y ago
Good point, I will have to swap out some testing libs but should be possible.

jacquesc 12y ago

I use Ruby and am massively jealous of this. Anyone know of good Ruby alternatives? thanks!

sprobertson 12y ago
There's Plunk, a "human friendly" Elasticsearch DSL: https://github.com/elbii/plunk https://github.com/elbii/plunk

skkbits 12y ago

I am kinda newbie in search technology field. Can someone please explain what is difference between haystack, elastic search to me ? How does this compare to search engines like google , yahoo and others from application prospective ? Where to use haystack, elastic search and where not ?

sjaaktrekhaak 12y ago

Is the precommit hook[1] a custom one or is it an off-the-shelf Django recommended hook? Looks like it has some neat tricks and code style checking. [1]: https://github.com/Yipit/pyeqs/blob/master/pre-commit https://github.com/Yipit/pyeqs/blob/master/pre-commit

andrewgross 12y ago
It is a modified version of what we use at Yipit. I stripped out mostly things related to working with Django/Flask apps. It comes in very handy when making sure we maintain style and push fewer broken commits. Feel free to re-use.