7 ms·
My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text sea
by vimda 5y ago
My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case
- nerdponx 5y agoThis has been my experience. Obviously different people use logs for different things, but in my case I'm usually looking for information about something bad that already happened, within a very specific window of time, and within a specific section of the application. 99% of the time, that means I am filtering until there are only a handful of entries that match, at which point I don't need full text search at all.
- patrakov 5y agoI am not really sure about this. A few days ago, a colleague asked me why a certain Google cloud instance does not exist. I did not know either, so I searched for this name in the Google audit log, and found when and by whom it was decommissioned. But it was a full-text search, given the instance name. I probably could do it (in theory) as a field match, if I knew which field it was, and which format it was in (I am talking about project/abc/location/xyz type of junk that precedes the actual instance name). And yes it was slow (this instance was deleted months ago, and Google tries to search the most recent logs first).
- nerdponx 5y agoThis sounds like the 1% of my experiences not served by filtering. Naturally your experiences will be different from mine!
- willvarfar 5y agoCompletely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.
- dd82 5y agoYou can update the index with the new field specification and reindex your content. Your complaint really doesn't make sense, how would you query an unindexed field? Elasticsearch is a _search_ engine, which means it needs to index content that is to be discoverable. What you're saying with unindexed fields is you're completely fine with those not being included in any search or filtering.
- dvdkon 5y agoExactly, it's a search engine. It probably doesn't make sense to use it as a storage engine for logs unless you need to search all of them efficiently.
- barrkel 5y agoYou use the index to identify a subset of records and scan those for unindexed criteria. It's ok to fail if the indexed criteria are not selective enough. In fact it's usually preferable to a long timeout.
- da_chicken 5y agoYour response is, "Why can't you perfectly predict which columns/keywords will be necessary later on, or otherwise re-index the whole system at the drop of a hat for one query? And why would you think a search engine would be able to perform an unindexed, ad hoc search?" Compared to my experience, you have a foundational difference of understanding with how systems are actually used.
- lathiat 5y agoI guess what they want is to use the elasticsearch query language but let it optionally do “expensive” non indexed filtering like a SQL database would let you do. Without knowing for sure I imagine they originally expected the application side to handle this but many of the current solutions don’t do that. And they expose and overload the elastic search query language as the primary search interface with no additional app logic. The elastic search query “is” the search application. Making some assumptions but might reconcile the different viewpoints on why it does or doesn’t make sense.