6 ms·

Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here) Just re

Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here) Just recently we merged support a new experimental index scheme called 'scorch'. This new index scheme is designed from the ground up to reduce index size and improve performance. It features: - a segment based approach, much like Lucene - vellum FTS for the term dictionary - https://github.com/couchbase/vellum https://github.com/couchbase/vellum - roaring bitmaps for the postings lists - https://github.com/RoaringBitmap/roaring https://github.com/RoaringBitmap/roaring - and compressed chunked integer storage for all the posting details It's still experimental at this point, but shows considerable indexing speedup, index size reduction, and similar query performance to the old index format used today. The code for this new index scheme can be found here: https://github.com/blevesearch/bleve/tree/master/index/scorch https://github.com/blevesearch/bleve/tree/master/index/scorc...

oelmekki 9y ago
Hi, There's a question I don't find an answer for by looking at homepage and documentation: does it handle concurrent queries? (I wonder, since I see a store is a single file) That is, is it something akin sqlite, meant to be used as an embedded engine for standalone applications, or is it fit to be used by a centralized api?
- mschoch 9y ago
  Concurrent queries are supported (not sure what you mean by store being a single file, it is a directory of many files). Concurrent indexing is also possible, so long as you can arrange to not put duplicate document ids into batches executing concurrently. As for usage, it is just a library, so it is typically embedded in a single process (though this can serve multiple clients concurrently). Distributing the index across multiple nodes is done at the application level. At Couchbase we do this with bleve in a separate project called 'cbft'. https://github.com/couchbase/cbft https://github.com/couchbase/cbft
  - oelmekki 9y ago
    > (not sure what you mean by store being a single file, it is a directory of many files) I see, my bad. I saw one binary file named "store" in the directory created by the example code, I thought it would be it. Thanks for explanation!
SAI_Peregrinus 9y ago
Since I didn't see it with a quick look, why call it bleve? Given the logo it's clearly a reference to Boiling Liquid Expanding Vapor Explosion, but that seems an odd choice of name with no relation to the project. Do you just like fire?
- mschoch 9y ago
  I was watching one of those engineering disaster shows and thought it would make a good name for a project. I didn't find any other software projects using the name, and it seemed like it would have decent googleability. The relationship to fire/explosions has given good themes for logos and sub-project names (like scorch).
Denzel 9y ago
The reason is most likely because they saw this on the front-page: https://news.ycombinator.com/item?id=16085873 https://news.ycombinator.com/item?id=16085873. I've noticed that people like to submit items that are tangentially related to those that popup on the front page.
wejick 9y ago
Are there any plan to have sharding mechanism implemented on the blevesearch? IMHO it should be on the application level, but having this on blevesearch would be good. Regarding segments based approach like lucene, means we will need to do segments merging which in my experience quite resource intensive. I don't actually know how blevesearch handles this prior segment based approach.
- millisecond 9y ago
  Not directly sharding but I've been working on a fork of Bleve that uses Cassandra as the backing store which allows for horizontal scaling of a single index: https://github.com/wrble/flock https://github.com/wrble/flock Still very much a work in progress but the core is functional.
  - ddorian43 9y ago
    see elassandra for something similar
- mschoch 9y ago
  Bleve has support for querying across multiple indexes (shards) but does not prescribe any mechanism to split the data. So, it's up the application to divide the data how it sees fit, but you can use Bleve functionality to execute the same query across multiple indexes and merge the results. Merging is required and is indeed somewhat resource intensive. Bleve's current indexing approach has no segments, instead all index data is serialized into a key/value store. This approach allowed us to experiment and plug-in a variety of implementations. Unfortunately, the key/value abstraction limits the way you interact with data, so there are a number of drawbacks. One key gain we get from the segmented approach vs the key/value store approach is that we no longer need to maintain a backindex to handle updates/deletes.
  - wejick 9y ago
    Yes and if I'm not mistaken that's what alias on bleve for. Not yet looking into scorch. So scorch would replace other storage engine like rocksdb and leveldb?

oelmekki 9y ago

Hi, There's a question I don't find an answer for by looking at homepage and documentation: does it handle concurrent queries? (I wonder, since I see a store is a single file) That is, is it something akin sqlite, meant to be used as an embedded engine for standalone applications, or is it fit to be used by a centralized api?

mschoch 9y ago
Concurrent queries are supported (not sure what you mean by store being a single file, it is a directory of many files). Concurrent indexing is also possible, so long as you can arrange to not put duplicate document ids into batches executing concurrently. As for usage, it is just a library, so it is typically embedded in a single process (though this can serve multiple clients concurrently). Distributing the index across multiple nodes is done at the application level. At Couchbase we do this with bleve in a separate project called 'cbft'. https://github.com/couchbase/cbft https://github.com/couchbase/cbft
- oelmekki 9y ago
  > (not sure what you mean by store being a single file, it is a directory of many files) I see, my bad. I saw one binary file named "store" in the directory created by the example code, I thought it would be it. Thanks for explanation!

SAI_Peregrinus 9y ago

Since I didn't see it with a quick look, why call it bleve? Given the logo it's clearly a reference to Boiling Liquid Expanding Vapor Explosion, but that seems an odd choice of name with no relation to the project. Do you just like fire?

mschoch 9y ago
I was watching one of those engineering disaster shows and thought it would make a good name for a project. I didn't find any other software projects using the name, and it seemed like it would have decent googleability. The relationship to fire/explosions has given good themes for logos and sub-project names (like scorch).

Denzel 9y ago

The reason is most likely because they saw this on the front-page: https://news.ycombinator.com/item?id=16085873 https://news.ycombinator.com/item?id=16085873. I've noticed that people like to submit items that are tangentially related to those that popup on the front page.

wejick 9y ago

Are there any plan to have sharding mechanism implemented on the blevesearch? IMHO it should be on the application level, but having this on blevesearch would be good. Regarding segments based approach like lucene, means we will need to do segments merging which in my experience quite resource intensive. I don't actually know how blevesearch handles this prior segment based approach.

millisecond 9y ago
Not directly sharding but I've been working on a fork of Bleve that uses Cassandra as the backing store which allows for horizontal scaling of a single index: https://github.com/wrble/flock https://github.com/wrble/flock Still very much a work in progress but the core is functional.
- ddorian43 9y ago
  see elassandra for something similar
mschoch 9y ago
Bleve has support for querying across multiple indexes (shards) but does not prescribe any mechanism to split the data. So, it's up the application to divide the data how it sees fit, but you can use Bleve functionality to execute the same query across multiple indexes and merge the results. Merging is required and is indeed somewhat resource intensive. Bleve's current indexing approach has no segments, instead all index data is serialized into a key/value store. This approach allowed us to experiment and plug-in a variety of implementations. Unfortunately, the key/value abstraction limits the way you interact with data, so there are a number of drawbacks. One key gain we get from the segmented approach vs the key/value store approach is that we no longer need to maintain a backindex to handle updates/deletes.
- wejick 9y ago
  Yes and if I'm not mistaken that's what alias on bleve for. Not yet looking into scorch. So scorch would replace other storage engine like rocksdb and leveldb?