4 ms·

Show HN: Code Indexer Loop

28 points by ricklamers 3y ago

This is a project I recently worked on at Definitive. We’re hoping to make it easier to semantically query (changing) source code files. We’d love to see people use it as a building block. If you have any suggestions please leave a comment or open a GitHub issue!

_boffin_ 3y ago
From the initial look, it looks like you're taking the work of SweepAI, a source-available repo, which gave(?) it to LlamaIndex(1) and then you're just providing a wrapper around the LlamaIndex that provides a way to do an Update statement. 1) https://docs.sweep.dev/blogs/chunking-improvements https://docs.sweep.dev/blogs/chunking-improvements
- deleted 3y ago
  [deleted]
- ricklamers 3y ago
  Sweep is mentioned as attribution in multiple places a) https://github.com/definitive-io/code-indexer-loop#attribution https://github.com/definitive-io/code-indexer-loop#attributi... b) https://github.com/definitive-io/code-indexer-loop/blob/fd9d4a65e475e9f41e782af7cedea3d0bc675af5/code_indexer_loop/code_splitter.py#L3 https://github.com/definitive-io/code-indexer-loop/blob/fd9d... The difference is packaging it as a consumable PyPI package that can easily be used in a project (they even call out for separating this out into a stand alone project but that they lack the time to do so: https://docs.sweep.dev/blogs/chunking-2m-files#future- https://docs.sweep.dev/blogs/chunking-2m-files#future- ) In addition, we expand and fix the implementation, for example it now supports limiting on token count instead of character count, and we fix some white space inconsistencies in parsing/chunk reconstruction.
  - _boffin_ 3y ago
    Looking forward to diving further in and seeing how you handle white spaces issues as I’ve had massive issues with that already
    - ricklamers 3y ago
      Please let me know if anything isn’t as expected and I’ll try to look into it
funcDropShadow 3y ago
Could anybody show some example queries and answers?
- ricklamers 3y ago
  Queries on https://github.com/pypa/flit/tree/main/flit_core/flit_core https://github.com/pypa/flit/tree/main/flit_core/flit_core (omitted tests/) https://pastebin.com/qyzWm6YC https://pastebin.com/qyzWm6YC https://pastebin.com/DDnDKGxd https://pastebin.com/DDnDKGxd

28 points by ricklamers 3y ago

_boffin_ 3y ago

From the initial look, it looks like you're taking the work of SweepAI, a source-available repo, which gave(?) it to LlamaIndex(1) and then you're just providing a wrapper around the LlamaIndex that provides a way to do an Update statement. 1) https://docs.sweep.dev/blogs/chunking-improvements https://docs.sweep.dev/blogs/chunking-improvements

deleted 3y ago
[deleted]
ricklamers 3y ago
Sweep is mentioned as attribution in multiple places a) https://github.com/definitive-io/code-indexer-loop#attribution https://github.com/definitive-io/code-indexer-loop#attributi... b) https://github.com/definitive-io/code-indexer-loop/blob/fd9d4a65e475e9f41e782af7cedea3d0bc675af5/code_indexer_loop/code_splitter.py#L3 https://github.com/definitive-io/code-indexer-loop/blob/fd9d... The difference is packaging it as a consumable PyPI package that can easily be used in a project (they even call out for separating this out into a stand alone project but that they lack the time to do so: https://docs.sweep.dev/blogs/chunking-2m-files#future- https://docs.sweep.dev/blogs/chunking-2m-files#future- ) In addition, we expand and fix the implementation, for example it now supports limiting on token count instead of character count, and we fix some white space inconsistencies in parsing/chunk reconstruction.
- _boffin_ 3y ago
  Looking forward to diving further in and seeing how you handle white spaces issues as I’ve had massive issues with that already
  - ricklamers 3y ago
    Please let me know if anything isn’t as expected and I’ll try to look into it

funcDropShadow 3y ago

Could anybody show some example queries and answers?

ricklamers 3y ago
Queries on https://github.com/pypa/flit/tree/main/flit_core/flit_core https://github.com/pypa/flit/tree/main/flit_core/flit_core (omitted tests/) https://pastebin.com/qyzWm6YC https://pastebin.com/qyzWm6YC https://pastebin.com/DDnDKGxd https://pastebin.com/DDnDKGxd