6 ms·

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human

by ekzhu 5y ago

I cannot continue reading after this following “declaration”… Author should take a look at the Wikipedia page for TF-IDF. > As someone who has a Ph.D. in Human-computer Interaction ;-), I feel like I am entitled to define a condition of "good" in relevance here. I hereby declare that: >> A good top-K algorithm should rank a document containing more user query terms higher than a document containing less number of user query terms. > This makes perfect sense. Right? Also, “most search engines” don’t use vector space model as the only way to rank result, for example, page rank. Edit: in some search scenarios finding the documents with the most query terms make sense, but Lucene can also rank using this metric. Still, applaud the author's effort in digging into research literature. Search relevance is very hard and standard off the shelf metrics like TF-IDF and page rank are often not enough. Good search usually requires deep understanding of the specific subject domain and hand-tuning tons of signals, many of which aren't even strictly based on search terms (e.g., previously purchased products on a store's website, geographic location, trending results).

Cavallium 5y ago
I also suggest you to look at his "benchmarks" code
- huahaiy 5y ago
  The benchmark is fine. Your so-called "fundamental" flaws are just superficial things, such as using a different time measure, using a different benchmark library, passing in a thread pool, and so on. These are immaterial to relative comparison, unless proven otherwise. I have repeated asked you to send the code that does things properly in your mind, and you refused to do so. So you are just trolling. Please stop hinder the progress of the industry through pointless trolling. Makes some positive contributions instead, and I have pointed out repeatedly how to make a positive contribution in this case. Please do. You are a university student, I am a seasoned computer scientist, past researcher and professor, I am busying running a startup. This will be the last time I will say this to you: please consider contributing something positive to the world.
softwaredoug 5y ago
As author of Relevant Search and contriburor of AI powered search I endorse this :) Relevance is really subjective, domain specific, requires intense amount of measurement and testing and many different ranking signals. Lucene is a toolbox for crafting many of these signals.
- softwaredoug 5y ago
  On second read-thru, I think the author is maybe(?) describing assumptions behind WAND and relevance algos that benefit from it? Maybe not some overarching statement about relevance per-se? But it's mixed in with statements about relevance / what Lucene does that are mostly incorrect... For example, he says > However, as you can see, this vector space model does not explicitly require a higher ranking document to contain more query terms than a lower ranking one. Well the way you get a higher similarity in a vector-space model is matching more terms. The caveat being that IDF and field length makes you also consider a term's specificity. So if you search for 'luke skywalker' you care more about the 'skywalker' match than the 'luke' match. But a match on BOTH 'luke skywalker' would score higher (field lengths being constant)
  - huahaiy 5y ago
    There's no guarantee in pure vector space model that it is the case. Your understanding is way off. This is a caveat that Lucene prominently put on their Web page. https://lucene.apache.org/core/3_5_0/scoring.html https://lucene.apache.org/core/3_5_0/scoring.html "Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on java-user@lucene.apache.org to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms." See? Lucene people know about it but they just do not think it is a problem. But I do. T-Wand, though also uses vector space model, makes sure it is the case.
    - softwaredoug 5y ago
huahaiy 5y ago
It's your loss then. It is actually smug, to suggest the author, in this case, me, a computer scientist and a past professor who taught Information Retrieval class for more than 3 years, and who just came up with a new search algorithm, to "take a look at Wikipedia page for TF-IDF". In case you have not read the article due to your smugness, my search algorithm also uses TF-IDF and vector space model. Please do not dis-understand what is going on here: I am running a startup, and I am also old enough to not care about publications as much as people who are younger or in academia. That's why I chose to reveal this in a blog post instead of hiding it until after my paper is published. Understood?
- antocv 5y ago
  Haters gonna hate, dont mind them. Your work is truly amazing, I learned a lot and you opened a whole new world of datalog as an aside.
  - huahaiy 5y ago
    Thank you. I am glad that you learned something from the article. I wrote the article for people like you, who are seeking knowledge and self improvement.
rory_isAdonk 5y ago
Yeah beyond the cringe of thinking a Ph.D really means anything, its just highlights the pure lack of Lucene knowledge.
- speedgoose 5y ago
  A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.
  - tgv 5y ago
    But a phd in HCI has very little relevance to information retrieval or computational linguistics.
    - johncearls 5y ago
      Exactly this. I have a PHD in CS and am a world class expert in multi-omic data integration and analysis. I'm happy to throw my weight around in that area, but I'd never point to my PhD to pontificate on Neural nets or systems or queuing theory or 99 percent of CS. If getting a PhD doesn't teach you how much you don't know and how hard it is to develop real expertise in any area, I think you wasted your PhD. Note, my PhD isn't on the value of PhDs so take it as you will.
      - huahaiy 5y ago
        I think I am pretty qualified to make my declaration, since it is about user experience. I also do know a lot about IR, because I taught Information Retrieval class for 3 years when I was teaching in university, I read research papers, and I just come up with a new search algorithm. It's just that some people cannot accept that there are people who can cross fields with ease, make contributions quickly, and move on to the next field that pit their interests. Yes, I am one of those people. In addition to HCI, I also published in the following areas: VR, DB, NLP, IR and Psychology. Sorry to hurt your feelings, but it is what it is. Accept it and move on.

by ekzhu 5y ago

Cavallium 5y ago

I also suggest you to look at his "benchmarks" code

huahaiy 5y ago
The benchmark is fine. Your so-called "fundamental" flaws are just superficial things, such as using a different time measure, using a different benchmark library, passing in a thread pool, and so on. These are immaterial to relative comparison, unless proven otherwise. I have repeated asked you to send the code that does things properly in your mind, and you refused to do so. So you are just trolling. Please stop hinder the progress of the industry through pointless trolling. Makes some positive contributions instead, and I have pointed out repeatedly how to make a positive contribution in this case. Please do. You are a university student, I am a seasoned computer scientist, past researcher and professor, I am busying running a startup. This will be the last time I will say this to you: please consider contributing something positive to the world.

softwaredoug 5y ago

As author of Relevant Search and contriburor of AI powered search I endorse this :) Relevance is really subjective, domain specific, requires intense amount of measurement and testing and many different ranking signals. Lucene is a toolbox for crafting many of these signals.

softwaredoug 5y ago
On second read-thru, I think the author is maybe(?) describing assumptions behind WAND and relevance algos that benefit from it? Maybe not some overarching statement about relevance per-se? But it's mixed in with statements about relevance / what Lucene does that are mostly incorrect... For example, he says > However, as you can see, this vector space model does not explicitly require a higher ranking document to contain more query terms than a lower ranking one. Well the way you get a higher similarity in a vector-space model is matching more terms. The caveat being that IDF and field length makes you also consider a term's specificity. So if you search for 'luke skywalker' you care more about the 'skywalker' match than the 'luke' match. But a match on BOTH 'luke skywalker' would score higher (field lengths being constant)
- huahaiy 5y ago
  There's no guarantee in pure vector space model that it is the case. Your understanding is way off. This is a caveat that Lucene prominently put on their Web page. https://lucene.apache.org/core/3_5_0/scoring.html https://lucene.apache.org/core/3_5_0/scoring.html "Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on java-user@lucene.apache.org to figure out why a document with five of our query terms scores lower than a different document with only one of the query terms." See? Lucene people know about it but they just do not think it is a problem. But I do. T-Wand, though also uses vector space model, makes sure it is the case.
  - softwaredoug 5y ago

huahaiy 5y ago

It's your loss then. It is actually smug, to suggest the author, in this case, me, a computer scientist and a past professor who taught Information Retrieval class for more than 3 years, and who just came up with a new search algorithm, to "take a look at Wikipedia page for TF-IDF". In case you have not read the article due to your smugness, my search algorithm also uses TF-IDF and vector space model. Please do not dis-understand what is going on here: I am running a startup, and I am also old enough to not care about publications as much as people who are younger or in academia. That's why I chose to reveal this in a blog post instead of hiding it until after my paper is published. Understood?

antocv 5y ago
Haters gonna hate, dont mind them. Your work is truly amazing, I learned a lot and you opened a whole new world of datalog as an aside.
- huahaiy 5y ago
  Thank you. I am glad that you learned something from the article. I wrote the article for people like you, who are seeking knowledge and self improvement.

rory_isAdonk 5y ago

Yeah beyond the cringe of thinking a Ph.D really means anything, its just highlights the pure lack of Lucene knowledge.

speedgoose 5y ago
A Ph.D means a lot for many people. It's years of work to make the science progress a tiny little bit on a topic you enjoy (supposedly). I think it's a lot harder and meaningful than writing scalable and reliable code as a team in a software company, just to give one example to compare with.
- tgv 5y ago
  But a phd in HCI has very little relevance to information retrieval or computational linguistics.
  - johncearls 5y ago
    Exactly this. I have a PHD in CS and am a world class expert in multi-omic data integration and analysis. I'm happy to throw my weight around in that area, but I'd never point to my PhD to pontificate on Neural nets or systems or queuing theory or 99 percent of CS. If getting a PhD doesn't teach you how much you don't know and how hard it is to develop real expertise in any area, I think you wasted your PhD. Note, my PhD isn't on the value of PhDs so take it as you will.
    - huahaiy 5y ago
      I think I am pretty qualified to make my declaration, since it is about user experience. I also do know a lot about IR, because I taught Information Retrieval class for 3 years when I was teaching in university, I read research papers, and I just come up with a new search algorithm. It's just that some people cannot accept that there are people who can cross fields with ease, make contributions quickly, and move on to the next field that pit their interests. Yes, I am one of those people. In addition to HCI, I also published in the following areas: VR, DB, NLP, IR and Psychology. Sorry to hurt your feelings, but it is what it is. Accept it and move on.