9 ms·

Show HN: TextRazor, a scriptable text mining API

118 points by tcwc 14y ago

steeve 14y ago
Cool demo, but how do you guys compare to, say, Alchemy API ? Also, no pricing => not cool.
- tcwc 14y ago
  Hey steeve we thought there were a few things missing in the competition. We've built a bunch of extra functionality such as more extensive relation and dependency parsing and contextual entailment generation, and use all that to build much more accurate entity and topic recognition, an area we think the others can be greatly improved on. We also expose all these results to a Prolog interpreter on our backend and allow you to add custom logic to mashup and extend all of our results, as well as provide a much easier integration experience. Totally agree with you on the pricing front, we're still finalising the details there. We're aiming to be fully transparent with both the technical and business side of things.
  - ismaelc 14y ago
    You can add billing/pricing tiers to your API now using Mashape http://www.mashape.com/ http://www.mashape.com/ (Disclaimer: I work for Mashape. Let me know if you need help!)
- PaulHoule 14y ago
  Recall is perhaps 300% better than Alchemy API with much better precision.
GotAnyMegadeth 14y ago
This looks amazing. Though when I put my name in it came up with: DBPedia types: Person Athlete Agent GolfPlayer owl#Thing
abaer 14y ago
Really impressive results. I put in some music reviews, and it did an excellent job of identifying artists, Genres, labels, etc. From an API perspective,there's not much out there that competes with this - and nothing with a modern API.
- typpo 14y ago
  There are a number of other competing APIs. One of the best is OpenCalais, which is owned by Thomson Reuters[1]. You can demo it with some text here: http://viewer.opencalais.com/ http://viewer.opencalais.com/ [1] http://www.opencalais.com http://www.opencalais.com
thejosh 14y ago
No pricing at all on the website is fishy for anyone who wants to use the free plan?
- mark_l_watson 14y ago
  I offer something similar, and I do have a pricing page: http://kbsportal.com/pricing http://kbsportal.com/pricing :-)
- tcwc 14y ago
  I can understand why you might think that, but we'll always have the free plan. We're still working out all the little pricing details.
habosa 14y ago
Wow, this seems incredible, signed up immediately. My mind is already spinning with all of the cool apps I could make with this. How hard would it be to allow this functionality offline for paid users? You could have some sort of packaged library which phones home to count requests used, but does the processing offline to take network latency out of the equation. Not sure if that's feasible, but it would be great.
anilshanbhag 14y ago
This is just like stanford parser. http://nlp.stanford.edu:8080/parser/ http://nlp.stanford.edu:8080/parser/ Why use TextRazor and pay for it ?
- mark_l_watson 14y ago
  The Stanford NLP tools are very good, and also GPLed, which works for a lot of projects. If the GPL doesn't work for you, the Apache OpenNLP project is also good.
  - mark_l_watson 14y ago
    BTW, it is not just having software packages to use: it is a ton of work obtaining and preparing training data. That said, Stanford NLP and OpenNLP tools come out of the box with trained models for tagging, entity name recognition, etc. For lots of uses, these pre-trained models will work well for you.
- tcwc 14y ago
  The Stanford parser is great, but isn't really the same. The Stanford entity recogniser is limited to the standard types of people, places, companies, but we identify and disambiguate into a far richer ontology from wikipedia, and can recognize topic abstractions that aren't explicitly mentioned. Also we found the Stanford tools (and the other open source NLP tools) were difficult to integrate into "production" apps for various reasons. One big one was performance - we aim to run the full parsing and extraction pipeline on an average news story in a few hundred milliseconds, which can be an order of magnitude faster than the others.
  - JPKab 14y ago
    How does your offering compare to Calais from Thomson-Reuters? Edit: To be specific, it looks very similar. What do you have that Calais doesn't?
    - mark_l_watson 14y ago
ses 14y ago
Being a big Prolog fan, I think this looks like an awesome product. This sort of textual analysis will become more important as time goes on. As the interest in search technologies grows, I think intelligent search (contextual queries, query answering, clustering, recommendations, meta-data extraction etc.) will start to appear in more end-user products. One question... whereabouts in the UK are you based?
- tcwc 14y ago
  Thanks, great to see the other Prolog fans coming out of the woodwork! Based in London.
polskibus 14y ago
That's exactly what I need to start working again on my algo trader! Seems to be working well with a sample of financial news extracts. Will definitely look into it further, thanks!
doktrin 14y ago
Very cool. This is essentially what I envisioned when I started work on http://www.textalyze.com http://www.textalyze.com. Haven't devoted much time to it over the last couple months, but this provides a lot of inspiration.
- wiradikusuma 14y ago
  Hi man, I tried my website (http://www.ngajakjalan.com http://www.ngajakjalan.com) but the analyzer probably reads my inlined JS and doesn't read the "done" version (I use AngularJS). Maybe you can use something like PhantomJS to extract websites content "as seen by human"?
beebs93 14y ago
Wow, nice demo. I agree with the others the lack of pricing is a bit disconcerting. Btw, anyone else read the title as "Trent Reznor"? I need a coffee...
- snake_plissken 14y ago
  totes on the Trent Reznor part.
adelevie 14y ago
This looks great. I'm building http://dokket.aws.af.cm http://dokket.aws.af.cm. It's a database filled with documents from the Federal Communications Commission. From day 1, I've been looking for smart ways to make use of the thousands of documents of unstructured text. The customization you offer seems to be the killer feature for me. I'll write a Ruby api wrapper if you give me an agreed-upon amount of usage when you settle on pricing. Feel free to email me (HN name @ gmail) if you're interested or just want to follow up for customer development purposes. Best of luck!
- richardofyork 14y ago
  I am also building an application that could use this service. So I tested TextRazor and the results were not good on two random articles I processed with the service. I really want this to work out, because my application needs this kind of technology to be reliable and accurate. I just processed the article at the link below, and the word "Tesla" was not captured as one of the topics. http://www.teslamotors.com/blog/most-peculiar-test-drive-follow http://www.teslamotors.com/blog/most-peculiar-test-drive-fol...
  - jsmcgd 14y ago
    It works for me.
    - richardofyork 14y ago
      I just tried it again and it worked this time. That is interesting: I don't know if they tweaked it a bit or not :), but I am feeling a bit better now, because my application really needs a reliable, accurate service like this.
      - tcwc 14y ago
        No tweaking, promise :) It's possible you hit an inconsistent server first time around, I'll have a dig on our end. Let me know if you have any other problems - toby@textrazor.com.
eurodance 14y ago
Well done.
movingahead 14y ago
This is very impressive stuff. I ran a news article through the demo, and the entity recognition was very impressive. Waiting for them to reveal more details on pricing.
PaulHoule 14y ago
This is the best Wikipedia-backed namex I've seen and I've seen a lot of them, worked on code for one, and even designed one. Awesome!
petercooper 14y ago
Question: What are people actually doing with technology like this right now? (i.e. who are the people who see this and think.. yay, I'll sign up now!)
- JPKab 14y ago
  It's certainly not going to be ideal for your typical CRUD app. Think about all of the information that is locked inside of unstructured text (MS word docs, pdf's come to mind), and then imagine if you can scan through thousands of documents, find the named entities, and then start connecting them together in queries. Obvious uses would be any kind of CMS. Investigative journalism is another.
- foobar2k 14y ago
  finance
- adelevie 14y ago
  Me. See my other comment in this thread. I've recently started exploring the Legal Informatics field. The problems in it are huge and typically involving adding some structure to lots and lots of unstructured text. Also, Peter, if you do end up reading this, great work on the stuff you do :) Big fan here!
- Mikushi 14y ago
  Extracting important keywords from thousands of pages for example. On our website we let user enter quite a lot of content, and being able to extract keywords and find patterns between user could be key in naming categories and such.
cpsales 14y ago
python nltk is all what I have to say.
thehodge 14y ago
I don't know if your just being slammed right now but I posted a request for pricing and haven't got anything.. it looks like a nice API to integrate into our system but it really needs clearer pricing..
jsmcgd 14y ago
Awesome stuff. It would be great to use something like this for suggesting relevant tags of content. Perhaps a WordPress or Drupal plugin to get the ball rolling.
sinzone 14y ago
Hi, would love to have this API listed on Mashape
atrilla 14y ago
Great work! But I missed the "sentiment analysis" flavour that used to be so popular some years ago with the NLP bunch... In this sense, I did something similar: http://dtminredis.housing.salle.url.edu:8080/EmoLib/ http://dtminredis.housing.salle.url.edu:8080/EmoLib/ and http://nlptools.atrilla.net/web/omsa.php http://nlptools.atrilla.net/web/omsa.php Drop me a line if I can be of any help!

118 points by tcwc 14y ago

steeve 14y ago

Cool demo, but how do you guys compare to, say, Alchemy API ? Also, no pricing => not cool.

tcwc 14y ago
Hey steeve we thought there were a few things missing in the competition. We've built a bunch of extra functionality such as more extensive relation and dependency parsing and contextual entailment generation, and use all that to build much more accurate entity and topic recognition, an area we think the others can be greatly improved on. We also expose all these results to a Prolog interpreter on our backend and allow you to add custom logic to mashup and extend all of our results, as well as provide a much easier integration experience. Totally agree with you on the pricing front, we're still finalising the details there. We're aiming to be fully transparent with both the technical and business side of things.
- ismaelc 14y ago
  You can add billing/pricing tiers to your API now using Mashape http://www.mashape.com/ http://www.mashape.com/ (Disclaimer: I work for Mashape. Let me know if you need help!)
PaulHoule 14y ago
Recall is perhaps 300% better than Alchemy API with much better precision.

GotAnyMegadeth 14y ago

This looks amazing. Though when I put my name in it came up with: DBPedia types: Person Athlete Agent GolfPlayer owl#Thing

abaer 14y ago

Really impressive results. I put in some music reviews, and it did an excellent job of identifying artists, Genres, labels, etc. From an API perspective,there's not much out there that competes with this - and nothing with a modern API.

typpo 14y ago
There are a number of other competing APIs. One of the best is OpenCalais, which is owned by Thomson Reuters[1]. You can demo it with some text here: http://viewer.opencalais.com/ http://viewer.opencalais.com/ [1] http://www.opencalais.com http://www.opencalais.com

thejosh 14y ago

No pricing at all on the website is fishy for anyone who wants to use the free plan?

mark_l_watson 14y ago
I offer something similar, and I do have a pricing page: http://kbsportal.com/pricing http://kbsportal.com/pricing :-)
tcwc 14y ago
I can understand why you might think that, but we'll always have the free plan. We're still working out all the little pricing details.

habosa 14y ago

Wow, this seems incredible, signed up immediately. My mind is already spinning with all of the cool apps I could make with this. How hard would it be to allow this functionality offline for paid users? You could have some sort of packaged library which phones home to count requests used, but does the processing offline to take network latency out of the equation. Not sure if that's feasible, but it would be great.

anilshanbhag 14y ago

This is just like stanford parser. http://nlp.stanford.edu:8080/parser/ http://nlp.stanford.edu:8080/parser/ Why use TextRazor and pay for it ?

mark_l_watson 14y ago
The Stanford NLP tools are very good, and also GPLed, which works for a lot of projects. If the GPL doesn't work for you, the Apache OpenNLP project is also good.
- mark_l_watson 14y ago
  BTW, it is not just having software packages to use: it is a ton of work obtaining and preparing training data. That said, Stanford NLP and OpenNLP tools come out of the box with trained models for tagging, entity name recognition, etc. For lots of uses, these pre-trained models will work well for you.
tcwc 14y ago
The Stanford parser is great, but isn't really the same. The Stanford entity recogniser is limited to the standard types of people, places, companies, but we identify and disambiguate into a far richer ontology from wikipedia, and can recognize topic abstractions that aren't explicitly mentioned. Also we found the Stanford tools (and the other open source NLP tools) were difficult to integrate into "production" apps for various reasons. One big one was performance - we aim to run the full parsing and extraction pipeline on an average news story in a few hundred milliseconds, which can be an order of magnitude faster than the others.
- JPKab 14y ago
  How does your offering compare to Calais from Thomson-Reuters? Edit: To be specific, it looks very similar. What do you have that Calais doesn't?
  - mark_l_watson 14y ago

ses 14y ago

Being a big Prolog fan, I think this looks like an awesome product. This sort of textual analysis will become more important as time goes on. As the interest in search technologies grows, I think intelligent search (contextual queries, query answering, clustering, recommendations, meta-data extraction etc.) will start to appear in more end-user products. One question... whereabouts in the UK are you based?

tcwc 14y ago
Thanks, great to see the other Prolog fans coming out of the woodwork! Based in London.

polskibus 14y ago

That's exactly what I need to start working again on my algo trader! Seems to be working well with a sample of financial news extracts. Will definitely look into it further, thanks!

doktrin 14y ago

Very cool. This is essentially what I envisioned when I started work on http://www.textalyze.com http://www.textalyze.com. Haven't devoted much time to it over the last couple months, but this provides a lot of inspiration.

wiradikusuma 14y ago
Hi man, I tried my website (http://www.ngajakjalan.com http://www.ngajakjalan.com) but the analyzer probably reads my inlined JS and doesn't read the "done" version (I use AngularJS). Maybe you can use something like PhantomJS to extract websites content "as seen by human"?

beebs93 14y ago

Wow, nice demo. I agree with the others the lack of pricing is a bit disconcerting. Btw, anyone else read the title as "Trent Reznor"? I need a coffee...

snake_plissken 14y ago
totes on the Trent Reznor part.

adelevie 14y ago

This looks great. I'm building http://dokket.aws.af.cm http://dokket.aws.af.cm. It's a database filled with documents from the Federal Communications Commission. From day 1, I've been looking for smart ways to make use of the thousands of documents of unstructured text. The customization you offer seems to be the killer feature for me. I'll write a Ruby api wrapper if you give me an agreed-upon amount of usage when you settle on pricing. Feel free to email me (HN name @ gmail) if you're interested or just want to follow up for customer development purposes. Best of luck!

richardofyork 14y ago
I am also building an application that could use this service. So I tested TextRazor and the results were not good on two random articles I processed with the service. I really want this to work out, because my application needs this kind of technology to be reliable and accurate. I just processed the article at the link below, and the word "Tesla" was not captured as one of the topics. http://www.teslamotors.com/blog/most-peculiar-test-drive-follow http://www.teslamotors.com/blog/most-peculiar-test-drive-fol...
- jsmcgd 14y ago
  It works for me.
  - richardofyork 14y ago
    I just tried it again and it worked this time. That is interesting: I don't know if they tweaked it a bit or not :), but I am feeling a bit better now, because my application really needs a reliable, accurate service like this.
    - tcwc 14y ago
      No tweaking, promise :) It's possible you hit an inconsistent server first time around, I'll have a dig on our end. Let me know if you have any other problems - toby@textrazor.com.

eurodance 14y ago

Well done.

movingahead 14y ago

This is very impressive stuff. I ran a news article through the demo, and the entity recognition was very impressive. Waiting for them to reveal more details on pricing.

PaulHoule 14y ago

This is the best Wikipedia-backed namex I've seen and I've seen a lot of them, worked on code for one, and even designed one. Awesome!

petercooper 14y ago

Question: What are people actually doing with technology like this right now? (i.e. who are the people who see this and think.. yay, I'll sign up now!)

JPKab 14y ago
It's certainly not going to be ideal for your typical CRUD app. Think about all of the information that is locked inside of unstructured text (MS word docs, pdf's come to mind), and then imagine if you can scan through thousands of documents, find the named entities, and then start connecting them together in queries. Obvious uses would be any kind of CMS. Investigative journalism is another.
foobar2k 14y ago
finance
adelevie 14y ago
Me. See my other comment in this thread. I've recently started exploring the Legal Informatics field. The problems in it are huge and typically involving adding some structure to lots and lots of unstructured text. Also, Peter, if you do end up reading this, great work on the stuff you do :) Big fan here!
Mikushi 14y ago
Extracting important keywords from thousands of pages for example. On our website we let user enter quite a lot of content, and being able to extract keywords and find patterns between user could be key in naming categories and such.

cpsales 14y ago

python nltk is all what I have to say.

thehodge 14y ago

I don't know if your just being slammed right now but I posted a request for pricing and haven't got anything.. it looks like a nice API to integrate into our system but it really needs clearer pricing..

jsmcgd 14y ago

Awesome stuff. It would be great to use something like this for suggesting relevant tags of content. Perhaps a WordPress or Drupal plugin to get the ball rolling.

sinzone 14y ago

Hi, would love to have this API listed on Mashape

atrilla 14y ago

Great work! But I missed the "sentiment analysis" flavour that used to be so popular some years ago with the NLP bunch... In this sense, I did something similar: http://dtminredis.housing.salle.url.edu:8080/EmoLib/ http://dtminredis.housing.salle.url.edu:8080/EmoLib/ and http://nlptools.atrilla.net/web/omsa.php http://nlptools.atrilla.net/web/omsa.php Drop me a line if I can be of any help!