Seems to me the hardest part is going to be the speech recognition or at least to the degree of understanding that "jarvis" has. What are you hoping to have it

Seems to me the hardest part is going to be the speech recognition or at least to the degree of understanding that "jarvis" has. What are you hoping to have it accomplish? Generally speaking the toolset that you "should" use for a project should determined by what your most proficient in. In the 1 minute of googling that I did I found the dragonfly python library which looks like should help you handle the speech recognition. https://code.google.com/p/dragonfly/ https://code.google.com/p/dragonfly/ But keep in mind your tackling a very complex problem, to try and build "jarvis". Plenty of people that are smarter than me, and probably you, have tried and failed to come up with a good solution. That's why "jarvis" is in the movie, but nothing quite as smart "jarvis" has been built yet. The closest thing that comes to my mind is IBM Watson, but that was built just for jeopardy.