5 ms·

How good is Google’s Instant Mix?

57 points by hillad 15y ago

Androsynth 15y ago
I have two thoughts on this: 1-As has been mentioned multiple times on HN recently, many of the best minds of our generation are working on the algorithms that go into products like genius. They are basically the same as recomendations, online advertisements, etc. Its not surprising that theyre really damn good at this point. 2-Google probably doesnt care about the product itself, more likely it just wants to mine the data that gets put into it. Like goog-411, the service is just a conduit to acquire data.
- nl 15y ago
  2-Google probably doesnt care about the product itself, more likely it just wants to mine the data that gets put into it. Like goog-411, the service is just a conduit to acquire data. To what end? Unless Google is trying to make sure that when its software becomes sentient and takes over the world it has good taste in music (heh) I can't see much use for the data beyond building a recommender.
- bad_user 15y ago
  Apple has data about purchased music from iTunes, with which it can do pretty damn good similarity scores. Google has no such data.
kefs 15y ago
So, Google should purchase The Echo Nest.
chapel 15y ago
He addresses this in his article, but you really can't evaluate competitors to your own service in such a manner. Not only were the testing criteria highly subjective, the fact that songs he considered 'WTF' in the lists were the negative marks. I know music is a very subjective subject, I really think he could have been more objective about the whole thing. Maybe using other peoples music collections and getting their own personal opinions on which songs work and which songs don't. Add to that, have other people rate the playlists he generated, so it wasn't just his opinion. Also, wtf about Genius only getting 10 marks against it for not doing Beatles playlists? I'm sorry, if you hold the criteria that any songs that are out of place are considered negative, no songs in a playlist should be worth 24 'WTF' points. Not that it matters in the comparison, and it was nowhere near close. I don't have access to the beta, and at this point don't really care about it, but this just screams as self promotion. I think it would have been a lot more respectable if he had been more objective concerning the tests he used.
- nl 15y ago
  Paul has done a lot of work in the field of playlist evaluation. I think that if you survey the researchers in the field, the "WTF test" would be considered fairly reasonable - especially for a quick-and-dirty evaluation. Can you point to any specific songs that he said were WTF's and you think aren't, or vice/versa? If not then it would appear to meet the objectivity criteria. Using his own music collection might be slightly more suspect. Changing that might have flipped the outcome of iTunes vs EchoNest, but wouldn't have changed the real news here: Google does really, really badly.
  - Tyrannosaurs 15y ago
    I think you can argue the individual songs but the overall findings are sound - Google is poor, EchoNest is good, iTunes is good bar one song (which is suspicious). The methodology is a bit finger in the air but then so are the findings.
- watty 15y ago
  I agree music is subjective and his results are biased. However, it's safe to say Google's attempts are terrible.
- brianwhitman 15y ago
  It is impossible to be objective about music. The point of his "WTF" test (not speaking for paul, but i do work with the guy) was to look at each song by each provider and ask "would most people agree this doesn't belong" erring very far on the generous inclusive side. You can definitely quibble about whether or not a particular song deserves a WTF mark but you can't argue that google's results are very terrible overall. You couldn't do a test with other collections because the beta is very limited right now-- I can only think of a few people I know that have access. But I can concur that my results are as terrible as his were. Yes, he works for EN (which he's very clear about) and yes it's a bit inside baseball showing a service that most people can't use (because they're not developers or customers) but you can scroll past our results & take it as a post titled "How is Apple so much better than Google at such a data driven task?" (I would have given the lack of beatles on genius -24 too! I also found a couple EN clunkers that I'd give a WTF to, but I'm a notorious jerk for those things, ask anyone i work with :)
andrewflnr 15y ago
It would be interesting to run these songs through Pandora and compare the results.
MikeCapone 15y ago
Could this be because this service is so new that they haven't yet had the benefit of gathering lots of data and stats from users?
- lordlarm 15y ago
  I too believe so. Let Google build their product first, it is still a beta, for a reason. Music recommendation is close to impossible without lots of data - and it becomes unfair to compare Google Music with iTunes Genius, which has been on the market in 3 years.
  - brianwhitman 15y ago
    (again, i'm the co-founder, grain of salt time, but check it out yourself) but The Echo Nest makes amazing similarity judgments and playlists without any of the data that you think Google needs to wait for. We crawl the web and analyze the audio and figure out what people are saying about music and what it sounds like. We don't need usage data at all to make recommendations. No usage data went into the EN results in the OP.
    - MikeCapone 15y ago
      Fair enough, but if their technique is based on mining data from users of the service, then let's wait a little while and do the comparison again.
tintin 15y ago
I think it would be more interesting to recommend music by mood instead of genre. Jazz doesn't mean a thing. When I'm happy I don't want to listen to down jazz although it's jazz. Maybe I would like some happy funk and metal as well.
Tyrannosaurs 15y ago
Anyone else curious about why the Beatles returned nothing on iTunes? [Note: All this applies to Beatles tracks ripped from CD rather than purchased via iTunes.] I've just tried Genius (updated immediately before) on a selection of Beatles tracks in iTunes - Eleanor Rigby, Yellow Submarine, Lucy in the Sky with Diamonds, Sgt. Peppers Lonely Hearts Club Band and not one of them produces a playlist. Even before they were selling Beatles tracks enough people will have ripped them from CD to have the data to produce playlists so it seems unlikely (but not impossible) that it's data related, which suggests that it's either an odd, very specific error or intentional. What's stranger is that for some (but not all) of the tracks it will make Genius recommendations to buy... Anyone any ideas? Some strange part of the Beatles licensing deal perhaps though that would be very odd as really who gains anything by that?

57 points by hillad 15y ago

I have two thoughts on this: 1-As has been mentioned multiple times on HN recently, many of the best minds of our generation are working on the algorithms that go into products like genius. They are basically the same as recomendations, online advertisements, etc. Its not surprising that theyre really damn good at this point. 2-Google probably doesnt care about the product itself, more likely it just wants to mine the data that gets put into it. Like goog-411, the service is just a conduit to acquire data.

nl 15y ago
2-Google probably doesnt care about the product itself, more likely it just wants to mine the data that gets put into it. Like goog-411, the service is just a conduit to acquire data. To what end? Unless Google is trying to make sure that when its software becomes sentient and takes over the world it has good taste in music (heh) I can't see much use for the data beyond building a recommender.
bad_user 15y ago
Apple has data about purchased music from iTunes, with which it can do pretty damn good similarity scores. Google has no such data.

kefs 15y ago

So, Google should purchase The Echo Nest.

chapel 15y ago

He addresses this in his article, but you really can't evaluate competitors to your own service in such a manner. Not only were the testing criteria highly subjective, the fact that songs he considered 'WTF' in the lists were the negative marks. I know music is a very subjective subject, I really think he could have been more objective about the whole thing. Maybe using other peoples music collections and getting their own personal opinions on which songs work and which songs don't. Add to that, have other people rate the playlists he generated, so it wasn't just his opinion. Also, wtf about Genius only getting 10 marks against it for not doing Beatles playlists? I'm sorry, if you hold the criteria that any songs that are out of place are considered negative, no songs in a playlist should be worth 24 'WTF' points. Not that it matters in the comparison, and it was nowhere near close. I don't have access to the beta, and at this point don't really care about it, but this just screams as self promotion. I think it would have been a lot more respectable if he had been more objective concerning the tests he used.

nl 15y ago
Paul has done a lot of work in the field of playlist evaluation. I think that if you survey the researchers in the field, the "WTF test" would be considered fairly reasonable - especially for a quick-and-dirty evaluation. Can you point to any specific songs that he said were WTF's and you think aren't, or vice/versa? If not then it would appear to meet the objectivity criteria. Using his own music collection might be slightly more suspect. Changing that might have flipped the outcome of iTunes vs EchoNest, but wouldn't have changed the real news here: Google does really, really badly.
- Tyrannosaurs 15y ago
  I think you can argue the individual songs but the overall findings are sound - Google is poor, EchoNest is good, iTunes is good bar one song (which is suspicious). The methodology is a bit finger in the air but then so are the findings.
watty 15y ago
I agree music is subjective and his results are biased. However, it's safe to say Google's attempts are terrible.
brianwhitman 15y ago
It is impossible to be objective about music. The point of his "WTF" test (not speaking for paul, but i do work with the guy) was to look at each song by each provider and ask "would most people agree this doesn't belong" erring very far on the generous inclusive side. You can definitely quibble about whether or not a particular song deserves a WTF mark but you can't argue that google's results are very terrible overall. You couldn't do a test with other collections because the beta is very limited right now-- I can only think of a few people I know that have access. But I can concur that my results are as terrible as his were. Yes, he works for EN (which he's very clear about) and yes it's a bit inside baseball showing a service that most people can't use (because they're not developers or customers) but you can scroll past our results & take it as a post titled "How is Apple so much better than Google at such a data driven task?" (I would have given the lack of beatles on genius -24 too! I also found a couple EN clunkers that I'd give a WTF to, but I'm a notorious jerk for those things, ask anyone i work with :)

andrewflnr 15y ago

It would be interesting to run these songs through Pandora and compare the results.

MikeCapone 15y ago

Could this be because this service is so new that they haven't yet had the benefit of gathering lots of data and stats from users?

lordlarm 15y ago
I too believe so. Let Google build their product first, it is still a beta, for a reason. Music recommendation is close to impossible without lots of data - and it becomes unfair to compare Google Music with iTunes Genius, which has been on the market in 3 years.
- brianwhitman 15y ago
  (again, i'm the co-founder, grain of salt time, but check it out yourself) but The Echo Nest makes amazing similarity judgments and playlists without any of the data that you think Google needs to wait for. We crawl the web and analyze the audio and figure out what people are saying about music and what it sounds like. We don't need usage data at all to make recommendations. No usage data went into the EN results in the OP.
  - MikeCapone 15y ago
    Fair enough, but if their technique is based on mining data from users of the service, then let's wait a little while and do the comparison again.

tintin 15y ago

I think it would be more interesting to recommend music by mood instead of genre. Jazz doesn't mean a thing. When I'm happy I don't want to listen to down jazz although it's jazz. Maybe I would like some happy funk and metal as well.

Tyrannosaurs 15y ago

Anyone else curious about why the Beatles returned nothing on iTunes? [Note: All this applies to Beatles tracks ripped from CD rather than purchased via iTunes.] I've just tried Genius (updated immediately before) on a selection of Beatles tracks in iTunes - Eleanor Rigby, Yellow Submarine, Lucy in the Sky with Diamonds, Sgt. Peppers Lonely Hearts Club Band and not one of them produces a playlist. Even before they were selling Beatles tracks enough people will have ripped them from CD to have the data to produce playlists so it seems unlikely (but not impossible) that it's data related, which suggests that it's either an odd, very specific error or intentional. What's stranger is that for some (but not all) of the tracks it will make Genius recommendations to buy... Anyone any ideas? Some strange part of the Beatles licensing deal perhaps though that would be very odd as really who gains anything by that?