14 ms·

Extra cloaking for malicious links? Bit.ly can't exactly check the original site to see if it's OK if it's hidden behind a funky redirection layer. I think that

by whughes 16y ago

Extra cloaking for malicious links? Bit.ly can't exactly check the original site to see if it's OK if it's hidden behind a funky redirection layer. I think that bit.ly was reasonable to have an interstitial, although maybe they should be more open about why and how to remove it.

tkaemming 16y ago
They could check the endpoint URL by following the HTTP headers on 301/302 redirects.
sjs 16y ago
It's a standard http 301 redirect, not a "funky redirection layer". /Users/sjs % curl -i http://xrl.in/33qj HTTP/1.1 301 Permanent Redirect Date: Sun, 09 May 2010 00:38:19 GMT Server: Apache Location: http://www.donationcoder.com/CodingSnacks/index.php Content-Length: 0 Content-Type: text/html
- thenduks 16y ago
  Right, but the problem is this scenario: Bit.ly see's your xrl.in and does a request. They find 301 and the location at donationcoder.com. They conclude "this site is ok". Later, the xrl.in url is changed to <malware link>. They aren't going to do a request to every url they're linking to on every click, obviously. So they'd only get the one chance. Now, I'm not actually sure that xrl.in lets you change links after shortening. The point is that bit.ly doesn't know either.
  - jbm 16y ago
    Bit.ly could check a link to no_malware_here.com, which thereafter adds a 301 that redirects to your_computer_now_has_aids.com. I think it is misleading to display that message based on the possibility of a redirection. Any page can do that, not just xrl.in.
    - thenduks 16y ago
      I think that's the point. They check for a redirection and show an interstitial. They've white-listed a few other shortener services (the ones they know are redirecting to actual sites and presumably do their own redirection checks on)... but as far as I understand it any url that you try to shorten that does an immediate 301 gets the interstitial first. If they let urls with redirects on them they can inadvertently bit.ly link directly to a malware site. They don't want to do that at all and take measures to prevent it. Checking urls against malware lists and not allowing redirects are just a couple, I'm sure there are more. As for being misleading in the interstitial itself... I don't think so. They've updated it to be more clear about the issues with this link: * Some URL-shorteners re-use their links, so bit.ly can't guarantee the validity of this link. * Some URL-shorteners allow their links to be edited, so bit.ly can't tell where this link will lead you. * Spam and malware is very often propagated by exploiting these loopholes, neither of which bit.ly allows for.
  - BrandonM 16y ago
    Easy solution: before serving that warning page for shortened URLs, dynamically determine the endpoint. If it does point to a bad site, then serve the warning page. Otherwise, do the right thing.
    - thenduks 16y ago
      That isn't so easy when there could be 15 redirects between bit.ly and the malware site.
  - sjs 16y ago
    I can submit a link to any URL in my control and swap it out for something else later. That can apply to every single link on the web. By that logic they should maintain a whitelist of not only redirection services such as shorteners, but for each and every domain. I'm all for whitelisting and being paranoid, but it just doesn't make sense here and it seems a bit like they're trying to make the competition look bad. This should really be a blacklist instead.
    - thenduks 16y ago
      I'm not entirely sure but I believe the idea is that any 301 will cause bit.ly to show the interstitial. So you're right, it does apply to any site on the internet. It's just that you mostly notice it on other shortener services that haven't been whitelisted (blacklist makes no sense because it would then be bit.ly's responsibility to know of and check every other shortener service out there).

by whughes 16y ago

tkaemming 16y ago

They could check the endpoint URL by following the HTTP headers on 301/302 redirects.

sjs 16y ago

It's a standard http 301 redirect, not a "funky redirection layer". /Users/sjs % curl -i http://xrl.in/33qj HTTP/1.1 301 Permanent Redirect Date: Sun, 09 May 2010 00:38:19 GMT Server: Apache Location: http://www.donationcoder.com/CodingSnacks/index.php Content-Length: 0 Content-Type: text/html

thenduks 16y ago
Right, but the problem is this scenario: Bit.ly see's your xrl.in and does a request. They find 301 and the location at donationcoder.com. They conclude "this site is ok". Later, the xrl.in url is changed to <malware link>. They aren't going to do a request to every url they're linking to on every click, obviously. So they'd only get the one chance. Now, I'm not actually sure that xrl.in lets you change links after shortening. The point is that bit.ly doesn't know either.
- jbm 16y ago
  Bit.ly could check a link to no_malware_here.com, which thereafter adds a 301 that redirects to your_computer_now_has_aids.com. I think it is misleading to display that message based on the possibility of a redirection. Any page can do that, not just xrl.in.
  - thenduks 16y ago
    I think that's the point. They check for a redirection and show an interstitial. They've white-listed a few other shortener services (the ones they know are redirecting to actual sites and presumably do their own redirection checks on)... but as far as I understand it any url that you try to shorten that does an immediate 301 gets the interstitial first. If they let urls with redirects on them they can inadvertently bit.ly link directly to a malware site. They don't want to do that at all and take measures to prevent it. Checking urls against malware lists and not allowing redirects are just a couple, I'm sure there are more. As for being misleading in the interstitial itself... I don't think so. They've updated it to be more clear about the issues with this link: * Some URL-shorteners re-use their links, so bit.ly can't guarantee the validity of this link. * Some URL-shorteners allow their links to be edited, so bit.ly can't tell where this link will lead you. * Spam and malware is very often propagated by exploiting these loopholes, neither of which bit.ly allows for.
- BrandonM 16y ago
  Easy solution: before serving that warning page for shortened URLs, dynamically determine the endpoint. If it does point to a bad site, then serve the warning page. Otherwise, do the right thing.
  - thenduks 16y ago
    That isn't so easy when there could be 15 redirects between bit.ly and the malware site.
- sjs 16y ago
  I can submit a link to any URL in my control and swap it out for something else later. That can apply to every single link on the web. By that logic they should maintain a whitelist of not only redirection services such as shorteners, but for each and every domain. I'm all for whitelisting and being paranoid, but it just doesn't make sense here and it seems a bit like they're trying to make the competition look bad. This should really be a blacklist instead.
  - thenduks 16y ago
    I'm not entirely sure but I believe the idea is that any 301 will cause bit.ly to show the interstitial. So you're right, it does apply to any site on the internet. It's just that you mostly notice it on other shortener services that haven't been whitelisted (blacklist makes no sense because it would then be bit.ly's responsibility to know of and check every other shortener service out there).