9 ms·
How paywalled sites permit access to visitors from social media sites and apps
- porker 9y agoInterestingly the FT's paywall is immune to this in my testing. Device fingerprinting?
- whyagaindavid 9y agoFT does not allow all articles free from a t.co link.
- animeseinfeld 9y agoNot that I condone doing it, Google Bot trick still works for FT.
- cupcakestand 9y agotl;dr: Wall Street Journal ended allowing special access for search engines through their paywall. By spoofing Twitter app's Referer and User Agent access is still possible and an included Chrome Extension script implements this idea.
- chipperyman573 9y agoI thought that google didn't allow websites to alter their appearance for google's UA.
- icebraining 9y agoI think the WSJ is just showing Google the reduced version as well.
- MarkSweep 9y agoI think the standard was, you could show the whole page to the google search engine so long as you showed the same page to someone clicking through from the search results. That's why the old trick was to search for the article on Google.
- donohoe 9y agoNope. Never been the case.
- Jaruzel 9y ago<rant> Personally, I'm finding the increased use of tl;dr here on HN annoying. I feel, that like myself, HN readers are intelligent enough to read and understand the articles without someone coming along and posting a summary, simply for up-votes. HN is all about the articles, and then the discussion on top. If people are finding the articles to hard to penetrate that they need a tl;dr summary, then maybe HN isn't the site for them? </rant>
- rovek 9y agoThis is exactly the kind of article I expect a tl;dr on, I actually looked for it to save myself opening the article, scanning for "referer" and then closing it immediately (which I did because the tl;dr was so low) I think this is a backlash from the self-congratulatory, self-indulgent tone medium has perpetuated, not through design I'm sure. Sometimes one just wants to validate what we expect/know and move on; personally I don't want more than 200 words to elaborate this headline. edit: typo
- icantdrive55 9y agoI have found the average HN'er who summarizes does a better job the the original author, in many cases. They might have a better understanding of the information, or just better writers? I don't like reading wordy articles. I don't like reading rushed, or poorly researched articles behind a paywall. Traditional journalists/authors have another problem on their hands these days. They have people out there summarizing their information, many times better, and more concise than the original article--for free. This whole change in the way things were once done is hurting all of us financially. There are so many times I come here and get such a better understanding of said article by reading the comments. (I do the think publications need to up their game a bit. It seems like too many decided to hire new college graduates, on the cheap, whom seem to reluctantly spit out a article, or they hire wordy authors who can't write well, but got the job because they know someone at the organization. This is not the time for any publication to hire employees kids, or practice any form form of nepotism. I understand they can't pay like they used to, but this is the time to up your game especially if you want us to pay. Even if they up their game--they might have lost the war. And in certain cities/locales a lot of important issues/information will go under the radar; which I find sad. Maybe the federal government should step in and fund certain publications, but in a hands off approach?)
- deleted 9y ago[deleted]
- Animats 9y agoThis only works, of course, for paywalls that want links in Twitter feeds to bypass their paywalls. I'd at least expect an ad at that point.
- NKCSS 9y agoif (details.url.includes(url)) { So, if I want to detect if you have this plugin installed, I load an image with ?plugin-test=wsj.com as any part in the url can have it to trigger. Might want to improve this...
- NKCSS 9y agoIf you add this to the manifest "permissions" section and go to http://pbb.nkcss.com/ http://pbb.nkcss.com/ you can see it's easily detected: "http://pbb.nkcss.com/*" And if I can do it in a few minutes, I'm sure those who have a paywall can do so as well.
- personjerry 9y agoSo why isn't this on the Chrome store?
- JamieF1 9y agoBecause the sites that have set up the paywall will report the extension and get it taken down, happened to an extension that I made a while back that had a similar idea.
- personjerry 9y agoWhy is it take-down-able?
- kahnpro 9y agoBecause it allows people to circumvent the paywall and WSJ will complain to Google. And then because Google can take down whatever they want...
- chii 9y agoWhich is why you don't put your extension as s Chrome extension, but as a tamper monkey script.
- _e 9y agoWhat is the best, in your opinion, site to find tamper/grease monkey scripts? I remember the landscape being fragmented some time ago.
- skinnymuch 9y agoYeah. Seems like when the main site went down, the scene never got things back on track with a proper aggregator or central site.
- crtasm 9y ago
- Kenji 9y agoIs it just me or are websites that implement these kinds of selective paywalls rarely worth visiting, let alone worth spending effort to get through the paywall?
- freddie_mercury 9y agoIt is just you.
- xU1ppskunDmy6oz 9y agoThe Wall Street Journal is a very, very good newspaper that brought us such amazing series as "What They Know", which I believe started the practice of referring to the use of trackers as "spying" (much to the dismay of advertisers), which has certainly helped deliver the message that tracking is something users should worry about.
- losteverything 9y agoIt's not you. For me, there is so much more to life than news. Not worth any extra effort
- whizzkid 9y agoWhile I think these kind of posts are really fun to read, they should also mention little bit more about why these are paywalled. People needs to understand that traditional newspaper is dying and in order to support real journalists, they need to make money. I am neither working for any news organisation nor talking about a specific news outlet. As far as I can see, they are trying hard not to annoy people but at the same time try to make some money for their work.
- 35bge57dtjku 9y agoThey don't get paid via ads like they used to?
- Rjevski 9y agoMost people block ads, as a side effect of the ads being nasty and irrelevant. Maybe they can start anew if they used bearable ads - something like The Deck (no affiliation, just an example).
- colejohnson66 9y agoPeople block ads. And for good reason. If a big website wants to make money from ads, they need to force the advertisers hand sometimes. OutBrain and Taboola are the worst.
- donohoe 9y agoBear in mind ads is just one revenue stream. There are times that ad market has collapsed and prices dropped. Having a second (or more) reliable and recurring revenue stream makes solid business sense and allows them to move away from soem of the crappier ad practices.
- tomjen3 9y agoI am okay with that. The quality of news produced is already so low it doesn't matter.
- PietdeVries 9y ago
- chrismorgan 9y agoSome code review just in case anyone is interested. (I don’t expect it to make it into the article as it was written six months ago.) This pattern: if (details.url.includes(url)) { return true; } return false; should be replaced by this: return details.url.includes(url); This pattern: array.map(someFunction).reduce(function(a, b) { return a ||b}, false) should be replaced by this: array.some(someFunction) (Note the semantics are slightly different—`.some` will break early, so it’s more efficient and equivalent provided there are no side-effects in the map function.) Taking both of these, the following: var useTwitter = VIA_TWITTER.map(function(url) { if (details.url.includes(url)) { return true; } return false; }) .reduce(function(a, b) { return a || b}, false); can be rewritten much more simply as: var useTwitter = VIA_TWITTER.some(function(url) { return details.url.includes(url); } You could even do it thus if you desired: var useTwitter = VIA_TWITTER.some(details.url.includes.bind(details.url)); … but that’s probably harder to read. I will mention arrow functions, however, which are pretty: var useTwitter = VIA_TWITTER.some(url => details.url.includes(url)); This part: details.requestHeaders.filter(function(header) { // block cookies by default if (header.name !== "Cookie") { return header; } }) `.filter` only cares about truthiness in its return value—as this code does it, undefined is false and an object is true. But you could simplify it: details.requestHeaders.filter(function(header) { // block cookies by default return header.name !== "Cookie"; }) Also in the original code’s usage of map, it’s not actually changing the values, only things inside them, so using `map` is wasteful (as it entails allocating a new array). You could just use `forEach`: var reqHeaders = …; reqHeaders.forEach(function(header) { if (header.name === "Referer") { header.value = setRefer(useTwitter); foundReferer = true; } if (header.name === "User-Agent") { header.value = setUserAgent(useTwitter); foundUA = true; } }); A remark on fine-tuning performance: when you access properties inside an object multiple times, it’s optimal to store it as a local variable to save having to look it up multiple times. (This is especially the case if the property is expensive to access.) Take the `blockCookies` method: function blockCookies(details) { for (var i = 0; i < details.responseHeaders.length; ++i) { if (details.responseHeaders[i].name === "Set-Cookie") { details.responseHeaders.splice(i, 1); } } return {responseHeaders: details.responseHeaders}; } This is accessing `details.responseHeaders` many times when it only needs to access it once. It is also accessing its `length` member once per iteration, rather than caching the length. Normally for that I’d say “store the length once up front,” but in this case the code is changing the array length in the loop, so that’d actually break things. On that note, the code as published is actually missing some cookies, because it removes an item from the array and then skips past the new element at that index. To fix that, you need to move the `++i` into the loop so it can be skipped if you do splice the array. Also in order to not need to access the length property many times you could iterate in reverse instead of forwards. I might write the whole function like this: function blockCookies(details) { var headers = details.responseHeaders; var i = headers.length - 1; while (i > -1) { if (headers[i].name === "Set-Cookie") { headers.splice(i, 1); } else { i--; } } return {responseHeaders: headers}; }
- allenleein 9y agoHow Google’s Web Crawler Bypasses Paywalls https://elaineou.com/2016/02/19/how-to-use-chrome-extensions-to-bypass-paywalls/ https://elaineou.com/2016/02/19/how-to-use-chrome-extensions...
- chinathrow 9y agoThat article links back to the one published on HN: "Update: A newer version of the chrome extension is available here."
- AznHisoka 9y agoI want bot neutrality damn it!
- cjg 9y agoThis is the kind of thing that eager prosecutors will turn into a CFAA charge.
- userbinator 9y agoI remember this trick many years ago was useful for bypassing download sites that otherwise obliged you to use their adware-filled "download manager" to get files or gave bonuses to such usrs (special user-agent). The words "User-Agent: MEGAUPLOAD 2.0" might bring back interesting memories for some here. ;-) It's trivial to do this with a filtering proxy, which means it works in all browsers. On the other side, I've unknowingly embedded images from image hosting sites that didn't allow "hotlinking", only to be told by other users they couldn't see them, because of the referer headers (or lack thereof) my usual configuration sends. IMHO things like this shouldn't be spread too widely... much like the fight between adblockers and anti-adblockers, it can only eventually lead to a more hostile computing environment.
- yalooze 9y agoThis bookmarklet works too: javascript:location.href='http://facebook.com/l.php?u='+location.href http://facebook.com/l.php?u='+location.href
- donohoe 9y agoAs someone who has worked on building two paywalls and still involved in them, please bear in mind a few things. 1. Publishers have ability to configure access on paywalls as they see fit. Whether based on referrer, UA, and a whole host of other attributes, history and so on. 2. Publishers don't care about a degree of paywall evasion. Studies show that people willing to pay will pay, and those who go to great lengths to evade won't ever subscribe. The question is then, do you want to waste expensive developer resources in an arms-race against ppl who'll never give you a cent, or do you want to spend that developer time enhancing the experience for those who will subscribe. 3. Full locked-down paywalls are known to be bad. Publishers still want to ensure their content is in the public conversation, and that means their content has to be accessible in soem form - or you strategically choose to follow a different business model. See: https://techcrunch.com/2010/11/02/times-paywall-4-million-readers/ https://techcrunch.com/2010/11/02/times-paywall-4-million-re...
- digi_owl 9y agoFirefox and ModifyHeaders is nice for this.
- mola 9y agoI am curious as how these sort of guides are taken as incouous fun reads while a guide to, say, shoplifting would seem less than legitimate.
- corybrown 9y agoBecause, unlike a news article, the marginal cost of the shoplifted good is nowhere near zero.
- amelius 9y agoAnd how about a guide to illegally copy Spotify tracks without an account?
- clarkenheim 9y agolink?
- amelius 9y agoJust hypothetical. I'm curious about the perceived value of music versus news articles.
- linkregister 9y agoI wouldn't be surprised if a shoplifting guide made it to the front page of HN. I mean, how many HNers have read the MIT lockpicking guide or the Anarchist's Cookbook? At least half?
- andrewla 9y agoI think a better analogy is sneaking into a movie theater, for which there are a large number of "innocuous fun" articles out there [1] (although not featured on HN so much) [1] https://www.google.com/search?q=how+to+sneak+into+a+movie+theater https://www.google.com/search?q=how+to+sneak+into+a+movie+th...
- rdiddly 9y agoBut this is about an arbitrary distinction between two groups of non-paying customers! However I dispute the "customer" analogy too. In fact, turning this tired and one-sided analogy of a "store" on its head, how about it's my browser, and while they're in it, they can play by my rules? There is nothing about a client-server paradigm (especially nowadays with thick clients) causing me to be "in their premises" or causing that to be a better analogy than their being in my premises[1]. The "online store" was just an analogy, and was how the www was sold to millions of profiteering dullards starting in the 90s, so naturally the idea has gotten a lot of traction and you can be forgiven for still thinking of it that way. [1]Neither is accurate of course - The truth of the matter is that my robot handshakes with their robot across space. (And then their robot asks the Twitter robot whether my robot has the special Twitter street cred badge.)
- netcan 9y agoPaywalled sites (particularly news/media sites) are playing at a messy game here. It’s similar to the free vs paid music. In the radio-records day, free radio play drove record sales revenue. Ideally they would sell records to anyone willing to pay, but give music away to anyone willing to listen. In the actual world, you can’t do this perfectly so you need another way for free & paid to co-exist. Once the medium changed to digital, all the rules get thrown out. Attempts to force digital to play by radio-&-record’s rules have been a slog. Many internet generations ago news sites were mad at Google for showing ads alongside their headlines on news.google. They even got some lawmakers to agree. Google offered to de-index them. Stalemate. Similar issue here. News sites want paid subscriptions if they can get them, so paywall. They also want the readers who won’t pay, so no paywall. Overall, I think paywalling is a semi-dead end. I don’t mean that it won’t work for any site, but it’ll probably be a niche revenue source like a print-only publication. Most news sites want to be part of the greatest, most relevant discussions. Those happen on the internet as a whole, not inside walled gardens. This mess free for some, paid for others mess is to much of a kludge to be the model.
- frandroid 9y agoIt's fine talking about messes, but _someone_ has to pay for the content, and display ads don't cut it. Paywalls will be the dominant form of monetization for most mainstream news orgs in the next 5 years. The second most important one will be the Guardian's supporter model, which is a paywall lite. It's still not clear this model will work at the Guardian is still bleeding money. I think eventually you'll see a re-arranging of press wire cooperatives such as AP and CP to limit dissemination of wire content for free without strings attached, forcing the hand of many.
- bigbugbag 9y agoThe content is usually crap used to brainwash people, so let the shareholders pay for it. Besides once _someone_ has paid then no one needs to pay ever again because sharing and digital copies. The business model is wrong and broken, and this failed business model is not our problem.
- JokerDan 9y agoI honestly prefer to not use pay walled websites. I also run adblock but in a whitelist mode, sites i often visit, find entertaining and worthy of any money from my activity, get whitelisted.
- gnicholas 9y agoAnother way to get free access to the WSJ, which doesn't require stepping into a legally/ethically grey area, is to use one of the handful of apps/extensions that has been granted free access. The WSJ approached me and offered this access to a project I'm building, Read Across The Aisle [1]. I've built an iOS app and Chrome extension, both of which are free. I don't know exactly what the WSJ gets out of this deal (we have not given them, or anyone else, any user data), but I think it's that they want to be associated with post-filter bubble projects. 1: http://www.readacrosstheaisle.com http://www.readacrosstheaisle.com
- bypasspaywalls 9y agoYou can download my Firefox plugin to bypass the WSJ paywall (also bypasses FT paywall): https://addons.mozilla.org/en-US/firefox/addon/bypasspaywalls/ https://addons.mozilla.org/en-US/firefox/addon/bypasspaywall... And if you want the Chrome version you will need to manually download it: http://bypasspaywalls.weebly.com/ http://bypasspaywalls.weebly.com/