8 ms·
BeGoneAds – A Python script that blocks ads by installing common hosts files
- AdmiralAsshat 7y agoPython scripts to modify system files make me a little skittish, even with source code available. I think I would just as soon grab the hosts file from https://www.someonewhocares.org/ https://www.someonewhocares.org/ and drop it in myself.
- Redoubts 7y agoI also don't understand systemd integration in the todo.
- anned20 7y agoThe goal here is to make it automatically update the hosts file once in a while
- deleted 7y ago[deleted]
- gumby 7y agoIsn’t that what cron is for?
- josteink 7y agoSystemd has a cron system too. I’m just guessing, but it might be easier to programmatically install a systemd cron job (and making sure it runs) than doing so for the old/conventional crond?
- gumby 7y agoJust using cron is back compatible
- Spivak 7y agoThings you get with systemd that you don't get with cron unless you implement them in the script. * Not running if the network is down. * Not running if the download path isn't available. * Running if the machine was off during the scheduled time. * Monitoring and retry logic. * Logging to syslog. * Resource constraints. * Random wait. It ends up being a lot of code factored out of the actual application.
- behringer 7y agoso instead of code in one place you have it in 2?
- JustSomeNobody 7y agoI don’t understand this logic. Can you explain why having code in more than one place is a bad thing?
- behringer 7y agoFor a simple script that does one thing, it's overkill.
- josteink 7y agoYes. So you keep that simple script simple, and you let systemd do the heavy lifting.
- behringer 7y agoBut systemd doesn't do any of the above listed things automatically... You'd need to write an entire script for sytstemd to take advantage of it. An entire script you could just have run with cron on boot.
- naikrovek 7y agoHe probably wants to learn more about systemd and needs/wants a project to do it. Yes, obviously cron would work. If systemd can do what this dev needs, what's the harm?
- anned20 7y agoYou're right, but I like the extensibilty of Systemd. Things like running every week or whenever the next start is. Or only if the network is online.
- deleted 7y ago[deleted]
- dvfjsdhgfv 7y agoIf I considered constructing a botnet of Linux workstations, your software would be an excellent candidate.
- farisjarrah 7y agoYou could say the same thing about Ansible, or Chef or Puppet or any of the other millions of systems automation tools available. There are lots of ways to misuse software, I don't think that that should preclude people from writing it.
- dvfjsdhgfv 7y agoI'm quite precise about about what these tools can and cannot do on my systems. Downloading random files form the Internet isn't on the list. When you hand over the control of the hosts file to someone else, you're basically transfer control of your DNS queries.
- michaelmior 7y agoAt least for HTTP, you're hopefully using TLS for anything important and failing if the certificate isn't valid. That certainly won't remove all the risks of losing control of your DNS, but it's one good safeguard.
- IggleSniggle 7y agoSome of my favorite hacks in the last year have been about using valid certs for bad actions. When you can have a cert from Microsoft (Azure), there’s a lot of things people will trust. Along similar lines, I think I heard that 30% of detected malware was signed with a “trusted” authority last year.
- stuntkite 7y agoDoesn’t automation of editing your hosts file potentially open people up to fairly insidious man in the middle attacks?
- joyjoyjoy 7y agoWhy?
- LASR 7y agoWhy not? Modifying system hosts configuration requires privileged file system access. The mindset here should be default deny.
- sh-run 7y agoYep, finding and modifying a script that runs with root privileges, but is writable by non-root users is the oldest privesc trick in the book. With the proper permissions something like this should be ok, but I'd tread lightly. Especially with something that dynamically updates your hosts file.
- anned20 7y agoIt won't be able to run if the user that is running it doesn't have the proper privileges. You could even protect the files by giving them other permissions so only the root user can use them.
- sh-run 7y agoAbove you mentioned setting this up as a scheduled job. In this case the job would need to run as root (or you'd need to assign the appropriate permissions in the sudoers file, but people are lazy). If a non-root user had write privileges to the file, they could modify the script and thereby gain root code execution. Naturally it's on the user to properly configure the permissions. I'm not saying this isn't a worthy project, I'm just adding to the discussion on why people should be cautious when running scripts with root permissions.
- joyjoyjoy 7y agoI don't understand the downvote. 1. What is so bad about python in specific? 2. If you worry about root privileges, required for modifying the host file, you can use app armor to put the thing on a leash
- digitalni 7y agoThat hosts file is depressing. Blocking trackers by a1.tracker.name, a2.tracker. name etc. Today it seems easier to have a whitelist rather than blacklist...
- barbecue_sauce 7y agoAnybody have a sense of the performance overhead of using hosts files versus a detached hardware solution like a pihole?
- LyndsySimon 7y agoI don't have any evidence (I've not attempted to benchmark it or anything), but my gut says that the stack is checking the hosts file first anyhow, so it shouldn't be much. It might actually be an improvement over a separate appliance.
- NikolaNovak 7y agoMy understanding is that difference is in scope, not performance. Hosts files will only affect the host (workstation/desktop/laptop etc) they're installed on. Things like piHole try to make it easy to apply the solution to all members of your network - which even in household cases these days can number in dozens, making it impractical to manage hosts files for all of them (This includes items like phones which are typically unfeasible to mess with hosts file).
- ycombonator 7y agoIt would be nice to see a performance hit based on the number of hosts entries.
- dredmorbius 7y agoAbout 3ms for 68k entries: https://news.ycombinator.com/item?id=20148457 https://news.ycombinator.com/item?id=20148457
- ImprovedSilence 7y agoPie hole also has a nice browser interface to debug blocked requests that are breaking the site you don’t want To be broken. Which happens inevitably when you pull together 10 different sources of blocked lists.... or just one persons whose ideal blacklist doesn’t match yours.
- 7y ago
- sherincall 7y agoI get that this is just someone's side project, I'm glad it exists and they're free to write it in their favorite language/environment and all; but the effort to actually run this is equivalent to actually copying the hosts files manually, and I already have all the dependencies installed. I could never get my non-techy parents to run this properly. If the goal of the project is actual adoption, a native executable without external dependencies would have been a much better option.
- anned20 7y agoThis is a todo, It's already on PyPI and I'm working on getting it packaged for all the main distros of Linux/Windows and MacOS.
- sherincall 7y agoI saw the todo item, but didn't realize it also included providing an executable for Windows and macOS. Thanks!
- inlined 7y agoSince accepting Host files from someone on the internet can be dangerous I dug into the code: The list of hosts to exclude comes from several sites here: https://github.com/anned20/begoneads/blob/2c90fcee221edf71f870b1282bad3d25ac151488/begoneads/begoneads.py#L15 https://github.com/anned20/begoneads/blob/2c90fcee221edf71f8... The actual application of the hosts file is here: https://github.com/anned20/begoneads/blob/2c90fcee221edf71f870b1282bad3d25ac151488/begoneads/hostsmanager.py#L32 https://github.com/anned20/begoneads/blob/2c90fcee221edf71f8... I missed something though. Is a simple domain name per line enough to send that content to /dev/null? I haven’t used that form in /etc/hosts. My primary concern was that this technique could be used to send ad traffic to a site that returns 404 but gathers metrics on the web regardless.
- anned20 7y agoThis is actually sent to the IP address 0.0.0.0, it roughly means that all the traffic of the listed hosts is routed back to localhost
- lelf 7y ago0.0.0.0 ≠ localhost
- inlined 7y agoIndeed. Traffic served from 0.0.0.0 can be seen from other machines. Traffic on 127.0.0.1 or localhost cannot. Important to know when you’re doing local development vs a local demo.
- sp332 7y agoIP addresses with 0 as the first octet are invalid and hopefully will not be routed. I prefer them for hosts files over localhost because localhost has to wait and time out, but 0.0.0.0 will fail right away.
- 7y ago
- daverobbins1 7y agoHow is this different or better than Steven Black's project? Repo: https://github.com/StevenBlack/hosts https://github.com/StevenBlack/hosts
- DyslexicAtheist 7y agomy first question too. Steve Black's updateHostesFile.py is extensible and can be automated and is well trusted and gets tested by a large community. I don't want to sound critical but would like to understand the value add in comparison. Both are in python as well so I don't get it
- fuzz4lyfe 7y agoI assume you wrote a hello world once or twice, what was the point of that?
- DyslexicAtheist 7y agosure I did but I didn't bother with a "Show HN", also this is well beyond the scope of a Hello World. It just made me wonder if OP hasn't studied what was already out there and if he did why no explanation or credit or mention of Steven's work ... even if he disagreed with Steven solution and chose to reimplement, it would have been interesting to understand what the motivation was. I'm not saying he shouldn't, but just that it would be nice to know what motivated his design and why he thinks it's better to redo ...
- daverobbins1 7y agoWho posts their hello world on HN? My question was a valid one given the circumstances.
- pvg 7y agoPeople posts simple projects, learning projects, projects that are inadvertently a dupe of some other project, projects that are deliberately patterned after existing projects, etc, etc, etc all the time.
- joyjoyjoy 7y agoI use host flash: http://host-flash.com/ http://host-flash.com/ Does anyone know an up-to date list for blocking social networks?
- DyslexicAtheist 7y agoSteve Black's hosts file ... just specify "-e social" or "--extension social" option, or use a "myhosts" file to name your own domains for a subset (e.g. all of facebook or whatever) https://github.com/StevenBlack/hosts https://github.com/StevenBlack/hosts
- jakeogh 7y agoNice to see projects using click! Here's another one to toss on the pile (works, I use it, supports wildcards, *nix only): https://github.com/jakeogh/dnsgate https://github.com/jakeogh/dnsgate
- zactato 7y agoSerious question. We all realize that the economics of the internet is largely fueled by ads, so why are we so keen to block them? It’s ad revenue that have allowed technology to flourish so strongly over the last two decades.
- dsswh 7y agoNot long ago, the economy was largely fueled by slavery. Yet we got rid of that.
- layoutIfNeeded 7y agoCitation needed
- briandear 7y agoNot exactly Godwin’s law, but pretty close. Comparing advertising and marketing to the ownership of human beings? Slavery infringed on the inalienable of human beings, the existence of advertising doesn’t take away my freedom or potentially subject me to beatings. It’s a ridiculous comparison. I am not a friend to intrusive ad-tech, but making a moral equivalence to slavery is to trivialize slavery. It’s like comparing parking tickets to the death penalty.
- jancsika 7y ago> Comparing advertising and marketing to the ownership of human beings? That's not exactly what happened here. OP repeated the beginning of a truism: "ads fund the development of the web while at the same time causing a whole host of severe problems for its users (individually and as a whole)." OP left a hole where the italized part of the truism should be. OP asked HN to fill that hole. On a side devoted to tech/software, it's either low effort or bad-faith to ask others to fill a hole in such a well-known truism. In light of this I offer up a countervailing law, "Loki's Law:" "If you leave a Hitler-sized hole in your argument, expect it to be filled accordingly."
- dsswh 7y agoIt's a valid comparison. Long ago it was ok to kill your enemy. Not long ago it was ok to have slaves. Today either is a sure way to end up in prison. Standards are rising. IT is a very new thing and the society and the laws are behind a bit. Adtech uses this to extract profit while it can. But this will end. Soon it will be a crime to store personal data: names, location, anything like that. GDPR is just the beginning. Adtech will fight, but it will lose. This business will disappear entirely, just like slave labor. In far future it will be a crime to be intrusive: any unwanted ads; and mining personal data will be seen like cannibalism today, i.e. even criminals will consider such people as freaks. Right now we are in the era of wild west in IT.
- rafaelvasco 7y agoReading the code one clearly sees why Python is so well suited for these kinds of applications, one-shot script executables: Really nice string ops, regex, file io etc. One of my favorite languages. The other is C# for everything else, that Python is not that suitable for: Huge complex codebases, type safeness, more strict performance requirements etc. Specially the static typing. The dynamism and lack of type annotations of Python really bothered me when I was developing a somewhat complex desktop app in it some years ago. I guess I'm a static typing guy with optional dynamism kinda person.
- misterdoubt 7y agoIf you haven't checked back lately, type annotations in Python are getting better and better. Built-in support via the typing module and a strong community package in mypy.
- firefoxd 7y agoI wish the hosts file could have an include directive. Since I regularly add or remove entries, the file becomes a mess.
- ris 7y ago> You ran WHAT script on your machine?!
- hlau 7y agoI'll be the first to admit that the existing advertising ecosystem is broken, primarily due to misaligned incentives across the board. But, given a choice, would you rather have a clearly labeled thing that you know is an ad transparently trying to influence you or a sneaky human billboard, err "influencer" coming up to you with an agenda along with tons of product placement in whatever you watch/read/listen to?
- harry8 7y agoThere's no either/or decision to be made here. You get compromised, paid for content with our without ads as well. Critical thinking I'd a requirement always.
- hlau 7y agoThere definitely is an either/or because blocking of one channel will naturally necessitate money/barter flowing to the other channel. One is at least transparent and regulated, the murky world of influence peddling isn't since it's hard for anyone to tell in the moment whether something is "organic" or not.
- harry8 7y agoNot when the other channel is already at capacity. And it is. Blocking ads has no effect on that. You never agreed to being tracked either, so blocking that is the right and proper thing to do. Blocking surveillance capitalism might push businesses toward honesty, it's at least with a shot.
- hlau 7y agoIf you think influencer marketing and product placement are already at capacity you have no idea how much worse it's about to get if ad blocking gets much worse. And the irony is that, by design, you won't know a good chunk of the time and other times it'll just merely be implied without being explicitly stated. Continued use of social networks, including this one, collects way more identifiable data than what the non-Google/FB/Amazon ad market collects. Ad blockers have had near 0 impact on FB's operations. Google and others have paid to ensure that their search ads still make it through most ad blockers. Blocking ads does not drive businesses to be more "honest". They'll just spend more on PR and influencers. And given how hostile this community is to ads and perhaps even marketing overall, (how YC ever backed a marketing or ad startup is beyond me), companies already realize that getting a fawning TC article purchased thru connections and favors and PR chicanery is going to be more effective than ad campaign even though the ad campaign is more honest, upfront and transparent with its agenda.
- deleted 7y ago[deleted]
- mehrdadn 7y agoHosts files slow down the system as well as the browser itself. Get/create a browser extension to actually block the request (at least while your browser supports this) so you get immediate results.
- bigend 7y agoIf you let someone else manage the hosts your computer resolves, you are trusting that someone as much as your ISP. A man in the middle.
- gregw2 7y agoI have cron jobs on my mac that update my hosts files (to block "addictive" sites in my case (not ads)). It doesn't really work. Browsers cache and use outside DNS servers despite the hosts files. Chrome and sometimes Safari don't really honor the hosts files 100% of the time. Every once in a while I google around to try and restore my control, try to tweak my browser settings but I have yet to find anything that makes using hosts files bulletproof.
- mywittyname 7y agoI think firewall rules would be your next line of defense. I'm not sure how configurable most home routers are though.
- appleflaxen 7y agohow does the list compare to the pi hole hostfile?