5 ms·
Show HN: Sleeping Spider – Web page change monitor
- akx 12y agoTypo in the footer: "Sleeping Sider"
- deleted 12y ago[deleted]
- sleepingspider 12y agoThank you. Fixed.
- etw 12y agoWhat's your twitter id. You're trending on PH.
- sleepingspider 12y agoThe product doesn't have a twitter as the name was occupied by others. :) Personally I don't use twitter neither. Email me if you need to contact.
- etw 12y agoIf you had a personal twitter account, I could have granted you commenting access on Product Hunt. That's why I contacted you.
- jmomo 12y agoCreepy sleepy drinky spider cider. I feel a children's book coming out of this.
- seivui 12y agoWhich timezone does "8 o'clock in the morning" consider?
- deleted 12y ago[deleted]
- sleepingspider 12y agoIt will try to detect your time zone with JavaScript. If that doesn't work, it falls back to Los Angeles and the user may contact the support to update this setting.
- callesgg 12y agoWhat is regarded as a change?
- sleepingspider 12y agoText content changes. For example the site corrected a typo or published a new post. Change <h1> to <h2> will not trigger an alert.
- lucasjans 12y agoAdding watch results in a white screen of nothing, no new watches added. I'm trying to monitor the Nexus 6 page :D https://play.google.com/store/devices/details/Nexus_6_32GB_Cloud_White?id=nexus_6_white_32gb https://play.google.com/store/devices/details/Nexus_6_32GB_C... For my use case, anything more than an hour delay wouldn't be practical. Ideally it would be up to the minute, but I'd imagine that would cause other issues for a service like this. In the past I just used a Chrome Extension. But living in Vietnam I have to make sure my VPN is up in order to see the North American content.
- sleepingspider 12y agoThe probe received a 403 forbidden error from google. I realized that google store forbid crawls. So a simple http request won't work. I was not aware of this situation before. I will see what I can do.
- _asciiker_ 12y agosimple, clean and useful. I'm trying it out. Good job!
- TheDong 12y agoDoes this site respect robots.txt? Does it render javascript content (so e.g. <div ng-view> angular sites work)? Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes? Will I get a notification if the page briefly 500 errors when you hit it, or does it have to get a 200 before it reports a change? Will I get a notification if a page disappears entirely? Does it scrape only once per day or more often and alert only once a day? What user-agent does it scrape from on the off-chance someone wants to block it? Does it diff binary data.. for example, if I ask it to watch an image url will it hash the image and notify me when it gets updated? Does it auto-detect pages with rss feeds or pubsub feeds and subscribe to them? Does it understand twitter so if I watch a user, it only alerts me of tweets, not e.g. a change in the number of followers? Does it have special code for any website whatsoever? Is the webpage parsing/diffing library / algorithm open source somewhere? Is there a monitization plan, and if not how can I trust it won't vanish? Is there a limit on the number of sites I can watch? I like the simplistic design, but it would be nice if there were more information.
- mattmanser 12y agoGo write your own then. I find this a bizarre response to a simple and easy to understand tool.
- TheDong 12y agoIf it's easy to understand then I guess you can answer my questions? I don't think "go write your own" is a legitimate response to my questions (if that's ever a legitimate response).
- baruch 12y agoI'm using websec as an existing tool to watch website changes. It's not perfect but it's good enough. For a while I had a webapp to do a better job but it never got any traction, most likely because I had no idea how to get traction. It's an interesting exercise to write such a tool though.
- 12y ago
- aw3c2 12y agoIf you prefer privacy and control over your own tools, there are the FLOSS http://specto.sourceforge.net/ http://specto.sourceforge.net/ or the free(as in no cost)ware http://www.cmcode.co.uk/webmon/ http://www.cmcode.co.uk/webmon/ for Windows
- PavlovsCat 12y agoThere is also the commercial, but very extensive Website-Watcher: http://aignes.com/ http://aignes.com/
- nousougi 12y agoI too used webmon until recently, but got tired of monitoring the monitoring tool on my desktop. Now I use ChangeDetect - http://www.changedetect.com/ http://www.changedetect.com/ Despite what is said on the homepage, it is NOT free. Good luck...
- netman21 12y agoIn the most amazing coincidence the first two pages I submitted had the same digest: 90c0568! One was the front page of a news site, the other was the management page of a company I follow. The mind boggles. Or, they have a bug. edit: as did the 3rd. And 0 bytes returned so probably being blocked. On a black list already?
- nacs 12y agoGetting the same digest (90c0568) with no response code and a 0 byte response size so looks like something is broken (the crawler maybe).
- sleepingspider 12y agoProblem fixed. All failed probes retried. Thanks!
- watson 12y agoIs it only for HTML, or will it alert on changes to JavaScript files for instance?
- sleepingspider 12y agoIt's only for HTML. But if you have a need for JavaScript files, just let me know, it's technically possible, I just don't understand the use case yet.
- zapt02 12y agoNice branding!
- ideaoverload 12y agoLooks very similiar to https://followthatpage.com https://followthatpage.com . Worked for me pretty well for non JS pages for last 3 years.
- girishso 12y agoI just built "Page Poller" for the Rails Rumble 2014, same concept! http://pagepoller.r14.railsrumble.com/ http://pagepoller.r14.railsrumble.com/
- sleepingspider 12y agoJust added HTTP authentication support.
- sleepingspider 12y ago> Does it diff binary data.. for example, if I ask it to watch an image url will it hash the image and notify me when it gets updated? Can you elaborate in what case you might want to do this?
- sleepingspider 12y ago> Does it understand twitter so if I watch a user, it only alerts me of tweets, not e.g. a change in the number of followers? What do you think if I provide a numeric filter option to watches, which filters out numbers on the page before comparing if turned on.
- sleepingspider 12y ago> Does it exclude or include comments... if I watch a blogpost with comments will every new comment cause an alert or only blogpost changes? I don't have a good solution for this problem yet. Let the user specify a css selector or xpath?