6 ms·

Building a Dark Web Crawler in Go

283 points by aadlani 7y ago

zhdc1 7y ago
Crawlers are fun! If you're new to the field and want something that's easy to set up & polite, I strongly recommend Apache Storm Crawler (https://github.com/DigitalPebble/storm-crawler https://github.com/DigitalPebble/storm-crawler).
sbmthakur 7y ago
A well written article with lot of technical details. Well done. However, I'm wondering what would be a good practical purpose of crawling dark web.
- creekorful 7y ago
  Thank you! There's no practical purpose for the crawler. It's more an educational project than anything.
  - warent 7y ago
    Weird, for some reason your comments are being instantly marked as "dead." I think there's some kind of filter that's tripping out for your account since it's new. I vouched for your two comments so hopefully everyone can see them now, but an admin (i.e. dang?) will need to look into this for a longer term solution.
    - creekorful 7y ago
      Thank you sir. Actually my other comments are invisible too. That's weird.
bureaucrat 7y ago
First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
- malux85 7y ago
  > First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?
  - creekorful 7y ago
    Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety
    - sandworm101 7y ago
      It's now just "Tor" and when accessing hidden services it isnt really a proxy. The Onion Router acronym went away back with the vidalia proxy. But i do miss the oldschool torbutton. It was fun.
      - creekorful 7y ago
        the proxy which I refer is the local SOCKS proxy used by tor browser / your applications to be routed through the network.
    - malux85 7y ago
      Thank you. > The dark web is a vast groups of services that cannot be accessed without using special software / proxy. Can you name a few for examples?
    - xtracto 7y ago
      > The dark web is a vast groups of services that cannot be accessed without using special software This takes me to the 1990's.... "groups of services that cannot be accessed without using special software" definitely matches NNTP, FTP, SMTP, SSH, HTTP, etc.
rolltiide 7y ago
I’ve been pretty surprised at how big hidden services have become Dread, the dark net reddit, is surprisingly vibrant I think its weird that people almost don't want to hear positive stories about dark net. It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope
- Phenomenit 7y ago
  I thought dread was dead?
  - rolltiide 7y ago
    Its not, hit up dark fail for the onion link to dark fail and browse the latest onion links
  - CryptoBanker 7y ago
    So is bitcoin they tell me...
fs111 7y ago
Any http-aware software that supports socks proxies can access information on hidden services, so any crawler can do it. I fail to see what is novel about that, except that it uses k8s and mongo and a catchy blog title.
jmnicolas 7y ago
I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.
- creekorful 7y ago
  You are right. That's why it's an educational project and not a public search engine
  - mellosouls 7y ago
    IANAL but "educational project" won't fly in court, and nor should it.
    - andrewjrhill 7y ago
      Programs like https://www.hacksplaining.com/ https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.
      - F147H34D 7y ago
        Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. I would tread lightly crawling the dark web. There are cases where the FBI has admitted to running services on TOR, to collect IP addresses: https://www.wired.com/2013/09/freedom-hosting-fbi/ https://www.wired.com/2013/09/freedom-hosting-fbi/
        apta 7y ago
        > Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. What about when the FBI/CIA does it? Genuine question.
Hitton 7y ago
Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just absolute ones.
- creekorful 7y ago
  Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and has to evolve. You can make a PR If you want to help me to improve it!
  - bluesign 7y ago
    I think 75000 comment is coming from stats[1]. [1] https://metrics.torproject.org/hidserv-dir-onions-seen.html https://metrics.torproject.org/hidserv-dir-onions-seen.html
  - deleted 7y ago
    [deleted]
  - akklesed 7y ago
    Offtopic nitpick: >Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is. v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite work to calculate real theoretical capacity. IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course). [1] https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions https://trac.torproject.org/projects/tor/wiki/doc/NextGenOni...
mschuster91 7y ago
To anyone experimenting with such stuff, take care and don't make your services publically available. Especially the dark web is full with highly illegal content such as child pornography and in some jurisdictions even "involuntary possession" such as in browser caches may be enough to convict you.
- creekorful 7y ago
  Do you think I should add a license in Github to mention that? To protect me and the users who will use the crawler?
  - weatherlight 7y ago
    yes.
Havoc 7y ago
Sounds like a recipe to score yourself a free FBI visit
- penagwin 7y ago
  Generally the FBI doesn't give a hoot until you start distributing illegal stuff....
  - fishtacos 7y ago
    What does suck is being put on IP blacklists by various providers for merely running a Tor relay, not an exit node. There are several websites I can only access through VPN because of my IP is associated with running a relay.
goatsi 7y ago
How well does it handle a gzip bomb? https://www.hackerfactor.com/blog/index.php?/archives/762-Attacked-Over-Tor.html https://www.hackerfactor.com/blog/index.php?/archives/762-At...
getpolarized 7y ago
Go is a horrible language in which to write a crawler. The main problem is that NLP and machine learning code simply isn't as prevalent and robust as it is in Java and Python.
- marcrosoft 7y ago
  Go is great for a crawler. What does NLP and ML have to do with crawling?
seisvelas 7y ago
I did the same in Racket when I made a Tor search engine. Here's the source code of the crawler! https://github.com/torgle/torgle/blob/master/backend/torgle.rkt https://github.com/torgle/torgle/blob/master/backend/torgle....
woodandsteel 7y ago
So how well would this thing work? What I am asking is what percentage of all the tor hidden service sites out there would get detected by it?

283 points by aadlani 7y ago

Crawlers are fun! If you're new to the field and want something that's easy to set up & polite, I strongly recommend Apache Storm Crawler (https://github.com/DigitalPebble/storm-crawler https://github.com/DigitalPebble/storm-crawler).

sbmthakur 7y ago

A well written article with lot of technical details. Well done. However, I'm wondering what would be a good practical purpose of crawling dark web.

creekorful 7y ago
Thank you! There's no practical purpose for the crawler. It's more an educational project than anything.
- warent 7y ago
  Weird, for some reason your comments are being instantly marked as "dead." I think there's some kind of filter that's tripping out for your account since it's new. I vouched for your two comments so hopefully everyone can see them now, but an admin (i.e. dang?) will need to look into this for a longer term solution.
  - creekorful 7y ago
    Thank you sir. Actually my other comments are invisible too. That's weird.

bureaucrat 7y ago

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

malux85 7y ago
> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?
- creekorful 7y ago
  Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety
  - sandworm101 7y ago
    It's now just "Tor" and when accessing hidden services it isnt really a proxy. The Onion Router acronym went away back with the vidalia proxy. But i do miss the oldschool torbutton. It was fun.
    - creekorful 7y ago
      the proxy which I refer is the local SOCKS proxy used by tor browser / your applications to be routed through the network.
  - malux85 7y ago
    Thank you. > The dark web is a vast groups of services that cannot be accessed without using special software / proxy. Can you name a few for examples?
  - xtracto 7y ago
    > The dark web is a vast groups of services that cannot be accessed without using special software This takes me to the 1990's.... "groups of services that cannot be accessed without using special software" definitely matches NNTP, FTP, SMTP, SSH, HTTP, etc.

rolltiide 7y ago

I’ve been pretty surprised at how big hidden services have become Dread, the dark net reddit, is surprisingly vibrant I think its weird that people almost don't want to hear positive stories about dark net. It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope

Phenomenit 7y ago
I thought dread was dead?
- rolltiide 7y ago
  Its not, hit up dark fail for the onion link to dark fail and browse the latest onion links
- CryptoBanker 7y ago
  So is bitcoin they tell me...

fs111 7y ago

Any http-aware software that supports socks proxies can access information on hidden services, so any crawler can do it. I fail to see what is novel about that, except that it uses k8s and mongo and a catchy blog title.

jmnicolas 7y ago

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

creekorful 7y ago
You are right. That's why it's an educational project and not a public search engine
- mellosouls 7y ago
  IANAL but "educational project" won't fly in court, and nor should it.
  - andrewjrhill 7y ago
    Programs like https://www.hacksplaining.com/ https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.
    - F147H34D 7y ago
      Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. I would tread lightly crawling the dark web. There are cases where the FBI has admitted to running services on TOR, to collect IP addresses: https://www.wired.com/2013/09/freedom-hosting-fbi/ https://www.wired.com/2013/09/freedom-hosting-fbi/
      - apta 7y ago
        > Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. What about when the FBI/CIA does it? Genuine question.

Hitton 7y ago

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just absolute ones.

creekorful 7y ago
Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and has to evolve. You can make a PR If you want to help me to improve it!
- bluesign 7y ago
  I think 75000 comment is coming from stats[1]. [1] https://metrics.torproject.org/hidserv-dir-onions-seen.html https://metrics.torproject.org/hidserv-dir-onions-seen.html
- deleted 7y ago
  [deleted]
- akklesed 7y ago
  Offtopic nitpick: >Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is. v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite work to calculate real theoretical capacity. IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course). [1] https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions https://trac.torproject.org/projects/tor/wiki/doc/NextGenOni...

mschuster91 7y ago

To anyone experimenting with such stuff, take care and don't make your services publically available. Especially the dark web is full with highly illegal content such as child pornography and in some jurisdictions even "involuntary possession" such as in browser caches may be enough to convict you.

creekorful 7y ago
Do you think I should add a license in Github to mention that? To protect me and the users who will use the crawler?
- weatherlight 7y ago
  yes.

Havoc 7y ago

Sounds like a recipe to score yourself a free FBI visit

penagwin 7y ago
Generally the FBI doesn't give a hoot until you start distributing illegal stuff....
- fishtacos 7y ago
  What does suck is being put on IP blacklists by various providers for merely running a Tor relay, not an exit node. There are several websites I can only access through VPN because of my IP is associated with running a relay.

goatsi 7y ago

How well does it handle a gzip bomb? https://www.hackerfactor.com/blog/index.php?/archives/762-Attacked-Over-Tor.html https://www.hackerfactor.com/blog/index.php?/archives/762-At...

getpolarized 7y ago

Go is a horrible language in which to write a crawler. The main problem is that NLP and machine learning code simply isn't as prevalent and robust as it is in Java and Python.

marcrosoft 7y ago
Go is great for a crawler. What does NLP and ML have to do with crawling?

seisvelas 7y ago

I did the same in Racket when I made a Tor search engine. Here's the source code of the crawler! https://github.com/torgle/torgle/blob/master/backend/torgle.rkt https://github.com/torgle/torgle/blob/master/backend/torgle....

woodandsteel 7y ago

So how well would this thing work? What I am asking is what percentage of all the tor hidden service sites out there would get detected by it?