4 ms·

Show HN: I created a search engine

1 points by nsturtz 6y ago

nsturtz 6y ago
I have written a search engine that works just like google but 100x safer and with no trackers, right now my database has over 3,238,313 rows of data yet is under 1/2 of a GB. It is under construction at https://sturtz.cf https://sturtz.cf it is written in PHP and MySQL. I wrote it to get my family off of Google and never thought it would go any where ever. But I working at it and here I am, growing by over 1 million links a day. You can find the Source at https://git.sturtz.cf https://git.sturtz.cf Thank you for reading this. :) The Crawler works but finding the URL with: (https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/src/master/admin/spider.php#L1351 https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...) if (mbsubstr($element->href, 0, 2) == "//") { $element->href = $urlparts['scheme'].":".$element->href; } if (mbsubstr($element->href, 0, 1) == "/") { $element->href = $base_url.$element->href; } if (mbsubstr($element->href, 0, 2) == "./") { $element->href = substr($element->href, 2); } if (mbsubstr($element->href, 0, 5) != "http:" && mbsubstr($element->href, 0, 6) != "https:" ) { $element->href = $base_url.$url_path.$element->href; } which looks for the href tag and will send the spider there to crawl it then it puts it into the database by some INSERT SQL commands: (https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/src/master/admin/spider.php#L509 https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...) if ($md5sum == '') { $stmt = $db->prepare( "INSERT INTO `".$mysql_table_prefix ."links` (site_id, url, title, description, fulltxt, " ."indexdate, size, md5sum, level) VALUES ( ? , ? , ? ," ." ? , ? , CURDATE(), ? , ? , ? )" ); You can find a working version at https://sturtz.cf https://sturtz.cf
- jrruethe 6y ago
  How do you determine which sites get crawled?
  - nsturtz 6y ago
    I use the CLI command (php spider.php -u <url> -f (full) -l (leave domain) -r (re-crawl) that is How I crawl. I crawl websites that I use on a daily basis, I also target websites that are for open source projects and the likes, but once I start my spider on a website it will just keep going and going and going hopping different websites. and if you want you can submit a url to nate.sturtz@sturtz.cf it just has to be a SFW website. I hope this answered your question.
  - nsturtz 6y ago
    @jruethe did this answer your question?

1 points by nsturtz 6y ago

I have written a search engine that works just like google but 100x safer and with no trackers, right now my database has over 3,238,313 rows of data yet is under 1/2 of a GB. It is under construction at https://sturtz.cf https://sturtz.cf it is written in PHP and MySQL. I wrote it to get my family off of Google and never thought it would go any where ever. But I working at it and here I am, growing by over 1 million links a day. You can find the Source at https://git.sturtz.cf https://git.sturtz.cf Thank you for reading this. :) The Crawler works but finding the URL with: (https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/src/master/admin/spider.php#L1351 https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...) if (mbsubstr($element->href, 0, 2) == "//") { $element->href = $urlparts['scheme'].":".$element->href; } if (mbsubstr($element->href, 0, 1) == "/") { $element->href = $base_url.$element->href; } if (mbsubstr($element->href, 0, 2) == "./") { $element->href = substr($element->href, 2); } if (mbsubstr($element->href, 0, 5) != "http:" && mbsubstr($element->href, 0, 6) != "https:" ) { $element->href = $base_url.$url_path.$element->href; } which looks for the href tag and will send the spider there to crawl it then it puts it into the database by some INSERT SQL commands: (https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/src/master/admin/spider.php#L509 https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...) if ($md5sum == '') { $stmt = $db->prepare( "INSERT INTO `".$mysql_table_prefix ."links` (site_id, url, title, description, fulltxt, " ."indexdate, size, md5sum, level) VALUES ( ? , ? , ? ," ." ? , ? , CURDATE(), ? , ? , ? )" ); You can find a working version at https://sturtz.cf https://sturtz.cf

jrruethe 6y ago
How do you determine which sites get crawled?
- nsturtz 6y ago
  I use the CLI command (php spider.php -u <url> -f (full) -l (leave domain) -r (re-crawl) that is How I crawl. I crawl websites that I use on a daily basis, I also target websites that are for open source projects and the likes, but once I start my spider on a website it will just keep going and going and going hopping different websites. and if you want you can submit a url to nate.sturtz@sturtz.cf it just has to be a SFW website. I hope this answered your question.
- nsturtz 6y ago
  @jruethe did this answer your question?