7 ms·
I have written a search engine that works just like google but 100x safer and with no trackers, right now my database has over 3,238,313 rows of data yet is und
by nsturtz 6y ago
I have written a search engine that works just like google but 100x safer and with no trackers, right now my database has over 3,238,313 rows of data yet is under 1/2 of a GB. It is under construction at https://sturtz.cf https://sturtz.cf it is written in PHP and MySQL. I wrote it to get my family off of Google and never thought it would go any where ever. But I working at it and here I am, growing by over 1 million links a day. You can find the Source at https://git.sturtz.cf https://git.sturtz.cf Thank you for reading this. :)
The Crawler works but finding the URL with: (https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/src/master/admin/spider.php#L1351 https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...)
if (mbsubstr($element->href, 0, 2) == "//") {
$element->href = $urlparts['scheme'].":".$element->href;
}
if (mbsubstr($element->href, 0, 1) == "/") {
$element->href = $base_url.$element->href;
}
if (mbsubstr($element->href, 0, 2) == "./") {
$element->href = substr($element->href, 2);
}
if (mbsubstr($element->href, 0, 5) != "http:"
&& mbsubstr($element->href, 0, 6) != "https:"
) {
$element->href = $base_url.$url_path.$element->href;
}
which looks for the href tag and will send the spider there to crawl it
then it puts it into the database by some INSERT SQL commands: (https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/src/master/admin/spider.php#L509 https://git.sturtz.cf/Sturtz_Network/sturtz-search-public/sr...)
if ($md5sum == '') {
$stmt = $db->prepare(
"INSERT INTO `".$mysql_table_prefix
."links` (site_id, url, title, description, fulltxt, "
."indexdate, size, md5sum, level) VALUES ( ? , ? , ? ,"
." ? , ? , CURDATE(), ? , ? , ? )"
);
You can find a working version at https://sturtz.cf https://sturtz.cf
- jrruethe 6y agoHow do you determine which sites get crawled?
- nsturtz 6y agoI use the CLI command (php spider.php -u <url> -f (full) -l (leave domain) -r (re-crawl) that is How I crawl. I crawl websites that I use on a daily basis, I also target websites that are for open source projects and the likes, but once I start my spider on a website it will just keep going and going and going hopping different websites. and if you want you can submit a url to nate.sturtz@sturtz.cf it just has to be a SFW website. I hope this answered your question.
- nsturtz 6y ago@jruethe did this answer your question?