josefcullhed

7 ms·

1.
by josefcullhed 4y ago
Great article, I really think programmers should be more conservative when it comes to new technology. You don't always have to use fancy react stuff when making simple websites and so on.. But I'm not sure about the whole "p
2.
by josefcullhed 4y ago
Great job with Marginalia. Do you plan to open source your data as well or only the code?
3.
by josefcullhed 5y ago
Thanks! The active index is running on 4 servers and we have one server for hosting the frontend and the api (the API is what is used by the frontend, ex: https://api.alexandria.org/?q=hacker%20news ) Then we have one filese
4.
by josefcullhed 5y ago
I just think that the timing is right. I think we are in a spot in time where it does not cost billions of dollars to build a search engine like it did 20 years ago. The relevant parts of the internet is probably shrinking and Moore's
5.
by josefcullhed 5y ago
This is actually not used yet. Working on implementing that as a factor.
6.
by josefcullhed 5y ago
We are currently just doing an intersection and then we make a lookup in a forward index to get the urls, titles and snippets. I actually don't know what roaring bitmaps are, please enlighten me :)
7.
by josefcullhed 5y ago
Founder here. We are using harmonic centrality instead of pagerank. But of course much more work needs to be done to make the search engine usable.
8.
by josefcullhed 5y ago
Yes it would be impossible to keep the index in RAM. >> It's hard for me to see how that could be done much faster unless you find a way to parallelize the process We actually parallelize the process. We do it by separating the U
9.
by josefcullhed 5y ago
Hi, Yes our documentation is probably pretty confusing. It works like this, the base score for all URLs to a specific domain is the harmonic centrality (hc). Then we have two indexes, one with URLs and one with links (we index the link text
10.
by josefcullhed 5y ago
Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare so you will be blocke
11.
by josefcullhed 5y ago
The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost. It takes us a couple of
12.
by josefcullhed 5y ago
Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are funding the servers ourse
13.
by josefcullhed 5y ago
Interesting, in Sweden I only got this story when I made the same searches: https://imgur.com/a/k1Avbtm