searchcord

6 ms·

1.
by searchcord 1y ago
This all depends on when the message is scraped. The scraper does not go back and check for edits/deletions. This is something that while could be added, would be incredibly inefficient.
2.
by searchcord 1y ago
> The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless u
3.
by searchcord 1y ago
lol. It's only on the homepage, not on any content pages.
4.
by searchcord 1y ago
This is something I have looked into. Unfortunately, every server uses a different format/bot. It might be possible to develop some sort of machine learning classifier to determine if it is a welcome message.
5.
by searchcord 1y ago
Telegram is already heavily monitored and scraped due to the large volume of illegal or extremely controversial activity that happens there. This is something I will look into though, my XDA threads rarely get any replies anymore. Thanks fo
6.
by searchcord 1y ago
This is something I have been trying to make as a way to learn about graph theory. If I can find a way to make it work efficiently, I will definitely add this.
7.
by searchcord 1y ago
> how are you going to handle the issues involved with breaking Discord's TOS? Not sure. I will solve that problem if and when Discord takes issue with Searchcord. > additionally, are these pages indexable? Yes, I would actually
8.
by searchcord 1y ago
This is something I have already completed but have not finished bug testing. The bot also includes functionality to recover any server in case it was nuked/wiped and Searchcord has a backup of it. It uses webhooks to resend the messag
9.
by searchcord 1y ago
Probably not. Discord will aggressively captcha you and every server has a different implementation of verification. It might be possible with a captcha solver and then some LLM to figure out the next steps.
10.
by searchcord 1y ago
Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This remo
11.
by searchcord 1y ago
Some very large servers are eligible for what Discord calls "discovery". This makes their data visible without joining the server. You can find a list of those on Discord's site here: https://discord.com/serve
12.
by searchcord 1y ago
Hey, This is interesting, I somehow missed this. Unfortunately, those are not full text searchable. Maybe I will download them and import them into Searchcord, with proper credit of course. Thanks for this!
13.
by searchcord 1y ago
I hope Searchcord helps you! <3
14.
by searchcord 1y ago
Hey, This is absolutely something I want to do, but at the guild level. The database itself is over 13TB which is much to large to create regular exports of. I will probably provide a SQLite export of each guild, regenerated each week/
15.
by searchcord 1y ago
Hey, Not exactly. Attachments are only fetched from Discord as the user requests them. This means that the vast majority of attachments are never stored on my server. Right now, I only have about 280TB of attachments locally on my own infra
16.
by searchcord 1y ago
Hey, Thanks for your suggestions. However, this does not work for a few reasons: 1. Joining servers is protected by increasingly difficult to solve captchas that have no commercially available solver. This is not a battle I want to fight. 2
17.
by searchcord 1y ago
Sure, but there's a few petabytes of attachments and over 63 billion messages. Feel free to use the API.
18.
by searchcord 1y ago
Thanks for your feedback! <3
19.
by searchcord 1y ago
I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any
20.
by searchcord 1y ago
Hey, Thanks for your feedback. For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable serve
21.
Show HN: A free, privacy preserving, archive of public Discord servers(searchcord.io)
70 points by searchcord 1y ago|81 comments