13 ms·

The HN/Firebase API doesn't make this easy. For https://hnstream.com https://hnstream.com I ended up crawling items to find the article.

esafak 11mo ago
The comments don't even have a thread ID?
- zamadatix 11mo ago
  Comment items look like https://hacker-news.firebaseio.com/v0/item/45533616.json?print=pretty https://hacker-news.firebaseio.com/v0/item/45533616.json?pri...: { "by" : "jkarneges", "id" : 45533018, "kids" : [ 45533616 ], "parent" : 45532549, "text" : "The HN/Firebase API doesn't make this easy. For <a href=\"https://hnstream.com\" rel=\"nofollow\">https://hnstream.com</a> I ended up crawling items to find the article.", "time" : 1760043552, "type" : "comment" } "parent" can either be the actual parent comment or the parent article, depending where in the comment chain you are.
  - esafak 11mo ago
    Perhaps @kogir, who was active on https://github.com/HackerNews/API https://github.com/HackerNews/API could add the thread id.
  - smusamashah 11mo ago
    https://jaytaylor.github.io/hn-live2 https://jaytaylor.github.io/hn-live2 is doing it though.
    - zamadatix 11mo ago
      As does hnstream.com from the sourced sample comment itself. Both just traverse the parent id until it's the root (article). It takes more queries, but the API is not rate limited.
      - krapp 11mo ago
        It wouldn't take more queries if the comments were cached. It could probably be done entirely in memory, HN's entire corpus can't be that large. If one were to start at the page endpoints (eg /topstories) one could add references to origin ids while preloading comments, and probably cover the most likely to be referenced ID, and even make traversal up the tree even more efficient.
ryandrake 11mo ago
Any tips on respectfully crawling HN so you don’t get throttled? I had an application idea that could not be served by the API (need karma values) so I started to write code to scrape but got rate limited pretty quickly.
- jkarneges 11mo ago
  I've had no trouble hitting the Firebase API at the speed items are created, with a 5 second delay between retries. For scraping HN directly, in my experience you have to go extremely slow, like 1 minute between fetching items. And if you get blocked, it may be better to wait a long time (minutes) before trying again rather than exponential backoff, in order to get out of the penalty box. You'll need a cache for sure.

esafak 11mo ago

The comments don't even have a thread ID?

zamadatix 11mo ago
Comment items look like https://hacker-news.firebaseio.com/v0/item/45533616.json?print=pretty https://hacker-news.firebaseio.com/v0/item/45533616.json?pri...: { "by" : "jkarneges", "id" : 45533018, "kids" : [ 45533616 ], "parent" : 45532549, "text" : "The HN/Firebase API doesn't make this easy. For <a href=\"https://hnstream.com\" rel=\"nofollow\">https://hnstream.com</a> I ended up crawling items to find the article.", "time" : 1760043552, "type" : "comment" } "parent" can either be the actual parent comment or the parent article, depending where in the comment chain you are.
- esafak 11mo ago
  Perhaps @kogir, who was active on https://github.com/HackerNews/API https://github.com/HackerNews/API could add the thread id.
- smusamashah 11mo ago
  https://jaytaylor.github.io/hn-live2 https://jaytaylor.github.io/hn-live2 is doing it though.
  - zamadatix 11mo ago
    As does hnstream.com from the sourced sample comment itself. Both just traverse the parent id until it's the root (article). It takes more queries, but the API is not rate limited.
    - krapp 11mo ago
      It wouldn't take more queries if the comments were cached. It could probably be done entirely in memory, HN's entire corpus can't be that large. If one were to start at the page endpoints (eg /topstories) one could add references to origin ids while preloading comments, and probably cover the most likely to be referenced ID, and even make traversal up the tree even more efficient.

ryandrake 11mo ago

Any tips on respectfully crawling HN so you don’t get throttled? I had an application idea that could not be served by the API (need karma values) so I started to write code to scrape but got rate limited pretty quickly.

jkarneges 11mo ago
I've had no trouble hitting the Firebase API at the speed items are created, with a 5 second delay between retries. For scraping HN directly, in my experience you have to go extremely slow, like 1 minute between fetching items. And if you get blocked, it may be better to wait a long time (minutes) before trying again rather than exponential backoff, in order to get out of the penalty box. You'll need a cache for sure.