5 ms·

/*/tree is only for directory listings. File contents will be under a /blob/ path, e.g. https://github.com/facebook/react/blob/main/AUTHORS https://github.com/f

by knute 4y ago

/*/tree is only for directory listings. File contents will be under a /blob/ path, e.g. https://github.com/facebook/react/blob/main/AUTHORS https://github.com/facebook/react/blob/main/AUTHORS, and should be, AFAIK, indexable. (mandatory disclaimer: I'm a GitHub employee, not speaking on behalf of the company)

staplung 4y ago
If they can't hit `/*/tree` is there a way to know the URLs of the files?
- kadoban 4y ago
  Sure, clone the git repo.
  - utopcell 4y ago
    GitHub would not be happy with Google cloning all repos, and many of them at a high frequency, in order to circumvent a robots.txt restriction.
    - kadoban 4y ago
      They're clever people, they could just do partial updates (pull instead of clone). I doubt it'd be that much of a strain.
- pancrufty 4y ago
  Direct links from crawlable pages
simonw 4y ago
I asked about this on the support forum a while ago and never got a satisfactory response: https://github.com/community/community/discussions/20958 https://github.com/community/community/discussions/20958