4 ms·

> Each task comes from a private production codebase that we licensed from a real-world company How does that work?

gruez 4d ago
From the same site: https://withspecific.com/company-data https://withspecific.com/company-data
traceroute66 4d ago
> How does that work? My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not. I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.
- InsideOutSanta 4d ago
  Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money. At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.
  - traceroute66 4d ago
    The point I'm making is that companies who are serious enough to want to keep their code-base in-house and off the various online repo services are also the kind of companies who are strict about what you can and cannot do with LLMs (if they permit use of LLMs at all). So it does not make sense that the same companies would then magically sign-off on allowing their entire codebase to be spoon-fed into a whole bunch of LLMs for benchmarking.
strobe 4d ago
lot of ads everywhere offering to buy your codebase of real product/star up even it long gone or failed (offer usually price per lines of code). So most likely that they have bunch of abandoned codebases between small and medium sizes and probably also some fake codebases as well.
- janaksunil 4d ago
  we manually vet all codebases and companies
janaksunil 4d ago
we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app store.

From the same site: https://withspecific.com/company-data https://withspecific.com/company-data

> How does that work? My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not. I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.

InsideOutSanta 4d ago
Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money. At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.
- traceroute66 4d ago
  The point I'm making is that companies who are serious enough to want to keep their code-base in-house and off the various online repo services are also the kind of companies who are strict about what you can and cannot do with LLMs (if they permit use of LLMs at all). So it does not make sense that the same companies would then magically sign-off on allowing their entire codebase to be spoon-fed into a whole bunch of LLMs for benchmarking.

strobe 4d ago

lot of ads everywhere offering to buy your codebase of real product/star up even it long gone or failed (offer usually price per lines of code). So most likely that they have bunch of abandoned codebases between small and medium sizes and probably also some fake codebases as well.

janaksunil 4d ago
we manually vet all codebases and companies

janaksunil 4d ago

we reached out to companies that were willing to license their codebases. every codebase we used had real users, one of them had 200k+ users and is currently top 100 on the app store.