10 ms·

Intelligence – A good collection of great OSINT Resources

223 points by g147 4y ago

profstasiak 4y ago
at the start of Russian invasion I helped geolocate one video, claiming that Russians entered Kherson. I found it after about one hour and posted on twitter tagging everyone I know. I got no feedback so got discouraged. But few months later, I noticed that France24 used my tweet to prove Russians entered Kherson and a lot of newspapers followed their lead. Pretty cool I was able to help in a minimally viable way to understand what was happening
- jasmer 4y ago
  Bravo. That is definitely probably the only true 'hacker badge' that I would really want. I would wear that little patch on my jacket at the little meeting for sure.
- g147 4y ago
  that's awesome man. oh and feel free to be encouraged but don't ever get discouraged based on other people's behavior or actions. experts like you are definitely needed in today's world. people will eventually appreciate your research.
- greggarious 4y ago
  [flagged]
  - dmix 4y ago
    A lot of people geolocate stuff on /r/combatfootage for fun because they like following the progress of the war. It's a personal challenge.
    - greggarious 4y ago
      > A lot of people geolocate stuff on /r/combatfootage for fun because they like following the progress of the war. It's a personal challenge. Or because they like gore and /r/watchpeopledie got got.
  - halfmatthalfcat 4y ago
    Living life in fear of reprisal isn’t much of a life. We all should strive to live up to our ideals and convictions, no matter the consequences. If we don’t then what are we actually doing here?
    - greggarious 4y ago
      > Living life in fear of reprisal isn’t much of a life. We all should strive to live up to our ideals and convictions, no matter the consequences. If we don’t then what are we actually doing here? I agree, so why not do something truly brave, like spark a riot outside the mayor's house with nothing more than a well timed GIF rather than waste time confirming what folks already know in forensic detail? (Sorry to be flippant, but it's been a long day and it's so late it's early.)
  - deleted 4y ago
    [deleted]
  - jacquesm 4y ago
    If Putin is going to come after every Westerner that both hates his guts and that has contributed in one form or another to the Ukrainian side of this war then he has his work cut out for him. I'd say OP is too low value a target from cost/benefit analysis point of view and the backlash of committing acts of war on Western territory could well be much, much worse than anything that Putin is willing to deal with at the moment. Putin has gotten away with the occasional assassination of Russians on foreign soil, with innocents as collateral damage, if he steps that up to a targeted assassination campaign of Western citizens then that would be a serious miscalculation on his part.
    - 2devnull 4y ago
      On the other hand cost/benefit can change, especially with new technology - machine learning, ai, quantum computing, etc… That may be why the military is so interested in encouraging it, to distribute (thin out) the risk.
    - jstarfish 4y ago
      > Putin has gotten away with the occasional assassination of Russians on foreign soil, with innocents as collateral damage, if he steps that up to a targeted assassination campaign of Western citizens then that would be a serious miscalculation on his part. On the other hand, targeting nobodies abroad would be a highly-effective form of stochastic terrorism inflicted with the unpredictability and resources of a state-sponsored serial killer. Anybody attempting attribution would be dismissed as crazy.
      - greggarious 4y ago
        >> Putin has gotten away with the occasional assassination of Russians on foreign soil, with innocents as collateral damage, if he steps that up to a targeted assassination campaign of Western citizens then that would be a serious miscalculation on his part. >On the other hand, targeting nobodies abroad would be a highly-effective form of stochastic terrorism I don't think that word means what you think it means. Stochastic terrorism is "The use of mass public communication, usually against a particular individual or group, which incites or inspires acts of terrorism which are statistically probable but happen seemingly at random."[1] What you're describing -- "murdering nobodies" is just run of the mill terrorism[2] where I'm from[3]. >§ 2717. Terrorism. >(a) General rule.--A person is guilty of terrorism if he commits a violent offense intending to do any of the following: >(1) Intimidate or coerce a civilian population. >(2) Influence the policy of a government by intimidation or coercion.= >(3) Affect the conduct of a government. Be more careful with your language, words have immense power. [1] https://en.wiktionary.org/wiki/stochastic_terrorism https://en.wiktionary.org/wiki/stochastic_terrorism [2] https://www.legis.state.pa.us/WU01/LI/LI/CT/HTM/18/00.027..HTM https://www.legis.state.pa.us/WU01/LI/LI/CT/HTM/18/00.027..H... [3] https://www.youtube.com/watch?v=Vl3jK8NCLFc https://www.youtube.com/watch?v=Vl3jK8NCLFc
- deleted 4y ago
  [deleted]
- noman-land 4y ago
  This is a great story. Don't get discouraged!
ameliorees 4y ago
The list is cool and full of interesting things. But as someone who has done OSINT a lot for journalism and legal investigations and built tools for the space I can say that people rely most on custom internal tools or the simplest/ most popular tooling among the community!
- asdadsdad 4y ago
  I feel like most of OSINT is more manual than these lists lead to believe. Or if not, most of it relies on already-built massive archives or tools that scrape those archives.
  - g147 4y ago
    absolutely. it's just useful for a quick overview. real research happens when you keep digging where others haven't.
  - anigbrowl 4y ago
    Agreed. This list is obviously tilted toward computer security; OSINT also targets nation states and political actors, and more often than not that's through non-technical means (ie looking for information carelessly left public rather than exploiting vulnerabilities to gain access to private information).
- realitygrill 4y ago
  Would you mind reaching out to me? I'm trying to learn more about how to do OSINT (with that particular slant!).
aristus 4y ago
Relatedly: is there software / database designs recommended for managing osint facts?
- z3c0 4y ago
  I use rolling snapshots. It helps protect against members dropping suddenly, as a lot of OSINT sources aren't intended to be and will only contain a sample of your target population. Opinions, priorities, or biases may change, and thus so will the data.
- g147 4y ago
  you mean like maltego or osintframework?
- epoch_100 4y ago
  There’s https://atlos.org https://atlos.org for visual investigations
axegon_ 4y ago
I mean... Including Kaspersky in any shape or form is a bit of a moot point but...
deleted 4y ago
[deleted]
GalenErso 4y ago
Don't forget the Noncredibledefense subreddit.
- g147 4y ago
  i did forget. thanks!
  - GalenErso 4y ago
    Hold up. It was a joke.
    - g147 4y ago
      good one
derefr 4y ago
Something I've always wondered: when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding? Sort of like how BigQuery has "BigQuery public datasets"? If not, seems like a missed opportunity for value-add / lock-in. And if they do provide such data, it'd be nice if one such platform would make either the scrapers, or the scraped data, open-source. It's all just cleaned+normalized forms of already-public data, after all. (Like how the Yellow Pages could always be seen as a cleaned+normalized form of phone numbers listed in public view on shop windows.) This is on my mind recently after noticing that there's actually no public dataset of "all WHOIS data for every domain on the Internet", despite WHOIS data being something explicitly designed for public querying. Almost certainly, intelligence agencies compile such a dataset themselves in order to fill in some gaps in cybercrime network-analysis graphs. Almost certainly multiple of them do, such that the number of crawlers doing it probably hurts the WHOIS services. Almost certainly the WHOIS services would be better off partnering with one of them to act as an "official crawler" to build such an index for everyone's use.
- schroeding 4y ago
  A public (historical) WHOIS dataset is a nightmare due to GDPR et al., though, isn't it? You couldn't opt-out of WHOIS (some TLDs just didn't allow for WHOIS privacy / proxy registrations) in the past, and it's IMO quite a hard sell that the public interest in the historic WHOIS data of some personal website should have any weight at all in this case.
  - posterboy 4y ago
    That's an interesting way to put it. There is some interest, no doubt, and there is some potential abusing that information. Since the information was committed knowingly and willingly to the public domain, though I do not know much about it to begin with, I don't see how GDPR applies at all.
    - schroeding 4y ago
captainmuon 4y ago
Since other people are asking about related tools... Do you know about a tool for managing personas (vulgo sockpuppets)? E.g. a little database where you store facts about fake identities: Name, age, location, hobbies, favorite topics, and so on. Maybe if you click on an identity, it will connect you with a preconfigured proxy or VPN and allow you to post as them on websites. I don't really have a use case for myself. I'm just curious because I read a couple of years ago that the US Army was developing something like that for propaganda, and wonder if the OSINT or sousveilance people have come up with a version. A less nefarious use of such technology would be to keep your (real) online personalities separate.
- 2devnull 4y ago
  A password manager?
- chli 4y ago
  Firefox Multi-Account Containers with Mozilla VPN would be a start.
- jstarfish 4y ago
  > Maybe if you click on an identity, it will connect you with a preconfigured proxy or VPN and allow you to post as them on websites. You can still be fingerprinted across "personas" when your useragent/screen resolution/IP address/proxy provider/etc. are all identical, since you're presumably doing all of this from the same device. This does not mean spin up a bunch of EC2 instances. Buy a bunch of different burner phones/old laptops from Goodwill or whatever-- use discrete hardware. Get SIM cards from different providers. Pay with Bitcoin or cash where possible. Etc. These are the sorts of things scammer sweatshops do to avoid accountability. They'll have literal walls full of phones wired up, each preconfigured for a particular identity. Even teenaged fraudsters like Eleanor Williams think to do this sort of stuff to maintain opsec.
freestate 4y ago
OSINT is the one of the last ways we humans have, to get some Data that is at least a bit honest. More importaint is that we know how to use and gather it. THX for your link i will take a look at it.
AtlasBarfed 4y ago
For better or worse, I'm surprised this hasn't happened yet: Track organizational structures and responsibility over time (especially public ones like government and public corporation executives). Then enable searches of events / transgressions and claimed fallguys so more responsibility can be traced/blame for these important figures who usually get off scot free. It would be useful to know "hey who likely was in charge of the divisions that suppressed early global warming research in oil companies" with a relatively simple search.
mellosouls 4y ago
Earlier discussion about a similar list: https://news.ycombinator.com/item?id=32951406 https://news.ycombinator.com/item?id=32951406
prosaic-hacker 4y ago
BGP Ranking site https://bgpranking.circl.lu/ https://bgpranking.circl.lu/ serve a 503 error Service unavailable. I do not know any more details.
aaron695 4y ago
[dead]

223 points by g147 4y ago

at the start of Russian invasion I helped geolocate one video, claiming that Russians entered Kherson. I found it after about one hour and posted on twitter tagging everyone I know. I got no feedback so got discouraged. But few months later, I noticed that France24 used my tweet to prove Russians entered Kherson and a lot of newspapers followed their lead. Pretty cool I was able to help in a minimally viable way to understand what was happening

jasmer 4y ago
Bravo. That is definitely probably the only true 'hacker badge' that I would really want. I would wear that little patch on my jacket at the little meeting for sure.
g147 4y ago
that's awesome man. oh and feel free to be encouraged but don't ever get discouraged based on other people's behavior or actions. experts like you are definitely needed in today's world. people will eventually appreciate your research.
greggarious 4y ago
[flagged]
- dmix 4y ago
  A lot of people geolocate stuff on /r/combatfootage for fun because they like following the progress of the war. It's a personal challenge.
  - greggarious 4y ago
    > A lot of people geolocate stuff on /r/combatfootage for fun because they like following the progress of the war. It's a personal challenge. Or because they like gore and /r/watchpeopledie got got.
- halfmatthalfcat 4y ago
  Living life in fear of reprisal isn’t much of a life. We all should strive to live up to our ideals and convictions, no matter the consequences. If we don’t then what are we actually doing here?
  - greggarious 4y ago
    > Living life in fear of reprisal isn’t much of a life. We all should strive to live up to our ideals and convictions, no matter the consequences. If we don’t then what are we actually doing here? I agree, so why not do something truly brave, like spark a riot outside the mayor's house with nothing more than a well timed GIF rather than waste time confirming what folks already know in forensic detail? (Sorry to be flippant, but it's been a long day and it's so late it's early.)
- deleted 4y ago
  [deleted]
- jacquesm 4y ago
  If Putin is going to come after every Westerner that both hates his guts and that has contributed in one form or another to the Ukrainian side of this war then he has his work cut out for him. I'd say OP is too low value a target from cost/benefit analysis point of view and the backlash of committing acts of war on Western territory could well be much, much worse than anything that Putin is willing to deal with at the moment. Putin has gotten away with the occasional assassination of Russians on foreign soil, with innocents as collateral damage, if he steps that up to a targeted assassination campaign of Western citizens then that would be a serious miscalculation on his part.
  - 2devnull 4y ago
    On the other hand cost/benefit can change, especially with new technology - machine learning, ai, quantum computing, etc… That may be why the military is so interested in encouraging it, to distribute (thin out) the risk.
  - jstarfish 4y ago
    > Putin has gotten away with the occasional assassination of Russians on foreign soil, with innocents as collateral damage, if he steps that up to a targeted assassination campaign of Western citizens then that would be a serious miscalculation on his part. On the other hand, targeting nobodies abroad would be a highly-effective form of stochastic terrorism inflicted with the unpredictability and resources of a state-sponsored serial killer. Anybody attempting attribution would be dismissed as crazy.
    - greggarious 4y ago
      >> Putin has gotten away with the occasional assassination of Russians on foreign soil, with innocents as collateral damage, if he steps that up to a targeted assassination campaign of Western citizens then that would be a serious miscalculation on his part. >On the other hand, targeting nobodies abroad would be a highly-effective form of stochastic terrorism I don't think that word means what you think it means. Stochastic terrorism is "The use of mass public communication, usually against a particular individual or group, which incites or inspires acts of terrorism which are statistically probable but happen seemingly at random."[1] What you're describing -- "murdering nobodies" is just run of the mill terrorism[2] where I'm from[3]. >§ 2717. Terrorism. >(a) General rule.--A person is guilty of terrorism if he commits a violent offense intending to do any of the following: >(1) Intimidate or coerce a civilian population. >(2) Influence the policy of a government by intimidation or coercion.= >(3) Affect the conduct of a government. Be more careful with your language, words have immense power. [1] https://en.wiktionary.org/wiki/stochastic_terrorism https://en.wiktionary.org/wiki/stochastic_terrorism [2] https://www.legis.state.pa.us/WU01/LI/LI/CT/HTM/18/00.027..HTM https://www.legis.state.pa.us/WU01/LI/LI/CT/HTM/18/00.027..H... [3] https://www.youtube.com/watch?v=Vl3jK8NCLFc https://www.youtube.com/watch?v=Vl3jK8NCLFc
deleted 4y ago
[deleted]
noman-land 4y ago
This is a great story. Don't get discouraged!

ameliorees 4y ago

The list is cool and full of interesting things. But as someone who has done OSINT a lot for journalism and legal investigations and built tools for the space I can say that people rely most on custom internal tools or the simplest/ most popular tooling among the community!

asdadsdad 4y ago
I feel like most of OSINT is more manual than these lists lead to believe. Or if not, most of it relies on already-built massive archives or tools that scrape those archives.
- g147 4y ago
  absolutely. it's just useful for a quick overview. real research happens when you keep digging where others haven't.
- anigbrowl 4y ago
  Agreed. This list is obviously tilted toward computer security; OSINT also targets nation states and political actors, and more often than not that's through non-technical means (ie looking for information carelessly left public rather than exploiting vulnerabilities to gain access to private information).
realitygrill 4y ago
Would you mind reaching out to me? I'm trying to learn more about how to do OSINT (with that particular slant!).

aristus 4y ago

Relatedly: is there software / database designs recommended for managing osint facts?

z3c0 4y ago
I use rolling snapshots. It helps protect against members dropping suddenly, as a lot of OSINT sources aren't intended to be and will only contain a sample of your target population. Opinions, priorities, or biases may change, and thus so will the data.
g147 4y ago
you mean like maltego or osintframework?
epoch_100 4y ago
There’s https://atlos.org https://atlos.org for visual investigations

axegon_ 4y ago

I mean... Including Kaspersky in any shape or form is a bit of a moot point but...

deleted 4y ago

[deleted]

GalenErso 4y ago

Don't forget the Noncredibledefense subreddit.

g147 4y ago
i did forget. thanks!
- GalenErso 4y ago
  Hold up. It was a joke.
  - g147 4y ago
    good one

derefr 4y ago

Something I've always wondered: when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding? Sort of like how BigQuery has "BigQuery public datasets"? If not, seems like a missed opportunity for value-add / lock-in. And if they do provide such data, it'd be nice if one such platform would make either the scrapers, or the scraped data, open-source. It's all just cleaned+normalized forms of already-public data, after all. (Like how the Yellow Pages could always be seen as a cleaned+normalized form of phone numbers listed in public view on shop windows.) This is on my mind recently after noticing that there's actually no public dataset of "all WHOIS data for every domain on the Internet", despite WHOIS data being something explicitly designed for public querying. Almost certainly, intelligence agencies compile such a dataset themselves in order to fill in some gaps in cybercrime network-analysis graphs. Almost certainly multiple of them do, such that the number of crawlers doing it probably hurts the WHOIS services. Almost certainly the WHOIS services would be better off partnering with one of them to act as an "official crawler" to build such an index for everyone's use.

schroeding 4y ago
A public (historical) WHOIS dataset is a nightmare due to GDPR et al., though, isn't it? You couldn't opt-out of WHOIS (some TLDs just didn't allow for WHOIS privacy / proxy registrations) in the past, and it's IMO quite a hard sell that the public interest in the historic WHOIS data of some personal website should have any weight at all in this case.
- posterboy 4y ago
  That's an interesting way to put it. There is some interest, no doubt, and there is some potential abusing that information. Since the information was committed knowingly and willingly to the public domain, though I do not know much about it to begin with, I don't see how GDPR applies at all.
  - schroeding 4y ago

captainmuon 4y ago

Since other people are asking about related tools... Do you know about a tool for managing personas (vulgo sockpuppets)? E.g. a little database where you store facts about fake identities: Name, age, location, hobbies, favorite topics, and so on. Maybe if you click on an identity, it will connect you with a preconfigured proxy or VPN and allow you to post as them on websites. I don't really have a use case for myself. I'm just curious because I read a couple of years ago that the US Army was developing something like that for propaganda, and wonder if the OSINT or sousveilance people have come up with a version. A less nefarious use of such technology would be to keep your (real) online personalities separate.

2devnull 4y ago
A password manager?
chli 4y ago
Firefox Multi-Account Containers with Mozilla VPN would be a start.
jstarfish 4y ago
> Maybe if you click on an identity, it will connect you with a preconfigured proxy or VPN and allow you to post as them on websites. You can still be fingerprinted across "personas" when your useragent/screen resolution/IP address/proxy provider/etc. are all identical, since you're presumably doing all of this from the same device. This does not mean spin up a bunch of EC2 instances. Buy a bunch of different burner phones/old laptops from Goodwill or whatever-- use discrete hardware. Get SIM cards from different providers. Pay with Bitcoin or cash where possible. Etc. These are the sorts of things scammer sweatshops do to avoid accountability. They'll have literal walls full of phones wired up, each preconfigured for a particular identity. Even teenaged fraudsters like Eleanor Williams think to do this sort of stuff to maintain opsec.

freestate 4y ago

OSINT is the one of the last ways we humans have, to get some Data that is at least a bit honest. More importaint is that we know how to use and gather it. THX for your link i will take a look at it.

AtlasBarfed 4y ago

For better or worse, I'm surprised this hasn't happened yet: Track organizational structures and responsibility over time (especially public ones like government and public corporation executives). Then enable searches of events / transgressions and claimed fallguys so more responsibility can be traced/blame for these important figures who usually get off scot free. It would be useful to know "hey who likely was in charge of the divisions that suppressed early global warming research in oil companies" with a relatively simple search.

mellosouls 4y ago

Earlier discussion about a similar list: https://news.ycombinator.com/item?id=32951406 https://news.ycombinator.com/item?id=32951406

prosaic-hacker 4y ago

BGP Ranking site https://bgpranking.circl.lu/ https://bgpranking.circl.lu/ serve a 503 error Service unavailable. I do not know any more details.

aaron695 4y ago

[dead]