Recon-ng automates time-consuming OSINT activities, like cutting and pasting. Recon-ng doesn't claim that all OSINT gathering can be conducted by its tool, but it can be used to automate much of the most popular kinds of harvesting, leaving more time for the things that still must be done manually.
Designed so that even the most junior Python developers can create searches of publicly available data and return good results, it has a very modular framework with a lot of built-in functionality. Common tasks like standardising output, interacting with databases, making web requests and managing API keys are all part of the interface. Instead of programming Recon-ng to perform searches, developers simply choose which functions they want it to perform and build an automated module in just a few minutes.
Recon-ng is free, open source software. The available wiki includes comprehensive information for getting started with the tool as well as best practices for using it.
One of the simplest tools to use on this list, theHarvester is designed to capture public information that exists outside of an organisation’s owned network. It can find incidental things on internal networks as well, but the majority of tools that it uses are outward facing. It would be effective as a reconnaissance step prior to penetration testing or similar exercises.
The sources that theHarvester uses include popular search engines like Bing and Google, as well as lesser known ones like dogpile, DNSdumpster and the Exalead meta data engine. It also uses Netcraft Data Mining and the AlienVault Open Threat Exchange. It can even tap the Shodan search engine to discover open ports on discovered hosts. In general, theHarvester tool gathers emails, names, subdomains, IPs and URLs.
TheHarvester can access most public sources without any special preparations. However, a few of the sources used require an API key. You must also have Python 3.6 or better in your environment.
Anyone can obtain theHarvester on GitHub. It’s recommended that you use a virtualenv to create an isolated Python environment when cloning it from there.
Shodan is a dedicated search engine used to find intelligence about devices like the billions that make up the internet of things (IoT) that are not often searchable, but happen to be everywhere these days. It can also be used to find things like open ports and vulnerabilities on targeted systems. Some other OSINT tools like theHarvester use it as a data source, though deep interaction with Shodan requires a paid account.
The number of places that Shodan can monitor and search as part of an OSINT effort is impressive. It’s one of the few engines capable of examining operational technology (OT) such as the kind used in industrial control systems at places like power plants and manufacturing facilities. Any OSINT gathering effort in industries that deploy both information technology and OT would miss a huge chunk of that infrastructure without a tool like Shodan.
In addition to IoT devices like cameras, building sensors and security devices, Shodan can also be turned to look at things like databases to see if any information is publicly accessible through paths other than the main interface. It can even work with videogames, discovering things like Minecraft or Counter-Strike: Global Offensive servers hiding on corporate networks where they should not be, and what vulnerabilities they generate.
Anyone can purchase a Freelancer license and use Shodan to scan up to 5,120 IP addresses per month, with a return of up to a million results. That costs $59 per month. Serious users can buy a Corporate license, which provides unlimited results and scanning of up to 300,000 IPs monthly. The Corporate version, which costs $899 per month, includes a vulnerability search filter and premium support.
Another freely available tool on GitHub, Metagoofil is optimised to extract metadata from public documents. Metagoofil can investigate almost any kind of document that it can reach through public channels including .pfd, .doc, .ppt, .xls and many others.
The amount of interesting data that Metagoofil can gather is impressive. Searches return things like the usernames associated with discovered documents, as well as real names if available. It also maps the paths of how to get to those documents, which in turn would provide things like server names, shared resources and directory tree information about the host organisation.
Everything that Metagoofil finds would be very useful for a hacker, who could use it to do things like launch brute-force password attacks or even phishing emails. Organisations that want to protect themselves could instead take the same OSINT gathered information and protect or hide it before a malicious actor can take the initiative.
For those who need to go really deep into the complex matrix of OSINT gathering, searchcode is a highly specialised search engine that looks for useful intelligence inside source code. This powerful engine is surprisingly the work of a single developer.
Because a repository of code needs to be first added to the program before becoming searchable, searchcode straddles the line between an OSINT tool and one designed to find things other than public information. However, it can still be considered an OSINT tool because developers can use it to discover problems associated with having sensitive information accessible inside code on either running apps or those that are still in development. In the latter case, those problems could be fixed prior to deployment into a production environment.
Although anything involving code is going to require more knowledge than, say, a Google search, searchcode does a great job of making its interface as easy to use as possible. Users simply type in their search fields and searchcode returns relevant results with search terms highlighted in the lines of code. Suggested searches include usernames, security flaws like eval $_GET calls, unwanted active functions like re.compile and special characters that can be used to launch code injection attacks.
Most of the time, the results returned by searchcode are self-explanatory. However, it’s possible to click through those results to find deeper information or matching problems if needed.
Relevant information isn’t always in English. Only about a quarter of internet users speak English as their primary language according to Statista, though various sources say as much as 55% of internet content is in English. The information you need might be in Chinese, Spanish or Tamil.
Babel X from Babel Street is a multilingual search tool for the public internet, including blogs, social media, message boards and news sites. It also searches the dark web, including Onion sites, and some deep web content that Babel X can access through agreements or licensing from the content owners. The product is able to geo-locate the source of information it finds, and it can perform text analysis to identify relevant results. Babel X is currently capable of searching in more than 200 languages.
Use cases where a multilingual search is useful include searching global news for situational awareness—for example, knowing trends in targeting for ransomware attacks. It can also be used to spot a company’s intellectual property for sale on a foreign website, or information that shows a key partner has been compromised. Customers have also used Babel X to find user handles of suspected attackers on non-English message boards.
The main Babel X product is cloud-based and allows customers to customise it by adding their own data sources to search. Babel Box is an on-premises version but lacks some features of Babel X, such as access to deep web data sources. Babel Channels, the lowest cost option, is a curated collection of data sources. A mobile app is available for all the options.
While these tools offer a wealth of OSINT data, there are many other tools and techniques available that help you fully understand your organisation's public footprint. An excellent resource for discovering more tools is the OSINT Framework, which offers a web-based interface that breaks down different topic areas of interest to OSINT researchers and connects you to the tools that can help you sniff out the info you need.
The tools that the OSINT Framework will point you to are all free of charge, though some require registration or have more fully featured paid versions available. Some are simply tools that help construct advanced Google searches that can yield a surprising amount of information. The OSINT Framework is maintained by Justin Nordine, and has a project page on GitHub.
Is OSINT illegal?
While OSINT techniques are often used by malicious hackers as reconnaissance before they launch an illegal attack, for the most part the tools and techniques themselves are perfectly legal—after all, they're designed to help you home in on data that's published or otherwise in the public view. Even government agencies are encouraged to use OSINT techniques to ferret out holes in their own cybersecurity defences.
Following the trail opened by these OSINT queries can get you into legal grey areas, however. Media Sonar has some good advice on how to stay on the right side of the law here. For instance, it's not illegal to access public areas of the dark web, and it can be important to do so if you're trying to determine if your organisation's data has been breached or stolen; but you shouldn't try to buy collections of stolen data as part of your research, or impersonate a law enforcement officer to shake information out of shady characters.
In general, it's important to develop a code of conduct in advance to guide your employees' behaviour on these expeditions, and to document everything you do to demonstrate that you're sticking to those guidelines and haven't broken any laws.
Closing down open source intelligence loopholes
Not every hack or intrusion involves advanced persistent threats or deep, sophisticated penetrations. Hackers, like everyone else, will take the easiest path to their objectives. There is no need to try to crack tight cybersecurity through many months of effort if the information they want is available through a publicly accessible channel. At the very least, sensitive information can be used as a shortcut to obtaining valid credentials or to help plan an effective intrusion with less effort or risk.
OSINT tools can help organisations get a grip on what information is available about them, their networks, data and users. Finding that information quickly is key since it would allow for its removal before someone can exploit it. These tools can be a strong boost during that most critical race.