Mastering OSINT| A Comprehensive Guide to Intelligence Gathering for Bug Hunters

Hey cyber learners,

I will teach you everything about OSINT and intelligence gathering in this blog series. I will share some practical tools and tips on how a hacker or a government agent performs an OSINT investigation. also, we will discuss the OSINT Framework, some real-life OSINT investigations, and how you can protect your identity from these Open Source Intelligence Gathering tools and remain anonymous from the government.

In this blog, we'll take a closer look at the importance of OSINT for bug hunters and how they can use it to improve the accuracy and efficiency of their assessments.

we will also take a look at some tools that bug hunters can use to gather and analyze information from publicly available sources. The goal of this blog is to provide practical information and tips on how to use each of these tools effectively in the Bug Hunting process.

At the end of this blog, I have also attached a practical Video on the usage of OSINT tools for bug hunting. I recommend you to watch that video after reading the blog because the video has some extra tips and commands.

• What is OSINT?
• Why OSINT is Important for Bug Hunters?
• Best OSINT Tools for Bug Hunters
• Google
• Waybackmachine
• Video Tutorial on OSINT Tools
• Conclusion
• Commonly Asked Questions

What is OSINT?

Open Source Intelligence (OSINT) is an essential tool for bug hunters in their quest to identify and exploit vulnerabilities in systems, applications, and networks. OSINT is the process of gathering and analyzing information from publicly available sources to gain a deeper understanding of a target and its environment.

OSINT helps bug hunters to save time by providing them with information about the target that can be used to prioritize testing efforts and make informed decisions about where to focus their attention. By using OSINT, bug hunters can gain a comprehensive understanding of the target and its environment, including organizational structure, technologies used, IP ranges, subdomains, email addresses, phone numbers, and network and domain configuration information.

for more information, you can read my previous blog on this topic What is OSINT? | OSINT Framework | Tools for OSINT | Best OSINT Techniques

Why OSINT is important for Bug Hunters?

Open Source Intelligence (OSINT) is a crucial aspect of bug hunting, Bug hunting is a process of finding and reporting vulnerabilities in systems, applications, and networks. OSINT refers to the collection and analysis of information from publicly available sources, including the internet, social media, public records, and other open-access databases.

OSINT is a critical tool for bug hunters because it provides a wealth of information about the target, including its systems, applications, and personnel. This information can be used to identify potential vulnerabilities, determine the scope of the target, and prioritize areas of attack. OSINT is also essential for understanding the target's environment, including its technology stack, development process, and security posture.

Now let's understand some practical aspects of OSINT in the bug-hunting process. There are multiple ways where OSINT will be very helpful when you are going to start your bug hunting. Below I have mentioned a few of them that I personally experienced in my bug-hunting journey.

Recon

The first step in any bug-hunting process is to perform recon or reconnaissance. OSINT can be used to gather information about the target, including its systems, applications, and personnel. This information can be used to identify potential attack surfaces, determine the target's scope, and prioritize areas of attack.

Vulnerability Assessment

OSINT can be used to gather information about potential vulnerabilities in the target's systems and applications. This information can be used to identify areas of weakness that can be exploited and to prioritize those areas for further investigation.

Understanding the Target Environment

OSINT can be used to gather information about the target's environment, including its technology stack, development process, and security posture. This information is critical for understanding the target's systems, and applications, and for identifying potential weaknesses, and areas of attack.

Prioritizing testing efforts

By using OSINT, bug hunters can reduce the time required to complete their assessments by prioritizing their testing efforts. They can use the information they gather to focus their attention on high-risk areas and make informed decisions about where to allocate their resources. This can help to increase the chances of finding critical vulnerabilities and reduce the time required to complete the assessment.

Ethical hacking practices

OSINT is an ethical hacking tool that allows bug hunters to gather information about their target without causing harm to the target system or data. This is in contrast to other methods, such as port scanning, which can cause damage to the target system and result in legal consequences. By following ethical hacking practices, bug hunters can ensure that their assessments are performed in a responsible manner and do not cause harm to the target.

Best OSINT tools for bug hunters

Unleashing the power of Open Source Intelligence (OSINT) starts with the right tools. Check out this list of my personal favourites:

Google

Google is the world's largest search engine and one of the most powerful OSINT tools available to bug hunters. Google can be used to gather information about the target, including company and organizational structure, technologies used, IP ranges and subdomains, email addresses, phone numbers, and network and domain configuration information. To get the most out of Google, bug hunters can use advanced search operators(google dorks), such as site:domain.com to search for specific information on a specific domain.

Question: What is Google Dork?    
Ans: Google Dorks is a term used to describe advanced Google search queries that can be used to find specific information from the Internet. By using specific keywords and operators, you can gather information about people, organizations, and infrastructure to help identify potential targets for further security research.

Let's try to get information on Paytm.in.

Now, if I want to search for admin panels of Paytm.in, I can simply use this dork:

site:paytm.in inurl:admin

In the above screenshot, you can see the results, you can visit those results according to your research, generally, such results it contains a private admin panel or some vulnerable login page.

Now after searching for admin panels, let's try to search for database leaks.

For that, I will use this dork:

site:paytm.in inurl:db

I opened one of these sites and got some interesting results. You need to open each and every link to investigate and find some useful data.

In the above screenshot, you can notice one of the links shows a directory listing which nowadays is considered a bug, also we found some files, this may contain sensitive information about database configuration.

Now, let's search for some open FTP servers.

For that I will simply use this dork:

site:paytm.in inurl:ftp

again, I opened one of these results and got this FTP server directory

Note: Finding such data is difficult and needs a lot of research, you may need to use various dorks to find some useful content.

as you can see in the above screenshot, there are several directories that may contain sensitive information. these small sensitive pieces of information are a gold mine for hackers.

Below I have shown some of my most used dork formats or syntax. You can use and customize them according to your requirements.

Some best google dorks for reconnaissance:

"site:" + [domain name] - to search for all pages within a specific website

"inurl:" + [keyword] - to search for pages with a specific word in the URL

"intitle:" + [keyword] - to search for pages with a specific word in the title

"filetype:" + [file extension] - to search for specific file types such as ".pdf" or ".doc"

"link:" + [domain name] - to find all pages that link to a specific website

"related:" + [domain name] - to find websites related to a specific website

"cache:" + [domain name] - to view the cached version of a website

"info:" + [domain name] - to view information about a specific website

"define:" + [word] - to search for definitions of a specific word or phrase

"intext:" + [keyword] - to search for pages containing a specific word or phrase in the page text.

Now if you are a beginner and don't know how you can use the above syntax to generate your own dorks so don't worry I have generated a few of them below.

you can use the above dorks to generate many different dork searches. some examples are listed below:

site:domain.com filetype:pdf
site:domain.com intitle:"index of"
site:domain.com inurl:admin
site:domain.com inurl:login
site:domain.com inurl:wp-content
site:domain.com intitle:"database dump"
site:domain.com intitle:"restricted access"
site:domain.com intitle:"upload"
site:domain.com intitle:"web server"
site:domain.com inurl:db
site:domain.com inurl:config
site:domain.com inurl:backup
site:domain.com intitle:"list files"
site:domain.com intitle:"server status"
site:domain.com intitle:"web server statistics"
site:domain.com intitle:"phpinfo"
site:domain.com inurl:"phpmyadmin"
site:domain.com inurl:"webmail"
site:domain.com inurl:"mail"
site:domain.com inurl:ftp
site:domain.com allinurl:@[mailbox_domain]

Here are a few practical tips for using Google for OSINT in bug hunting:

1. Use advanced search operators(Google dorks): Google provides a number of advanced search operators that allow you to refine your search and find specific information. Some commonly used operators include "site:", "intitle:", "intext:", "filetype:", and "cache:". For example, using the "site:" operator followed by the target's domain name allows you to search for information only on that specific website.
2. Search for subdomains: Subdomains can contain sensitive information or potential attack surfaces, so it's important to search for them as part of your OSINT investigation. To search for subdomains, use the "site:" operator followed by the target's domain name and a wildcard (*).
3. Search for sensitive information: Accidentally leaked credentials or other sensitive information can provide valuable information for a bug hunter. To search for sensitive information, use search operators such as "intext:" or "filetype:".
4. Search for information about security incidents: Information about previous security incidents, such as data breaches or vulnerabilities, can provide insight into the target's security posture and identify potential attack surfaces. To search for this information, use search operators such as "site:" and "intitle:".
5. Find hidden or obscured information: Google can also be used to find hidden or obscured information about a target, such as information about the target's employees or previous versions of the target's website. To find this information, use search operators such as "cache:" or "site:".

In practice, these tips can be combined in different ways to effectively use Google for OSINT in bug hunting.

You might be thinking real hackers are not using this they might use different tools. Let' Me show you some real-world examples that will show you the power of Google Dorking.

Below I have shown two real-world examples of bug hunting. Let's see some case studies.

Real-world examples:

July 16, 2019, 5:30 pm +0530

In this case, the reporter discovered a pdf file on some privet U.S.Defance server, that outlines various information corresponding to military members. It reveals information on the date of birth, where they were born, marriage status, race, children/dependents, etc.

It's a high-severity vulnerability, and very easy to exploit. attacker only have to visit https://<privet>/wp-content/uploads/2018/12/<privet>.

Read More: PII leakage-Full SSN on (U.S. Dept Of Defense)

May 26, 2021, 9:49 pm +0530

Khan Academy is a very famous edu-tech platform. In this case, the attacker used this particular google dork site:khanacademy.org/join/* to enumerate all the links to joining classes.

so, everyone is able to access paid khan academy classes without paying a single rupee. with simple google dork. this bug has the power to destroy the whole business model of khan academy because all their revenue comes from paid courses. this is also a high-severity bug.

Read More: Enumerate all the class codes via google dorking (Khan Academy)

It's important to note that while Google can be a powerful tool for OSINT, it's just one part of a larger investigation. To increase your chances of finding and reporting bugs, it's important to also use other OSINT tools and techniques, such as social media searches, WHOIS lookups, and passive DNS analysis.

Waybackmachine

We can use this tool to actually look into some sensitive files that existed way back. Wayback Machine was developed by the Internet Archive. in short, it's a digital archive of the World Wide Web that provides access to billions of web pages and snapshots of websites dating back to 1996. For bug hunters, the Wayback Machine can be a valuable tool for gathering information about the target's web presence and history, and for identifying potential vulnerabilities.

By using the Wayback Machine, bug hunters can view the evolution of a target's website over time, and can identify changes in the website's design, functionality, and content. This information can be used to reconstruct the target's web presence and to identify potential vulnerabilities that may have been introduced or changed over time.

Now, let's try to get information on facebook.com.

as you can see in the below image, there are a total of 52,87,230 pages archived from facebook.com

After some time, I found out that the facebook.com domain was used by a company "about face" in 2000. Same you can observe in the below screenshot.

This is a snapshot of Facebook in his starting days.

Let's search for robots.txt in 2004. To find the robot.txt you just need to type domain name/robots.txt. for example facebook.com/robots.txt

Facebook removed many endpoints from that file now, but for your target, sometimes you may find outdated APIs or some vulnerable endpoints in the Wayback Machine that might be helpful in your research.

Video Tutorial on OSINT Tools

Below I have created a video for you on OSINT bug hunting where I have practically demonstrated above mentioned tools and also shown some extra investigation tips and commands.

Go check out this video.

Practical tips to use Waybackmachine effectively:

1. robots.txt: Old robots.txt data contains a lot of juicy endpoints. In this file, businesses will identify endpoints that they do not want Google to index. Over time, they may also add and remove endpoints. All of it is stored on WayBackMachine! Old endpoints that may have been removed in 2018 might still be functioning and have entertaining features to experiment with.
2. more endpoints: Almost all of the endpoints on the target's website were listed in the robots.txt file. I was able to boost my recon from 400 endpoints to 7000+ using Wayback Machine, which allowed me to collect data from the previous 7 years. This enabled me to capture several bugs and bounties.
3. endpoints on the single domain: Additionally, WayBackMachine provides a list of all the endpoints it has scraped for the domain you have selected. For example, you could search for every endpoint WayBackMachine has archived for www.techofide.com.
4. raw HTML: The next strategy does need some manual testing. Since WayBackMachine has the raw HTML for websites, it is possible to get the HTML from the last seven years of https://www.hackerone.com.To extract information like the parameter names used on each page, you can start manually searching through these. For instance, you might come across a three-year-old input.
5. reused codes: You can test current known parameters on past endpoints that WayBackMachine has saved in addition to trying to uncover ancient (but new to you!) parameters. Keep in mind that programmers like to reuse code!
6. more data: WayBackMachine has a lot more data archived than people are aware of, all it takes is sifting through the data to find the crucial pieces.

In conclusion, the main things you're looking for are:

• Old endpoints that may still work. Old code = bugs!
• Old parameters that may be re-used
• Old API keys referenced

There are various automated tools that will help you to investigate a few of them I referred to below. If you are a beginner then I suggest you do it manually as I demonstrated:

• waybackurls
• waybackrobots
• waybackurls by tomnomnom
• getallurls (gau)

If you are thinking it is a very difficult task and time taking so test it consumes time and that is what happens in bug hunting you need to research a lot and touch each and every aspect of the website. The Wayback Machine is just like time travel that allows you to see the website for past and present. 

Below I have shown some real-world bug bounties that were discovered with the help of this Wayback technique,

Real-world examples:

January 2, 2021, 2:08 pm +0530

simplenote is a note-saving platform like Pastebin. it has both options, user can save his note privately or publicly. when the user saves his note in private, it should not be accessible to the public.
now, let's take the example of user kanav.

kanav is an old user of simplenote. in 2020, he saved a note "very secret password123" publicly, because that time this information is not that sensitive.

now in 2023, kanav realized that he leaked his very sensitive password123 online. so, he turned his note from public to privet.

also in 2023, an attacker simply searched simplenote.com in the Waybackmachine, and he opened the 2020 archive, now kanav's password is public.

this is a very sensitive bug, over 50000 user's sensitive informations are leaked publicly. simplenote gave a $150 bounty to this hacker.

Read More: information disclosure lead to disclose user's private notes

This is a write-up of a hacker, How he earned 1K$ only using Waybackurls.

How I Scored 1K Bounty Using Waybackurls

Conclusion

In conclusion, OSINT is an essential tool for bug hunters and can significantly improve the accuracy and efficiency of their assessments. By gathering and analyzing information from publicly available sources, bug hunters can gain a deeper understanding of the target and its environment, prioritize their testing efforts, and follow ethical hacking practices. Whether you're a seasoned bug hunter or just starting out, incorporating OSINT into your assessments is sure to improve your results.

So we've discussed the practical use of Google and Wayback Machine in bug hunting. I have shared my methodology for using these tools effectively in bug-hunting investigations. However, these are just a few OSINT tools that bug hunters can use for reconnaissance.

In the next blog in this series, we will cover some more OSINT tools that bug hunters can use to gather information about potential targets.

Stay tuned for our next blog, where we will dive deeper into the world of OSINT and bug hunting!