find HIDDEN urls!! (subdomain enumeration hacking) // ft. HakLuke

Video Notes:

Learning how to hack is fun, but having people spy on your internet traffic is not. Get 3 months of PIA for free when you sign up for a 2 year plan!(ps. That’s only $2.11 a month) https://ntck.co/PIA

In this video Chuck is going to talk to you about Hakrawler, an awesome Subdomain Enumeration tool created by Hakluke! Chuck is going to walk you through how to install Hawkrawler and another Subdomain Enumeration tool called Gau on a Kali Linux machine using Docker. 🔥🔥Join the NetworkChuck Academy!: https://ntck.co/NCAcademy

0:00 ⏩ Intro

1:23 ⏩ What are we doing here and why are we doing it?

3:13 ⏩ What is Hakrawler?

4:44 ⏩ Getting Hakrawler installed

5:51 ⏩ How do we use Hakrawler?

7:54 ⏩ How is Hakrawler working?

9:15 ⏩ Don’t be afraid and to tinker!

10:52 ⏩ How is your privacy?

12:30 ⏩ Two tools in one video!?

13:00 ⏩ Installing and using Gau

14:28 ⏩ Outro

Every website out there has hidden URLs, like check it out. When you go to reddit.com, there’s more than just this one URL. There are hidden secrets that we’re going to find, because if I run a tool like this, we’re gonna find a lot more hidden secrets. Dang. It’s still going, oh, by the way, I’m gonna show you how to use that tool. So get your copy. Ready. What’s going on guys. Welcome back to network. Chuck in today’s video, we’re gonna cover really two things. First. What I already showed you finding hidden URLs in a process called subdomain enumeration. And I’m gonna show you how to use and install two tools to make this thing happen, to find all the hidden secrets of the internet. But disclaimer, there are rules you have to obey you. Can’t just use these tools on anybody. You need permission. And we’ll talk about that here in a bit. 

Oh, by the way, this whole process of subdomain enumeration, this isn’t just for hackers. It’s for you. If you wanna keep your stuff safe or if you’re just curious. And the second thing we’re gonna talk about is really the thing I’m most excited about and that’s creating your own tools and the value behind that. In fact, one of the tools I’m showing you today, I got a chance to interview that creator. Now, before I show you how to use these tools, let’s talk a bit about what we’re doing here and why we’re doing it because yeah, I showed you that reddit.com does have more to it than just the, you know, the reddit.com. So for example, we might have a domain network, chuck.com, which in most cases, this domain will point to one location, most likely a server. But what we’ll also see is other hidden stuff. 

If we do a search with a tool like I just showed you. So for example here, if you look through our, our stuff here, I did a search on my own domain. Now don’t you go searching mine. I’m not giving you permission. And that’s the key here. You need permission again. We’ll talk more about that here in a second. But notice here, I’ve got a learn.network, chuck.com situation here. This is what we’re talking about when we refer to a sub domain and what makes these special is that these could point to anything. They could be anything. This learn.network, chuck.com could point to an entirely different system, an entirely different server. And that right there is the goal of subdomain enumeration in a wider scope. This is called recon discovering all the targets that you could possibly find vulnerabilities on that you can hack. Now, this is valuable in a lot of ways. 

First, you may have a company. You may have a website where you’re like, Hey, I wonder what I have going on. <laugh> you may have created this subdomain. That’s not used for anything. That’s actually pretty dangerous. I’ll have another video talking about how we can hack that. And on the other hand, you may be a hacker, maybe be doing some bug bounty, and it’s your goal to attack a company and find all the possible endpoints that you could hack. And this is actually pretty stinking cool. If you’re unfamiliar with the bug bounty world, companies will actually pay you to hack them. They’ll partner with companies like hacker one, which you can go and sign up for right now, free account and companies like, ah, I don’t know. You ever heard of Snapchat tender Spotify Reddit. And if you take a look at the details, they’ll tell you what you’re allowed to hack. 

They’ll be an entire scope of what you’re allowed and what you’re not allowed to do. So currently at this moment in time, you could take a look at the scope for Reddit and do a few fun things. Keeping in mind the goal here would be to use tools to find targets and using scanning tools like I’m about to show you. You might find things like, I don’t know, vulnerable.reddit.com, which just is that a real thing? Probably not. And that might be a server that has a massive vulnerability. And if you find it, you can write a report on it, submit it to hacker one and write it and they’ll give you some money if it’s actually a thing. So now let’s step forward and actually install one of these tools. The first one we’re looking at is a tool called hack rawer from a amazing dude named hack, Luke, Luke, go ahead and tell us more about what hack roller 

Is. Hack crawler is a web crawler. So basically, uh, you feed it, uh, a website or many websites and it will navigate to those websites. And then it will find all of the links in that page. And then it will navigate to those links and it will navigate to the links in those pages and those pages and those pages, um, to as many levels down as you specify. So, um, as a, as a hacker, I guess, um, one of the things that you are looking for is, um, just coverage over web applications and making sure you, you can cover things or, or discover things, um, with the least amount of manual work possible because it’s quite, uh, it’s quite a daunting task to like discover every single, uh, link or, or feature in a, in a web application manually. 

Now, hack crawler is a tool you have to be careful with because it is very powerful and it’s using more of what we’ve refer to as active enumeration, meaning it’s actually going out into the domains we’re talking about here. So for example, network, chuck.com. I can write my own domain. Oh my gosh. And it’s kind of creepy crawlies, just crawling it like a spider, trying to figure out all the stuff. And I’m pointing this out because not all scopes include active enumeration. Keep that in mind, be careful, be cautious. So anyways, how do we install this? There’s a number of ways, but I found my favorite way is using Docker. If you don’t know what Docker is, I’ve got a wonderful video up here or over here. I never know. And I’ll never take the time to remember. So grab you some Linux call Linux is a fun one to have maybe Ubuntu and make sure you have Docker installed. 

I don’t have it installed right now. So I’m gonna do it right now. First I will do pseudo a P T blah, update to update all my delicious repositories. I’ve been saying delicious a lot. I’m just gonna take a sip of coffee real quick. Oh, wrong password. There we go. And then I’ll type in pseudo a PT install, docker.io. And I’ll do a dash Y at the end. This should work for most people. Keeping in mind, this is a Debbie and based installation. I’m showing you hit enter quick sip of coffee while it’s doing its thing. And it might take a second because Docker is pure magic and you have to let the magic just do its thing. I’m trust me. And it’s done cool. Now for this install, I’m literally taking the code from hack Luke’s GitHub for a local Docker install using build. 

I found this is the best way to do it. Every other way is kind of frustrating. So this will give you the latest and greatest and just, just do that. So if your code looks like this, as I’m pacing, it, you’re good to go hit, enter and permission denied. I need some pseudo action specifically right in front of my pseudo or my Docker build. And my Docker run. Gotta give yourself permission. There we go. Now we can do it doing its thing. Go laying. Did that rhyme. I don’t know. And if you’re curious, what it’s doing right now is pulling down all the Docker containers or images. It will need to build out this Docker container thing. And mine is done. Now you’ll know yours is pretty good to go. If you immediately get some help icon or help information. So how to use hat crawl or everything, dash D and uh, how deep to go, how deep to crawl, but all that stuff. 

So, you know, it works if that happened. So now how do we use it and how does it work? Let’s talk about that real quick first. Let’s just try it out. So I wanna do my own domain. I’ll start with an echo and I’m echoing out what domain I want it to use. It’ll be HTPs S colon w whack network, chuck.com. You use something else, please don’t use mine and then I’ll do a pipe. And then now for my Docker command, I will need some pseudo action here. So pseudo Docker run, this is actually running a container. I’ll feed us some parameters, dash RM dash. I not gonna cover that right now, specify our container, which would be hack Luke slash hack roller. And really that’s all you need at this point, just for a basic scan. And it’ll just feed it right into your, into the terminal party. Hardy, look at it, go, and that was wicked fast. Now what it’s doing is actually again, it’s more active and it’s kind of different. This is the reason that Luke created this tool himself. 

Uh, I always, you know, some, sometimes I, I make my own tools. You know, I usually don’t spend more than a couple of days making a tool, um, or, or even like one day. Um, but it’s usually just to scratch my own niche as they say. So like, I’ll be doing some particular task and I just don’t know of a good way to do that or an effective way to do that. So I just write a quick go link tool. 

Now, as Luke mentioned earlier, this tool’s kind of nuts. If you notice it actually went out to every URL, it could find on my, my website and went to it and then it found the other URLs or links that it’s referencing to and went to those as well. And it also found, um, JavaScript files, which are super juicy and handy when you are, uh, doing some hacking and bug bounty. So I notice here it’s going out to, uh, URLs. So look, it found my store and all the different accessories and merch I sell. I can also throw in the parameter dash subs to specify, Hey, I want, I want you to try to look the subdomains and get them. So go ahead and do that right. And check this out. It’ll actually go to like my Twitter links and stuff, which is kind of crazy. 

Yeah. Like it’ll navigate out and find like, oh, here’s this YouTube, here’s this Instagram, it’s a powerful tool. Luke has this great medium article that can go deeper on it for you. And here’s specifically how it’s working. It’s querying the way back machine, which if you don’t know what that is, the way back machine kind of keeps in archive of the web. So a website as you see it now, and then a website as you may have seen it five years ago. So this is literally a time machine looking at your LS and times past, it’s also parsing the robots TXT file and the site map XML files. And of course it is spidering the application, which if you’re rack neph, phobic like me, just that phrase makes you freak out a little bit. Now I wanna touch on this real quick. Luke built this tool himself. 

He did it because he found a need that the other tools out there just weren’t meeting. And I just, I love this about the hacking community and really just the it community in general. It’s that mentality of like, okay, I need to tinker. I need to hack. And if there’s not a tool available for me, I can just make one myself. And it’s that mentality that I want you to just grasp, grab onto. And that I love what Luke said about the language you use. So this particular tool, normally tools are written in Python, cuz it’s, Python’s amazing. But for this, he used go Lang. Now Goling is from Google. It’s a very, very fast language. Like that’s, if you notice like we, this program’s very, very fast. That’s why he wrote it in go Lang, but also he wanted to learn Golan. 

So I, I knew Python before I knew go Lang. Um, but yeah, ultimately, you know, go Lang has native concurrency and that’s the main reason I use it. Um, also, you know, because it’s a compiled language, it’s just a bit faster, but you know, it really depends. Um, you, you can certainly write the same tool in Python and Golan and the Golan one might be slower if you don’t code it in a way that is particularly, um, effective or, or, um, efficient. So it just, it just really depends. Um, but yeah, I just, uh, I, I, I can write in Python as well. Um, quite well, but, but I end up writing everything in Golan these days just because I don’t know, I just like it being compiled. I like my tools to be compiled and um, I just really like go Lang as a language. Um, but yeah, the main thing is, is concurrency. That’s what keeps me with go Lang um, yeah, it’s just, just faster. And a lot of the things that I do, like a lot of the things that I code, I code them because I want to do them over a bunch of different targets or, or something like that. So, um, being, being concurrent is like really important, I guess. 

So Luke chose goaling for this project to also teach himself goaling. So for those of you getting into it, learning, hacking, learning, networking, learning, cloud, whatever it is, don’t be afraid to do stuff like this. I don’t care if there’s already a tool out there that does what you want to do. In fact, this hack crawl or tool it’s does some new stuff, but it’s not completely novel. There are other tools that do subdomain enumeration, 

Oh, this, this other person’s tool already does this. And I’m like, well, if I knew that that existed, I wouldn’t have done it myself. You know, I would’ve just used that. Um, so, so really like, um, yeah, it’s, it’s more of a, it’s more of a convenience thing for me. Like if I can, if I can find any task I do regularly that is, um, you know, could be automated, um, without too much, too much time investment, then, then I’ll, I’ll basically write a tool for it. So it’s, it’s basically being lazy, you know, <laugh>, it’s just automating things that are, that I, that I do repetitively or that are, that are not particularly, uh, fun to do or whatever, if I can automate that, I will, 

It doesn’t matter if someone’s already made this tool, make your own, add a little bit of change to it, do something cool that you’re trying to learn, or just add a feature to something. A 

Lot of people that I, I speak to about this stuff that are afraid to release their code because they think it’s gonna get picked to pieces. Um, and it’s the same with writing blogs, releasing videos. Mm. Um, anything like that, you know, people are just so scared to start. And I always say like, just do it just start. Um, nobody, nobody cares. <laugh> nobody cares if it sucks. Um, you know, and you know, maybe there’ll be one or two people, always that kind of a negative, um, towards what you’re doing, but in the end, um, it doesn’t matter as long as you, as long as your heart’s in the right place. And you’re releasing your, your, you know, being open with the things that you’re creating. I think people will always be the majority of people will always appreciate what you’re doing. 

So hack, crawler, amazing tool that you can use right now written by an amazing guy who I think is an inspiration for all of us to go, Hey, let’s just, let’s just make some stuff. And, and, and don’t be afraid to break it. Don’t be afraid to put yourself out there. It’s all part of the learning process. That’s part of like getting yourself in the community. And by the way, our interview is a bit longer than what I’ve showed you here. If you wanna see the entire thing, I gotta link below and also please go follow hack, Luke. Dude’s amazing. And his first name missed at hack. I mean, his first name is obviously Luke, but anyways, go follow him. He’s cool. Hey, Hey. Hey, real quick. I wanna talk to you for a second about your privacy. Are you using a VPN? You better be using a VPN. 

So let’s take a quick coffee break and talk about our sponsor private internet access, private internet access is what I use for VPN a virtual private network. And as you’re doing all these crazy hacking things, scanning people and such, you need to have a VPN on because as you may know, whenever you do anything on the internet, no matter what website you go to or what website you’re scanning, they can see you, everything about you, your IP address, they know where you live. So you wanna do your best right now to hide that using a VPN like private internet access will enable you to hide your IP address. You’ll be using one of their IP addresses. So it’s keeping you hidden and also your data will be encrypted. You don’t want other hackers seeing what you’re doing. And I know what you’re probably thinking. Well, can’t private internet access. 

See what I’m doing? Want to keep a record of where I’m going and just give it away to the police or anybody who wants it. And the answer is no, they have a strict, no log policy. So you’re safe. And please don’t just use it for your computer. Use it for all your devices. You can put it on your phone, your Android, your iPhone, your Mac, your toilet coming soon. No, I’m just playing. They don’t, they don’t have a toilet app yet, yet. In fact, with one plan, you can protect up to 10 devices at one time and they do have a 30 day money back guarantee. So just, you know, just do it. So don’t surf the internet all raw and exposed. Cover your sofa. Use VPN check ’em out. Link below private internet access.com for slash network Chuck, or just click the link. 

Cause the links are easier. You’ll get complete digital privacy for less than $3 a month. Plus three months free. That’s roughly about $2 and 59 cents a month, just 78% off. Okay. Just, just do it. Now. I do wanna show you one more tool for SubD domain enumeration and just getting all the URLs. And that’s actually the name of this next tool it’s called GAU or gal. Get all URLs, very straightforward name from a guy named CDL that says hacker name. I think he was interviewed by NA sack here, but his story was much the same as hack Luke. There were things that he wanted to get done that he needed to automate. And there, there wasn’t a tool available. So he just wrote it himself. And this thing is pretty popular. Let’s play with it real quick. So again, same story. Make sure you got Lennox. 

Callie. Lennox is always fun to play with. And just like before with hat crawler, my preferred method to install, this is with Docker. Docker’s awesome. I will link the GitHub page to this tool below and really all you need is to run this one command. Once you have Docker installed, I wanna paste that here, making sure I got some pseudo action at the beginning and let’s party. Oh, wait pass. There we go. So it it’ll go. It’ll go. Hey, I, I can’t find the image. I’m gonna go, oh God, it’s already done. <laugh> I can’t even tell you. So it went and found the images pulled them down, installed it. Ready to go. Now, as I mentioned before, hack crawler is more of a active enumeration gal or GI all URLs is more passive enumeration. And what that means in let’s look at his GitHub real quick is that it’s not actually going out to the web server. 

It’s not like, let me hit you up domain and bombard it with spiders. No more, no spiders here. Thank goodness. What it’s actually doing is querying databases that already have this information. So it FES known URLs from things like alien vaults, open thread exchange. It also uses the way back machine and it uses common crawl, which this is very cool. It’s actually hosted on AWS’s, um, servers for free. They literally have an open repository of websites they’ve crawled. So that’s, that’s super cool. So no crawling here. It just accesses that database, all that data. So let’s try it out on my website for this pretty simple pseudo Docker run, do a dash dash RM dash I and then the name of the Docker container, which will be SX security, four slash GAU. And then just the domain. That’s simple network, chuck.com go. And it’s doing a lot of stuff right now and giving us a ton of information, almost too much information to where, like, you don’t know what to do with it, by the way, in this video, I’m not gonna show you how to parsh through this information and figure out how to use it, to find hacking targets. 

That’s another video for another time. If you do wanna learn that I do have a bug bounty course from Nahham sack at network Chuck academy linked below. So two tools that you can use for subdomain in admiration, basically just finding all the hidden URLs you can for a given domain. They are very handy for your own sites to discover, Hey, what do I have out there? What am I putting out there? Maybe there are some things I don’t want to be out there that I don’t want to be found. It’s better that I find them first, right? So maybe do a search on your own stuff and also manage a hacker as a bug bounty hunter. These things are invaluable. They save you time. They help you automate and they inspire you to write your own tools, looking at the authors of these tools. That’s what they did. 

And I think you should do the same thing. You could start just by forking, which is the oddest term, but you can fork one of their repositories and just add a few more things to it. Just make sure you share that same change to the community. Anyways. That’s all I have for this video. Thanks for having some coffee with me and learning a few things about stuff. Oh, by the way, have you hacked the YouTube algorithm today? Let’s make sure you do with that, like button notification, bell comment, subscribe. You gotta hack YouTube today. Ethically of course.

Check Out Network Chuck's Coffee and MERCH Shop