Crawling A Domain For Unknown Links

[mantraminator]

I have a problem. On a domain, lets name it www.example.com are a lot of Files uploaded, lets say in www.example.com/file.MOV, they are all downloadable but I forgot/ dont know some links of the files, so is there any programm or technique which could "brute-force-try" possible links?

I mean, i could also try out the links (e.g. www.example.com/file1.MOV, www.example.com/file2.MOV) Is there any tool that could do this work for me?

or could there be any search engines or whatever that could show me all the subfolders or files that are on this domain?

[Falcon1986]

When you say "brute force" this leads me to think that the website is probably not openly showing links to their media perhaps because they either only want you to be able to view it or having them publicly displayed may result in someone crawling their entire site and performing a mass download of everything, which would seriously impact their bandwidth or server.

As a website administrator myself I understand this and take actions to prevent it such as restricting directory viewing, hotlinking, etc. If I were to let anyone be able to download any and everything from my website, that would pose a serious security issue since some files in there are configuration files that contain usernames and passwords, for example.

What you are trying to do would probably be considered as leeching which may be discourteous to the hosting provider or their client. Now if these links are made publicly made available by the content provider then that is a different story. They can probably afford to do this if they have the bandwidth and server resources to allow mass downloads.

[Matt]

Sounds like you're looking for files you aren't supposed to have..

*Matt will be watching this thread.

[hitest]

Sounds like you're looking for files you aren't supposed to have..

*Matt will be watching this thread.

Yeah, well-said, Matt:-) When I hear the words brute-force I think hacking.

[jcl]

"Brute force" would mean testing every possible URL that fits the pattern. There are at least 10⁶¹³ potentially working URLs that fit the pattern "www.example.com/*".

[mantraminator]

Sounds like you're looking for files you aren't supposed to have..

*Matt will be watching this thread.

Yeah, well-said, Matt:-) When I hear the words brute-force I think hacking.

lol is it possible to hack I site by finding a lot of Links? IF I want to find hack and damaging Programms I would search on an illegal crack/hack site and not here

"Brute force" would mean testing every possible URL that fits the pattern. There are at least 10⁶¹³ potentially working URLs that fit the pattern "www.example.com/*".

i know, but I already had the links once I just lost them, and as there is a big community the webmasters have no time and no want to give me the links, but there is (for me atleast) very important informations on this links. So I know nealry what the links are called and how long they are so I can specialize the search extremely, and I would do a very low search/sec so the domain would not get disturbed

[jcl]

lol is it possible to hack I site by finding a lot of Links?

It's possible to violate the site operator's expectation that the files won't be accessed by random people, but that expectation is so unreasonable, and the 'attack' so banal, that I wouldn't call it hacking. Even security through obscurity should be robust against typos.

i know, but I already had the links once I just lost them, and as there is a big community the webmasters have no time and no want to give me the links, but there is (for me atleast) very important informations on this links.

Pr0nz?

Edited April 22, 2008 by jcl

[mantraminator]

Pr0nz?

lol no, actually if you wanna know it, its spiritual stuff, just important for me, pr0nz can be found everywhere , so can anybody help me?

Edited April 22, 2008 by mantraminator

[iccaros]

you could create a script with wget.. but this would work if you knew what you were looking for.. with directory listing turned off, you can not just list the contents of a directory on a web server..

http://en.wikipedia.org/wiki/Webserver_directory_index

[hitest]

so can anybody help me?

Perhaps you should send an e-mail to the site administrator and see if he/she will let you look at the contents of the web directory. Explain your issue and perhaps you'll get access to your stuff.

[jcl]

you could create a script with wget..

FWIW, with wget I get around 130 HEADs per second on loopback, around 150 on my LAN, and around 7 on the interwebs at large. The process seems to be latency-bound: CPU and network load are negligible, almost imperceptible, in all cases. So, I suppose probing might not kill the target server, but if hitting it as hard as possible only yields 5 HEADs per second....

[mantraminator]

wow this program seems to be what i search (though im not sure cause i cant really understand it) @jcl have you already made a script that can search for links?

[jcl]

wow this program seems to be what i search (though im not sure cause i cant really understand it) @jcl have you already made a script that can search for links?

No and no. wget doesn't do what you want and the 'script' -- it was a one-liner -- I used for the benchmark definitely doesn't do what you want.

[iccaros]

you may be able to hack this to do what you need.

http://69.10.233.10/KB/IP/davwebspider.aspx

you need some C# skills or any c language and an ability to code. I think you have to sign up at the code project to download code?

sharpdevelop is a great IDE for .NET...

[JSKY]

Just exactly which website are you trying to find this info on?

And just which files are you trying to retrieve? Most files loaded on a site will have links somewhere to the info. New or old.

If it's spiritual stuff, surly it can't be this much of a secret. If you trying to break into the Scientology`s website. Anon has been doing a good job of that allready.