Is there a way to search an entire site for instances of URLs in the text?

Question

Can you type in the homepage URL into a search engine or program or something and then it gives you a list of every URL listed in the text on every page within the website (publicly accessible pages only, of course)?

I don't know if you can use URLs for spam purposes but I have no intentions of trying to get URLs to spam. I can't even think of a way you can spam URLs but just in case, I wanted to make it clear I'm not planning to do that.

Answer 1

There is sitemaps generation website http://www.xml-sitemaps.com/ not sure this is the one you are looking for.

Hope this help.

Answer 2

If you're looking at writing your own utility for this then the easiest way to accomplish searching is to use a regular expression.



http://en.wikipedia.org/wiki/Regular_expression



These are formal constructs that allow you to match specific strings. You could, for example, look for every instance of "www.*.*" (where * means anything), or you could look for something even more specific like "www.*.* but not www.*.edu".



After you can match (or follow) links it's just a matter of recursively hitting every link in a page.



If you know your project is in a specific language (like PHP or Perl) then you can look up a specific reference for those languages. Almost every major language has the ability to do Regular Expressions.

Answer 3

Yes, there is. Whip up a Greasemonkey script to do the job. You'll want to call getElementsByTagName on , store the results, increment a counter N, change location.url to the Nth element in your stored array, create a new array containing the original array concatenated with a new list of all links on the new page and overwrite the stored array with this, repeat until there are no new links found.

Answer 4

This is probably the long way to do it but I would, after I'm already at a webpage, go to view then page source then ctrl+f and type www. And it'll show you where all the weblinks are at.

Answer 5

google does that