WayURLS is a powerful CLI tool that allows you to effortlessly fetch URLs from the Wayback Machine, Common Crawl, and VirusTotal. With options to customize your fetch dates, exclude subdomains, and output results to a file, it streamlines your web research and data analysis tasks, making it an essential tool for developers and data enthusiasts.
Wayurls is an innovative Command-Line Interface (CLI) tool designed to seamlessly fetch URLs from popular web archiving sources such as the Wayback Machine, Common Crawl, and VirusTotal.
Key Features
- Multisource Fetching: Effortlessly retrieve URLs from multiple reputable sources, giving you comprehensive access to archived content.
- Flexible Options: Customize your URL fetching process with various options:
-d: Display fetch dates in the results.-t <domain|file>: Specify a target domain or use a file containing a list of domains.-n: Exclude subdomains for a cleaner result set.-o <file>: Output your results to a file, defaulting to standard output.-v: List all crawled versions of the URLs, or get details from VirusTotal with the-vt <key>option using your API key.
Usage Examples
Unlock the power of Wayurls to access archived URLs with straightforward commands:
# Fetch URLs for a single domain
wayurls example.com
# Fetch URLs from a list of domains
wayurls -t domains.txt -o results.txt
# Fetch URLs with dates, excluding subdomains
wayurls -d -n -t example.com
# List crawled versions of URLs
wayurls -v example.com
Wayurls invites contributions and improvements from the community, fostering collaborative development for enhanced URL management and archive retrieval.
Harness the potential of historical web data effortlessly with Wayurls!