A java web spider.
Arale can download entire web sites or specific resources from the web. Arale can also render dynamic sites to static pages. I wrote this utility in 2001 to familiarize myself with the java.net.* package. I'm not actively maintaining it anymore, the code does not shine but the spider is working fine.
Features
- Download and scan user-defined file types.
- Rename dynamic resources. Encode query strings into filenames.
- Set the number of simultaneous connections.
- Options for minimum and maximum file size.
- Domain depth support.
While many bots around are focused on page indexing, Arale is primarly designed for personal use. It fits the needs of advanced web surfers and web developers. Some real life cases are:
- downloading only images, videos, mp3 or zip files from a site.
- manuals, articles, ebooks fragmented in many files to discourage download.
- user-unfriendly sites. Popups, banners and tricky scripts annoying you before you can download a resource.
Multithreaded means that Arale can download more than one file simultaneously. Arale can easily saturate your bandwidth, thus providing the fastest possible download speed for your internet connection.
Currently Arale is a command-line tool. It would be nice to develop a GUI for it.
|
