Skip to content

Latest commit

 

History

History
17 lines (12 loc) · 428 Bytes

README.md

File metadata and controls

17 lines (12 loc) · 428 Bytes

Crawler

This package downloads a HTML part of a web page and follows its links to download related pages into ./storage directory.

Usage

php ./composer.phar install

php bin/app.php --url=https://spiegel.de

By default, it downloads 100 pages with 5 parallel processes.

To change that, provide additional CLI arguments:

php bin/app.php --url=https://spiegel.de --max-pages=500 --max-processes=30