Skip to content

Latest commit

 

History

History
34 lines (23 loc) · 751 Bytes

README.md

File metadata and controls

34 lines (23 loc) · 751 Bytes

Openings

Experimenting with scraping job openings from the web.

Prerequisites

  • Python 3.6+ (was developed using python 3.8)
  • pip/virtualenv installed

Installing

Create a virtualenv and install requirements. You can do this by running

virtualenv env
. env/bin/activate
pip install -r requirements

Running spiders

There are two spiders that you can use for now, one for Workable-backed career pages and one for Recruitee.

To run Workable-based crawler, edit openings/spiders/workable.py run:

scrapy runspider jobscrapper/spiders/workable.py

Similarly for recruitee just run:

scrapy runspider jobscrapper/spiders/recruitee.py

All jobs will be stored under data/, on a separate file per company.