Welcome to Clinic Web Scraper! This tool is designed to scrape clinic data for various clinics in Singapore, previously available on https://www.hcidirectory.gov.sg/hcidirectory/. As of 25/10/2023, the website has migrated to a different domain, deprecating this script 😧.
- Python 3.x installed on your system
- A web browser (preferably Microsoft Edge, as the driver for Microsft Edge is already available with this repository)
-
Download the Repository You can clone the repository using the following command:
git clone https://github.com/AlagappanRa/Clinic-Web-Scraper -
Navigate to the Repository on Your Local Machine Navigate to the directory where you cloned the repository:
cd path_to_directory/Clinic-Web-Scraper -
You need to install Selenium for the web scraping functionality. Run the following command:
pip install selenium -
Install Web Driver (Skip this step if you already have a driver installed OR will be using microsoft edge)
-
ChromeDriver for Chrome
-
GeckoDriver for Firefox
-
Download the appropriate driver, unzip it, and place it in a known location on your system.
-
You might need to provide the path to this driver in the script or add it to your system's PATH.
- Run the Script If you are using VSCode:
- Open the repository in VSCode
- Ensure you have the Python extension installed in VSCode
- Open the Python script file
- Click the 'Run Python File in Terminal' option (usually present at the top-right corner of the script editor)
- Alternatively, you can run the script from the terminal:
python scraper.py
- Ensure you are respecting the robots.txt of the website when scraping.
- Web scraping may be subject to legal and ethical considerations. Always ensure you have the right to scrape and use the data you're collecting.
- The structure of websites can change over time. If the script stops working, the website's structure might have changed, and the script might need adjustments.
If you find any issues or have suggestions for improvements, please open an issue or submit a pull request. Contributions are welcome!
This project is licensed under the MIT License.
Happy scraping! And always remember to scrape responsibly.