Skip to content

Commit

Permalink
Update Readme
Browse files Browse the repository at this point in the history
  • Loading branch information
Pandalei97 committed Dec 1, 2023
1 parent 518cff0 commit 25908cd
Show file tree
Hide file tree
Showing 6 changed files with 13 additions and 12 deletions.
25 changes: 13 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,18 +83,19 @@ This endpoint allows you to create a new website configuration end execute a cra
POST http://127.0.0.1:8080/api/websites
```

| Parameter | Type | Description |
| :-------- | :------- | :------------------------- |
| `url` | `string` | **Required**. Starting url to crawl from |
| `depth` | `integer` | Maximum depth to crawl (**Default**: 2) |
| `limit` | `integer` | Maximum pages to crawl (**Default**: 400) |
| `headers` | `dict[str, str]` | Headers that will be passed to all crawl requests (**Default**: {})|
| `lighthouse` | `MetadataConfig` | Lighthouse configuration (**Default**: {'enabled':True, 'depth' 0}) |
| `technologies` | `MetadataConfig` | Technologies configuration (**Default**: {'enabled': False}) |
| `responsiveness` | `MetadataConfig` | Responsiveness configuration (**Default**: {'enabled': False}) |
| `carbon_footprint` | `MetadataConfig` | Carbon Footprint configuration (**Default**: {'enabled': False}) |
| `tags` | `list[str]` | List of tags to associate to this website (**Default**: [])|
| `crawl_every` | `integer` | Time to wait between each crawl (In days, >= 0, **Default**: 30)|
| Parameter | Type | Description |
|:-------------------|:-----------------|:--------------------------------------------------------------------|
| `url` | `string` | **Required**. Starting url to crawl from |
| `depth` | `integer` | Maximum depth to crawl (**Default**: 2) |
| `limit` | `integer` | Maximum pages to crawl (**Default**: 400) |
| `use_playwright` | `boolean` | Activation of scrapy-playwright (**Default**: false) |
| `headers` | `dict[str, str]` | Headers that will be passed to all crawl requests (**Default**: {}) |
| `lighthouse` | `MetadataConfig` | Lighthouse configuration (**Default**: {'enabled':True, 'depth' 0}) |
| `technologies` | `MetadataConfig` | Technologies configuration (**Default**: {'enabled': False}) |
| `responsiveness` | `MetadataConfig` | Responsiveness configuration (**Default**: {'enabled': False}) |
| `carbon_footprint` | `MetadataConfig` | Carbon Footprint configuration (**Default**: {'enabled': False}) |
| `tags` | `list[str]` | List of tags to associate to this website (**Default**: []) |
| `crawl_every` | `integer` | Time to wait between each crawl (In days, >= 0, **Default**: 30) |

**MetadataConfig**

Expand Down
Binary file modified demo/demo.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified demo/flower.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified demo/minio.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified demo/mongodb_crawl_configuration.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified demo/mongodb_crawl_progression.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 25908cd

Please sign in to comment.