Skip to content

Commit

Permalink
Merge branch 'main' into main
Browse files Browse the repository at this point in the history
  • Loading branch information
rockeynebhwani authored Dec 13, 2024
2 parents fd9f657 + 378d39f commit 94a3987
Show file tree
Hide file tree
Showing 106 changed files with 4,409 additions and 9,251 deletions.
1 change: 0 additions & 1 deletion .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ module.exports = {
extends: [
'@nuxtjs',
'prettier',
'prettier/vue',
'plugin:prettier/recommended',
'plugin:nuxt/recommended',
'plugin:json/recommended',
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/technology-request.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ assignees: ''

---

You may request a new technology to be added but chances of having it included are greatly improved if you submit a pull request. Please refer to the [contributing guide](https://github.com/AliasIO/wappalyzer/blob/master/CONTRIBUTING.md).
You may request a new technology to be added but chances of having it included are greatly improved if you submit a pull request. Please refer to the [contributing guide](https://github.com/HTTPArchive/wappalyzer/blob/main/CONTRIBUTING.md).
10 changes: 10 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<!-- Add any related issues and a description of the changes proposed in the pull request. -->
Resolves \#1

Description of the changes...

---
<!-- List the pages that should be automatically tested as part of your custom metric changes. -->
**Test websites**:

- https://example.com/
28 changes: 28 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Linting

on:
pull_request_target:
branches:
- main
workflow_dispatch:

jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0

- name: Install dependencies
run: yarn install

- name: Validate
run: yarn run validate

- name: Run WebPageTest with unit tests
id: unit-test
run: yarn run lint
89 changes: 89 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: Tests

on:
pull_request_target:
branches:
- main
paths:
- "src/technologies/*.json"
- "src/categories.json"
- "src/groups.json"
- "tests/*.js"
workflow_dispatch:

jobs:
test:
name: WebPageTest Test Cases
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0

- name: Install dependencies
run: yarn install

- name: Validate
run: yarn run validate

- name: Run WebPageTest with unit tests
id: unit-test
env:
WPT_SERVER: "webpagetest.httparchive.org"
WPT_API_KEY: ${{ secrets.HA_API_KEY }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: yarn run test

- name: Run WebPageTest for more websites
id: wpt-test
env:
WPT_SERVER: "webpagetest.httparchive.org"
WPT_API_KEY: ${{ secrets.HA_API_KEY }}
PR_NUMBER: ${{ github.event.pull_request.number }}
PR_BODY: ${{ github.event.pull_request.body }}
run: |
# Read PR body into an array, removing line breaks and carriage returns
lines=()
while IFS= read -r line; do
lines+=("${line//[$'\r\n']}")
done <<< "$PR_BODY"
# Find the index of the line after "**Test websites**:"
start_index=-1
for ((i=0; i<${#lines[@]}; i++)); do
if [[ "${lines[$i]}" == *"**Test websites**:"* ]]; then
start_index=$((i + 1))
break
fi
done
# If the index is valid, then parse the URLs
if [ $start_index -gt -1 ]; then
# Initialize an array for URLs
URLS=()
url_pattern="((http|https|ftp):\/\/[a-zA-Z0-9.-]+(\.[a-zA-Z]{2,4})(\/[a-zA-Z0-9_.-]+)*(\/?)(\?[a-zA-Z0-9_.-]+=[a-zA-Z0-9%_.-]+)*(\#?)([a-zA-Z0-9%_.-=]+)*)"
for ((i=start_index; i<${#lines[@]}; i++)); do
if [[ ${lines[$i]} =~ $url_pattern ]]; then
URLS+=("${BASH_REMATCH[1]}")
fi
done
# Run WebPageTest for each URL
for TEST_WEBSITE in "${URLS[@]}"; do
echo "::group::Running WPT test for $TEST_WEBSITE"
node tests/wpt.js "$TEST_WEBSITE"
echo "::endgroup::"
done
else
echo "No websites found."
fi
- name: Add comment with results
uses: mshick/add-pr-comment@v2
if: steps.unit-test.outcome == 'success' || steps.wpt-test.outcome == 'success'
with:
refresh-message-position: true
message-path: test-results.md
45 changes: 45 additions & 0 deletions .github/workflows/upload.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Tests

on:
push:
branches:
- main
paths:
- "src/technologies/*.json"
- "src/categories.json"
- "src/groups.json"
workflow_dispatch:

jobs:
test:
name: Test and upload to GCP
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0

- name: Install dependencies
run: yarn install

- name: Validate
run: yarn run validate

- name: Run WebPageTest with unit tests
id: unit-test
env:
WPT_SERVER: "webpagetest.httparchive.org"
WPT_API_KEY: ${{ secrets.HA_API_KEY }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: yarn run test

- name: Upload to GCP
id: upload
env:
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
run: |
echo $GCP_SA_KEY > /tmp/gcp_key.json
yarn run upload
30 changes: 0 additions & 30 deletions .github/workflows/validate.yml

This file was deleted.

9 changes: 1 addition & 8 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,22 +1,15 @@
/build/*
/src/images/icons/converted/*
/src/manifest.json
/src/manifest.bak.json

node_modules
package-lock.json

!.gitkeep

Thumbs.db
Desktop.ini
*.DS_Store
*.log
._*
tags
tags.*
.idea
/nbproject/private/
src/out.json

keys.json
.env
20 changes: 8 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Contributing

Wappalyzer is an [GPLv3 licensed](https://github.com/wappalyzer/wappalyzer/blob/master/LICENSE), open source project written in JavaScript. Anyone is welcome to contribute.
Wappalyzer is an [GPLv3 licensed](https://github.com/HTTPArchive/wappalyzer/blob/main/LICENSE), open source project written in JavaScript. Anyone is welcome to contribute.

## Getting started

To get started, see the [README](https://github.com/wappalyzer/wappalyzer/blob/master/README.md).
To get started, see the [README](https://github.com/HTTPArchive/wappalyzer/blob/main/README.md).

## Submitting changes

Expand All @@ -15,23 +15,19 @@ To get started, see the [README](https://github.com/wappalyzer/wappalyzer/blob/m

## Adding a new technology

Wappalyzer uses [regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions) to fingerprint technologies. Refer to the [specification](https://github.com/wappalyzer/wappalyzer/blob/master/README.md#specification) for detail.
Wappalyzer uses [regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions) to fingerprint technologies. Refer to the [specification](https://github.com/HTTPArchive/wappalyzer/blob/main/README.md#specification) for detail.

- Add a new block to [`src/technologies/*.json`](https://github.com/wappalyzer/wappalyzer/blob/master/src/technologies). The filename should match the first letter of the technology name (a-z). Use `_.json` if the first character is a number or symbol.
- Add an icon to [`src/images/icons`](https://github.com/wappalyzer/wappalyzer/tree/master/src/images/icons). The image must be square, either SVG or PNG (32 x 32 pixels).
- Add a new block to [`src/technologies/*.json`](https://github.com/HTTPArchive/wappalyzer/blob/main/src/technologies). The filename should match the first letter of the technology name (a-z). Use `_.json` if the first character is a number or symbol.
- Add an icon to [`src/images/icons`](https://github.com/HTTPArchive/wappalyzer/tree/master/src/images/icons). The image must be square, either SVG or PNG (32 x 32 pixels).

Only widely used technologies are accepted. When creating a pull request, include ten or more links to websites that use the application, a GitHub page with at least 1,000 stars or anything that will help establish the size of the user base.

## Adding a new category

Please [open an issue on GitHub](https://github.com/wappalyzer/wappalyzer/issues) first to discuss the need for a new category.
Please [open an issue on GitHub](https://github.com/HTTPArchive/wappalyzer/issues) first to discuss the need for a new category.

To add a category, edit [`src/categories.json`](https://github.com/wappalyzer/wappalyzer/blob/master/src/categories.json) and update every [locale](https://github.com/wappalyzer/wappalyzer/tree/master/src/_locales). You may use the English category name in all of them.

## Adding a new translation

To add a new translation, copy the `en` folder in [`src/_locales`](https://github.com/wappalyzer/wappalyzer/tree/master/src/_locales), rename it to the relevant two-letter country code and update the containing `messages.json` file.
To add a category, edit [`src/categories.json`](https://github.com/HTTPArchive/wappalyzer/blob/main/src/categories.json) and update every [locale](https://github.com/HTTPArchive/wappalyzer/tree/master/src/_locales). You may use the English category name in all of them.

## Adding a new feature

Please [open an issue on GitHub](https://github.com/wappalyzer/wappalyzer/issues) first. New features and large changes are rarely accepted without prior discussion.
Please [open an issue on GitHub](https://github.com/HTTPArchive/wappalyzer/issues) first. New features and large changes are rarely accepted without prior discussion.
23 changes: 10 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
[![Validate](https://github.com/wappalyzer/wappalyzer/actions/workflows/validate.yml/badge.svg)](https://github.com/wappalyzer/wappalyzer/actions/workflows/validate.yml)

[![Test](https://github.com/HTTPArchive/wappalyzer/actions/workflows/test.yml/badge.svg)](https://github.com/HTTPArchive/wappalyzer/actions/workflows/test.yml)

[![Lint](https://github.com/HTTPArchive/wappalyzer/actions/workflows/lint.yml/badge.svg)](https://github.com/HTTPArchive/wappalyzer/actions/workflows/lint.yml)

[![Github Sponsor](https://img.shields.io/static/v1?label=Sponsor&message=%E2%9D%A4&logo=GitHub&link=https://github.com/sponsors/AliasIO)](https://github.com/sponsors/AliasIO)

<a href="https://www.wappalyzer.com/?utm_source=readme&utm_medium=github&utm_campaign=wappalyzer"><img src="https://www.wappalyzer.com/images/logo/icon_192.png" height="72" alt="Wappalyzer" align="left" /></a>
Expand All @@ -13,12 +17,11 @@

- [Git](https://git-scm.com)
- [Node.js](https://nodejs.org) version 14 or higher
- [Yarn](https://yarnpkg.com)

## Quick start

```sh
git clone https://github.com/wappalyzer/wappalyzer.git
git clone https://github.com/HTTPArchive/wappalyzer.git
cd wappalyzer
yarn install
```
Expand All @@ -27,24 +30,18 @@ yarn install

### Chrome extension

- Go to `about:extensions`
- Go to `chrome:extensions`
- Enable 'Developer mode'
- Click 'Load unpacked'
- Select `src`

### Firefox extension

- Go to `about:debugging#/runtime/this-firefox`
- Click 'Load Temporary Add-on'
- Select `src/manifest.json`

## Specification

A long list of [regular expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions) is used to identify technologies on web pages. Wappalyzer inspects HTML code, as well as JavaScript variables, response headers and more.

Patterns (regular expressions) are kept in [`src/technologies/`](https://github.com/wappalyzer/wappalyzer/blob/master/src/technologies). The following is an example of an application fingerprint.
Patterns (regular expressions) are kept in [`src/technologies/`](https://github.com/HTTPArchive/wappalyzer/tree/main/src/technologies). The following is an example of an application fingerprint.

#### Example
### Example

```json
"Example": {
Expand Down Expand Up @@ -104,7 +101,7 @@ Patterns (regular expressions) are kept in [`src/technologies/`](https://github.

## JSON fields

Find the JSON schema at [`schema.json`](https://github.com/wappalyzer/wappalyzer/blob/master/schema.json).
Find the JSON schema at [`schema.json`](https://github.com/HTTPArchive/wappalyzer/blob/main/schema.json).

### Required properties

Expand Down
45 changes: 0 additions & 45 deletions bin/build.js

This file was deleted.

Loading

0 comments on commit 94a3987

Please sign in to comment.