Data-trace-competition

Scripts in the file of analyse-tools

1. dealShortUrl.py

reverse the most common short urls to the original urls

2. shortUrl.py

reverse the rest short urls to the original urls

3. getDomain.py

web scraping to get domain

4. getDNSInfo.py

web scraping to get dns info

5. getWhoisInfo.py

web scraping to get whois info

6. simHash.py

compute the simHash of urls

7. textDistance.py

calculate the text distances of urls based on the simHash

8. shang.py

calculate the entropy of texts

9. countKey.py

count the numbers of the keys of urls

Scripts in the root directory

---for trace1---

Use xgboost to train and predict the data

1. trace1.py

---for trace2---

Use data mining to cluster the data

1. ultis.py
2. trace2-1-step.py
3. trace2-2-step.py
4. trace2-3-step.py
5. trace2-4-step.py
6. trace2-5-step.py