Skip to content

Commit

Permalink
Delivery of REST API to master (#123)
Browse files Browse the repository at this point in the history
* #32 : Reset Travis badge link

* #32 : More work on REST API security

* Moved security code out of v1 package hierarchy.
* Added database model + SQL.
* Reduced INFO logging in Maven build output.

* #32 : Added JWT user service and factory class

* #32 : Added JWT utils class

* #32 : Added JWT authentication request and response

* #32 : Added JWT AuthenticationEntryPoint

* #32 : Added JWT Authentication filter

* #32 : Fixed broken test for JWT Authentication filter

* #32 : Removed unused annotation

* #32 : Added Authentication Controller

* #32 : Wired up JWT authentication for Email Alerts config endpoint

TODO:
* Rest of controllers need updating.
* Add tests for RBAC on mutating ops.
* Sanitise properties.

* #32 : Typos n polish

* #32 : Wired up JWT authentication for remaining config endpoints

TODO:
* Add tests for RBAC on mutating ops.
* Sanitise properties.

* #32 : Wired up JWT authentication for runtime endpoints

TODO:
* Add tests for RBAC on mutating ops.
* Sanitise properties.

* #32 : Added tests for RBAC on mutating ops

TODO:
* Fix getUsername() NPE.
* Fix Gradle build.
* Sanitise properties.
* Swagger API docs.
* README updates.

* #32 : Fixed getUsername() NPE in controller log statements

TODO:
* Fix Gradle build.
* Sanitise properties.
* Swagger API docs.
* README updates.

* #32 : Fixed getUsername() NPE in controller log statements

TODO:
* Fix Gradle build.
* Sanitise properties.
* Swagger API docs.
* README updates.

* #32 : Fixed Gradle build

TODO:
* Swagger API docs.
* Sanitise app config properties.
* README updates.

* #32 : Fixed some Sonar issues

TODO:
* Swagger API docs.
* Sanitise app config properties.
* README updates.

* #32 : Fixed a few more Sonar issues

TODO:
* Swagger API docs.
* Sanitise app config properties.
* README updates.

* #32 : Started work on Swagger docs

* #32 : Some class renaming + config init checks

* #32 : Cleanup of Swagger docs

TODO:
* Update Gradle build
* Add authentication to Swagger UI
* README updates
* Fix Sonar issues

* #32 : Started introducing some JBehave BDD tests

* #32 : Started introducing some JBehave BDD tests - POM cleanup

* #32 : More JBehave BDD tests

* #32 : Cleanup of JBehave tests

* #32 : Added more JBehave BDD tests + code cleanup

* #32 : Removed commented out code

* #32 : Don't skip rest of JBehave tests after first failure

* #32 : README updates for using REST API

* #32 : More README updates for using REST API

* #32 : Added Authorize button to Swagger docs for trying endpoints

* #32 : Set token expires time to 10 mins + doc updates

* #32 : Fixed checkstyle errors

* #32 : Fixed some Sonar issues

* #32 : Fixed some Sonar issues

* #32 : Minor README updates

* #32 : Attempt to fix Sonar coverage

* #32 : Fix path to XML JaCoco reports

* #32 : Still trying to fix generation of Sonar coverage reports...

* #32 : Still trying to fix generation of Sonar coverage reports...

* #32 : Pin-pointing fix for creating Sonar coverage reports

* #32 : Pin-pointing fix for creating Sonar coverage reports

* #32 : Pin-pointing fix for creating Sonar coverage reports

* #32 : Pin-pointing fix for creating Sonar coverage reports

* #32 : FIXED - creating Sonar coverage reports

* #32 : Fixed some Sonar issues

* #32 : Fixed checkstyle issue

* #32 : Cleanup of Swagger docs

* #32 : Fixed more Sonar issues

* #32 : Some inspiration

* #32 : Some inspiration

* #32 : REST API README updates

* #32 : REST API README section tweaks

* #32 : Increased unit test coverage. Gradle build needs fixing

* #32 : Started fixing Gradle build - more to come ;-/

* #32 : Removed JBehave stuff - moved to new repo.

* #32 : Disabled REST Controller tests temporarily in Gradle

For some reason, they've started failing in Gradle build.
Work fine in maven and IDE. Needs investigating...

* #32 : Dependency updates for maven build

* #32 : Dependency updates for maven build - missed a couple

* #32 : Dependency updates for Gradle build

* #32 : Fixed REST API tests in Gradle

... or rather they just started working again after the dependency
updates! ;-o

* #32 : Fixed Sonar issue

* #32 : Fixed broken test

* #32 : Include unit tests in checkstyle checks

* #32 : Added maven checkstyle reports

* #32 : Disabled OKCoin IT test - v1 API no longer available.

OKCoin adapter needs updating.

* #32 : Removed bdd profile

* #32 : Added missing boot starters to parent pom

* #32 : Gradle cleanup

* #32 : README updates

* #32 : Ignore REST API keystore

* #32 : Updated TLS docs

* #32 : Fixed more Sonar issues

* #32 : Fixed more Sonar issues

* #32 : Using correct naming convention

* #32 : Revert last change

* #32 : Fixed checkstyle issue
  • Loading branch information
gazbert authored Apr 3, 2020
1 parent 9cc7a23 commit 4688978
Show file tree
Hide file tree
Showing 99 changed files with 4,924 additions and 732 deletions.
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,7 @@ hs_err_pid*

# IT test output
bxbot-exchanges/out

# Ignore personal keystore used for REST API TLS
**/keystore*

Binary file modified .mvn/wrapper/maven-wrapper.jar
Binary file not shown.
3 changes: 2 additions & 1 deletion .mvn/wrapper/maven-wrapper.properties
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
distributionUrl=https://repo1.maven.org/maven2/org/apache/maven/apache-maven/3.6.1/apache-maven-3.6.1-bin.zip
distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.zip
wrapperUrl=https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar
2 changes: 2 additions & 0 deletions 3rd-party-licenses.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
* JavaMail - https://glassfish.java.net/public/CDDL+GPL_1_1.html
* Spring Boot - https://github.com/spring-projects/spring-boot/blob/master/LICENSE.txt
* Snake YAML - https://bitbucket.org/asomov/snakeyaml/src/default/LICENSE.txt
* Springfox - https://github.com/springfox/springfox/blob/master/LICENSE

## Build & Test

Expand All @@ -22,5 +23,6 @@
* SpotBugs - https://spotbugs.github.io/index.html
* JaCoCo - https://www.jacoco.org/license.html
* SonarQube - https://www.sonarqube.org/downloads/license
* JBehave - https://jbehave.org/license.html


119 changes: 108 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# BX-bot

[![Build Status](https://travis-ci.org/gazbert/bxbot.svg?branch=master)](https://travis-ci.org/gazbert/bxbot)
[![Build Status](https://travis-ci.org/gazbert/bxbot.svg?branch=bxbot-restapi)](https://travis-ci.org/gazbert/bxbot)
[![Sonarcloud Status](https://sonarcloud.io/api/project_badges/measure?project=gazbert_bxbot&metric=alert_status)](https://sonarcloud.io/dashboard?id=gazbert_bxbot)
[![Join the chat at https://gitter.im/BX-bot/Lobby](https://badges.gitter.im/BX-bot/Lobby.svg)](https://gitter.im/BX-bot/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

Expand Down Expand Up @@ -29,6 +29,7 @@ traded at the [spot price](http://www.investopedia.com/terms/s/spotprice.asp).
and released under the [MIT license](http://opensource.org/licenses/MIT).

## Architecture

![bxbot-core-architecture.png](./docs/bxbot-core-architecture.png)

- **Trading Engine** - the execution unit. It provides a framework for integrating Exchange Adapters and executing
Expand Down Expand Up @@ -96,7 +97,7 @@ and evaluate the bot, Docker is the way to go.
1. Install [Docker](https://docs.docker.com/engine/installation/) on the machine you want to run the bot.
1. Fetch the BX-bot image from [Docker Hub](https://hub.docker.com/r/gazbert/bxbot/): `docker pull gazbert/bxbot:x.x.x` -
replace `x.x.x` with the [Release](https://github.com/gazbert/bxbot/releases) version of the bot you want to run, e.g.
`docker pull gazbert/bxbot:0.12.1`
`docker pull gazbert/bxbot:1.0.0`
1. Run the Docker container: `docker container run --name bxbot-x.x.x -it gazbert/bxbot:x.x.x bash`
1. Change into the bot's directory: `cd bxbot*`
1. Configure the bot as described in step 4 of the previous [Maven](#maven) section.
Expand Down Expand Up @@ -135,6 +136,7 @@ Clone the repo locally (master branch).
bxbot-trading-api, bxbot-strategy-api, and bxbot-exchange-api modules.

## Issue & Change Management

Issues and new features are managed using the project [Issue Tracker](https://github.com/gazbert/bxbot/issues) -
submit bugs here.

Expand All @@ -152,6 +154,8 @@ The SNAPSHOT builds on master are active development builds, but the tests shoul
be deployable.

## User Guide
_"Change your opinions, keep to your principles; change your leaves, keep intact your roots."_ - Victor Hugo

### Configuration
The bot provides a simple plugin framework for:

Expand Down Expand Up @@ -390,8 +394,7 @@ All fields are mandatory unless stated otherwise.
Sample SMTP config for using a Gmail account is shown above - all elements within `smtpConfig` are mandatory.

### How do I write my own Trading Strategy?
_"Battle not with monsters, lest ye become a monster, and if you gaze into the abyss, the abyss gazes also into you."_ -
Friedrich Nietzsche
_"I was seldom able to see an opportunity until it had ceased to be one."_ - Mark Twain

The best place to start is with the
[`ExampleScalpingStrategy`](./bxbot-strategies/src/main/java/com/gazbert/bxbot/strategies/ExampleScalpingStrategy.java) -
Expand Down Expand Up @@ -455,9 +458,10 @@ BX-bot jar. You can also create your own jar for your strats, e.g. `my-strats.ja
runtime classpath - see the _[Installation Guide](#the-manual-way)_ for how to do this.

### How do I write my own Exchange Adapter?
_"I was seldom able to see an opportunity until it had ceased to be one."_ - Mark Twain
_"Battle not with monsters, lest ye become a monster, and if you gaze into the abyss, the abyss gazes also into you."_ -
Friedrich Nietzsche

The best place to start is with one of the inbuilt Exchange Adapters - see the latest
It's not easy, and can be frustrating at times, but a good place to start is with one of the inbuilt Exchange Adapters - see the latest
[`BitstampExchangeAdapter`](./bxbot-exchanges/src/main/java/com/gazbert/bxbot/exchanges/BitstampExchangeAdapter.java)
for example. There is also an Exchange Adapter specific channel on [Gitter](https://gitter.im/BX-bot/exchange-adapters).

Expand Down Expand Up @@ -531,11 +535,104 @@ at `info`. You can change this default logging configuration in the [`config/log
We recommend running at `info` level, as `debug` level logging will produce a *lot* of
output from the Exchange Adapters; it's very handy for debugging, but not so good for your disk space!

## Coming Soon... (Definitely Maybe)
The following features are in the pipeline:
### REST API
_"Enlightenment means taking full responsibility for your life."_ - William Blake

The bot has a REST API that allows you to remotely:

* View and update Engine, Exchange, Markets, Strategy, and Email Alerts config.
* View and download the log file.
* Restart the bot - this is necessary for any config changes to take effect.

It has role based access control
([RBAC](https://en.wikipedia.org/wiki/Role-based_access_control)): Users can view config and the
logs, but only administrators can update config and restart the bot.

- A REST API for administering the bot. It's being developed on the
[bxbot-restapi](https://github.com/gazbert/bxbot/tree/bxbot-restapi) branch.
- A UI built with [React](https://reactjs.org/) - it will consume the REST API.
It is secured using [JWT](https://jwt.io/) over [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security).

You can view the [Swagger](https://swagger.io/tools/swagger-ui/) docs at:
[http://localhost:8080/swagger-ui.html](http://localhost:8080/swagger-ui.html) once you've configured
and started the bot.

#### Configuration
The REST API is disabled by default to prevent accidental exposure of unencrypted traffic over public networks.
To enable it, you need to change the `server.port` in the
[./config/application.properties](./config/application.properties) from '-1' to the port you want
the bot to listen on - see the _[TLS](#tls)_ section below if you plan on accessing the REST API over a public network.

You _must_ change the `bxbot.restapi.jwt.secret` value in the
[./config/application.properties](./config/application.properties) before using the REST API over a public network.
This is the key that is used to sign your web tokens - the JWTs are signed using the HS512 algorithm.

Other interesting configuration in the [./config/application.properties](./config/application.properties) includes:

* `bxbot.restapi.maxLogfileLines` - the maximum number of lines to be returned in a view log file request.
(For a head request, the end of the file is truncated; for a tail request the start of the file is truncated).

* `bxbot.restapi.maxLogfileDownloadSize` - the maximum size of the logfile to download.
If the size of the logfile exceeds this limit, the end of the file will be truncated.

* `bxbot.restapi.jwt.expiration` - the expires time of the JWT. Set to 10 mins. Be sure you know the
risks if you decide to extend the expiry time.

#### Users
You _must_ change the `PASSWORD` values in the
[./bxbot-rest-api/src/main/resources/import.sql](./bxbot-rest-api/src/main/resources/import.sql)
before using the REST API over a public network - see instructions in the file on how to
[bcrypt](https://en.wikipedia.org/wiki/Bcrypt) your passwords.

2 users have been set up out of the box: `user` and `admin`. These users have `user` and `admin`
roles respectively. Passwords are the same as the usernames - remember to change these :-)

When the bot starts up, Spring Boot will load the `import.sql` file and store the users and their
access rights in its [H2](https://www.h2database.com/html/main.html) in-memory database.

#### Authentication
The REST API endpoints require a valid JWT to be passed in the `Authorization` header of any requests.

To obtain a JWT, your REST client needs to call the `/api/token` endpoint with a valid username/password
contained in the `import.sql` file. See the
[Authentication](http://localhost:8080/swagger-ui.html#/Authentication/getTokenUsingPOST)
Swagger docs for how to do this.

The returned JWT expires after 10 mins. Your client should call the `/api/refresh` endpoint with the
JWT before it expires in order to get a new one. Alternatively, you can re-authenticate using the
`/api/token` endpoint.

#### TLS
The REST API _must_ be configured to use TLS before accessing it over a public network.

You will need to
[create a keystore](https://docs.oracle.com/en/java/javase/11/tools/keytool.html) - the command to
create a [PKCS12](https://en.wikipedia.org/wiki/PKCS_12) self-signed certificate is shown below:

``` bash
keytool -genkeypair -alias rest-api-keystore -keyalg RSA -keysize 2048 -storetype PKCS12 -keystore keystore.p12 -validity 3650
```

The keystore must be on the app's classpath - you can put it in
the [./src/main/resources](./src/main/resources) and re-build the app to get up and running fast.
For a Production system, you'll want to replace the self-signed certificate with a
CA signed certificate.

The 'TLS Configuration' section in the [./config/application.properties](./config/application.properties)
file needs the following properties set:

``` properties
# Spring Boot profile for REST API.
# Must use https profile in Production environment.
spring.profiles.active=https
# SSL (TLS) configuration to secure the REST API.
# Must be enabled in Production environment.
server.port=8443
security.require-ssl=true
server.ssl.key-store=classpath:keystore.p12
server.ssl.key-store-password=secret
server.ssl.key-store-type=PKCS12
```

## Coming Soon... (Definitely Maybe)
A UI built with [React](https://reactjs.org/) - it will consume the REST API.

See the [Project Board](https://github.com/gazbert/bxbot/projects/2) for timescales and progress.
93 changes: 57 additions & 36 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -1,20 +1,18 @@
buildscript {

repositories {
jcenter()
mavenLocal()
mavenCentral()
}

dependencies {
classpath("io.spring.gradle:dependency-management-plugin:1.0.7.RELEASE")
classpath("org.springframework.boot:spring-boot-gradle-plugin:2.1.4.RELEASE")
classpath("io.spring.gradle:dependency-management-plugin:1.0.9.RELEASE")
classpath("org.springframework.boot:spring-boot-gradle-plugin:2.1.13.RELEASE")
classpath "gradle.plugin.com.github.spotbugs:spotbugs-gradle-plugin:2.0.0"
}
}

plugins {
id 'com.gradle.build-scan' version '2.2.1'
id "org.sonarqube" version "2.7.1"
id "org.sonarqube" version "2.8"
id 'jacoco'
}

Expand All @@ -25,52 +23,76 @@ buildScan {
}

ext.versions = [
springBootVersion: '2.1.4.RELEASE',
springCloudVersion: '2.1.1.RELEASE',
springTxVersion : '5.1.6.RELEASE'
springBootVersion : '2.1.13.RELEASE',
springCloudVersion : '2.1.5.RELEASE',
springTxVersion : '5.1.14.RELEASE',
springFoxVersion : '2.9.2',
hibernateVaildatorVersion: '6.1.2.Final',
jaxbVersion : '2.3.1',
javaxMailVersion : '1.6.2'
]

ext.libraries = [
spring_boot_starter : dependencies.create("org.springframework.boot:spring-boot-starter:" + ext.versions.springBootVersion) {
spring_boot_starter : dependencies.create("org.springframework.boot:spring-boot-starter:" + ext.versions.springBootVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
spring_boot_starter_log4j2 : dependencies.create("org.springframework.boot:spring-boot-starter-log4j2:" + ext.versions.springBootVersion),
spring_boot_starter_web : dependencies.create("org.springframework.boot:spring-boot-starter-web:" + ext.versions.springBootVersion) {
spring_boot_starter_log4j2 : dependencies.create("org.springframework.boot:spring-boot-starter-log4j2:" + ext.versions.springBootVersion),
spring_boot_starter_web : dependencies.create("org.springframework.boot:spring-boot-starter-web:" + ext.versions.springBootVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
spring_boot_starter_actuator: dependencies.create("org.springframework.boot:spring-boot-starter-actuator:" + ext.versions.springBootVersion) {
spring_boot_starter_data_jpa : dependencies.create("org.springframework.boot:spring-boot-starter-data-jpa:" + ext.versions.springBootVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
spring_boot_starter_security: dependencies.create("org.springframework.boot:spring-boot-starter-security:" + ext.versions.springBootVersion) {
spring_boot_starter_data_rest : dependencies.create("org.springframework.boot:spring-boot-starter-data-rest:" + ext.versions.springBootVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
spring_cloud_starter: dependencies.create("org.springframework.cloud:spring-cloud-starter:" + ext.versions.springCloudVersion) {
spring_boot_starter_actuator : dependencies.create("org.springframework.boot:spring-boot-starter-actuator:" + ext.versions.springBootVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
spring_tx : dependencies.create("org.springframework:spring-tx:" + ext.versions.springTxVersion),
google_guava : dependencies.create("com.google.guava:guava:27.1-jre"),
google_gson : dependencies.create("com.google.code.gson:gson:2.8.5"),
javax_mail_api : dependencies.create("javax.mail:javax.mail-api:1.6.2"),
javax_mail_sun : dependencies.create("com.sun.mail:javax.mail:1.6.2"),
javax_xml_api : dependencies.create("javax.xml.bind:jaxb-api:2.3.1"),
javax_xml_impl : dependencies.create("com.sun.xml.bind:jaxb-impl:2.3.1"),
snake_yaml : dependencies.create("org.yaml:snakeyaml:1.24"),

junit : dependencies.create("junit:junit:4.12"),
powermock_junit : dependencies.create("org.powermock:powermock-module-junit4:2.0.2"),
powermock_api_easymock : dependencies.create("org.powermock:powermock-api-easymock:2.0.2"),
easymock : dependencies.create("org.easymock:easymock:4.0.2"),

spring_boot_starter_test : dependencies.create("org.springframework.boot:spring-boot-starter-test:" + ext.versions.springBootVersion) {
spring_boot_starter_security : dependencies.create("org.springframework.boot:spring-boot-starter-security:" + ext.versions.springBootVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
awaitility : dependencies.create("org.awaitility:awaitility:3.1.6"),
spring_cloud_starter : dependencies.create("org.springframework.cloud:spring-cloud-starter:" + ext.versions.springCloudVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
spring_tx : dependencies.create("org.springframework:spring-tx:" + ext.versions.springTxVersion),
jjwt : dependencies.create("io.jsonwebtoken:jjwt:0.9.1"),
google_guava : dependencies.create("com.google.guava:guava:28.2-jre"),
google_gson : dependencies.create("com.google.code.gson:gson:2.8.6"),
h2 : dependencies.create("com.h2database:h2:1.4.199"),
javax_mail_api : dependencies.create("javax.mail:javax.mail-api:" + ext.versions.javaxMailVersion),
javax_mail_sun : dependencies.create("com.sun.mail:javax.mail:" + ext.versions.javaxMailVersion),
javax_xml_api : dependencies.create("javax.xml.bind:jaxb-api:" + ext.versions.jaxbVersion),
javax_xml_impl : dependencies.create("com.sun.xml.bind:jaxb-impl:" + ext.versions.jaxbVersion),
snake_yaml : dependencies.create("org.yaml:snakeyaml:1.26"),

springfox_swagger2 : dependencies.create("io.springfox:springfox-swagger2:" + ext.versions.springFoxVersion),
springfox_swagger_ui : dependencies.create("io.springfox:springfox-swagger-ui:" + ext.versions.springFoxVersion),
springfox_bean_validators : dependencies.create("io.springfox:springfox-bean-validators:" + ext.versions.springFoxVersion),
swagger_annotations : dependencies.create("io.swagger:swagger-annotations:1.6.0"),

validation_api : dependencies.create("javax.validation:validation-api:2.0.0.Final"),
hibernate_validator : dependencies.create("org.hibernate.validator:hibernate-validator:" + ext.versions.hibernateVaildatorVersion),
hibernate_validator_annotation_processor: dependencies.create("org.hibernate.validator:hibernate-validator-annotation-processor:" + ext.versions.hibernateVaildatorVersion),

junit : dependencies.create("junit:junit:4.13"),
powermock_junit : dependencies.create("org.powermock:powermock-module-junit4:2.0.6"),
powermock_api_easymock : dependencies.create("org.powermock:powermock-api-easymock:2.0.6"),
easymock : dependencies.create("org.easymock:easymock:4.2"),

spring_boot_starter_test : dependencies.create("org.springframework.boot:spring-boot-starter-test:" + ext.versions.springBootVersion) {
exclude module: "spring-boot-starter-logging"
force = true
},
spring_security_test : dependencies.create("org.springframework.security:spring-security-test:5.2.2.RELEASE"),
awaitility : dependencies.create("org.awaitility:awaitility:4.0.2")
]

allprojects {
Expand All @@ -82,7 +104,7 @@ allprojects {
apply plugin: "jacoco"

group = 'com.gazbert.bxbot'
version = '0.12.2-SNAPSHOT'
version = '1.0.0-SNAPSHOT'
}

subprojects {
Expand All @@ -93,8 +115,7 @@ subprojects {
targetCompatibility = 1.11

repositories {
mavenLocal()
maven { url "http://repo.maven.apache.org/maven2" }
mavenCentral()
}

test {
Expand Down Expand Up @@ -137,11 +158,11 @@ subprojects {
}

jacoco {
toolVersion = "0.8.3"
toolVersion = "0.8.5"
}
jacocoTestReport {
reports {
xml.enabled false
xml.enabled true
csv.enabled false
html.enabled true
}
Expand Down
Loading

0 comments on commit 4688978

Please sign in to comment.