Start working on new REST API. Refactor logic in domains. (#497)

* added 404 template

* added auth domain

* added embed file for frontend

* added base config and dependencies

* added basic new http server

* added separated server command

* updated go modules

* removed modd file

* Added shortcut to send internal server error response

* Added JWT support to Auth Domain

* Added JWT support to API

* docs: added comments to response struct

* naming

* inline returns

* updated dependencies

* production logger

* bookmarks endpoint

* reverted old views api path

* frontend for api v1

* proper 404 error (not working atm)

* use response

* removed 404 html

* server error handler

* login and basic auth

* adjusted session duration

* properly retrieve tags

* properly delete bookmark

* cleanup

* archiver domain

* debug routes

* bookmark routes

* expiration by parameter

* move to logrus

* logout

* frontend cache

* updated dependencies

* http: migrated to gin

* linted

* Added version command

* unit tests, docs

* response test utils and tests

* remove logout handler

* auth

* createtag

* improved http test utilities

* assert message equals

* Remove 1.19 from test matrix

* moved api to v1 folder

* docs: contribute docs

* updated makefile

* updated usage docs

* warn in server command

* updaed docs with shiori version command

* Updated documentation

* deps: update

.github/workflows/_test.yml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        go: ["1.19", "1.20"]
+        go: ["1.20"]
     services:
       postgres:
         image: postgres:15
@@ -33,10 +33,15 @@ jobs:
     name: Go ${{ matrix.go }} unit tests
     steps:
     - uses: actions/checkout@24cb9080177205b6e8c946b17badbe402adc938f # v3.4.0
+
     - name: Setup go
       uses: actions/setup-go@6edd4406fa81c3da01a34fa6f6343087c207a568 # v3.5.0
       with:
         go-version: ${{ matrix.go }}
+
+    - name: Set up gotestfmt
+      uses: gotesttools/gotestfmt-action@65f1d2228f06cc5e828b84597440fbd063d12ea2 # v2.1.0
+
     - uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # 3.3.1
       with:
         path: |
@@ -46,8 +51,10 @@ jobs:
         restore-keys: |
           golangci-lint.cache-{interval_number}-
           golangci-lint.cache-
-    - run: go test ./...
+
+    - run: make unittest
       env:
         SHIORI_TEST_PG_URL: "postgres://shiori:shiori@localhost:5432/shiori?sslmode=disable"
         SHIORI_TEST_MYSQL_URL: "shiori:shiori@(localhost:3306)/shiori"
+
     - run: CGO_ENABLED=0 go build -tags osusergo,netgo -ldflags="-s -w -X main.version=$(git describe --tags) -X main.date=$(date --iso-8601=seconds)"

.golangci.yml

@@ -0,0 +1,31 @@
+# Docs: https://golangci-lint.run/usage/configuration/#config-file
+
+run:
+  timeout: 5m
+  skip-dirs:
+    - internal/mocks
+
+issues:
+  max-issues-per-linter: 0
+  max-same-issues: 0
+
+linters-settings:
+  gofmt:
+    simplify: true
+  govet:
+    enable-all: true
+    disable:
+      - fieldalignment
+
+linters:
+  disable-all: true
+  enable:
+    - gofmt
+    - gosimple
+    # - govet # Re-enable when all shadow declarations are fixed
+    - ineffassign
+    - predeclared
+    - exportloopref
+    - staticcheck
+    - unconvert
+    - unused

.goreleaser.yaml

@@ -5,6 +5,7 @@ builds:
   - binary: shiori
     env:
       - CGO_ENABLED=0
+      - GIN_MODE=release
     tags:
       - netgo
       - osusergo

Makefile

@@ -0,0 +1,66 @@
+GO ?= $(shell command -v go 2> /dev/null)
+BASH ?= $(shell command -v bash 2> /dev/null)
+
+# Development
+SHIORI_DIR ?= dev-data
+
+# Testing
+GO_TEST_FLAGS ?= -v -race
+GOTESTFMT_FLAGS ?=
+
+# Build
+CGO_ENABLED ?= 0
+BUILD_TIME := $(shell date -u +%Y%m%d.%H%M%S)
+BUILD_HASH := $(shell git describe --tags)
+BUILD_TAGS ?= osusergo,netgo
+LDFLAGS += -s -w -X main.version=$(BUILD_HASH) -X main.date=$(BUILD_TIME)
+
+# Development
+GIN_MODE ?= debug
+SHIORI_DEVELOPMENT ?= true
+
+# Help documentatin à la https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html
+.PHONY: help
+help:
+	@cat Makefile | grep -v '\.PHONY' |  grep -v '\help:' | grep -B1 -E '^[a-zA-Z0-9_.-]+:.*' | sed -e "s/:.*//" | sed -e "s/^## //" |  grep -v '\-\-' | sed '1!G;h;$$!d' | awk 'NR%2{printf "\033[36m%-30s\033[0m",$$0;next;}1' | sort
+
+## Cleans up build artifacts
+.PHONY: clean
+clean:
+	rm -rf dist
+
+## Runs the legacy http API for local development
+.PHONY: serve
+serve:
+	SHIORI_DEVELOPMENT=$(SHIORI_DEVELOPMENT) SHIORI_DIR=$(SHIORI_DIR) go run main.go serve
+
+## Runs server for local development
+.PHONY: run-server
+run-server:
+	GIN_MODE=$(GIN_MODE) SHIORI_DEVELOPMENT=$(SHIORI_DEVELOPMENT) SHIORI_DIR=$(SHIORI_DIR) go run main.go server
+
+## Generate swagger docs
+.PHONY: swagger
+swagger:
+	swag init
+
+## Run linter
+.PHONY: lint
+lint:
+	golangci-lint run
+
+## Run unit tests
+.PHONY: unittest
+unittest:
+	GIN_MODE=$(GIN_MODE) GO_TEST_FLAGS="$(GO_TEST_FLAGS)" GOTESTFMT_FLAGS="$(GOTESTFMT_FLAGS)" $(BASH) -xe ./scripts/test.sh
+
+## Build binary
+.PHONY: build
+build: clean
+	GIN_MODE=$(GIN_MODE) goreleaser build --rm-dist --snapshot
+
+## Creates a coverage report
+.PHONY: coverage
+coverage:
+	$(GO) test $(GO_TEST_FLAGS) -coverprofile=coverage.txt ./...
+	$(GO) tool cover -html=coverage.txt

docs/API.md

@@ -1,5 +1,7 @@
 This is a brief explanation of Shiori's API. For more examples you can import this [collection](https://github.com/go-shiori/shiori/blob/master/docs/postman/shiori.postman_collection.json) in Postman.
 
+> ⚠️ **This is the documentation for the old API. This API is deprecated and will be removed in the future. Please refer and start migrating to the [API v1](./APIv1.md) instead.**
+
 <!-- TOC -->
 
 - [Auth](#auth)

docs/APIv1.md

@@ -0,0 +1,18 @@
+# API v1
+
+> ℹ️ **This is the documentation for the new API. This API is still in development and though the finished endpoints should not change please consider that breaking changes may occur once its properly released. If you are looking for the current API, please [see here](./API.md).**
+
+The new API is an ongoing effort to migrate the current API to a more modern and standard API.
+
+The main goals of this new API are:
+- Ease of development
+- Use of a [modern framework](https://gin-gonic.com)
+- Use of a [standard API specification](https://swagger.io/specification/)
+- Self-documented API using [Swag](https://github.com/swaggo/swag)
+- Improved authentication and sessions using [JWT](https://jwt.io)
+- Deduplicate code between the webserver and the API by refactoring the logic into domains
+- Improve testability by using interfaces and dependency injection
+
+The current status of this new API can be checked [here](https://github.com/go-shiori/shiori/issues/640).
+
+Since the API is self-docummented, you can check the API documentation by [running the server locally](./Contribute.md#running-the-server-locally) and visiting the [`/swagger/index.html` endpoint](http://localhost:8080/swagger/index.html).

docs/Contribute.md

@@ -1,5 +1,50 @@
 # Contribute
 
+1. [Running the server locally](#running-the-server-locally)
+2. [Updating the API documentation](#updating-the-api-documentation)
+3. [Lint the code](#lint-the-code)
+4. [Running tests](#running-tests)
+
+## Running the server locally
+
+To run the current development server with the defaults you can run the following command:
+
+```bash
+make serve
+```
+
+If you want to run the refactored server, you can run the following command:
+
+```bash
+make run-server
+```
+
+> **ℹ️ Note:** For more information into what the _refactored server_ means, please check this issue: https://github.com/go-shiori/shiori/issues/640
+
+## Updating the API documentation
+
+> **ℹ️ Note:** This only applies for the Rest API documentation under the `internal/http` folder, **not** the one under `internal/webserver`.
+
+If you make any changes to the Rest API endpoints, you need to update the swagger documentation. In order to do that, you need to have installed [swag](https://github.com/swaggo/swag).
+
+Then, run the following command:
+
+```bash
+make swagger
+```
+
+## Lint the code
+
+In order to lint the code, you need to have installed [golangci-lint](https://golangci-lint.run).
+
+After that, run the following command:
+
+```bash
+make lint
+```
+
+If any errors are found please fix them before submitting your PR.
+
 ## Running tests
 
 In order to run the test suite, you need to have running a local instance of MariaDB and PostgreSQL.
@@ -12,12 +57,12 @@ docker-compose up -d mariadb postgres
 After that, provide the `SHIORI_TEST_PG_URL` and `SHIORI_TEST_MYSQL_URL` environment variables with the connection string to the databases:
 
 ```
-SHIORI_TEST_PG_URL=postgres://postgres:postgres@127.0.0.1:5432/shiori?sslmode=disable
+SHIORI_TEST_PG_URL=postgres://shiori:shiori@127.0.0.1:5432/shiori?sslmode=disable
 SHIORI_TEST_MYSQL_URL=shiori:shiori@tcp(127.0.0.1:3306)/shiori
 ```
 
 Finally, run the tests with the following command:
 
 ```bash
-go test ./...
+make unittest
 ```

docs/Usage.md

@@ -63,7 +63,9 @@ Available Commands:
   pocket      Import bookmarks from Pocket's exported HTML file
   print       Print the saved bookmarks
   serve       Serve web interface for managing bookmarks
+  server      Run the Shiori webserver [alpha]
   update      Update the saved bookmarks
+  version     Output the shiori version
 
 Flags:
   -h, --help       help for shiori