Skip to content

Commit

Permalink
Merge pull request #158 from tetherless-world/master
Browse files Browse the repository at this point in the history
New Release
  • Loading branch information
jpmccu authored Nov 26, 2019
2 parents bb83aad + 89cf0d1 commit c1d6ed7
Show file tree
Hide file tree
Showing 88 changed files with 2,033 additions and 663 deletions.
2 changes: 1 addition & 1 deletion .circleci/test-integration.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ if [ "$(ls results/results*.xml | wc -l)" -lt "1" ]; then
echo "Integration test results.xml does not exist, exiting abnormally"
exit 1
fi
if [ "$(grep -c 'failure ' results/results*.xml)" -ge 1 ]; then
if [ "$(grep -o 'failure ' results/results*.xml | wc -l)" -ge 1 ]; then
echo "Integration test results.xml has failures, exiting abnormally"
exit 1
fi
Expand Down
1 change: 1 addition & 0 deletions default_vocab.ttl
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,7 @@ owl:Ontology rdfs:label "Ontology";
whyis:hasView "ontology_view.html".

np:Nanopublication a owl:Class;
whyis:hasDescribe "nanopub_describe.json";
whyis:hasView "nanopublication_view.html".

# <search> a whyis:searchView.
Expand Down
21 changes: 21 additions & 0 deletions docker/compose/split/db-dev/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
version: '3'
services:
redis:
container_name: "redis"
image: "redis:3.0.6-alpine"
ports:
- "127.0.0.1:6379:6379"
volumes:
- ./../../../../../../data/redis:/data
blazegraph:
build:
args:
WHYIS_IMAGE_TAG: ${WHYIS_IMAGE_TAG:-latest}
context: ../../../..
dockerfile: docker/image/whyis-blazegraph/Dockerfile
container_name: "blazegraph"
image: tetherlessworld/whyis-blazegraph:${WHYIS_IMAGE_TAG:-latest}
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./../../../../../../data:/data
6 changes: 2 additions & 4 deletions docker/compose/split/db/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ services:
redis:
container_name: "redis"
image: "redis:3.0.6-alpine"
ports:
- "127.0.0.1:6379:6379"
expose:
- 6379
volumes:
- ./../../../../../../data/redis:/data
blazegraph:
Expand All @@ -15,7 +15,5 @@ services:
dockerfile: docker/image/whyis-blazegraph/Dockerfile
container_name: "blazegraph"
image: tetherlessworld/whyis-blazegraph:${WHYIS_IMAGE_TAG:-latest}
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./../../../../../../data:/data
2 changes: 2 additions & 0 deletions docker/image/whyis-blazegraph/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,5 @@ USER jetty
ENV JAVA_OPTIONS="-ea -Xmx4g -server -XX:+UseParallelOldGC -Djava.awt.headless=true -Dorg.eclipse.jetty.server.Request.maxFormContentSize=200000000"

WORKDIR /var/lib/jetty

EXPOSE 8080
2 changes: 2 additions & 0 deletions docker/image/whyis/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ RUN npm run build
# Deploy image from here
FROM tetherlessworld/whyis-deps:$WHYIS_IMAGE_TAG

RUN chmod 777 /data

# Apache configuration
COPY /puppet/files/etc/apache2/sites-available/000-default.conf /apache.conf
RUN sed '1d' /apache.conf > /etc/apache2/sites-available/000-default.conf
Expand Down
29 changes: 21 additions & 8 deletions docs/_data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,31 @@
- title: Home
url: /

- title: Basic Installation
url: /install
- title: Installation
sublinks:
- title: Quick Start With Docker
url: /tutorial

- title: Basic Installation
url: /install

- title: Docker Installation
url: /docker
- title: Docker Installation
url: /docker

- title: Customizing Whyis
sublinks:
- title: Configuring Whyis
url: /config

- title: Whyis Views
url: /views
- title: Command Line Reference
url: /commands

- title: Inference Agents
url: /inference
- title: Creating Whyis Views
url: /views

- title: Creating Inference Agents
url: /inference

- title: Use Cases
url: /usecases

Expand Down
207 changes: 207 additions & 0 deletions docs/commands.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@

# Using the Whyis command-line interface

To peform the following administrative tasks, if you're not running Whyis directly, you will need to connect to the device you are running it on.

If you're using a virtual machine with Vagrant:
```
vagrant ssh
```

If you're using a Docker container:
```
docker exec -it <container name> bash
```

Once you are in the server, you need to change to the **whyis** user, go to the whyis app directory, and activate the python virtual environment:

```
sudo su - whyis
cd /apps/whyis
source venv/bin/activate
```

To see a brief description of the Whyis subcommands:

```
python manage.py -?
```

To see more in-depth information about a subcommand and its options:

```
python manage.py <subcommand> -?
```

### Configure Whyis
**If you are using an existing Whyis knowledge graph, this step is not needed. Instead, go to the install instructions for the graph you wish to install.**

Whyis is built on the Flask web framework, and most of the Flask authentication options are available to configure in Whyis.
A configuration script will walk you through the configuration process and make a project directory for you.
Change the default values as needed. The SECRET_KEY and SECURITY_PASSWORD_SALT are randomly generated at runtime, so you shouldn't need to change those.

Run the following command:
```
python manage.py configure
```

This will run the configuration script, which will allow you to enter the desired configuration options from stdin, which will look like this:
```
project_name [My Knowledge Graph]:
project_short_description [An example knowledge graph configuration.]:
project_slug [my_knowledge_graph]:
location [/apps/my_knowledge_graph]:
author [J. Doe]:
email [[email protected]]:
linked_data_prefix [http://localhost]:
version [0.1]:
packages []:
SECRET_KEY [J00F5f80rGSbvpUo9oBFAtksmrd7ef8u]:
SECURITY_PASSWORD_SALT [JDyCyPu0KEu/fdJr4CbG65VhCtGugwCu]:
$
```

This will create a project skeleton for you at `location` (here, `/apps/my_knowledge_graph`). The files are:

* **config.py** - Main configuration file for Whyis.
* **vocab.ttl** - Vocabulary file for configuring custom Whyis views.
* **templates/** - Directory for storing Whyis view templates.
* **my_knowledge_graph/** - Project source directory. Put any python code in here.
* **agent.py** - An empty inference agent module.
* **static/** - Files that are served up at `{linked_data_prefix}/cdn/` as static files.
* **css/** - Project-specific CSS files.
* **my_knowledge_graph.css** - Default empty project-specific CSS file.
* **html/** - Project-specific static HTML files, like for Angular.js templates.
* **js/** - Project-specific javascript files.
* **my_knowledge_graph.js** - Default empty project-specific javascript file.
* **setup.py** - File for installation using pip.

Change directories into the project dir and install it into your virtualenv as follows. Be sure your virtualenv is activated first, by running `source venv/bin/activate` if you have not already done so.

```
cd /apps/my_knowledge_graph
pip install -e .
```

Restart apache and celeryd as a privileged user (not whyis) to have the configuration take effect:

```
sudo service apache2 restart
sudo service celeryd restart
```

### Add a User

Registration is available on the website for users, but it's easy to add a user to the knowledge graph from the command line.
Perform this task as the `whyis` user from the `/apps/whyis` directory.
Use `--roles=admin` to make the user an administrator.

```
python manage.py createuser -e <email> -p <password (can change later)> -f <first name> -l <last name> -u <user handle> --roles=admin
```

### Modify a User

You can change the roles a user has from the command line as well, but you'll need their user handle.
For instance, you can add a user to a role like this:

```
python manage.py updateuser -u <user handle> --add_roles=admin
```

You can remove them from a role like this:

```
python manage.py updateuser -u <user handle> --remove_roles=admin
```

Changing a password is also simple:

```
python manage.py updateuser -u <user handle> -p <new password>
```

### Run in development mode

Whyis can be run on a different port to enable debugging. You will see output from the log in the console and will be able to examine stack traces inside the browser.

```
python manage.py runserver -h 0.0.0.0
```

### Loading Knowledge

Knowledge can be added to Whyis using a command as well. This can be used to inject states that trigger larger-scale knowledge ingestion using [SETLr](https://github.com/tetherless-world/setlr/wiki/SETLr-Tutorial), or can simply add manually curated knowledge.
If the RDF format supports named graphs, and the graphs are nanopublications, the nanopublications will be added as-is.
If there are no explicit nanopublications, or if the RDF format is triples-only, each graph (including the default one), is treated as a separate nanopublication assertion.
The PublicationInfo will contain some minimal provenance about the load, and each assertion will be the graphs contained in the file.

```
python manage.py load -i <input file> -f <turtle|trig|json-ld|xml|nquads|nt|rdfa>
```

### Retire a nanopublication

To remove a nanopublication from the Whyis knowledge graph, use the following command:

```
python manage.py retire -n <nanopub URI>
```

This will also retire anything that *prov:wasDerivedFrom* the retired nanopublication, which may happen recursively.

Note that retired nanopublications are still accessible as linked data from a file archive that stores all nanopublications that have been published in the knowledge graph.

### Run tests on Whyis

To run the test suite of Whyis (unit tests, integration tests, API tests):
```
python manage.py test
```

This will run every test that is specified in the source code in a `.py` file whose name begins with `test_`.

To specify an alternate pattern for filenames, use the `--test` option. This should be specified as a filename without an extension. For example, to run the test suite specified in `tests/integration/test_login.py`:

```
python manage.py test --test test_login
```

Test files can also be specified using glob patterns. For example, to run the group of tests in `tests/api/view` whose filenames all end in `_json_view.py`, the command is:
```
python manage.py test --test test_*_json_view
```

### Run an inference agent

To run an inference agent on the Whyis knowledge graph:

```
python manage.py testagent -a <agent>
```

where `agent` is specified as a dotted Python path. Currently-supported agents include `agents.nlp.EntityExtractor`, `agents.nlp.EntityResolver`, `agents.nlp.IDFCalculator`, and `agents.nlp.HTML2Text`.



### Remove the currently-installed Whyis application

This command will remove the Whyis application that is currently installed, if there is one. Currently, having multiple apps installed into the same Whyis instance simultaneously is undefined behavior, so it is important to run this command before creating a new app.

```
python manage.py uninstallapp
```

### View a list of valid routes in the API

```
python manage.py listroutes
```

### Open a Python shell inside the application context

```
python manage.py shell
```

## [Next: Creating Whyis Views](views)
28 changes: 28 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Configuring Whyis

Whyis is configured within a generated app using the file `config.py`. This includes a number of options that can be added or modified.
The file contains a baseline config object with customizations for production, development, and testing.

## Enabling Anonymous Access

By defualt, Whyis requires users to be authenticated to access the knowledge graph.
Read-only anonymous access can be enabled by adding the following entry:

```
DEFAULT_ANONYMOUS_READ=True,
```

## Using BlazeGraph Bulk Loading

Whyis can use the Blazegraph custom bulk loader API if configured.
To use it with the default configuration, add the following entries to the config object:

```
knowledge_useBlazeGraphBulkLoad = True,
knowledge_bulkLoadEndpoint = 'http://localhost:8080/blazegraph/dataloader',
knowledge_BlazeGraphProperties = '/apps/whyis/knowledge.properties',
load_dir = '/data/loaded',
knowledge_bulkLoadNamespace = 'knowledge',
```

Loading

0 comments on commit c1d6ed7

Please sign in to comment.