Add support for DRAFT version 1.1 and improve validation

zimeon · Apr 26, 2022 · a5bf127 · a5bf127
2 parents 793deac + c9a27dc
commit a5bf127
Show file tree

Hide file tree

Showing 151 changed files with 940 additions and 355 deletions.
diff --git a/.github/workflows/action.yml b/.github/workflows/action.yml
@@ -0,0 +1,38 @@
+name: Tests for ocfl-py
+
+on: [push]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.6", "3.7", "3.8", "3.9"]
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: true
+          recursive: true
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install pytest pycodestyle pydocstyle pylint  # For testing and linting
+      - name: Lint with pycodestyle
+        run: |
+          pycodestyle --ignore=E501,W503 *.py ocfl tests
+      - name: Lint with pydocstyle
+        run: |
+          pydocstyle *.py ocfl tests
+      - name: Lint with pyliny
+        run: |
+          pylint --disable=unneeded-not,line-too-long,unnecessary-semicolon,trailg-whitespace,missing-final-newline,bad-indentation,multiple-statements,bare-except,missing-module-docstring,missing-class-docstring,missing-function-docstring,W0511,W0622,W0707,C0103,C0209,R0902,R0911,R0912,R0913,R0914,R0915,R1702 *.py ocfl tests
+      - name: Test with pytest
+        run: |
+          pytest
diff --git a/.travis.yml b/.travis.yml
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,13 +1,17 @@
 # ocfl-py changelog
 
-## 2021-??-?? v1.2.3
+## 2022-04-26 v1.3.0
 
+  * Add preliminary handling of OCFL v1.1 (https://github.com/zimeon/ocfl-py/issues/81)
   * Additional validation improvements:
     * Checks between version state in different version inventories
     * Check to see is extra directories look like version directories
     * Fix URI scheme syntax check (https://github.com/zimeon/ocfl-py/issues/75)
     * Check extensions directory in storage root (https://github.com/zimeon/ocfl-py/issues/76)
   * Use additional fixtures in https://github.com/OCFL/fixtures for tests
+  * Improve handling of inconsistent state between version inventories errors (E066, https://github.com/zimeon/ocfl-py/issues/85)
+  * Improve handling of fixity block issues (E056/E111, https://github.com/zimeon/ocfl-py/issues/79, https://github.com/zimeon/ocfl-py/issues/87)
+  * Running tests on Python 3.6, 3.7, 3.8, 3.9
 
 ## 2021-04-26 v1.2.2
 

diff --git a/README b/README
@@ -2,24 +2,27 @@
 ocfl-py - An OCFL implementation in Python
 ==========================================
 
-.. image:: https://travis-ci.com/zimeon/ocfl-py.svg?branch=main
-  :target: https://travis-ci.com/zimeon/ocfl-py
-.. image:: https://coveralls.io/repos/github/zimeon/ocfl-py/badge.svg?branch=main
-  :target: https://coveralls.io/github/zimeon/ocfl-py?branch=main
-
-Attempts to follow `OCFL specification v1.0
-<https://ocfl.io/1.0/spec/>`_, see `implementation status for errors and warnings
+`ocfl-py` is a Python reference implementation of the Oxford Common File Layout
+(OCFL). It also provides a number of command-line tools that may be useful for
+validating and manipulating OCFL Objects and OCFL Storage Roots. The code
+attempts to follow the current `Draft OCFL specification v1.1
+<https://ocfl.io/draft/spec/>`_ while retaining support for `v1.0
+<https://ocfl.io/1.0/spec/>`_. See `implementation status for errors and warnings
 <https://github.com/zimeon/ocfl-py/blob/main/docs/validation_status.md>`_ for
 details of validator support.
 
+See also `OCFL-Core
+<https://github.com/inveniosoftware/ocflcore>`_ which is another Python implementation
+of OCFL, designed to support the InvenioRDM repository, and `other OCFL implementations
+<https://github.com/OCFL/spec/wiki/Implementations>`_.
+
 Installing
 ----------
 
 This code requires Python 3.
 
-This code attempts to support the OCFL specification v1.0 and additional
-developments. To get the most up to date version check out the ``main``
-branch from github (or if you are reckless you can try the ``develop`` branch).
+This code attempts to support the OCFL specifications v1.1 draft and v1.0. To get
+the most up to date version check out the ``main`` branch from github.
 
 I hope to keep a fairly current version on `PyPI
 <https://pypi.org/project/ocfl-py/>`_, which can be installed or upgraded
@@ -33,7 +36,7 @@ The latest version is in the `main` branch on `github
 Use
 ---
 
-There should then be three scripts available:
+There should then be four command-line scripts available:
 
 - ``ocfl-validate.py`` - validate OCFL objects, OCFL storage roots or standalone OCFL inventory files
 - ``ocfl-object.py`` - build, manipulate, extract from or validate an OCFL object
@@ -60,7 +63,7 @@ for guidelines for contributing.
 Copyright and License
 ---------------------
 
-Copyright 2018--2021 Simeon Warner and `contributors
+Copyright 2018--2022 Simeon Warner and `contributors
 <https://github.com/zimeon/ocfl-py/graphs/contributors>`_.
 Provided under the MIT license, see `LICENSE.txt
 <https://github.com/zimeon/ocfl-py/blob/main/LICENSE.txt>`_.
diff --git a/docs/demo_build_spec_examples.md b/docs/demo_build_spec_examples.md
@@ -90,7 +90,7 @@ WARNING:ocfl.object:### Inventory for v1{
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-04-26T13:25:35.957146Z",
+      "created": "2022-04-26T14:47:17.394880Z",
       "state": {
         "7dcc352f96c56dc...c31": [
           "foo/bar.xml"
@@ -157,7 +157,7 @@ WARNING:ocfl.object:### Inventory for v2{
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-04-26T13:25:35.957146Z",
+      "created": "2022-04-26T14:47:17.394880Z",
       "state": {
         "7dcc352f96c56dc...c31": [
           "foo/bar.xml"
@@ -171,7 +171,7 @@ WARNING:ocfl.object:### Inventory for v2{
       }
     },
     "v2": {
-      "created": "2021-04-26T13:25:35.958734Z",
+      "created": "2022-04-26T14:47:17.397084Z",
       "state": {
         "4d27c86b026ff70...b53": [
           "foo/bar.xml"
@@ -236,7 +236,7 @@ WARNING:ocfl.object:### Inventory for v3{
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-04-26T13:25:35.957146Z",
+      "created": "2022-04-26T14:47:17.394880Z",
       "state": {
         "7dcc352f96c56dc...c31": [
           "foo/bar.xml"
@@ -250,7 +250,7 @@ WARNING:ocfl.object:### Inventory for v3{
       }
     },
     "v2": {
-      "created": "2021-04-26T13:25:35.958734Z",
+      "created": "2022-04-26T14:47:17.397084Z",
       "state": {
         "4d27c86b026ff70...b53": [
           "foo/bar.xml"
@@ -262,7 +262,7 @@ WARNING:ocfl.object:### Inventory for v3{
       }
     },
     "v3": {
-      "created": "2021-04-26T13:25:35.959946Z",
+      "created": "2022-04-26T14:47:17.399480Z",
       "state": {
         "4d27c86b026ff70...b53": [
           "foo/bar.xml"

diff --git a/docs/demo_ocfl_object_script.md b/docs/demo_ocfl_object_script.md
@@ -10,7 +10,7 @@ The `--version` argument will show version number and exit
 
 ```
 > python ocfl-object.py --version
-ocfl-object.py is part of ocfl-py version 1.2.2
+ocfl-object.py is part of ocfl-py version 1.3.0
 ```
 
 
@@ -34,7 +34,7 @@ WARNING:ocfl.object:### Inventory for v1{
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-04-26T13:25:37.573161Z",
+      "created": "2022-04-26T14:47:18.794059Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
@@ -65,7 +65,7 @@ WARNING:ocfl.object:### Inventory for v1{
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-04-26T13:25:38.047154Z",
+      "created": "2022-04-26T14:47:19.184724Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
@@ -90,15 +90,15 @@ WARNING:ocfl.object:### Inventory for v2{
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-04-26T13:25:38.047154Z",
+      "created": "2022-04-26T14:47:19.184724Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
         ]
       }
     },
     "v2": {
-      "created": "2021-04-26T13:25:38.047821Z",
+      "created": "2022-04-26T14:47:19.185642Z",
       "state": {
         "296e72b8fd5f7f0ac1473993600ae34953d5dab646f17e7b182b8648aff830d7bf01b56490777cb3e72b33fcc1ae520506badea1032252d1a55fd7362e269975": [
           "a_file.txt"
@@ -123,23 +123,23 @@ WARNING:ocfl.object:### Inventory for v3{
   "type": "https://ocfl.io/1.0/spec/#inventory",
   "versions": {
     "v1": {
-      "created": "2021-04-26T13:25:38.047154Z",
+      "created": "2022-04-26T14:47:19.184724Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"
         ]
       }
     },
     "v2": {
-      "created": "2021-04-26T13:25:38.047821Z",
+      "created": "2022-04-26T14:47:19.185642Z",
       "state": {
         "296e72b8fd5f7f0ac1473993600ae34953d5dab646f17e7b182b8648aff830d7bf01b56490777cb3e72b33fcc1ae520506badea1032252d1a55fd7362e269975": [
           "a_file.txt"
         ]
       }
     },
     "v3": {
-      "created": "2021-04-26T13:25:38.048884Z",
+      "created": "2022-04-26T14:47:19.186428Z",
       "state": {
         "43a43fe8a8a082d3b5343dfaf2fd0c8b8e370675b1f376e92e9994612c33ea255b11298269d72f797399ebb94edeefe53df243643676548f584fb8603ca53a0f": [
           "a_file.txt"

diff --git a/docs/demo_ocfl_sidecar_script.md b/docs/demo_ocfl_sidecar_script.md
@@ -10,7 +10,7 @@ The `--version` argument will show version number and exit
 
 ```
 > python ocfl-sidecar.py --version
-ocfl-sidecar.py is part of ocfl-py version 1.2.2
+ocfl-sidecar.py is part of ocfl-py version 1.3.0
 ```
 
 

diff --git a/docs/demo_ocfl_store_script.md b/docs/demo_ocfl_store_script.md
@@ -10,7 +10,7 @@ The `--version` argument will show version number and exit (but we still tave to
 
 ```
 > python ocfl-store.py --version --root=tmp/root --list
-ocfl-store.py is part of ocfl-py version 1.2.2
+ocfl-store.py is part of ocfl-py version 1.3.0
 ```
 
 

diff --git a/docs/demo_using_bagit_bags.md b/docs/demo_using_bagit_bags.md
@@ -161,8 +161,8 @@ Taking the newly created OCFL object `/tmp/obj` we can `--extract` the `v4` cont
 INFO:ocfl.object:Extracted v4 into tmp/extracted_v4
 INFO:bagit:Creating bag for directory tmp/extracted_v4
 INFO:bagit:Creating data directory
-INFO:bagit:Moving my_content to /privatetmp/extracted_v4/tmp32x9b1fz/my_content
-INFO:bagit:Moving /privatetmp/extracted_v4/tmp32x9b1fz to data
+INFO:bagit:Moving my_content to /privatetmp/extracted_v4/tmpmnmylt6l/my_content
+INFO:bagit:Moving /privatetmp/extracted_v4/tmpmnmylt6l to data
 INFO:bagit:Using 1 processes to generate manifests: sha512
 INFO:bagit:Generating manifest lines for file data/my_content/dracula.txt
 INFO:bagit:Generating manifest lines for file data/my_content/dunwich.txt
@@ -184,12 +184,12 @@ We note that the OCFL object had only one `content` file in `v4` but the extract
 diff -r tmp/extracted_v4/bag-info.txt tests/testdata/bags/uaa_v4/bag-info.txt
 1,2c1
 < Bag-Software-Agent: bagit.py v1.8.1 <https://github.com/LibraryOfCongress/bagit-python>
-< Bagging-Date: 2021-04-26
+< Bagging-Date: 2022-04-26
 ---
 > Bagging-Date: 2020-01-04
 diff -r tmp/extracted_v4/tagmanifest-sha512.txt tests/testdata/bags/uaa_v4/tagmanifest-sha512.txt
 2c2
-< 18c8191c83d80fdcf836098d3392665caf2f23cb1d1ec20ec8dd5af473b695245f0f991a579e045a04a736b8c2a6c963556412b3f510038f424322e882a8e881 bag-info.txt
+< 7dd1a757393da8b7513d149e185edd05598dfe48dd20ba5dde166f29eaf59bbf08c173718b3bc2976945c7a48aa52516c69a7d57a7c3b2b796247ad67e28f4bf bag-info.txt
 ---
 > 10624e6d45462def7af66d1a0d977606c7b073b01809c1d42258cfab5c34a275480943cbe78044416aee1f23822cc3762f92247b8f39b5c6ddc5ae32a8f94ce5 bag-info.txt
 ```
-Original file line number
+Diff line change
@@ Expand Up / @@ -10,7 +10,7 @@ The `--version` argument will show version number and exit @@
     ```
     > python ocfl-sidecar.py --version
-    ocfl-sidecar.py is part of ocfl-py version 1.2.2
+    ocfl-sidecar.py is part of ocfl-py version 1.3.0
     ```
@@ Expand Down @@