I want to deploy a shared service that has no Azure infrastructure, j…

…ust a connection URI

Fixes microsoft#2485

.github/actions/devcontainer_run_command/action.yml

@@ -198,6 +198,7 @@ runs:
           -e TF_VAR_application_admin_client_id \
           -e TF_VAR_application_admin_client_secret \
           -e TF_VAR_arm_subscription_id="${{ inputs.ARM_SUBSCRIPTION_ID }}" \
+          -e SWAGGER_UI_CLIENT_ID="${{ inputs.TF_VAR_swagger_ui_client_id }}" \
           -e TF_VAR_swagger_ui_client_id \
           -e TF_VAR_core_address_space \
           -e TF_VAR_tre_address_space \

.github/workflows/build_validation_develop.yml

@@ -73,3 +73,5 @@ jobs:
           VALIDATE_BASH_EXEC: true
           VALIDATE_GITHUB_ACTIONS: true
           VALIDATE_DOCKERFILE_HADOLINT: true
+          VALIDATE_TSX: true
+          VALIDATE_TYPESCRIPT_ES: true

.github/workflows/deploy_tre_reusable.yml

@@ -679,11 +679,59 @@ jobs:
           TF_VAR_stateful_resources_locked:
             "${{ github.ref == 'refs/heads/main' && inputs.prRef == '' && true || false }}"
 
+  deploy_ui:
+    name: Deploy UI
+    runs-on: ubuntu-latest
+    needs: [deploy_tre]
+    environment: ${{ inputs.environmentName }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          persist-credentials: false
+          # if the following values are missing (i.e. not triggered via comment workflow)
+          # then the default checkout will apply
+          ref: ${{ inputs.prRef }}
+      - name: Deploy UI
+        uses: ./.github/actions/devcontainer_run_command
+        with:
+          COMMAND: "make build-and-deploy-ui"
+          ACTIONS_ACR_NAME: ${{ secrets.ACTIONS_ACR_NAME }}
+          ACTIONS_ACR_URI: ${{ secrets.ACTIONS_ACR_URI }}
+          ACTIONS_ACR_PASSWORD: ${{ secrets.ACTIONS_ACR_PASSWORD }}
+          ACTIONS_DEVCONTAINER_TAG: ${{ secrets.ACTIONS_DEVCONTAINER_TAG }}
+          ARM_TENANT_ID: "${{ secrets.ARM_TENANT_ID }}"
+          ARM_CLIENT_ID: "${{ secrets.ARM_CLIENT_ID }}"
+          ARM_CLIENT_SECRET: "${{ secrets.ARM_CLIENT_SECRET }}"
+          ARM_SUBSCRIPTION_ID: "${{ secrets.ARM_SUBSCRIPTION_ID }}"
+          API_CLIENT_ID: "${{ secrets.API_CLIENT_ID }}"
+          AAD_TENANT_ID: "${{ secrets.AAD_TENANT_ID }}"
+          TEST_APP_ID: "${{ secrets.TEST_APP_ID }}"
+          TEST_WORKSPACE_APP_ID: "${{ secrets.TEST_WORKSPACE_APP_ID }}"
+          TEST_WORKSPACE_APP_SECRET: "${{ secrets.TEST_WORKSPACE_APP_SECRET }}"
+          TEST_ACCOUNT_CLIENT_ID: "${{ secrets.TEST_ACCOUNT_CLIENT_ID }}"
+          TEST_ACCOUNT_CLIENT_SECRET: "${{ secrets.TEST_ACCOUNT_CLIENT_SECRET }}"
+          TRE_ID: "${{ secrets.TRE_ID }}"
+          LOCATION: ${{ secrets.LOCATION }}
+          ACR_NAME: ${{ secrets.ACR_NAME }}
+          TF_VAR_terraform_state_container_name: ${{ secrets.TF_STATE_CONTAINER }}
+          TF_VAR_mgmt_resource_group_name: ${{ secrets.MGMT_RESOURCE_GROUP }}
+          TF_VAR_mgmt_storage_account_name: ${{ secrets.STATE_STORAGE_ACCOUNT_NAME }}
+          TF_VAR_core_address_space: ${{ secrets.CORE_ADDRESS_SPACE }}
+          TF_VAR_tre_address_space: ${{ secrets.TRE_ADDRESS_SPACE }}
+          TF_VAR_swagger_ui_client_id: "${{ secrets.SWAGGER_UI_CLIENT_ID }}"
+          TF_VAR_api_client_id: "${{ secrets.API_CLIENT_ID }}"
+          TF_VAR_api_client_secret: "${{ secrets.API_CLIENT_SECRET }}"
+          TF_VAR_application_admin_client_id: "${{ secrets.APPLICATION_ADMIN_CLIENT_ID }}"
+          TF_VAR_application_admin_client_secret: "${{ secrets.APPLICATION_ADMIN_CLIENT_SECRET }}"
+          TF_VAR_stateful_resources_locked:
+            "${{ github.ref == 'refs/heads/main' && inputs.prRef == '' && true || false }}"
+
   e2e_tests_smoke:
     name: "Run E2E Tests (Smoke)"
     runs-on: ubuntu-latest
     environment: ${{ inputs.environmentName }}
-    needs: [deploy_shared_services, register_bundles]
+    needs: [deploy_shared_services, register_bundles, deploy_ui]
     timeout-minutes: 10
     steps:
       - name: Checkout

CHANGELOG.md

@@ -5,8 +5,26 @@
 
 **BREAKING CHANGES & MIGRATIONS**:
 
+* Firewall now blocks terraform/hasicorp domains ([#2590](https://github.com/microsoft/AzureTRE/pull/2590)). **Migration** is manual - update the templateVersion of `tre-shared-service-firewall` resource in Cosmos to `0.5.0`. Check the PR for more details.
+
+FEATURES:
+
 *
 
+ENHANCEMENTS:
+
+* Cancelling an Airlock request triggers deletion of the request container and files ([#2584](https://github.com/microsoft/AzureTRE/pull/2584))
+
+BUG FIXES:
+
+* Resource processor error on deploying user-resource: TypeError: 'NoneType' object is not iterable ([#2569](https://github.com/microsoft/AzureTRE/issues/2569))
+
+## 0.4.3 (September 12, 2022)
+
+**BREAKING CHANGES & MIGRATIONS**:
+
+* Remove support for Nexus V1 ([#2580](https://github.com/microsoft/AzureTRE/pull/2580)). Please migrate to the newer version as described [here](https://microsoft.github.io/AzureTRE/tre-admins/setup-instructions/configuring-shared-services/).
+
 FEATURES:
 
 *
@@ -20,12 +38,45 @@ ENHANCEMENTS:
 * Airlock requests contain a field with information about the files that were submitted ([#2504](https://github.com/microsoft/AzureTRE/pull/2504))
 * UI - Operations and notifications stability improvements ([[#2530](https://github.com/microsoft/AzureTRE/pull/2530))
 * UI - Initial implemetation of Workspace Airlock Request View ([#2512](https://github.com/microsoft/AzureTRE/pull/2512))
+* Add ability to automatically create Azure AD groups for each application role. Requires API version 0.4.30 or later ([#2532](https://github.com/microsoft/AzureTRE/pull/2532))
+* Add `is_expsed_externally` option to Azure ML Workspace Service ([#2548](https://github.com/microsoft/AzureTRE/pull2548))
+* Azure ML workspace service assigns Azure ML Data Scientist role to Workspace Researchers ([#2539](https://github.com/microsoft/AzureTRE/pull/2539))
+* UI is deployed by default ([#2554](https://github.com/microsoft/AzureTRE/pull/2554))
+* Remove manual/makefile option to install Gitea/Nexus ([#2573](https://github.com/microsoft/AzureTRE/pull/2573))
+* Exact Terraform provider versions in bundles ([#2579](https://github.com/microsoft/AzureTRE/pull/2579))
+* Stabilize E2E tests by issuing the access token prior using it, hence, reducing the change of expired token ([#2572](https://github.com/microsoft/AzureTRE/pull/2572))
 
 BUG FIXES:
 
 * API health check is also returned by accessing the root path at / ([#2469](https://github.com/microsoft/AzureTRE/pull/2469))
 * Temporary disable AppInsight's private endpoint in base workspace ([#2543](https://github.com/microsoft/AzureTRE/pull/2543))
 * Resource Processor execution optimization (`porter show`) for long-standing services ([#2542](https://github.com/microsoft/AzureTRE/pull/2542))
+* Move AML Compute deployment to use AzApi Terraform Provider {[#2555]((https://github.com/microsoft/AzureTRE/pull/2555))
+* Invalid token exceptions in the API app are catched, throwing 401 instead of 500 Internal server error ([#2572](https://github.com/microsoft/AzureTRE/pull/2572))
+
+COMPONENTS:
+
+| name | version |
+| ----- | ----- |
+| devops | 0.4.0 |
+| core | 0.4.23 |
+| tre-workspace-base | 0.3.28 |
+| tre-workspace-unrestricted | 0.1.9 |
+| tre-service-mlflow | 0.3.7 |
+| tre-service-innereye | 0.3.5 |
+| tre-workspace-service-gitea | 0.3.8 |
+| tre-workspace-service-mysql | 0.1.2 |
+| tre-service-guacamole-linuxvm | 0.4.14 |
+| tre-service-guacamole-windowsvm | 0.4.8 |
+| tre-service-guacamole | 0.4.5 |
+| tre-user-resource-aml-compute-instance | 0.3.2 |
+| tre-service-azureml | 0.4.8 |
+| tre-shared-service-cyclecloud | 0.2.6 |
+| tre-shared-service-gitea | 0.3.14 |
+| tre-shared-service-airlock-notifier | 0.1.2 |
+| tre-shared-service-certs | 0.1.3 |
+| tre-shared-service-sonatype-nexus | 2.1.6 |
+| tre-shared-service-firewall | 0.4.3 |
 
 ## 0.4.2 (August 23, 2022)
 
@@ -61,7 +112,6 @@ COMPONENTS:
 | ----- | ----- |
 | devops | 0.4.0 |
 | core | 0.4.18 |
-| tre-workspace-base | 0.3.19 |
 | tre-workspace-base | 0.3.25 |
 | tre-service-mlflow | 0.3.5 |
 | tre-service-innereye | 0.3.3 |

Makefile

@@ -11,18 +11,13 @@ LINTER_REGEX_INCLUDE?=all # regular expression used to specify which files to in
 target_title = @echo -e "\n\e[34m»»» 🧩 \e[96m$(1)\e[0m..."
 
 all: bootstrap mgmt-deploy images tre-deploy
-tre-deploy: deploy-core build-and-deploy-ui deploy-shared-services db-migrate show-core-output
+tre-deploy: deploy-core build-and-deploy-ui firewall-install db-migrate show-core-output
 
 images: build-and-push-api build-and-push-resource-processor build-and-push-airlock-processor
 build-and-push-api: build-api-image push-api-image
 build-and-push-resource-processor: build-resource-processor-vm-porter-image push-resource-processor-vm-porter-image
 build-and-push-airlock-processor: build-airlock-processor push-airlock-processor
 
-deploy-shared-services: firewall-install
-	. ${MAKEFILE_DIR}/devops/scripts/load_env.sh ./templates/core/.env \
-	&& if [ "$${DEPLOY_GITEA}" == "true" ]; then $(MAKE) gitea-install; fi \
-	&& if [ "$${DEPLOY_NEXUS}" == "true" ]; then $(MAKE) nexus-install; fi
-
 # to move your environment from the single 'core' deployment (which includes the firewall)
 # toward the shared services model, where it is split out - run the following make target before a tre-deploy
 # This will remove + import the resource state into a shared service
@@ -107,7 +102,10 @@ prepare-tf-state:
 deploy-core: tre-start
 	$(call target_title, "Deploying TRE") \
 	&& . ${MAKEFILE_DIR}/devops/scripts/check_dependencies.sh nodocker,env,auth \
-	&& if [[ "$${TF_LOG}" == "DEBUG" ]]; then echo "TF DEBUG set - output supressed - see tflogs container for log file" && cd ${MAKEFILE_DIR}/templates/core/terraform/ && ./deploy.sh 1>/dev/null 2>/dev/null; else cd ${MAKEFILE_DIR}/templates/core/terraform/ && ./deploy.sh; fi;
+	&& if [[ "$${TF_LOG}" == "DEBUG" ]]; \
+		then echo "TF DEBUG set - output supressed - see tflogs container for log file" && cd ${MAKEFILE_DIR}/templates/core/terraform/ \
+			&& ./deploy.sh 1>/dev/null 2>/dev/null; \
+		else cd ${MAKEFILE_DIR}/templates/core/terraform/ && ./deploy.sh; fi;
 
 letsencrypt:
 	$(call target_title, "Requesting LetsEncrypt SSL certificate") \
@@ -168,9 +166,11 @@ lint:
 		-e VALIDATE_BASH_EXEC=true \
 		-e VALIDATE_GITHUB_ACTIONS=true \
 		-e VALIDATE_DOCKERFILE_HADOLINT=true \
+		-e VALIDATE_TSX=true \
+    -e VALIDATE_TYPESCRIPT_ES=true \
 		-e FILTER_REGEX_INCLUDE=${LINTER_REGEX_INCLUDE} \
 		-v $${LOCAL_WORKSPACE_FOLDER}:/tmp/lint \
-		github/super-linter:slim-v4.9.5
+		github/super-linter:slim-v4.9.6
 
 lint-docs:
 	LINTER_REGEX_INCLUDE='./docs/.*\|./mkdocs.yml' $(MAKE) lint
@@ -247,7 +247,9 @@ bundle-register:
 	&& az acr login --name $${ACR_NAME}	\
 	&& . ${MAKEFILE_DIR}/devops/scripts/get_access_token.sh \
 	&& cd ${DIR} \
-	&& ${MAKEFILE_DIR}/devops/scripts/register_bundle_with_api.sh --acr-name "$${ACR_NAME}" --bundle-type "$${BUNDLE_TYPE}" --current --insecure --tre_url "$${TRE_URL:-https://$${TRE_ID}.$${LOCATION}.cloudapp.azure.com}" --verify --workspace-service-name "$${WORKSPACE_SERVICE_NAME}"
+	&& ${MAKEFILE_DIR}/devops/scripts/register_bundle_with_api.sh --acr-name "$${ACR_NAME}" --bundle-type "$${BUNDLE_TYPE}" \
+		--current --insecure --tre_url "$${TRE_URL:-https://$${TRE_ID}.$${LOCATION}.cloudapp.azure.com}" --verify \
+		--workspace-service-name "$${WORKSPACE_SERVICE_NAME}"
 
 workspace_bundle = $(MAKE) bundle-build bundle-publish bundle-register \
 	DIR="${MAKEFILE_DIR}/templates/workspaces/$(1)" BUNDLE_TYPE=workspace
@@ -273,18 +275,6 @@ firewall-install:
 	$(MAKE) bundle-build bundle-publish bundle-register deploy-shared-service \
 	DIR=${MAKEFILE_DIR}/templates/shared_services/firewall/ BUNDLE_TYPE=shared_service
 
-nexus-install:
-	$(MAKE) bundle-build bundle-publish bundle-register deploy-shared-service \
-	DIR="${MAKEFILE_DIR}/templates/shared_services/certs" BUNDLE_TYPE=shared_service PROPS="--domain_prefix nexus --cert_name nexus-ssl" \
-	&& $(MAKE) bundle-build bundle-publish bundle-register deploy-shared-service \
-	DIR=${MAKEFILE_DIR}/templates/shared_services/sonatype-nexus-vm/ BUNDLE_TYPE=shared_service PROPS="--ssl_cert_name nexus-ssl"
-
-gitea-install:
-	$(MAKE) bundle-build bundle-publish bundle-register deploy-shared-service DIR=${MAKEFILE_DIR}/templates/shared_services/gitea/ BUNDLE_TYPE=shared_service
-
-temp-do-upload:
-	$(MAKE) static-web-upload DIR=${MAKEFILE_DIR}/dummy
-
 static-web-upload:
 	$(call target_title, "Uploading to static website") \
 	&& . ${MAKEFILE_DIR}/devops/scripts/check_dependencies.sh nodocker,env,auth \
@@ -297,7 +287,7 @@ build-and-deploy-ui:
 	&& . ${MAKEFILE_DIR}/devops/scripts/check_dependencies.sh nodocker,env,auth \
 	&& pushd ${MAKEFILE_DIR}/templates/core/terraform/ > /dev/null && . ./outputs.sh && popd > /dev/null \
 	&& . ${MAKEFILE_DIR}/devops/scripts/load_env.sh ${MAKEFILE_DIR}/templates/core/private.env \
-	&& if [ "$${DEPLOY_UI}" == "true" ]; then ${MAKEFILE_DIR}/devops/scripts/build_deploy_ui.sh; else echo "UI Deploy skipped as DEPLOY_UI not true"; fi \
+	&& if [ "$${DEPLOY_UI}" != "false" ]; then ${MAKEFILE_DIR}/devops/scripts/build_deploy_ui.sh; else echo "UI Deploy skipped as DEPLOY_UI is false"; fi \
 
 prepare-for-e2e:
 	$(call workspace_bundle,base) \

airlock_processor/BlobCreatedTrigger/__init__.py

@@ -14,7 +14,7 @@
 
 def main(msg: func.ServiceBusMessage,
          stepResultEvent: func.Out[func.EventGridOutputEvent],
-         toDeleteEvent: func.Out[func.EventGridOutputEvent]):
+         dataDeletionEvent: func.Out[func.EventGridOutputEvent]):
 
     logging.info("Python ServiceBus topic trigger processed message - A new blob was created!.")
     body = msg.get_body().decode('utf-8')
@@ -74,13 +74,13 @@ def main(msg: func.ServiceBusMessage,
     logging.info(f"copied from history: {copied_from}")
 
     # signal that the container where we copied from can now be deleted
-    toDeleteEvent.set(
+    dataDeletionEvent.set(
         func.EventGridOutputEvent(
             id=str(uuid.uuid4()),
             data={"blob_to_delete": copied_from[-1]},  # last container in copied_from is the one we just copied from
             subject=request_id,
-            event_type="Airlock.ToDelete",
+            event_type="Airlock.DataDeletion",
             event_time=datetime.datetime.utcnow(),
-            data_version="1.0"
+            data_version=constants.DATA_DELETION_EVENT_DATA_VERSION
         )
     )

airlock_processor/BlobCreatedTrigger/function.json

@@ -19,9 +19,9 @@
     },
     {
       "type": "eventGrid",
-      "name": "toDeleteEvent",
-      "topicEndpointUri": "EVENT_GRID_TO_DELETE_TOPIC_URI_SETTING",
-      "topicKeySetting": "EVENT_GRID_TO_DELETE_TOPIC_KEY_SETTING",
+      "name": "dataDeletionEvent",
+      "topicEndpointUri": "EVENT_GRID_DATA_DELETION_TOPIC_URI_SETTING",
+      "topicKeySetting": "EVENT_GRID_DATA_DELETION_TOPIC_KEY_SETTING",
       "direction": "out"
     }
   ]

airlock_processor/ToDeleteTrigger/__init__.py → airlock_processor/DataDeletionTrigger/__init__.py

@@ -15,8 +15,13 @@ def delete_blob_and_container_if_last_blob(blob_url: str):
         credential=credential)
     container_client = blob_service_client.get_container_client(container_name)
 
+    if not blob_name:
+        logging.info(f'No specific blob specified, deleting the entire container: {container_name}')
+        container_client.delete_container()
+        return
+
     # If it's the only blob in the container, we need to delete the container too
-    # Check how many blobs are in the container (note: this exausts the generator)
+    # Check how many blobs are in the container (note: this exhausts the generator)
     blobs_num = sum(1 for _ in container_client.list_blobs())
     logging.info(f'Found {blobs_num} blobs in the container')
 
@@ -33,7 +38,7 @@ def delete_blob_and_container_if_last_blob(blob_url: str):
 
 def main(msg: func.ServiceBusMessage):
     body = msg.get_body().decode('utf-8')
-    logging.info(f'Python ServiceBus queue trigger processed mesage: {body}')
+    logging.info(f'Python ServiceBus queue trigger processed message: {body}')
     json_body = json.loads(body)
 
     blob_url = json_body["data"]["blob_to_delete"]

airlock_processor/ToDeleteTrigger/function.json → airlock_processor/DataDeletionTrigger/function.json

@@ -6,7 +6,7 @@
       "name": "msg",
       "type": "serviceBusTrigger",
       "direction": "in",
-      "queueName": "%AIRLOCK_TO_DELETE_QUEUE_NAME%",
+      "queueName": "%AIRLOCK_DATA_DELETION_QUEUE_NAME%",
       "connection": "SB_CONNECTION_STRING"
     }
   ]