Documentation_Erica.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**Exploring the data - json side**\n",
        "\n",
        "Starting from the csv and json files that we have as exemplar data, we decide to study more in details the configuration and the structure of IIIF \"International Image Interoperability Framework\" format.\n",
        "\n",
        "https://iiif.io/api/presentation/3.0/#1-introduction\n",
        "\n",
        "\n",
        "Here we can find more information related to the hierarchical structure of the json files -> collection, manifest and canvas -> with ids, type, label and items.\n",
        "\n",
        "In our case we have the file \"collection-1.json\" which contains ONE collection with label \"Works of Dante Alighieri\" which itself contains only ONE manifest with label \"Il Canzoniere\" with 239 canvases inside.\n",
        "\n",
        "The file \"collection-2.json\" contains ONE collection with label \"Fondo Giuseppe Raimondi\" which contains inside TWO manifests. The first one with label \"Raimondi, Giuseppe. Quaderno manoscritto, \"Caserma Scalo : 1930-1968\"\" (https://dl.ficlit.unibo.it/s/lib/item/19428) which have 16 canvases inside. The second one with label \"Raimondi, Giuseppe. Quaderno manoscritto, \"La vecchia centrale termica. Aprile 965\"\" (https://dl.ficlit.unibo.it/s/lib/item/19425) with other 16 canvases inside itself. The whole collection contains 32 canvases. \n",
        "\n",
        "One of the first things I analysed was the structure of json files, which look like dictionaries containing lists of dictionaries within them.\n",
        "Following this cascade of internal lists, one arrives at the canvases contained in each manifest. \n",
        "\n",
        "Still continuing my observation of the data, I went to look up the meaning of the attribute 'label' in the IIIF documentation to better understand what it meant.\n",
        "\n",
        "https://iiif.io/api/presentation/3.0/#language-of-property-values\n",
        "\n",
        "Inside the label is contained a dictionary which has as its key 'none' and as its value a list containing a string with a name or a description. The key, in this case 'none', is where the language in which the label is written should be indicated, following the BCP 47 language code for the language (https://www.rfc-editor.org/info/bcp47). So the language should be indicated with 'en', 'it', 'fr' etc. In our case \"none\" is used when the language is not reported or is unknown.\n"
      ],
      "metadata": {
        "id": "NslJc1kO0Kq2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Building the CollectionProcessor**\n",
        "\n",
        "It is son of class Processor.\n",
        "\n",
        "The CollectionProcessor has as method:\n",
        "\n",
        "uploadData: it takes in input the path of a JSON file containing collections (with manifests and canvases) and uploads them in the database. This method can be called everytime there is a need to upload collections in the database.\n",
        "\n"
      ],
      "metadata": {
        "id": "KFUV3bxQPpzJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CollectionProcessor(Processor):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "\n",
        "    def uploadData(self, path: str):\n",
        "        try: \n",
        "            base_url = \"https://github.com/n1kg0r/ds-project-dhdk/\"  \n",
        "            my_graph = Graph()\n",
        "\n",
        "            # define namespaces \n",
        "            nikCl = Namespace(\"https://github.com/n1kg0r/ds-project-dhdk/classes/\")\n",
        "            nikAttr = Namespace(\"https://github.com/n1kg0r/ds-project-dhdk/attributes/\")\n",
        "            nikRel = Namespace(\"https://github.com/n1kg0r/ds-project-dhdk/relations/\")\n",
        "            dc = Namespace(\"http://purl.org/dc/elements/1.1/\")\n",
        "\n",
        "            my_graph.bind(\"nikCl\", nikCl)\n",
        "            my_graph.bind(\"nikAttr\", nikAttr)\n",
        "            my_graph.bind(\"nikRel\", nikRel)\n",
        "            my_graph.bind(\"dc\", dc)\n",
        "            \n",
        "            with open(path, mode='r', encoding=\"utf-8\") as jsonfile:\n",
        "                json_object = load(jsonfile)\n",
        "            \n",
        "            #CREATE GRAPH\n",
        "            if type(json_object) is list: #CONTROLLARE!!!\n",
        "                for collection in json_object:\n",
        "                    create_Graph(collection, base_url, my_graph)\n",
        "            \n",
        "            else:\n",
        "                create_Graph(json_object, base_url, my_graph)\n",
        "            \n",
        "                    \n",
        "            #DB UPTDATE\n",
        "            store = SPARQLUpdateStore()\n",
        "\n",
        "            endpoint = self.getDbPathOrUrl()\n",
        "\n",
        "            store.open((endpoint, endpoint))\n",
        "\n",
        "            for triple in my_graph.triples((None, None, None)):\n",
        "                store.add(triple)\n",
        "            store.close()\n",
        "            \n",
        "            with open('Graph_db.ttl', mode='a', encoding='utf-8') as f:\n",
        "                f.write(my_graph.serialize(format='turtle'))\n",
        "\n",
        "            return True\n",
        "        \n",
        "        except Exception as e:\n",
        "            print(str(e))\n",
        "            return False\n"
      ],
      "metadata": {
        "id": "SWRrLHaOKpmu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I created the method with a \"try and except\" function that returns True if the process of uploading data ends well, or False if it ends with an error + it will write also the error itself to clarify why the method is not working properly.\n",
        "\n",
        "I used rdflib to create the RDF graph and RDF statements.\n",
        "From this library I used the Classes Graph and Namespace in CollectionProcessor, the I will use other classes and properties in CreateGraph, a side function I've created for populating the Graph.\n",
        "\n",
        "Class Graph() -> https://rdflib.readthedocs.io/en/stable/intro_to_parsing.html -> to create a new (set as empty) RDF Graph\n",
        "\n",
        "Class Namespace -> \n",
        "https://rdflib.readthedocs.io/en/stable/apidocs/rdflib.namespace.html#rdflib.namespace.Namespace -> In RDF, namespaces are used to provide an abbreviation or prefix for URIs (Uniform Resource Identifiers) that identify resources in the RDF graph. By using a namespace, a prefix or keyword can be defined to represent the complete URI. This simplifies writing and reading RDF graphs, as URIs can be quite long and complex.\n",
        "\n",
        "I created four different Namespace. One for Classes, one for Attributes, one for Relations, and one for DublinCore (used for its property \"identifier\").\n",
        "As you can see, I choose to define my personal URIs for almost everything. The only property I reuse from an already existing standard is \"identifier\" from Dublin Core -> http://purl.org/dc/elements/1.1/identifier\n",
        "This choice was made in order to remain faithful to the lexicon required by the UML.\n",
        "\n",
        "Before taking this decision I also read the documentation of the ontology proposed by IIIF, the Shared Canvas Data Model ->\n",
        "https://iiif.io/api/model/shared-canvas/1.0/#Namespaces\n",
        "Here there are some classes and properties useful for the description of a IIIF document as sc:Canvas or sc:Manifest or the properties \"sc:hasSequences\" or \"sc:hasCanvases\" where \"sc:hasSequences\" is used to represent the set of sequences within a manifest, while \"sc:hasCanvases\" is used to represent the set of canvases within a sequence or manifest.\n",
        "Studying the documentation of this IIIF-specific ontology, it did not seem suitable for our project as there were no specific classes for representing \"Collection\" and the relationship between Collection and Manifest or Collection and Canvas was not considered. I therefore preferred to create custom URIs to be more free in the creation of the Graph.\n",
        "\n",
        "Then I use the binding method (.bing) to bound the prefixes in the Graph -> https://rdflib.readthedocs.io/en/stable/namespaces_and_bindings.html\n",
        "\n",
        "Then I open the json file and I read it.\n",
        "\n",
        "Then I created a if-else statement that allow me to check whether there is a list in the json file or not.\n",
        "This is because, as mentioned above, in our starting json files there are dictionaries each expressing a single collection, but I have also evaluated the possible case in which in a json file there are several collections (expressed through dictionaries) contained one by one within a list. \n",
        "With this if-else statement it is possible to check for the presence of a list and, if it's True, iterate over the list and create a triple for each collection.\n",
        "\n",
        "Then I import the class SPARQLUpdateStore from rdflib.plugins.stores.sparqlstore to upload the graph persistently on our triplestore. We specify the endpoint of our triplestore (using method from Processor) and we store there all out triples usign a for loop. Then we close the connection.\n",
        "\n",
        "After that, I added \"with open\" function to store the rdf graph as a turtle file. Mode is equal to \"a\" that stands for \"append\" cause I wanted to store the update of the graph in the same file.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "mW__OQ14RFtY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Create Graph Function**"
      ],
      "metadata": {
        "id": "D91G895EFmT3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from rdflib import Graph, URIRef, RDF, Literal, Namespace\n",
        "\n",
        "def create_Graph(json_object:dict, base_url, my_graph:Graph):\n",
        "    \n",
        "    # create an internal id for the collections using an external counter\n",
        "    # .strip is for removing eventually white space\n",
        "    with open('collection_counter.txt', 'r', encoding='utf-8') as a:\n",
        "        collection_counter = int(a.read().strip())\n",
        "\n",
        "    # create an internal id for the manifest using an external counter\n",
        "    with open('manifest_counter.txt', 'r', encoding='utf-8') as b:\n",
        "        manifest_counter = int(b.read().strip())\n",
        "\n",
        "    # create an internal id for the canvases using an external counter\n",
        "    with open('canvas_counter.txt', 'r', encoding='utf-8') as c:\n",
        "        canvas_counter = int(c.read().strip())\n",
        "\n",
        "    # define namespaces \n",
        "    nikCl = Namespace(\"https://github.com/n1kg0r/ds-project-dhdk/classes/\")\n",
        "    nikAttr = Namespace(\"https://github.com/n1kg0r/ds-project-dhdk/attributes/\")\n",
        "    nikRel = Namespace(\"https://github.com/n1kg0r/ds-project-dhdk/relations/\")\n",
        "    dc = Namespace(\"http://purl.org/dc/elements/1.1/\")\n",
        "\n",
        "    # classes\n",
        "    Collection = nikCl[\"Collection\"]\n",
        "    Manifest = nikCl[\"Manifest\"]\n",
        "    Canvas = nikCl[\"Canvas\"]\n",
        "\n",
        "    # attributes related to classes\n",
        "    label = nikAttr[\"label\"]\n",
        "\n",
        "    # relations among classes\n",
        "    items = nikRel[\"items\"]\n",
        "    has_id = dc[\"identifier\"]\n",
        "\n",
        "    # create a variable for the id\n",
        "    collection_id = json_object['id'] \n",
        "\n",
        "    # create an internal id with an external counter\n",
        "    collection_counter += 1\n",
        "    collection_IntId = json_object['type'] + f\"_{collection_counter}\"\n",
        "    Coll_internalId = URIRef(base_url + collection_IntId)\n",
        "\n",
        "    # create a list from the dictionary of label and catch the value of the key \"none\" (language) in a variable\n",
        "    label_list = list(json_object['label'].values())  \n",
        "    value_label = label_list[0][0]\n",
        "\n",
        "    # remove the square brackets from the label value\n",
        "    value_label = str(value_label)\n",
        "\n",
        "    # create the graph with the triples\n",
        "    my_graph.add((Coll_internalId, has_id, Literal(collection_id)))\n",
        "    my_graph.add((Coll_internalId, RDF.type, Collection))\n",
        "    my_graph.add((Coll_internalId, label, Literal(str(value_label))))\n",
        "\n",
        "    \n",
        "    # second step is entering the collection items list (enter the manifest) -> entering a list of dictionaries\n",
        "    # here i take the id and I store it in a variable\n",
        "    for manifest in json_object[\"items\"]:\n",
        "        manifest_id = manifest['id']\n",
        "\n",
        "        # here i raise the counter for the manifest internal id\n",
        "        manifest_counter += 1\n",
        "        manifest_IntId = manifest['type'] + f\"_{manifest_counter}\" \n",
        "        Man_internalId = URIRef(base_url + manifest_IntId)\n",
        "\n",
        "        #add the \"has Item\" to connect Collection to Manifest\n",
        "        my_graph.add((Coll_internalId, items, Man_internalId))\n",
        "\n",
        "        # create a list from the dictionary of label and catch the value of the key \"none\" (language) in a variable\n",
        "        M_label_list = list(manifest['label'].values())  \n",
        "        M_value_label = M_label_list[0][0]\n",
        "\n",
        "        # remove the square brackets from the label value\n",
        "        M_value_label = str(M_value_label)\n",
        "        \n",
        "\n",
        "        # create the graph with the triples\n",
        "        my_graph.add((Man_internalId, has_id, Literal(manifest_id)))\n",
        "        my_graph.add((Man_internalId, RDF.type, Manifest))\n",
        "        my_graph.add((Man_internalId, label, Literal(str(M_value_label))))\n",
        "\n",
        "        # third step is entering the manifest items list (enter the canvases) -> entering a list of dictionaries\n",
        "        # here i take the id and I store it in a variable\n",
        "        for canvas in manifest[\"items\"]:\n",
        "            canvas_id = canvas['id']\n",
        "\n",
        "            # here i raise the counter for the manifest internal id\n",
        "            canvas_counter += 1\n",
        "            canvas_IntId = canvas['type'] + f\"_{canvas_counter}\" \n",
        "            Can_internalId = URIRef(base_url + canvas_IntId)\n",
        "\n",
        "            #add the \"has Item\" to connect Collection to Manifest\n",
        "            my_graph.add((Man_internalId, items, Can_internalId))\n",
        "\n",
        "            # create a list from the dictionary of label and catch the value of the key \"none\" (language) in a variable\n",
        "            C_label_list = list(canvas['label'].values())  \n",
        "            C_value_label = C_label_list[0][0]\n",
        "\n",
        "            # remove the square brackets from the label value\n",
        "            C_value_label = str(C_value_label)\n",
        "\n",
        "\n",
        "            # create the graph with the triples\n",
        "            my_graph.add((Can_internalId, has_id, Literal(canvas_id)))\n",
        "            my_graph.add((Can_internalId, RDF.type, Canvas))\n",
        "            my_graph.add((Can_internalId, label, Literal(str(C_value_label))))\n",
        "\n",
        "\n",
        "    #upload the counters text file\n",
        "    with open('collection_counter.txt', 'w') as a:\n",
        "        a.write(str(collection_counter))\n",
        "\n",
        "    with open('manifest_counter.txt', 'w') as b:\n",
        "        b.write(str(manifest_counter))\n",
        "\n",
        "    with open('canvas_counter.txt', 'w') as c:\n",
        "        c.write(str(canvas_counter))"
      ],
      "metadata": {
        "id": "i_FNFmFDir-4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I've created this separated function to build the structure of the graph.\n",
        "First thing that this function does is opening three text files that I've called \"counters\", one for collections, one for manifests and one for canvases. \n",
        "The I recall the namespaces and I create the specific classes, attributes and relations that I neeeded. \n",
        "\n",
        "I raise the collection counter of 1 every time I use the create_graph function, entering the first dictionary. \n",
        "\n",
        "Here I use also URIRef, Literal and RDF from rdflib.\n",
        "\n",
        "With two for loop I entered first the Manifest and then the Canvas lists of dictionary.\n",
        "\n",
        "At the end of the function I update the text files of the counters. "
      ],
      "metadata": {
        "id": "dGUElSGmG3nS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Cleaning functions**\n",
        "\n",
        "I also developed two extra-functions to clean the counters and to clean the blazegraph database. \n",
        "\n",
        "They were useful during the testing phase of our code and represent a very easy way to start from zero with the creation of the graph. "
      ],
      "metadata": {
        "id": "pMEobIYgKugl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "def clear_counter():\n",
        "    with open('collection_counter.txt', 'w') as a:\n",
        "        a.write('0')\n",
        "\n",
        "    with open('manifest_counter.txt', 'w') as b:\n",
        "        b.write('0')\n",
        "\n",
        "    with open('canvas_counter.txt', 'w') as c:\n",
        "        c.write('0')\n",
        "\n",
        "def clear_blazegraph_database(db_path):\n",
        "    url = f\"{db_path}/sparql\"\n",
        "    query = \"DELETE WHERE { ?s ?p ?o }\"\n",
        "\n",
        "    response = requests.post(url, data=query, headers={\"Content-Type\": \"application/sparql-update\"})\n",
        "\n",
        "    if response.status_code == 200:\n",
        "        print(\"Database cleared successfully.\")\n",
        "    else:\n",
        "        print(\"Failed to clear the database.\")\n",
        "\n",
        "# RUN\n",
        "blazegraph_path = \"http://127.0.0.1:9999/blazegraph\"\n",
        "clear_blazegraph_database(blazegraph_path)\n",
        "clear_counter()"
      ],
      "metadata": {
        "id": "fdi5eK3FLL50"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first function write \"0\" inside all the three counters text files.\n",
        "\n",
        "The second one takes in input the path of the database that we want to clear and it make a SPARQL Update Query via HTTP Post request.\n",
        "The response from the server is stored in the response variable. \n",
        "If the status of the response is equal to 200 (indicating a successful request), it prints \"Database cleared successfully.\" Otherwise, it prints \"Failed to clear the database.\""
      ],
      "metadata": {
        "id": "zPGyu-V6LnNc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**TriplestoreQueryProcessor**"
      ],
      "metadata": {
        "id": "GYBCR5fpLQxE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class TriplestoreQueryProcessor(QueryProcessor):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "\n",
        "    def getAllCanvases(self):\n",
        "\n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_canvases = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/> \n",
        "\n",
        "        SELECT ?canvas ?id ?label\n",
        "        WHERE {\n",
        "            ?canvas a nikCl:Canvas;\n",
        "            dc:identifier ?id;\n",
        "            nikAttr:label ?label.\n",
        "        }\n",
        "        \"\"\"\n",
        "\n",
        "        df_sparql_getAllCanvases = get(endpoint, query_canvases, True)\n",
        "        return df_sparql_getAllCanvases\n",
        "\n",
        "    def getAllCollections(self):\n",
        "\n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_collections = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/> \n",
        "\n",
        "        SELECT ?collection ?id ?label\n",
        "        WHERE {\n",
        "           ?collection a nikCl:Collection;\n",
        "           dc:identifier ?id;\n",
        "           nikAttr:label ?label .\n",
        "        }\n",
        "        \"\"\"\n",
        "\n",
        "        df_sparql_getAllCollections = get(endpoint, query_collections, True)\n",
        "        return df_sparql_getAllCollections\n",
        "\n",
        "    def getAllManifests(self):\n",
        "\n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_manifest = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/>\n",
        "\n",
        "        SELECT ?manifest ?id ?label\n",
        "        WHERE {\n",
        "           ?manifest a nikCl:Manifest ;\n",
        "           dc:identifier ?id ;\n",
        "           nikAttr:label ?label .\n",
        "        }\n",
        "        \"\"\"\n",
        "\n",
        "        df_sparql_getAllManifest = get(endpoint, query_manifest, True)\n",
        "        return df_sparql_getAllManifest\n",
        "\n",
        "    def getCanvasesInCollection(self, collectionId: str):\n",
        "\n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_canInCol = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/>\n",
        "\n",
        "        SELECT ?canvas ?id ?label \n",
        "        WHERE {\n",
        "            ?collection a nikCl:Collection ;\n",
        "            dc:identifier \"%s\" ;\n",
        "            nikRel:items ?manifest .\n",
        "            ?manifest a nikCl:Manifest ;\n",
        "            nikRel:items ?canvas .\n",
        "            ?canvas a nikCl:Canvas ;\n",
        "            dc:identifier ?id ;\n",
        "            nikAttr:label ?label .\n",
        "        }\n",
        "        \"\"\" % collectionId\n",
        "\n",
        "        df_sparql_getCanvasesInCollection = get(endpoint, query_canInCol, True)\n",
        "        return df_sparql_getCanvasesInCollection\n",
        "\n",
        "    def getCanvasesInManifest(self, manifestId: str):\n",
        "\n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_canInMan = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/> \n",
        "\n",
        "        SELECT ?canvas ?id ?label\n",
        "        WHERE {\n",
        "            ?manifest a nikCl:Manifest ;\n",
        "            dc:identifier \"%s\" ;\n",
        "            nikRel:items ?canvas .\n",
        "            ?canvas a nikCl:Canvas ;\n",
        "            dc:identifier ?id ;\n",
        "            nikAttr:label ?label .\n",
        "        }\n",
        "        \"\"\" % manifestId\n",
        "\n",
        "        df_sparql_getCanvasesInManifest = get(endpoint, query_canInMan, True)\n",
        "        return df_sparql_getCanvasesInManifest\n",
        "\n",
        "\n",
        "    def getManifestsInCollection(self, collectionId: str):\n",
        "\n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_manInCol = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/>  \n",
        "\n",
        "        SELECT ?manifest ?id ?label\n",
        "        WHERE {\n",
        "            ?collection a nikCl:Collection ;\n",
        "            dc:identifier \"%s\" ;\n",
        "            nikRel:items ?manifest .\n",
        "            ?manifest a nikCl:Manifest ;\n",
        "            dc:identifier ?id ;\n",
        "            nikAttr:label ?label .\n",
        "        }\n",
        "        \"\"\" % collectionId\n",
        "\n",
        "        df_sparql_getManifestInCollection = get(endpoint, query_manInCol, True)\n",
        "        return df_sparql_getManifestInCollection\n",
        "    \n",
        "\n",
        "    def getEntitiesWithLabel(self, label: str): \n",
        "            \n",
        "\n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_entityLabel = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/>\n",
        "\n",
        "        SELECT ?entity ?type ?label ?id\n",
        "        WHERE {\n",
        "            ?entity nikAttr:label \"%s\" ;\n",
        "            a ?type ;\n",
        "            nikAttr:label ?label ;\n",
        "            dc:identifier ?id .\n",
        "        }\n",
        "        \"\"\" % remove_special_chars(label)\n",
        "\n",
        "        df_sparql_getEntitiesWithLabel = get(endpoint, query_entityLabel, True)\n",
        "        return df_sparql_getEntitiesWithLabel\n",
        "    \n",
        "\n",
        "    def getEntitiesWithCanvas(self, canvasId: str): \n",
        "            \n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_entityCanvas = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/> \n",
        "\n",
        "        SELECT ?id ?label ?type\n",
        "        WHERE {\n",
        "            ?entity dc:identifier \"%s\" ;\n",
        "            dc:identifier ?id ;\n",
        "            nikAttr:label ?label ;\n",
        "            a ?type .\n",
        "        }\n",
        "        \"\"\" % canvasId\n",
        "\n",
        "        df_sparql_getEntitiesWithCanvas = get(endpoint, query_entityCanvas, True)\n",
        "        return df_sparql_getEntitiesWithCanvas\n",
        "    \n",
        "    def getEntitiesWithId(self, id: str): \n",
        "            \n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_entityId = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/> \n",
        "\n",
        "        SELECT ?id ?label ?type\n",
        "        WHERE {\n",
        "            ?entity dc:identifier \"%s\" ;\n",
        "            dc:identifier ?id ;\n",
        "            nikAttr:label ?label ;\n",
        "            a ?type .\n",
        "        }\n",
        "        \"\"\" % id\n",
        "\n",
        "        df_sparql_getEntitiesWithId = get(endpoint, query_entityId, True)\n",
        "        return df_sparql_getEntitiesWithId\n",
        "    \n",
        "\n",
        "    def getAllEntities(self): \n",
        "            \n",
        "        endpoint = self.getDbPathOrUrl()\n",
        "        query_AllEntities = \"\"\"\n",
        "        PREFIX dc: <http://purl.org/dc/elements/1.1/> \n",
        "        PREFIX nikAttr: <https://github.com/n1kg0r/ds-project-dhdk/attributes/> \n",
        "        PREFIX nikCl: <https://github.com/n1kg0r/ds-project-dhdk/classes/> \n",
        "        PREFIX nikRel: <https://github.com/n1kg0r/ds-project-dhdk/relations/> \n",
        "\n",
        "        SELECT ?entity ?id ?label ?type\n",
        "        WHERE {\n",
        "            ?entity dc:identifier ?id ;\n",
        "                    dc:identifier ?id ;\n",
        "                    nikAttr:label ?label ;\n",
        "                    a ?type .\n",
        "        }\n",
        "        \"\"\" \n",
        "\n",
        "        df_sparql_getAllEntities = get(endpoint, query_AllEntities, True)\n",
        "        return df_sparql_getAllEntities"
      ],
      "metadata": {
        "id": "HiuaXDz7NXx9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here I use a facility of Pandas that permits to interact with a SPARQL endpoint provided by an RDF triplestore (in our case Blazegraph) -> the library sparql_dataframe -> with this library the answer of our query will be shows as a Pandas dataframe. \n",
        "We use the method \"get\", that takes in input three parameters, to perform this operation. The input parameters are: the URL of the SPARQL endpoint to contact, the query to execute, and a boolean that specify if to contact the SPARQL endpoint using th PostHTTP method.\n",
        "\n",
        "For some of these queries I've used \"%s\" and \"%\" that are part of a string formatting in Python. They are used to dynamically insert the \"input\" value into the query.\n",
        "\n",
        "In the query \"%s\" is placed within the query string as a placeholder for the input value. Then, later in the query, the \"%\" character is used to indicate that you want to replace %s with the actual input value.\n",
        "\n",
        "In \"getEntitiesWithLabel\" I used also an external function to clean the string and escape some problematic characters."
      ],
      "metadata": {
        "id": "J9q88S9dNpC5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Clean String Function**"
      ],
      "metadata": {
        "id": "FMmBUlIJdjzq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def remove_special_chars(s: str) -> str:\n",
        "    if '\\\"' in s:\n",
        "        return s.replace('\\\"', '\\\\\\\"')\n",
        "    elif '\"' in s:\n",
        "        return s.replace('\"', '\\\\\\\"')\n",
        "    else:\n",
        "        return s"
      ],
      "metadata": {
        "id": "6v60-loadoeu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "With this function I'm escaping the double-quotes '\"' and the slash + double-quotes '\\\"' that we can meet inside the labels of the json files in order to produce safe strings to insert them in the SPARQL queries withouth problems.\n"
      ],
      "metadata": {
        "id": "VAhUun50dvGS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Last 4 methods of the GenericQueryProcessor**"
      ],
      "metadata": {
        "id": "6WkuhQlKeTKm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def getEntitiesWithLabel(self, label):\n",
        "        \n",
        "        graph_db = DataFrame()\n",
        "        relation_db = DataFrame()\n",
        "        result = list()\n",
        "\n",
        "        for processor in self.queryProcessors:\n",
        "            if isinstance(processor, TriplestoreQueryProcessor):\n",
        "                graph_to_add = processor.getEntitiesWithLabel(label)\n",
        "                graph_db = concat([graph_db, graph_to_add], ignore_index= True)\n",
        "            elif isinstance(processor, RelationalQueryProcessor):\n",
        "                relation_to_add = processor.getEntities()\n",
        "                relation_db = concat([relation_db, relation_to_add], ignore_index=True)\n",
        "            else:\n",
        "                break\n",
        "        \n",
        "        \n",
        "        if not graph_db.empty: #check if the call got some result\n",
        "            df_joined = merge(graph_db, relation_db, left_on=\"id\", right_on=\"id\") #create the merge with the two db\n",
        "            grouped = df_joined.groupby(\"id\").agg({\n",
        "                                                        \"label\": \"first\",\n",
        "                                                        \"title\": \"first\",\n",
        "                                                        \"creator\": lambda x: \"; \".join(x)\n",
        "                                                    }).reset_index() #this is to avoid duplicates when we have more than one creator\n",
        "            grouped_fill = grouped.fillna('')\n",
        "            sorted = grouped_fill.sort_values(\"id\") #sorted for id\n",
        "            \n",
        "            if not sorted.empty:\n",
        "                for row_idx, row in sorted.iterrows():\n",
        "                    id = row[\"id\"]\n",
        "                    label = label\n",
        "                    title = row[\"title\"]\n",
        "                    creators = row['creator'].split('; ')\n",
        "                    entities = EntityWithMetadata(id, label, title, creators)\n",
        "                    result.append(entities)            \n",
        "            \n",
        "                return result\n",
        "\n",
        "            else:\n",
        "                for _, row in graph_db.iterrows():\n",
        "                    id = row[\"id\"]\n",
        "                    label = label\n",
        "                    title = \"\"\n",
        "                    creators = \"\"\n",
        "                    entities = EntityWithMetadata(id, label, title, creators)\n",
        "                    result.append(entities)\n",
        "                return result\n",
        "        return result\n",
        "                "
      ],
      "metadata": {
        "id": "gIDuA-tOeayB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first is \"getEntitiesWithLabel\": it returns a list of objects having class EntityWithMetadata, included in the databases accessible via the query processors, related to the entities having, as label, the input label.\n",
        "\n",
        "I create two empty dataframe and I iterate every processor that are present in the list \"queryProcessor\", by dividing them according to the type of processor they are (either TriplestoreQueryProcessor or RelationalQueryProcessor), thus either linked to a graph database or linked to a relational database.\n",
        "\n",
        "If they are istances of TriplestoreQP I use the method \"getEntitiesWithLabel\" already developed in the TriplestoreQP and I store the result in a variable that I concat with one of the empty Dataframes in order to collect data from different processors all together in the same dataframe.\n",
        "The method \"getEntitiesWithLabel\" returns the internalId, the type, the label and the id of an entity that got as label the input label.\n",
        "I do the same for RelationalQP, but here, instead of \"getEntitiesWithLabel\", I use \"getEntities\" from the RelationalQP, that returns internalId, id, creators and title of all the entities.\n",
        "\n",
        "If the graphDB has got some result inside the dataframe (it is the one that contains \"label\", so it's the most important ot check) we can proceed with merging the dataframes with the column \"id\" as merging column.\n",
        "\n",
        "After that, I proceed with an operation of grouping.\n",
        "- \"grouped\" is the new DataFrame that is created to contain the results of the grouping and aggregation operation.\n",
        "- groupby(\"id\") -> performs the grouping of rows in the DataFrame df_joined based on the column \"id\". This means that groups of rows with the same \"id\" value will be created.\n",
        "- agg({...}) -> This method is called on the grouped DataFrame to perform data aggregation within each group. The arguments passed to \"agg\" specify the columns to be aggregated and the aggregation operations to be applied.\n",
        "- \"label\": \"first\": This line specifies that the first value of the group will be selected as the aggregate value for the \"label\" column of the grouped DataFrame. In other words, the first value of the \"label\" column within each group is taken.\n",
        "- same is for \"title\".\n",
        "- \"creator\": lambda x: '; '.join(x) -> This line specifies that a lambda function is defined for the column \"creator\" and applied to each group. The lambda function takes the entire set of values of the \"creator\" column as input and uses '; '.join(x) to join the values into a single string separated by '; '.\n",
        "- .reset_index() -> At the end of the aggregation operation, the reset_index() method is called to reset the index of the grouped DataFrame, so that the \"id\" column is returned as an ordinary column instead of an index.\n",
        "\n",
        "Finally I use \"fillna('')\" to fill in any empty cells and sort_values(\"id\") to sort the result by the ids.\n",
        "\n",
        "Then I check if the sorted df has something inside, if True, we can proceed with an iterrow for loop, checking every row of every column and creating some variables. For the row inside the column \"creator\" I split the values that I've previously merged together in a string, separated with \"; \". \n",
        "The splitting operation returns a list of strings. With these variables I create the EntityWithMetadata objects and I append them to the result list. \n",
        "\n",
        "If the sorted df is empty (so there are data coming only from the graph db) we can do the same iterrow for loop but this time only with the graph_db dataframe, and we will insert some empty strings for title and creators.\n"
      ],
      "metadata": {
        "id": "kgnw32bjge2A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def getEntitiesWithTitle(self, title):\n",
        "\n",
        "        graph_db = DataFrame()\n",
        "        relation_db = DataFrame()\n",
        "        result = list()\n",
        "\n",
        "        for processor in self.queryProcessors:\n",
        "            if isinstance(processor, TriplestoreQueryProcessor):\n",
        "                graph_to_add = processor.getAllEntities()\n",
        "                graph_db = concat([graph_db ,graph_to_add], ignore_index= True)\n",
        "            elif isinstance(processor, RelationalQueryProcessor):\n",
        "                relation_to_add = processor.getEntitiesWithTitle(title)\n",
        "                relation_db = concat([relation_db, relation_to_add], ignore_index=True)\n",
        "            else:\n",
        "                break        \n",
        "        \n",
        "\n",
        "        if not graph_db.empty:\n",
        "            df_joined = merge(graph_db, relation_db, left_on=\"id\", right_on=\"id\")\n",
        "            grouped = df_joined.groupby(\"id\").agg({\n",
        "                                                        \"label\": \"first\",\n",
        "                                                        \"title\": \"first\",\n",
        "                                                        \"creator\": lambda x: \"; \".join(x)\n",
        "                                                    }).reset_index() #this is to avoid duplicates when we have more than one creator\n",
        "            grouped_fill = grouped.fillna('')\n",
        "            # sorted = grouped_fill.sort_values(\"id\")\n",
        "\n",
        "            for _, row in grouped_fill.iterrows():\n",
        "                id = row[\"id\"]\n",
        "                label = row[\"label\"]\n",
        "                title = row['title']\n",
        "                creators = row['creator'].split('; ')\n",
        "                entities = EntityWithMetadata(id, label, title, creators)\n",
        "                result.append(entities)\n",
        "\n",
        "        return result"
      ],
      "metadata": {
        "id": "F54aF9lbqh8_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The second is \"getEntitiesWithTitle\" -> it returns a list of objects having class EntityWithMetadata, included in the databases accessible via the query processors, related to the entities having, as title, the input title.\n",
        "\n",
        "The very first part is the same as the method before.\n",
        "But here we use the method \"getAllEntities\" from the TriplestoreQP that returns internalId, id, label and type of all the entities.\n",
        "And the method \"getEntitiesWithTitle(title)\" from the RelationalQP that returnsinternalId, id , creator and title of the entities that correspond to the input title. \n",
        "\n",
        "The I check if the graph_db has something inside and I merge the dataframe together by the \"id\" column.\n",
        "Then I used the same function than before \"groupby\". \n",
        "And I proceed with an iterrows for loop to take all the variables to fill EntitiyWithMetadata arguments and I append the objects to the result list.\n"
      ],
      "metadata": {
        "id": "-5ZCfdhUrIEh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        " def getImagesAnnotatingCanvas(self, canvasId):\n",
        "\n",
        "        graph_db = DataFrame()\n",
        "        relation_db = DataFrame()\n",
        "        result = list()\n",
        "\n",
        "        for processor in self.queryProcessors:\n",
        "\n",
        "            if isinstance(processor, TriplestoreQueryProcessor):\n",
        "                graph_to_add = processor.getEntitiesWithCanvas(canvasId)\n",
        "                graph_db = concat([graph_db,graph_to_add], ignore_index= True)\n",
        "            elif isinstance(processor, RelationalQueryProcessor):\n",
        "                relation_to_add = processor.getAllAnnotations()\n",
        "                relation_db = concat([relation_db, relation_to_add], ignore_index=True)\n",
        "            else:\n",
        "                break\n",
        "\n",
        "        if not graph_db.empty:\n",
        "            df_joined = merge(graph_db, relation_db, left_on=\"id\", right_on=\"target\")\n",
        "\n",
        "            for _, row in df_joined.iterrows():\n",
        "                id = row[\"body\"]\n",
        "                images = Image(id)\n",
        "            result.append(images)\n",
        "\n",
        "        return result"
      ],
      "metadata": {
        "id": "hpO7zI8NtF8K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The third is \"getImagesAnnotatingCanvas\" -> it returns a list of objects having class Image, included in the databases accessible via the query processors, that are body of the annotations targetting the canvaes specified by the input identifier.\n",
        "\n",
        "Here I use the method \"getEntitiesWithCanvas\" from the TriplestoreQP that returns id, label and type of the canvas corresponding to the input canvasId.\n",
        "And I use \"getAllAnnotations()\" from the RelationalQP that returns everything from the Annotation table so id, body, target and motivation.\n",
        "\n",
        "Then I merge the df with the column \"id\" from the graph_db and the column \"target\" from the relational_db.\n",
        "\n",
        "I do an iterrows for loop and i create the variable \"id\" taking data from the column \"body\" to fill the arguments of the Class Image. Finally, I append the entities to the result list."
      ],
      "metadata": {
        "id": "tQq0bRpxtOho"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def getManifestsInCollection(self, collectionId):\n",
        "\n",
        "        graph_db = DataFrame()\n",
        "        relation_db = DataFrame()\n",
        "        result = list()\n",
        "        \n",
        "        for processor in self.queryProcessors:\n",
        "            if isinstance(processor, TriplestoreQueryProcessor):\n",
        "                graph_to_add = processor.getManifestsInCollection(collectionId)\n",
        "                graph_db = concat([graph_db,graph_to_add], ignore_index= True)\n",
        "            elif isinstance(processor, RelationalQueryProcessor):\n",
        "                relation_to_add = processor.getEntities()\n",
        "                relation_db = concat([relation_db, relation_to_add], ignore_index=True)\n",
        "            else:\n",
        "                break\n",
        "        \n",
        "\n",
        "        if not graph_db.empty:\n",
        "            df_joined = merge(graph_db, relation_db, left_on=\"id\", right_on=\"id\") \n",
        "\n",
        "            for _, row in df_joined.iterrows():\n",
        "\n",
        "                graph_db_canvas = DataFrame()\n",
        "\n",
        "                for processor in self.queryProcessors:\n",
        "                    if isinstance(processor, TriplestoreQueryProcessor):\n",
        "                        graph_to_add_canvas = processor.getCanvasesInManifest(row['id'])\n",
        "                        graph_db_canvas = concat([graph_db_canvas,graph_to_add_canvas], ignore_index= True).drop_duplicates()\n",
        "\n",
        "                        df_joined_canvas = merge(graph_db_canvas, \n",
        "                                                relation_db,\n",
        "                                                how='left',\n",
        "                                                left_on='id',\n",
        "                                                right_on='id'\n",
        "                                                ).fillna('')\n",
        "        \n",
        "        \n",
        "                        df_joined_canvas['creator'] =  df_joined_canvas.groupby(['canvas','id','label', 'entityId', 'title'])['creator'].transform(lambda x: '; '.join(x))\n",
        "                        df_joined_canvas = df_joined_canvas.drop_duplicates()\n",
        "\n",
        "                        canvases_list = [\n",
        "                                Canvas(row1['id'], \n",
        "                                row1['label'], \n",
        "                                row1['title'],\n",
        "                                row1['creator'].split('; ')) \n",
        "                                for _, row1 in df_joined_canvas.iterrows()\n",
        "                            ]\n",
        "                            \n",
        "                        result.append(\n",
        "                                Manifest(row[\"id\"],\n",
        "                                    row[\"label\"], \n",
        "                                    canvases_list,\n",
        "                                    row['title'], \n",
        "                                    row['creator'].split('; ')\n",
        "                                    ) \n",
        "                            )\n",
        "\n",
        "        return result\n",
        "            "
      ],
      "metadata": {
        "id": "aY89pOQhuXR-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The last is the method \"getManifestInCollection\" -> it returns a list of objects having class Manifest, included in the databases accessible via the query processors, that are contained in the collection identified by the input identifier.\n",
        "\n",
        "Here I use the method \"getManifestsInCollection(collectionId)\" from the TriplestoreQP that returns internalId, id, label. And I use \"getEntities()\" from the RelationalQP, that returns internalId, id, creators and title of all the entities.\n",
        "\n",
        "If the graph_db DataFrame is not empty, it merges (inner join) the graph_db and relation_db DataFrames based on their \"id\" columns. \n",
        "\n",
        "The code iterates over each row of the df_joined DataFrame.\n",
        "A new DataFrame, \"graph_db_canvas\", is initialized for each iteration to store the results specific to canvases.\n",
        "Another loop iterates over each query processor.\n",
        "If the processor is an instance of TriplestoreQP, it calls the \"getCanvasesInManifest\" method of that processor with the \"id\" value from the current row. The result is stored in \"graph_to_add_canvas\". The \"graph_db_canvas\" DataFrame is concatenated with \"graph_to_add_canvas\" to accumulate the results, and duplicates are dropped.\n",
        "\n",
        "Then, the \"graph_db_canvas\" DataFrame is merged with the relation_db DataFrame based on the common column \"id\". Missing values are filled with empty strings.\n",
        "\n",
        "At the end the canvases_list is inserted in the items arguments in the Class Manifest."
      ],
      "metadata": {
        "id": "AZeeLXBKudRq"
      }
    }
  ]
}