diff --git a/docs/docs/devs/devhandbook.md b/docs/docs/devs/devhandbook.md new file mode 100644 index 000000000..729834cc6 --- /dev/null +++ b/docs/docs/devs/devhandbook.md @@ -0,0 +1,719 @@ +# Clowder Developer Handbook + +# 1. Change Clowder Core + +## 1.1 Backend + +### 1.1.1 API Routes + +- Backend API routes interact with the frontend. + +- A controller exists for each type of object (file, dataset, users, + etc) + +- Routes take arguments from the user and dependencies and handle + dependencies. + +- Run Codegen when you add new routes/models to the backend to update + the front end. That will add new methods to the various + frontend/openapi/services and models. + + - npm run codegen:v2:dev + +- Routes are annotated with \@get \@post \@put in FastAPI, whatever + the type of request is being made. The response_model tells you what + the method returns. + +Here we see the method - what the arguments are. This is the +get_datasets method that populates the datasets you see in the 'Explore' +page. + +```python +@router.get("", response_model=Paged) +async def get_datasets( + user_id=Depends(get_user), + skip: int = 0, + limit: int = 10, + mine: bool = False, + admin=Depends(get_admin), + enable_admin: bool = False, + admin_mode: bool = Depends(get_admin_mode), +): +``` + +A query is created depending on these values. We see the default values +here for 'skip', 'limit' 'mine' and 'enable_admin' and 'admin_mode.' +Some are passed in as parameters, others come from dependencies like +Depends(get_user). Here we see that since 'mine' is true, we check that +the creator.email is the same as the current user_id from +Depends(get_user). Below we see the query that will be used to find the +results using the DatasetDBViewList db table. + +```python +if mine: + query.append(DatasetDBViewList.creator.email == user_id) +datasets_and_count = ( + await DatasetDBViewList.find(*query) + .aggregate( + [_get_page_query(skip, limit, sort_field="created", ascending=False)], + ) + .to_list() +) + +``` + +The results are returned as 'Paged' and as a dict for the front end. An +object would be returned as + +A dictionary as well. + +```python +page = Paged( + metadata=page_metadata, + data=[ + DatasetOut(id=item.pop("_id"), **item) + for item in datasets_and_count[0]["data"] + ], +) + +return page.dict() +``` + +[Link](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/backend/app/routers/datasets.py#L220C1-L278C23) +to full method on github. + +To interact with the backend as a user directly, you can create an API +key for a user. Postman app or python requests are two options for +interacting with the back end directly instead of through the frontend +interface. When adding new methods, it can be helpful to test the +backend separately from the front end. + +### 1.1.2 Models + +Models folder (under +[backend/app/models/](https://github.com/clowder-framework/clowder2/tree/bb35e0b5040443abb847e658ad61f622f189f6bb/backend/app/models)) +holds data models. + +```python +class DatasetBaseCommon(DatasetBase): + creator: UserOut + created: datetime = Field(default_factory=datetime.utcnow) + modified: datetime = Field(default_factory=datetime.utcnow) + status: str = DatasetStatus.PRIVATE.name + user_views: int = 0 + downloads: int = 0 + thumbnail_id: Optional[PydanticObjectId] = None + origin_id: Optional[PydanticObjectId] = None + standard_license: bool = True + license_id: Optional[str] = None +``` + +If you wish to add a field in any existing model, locate the model you +wish to change and add it with corresponding types, and if needed, a +default value. + +E.g. I wish to have a "favorite" field stored in each of the dataset +documents. + +```python +class DatasetBaseCommon(DatasetBase): + ... + favorite: bool = False + + +``` + +#### 1.1.2.1 Conventions + +{XXX}**DBViewList** - View Models. These models are used to create views +in MongoDB, which aggregate and transform data. + +{XXX}**DB** - These models are directly tied to the Documents in MongoDB +collections.\ +{XXX}**In** - These models are used for incoming data, such as defining +the request body for the API endpoint. + +{XXX}**Out** - These models are used for defining the data out, such as +API responses. + +#### 1.1.2.2 Use models in the Routes + +E.g. Define the response model, the endpoint will automatically cast the +return object into the model shape + +E.g. Define the request body shape. Dataset_in must match the shape of +the DatasetIn model otherwise it will complain. + +```python +@router.post("", response_model=DatasetOut) +async def save_dataset( + dataset_in: DatasetIn, + license_id: str, + user=Depends(get_current_user), + es: Elasticsearch = Depends(dependencies.get_elasticsearchclient), +): + ... + return dataset.dict() +``` + +## 1.2 Frontend + +### 1.2.1 Components + +The frontend consists of various React components, each serving specific +functions. **Before developing a new component, please review the +existing ones to see if any can be reused.** + +![](devhandbookmedia/image12.png) + +#### 1.2.1.1 Reuse Components + +Components are often reused. Both Dataset and File use the same +DisplayMetadata component. The component takes the resourceType as an +argument, that can be a dataset or file. Reuse of components can save +time and keep design and function consistent. + +```javascript +import FilesTable from "../files/FilesTable"; +import {MetadataIn} from "../../openapi/v2"; +import {DisplayMetadata} from "../metadata/DisplayMetadata"; +import {DisplayListenerMetadata} from "../metadata/DisplayListenerMetadata"; +import {EditMetadata} from "../metadata/EditMetadata"; +``` + +Here we see the import of DisplayMetadata. This component is used for +Files or Datasets. Below we see arguments for resourceId and +resourceType, which can be file or dataset. + +```javascript +export const DisplayMetadata = (props: MetadataType) => { + const { + updateMetadata, + deleteMetadata, + resourceType, + resourceId, + publicView, + } = props; +``` + +#### 1.2.1.2 Styled Components + +We provide a set of basic Clowder-styled components (e.g., Button, +Select, Title, Footnote). These can be used just like MUI components, +with styling controlled here and props still passable. Found those +components +[here](https://github.com/clowder-framework/clowder2/tree/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/components/styledComponents). + +For example, the ClowderSelect component: + +```javascript +import {ClowderSelect} from "../styledComponents/ClowderSelect"; + +... + + { + setSelectedVersionNum(event.target.value); + setSnackBarMessage(`Viewing version ${event.target.value}`); + setSnackBarOpen(true); + }} + > + {fileVersions.map((fileVersion) => { + return ( + + {fileVersion.version_num} + + ); + })} + + + +```` + +#### 1.2.1.3 Auth Wrapper + +Some components show different things based on user permissions. Here is +an +[example](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/components/datasets/Dataset.tsx#L456) + +Using ``. We see that only the **owner**, **editor** or +**uploader** can have access to the "Add Metadata" component. + +```javascript + + + + + + +``` + +#### 1.2.1.4 Full Example + +Take +[DisplayMetadata](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/components/metadata/DisplayMetadata.tsx#L31) +as an example: + +```javascript +export const DisplayMetadata = (props: MetadataType) => { + const { + updateMetadata, + deleteMetadata, + resourceType, + resourceId, + publicView, + } = props; + + const dispatch = useDispatch(); + + const getMetadatDefinitions = ( + name: string | null, + skip: number, + limit: number + ) => dispatch(fetchMetadataDefinitions(name, skip, limit)); + +``` + +[Here](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/components/metadata/DisplayMetadata.tsx#L92-L106) +we see that whether we have a dataset or file, we use different methods +to get the metadata. This is how components can be reused. There are +also options for publicView if the user is not logged in. + +```javascript +useEffect(() => { + if (resourceType === "dataset") { + if (publicView) { + listPublicDatasetMetadata(resourceId); + } else { + listDatasetMetadata(resourceId); + } + } else if (resourceType === "file") { + if (publicView) { + listPublicFileMetadata(resourceId); + } else { + listFileMetadata(resourceId); + } + } +}, [resourceType, resourceId]); + +``` + +These work by [importing +actions](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/components/metadata/DisplayMetadata.tsx#L5-L12) +from frontend/actions + +```javascript +import { + fetchDatasetMetadata, + fetchFileMetadata, + fetchMetadataDefinitions, + fetchPublicFileMetadata, + fetchPublicMetadataDefinitions, +} from "../../actions/metadata"; +``` + +And we can see the call in +[frontend/src/actions/metadata.js](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/actions/metadata.js#L7-L31) +that calls a method from the Services + +```javascript +export function fetchMetadataDefinitions(name, skip, limit) { + return (dispatch) => { + return V2.MetadataService.getMetadataDefinitionListApiV2MetadataDefinitionGet( + name, + skip, + limit + ) + .then((json) => { + dispatch({ + type: RECEIVE_METADATA_DEFINITIONS, + metadataDefinitionList: json, + receivedAt: Date.now(), + }); + }) + .catch((reason) => { + dispatch( + handleErrors(reason, fetchMetadataDefinitions(name, skip, limit)) + ); + }); + }; +} + +``` + +And looking at the service in +[/frontend/src/openapi/v2/MetadataService](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/openapi/v2/services/MetadataService.ts#L26-L43) +.ts, +we see it calling the back end routes. + +```javascript +public +static +getMetadataDefinitionListApiV2MetadataDefinitionGet( + name ? : string, + skip ? : number, + limit +: +number = 2, +): +CancelablePromise < Paged > { + return __request({ + method: 'GET', + path: `/api/v2/metadata/definition`, + query: { + 'name': name, + 'skip': skip, + 'limit': limit, + }, + errors: { + 422: `Validation Error`, + }, + }); +``` + +### 1.2.2 Redux + +![](devhandbookmedia/image10.png) + +- Add an action under the action folder + [here](https://github.com/clowder-framework/clowder2/tree/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/actions) + +- Update the reducer + [here](https://github.com/clowder-framework/clowder2/tree/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/reducers) + +- Access the action and state in component use hooks. + +```javascript + +import {useDispatch, useSelector} from "react-redux"; + +... + +const dispatch = useDispatch(); +const listFileSummary = (fileId: string | undefined) => + dispatch(fetchFileSummary(fileId)); + +const file = useSelector((state: RootState) => state.file); + +``` + +# 2. Write New Extractors + +### 2.1 Components of extractors + + + +- requirements.txt contains the required set of libraries to install + (It always need pyclowder, this is library used to connect to + Clowder) + +``` +pyclowder==3.0.7 +``` + +- extractor_info.json contains the details about the extractors + +```json + +{ + "@context": "http://clowder.ncsa.illinois.edu/contexts/extractors.jsonld", + "name": "ncsa.wordcount-3", + "version": "2.0", + "description": "WordCount extractor. Counts the number of characters, words and lines in the text file that was uploaded.", + "author": "Rob Kooper ", + "contributors": [], + "contexts": [ + { + "lines": "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#lines", + "words": "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#words", + "characters": "http://clowder.ncsa.illinois.edu/metadata/ncsa.wordcount#characters" + } + ], + "repository": [ + { + "repType": "git", + "repUrl": "https://opensource.ncsa.illinois.edu/stash/scm/cats/pyclowder.git" + } + ], + "process": { + "file": [ + "text/*", + "application/json" + ] + }, + "external_services": [], + "dependencies": [], + "bibtex": [] +} +``` + +- .py file contains the code for your extractor + +```python + +class WordCount(Extractor): + """Count the number of characters, words and lines in a text file.""" + + def __init__(self): + Extractor.__init__(self) + + # add any additional arguments to parser + # self.parser.add_argument('--max', '-m', type=int, nargs='?', default=-1, + # help='maximum number (default=-1)') + + # parse command line and load default logging configuration + self.setup() + + # setup logging for the extractor + logging.getLogger('pyclowder').setLevel(logging.DEBUG) + logging.getLogger('__main__').setLevel(logging.DEBUG) + +``` + +- README contains the basic instructions to run the extractor + +- **Dockerfile contains a set of instructions to build a Docker image + out of the extractor** + +More details are [here](https://github.com/clowder-framework/pyclowder) +about creating an extractor + +### 2.2 Starting the extractor + +#### 2.2.1 Production mode + +- Create Dockerfile and build docker container + +- Add to + [docker-compose.extractors.yml](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/docker-compose.extractors.yml). + + - Make sure to include default environment variables + **CLOWDER_VESERION**, and **RABBITMQ_URI**. + + - Set the networks to **clowder2** + + For example: + +```yaml +sentiment-analysis: + image: socialmediamacroscope/sentiment_analysis_extractor:latest + environment: + CLOWDER_VERSION: 2 + RABBITMQ_URI: amqp://guest:guest@rabbitmq:5672/%2F + networks: + - clowder2 + restart: unless-stopped +``` + +- Run `docker-compose -f docker-compose.yml -f docker-compose.extractors.yml -p clowder-prod up` + +#### 2.2.2 Dev mode + +Run the main file in your extractor in the IDE, with environment +parameters set. Similar to above, **CLOWDER_VERSION** and +**RABBITMQ_URI** are required. + +![](devhandbookmedia/image14.png) + +Alternatively, using command line + +```shell +export PYTHONUNBUFFERED=1 +export CLOWDER_VERSION=2 +export IMAGE_BINARY=/usr/local/bin/convert +python run pyclowder/sample-extractors/image-preview-v2/binary_extractor.py +``` + +### 2.3 Check if extractor registered + +- Log in + +- Turn on admin mode + +- Go to extractors tab at the side menu, you should be able to see + your extractors with information + +![](devhandbookmedia/image11.png) +![](devhandbookmedia/image13.png) + +- Go to the file/dataset that are compatible with that extractor. + Submit and watch the logs + +![](devhandbookmedia/image9.png) + +### 2.4 Create Custom Widget submitting to Extractor + +#### 2.4.1 Modifying Widget for RSJF + +In order to create custom widgets, we need to take the following steps: + +- After you created your component make sure, there is an onChange + function, this onChange function. The component will use it to set + the parameters. In my example, I am creating an Image Annotator, so + I need to save the name and points of the annotation + +> ![](devhandbookmedia/image2.png) +> +> ![](devhandbookmedia/image4.png) + +- Create a component in a directory (I use StyledComponents), this + offers us to customize the component for the extractor but mainly + acts as an intermediary between the rsjf form and the component. The + form passes an object which contains a parameter and title that I + use to create a title + +> ![](devhandbookmedia/image3.png) + +#### 2.4.2 Adding to the RSJF + +Now the widget is ready for the form, our form for extractors is in the +file SubmitExtraction.tsx. The steps are: + +Import the widget component and add to the widgets object that gives the +form a map to the different custom widgets + +![](devhandbookmedia/image1.png) + +In order to map the parameter to the widget, we use the second parameter +for the form uiSchema. In the extractor-info.json, we give the parameter +have a property called format that tells the form to map this widget to +the extractor. + +We use a for loop in the file here to go through the parameters and add +it to the uiSchema if a parameter needs a custom widget + +![](devhandbookmedia/image6.png) + +#### 2.4.3 In the extractor-info.json + +In the extractor json that renders the form, create the parameter that +can map to this widget. So, since in my case, I use the field +ImageAnnotator to map to this widget, I create the following + +```json +"parameters": { +"schema": { +"Annotations": { +"type": "object", +"title": "Annotate image", +"description": "Annotate image", +"format": "ImageAnnotator" +} +} +} +``` + +When we open the extractor, we should the be able to see the widget, + +![](devhandbookmedia/image5.png) + +![](devhandbookmedia/image7.png) + +When you submit the extractor, you can see the parameters are passed in +if you extractor script logs the parameters it receives + +![](devhandbookmedia/image8.png) + +# 3. Custom Visualization Component + +You can refer to this PR that adds word cloud visualization as an end to +end example: +[https://github.com/clowder-framework/clowder2/pull/786](https://github.com/clowder-framework/clowder2/pull/786) + +### 3.1 Create Folder + +under [visualization +folder](https://github.com/clowder-framework/clowder2/tree/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/components/visualizations) + +### 3.2 Create the main component + +This component could expect 3 props: ***fileId, visualizationId, +publicView*** + +***If fileId specified → Raw file bytes (below ) is used to*** + +***If visualizationId specified → Visualization can be generated by +(processed) visualization data source*** + +***If publicView is True → Visualization can be viewed on public +dataset*** + +```javascript +type +VisProps = { + fileId? : string; + visualizationId? : string; + publicView? : boolean | false; +}; + +// E.g. +export default function Image(props: VisProps) { + const {fileId, visualizationId, publicView} = props; + +``` + +### 3.3 Create manifest.json + +- Ensure the \"main\" is directed to the component. + +- Add visConfig fields, ensuring the \"name\" matches the folder name. + +- List the \"mainType\" and \"mimeType\" that the visualization widget + supports. + +```json + +{ + "name": "vega-spec", + "version": "1.0.0", + "description": "A simple visualization using vegalite and the spec in json from vis data.", + "main": "VegaSpec.tsx", + "dependencies": { + "clowder2-core": "1.0.0", + "react": "^17.0.2", + "react-dom": "^17.0.2", + "vega": "^5.20.2" + }, + "visConfig": { + "name": "VegaSpec", + "mainType": "application", + "mimeTypes": [ + "application/json" + ], + "fields": [] + } +} +``` + +### 3.4 Register the component + +Add code block in +[visualization.config.ts](https://github.com/clowder-framework/clowder2/blob/bb35e0b5040443abb847e658ad61f622f189f6bb/frontend/src/visualization.config.ts). +Place the correct manifest.json + +For example: + +```javascript +const configWordCloudSpec = require("./components/visualizations/VegaSpec/manifest.json"); + +visComponentDefinitions.push({ + name: configWordCloudSpec.name, + mainType: configWordCloudSpec.visConfig.mainType, + mimeTypes: configWordCloudSpec.visConfig.mimeTypes, + component: React.createElement(registerComponent(configWordCloudSpec)), +}); + +``` diff --git a/docs/docs/devs/devhandbookmedia/image1.png b/docs/docs/devs/devhandbookmedia/image1.png new file mode 100644 index 000000000..58bb030b4 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image1.png differ diff --git a/docs/docs/devs/devhandbookmedia/image10.png b/docs/docs/devs/devhandbookmedia/image10.png new file mode 100644 index 000000000..a0167f473 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image10.png differ diff --git a/docs/docs/devs/devhandbookmedia/image11.png b/docs/docs/devs/devhandbookmedia/image11.png new file mode 100644 index 000000000..71cdf2a63 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image11.png differ diff --git a/docs/docs/devs/devhandbookmedia/image12.png b/docs/docs/devs/devhandbookmedia/image12.png new file mode 100644 index 000000000..ff66e164d Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image12.png differ diff --git a/docs/docs/devs/devhandbookmedia/image13.png b/docs/docs/devs/devhandbookmedia/image13.png new file mode 100644 index 000000000..60da05202 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image13.png differ diff --git a/docs/docs/devs/devhandbookmedia/image14.png b/docs/docs/devs/devhandbookmedia/image14.png new file mode 100644 index 000000000..da7240f01 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image14.png differ diff --git a/docs/docs/devs/devhandbookmedia/image2.png b/docs/docs/devs/devhandbookmedia/image2.png new file mode 100644 index 000000000..97e7d79d2 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image2.png differ diff --git a/docs/docs/devs/devhandbookmedia/image3.png b/docs/docs/devs/devhandbookmedia/image3.png new file mode 100644 index 000000000..723ee7a0f Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image3.png differ diff --git a/docs/docs/devs/devhandbookmedia/image4.png b/docs/docs/devs/devhandbookmedia/image4.png new file mode 100644 index 000000000..e89ca2e6b Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image4.png differ diff --git a/docs/docs/devs/devhandbookmedia/image5.png b/docs/docs/devs/devhandbookmedia/image5.png new file mode 100644 index 000000000..734cffd35 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image5.png differ diff --git a/docs/docs/devs/devhandbookmedia/image6.png b/docs/docs/devs/devhandbookmedia/image6.png new file mode 100644 index 000000000..78870d9ef Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image6.png differ diff --git a/docs/docs/devs/devhandbookmedia/image7.png b/docs/docs/devs/devhandbookmedia/image7.png new file mode 100644 index 000000000..7400cee80 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image7.png differ diff --git a/docs/docs/devs/devhandbookmedia/image8.png b/docs/docs/devs/devhandbookmedia/image8.png new file mode 100644 index 000000000..5bb167b24 Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image8.png differ diff --git a/docs/docs/devs/devhandbookmedia/image9.png b/docs/docs/devs/devhandbookmedia/image9.png new file mode 100644 index 000000000..9c3d8613e Binary files /dev/null and b/docs/docs/devs/devhandbookmedia/image9.png differ diff --git a/docs/docs/devs/devsetup.md b/docs/docs/devs/devsetup.md new file mode 100644 index 000000000..665cb3c32 --- /dev/null +++ b/docs/docs/devs/devsetup.md @@ -0,0 +1,767 @@ +# Clowder V2 Dev Environment Setup + +## 1. Explore Github Repository + +### 1.1 Prerequisites + +- Understanding how git works + - For more information, read this [https://www.atlassian.com/git](https://www.atlassian.com/git) + +- Create a GitHub Account + - If you do not have a GitHub account, create one [https://github.com/signup](https://github.com/signup) + +### 1.2 GitHub Basics + +- **Repositories** are centralized locations for project files. + +- **Issues** are used to track tasks, bugs, and feature requests. + +- **Pull** **Requests** **(PRs)**: Proposed changes to a repository. + +![](./devsetupmedia/image43.png) + +### 1.3 Creating Issues in Clowder2 + +- Navigate to the Clowder2 repository: + [https://github.com/clowder-framework/clowder2/](https://github.com/clowder-framework/clowder2/) + +- Click on the \"Issues\" tab. + +- Click the \"New issue\" button. + +- Fill in the issue template with a descriptive title and detailed + description. + +- Click \"Submit new issue.\" + +- Optional: Decide assignees and pick appropriate labels + +![](./devsetupmedia/image30.png) +![](./devsetupmedia/image38.png) +![](./devsetupmedia/image17.png) + +### 1.4 Creating a Branch Related to an Issue Using GitHub Interface + +- Click on the \"Issues\" tab and open the issue for which you want to + create a branch. + +- On the issue page, under development section on the right side + panel, click the \"Create a branch\" link + +- In the pop up window, double check if it is the correct repository + destination "clowder-framework/clowder2". In rare case, you can + create a branch in another clowder affiliated repository + +- The default branch will be created based on the "main" branch, but + you can change branch source if you would like to branch off a + specific branch. + +- Once you click the "Create branch" button, a new branch will be + created with a name referencing the issue number. + +![](./devsetupmedia/image7.png) + +### 1.5 Opening a Pull Request (PR) or Draft PR + +- Make your code changes and commit them. + + - Using the command line tool, run `git commit -m ""` + - **Alternatively, you can use PyCharm to commit, which will be covered in the next section.** + +- Push the branch to GitHub + + - If using the command line, run `git push origin ` + - **Alternatively, you can use PyCharm to push, which will be covered in the next section.** + +![](./devsetupmedia/image24.png) + +- Navigate to the repository on GitHub. Alternatively, use the link + provided in the command line after you push your branch. + +- Find the \"Pull requests\" tab, then click the \"New pull request\" + button. + +- Select the correct base branch (the branch to be merged into) and + your branch. Review the changes, and if everything looks correct, + click the \"Create pull request\" button. + +- Fill in the PR template with a descriptive title and detailed + description of the changes. + +- Appoint reviewers. + +- Select \"Create draft pull request\" if your work is not yet + complete; otherwise, click \"Create pull request.\" + +![](./devsetupmedia/image21.png) + +![](./devsetupmedia/image9.png) + +![](./devsetupmedia/image18.png) + +### 1.6 Review Process + +- Once the PR is created, assign other developers for peer review. The + PR will also require review from the project maintainers. + +- Reviewers should provide feedback and/or approve the changes. + Navigate to the "Files changed" tab and click "Review changes" + button to provide feedback/ + +- To provide inline feedback, click on the lines directly. + +- The PR author needs to address any feedback by making changes and + pushing them to the branch. + +- Click \"Resolve conversation\" to indicate that the feedback has + been addressed. + +- Once approved, AND all the github actions (checks) passed, the PR + can be merged into the main branch. + +- If there are multiple commits, consider using the \"Squash and + merge\" option. + +![](./devsetupmedia/image15.png) + +![](./devsetupmedia/image32.png) + +![](./devsetupmedia/image20.png) + +![](./devsetupmedia/image49.png) + +## 2. Set up Integrated Development Environment (IDE) (PyCharm) + +### 2.1 Download PyCharm + +- Download PyCharm Community Edition + ([https://www.jetbrains.com/pycharm/download/](https://www.jetbrains.com/pycharm/download/)). + It should select the right PyCharm version for your computer. + +- Installing PyCharm community edition. + +- **If you have access to Professional Edition**, feel free to use + that since it has more comprehensive features + +### 2.2 Opening the Clowder2 Project in PyCharm + +- Clone the Clowder2 GitHub repository project via command line + (terminal). + + - On the command line, make sure that you have git installed. + + - Note that on the GitHub page, there is a button 'Code.' Click on + that and you will be able to copy the HTTPS link to the Clowder2 + GitHub repository. + + - The command for cloning is: `git clone https://github.com/clowder-framework/clowder2.git` + +- Once cloned, you can click on the 'Open' after you open PyCharm, and + navigate to the folder where Clowder2 was cloned and open the + project. + +![](./devsetupmedia/image40.png) + +![](./devsetupmedia/image36.png) + +Alternatively + +- After you open PyCharm, click on "**Get from VCS**" button + +- Select "**Git**" as the version control option and put in the Clowder's GitHub repository URL + +- Choose a Directory path which is empty + +- Click "Clone" + +![](./devsetupmedia/image19.png) + +![](./devsetupmedia/image44.png) + +### 2.3 Switching to a Branch + +Within PyCharm, under Git \> Branches you have the option of either +creating a branch from the one you are on, or from checking out remote +branches. + +![](./devsetupmedia/image34.png) + +### 2.4 Comparing Files Across Branches + +PyCharm also allows you to compare files across branches. Click on +"**Git**" and then "**Current File**" and you can use **"Compare with +Branch..."** to compare file changes with other branches. "**Fetch**" +will fetch all remote branches. If you are unable to check out a new +branch, make sure you run fetch first to bring you up to date with the +latest changes in the GitHub repository. Also take note of the option +**"Merge...".** Periodically, you may need to merge the *main* branch +onto your local development branch to bring the changes that may have +got merged on the *main* branch. + +![](./devsetupmedia/image10.png) + +### 2.5 Commit and Push + +Below we see options called **"Commit..."** and **"Push..."**. You +should commit and push your work often (at least daily or after you've +made significant changes) to make sure that your changes are saved and +will not be lost. The commits you make will be visible on GitHub, and +that way other people can test your branch. + +![](./devsetupmedia/image14.png) + +![](./devsetupmedia/image51.png) + +![](./devsetupmedia/image23.png) + +## 3. Install Libraries + +For our project, we have two main parts: the backend and the frontend. + +- **Backend:** We use Python and a framework called + [FastAPI](https://fastapi.tiangolo.com/), which helps + us create the backend of our application. To run our backend + efficiently, we use the + [Uvicorn](https://www.uvicorn.org/) server. To manage + and install all the necessary tools and libraries for the backend, + we use a package manager + [Pipenv](https://pipenv.pypa.io/en/latest/). + +- **Frontend:** We use [React](https://react.dev/), a + popular library for building user interfaces, to create the frontend + of our application. The frontend module is developed using + TypeScript, React, Material UI, Redux, webpack, Node.js. To manage + and install all the tools and libraries needed for the frontend, we + use a package manager called + [NPM](https://www.npmjs.com/) (Node Package Manager). + +### 3.1 Install Backend Dependencies + +#### 3.1.1 Python + +Install Python [here](https://www.python.org/downloads/). Download the specific architecture version. We recommend +using Python 3.9 + +![](./devsetupmedia/image25.png) + +Open the downloaded file and follow the installation instructions. After +the installation, open a terminal or command prompt. + +Type `python --version` to check that Python is installed and to see the +version number. + +![](./devsetupmedia/image5.png) + +#### 3.1.2 Install Pipenv + +`pip install --user pipenv` + +Pipenv provides a single, unified interface that allows you to manage +your Python project dependencies and virtual environments with ease. + +- It uses a **Pipfile** to specify the packages you need and a + **Pipfile.lock** to lock down the exact versions of these packages + to ensure reproducibility. + +- It automatically creates and manages a **virtual environment** for + your projects, ensuring that your dependencies are isolated from + other projects on your mac + +We have 2 Pipfiles + +- [clowder2/Pipfile](https://github.com/clowder-framework/clowder2/blob/main/Pipfile) - + contains the libraries needed for overall app + +- [clowder2/backend/Pipfile](https://github.com/clowder-framework/clowder2/blob/main/backend/Pipfile) - + contains all libraries specific to backend + +#### 3.1.3 Install libraries using pipenv + +Below are the steps to set up **backend specific libraries**: + +In the PyCharm Terminal you could try below commands to install +dependencies for Clowder v2: + +1. `cd backend` → setup backend environment + +2. `pipenv install` → installs all dependencies + +![](./devsetupmedia/image6.png) + +3. `pipenv install --dev` → installs all dev dependencies + +4. `pipenv uninstall ` → uninstalls the particular + dependency + +5. `pipenv uninstall ---all` → uninstalls all dependencies + +6. To add/update the dependencies version, make sure to update + **Pipfile** and run pipenv install + +7. For a clean restart, delete **Pipfile.lock** file and run pipenv + install + +#### 3.1.4 Connect virtual environment in PyCharm + +Go to File \> Settings (or PyCharm \> Settings on macOS) \> Project \> Python Interpreter +> +![](./devsetupmedia/image46.png) + +"Add Interpreter" → "Add Local Interpreter..." +![](./devsetupmedia/image27.png) + +Pick Virtualenv Environment → Pick Existing → **Click the ...** +![](./devsetupmedia/image35.png) + +Find the virtual environment python path. + +E.g. My virtual environment python path lives at: + +*/Users/cwang138/.virtualenvs/backend-vtWdOmS7/bin/python* +![](./devsetupmedia/image42.png) + +### 3.2 Install Frontend Dependencies + +#### 3.2.1 Install Node (and Npm). + +Download +[https://nodejs.org/en/download/package-manager](https://nodejs.org/en/download/package-manager) + +We recommend using Node v16.15 LTS. + +![](./devsetupmedia/image41.png) + +#### 3.2.2 Npm install dependencies + +[npm](https://www.npmjs.com/) stands for Node Package +Manager. It\'s a package manager for JavaScript, and it\'s the default +package manager for the Node.js runtime environment. It helps developers +manage and share reusable code packages (modules), making it easier to +install, remove libraries and dependencies in your projects. + +The libraries and dependencies are specified in **package.json** and the +exact versions are locked down using **package_lock.json.** In addition +to libraries, you can define scripts under the "script" field in +package.json. + +Here are the links to +[clowder2/frontend/package.json](https://github.com/clowder-framework/clowder2/blob/main/frontend/package.json) +and +[clowder2/frontend/package-lock.json](https://github.com/clowder-framework/clowder2/blob/main/frontend/package-lock.json) + +Useful commands to install the dependencies for Clowder frontend : + +1. `npm install` → installs all dependencies + +![](./devsetupmedia/image26.png) + +2. `npm install ` → install specific dependency + +3. `npm uninstall ` → uninstalls the particular dependency + +4. `npm uninstall` → uninstalls all dependencies + +5. To add/update the dependencies version, **make sure to update + package.json, and run npm install** + +6. For a clean restart, delete package-lock.json and node-modules, and + run npm install + +7. npm run codegen:v2:dev → run this whenever you make any changes in + backend classes/functions and that needs to be reflected in frontend + classes/functions + +> ![](./devsetupmedia/image1.png) + +## 4. Set up Docker + +[**Docker**](http://docker.com) is a platform that allows you to package +applications and their dependencies into portable, isolated units called +containers. + +- A docker image is like a blueprint for a container, containing the + application code, libraries, and environment settings needed to run + the application. + +- A docker container is the running instance of a Docker image, + providing an isolated environment where the application runs. + +![](./devsetupmedia/image47.png) + +### 4.1 Installing Docker + +- You will need to install docker first, this can be done by following + the instructions at: + + - [Install Docker Desktop on Mac](https://docs.docker.com/desktop/install/mac-install/) + + - [Install Docker Desktop on Windows](https://docs.docker.com/desktop/install/windows-install/) + +> If the docker desktop shows "Docker Desktop stopped....", you might +> have to install wsl (windows subsystem linux). This is done using: wsl +> --install --distribution ubuntu + +- [Install Docker Desktop on + Linux](https://docs.docker.com/desktop/install/linux-install/) + + +- Once you have installed docker, please see below to test it to make + sure it works as expected. + +#### *4.1.1 Troubleshooting Docker on Windows (Skip if not applicable)* + +*When installing Docker for Windows, please make sure that you have all +updates installed. When running any of the docker commands, I\'d +recommend using the powershell (no need for administrator). Main reason +is that I will use it to get the current working directory, under the +MS-DOS command you will need to use instead.* + +*1. install wsl from powershell with Admin rights: wsl --install and +reboot your system this will install ubuntu as well after the reboot\ +this can be removed using wsl --unregister ubuntu* + +*2. update wsl `wsl -update` and restart wsl `wsl --shutdown`* + +*3. check your version `wsl --status`* + +#### 4.1.2 Testing Docker Installation + +To test your installation you should be able to use the following from +the command line: + +docker run docker/whalesay cowsay Hello World + +This should generate the following output: + +![](./devsetupmedia/image28.png) + +#### 4.1.3 Useful Docker Command Line + +The docker application takes a second argument that is indicates what to +do: + +- `docker run` → creates a container from an image and starts the + default application in the container. The run command takes + additional flags, please look + [here](https://docs.docker.com/reference/cli/docker/container/run/) + for details.. + +- `docker images` → are what is used to start a container, these are + pulled from a server and contain everything needed to run the application (think a zip file of a linux + filesystem). + +- `ls` : shows a list of all images you have downloaded. + +- `rm` : will remove an image (you can always download it again). + +- `docker pull` → will download an image without running it, this is + also used to make sure you have the latest image downloaded. + +- `docker logs` → shows the output from a container, you can do this + after the container is finished, or if the container runs in the + background allows you to see the output of the container (think of + the tail command in unix). + +- `docker rm` → removes a stopped container, this is a cleanup process. + You will free up the disk space that the stopped container takes up. + +- `docker stop` → stops a container that is running. + +### 4.2 Docker Compose Explained + +[Docker Compose](https://docs.docker.com/compose/) is a +tool for defining and running multi-container applications. It is the +key to unlocking a streamlined and efficient development and deployment +experience. A +[docker-compose.yam](https://github.com/clowder-framework/clowder2/blob/main/docker-compose.yml)l +file is used to define and manage multi-container Docker applications as +a single service. This file is mainly for production. + +#### 4.2.1 For Development + +We have a separate +[docker-compose.dev.yml](https://github.com/clowder-framework/clowder2/blob/main/docker-compose.dev.yml) +where we have separated the backend and frontend from the other docker +containers. We run the backend and frontend separately in PyCharm to +make the development process easy. + +#### 4.2.2 Run Clowder Docker Containers + +1. `docker-compose -f -p up + -d --build` → create and build all the containers + +2. `docker-compose -f -p down` → stops all containers + +**We have a +[script](https://github.com/clowder-framework/clowder2/blob/main/docker-dev.sh) +to make it easier:** + +**`./docker-dev.sh up` → internally runs the above command 1** + +**`./docker-dev.sh down` →internally runs the above command 2** + +### 4.3 View Containers in Docker Desktop + +![](./devsetupmedia/image11.png) + +### *4.4 Delete Storage and Restart (Optional)* + +- *If you ever need to delete all the storage data for the containers, + please delete all the respective volumes in the docker desktops (see + screenshots).* + +- *Alternatively, you can use the command line to run `docker-compose + -f -p down -v`* + +- *After volumes have been successfully removed, run the docker + compose up command to start over.* + +![](./devsetupmedia/image4.png) + +## 5. Run/Debug Clowder2 Full Stack + +By following these steps, you can start both the frontend and backend, +along with the necessary Docker containers, and run the Clowder 2 stacks +using PyCharm. + +### 5.1 Start Docker Dependencies + +Ensure that the necessary dependencies are running. This includes +**Elasticsearch, MongoDB, RabbitMQ, extractor heartbeat listeners, +and/or extractors**. Use the following command to start them + +`sh docker-dev.sh up` + +(Details please refer to previous section: [Run Clowder Docker +Containers](#run-clowder-docker-containers)) + +Verify that the dependencies are running by running: `docker ps` + +### 5.2 Run Clowder 2 Backend + +In PyCharm, navigate to the left panel where you will see the "Run" +menu (located between "Refactor" and "Tools"). This menu contains +all the configurations needed to run Clowder. + +**Select and Run Configuration:** On the right side of the PyCharm +interface, you will see the name of your selected run configuration. +Next to it, there are buttons for running the configuration. Ensure that +uvicorn is selected. The run button is represented by an arrow. + +![](./devsetupmedia/image29.png) + +To run the backend, choose **'uvicorn'** under the run/debug +configurations and choose run. + +![](./devsetupmedia/image13.png) + +### 5.3 Run Clowder 2 Frontend + +To run the front end, run the **'npm start:dev'** configuration. **After +that, Clowder2 should open up at localhost:3000 in your browser.\ +**![](./devsetupmedia/image2.png) + +### 5.4 Debugging using PyCharm + +#### 5.4.1 Debugging the Backend + +- Debugging the backend is almost the same as running. Simply choose + **debug** instead of run. **Set breakpoints** somewhere in the + backend code and PyCharm will stop at those breakpoints. + +#### 5.4.2 Debugging the Frontend + +- Make sure you have a running backend first + +- Make sure to run the frontend as instructed in the above section. + +- Additionally, set up "**Javascript Debug**" and run in "**Debug**" + mode + +- **This will open a new browser window.** In order to debug the front + end, you must use **the new browser** window that opens up. The + original one opened by the usual run config will not have any + effect. Then you can put breakpoints in the front end and will stop + when they are reached. + +![](./devsetupmedia/image3.png) + +Here is a screenshot of what you would see in the backend debug mode +when a breakpoint is reached. You can see the method that we are in +along with the values. + +![](./devsetupmedia/image12.png) + +Here\'s a look at what you would see when debugging the frontend +JavaScript. In this example, we\'re examining the \`**useEffect**\` +method within the main thread. The debugger shows the current +\`**datasetId**\` and \`**folderId**\`. + +![](./devsetupmedia/image16.png) + +## 6. Pre-commits + +Pre-commit hooks are used to identify and fix simple code issues before +submission to code review, allowing reviewers to focus on more +significant architectural changes. Clowder2 pre-commit hooks include +below automatic formatting and checks: + +- Python style checks and formatting using **black, isort, and + flake8**. + +- Custom script to create openapi.json file from the backend FastAPI + app. + +- Eslint and prettier for frontend code. + +### 6.1 Install Pre-commit Hooks + +- Run pip install pre-commit to install pre-commit. + +- Navigate to the **root directory of the repository** and run + pre-commit install to install the pre-commit hooks. + +### 6.2 Automatic Hook Execution + +- The hooks will run automatically on every commit. + +- If any of the hooks fail, the commit will be aborted. + +- Hooks are stored in the .git/hooks directory and will be in place + for all future commits and branches. + +![](./devsetupmedia/image50.png) + +### 6.3 Manual Hook Execution + +- To run the hooks manually, use `pre-commit run --all-files` in the + command line. You should see a list of checks. If any fail, please + investigate the issues and fix them accordingly. + +![](./devsetupmedia/image31.png) + +### 6.4 Skipping Hooks (Not recommended) + +- To skip the hooks, use `git commit --no-verify` or in PyCharm, click + on the cog icon in the PyCharm commit interface and uncheck \"Run + Git hooks\". + +## 7. Optional + +### 7.1 Visual Studio Code (VS Code) + +Feel free to use Visual Studio Code if you're already familiar with it. +Like Pycharm, VS Code is a free, lightweight, and powerful code editor. +It supports a wide range of programming languages and comes with +features like syntax highlighting, debugging, version control +integration, and extensions to enhance functionality. If you haven\'t +already, download and install Visual Studio Code from the [official +website](https://code.visualstudio.com/). + +### 7.2 Setup Github CoPilot with VS Code + +If you have [Github +CoPilot](https://github.com/features/copilot/), you can +integrate it with VS Code and it can help you generate a lot of the +coding. + +#### 7.2.1 Step 1: Install GitHub Copilot Extension + +- Open VS Code. + +- Go to Extensions View: + + - Click on the Extensions icon in the Activity Bar on the side of + the window, or press Ctrl+Shift+X (Windows/Linux) or Cmd+Shift+X + (macOS). + +> ![](./devsetupmedia/image33.png) + +- Search for GitHub Copilot: + + - In the search bar, type GitHub Copilot. + +- Install the Extension: + + - Click on the Install button next to the GitHub Copilot + extension. + +![](./devsetupmedia/image22.png) + +#### 7.2.2 Step 2: Sign in to GitHub + +1. Open Command Palette: + +- Press Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS) to open + the Command Palette. + +![](./devsetupmedia/image37.png) + +2. Sign In: + +- Type GitHub: Sign in and select the option. + +- Follow the prompts to sign in with your GitHub account. + +#### 7.2.3 Step 3: Enable GitHub Copilot + +1. Open Command Palette: + + - Press Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS). + +2. Enable Copilot: + + - Type GitHub Copilot: Enable and select the option. + +#### 7.2.4 Step 4: Start Using GitHub Copilot + +- Open a file: Open a file in a supported language (e.g., Python, + JavaScript). + +- Begin typing: As you type, GitHub Copilot will provide suggestions. + You can accept suggestions by pressing Tabor Enter. + +> ![](./devsetupmedia/image39.png) +> +> ![](./devsetupmedia/image8.png) +> +> ![](./devsetupmedia/image45.png) + +### 7.3 Browser inspect mode + +The inspect mode gives access to a list of tools that collectively help +developers diagnose and fix issues, optimize performance, and ensure the +best user experience. Let's take Chrome for example: + +#### 7.3.1 Accessing Inspect Mode + +- Right-click on a webpage and select \"Inspect\" from the context + menu. + +- Alternatively, press `Ctrl+Shift+I` (Windows/Linux) or + `Cmd+Option+I` (Mac). + +![](./devsetupmedia/image48.png) + +#### 7.3.2 Elements Tab + +- View and modify the HTML structure and CSS styles of a webpage. + +- Hover over elements to see their dimensions and applied styles. + +- Edit HTML or CSS directly and see changes in real-time. + +#### 7.3.3 Console Tab + +- Useful for debugging scripts and interacting with the page's + JavaScript context. + +#### 7.3.4 Network Tab + +- Monitor network activity and inspect requests and responses. + +- Filter requests by type (e.g., FETCH/XHR, JS, CSS, Images) to focus + on specific resources. + +- Click each entry to view detailed information about each request, + such as headers, payload, and timing. diff --git a/docs/docs/devs/devsetupmedia/image1.png b/docs/docs/devs/devsetupmedia/image1.png new file mode 100644 index 000000000..72653089d Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image1.png differ diff --git a/docs/docs/devs/devsetupmedia/image10.png b/docs/docs/devs/devsetupmedia/image10.png new file mode 100644 index 000000000..4c8fe4843 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image10.png differ diff --git a/docs/docs/devs/devsetupmedia/image11.png b/docs/docs/devs/devsetupmedia/image11.png new file mode 100644 index 000000000..f5c3ca9d4 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image11.png differ diff --git a/docs/docs/devs/devsetupmedia/image12.png b/docs/docs/devs/devsetupmedia/image12.png new file mode 100644 index 000000000..6899e9435 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image12.png differ diff --git a/docs/docs/devs/devsetupmedia/image13.png b/docs/docs/devs/devsetupmedia/image13.png new file mode 100644 index 000000000..04b057611 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image13.png differ diff --git a/docs/docs/devs/devsetupmedia/image14.png b/docs/docs/devs/devsetupmedia/image14.png new file mode 100644 index 000000000..fb80ee90c Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image14.png differ diff --git a/docs/docs/devs/devsetupmedia/image15.png b/docs/docs/devs/devsetupmedia/image15.png new file mode 100644 index 000000000..374af4f30 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image15.png differ diff --git a/docs/docs/devs/devsetupmedia/image16.png b/docs/docs/devs/devsetupmedia/image16.png new file mode 100644 index 000000000..12ed56ffe Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image16.png differ diff --git a/docs/docs/devs/devsetupmedia/image17.png b/docs/docs/devs/devsetupmedia/image17.png new file mode 100644 index 000000000..03d68f906 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image17.png differ diff --git a/docs/docs/devs/devsetupmedia/image18.png b/docs/docs/devs/devsetupmedia/image18.png new file mode 100644 index 000000000..7cb3a3953 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image18.png differ diff --git a/docs/docs/devs/devsetupmedia/image19.png b/docs/docs/devs/devsetupmedia/image19.png new file mode 100644 index 000000000..2d33e2350 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image19.png differ diff --git a/docs/docs/devs/devsetupmedia/image2.png b/docs/docs/devs/devsetupmedia/image2.png new file mode 100644 index 000000000..1a3b7deec Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image2.png differ diff --git a/docs/docs/devs/devsetupmedia/image20.png b/docs/docs/devs/devsetupmedia/image20.png new file mode 100644 index 000000000..1c2cffd3c Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image20.png differ diff --git a/docs/docs/devs/devsetupmedia/image21.png b/docs/docs/devs/devsetupmedia/image21.png new file mode 100644 index 000000000..ebbb2d9d2 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image21.png differ diff --git a/docs/docs/devs/devsetupmedia/image22.png b/docs/docs/devs/devsetupmedia/image22.png new file mode 100644 index 000000000..c123ef9be Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image22.png differ diff --git a/docs/docs/devs/devsetupmedia/image23.png b/docs/docs/devs/devsetupmedia/image23.png new file mode 100644 index 000000000..62fbfa65f Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image23.png differ diff --git a/docs/docs/devs/devsetupmedia/image24.png b/docs/docs/devs/devsetupmedia/image24.png new file mode 100644 index 000000000..63264bceb Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image24.png differ diff --git a/docs/docs/devs/devsetupmedia/image25.png b/docs/docs/devs/devsetupmedia/image25.png new file mode 100644 index 000000000..7cd668f70 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image25.png differ diff --git a/docs/docs/devs/devsetupmedia/image26.png b/docs/docs/devs/devsetupmedia/image26.png new file mode 100644 index 000000000..e09cae279 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image26.png differ diff --git a/docs/docs/devs/devsetupmedia/image27.png b/docs/docs/devs/devsetupmedia/image27.png new file mode 100644 index 000000000..537377fc0 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image27.png differ diff --git a/docs/docs/devs/devsetupmedia/image28.png b/docs/docs/devs/devsetupmedia/image28.png new file mode 100644 index 000000000..47482eb8e Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image28.png differ diff --git a/docs/docs/devs/devsetupmedia/image29.png b/docs/docs/devs/devsetupmedia/image29.png new file mode 100644 index 000000000..994004d33 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image29.png differ diff --git a/docs/docs/devs/devsetupmedia/image3.png b/docs/docs/devs/devsetupmedia/image3.png new file mode 100644 index 000000000..4196f9346 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image3.png differ diff --git a/docs/docs/devs/devsetupmedia/image30.png b/docs/docs/devs/devsetupmedia/image30.png new file mode 100644 index 000000000..56123bef9 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image30.png differ diff --git a/docs/docs/devs/devsetupmedia/image31.png b/docs/docs/devs/devsetupmedia/image31.png new file mode 100644 index 000000000..865ce2ddf Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image31.png differ diff --git a/docs/docs/devs/devsetupmedia/image32.png b/docs/docs/devs/devsetupmedia/image32.png new file mode 100644 index 000000000..599bf53e7 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image32.png differ diff --git a/docs/docs/devs/devsetupmedia/image33.png b/docs/docs/devs/devsetupmedia/image33.png new file mode 100644 index 000000000..323a50565 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image33.png differ diff --git a/docs/docs/devs/devsetupmedia/image34.png b/docs/docs/devs/devsetupmedia/image34.png new file mode 100644 index 000000000..36f28378b Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image34.png differ diff --git a/docs/docs/devs/devsetupmedia/image35.png b/docs/docs/devs/devsetupmedia/image35.png new file mode 100644 index 000000000..f2a3a39bb Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image35.png differ diff --git a/docs/docs/devs/devsetupmedia/image36.png b/docs/docs/devs/devsetupmedia/image36.png new file mode 100644 index 000000000..640263845 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image36.png differ diff --git a/docs/docs/devs/devsetupmedia/image37.png b/docs/docs/devs/devsetupmedia/image37.png new file mode 100644 index 000000000..acf381455 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image37.png differ diff --git a/docs/docs/devs/devsetupmedia/image38.png b/docs/docs/devs/devsetupmedia/image38.png new file mode 100644 index 000000000..806a8b7b3 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image38.png differ diff --git a/docs/docs/devs/devsetupmedia/image39.png b/docs/docs/devs/devsetupmedia/image39.png new file mode 100644 index 000000000..ebf3da3c0 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image39.png differ diff --git a/docs/docs/devs/devsetupmedia/image4.png b/docs/docs/devs/devsetupmedia/image4.png new file mode 100644 index 000000000..d654ab1f0 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image4.png differ diff --git a/docs/docs/devs/devsetupmedia/image40.png b/docs/docs/devs/devsetupmedia/image40.png new file mode 100644 index 000000000..1098b4d51 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image40.png differ diff --git a/docs/docs/devs/devsetupmedia/image41.png b/docs/docs/devs/devsetupmedia/image41.png new file mode 100644 index 000000000..79951afc5 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image41.png differ diff --git a/docs/docs/devs/devsetupmedia/image42.png b/docs/docs/devs/devsetupmedia/image42.png new file mode 100644 index 000000000..d47c0401d Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image42.png differ diff --git a/docs/docs/devs/devsetupmedia/image43.png b/docs/docs/devs/devsetupmedia/image43.png new file mode 100644 index 000000000..675ca030d Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image43.png differ diff --git a/docs/docs/devs/devsetupmedia/image44.png b/docs/docs/devs/devsetupmedia/image44.png new file mode 100644 index 000000000..25a5515df Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image44.png differ diff --git a/docs/docs/devs/devsetupmedia/image45.png b/docs/docs/devs/devsetupmedia/image45.png new file mode 100644 index 000000000..128f45a88 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image45.png differ diff --git a/docs/docs/devs/devsetupmedia/image46.png b/docs/docs/devs/devsetupmedia/image46.png new file mode 100644 index 000000000..90df72f43 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image46.png differ diff --git a/docs/docs/devs/devsetupmedia/image47.png b/docs/docs/devs/devsetupmedia/image47.png new file mode 100644 index 000000000..34bfbfa6b Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image47.png differ diff --git a/docs/docs/devs/devsetupmedia/image48.png b/docs/docs/devs/devsetupmedia/image48.png new file mode 100644 index 000000000..45f94c333 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image48.png differ diff --git a/docs/docs/devs/devsetupmedia/image49.png b/docs/docs/devs/devsetupmedia/image49.png new file mode 100644 index 000000000..ecad2bbee Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image49.png differ diff --git a/docs/docs/devs/devsetupmedia/image5.png b/docs/docs/devs/devsetupmedia/image5.png new file mode 100644 index 000000000..893ee0de1 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image5.png differ diff --git a/docs/docs/devs/devsetupmedia/image50.png b/docs/docs/devs/devsetupmedia/image50.png new file mode 100644 index 000000000..83e3ea9b3 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image50.png differ diff --git a/docs/docs/devs/devsetupmedia/image51.png b/docs/docs/devs/devsetupmedia/image51.png new file mode 100644 index 000000000..2d42dc34d Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image51.png differ diff --git a/docs/docs/devs/devsetupmedia/image6.png b/docs/docs/devs/devsetupmedia/image6.png new file mode 100644 index 000000000..f3a10c03f Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image6.png differ diff --git a/docs/docs/devs/devsetupmedia/image7.png b/docs/docs/devs/devsetupmedia/image7.png new file mode 100644 index 000000000..a820180e9 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image7.png differ diff --git a/docs/docs/devs/devsetupmedia/image8.png b/docs/docs/devs/devsetupmedia/image8.png new file mode 100644 index 000000000..b99976e00 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image8.png differ diff --git a/docs/docs/devs/devsetupmedia/image9.png b/docs/docs/devs/devsetupmedia/image9.png new file mode 100644 index 000000000..d523166e4 Binary files /dev/null and b/docs/docs/devs/devsetupmedia/image9.png differ diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 9cabe4e1e..7f88ca003 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -8,16 +8,20 @@ nav: - Visualizations: users/visualizations.md - Developers: - Getting started: devs/getstarted.md + - Development Environment Setup: devs/devsetup.md + - Development Handbook: devs/devhandbook.md - Architecture: devs/architecture.md - Frontend: - devs/frontend/lazyloading.md - Keycloak: devs/keycloak.md - - Metadata: devs/metadata.md - Mongo: devs/mongo-fastapi.md - PyCharm: devs/pycharm.md - Elasticsearch: devs/elasticsearch.md - - Listeners: devs/listeners.md - Standalone Docker Deployment: devs/standalone-docker-deployment.md + - Design Docs: + - Listeners: devs/listeners.md + - Metadata: devs/metadata.md + - Dataset Versioning: devs/dataset-versioning.md - Proposals: - Storage: devs/storage.md - Admins: