add customization and walkthrough

docs/docs/how-tos/monitoring.md

@@ -16,21 +16,51 @@ monitoring:
 
 ### Grafana
 
-Grafana is a leading open-source platform for monitoring and observability. It provides rich visualization
-tools and dashboards for analyzing and monitoring metrics from various data sources.
+[Grafana](https://grafana.com/) is a leading open-source platform for monitoring and observability.
+It provides rich visualization tools and dashboards for analyzing and monitoring metrics from various data sources.
 
 ### Prometheus
 
-Prometheus is a popular open-source monitoring and alerting toolkit. It collects metrics from configured targets,
-stores them efficiently, and allows querying them in real-time.
+[Prometheus](https://prometheus.io/) is a popular open-source monitoring and alerting toolkit. It collects
+metrics from configured targets, stores them efficiently, and allows querying them in real-time.
 
 ### Loki
 
-Loki is a horizontally-scalable, highly available, multi-tenant log aggregation system inspired by Prometheus.
-It is designed to be very cost-effective and easy to operate, as it does not index the contents of the logs,
-but rather a set of labels for each log stream.
+[Loki](https://grafana.com/docs/loki/latest/) is a horizontally-scalable, highly available, multi-tenant log
+aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it
+does not index the contents of the logs, but rather a set of labels for each log stream. Below is a step-by-step
+walkthrough of how to view JupyterHub pod logs in Grafana Loki:
 
-#### Deployment Configuration:
+Go to explore section of monitoring of your Nebari installation at:
+
+https://{your-nebari-domain}/monitoring/
+
+![Grafana Explore Page](/img/how-tos/1_grafana-explore.png)
+
+Select Loki Data source at the top:
+
+![Grafana Select Loki](/img/how-tos/2_grafana-select-loki.png)
+
+Click on the Log browser and select labels to search in, in this case we have selected pod:
+
+![Grafana Select Loki](/img/how-tos/3_grafana-log-browser-pod.png)
+
+Type the pod name initials for the pod you're looking to search logs for, in this case we are looking
+for hub pod:
+
+![Grafana Select Loki](/img/how-tos/4_grafana-log-search-pod.png)
+
+Select the pod from the list of pods and then click on "Show logs":
+
+![Grafana Select Loki](/img/how-tos/5_grafana-log-select-pod.png)
+
+After clicking on "Show logs", you should be able to see logs for JupyterHub pod as shown below:
+
+![Grafana Select Loki](/img/how-tos/6_grafana-view-pod-logs.png)
+
+You can also filter by time, by clicking on the time filter on top right, next to "Run query".
+
+## Terraform Overrides
 
 Nebari provides its users with the ability to customize the deployment of various component
 and Loki is one of them. Loki deployment is made up of three fundamental components:
@@ -40,8 +70,6 @@ and Loki is one of them. Loki deployment is made up of three fundamental compone
 - MinIO: a Kubernetes-native high-performance object storage server which is designed for large-scale
   private cloud infrastructure and compatible with Amazon S3.
 
-## Terraform Overrides
-
 ```yaml
 monitoring:
   enabled: true
@@ -50,3 +78,52 @@ monitoring:
     promtail: <PROMTAIL-HELM-CHART-VALUES-OVERRIDE>
     minio: <MINIO-HELM-CHART-VALUES-OVERRIDE>
 ```
+
+Below are some examples of customizing your loki deployment via terraform overrides:
+
+### Log Storage
+
+You can customize the size of persistent volume for logs storage, by setting up the value for
+`minio.persistence.size` as shown below:
+
+```yaml
+monitoring:
+enabled: true
+overrides:
+  minio:
+    persistence:
+      size: 100Gi
+```
+
+### Log Retention Period
+
+Since the storage for MinIO is predefined and is not infinitely scalable out-of-the-box like
+cloud storage solutions like S3, GCS, etc, unless you keep increasing minio persistent volume size.
+It is a good idea to clear up old logs to make space for new ones, Nebari does this by default with
+60 days retention period. You can customize the log retention period by overriding the default
+value as follows:
+
+```yaml
+monitoring:
+enabled: true
+overrides:
+  loki:
+    loki:
+      limits_config:
+        # The minimum retention period is 24h.
+        retention_period: 90d
+```
+
+This may not be a suitable solution for many cases, like if your organization needs to keep
+all the logs forever, then you may want to use a very large persistent volume and disable
+deletion of old logs. You can disable deletion via:
+
+```yaml
+monitoring:
+enabled: true
+overrides:
+  loki:
+    loki:
+      compactor:
+        retention_enabled: false
+```