KVStore Tools

README.md

@@ -125,6 +125,7 @@ This section contains additional reference documentation.
 
 Note: Any task with an **adhoc** prefix means that it can be used independently as a `deployment_task` in a playbook. You can use the tasks to resolve various Splunk problems or perform one-time activities, such as decommissioning an indexer from an indexer cluster.
 
+- **adhoc_backup_kvstore.yml** - Backup your KVStore to a given point - use the var `archive_name` to specify a tar name other than the default.
 - **adhoc_clean_dispatch.yml** - This task is intended to be used for restoring service to search heads should the dispatch directory become full. You should not need to use this task in a healthy environment, but it is at your disposal should the need arise. The task will stop splunk, remove all files in the dispatch directory, and then start splunk.
 - **adhoc_configure_hostname** - Configure a Splunk server's hostname using the value from inventory_hostname. It configures the system hostname, serverName in server.conf and host in inputs.conf. All Splunk configuration changes are made using the ini_file module, which will preserve any other existing configurations that may exist in server.conf and/or inputs.conf.
 - **adhoc_decom_indexer.yml** - Executes a splunk offline --enforce-counts command. This is useful when decommissioning one or more indexers from an indexer cluster.
@@ -141,6 +142,7 @@ Note: Any task with an **adhoc** prefix means that it can be used independently
 - **configure_idxc_manager.yml** - Configures a Splunk host to act as a manager node using `splunk_idxc_rf`, `splunk_idxc_sf`, `splunk_idxc_key`, and `splunk_idxc_label`.
 - **configure_idxc_member.yml** - Configures a Splunk host as an indexer cluster member using `splunk_uri_cm`, `splunk_idxc_rep_port`, and `splunk_idxc_key`.
 - **configure_idxc_sh.yml** - Configures a search head to join an existing indexer cluster using `splunk_uri_cm` and `splunk_idxc_key`.
+- **configure_kvstore.yml** - Disables KVStore when disabled by `splunk_enable_kvstore` and sets vars related to KVStore in `server.conf` configured in the defaults, like `splunk_kvstore_storage` and `splunk_oplog_size`
 - **configure_license.yml** - Configure the license group to the `splunk_license_group` variable defined. Default is `Trial`. Available values are "Trial, Free, Enterprise, Forwarder, Manager or Peer. If set to `Peer`, the `splunk_uri_lm` must be defined. Note: This could also be accomplished using configure_apps.yml with a git repository.
 - **configure_os.yml** - Increases ulimits for the splunk user and disables Transparent Huge Pages (THP) per Splunk implementation best practices.
 - **configure_serverclass.yml** - Generates a new serverclass.conf file from the serverclass.conf.j2 template and installs it to $SPLUNK_HOME/etc/system/local/serverclass.conf.
@@ -160,6 +162,7 @@ Note: Any task with an **adhoc** prefix means that it can be used independently
 - **install_splunk.yml** - *Do not call install_splunk.yml directly! Use check_splunk.yml* - Called by check_splunk.yml to install/upgrade Splunk and Splunk Universal Forwarders, as well as perform any initial configurations. This task is called by check_splunk.yml when the check determines that Splunk is not currently installed. This task will create the splunk user and splunk group, configure the bash profile for the splunk user (by calling configure_bash.yml), configure THP and ulimits (by calling configure_os.ym), download and install the appropriate Splunk package (by calling download_and_unarchive.yml), configure a common splunk.secret (by calling configure_splunk_secret.yml, if configure_secret is defined), create a deploymentclient.conf file with the splunk_ds_uri and clientName (by calling configure_deploymentclient.yml, if clientName is defined), install a user-seed.conf with a prehashed admin password (if used_seed is defined), and will then call the post_install.yml task. See post_install.yml entry for details on post-installation tasks.
 - **install_utilities.yml** - Installs Linux packages that are useful for troubleshooting Splunk-related issues when `install_utilities: true` and `linux_packages` is defined with a list of packages to install.
 - **configure_dmesg.yml** - Some distros restrict access to read  `dmesg` for non-root users. This allows the `splunk` user to run the `dmesg` command. Defaults to `false`.
+- **kvstore_upgrade.yml** - Upgrades a KVStore storage backend and/or server version on either a single or distributed instance.
 - **main.yml** - This is the main task that will always be called when executing this role. This task sets the appropriate variables for full vs uf packages, sends a Slack notification about the play if the slack_token and slack_channel are defined, checks the current boot-start configuration to determine if it's in the expected state, and then includes the task from the role to execute against, as defined by the value of the deployment_task variable. The deployment_task variable should be defined in your playbook(s). Refer to the included example playbooks to see this in action.
 - **post_install.yml** - Executes post-installation tasks. Performs a touch on the .ui_login file which disables the first-time login prompt to change your password, ensures that `splunk_home` is owned by the correct user and group, and optionally configures three scripts to: cleanup crash logs and old diags (by calling add_crashlog_script.yml and add_diag_script.yml, respectively), and a pstack generation shell script for troubleshooting purposes (by calling add_pstack_script.yml). This task will install various Linux troubleshooting utilities (by calling install_utilities.yml) when `install_utilities: true`.
 - **set_maintenance_mode.yml** - Enables or disables maintenance mode on a cluster manager. Intended to be called by playbooks for indexer cluster upgrades/maintenance. Requires the `state` variable to be defined. Valid values: enabled, disabled

roles/splunk/defaults/main.yml

@@ -69,6 +69,10 @@ splunk_shc_target_group: shc
 splunk_shc_deployer: "{{ groups['shdeployer'] | first }}" # If you manage multiple SHCs, configure the var value in group_vars
 splunk_shc_uri_list: "{% for h in groups[splunk_shc_target_group] %}https://{{ hostvars[h].ansible_fqdn }}:{{ splunkd_port }}{% if not loop.last %},{% endif %}{% endfor %}" # If you manage multiple SHCs, configure the var value in group_vars
 start_splunk_handler_fired: false # Do not change; used to prevent unnecessary splunk restarts
+splunk_enable_kvstore: true
+splunk_kvstore_storage: undefined # Can be defined here or at the group_vars level - accepted values: "wiredTiger" or "undefined", which leaves as default
+splunk_kvstore_version: undefined # Can be defined here or at the group_vars level - accepted values: 4.2 or "undefined", which leaves as default1
+splunk_oplog_size: 1000 # Default for Splunk Enterprise - should be changed at the group_vars level only at the behest of Splunk support with special care taken
 # Linux and scripting related vars
 add_crashlog_script: false # Set to true to install a script and cron job to automatically cleanup splunk crash logs older than 7 days
 add_diag_script: false # Set to true to install a script and cron job to automatically cleanup splunk diag files older than 30 days

roles/splunk/tasks/adhoc_backup_kvstore.yml

@@ -0,0 +1,15 @@
+---
+- name: Backup KVStore on desired host
+  ansible.builtin.command: |
+    {{ splunk_home }}/bin/splunk backup kvstore {{ archive_name | default("") }}
+  become: true
+  become_user: "{{ splunk_nix_user }}"
+  register: splunk_kvstore_backup_out
+  changed_when: splunk_kvstore_backup_out.rc == 0
+  failed_when: splunk_kvstore_backup_out.rc != 0
+
+- name: Check that backup has finished
+  ansible.builtin.command: |
+    {{ splunk_home }}/bin/splunk splunk show kvstore-status | grep backupRestoreStatus | sed -r 's/\s+backupRestoreStatus : //g'
+  register: splunk_kvstore_status_out
+  until: "{{ splunk_kvstore_status_out.stdout }} == 'Ready'"

roles/splunk/tasks/configure_kvstore.yml

@@ -0,0 +1,28 @@
+---
+- name: Disable KVStore if specified
+  include_tasks: kvstore_disable.yml
+  when: not splunk_enable_kvstore
+
+- name: Configure initial KVStore storage engine in server.conf
+  community.general.ini_file:
+    path: "{{ splunk_home }}/etc/system/local/server.conf"
+    section: kvstore
+    option: storageEngine
+    value: "{{ splunk_kvstore_storage }}"
+    owner: "{{ splunk_nix_user }}"
+    group: "{{ splunk_nix_group }}"
+    mode: 0644
+  become: true
+  when:
+    - splunk_kvstore_storage == "wiredTiger"
+    - splunk_enable_kvstore
+
+- name: Configure initial KVStore oplog size in server.conf
+  community.general.ini_file:
+    path: "{{ splunk_home }}/etc/system/local/server.conf"
+    section: kvstore
+    option: oplogSize
+    value: "{{ splunk_oplog_size }}"
+  become: true
+  become_user: "{{ splunk_nix_user }}"
+  when: splunk_enable_kvstore

roles/splunk/tasks/kvstore_disable.yml

@@ -0,0 +1,11 @@
+---
+- name: Disable kvstore
+  community.general.ini_file:
+    path: "{{ splunk_home }}/etc/system/local/server.conf"
+    section: kvstore
+    option: disabled
+    value: "true"
+  become: true
+  become_user: "{{ splunk_nix_user }}"
+  when: not splunk_enable_kvstore
+  notify: restart splunk

roles/splunk/tasks/kvstore_upgrade.yml

@@ -0,0 +1,152 @@
+---
+- name: Set fact for kvstore Upgrade
+  ansible.builtin.set_fact:
+    splunk_upgrade_server: false
+
+- name: Check the current kvstore storage backend
+  ansible.builtin.command: |
+    {{ splunk_home }}/bin/splunk show kvstore-status --verbose | grep storageEngine | sed -r 's/\s+storageEngine : //g'
+  become: true
+  become_user: "{{ splunk_nix_user }}"
+  register: splunk_kvstore_backend_out
+  changed_when: splunk_kvstore_backend_out.rc == 0
+  failed_when: splunk_kvstore_backend_out.rc != 0
+
+- name: Check the current kvstore server version
+  ansible.builtin.command: |
+    {{ splunk_home }}/bin/splunk show kvstore-status --verbose | grep serverVersion | sed -r 's/\s+serverVersion : //g'
+  become: true
+  become_user: "{{ splunk_nix_user }}"
+  register: splunk_current_server_version_out
+  changed_when: splunk_current_server_version_out.rc == 0
+  failed_when: splunk_current_server_version_out.rc != 0
+
+- name: Debug print kvstore backend engine
+  ansible.builtin.debug:
+    var: splunk_kvstore_backend_out.stdout
+    verbosity: 2
+
+- name: Debug print kvstore backend engine
+  ansible.builtin.debug:
+    var: splunk_current_server_version_out.stdout
+    verbosity: 2
+
+- name: Upgrade KVstore if needed
+  block:
+    - name: Backup KVStore
+      include_tasks: adhoc_backup_kvstore.yml
+      vars:
+        - archive_name: "-archiveName preAnsibleVersionUpgradeBackup"
+
+    - name: Perform single-install upgrade steps
+      block:
+        - name: Perform < 9.0 migration steps
+          block:
+            - name: Define storage migration as true in server.conf
+              community.general.ini_file:
+                path: "{{ splunk_home }}/etc/system/local/server.conf"
+                section: kvstore
+                option: storageEngineMigration
+                value: "true"
+              become: true
+              become_user: "{{ splunk_nix_user }}"
+
+            - name: Start storage engine migration on single instance
+              ansible.builtin.command: |
+                {{ splunk_home }}/bin/splunk migrate kvstore-storage-engine --target-engine wiredTiger --enable-compression
+              register: splunk_migration_single_early_out
+              changed_when: splunk_migration_single_early_out.rc == 0
+              failed_when: splunk_migration_single_early_out.rc != 0
+              become: true
+              become_user: "{{ splunk_nix_user }}"
+          when: splunk_package_version is version(9.0, '<')
+
+        - name: Perform >= 9.0 upgrade if necessary
+          ansible.builtin.command: |
+            {{ splunk_home }}/bin/splunk migrate migrate-kvstore
+          register: splunk_migration_single_early_out
+          changed_when: splunk_migration_single_early_out.rc == 0
+          failed_when: splunk_migration_single_early_out.rc != 0
+          become: true
+          become_user: "{{ splunk_nix_user }}"
+          when:
+            - splunk_package_version is version(9.0, '>=')
+            - splunk_current_server_version_out.stdout is version(4.2 '<')
+      when: splunk_shc_target_group not in group_names
+
+    - name: Perform SHC KVStore upgrade
+      block:
+        - name: Perform SHC pre-migration Steps
+          ansible.builtin.command: |
+            {{ splunk_home }}/bin/splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun true
+          register: splunk_shc_pre_steps_out
+          changed_when: splunk_shc_pre_steps_out.rc == 0
+          failed_when: splunk_shc_pre_steps_out.rc != 0 
+          become: true
+          become_user: "{{ splunk_nix_user }}"
+
+        - name: Start Backend migration
+          ansible.builtin.command: |
+            {{ splunk_home }}/bin/splunk start-shcluster-migration kvstore -storageEngine wiredTiger -clusterPerc 50
+          register: splunk_shc_kvstore_backend_migration_out
+          changed_when: splunk_shc_kvstore_backend_migration_out.rc == 0
+          failed_when: splunk_shc_kvstore_backend_migration_out.rc != 0
+          become: true
+          become_user: "{{ splunk_nix_user }}"
+
+        - name: Make sure migration is successful
+          ansible.builtin.command: |
+            {{ splunk_home }}/bin/splunk show shcluster-kvmigration-status | sed -r 's/\s+migrationStatus : //g'
+          register: splunk_kvstore_migration_status_out
+          changed_when: splunk_kvstore_migration_status_out.rc == 0
+          failed_when: splunk_kvstore_migration_status_out.rc != 0
+          become: true
+          become_user: "{{ splunk_nix_user }}"
+          until: "{{ splunk_kvstore_migration_status_out.stdout }} == 'notStarted'"
+
+        - name: Perform SHC pre-upgrade steps
+          ansible.builtin.command: |
+            {{ splunk_home }}/bin/splunk start-shcluster-upgrade kvstore -version 4.2 -isDryRun true
+          register: splunk_kvstore_version_check_out
+          changed_when: splunk_kvstore_version_check_out.rc == 0
+          failed_when: splunk_kvstore_version_check_out.rc != 0
+          become: true
+          become_user: "{{ splunk_nix_user }}"
+
+        - name: Start Version upgrade
+          ansible.builtin.command: |
+            {{ splunk_home }}/bin/splunk start-shcluster-upgrade kvstore -version 4.2
+          register: splunk_kvstore_version_upgrade_out
+          changed_when: splunk_kvstore_version_upgrade_out.rc == 0
+          failed_when: splunk_kvstore_version_upgrade_out.rc != 0
+          become: true
+          become_user: "{{ splunk_nix_user }}"
+
+        - name: Make sure upgrade is successful
+          ansible.builtin.command: |
+            {{ splunk_home }}/bin/splunk show kvstore-status --verbose | sed -r 's/\s+serverVersion : //g'
+          register: splunk_kvstore_version_status_out
+          changed_when: splunk_kvstore_version_status_out.rc == 0
+          failed_when: splunk_kvstore_version_status_out.rc != 0
+          become: true
+          become_user: "{{ splunk_nix_user }}"
+          until: "{{ splunk_kvstore_version_status_out.stdout is version(4.2, '>=') }}"
+      when: splunk_shc_target_group in group_names
+
+    - name: Clean up older binaries from older versions
+      ansible.builtin.file:
+        path: "{{ splunk_home }}/bin/{{ item }}"
+        state: absent
+      loop:
+        - mongod-3.6
+        - mongod-4.0
+        - mongodump-3.6
+        - mongorestore-3.6
+      become: true
+      become_user: "{{ splunk_nix_user }}"
+  when: 
+    - splunk_enable_kvstore
+    - splunk_kvstore_storage == "wiredTiger"
+    - "'full' in group_names"
+    - splunk_kvstore_backend_out.stdout != "wiredTiger"
+    - splunk_current_server_version_out.stdout is version(4.2 '<')

roles/splunk/tasks/post_install.yml

@@ -30,3 +30,6 @@
 - name: Install additional utilities and troubleshooting tools
   include_tasks: install_utilities.yml
   when: install_utilities
+
+- name: Configure KVStore vars
+  include_tasks: configure_kvstore.yml