Extended IDX Cluster manager and member task, adjust readme

README.md

@@ -112,6 +112,7 @@ The following example playbooks have been included in this project for your refe
 - **splunk_app_install.yml** - Install or upgrade apps on Splunk hosts using the configure_apps.yml task in the splunk role. Note that the apps you want to deploy should be defined in either host_vars or group_vars, along with a splunk_app_deploy_path. Refer to the documentation for app deployment for details.
 - **splunk_install_or_upgrade.yml** - Install or upgrade Splunk (or Splunk UFs) on hosts using the check_splunk.yml task in the splunk role.
 - **splunk_shc_deploy.yml** - Installs Splunk and initializes search head clustering on a shdeployer and group of hosts that will serve as a new search head cluster.
+- **splunk_idx_deploy.yml** - Installs Splunk and initializes indexer clustering on cluster manager and indexers. It's possible to activate multisite cluster with variables.
 - **splunk_upgrade_full_stack.yml** - Example playbook that demonstrates how to upgrade an entire Splunk deployment with a single-site indexer cluster and a search head cluster using the splunk role. Note: This playbook does not upgrade forwarders, although you could easily add an extra play to do that.
 
 ## Extended Documentation
@@ -139,8 +140,8 @@ Note: Any task with an **adhoc** prefix means that it can be used independently
 - **configure_deploymentclient.yml** - Generates a new deploymentclient.conf file from the deploymentclient.conf.j2 template and installs it to $SPLUNK_HOME/etc/system/local/deploymentclient.conf. This task is included automatically during new installations when values have been configured for the `clientName` and `splunk_uri_ds` variables.
 - **configure_dmc.yml** - Configures the DMC as an Indexer Peer in SH mode, adds hosts to the host as search peers, and configures the host MC in auto mode
 - **configure_facl.yml** - Configure file system access control lists (FACLs) to allow the splunk user to read /var/log files and add the splunk user's group to /etc/audit/auditd.conf to read /var/log/audit/ directory. This allows the splunk user to read privileged files from a non-privileged system account. Note: This task is performed automatically during new installations when splunk is installed as a non-root user.
-- **configure_idxc_manager.yml** - Configures a Splunk host to act as a manager node using `splunk_idxc_rf`, `splunk_idxc_sf`, `splunk_idxc_key`, and `splunk_idxc_label`.
-- **configure_idxc_member.yml** - Configures a Splunk host as an indexer cluster member using `splunk_uri_cm`, `splunk_idxc_rep_port`, and `splunk_idxc_key`.
+- **configure_idxc_manager.yml** - Configures a Splunk host to act as a manager node using `splunk_idxc_rf`, `splunk_idxc_sf`, `splunk_idxc_key`, `splunk_idxc_label` and `splunk_idxc_multisite`. For multisite clusters are additional variables necessary like `splunk_idxc_available_sites`, `splunk_idxc_site_mappings`, `splunk_idxc_site_rf`, `splunk_idxc_site_sf` and `splunk_idxc_site_affinity`.
+- **configure_idxc_member.yml** - Configures a Splunk host as an indexer cluster member using `splunk_uri_cm`, `splunk_idxc_rep_port`, `splunk_idxc_key` and `splunk_idxc_site_affinity`.
 - **configure_idxc_sh.yml** - Configures a search head to join an existing indexer cluster using `splunk_uri_cm` and `splunk_idxc_key`.
 - **configure_license.yml** - Configure the license group to the `splunk_license_group` variable defined. Default is `Trial`. Available values are "Trial, Free, Enterprise, Forwarder, Manager or Peer. If set to `Peer`, the `splunk_uri_lm` must be defined. Note: This could also be accomplished using configure_apps.yml with a git repository.
 - **configure_os.yml** - Increases ulimits for the splunk user and disables Transparent Huge Pages (THP) per Splunk implementation best practices.

playbooks/splunk_idxc_deploy.yml

@@ -21,3 +21,10 @@
     - ../roles/splunk
   vars:
     - deployment_task: configure_idxc_member.yml
+
+- hosts:
+    - search
+  roles:
+    - ../roles/splunk
+  vars:
+    - deployment_task: configure_idxc_sh.yml

roles/splunk/defaults/main.yml

@@ -52,11 +52,17 @@ git_version: master # Configure default version to clone, overridable inside the
 app_relative_path: # set a sub-path you want to sync within a repo. If the repo contains multiple apps in the root directory, just set this to a trailing slash.
 splunk_app_deploy_path: undefined # Path under $SPLUNK_HOME/ to deploy apps to - Note that this may be set in group_vars, host_vars, playbook vars, or inside the git_apps dictionary within host_vars
 # IDXC Vars
-splunk_idxc_key: mypass4symmkey
+splunk_idxc_key: undefined # Use ansible-vault encrypt_string, e.g. ansible-vault encrypt_string --ask-vault-pass 'var_value_to_encrypt' --name 'var_name'
+splunk_idxc_multisite: false
 splunk_idxc_rf: 2
 splunk_idxc_sf: 2
+splunk_idxc_available_sites: "site1"
+splunk_idxc_site_mappings: "" # Default are no mappings
+splunk_idxc_site_rf: "origin:1,total:2"
+splunk_idxc_site_sf: "origin:1,total:2"
 splunk_idxc_rep_port: 9887
 splunk_idxc_label: myidxc
+splunk_idxc_site_affinity: "site0" # Default to site0 for Searchheads and Forwarders, and need to be set to != site0 on manager
 splunk_apply_cluster_bundle_retries: 3 # How many times to retry indexer cluster bundle apply if it fails
 splunk_apply_cluster_bundle_delay: 60 # Delay in seconds between retries to apply indexer cluster bundle
 splunk_apply_shcluster_bundle_retries: 3 # How many times to retry SHC bundle apply if it fails

roles/splunk/tasks/check_decrypted_secret.yml

@@ -5,6 +5,7 @@
   become: true
   become_user: "{{ splunk_nix_user }}"
   changed_when: false
+  check_mode: false
   no_log: true
 
 - name: "Decrypt {{ req_secret_option }} of {{ req_secret_conf }}.conf [{{ req_secret_section }}]"
@@ -13,5 +14,6 @@
   become: true
   no_log: true
   changed_when: false
+  check_mode: false
   when:
     - encrypted_secret_value.rc == 0 and encrypted_secret_value.stdout != ""

roles/splunk/tasks/configure_idxc_manager.yml

@@ -8,7 +8,19 @@
 # https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Configuremanagerwithserverconf
 - name: Setting clustering mode based on Splunk version number
   set_fact:
-    mode_value: "{% if splunk_version_release | float < 8.1 %}master{% else %}manager{% endif %}"
+    mode_value: "{{ splunk_version_release is version('8.1', '<') | ternary('master', 'manager') }}"
+
+- name: Setting site affinity of cluster manager if not set
+  set_fact:
+    # This is just a fallback method, if someone forgot to set the right site for the cluster manager, as site0 is default
+    site_affinity: "{% if splunk_idxc_multisite and splunk_idxc_site_affinity == 'site0' %}site1{% else %}{{ splunk_idxc_site_affinity }}{% endif %}"
+
+- name: Extract encrypted value
+  include_tasks: check_decrypted_secret.yml
+  vars:
+    req_secret_conf: server
+    req_secret_section: clustering
+    req_secret_option: pass4SymmKey
 
 - name: Configure clustering stanza for cluster manager node
   ini_file:
@@ -25,8 +37,51 @@
     - wait for splunkd
   no_log: true
   loop:
-    - { option: "mode", value: "{{ mode_value}}" }
+    - { option: "mode", value: "{{ mode_value }}" }
     - { option: "replication_factor", value: "{{ splunk_idxc_rf }}" }
     - { option: "search_factor", value: "{{ splunk_idxc_sf }}" }
-    - { option: "pass4SymmKey", value: "{{ splunk_idxc_key }}" }
     - { option: "cluster_label", value: "{{ splunk_idxc_label }}" }
+    - { option: "multisite", value: "{{ splunk_idxc_multisite | ternary('true', 'false') }}" }
+    - { option: "available_sites", value: "{{ splunk_idxc_available_sites }}" }
+    - { option: "site_replication_factor", value: "{{ splunk_idxc_site_rf }}" }
+    - { option: "site_search_factor", value: "{{ splunk_idxc_site_sf }}" }
+    - { option: "site_mappings", value: "{{ splunk_idxc_site_mappings }}" }
+
+- name: Configure clustering stanza psk for cluster manager node
+  ini_file:
+    path: "{{ splunk_home }}/etc/system/local/server.conf"
+    section: clustering
+    option: "{{ item.option }}"
+    value: "{{ item.value }}"
+    mode: 0644
+    owner: "{{ splunk_nix_user }}"
+    group: "{{ splunk_nix_group }}"
+  become: true
+  when:
+    - splunk_idxc_key != "undefined"
+    - encrypted_secret_value.stdout == "" or (splunk_idxc_key != decrypted_secret_value.stdout | default(''))
+  notify: 
+    - restart splunk
+    - wait for splunkd
+  no_log: true
+  loop:
+    - { option: "pass4SymmKey", value: "{{ splunk_idxc_key }}" }
+
+- name: Configure general stanza on manager node
+  ini_file:
+    path: "{{ splunk_home }}/etc/system/local/server.conf"
+    section: general
+    option: "{{ item.option }}"
+    value: "{{ item.value }}"
+    mode: 0644
+    owner: "{{ splunk_nix_user }}"
+    group: "{{ splunk_nix_group }}"
+  become: true
+  notify: 
+    - restart splunk
+    - wait for splunkd
+  no_log: true
+  when:
+    - splunk_idxc_multisite
+  loop:
+    - { option: "site", value: "{{ site_affinity }}" }

roles/splunk/tasks/configure_idxc_member.yml

@@ -8,16 +8,19 @@
 # https://docs.splunk.com/Documentation/Splunk/8.0.9/Indexer/ConfigurepeerswithCLI
 - name: Setting clustering mode based on Splunk version number
   set_fact:
-    mode_value: "{% if splunk_version_release | float < 8.1 %}slave{% else %}peer{% endif %}"
+    mode_value: "{{ splunk_version_release is version('8.1', '<') | ternary('slave', 'peer') }}"
+    manager_uri_call: "{{ splunk_version_release is version('8.1', '<') | ternary('master_uri', 'manager_uri') }}"
 
 - name: Configure idxc member
-  command: "{{ splunk_home }}/bin/splunk edit cluster-config -mode {{ mode_value }} -auth {{ splunk_auth }} -master_uri {{ splunk_uri_cm }} -replication_port {{ splunk_idxc_rep_port }} -secret {{ splunk_idxc_key }}"
+  command: "{{ splunk_home }}/bin/splunk edit cluster-config -mode {{ mode_value }} {% if splunk_idxc_multisite %}-site {{ splunk_idxc_site_affinity }}{% endif %} -auth {{ splunk_auth }} -{{ manager_uri_call }} {{ splunk_uri_cm }} -replication_port {{ splunk_idxc_rep_port }} -secret {{ splunk_idxc_key }}"
   become: true
   become_user: "{{ splunk_nix_user }}"
   register: idxc_peer_init_result
   changed_when: idxc_peer_init_result.rc == 0
   failed_when: idxc_peer_init_result.rc != 0
-  notify: restart splunk
+  notify:
+    - restart splunk
+    - wait for splunkd
   no_log: true
   until: idxc_peer_init_result.rc == 0
   retries: 6

roles/splunk/tasks/configure_idxc_sh.yml

@@ -1,13 +1,38 @@
 ---
+- name: Run splunk version command to check currently installed version
+  include_tasks: check_splunk_version.yml
+
+- name: Setting clustering mode based on Splunk version number
+  set_fact:
+    manager_uri_call: "{{ splunk_version_release is version('8.1', '<') | ternary('master_uri', 'manager_uri') }}"
+    manager_command_call: "{{ splunk_version_release is version('8.1', '<') | ternary('cluster-master', 'cluster-manager') }}"
+
+# TODO: Maybe switch to config file, this is the cleaner solution and would work with multiple indexer clusters
 - name: Configure search head to join indexer cluster
-  command: "{{ splunk_home }}/bin/splunk edit cluster-config -mode searchhead -master_uri {{ splunk_uri_cm }} -secret {{ splunk_idxc_key }} -auth {{ splunk_auth }}"
+  command: "{{ splunk_home }}/bin/splunk edit cluster-config -mode searchhead -{{ manager_uri_call }} {{ splunk_uri_cm }} -secret {{ splunk_idxc_key }} -auth {{ splunk_auth }}"
   become: true
   become_user: "{{ splunk_nix_user }}"
   register: idxc_sh_join_result
   changed_when: idxc_sh_join_result.rc == 0
   failed_when: idxc_sh_join_result.rc != 0
-  notify: restart splunk
+  notify:
+    - restart splunk
+    - wait for splunkd
   no_log: true
   until: idxc_sh_join_result.rc == 0
   retries: 6
   delay: 5
+
+# Error Code 22, means no change in current configuration
+- name: Configure search head site affinity
+  command: "{{ splunk_home }}/bin/splunk edit {{ manager_command_call }} {{ splunk_uri_cm }} -site {{ splunk_idxc_site_affinity }} -auth {{ splunk_auth }}"
+  when: splunk_idxc_multisite
+  become: true
+  become_user: "{{ splunk_nix_user }}"
+  register: idxc_sh_join_result
+  changed_when: idxc_sh_join_result.rc == 0
+  failed_when: idxc_sh_join_result.rc != 0 and idxc_sh_join_result.rc != 22
+  no_log: true
+  until: idxc_sh_join_result.rc == 0 or idxc_sh_join_result.rc == 22
+  retries: 6
+  delay: 5