Actions check for static primary feature

docs/SETTINGS.rst

@@ -17,8 +17,9 @@ Dynamic configuration is stored in the DCS (Distributed Configuration Store) and
 -  **maximum\_lag\_on\_failover**: the maximum bytes a follower may lag to be able to participate in leader election.
 -  **maximum\_lag\_on\_syncnode**: the maximum bytes a synchronous follower may lag before it is considered as an unhealthy candidate and swapped by healthy asynchronous follower. Patroni utilize the max replica lsn if there is more than one follower, otherwise it will use leader's current wal lsn. Default is -1, Patroni will not take action to swap synchronous unhealthy follower when the value is set to 0 or below. Please set the value high enough so Patroni won't swap synchrounous follower fequently during high transaction volume.
 -  **max\_timelines\_history**: maximum number of timeline history items kept in DCS.  Default value: 0. When set to 0, it keeps the full history in DCS.
--  **master\_start\_timeout**: the amount of time a primary is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for primary failure is: loop\_wait + master\_start\_timeout + loop\_wait, unless master\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
-- **master\_stop\_timeout**: The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
+-  **master\_start\_timeout**: the amount of time a master is allowed to recover from failures before failover is triggered (in seconds). Default is 300 seconds. When set to 0 failover is done immediately after a crash is detected if possible. When using asynchronous replication a failover can cause lost transactions. Worst case failover time for master failure is: loop\_wait + master\_start\_timeout + loop\_wait, unless master\_start\_timeout is zero, in which case it's just loop\_wait. Set the value according to your durability/availability tradeoff.
+-  **master\_stop\_timeout**: the number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled. When set to > 0 and the synchronous_mode is enabled, Patroni sends SIGKILL to the postmaster if the stop operation is running for more than the value set by master_stop_timeout. Set the value according to your durability/availability tradeoff. If the parameter is not set or set <= 0, master_stop_timeout does not apply.
+-  **static\_primary**: enables a few optimizations to ensure that a cluster configured with a static primary will not unnecessarily demote the cluster primary. This is useful for cases where a cluster is running as a single-node cluster. When this value is set, replicas will refuse to boot until the config value is removed from DCS config. **WARNING:** this feature is currently designed for use in single-node Patroni clusters. If additional members are added to a cluster with a configured static primary, those members must be running Patroni >= 2.2.0, else they will eventually violate cluster consensus, which is not safe for your data. The current behavior of replicas added to a cluster with a static primary is to shutdown. Better support for replicas with a static primary is being planned.
 -  **synchronous\_mode**: turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election. Synchronous mode makes sure that successfully committed transactions will not be lost at failover, at the cost of losing availability for writes when Patroni cannot ensure transaction durability. See :ref:`replication modes documentation <replication_modes>` for details.
 -  **synchronous\_mode\_strict**: prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the primary. See :ref:`replication modes documentation <replication_modes>` for details.
 -  **postgresql**:
@@ -183,7 +184,7 @@ ZooKeeper
 -  **key**: (optional) File with the client key.
 -  **key_password**: (optional) The client key password.
 -  **verify**: (optional) Whether to verify certificate or not. Defaults to ``true``.
--  **set_acls**: (optional) If set, configure Kazoo to apply a default ACL to each ZNode that it creates. ACLs will assume 'x509' schema and should be specified as a dictionary with the principal as the key and one or more permissions as a list in the value.  Permissions may be one of ``CREATE``, ``READ``, ``WRITE``, ``DELETE`` or ``ADMIN``.  For example, ``set_acls: {CN=principal1: [CREATE, READ], CN=principal2: [ALL]}``. 
+-  **set_acls**: (optional) If set, configure Kazoo to apply a default ACL to each ZNode that it creates. ACLs will assume 'x509' schema and should be specified as a dictionary with the principal as the key and one or more permissions as a list in the value.  Permissions may be one of ``CREATE``, ``READ``, ``WRITE``, ``DELETE`` or ``ADMIN``.  For example, ``set_acls: {CN=principal1: [CREATE, READ], CN=principal2: [ALL]}``.
 
 .. note::
     It is required to install ``kazoo>=2.6.0`` to support SSL.

docs/releases.rst

@@ -3,6 +3,15 @@
 Release notes
 =============
 
+Version 2.2.0
+-------------
+
+**New features**
+
+- Added support for ``static_primary`` configuration (Anthony Dodd)
+
+  This can be configured using the ``static_primary=<name>`` config value, which enables a few optimizations to ensure that a cluster configured with a static primary will not unnecessarily demote the cluster primary. This is useful for cases where a cluster is running as a single-node cluster. When this value is set, replicas will refuse to boot until the config value is removed from DCS config.
+
 Version 2.1.4
 -------------
 
@@ -1108,7 +1117,7 @@ Version 1.6.1
 
 - Kill all children along with the callback process before starting the new one (Alexander Kukushkin)
 
-  Not doing so makes it hard to implement callbacks in bash and eventually can lead to the situation when two callbacks are running at the same time. 
+  Not doing so makes it hard to implement callbacks in bash and eventually can lead to the situation when two callbacks are running at the same time.
 
 - Fix 'start failed' issue (Alexander Kukushkin)
 

features/environment.py

@@ -667,6 +667,13 @@ def start(self, name, max_wait_limit=40, custom_config=None):
                                                       self._output_dir, custom_config)
         self._processes[name].start(max_wait_limit)
 
+    def start_with_expected_failure(self, name, max_wait_limit=40, custom_config=None):
+        try:
+            self.start(name, max_wait_limit, custom_config)
+            assert False, 'expected startup to fail'
+        except Exception:
+            pass
+
     def __getattr__(self, func):
         if func not in ['stop', 'query', 'write_label', 'read_label', 'check_role_has_changed_to',
                         'add_tag_to_config', 'get_watchdog', 'patroni_hang', 'backup']:

features/static_primary.feature

@@ -0,0 +1,21 @@
+Feature: static primary
+  We should check that static primary behavior is safe
+
+  Scenario: check static primary config in dcs blocks replica from starting
+    Given I start postgres0 as static primary
+    Then postgres0 is a leader after 10 seconds
+    And there is a non empty initialize key in DCS after 15 seconds
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "loop_wait": 2}
+    Then I receive a response code 200
+    When I start postgres1 with a configured static primary will not boot after 20 seconds
+    And I start postgres2 with a configured static primary will not boot after 20 seconds
+    And "sync" key not in DCS after waiting 20 seconds
+    And "members/postgres1" is stopped and uninitialized after waiting 10 seconds
+    # NOTE: no need to wait an additional 10 seconds here.
+    And "members/postgres2" is stopped and uninitialized after waiting 1 seconds
+
+  Scenario: check removing static primary config from dcs allows replica startup
+    Given I issue a PATCH request to http://127.0.0.1:8008/config with {"static_primary": null}
+    Then "sync" key in DCS has leader=postgres0 after 20 seconds
+    And "members/postgres1" key in DCS has state=running after 10 seconds
+    And "members/postgres2" key in DCS has state=running after 10 seconds

features/steps/static_primary.py

@@ -0,0 +1,53 @@
+import json
+
+from behave import step
+from time import sleep
+
+
+@step('I start {name:w} as static primary')
+def start_patroni_with_static_primary(context, name):
+    return context.pctl.start(name, custom_config={'bootstrap': {'dcs': {'static_primary': name}}})
+
+
+@step('I start {name:w} with a configured static primary will not boot after {time_limit:d} seconds')
+def start_patroni_as_replica_with_static_primary(context, name, time_limit):
+    return context.pctl.start_with_expected_failure(name, max_wait_limit=time_limit)
+
+
+@step('"{name}" key not in DCS after waiting {time_limit:d} seconds')
+def check_member_not_present(context, name, time_limit):
+    sleep(time_limit)
+    found_value = False
+    try:
+        res = json.loads(context.dcs_ctl.query(name))
+        if res is not None:
+            found_value = True
+    except Exception:
+        pass
+
+    if found_value:
+        print("found value under DCS key {}: {}".format(name, res))
+        assert False, "found value under DCS key {} after {} seconds".format(name, time_limit)
+    return True
+
+
+@step('"{name}" is stopped and uninitialized after waiting {time_limit:d} seconds')
+def member_is_stopped_and_uninitialized(context, name, time_limit):
+    sleep(time_limit)
+    value = None
+    try:
+        value = json.loads(context.dcs_ctl.query(name))
+        print("response from dcs_ctl.query({}): {}".format(name, value))
+    except Exception:
+        pass
+
+    if value is None:
+        assert False, "context.dcs_ctl.query({}) unexpectedly returned None".format(name)
+
+    state = value.get("state")
+    role = value.get("role")
+    if state != "stopped":
+        assert False, "{} has state {}, expected 'stopped', after {} seconds".format(name, state, time_limit)
+    if role != "uninitialized":
+        assert False, "{} has role {}, expected 'uninitialized', after {} seconds".format(name, role, time_limit)
+    return True

patroni/config.py

@@ -65,6 +65,7 @@ class Config(object):
         'check_timeline': False,
         'master_start_timeout': 300,
         'master_stop_timeout': 0,
+        'static_primary': None,
         'synchronous_mode': False,
         'synchronous_mode_strict': False,
         'synchronous_node_count': 1,
@@ -234,7 +235,7 @@ def _safe_copy_dynamic_configuration(self, dynamic_configuration):
                     if name in self.__DEFAULT_CONFIG['standby_cluster']:
                         config['standby_cluster'][name] = deepcopy(value)
             elif name in config:  # only variables present in __DEFAULT_CONFIG allowed to be overridden from DCS
-                if name in ('synchronous_mode', 'synchronous_mode_strict'):
+                if name in ('synchronous_mode', 'synchronous_mode_strict', 'static_primary'):
                     config[name] = value
                 else:
                     config[name] = int(value)

patroni/ha.py

@@ -114,6 +114,13 @@ def is_paused(self):
     def check_timeline(self):
         return self.check_mode('check_timeline')
 
+    def get_static_primary_config(self):
+        if self.cluster and self.cluster.config and self.cluster.config.modify_index:
+            config = self.cluster.config.data
+        else:
+            config = self.patroni.config.dynamic_configuration
+        return config.get('static_primary')
+
     def get_standby_cluster_config(self):
         if self.cluster and self.cluster.config and self.cluster.config.modify_index:
             config = self.cluster.config.data
@@ -128,6 +135,18 @@ def is_leader(self):
         with self._is_leader_lock:
             return self._is_leader > time.time()
 
+    def is_static_primary(self):
+        """Check if this node is configured as the static primary of the cluster."""
+        name = self.state_handler.name
+        static_primary = self.get_static_primary_config()
+        if static_primary is None or name is None:
+            return False
+        return static_primary == name
+
+    def is_static_primary_configured(self):
+        """Check if the Patroni cluster has been configured with a static primary."""
+        return self.get_static_primary_config() is not None
+
     def set_is_leader(self, value):
         with self._is_leader_lock:
             self._is_leader = time.time() + self.dcs.ttl if value else 0
@@ -702,6 +721,8 @@ def _is_healthiest_node(self, members, check_replication_lag=True):
     def is_failover_possible(self, members, check_synchronous=True, cluster_lsn=None):
         ret = False
         cluster_timeline = self.cluster.timeline
+        if self.is_static_primary():
+            return ret
         members = [m for m in members if m.name != self.state_handler.name and not m.nofailover and m.api_url]
         if check_synchronous and self.is_synchronous_mode():
             members = [m for m in members if self.cluster.sync.matches(m.name)]
@@ -1009,6 +1030,9 @@ def process_unhealthy_cluster(self):
                         'promoted self to leader by acquiring session lock'
                     )
             else:
+                if self.is_static_primary():
+                    return 'no action as cluster is in static single node config mode'
+
                 return self.follow('demoted self after trying and failing to obtain lock',
                                    'following new leader after trying and failing to obtain lock')
         else:
@@ -1021,6 +1045,8 @@ def process_unhealthy_cluster(self):
             if self.patroni.nofailover:
                 return self.follow('demoting self because I am not allowed to become master',
                                    'following a different leader because I am not allowed to promote')
+            if self.is_static_primary():
+                return 'no action as cluster is in static single node config mode'
             return self.follow('demoting self because i am not the healthiest node',
                                'following a different leader because i am not the healthiest node')
 
@@ -1061,6 +1087,9 @@ def process_healthy_cluster(self):
                 if self.state_handler.is_leader():
                     if self.is_paused():
                         return 'continue to run as master after failing to update leader lock in DCS'
+                    if self.is_static_primary():
+                        return 'continue to run as master after failing to update leader lock in DCS \
+                                due to static_primary config'
                     self.demote('immediate-nolock')
                     return 'demoted self because failed to update leader lock in DCS'
                 else:
@@ -1364,6 +1393,14 @@ def _run_cycle(self):
                 self.state_handler.reset_cluster_info_state(None, self.patroni.nofailover)
                 raise
 
+            # If the cluster has been configured with a static primary,
+            # and we are not that primary, then do not proceed.
+            if self.is_static_primary_configured() and not self.is_static_primary():
+                self.shutdown()
+                return 'patroni cluster is configured with a static primary, \
+                        and this node is not the primary, shutting down and \
+                        refusing to start'
+
             if self.is_paused():
                 self.watchdog.disable()
                 self._was_paused = True
@@ -1517,7 +1554,8 @@ def _run_cycle(self):
         except DCSError:
             dcs_failed = True
             logger.error('Error communicating with DCS')
-            if not self.is_paused() and self.state_handler.is_running() and self.state_handler.is_leader():
+            if not self.is_paused() and self.state_handler.is_running() \
+                    and self.state_handler.is_leader() and not self.is_static_primary():
                 self.demote('offline')
                 return 'demoted self because DCS is not accessible and i was a leader'
             return 'DCS is not accessible'

patroni/validator.py

@@ -329,8 +329,9 @@ def assert_(condition, message="Wrong value"):
         Optional("ttl"): int,
         Optional("loop_wait"): int,
         Optional("retry_timeout"): int,
-        Optional("maximum_lag_on_failover"): int
-        },
+        Optional("maximum_lag_on_failover"): int,
+        Optional("static_primary"): str
+    },
     "pg_hba": [str],
     "initdb": [Or(str, dict)]
   },

patroni/version.py

@@ -1 +1 @@
-__version__ = '2.1.4'
+__version__ = '2.2.0'

tests/test_config.py

@@ -81,7 +81,7 @@ def test_reload_local_configuration(self):
     @patch('shutil.move', Mock(return_value=None))
     @patch('json.dump', Mock())
     def test_save_cache(self):
-        self.config.set_dynamic_configuration({'ttl': 30, 'postgresql': {'foo': 'bar'}})
+        self.config.set_dynamic_configuration({'ttl': 30, 'static_primary': 'baz', 'postgresql': {'foo': 'bar'}})
         with patch('os.fdopen', Mock(side_effect=IOError)):
             self.config.save_cache()
         with patch('os.fdopen', MagicMock()):
@@ -99,6 +99,13 @@ def test_standby_cluster_parameters(self):
         for name, value in dynamic_configuration['standby_cluster'].items():
             self.assertEqual(self.config['standby_cluster'][name], value)
 
+    def test_static_primary_parameter(self):
+        dynamic_configuration = {
+            'static_primary': 'foobar'
+        }
+        self.config.set_dynamic_configuration(dynamic_configuration)
+        self.assertEqual(self.config['static_primary'], 'foobar')
+
     @patch('os.path.exists', Mock(return_value=True))
     @patch('os.path.isfile', Mock(side_effect=lambda fname: fname != 'postgres0'))
     @patch('os.path.isdir', Mock(return_value=True))