Child pages
  • Reducing Your Cluster's Capacity - Flex Down

This is documentation for a previous version of ClustrixDB. Documentation for the latest version can be found here

Skip to end of metadata
Go to start of metadata

On occasion, you may need to reduce your cluster's capacity. There could be many potential reasons. For example, you may wish to remove a node or node(s) 

  • To reduce operating costs following a peak event (i.e. following Cyber-Monday).
  • To allocate servers for other purposes.
  • To eliminate failing hardware. (See ALTER CLUSTER to drop a permanently failed node.)

The process to downsize your cluster within ClustrixDB is a simple three step process.  

Reminder: ClustrixDB requires a minimum of three nodes to support production systems. Going from three or more nodes to a single node is not supported via the steps outlined on this page.

Clustrix recommends running this process while logged on to a node other than one you wish to drop.

Step 1: Initiate SOFTFAIL

Marking a node as softfailed directs the Clustrix Rebalancer to move all data from the node(s) specified to others within the cluster. The Rebalancer proceeds in the background while the database continues to serve your ongoing production needs.

If necessary, determine the nodeid assigned to a given IP or hostname by running the following SQL select.

sql> SELECT * FROM system.nodeinfo ORDER BY nodeid; 

To initiate a SOFTFAIL, use the following syntax.

ALTER CLUSTER SOFTFAIL nodeid [, nodeid] ...

The SOFTFAIL operation will issue an error if there is not sufficient space to complete the softfail or if the softfail would leave the cluster unable to protect data should an additional node be lost. Clustrix recommends the following to resolve such errors:

  • Remove data from your cluster per the guidelines of the error message.
  • Reissue the SOFTFAIL request.

To cancel a SOFTFAIL process before it completes, use the following syntax. Your system will be restored to its prior configuration.

ALTER CLUSTER UNSOFTFAIL nodeid [, nodeid] ...  

Step 2: Monitor the SOFTFAIL Process

Once marked as softfailed, the Rebalancer moves data from the softfailed node(s). The Rebalancer process runs in the background while foreground processing continues to serve your production workload. 

The following will allow you to monitor the progress of the SOFTFAIL process.

a) Run the following query from a SQL prompt to verify that the node(s) you specified are indeed marked for removal.

sql> SELECT * FROM system.softfailed_nodes;

b) The system.softfailing_containers tables will show the list of containers that are slated to be moved as part of the SOFTFAIL operation. When this query returns 0 rows, the data migration is complete.

sql> SELECT * FROM system.softfailing_containers;

c) This query joins the tables referenced above and allows you to view the softfailed node(s) that are ready for removal. The query will return zero rows if the Rebalancer has not yet finished moving data from the softfailed node(s). The list will display vacated nodes once all their containers have been relocated. Proceed to the next step only when the results of this query include all the nodes you are softfailing.

sql> SELECT * FROM system.softfailed_nodes WHERE nodeid NOT IN (SELECT DISTINCT nodeid FROM system.softfailing_containers); 

d) Finally, once the SOFTFAIL is complete for all nodes, the clustrix.log file will contain the following message:

softfailing nodes are ready to be removed: <list of node ids>

Step 3: Remove Softfailed Node(s) from Your Cluster

The final step is to remove the node(s) from the cluster. To do this, issue the following SQL command.

sql> ALTER CLUSTER REFORM; 

This automatically removes softfailed node(s) from your cluster. There will be a brief interruption of service while the node(s) are removed.