Rebalance or Failover processing ongoing on some/all Eventing nodes, creating new functions, deployment or undeployment of existing functions is not allowed

Eventing
1

I tested the cluster (3 servers) with Eventing function that changes the document in case of its change. As a result of a large number of changes in the documents, it gave an error and asked to make a rebalance.
After the rebalance evetings are failed,

When changing Evetings it gives an error
deploy failed: {“name”:“ERR_REBALANCE_OR_FAILOVER_ONGOING”,“code”:36,“description”:“Rebalance or Failover processing ongoing on some/all Eventing nodes, creating new functions, deployment or undeployment of existing functions is not allowed”}

Please, help!
How can I remove all Evetings from Cluster ?

2

Can you show the error message?

3

The message is : deploy failed: {“name”:“ERR_REBALANCE_OR_FAILOVER_ONGOING”,“code”:36,“description”:“Rebalance or Failover processing ongoing on some/all Eventing nodes, creating new functions, deployment or undeployment of existing functions is not allowed”}

Screenshot 2024-08-28 093101

4

From the error message, creating new functions, deployment or undeployment of existing functions is not allowed during the rebalance. So either wait for the rebalance to complete or stop it.

5

I made rebalance three times but I have same result, error after rebalance in attach

{
  "data": {
    "stageInfo": {
      "backup": {
        "startTime": false,
        "completedTime": false,
        "timeTaken": false
      },
      "analytics": {
        "totalProgress": 100,
        "perNodeProgress": {
          "ns_1@65.21.128.51": 1,
          "ns_1@65.21.128.112": 1,
          "ns_1@65.21.129.105": 1
        },
        "startTime": "2024-08-28T20:37:21.801Z",
        "completedTime": "2024-08-28T20:37:23.866Z",
        "timeTaken": 2065
      },
      "eventing": {
        "startTime": "2024-08-28T20:37:23.866Z",
        "completedTime": false,
        "timeTaken": 35
      },
      "search": {
        "totalProgress": 100,
        "perNodeProgress": {
          "ns_1@65.21.128.51": 1,
          "ns_1@65.21.128.112": 1,
          "ns_1@65.21.129.105": 1
        },
        "startTime": "2024-08-28T20:36:43.212Z",
        "completedTime": "2024-08-28T20:36:43.391Z",
        "timeTaken": 180
      },
      "index": {
        "totalProgress": 100,
        "perNodeProgress": {
          "ns_1@65.21.128.51": 1,
          "ns_1@65.21.128.112": 1,
          "ns_1@65.21.129.105": 1
        },
        "startTime": "2024-08-28T20:36:43.391Z",
        "completedTime": "2024-08-28T20:37:21.801Z",
        "timeTaken": 38410
      },
      "data": {
        "totalProgress": 100,
        "perNodeProgress": {
          "ns_1@65.21.128.51": 1,
          "ns_1@65.21.128.112": 1,
          "ns_1@65.21.129.105": 1
        },
        "startTime": "2024-08-28T20:36:34.608Z",
        "completedTime": "2024-08-28T20:36:43.212Z",
        "timeTaken": 8604
      },
      "query": {
        "startTime": false,
        "completedTime": false,
        "timeTaken": false
      }
    },
    "rebalanceId": "52bc0ae14e01a3bfbf49012c956bbbce",
    "nodesInfo": {
      "active_nodes": [
        "ns_1@65.21.128.112",
        "ns_1@65.21.128.51",
        "ns_1@65.21.129.105"
      ],
      "keep_nodes": [
        "ns_1@65.21.128.112",
        "ns_1@65.21.128.51",
        "ns_1@65.21.129.105"
      ],
      "eject_nodes": [],
      "delta_nodes": [],
      "failed_nodes": []
    },
    "masterNode": "ns_1@65.21.128.112",
    "startTime": "2024-08-28T20:36:34.603Z",
    "completedTime": "2024-08-28T20:37:23.901Z",
    "timeTaken": 49298,
    "completionMessage": "Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {'EXIT',<0.14782.7413>,\n                                {{badmatch,\n                                  {error,\n                                   {bad_nodes,eventing,prepare_rebalance,\n                                    [{'ns_1@65.21.128.112',\n                                      {error,\n                                       {unknown_error,\n                                        <<\"Eventing Rebalance or Failover processing ongoing on nodeId: 0613a8524bcc1e4c225cdab1105e61c3\">>}}},\n                                     {'ns_1@65.21.128.51',\n                                      {error,\n                                       {unknown_error,\n                                        <<\"Eventing Rebalance or Failover processing ongoing on nodeId: a895243f010facdd736da7a5646e4a8b\">>}}}]}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,\"src/service_rebalancer.erl\"},\n                                    {line,158}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,\"proc_lib.erl\"},{line,211}]}]}}}}."
  },
  "status": 200,
  "config": {
    "method": "GET",
    "transformRequest": [
      null
    ],
    "transformResponse": [
      null
    ],
    "jsonpCallbackParam": "callback",
    "url": "/logs/rebalanceReport",
    "cache": true,
    "headers": {
      "Accept": "application/json, text/plain, */*",
      "invalid-auth-response": "on",
      "Cache-Control": "no-cache",
      "Pragma": "no-cache",
      "ns-server-ui": "yes"
    },
    "timeout": {
      "$$state": {
        "status": 0,
        "pending": [
          [
            {
              "$$state": {
                "status": 0
              }
            },
            null,
            null,
            null
          ]
        ]
      }
    }
  },
  "statusText": "OK",
  "xhrStatus": "complete"
}
6

Acoording the the message, the rebalance was not complete.

7

I see this but repeated rebalances lead to the same errors, Is it possible to restart the events ?

8

Yes. So stop rebalancing. Let the current rebalance complete - or stop it.
Then, when no rebalance is running you can manipulate eventing.