Hybrid mode

Hybrid mode, also known as Control Plane/Data Plane separation (CP/DP), is a deployment model that splits all Kong Gateway nodes in a cluster into one of two roles:

• Control Plane (CP) nodes, where configuration is managed and the Admin API is served from
• Data Plane (DP) nodes, which serve traffic for the proxy

In hybrid mode, the database only needs to exist on Control Plane nodes.

Each DP node is connected to one of the CP nodes, and only the CP nodes are directly connected to a database. Instead of accessing the database contents directly, the DP nodes maintain a connection with CP nodes to receive the latest configuration.

Konnect runs in hybrid mode. In this case, Kong manages the database for you, so you can’t access it directly. This means you can’t manage Konnect configuration via kong.conf like you can for Kong Gateway, as Kong handles that configuration. Additionally, Konnect uses the Control Plane Config API to manage Control Planes while Kong Gateway uses the Admin API.

The following diagram shows what Kong Gateway looks like in self-managed hybrid mode:

 
flowchart TD

A[(Database)]
B( Control Plane 
 #40;Kong Gateway instance#41;)
C( Data Plane 3
 #40;Kong Gateway instance#41;)
D( Data Plane 1
 #40;Kong Gateway instance#41;)
E( Data Plane 2
 #40;Kong Gateway instance#41;)

subgraph id1 [Self-managed CP node]
A---B
end

B --Kong proxy 
configuration---> id2 & id3

subgraph id2 [Self-managed on-prem]
C
end

subgraph id3 [Self-managed cloud]
D
E
end

style id1 stroke-dasharray:3,rx:10,ry:10
style id2 stroke-dasharray:3,rx:10,ry:10
style id3 stroke-dasharray:3,rx:10,ry:10

  

Figure 1: In self-managed hybrid mode, the Control Plane and Data Planes are hosted on different nodes. The Control Plane connects to the database, and the Data Planes receive configuration from the Control Plane.

When you create a new Data Plane node, it establishes a connection to the Control Plane. The Control Plane listens on port 8005 (Kong Gateway) or 443 (Konnect) for connections and tracks any incoming data from its Data Planes.

Once connected, every API or Kong Manager/Konnect UI action on the Control Plane triggers an update to the Data Planes in the cluster.

Hybrid mode deployments have the following benefits:

• Deployment flexibility: Users can deploy groups of Data Planes in different data centers, geographies, or zones without needing a local clustered database for each DP group.
• Increased reliability: The availability of the database doesn’t affect the availability of the Data Planes. Each DP caches the latest configuration it received from the Control Plane on local disk storage, so if CP nodes are down, the DP nodes keep functioning.
  • While the CP is down, DP nodes constantly try to reestablish communication.
  • DP nodes can be restarted while the CP is down, and still proxy traffic normally.
• Traffic reduction: Drastically reduces the amount of traffic to and from the database, since only CP nodes need a direct connection to the database.
• Increased security: If one of the DP nodes is compromised, an attacker won’t be able to affect other nodes in the Kong Gateway cluster.
• Ease of management: Admins only need to interact with the CP nodes to control and monitor the status of the entire Kong Gateway cluster.

You can run Kong Gateway in hybrid mode on any platform where Kong Gateway is supported, including Konnect.

You can run Kong Gateway on Kubernetes in hybrid mode with or without the Kong Ingress Controller. This uses Kubernetes as a runtime for your data planes.

Running Kong Gateway in hybrid mode is commonly referred to as “Kong on Kubernetes”. Running Kong Gateway with Kong Ingress Controller is commonly referred to as “Kong for Kubernetes”, as it provides a Kubernetes native way of configuring Kong entities using Kong Ingress Controller. Configuring Kong on Kubernetes is identical to deploying Kong Gateway running on a virtual machine or bare metal.

Configuring a hybrid mode deployment with Kong Ingress Controller should only be used in a small set of circumstances. We recommend using hybrid mode without Kong Ingress Controller, or DB-less mode with Kong Ingress Controller, unless you’ve been otherwise advised by a member of the Kong team.

For the full Kubernetes hybrid mode documentation, see hybrid mode in the kong/charts repository.

Depending on where you’re running hybrid mode, the following CP/DP versioning compatibility applies:

• Kong-managed in Konnect: Control planes only allow connections from Data Planes with the exact same version of the Control Plane.
• Self-managed in Kong Gateway: Control planes only allow connections from Data Planes with the same major version. Control planes won’t allow connections from Data Planes with newer minor versions.

For example, a Kong Gateway v3.9.0.1 Control Plane:

Plugin version compatibility

For every plugin that is configured on the Control Plane, new configs are only pushed to Data Planes that have those configured plugins installed and loaded. The major version of those configured plugins must be the same on both the Control Planes and Data Planes. Also, the minor versions of the plugins on the Data Planes can’t be newer than versions installed on the Control Planes. Similar to Kong Gateway version checks, plugin patch versions are also ignored when determining compatibility.

For instance, a new version of Kong Gateway includes a new plugin offering, and you update your Control Plane with that version. You can still send configurations to your Data Planes that are on a less recent version as long as you haven’t added the new plugin offering to your configuration. If you add the new plugin to your configuration, you will need to update your Data Planes to the newer version for the Data Planes to continue to read from the Control Plane.

If the compatibility checks fail, the Control Plane stops pushing out new config to the incompatible Data Planes to avoid breaking them. In Konnect, you will see compatibility errors on the Control Plane that has conflicts. See version compatibility in Control Planes for all errors and resolutions.

If a config can not be pushed to a Data Plane due to failure of the compatibility checks, the Control Plane will contain warn level lines in the error.log similar to the following:

unable to send updated configuration to DP node with hostname: localhost.localdomain ip: 127.0.0.1 reason: version mismatches, CP version: 2.2 DP version: 2.1
unable to send updated configuration to DP node with hostname: localhost.localdomain ip: 127.0.0.1 reason: CP and DP does not have same set of plugins installed or their versions might differ

The following API endpoints return the version of the Data Plane node and the latest config hash the node is using:

• On-prem Kong Gateway: /clustering/data-planes Admin API
• Konnect: /expected-config-hash Control Plane Config API This data helps detect version incompatibilities from the Control Plane side.

If Control Plane nodes are down, the Data Plane will keep functioning. Data plane caches the latest configuration it received from the Control Plane on the local disk. In case the Control Plane stops working, the Data Plane will keep serving requests using cached configurations. It does so while constantly trying to reestablish communication with the Control Plane.

This means that the Control Plane nodes can be stopped even for extended periods of time, and the Data Plane will still proxy traffic normally. Data plane nodes can be restarted while in disconnected mode, and will load the last configuration in the cache to start working. When the Control Plane is brought up again, the Data Plane nodes will contact them and resume connected mode.

You can also configure Data Plane resiliency in case of Control Plane outages.

Disconnected mode in on-prem deployments

The viability of the Data Plane while disconnected means that Control Plane updates or database restores can be done with peace of mind. First bring down the Control Plane, perform all required downtime processes, and only bring up the Control Plane after verifying the success and correctness of the procedure. During that time, the Data Plane will keep working with the latest configuration.

A new Data Plane node can be provisioned during Control Plane downtime. This requires either copying the LMDB directory (dbless.lmdb) from another Data Plane node, or using a declarative configuration. In either case, if it has the role of "data_plane", it will also keep trying to contact the control plane until it’s up again.

To change a disconnected Data Plane node’s configuration in self-managed hybrid mode, you must:

• Remove the LMDB directory (dbless.lmdb)
• Ensure the declarative_config parameter or the KONG_DECLARATIVE_CONFIG environment variable is set
• Set the whole configuration in the referenced YAML file

Data plane cache configuration

By default, Data Planes store their configuration to the file system in an unencrypted LMDB database, dbless.lmdb, in Kong Gateway’s prefix path. You can also choose to encrypt this database.

If encrypted, the Data Plane uses the cluster certificate key to decrypt the LMDB database on startup.

When using hybrid mode, you may encounter the following limitations.

Configuration inflexibility

In Kong Gateway 3.9.x or earlier, whenever you make changes to Kong Gateway entity configuration on the Control Plane, it immediately triggers a cluster-wide update of all Data Plane configurations. This can cause performance issues.

You can enable incremental configuration sync for improved performance in Kong Gateway 3.10.x or later. When a configuration changes, instead of sending the entire configuration set for each change, Kong Gateway only sends the parts of the configuration that have changed.

See the incremental configuration sync documentation to learn more.

Plugin incompatibility

When plugins are running on a Data Plane in hybrid mode, there is no API exposed directly from that DP. Since the Admin API is only exposed from the Control Plane, all plugin configuration has to occur from the CP. Due to this setup, and the configuration sync format between the CP and the DP, some plugins have limitations in hybrid mode:

• Key Auth Encrypted: The time-to-live setting (ttl), which determines the length of time a credential remains valid, does not work in hybrid mode.
• Rate Limiting, Rate Limiting Advanced, Response Rate Limiting, AI Rate Limiting Advanced, and Service Protection: These plugins don’t support the cluster strategy/policy in hybrid mode. One of the local or redis strategies/policies must be used instead.
• GraphQL Rate Limiting Advanced: This plugin doesn’t support the cluster strategy in hybrid mode. The redis strategy must be used instead.
• OAuth 2.0 Authentication: This plugin is not compatible with hybrid mode. For its regular workflow, the plugin needs to both generate and delete tokens, and commit those changes to the database, which is not possible with CP/DP separation.

Custom plugins

Custom plugins (either your own plugins or third-party plugins that are not shipped with Kong Gateway) need to be installed on both the Control Plane and the data plane in hybrid mode.

Consumer Groups

The ability to scope plugins to consumer groups was added in Kong Gateway version 3.4. Running a mixed-version Kong Gateway cluster (3.4 Control Plane, and <=3.3 Data Planes) is not supported when using consumer group scoped plugins.

Load balancing

There is no automated load balancing for connections between the Control Plane and the Data Plane. You can load balance manually by using multiple Control Planes and redirecting the traffic using a TCP proxy.

Several read-only endpoints from the Admin API are exposed to the Status API on Data Planes, including the following:

• GET /upstreams/{upstream}/targets/
• GET /upstreams/{upstream}/health/
• GET /upstreams/{upstream}/targets/all/
• GET /upstreams/{upstream}/targets/{target}

See Upstream objects in the Admin API and Control Plane Config API documentation for more information about the endpoints.

Because the Keyring module encrypts data in the database, it can’t encrypt data on Data Plane nodes, since these nodes run without a database and get data from the Control Plane.

Use the following configuration properties to configure Kong Gateway in hybrid mode:

The following properties are used differently between shared and PKI modes: