Authentication

Redpanda Cloud ensures the highest level of authentication for both users and services.

User authentication

Redpanda provides user authentication to your Redpanda organization through email or single sign-on.

Email

Redpanda Cloud can authenticate users with emails and passwords. Passwords are hashed (a one-way function that makes the original value unrecoverable, and effectively encrypted) and salted at rest using bcrypt.

Single sign-on

Redpanda Cloud can authenticate users with single sign-on (SSO) to an OIDC-based identity provider (IdP). Redpanda integrates with any OIDC-compliant IdP that supports discovery, including Okta, Microsoft Entra ID, Auth0, and Active Directory Federation Services (AD-FS). After SSO is enabled for an organization, new users in that organization can authenticate with SSO.

You must integrate your IdP with Redpanda Cloud to use SSO. On the Users page, users with admin permission see a Single sign-on tab and can add connections for up to two different IdPs. Enter the client ID, client secret, and discovery URI for the IdP. (See your IdP documentation for these values. The discovery URI may be called something different, like the well known URL or the issuer_url.)

By default, the connection is added in a disabled state. Edit the connection to enable it. You can choose to enable auto-enroll in the connection, which provides new users signing in from that IdP access to your Redpanda organization. When you enable auto-enroll, you select to assign a read, write, or admin role to users who log in with that IdP. Set up is different for most IdPs.

Deleting an SSO connection also deletes all users attached to it.

Integrate with Okta

To integrate with Okta, follow the Okta documentation to create an application within Okta for Redpanda. The Redpanda callback location (that is, the redirect location where Okta sends the user) is the following:

https://auth.prd.cloud.redpanda.com/login/callback

Okta provides the following fields required for SSO configuration on the Redpanda Users page: clientId, clientSecret, and discoveryUrl. The discovery URL for Okta generally looks like the following (where an_id could be “default”):

https://<orgname>.okta.com/oauth2/<an_id>/.well-known/openid-configuration

Integrate with Microsoft Entra ID

To integrate with Azure Entra ID, follow the Microsoft documentation to create an OIDC enterprise (web) application:

  1. In the Entra App Gallery, on the Create your own application page:

    1. Select Register an application to integrate with Microsoft Entra ID.

    2. For the name of your app, enter Redpanda Cloud.

    3. Click Create.

  2. On the Register an application page:

    1. For Supported account types, select Accounts in this organizational directory only (Redpanda only - Single tenant).

    2. For Redirect URI, select Web platform with the Callback URL found in Redpanda Cloud. In Redpanda Cloud, navigate to Users: Single sign-on, and click Add connection. Copy the Callback URL, and paste it into the Azure Redirect URI field.

    3. Click Register.

  3. On the Azure Enterprise applications page, you can now search for the Redpanda Cloud app to assign users access to Redpanda Cloud.

  4. On the Azure app for Redpanda Cloud, click Endpoints, and copy the OpenID Connect metadata document URL endpoint.

  5. In Redpanda Cloud, on Users: Single sign-on, paste that endpoint address into the Discovery URI field. Then, complete the SSO configuration:

    1. For Client ID, copy and paste the Application (client) ID from the Azure app for Redpanda Cloud.

    2. For Client secret, copy and paste the secret you get from adding a client secret on the Certificates & secrets page for the Azure app for Redpanda Cloud.

    3. For Realm, enter your Azure Entra ID tenant domain name.

    4. Click Save.

    5. On the Redpanda Cloud SSO page, edit your new Entra ID connection to enable SSO.

      Users with an email address with that realm (domain) can now access your Redpanda Cloud account.

      You can continue to configure your Azure Enterprise app page for Redpanda Cloud, but there is no need to configure JWT-based claims or API permissions.

Service authentication

Each Redpanda Cloud data plane runs its own dedicated agent, which authenticates and connects against the control plane over a single TLS 1.2 encrypted TCP connection.

Different Redpanda APIs support different authentication methods. For GCP, you can simultaneously enable mTLS and SASL for Kafka API, and mTLS and Basic authentication for the HTTP APIs (HTTP Proxy and Schema Registry). If you enable both mTLS and SASL on GCP clusters, Redpanda creates two distinct listeners: an mTLS listener operating on one port and a SASL listener operating on a different port.

Table 1. Redpanda APIs authentication methods
Cloud provider API Supported authentication methods

AWS

Kafka API

  • SASL

    • SASL/SCRAM

    • SASL/PLAIN

  • mTLS

HTTP Proxy

  • Basic authentication

  • mTLS

Schema Registry

  • Basic authentication

  • mTLS

GCP

See Enable mTLS and SASL.

Kafka API

  • SASL

    • SASL/SCRAM

    • SASL/PLAIN

  • mTLS

HTTP Proxy

  • Basic authentication

  • mTLS

Schema Registry

  • Basic authentication

  • mTLS

Azure

Kafka API

  • SASL

    • SASL/SCRAM

    • SASL/PLAIN

HTTP Proxy

  • Basic authentication

Schema Registry

  • Basic authentication

  • Kafka API: Redpanda Cloud enables SASL/SCRAM authentication over TLS 1.2 as well as mTLS to authenticate Kafka clients connecting to Redpanda clusters over the TCP endpoint or listener.

  • HTTP Proxy and Schema Registry: Authentication is done through an HTTP Basic Authentication header encrypted over TLS 1.2.

The following features use IAM policies to generate dynamic and short-lived credentials to interact with cloud provider APIs:

  • Data plane agent

  • Tiered Storage

  • Redpanda Console

  • Managed connectors

IAM policies have constrained permissions so that each service can only access or manage its own data plane-scoped resources, following the principle of least privilege.

Configure service authentication

Redpanda Cloud supports mTLS or SASL authentication for Kafka API, HTTP Proxy, and Schema Registry.

When you create a new cluster using the Cloud UI, the cluster is enabled by default with SASL for Kafka API, and Basic authentication for HTTP Proxy and Schema Registry.

Requirements

mTLS authentication is supported on AWS and GCP clusters only.

If you want to enable mTLS authentication:

  • You must use the Cloud API to create a new mTLS-enabled cluster.

  • You must also use the Cloud API to update an existing cluster to switch to mTLS authentication for Kafka API.

  • You can use the Cloud UI to update an existing cluster to switch to mTLS authentication for HTTP Proxy and Schema Registry only.

  • To enable mTLS and SASL (or Basic authentication) simultaneously on GCP clusters, you must use the Cloud API to create a new cluster or update an existing cluster.

To configure service authentication in your cluster using the Cloud API, you must have the following:

  • A service account in your Redpanda organization. You must have administrative privileges to create a service account.

  • Access to the Cloud API to enable mTLS or both SASL and mTLS. You use the service account to obtain an access token. You then make a request to the Control Plane API, passing the access token in the authorization header.

Authenticate to the Cloud API

  1. Create a service account in your organization, if you haven’t already. Go to the Service account tab of the Organization IAM page in the Redpanda Cloud UI and click Create service account to create a service account. Enter a name and description.

  2. Retrieve the client ID and secret by clicking Copy ID and Copy Secret.

  3. Obtain an access token by making a POST request to https://auth.prd.cloud.redpanda.com/oauth/token with the ID and secret in the request body.

    AUTH_TOKEN=`curl -s --request POST \
    --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \
    --header 'content-type: application/x-www-form-urlencoded' \
    --data grant_type=client_credentials \
    --data client_id=<client-id> \
    --data client_secret=<client-secret> \
    --data audience=cloudv2-production.redpanda.cloud | jq -r .access_token`

    Make sure to replace the following variables:

    Placeholder variable Description

    <client-id>

    Client ID.

    <client-secret>

    Client secret.

Enable mTLS authentication

For clusters with mTLS authentication, Redpanda creates a dedicated mTLS-enabled listener for each API service (Kafka API, HTTP Proxy, or Schema Registry) where you’ve enabled this authentication method. After you enable mTLS, get the API endpoints and verify that mTLS authentication is in effect.

If you enable mTLS authentication, you cannot disable it later.

Create a new cluster with mTLS enabled

  1. Follow the steps to to create a resource group and network for BYOC or Dedicated, if you haven’t already. You’ll need the resource group ID and network ID to create a cluster in the next step.

  2. Make a POST /v1/clusters/{cluster.id} request to create a new cluster with mTLS enabled.

The following example enables mTLS for Kafka API. To enable mTLS for HTTP Proxy and Schema Registry, add the http_proxy.mtls and schema_registry.mtls fields to the request body. You can choose to enable mTLS for any combination of the three services.
Show example request to enable mTLS for Kafka API 
CLUSTER_CREATE_BODY=`cat << EOF
{
  "cluster": {
    "cloud_provider": "CLOUD_PROVIDER_GCP",
    "connection_type": "CONNECTION_TYPE_PRIVATE",
    "name": "<cluster-name>",
    "resource_group_id": "<resource-group-id>",
    "network_id": "<network-id>",
    "region": "<region>",
    "zones": [ <zones> ],
    "throughput_tier": "<tier>",
    "type": "<cluster-type>",
    "kafka_api": {
       "mtls": {
          "enabled": true,
          "ca_certificates_pem": ["<ca-certificate-pem>"],
          "principal_mapping_rules": ["<principal-mapping-rule>"]
       }
    }
  }
}
EOF`
curl -v -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_CREATE_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`

Make sure to replace the following variables:

Placeholder variable Description

<cluster-id>

ID of the Redpanda cluster.

<cluster-name>

Name of the Redpanda cluster.

<resource-group-id>

ID of the resource group.

<network-id>

ID of the network.

<region>

The region where the cluster is created. For example, us-central1.

<zones>

The zones where the cluster is created. For example, ["us-central1-a", "us-central1-b", "us-central1-c"].

<tier>

The usage tier of the cluster. For example, .

<cluster-type>

The Redpanda cluster type, TYPE_BYOC or TYPE_DEDICATED.

<ca-certificate-pem>

A trusted Kafka client CA certificate in PEM format. The ca_certificates_pem field accepts a list of certificates.

<principal-mapping-rule>

Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals.

For example, the mapping rule RULE:.*CN=([^,]+).*/\\$1/ maps the following certificate subject to a principal named test:

Subject: C=US, ST=IL, L=Chicago, O=redpanda, OU=cloud, CN=test, emailAddress=test123@redpanda.com

See Configure Authentication for more details on principal mapping rules. The principal_mapping_rules field accepts a list of rules.

The Create Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET request to the following endpoint:

curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>

When the operation state is COMPLETED, you can verify that mTLS is enabled for the API endpoints.

Update an existing cluster to use mTLS

Make a PATCH /v1/clusters/{cluster.id} request to enable mTLS for the Kafka API on a cluster.

The following code block shows a request to enable mTLS for Kafka API. To enable mTLS for HTTP Proxy and Schema Registry, add the http_proxy.mtls and schema_registry.mtls fields to the request body:

Show example request 
CLUSTER_PATCH_BODY=`cat << EOF
{
  "kafka_api": {
     "mtls": {
        "enabled": true,
        "ca_certificates_pem": ["<ca-certificate-pem>"],
        "principal_mapping_rules": ["<principal-mapping-rule>"]
     }
  }
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`

Make sure to replace the following variables:

Placeholder variable Description

<cluster-id>

ID of Redpanda cluster.

<ca-certificate-pem>

A trusted Kafka client CA certificate in PEM format. The ca_certificates_pem field accepts a list of certificates.

<principal-mapping-rule>

Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals.

For example, the mapping rule RULE:.*CN=([^,]+).*/\\$1/ maps the following certificate subject to a principal named test:

Subject: C=US, ST=IL, L=Chicago, O=redpanda, OU=cloud, CN=test, emailAddress=test123@redpanda.com

See Configure Authentication for more details on principal mapping rules. The principal_mapping_rules field accepts a list of rules.

The Update Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET request to the following endpoint:

curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>

When the operation state is COMPLETED, you can verify that mTLS is enabled for the API endpoints.

Enable mTLS and SASL

Enabling mTLS and SASL simultaneously is available for GCP clusters only. To unlock this feature for your account, contact your Customer Success Manager.

You can choose to enable mTLS and SASL simultaneously for the Kafka API, and mTLS and Basic authentication for HTTP Proxy and Schema Registry. The sasl field in the API request examples toggle both SASL and Basic authentication.

Create a new cluster with both mTLS and SASL enabled

  1. Follow the steps to to create a resource group and network for BYOC or Dedicated, if you haven’t already done so. You’ll need the resource group ID and network ID to create a cluster in the next step.

  2. Make a POST /v1/clusters/{cluster.id} request to create a new cluster with both mTLS and SASL or Basic authentication enabled.

    You can enable mTLS and SASL or Basic authentication for any combination of the three services. For example, if you want to enable mTLS and SASL simultaneously for Kafka API and mTLS and Basic authentication simultaneously for Schema Registry only, leave out the entire http_proxy block from the request body. If you want to enable mTLS only for Kafka API, and mTLS and Basic authentication for HTTP Proxy and Schema Registry, leave out the kafka_api.sasl field.

    Show example request 
    CLUSTER_CREATE_BODY=`cat << EOF
{
  "cluster": {
    "cloud_provider": "CLOUD_PROVIDER_GCP",
    "connection_type": "CONNECTION_TYPE_PRIVATE",
    "name": "<cluster-name>",
    "resource_group_id": "<resource-group-id>",
    "network_id": "<network-id>",
    "region": "<region>",
    "zones": [ <zones> ],
    "throughput_tier": "<tier>",
    "type": "<cluster-type>",
    "kafka_api": {
       "mtls": {
          "enabled": true,
          "ca_certificates_pem": ["<ca-certificate-pem>"],
          "principal_mapping_rules": ["<principal-mapping-rule>"]
       },
       "sasl": {
          "enabled": true
       }
    },
    "http_proxy": {
       "mtls": {
          "enabled": true,
          "ca_certificates_pem": ["<ca-certificate-pem>"]
       },
       "sasl": {
          "enabled": true
       }
    },
    "schema_registry": {
       "mtls": {
          "enabled": true,
          "ca_certificates_pem": ["<ca-certificate-pem>"]
       },
       "sasl": {
          "enabled": true
       }
    }
  }
}
EOF`
curl -v -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_CREATE_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`

    Make sure to replace the following variables:

    Placeholder variable Description

    <cluster-id>

    ID of Redpanda cluster.

    <cluster-name>

    Name of the Redpanda cluster.

    <resource-group-id>

    ID of the resource group.

    <network-id>

    ID of the network.

    <region>

    The region where the cluster is created. For example, us-central1.

    <zones>

    The zones where the cluster is created. For example, ["us-central1-a", "us-central1-b", "us-central1-c"].

    <tier>

    The usage tier of the cluster. For example, .

    <cluster-type>

    The Redpanda cluster type, TYPE_BYOC or TYPE_DEDICATED.

    <ca-certificate-pem>

    A trusted Kafka client CA certificate in PEM format. The ca_certificates_pem field accepts a list of certificates.

    <principal-mapping-rule>

    Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals.

    For example, the mapping rule RULE:.*CN=([^,]+).*/\\$1/ maps the following certificate subject to a principal named test:

    Subject: C=US, ST=IL, L=Chicago, O=redpanda, OU=cloud, CN=test, emailAddress=test123@redpanda.com

    See Configure Authentication for more details on principal mapping rules. The principal_mapping_rules field accepts a list of rules.

The Create Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET request to the following endpoint:

curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>

When the operation state is COMPLETED, you can verify that mTLS is enabled for the API endpoints.

Update an existing cluster to use mTLS and SASL

Make a PATCH /v1/clusters/{cluster.id} request to enable mTLS and SASL on an existing cluster.

You can choose to enable mTLS and SASL or Basic authentication for any combination of the three services. For example, if you want to enable mTLS and SASL simultaneously for Kafka API and mTLS and Basic authentication simultaneously for Schema Registry only, leave out the entire http_proxy block from the request body. If you want to enable mTLS only for Kafka API, and mTLS and Basic authentication for HTTP Proxy and Schema Registry, leave out the kafka_api.sasl field.

Show example request 
CLUSTER_PATCH_BODY=`cat << EOF
{
  "kafka_api": {
     "mtls": {
        "enabled": true,
        "ca_certificates_pem": ["<ca-certificate-pem>"],
        "principal_mapping_rules": ["<principal-mapping-rule>"]
     },
     "sasl": {
        "enabled": true
     }
  },
  "schema_registry": {
     "mtls": {
        "enabled": true,
        "ca_certificates_pem": ["<ca-certificate-pem>"]
     },
     "sasl": {
        "enabled": true
     }
  },
  "http_proxy": {
     "mtls": {
        "enabled": true,
        "ca_certificates_pem": ["<ca-certificate-pem>"]
     },
     "sasl": {
        "enabled": true
     }
  }
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`

Make sure to replace the following variables:

Placeholder variable Description

<cluster-id>

ID of Redpanda cluster.

<ca-certificate-pem>

A trusted Kafka client CA certificate in PEM format. The ca_certificates_pem field accepts a list of certificates.

<principal-mapping-rule>

Configurable rule for mapping the Distinguished Name of Kafka client certificates to Kafka principals.

For example, the mapping rule RULE:.*CN=([^,]+).*/\\$1/ maps the following certificate subject to a principal named test:

Subject: C=US, ST=IL, L=Chicago, O=redpanda, OU=cloud, CN=test, emailAddress=test123@redpanda.com

See Configure Authentication for more details on principal mapping rules. The principal_mapping_rules field accepts a list of rules.

The Update Cluster endpoint returns a long-running operation. You can check the status of the operation by making a GET request to the following endpoint:

curl -H "Authorization: Bearer $AUTH_TOKEN" https://api.redpanda.com/v1/operations/<operation-id>

When the operation state is COMPLETED, you can verify that mTLS is enabled for the API endpoints.

Update an existing cluster to disable SASL

If you enabled mTLS and SASL on a cluster, you can disable SASL by making a PATCH /v1/clusters/{cluster.id} request:

Show example request 
CLUSTER_PATCH_BODY=`cat << EOF
{
  "kafka_api": {
     "sasl": {
        "enabled": false
     }
  }
}
EOF`
curl -v -X PATCH \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
-d "$CLUSTER_PATCH_BODY" https://api.redpanda.com/v1/clusters/<cluster-id>`

Retrieve API endpoints

Retrieve the mTLS and SASL-enabled endpoints by calling the GET /v1/clusters/{id} endpoint, passing the cluster ID as a parameter.

curl -X GET "https://api.redpanda.com/v1/clusters/<cluster-id>" \
 -H "accept: application/json"\
 -H "content-type: application/json" \
 -H "authorization: Bearer ${AUTH_TOKEN}"

The API endpoints are returned in the response body in the following fields:

API Field Example

Kafka API

kafka_api.all_seed_brokers

  • sasl: seed-2f92c489.d040oh0mf339m7q5uu0g.byoc.ign.cloud.redpanda.com:9092

  • mtls: seed-2f92c489.d040oh0mf339m7q5uu0g.byoc.ign.cloud.redpanda.com:9093

HTTP Proxy

http_proxy.all_urls

  • sasl: https://pandaproxy-ce24d80a.d040oh0mf339m7q5uu0g.byoc.ign.cloud.redpanda.com:30082

  • mtls: https://pandaproxy-ce24d80a.d040oh0mf339m7q5uu0g.byoc.ign.cloud.redpanda.com:30083

Schema Registry

schema_registry.all_urls

  • sasl: https://schema-registry-20b02d09.d040oh0mf339m7q5uu0g.byoc.ign.cloud.redpanda.com:30081

  • mtls: https://schema-registry-20b02d09.d040oh0mf339m7q5uu0g.byoc.ign.cloud.redpanda.com:30080

Verify mTLS for Kafka API connections

To verify that mTLS is enabled for Kafka API, run the following rpk command without providing a security certificate or private key:

rpk cluster info --tls-enabled

You should get the following error:

unable to request metadata: remote error: tls: certificate required

When you consume, produce to, or manage topics using rpk, you must provide a client certificate and private key. You may use the --tls-cert and --tls-key options, or environment variables with rpk.

rpk topic create test-topic --tls-enabled --tls-cert=/path/to/tls.crt --tls-key=/path/to/tls.key

Verify mTLS for HTTP Proxy and Schema Registry

To verify that mTLS is enabled for the HTTP Proxy and Schema Registry, run the following curl commands, without providing a security certificate or key:

# Run the following to verify HTTP Proxy
curl -u $USERNAME:$PASSWORD -k -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 -d '{"records":[{"test":"hello"},{"test":"world"}]}' $HTTP_PROXY_MTLS_URL/topics/<topic-name>

# Run the following to verify Schema Registry
curl -u $USERNAME:$PASSWORD -k -H "Content-Type: application/vnd.schemaregistry.v1+json" $SCHEMA_REGISTRY_MTLS_URL/subjects/<subject-name>/versions/1

You should get an error indicating that the certificate is required.

To successfully connect to the HTTP Proxy and Schema Registry, you must provide a client certificate and private key. The following curl commands show example requests to mTLS-enabled endpoints using test as the username and 12345 as the password.

# HTTP Proxy
curl -u test:12345 -k --cert cert.pem --key key.pem -H "Content-Type: application/vnd.kafka.json.v2+json" --sslv2 --http2 https://pandaproxy-45f811b1.cge5asc6006u7fvep0q0.fmc.dev.cloud.redpanda.com:30082/topics

# Schema Registry
curl -u test:12345 -k --cert cert.pem --key key.pem https://schema-registry-15d24f32.cge5asc6006u7fvep0q0.fmc.dev.cloud.redpanda.com:30081/subjects/Kafka-value/versions/1

Verify SASL

To verify that SASL is enabled for Kafka API, run the following rpk command:

rpk topic create test-topic --tls-enabled --user <username> --password <password>

The command should succeed, and you should be able to create a topic named test-topic.

Suggested reading