Deploy OceanBase Database in a Kubernetes cluster

This topic describes how to deploy OceanBase Database in a Kubernetes environment by using ob-operator. Currently, only Kubernetes V1.23.6 is tested, and if you encounter any issues while deploying other versions, feel free to contact us.

The topic is divided into three parts:

1. Deploy ob-operator
2. Deploy OceanBase Database
3. Monitor OceanBase Database

Deploy ob-operator

Using ob-operator can simplify the deployment and operation of OceanBase Database in a Kubernetes cluster. The following two ways are available to deploy ob-operator:

Deploy ob-operator with Helm

You can deploy ob-operator by running the following commands:

helm repo add ob-operator https://oceanbase.github.io/ob-operator/
helm install ob-operator ob-operator/ob-operator --namespace=oceanbase-system --create-namespace  --version=1.1.0

Deploy ob-operator with configuration files

You can deploy ob-operator with configuration files by running the following commands:

# deploy CRD
kubectl apply -f https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/crd.yaml

# deploy ob-operator
kubectl apply -f https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/operator.yaml

Customize ob-operator

If you need to customize ob-operator, download the configuration file by running the following commands:

# download the config file
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/crd.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/operator.yaml

After you modify the configuration file, deploy ob-operator by running the following commands:

# after making some modification
kubectl apply -f crd.yaml
kubectl apply -f operator.yaml

Deploy OceanBase Database

Preparations

Configure labels for Kubernetes nodes

ob-operator uses nodeSelector in the obcluster.yaml configuration file of OceanBase Database to determine the distribution of OBServer nodes. Therefore, you need to configure labels for the Kubernetes nodes first.

You can set the label for a node by running the following command:

kubectl label node <node_name> <label_key>=<label_value>

# for example
kubectl label node node1 ob.zone=zone1

Deploy local-path-provisioner

To deploy OceanBase Database by using ob-operator, create a Persistent Volume Claim (PVC) as storage. For optimal performance, local storage is recommended. This topic uses local-path-provisioner to manage PVC.

Use the following command to deploy local-path-provisioner:

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.24/deploy/local-path-storage.yaml

For more information, see local-path-provisioner.

Deploy OceanBase Database

The OceanBase cluster is defined in a YAML configuration file. You can customize the configuration file provided by ob-operator to suit your needs. Use the following command to download the obcluster.yaml configuration file for OceanBase Database:

wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/obcluster.yaml

The sample content of the configuration file is shown below.

apiVersion: cloud.oceanbase.com/v1
kind: OBCluster
metadata:
  name: ob-test
  namespace: obcluster
spec:
  imageRepo: oceanbase/oceanbase-cloud-native
  tag: 4.1.0.0-100000192023032010
  imageObagent: oceanbase/obagent:1.2.0
  clusterID: 1
  topology:
    - cluster: cn
      zone:
      - name: zone1
        region: region1
        nodeSelector:
          ob.zone: zone1
        replicas: 1
      - name: zone2
        region: region1
        nodeSelector:
          ob.zone: zone2
        replicas: 1
      - name: zone3
        region: region1
        nodeSelector:
          ob.zone: zone3
        replicas: 1
      parameters:
        - name: log_disk_size
          value: "40G"
  resources:
    cpu: 2
    memory: 10Gi
    storage:
      - name: data-file
        storageClassName: "local-path"
        size: 50Gi
      - name: data-log
        storageClassName: "local-path"
        size: 50Gi
      - name: log
        storageClassName: "local-path"
        size: 30Gi
      - name: obagent-conf-file
        storageClassName: "local-path"
        size: 1Gi
    volume:
        name: backup
        nfs:
          server: ${nfs_server_address}
          path: /opt/nfs
          readOnly: false

Parameters in the sample content are described as follows:

• imageRepo: The image repository of OceanBase Database.

• tag: The tag of the OceanBase image.

• imageObagent: The image of OBAgent, which is used to collect monitoring data for the OceanBase cluster.

• cluster: To deploy an OceanBase cluster in a Kubernetes cluster, set the value of this parameter to the same as that of the startup parameter --cluster-name of ob-operator. The default value is cn.

• nodeSelector: The node selection strategy for the current Zone, which matches the node based on its labels. This configuration is used to configure the Kubernetes node labels in the preparation steps mentioned earlier.

• parameters: The custom parameters of OceanBase Database. Specify the parameters as needed.

• cpu: The number of pod CPU cores. We recommend that you set the value to an integer greater than 2. A value smaller than 2 may cause system exceptions.

• memory: The pod memory size. We recommend that you set the value to an integer greater than 10 GiB. A value smaller than 10 GiB may cause system exceptions.

• data-file: The data storage configuration of OceanBase Database. You can specify the size and storageClassName to use as the data directory for OceanBase Database. We recommend that you configure a value at least 3 times the size of memory or larger.

• data-log: The log storage configuration of OceanBase Database. You can specify the size and storageClassName to use as the Clog directory for OceanBase Database. We recommend that you configure a value at least 3 times the size of memory or larger.

• log: The process log storage configuration of OceanBase Database. You can specify the size and storageClassName to use as the directory for storing process logs for OceanBase Database. We recommend that you configure a value of at least 10GiB or larger.

• obagent-conf-file: The storage configuration of OBAgent configuration files, which is used to store OBAgent configuration files. A smaller size, generally around 1GiB, is sufficient.

• volume: The storage for backup data. If no backup requirement exists, this can be left unconfigured. However, once the cluster is created, it cannot be added later, so we recommend that you plan for this before deployment.

After you modify the configuration file, run the following command to deploy the cluster:

kubectl apply -f obcluster.yaml

Deploy OceanBase Database Proxy (ODP)

ODP is defined by using yaml configuration files. You can customize these files by referring to the configuration files of ob-operator. To download the ODP configuration files, run the following command:

wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/obproxy/deployment.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/obproxy/service.yaml

The sample content of the deployment.yaml file is shown below.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: obproxy
  namespace: obcluster
spec:
  selector:
    matchLabels:
      app: obproxy
  replicas: 2
  template:
    metadata:
      labels:
        app: obproxy
    spec:
      containers:
        - name: obproxy
          image: oceanbase/obproxy-ce:4.1.0.0-7
          ports:
            - containerPort: 2883
              name: "sql"
            - containerPort: 2884
              name: "prometheus"
          env:
            - name: APP_NAME
              value: helloworld
            - name: OB_CLUSTER
              value: ob-test
            - name: RS_LIST
              value: $RS_LIST
          resources:
            limits:
              memory: 2Gi
              cpu: "1"

Main environment variables in the content above are described as follows:

• APP_NAME: The name of the ODP application.

• OB_CLUSTER: The name of the OceanBase cluster that the ODP connects to.

• RS_LIST: The rs_list of the OceanBase cluster, which has the format ${ip1}:${port1};${ip2}:${port2};${ip3}:${port3}. Replace values in this format with actual ones. The following two ways are available to obtain the address of rs_list:

  • Method 1: Connect to the OceanBase cluster and run the show parameters like 'rootservice_list'; command to view it.

  • Method 2: Use the kubectl get RootService rs-${cluster_name} -n ${namespace} -o yaml command to query all RootService addresses, and then concatenate them into a complete rs_list. Replace ${cluster_name} and ${namespace} with the actual OceanBase cluster name and namespace.

The service.yaml file opens two ports: one for SQL connection and the other for monitoring data collection. Here is what the service.yaml file contains:

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: obproxy-service
  namespace: obcluster
spec:
  type: NodePort
  selector:
    app: obproxy
  ports:
    - name: "sql"
      port: 2883
      targetPort: 2883
      nodePort: 30083
    - name: "prometheus"
      port: 2884
      targetPort: 2884
      nodePort: 30084

After you modify the configuration file, run the following commands for deployment:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

Connect to OceanBase Database

We recommend that you connect to OceanBase Database through ODP. After you deploy OceanBase Database and ODP, run the following command to obtain the service connection address of ODP:

kubectl get svc ${servicename} -n ${namespace}

# for example
kubectl get svc obproxy-service -n obcluster

# output
NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
obproxy-service   NodePort   xxx.xxx.xxx.xxx   <none>        2883:30083/TCP,2884:30084/TCP   1m

You can make a connection using CLUSTER-IP and PORT. The corresponding connection commands are as follows:

# use clusterip without password
obclient -hxxx.xxx.xxx.xxx -P2883 -uroot@sys oceanbase -A -c

# use clusterip with password
obclient -hxxx.xxx.xxx.xxx -P2883 -uroot@sys -p oceanbase -A -c

# using nodeport without password
obclient -h<node_ip> -P30083 -uroot@sys oceanbase -A -c

# using nodeport with password
obclient -h<node_ip> -P30083 -uroot@sys -p oceanbase -A -c

Monitor OceanBase Database

Deploy Prometheus

Download the Prometheus configuration files by running the following command and customize them by referring to the Prometheus configuration files provided by ob-operator:

wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/prometheus/cluster-role.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/prometheus/cluster-role-binding.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/prometheus/configmap.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/prometheus/deployment.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/prometheus/service.yaml

Prometheus selects the address for collecting data based on the name and port of the service in Kubernetes. You can configure filtering for data collection using regular expressions, as shown in the configuration in the configmap.yaml file:

# Configurations in configmap.yaml
scrape_configs:
  - job_name: 'obagent-monitor-basic'
    kubernetes_sd_configs:
      - role: endpoints
    metrics_path: '/metrics/ob/basic'
    relabel_configs:
    - source_labels: [__meta_kubernetes_endpoints_name,__meta_kubernetes_endpoint_port_name]
      regex: svc-monitor-ob-test;monagent
      action: keep
  - job_name: 'obagent-monitor-extra'
    kubernetes_sd_configs:
      - role: endpoints
    metrics_path: '/metrics/ob/extra'
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_pod_container_port_name]
      regex: svc-monitor-ob-test;monagent
      action: keep
  - job_name: 'obagent-monitor-host'
    kubernetes_sd_configs:
      - role: endpoints
    metrics_path: '/metrics/node/host'
    relabel_configs:
    - source_labels: [__meta_kubernetes_endpoints_name,__meta_kubernetes_endpoint_port_name]
      regex: svc-monitor-ob-test;monagent
      action: keep
  - job_name: 'proxy-monitor'
    kubernetes_sd_configs:
      - role: endpoints
    metrics_path: '/metrics'
    relabel_configs:
    - source_labels: [__meta_kubernetes_endpoints_name,__meta_kubernetes_endpoint_port_name]
      regex: obproxy-service;prometheus
      action: keep

The configurations are described as follows:

• Service and port:

  • OceanBase: ob-operator creates a service based on the OceanBase cluster name to obtain the exporter address. The naming rule for the service is svc-monitor-${obcluster_name}, and the port name is monagent. If you customized the cluster name during deployment, modify the relevant configuration according to the actual cluster name.

  • ODP: You need to configure the corresponding ODP service and port name.

• Request path for monitoring metrics:

  • OceanBase cluster: The request paths are /metrics/ob/basic and /metrics/ob/extra.

  • ODP cluster: ODP supports the ability to expose monitoring metrics using the Prometheus protocol, and the request path is /metrics.

After you modify the configuration files, deploy Prometheus by running the following command:

kubectl apply -f cluster-role.yaml
kubectl apply -f cluster-role-binding.yaml
kubectl apply -f configmap.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

Verification

Open the Prometheus address in a browser and navigate to Status > Targets to verify that the Endpoint under proxy-monitor, obagent-monitor-host, obagent-monitor-basic, and obagent-monitor-extra are all in the up state.

Click on Graph to enter PromQL expressions and perform queries.

Deploy Grafana

Download the Grafana configuration files by running the following command and customize them by referring to the Prometheus configuration files provided by ob-operator:

wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/grafana/configmap.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/grafana/pvc.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/grafana/deployment.yaml
wget https://raw.githubusercontent.com/oceanbase/ob-operator/master/deploy/grafana/service.yaml

The configmap.yaml file contains key configuration information about Grafana, including the address of the Prometheus data source. If Prometheus was customized during deployment, you need to configure the actual service address.

# Key configurations in configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: obcluster
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access":"proxy",
                "editable": true,
                "name": "prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://svc-prometheus.obcluster.svc:8080",
                "version": 1,
                "isDefault": true
            }
        ]
    }

The Grafana configuration uses the service address of Prometheus as the data source configuration (url field in datasources). You need to fill this in according to the actual deployment of Prometheus.

After you modify the configuration files, deploy Grafana by running the following command:

kubectl apply -f configmap.yaml
kubectl apply -f pvc.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

Verification

Open the Grafana address in a browser and log on with the admin user. The default password is also admin, and you will be prompted to change it the first time you log on. The Grafana configuration includes the OceanBase Database dashboard by default, which you can view by opening the following link in a browser:

http://${node_ip}:${node_port}/d/oceanbase

You can view the host metrics and ODP monitoring templates by adding the following dashboard IDs.

• Host: 15216

• ODP: 15354