Connect External OpenSearch Dashboard to AWS OpenSearch Domain With Helm

gowebing 2024-12-02

Last year, AWS announced the renaming of Amazon Elasticsearch Service to Amazon OpenSearch Service, with OpenSearch being the successor to the distributed search and analytics cluster service.

Why Did Amazon Change From Elasticsearch to OpenSearch?

On January 21, 2021, Elastic NV announced that they would change their software licensing strategy and not release new versions of Elasticsearch and Kibana under the permissive ALv2 license. Instead, Elastic is releasing Elasticsearch and Kibana under the Elastic license, with source code available under the Elastic License or Server Side Public License (SSPL). These licenses are not open source. Because of this, Amazon decided to create and maintain a fork from the last ALv2 version of Elasticsearch and Kibana. The fork is called OpenSearch and is available under ALv2.

The updated policy is available in the official documentation.

How It Works

When you create an OpenSearch domain, you are creating a cluster. This cluster is composed of several EC2 instances, which are equivalent to a node. Each node stores data and processes query requests.

For productive environments, there are master nodes that manage the operations of the nodes. They manage the status, health of each node, and shards distribution.

However, you won't see any EC2 instances in the EC2 service. This is because OpenSearch Service is a managed service (SaaS) by AWS. For that reason, you will only be able to edit the cluster configuration (number of nodes, instance types, etc.).

Connect External OpenSearch Dashboard to AWS OpenSearch Domain With Helm

Requirements

Your Environment:

Git
Visual Studio Code or another similar code editor tool
kubectl and the kubeconfig of the EKS cluster configured. You can check out more about it in the Kubernetes documentation.
Helm

Your AWS Account:

Create an OpenSearch domain
HTTPS connection enabled between the SGs of the OpenSearch domain and the EKS nodes of the different VPCs

Steps

First, clone the OpenSearch Helm chart repository.
Open the project with Visual Studio Code and go to the helm-charts/charts/opensearch-dashboards/ directory to edit the values.yaml file.
Before going into the details of the helm chart configuration, it is important to create a secret in the namespace where OpenSearch Dashboard will be deployed which will contain the user and password to access the portal.

kubectl create secret generic opensearchdashboards-auth --from-literal=username="myfirstuser" --from-literal=password="myfirstpasswd" [OPTIONAL: -n opensearch]

4. Open the file values.yaml and apply the following configuration:

Sets the domain endpoint of the Amazon OpenSearch service as the value for the opensearchHosts parameter. You can get it from the General Information panel:

General Information Panel

     YAML 
   
   opensearchHosts: "https://vpc-dump-domain-xvu6q2qt7vjs2ueaipnotcpyui.eu-north-1.es.amazonaws.com/_dashboards"

5. Then set the name and the version of the OpenSearch Dashboards docker image. The following example uses the official image obtained from Docker Hub but you can use one from a private repository (ECR, Artifactory, NexusOSS...).

     YAML 
   
   image: "opensearchproject/opensearch-dashboards"
imageTag: "1.2.0"
imagePullPolicy: "IfNotPresent"

6. Define the name of the secret which contains the username/password credentials created in the previous steps to link it to the dashboard:

     YAML 
   
   opensearchAccount:
  secret: "opensearchdashboards-opensearch-auth"
  keyPassphrase:
    enabled: false

7. Finally, as an optional step, you can configure your ingress to enable external service to the pod. In the following example, Traefik has been used as a reverse proxy and balancer in the cluster.

     YAML 
   
 
 
   ingress:
  enabled: true
  annotations:
    external-dns.alpha.kubernetes.io/target: traefik-int.cloud.myprivatedomain.com
    kubernetes.io/ingress.class: traefik
  hosts:
    - host: opensearchdashboard.cloud.myprivatedomain.com
      paths:
        - path: / 
  

In case you don't want to configure this part, you can enable access from your local environment using the command. After running the following command, you will be able to check the dashboard access from localhost:8080 (Must be deployed first).

     Shell 
   
   kubectl port-forward -n opensearch [OS_DASHBOARD_POD_NAME] 8080:5601

Once the helm chart has been configured with the correct values, it is time to open a terminal at the same level as the values.yaml file and execute helm install to deploy the OpenSearch Dashboard chart in your EKS cluster.

     Shell 
   
   helm install opensearchdashboards . -n opensearch [OPTIONAL: --create-namespace]

After a few seconds, you will see that it has been successfully deployed and you will be able to access the OpenSearch dashboard from the browser:

Now, it will be time to synchronize with the domain indexes and start visualizing and processing the data.