NGINX Ingress Controller Configuration In AKS
Ingress Controller
Ingress controller is a necessary Kubernetes feature that plays a vital role in the functioning of Ingress resources. The Ingress resources deployed in the cluster are controlled by the Ingress controller. As mentioned in Kubernetes Ingress controller page: "Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically when a cluster is created." It must be deployed into the cluster manually and configured according to the prescribed requirements.
NGINX Ingress Controller for Kubernetes
According to the NGINX Ingress Controller page, "The NGINX Ingress Controller for Kubernetes provides enterprise-grade delivery of services for Kubernetes applications, with benefits for users of both open source NGINX and Kubernetes." We are using this NGINX Ingress Controller.
Deploying a NGINX Ingress Controller
kubectl apply -f
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
---
This will deploy the Ingress Controller deployment and other associated resources in the ingress-nginx namespace.
kubectl apply -f
:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
---
This will deploy the Ingress Controller LoadBalancer type Kubernetes service in the ingress-nginx namespace.
Ingresses
Ingress is a Kubernetes object that allows access to your Kubernetes services from outside the Kubernetes cluster. You can configure access by creating a set of rules that defines which inbound internet traffic must reach which Kubernetes service in the cluster.
For Ingress resources to work within a cluster you must deploy an Ingress Controller prior to deploying an Ingress. The Ingress controller allows the Internet traffic to enter in the Kubernetes cluster. Once it is set up correctly we can create Ingress resources in the cluster and route internet traffic to the services. Note that Ingress resources are namespace-specific.
Below is a sample code for an Ingress resource:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: service1-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "*"
spec:
rules:
- host: aks.example.com
http:
paths:
- backend:
serviceName: service1
servicePort: 8089
Common Issues Faced when Using Nginx Ingress Controller with Default Settings
Headers and Body play vital roles in performing operations to an API service running in AKS. Ingresses allow the services running in AKS to talk to the outside world seamlessly. However one might face blockers when appropriate annotations and custom headers are not allowed in the configurations of the Nginx Ingress controller while the traffic hits the API service. There are a couple of ways to manipulate the configurations of the Ingress controller. One is by adding config maps that are connected to the Ingress controller deployment which affects the entire cluster, and the other is by adding necessary annotations to the Ingress resource, this way of manipulating configurations can affect only the namespace in which the Ingress resource is deployed.
Following are some of the common issues which we faced.
Rewrite Target
In the Ingress resource one must add nginx.ingress.kubernetes.io/rewrite-target annotations to redirect the internet traffic to a specific service of the application deployed in the cluster. The path
key in the below YAML snippet has the value with which the URL will get rewritten. For instance, if the URL is http://aks.example.com/service1, then the URL will get rewritten as http://aks.example.com and then accessed inside the pod.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: service1-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: aks.example.com
http:
paths:
- path: /service1
backend:
serviceName: service1
servicePort: 80
Blocked Custom-Headers
Ingresses, by default block the custom header which are used by a API service running in the AKS. These custom headers must be allowed through ingress controller so that it reaches the API service from the outside world. To fix this issue we make use of annotations and config maps that adhere to the Nginx ingress controller.
We create this config map following is YAML code for the config map.
apiVersion: v1
data:
X-Different-Name: "true"
X-Request-Start: t=${msec}
X-Using-Nginx-Controller: "true"
kind: ConfigMap
metadata:
name: custom-headers
namespace: ingress-nginx
We also need to add the annotation to the Ingress of that service as shown below.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/cors-allow-headers: Ocp-Apim-Subscription-Key,Ocp-Apim-Trace, Authorization, Content-Type
Ocp-Apim-Subscription-Key and Ocp-Apim-Trace are custom headers which we need to use to make use of Azure API management resource.
Underscores in Headers Are Not Allowed
Application developers have a habit of creating API services which accept headers that may have underscores in them, by default, the Nginx ingress controllers do not accept custom headers and headers containing underscores. To fix this issue, we edit the config maps that adheres to Nginx ingress controller.
When you create an Ingress controller it also creates a default config map know as nginx-configuration we edit this config map and add data to it. Following is YAML code for the config map.
apiVersion: v1
data:
enable-underscores-in-headers: "true"
proxy-set-headers: ingress-nginx/custom-headers
kind: ConfigMap
metadata:
labels:
app: ingress-nginx
name: nginx-configuration
namespace: ingress-nginx
Limited Proxy Body Size
In some cases, the body of an API service requires data as a payload which is of larger size than the limit. To increase the size of data supplied as the payload of the API we need to increase the PROXY BODY SIZE
allowed by the Ingresses.
To fix this issue we need to use the annotations in the ingresses. We use nginx.ingress.kubernetes.io/proxy-body-size: 10m. annotation in the Ingress of that particular service.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 10m
Enable Cross-Origin Resource Sharing (CORS) in Ingress
According to Wikipedia, "Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served".
To enable CORS we need to add the three main annotations in the ingresses as shown below.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/cors-allow-methods: '*'
nginx.ingress.kubernetes.io/cors-allow-origin: '*'
nginx.ingress.kubernetes.io/enable-cors: "true"
Similarly, based on application needs there are many annotations which you can use to enable the features in the Ingress.
For more information on annotation of Ingress follow the link.