Using Internal Nginx Ingress with Azure Kubernetes Service

As we know, we need an ingress controller in Kubernetes to route traffic through a single IP address and for TLS termination operations.

Although various Service Mesh technologies are preferred for these types of operations, in some cases we need an ingress controller like Nginx depending on the type and size of the system. For example, if the domain is not too complex or if we are using a Windows node in the Kubernetes cluster.

Sidecar proxies can be a problem.

In this article context, I will try to address how we can use an internal nginx ingress controller (open-source one) on Azure Kubernetes Service (AKS), which is not publicly exposed to the outside.

Before we begin:

Latest Azure CLI
Azure Kubernetes Service
Helm 3 CLI

NOTE: You can follow the AKS installation steps here.

Creating Nginx Ingress Controller

We will use the following command line in order to perform the installation process of the nginx controller within the AKS cluster.

kubectl create namespace ingress-private

helm install my-ingress stable/nginx-ingress \
    --namespace=ingress-private \
    --set rbac.create=true \
    --set controller.service.loadBalancerIP="10.240.2.2" \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io\/os"=linux \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"="true"

There are a few things that we should take into account.

If RBAC is not enabled in the AKS cluster, we need to set it to “false”.
We need to provide an unused internal IP address within the virtual network as a load balancer IP address in order to not create the ingress controller with a dynamic public IP.
Then we need to add the service.beta.kubernetes.io/azure-load-balancer-internal: “true” annotation to specify that it will be an internal load balancer.
Also, the nginx ingress controller and nginx ingress default backend must be scheduled on the “linux” node. For this reason, we need to define the node selector as “linux“.

Now let’s run the following command line and wait for the nginx ingress controller to get the “EXTERNAL-IP” address as follows.

kubectl get svc -n ingress-private -w

Creating the Helm Chart

Let’s create a helm chart called “mytodoapp-api” to deploy and test a sample application on the AKS.

helm create mytodoapp-api

Then we need to update “values.yaml” file of the chart as follows.

# Default values for mytodoapp-api.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: mytodotestregistry.azurecr.io/mytodoapp-api:01
  pullPolicy: IfNotPresent

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: true
  annotations: {
    kubernetes.io/ingress.class: "nginx",
    nginx.ingress.kubernetes.io/rewrite-target: /$1
  }
  hosts:
    - host:
      paths:
        - /mytodoapp-api/?(.*)
  tls: []

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

nodeSelector:
  beta.kubernetes.io/os: windows

tolerations: []

affinity: {}

In the “image.repository” section, I have specified that I will use a windows container image, which contains an ASP.NET Framework Web API project, in order to perform an example. Then, in the “nodeSelector” section, I have performed the necessary definition in order to be get scheduled this pod in the “windows” node.

The important point here is the “ingress” settings. We need to add kubernetes.io/ingress.class: “nginx” annotation in order for nginx to find ingress automatically. Also, “rewrite-target” annotation for path forwarding.

After adding the annotations, in the “paths” section, we have specified the path that we want the application to be routed to.

Now, let’s deploy this chart as follows.

helm install mytodoapp-api mytodoapp-api

Let’s test it

Now, in order to perform test operation, let’s forward the port “80” of the nginx ingress controller to local “8080” port as follows.

kubectl port-forward svc/my-ingress-nginx-ingress-controller -n ingress-private 8080:80

Then, in order to test the path-based routing, let’s send a request to the path we have specified for the sample API in the helm.

As we have seen above, the nginx ingress controller has performed the routing operation of the request to the API which we specified.

References

https://docs.microsoft.com/en-us/azure/aks/ingress-internal-ip?WT.mc_id=DT-MVP-5003382
https://github.com/helm/charts/tree/master/stable/nginx-ingress