We recently made the switch from GKE to EKS. While both use Kubernetes under the hood, the experience is vastly different. GKE offers more visual interface guidance while Amazon seems to provide more documentation resources like their new video show ‘containers from the couch’.
One of the first challenges you’ll come across with either platform is the experience of linking your domain to your k8s cluster. Ideally, you don’t want to deal with any certificates and SSL traffic yourself, and in theory, there’s always a fully hosted solution by your cloud provider. In reality, you might struggle with load balancers on either the network or the application layer, you might have difficulties getting any higher-level protocols like HTTP2 and WebSockets to work, and getting the cloud provider to acknowledge your ownership of your domain is another can of worms.
With our new EKS setup, I wanted to give Fargate a try. I really like the notion of not constantly worrying about the knapsack-style issue of using up all your nodes effectively while still having computing resources to spare for a sudden traffic spike. I really didn’t expect all the roadblocks that come with that decision. In this article, I want to talk about one specific issue.
Because Fargate seems to alter some of the DNS fundamentals of how Kubernetes works on AWS, there are some applications that don’t work out of the box (or at all). Surprisingly, ingress is one of them. When researching SSL/TLS ingress for EKS, you’ll quickly find the AWS LoadBalancer Controller, formerly known as AWS ALB Ingress Controller. It’s an actively developed Kubernetes service, which acts as the bridge between your Ingress manifests and the AWS configurations to configure load balancers and target groups.
In theory, the setup is simple. You use eksctl to spin up a cluster, apply some YAML files to install the controller, and you’re off to the races. Your first ingress is just one manifest away. In practice, the setup guide is pretty outdated and uses v1 of the controller which lacks some really important functionality. And that thing is from March! Hats off to the engineers at work here, this project is moving fast.
Now the v2 guide offers two options. The helm way seems really straightforward, but Helm is a pretty big dependency to keep track of and I’d avoid installing it just for one service. The development speed of Helm is also really impressive and you have to update it constantly.
The other option, YAML manifests, seems like a less preferred way because it doesn’t go into quite as many details and doesn’t work on Fargate. One of the dependencies of the controller is a self-signed TLS certificate. When installing it via manifests, cert-manager is used for this step and it’s an even bigger dependency than Helm which doesn’t even work for our use case at all.
So to work around all these limitations, I changed most of the controller manifest to work without cert-manager and use a certificate from a file instead. You can find all of the setup in the repo, but here’s the gist of it:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.4.0
creationTimestamp: null
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: targetgroupbindings.elbv2.k8s.aws
spec:
additionalPrinterColumns:
- JSONPath: .spec.serviceRef.name
description: The Kubernetes Service's name
name: SERVICE-NAME
type: string
- JSONPath: .spec.serviceRef.port
description: The Kubernetes Service's port
name: SERVICE-PORT
type: string
- JSONPath: .spec.targetType
description: The AWS TargetGroup's TargetType
name: TARGET-TYPE
type: string
- JSONPath: .spec.targetGroupARN
description: The AWS TargetGroup's Amazon Resource Name
name: ARN
priority: 1
type: string
- JSONPath: .metadata.creationTimestamp
name: AGE
type: date
group: elbv2.k8s.aws
names:
categories:
- all
kind: TargetGroupBinding
listKind: TargetGroupBindingList
plural: targetgroupbindings
singular: targetgroupbinding
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
description: TargetGroupBinding is the Schema for the TargetGroupBinding API
properties:
apiVersion:
description:
"APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources"
type: string
kind:
description:
"Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"
type: string
metadata:
type: object
spec:
description: TargetGroupBindingSpec defines the desired state of TargetGroupBinding
properties:
networking:
description:
networking provides the networking setup for ELBV2 LoadBalancer
to access targets in TargetGroup.
properties:
ingress:
description:
List of ingress rules to allow ELBV2 LoadBalancer to
access targets in TargetGroup.
items:
properties:
from:
description:
List of peers which should be able to access
the targets in TargetGroup. At least one NetworkingPeer
should be specified.
items:
description:
NetworkingPeer defines the source/destination
peer for networking rules.
properties:
ipBlock:
description:
IPBlock defines an IPBlock peer. If specified,
none of the other fields can be set.
properties:
cidr:
description:
CIDR is the network CIDR. Both IPV4
or IPV6 CIDR are accepted.
type: string
required:
- cidr
type: object
securityGroup:
description:
SecurityGroup defines a SecurityGroup peer.
If specified, none of the other fields can be set.
properties:
groupID:
description: GroupID is the EC2 SecurityGroupID.
type: string
required:
- groupID
type: object
type: object
type: array
ports:
description:
List of ports which should be made accessible
on the targets in TargetGroup. If ports is empty or unspecified,
it defaults to all ports with TCP.
items:
properties:
port:
anyOf:
- type: integer
- type: string
description:
The port which traffic must match. When
NodePort endpoints(instance TargetType) is used, this
must be a numerical port. When Port endpoints(ip TargetType)
is used, this can be either numerical or named port
on pods. if port is unspecified, it defaults to all
ports.
x-kubernetes-int-or-string: true
protocol:
description:
The protocol which traffic must match.
If protocol is unspecified, it defaults to TCP.
enum:
- TCP
- UDP
type: string
type: object
type: array
required:
- from
- ports
type: object
type: array
type: object
serviceRef:
description: serviceRef is a reference to a Kubernetes Service and ServicePort.
properties:
name:
description: Name is the name of the Service.
type: string
port:
anyOf:
- type: integer
- type: string
description: Port is the port of the ServicePort.
x-kubernetes-int-or-string: true
required:
- name
- port
type: object
targetGroupARN:
description:
targetGroupARN is the Amazon Resource Name (ARN) for the
TargetGroup.
type: string
targetType:
description:
targetType is the TargetType of TargetGroup. If unspecified,
it will be automatically inferred.
enum:
- instance
- ip
type: string
required:
- serviceRef
- targetGroupARN
type: object
status:
description: TargetGroupBindingStatus defines the observed state of TargetGroupBinding
properties:
observedGeneration:
description: The generation observed by the TargetGroupBinding controller.
format: int64
type: integer
type: object
type: object
version: v1alpha1
versions:
- name: v1alpha1
served: true
storage: false
- name: v1beta1
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-leader-election-role
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
- aws-load-balancer-controller-leader
verbs:
- get
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-role
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods/status
verbs:
- patch
- update
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- services/status
verbs:
- patch
- update
- apiGroups:
- elbv2.k8s.aws
resources:
- targetgroupbindings
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- elbv2.k8s.aws
resources:
- targetgroupbindings/status
verbs:
- patch
- update
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- patch
- update
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- patch
- update
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-leader-election-rolebinding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: aws-load-balancer-controller-leader-election-role
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: aws-load-balancer-controller-role
subjects:
- kind: ServiceAccount
name: aws-load-balancer-controller
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
name: aws-load-balancer-controller
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: aws-load-balancer-controller
spec:
containers:
- args:
- --cluster-name=$CLUSTER_NAME
- --ingress-class=alb
- --aws-vpc-id=$VPC_ID
- --aws-region=$AWS_REGION
image: amazon/aws-alb-ingress-controller:v2.0.1
livenessProbe:
failureThreshold: 2
httpGet:
path: /healthz
port: 61779
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 10
name: controller
resources:
limits:
cpu: 200m
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
securityContext:
fsGroup: 1337
serviceAccountName: aws-load-balancer-controller
terminationGracePeriodSeconds: 10
volumes:
- name: cert
secret:
defaultMode: 420
secretName: aws-load-balancer-webhook-tls