Author: Sanders

Eventually i would like to get K8 cluster based on AWS (EKS)

Official AWS ALB documentation

But…it has some pitfalls with the roles, so let’s figure out

What needs?

1. Install eksctl to local machine (it needs to setup cluster from terminal instead of UI)
2. Prepare simple eksctl yaml file:

# A simple example of ClusterConfig object:
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: clusterName
  region: us-east-1
  version: "1.17"

vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: true

nodeGroups:
  - name: ng-1
    instanceType: t2.large
    desiredCapacity: 1

fargateProfiles:
  - name: fp-default
    selectors:
      - namespace: default
      - namespace: kube-system

cloudWatch:
  clusterLogging:
    enableTypes: ["audit", "authenticator", "controllerManager", "scheduler"]

3. Apply deployment yaml with ingress and service blocks

Example of the ingress part

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: yourapp-ingress
  labels:
    app: yourapp-app
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing

4. Install aws-load-balancer

Here i created the script to create all appropriate roles

#!/bin/bash

REGION=eu-central-1
CLUSTER=cka-cert
ACCOUNT=YOUR_ACCOUNT_ID
VPC_ID=$(eksctl get cluster cka-cert --output json | jq '.[0] .ResourcesVpcConfig .VpcId' | xargs)

#stop on any error
set -e

eksctl utils associate-iam-oidc-provider \
    --region ${REGION} \
    --cluster $CLUSTER \
    --approve

curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.1.0/docs/install/iam_policy.json

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam-policy.json

eksctl create iamserviceaccount \
  --cluster=$CLUSTER \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::${ACCOUNT}:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

helm repo add eks https://aws.github.io/eks-charts
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

#EC2 meta-data is not available for Fargate. So needs to set vpc
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \
  --set clusterName=${CLUSTER} \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=${REGION} \
  --set vpcId=${VPC_ID}

1. If you see the following error:

couldn't auto-discover subnets: unable to discover at least one subnet

Need to add into ingress yaml configuration the following annotation:

alb.ingress.kubernetes.io/subnets: <Public_subnetID 1>,<Public_subnetID 2>,<Public_subnetID 3>

2. This error means, that service role is not setup or not enough policy permissions:

UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403

Run the following command:

Helm 3:

helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update

helm install nginx-ingress nginx-stable/nginx-ingress --set controller.service.externalIPs={yourIp} --set controller.service.type=NodePort


by google
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm install project-ingress ingress-nginx/ingress-nginx

metalb

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/baremetal/deploy.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.2/deploy/static/provider/baremetal/service-nodeport.yaml

kubectl -n ingress-nginx edit service/ingress-nginx 

and add the following lines:

spec:
  externalIPs:
  - X.X.X.X
  - Y.Y.Y.Y

How i do it?

1. Create centos machine in Hetzner Control panel
2. Setup basic things like a time, hostname by ansible script.
3. Install docker by ansible script
4. Install kubernetes on master and worker nodes by ansible script.

Run the following commands as kubeadm suggested:

[root@k8s-master ~]# kubeadm init

[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config

Then install container’s networks:

We can choose flannel, weave, etc.

[root@k8s-master ~]# export kubever=$(kubectl version | base64 | tr -d '\n')
[root@k8s-master ~]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"
serviceaccount "weave-net" created
clusterrole "weave-net" created
clusterrolebinding "weave-net" created
daemonset "weave-net" created
[root@k8s-master ~]#

Connect workers to master by join command:

[root@worker-node1 ~]# kubeadm join 16.3.215.160:6443 --token abd0v3.pmhcans1g3wz3jhs  --discovery-token-ca-cert-hash sha256:841cac306cc7fd07831613e63c85

ECR private repositories

Link – Create a Secret by providing credentials on the command line