diff --git a/README.md b/README.md index dd2e8a5..ec8df38 100644 --- a/README.md +++ b/README.md @@ -11,17 +11,11 @@ * [Disable Traefik-ingress](#disable-traefik-ingress) * [Enable NGINX-ingress](#enable-nginx-ingress) * [Installation](#install-nginx-ingress) - * [Change service type from NodePort to LoadBalancer](#nginx-ingress-loadbalancer) - * [Enable nginx-ingress tcp- and udp-services for apps other than http/s](#nginx-ingress-tcp-udp-enabled) - * [Enable client-IP transparency and expose TCP-port 9000](#enable-client-ip-transp-expose-tcp-9000) - * [Deploy my-nginx-service](#deploy-my-nginx-service) - * [Stick the nginx-ingress controler and my-nginx app together](#stick-nginx-ingress-and-tcp-service) - * [Test exposed app on TCP-port 9000](#test-nginx-ingress-and-tcp-service) * [Cert-Manager (references ingress controller)](#cert-manager) * [Installation](#cert-manager-install) * [Let´s Encrypt issuer](#cert-manager-le-issuer) + * [Deploying a LE-certificate](#cert-manager-ingress) * [Troubleshooting](#cert-manager-troubleshooting) -* [Running DaemonSets on `hostPort`](#running-daemonsets) * [HELM charts](#helm) * [Create a chart](#helm-create) * [Install local chart without packaging](#helm-install-without-packaging) @@ -30,7 +24,13 @@ * [Get status of deployed chart](#helm-status) * [Get deployment history](#helm-history) * [Rollback](#helm-rollback) - +* [Examples](#examples) + * [Enable nginx-ingress tcp- and udp-services for apps other than http/s](#nginx-ingress-tcp-udp-enabled) + * [Enable client-IP transparency and expose TCP-port 9000](#enable-client-ip-transp-expose-tcp-9000) + * [Deploy my-nginx-service](#deploy-my-nginx-service) + * [Stick the nginx-ingress controler and my-nginx app together](#stick-nginx-ingress-and-tcp-service) + * [Test exposed app on TCP-port 9000](#test-nginx-ingress-and-tcp-service) + * [Running DaemonSets on `hostPort`](#running-daemonsets) # Install k3s https://k3s.io/: @@ -81,9 +81,6 @@ spec: max: cpu: 500m memory: 1Gi - min: - cpu: 10m - memory: 4Mi type: Container ``` `kubectl apply -f namespace-devel-limitranges.yaml` @@ -115,9 +112,6 @@ spec: max: cpu: 500m memory: 1Gi - min: - cpu: 10m - memory: 4Mi type: Container ``` `kubectl apply -f namespace-staging-limitranges.yaml` @@ -143,9 +137,6 @@ spec: - defaultRequest: cpu: 50m memory: 4Mi - min: - cpu: 10m - memory: 4Mi type: Container ``` `kubectl apply -f namespace-prod-limitranges.yaml` @@ -173,225 +164,32 @@ Finally `systemctl daemon-reload` and `systemctl restart k3s` ## Enable NGINX-ingress ### Installation -https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal +https://kubernetes.github.io/ingress-nginx/deploy/#using-helm -### Change service type from NodePort to LoadBalancer -`kubectl edit service -n ingress-nginx ingress-nginx-controller` and change `type: NodePort` to `type: LoadBalancer` - -Port 80 and 443 should listen now on an *External-IP* `kubectl get all --all-namespaces`: ``` -[...] -NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -[...] -ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.43.174.128 443/TCP 35m -ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.237.255 10.62.94.246 80:30312/TCP,443:30366/TCP 35m -[...] -``` -Test: `curl -s http://` should return well known nginx-404-page: -``` -dominik@muggler:~$ curl -s http://10.62.94.246 - -404 Not Found - -

404 Not Found

-
nginx/1.19.1
- - +kubectl create ns ingress-nginx +helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +helm install my-release ingress-nginx/ingress-nginx -n ingress-nginx ``` -### Enable nginx-ingress tcp- and udp-services for apps other than http/s -Docs: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ +`kubectl -n ingress-nginx get all`: +``` +NAME READY STATUS RESTARTS AGE +pod/svclb-my-release-ingress-nginx-controller-m6gxl 2/2 Running 0 110s +pod/my-release-ingress-nginx-controller-695774d99c-t794f 1/1 Running 0 110s -`kubectl edit deployment -n ingress-nginx ingress-nginx-controller` and search for `spec:`/`template`/`spec`/`containers` section: -``` -[...] -spec: -[...] - template: - metadata: - creationTimestamp: null - labels: - app.kubernetes.io/component: controller - app.kubernetes.io/instance: ingress-nginx - app.kubernetes.io/name: ingress-nginx - spec: - containers: - - args: - - /nginx-ingress-controller - - --election-id=ingress-controller-leader - - --ingress-class=nginx - - --configmap=ingress-nginx/ingress-nginx-controller - - --validating-webhook=:8443 - - --validating-webhook-certificate=/usr/local/certificates/cert - - --validating-webhook-key=/usr/local/certificates/key ->>> ADD - - --tcp-services-configmap=ingress-nginx/tcp-services - - --udp-services-configmap=ingress-nginx/udp-services -<<< ADD - env: -[...] -``` +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +service/my-release-ingress-nginx-controller-admission ClusterIP 10.43.116.191 443/TCP 110s +service/my-release-ingress-nginx-controller LoadBalancer 10.43.55.41 192.168.178.116 80:31110/TCP,443:31476/TCP 110s -## Enable client-IP transparency and expose TCP-port 9000 -Enable client-IP transparency (X-Original-Forwarded-For) and expose my-nginx app on nginx-ingress TCP-port 9000: `kubectl edit service -n ingress-nginx ingress-nginx-controller` -Find the `ports:`-section of the `ingress-nginx-controller` service and *ADD* the definition for port 9000: -``` -[...] -spec: - clusterIP: 10.43.237.255 ->>> CHANGE externalTrafficPolicy from Cluster to Local if original client-IP is desirable - externalTrafficPolicy: Local -<<< CHANGE - ports: - - name: http - nodePort: 30312 - port: 80 - protocol: TCP - targetPort: http - - name: https - nodePort: 30366 - port: 443 - protocol: TCP - targetPort: https ->>> ADD - - name: proxied-tcp-9000 - port: 9000 - protocol: TCP - targetPort: 9000 -<<< ADD -[...] -``` -Verify nginx-ingress-controller is a Loadbalancer and listening on port 9000 with `kubectl get services -n ingress-nginx`: -``` -[...] -NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE -[...] -ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.237.255 10.62.94.246 80:30312/TCP,443:30366/TCP,9000:31460/TCP 71m -[...] -``` +NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE +daemonset.apps/svclb-my-release-ingress-nginx-controller 1 1 1 1 1 110s -### Deploy my-nginx deployment and service -my-nginx-deployment.yml: -``` -apiVersion: apps/v1 -kind: Deployment -metadata: - name: my-nginx -spec: - selector: - matchLabels: - run: my-nginx - replicas: 1 - template: - metadata: - labels: - run: my-nginx - spec: - containers: - - name: my-nginx - image: nginx:alpine - ports: - - containerPort: 80 ---- -apiVersion: v1 -kind: Service -metadata: - name: my-nginx - labels: - run: my-nginx -spec: - ports: - - port: 80 - protocol: TCP - selector: - run: my-nginx -``` -Apply with `kubectl apply -f my-nginx-deployment.yml`: -``` -deployment.apps/my-nginx created -service/my-nginx created -``` -Test: `kubectl get all | grep my-nginx`: -``` -pod/my-nginx-65c68bbcdf-xkhqj 1/1 Running 4 2d7h -service/my-nginx ClusterIP 10.43.118.13 80/TCP 2d7h -deployment.apps/my-nginx 1/1 1 1 2d7h -replicaset.apps/my-nginx-65c68bbcdf 1 1 1 2d7h -``` +NAME READY UP-TO-DATE AVAILABLE AGE +deployment.apps/my-release-ingress-nginx-controller 1/1 1 1 110s -## Stick the nginx-ingress-controler and my-nginx app together -Finally, the nginx-ingress controller needs a port-mapping pointing to the my-nginx app. This will be done with a config-map `nginx-ingress-tcp-services-config-map.yml`, referenced earlier in the nginx-ingress deployment definition: -``` ---- -apiVersion: v1 -kind: ConfigMap -metadata: - name: tcp-services - namespace: ingress-nginx -data: - "9000": default/my-nginx:80 -``` -Apply with `kubectl apply -f nginx-ingress-tcp-services-config-map.yml`: -``` -configmap/tcp-services created -``` -Subsequently the config-map can be edited with `kubectl -n ingress-nginx edit configmap tcp-services` - -**Changes to config-maps do not take effect on running pods! A re-scale to 0 and back can solve this problem: https://stackoverflow.com/questions/37317003/restart-pods-when-configmap-updates-in-kubernetes** - -## Test exposed app on TCP-port 9000 -``` -dominik@muggler:~$ curl -s http://10.62.94.246:9000 - - - -Welcome to nginx! - - - -

Welcome to nginx!

-

If you see this page, the nginx web server is successfully installed and -working. Further configuration is required.

- -

For online documentation and support please refer to -nginx.org.
-Commercial support is available at -nginx.com.

- -

Thank you for using nginx.

- - -``` -Check logs of ingress-nginx-controller POD: -``` -root@k3s-master:~# kubectl get pods --all-namespaces |grep ingress-nginx -[...] -ingress-nginx ingress-nginx-controller-d88d95c-khbv4 1/1 Running 0 4m36s -[...] -``` -``` -root@k3s-master:~# kubectl logs ingress-nginx-controller-d88d95c-khbv4 -f -n ingress-nginx -[...] -[10.62.94.1] [23/Aug/2020:16:38:33 +0000] TCP 200 850 81 0.001 -[...] -``` -Check logs of my-nginx POD: -``` -root@k3s-master:/k3s# kubectl get pods -NAME READY STATUS RESTARTS AGE -my-nginx-65c68bbcdf-xkhqj 1/1 Running 0 90m -``` -``` -kubectl logs my-nginx-65c68bbcdf-xkhqj -f -[...] -10.42.0.18 - - [23/Aug/2020:16:38:33 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.0" "-" -[...] +NAME DESIRED CURRENT READY AGE +replicaset.apps/my-release-ingress-nginx-controller-695774d99c 1 1 1 110s ``` # Cert-Manager (references ingress controller) @@ -456,7 +254,7 @@ spec: ``` `kubectl apply -f lets-encrypt-cluster-issuers.yaml` -## Deploying a LE-certificate +## Deploying a LE-certificate All you need is an `Ingress` resource of class `nginx` which references a ClusterIssuer (`letsencrypt-prod-issuer`) resource: ``` apiVersion: networking.k8s.io/v1beta1 @@ -505,66 +303,7 @@ kubectl -n describe challenge After successfull setup perform a TLS-test: `https://www.ssllabs.com/ssltest/index.html` -# Running DaemonSets on `hostPort` -* Docs: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ -* Good article: https://medium.com/stakater/k8s-deployments-vs-statefulsets-vs-daemonsets-60582f0c62d4 -In this case configuration of networking in context of services is not needed. - -This setup is suitable for legacy scenarios where static IP-address are required: -* inbound mailserver -* dns server - -``` -kind: DaemonSet -apiVersion: apps/v1 -metadata: - name: netcat-daemonset - labels: - app: netcat-daemonset -spec: - selector: - matchLabels: - app: netcat-daemonset - template: - metadata: - labels: - app: netcat-daemonset - spec: - containers: - - command: - - nc - - -lk - - -p - - "23456" - - -v - - -e - - /bin/true - env: - - name: DEMO_GREETING - value: Hello from the environment - image: dockreg-zdf.int.zwackl.de/alpine/latest/amd64:prod - imagePullPolicy: Always - name: netcat-daemonset - ports: - - containerPort: 23456 - hostPort: 23456 - protocol: TCP - resources: - limits: - cpu: 500m - memory: 64Mi - requests: - cpu: 50m - memory: 32Mi - restartPolicy: Always - securityContext: {} - terminationGracePeriodSeconds: 30 - updateStrategy: - rollingUpdate: - maxUnavailable: 1 - type: RollingUpdate -``` # HELM charts Docs: @@ -697,3 +436,263 @@ NOTES: echo "Visit http://127.0.0.1:8080 to use your application" kubectl --namespace default port-forward $POD_NAME 8080:80 ``` + +# Examples +## Enable nginx-ingress tcp- and udp-services for apps other than http/s +Docs: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ + +`kubectl -n ingress-nginx edit deployment.apps/my-release-ingress-nginx-controller` and search for `spec:`/`template`/`spec`/`containers` section: +``` +[...] +spec: +[...] + template: + metadata: + creationTimestamp: null + labels: + app.kubernetes.io/component: controller + app.kubernetes.io/instance: ingress-nginx + app.kubernetes.io/name: ingress-nginx + spec: + containers: + - args: + - /nginx-ingress-controller + - --election-id=ingress-controller-leader + - --ingress-class=nginx + - --configmap=ingress-nginx/ingress-nginx-controller + - --validating-webhook=:8443 + - --validating-webhook-certificate=/usr/local/certificates/cert + - --validating-webhook-key=/usr/local/certificates/key +>>> ADD + - --tcp-services-configmap=ingress-nginx/tcp-services + - --udp-services-configmap=ingress-nginx/udp-services +<<< ADD + env: +[...] +``` + +## Enable client-IP transparency and expose TCP-port 9000 +Enable client-IP transparency (X-Original-Forwarded-For) and expose my-nginx app on nginx-ingress TCP-port 9000 + +`kubectl edit service -n ingress-nginx ingress-nginx-controller` + +Find the `ports:`-section of the `ingress-nginx-controller` service and *ADD* the definition for port 9000: +``` +[...] +spec: + clusterIP: 10.43.237.255 +>>> CHANGE externalTrafficPolicy from Cluster to Local if original client-IP is desirable + externalTrafficPolicy: Local +<<< CHANGE + ports: + - name: http + nodePort: 30312 + port: 80 + protocol: TCP + targetPort: http + - name: https + nodePort: 30366 + port: 443 + protocol: TCP + targetPort: https +>>> ADD + - name: proxied-tcp-9000 + port: 9000 + protocol: TCP + targetPort: 9000 +<<< ADD +[...] +``` +Verify nginx-ingress-controller is listening on port 9000 with `kubectl -n ingress-nginx get service`: +``` +[...] +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +[...] +my-release-ingress-nginx-controller LoadBalancer 10.43.55.41 192.168.178.116 80:31110/TCP,443:31476/TCP 9m6s +[...] +``` + +## Deploy my-nginx deployment and service +my-nginx-deployment.yml: +``` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-nginx +spec: + selector: + matchLabels: + run: my-nginx + replicas: 1 + template: + metadata: + labels: + run: my-nginx + spec: + containers: + - name: my-nginx + image: nginx:alpine + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + name: my-nginx + labels: + run: my-nginx +spec: + ports: + - port: 80 + protocol: TCP + selector: + run: my-nginx +``` +Apply with `kubectl apply -f my-nginx-deployment.yml`: +``` +deployment.apps/my-nginx created +service/my-nginx created +``` +Test: `kubectl get all | grep my-nginx`: +``` +pod/my-nginx-65c68bbcdf-xkhqj 1/1 Running 4 2d7h +service/my-nginx ClusterIP 10.43.118.13 80/TCP 2d7h +deployment.apps/my-nginx 1/1 1 1 2d7h +replicaset.apps/my-nginx-65c68bbcdf 1 1 1 2d7h +``` + +## Stick the nginx-ingress-controler and my-nginx app together +Finally, the nginx-ingress controller needs a port-mapping pointing to the my-nginx app. This will be done with a config-map `nginx-ingress-tcp-services-config-map.yml`, referenced earlier in the nginx-ingress deployment definition: +``` +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tcp-services + namespace: ingress-nginx +data: + "9000": default/my-nginx:80 +``` +Apply with `kubectl apply -f nginx-ingress-tcp-services-config-map.yml`: +``` +configmap/tcp-services created +``` +Subsequently the config-map can be edited with `kubectl -n ingress-nginx edit configmap tcp-services` + +**Changes to config-maps do not take effect on running pods! A re-scale to 0 and back can solve this problem: https://stackoverflow.com/questions/37317003/restart-pods-when-configmap-updates-in-kubernetes** + +## Test exposed app on TCP-port 9000 +``` +dominik@muggler:~$ curl -s http://10.62.94.246:9000 + + + +Welcome to nginx! + + + +

Welcome to nginx!

+

If you see this page, the nginx web server is successfully installed and +working. Further configuration is required.

+ +

For online documentation and support please refer to +nginx.org.
+Commercial support is available at +nginx.com.

+ +

Thank you for using nginx.

+ + +``` +Check logs of ingress-nginx-controller POD: +``` +root@k3s-master:~# kubectl get pods --all-namespaces |grep ingress-nginx +[...] +ingress-nginx ingress-nginx-controller-d88d95c-khbv4 1/1 Running 0 4m36s +[...] +``` +``` +root@k3s-master:~# kubectl logs ingress-nginx-controller-d88d95c-khbv4 -f -n ingress-nginx +[...] +[10.62.94.1] [23/Aug/2020:16:38:33 +0000] TCP 200 850 81 0.001 +[...] +``` +Check logs of my-nginx POD: +``` +root@k3s-master:/k3s# kubectl get pods +NAME READY STATUS RESTARTS AGE +my-nginx-65c68bbcdf-xkhqj 1/1 Running 0 90m +``` +``` +kubectl logs my-nginx-65c68bbcdf-xkhqj -f +[...] +10.42.0.18 - - [23/Aug/2020:16:38:33 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.0" "-" +[...] +``` + +## Running DaemonSets on `hostPort` +* Docs: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ +* Good article: https://medium.com/stakater/k8s-deployments-vs-statefulsets-vs-daemonsets-60582f0c62d4 + +In this case configuration of networking in context of services is not needed. + +This setup is suitable for legacy scenarios where static IP-address are required: +* inbound mailserver +* dns server + +``` +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: netcat-daemonset + labels: + app: netcat-daemonset +spec: + selector: + matchLabels: + app: netcat-daemonset + template: + metadata: + labels: + app: netcat-daemonset + spec: + containers: + - command: + - nc + - -lk + - -p + - "23456" + - -v + - -e + - /bin/true + env: + - name: DEMO_GREETING + value: Hello from the environment + image: dockreg-zdf.int.zwackl.de/alpine/latest/amd64:prod + imagePullPolicy: Always + name: netcat-daemonset + ports: + - containerPort: 23456 + hostPort: 23456 + protocol: TCP + resources: + limits: + cpu: 500m + memory: 64Mi + requests: + cpu: 50m + memory: 32Mi + restartPolicy: Always + securityContext: {} + terminationGracePeriodSeconds: 30 + updateStrategy: + rollingUpdate: + maxUnavailable: 1 + type: RollingUpdate +``` \ No newline at end of file