dominik/k3s
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

paragraph order changed

Browse Source
This commit is contained in:
Dominik Chilla 2020-09-25 23:32:25 +02:00
parent c6fbdee6e4
commit 2541b4083d
1 changed files with 287 additions and 288 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

575
README.md
View File

@ -11,17 +11,11 @@
* [Disable Traefik-ingress](#disable-traefik-ingress) * [Disable Traefik-ingress](#disable-traefik-ingress)
* [Enable NGINX-ingress](#enable-nginx-ingress) * [Enable NGINX-ingress](#enable-nginx-ingress)
* [Installation](#install-nginx-ingress) * [Installation](#install-nginx-ingress)
* [Change service type from NodePort to LoadBalancer](#nginx-ingress-loadbalancer)
* [Enable nginx-ingress tcp- and udp-services for apps other than http/s](#nginx-ingress-tcp-udp-enabled)
* [Enable client-IP transparency and expose TCP-port 9000](#enable-client-ip-transp-expose-tcp-9000)
* [Deploy my-nginx-service](#deploy-my-nginx-service)
* [Stick the nginx-ingress controler and my-nginx app together](#stick-nginx-ingress-and-tcp-service)
* [Test exposed app on TCP-port 9000](#test-nginx-ingress-and-tcp-service)
* [Cert-Manager (references ingress controller)](#cert-manager) * [Cert-Manager (references ingress controller)](#cert-manager)
* [Installation](#cert-manager-install) * [Installation](#cert-manager-install)
* [Let´s Encrypt issuer](#cert-manager-le-issuer) * [Let´s Encrypt issuer](#cert-manager-le-issuer)
* [Deploying a LE-certificate](#cert-manager-ingress)
* [Troubleshooting](#cert-manager-troubleshooting) * [Troubleshooting](#cert-manager-troubleshooting)
* [Running DaemonSets on `hostPort`](#running-daemonsets)
* [HELM charts](#helm) * [HELM charts](#helm)
* [Create a chart](#helm-create) * [Create a chart](#helm-create)
* [Install local chart without packaging](#helm-install-without-packaging) * [Install local chart without packaging](#helm-install-without-packaging)
@ -30,7 +24,13 @@
* [Get status of deployed chart](#helm-status) * [Get status of deployed chart](#helm-status)
* [Get deployment history](#helm-history) * [Get deployment history](#helm-history)
* [Rollback](#helm-rollback) * [Rollback](#helm-rollback)
* [Examples](#examples)
* [Enable nginx-ingress tcp- and udp-services for apps other than http/s](#nginx-ingress-tcp-udp-enabled)
* [Enable client-IP transparency and expose TCP-port 9000](#enable-client-ip-transp-expose-tcp-9000)
* [Deploy my-nginx-service](#deploy-my-nginx-service)
* [Stick the nginx-ingress controler and my-nginx app together](#stick-nginx-ingress-and-tcp-service)
* [Test exposed app on TCP-port 9000](#test-nginx-ingress-and-tcp-service)
* [Running DaemonSets on `hostPort`](#running-daemonsets)
# Install k3s <a name="install-k3s"></a> # Install k3s <a name="install-k3s"></a>
https://k3s.io/: https://k3s.io/:
@ -81,9 +81,6 @@ spec:
max: max:
cpu: 500m cpu: 500m
memory: 1Gi memory: 1Gi
min:
cpu: 10m
memory: 4Mi
type: Container type: Container
``` ```
`kubectl apply -f namespace-devel-limitranges.yaml` `kubectl apply -f namespace-devel-limitranges.yaml`
@ -115,9 +112,6 @@ spec:
max: max:
cpu: 500m cpu: 500m
memory: 1Gi memory: 1Gi
min:
cpu: 10m
memory: 4Mi
type: Container type: Container
``` ```
`kubectl apply -f namespace-staging-limitranges.yaml` `kubectl apply -f namespace-staging-limitranges.yaml`
@ -143,9 +137,6 @@ spec:
- defaultRequest: - defaultRequest:
cpu: 50m cpu: 50m
memory: 4Mi memory: 4Mi
min:
cpu: 10m
memory: 4Mi
type: Container type: Container
``` ```
`kubectl apply -f namespace-prod-limitranges.yaml` `kubectl apply -f namespace-prod-limitranges.yaml`
@ -173,225 +164,32 @@ Finally `systemctl daemon-reload` and `systemctl restart k3s`
## Enable NGINX-ingress <a name="enable-nginx-ingress"></a> ## Enable NGINX-ingress <a name="enable-nginx-ingress"></a>
### Installation <a name="install-nginx-ingress"></a> ### Installation <a name="install-nginx-ingress"></a>
https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal https://kubernetes.github.io/ingress-nginx/deploy/#using-helm
### Change service type from NodePort to LoadBalancer <a name="nginx-ingress-loadbalancer"></a>
`kubectl edit service -n ingress-nginx ingress-nginx-controller` and change `type: NodePort` to `type: LoadBalancer`
Port 80 and 443 should listen now on an *External-IP* `kubectl get all --all-namespaces`:
``` ```
[...] kubectl create ns ingress-nginx
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
[...] helm install my-release ingress-nginx/ingress-nginx -n ingress-nginx
ingress-nginx service/ingress-nginx-controller-admission ClusterIP 10.43.174.128 <none> 443/TCP 35m
ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.237.255 10.62.94.246 80:30312/TCP,443:30366/TCP 35m
[...]
```
Test: `curl -s http://<External-IP>` should return well known nginx-404-page:
```
dominik@muggler:~$ curl -s http://10.62.94.246
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>
``` ```
### Enable nginx-ingress tcp- and udp-services for apps other than http/s <a name="nginx-ingress-tcp-udp-enabled"></a> `kubectl -n ingress-nginx get all`:
Docs: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/ ```
NAME READY STATUS RESTARTS AGE
pod/svclb-my-release-ingress-nginx-controller-m6gxl 2/2 Running 0 110s
pod/my-release-ingress-nginx-controller-695774d99c-t794f 1/1 Running 0 110s
`kubectl edit deployment -n ingress-nginx ingress-nginx-controller` and search for `spec:`/`template`/`spec`/`containers` section: NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
``` service/my-release-ingress-nginx-controller-admission ClusterIP 10.43.116.191 <none> 443/TCP 110s
[...] service/my-release-ingress-nginx-controller LoadBalancer 10.43.55.41 192.168.178.116 80:31110/TCP,443:31476/TCP 110s
spec:
[...]
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
spec:
containers:
- args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --ingress-class=nginx
- --configmap=ingress-nginx/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
>>> ADD
- --tcp-services-configmap=ingress-nginx/tcp-services
- --udp-services-configmap=ingress-nginx/udp-services
<<< ADD
env:
[...]
```
## Enable client-IP transparency and expose TCP-port 9000 <a name="enable-client-ip-transp-expose-tcp-9000"></a> NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
Enable client-IP transparency (X-Original-Forwarded-For) and expose my-nginx app on nginx-ingress TCP-port 9000: `kubectl edit service -n ingress-nginx ingress-nginx-controller` daemonset.apps/svclb-my-release-ingress-nginx-controller 1 1 1 1 1 <none> 110s
Find the `ports:`-section of the `ingress-nginx-controller` service and *ADD* the definition for port 9000:
```
[...]
spec:
clusterIP: 10.43.237.255
>>> CHANGE externalTrafficPolicy from Cluster to Local if original client-IP is desirable
externalTrafficPolicy: Local
<<< CHANGE
ports:
- name: http
nodePort: 30312
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 30366
port: 443
protocol: TCP
targetPort: https
>>> ADD
- name: proxied-tcp-9000
port: 9000
protocol: TCP
targetPort: 9000
<<< ADD
[...]
```
Verify nginx-ingress-controller is a Loadbalancer and listening on port 9000 with `kubectl get services -n ingress-nginx`:
```
[...]
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
[...]
ingress-nginx service/ingress-nginx-controller LoadBalancer 10.43.237.255 10.62.94.246 80:30312/TCP,443:30366/TCP,9000:31460/TCP 71m
[...]
```
### Deploy my-nginx deployment and service <a name="deploy-my-nginx-service"></a> NAME READY UP-TO-DATE AVAILABLE AGE
my-nginx-deployment.yml: deployment.apps/my-release-ingress-nginx-controller 1/1 1 1 110s
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 1
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx:alpine
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
```
Apply with `kubectl apply -f my-nginx-deployment.yml`:
```
deployment.apps/my-nginx created
service/my-nginx created
```
Test: `kubectl get all | grep my-nginx`:
```
pod/my-nginx-65c68bbcdf-xkhqj 1/1 Running 4 2d7h
service/my-nginx ClusterIP 10.43.118.13 <none> 80/TCP 2d7h
deployment.apps/my-nginx 1/1 1 1 2d7h
replicaset.apps/my-nginx-65c68bbcdf 1 1 1 2d7h
```
## Stick the nginx-ingress-controler and my-nginx app together <a name="stick-nginx-ingress-and-tcp-service"></a> NAME DESIRED CURRENT READY AGE
Finally, the nginx-ingress controller needs a port-mapping pointing to the my-nginx app. This will be done with a config-map `nginx-ingress-tcp-services-config-map.yml`, referenced earlier in the nginx-ingress deployment definition: replicaset.apps/my-release-ingress-nginx-controller-695774d99c 1 1 1 110s
```
---
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: ingress-nginx
data:
"9000": default/my-nginx:80
```
Apply with `kubectl apply -f nginx-ingress-tcp-services-config-map.yml`:
```
configmap/tcp-services created
```
Subsequently the config-map can be edited with `kubectl -n ingress-nginx edit configmap tcp-services`
**Changes to config-maps do not take effect on running pods! A re-scale to 0 and back can solve this problem: https://stackoverflow.com/questions/37317003/restart-pods-when-configmap-updates-in-kubernetes**
## Test exposed app on TCP-port 9000 <a name="test-nginx-ingress-and-tcp-service"></a>
```
dominik@muggler:~$ curl -s http://10.62.94.246:9000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```
Check logs of ingress-nginx-controller POD:
```
root@k3s-master:~# kubectl get pods --all-namespaces |grep ingress-nginx
[...]
ingress-nginx ingress-nginx-controller-d88d95c-khbv4 1/1 Running 0 4m36s
[...]
```
```
root@k3s-master:~# kubectl logs ingress-nginx-controller-d88d95c-khbv4 -f -n ingress-nginx
[...]
[10.62.94.1] [23/Aug/2020:16:38:33 +0000] TCP 200 850 81 0.001
[...]
```
Check logs of my-nginx POD:
```
root@k3s-master:/k3s# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-nginx-65c68bbcdf-xkhqj 1/1 Running 0 90m
```
```
kubectl logs my-nginx-65c68bbcdf-xkhqj -f
[...]
10.42.0.18 - - [23/Aug/2020:16:38:33 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.0" "-"
[...]
``` ```
# Cert-Manager (references ingress controller) <a name="cert-manager"></a> # Cert-Manager (references ingress controller) <a name="cert-manager"></a>
@ -456,7 +254,7 @@ spec:
``` ```
`kubectl apply -f lets-encrypt-cluster-issuers.yaml` `kubectl apply -f lets-encrypt-cluster-issuers.yaml`
## Deploying a LE-certificate ## Deploying a LE-certificate <a name="cert-manager-ingress"></a>
All you need is an `Ingress` resource of class `nginx` which references a ClusterIssuer (`letsencrypt-prod-issuer`) resource: All you need is an `Ingress` resource of class `nginx` which references a ClusterIssuer (`letsencrypt-prod-issuer`) resource:
``` ```
apiVersion: networking.k8s.io/v1beta1 apiVersion: networking.k8s.io/v1beta1
@ -505,66 +303,7 @@ kubectl -n <stage> describe challenge <object>
After successfull setup perform a TLS-test: `https://www.ssllabs.com/ssltest/index.html` After successfull setup perform a TLS-test: `https://www.ssllabs.com/ssltest/index.html`
# Running DaemonSets on `hostPort` <a name="running-daemonsets"></a>
* Docs: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
* Good article: https://medium.com/stakater/k8s-deployments-vs-statefulsets-vs-daemonsets-60582f0c62d4
In this case configuration of networking in context of services is not needed.
This setup is suitable for legacy scenarios where static IP-address are required:
* inbound mailserver
* dns server
```
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: netcat-daemonset
labels:
app: netcat-daemonset
spec:
selector:
matchLabels:
app: netcat-daemonset
template:
metadata:
labels:
app: netcat-daemonset
spec:
containers:
- command:
- nc
- -lk
- -p
- "23456"
- -v
- -e
- /bin/true
env:
- name: DEMO_GREETING
value: Hello from the environment
image: dockreg-zdf.int.zwackl.de/alpine/latest/amd64:prod
imagePullPolicy: Always
name: netcat-daemonset
ports:
- containerPort: 23456
hostPort: 23456
protocol: TCP
resources:
limits:
cpu: 500m
memory: 64Mi
requests:
cpu: 50m
memory: 32Mi
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
```
# HELM charts <a name="helm"></a> # HELM charts <a name="helm"></a>
Docs: Docs:
@ -697,3 +436,263 @@ NOTES:
echo "Visit http://127.0.0.1:8080 to use your application" echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:80 kubectl --namespace default port-forward $POD_NAME 8080:80
``` ```
# Examples <a name="examples"></a>
## Enable nginx-ingress tcp- and udp-services for apps other than http/s <a name="nginx-ingress-tcp-udp-enabled"></a>
Docs: https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/
`kubectl -n ingress-nginx edit deployment.apps/my-release-ingress-nginx-controller` and search for `spec:`/`template`/`spec`/`containers` section:
```
[...]
spec:
[...]
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
spec:
containers:
- args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --ingress-class=nginx
- --configmap=ingress-nginx/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
>>> ADD
- --tcp-services-configmap=ingress-nginx/tcp-services
- --udp-services-configmap=ingress-nginx/udp-services
<<< ADD
env:
[...]
```
## Enable client-IP transparency and expose TCP-port 9000 <a name="enable-client-ip-transp-expose-tcp-9000"></a>
Enable client-IP transparency (X-Original-Forwarded-For) and expose my-nginx app on nginx-ingress TCP-port 9000
`kubectl edit service -n ingress-nginx ingress-nginx-controller`
Find the `ports:`-section of the `ingress-nginx-controller` service and *ADD* the definition for port 9000:
```
[...]
spec:
clusterIP: 10.43.237.255
>>> CHANGE externalTrafficPolicy from Cluster to Local if original client-IP is desirable
externalTrafficPolicy: Local
<<< CHANGE
ports:
- name: http
nodePort: 30312
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 30366
port: 443
protocol: TCP
targetPort: https
>>> ADD
- name: proxied-tcp-9000
port: 9000
protocol: TCP
targetPort: 9000
<<< ADD
[...]
```
Verify nginx-ingress-controller is listening on port 9000 with `kubectl -n ingress-nginx get service`:
```
[...]
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
[...]
my-release-ingress-nginx-controller LoadBalancer 10.43.55.41 192.168.178.116 80:31110/TCP,443:31476/TCP 9m6s
[...]
```
## Deploy my-nginx deployment and service <a name="deploy-my-nginx-service"></a>
my-nginx-deployment.yml:
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 1
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx:alpine
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
ports:
- port: 80
protocol: TCP
selector:
run: my-nginx
```
Apply with `kubectl apply -f my-nginx-deployment.yml`:
```
deployment.apps/my-nginx created
service/my-nginx created
```
Test: `kubectl get all | grep my-nginx`:
```
pod/my-nginx-65c68bbcdf-xkhqj 1/1 Running 4 2d7h
service/my-nginx ClusterIP 10.43.118.13 <none> 80/TCP 2d7h
deployment.apps/my-nginx 1/1 1 1 2d7h
replicaset.apps/my-nginx-65c68bbcdf 1 1 1 2d7h
```
## Stick the nginx-ingress-controler and my-nginx app together <a name="stick-nginx-ingress-and-tcp-service"></a>
Finally, the nginx-ingress controller needs a port-mapping pointing to the my-nginx app. This will be done with a config-map `nginx-ingress-tcp-services-config-map.yml`, referenced earlier in the nginx-ingress deployment definition:
```
---
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: ingress-nginx
data:
"9000": default/my-nginx:80
```
Apply with `kubectl apply -f nginx-ingress-tcp-services-config-map.yml`:
```
configmap/tcp-services created
```
Subsequently the config-map can be edited with `kubectl -n ingress-nginx edit configmap tcp-services`
**Changes to config-maps do not take effect on running pods! A re-scale to 0 and back can solve this problem: https://stackoverflow.com/questions/37317003/restart-pods-when-configmap-updates-in-kubernetes**
## Test exposed app on TCP-port 9000 <a name="test-nginx-ingress-and-tcp-service"></a>
```
dominik@muggler:~$ curl -s http://10.62.94.246:9000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```
Check logs of ingress-nginx-controller POD:
```
root@k3s-master:~# kubectl get pods --all-namespaces |grep ingress-nginx
[...]
ingress-nginx ingress-nginx-controller-d88d95c-khbv4 1/1 Running 0 4m36s
[...]
```
```
root@k3s-master:~# kubectl logs ingress-nginx-controller-d88d95c-khbv4 -f -n ingress-nginx
[...]
[10.62.94.1] [23/Aug/2020:16:38:33 +0000] TCP 200 850 81 0.001
[...]
```
Check logs of my-nginx POD:
```
root@k3s-master:/k3s# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-nginx-65c68bbcdf-xkhqj 1/1 Running 0 90m
```
```
kubectl logs my-nginx-65c68bbcdf-xkhqj -f
[...]
10.42.0.18 - - [23/Aug/2020:16:38:33 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.0" "-"
[...]
```
## Running DaemonSets on `hostPort` <a name="running-daemonsets"></a>
* Docs: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
* Good article: https://medium.com/stakater/k8s-deployments-vs-statefulsets-vs-daemonsets-60582f0c62d4
In this case configuration of networking in context of services is not needed.
This setup is suitable for legacy scenarios where static IP-address are required:
* inbound mailserver
* dns server
```
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: netcat-daemonset
labels:
app: netcat-daemonset
spec:
selector:
matchLabels:
app: netcat-daemonset
template:
metadata:
labels:
app: netcat-daemonset
spec:
containers:
- command:
- nc
- -lk
- -p
- "23456"
- -v
- -e
- /bin/true
env:
- name: DEMO_GREETING
value: Hello from the environment
image: dockreg-zdf.int.zwackl.de/alpine/latest/amd64:prod
imagePullPolicy: Always
name: netcat-daemonset
ports:
- containerPort: 23456
hostPort: 23456
protocol: TCP
resources:
limits:
cpu: 500m
memory: 64Mi
requests:
cpu: 50m
memory: 32Mi
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
```