chapter2.tex (19787B)
1 A Helm Package follows a strict structure of files and folders. 2 The typical structure looks like this\footnote{The test-connection.yaml file can be named anything. 3 This is the default name you get in the official template.}: 4 \dirtree{% 5 .1 . 6 .2 Chart.yaml. 7 .2 README.md. 8 .2 templates. 9 .3 \_helpers.tpl. 10 .3 NOTES.txt. 11 .3 tests. 12 .4 test-connection.yaml. 13 .2 values.yaml. 14 } 15 16 \section{The \enquote{Chart.yaml}} 17 This is the file defining the metadata of the Helm Chart. 18 Important fields are the name, description, type, version and appVersion fields. 19 Most of these are self-explanatory. 20 Below the special fields will be explained. 21 22 \begin{figure}[h] 23 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 24 apiVersion: v2 25 name: matrix-neoboard-widget 26 description: A whiteboard widget for the Element messenger 27 type: application 28 version: 0.1.0 29 appVersion: "0.0.0" 30 home: https://github.com/nordeck/matrix-neoboard 31 \end{minted} 32 \caption{A simple application Chart.yaml}\label{code:Chart.yaml} 33 \end{figure} 34 35 \subsection{The \emph{appVersion} field} 36 The appVersion field is referring to the version of the application. 37 It can be different from the version of the Helm chart which is defined in the version field and is expected to contain the tag of the application's docker image. 38 Be aware that this is not semver\cite{helmauthorsAppVersionsField} or similar but instead is an opaque string as Helm won't make assumptions about the version of an application. 39 \subsection{The \emph{maintainers} field} 40 Another section of the Chart.yaml is the \enquote{maintainers} field which allows you to set the maintainers of the application. 41 This is not mandatory but useful if you publish this Helm chart to some places. 42 It is an array which contains a name, an email and a url field and is meant to contain each maintainer that works on the chart. 43 However, in a company setting it can also be used to just refer to the company instead. 44 45 \section{The \enquote{values.yaml}} 46 47 \subsection{Images} 48 49 The core of every application in Kubernetes is the image used for deploying it. 50 This is being done in the \enquote{image} section of the \gls{values}. 51 52 \begin{figure}[h] 53 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 54 # This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/ 55 image: 56 repository: nginx 57 # This sets the pull policy for images. 58 pullPolicy: IfNotPresent 59 # Overrides the image tag whose default is the chart appVersion. 60 tag: "" 61 # This is for the secretes for pulling an image from a private repository more information can be found here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ 62 imagePullSecrets: [] 63 \end{minted} 64 \caption{The \enquote{image} section of the \gls{values}}\label{code:image_section} 65 \end{figure} 66 67 Things to note here are the 3 fields it should contain: 68 69 \begin{enumerate} 70 \item{ 71 The repository which sets the image name. 72 This would also include things like \enquote{ghcr.io} or other custom repositories used. 73 } 74 \item{ 75 The \enquote{pullPolicy} which defines how often it is pulled 76 By default this should be \enquote{IfNotPresent}. 77 For latest tags it automatically defaults however to \enquote{Always} which, as the name says, will always pull the image when a \Gls{pod} is started. 78 } 79 \item{ 80 The \enquote{tag} field defines the value after the colon in a docker image. 81 This should stay as an empty string by default since it will be pulled from the chart's \enquote{appVersion} field usually. 82 It is meant to allow a consumer to change this if they need to. 83 } 84 \end{enumerate} 85 86 Additionally, there is the \enquote{imagePullSecrets} field which allows you to pull from private repositories. For more information on this take a look at \url{https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/}. 87 88 89 \subsection{Service Account} 90 91 \begin{figure}[h] 92 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 93 # This section builds out the service account more information can be found here: https://kubernetes.io/docs/concepts/security/service-accounts/ 94 serviceAccount: 95 # Specifies whether a service account should be created 96 create: true 97 # Automatically mount a ServiceAccount's API credentials? 98 automount: true 99 # Annotations to add to the service account 100 annotations: {} 101 # The name of the service account to use. 102 # If not set and create is true, a name is generated using the fullname template 103 name: "" 104 \end{minted} 105 \caption{The \enquote{serviceAccount} section of the \gls{values}}\label{code:service_account_section} 106 \end{figure} 107 108 Service Accounts are required for accessing the resources of the \gls{k8s} cluster itself. 109 They are scoped accounts to the cluster and require most likely more setup in the templates to actually be useful. 110 They are commonly used by operators or similar things which listen to or write to resources in the cluster. 111 112 \subsection{Service and Ingress} 113 114 Ingresses and services usually come in pairs in a production ready application. 115 116 \begin{figure}[h] 117 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 118 service: 119 # This sets the service type more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types 120 type: ClusterIP 121 # This sets the ports more information can be found here: https://kubernetes.io/docs/concepts/services-networking/service/#field-spec-ports 122 port: 80 123 124 # This block is for setting up the ingress for more information can be found here: https://kubernetes.io/docs/concepts/services-networking/ingress/ 125 ingress: 126 enabled: false 127 className: "" 128 annotations: {} 129 # kubernetes.io/ingress.class: nginx 130 # kubernetes.io/tls-acme: "true" 131 hosts: 132 - host: chart-example.local 133 paths: 134 - path: / 135 pathType: ImplementationSpecific 136 tls: [] 137 # - secretName: chart-example-tls 138 # hosts: 139 # - chart-example.local 140 \end{minted} 141 \caption{The \enquote{service} section and the \enquote{ingress} section of the \gls{values}}\label{code:service_and_ingress_section} 142 \end{figure} 143 144 A service consists of the service type and a port. 145 The type of service generally will be \enquote{ClusterIP} for production clusters. 146 However, there are users which might want to prefer \enquote{NodePort} as a type instead of using an Ingress. 147 Therefor it should be left as an option for users. 148 149 For the port on the other hand it depends on the application. 150 It is only useful to be kept as an option if the container image at runtime allows adjusting it. 151 This is incredibly useful for \enquote{NodePort} users and should be preferred. 152 It should default however to the application default to avoid confusion in case an end-user has to debug it. 153 You can find more information about the service resource in \gls{k8s} at \url{https://kubernetes.io/docs/concepts/services-networking/service}. 154 155 \bigskip 156 The ingress of a Helm chart is used to represent the public access point of an application. 157 It is used to define one or multiple hosts or subpaths for the application. 158 They should stay in the same pattern as above since ingresses do depend a lot on the user and their ingress software used. 159 Additionally this is where the secrets and hosts for the tls certificate attached to the ingress are defined. 160 It is safe to assume that a \gls{k8s} cluster has means to provide this certificate and therefor should not be part of the application chart itself. 161 162 \subsection{Volumes} 163 164 Volumes and volume mounts are a way to describe storage in \gls{k8s}. 165 In terms of a Helm chart there are 2 things to differenciate here. 166 On the one side we have mandatory storage which an application requires. 167 Usually this is being named \enquote{persistence} in a Helm chart and takes just the storage size and access modes definition. 168 169 For example: 170 171 \begin{figure}[h] 172 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 173 ## Persistence configuration for the media repository function. 174 ## This PVC will be mounted in either Synapse or a media_repo worker. 175 ## 176 ## NB; If you want to be able to scale this, you will have to set the 177 ## accessMode to RWX/ReadWriteMany. 178 ## 179 persistence: 180 enabled: true 181 # existingClaim: synapse-data 182 183 # storageClass: "-" 184 accessMode: ReadWriteOnce 185 size: 10Gi 186 \end{minted} 187 \caption{The \enquote{persistence} section of the \gls{values}}\label{code:persistence_section} 188 \end{figure} 189 190 This example defines that the persistence is enabled, does not use an existing \Gls{pvc}. 191 It also says that we use the \enquote{ReadWriteOnce} access mode meaning only one pod at a time can use it\cite{KubernetesPersistentVolume}. 192 Last but not least it also defines that the size must be \qty{10}{\giga\byte} for this storage. 193 The storage class is not set which means the cluster default is used. 194 As there can be multiple storage providers it is desirable to have this as an option to allow a user to change this based on their cluster. 195 196 Additionally, one could also not set \enquote{size} and \enquote{accessMode} and instead define an existing \gls{pvc}. 197 That way the underlying \gls{pvc} will not be managed by the chart. 198 \bigskip 199 200 In addition to required storage, there is the option to provide means to add optional user defined storage. 201 The use for this depends on the application type. 202 It allows defining the same volumes and volumeMounts definitions as explained\cite{Pods} in the \gls{pod} resource. 203 As we do not know if in the end the user keeps using our image or forks it, it is a good idea to allow this flexiblity. 204 205 Usually it looks something like this example: 206 \begin{figure}[h] 207 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 208 # Additional volumes on the output Deployment definition. 209 volumes: [] 210 # - name: foo 211 # secret: 212 # secretName: mysecret 213 # optional: false 214 # Additional volumeMounts on the output Deployment definition. 215 216 volumeMounts: [] 217 # - name: foo 218 # mountPath: "/etc/foo" 219 # readOnly: true 220 \end{minted} 221 \caption{The \enquote{volumes} section and the \enquote{volumeMounts} section of the \gls{values}}\label{code:volumes_section} 222 \end{figure} 223 224 \subsection{Security Contexts} 225 Security Contexts allow changing the rules of the sandbox. 226 They get as is added to a \Gls{deployment resource}. 227 228 \begin{figure}[h] 229 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 230 podSecurityContext: {} 231 # fsGroup: 2000 232 233 securityContext: {} 234 # capabilities: 235 # drop: 236 # - ALL 237 # readOnlyRootFilesystem: true 238 # runAsNonRoot: true 239 # runAsUser: 1000 240 \end{minted} 241 \caption{The security context sections of the \gls{values}}\label{code:security_section} 242 \end{figure} 243 244 Notable things you might want by default but need Docker adjustments are: 245 246 \begin{enumerate} 247 \item{ 248 \enquote{readOnlyRootFilesystem} should be enabled. 249 This ensures that the system never writes to the temporary filesystem of the pod. 250 As a result of this an attacker has a harder time to inject changes into a Pod which would affect the end-user in case a container has been compromised. 251 For \enquote{/tmp} you should prefer an \enquote{emptyDir} volume instead. 252 } 253 \item{ 254 \enquote{runAsNonRoot}, \enquote{fsGroup} and \enquote{runAsUser} should be set to a non root user. 255 This requires docker changes to work. 256 This ensures that escaping the sandbox is made harder than it would be with a root user. 257 } 258 \item{ 259 \enquote{capabilities} should default to dropping all. 260 Ideally these should be tightly scoped. 261 Depending on the application common ones are network related and chroot related capabilities with webservers. 262 } 263 \end{enumerate} 264 265 \subsection{Resources} 266 Resources allow \gls{k8s} to better schedule the pods spawned by the application. 267 268 I suggest \url{https://home.robusta.dev/blog/stop-using-cpu-limits} for further information on this. 269 270 \begin{figure}[h] 271 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 272 resources: {} 273 # We usually recommend not to specify default resources and to leave this as a conscious 274 # choice for the user. This also increases chances charts run on environments with little 275 # resources, such as Minikube. If you do want to specify resources, uncomment the following 276 # lines, adjust them as necessary, and remove the curly braces after 'resources:'. 277 # limits: 278 # cpu: 100m 279 # memory: 128Mi 280 # requests: 281 # cpu: 100m 282 # memory: 128Mi 283 \end{minted} 284 \caption{The \enquote{resources} section of the \gls{values}}\label{code:resources_section} 285 \end{figure} 286 287 \subsection{Probes} 288 \Gls{k8s} comes with 3 types of probes\cite{ConfigureLivenessReadiness}. 289 2 of these are commonly used in applications, which are the \enquote{readiness} and the \enquote{liveness} probes. 290 The \enquote{startup} probe is only used when an application is slow to start and there is a longer waiting time to be expected. 291 292 Generally you want to have a health endpoint for this on your application which can be pinged. 293 Alternatively this also can be a command within the pod. 294 Usually this also is a command that can be fixed for the Helm chart, but when you do that you should have the timeouts exposed in the \gls{values}. 295 296 \subsection{Auto scaling} 297 Auto scaling is the automation of replication. 298 For this to work your application needs to support being able to be horizontally scalable. 299 This means it can run multiple times next to each other without causing inconsistent state or other side effects that may affect a user. 300 301 With auto scaling you can make it automatically add pods or remove pods based on the demand which can be measured using CPU or memory usage. 302 It also allows you to define the maximum and minimum replicas. 303 304 \begin{figure}[h] 305 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 306 #This section is for setting up autoscaling more information can be found here: https://kubernetes.io/docs/concepts/workloads/autoscaling/ 307 autoscaling: 308 enabled: false 309 minReplicas: 1 310 maxReplicas: 100 311 targetCPUUtilizationPercentage: 80 312 # targetMemoryUtilizationPercentage: 80 313 \end{minted} 314 \caption{The \enquote{autoscaling} section of the \gls{values}}\label{code:autoscaling_section} 315 \end{figure} 316 317 \subsection{Misc} 318 Apart from well defined sections there are also some ungrouped fields available. 319 320 The \enquote{replicaCount} here is the amount of pods being spawned. 321 Contrary to auto scaling this is a constant amount which \gls{k8s} will always try to aim for. 322 It however will respect the resource requests and limits. 323 This means that it might not be able to fulfill the amount if there are not enough resources available on a cluster. 324 325 \bigskip 326 There are also overrides which allow an admin to rename the deployed chart. 327 This is useful when a chart changed behavior and the calculation of a name changed or when it was migrated from another deployment and namespace of deployment changed which usually is part of the full name. 328 329 \bigskip 330 Annotations and labels are useful for various things like interaction with external tooling. 331 They are usually deployment specific. 332 333 \bigskip 334 335 \enquote{nodeSelector}, \enquote{tolerations} and \enquote{affinity} are generally used to influence how an application is being deployed within a cluster. 336 The nodeSelector value can be used to for example deploy it on a specific node with a specific label. 337 338 The tolerations are helping with nodes which have taints. 339 Taints prevent scheduling unless tolerated. 340 A common example of a taint is the control plane taint for the control plane nodes of the cluster. 341 It is usually used to restrict a node for important core tasks of the cluster. 342 343 \enquote{Affinity} is another more flexible way for the application to define where it can get scheduled. 344 A common usecase is to use anti-affinity where pods \enquote{repel} each other so you have redundancy of replicas across the cluster. 345 You can configure it to not allow a second pod of the same application on the same node. 346 \begin{figure}[h] 347 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{yaml} 348 # This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ 349 replicaCount: 1 350 351 # This is to override the chart name. 352 nameOverride: "" 353 fullnameOverride: "" 354 355 # This is for setting Kubernetes Annotations to a Pod. 356 # For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/ 357 podAnnotations: {} 358 # This is for setting Kubernetes Labels to a Pod. 359 # For more information checkout: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ 360 podLabels: {} 361 362 nodeSelector: {} 363 364 tolerations: [] 365 366 affinity: {} 367 \end{minted} 368 \caption{Values which affect the pods or the deployment but are not in a specific group of things}\label{code:misc_values} 369 \end{figure} 370 371 \section{The \enquote{NOTES.txt}} 372 The \enquote{NOTES.txt} is a special template file, which allows you to display a message at the end of an installation or upgrade. 373 It behaves like any other template and is then rendered to the console after a successful deployment. 374 375 Commonly it contains information about possible manual tasks you want to take like creating the initial user and also the information where you can reach the deployment. 376 377 \begin{figure}[h] 378 \begin{minted}[numbers=left, frame=lines,breaklines,breakanywhere,samepage=false]{jinja} 379 1. Get the application URL by running these commands: 380 {{- if .Values.ingress.enabled }} 381 {{- range $host := .Values.ingress.hosts }} 382 {{- range .paths }} 383 http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }} 384 {{- end }} 385 {{- end }} 386 {{- else if contains "NodePort" .Values.service.type }} 387 export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "foo.fullname" . }}) 388 export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") 389 echo http://$NODE_IP:$NODE_PORT 390 {{- else if contains "LoadBalancer" .Values.service.type }} 391 NOTE: It may take a few minutes for the LoadBalancer IP to be available. 392 You can watch its status by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "foo.fullname" . }}' 393 export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "foo.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") 394 echo http://$SERVICE_IP:{{ .Values.service.port }} 395 {{- else if contains "ClusterIP" .Values.service.type }} 396 export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "foo.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") 397 export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}") 398 echo "Visit http://127.0.0.1:8080 to use your application" 399 kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT 400 {{- end }} 401 \end{minted} 402 \caption{A standard NOTES.txt}\label{code:NOTES.txt} 403 \end{figure} 404 405 \section{\_helpers.tpl} 406 407 The helpers file is a file which does not get rendered. 408 It however contains a bunch of useful global variables in its generated default form. 409 Important things like the deployment's full name, common labels for identifying the deployment, name of the service account and a combination of name and version. 410 Beyond that it can be extended for anything that may be needed to be consistent across multiple templates.