DevOps Journal – Blog

Helm Multi-Values Plugin with ArgoCD's CMP

Tue, 23 Apr 2024 00:00:00 +0000

Introduction

With the current version of Helm, it’s not possible to merge multiple values files. It will override the list from the preceding values file. While trying to solve this problem, I stumbled upon this custom plugin that can take a folder of values files and split them into individual files. This plugin is what I was looking for to solve my problem. Finally, I want to integrate this plugin into my ArgoCD instance.

You’ll need to perform the following steps:

Install Helm Plugin using InitContainer of the repo server.
The configuration management plugin installation entails two steps. First, create a ConfigMap of your CMP configuration, then configure the plugin via sidecar for the repo server. For additional details, see the ArgoCD documentation.
Finally, create an Argo Application to test the multi-values plugin.

Prerequisites

Make sure you have access to the OpenShift cluster with cluster-admin privileges.
The OpenShift GitOps Operator must be installed and configured
It would help if you had some understanding of ArgoCD and Helm Charts.

Install Helm Plugin in repo server

We’ll use the custom tooling methodology to add the helm multivalues plugin to the repo server. Update the ArgoCD operator CR to add an init container to the repo spec and volumes to install the helm plugin. We must also add Helm environment variables to override default data, config, and cache paths.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
 name: openshift-gitops
 namespace: openshift-gitops
spec:
 repo:
 volumes:
 - configMap:
 name: helm-multivalues-plugin
 name: helm-multivalues-plugin
 - name: helm-custom-config
 emptyDir: {}
 env:
 - name: HELM_CACHE_HOME
 value: /.config/helm/cache
 - name: HELM_CONFIG_HOME
 value: /.config/helm/cache
 - name: HELM_DATA_HOME
 value: /.config/helm/data
 initContainers:
 - name: install-helm-multivalues-plugin
 image: registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:baec6d73a77b832df8131bac3c5a86dc405ef89f600e27a22f164ed3c72816db
 command: [sh, -c]
 args:
 - "helm plugin install https://github.com/nico-ulbricht/helm-multivalues"
 env:
 - name: HELM_CACHE_HOME
 value: /helm-custom-config/cache
 - name: HELM_CONFIG_HOME
 value: /helm-custom-config/cache
 - name: HELM_DATA_HOME
 value: /helm-custom-config/data
 volumeMounts:
 - mountPath: /helm-custom-config
 name: helm-custom-config
 volumeMounts:
 - mountPath: /.config/helm
 name: helm-custom-config

Configure Argo CMP

Next, create a ConfigMap with the configuration for the plugin. The generate command under spec section runs in the Application source directory each time manifests need to generate output. As per the plugin documentation, we need to run the helm multivalues [ORIGINAL_COMMAND] -f [VALUES_FOLDER_PATH] [ORIGINAL_ARGS] to generate the output as a stdout

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


apiVersion: v1
kind: ConfigMap
metadata:
 name: helm-multivalues-plugin
 namespace: openshift-gitops
data:
 plugin.yaml: |
 apiVersion: argoproj.io/v1alpha1
 kind: ConfigManagementPlugin
 metadata:
 name: helm-multivalues-plugin
 spec:
 generate:
 command: [sh, -c, 'helm multivalues template $ARGOCD_APP_NAME $ARGOCD_ENV_CHART_PATH --values $ARGOCD_ENV_VALUES_FILE -f $ARGOCD_ENV_VALUES_FOLDER']

The next step is to add the sidecar to the ArgoCD operator CR.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
 name: openshift-gitops
 namespace: openshift-gitops
spec:
 repo:
 sidecarContainers:
 - name: helm-multivalues-plugin
 command: [/var/run/argocd/argocd-cmp-server]
 image: registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:baec6d73a77b832df8131bac3c5a86dc405ef89f600e27a22f164ed3c72816db
 env:
 - name: HELM_CACHE_HOME
 value: /.config/helm/cache
 - name: HELM_CONFIG_HOME
 value: /.config/helm/cache
 - name: HELM_DATA_HOME
 value: /.config/helm/data
 volumeMounts:
 - mountPath: /var/run/argocd
 name: var-files
 - mountPath: /home/argocd/cmp-server/plugins
 name: plugins
 - mountPath: /tmp
 name: tmp
 - mountPath: /home/argocd/cmp-server/config/plugin.yaml
 subPath: plugin.yaml
 name: helm-multivalues-plugin
 - mountPath: /.config/helm
 name: helm-custom-config
 volumes:
 - configMap:
 name: helm-multivalues-plugin
 name: helm-multivalues-plugin
 - name: helm-custom-config
 emptyDir: {}
 env:
 - name: HELM_CACHE_HOME
 value: /.config/helm/cache
 - name: HELM_CONFIG_HOME
 value: /.config/helm/cache
 - name: HELM_DATA_HOME
 value: /.config/helm/data
 initContainers:
 - name: install-helm-multivalues-plugin
 image: registry.redhat.io/openshift-gitops-1/argocd-rhel8@sha256:baec6d73a77b832df8131bac3c5a86dc405ef89f600e27a22f164ed3c72816db
 command: [sh, -c]
 args:
 - "helm plugin install https://github.com/nico-ulbricht/helm-multivalues"
 env:
 - name: HELM_CACHE_HOME
 value: /helm-custom-config/cache
 - name: HELM_CONFIG_HOME
 value: /helm-custom-config/cache
 - name: HELM_DATA_HOME
 value: /helm-custom-config/data
 volumeMounts:
 - mountPath: /helm-custom-config
 name: helm-custom-config
 volumeMounts:
 - mountPath: /.config/helm
 name: helm-custom-config

A few things to review:

The code snippet above uses an argocd image to mount the plugin we installed in the earlier step and plugin.yaml in the container using the ConfigMap we just created.
Other mounts from Argo CD are necessary, like /var/run/argocd (that contains the argocd-cmp-server command that needs to run) and /home/argocd/cmp-server/plugins.

Apply the changes; Operator will rollout a latest changes of the repo-server pod.

Argo Application to Validate

Let’s set up an ArgoCD application to test the changes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: tenant-gitops-config
 namespace: openshift-gitops
spec:
 destination:
 server: "https://kubernetes.default.svc"
 project: default
 source:
 path: .
 repoURL: "https://github.com/Vikaspogu/openshift-multicluster.git"
 targetRevision: main
 plugin:
 name: helm-multivalues-plugin
 env:
 - name: CHART_PATH
 value: helm/charts/tenant-gitops
 - name: VALUES_FOLDER
 value: kustomize/cluster-overlays/dev-acm/tenant-gitops-helm/apps
 - name: VALUES_FILE
 value: kustomize/cluster-overlays/dev-acm/tenant-gitops-helm/values.yaml
 syncPolicy:
 retry:
 backoff:
 duration: 120s
 factor: 3
 maxDuration: 10m0s
 limit: 10
 syncOptions:
 - SkipDryRunOnMissingResource=true
 - Validate=false
 - ApplyOutOfSyncOnly=true
 - RespectIgnoreDifferences=true
 - Prune=true

Deployed Helm Chart Application

Success! The Config Management Plugin ran the commands in the generate section to render out the YAML and Argo CD applied that YAML to the destination cluster.

Resources

Helm Chart Repository
Helm MultiValues Folder

Helm Chart Testing on OpenShift Pipelines

Sun, 17 Mar 2024 00:00:00 +0000

Introduction

OpenShift Pipelines is a cloud-native CI/CD solution built on Tekton that allows you to define and run pipelines for your applications. Helm charts are a convenient way to package and deploy applications on Kubernetes, including OpenShift. To ensure the quality and reliability of Helm charts, it’s important to perform tests on them. In this blog, we will explore using the Chart Testing tool to perform tests on Helm charts within OpenShift Pipelines.

Prerequisites

Before we begin, make sure you have the following prerequisites installed:

OpenShift Cluster
OpenShift Pipelines Operator installed
Understanding of OpenShift Pipelines and Tasks

Chart Testing Tasks

ct is the the tool for testing Helm charts. It is meant to be used for linting and testing pull requests. It automatically detects charts changed against the target branch.

The ct install command includes helm install and helm test; helm test performs port-forwarding to validate the installation’s success. Since the chart testing commands are run inside a pod, we need to ensure it can connect to the cluster to perform those tasks. If the pod cannot connect to the cluster, you’ll see the error below.

The connection to the server localhost:8080 was refused - did you specify the right host or port?
Error printing details: failed waiting for process: exit status 1
Error printing logs: failed running process: exit status 1
Deleting release "deploy-jgyw6ew05y"...
release "deploy-jgyw6ew05y" uninstalled
------------------------------------------------------------------------------------------------------------------------
 ✖︎ deploy => (version: "0.1.0", path: "deploy") > failed running process: exit status 1
------------------------------------------------------------------------------------------------------------------------
failed installing charts: failed processing charts
Error: failed installing charts: failed processing charts

To solve the above error, we’ll follow below steps:

Create a new service account and secret token from the service account.
Grant appropriate permissions to the service account.
Create a kubeconfig file using the kubeconfig creator task. By mounting the service account token as an environment variable from the secret in the task
This task’s output will be the kubeconfig file stored in the workspace and shared across tasks.

Create a new service account

oc create sa ct-helm-task

Create a token secret from serviceaccount,

oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: ct-helm-task
 annotations:
 kubernetes.io/service-account.name: ct-helm-task
type: kubernetes.io/service-account-token
EOF

Grant permissions to the service account

oc policy add-role-to-user edit -z ct-helm-task -n pipelines-helm

The task will use the kubeconfigwriter image and the provided parameters to create a kubeconfig file that can be used by other tasks in the pipeline to access the target cluster. The kubeconfig will be placed at /workspace/<workspace-name>/kubeconfig.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
 annotations:
 tekton.dev/categories: 'Deployment, Kubernetes'
 tekton.dev/displayName: kubeconfig creator
 tekton.dev/pipelines.minVersion: 0.12.1
 tekton.dev/platforms: 'linux/amd64,linux/s390x,linux/ppc64le'
 tekton.dev/tags: deploy
 name: kubeconfig-creator
spec:
 params:
 - default: ''
 description: name of the cluster
 name: name
 type: string
 - description: address of the cluster
 name: url
 type: string
 - default: '<CLUSTER_URL>'
 description: bearer token for authentication to the cluster
 name: token
 type: string
 - default: 'true'
 description: >-
 to indicate server should be accessed without verifying the TLS
 certificate
 name: insecure
 type: string
 steps:
 - args:
 - '-clusterConfig'
 - >-
 { "name":"$(params.name)", "url":"$(params.url)",
 "username":"$(params.username)", "password":"$(params.password)",
 "cadata":"$(params.cadata)",
 "clientKeyData":"$(params.clientKeyData)",
 "clientCertificateData":"$(params.clientCertificateData)",
 "namespace":"$(params.namespace)", "token":"$(SA_TOKEN)",
 "Insecure":$(params.insecure) }
 - '-destinationDir'
 - $(workspaces.output.path)
 command:
 - /ko-app/kubeconfigwriter
 env:
 - name: SA_TOKEN
 valueFrom:
 secretKeyRef:
 key: token
 name: ct-helm-task
 image: >-
 gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/kubeconfigwriter@sha256:b2c6d0962cda88fb3095128b6202da9b0e6c9c0df3ef6cf7863505ffd25072fd
 name: write
 resources: {}
 workspaces:
 - name: output

Finally, create a task using the chart-testing image. We’ll export the KUBECONFIG variable with the kubeconfig location from the workspace to the helm chart testing task.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
 name: helm-ci-task
spec:
 description: >-
 This task can be used to replace fields in YAML files. For example for altering helm charts on GitOps repos.
 workspaces:
 - name: source
 description: A workspace that contains the file which needs to be altered.
 params:
 - name: image
 type: string
 description: The chart testing image to use.
 default: quay.io/helmpack/chart-testing:v3.7.1
 - name: chart-dirs
 type: string
 description: directory of the charts
 default: "charts/"
 - name: charts
 type: string
 description: name of the chart to test
 default: ""
 - name: namespace
 type: string
 description: namespace to run ci tests
 default: "helm-tekton-pipeline"
 - name: ct-config-file
 type: string
 description: config file for running ct commands
 default: "ct.yaml"
 steps:
 - name: ct-script
 image: quay.io/helmpack/chart-testing:v3.5.1
 workingDir: $(workspaces.source.path)
 script: |
 ## export kubeconfig file from previous step
 export KUBECONFIG="$(workspaces.source.path)/kubeconfig"
 ct lint-and-install --chart-dirs $(params.chart-dirs) --charts $(params.charts) --namespace $(params.namespace) --config $(params.ct-config-file)

Setting Up OpenShift Pipeline

Create a pipeline resource to perform git-clone of the helm chart repository, then create a kubeconfig file and chart-testing task to run lint and install to validate the chart changes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
 name: helm-ct-pipeline
spec:
 workspaces:
 - name: shared-workspace
 params:
 - name: git-url
 type: string
 description: url of the git repo for the code of deployment
 - name: git-revision
 type: string
 description: revision to be used from repo of the code for deployment
 default: master
 tasks:
 - name: fetch-repository
 taskRef:
 name: git-clone
 kind: ClusterTask
 workspaces:
 - name: output
 workspace: shared-workspace
 params:
 - name: url
 value: $(params.git-url)
 - name: subdirectory
 value: ""
 - name: deleteExisting
 value: "true"
 - name: revision
 value: $(params.git-revision)
 - name: kubeconfig-creator
 params:
 - name: url
 value: 'https://api.cluster.example.com:6443'
 runAfter:
 - fetch-repository
 taskRef:
 kind: Task
 name: kubeconfig-creator
 workspaces:
 - name: output
 workspace: shared-workspace
 - name: run-ct-test
 taskRef:
 name: helm-ci-task
 kind: Task
 params:
 - name: charts
 value: deploy
 workspaces:
 - name: source
 workspace: shared-workspace
 runAfter:
 - kubeconfig-creator

Conclusion

In this blog, we have demonstrated how to perform tests on Helm charts using the Chart Testing tool within OpenShift Pipelines. By setting up a pipeline with tasks for linting and testing, you can ensure the quality and reliability of your Helm charts before deploying them to your OpenShift cluster.

Setting Up Cloudflared Tunnel on OpenShift

Mon, 11 Mar 2024 00:00:00 +0000

Introduction

Cloudflare’s Tunnel, powered by Cloudflared, provides a secure way to expose your web applications to the internet without exposing your origin server’s IP address. This guide walks us through the steps to set up a Cloudflared tunnel on OpenShift, a popular container platform.

Important

This setup will expose the OpenShift console and Ingress routes only, not the API server, for oc login

Prerequisites

An OpenShift cluster, either on-premises or in a cloud provider
Cloudflare account
Basic knowledge of OpenShift and Cloudflare

Install Cloudflared on Your Local Machine

Download and Install the Cloudflared binary for your operating system from the Cloudflare website.

brew install cloudflared

Verify the installation

$ cloudflared version
cloudflared version 2024.2.1 (built 2024-02-20T16:25:25Z)

Create a Cloudflare Tunnel

Authenticate Cloudflared: A browser window will open and prompt you to log in to your Cloudflare account. After logging in, select your hostname.

cloudflared tunnel login

Create a tunnel and give it a name. From the command’s output, note the Tunnel’s UUID and the path to your Tunnel’s credentials file.

cloudflared tunnel create openshift

Configure Cloudflared for OpenShift

Create an OpenShift secret from the Tunnel’s credentials file.

oc create secret generic cloudflared --from-file ~/.cloudflared/<tunnel_id>.json

To deploy the cloudflared tunnel image, I’ll be using a helm chart

Create a values.yaml with a cloudflared configuration.

replicaCount: 3
image:
 repository: docker.io/cloudflare/cloudflared
 tag: "2024.2.1"
cloudflared:
 tunnelID: "<REPLACE_WITH_TUNNEL_ID>"
 existingSecret: cloudflared
 ingress:
 - hostname: "oauth-openshift.apps.<DOMAIN>"
 originRequest:
 noTLSVerify: true
 service: https://oauth-openshift.openshift-authentication
 - hostname: "*.apps.<DOMAIN>"
 originRequest:
 noTLSVerify: true
 service: https://router-internal-default.openshift-ingress.svc.cluster.local
 - service: http_status:404

Install helm chart

helm repo add xunholy-charts https://xunholy.github.io/charts/
helm install cloudflared xunholy-charts/cloudflared -f values.yaml -n networking

Cloudflared will authenticate with Cloudflare using the credentials in the configuration file and establish a secure tunnel to your OpenShift service.

Verify that cloudflared pods are running.

oc get pods
NAME READY STATUS RESTARTS AGE
cloudflared-fd8478945-l7xhc 1/1 Running 0 119m
cloudflared-fd8478945-v9nq5 1/1 Running 0 119m
cloudflared-fd8478945-wt2z7 1/1 Running 0 119m

Create a DNS record

Follow this guide to create a DNS record for the tunnel. For the CNAME record name, add *.apps.<DOMAIN_NAME> with the target as <TUNNEL_ID>,cfargotunnel.com

Verify the Tunnel

Open a web browser and navigate to the OpenShift console.

Congratulations! You have successfully set up a Cloudflared tunnel on OpenShift, which allows you to expose your web application securely to the internet.

Getting Started with Red Hat Developer Hub - Part 2

Sun, 14 Jan 2024 00:00:00 +0000

In this continuation of our getting started with Developer Hub series, we’re diving into the art of crafting templates. These templates will not only streamline your workflow but also serve as the cornerstone for automated deployments.

High-level overview

Overview

The developer initiates the workflow by leveraging the backstage self-service template. This template, in turn, pulls essential skeleton resources from a dedicated software template repository, ensuring consistency and best practices. The developer’s actions trigger the creation of a pull request to the GitOps repository, introducing version-controlled changes to the infrastructure. An admin reviews and approves the pull request. Then, ArgoCD, a continuous delivery tool, provides the database resources according to the approved changes. This seamless integration of backstage, GitOps, and ArgoCD accelerates the development lifecycle. It fosters a collaborative environment where developers and administrators work harmoniously to deliver robust and scalable database solutions.

Define Your Infrastructure

Before diving into templates, define the infrastructure you want to provision. In our case, let’s consider provisioning a PostgreSQL database using CloudNative-PG. Let’s start with the installation of the CloudNative-PG operator on OpenShift cluster.

Prerequisites

Ensure you have access to the cluster with cluster-admin privileges.
Verify access to the OpenShift Container Platform web console.

Procedure

Log in to the OpenShift Container Platform web console.
Navigate to Operators → OperatorHub.
Enter cloudnativepg into the filter box.
Select the CloudNativePG and click Install.

CloudnativePG Operator

CloudnativePG Operator installation

Backstage Software Template

Create a software template under a kind Template. This template will automate the process of opening a GitHub pull request for provisioning.

I recommend referring to documentation on creating templates and using Template Editor from the developer hub for quick feedback.

Template Editor Menu

Template Editor forms

The outcome of the template would be as follows, based on the flow we discussed in the overview. Feel free to modify the YAML as required and test the changes in the template form editor for quicker feedback

---
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: postgresql-cluster
 title: Postgresql Cluster template
 description: scaffolder v1beta3 template to create pgsql cluster on existing git repository
 tags:
 - database
 - postgresql
 links:
 - title: Documentation
 url: https://backstage.io/docs/features/software-templates
 icon: docs
 - title: Source
 url: https://github.com/Vikaspogu/software-templates/blob/main/scaffolder-templates/pgsql-cluster/template.yaml
 icon: github
spec:
 owner: group:ops
 type: service
 parameters:
 - title: Postgresql Definition
 required:
 - namespace
 properties:
 environment:
 title: Select environment
 type: string
 default: dev
 enum:
 - prod
 - stage
 - dev
 enumNames:
 - 'production (prod)'
 - 'staging (stage)'
 - 'development (dev)'
 cluster:
 title: Select openshift cluster
 type: string
 default: acm
 enum:
 - acm
 - dev
 - prod
 enumNames:
 - 'acm (acm)'
 - 'development (dev)'
 - 'production (prod)'
 namespace:
 title: Namespace
 type: string
 description: Namespace of the postgresql cluster
 create_namespace:
 title: Create a namespace
 type: boolean
 instances:
 title: Instances
 type: integer
 description: Number of instances required in the cluster
 default: 1
 storage_size:
 title: Storage Size
 type: string
 description: Size of the storage (ex. 1Gi)
 default: 1Gi
 owner:
 title: Owner
 type: string
 description: Owner of the component
 ui:field: OwnerPicker
 ui:options:
 catalogFilter:
 kind: Group
 - title: Provide information about the GitHub location
 required:
 - repoOwner
 - repoName
 - owner
 properties:
 repoOwner:
 title: Repository Owner
 type: string
 default: vikaspogu
 owner:
 title: Owner
 type: string
 description: Owner of the component
 repoName:
 title: Repository Name
 type: string
 default: cluster-components

 steps:
 - id: fetch-base
 name: Fetch Base
 action: fetch:template
 input:
 url: ./skeleton
 values:
 cluster: ${{parameters.cluster}}
 environment: ${{parameters.environment}}
 name: ${{parameters.namespace}}-${{parameters.environment}}-db
 namespace: ${{parameters.namespace}}-${{parameters.environment}}
 instances: ${{parameters.instances}}
 storage_size: ${{parameters.storage_size}}
 create_namespace: ${{parameters.create_namespace}}
 repoOwner: ${{ parameters.repoOwner }}
 repoName: ${{ parameters.repoName }}
 owner: ${{ parameters.owner }}
 applicationType: resources

 - id: publish
 name: Create a pull request
 action: publish:github:pull-request
 input:
 repoUrl: github.com?owner=${{ parameters.repoOwner }}&repo=${{ parameters.repoName }}
 branchName: pqsql/${{parameters.namespace}}-${{parameters.environment}}
 title: "Create a new pgsql database in ${{parameters.namespace}}"
 description: |
 ### Requesting new pgsql database in ${{parameters.namespace}}

 This pull request is generated by IDP catalog

 - id: label_pr
 name: Add labels to PR
 action: github:issues:label
 input:
 repoUrl: github.com?owner=${{ parameters.repoOwner }}&repo=${{ parameters.repoName }}
 number: '${{ steps.publish.output.pullRequestNumber }}'
 labels:
 - postgresql
 - created-by-backstage
 - ${{ parameters.environment }}
 - ${{parameters.namespace}}

 output:
 links:
 - title: 'Pull request link'
 url: ${{ steps.publish.output.remoteUrl }}
 icon: github

Adding template

To incorporate the template files, I’m adopting a static location configuration. In this approach, we specify the template URL within the catalog section of the app-config-rhdh configmap, as demonstrated below

catalog:
 rules:
 - allow: [Component, System, API, Resource, Location, Template]
 locations:
 - type: url
 target: https://github.com/Vikaspogu/software-templates/blob/main/scaffolder-templates/pgsql-cluster/template.yaml
 rules:
 - allow: [Template]

After updating the configmap, recreate the pod. Once the pod is ready, you’ll discover the template in the create menu.

Available Templates

Choose the Template

Fill out information

Create template

Success

Pull request

ArgoCD Application Manifest

Create an ArgoCD application manifest that corresponds to your Backstage template. This manifest specifies the desired state of your application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: cluster-components
 namespace: openshift-gitops
 labels:
 type: config
spec:
 destination:
 server: "https://kubernetes.default.svc"
 project: default
 source:
 directory:
 recurse: true
 exclude: "**/catalog-info.yaml"
 path: acm
 repoURL: "https://github.com/Vikaspogu/cluster-components"
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true
 - RespectIgnoreDifferences=true
 - ApplyOutOfSyncOnly=true

Test Your Workflow

Merge the pull request generated from the template, then witness ArgoCD autonomously deploying the changes according to the updated desired state. Confirm that the PostgreSQL database cluster has been provisioned as anticipated.

Conclusion

Congratulations! You’ve now crafted a powerful template that seamlessly integrates Backstage and ArgoCD. This template not only simplifies your deployment workflow but also introduces automation through GitHub pull requests.

Getting Started with Red Hat Developer Hub - Part 1

Sat, 13 Jan 2024 00:00:00 +0000

Why I Wanted an Internal Developer Portal

Every platform team eventually faces the same question: how do developers discover what’s available? We had ArgoCD for deployments, Open Cluster Management for multi-cluster visibility, and GitHub for code—but no single place to see it all.

I’d heard about Backstage for years but never had time to evaluate it. When Red Hat released Developer Hub (their supported Backstage distribution), I decided to try building a self-service portal where developers could provision databases without filing tickets.

This post covers what I learned deploying it—including the configuration gotchas that aren’t obvious from the docs.

Choosing Red Hat Developer Hub Over Vanilla Backstage

Why not just run Backstage directly?

I tried. Backstage requires you to build and maintain your own container image with plugins baked in. Every plugin update means rebuilding. For a team with time to invest, this is fine. For a quick evaluation, it’s friction.

Developer Hub ships pre-built with dynamic plugins—you enable them via config, not code. The tradeoff is less flexibility, but I wanted to validate the concept before committing to custom development.

Installation: What the Docs Don’t Emphasize

The Helm install itself is straightforward:

helm repo add openshift-helm-charts https://charts.openshift.io/
helm show values openshift-helm-charts/redhat-developer-hub > values.yaml

Update the global.clusterRouterBase according to OpenShift router host and modify values if needed

1
2
3
4
5
6
7
8


global:
 auth:
 backend:
 enabled: true
 existingSecret: ""
 value: ""
 clusterRouterBase: apps.example.com
...

Install the helm chart with modified values

oc new-project developer-hub
helm install developer-hub developer-hub/developer-hub -f values.yaml

Wait for the developer hub and postgresql database pods to be in ready status

oc get pods
NAME READY STATUS RESTARTS AGE
developer-hub-64d8cff99c-k8k9r 1/1 Running 0 2d
developer-hub-postgresql-0 1/1 Running 0 2d

Navigate to the developer hub in a web browser by grabbing the route host information as shown below

oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
developer-hub developer-hub-developer-hub.apps.example.com / developer-hub http-backend edge/Redirect None

At this point you have a running Developer Hub—but it’s useless. It’s an empty portal with no integrations. The real work starts now.

The Plugin Problem I Didn’t Anticipate

Backstage’s value comes from plugins. Developer Hub ships with many pre-installed but disabled. My goal was to integrate:

GitHub for authentication (so developers use their existing credentials)
Kubernetes plugin to show workloads
Open Cluster Management to display all managed clusters
ArgoCD to show deployment sync status

What I learned: Enabling plugins is easy. Configuring them to actually work together is where you’ll spend your time. Each plugin needs its own authentication, and the documentation assumes you’re doing one at a time.

Enable Plugins

Developer Hub images come with dynamic plugins pre-installed but disabled. Enable them in your Helm values:

Gotcha: The package paths must match exactly what’s in the image. These paths change between versions. If you upgrade Developer Hub and plugins stop working, check if the paths changed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


global:
 auth:
 backend:
 enabled: true
 existingSecret: ""
 value: ""
 clusterRouterBase: apps.example.com
 dynamic:
 includes:
 - dynamic-plugins.default.yaml
 plugins:
 - disabled: false
 package: ./dynamic-plugins/dist/backstage-plugin-catalog-backend-module-github-dynamic
 - disabled: false
 package: ./dynamic-plugins/dist/roadiehq-backstage-plugin-security-insights
 - disabled: false
 package: ./dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic
 - disabled: false
 package: ./dynamic-plugins/dist/backstage-plugin-kubernetes
 - disabled: false
 package: ./dynamic-plugins/dist/roadiehq-backstage-plugin-argo-cd-backend-dynamic
 - disabled: false
 package: ./dynamic-plugins/dist/roadiehq-scaffolder-backend-argocd-dynamic
 - disabled: false
 package: ./dynamic-plugins/dist/roadiehq-backstage-plugin-argo-cd
 - disabled: false
 package: ./dynamic-plugins/dist/janus-idp-backstage-plugin-ocm
 - disabled: false
 package: ./dynamic-plugins/dist/janus-idp-backstage-plugin-ocm-backend-dynamic
 - disabled: false
 package: ./dynamic-plugins/dist/roadiehq-scaffolder-backend-module-utils-dynamic
...

Re-install the chart with updated values

helm upgrade --install developer-hub developer-hub/developer-hub -f values.yaml

Verify if plugins are installed in the install-dynamic-plugins init container within the Developer Hub pod’s log

======= Skipping disabled dynamic plugin ./dynamic-plugins/dist/backstage-plugin-scaffolder-backend-module-gitlab-dynamic
======= Installing dynamic plugin ./dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic
==> Grabbing package archive through `npm pack`
==> Removing previous plugin directory /dynamic-plugins-root/backstage-plugin-kubernetes-backend-dynamic-0.13.0
==> Extracting package archive /dynamic-plugins-root/backstage-plugin-kubernetes-backend-dynamic-0.13.0.tgz
==> Removing package archive /dynamic-plugins-root/backstage-plugin-kubernetes-backend-dynamic-0.13.0.tgz
==> Merging plugin-specific configuration
==> Successfully installed dynamic plugin /opt/app-root/src/dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic
======= Installing dynamic plugin ./dynamic-plugins/dist/backstage-plugin-kubernetes
==> Grabbing package archive through `npm pack`
==> Removing previous plugin directory /dynamic-plugins-root/backstage-plugin-kubernetes-0.11.0
==> Extracting package archive /dynamic-plugins-root/backstage-plugin-kubernetes-0.11.0.tgz
==> Removing package archive /dynamic-plugins-root/backstage-plugin-kubernetes-0.11.0.tgz
==> Merging plugin-specific configuration
==> Successfully installed dynamic plugin /opt/app-root/src/dynamic-plugins/dist/backstage-plugin-kubernetes
======= Installing dynamic plugin ./dynamic-plugins/dist/janus-idp-backstage-plugin-topology

Plugin Integrations

In the spirit of brevity and to ensure accuracy, I’ve posted the configuration code snippets for your convenience. However, for a more detailed and nuanced understanding of the integration process, I highly recommend referring to the official product documentation.

Important

Integration code snippets are added to the configmap; for reference, here is the complete configmap

Externalize configuration

Create a configmap to add configuration to the developer hub as described in the documentation

ConfigMap

Mount the ConfigMap, add extraAppConfig to the values file under upstream.backstage section

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


upstream:
 backstage:
 appConfig:
 app:
 baseUrl: 'https://{{- include "janus-idp.hostname" . }}'
 backend:
 baseUrl: 'https://{{- include "janus-idp.hostname" . }}'
 cors:
 origin: 'https://{{- include "janus-idp.hostname" . }}'
 database:
 connection:
 password: ${POSTGRESQL_ADMIN_PASSWORD}
 user: postgres
 auth:
 keys:
 - secret: ${BACKEND_SECRET}
 args:
 - "--config"
 - dynamic-plugins-root/app-config.dynamic-plugins.yaml
 command: []
 extraAppConfig:
 - configMapRef: app-config-rhdh
 filename: app-config-rhdh.yaml

Re-install the chart with updated values

helm upgrade --install developer-hub developer-hub/developer-hub -f values.yaml

GitHub Integration

Seamlessly connect your projects with GitHub by configuring the integration.

Create a secret named rhdh-secrets as mentioned in documentation with GitHub clientId, secret and token

oc create secret generic rhdh-secrets --from-literal=GITHUB_APP_CLIENT_ID=dummy --from-literal=GITHUB_APP_CLIENT_SECRET=dummy --from-literal=GITHUB_TOKEN=dummy

Mount the rhdh-secrets, add extraEnvVarsSecrets to the values file under upstream.backstage section

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


upstream:
 backstage:
 appConfig:
 app:
 baseUrl: 'https://{{- include "janus-idp.hostname" . }}'
 backend:
 baseUrl: 'https://{{- include "janus-idp.hostname" . }}'
 cors:
 origin: 'https://{{- include "janus-idp.hostname" . }}'
 database:
 connection:
 password: ${POSTGRESQL_ADMIN_PASSWORD}
 user: postgres
 auth:
 keys:
 - secret: ${BACKEND_SECRET}
 args:
 - "--config"
 - dynamic-plugins-root/app-config.dynamic-plugins.yaml
 command: []
 extraEnvVarsSecrets:
 - rhdh-secrets
 extraAppConfig:
 - configMapRef: app-config-rhdh
 filename: app-config-rhdh.yaml

Update the app-config-rhdh configmap and add below configuration

integrations:
 github:
 - host: github.com
 token: ${GITHUB_TOKEN}
auth:
 # see https://backstage.io/docs/auth/ to learn about auth providers
 environment: development
 providers:
 github:
 development:
 clientId: ${GITHUB_APP_CLIENT_ID}
 clientSecret: ${GITHUB_APP_CLIENT_SECRET}

Re-installing the chart will recreate a new pod with updated values and configmap

helm upgrade --install developer-hub developer-hub/developer-hub -f values.yaml

GitHub Auth

Navigating Clusters with Open Cluster Management

Open Cluster Management extends our reach to manage and monitor multiple clusters effortlessly. To enable we can add kubernetes configuration and reference the cluster in catalog provider

We are setting up authentication for the Kubernetes cluster using the serviceaccount token. Create a serviceaccount in the developer-hub namespace

oc create sa developer-hub -n developer-hub

Obtain a long-lived token by creating a secret

oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
 name: developer-hub-sa
 namespace: developer-hub
 annotations:
 kubernetes.io/service-account.name: developer-hub
type: kubernetes.io/service-account-token
EOF

Add a new entry in the extraEnvVars to mount the serviceaccount token secret as an environment variables in the values file under upstream.backstage section

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


upstream:
 backstage:
 ...
 extraEnvVarsSecrets:
 - rhdh-secrets
 extraAppConfig:
 - configMapRef: app-config-rhdh
 filename: app-config-rhdh.yaml
 extraEnvVars:
 - name: BACKEND_SECRET
 valueFrom:
 secretKeyRef:
 key: backend-secret
 name: '{{ include "janus-idp.backend-secret-name" $ }}'
 - name: POSTGRESQL_ADMIN_PASSWORD
 valueFrom:
 secretKeyRef:
 key: postgres-password
 name: '{{- include "janus-idp.postgresql.secretName" . }}'
 - name: KUBE_TOKEN
 valueFrom:
 secretKeyRef:
 key: token
 name: developer-hub-sa

Update the app-config-rhdh configmap with below configuration

kubernetes:
 serviceLocatorMethod:
 type: "multiTenant"
 clusterLocatorMethods:
 - type: "config"
 clusters:
 - url: https://api.example.com:6443
 name: acm
 authProvider: "serviceAccount"
 skipTLSVerify: true
 serviceAccountToken: ${KUBE_TOKEN}
 dashboardApp: openshift
 dashboardUrl: https://console-openshift-console.apps.example.com/
catalog:
 providers:
 ocm:
 default:
 kubernetesPluginRef: acm #same as the clusters name in kubernetes section
 name: multiclusterhub
 owner: group:ops
 schedule:
 frequency:
 seconds: 10
 timeout:
 seconds: 60

Re-installing the chart will recreate a new pod with updated values and mount serviceaccount token secret as an environment variable

helm upgrade --install developer-hub developer-hub/developer-hub -f values.yaml

Multicluster

Synchronizing with ArgoCD

The ArgoCD Backstage plugin provides synced, health status and updates the history of your services to your Developer Portal.

Generate a new token from the ArgoCD instance

ArgoCD token

Create a new secret from the ArgoCD token

oc create secret generic argocd-token --from-literal=ARGOCD_TOKEN=dummy

Add a new entry in the extraEnvVars to mount the secret as an environment variables in the values file under upstream.backstage section

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


upstream:
 backstage:
 ...
 extraEnvVarsSecrets:
 - rhdh-secrets
 extraAppConfig:
 - configMapRef: app-config-rhdh
 filename: app-config-rhdh.yaml
 extraEnvVars:
 - name: BACKEND_SECRET
 valueFrom:
 secretKeyRef:
 key: backend-secret
 name: '{{ include "janus-idp.backend-secret-name" $ }}'
 - name: POSTGRESQL_ADMIN_PASSWORD
 valueFrom:
 secretKeyRef:
 key: postgres-password
 name: '{{- include "janus-idp.postgresql.secretName" . }}'
 - name: KUBE_TOKEN
 valueFrom:
 secretKeyRef:
 key: token
 name: developer-hub-sa
 - name: ARGOCD_TOKEN
 valueFrom:
 secretKeyRef:
 key: ARGOCD_TOKEN
 name: argocd-token

Update the app-config-rhdh configmap with below configuration

argocd:
 appLocatorMethods:
 - instances:
 - name: main
 token: ${ARGOCD_TOKEN}
 url: https://openshift-gitops-server-openshift-gitops.apps.example.com
 type: config

Re-installing the chart

helm upgrade --install developer-hub developer-hub/developer-hub -f values.yaml

ArgoCD

What I Learned Setting This Up

Time investment: The initial deployment took 30 minutes. Getting all four integrations working took two days. Most of that was understanding how secrets flow between the Helm values, ConfigMaps, and the actual Backstage config.

The biggest gotcha: Every integration needs its own service account or token, and they all have different permission requirements. The Kubernetes plugin needs cluster-wide read access. The ArgoCD plugin needs a token with specific RBAC. Plan for this before you start.

Is it worth it? For a home lab exploration, absolutely—I learned a lot about how Backstage works. For production, you need to budget significant time for ongoing maintenance. Plugins break across Backstage upgrades, and the ecosystem moves fast.

What’s Next

In Part 2, I’ll cover the part that actually delivers value to developers: software templates. We’ll build a self-service workflow where developers can provision a CloudNative-PG database cluster by filling out a form—no tickets, no waiting.

Upgrading CloudNative-PG: PostgreSQL 15 to 16

Fri, 12 Jan 2024 00:00:00 +0000

Introduction

In the ever-evolving landscape of cloud-native technologies, staying up-to-date with the latest software versions is crucial for ensuring optimal performance, security, and access to new features. This blog post will guide you through the process of upgrading a PostgreSQL database instance from version 15 to version 16 in a cloud-native environment.

Before You Begin

Before diving into the upgrade process, it’s essential to perform a few preliminary steps to ensure a smooth transition:

Backup Your Database

Begin by creating a comprehensive backup of your PostgreSQL database. This step is critical in case anything goes wrong during the upgrade process, allowing you to restore your database to its previous state.

Check Compatibility

Verify that your applications and any other dependencies are compatible with PostgreSQL 16. Review release notes and documentation to identify any potential issues that may arise.

Update Extensions and Dependencies

Make sure that all extensions and dependencies are compatible with PostgreSQL 16. Update them as needed to avoid compatibility issues after the upgrade.

Upgrading Cloud-Native PostgreSQL Database

Review Release Notes

Start by carefully reviewing the release notes for PostgreSQL 16. Take note of any changes, new features, and potential issues that may affect your database.

Stop Database Activity

Before initiating the upgrade, stop all database activity to prevent any data inconsistencies or corruption during the process. Notify users in advance to minimize disruptions.

Upgrading PostgreSQL Cluster

This post doesn’t cover in-place upgrades; instead, it will require you to spin a newer postgresql cluster with version 16 and bootstrap from another cluster.

Use the new Postgres version 16 as the cluster image

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: upgrade-cluster-16
spec:
 instances: 3
 imageName: ghcr.io/cloudnative-pg/postgresql:16.1-13

Add external cluster configuration to perform a monolith import using the version 15 cluster as the connection parameters.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: upgrade-cluster-16
spec:
 instances: 3
 imageName: ghcr.io/cloudnative-pg/postgresql:16.1-13
 externalClusters:
 - name: cluster-pg96
 connectionParameters:
 # Use the correct IP or host name for the source database
 host: pg96.local
 user: postgres
 dbname: postgres
 password:
 name: cluster-pg96-superuser
 key: password

Configure bootstrap source with cluster name from above

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: upgrade-cluster-16
spec:
 instances: 3
 imageName: ghcr.io/cloudnative-pg/postgresql:16.1-13
 externalClusters:
 - name: cluster-pg96
 connectionParameters:
 # Use the correct IP or host name for the source database
 host: pg96.local
 user: postgres
 dbname: postgres
 password:
 name: cluster-pg96-superuser
 key: password
 bootstrap:
 initdb:
 import:
 type: monolith
 databases:
 - "*"
 roles:
 - "*"
 source:
 externalCluster: cluster-pg96

Add configuration to perform backup after the bootstrap import to an object store

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: upgrade-cluster-16
spec:
 instances: 3
 imageName: ghcr.io/cloudnative-pg/postgresql:16.1-13
 externalClusters:
 - name: cluster-pg96
 connectionParameters:
 # Use the correct IP or host name for the source database
 host: pg96.local
 user: postgres
 dbname: postgres
 password:
 name: cluster-pg96-superuser
 key: password
 bootstrap:
 initdb:
 import:
 type: monolith
 databases:
 - "*"
 roles:
 - "*"
 source:
 externalCluster: cluster-pg96
 backup:
 barmanObjectStore:
 destinationPath: "<destination path here>"
 serverName: upgrade-cluster-16
 s3Credentials:
 accessKeyId:
 name: aws-creds
 key: ACCESS_KEY_ID
 secretAccessKey:
 name: aws-creds
 key: ACCESS_SECRET_KEY
 retentionPolicy: "30d"

Apply changes to the cluster and wait for the cluster to be in a healthy state
```
kubectl apply -f upgrade-cluster-16.yaml
```

Finally, swap the bootstrap recovery source to the new backup generated from earlier steps

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: main
spec:
 instances: 3
 imageName: ghcr.io/cloudnative-pg/postgresql:16.1-13
 bootstrap:
 recovery:
 source: upgrade-cluster-16

Delete the cluster created in steps 1-4

kubectl delete -f upgrade-cluster-16.yaml

Conclusion

Upgrading a cloud-native PostgreSQL database instance from version 15 to 16 requires careful planning and execution. By following the steps outlined in this blog post, you can ensure a seamless transition, taking advantage of the latest features and improvements while maintaining the integrity of your data. Keep in mind that each cloud provider may have specific guidelines or tools for managing PostgreSQL upgrades, so consult their documentation for additional guidance.

SOPS Ansible Execution Image

Sat, 29 Apr 2023 00:00:00 +0000

Introduction

Today, I am excited to share how to set up an Ansible execution environment to control SOPS secrets. The Ansible execution image is a pre-built container containing all the necessary components for decrypting SOPS secrets in Ansible playbooks.

Ansible Builder

Ansible Builder is a tool that can help with building the image

Create a working directory

Let’s start by creating a new working directory

mkdir custom-ansible-execution; cd custom-ansible-execution

Builder definition

Create an execution-environment.yml file. ansible-builder build container images with the definition file execution-environment.yml

In Ansible Builder, the `additional_build_steps`` configuration allows you to specify additional commands or steps to be executed during the build process. This can be useful for performing custom tasks, installing specific dependencies, or executing scripts as part of the Ansible collection build.

Highlighted code block shows how to add SOPS rpm package via additional build steps

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


version: 1
build_arg_defaults:
 EE_BASE_IMAGE: 'registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:latest'
additional_build_steps:
 prepend: |
 RUN rpm -i https://github.com/mozilla/sops/releases/download/v3.7.3/sops-3.7.3-1.x86_64.rpm && \
 sops -v && \
 mkdir -p /home/runner/.config/sops/age
dependencies:
 galaxy: requirements.yml

Ansible collections

Create and add the SOPS collection to the requirements.yml

collections:
 - community.sops

Create context files

ansible-buidler create Creates a build context, which can be used by podman to build an image.

ansible-builder create
Complete! The build context can be found at: custom-ansible-execution/context

Build and push image

Build the container image using Podman and push to a registry

podman build -f context/Containerfile -t localhost/custom-ee:1.0 context
podman push ...

Execution environments in jobs

Follow this guide to add an execution environment in Jobs

Mount options

Mount the SOPS keys to decrypt using Execution environment mount options

Copy sops keys

Finally, SSH into the ansible controller and change to awx user to copy the key into the appropriate directory

ssh aap-controller
sudo su - awx
sudo mkdir -p /var/lib/awx/.sops-key
sudo mv keys.txt /var/lib/awx/.sops-key

Thank you for taking the time to read my blog post. Until next time, take care!

Upgrading PostgreSQL in Kubernetes

Sat, 21 Jan 2023 00:00:00 +0000

I have been running an instance of bitnami’s version of Postgresql 12 for a while, and I have recently upgraded/migrated data from version 12 to 15.

This post doesn’t cover in-place upgrades; instead, it will require you to spin a newer container version.

Deploy New Container Image

The first step is to deploy a new Postgres container using the updated image version.

This container MUST NOT mount the same volume as the older Postgres container.

Instead, it will need to mount a new volume for the database. The new Postgres server will fail if you mount the older Postgres server volume. This is because Postgres requires the data to be migrated before it can load it.

Backup

Install the PostgreSQL command line tool on your local machine

brew install PostgreSQL

Start Kubernetes port-forward for postgres service to expose the port on the local system to perform a data dump

kubectl port-forward svc/postgresql-kube 5432 &

Run pg_dumpall, a utility for writing out (dumping) all of your PostgreSQL databases of a cluster

pg_dumpall -h 127.0.0.1 -p 5432 -U postgres -W -f database.sql

If everything goes successfully, a new file is created on the local filesystem.

Import

We’ll use the Kubernetes copy(cp) command to transfer the dump file into the container.

kubectl cp database.sql postgresql-0:/tmp/database.sql

Open a terminal shell on the new postgresql container.

kubectl exec -it postgresql-0 bash

Finally, import the databases.

I have no name!@postgresql-0:/$ psql -U postgres -W -f /tmp/database.sql

Verify the import and stop the old container. You have completed the upgrade successfully now. Congrats!

Forward OpenShift Audit Logs to Splunk Using NewRelic Fluent Bit

Wed, 11 Jan 2023 00:00:00 +0000

It is common for organizations to send logs to multiple systems—for example, container logs to NewRelic, and audit logs to Splunk for the InfoSec team.

Deploying NewRelic using helm chart will also deploy the NewRelic logging deamonset, which utilizes a custom Fluent Bit image with a NewRelic output plugin to forward logs easily. Since the NewRelic Fluent Bit image uses an upstream fluent bit as a base image, it is pre-baked with the Splunk plugin.

Now, let’s look at how to configure both NewRelic for container logs and Splunk for audit logs using the helm chart.

Three things have to be configured in helm chart values

Configure tail input for audit log paths
Update the Match tag in the Kubernetes filter to match all records that start with audit.*
Configure Splunk output with HEC token and host details

Configure audit input

Input Plugins gather information from different sources; some collect data from log files. tail input plugin allows monitoring one or several text files. It has similar behavior to the tail -f shell command.

In this case, we want to tail /var/log/audit/audit.log; to configure new input in NewRelic logging. Add tail input for audit logs in values.yaml file under the fluentBit config -> inputs section, as shown below.

fluentBit:
 config:
 inputs: |
 [INPUT]
 Name tail
 Tag kube.*
 Path ${PATH}
 Parser ${LOG_PARSER}
 DB ${FB_DB}
 Mem_Buf_Limit 7MB
 Skip_Long_Lines On
 Refresh_Interval 10
 [INPUT]
 Name tail
 Tag audit.*
 Path "/var/log/audit/audit.log"
 Parser ${LOG_PARSER}
 DB ${FB_DB}
 Mem_Buf_Limit 7MB
 Skip_Long_Lines On
 Refresh_Interval 10

Update Match rule

A Match represents a simple rule to select Events whose Tags match a defined rule.

In this scenario, we are checking the audit.* tags. Only one unique filter is allowed; we’ll append it to the existing kube.* tag.

fluentBit:
 config:
 [FILTER]
 Name kubernetes
 Match kube.* audit.*
 Kube_URL https://kubernetes.default.svc.cluster.local:443
 Buffer_Size ${K8S_BUFFER_SIZE}
 K8S-Logging.Exclude ${K8S_LOGGING_EXCLUDE}
 [FILTER]
 Name record_modifier
 Match *
 Record cluster_name ${CLUSTER_NAME}

Configure Splunk output

The output interface allows the definition of destinations for the data. For example, our destination for audit logs is Splunk.

To configure new output in NewRelic logging. Add splunk output for audit.* logs in the values.yaml file under the fluentBit config -> outputs section, as shown below.

fluentBit:
 config:
 outputs: |
 [OUTPUT]
 Name newrelic
 Match *
 licenseKey ${LICENSE_KEY}
 endpoint ${ENDPOINT}
 lowDataMode ${LOW_DATA_MODE}
 Retry_Limit ${RETRY_LIMIT}
 [OUTPUT]
 Name splunk
 Match audit.*
 Host splunk-hec.host
 Port 443
 TLS On
 Splunk_Token ${SPLUNK_TOKEN} #Create a secret with Splunk token and mount into container

Finally, recreate the logging pods. Audit logs should be routed to Splunk along with container logs to NewRelic.

Why I Switched from FluxCD to ArgoCD (And Almost Switched Back)

Wed, 10 Aug 2022 00:00:00 +0000

The Problem I Was Actually Solving

I’d been running FluxCD for two years. Secrets management was seamless—FluxCD has native SOPS integration, and it just worked. But my team standardized on ArgoCD, and I needed to migrate.

What I thought would be a weekend project turned into a week of debugging. Here’s what actually happened.

Why Not Just Use Sealed Secrets or External Secrets?

Before diving into SOPS, I evaluated the alternatives:

Sealed Secrets: Requires the controller to be running. If you’re bootstrapping a cluster from scratch, you have a chicken-and-egg problem. I wanted secrets in git that work even for cluster bootstrapping.

External Secrets Operator: Great if you’re already invested in Vault or AWS Secrets Manager. I wasn’t, and I didn’t want to add another dependency for a home lab.

SOPS: Secrets encrypted in git, decrypted at apply time. No external dependencies. This is what I wanted.

What I Tried First (And Why It Failed)

Attempt 1: Following the KSOPS README

The KSOPS documentation shows a clean integration. I copied it verbatim.

Result: ArgoCD showed Unknown sync status. No errors in the UI, just… nothing.

The actual problem: The docs assume you’re running vanilla ArgoCD. I was using the OpenShift GitOps Operator, which runs the repo-server with a non-root user. The KSOPS binary couldn’t execute because of permission issues that never surfaced as clear errors.

Attempt 2: Using PGP Instead of Age

I initially tried PGP because that’s what most tutorials showed. Generating keys, importing them, managing keyrings—it was a mess.

The actual problem: PGP key management is painful. SOPS now recommends Age encryption, which uses simple file-based keys. I wasted a day before discovering this.

The Solution That Actually Works on OpenShift

Here’s the configuration that works with the OpenShift GitOps Operator. The key differences from the standard docs:

1. Age Over PGP

Generate an Age key (this is the entire “key management” process):

age-keygen -o age.agekey
# Public key: age1helqcqsh9464r8chnwc2fzj8uv7vr5ntnsft0tn45v2xtz0hpfwq98cmsg

Store it as a secret:

cat age.agekey | kubectl create secret generic sops-age \
 --namespace=openshift-gitops \
 --from-file=key.txt=/dev/stdin

2. The ArgoCD CR Configuration

This is the part that differs from most tutorials. Note the specific paths—these matter on OpenShift:

repo:
 env:
 - name: XDG_CONFIG_HOME
 value: /.config
 - name: SOPS_AGE_KEY_FILE
 value: /.config/sops/age/keys.txt
 volumes:
 - name: custom-tools
 emptyDir: {}
 - name: sops-age
 secret:
 secretName: sops-age
 initContainers:
 - name: install-ksops
 image: viaductoss/ksops:v3.0.2
 command: ["/bin/sh", "-c"]
 args:
 - 'echo "Installing KSOPS..."; cp ksops /custom-tools/; cp $GOPATH/bin/kustomize /custom-tools/; echo "Done.";'
 volumeMounts:
 - mountPath: /custom-tools
 name: custom-tools
 volumeMounts:
 - mountPath: /usr/local/bin/kustomize
 name: custom-tools
 subPath: kustomize
 - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
 name: custom-tools
 subPath: ksops
 - mountPath: /.config/sops/age/keys.txt
 name: sops-age
 subPath: keys.txt

Why XDG_CONFIG_HOME? Kustomize looks for plugins in $XDG_CONFIG_HOME/kustomize/plugin/. Without this, it defaults to ~/.config, which doesn’t exist in the container.

Why the specific mount paths? KSOPS must be at the exact path Kustomize expects for the plugin API version (viaduct.ai/v1/ksops). Get this wrong and you’ll get silent failures.

What I’d Do Differently

Start with Age, not PGP. The documentation ecosystem hasn’t caught up, but Age is simpler in every way.
Test locally first. Before touching ArgoCD, verify your SOPS + Kustomize setup works with kustomize build on your laptop.
Check the repo-server logs. The ArgoCD UI hides most errors. The real debugging happens in kubectl logs -n openshift-gitops deployment/openshift-gitops-repo-server.
Consider if you need this complexity. For a home lab, this works great. For production with a team, External Secrets Operator with a proper secrets manager might be worth the additional dependency.

The Takeaway

GitOps secrets management isn’t a solved problem—it’s a series of tradeoffs. SOPS gives you encrypted secrets in git with no external dependencies, but the ArgoCD integration requires understanding how Kustomize plugins work and where ArgoCD expects to find them.

The official docs get you 80% there. The last 20% depends on your specific ArgoCD deployment method, and that’s where most people get stuck.

Configure InfluxDB with Grafana

Mon, 08 Aug 2022 00:00:00 +0000

I recently started using proxmox and was planning on monitoring metrics using InfluxDB and Grafana, which I have already deployed on my Kubernetes cluster. However, during that process, I encountered two issues:

Authentication
“Database not found”

Authentication

For the Authentication issue, I found out that you have to use the Authorization header with the Token yourAuthToken value to access the bucket. So, configure the header name in the jsonData field, and we should configure the header value in secureJsonData for grafana helm chart values.

secureJsonData:
 httpHeaderValue1: 'Token ${INFLUXDB_TOKEN}'
jsonData:
 httpHeaderName1: 'Authorization'

Complete datasources helm values

datasources:
 datasources.yaml:
 apiVersion: 1
 deleteDatasources:
 - name: Loki
 orgId: 1
 datasources:
 - name: Prometheus
 type: prometheus
 url: http://prometheus-operated:9090
 access: proxy
 isDefault: true
 - name: InfluxDB
 type: influxdb
 url: "http://influxdb.default.svc.cluster.local:80"
 database: "default"
 secureJsonData:
 httpHeaderValue1: 'Token ${INFLUXDB_TOKEN}'
 access: proxy
 jsonData:
 httpHeaderName1: 'Authorization'
 httpMode: GET

“Database not found”

This one took me a while to figure out. In InfluxDB v1, data gets stored in databases and retention policies. In InfluxDB v2, data gets stored in buckets. Because InfluxQL uses the v1 data model, before querying in InfluxQL, we must map a bucket to a database and retention policy. Find more information here.

As shown below, I am using the InfluxDB API method to create DBRP mappings for unmapped buckets.

curl --request POST http://localhost:8086/api/v2/dbrps \
 --header "Authorization: Token YourAuthToken" \
 --header 'Content-type: application/json' \
 --data '{
 "bucketID": "00oxo0oXx000x0Xo",
 "database": "example-db",
 "default": true,
 "orgID": "00oxo0oXx000x0Xo",
 "retention_policy": "example-rp"
 }'

Make sure to update Host, Token, bucketID, database, orgID, retention_policy with your values.

Lastly, go to Grafana, choose the InfluxDB data source, and try to run a query to see that the setup process was successful.

PXE Boot with Synology NAS

Fri, 13 May 2022 00:00:00 +0000

In this post, I have documented the steps I followed to install RHEL 8 by booting from a PXE server over the network with a Kickstart file using Synology NAS as TFTP, and HTTP server.

Configure TFTP
Install & Configure HTTP server
PXE Boot
- Prepare Installation
- Kickstart file
- PXE boot

TFTP

Trivial File Transfer Protocol (TFTP) is a simple file transfer protocol, generally used for transferring configurations or boot files when authentication is not required.

Synology NAS comes with TFTP file services.

Configure TFTP

Go to Control Panel > File Services > Advanced

Select Enable TFTP Service
In the TFTP root folder field, specify which folder on the Synology NAS can be accessed by TFTP clients

HTTP server

We also need a service to host our image repository; we can use FTP or HTTP, or NFS to host our image repository. Synology NAS comes with web hosting features. In Web Station, we will create a port-based virtual host to host the contents of our image repository.

Install Packages

Install the following packages from the Synology package center

Web Station
Apache HTTP Server 2.4

Configure

Open Web Station and go to Web Service Portal > Select Create Service Portal

Select Virtual Host

In the virtual host wizard

Select Port-based with protocol as HTTP and enter port number 8080
Select the Document root directory, which is the exact PXE location as TFTP
Select Apache HTTP Server as the back-end server
Click Create to save settings

We need to enable simple directory browsing on the virtual host to access the contents of images and configuration files to perform PXE Boot.

In the Document root directory, create a file named .htaccess with the following content.

Options +Indexes

Navigate to IP:8080, and you should see a similar screen with files in the folder.

PXE Boot

Prepare Installation

Download the ISO image file of the complete installation DVD
Create a directory named images/rhel/8.6 under the root directory of the TFTP/HTTP server
Mount the ISO image
Copy all the files from the ISO to the previously created directory on TFTP/HTTP server. You will need around 10GB of storage to copy all the files from the ISO to the local directory.

mount rhel-8.6-x86_64-dvd.iso /mnt
scp -pr /mnt/* nas-user@nas-address:/media/PXE/images/rhel/8.6

To perform the UEFI PXE Boot installation, we will need the following PXE boot files.

grubx64.efi provided by grub2-efi-x64 rpm
shimx64.efi provided by shim-x64 rpm
BOOTX64.EFI provided by shim-x64 rpm

cp -pr /mnt/BaseOS/Packages/grub2-efi-x64-2.02-81.el8.x86_64.rpm /tmp
cp -pr /mnt/BaseOS/Packages/shim-x64-15-11.el8.x86_64.rpm /tmp/

Extract the packages

cd /tmp
rpm2cpio shim-x64-15-11.el8.x86_64.rpm | cpio -dimv
rpm2cpio grub2-efi-x64-2.02-81.el8.x86_64.rpm | cpio -dimv

Copy the EFI boot images from the /tmp directory to the PXE root directory

scp /tmp/boot/efi/EFI/BOOT/BOOTX64.EFI nas-user@nas-address:/media/PXE
scp /tmp/boot/efi/EFI/centos/shimx64.efi nas-user@nas-address:/media/PXE
scp /tmp/boot/efi/EFI/centos/grubx64.efi nas-user@nas-address:/media/PXE

Add a configuration file named grub.cfg to the TFTP/HTTP root directory. We need initrd and vmlinuz files to load the Operating System until the hard disk and other interfaces are detected. In the grub.cfg take note of linuxefi and initrdefi. Please update it to TFTP/HTTP image directory location.

set timeout=30
menuentry 'Install RHEL 8.6 on T7910' {
 linuxefi images/rhel/8.6/images/pxeboot/vmlinuz inst.stage2=http://192.168.100.160:8080/images/rhel/8.6 quiet
 initrdefi images/rhel/8.6/images/pxeboot/initrd.img
}

Kickstart file

Next, we will create our kickstart file for an unattended automated installation. I have used the kickstart configuration tool available at https://access.redhat.com/labs/kickstartconfig/ in the Red Hat Customer Portal Labs. This tool will walk you through basic configuration and allows you to download the resulting Kickstart file.

Create a new directory named ks under the root directory of the TFTP/HTTP server to store the kickstart file for the UEFI PXE Boot purpose
Copy the downloaded kickstart file into the previously created location

scp kickstart.cfg nas-user@nas-address:/media/PXE/ks

Update the grub.cfg to use the kickstart file during installation. For the inst.ks= boot option, specify kickstart location

The final grub.cfg will look like this:

set timeout=30
menuentry 'Install RHEL 8.6 on T7910' {
 linuxefi images/rhel/8.6/images/pxeboot/vmlinuz inst.ks=http://192.168.100.160:8080/ks/rhel8-t7910.cfg inst.stage2=http://192.168.100.160:8080/images/rhel/8.6 quiet
 initrdefi images/rhel/8.6/images/pxeboot/initrd.img
}

PXE boot

We are ready to perform UEFI PXE Boot. The shortcut button to boot over the network may vary for different hardware, F12 on most common hardware.

If your UEFI PXE boot server configuration is accurate, then the TFTP files should acquire successfully.

Helm Alternative for Multiple If-Else Conditions

Thu, 08 Jul 2021 00:00:00 +0000

Recently I was in a situation where I ended up writing multiple if-else statements in the helm template. As a result, the code block wasn’t clean, and I began to explore alternative ways to achieve the same behavior concisely.

Helm dict function perfectly fits my use case.

Before

{{- if eq .Values.imageType "java" }}
 name: openjdk:latest
{{- else if eq .Values.imageType "nodejs" }}
 name: nodejs:latest
{{- else if eq .Values.imageType "golang" }}
 name: golang:latest
{{- else if eq .Values.imageType "python" }}
 name: python:latest
{{- end }}

After

{{- $imageTypes := dict }}
{{- $_ := set $imageTypes "java" "openjdk:latest" }}
{{- $_ := set $imageTypes "nodejs" "nodejs:latest" }}
{{- $_ := set $imageTypes "python" "openjdk:latest" }}
{{- $_ := set $imageTypes "java" "openjdk:latest" }}
...
 name: {{ get $imageTypes .Values.imageType }}

Build Multi-Arch Docker Images Using Tekton

Tue, 19 Jan 2021 00:00:00 +0000

There are three different strategies to build multi-platform images that Buildx and Dockerfiles support:

Using the QEMU emulation support in the kernel
Building on multiple native nodes using the same builder instance
Using a stage in Dockerfile to cross-compile to different architectures

I’ll focus on the first option, cross-building with emulation using buildx.

docker buildx build --platform linux/amd64,linux/arm64 .

Setup

Steps might differ from platform to platform, but the pipelines will remain the same. So I’ve constructed a basic buildx setup for the TektonCD task.

Pipeline

I am taking the upstream docker-build task and modifying it to my needs

What’s modified

Enable DOCKER_BUILDKIT and DOCKER_CLI_EXPERIMENTAL
Install docker-buildx plugin
Run the latest docker/binfmt tag to use its qemu parts
Docker login (using secret for values)
Docker build with docker buildx build --platform
Create a secret holding docker username and token

kubectl create secret generic docker-token \
 --from-literal=username="${CONTAINER_REGISTRY_USER}" \
 --from-literal=password="${CONTAINER_REGISTRY_PASSWORD}"

apiVersion: tekton.dev/v1beta1
kind: ClusterTask
metadata:
 name: docker-buildx
 labels:
 app.kubernetes.io/version: "0.1"
 annotations:
 tekton.dev/pipelines.minVersion: "0.12.1"
 tekton.dev/tags: docker, build-image, push-image, dind, buildx, multi-arch
 tekton.dev/displayName: docker-buildx
spec:
 description: >-
 This task will build and push an image using docker buildx.
 The task will build an out image out of a Dockerfile.
 This image will be pushed to an image registry.
 The image will be built and pushed using a dind sidecar over TCP+TLS.
 params:
 - name: image
 description: Reference of the image docker will produce.
 - name: builder_image
 description: The location of the docker builder image.
 default: docker.io/library/docker:19.03.14
 - name: dockerfile
 description: Path to the Dockerfile to build.
 default: ./Dockerfile
 - name: context
 description: Path to the directory to use as context.
 default: .
 - name: build_extra_args
 description: Extra parameters passed for the build command when building images.
 default: "--platform linux/amd64,linux/arm/v7 --no-cache"
 - name: push_extra_args
 description: Extra parameters passed for the push command when pushing images.
 default: "--push"
 - name: insecure_registry
 description: Allows the user to push to an insecure registry that has been specified
 default: ""
 - name: docker-token-secret
 type: string
 description: name of the secret holding the docker-token
 default: docker-token
 workspaces:
 - name: source
 results:
 - name: IMAGE_DIGEST
 description: Digest of the image just built.
 steps:
 - name: buildx-build-push
 image: $(params.builder_image)
 env:
 # Connect to the sidecar over TCP, with TLS.
 - name: DOCKER_HOST
 value: tcp://localhost:2376
 # Verify TLS.
 - name: DOCKER_TLS_VERIFY
 value: '1'
 # Use the certs generated by the sidecar daemon.
 - name: DOCKER_CERT_PATH
 value: /certs/client
 - name: DOCKERHUB_USER
 valueFrom:
 secretKeyRef:
 name: $(params.docker-token-secret)
 key: username
 - name: DOCKERHUB_PASS
 valueFrom:
 secretKeyRef:
 name: $(params.docker-token-secret)
 key: password
 workingDir: $(workspaces.source.path)
 script: |
 # install depends
 apk add curl jq

 # enable experimental buildx features
 export DOCKER_BUILDKIT=1
 export DOCKER_CLI_EXPERIMENTAL=enabled

 # Download latest buildx bin from github
 mkdir -p ~/.docker/cli-plugins/
 BUILDX_LATEST_BIN_URI=$(curl -s -L https://github.com/docker/buildx/releases/latest | grep 'linux-amd64' | grep 'href' | sed 's/.*href="/https:\/\/github.com/g; s/amd64".*/amd64/g')
 curl -s -L ${BUILDX_LATEST_BIN_URI} -o ~/.docker/cli-plugins/docker-buildx
 chmod a+x ~/.docker/cli-plugins/docker-buildx

 # Get and run the latest docker/binfmt tag to use its qemu parts
 BINFMT_IMAGE_TAG=$(curl -s https://registry.hub.docker.com/v2/repositories/docker/binfmt/tags | jq '.results | sort_by(.last_updated)[-1].name' -r)
 docker run --rm --privileged docker/binfmt:${BINFMT_IMAGE_TAG}

 docker context create tls-environment
 # create the multibuilder
 docker buildx create --name multibuilder --use tls-environment
 docker buildx use multibuilder

 # login to a registry
 echo ${DOCKERHUB_PASS} | docker login -u ${DOCKERHUB_USER} --password-stdin

 # build the containers and push them to the registry then display the images
 docker buildx build $(params.build_extra_args) \
 -f $(params.dockerfile) -t $(params.image) $(params.context) $(params.push_extra_args)

 volumeMounts:
 - mountPath: /certs/client
 name: dind-certs
 sidecars:
 - image: docker:19.03.14-dind
 name: server
 args:
 - --storage-driver=vfs
 - --userland-proxy=false
 - --debug
 securityContext:
 privileged: true
 env:
 # Write generated certs to the path shared with the client.
 - name: DOCKER_TLS_CERTDIR
 value: /certs
 volumeMounts:
 - mountPath: /certs/client
 name: dind-certs
 # Wait for the dind daemon to generate the certs it will share with the
 # client.
 readinessProbe:
 periodSeconds: 1
 exec:
 command: ['ls', '/certs/client/ca.pem']
 volumes:
 - name: dind-certs
 emptyDir: {}

Make sure you have a base image that supports multi-arch like alpine or one of these base images

Tekton Triggers and CEL Interceptors

Tue, 19 Jan 2021 00:00:00 +0000

Tekton Triggers work by having EventListeners receive incoming webhook notifications, processing them using an Interceptor, and creating Kubernetes resources from templates if the interceptor allows it, with the extraction of fields from the body of the webhook

CEL Interceptors can filter or modify incoming events. For example, you can truncate the commit id from the webhook body.

apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
 name: shared-listener
 namespace: default
spec:
 serviceAccountName: build-bot
 triggers:
 - name: shared-pipeline-trigger
 interceptors:
 - cel:
 overlays:
 - key: intercepted.commit_id_short
 expression: "body.head_commit.id.truncate(7)"
 bindings:
 - ref: pipeline-binding
 template:
 ref: pipeline-template

The applied overlay uses an extension body in the binding. As shown in below example as $(extensions.<overlay_key>)

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
 name: pipeline-binding
 namespace: default
spec:
 params:
 - name: git-repo-name
 value: $(body.repository.name)
 - name: git-repo-url
 value: $(body.repository.url)
 - name: image-tag
 value: $(extensions.intercepted.commit_id_short)

GitOps with Tekton and ArgoCD

Thu, 14 Jan 2021 00:00:00 +0000

In this post, I will:

Use Tekton to build and publish an image to the docker registry
Trigger a Tekton pipeline from GitHub
Use ArgoCD to deploy an application

Getting Started

Pre-requisites:

An OpenShift 4 cluster with ArgoCD and OpenShift Pipelines installed. If not, you follow instructions to OpenShift Pipelines and ArgoCD Operator
Basic understanding of ArgoCD and Tekton concepts

Create an OpenShift project called node-web-project

oc new-project node-web-project

Using Private Registries

Create a docker secret with registry authentication details

oc create secret docker-registry container-registry-secret \
 --docker-server=$CONTAINER_REGISTRY_SERVER \
 --docker-username=$CONTAINER_REGISTRY_USER \
 --docker-password=$CONTAINER_REGISTRY_PASSWORD -n node-web-project

Create a service account named build-bot

oc create sa -n node-web-project build-bot
serviceaccount/build-bot created

Add docker secret container-registry-secret to newly created service account build-bot

oc patch serviceaccount build-bot \
 -p '{"secrets": [{"name": "container-registry-secret"}]}'
serviceaccount/build-bot patched

Verify if the service account has the secret added:

oc get sa -n node-web-project build-bot -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
 creationTimestamp: "2021-01-12T03:34:32Z"
 name: build-bot
 namespace: node-web-project
 resourceVersion: "53879"
 selfLink: /api/v1/namespaces/node-web-project/serviceaccounts/build-bot
 uid: 628067fd-91d1-4cdd-b6a6-88b4f7280ff0
secrets:
- name: container-registry-secret
- name: build-bot-token-8nl2v

Create Pipeline

A Pipeline is a collection of tasks you want to run as part of your workflow. Each Task in a Pipeline gets executed in a pod and runs in parallel by default. However, you can specify the order by using runAfter.

The below pipeline consists of three tasks.

Cloning source code
Build and push images using the Buildah cluster task
Synchronize Argo deployment

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
 name: node-web-app-pipeline
 namespace: node-web-project
spec:
 workspaces:
 - name: shared-workspace
 params:
 - name: node-web-app-source
 - name: node-web-app-image
 tasks:
 - name: fetch-repository
 taskRef:
 kind: ClusterTask
 name: git-clone
 workspaces:
 - name: output
 workspace: shared-workspace
 params:
 - name: url
 value: $(params.node-web-app-source)
 - name: build-and-publish-image
 params:
 - name: IMAGE
 value: $(params.node-web-app-image)
 - name: TLSVERIFY
 value: 'false'
 taskRef:
 kind: ClusterTask
 name: buildah
 runAfter:
 - fetch-repository
 workspaces:
 - name: source
 workspace: shared-workspace
 - name: sync-application
 taskRef:
 name: argocd-task-sync-and-wait
 params:
 - name: application-name
 value: node-web-app
 - name: flags
 value: --insecure --grpc-web
 - name: argocd-version
 value: v1.7.11
 runAfter:
 - build-and-publish-image

Triggers

Now that the pipeline is ready, the next step is to set up a GitHub webhook to trigger the pipeline. But, first, we need to create the following resources:

A TriggerTemplate act as a blueprint for pipeline resources. We can also use it to define parameters that can be substituted anywhere within the resource template(s).

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
 name: pipeline-template
 namespace: node-web-project
spec:
 params:
 - name: git-repo-url
 - name: git-repo-name
 - name: git-revision
 - name: image-name
 resourcetemplates:
 - apiVersion: tekton.dev/v1beta1
 kind: PipelineRun
 metadata:
 generateName: $(tt.params.git-repo-name)-pipeline-run-
 namespace: node-web-project
 spec:
 params:
 - name: source
 value: $(tt.params.git-repo-url)
 - name: image
 value: $(tt.params.image-name)
 pipelineRef:
 name: node-web-app-pipeline
 serviceAccountName: build-bot
 timeout: 1h0m0s
 workspaces:
 - name: shared-workspace
 persistentVolumeClaim:
 claimName: tekton-workspace-pvc

A TriggerBinding binds the incoming event data to the template (i.e., git URL, repo name, revision, etc….)

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
 name: pipeline-binding
 namespace: node-web-project
spec:
 params:
 - name: git-repo-name
 value: $(body.repository.name)
 - name: git-repo-url
 value: $(body.repository.url)
 - name: git-revision
 value: $(body.head_commit.id)
 - name: image-name
 value: docker.io/vikaspogu/$(body.repository.name):$(body.head_commit.id)

An EventListener that will create a pod application bringing together a binding and a template

apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
 name: el-node-web-app
 namespace: default
spec:
 serviceAccountName: build-bot
 triggers:
 - name: node-web-app-trigger
 bindings:
 - ref: pipeline-binding
 template:
 ref: pipeline-template

An OpenShift Route to expose the Event Listener

$ oc expose svc el-node-web-app
$ echo "URL: $(oc get route el-node-web-app --template='http://{{.spec.host}}')"
http://el-node-web-app-node-web-project.apps.cluster-7d51.sandbox659.opentlc.com

You can learn about Tekton Triggers and OpenShift Pipelines

Create ArgoCD App for Web App Resources

Create an ArgoCD application via the GUI or command line

Project: default
cluster: (URL Of your OpenShift Cluster)
namespace should be the name of your OpenShift Project
repo URL: git repo
Target Revision: Head
PATH: deployment
AutoSync Disabled

Configure Webhooks

GitHub webHook for Tekton event listener to start a Tekton build on git push.

Make a code change and commit, look at the build

Push an empty commit to the repo

git commit -m "empty-commit" --allow-empty && git push

In OpenShift Console, you should see a pipeline run
Once the pipeline is finished. Use the OpenShift route to verify the app

Workspace PVC

It would be best if you had a ReadWriteMany PVC to use with multiple pipelines.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: tekton-workspace-pvc
 namespace: default
spec:
 accessModes:
 - ReadWriteMany
 resources:
 requests:
 storage: 2Gi

Setting Up Docker Buildx on Linux

Sat, 05 Dec 2020 00:00:00 +0000

Docker buildx on Linux

Installation instructions for Ubuntu

To execute docker commands without sudo. We need to add the username to the docker group.

Find the current username by typing who
Add username to docker group
Reboot system

$ who
vikaspogu :0 2020-12-05 10:10 (:0)
$ sudo usermod -aG docker vikaspogu
$ sudo reboot

Enable buildx

Create a new config.json file under the ~.docker folder and enable the experimental feature

$ mkdir ~/.docker
$ vim ~/.docker/config.json
{
 "experimental": "enabled"
}

Now you should be able to do multi-arch builds

docker buildx build -t vikaspogu/test:v0.0.1 . --push --platform linux/arm64

If you encounter the below error when trying to build the image using buildx. It would help if you created new build instance

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/arm64 not supported

Create build instance

Follow the below steps to create a new build instance for arm64.

List existing buildx platforms
Create a new build instance named arm64
Inspect build instance to verify arm64 platform exists
Set builder instance
Build multi-platform image

$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
default docker
 default default running linux/amd64, linux/386

# create the builder
$ docker buildx create --name amr64 --platform linux/amd64,linux/arm64

# start the buildx
$ docker buildx inspect arm64 --bootstrap
Name: arm64
Driver: docker-container

Nodes:
Name: arm640
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64*, linux/arm64*, linux/riscv64, linux/ppc64le, linux/386, linux/arm/v7, linux/arm/v6, linux/s390x

# set current builder instance
$ docker buildx use amr64

# build the multi images
$ docker buildx build -t vikaspogu/test:v0.0.1 . --push --platform linux/arm64

If you encounter /bin/sh: Invalid ELF image for this architecture error. Run the following docker image and then build

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Boot from USB Through GRUB Menu

Tue, 01 Dec 2020 00:00:00 +0000

First, make sure you have secure boot disabled from the firmware settings. Once you are in the grub command line, type ls to list all partitions

grub>ls
(hd0) (hd0,gpt1) (hd1) (hd1,gpt8) (cd0))

Type ls (cd0) to get the UUID of the device.

grub>ls (hd0,gpt1)
Partition hd0,gpt1: Filesystem type fat - Label `CES_X64FREV`, UUID 4099-DBD9 Partition start-512 Sectors...

Note the UUID of your USB drive, shown in the above command

Type the following commands

insmod part_gpt
insmod fat
insmod search_fs_uuid
insmod chain
search --fs-uuid --set=root 409-DBD9

Replace the UUID of your device

Now we select the EFI file to boot from

chainloader /efi/boot/bootx64.efi
boot

That’s it; that should boot the USB drive.

End User Auth with OpenShift Service Mesh and Keycloak

Thu, 12 Nov 2020 00:00:00 +0000

In this article, I will share the setup for enabling Authentication and Authorization in OpenShift Service Mesh with Keycloak.

Installing OpenShift Service Mesh

Follow the Installing Red Hat OpenShift Service Mesh guide for setup

Enable the following configuration in your ServiceMeshControlPlane resource

Strict mTLS across the mesh
Automatic istio route creation

apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
spec:
 version: v1.1
 security:
 controlPlane:
 mtls: true
 gateways:
 OpenShiftRoute:
 enabled: true

Keycloak

Keycloak is an open-source identity and access management application that uses open protocols and is easily integrated with other providers. It is the open-source project base of Red Hat Single Sign-on

Deploying Red Hat Single Sign-on

The easiest way to deploy SSO is from the operator hub.

Follow the Keycloak identity provider article for adding a new security realm, client, role, user

Deploying Bookinfo example application

Create a new namespace

oc new-project bookinfo

Edit the default Service Mesh Member Roll YAML and add bookinfo to the member’s list

apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
 name: default
spec:
 members:
 - bookinfo

From the CLI, deploy the Bookinfo application in the bookinfo project by applying the bookinfo.yaml file

oc apply -n bookinfo -f https://raw.githubusercontent.com/Maistra/istio/maistra-2.0/samples/bookinfo/platform/kube/bookinfo.yaml

Create the ingress gateway by applying the bookinfo-gateway.yaml file

oc apply -n bookinfo -f https://raw.githubusercontent.com/Maistra/istio/maistra-2.0/samples/bookinfo/networking/bookinfo-gateway.yaml

Set the value for the GATEWAY_URL parameter

oc get routes -n istio-system | grep bookinfo
export GATEWAY_URL=$(oc -n istio-system get route bookinfo-gateway-pl2rw -o jsonpath='{.spec.host}')

Adding default destination rules

I have enabled global mutual TLS in the control plane, so I’ll deploy the destination rule with all mtls

oc apply -n bookinfo -f https://raw.githubusercontent.com/Maistra/istio/maistra-2.0/samples/bookinfo/networking/destination-rule-all-mtls.yaml

Verifying the Bookinfo installation

Run the following command to confirm that the Bookinfo application is up and running

curl -o /dev/null -s -w "%{http_code}\n" http://$GATEWAY_URL/productpage

Authentication

Enabling User-End Authentication

Now it is time to enable end-user authentication.

The first thing you need to do is validate that it is possible to communicate between all services without authentication.

curl -k -o /dev/null -w "%{http_code}" http://$GATEWAY_URL/productpage
200

You can create the end-user authentication policy.

cat <<EOF | oc apply -n bookinfo -f -
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
 name: "productpage-jwt"
 namespace: "bookinfo"
spec:
 targets:
 - name: productpage
 peers:
 - mtls: {}
 origins:
 - jwt:
 issuer: "https://keycloak-sso.apps.amp01.lab.amp.aapaws/auth/realms/istio"
 jwksUri: "https://keycloak-sso.apps.amp01.lab.amp.aapaws/auth/realms/istio/protocol/openid-connect/certs"
 audiences:
 - customer
 triggerRules:
 - excludedPaths:
 - prefix: /healthz
 principalBinding: USE_ORIGIN
EOF

NOTE: If you see Origin authentication failed. after passing the correct access token and URL. Verify your istio-pilot pods for any x509: certificate signed by unknown authority; if that is the case, follow this workaround

Then let’s rerun the curl.

curl -k http://$GATEWAY_URL/productpage

And you will see something like

Origin authentication failed.

Set the value for the TOKEN parameter from keyclock

export TOKEN=$(curl -sk --data "username=demo&password=demo&grant_type=password&client_id=istio" https://keycloak-sso.apps.amp01.lab.amp.aapaws/auth/realms/istio/protocol/openid-connect/token | jq ".access_token")

Then let’s rerun the curl, this time with the token.

curl -k -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $TOKEN" http://$GATEWAY_URL/productpage
200

Authorization

Create a deny-all policy in the namespace. The policy doesn’t have a selector field, which applies to every workload in the namespace. The spec: field in the policy has the empty value {}, this means that no traffic is permitted, effectively denying all requests

$ cat <<EOF | oc apply -n bookinfo -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: deny-all
spec:
 {}
EOF

Once the policy takes effect, verify that mesh rejected the curl connection to the workload.

$ curl -k -H "Authorization: Bearer $TOKEN" http://$GATEWAY_URL/productpage
RBAC: access denied

To give read access to the product page workload, create the policy that applies to the workload with label app: product page and allows users with roles customer to access it with all method.

$ cat <<EOF | oc apply -n bookinfo -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: "productpage-authz"
 namespace: "bookinfo"
spec:
 selector:
 matchLabels:
 app: productpage
 rules:
 - to:
 - operation:
 methods: ["*"]
 when:
 - key: request.auth.claims[roles]
 values: ["customer"]
EOF

Wait for the newly defined policy to take effect.

After the policy takes effect, verify the connection to the httpbin workload succeeds.

curl -k -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $TOKEN" http://$GATEWAY_URL/productpage
200

However, you can see the following errors on the page.

Error fetching product details
Error fetching product reviews on the page

Run the following command to create the details-viewer policy to allow the productpage workload, which issues requests using the cluster.local/ns/bookinfo/sa/bookinfo-productpage service account, to access the details workload via GET methods.

oc apply -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: "details-viewer"
 namespace: bookinfo
spec:
 selector:
 matchLabels:
 app: details
 rules:
 - from:
 - source:
 principals: ["cluster.local/ns/bookinfo/sa/bookinfo-productpage"]
 to:
 - operation:
 methods: ["GET"]
EOF

Run the following command to create a policy reviews-viewer to allow the productpage workload, which issues requests using the cluster.local/ns/bookinfo/sa/bookinfo-productpage service account, to access the reviews workload through GET methods

oc apply -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: "reviews-viewer"
 namespace: bookinfo
spec:
 selector:
 matchLabels:
 app: reviews
 rules:
 - from:
 - source:
 principals: ["cluster.local/ns/bookinfo/sa/bookinfo-productpage"]
 to:
 - operation:
 methods: ["GET"]
EOF

Point your browser at the Bookinfo productpage (http://$GATEWAY_URL/productpage). Now, you should see the “Bookinfo Sample” page with “Book Details” on the lower left part and “Book Reviews” on the lower right part.

However, in the “Book Reviews” section, you’ll see an error Rating service is currently unavailable. This error is because the reviews workload doesn’t permit access to the ratings workload. To fix this issue, you need to grant the reviews workload access to the ratings workload. Next, configure a policy to grant access to the reviews workload.

Run the following command to create the ratings-viewer policy to allow the reviews workload, which issues requests using the cluster.local/ns/bookinfo/sa/bookinfo-reviews service account, to access the ratings workload through GET methods

oc apply -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
 name: "ratings-viewer"
 namespace: bookinfo
spec:
 selector:
 matchLabels:
 app: ratings
 rules:
 - from:
 - source:
 principals: ["cluster.local/ns/bookinfo/sa/bookinfo-reviews"]
 to:
 - operation:
 methods: ["GET"]
EOF

Point your browser at the Bookinfo productpage (http://$GATEWAY_URL/productpage). You should see the “black” and “red” ratings in the “Book Reviews” section.

Syslog with Loki Promtail

Wed, 04 Nov 2020 00:00:00 +0000

Enable Syslog, do this on each host, and replace target IP (and maybe port) with your Syslog externalIP that is in helm values for promtail

promtail:
 serviceMonitor:
 enabled: true
 extraScrapeConfigs:
 - job_name: syslog
 syslog:
 listen_address: 0.0.0.0:1514
 label_structured_data: yes
 labels:
 job: "syslog"
 relabel_configs:
 - source_labels: ["__syslog_connection_ip_address"]
 target_label: "ip_address"
 - source_labels: ["__syslog_message_severity"]
 target_label: "severity"
 - source_labels: ["__syslog_message_facility"]
 target_label: "facility"
 - source_labels: ["__syslog_message_hostname"]
 target_label: "host"
 syslogService:
 enabled: true
 type: LoadBalancer
 port: 1514
 externalIPs:
 - 192.168.42.155

Create file /etc/rsyslog.d/50-promtail.conf with the following content:

module(load="omprog")
module(load="mmutf8fix")
action(type="mmutf8fix" replacementChar="?")
action(type="omfwd" protocol="tcp" target="192.168.42.155" port="1514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on")

Restart rsyslog and view status

sudo systemctl restart rsyslog
sudo systemctl status rsyslog

In Grafana, on the explore tab, you should now be able to view your host’s logs, e.g., this query {host="k3s-master"}.

Helm Join Strings in a Named Template

Thu, 08 Oct 2020 00:00:00 +0000

The key benefit of Helm is that it helps reduce the configuration a user needs to provide to deploy applications to Kubernetes. With Helm, we can have a single chart that can deploy all the microservices.

Unique ServiceAccount

Recently we wanted to create a unique service account for each microservice, so all microservices don’t share the same service account using the helm template. In our example, we will append the release name with the service account name.

Using join function

{{ (list .Values.serviceAccount.name (include "openjdk.fullname" .) | join "-") }}

Adding fail condition

{{- if .Values.serviceAccount.name -}} 
 {{ (list .Values.serviceAccount.name (include "openjdk.fullname" .) | join "-") }} 
{{- else -}} 
 {{- fail "Please enter a name for service account to create." }} 
{{- end -}}

Full example

{{- define "openjdk.serviceAccountName" -}}
{{- if .Values.serviceAccount.create -}} 
 {{- if .Values.serviceAccount.name -}} 
 {{ (list .Values.serviceAccount.name (include "openjdk.fullname" .) | join "-") }} 
 {{- else -}} 
 {{- fail "Please enter a name for service account to create." }} 
 {{- end -}}
{{- else -}} 
 {{- if .Values.serviceAccount.name -}} 
 {{ (list .Values.serviceAccount.name (include "openjdk.fullname" .) | join "-") }} 
 {{- else -}} 
 "default"
 {{- end -}}
{{- end -}}
{{- end -}}

Another way to manipulate strings in helpers.tpl is using printf

{{- define "ocp-openjdk.hostname" -}}
{{- printf "%s-%s%s" .Release.Name .Release.Namespace (.Values.subdomain | default ".apps.amp01.nonprod" ) -}}
{{- end -}}

Resources

Sprig functions

AdGuard Home on Kubernetes

Wed, 19 Aug 2020 00:00:00 +0000

AdGuard

AdGuard Home is a network-wide software for blocking ads & tracking

Adguard is similar to Pi-Hole with more features. Comparison of Adguard to Pi-Hole

The below configuration has worked for me. I am a big fan of helm charts, and I’ll be using AdGuard chart

Configure chart

Pull chart locally

helm repo add billimek https://billimek.com/billimek-charts/
helm fetch billimek/adguard-home

Update deployment to use hostNetwork

#templates/deployment.yaml
...
 spec:
 hostNetwork: true
 securityContext:
...

Enable configAsCode, update bind_host to Kubernetes host IP in values.yaml

# values.yaml
configAsCode:
 enabled: true
 config:
 bind_host: 192.168.0.101
 bind_port: 3000
 dns:
 bind_host: 192.168.0.101

Update securityContext to run as a privileged pod, drop all capabilities and add NET_BIND_SERVICE

# values.yaml
securityContext:
 privileged: true
 capabilities:
 drop:
 - ALL
 add:
 - NET_BIND_SERVICE

Add nodeSelector to assign a pod to that node.

# values.yaml
nodeSelector:
 kubernetes.io/hostname: node3

Deploy helm chart

helm install adguard-home

Wait for AdGuard pods.

kubectl get pods
NAME READY STATUS RESTARTS AGE
adguard-adguard-home-67975f7768-v6bg9 1/1 Running 0 43h

Configure your devices to use your AdGuard Home

Once we’ve established that AdGuard Home has deployed, you can use it on other computers in your network by changing their system DNS settings to use the Kubernetes node’s IP address (which is 192.168.0.101 in our case).

Configure Jenkins Pipeline Job with JCasC

Tue, 18 Aug 2020 00:00:00 +0000

In one of my previous posts, I discussed configuring multibranch pipeline seed jobs using Jenkins configuration as code plugin

Below is an example of configuring a declarative pipeline job from a bitbucket repo, running at midnight every day

- script: >
 pipelineJob('sample-job') {
 definition {
 cpsScm {
 scriptPath 'job1/Jenkinsfile' ## If the Jenkins job is in a nested folder
 scm {
 git {
 remote {
 url 'https://github.com/Vikaspogu/sample-repo'
 credentials 'sample-creds'
 }
 branch '*/master'
 extensions {}
 }
 }
 triggers {
 cron('@midnight')
 }
 }
 }
 }

OPA Gatekeeper on OpenShift

Tue, 18 Aug 2020 00:00:00 +0000

Every organization has policies. Some are essential to meet governance and legal requirements. Others help ensure adherence to best practices and institutional conventions. Attempting to ensure compliance manually would be error-prone and frustrating.

OPA allows users to a specific policy as code using OPA’s policy language Rego

In this post, I’ll share my experience deploying OPA Gatekeeper on OpenShift and creating a few policies for demonstrations. This post is not an introduction to OPA refer to for an intro

Gatekeeper introduces native Kubernetes CRDs for instantiating policies.

Installation

For installation, make sure you have cluster-admin permissions.

Let’s start by adding the admission.gatekeeper.sh/ignore label to non-user namespaces so that all the resources in the labeled project are exempted from the admission webhook.

oc login --token=l4xjpLh0e722B2_i7iWAbPsUNOb6vPDaAXnqhH563oU --server=https://api.cluster-1d4d.sandbox702.opentlc.com:6443

for namespace in $(oc get namespaces -o jsonpath='{.items[*].metadata.name}' | xargs); do
 if [[ "${namespace}" =~ OpenShift.* ]] || [[ "${namespace}" =~ kube.* ]] || [[ "${namespace}" =~ default ]]; then
 oc patch namespace/${namespace} -p='{"metadata":{"labels":{"admission.gatekeeper.sh/ignore":"true"}}}'
 else
 # Probably a user project, so leave it alone
 echo "Skipping: ${namespace}"
 fi
done

Deploying a Release using Prebuilt Image

Deploy Gatekeeper with a prebuilt image

oc apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Remove securityContext and annotations from the deployments.

oc patch Deployment/gatekeeper-audit --type json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations"}]' -n gatekeeper-system
oc patch Deployment/gatekeeper-controller-manager --type json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations"}]' -n gatekeeper-system
oc patch Deployment/gatekeeper-audit --type json --patch '[{ "op": "remove", "path": "/spec/template/spec/containers/0/securityContext" }]' -n gatekeeper-system
oc patch Deployment/gatekeeper-controller-manager --type json --patch '[{ "op": "remove", "path": "/spec/template/spec/containers/0/securityContext" }]' -n gatekeeper-system

Wait for Gatekeeper to be ready.

oc get pods -n gatekeeper-system
NAME READY STATUS RESTARTS AGE
gatekeeper-audit-7c84869dbf-r5p87 1/1 Running 0 2m34s
gatekeeper-controller-manager-ff58b6688-9tbxx 1/1 Running 0 3m23s
gatekeeper-controller-manager-ff58b6688-njfnd 1/1 Running 0 2m54s
gatekeeper-controller-manager-ff58b6688-vlrsl 1/1 Running 0 3m11s

Gatekeeper uses the OPA Constraint Framework to describe and enforce policy

Defining constraints

Users can define constraints by creating a CRD (CustomResourceDefinition) with the template of the constraint they want. For example, let’s look at a template that only enforces users must create secure routes on the cluster.

apiVersion: templates. Gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
 name: k8sallowedroutes
spec:
 crd:
 spec:
 names:
 kind: K8sAllowedRoutes
 targets:
 - target: admission.k8s.gatekeeper.sh
 rego: |
 package k8sallowedroutes

 violation[{"msg": msg}] {
 not input.review.object.spec.tls
 msg := sprintf("'%v' route must be a secured route. non secured routes are not permitted", [input.review.object.metadata.name])
 }

It will get the route and check if the tls object is specified. If this is not true, the violation block continues, and the violation is triggered with the corresponding message.

Enforcing constraints

Define a template using earlier constraints to enforce.

apiVersion: constraints. Gatekeeper.sh/v1beta1
kind: K8sAllowedRoutes
metadata:
 name: secure-route
spec:
 match:
 kinds:
 - apiGroups: ["route.OpenShift.io"]
 kinds: ["Route"]

The above policy uses the CRD “K8sAllowedRoutes”, which we had already defined. Enforcement takes place by matching the API group.

Some constraints are impossible to write without access to more states than the object under test. For example, it is impossible to know if a route’s hostname is unique among all routes unless a rule has access to all other routes. To make such laws possible, we enable syncing of data into OPA.

apiVersion: config. Gatekeeper.sh/v1alpha1
kind: Config
metadata:
 name: config
 namespace: "gatekeeper-system"
spec:
 sync:
 syncOnly:
 - group: ""
 version: "v1"
 kind: "Namespace"
 - group: "route.OpenShift.io"
 version: "v1"
 kind: "Route"

Let’s create another policy that prevents conflicting routes from being created.

apiVersion: templates. Gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
 name: k8suniqueroutehost
spec:
 crd:
 spec:
 names:
 kind: K8sUniqueRouteHost
 targets:
 - target: admission.k8s.gatekeeper.sh
 rego: |
 package k8suniqueroutehost

 identical(obj, review) {
 obj.metadata.namespace == review.object.metadata.namespace
 obj.metadata.name == review.object.metadata.name
 }

 violation[{"msg": msg}] {
 input.review.kind.kind == "Route"
 re_match("^(route.OpenShift.io)$", input.review.kind.group)
 host := input.review.object.spec.host
 other := data.inventory.namespace[ns][otherapiversion]["Route"][name]
 re_match("^(route.OpenShift.io)/.+$", otherapiversion)
 other.spec.host == host
 not identical(other, input.review)
 msg := sprintf("Route host conflicts with an existing route <%v>", [host])
 }

apiVersion: constraints. Gatekeeper.sh/v1beta1
kind: K8sUniqueRouteHost
metadata:
 name: unique-route-host
spec:
 match:
 kinds:
 - apiGroups: ["route.OpenShift.io"]
 kinds: ["Route"]

Resources

Jenkins Pipeline Date Helper Functions

Tue, 07 Jul 2020 00:00:00 +0000

The below snippet shows how to calculate days between two dates in the Jenkins pipeline. @NonCPS annotation is functional when you have methods that use objects that aren’t serializable.

import java.text.SimpleDateFormat

stages {
 stage("Calculate days") {
 steps {
 script {
 def daysRemaining = getRemainingDays("2020-06-25")
 }
 }
 }
 }
}

@NonCPS
def getRemainingDays(previousDate){
 def currentDate = new Date()
 String currentTimeFormat= currentDate.format("yyyy-MM-dd")
 def oldDate = new SimpleDateFormat("yyyy-MM-dd").parse(previousDate)
 return currentTimeFormat-oldDate
}

Calculate if the date is greater than 14 days

import java.time.LocalDate
import java.time.format.DateTimeFormatter

stages {
 stage("Check if date is older than 14 days") {
 steps {
 script {
 if (checkIfOtherThan("2020-06-25")){
 println "Date is greater than 14 days"
 }
 }
 }
 }
 }
}

@NonCPS
def checkIfOlderThan(previousDate){
 def dateFormat = DateTimeFormatter.ofPattern("yyyy-MM-dd")
 def currentDate = LocalDate.now().format(dateFormat);
 def projectpreviousDate = LocalDate.parse(previousDate, dateFormat)
 if (LocalDate.parse(currentDate, dateFormat).minusDays(14) > projectpreviousDate) {
 return true;
 }
 return false;
}

OpenShift Jenkins Configuration via JCasC Plugin

Mon, 06 Jul 2020 00:00:00 +0000

Deploying Jenkins on Kubernetes provides significant benefits over a standard VM-based deployment. For example, we are gaining the ability to have project-specific Jenkins slaves (agents) on demand instead of having a pool of VMs idle waiting for a job.

In vanilla Kubernetes, we can deploy Jenkins using Helm, and In OpenShift, we can deploy Jenkins via the developer’s catalog.

As everyone has experienced, setting up Jenkins is a complex process. Both Jenkins and its plugins require some tuning and Configuration, with dozens of parameters to set within the web UI manage section.

Jenkins Config as code

Configuration as code gives you an opinionated way to configure Jenkins based on yaml files

This post will cover Jenkins configuration code on OpenShift.

Mount Jenkins config as a configmap and load configuration
Updated configuration file to automate the creation of
- credentials
- script approval signatures
- shared libraries
- multibranch pipeline seed jobs

Install Plugin

First, let us install configuration-as-code plugins in Jenkins. In OpenShift, you can easily install plugins by adding the INSTALL_PLUGINS environment variable to the deployment config

....
env:
 - name: INSTALL_PLUGINS
 value: "configuration-as-code:1.35,configuration-as-code-support:1.18,configuration-as-code-groovy:1.1

Create ConfigMap

Create a configmap from jenkins-config yaml and mount it as a volume at /var/jenkins_config location. Configuration can be now loaded from /var/jenkins_config/jenkins-config.yaml path

oc create configmap jenkins-config --from-file jenkins-config.yaml

....
 volumeMounts:
 - mountPath: /var/jenkins_config
 name: jenkins-config
dnsPolicy: ClusterFirst
restartPolicy: Always
volumes:
 - name: jenkins-config
 configMap:
 name: jenkins-config

Mount Secret

Since we can have Jenkins configuration in the git repo, we don’t want to hardcode the secrets as it is a security risk. The easiest way to automate credentials in Jenkins configuration is to create an OpenShift secret, and add that secret as an environment variable to the deployment config

oc create secret generic github-ssh --from-file=ssh-privatekey=github-ssh/
oc create secret generic jenkins-credentials --from-literal=OCP_SA="qwerty26sds99ie9kcsd" --from-literal=GITHUB_PASSWORD="dummypassword"

Mount secrets into the container

....
env:
....
- name: OCP_SA
 valueFrom:
 secretKeyRef:
 name: jenkins-credentials
 key: OCP_SA
- name: GITHUB_SSH
 valueFrom:
 secretKeyRef:
 name: github-ssh
 key: ssh-privatekey
- name: GITHUB_PASSWORD
 valueFrom:
 secretKeyRef:
 name: jenkins-credentials
 key: GITHUB_PASSWORD
....

credentials:
 system:
 domainCredentials:
 - credentials:
 - fileSystemServiceAccountCredential:
 id: "1a12dfa4-7fc5-47a7-aa17-cc56572a41c7"
 scope: GLOBAL
 - basicSSHUserPrivateKey:
 description: "github ssh credentials"
 id: "github-ssh"
 privateKeySource:
 directEntry:
 privateKey: ${GITHUB_SSH}
 scope: GLOBAL
 username: "github-user"
 - OpenShiftToken:
 id: "jenkins-ocp-token"
 scope: GLOBAL
 secret: ${OCP_SA}
 - usernamePassword:
 description: "github user credentials"
 id: "github-user"
 password: ${GITHUB_PASSWORD}
 scope: GLOBAL
 username: "github-user"

Example Configs

Security script approval

security:
 scriptApproval:
 approvedSignatures:
 - "method java.text.DateFormat parse java.lang.String"

Global shared library

unclassified:
 globalLibraries:
 libraries:
 - defaultVersion: "master"
 name: "shared-pipeline"
 retriever:
 modernSCM:
 scm:
 git:
 credentialsId: "github-ssh"
 id: "shared-pipeline"
 remote: "git@github.com:vikaspogu/jenkins-shared.git"

Multibranch job

Multibranch seed job with periodic polling, traits for branch discovery

jobs:
 - script: >
 multibranchPipelineJob("sample-repo") {
 branchSources {
 branchSource {
 source {
 github {
 //This is a unique identifier; if not set, the pipeline will be
 //indexed, and jobs will be kicked off every time new config is applied
 id("5bb970c2-766b-4588-8cf8-e077bfec23a0")
 credentialsId("github-user")
 repoOwner("vikaspogu")
 repository("sample-repo")
 traits {
 cloneOptionTrait {
 extension {
 shallow (false)
 noTags (false)
 reference (null)
 depth(1)
 honorRefspec (false)
 timeout (10)
 }
 }
 }
 }
 }
 }
 }
 configure {
 def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait' {
 strategyId(3) // detect all branches -refer the plugin source code for various options
 }
 }
 configure {
 def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait' {
 strategyId(2)
 }
 }
 configure {
 def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.TagDiscoveryTrait' {}
 }
 orphanedItemStrategy {
 discardOldItems {
 numToKeep(15)
 }
 }
 triggers {
 periodic(2)
 }
 }

Raspberry Pi Garage Opener on K3s Cluster

Sun, 05 Jul 2020 00:00:00 +0000

Many articles are out there which demonstrate how to use a raspberry pi as a DIY garage door opener project. Few are outdated and not deployed using container images. I found a few reasonable solutions on google, but I couldn’t run them on the Kubernetes cluster due to older packages or insufficient information. I decided to build my solution from different sources of information I found

What we’ll cover in this post

setup nodejs project to simulate a button
create a container from nodejs application
deploy the container on the Kubernetes cluster

Kudos! to the author for this excellent article for showing us how to connect the relay and magnetic switch to raspberry pi for our purpose.

Once finished wiring up all components. Let’s start setting up our Nodejs application.

Application setup

Create an npm project

mkdir garage-pi && cd garage-pi
npm init -y

We’ll be using node-rpio package, which provides access to the Raspberry Pi GPIO interface

Install node-rpio and express packages

npm i rpio express -S

Create an express app that starts on port 8080 in server.js file

"use strict";

const express = require("express");
const rpio = require("rpio");

const app = express();
const PORT = 8080;
app.use("/assets", express.static("assets"));

app.listen(PORT);
console.log("Running on http://localhost:" + PORT);

The below code lets you simulate a button press; In our case, PIN is 19. First, we want to output to low and set the pin to high after 1000ms. Please refer to rpio repo for more explanation

const openPin = process.env.OPEN_PIN || 19;
const relayPin = process.env.RELAY_PIN || 11;

app.get("/relay", function (req, res) {
 // Simulate a button press
 rpio.write(relayPin, rpio.LOW);
 setTimeout(function () {
 rpio.write(relayPin, rpio.HIGH);
 res.send("done");
 }, 1000);
});

To get the state of pin.

function getState() {
 return {
 open: !rpio.read(openPin),
 };
}

app.get("/status", function (req, res) {
 res.send(JSON.stringify(getState()));
});

Complete server.js file

"use strict";

const express = require("express");
const rpio = require("rpio");

const app = express();
const PORT = 8080;
const openPin = process.env.OPEN_PIN || 19;
const relayPin = process.env.RELAY_PIN || 11;

app.use("/assets", express.static("assets"));

function getState() {
 return {
 open: !rpio.read(openPin),
 };
}

app.get("/status", function (req, res) {
 res.send(JSON.stringify(getState()));
});

app.get("/relay", function (req, res) {
 // Simulate a button press
 rpio.write(relayPin, rpio.LOW);
 setTimeout(function () {
 rpio.write(relayPin, rpio.HIGH);
 res.send("done");
 }, 1000);
});

app.listen(PORT);
console.log("Running on http://localhost:" + PORT);

Container image

Create a dockerfile with multi-stage builds using nodejs arm image
Install the necessary python package for rpio
Build docker image
Publish the docker image to your repo in docker hub
Create new Kubernetes deployment and service

# Fetch node_modules for backend; nothing here except
# the node_modules dir ends up in the final image
FROM arm32v7/node:12.18-alpine as builder
RUN mkdir /app
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY package.json /app/package.json
RUN apk add --no-cache make gcc g++ python && \
 npm install --production --silent && \
 apk del make gcc g++ python
RUN npm install

# Add the files to the arm image
FROM arm32v7/node:12.18-alpine
RUN mkdir /app
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH

# Same as earlier, be specific or copy everything
ADD package.json /app/package.json
ADD package-lock.json /app/package-lock.json
ADD . /app

COPY --from=builder /app/node_modules /app/node_modules

ENV PORT=8080
EXPOSE 8080
CMD [ "npm", "start" ]

Docker buildx feature lets you build arm-based images on mac or windows system

docker buildx build --platform linux/arm64 -t <docker-username>/garage-pi .
docker push <docker-username>/garage-pi

Create a new deployment named garage-pi that runs the earlier published image.

kind: Deployment
apiVersion: apps/v1
metadata:
 name: garage-pi
 labels:
 app.kubernetes.io/name: garage-pi
spec:
 replicas: 1
 selector:
 matchLabels:
 app.kubernetes.io/instance: garage-pi
 app.kubernetes.io/name: garage-pi
 template:
 metadata:
 creationTimestamp: null
 labels:
 app.kubernetes.io/instance: garage-pi
 app.kubernetes.io/name: garage-pi
 spec:
 volumes:
 - name: dev-snd
 hostPath:
 path: /dev/mem
 type: ''
 containers:
 - name: garage-pi
 image: '<docker-username>/garage-pi:latest' ##update username here
 ports:
 - name: http
 containerPort: 8080
 protocol: TCP
 resources: {}
 volumeMounts:
 - name: dev-snd
 mountPath: /dev/mem
 livenessProbe:
 httpGet:
 path: /
 port: http
 scheme: HTTP
 readinessProbe:
 httpGet:
 path: /
 port: http
 scheme: HTTP
 imagePullPolicy: Always
 securityContext:
 privileged: true
 restartPolicy: Always

Create a service for a garage-pi deployment, which serves on port 8080 and connects to the containers on port 8080.

kubectl expose deployment nginx --port=8080 --target-port=8080

That’s it; you should be able to hit the endpoint and simulate button click!

Frontend

For frontend of application, we’ll use pugjs templating engine

Install pug package

npm i pug -S

add the views folder in the root directory and create a new index.pug file in the views folder
wire up templating engine with application
render index page on root / endpoint
integrate button with /relay endpoint, which will open/closed garage door

doctype html
head
meta(charset='utf-8')
meta(name='viewport', content='width=device-width, initial-scale=1, shrink-to-fit=no')
meta(name='description', content='')
meta(name='author', content='')
title Garage Opener
.text-center
form.form-signin(method='POST', action='/relay')
h1.h1.mb-2.font-weight-normal(style='color: #FFFFFF') Garage Door
.text-center
.form-signin
.text-center
#open
h1.h2.mb-3.font-weight-normal(style='color: #CF6679') The Door is Open &#xF62B;
button#mainButton.btn.btn-lg.btn-primary.btn-block(type='submit') Close

app.set("views", path.join(__dirname, "views"));
app.set("view engine", "pug");

app.get("/", function (req, res) {
 res.render("index", getState());
});

Twilio Integration (Optional)

Install dotenv, twilio and node-schedule packages

npm i dotenv node-schedule twilio -S

Create a .env file at the root of the project and add your Twilio auth key, account sid, and phone number

TWILIO_ACCOUNT_SID=
TWILIO_AUTH_TOKEN=
TWILIO_PHONE_NUMBER=

Create a twilio.js file, add the below code

require("dotenv").config();

const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;

const sendSms = (phone, message) => {
 const client = require("twilio")(accountSid, authToken);
 client.messages
 .create({
 body: message,
 from: process.env.TWILIO_PHONE_NUMBER,
 to: phone,
 })
 .then((message) => console.log(message.sid));
};

module.exports = sendSms;

Schedule a job for every 15mins to check the state of the garage door. If the door is open, we’ll send a message

const sendSms = require("./twilio");
...
...
schedule.scheduleJob("*/15 * * * *", function () {
 var status = JSON.parse(JSON.stringify(getState()));
 if (status.open) {
 sendSms("<YOUR-NUMBER>", "Garage door is open 🔥");
 }
});

Slack Bot with Node.js

Sat, 04 Jul 2020 00:00:00 +0000

Build your slack bot in a few steps. In this post, we’ll navigate the process of creating the bot.

Slack setup

First, create a slack workspace

Give your workspace a name

Create a new bot at slack apps

Give your new application a name
Choose the workspace you created before installing the bot application

Then go to the Features > OAuth & Permissions screen to scroll down to Bot Token Scopes to specify the OAuth scopes, and select app_mentions and chat_write to enable the bot to send messages.

Before jumping into the application setup, copy the signing secret and verification token from the basic information page. We’ll be using this later in our NodeJS application.

Application setup

Create an npm project, install @slack/bolt and dotenv packages

mkdir test-bot && cd test-bot
npm init -y
npm i dotenv @slack/bolt -S

Add start command to scripts if necessary.

...
"scripts": {
 "start": "node index.js"
}

Create a .env file and add SLACK_SIGNING_SECRET, SLACK_BOT_TOKEN

Note: Don’t commit this file to any repo

SLACK_BOT_TOKEN= #token goes here
SLACK_SIGNING_SECRET= #signing secret goes here

In your index.js file, require the Bolt package, and initialize an app with credentials.

require("dotenv").config();
const { App } = require("@slack/bolt");

const bot = new App({
 signingSecret: process.env.SLACK_SIGNING_SECRET,
 token: process.env.SLACK_BOT_TOKEN,
 endpoints: "/slack/events",
});

(async () => {
 // Start the app
 await bot.start(process.env.PORT || 3000);

 console.log("⚡️ Bolt app is running!");
})();

Deploy the application to a live server like ngrok.

Event Setup

We’ll need to subscribe to events so that when a Slack event happens (like a user mentions an app), the app server will receive an event payload.

Go to Event Subscriptions from the left-hand menu, and turn the toggle switch on to enable events
Enter your Request URL

Subscribe to app_mention event

Install app to workspace

You should see the bot in your workspace now!

Handling Events

To listen to any Events API events from Slack, use the event() method. This method allows your app to take action on Slack events. In this scenario, it’s triggered when a user mentions the app.

bot.event("app_mention", async ({ context, event }) => {
 try {
 const command = event.text;
 let reply;
 if (command.includes("Hi")) {
 reply = `Hi <@${event.user}>, you mentioned me`;
 } else {
 reply = "How can I help you?";
 }
 await bot.client.chat.postMessage({
 token: context.botToken,
 channel: event.channel,
 text: `${reply}`,
 });
 } catch (e) {
 console.log(`error responding ${e}`);
 }
});

Okay, let’s try the app!

Add the app to a channel and mention the app. You should see a response from the bot!

Troubleshooting

Reinstall the app if you don’t see any responses from the bot

Pi-hole on K3s Cluster

Sat, 07 Mar 2020 00:00:00 +0000

Pi-hole is a fantastic tool that blocks DNS requests to ad servers. That means you can surf the web without looking at ads on every page.

Pi-Hole in Kubernetes

We are going to deploy a modified version of this pihole helm chart

Let’s start by cloning the repo.

git clone https://github.com/ChrisPhillips-cminion/pihole-helm.git
cd pihole-helm

We now need to make a few updates to the chart.

Update ServerIP with container host IP
Update image to v5.1.1 tag
Add WEB_PASSWORD, and TZ environment variables if needed
Update values.yaml accordingly

spec:
 replicas: 1
 template:
 metadata:
 labels:
 app: {{ template "fullname" . }}
 spec:
 # hostNetwork: true
 hostAliases:
 - ip: 127.0.0.1
 hostnames:
 - pi.hole
 nodeSelector:
 kubernetes.io/hostname: randomstore
 containers:
 - name: {{ .Chart.Name }}
 image: pihole/pihole:v5.1.1
 imagePullPolicy: {{ .Values.image.pullPolicy }}
 stdin: true
 tty: true
 resources:
 limits:
 memory: 1Gi
 env:
 - name: 'ServerIP'
 value: '192.168.1.132'
 - name: 'DNS1'
 value: '8.8.8.8'
 - name: 'DNS2'
 value: '8.8.4.4'
 - name: TZ
 value: "America/New_York"
 - name: WEBPASSWORD
 value: "somepassword"

#values.yaml
configData: |-
 server=/local/192.168.1.1
 address=/.vikaspogu.com/192.168.1.132
ingress:
 host: pi-hole.vikaspogu.com

Install chart

Create a new namespace (optional)
Install chart in namespace

kubectl create ns pi-hole
helm install pi-hole.

Wait for pods

kubectl get pods
NAME READY STATUS RESTARTS AGE
pi-hole-pihole-5bb56b5bd-b2wl7 1/1 Running 0 61m

Navigate to the ingress route in my case (pi-hole.vikaspogu.com) and log in with WEBPASSWORD used in the deployment

DNS Server

Configure Verizon FiOS router to use Pi-Hole as the DNS server:

On the top navigation menu
- Click My Network
On the left menu list
- Click Network Connections
Click Broadband Connection (Ethernet/Coax)>Settings
- Click the drop-down for DNS Server and select “Use The Following DNS Server Addresses”
- Type in the static IP Address of your pi (Or Pi-hole server)
- Click Apply

Jenkins OpenShift OAuth SSL

Mon, 10 Feb 2020 00:00:00 +0000

Jenkins SSL

Problem

I recently encountered an issue while authenticating to OpenShift Jenkins using the OpenShift OAuth plugin, where trusted certificates provided by CA aren’t included in the default JRE TrustStore.

Logs from Jenkins pod

2020-02-10 21:19:07.335+0000 [id=17] INFO o.o.j.p.o.OpenShiftOAuth2SecurityRealm#transportToUse: OpenShift OAuth got an SSL error when accessing the issuer's token endpoint when using the SA certificate2020-02-10 21:19:07.348+0000 [id=17] INFO o.o.j.p.o.OpenShiftOAuth2SecurityRealm#transportToUse: OpenShift OAuth provider token endpoint failed unexpectedly using the JVMs default keystore
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
 at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
 at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
 at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
Caused: sun.security.validator.ValidatorException: PKIX path building failed
 at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
 at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
 at java.base/sun.security.validator.Validator.validate(Validator.java:264)
 at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
 at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
 at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
 at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:629)
Caused: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
 at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)

Solution

By default, Java Applications (as Jenkins) use the JVM TrustStore. Therefore, if a Java Application needs to use an additional TrustStore, it must be configured.

First, create a secret with the JKS Keystore and password.

oc create secret generic jenkins-https-jks --from-literal=https-jks-password=changeit \
--from-file=custom-keystore.jks

Mount jenkins-https-jks keystore secret as a volume to /var/jenkins_keystore location.

...
 terminationMessagePath: /dev/termination-log
 terminationMessagePolicy: File
 volumeMounts:
 - mountPath: /var/lib/jenkins
 name: jenkins-data
 - mountPath: /var/jenkins_keystore
 name: jenkins-https-keystore
 serviceAccount: jenkins
 serviceAccountName: jenkins
 volumes:
 - name: jenkins-data
 persistentVolumeClaim:
 claimName: jenkins
 - name: jenkins-https-keystore
 secret:
 defaultMode: 420
 items:
 - key: custom-keystore.jks
 path: custom-keystore.jks
 secretName: jenkins-https-jks
...

Add the certificate to Jenkins as startup parameters; we can configure the Jenkins server to add the following JAVA properties to the JAVA_TOOL_OPTIONS environment variable.

-Djavax.net.ssl.trustStore=/var/jenkins_keystore/custom-keystore.jks \
-Djavax.net.ssl.trustStorePassword=changeit

spec:
 containers:
 - env:
 - name: JENKINS_HTTPS_KEYSTORE_PASSWORD
 valueFrom:
 secretKeyRef:
 key: https-jks-password
 name: jenkins-https-jks
 - name: JAVA_TOOL_OPTIONS
 value: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true
 -Djavax.net.ssl.trustStore=/var/jenkins_keystore/custom-keystore.jks -Djavax.net.ssl.trustStorePassword=$(JENKINS_HTTPS_KEYSTORE_PASSWORD)

The final deployment config should look like this:

apiVersion: apps.OpenShift.io/v1
kind: DeploymentConfig
metadata:
 annotations:
 template.alpha.OpenShift.io/wait-for-ready: "true"
 creationTimestamp: "2020-02-10T17:49:20Z"
 generation: 10
 labels:
 app: jenkins-persistent
 name: jenkins
spec:
 replicas: 1
 revisionHistoryLimit: 10
 selector:
 name: jenkins
 strategy:
 activeDeadlineSeconds: 21600
 recreateParams:
 timeoutSeconds: 600
 resources: {}
 type: Recreate
 template:
 metadata:
 creationTimestamp: null
 labels:
 name: jenkins
 spec:
 containers:
 - env:
 - name: OpenShift_ENABLE_OAUTH
 value: "true"
 - name: OpenShift_ENABLE_REDIRECT_PROMPT
 value: "true"
 - name: DISABLE_ADMINISTRATIVE_MONITORS
 value: "false"
 - name: KUBERNETES_MASTER
 value: https://kubernetes.default:443
 - name: KUBERNETES_TRUST_CERTIFICATES
 value: "true"
 - name: JENKINS_SERVICE_NAME
 value: jenkins
 - name: JNLP_SERVICE_NAME
 value: jenkins-jnlp
 - name: ENABLE_FATAL_ERROR_LOG_FILE
 value: "false"
 - name: JENKINS_UC_INSECURE
 value: "false"
 - name: JENKINS_HTTPS_KEYSTORE_PASSWORD
 valueFrom:
 secretKeyRef:
 key: https-jks-password
 name: jenkins-https-jks
 - name: JAVA_TOOL_OPTIONS
 value: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true
 -Djavax.net.ssl.trustStore=/var/jenkins_keystore/custom-keystore.jks -Djavax.net.ssl.trustStorePassword=$(JENKINS_HTTPS_KEYSTORE_PASSWORD)
 image: image-registry.OpenShift-image-registry.svc:5000/OpenShift/jenkins@sha256:dd5f1c5d14a8a72aa4ca51224c26a661c2e4f19ea3e5f9b7d8343f4952de5f0d
 imagePullPolicy: IfNotPresent
 livenessProbe:
 failureThreshold: 2
 httpGet:
 path: /login
 port: 8080
 scheme: HTTP
 initialDelaySeconds: 420
 periodSeconds: 360
 successThreshold: 1
 timeoutSeconds: 240
 name: jenkins
 readinessProbe:
 failureThreshold: 3
 httpGet:
 path: /login
 port: 8080
 scheme: HTTP
 initialDelaySeconds: 3
 periodSeconds: 10
 successThreshold: 1
 timeoutSeconds: 240
 resources:
 limits:
 memory: 1Gi
 securityContext:
 capabilities: {}
 privileged: false
 terminationMessagePath: /dev/termination-log
 terminationMessagePolicy: File
 volumeMounts:
 - mountPath: /var/lib/jenkins
 name: jenkins-data
 - mountPath: /var/jenkins_keystore
 name: jenkins-https-keystore
 dnsPolicy: ClusterFirst
 restartPolicy: Always
 schedulerName: default-scheduler
 securityContext: {}
 serviceAccount: jenkins
 serviceAccountName: jenkins
 terminationGracePeriodSeconds: 30
 volumes:
 - name: jenkins-data
 persistentVolumeClaim:
 claimName: jenkins
 - name: jenkins-https-keystore
 secret:
 defaultMode: 420
 items:
 - key: custom-keystore.jks
 path: custom-keystore.jks
 secretName: jenkins-https-jks
 test: false
 triggers: []
status: {}

Start a new deployment.

oc rollout latest jenkins

Once new pods are running, login will redirect to the Jenkins home page.

Measure Raspberry Pi Temperature with Telegraf, InfluxDB, and Grafana on K3s

Wed, 01 Jan 2020 00:00:00 +0000

In my previous post, I went through the k3s cluster home setup. Now, I’ll show how to measure the temperature of those Raspberry Pi’s using Telegraf, Influxdb, Grafana, and Helm charts.

Why Telegraf?

Telegraf has a plugin called exec, which can execute the commands on the host machine at a specific interval and parses those metrics from their output in any one of the accepted input data formats.

First, deploy the influxdb time series database chart.

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
 name: influxdb
 namespace: kube-system
spec:
 chart: stable/influxdb
 targetNamespace: monitoring

Get Pi temperature

I found this one-liner /sys/class/thermal/thermal_zone0/temp, which returns the temperature of the Pi; divide the output by 1000 to get a result in °C and use awk to have a float value.

awk '{print $1/1000}' /sys/class/thermal/thermal_zone0/temp

Update Chart values

Update chart values, add [inputs.exec] to config, and deploy it

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
 name: telegraf
 namespace: kube-system
spec:
 chart: stable/telegraf
 targetNamespace: monitoring
 valuesContent: |-
 replicaCount: 2
 image:
 repo: "telegraf"
 tag: "latest"
 pullPolicy: IfNotPresent
 env:
 - name: HOSTNAME
 valueFrom:
 fieldRef:
 fieldPath: spec.nodeName
 config:
 inputs:
 - exec:
 commands: ["awk '{print $1/1000}' /sys/class/thermal/thermal_zone0/temp"]
 name_override: "rpi_temp"
 data_format: "value"
 data_type: "float"

Add Datasource

Once influxdb and telegraf pods are ready, add influxdb Datasource in grafana.

Grafana

For Grafana visualization, import this dashboard.

K3s Cluster with Raspberry Pi and Traefik

Tue, 31 Dec 2019 00:00:00 +0000

In this post, I’ll share my home lab setup for Rancher’s k3s Kubernetes cluster.

Use-case:

A web interface to be accessible outside of my home so I could check and manage devices while away
Way to manage dynamic DNS since I don’t have a static I.P.

Setup

Setting up a master + single node Kubernetes cluster
Deploying DNS updater as a Kubernetes CronJob object
Deploying Traefik as a Kubernetes Ingress Controller and configuring it to manage SSL with Let’s Encrypt

Setting up a Pi Kubernetes Cluster

I followed an excellent guide by Alex Ellis here to initialize a cluster on the master and then join a single node.

k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
pi-node1 Ready <none> 3d v1.16.3-k3s.2
pi-master Ready master 3d v1.16.3-k3s.2

DNS and Routing

Add a DNS entry for the wildcard domain *.home.vikaspogu.com to point at the dynamic I.P.
Open ports 80 and 443 on the router’s firewall

At this point, a short dig on the domain should return your dynamic I.P.

$ dig +short test.home.vikaspogu.com
X.X.X.X.

I found this script online, which will update the DNS record if Dynamic I.P. is changed.

#!/bin/sh
zone=example.com
# dnsrecord is the A record which the script will update
dnsrecord=www.example.com
EMAIL=me@cloudflare.com
API_KEY=1234567890abcdef1234567890abcdef
# Get the current external I.P. address
ip=$(dig +short <CURRENT EXTERNAL IP>)
echo "Current IP is $ip"
if host $dnsrecord 1.1.1.1 | grep "has address" | grep "$ip"; then
 echo "$dnsrecord is currently set to $ip; no changes needed."
 exit
fi
# if here, the DNS record needs updating
# get the zone id for the requested zone
zoneid=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=$zone&status=active" \
 -H "X-Auth-Email: $EMAIL" \
 -H "X-Auth-Key: $API_KEY" \
 -H "Content-Type: application/json" | jq -r '{"result"}[] | .[0] | .id')
echo "Zoneid for $zone is $zoneid"
# get the DNS record id
dnsrecordid=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records?type=A&name=$dnsrecord" \
 -H "X-Auth-Email: $EMAIL" \
 -H "X-Auth-Key: $API_KEY" \
 -H "Content-Type: application/json" | jq -r '{"result"}[] | .[0] | .id')
echo "DNSrecordid for $dnsrecord is $dnsrecordid"
# update the record
curl -s -X PUT "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records/$dnsrecordid" \
 -H "X-Auth-Email: $EMAIL" \
 -H "X-Auth-Key: $API_KEY" \
 -H "Content-Type: application/json" \
 --data "{\"type\":\"A\",\"name\":\"$dnsrecord\",\"content\":\"$ip\",\"ttl\":1,\"proxied\":false}" | jq

Create a configmap from the script, secret with Cloudflare EMAIL and GLOBAL_API_TOKEN.

$ k3s create configmap update-script --from-file=cloudfare-dns-update.sh
$ k3s kubectl create secret generic cloudflare --from-literal=email=me@cloudflare.com \
--from-literal=api_key=1234567890abcdef1234567890abcdef

Now create a Kubernetes cronjob to update the DNS record to the correct address.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
 name: dns-update
 namespace: default
spec:
 schedule: "0 0 * * *"
 concurrencyPolicy: Forbid
 jobTemplate:
 spec:
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: dns-update
 image: e2eteam/dnsutils:1.1-linux-arm
 command: ["/bin/sh", "-c", "/scripts/cloudfare-dns-update.sh"]
 env:
 - name: API_KEY
 valueFrom:
 secretKeyRef:
 name: cloudflare
 key: api_key
 - name: EMAIL
 valueFrom:
 secretKeyRef:
 name: cloudflare
 key: email
 volumeMounts:
 - name: config-volume
 mountPath: /scripts
 volumes:
 - name: config-volume
 configMap:
 name: update-script
 defaultMode: 0744

Traefik and Let’s Encrypt

With a functioning cluster and the networking setup complete, the next task is to deploy a reverse proxy to manage the application routing.

In Kubernetes, we can deploy an Ingress Controller to achieve this. An Ingress Controller implements a reverse proxy that listens for changes to KubernetesIngress resources and updates its configuration accordingly.

Traefik provides detailed instructions on Kubernetes implementation, but I customized it slightly to get things working with my setup.

First, create RoleBinding.

k3s kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml

Configure Let’s Encrypt to support HTTPS endpoint and automatically fetch certificates. I used Cloudflare as the DNS provider, configuring Traefik to use DNS records for domain validation.

[acme]
 email = "me@cloudflare.com"
 storage="./acme.json"
 entryPoint = "https"
 acmeLogging=true
 [acme.dnsChallenge]
 provider = "cloudflare"
 delayBeforeCheck = 0
 [[acme.domains]]
 main = "*.home.vikaspogu.com"
 sans = ["home.vikaspogu.com"]

The Deployment objects look like this:

apiVersion: v1
data:
 traefik.toml: |
 # traefik.toml
 logLevel = "info"
 debug = true
 insecureSkipVerify = true
 defaultEntryPoints = ["http","https"]
 [entryPoints]
 [entryPoints.http]
 address = ":80"
 [entryPoints.http.redirect]
 entryPoint = "https"
 [entryPoints.https]
 address = ":443"
 [entryPoints.https.tls]
 [api]
 dashboard = true
 [kubernetes]
 [acme]
 email = "me@cloudflare.com"
 storage="./acme.json"
 entryPoint = "https"
 acmeLogging=true
 [acme.dnsChallenge]
 provider = "cloudflare"
 delayBeforeCheck = 0
 [[acme.domains]]
 main = "*.home.vikaspogu.com"
 sans = ["home.vikaspogu.com"]
kind: ConfigMap
metadata:
 labels:
 app: traefik
 name: traefik-config
 namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
 namespace: default
 name: traefik-ingress-controller
 labels:
 app: traefik
---
kind: Deployment
apiVersion: apps/v1
metadata:
 namespace: default
 name: traefik
 labels:
 app: traefik
spec:
 replicas: 1
 selector:
 matchLabels:
 app: traefik
 template:
 metadata:
 labels:
 app: traefik
 spec:
 serviceAccountName: traefik-ingress-controller
 containers:
 - name: traefik
 image: traefik:1.7.4
 ports:
 - name: http
 containerPort: 80
 - name: https
 containerPort: 443
 - name: admin
 containerPort: 8080
 env:
 - name: CF_API_EMAIL
 valueFrom:
 secretKeyRef:
 name: cloudflare
 key: email
 - name: CF_API_KEY
 valueFrom:
 secretKeyRef:
 name: cloudflare
 key: api_key
 volumeMounts:
 - mountPath: "/config"
 name: "config"
 args:
 - --configfile=/config/traefik.toml
 volumes:
 - name: config
 configMap:
 name: traefik-config
---
kind: Service
apiVersion: v1
metadata:
 name: traefik-ingress-service
 namespace: default
spec:
 selector:
 app: traefik
 ports:
 - protocol: TCP
 port: 80
 name: http
 - protocol: TCP
 port: 443
 name: https
 - protocol: TCP
 port: 8080
 name: admin
 externalIPs:
 - 192.168.0.101 # This is the node address

Deploy Ingress controller for traefik dashboard.

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: traefik-web-ui
 namespace: default
spec:
 rules:
 - host: traefik.home.vikaspogu.com
 http:
 paths:
 - path: /
 backend:
 serviceName: traefik-ingress-service
 servicePort: admin

Voila!

Deploy Kubernetes dashboard

Follow these instructions to deploy dashboard U.I..

Create an ingress controller to access the dashboard.

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: kubernetes-dashboard-ingress
 namespace: kubernetes-dashboard
spec:
 rules:
 - host: kubernetes-dashboard.home.vikaspogu.com
 http:
 paths:
 - backend:
 serviceName: kubernetes-dashboard
 servicePort: 443

Permission Denied Pushing to OpenShift Registry

Wed, 18 Dec 2019 00:00:00 +0000

Recently, I ran into an issue where pushing images to the docker registry after a build fails.

Pushing image docker-registry.default.svc:5000/simple-go-build/simple-go:latest ...
Registry server Address:
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: received unexpected HTTP status: 500 Internal Server Error

Registry pods logs show permission denied.

err.code=UNKNOWN err.detail="filesystem: mkdir /registry/docker/registry/v2/repositories/simple-go-build/simple-go/_uploads/c34415b4-c6d8-42ba-9854-aee449efd984: permission denied"

One of the Red Hat solutions articles suggested verifying the file ownership of the files and directories in the volume and comparing it to the uid of the registry.

Changing the owner recursively to the uid of the registry fixed the issue

root@master# chown -R 1001 /exports/registry/docker/

Basic Authentication in Go with Gin

Mon, 16 Dec 2019 00:00:00 +0000

This short post looks at adding basic authentication to GoLang applications. Below example application uses the gin web framework.

Let’s start by creating a gin router with default middleware. By default, it serves on :8080 unless we define a PORT environment variable.

func main(){
 r := gin.Default()
 r.GET("/getAllUsers", basicAuth, handlers.UsersList)
 _ = r.Run()
}

Now that we have our primary route let us create a method to add authentication logic. First, get basic auth credentials from the context request and validate them. The browser will prompt an authentication window for username and password details if the user is not authenticated.

func basicAuth(c *gin.Context) {
 // Get the Basic Authentication credentials
 user, password, hasAuth := c.Request.BasicAuth()
 if hasAuth && user == "testuser" && password == "testpass" {
 log.WithFields(log.Fields{
 "user": user,
 }).Info("User authenticated")
 } else {
 c.Abort()
 c.Writer.Header().Set("WWW-Authenticate", "Basic realm=Restricted")
 return
 }
}

Run application

go run main.go

Verify if authentication works

curl -X GET "http://testuser:testpass@localhost:8080/getAllUsers"

Create and Add NFS Storage to RHV

Sat, 02 Nov 2019 00:00:00 +0000

Set up NFS shares that will serve as storage domains on a Red Hat Enterprise Linux server.

Add a firewall rule to RHEL server.

firewall-cmd --permanent --add-service nfs
firewall-cmd --permanent --add-service mountd
firewall-cmd --reload

Create and add group kvm; set the ownership of your exported directories to 36:36, which gives vdsm:kvm ownership; change the director’s permission so that read and write access is granted to the owner.

groupadd kvm -g 36
useradd vdsm -u 36 -g 36
mkdir -pv /home/rhv-data-vol
chown -R 36:36 /home/rhv-data-vol
chmod 0755 /home/rhv-data-vol

Add newly created directory to /etc/exports file, which controls file systems exported to remote hosts and specifies options.

echo "/home/rhv-data-vol 192.168.10.1(rw,sync)" >> /etc/exports
exportfs -av

Start, enable NFS server; checklist of export server using showmount

systemctl start nfs
systemctl enable nfs
systemctl status nfs
showmount -e

Configuring Couchbase SSL for Dynamic Certificates in OpenShift

Wed, 28 Aug 2019 00:00:00 +0000

Couchbase SSL

Suppose you have followed dynamic creation of java keystores in OpenShift post and wondered how to use similar concepts for couchbase database and a java application. This post will help you.

Couchbase setup

Here is the couchbase documentation for configuring server-side certificates, we are interested in last few steps since OpenShift will generate key and cert by adding an annotation to the couchbase service.

Note: By adding this annotation, you can dynamically create certificates service.alpha.OpenShift.io/serving-cert-secret-name: couchbase-db-certs

couchbase service looks like this:

apiVersion: v1
kind: Service
metadata:
 annotations:
 kubectl.kubernetes.io/last-applied-configuration: ""
 service.alpha.OpenShift.io/serving-cert-secret-name: couchbase-db-certs
 labels:
 component: couchbase-db
 name: couchbase-db
 namespace: myproject
spec:
 ports:
 - name: consolerest
 port: 8091
 protocol: TCP
 targetPort: 8091

To convert certificates as couchbase expects, we will use an init container.

We will use an emptyDir volume to store the cert and key in the /opt/couchbase/var/lib/couchbase/inbox/ location so that couchbase can access them.

Init container will run a sequence of commands to split certificate, place them into /opt/couchbase/var/lib/couchbase/inbox/ location, and name cert file as chain.pem, key as pkey.key.

Init container will look as follows:

initContainers:
 - args:
 - "-c"
 - >-
 csplit -z -f crt- $crtfile '/-----BEGIN CERTIFICATE-----/' '{*}'
 && for file in crt-*; do cat $file >
 /opt/couchbase/var/lib/couchbase/inbox/service-$file; done && cat
 $crtfile > /opt/couchbase/var/lib/couchbase/inbox/chain.pem && cat
 $keyfile > /opt/couchbase/var/lib/couchbase/inbox/pkey.key
 command:
 - /bin/bash
 env:
 - name: keyfile
 value: /var/run/secrets/OpenShift.io/services_serving_certs/tls.key
 - name: crtfile
 value: /var/run/secrets/OpenShift.io/services_serving_certs/tls.crt
 - name: password
 value: changeit
 image: registry.access.redhat.com/redhat-sso-7/sso72-OpenShift:latest
 imagePullPolicy: Always
 name: couchbase-ssl
 volumeMounts:
 - mountPath: /var/run/secrets/OpenShift.io/services_serving_certs
 name: couchbase-db-certs
 - mountPath: /opt/couchbase/var/lib/couchbase/inbox/
 name: couchbase-ssl-volume
volumes:
 - emptyDir: {}
 name: couchbase-ssl-volume
 - name: couchbase-db-certs
 secret:
 defaultMode: 420
 secretName: couchbase-db-certs

Next, add couchbase-ssl-volume emptyDir volume mount to the actual container so the file can be accessed by couchbase.

spec:
 containers:
 - env:
 ...
 volumeMounts:
 - mountPath: /opt/couchbase/var/lib/couchbase/inbox/
 name: couchbase-ssl-volume

I am using a rhel7-couchbase image; on startup, it runs an initialization script to set up the cluster; at that time, we will upload the certificate and activate it using these commands.

couchbase-cli ssl-manage -c http://localhost:8091 -u Administrator \
-p password --upload-cluster-ca=${SERVICE_CERT}
couchbase-cli ssl-manage -c http://localhost:8091 -u Administrator \
-p password --set-node-certificate

Pass the cert location as environment variable SERVICE_CERT in deployment config.

- env:
 - name: SERVICE_CERT
 value: /opt/couchbase/var/lib/couchbase/inbox/service-crt-01

Verify logs on the container

SUCCESS: Uploaded cluster certificate to http://localhost:8091
SUCCESS: Node certificate set

We can also verify in couchbase UI

At this point, we completed the couchbase setup.

Application setup

We will be using same steps as SSL client from dynamically-creating-java-keystores-OpenShift post

To make a secure connection to the couchbase, it will need the trust store generated by the pem-to-truststore initContainer. Here is the client’s app deployment config:

- apiVersion: v1
 kind: DeploymentConfig
 metadata:
 labels:
 app: ssl-client
 name: ssl-client
 spec:
 replicas: 1
 selector:
 deploymentconfig: ssl-client
 template:
 metadata:
 labels:
 app: ssl-client
 deploymentconfig: ssl-client
 spec:
 containers:
 - name: ssl-client
 image: ssl-client
 imagePullPolicy: Always
 env:
 - name: JAVA_OPTIONS
 value: -Djavax.net.ssl.trustStore=/var/run/secrets/java.io/keystores/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit
 - name: POD_NAMESPACE
 valueFrom:
 fieldRef:
 apiVersion: v1
 fieldPath: metadata.namespace
 volumeMounts:
 - mountPath: /var/run/secrets/java.io/keystores
 name: keystore-volume
 initContainers:
 - name: pem-to-truststore
 image: registry.access.redhat.com/redhat-sso-7/sso71-OpenShift:1.1-16
 env:
 - name: ca_bundle
 value: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
 - name: truststore_jks
 value: /var/run/secrets/java.io/keystores/truststore.jks
 - name: password
 value: changeit
 command: ["/bin/bash"]
 args:
 [
 "-c",
 "csplit -z -f crt- $ca_bundle '/-----BEGIN CERTIFICATE-----/' '{*}' && for file in crt-*; do keytool -import -noprompt -keystore $truststore_jks -file $file -storepass changeit -alias service-$file; done",
 ]
 volumeMounts:
 - mountPath: /var/run/secrets/java.io/keystores
 name: keystore-volume
 volumes:
 - emtpyDir: {}
 name: keystore-volume

The next step is to enable encryption and pass the path and password of the truststore generated by the initContainer

CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder().sslEnabled(true)
 .sslTruststoreFile("/var/run/secrets/java.io/keystores/truststore.jks")
 .sslTruststorePassword("changeit").build();
cachedCluster = CouchbaseCluster.create(env, "couchbase-db")
 .authenticate("Administrator", "password");

Deploy your application; if successful, you should see similar output in container logs

2019-08-28 15:33:27.952 INFO 1 --- [cTaskExecutor-1]
com.couchbase.client.core.CouchbaseCore : CouchbaseEnvironment:
{sslEnabled=true, sslKeystoreFile='null', sslTruststoreFile='/var/run/secrets/java.io/keystores/truststore.jks',
sslKeystorePassword=false, sslTruststorePassword=true,
sslKeystore=null, sslTruststore=null, bootstrapHttpEnabled=true,
bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091,
...

TektonCD on OpenShift

Fri, 09 Aug 2019 00:00:00 +0000

Note: This post uses the deprecated v1alpha1 API and PipelineResources. Modern Tekton uses v1 API with Workspaces instead of PipelineResources. See the Tekton documentation for current best practices.

Recently I came across tektoncd project, The Tekton Pipelines project provides Kubernetes-style resources for declaring CI/CD-style pipelines caught my attention, and I started playing with it.

Basic Concepts

To create a Tekton pipeline, one does the following:

Create custom or install existing reusable Tasks
Create a Pipeline and PipelineResources to define your application’s delivery pipeline
Create a PipelineRun to instantiate and invoke the pipeline

Installing Tekton on OpenShift

oc new-project tekton-pipelines
oc adm policy add-scc-to-user anyuid -z tekton-pipelines-controller
oc apply --filename https://storage.googleapis.com/tekton-releases/latest/release.yaml

Install dashboard

The dashboard is a general-purpose, web-based UI for Tekton Pipelines. Tekton also has cli client

curl -L https://github.com/tektoncd/dashboard/releases/download/v0/gcr-tekton-dashboard.yaml | oc apply -f -

Expose tekton-dashboard service as a route

oc expose service tekton-dashboard \
 -n tekton-pipelines \
 --name "tekton-dashboard" \
 --port="http"

Create pipeline

This pipeline builds an image from the source and starts a new rollout.

Create a new project

oc new-project tektontutorial

Create a service account for running pipelines and enable it to run privileged pods for building images

oc create serviceaccount pipeline
oc adm policy add-scc-to-user privileged -z pipeline
oc adm policy add-role-to-user edit -z pipeline

Optionally, create OpenShift objects(i.e DeploymentConfig, ImageStream, Service, Route)

---
apiVersion: image.OpenShift.io/v1
kind: ImageStream
metadata:
 labels:
 app: go-sample
 name: go-sample
---
apiVersion: apps.OpenShift.io/v1
kind: DeploymentConfig
metadata:
 labels:
 app: go-sample
 name: go-sample
spec:
 replicas: 1
 revisionHistoryLimit: 10
 selector:
 app: go-sample
 deploymentconfig: go-sample
 strategy:
 activeDeadlineSeconds: 21600
 resources: {}
 rollingParams:
 intervalSeconds: 1
 maxSurge: 25%
 maxUnavailable: 25%
 timeoutSeconds: 600
 updatePeriodSeconds: 1
 type: Rolling
 template:
 metadata:
 labels:
 app: go-sample
 deploymentconfig: go-sample
 spec:
 containers:
 - image: go-sample:latest
 imagePullPolicy: Always
 livenessProbe:
 failureThreshold: 3
 httpGet:
 path: /
 port: 8080
 scheme: HTTP
 initialDelaySeconds: 45
 periodSeconds: 10
 successThreshold: 1
 timeoutSeconds: 1
 name: go-sample
 ports:
 - containerPort: 8080
 protocol: TCP
 readinessProbe:
 failureThreshold: 3
 httpGet:
 path: /
 port: 8080
 scheme: HTTP
 initialDelaySeconds: 45
 periodSeconds: 10
 successThreshold: 1
 timeoutSeconds: 5
 resources: {}
 terminationMessagePath: /dev/termination-log
 terminationMessagePolicy: File
 dnsPolicy: ClusterFirst
 restartPolicy: Always
 schedulerName: default-scheduler
 securityContext: {}
 terminationGracePeriodSeconds: 30
 test: false
 triggers:
 - imageChangeParams:
 containerNames:
 - go-sample
 from:
 kind: ImageStreamTag
 name: go-sample:latest
 namespace: tektontutorial
 type: ImageChange
 - type: ConfigChange
---
apiVersion: v1
kind: Service
metadata:
 labels:
 app: go-sample
 name: go-sample
spec:
 ports:
 - name: 8080-tcp
 port: 8080
 protocol: TCP
 targetPort: 8080
 selector:
 app: go-sample
 deploymentconfig: go-sample
 sessionAffinity: None
 type: ClusterIP
---
apiVersion: route.OpenShift.io/v1
kind: Route
metadata:
 labels:
 app: go-sample
 name: go-sample
spec:
 port:
 targetPort: 8080-tcp
 to:
 kind: Service
 name: go-sample
 weight: 100

oc apply -f go-sample-template.yml

The deployment will not be complete since there are no container images for the go-sample app.

Create a s2i-go, OpenShift-cli task. You can find more examples of reusable tasks in the Tekton Catalog and OpenShift Catalog repositories.

Note: Tasks consist of several steps that get executed sequentially. The pipeline will perform each task in a separate container within the same pod.

oc apply -f https://raw.githubusercontent.com/OpenShift/pipelines-catalog/master/s2i-go/s2i-go-task.yaml
oc create -f https://raw.githubusercontent.com/tektoncd/catalog/master/OpenShift-client/OpenShift-client-task.yaml

Create pipeline resources

---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
 name: go-sample-image
spec:
 type: image
 params:
 - name: url
 value: image-registry.OpenShift-image-registry.svc:5000/tektontutorial/go-sample
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
 name: go-sample-git
spec:
 type: git
 params:
 - name: url
 value: https://github.com/go-training/helloworld

oc apply -f resources.yml

Create pipeline; pipelines have two tasks here build and deploy. Build uses s2i-go task to create an image from source and deploy uses OpenShift-client task to rollout latest deployment of go-sample

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
 name: go-sample-pipeline
spec:
 resources:
 - name: app-git
 type: git
 - name: app-image
 type: image
 tasks:
 - name: build
 taskRef:
 name: s2i-go
 kind: Task
 params:
 - name: TLSVERIFY
 value: "false"
 resources:
 inputs:
 - name: source
 resource: app-git
 outputs:
 - name: image
 resource: app-image
 - name: deploy
 taskRef:
 name: OpenShift-client
 kind: ClusterTask
 runAfter:
 - build
 params:
 - name: ARGS
 value: "rollout latest go-sample"

oc apply -f go-sample-pipeline.yml

Start pipeline on the dashboard or by tektoncli

tkn pipeline start go-sample-pipeline \
 -r app-git=go-sample-git -r app-image=go-sample-image \
 -s pipeline

Click on Create PipelineRun, select pipeline, resources, and service account and start the pipeline.

Once finished, you should see all tasks marked in green.

The go-sample deployment should have one running pod.

Go JWT Authentication with Keycloak

Sat, 27 Jul 2019 00:00:00 +0000

I recently worked on a React project with Go backend using Gin web framework. Keycloak was the authentication mechanism for the front end; I also wanted to secure the back end using JSON Web Tokens, which Keycloak provided on every login. JWT verification setup in the Go application was easy.

First, copy the RS256 algorithm public key value from Keycloak.

Send the token as an Authorization header.

axios
 .get(BACKEND_URL.concat("sampleendpoint"), {
 headers: {
 Authorization: this.state.token
 }
 })
 .then(res => {});

Now Go-backend setup; let’s install the jwt-go, gin-cors libraries:

$ go get -u github.com/dgrijalva/jwt-go
$ go get -u github.com/gin-contrib/cors

Add cors config to the router to allow the authorization header.

router.Use(cors.New(cors.Config{
 AllowOrigins: []string{"*"},
 AllowMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTIONS", "HEAD"},
 AllowHeaders: []string{"Origin", "content-type", "accept", "authorization"},
 ExposeHeaders: []string{"Content-Length"},
 AllowCredentials: true,
 MaxAge: 12 * time.Hour,
}))

Let’s create a custom handler; add the public key from Keycloak and pass it to ParseRSAPublicKeyFromPEM, which will return a key. The key and token are then validated.

func VerifyToken(c *gin.Context) {
 SecretKey := "-----BEGIN CERTIFICATE-----\n"+
 "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApn
 ......
 +wnyuCHaCHp8P1yCnwIDAQAB" + "\n-----END CERTIFICATE-----"

 reqToken := c.GetHeader("Authorization")

 key, er := jwt.ParseRSAPublicKeyFromPEM([]byte(SecretKey))
 if er != nil {
 fmt.Println(er)
 c.Abort()
 c.Writer.WriteHeader(http.StatusUnauthorized)
 c.Writer.Write([]byte("Unauthorized"))
 return
 }

 token, err := jwt.Parse(reqToken, func(token *jwt.Token) (interface{}, error) {
 // Don't forget to validate the alg is what you expect:
 if _, ok := token.Method.(*jwt.SigningMethodRSA); !ok {
 return nil, fmt.Errorf("Unexpected signing method: %v", token.Header["alg"])
 }
 return key, nil
 })

 if err != nil {
 fmt.Println(err)
 c.Abort()
 c.Writer.WriteHeader(http.StatusUnauthorized)
 c.Writer.Write([]byte("Unauthorized"))
 return
 }

 if _, ok := token.Claims.(jwt.MapClaims); ok && token.Valid {
 fmt.Println("token is valid")
 }
}

Add the handler to the route.

router.GET("/sample", VerifyToken(), handlers.SampleEndpoint)

That’s it; if the token is valid, you will get the data from the backend, or else you’ll see 401 Unauthorized.

React App with Red Hat SSO or Keycloak

Thu, 25 Jul 2019 00:00:00 +0000

Note: This post uses React class components. Modern React applications should use functional components with hooks. Consider using @react-keycloak/web for a hooks-based approach.

This post will show you how to secure a React app using Red Hat SSO (upstream Keycloak). In this case, OpenID-connect is my identity provider.

Install the official Keycloak js adapter

npm i keycloak-js --save

Add host and port information to the client; in my case, it’s localhost:9000

In App.js, add a JavaScript object with the required configuration; you will find these configurations under Clients->Installation.

//keycloak init options
const initOptions = {
 url: "https://localhost:8080/auth",
 realm: "test",
 clientId: "react-app",
 onLoad: "login-required"
};

By default, to authenticate, you need to call the login function. However, there are two options available to make the adapter automatically authenticate. First, you can pass login-required or check-sso to the init function. login-required will authenticate the client if the user is logged-in to {project_name} or display the login page if not. check-sso will only authenticate the client if the user is already logged-in; if the user is not logged in, the browser will be redirected back to the application and remain unauthenticated.

componentDidMount() {
 let keycloak = Keycloak(initOptions);
 keycloak.init({ onLoad: initOptions.onLoad }).success(authenticated => {});
}

Finally

import * as Keycloak from "keycloak-js";

//keycloak init options
const initOptions = {
 url: "https://localhost:8080/auth",
 realm: "test",
 clientId: "react-app",
 onLoad: "login-required"
};

class App extends React.Component {
 constructor(props) {
 super(props);
 this.state = {
 keycloak: null,
 authenticated: false
 };
 }
 componentDidMount() {
 let keycloak = Keycloak(initOptions);
 keycloak.init({ onLoad: initOptions.onLoad }).success(authenticated => {
 this.setState({ keycloak: keycloak, authenticated: authenticated });
 });
 }
 render() {
 const { keycloak, authenticated } = this.state;
 if (keycloak) {
 if (authenticated) {
 return (
 <React.Fragment>
 <AppRouter />
 </React.Fragment>
 );
 }
 }
 return <div>Initializing SS0...</div>;
 }
}

Run your app

$ npm run start

Navigate to http://localhost:9000; you should see the login page if the user is not authenticated.

Heketi JWT token expired error

Tue, 16 Jul 2019 00:00:00 +0000

Note: This post is archived. Heketi and GlusterFS have been deprecated in favor of modern storage solutions like Rook-Ceph, OpenEBS, or cloud-native storage providers.

Recently I encountered a JWT token expired error on the heketi pod in the OpenShift cluster.

[jwt] ERROR 2019/07/16 19:17:14 heketi/middleware/jwt.go:66:middleware.(*HeketiJwtClaims).Valid: exp validation failed: Token is expired by 1h48m59s

After a lot of google searches, I synchronized clocks across the pod running heketi and the master nodes, which solved the issue

ntpdate -q 0.rhel.pool.ntp.org; systemctl restart ntpd

PatternFly Setup in React Application

Mon, 01 Jul 2019 00:00:00 +0000

Note: This post references PatternFly v4. PatternFly is now at v5+ with significant API changes. See the PatternFly documentation for current installation and usage guides.

To integrate PatternFly framework into a React application, create a new project or use an existing one

npx create-react-app patternfly-setup-react

Install patternfly dependencies react-core, react-table and patternfly

npm i --save @patternfly/patternfly \
 @patternfly/react-core @patternfly/react-table

Note: Import base.css and patternfly.css in your project, or some components may diverge in appearance

//These imports are a must to render CSS
import "@patternfly/react-core/dist/styles/base.css";
import "@patternfly/patternfly/patternfly.css";

To make sure everything is working correctly, update App.js with a demo layout from documentation

import React from "react";
import {
 Avatar,
 Brand,
 Button,
 ButtonVariant
 ...
 ...
} from "@patternfly/react-core";
import accessibleStyles from "@patternfly/react-styles/css/utilities/Accessibility/accessibility";
import spacingStyles from "@patternfly/react-styles/css/utilities/Spacing/spacing";
import { css } from "@patternfly/react-styles";
import { BellIcon, CogIcon } from "@patternfly/react-icons";

//These imports are a must to render CSS
import "@patternfly/react-core/dist/styles/base.css";
import "@patternfly/patternfly/patternfly.css";

class App extends React.Component {
...
...
 return (
 <React.Fragment>
 <Page
 header={Header}
 sidebar={Sidebar}
 isManagedSidebar
 skipToContent={PageSkipToContent}
 >
 <PageSection variant={PageSectionVariants.light}>
 <TextContent>
 <Text component="h1">Main Title</Text>
 <Text component="p">
 Body text should be Overpass Regular at 16px. It should have leading of 24px because <br />
 of its relative line height of 1.5.
 </Text>
 </TextContent>
 </PageSection>
 <PageSection>
 <Gallery gutter="md">
 {Array.apply(0, Array(10)).map((x, i) => (
 <GalleryItem key={i}>
 <Card>
 <CardBody>This is a card</CardBody>
 </Card>
 </GalleryItem>
 ))}
 </Gallery>
 </PageSection>
 </Page>
 </React.Fragment>
 );
 }
}
export default App;

Start the application

npm start

You should see a patternfly design like this!

Authenticate a Node.js Application with LDAP

Wed, 15 May 2019 00:00:00 +0000

This post demonstrates how to authenticate a user against LDAP.

Let’s start by installing basic-auth and ldapauth-fork packages

npm install ldapauth-fork
npm install basic-auth

Steps for implementation;

Add packages
Create an LDAP variable with authentication configuration
Basic auth should prompt for your username and password. Once the user is found, verify the given password by trying to bind the user client with the found LDAP user object and the given password.

const auth = require("basic-auth");
var LdapAuth = require("ldapauth-fork");

var ldap = new LdapAuth({
 url: "ldap://ldap-url:389",
 bindDN: "uid=rc,ou=AppAccounts,ou=People,ou=Entsys,dc=example.com",
 bindCredentials: "credentials",
 searchBase: "ou=entsys,dc=example.com",
 searchFilter: "(uid={{username}})",
 reconnect: true
});

app.use("/API/admin/", (req, res, next) => {
 const credentials = auth(req);
 if (credentials) {
 LDAP.authenticate(credentials.name, credentials.pass, function(err, user) {
 if (err) {
 console.log(err.message);
 return res
 .status("401")
 .set({ "WWW-Authenticate": 'Basic realm="Access Denied"' })
 .end("access denied");
 }
 req.user = user;
 next();
 });
 } else {
 return res
 .status("401")
 .set({ "WWW-Authenticate": 'Basic realm="Access Denied"' })
 .end("access denied");
 }
});

Visit basic-auth, ldapauth-fork packages for more information on configuration.

Deleting an OpenShift Project Stuck in Terminating State

Wed, 15 May 2019 00:00:00 +0000

Recently I faced an issue where one of my projects got stuck in a terminating state for days. The workaround below fixed the problem.

Export OpenShift project as a JSON Object

oc get project delete-me -o json > ns-without-finalizers.json

Replace below from

spec:
 finalizers:
 - kubernetes

spec:
 finalizers: []

On one of the master nodes, execute these commands.

kubectl proxy & PID=$!
curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize \
-H "Content-Type: application/json" --data-binary @ns-without-finalizers.json
kill $PID

Spring Boot Metrics with Prometheus and Grafana in OpenShift

Wed, 15 May 2019 00:00:00 +0000

Spring Boot Metrics

This post will discuss how to monitor spring boot application metrics using Prometheus and Grafana.

Prometheus

Prometheus is a monitoring system that collects metrics from configured targets at intervals.

Grafana

Grafana is an open-source metric analytics & visualization tool.

Micrometer

The micrometer is a metrics instrumentation library for JVM-based applications.

Spring Boot Actuator

Spring Boot Actuator helps you monitor and manage your application when it’s pushed to production. You can control and monitor your application using HTTP or JMX endpoints.

Setup

Enable Prometheus metrics by adding dependencies in pom.xml

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
 <groupId>io.micrometer</groupId>
 <artifactId>micrometer-core</artifactId>
 <version>1.1.4</version>
</dependency>
<dependency>
 <groupId>io.micrometer</groupId>
 <artifactId>micrometer-registry-prometheus</artifactId>
 <version>1.0.6</version>
</dependency>

By default Prometheus endpoint is not available and must be enabled in application.properties. You can find more configurations at spring-boot docs

#Metrics related configurations
management.endpoint.metrics.enabled=true
management.endpoints.web.exposure.include=*
management.endpoint.prometheus.enabled=true
management.metrics.export.prometheus.enabled=true
management.metrics.distribution.percentiles-histogram.http.server.requests=true
management.metrics.distribution.sla.http.server.requests=1ms,5ms
management.metrics.distribution.percentiles.http.server.requests=0.5,0.9,0.95,0.99,0.999

Optionally you can configure any number with the MeterRegistryCustomizer registry (such as applying common tags).

@Bean
MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
 return registry -> registry.config().commonTags("application", "sample-app");
}

Create a new project; deploy the application and Prometheus in OpenShift.

$ oc project myproject
$ oc new-app redhat-openjdk18-OpenShift~<git_repo_URL> -n sample-app
oc new-app prom/prometheus -n prometheus

To keep the Prometheus image and configuration decoupled, use the ConfigMap object to inject the Prometheus deployment with the appropriate configuration data.

cat <<'EOF' > prometheus.yml
global:
 scrape_interval: 5s
 evaluation_interval: 5s

scrape_configs:
 - job_name: 'sample-app'
 metrics_path: '/actuator/prometheus'
 static_configs:
 - targets: ['sample-app:8080']

EOF

oc create configmap prom-config-example --from-file=prometheus.yml

Next, edit the deployment configuration for Prometheus to include this ConfigMap.

oc edit dc/prometheus

Add new volume and volume mount.

- name: prom-config-example-volume
 configMap:
 name: prom-config-example
 defaultMode: 420

- name: prom-config-example-volume
 mountPath: /etc/prometheus/

Use an OpenShift Template to run Grafana with persistent storage.

$ oc process -f https://gist.githubusercontent.com/Vikaspogu/4a67495acf8dba5dc94837e031129fde/raw/e88f42515c6ed101c9554c7c2425794e80e10a64/OpenShift-grafana.yaml | oc apply -f-

Once deployed, log in to Grafana using the Route provided in the Template and using the default account admin with password admin (it may be a good idea to change the password after this).

Grafana Data Source

The Grafana template automatically provisions a Prometheus data source, App-Prometheus, which connects to http://prometheus:9090 via a proxy connection.
This works if there is a Prometheus service (called Prometheus) in the same project as Grafana. If this is not the case, it is necessary to edit the data source to point to the appropriate location.

Grafana Dashboard

The Grafana template automatically provisions sample dashboards. These dashboards are not comprehensive, but you can use them as a starting point for further customization.

You can find more official & community built grafana dashboards here

Debugging a .NET Core Application on OpenShift

Tue, 14 May 2019 00:00:00 +0000

This post concerns remote debugging an ASP.NET Core application on OpenShift using Visual Studio Code. You can use any Microsoft proprietary debugger engine vsdbg with Visual Studio Code.

First, list the available .Net application pods using the oc command.

$ oc get pod
 NAME READY STATUS RESTARTS AGE
 MY_APP_NAME-3-1xrsp 0/1 Running 0 6s
$ oc rsh MY_APP_NAME-3-1xrsp
sh-4.2$ curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l /opt/app-root/vsdbg -r linux-x64

Note: If your container is running behind a corporate proxy and cannot access the internet, you’ll have to build a base dotnet image with the installed debugger engine vsdbg.

Create (or open) the .vscode/launch.json file inside the source directory of the application (i.e., at the same level as the project folder), then add the following:

{
 "version": "0.1.0",
 "configurations": [
 {
 "name": ".NET Core OpenShift Pod Remote Attach",
 "type": "coreclr",
 "request": "attach",
 "processId": "1",
 "pipeTransport": {
 "pipeProgram": "oc",
 "pipeArgs": ["exec", "-it", "<replace-with-pod-name>", "--"],
 "quoteArgs": false,
 "debuggerPath": "/opt/app-root/vsdbg/vsdbg",
 "pipeCwd": "${workspaceRoot}"
 },
 "justMyCode": false,
 "sourceFileMap": { "/opt/app-root/src": "${workspaceRoot}" }
 }
 ]
}

In Launch.json, replace <replace-with-pod-name> with the pod’s name.

Confirm the PID of the dotnet process in the container. If different, replace the processId in launch.json with the appropriate value. Usually, this value is 1.

If the application is built with the Release configuration, the default for .NET Core S2I builder images, justMyCode should be false.

As S2I images build the source code in the /opt/app-root/src folder, we should specify this path for sourceFileMap.

Start debugging the .NET Core app by switching to the debug window, then select .NET Core OpenShift Pod Remote Attach as the configuration and click on the green play button (as shown below), or press the F5 key.

Debug .NET CORE

Remember to add the breakpoints in the source code to see debugging in action!

Debugging a Java Application in OpenShift

Tue, 14 May 2019 00:00:00 +0000

This post will discuss debugging a JAVA application running inside a container.

Red Hat container images

When you bootstrap your JVM, you should have a way to enable JVM to debug. For example, Red Hat S2I images allow you to control classpath and debugging via environment variables.

# Set debug options if required
if [ x"${JAVA_DEBUG}" != x ] && [ "${JAVA_DEBUG}" != "false" ]; then
 java_debug_args="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${JAVA_DEBUG_PORT:-5005}"
fi

Setting the JAVA_DEBUG environment variable inside the container to true will append debug args to the JVM startup command
Configure port forwarding so that you can connect to your application from a remote debugger

If you are using the tomcat image, replace the JAVA_DEBUG environment variable with DEBUG

Using the oc command, list the available deployment configurations:

oc get dc

Enable Debug

Set the JAVA_DEBUG environment variable in the deployment configuration of your application to true, which configures the JVM to open the port number 5005 for debugging.

oc set env dc/MY_APP_NAME JAVA_DEBUG=true

Disabling the health checks is not mandatory but recommended because a pod could be restarted during remote debugging while the process is paused. You can remove the readiness check to prevent a forced restart.

Redeploy

Redeploy the application if it is not set to redeploy automatically on configuration change.

oc rollout latest dc/MY_APP_NAME

Configure port forwarding from your local machine to the application pod. List the currently running pods and find one containing your application. $LOCAL_PORT_NUMBER is an unused port number of your choice on your local machine. Remember this number for the remote debugger configuration.

If you are using the tomcat image, replace the port 5005 with 8000

oc get pod
NAME READY STATUS RESTARTS AGE
MY_APP_NAME-3-1xrsp 1/1 Running 0 6s
...
oc port-forward MY_APP_NAME-3-1xrsp $LOCAL_PORT_NUMBER:5005

IntelliJ Config

Create a new debug configuration for your application in `IntelliJ IDE:

Click Run → Edit Configurations
In the list of configurations, add Remote. Creates a new remote debugging configuration
Enter a suitable name for the configuration in the name field
Set the port field to the port number that your application is listening on for debugging
Click Apply
Click Run -> Debug -> Select Profile

When done debugging, unset the JAVA_DEBUG environment variable in your application pod.

oc set env dc/MY_APP_NAME JAVA_DEBUG-

Non-Red Hat container images

If you are using the OpenJDK image to build an application, update ENTRYPOINT as below to pass options to the JVM through the $JAVA_OPTS environment variable

FROM openjdk:11.0.3-jdk-slim
RUN mkdir /usr/myapp
COPY target/java-kubernetes.jar /usr/myapp/app.jar
WORKDIR /usr/myapp
EXPOSE 8080
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -jar app.jar" ]

And then set deployments JAVA_OPTS environment variable.

oc set env deployment MY_APP_NAME JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,address=*:5005,server=y,suspend=n

Profiling a Java Application in OpenShift Container

Tue, 14 May 2019 00:00:00 +0000

Sometimes writing code that runs is not enough. We might want to know what goes on internally, such as memory allocation, consequences of using one coding approach over another, implications of concurrent executions, areas to improve performance, etc. We can use profilers for this.

In this post, I’ll discuss using YourKit-JavaProfiler inside a container.

Since my sample application is built using OpenShift S2I process and pushed into OpenShift internal registry, I’ll have to pull the image locally.

docker login -p $(oc whoami --show-token) -u admin docker-registry.example.com
docker pull docker-registry.example.com/myproject/sample-app:latest

Create a new Dockerfile, add a few lines to install YourKit Java Profiler agents, and expose the profiler agent port.

FROM docker-registry.example.com/myproject/sample-app:latest
RUN wget https://www.yourkit.com/download/docker/YourKit-JavaProfiler-2019.1-docker.zip -P /tmp/ && \
 unzip /tmp/YourKit-JavaProfiler-2019.1-docker.zip -d /usr/local && \
 rm /tmp/YourKit-JavaProfiler-2019.1-docker.zip
EXPOSE 10001

Build and push the image into the registry

docker build . -t docker-registry.example.com/myproject/sample-app-profiler:latest
docker push docker-registry.example.com/myproject/sample-app-profiler:latest

Update the image in the deployment configuration.

Load the agent into the JVM by adding a JAVA_TOOL_OPTIONS environment variable in the deployment configuration.

$ oc set env dc/MY_APP_NAME JAVA_TOOL_OPTIONS=-agentpath:/usr/local/YourKit-JavaProfiler-2019.01/bin/linux-x86-64/libyjpagent.so=port=10001,listen=all
oc rollout latest dc/MY_APP_NAME

Once the deployment is completed, run oc port forwarding from your local machine to the application pod.

$ oc get pod
NAME READY STATUS RESTARTS AGE
MY_APP_NAME-3-1xrsp 1/1 Running 0 6s
...
$ oc port-forward MY_APP_NAME-3-1xrsp 10001:10001

Add a connection in the profiler with localhost:10001, and you’re all set.