Catégorie : Kubernetes (AKS)

Planned Maintenance Windows in AKS

Hi!

Azure Kubernetes Service (AKS) now supports planned maintenance windows. This allows you to specify planned maintenance windows so you can control when weekly releases or proactive maintenance events may be performed in your clusters and minimize interruptions during hours when you need your clusters to be available.

When using Planned Maintenance, the following restrictions apply:

  • AKS reserves the right to break these windows for unplanned/reactive maintenance operations that are urgent or critical.
  • Currently, performing maintenance operations are considered best-effort only and are not guaranteed to occur within a specified window.
  • Updates cannot be blocked for more than seven days.

Allow maintenance on every Monday at 1:00am to 2:00am

az aks maintenanceconfiguration add -g MyResourceGroup --cluster-name myAKSCluster --name default --weekday Monday --start-hour 1

Update an existing maintenance window

az aks maintenanceconfiguration update -g MyResourceGroup --cluster-name myAKSCluster --name default --weekday Monday --start-hour 1

List all maintenance windows in an existing cluster

az aks maintenanceconfiguration list -g MyResourceGroup --cluster-name myAKSCluster

Show a specific maintenance configuration window in an AKS cluster

az aks maintenanceconfiguration show -g MyResourceGroup --cluster-name myAKSCluster --name default

Delete a certain maintenance configuration window in an existing AKS Cluster

az aks maintenanceconfiguration delete -g MyResourceGroup --cluster-name myAKSCluster --name default

Maxime.

AKS | OPA Gatekeeper Dashboard

Hi!

In a previous article, I show you how you can deploy a OPA Gatekeeper solution in your AKS cluster. We saw together how we can monitor the number of OPA gatekeeper violation in a second article.

In this article I will show how you can deploy a dashboard to monitor your OPA Gatekeeper violations. I will recommend you to use the solution Gatekeeper Policy Manager (GPM) created by Sighupio.

It’s very easy to deploy this solution, please clone the following repository and run this command to deploy the solution:

kubectl apply -k .

Once you’ve deployed the application, if you haven’t set up an ingress, you can access the web-UI using port-forward:

kubectl -n gatekeeper-system port-forward  svc/gatekeeper-policy-manager 8080:80

For a production usage of this solution, I recommend you to configure the OIDC authentication.

Maxime.

AKS | OPA Gatekeeper Monitoring

Hi,

In this article, I will show you how you can configure a Prometheus and Grafana solution to monitor your OPA Gatekeeper policies. The requirement is to have an existing Prometheus and Grafana stack deployed. If you don’t have an OPA Gatekeeper deployed in your AKS cluster, please follow the step of this article.

By default, when you deploy OPA Gatekeeper inside your kubernetes cluster, some OPA Gatekeeper metrics are already exposed for you! The idea is to consume these metrics via Prometheus and use Grafana to create a dashboard. In this example, we will create a new Grafana Dashboard to expose the number of violation of our dry-run OPA policies.

We will use the prometheus scrape feature to collect the OPA metrics. I will recommend you to edit the configuration of the OPA gatekeeper audit pod and add these following configuration lines:

  • prometheus.io/scrape: « true »
  • prometheus.io/port: « 8888 »
➜  ~ kubectl get pods --namespace gatekeeper-system
 NAME                                             READY   STATUS    RESTARTS   AGE
 gatekeeper-audit-576f6d6f8d-p5nvk                1/1     Running   0          18h
 gatekeeper-controller-manager-85d8bf48c9-5j2f5   1/1     Running   0          6d
 gatekeeper-controller-manager-85d8bf48c9-v2d92   1/1     Running   1          6d1h
 gatekeeper-controller-manager-85d8bf48c9-z924v   1/1     Running   0          18h
 gatekeeper-policy-manager-5bf4586996-2cmw9       1/1     Running   0          18h

➜  ~ kubectl edit pods gatekeeper-audit-576f6d6f8d-p5nvk --namespace gatekeeper-system
apiVersion: v1
 kind: Pod
 metadata:
   annotations:
     container.seccomp.security.alpha.kubernetes.io/manager: runtime/default
     prometheus.io/scrape: "true"
     prometheus.io/port: "8888"

When it’s done, we can see in the prometheus target a new resource:

Now, we can create a new Kubernetes dashboard from Grafana and add a metric to display only the gatekeeper violations for our dry-run policies:

Metrics: gatekeeper_violations{control_plane="audit-controller", enforcement_action="dryrun"}

In conclusion, we saw how you can configure Prometheus and Grafana to monitor the number of OPA gatekeeper violations. Do not hesitate to read the official documentation of OPA gatekeeper, other metrics are available to help you to monitor your OPA gatekeeper solution (Current number of known constraints, Number of observed constraint templates, …).

Maxime.