Authenticating Kubernetes in Active Directory—the saga...

Part 1…

Who is it for…

For system administrators, devops operators and engineers, technical and administrative leaders, the curious and the lazy ;-)….

In this article, technical jargon, command lines, code fragments, and access to source codes authored by me and third parties have been applied, with all their rights and licenses reserved.

The beginning…

It arose as a need to;

“ — We do not have a robust platform to orchestrate our containers/microservices;
 — We need to give the development team access to the services, logs, and applications of each namespace but without releasing access to the console or to all core components of kubernetes, in fact we want to separate the teams of each system into their respective namespaces;
 — We already have a directory service, I don't want another one …
 —Creating a user and administering them in Kubernetes is too complex …. (huhUAhuHAuHha, lazy, go study).”

So, for you, DevOps Engineer, who needs to do this integration or wants to test how it works and not implement an Openshift/OKD or Rancher, or other solution; this might be useful sooner or later.

Not that PaaS solutions do not meet needs and/or expectations, but that their support cost, learning curve, or even orchestration layers weigh at the time of administration and the good old "roots" and pure Kubernetes is preferred, and yes, authenticating somewhere, keeping my SSO (single sign on).

Accessing the console of the master servers or even the cost of a bastion machine just for this purpose, management of kubernetes components; is often expensive, insecure with complex auditing, questionable among other points.

The subject is rarely discussed and search results on the internet bring us incomplete, cloudy information, with a lack of options for distinct realities and leave a lot to be desired in terms of answers and questions and doubts from readers.

If you have any questions or opinions, please feel free to express them and contact me, as far as I can and to the best of my ability I will help you :-)

Without further ado, let's get down to business... hands-on, oops, hand on the keyboard (because mouse is for lazy people)…

Available options …

After almost dying from lack of some 3 sleepless nights, I managed to apply 2 distinct solutions but which inside, use the same components;

• Tremolo Security's Openunison

• DEX + Gangway + Oauth2-proxy

I confess that both completely meet the needs, there are differences in implementation and understanding but they work.

Openunison, from what I researched, is a java solution, written by Tremolo Security with DEX encapsulated for authentication in AD and then manipulates, creates token, and releases access to kubernetes.

The dex + gangway + oauth-proxy solution is lighter, written in GO in all 3 layers, that is, light and simple but requires sweat for customization if you want to implement it as an authentication portal for your users.

TL;DR;

Prereqs to implement the Openunison solution

1 — Kubernetes installed and with kube-apiserver configured for authentication using RBAC (calm down, I'll show you how to do it), Kubernetes Dashboard also implemented;

2 — SSL/TLS certificates, self-signed or signed by "heavenly" entities (also called valid) for your URLs;

• Main access URL (site where users will authenticate), is the Openunison portal/application, which will generate and manage your access token to kubernetes, both via kubectl on windows or linux or via dashboard.

• Kubernetes dashboard URL. is your dashboard URL

3 — User accounts and respective linked groups to validate the solution.

4 — Kubernetes Ingress Controller, look for NGINX, it works, has vast documentation. Although I professionally use Traefik 2.0 , this one still does not provide all the facilities of NGINX.

5 — regarding the kubernetes dashboard, already installed and with ingress created for its external access, as insecure, on port 80.

6 — root access, of couuuurse…

Creating configuration base

On your Master server or where you have the kubectl operational and with the configuration for your cluster, create two folders ; orchestra-configmaps and orchestra-secrets.

[root@server1]# mkdir deploy && cd deploy
[root@server1 deploy]# mkdir orchestra-configmaps orchestra-secrets

In the folder orchestra-configmaps, create an input.props file with the following content:

[root@server1 deploy/orchestra-configmaps]# cat input.props
OU_HOST=portaldousuario.meudominio.net
K8S_DASHBOARD_HOST=dash-k8s.meudominio.net
K8S_URL=https://k8s-master1.meudominio.net:6443
AD_BASE_DN=cn=users,dc=ent2k12,dc=domain,dc=com
AD_HOST=192.168.2.75
AD_PORT=636
AD_BIND_DN=cn=Administrator,cn=users,dc=ent2k12,dc=domain,dc=com
AD_CON_TYPE=ldaps
SRV_DNS=false
OU_CERT_OU=k8s
OU_CERT_O=My Company
OU_CERT_L=Atlantis
OU_CERT_ST=Bahia
OU_CERT_C=BR
USE_K8S_CM=true
SESSION_INACTIVITY_TIMEOUT_SECONDS=900
MYVD_CONFIG_PATH=WEB-INF/myvd.conf
[root@server1 deploy/orchestra-configmaps]#

Inside orchestra-secrets create another input.props file with the following content;

[root@server1 deploy/orchestra-secrets]# cat input.props
AD_BIND_PASSWORD=password
unisonKeystorePassword=start123
[root@server1 deploy/orchestra-secrets]#

Where [root@server1 deploy/orchestra-configmaps]# cat input.props is

Property — — — — — — — —Description
OU_HOST ……………………….The URL to access the token generation system. The user will access this initial URL.
K8S_DASHBOARD_HOST …….The URL of your kubernetes dashboard. It is the URL that you previously registered in your ingress to access the k8s dashboard externally. NOTE: OU_HOST and K8S_DASHBOARD_HOST MUST share the same DNS domain. Both OU_HOST and K8S_DASHBOARD_HOST will route to the Openunison deploy on the same ingress.
K8S_URL ……………………….The URL of your kubernetes, kube-apiserver.
AD_BASE_DN ………………….Starting search point of your AD.
AD_HOST ………………………The hostname or IP or the VIP of your AD. If you are using SRV records to point to hosts, this hostname must be the fqdn, hostname-vip.dominio.
AD_PORT………………………. The TCP port of AD, 636 for SSL.
AD_BIND_DN …………………..The account created that can search the AD.
AD_CON_TYPE.….….….….…..ldaps for SSL, ldap for plain text
SRV_DNS ……………………….If true, OpenUnison will look in DNS for the entry for the domain SRV DNS.
OU_CERT_OU ………………….OU is the display attribute of the internal certificate of the Openunison application.
OU_CERT_O ……………………O is another attribute of the internal certificate.
OU_CERT_L …………………….L is another attribute of the internal certificate.
OU_CERT_ST…………………...ST is another attribute of the certificate.
OU_CERT_C……………………..C is another attribute of the certificate.
USE_K8S_CM……………………If true, to use your kubernetes builtin certificate. false if your installation is Rancher or Canonical.
SESSION_INACTIVITY_TIMEOUT_SECONDS…….How many seconds of inactivity before the session is ended, inside the Openunison application, not in the kubernetes dashboard.
MYVD_CONFIG_PATH…………For multiple ADs, or LDAP edit this file, more information on the Tremolo Security website.
K8S_DASHBOARD_NAMESPACE…..What is the namespace of your dashboard. For dashboard version 1.x it is kube-system, for version 2.x it is kubernetes-dashboard
K8S_CLUSTER_NAME…………Optional If specified, the name of your kubernetes cluster within ./kube-config. The default is kubernetes.

And on [root@server1 deploy/orchestra-configmaps]# cat input.props

Property — — — — — — — — — — Description
AD_BIND_PASSWORD.…………….The password for AD_BIND_DN.
unisonKeystorePassword…..……..The password, to your liking, for the application's java keystore.

With this ready, you now need to copy the certificates of the above URLs to the orchestra-configmaps folder, in .pem format (which is a copy with .pem extension of a .crt certificate. That is, copy the .crt file of your self-signed certificate to the file with same name, but with .pem extension, for those who didn't understand, here is the command, lazy person ;-)

[root@server1]# cat portaldousuario.meudominio.net.crt
-----BEGIN CERTIFICATE-----
LKlkJHopoipiASAsaasakjhJKLhakJASSAAdd.........
.............bla bla bla
-----END CERTIFICATE-----
[root@server1]#
[root@server1]# cp portaldousuario.meudominio.net.crt deploy/orchestra-configmaps/portaldousuario.meudominio.net.pem
[root@server1]#
[root@server1]# cp dash-k8s.meudominio.net.crt deploy/orchestra-configmaps/dash-k8s.meudominio.net.pem

That's right, the .pem is identical to the .crt, no .key, no conversion, no fuss ;-)

If you are not familiar with certificates, I created a script that does this, you can download it from my github. If you look a bit closer, you will see the logic of this script is easy, which makes understanding the use of openssl for generating and maintaining certificates a trivial task.

Openunison Deploy by Tremolo Security

Execute inside the deploy directory;

[root@server1]# curl https://raw.githubusercontent.com/TremoloSecurity/kubernetes-artifact-deployment/master/src/main/bash/deploy_openunison.sh | bash -s /path/to/orchestra-configmaps /path/to/orchestra-secrets https://raw.githubusercontent.com/OpenUnison/openunison-k8s-login-activedirectory/master/src/main/yaml/artifact-deployment.yaml

@#$% Do not press ENTER. Let's pay attention to the command;

Notice here, that the path to the configmaps and secrets directories MUST BE THE FULL PATH, and without "/" at the end.

:-) You can press ENTER …. ;-)

Let's analyze what the Tremolo folks did; they created a javascript that;

• reads configuration, input.props, and certificates,

• creates a namespace, openunison-deploy

• creates secrets for configurations, certificates, internal key of the application,

• from the execution of this javascript, with all these parameters, a new namespace is created, openunison.

• creates configmaps, secrets, users, tokens, roles, rolebindings, apiservices, deployments and more other components for the openunison namespace,

• loads images from their hub.docker. ( I scanned it as a security measure and there were no serious problems ) If you feel safer, recompile the image removing the shells, but keeping the ports, endpoints and if possible, keep the labels to give proper credit to Tremolo. After all, they created it. ;-)

Your console will look like this;

namespace/openunison-deploy created
configmap/extracerts created
secret/input created
clusterrolebinding.rbac.authorization.k8s.io/artifact-deployment created
job.batch/artifact-deployment created
NAME READY STATUS RESTARTS AGE
artifact-deployment-jzmnr 0/1 Pending 0 0s
artifact-deployment-jzmnr 0/1 Pending 0 0s
artifact-deployment-jzmnr 0/1 ContainerCreating 0 0s
artifact-deployment-jzmnr 1/1 Running 0 4s
artifact-deployment-jzmnr 0/1 Completed 0 15s

When the artifact-deployment pod becomes “Completed”, you can exit with Ctrl+C.

Let's check the logs and do what the conductor says;

[root@server1 ~]# kubectl get po -n openunison-deploy
NAME READY STATUS RESTARTS AGE
artifact-deployment-jzmnr 0/1 Completed 0 2d1h
[root@server1 ~]# kubectl logs artifact-deployment-jzmnr -n openunison-deploy
Processing - '/etc/extracerts/........blablabla.....
Artifacts Created, to configure the API server run 'kubectl describe configmap api-server-config -n openunison'
[root@server1 ~]#
[root@server1 ~]# kubectl describe configmap api-server-config -n openunison
Name: api-server-config
Namespace: openunison
Labels: <none>
Annotations: <none>

Data
====
oidc-api-server-flags:
----
--oidc-issuer-url=https://portaldousuario.meudominio.net/auth/idp/k8sIdp
--oidc-client-id=kubernetes
--oidc-username-claim=sub
--oidc-groups-claim=groups
--oidc-ca-file=/etc/kubernetes/pki/ou-ca.pem
Events: <none>
[root@bastion-0 ~]#

Check the new namespace and if you have the two Pods openunison-orchestra and openunison-operator up and running, see their logs….

[root@server1 ]# kubectl get pod -n openunison
NAME READY STATUS RESTARTS AGE
openunison-operator-757cf446d9-dzkml 1/1 Running 0 20s
openunison-orchestra-58cd55577d-9n5c9 1/1 Running 0 20s
[root@server1 ]#

Inside a secret, ou-tls-certificate, there is a certificate and a private key, created based on the other two .pem of the URLs that will be accessible, the kubernetes dashboard itself and the authentication and token generation portal that were made available in the previous step.

You need to rescue them and make them available for kube-apiserver use.

There are two keys in this secret, tls.crt and tls.key, copy, echo and decode, saving the tls.crt with extension .pem, as follows;

[root@server1 ~]# kubectl get secret ou-tls-certificate -n openunison -o yaml
apiVersion: v1
data:
tls.crt: LS0tLS1CRUdJ...blablabla 0tLS0tCg==
tls.key: LS0tLS1CRUd....blablabla...LS0tLS0K
kind: Secret
metadata:
creationTimestamp: "2019-and smokeZ"
labels:
operated-by: openunison-operator
tremolo_operator_created: "true"
name: ou-tls-certificate
namespace: openunison
resourceVersion: "1479068"
selfLink: /api/v1/namespaces/openunison/secrets/ou-tls-certificate
uid: 1020959cxxxxxxxxxxxxxxxxx572af
type: kubernetes.io/tls
[root@server1 ~]# echo "LS0tLS...blablabla....RVJ0FURS0tLS0tCg==" | base64 -d > /etc/kubernetes/pki/ou-ca.pem
[root@server1 ~]#
[root@server1 ~]# echo "LS0tLS1CR....blablabla.....BVS0VZLS0tLS0K" | base64 -d > /etc/kubernetes/pki/ou-ca.key
[root@server1 ~]# chmod 0600 /etc/kubernetes/pki/ou-ca.key
[root@server1 ~]# ls -la /etc/kubernetes/pki/ou-ca.*
-rw------- 1 root root 1704 Feb 31 15:57 ou-ca.key
-rw-r--r-- 1 root root 1498 Feb 31 15:56 ou-ca.pem
[root@server1 ~]#

If you want to verify what is in the certificate, execute;

[root@server1 ~]# openssl x509 -in /etc/kubernetes/pki/ou-ca.pem -text -noout
(...) output omitted
your certificates, who signed, validity, issued on, etc...
(...)
[root@server1 ~]#

All good, let's make kubernetes recognize your new authenticator, this will generate an unavailability of a few seconds for kubectl, do it carefully, services will continue to work, applications as well, so, do that ...

[root@server1 ~]# vi /etc/kubernetes/manifests/kube-apiserver.yaml

add these lines in the apiserver arguments

containers:
- command:
- kube-apiserver
- --advertise-address=172.28.39.11
- --oidc-issuer-url=https://portaldousuario.meudominio.net/auth/idp/k8sIdp
- --oidc-client-id=kubernetes
- --oidc-username-claim=sub
- --oidc-groups-claim=groups
- --oidc-ca-file=/etc/kubernetes/pki/ou-ca.pem

Wait for the kube-apiserver to return;

[root@server1 ~]# watch -n1 kubectl get pod -n kube-system

Every 1.0s: kubectl get po -n kube-system Thu Feb 31 17:19:42 2019

NAME READY STATUS RESTARTS AGE
coredns-584795fc57-cxctl 1/1 Running 9 10w
coredns-584795fc57-rv5ql 1/1 Running 9 10w
etcd-bastion-0 1/1 Running 0 14w
kube-apiserver-bastion-0 1/1 Running 0 2s
kube-controller-manager-bastion-0 1/1 Running 7 14w
kube-proxy-nswkh 1/1 Running 0 14w
kube-scheduler-bastion-0 1/1 Running 7 14w
metrics-server-6dcf9864f6-xzhn4 1/1 Running 0 6d2h
weave-net-k84rs 2/2 Running 0 14w
[root@server1 ~]#

First access… almost there…

Before attempting your access, make sure ground has been prepared in AD.

• a group of users which will be one of those configured in kubernetes for access, e.g. k8s-admins

• a user who is part of the above group, e.g. usr_euzinho

Check if your ingress for Openunison has been created and is operational;

[root@server1 ~]# kubectl get ingress -n openunison
NAME HOSTS ADDRESS PORTS AGE
openunison portaldousuario.meudominio.net,dash-k8s.meudominio.net 10.105.115.198 80, 443 12m
[root@server1 ~]#

Access the URL ; https://portaldosuario.meudominio.net

You should supply your AD user and password. But you still won't gain access, rules inside kubernetes need to be created, but you can already see your token and additional commands to create your ./kube/config or kubectl windows command if you want to access k8s through your station.

There is a problem there, and we need to solve it before continuing, this openunison ingress created an entry for your dashboard that already exists, remove it, by editing the ingress.

[root@server1 ~]# kubectl create -f << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: activedirectory-cluster-admins
subjects:
- kind: Group
name: "CN=k8s-admins,CN=Users,DC=ent2k12,DC=domain,DC=com"
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
EOF
Success!!! ClusterRoleBinding activedirectory-cluster-admin created.
[root@server1 ~]#

• On the portal, perform logout and access again.

• Click on Kubernetes Tokens.

• Copy the id_token, return to the main page and click on Kubernetes Dashboard.

• On the login page of Kubernetes Dashboard, select token and paste your id_token, voila, full access to resources.

Did you see that nothing worked? -) It's because we haven't yet created the access rules for this AD group/user.

At this moment, only token creation, certificates and peripheral commands work, but they are not valid because the access correlation within kubernetes does not exist. So let's understand to create them.

Limit users and groups

For this you need to, in AD, create groups with understandable names and bind users to them. e.g.: k8s-admins, k8s-ns-site-a-views, k8s-ns-app-b-admins, etc…

It is not necessary to initialize the group with k8s, as long as you know which groups you want to release access to. Therefore, they can be existing ones, as long as there is coherence in your organization.

Once done, we need to decide which is the most lazy way of administering kubernetes.

• By granting access to AD groups, avoiding the work of creating a rule for each user — THIS IS THE RECOMMENDED WAY, or;

• By granting for each new user in the environment, forcing a rule to be created (clusterrolebinding for entire cluster or rolebinding for namespaces) for each of them, in addition to being careful to remove their access in case of resignation, promotion, E.T. abduction, among others.

If you don't know, see how kubernetes RBAC works, if you already know, you can skip this part.

Finally when you want to give access to users, groups etc, via RBAC of course, you need to link them to a rule (role) and its effective permission (binding). This can be done in the namespace scope or for the cluster as a whole.

In summary, 3 examples of how to create access;

- Concession by AD Group.

You already created a group called k8s-app-loja, and another group k8s-noc-viewers. To give appropriate access to users of these groups, create the files below and apply them.

[root@server1 ~]# kubectl create -f << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: k8s-noc-viewers
subjects:
- kind: Group
name: "CN=k8s-noc-viewers,CN=Users,DC=ent2k12,DC=domain,DC=com"
roleRef:
kind: ClusterRole
name: view
apiGroup: rbac.authorization.k8s.io
EOF
clusterrolebinding.rbac.authorization.k8s.io/k8s-noc-viewers created
[root@server1 ~]#
[root@server1 ~]#
[root@server1 ~]# kubectl create -f << EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: k8s-app-loja-viewers
namespace: app-loja
subjects:
- kind: Group
name: "CN=k8s-app-loja-viewers,CN=Users,DC=ent2k12,DC=domain,DC=com"
roleRef:
kind: Role
name: viewer-app-loja
apiGroup: rbac.authorization.k8s.io
EOF
rolebinding.rbac.authorization.k8s.io/k8s-app-loja-viewers created
[root@server1 ~]#

From the example above you can get an idea of what you need for your accesses, be creative, record your changes and document in your repository, share.

- Concession by AD Users.

The same way as above, but instead of groups, we specify users;

[root@server1 ~]# kubectl create -f << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: raimundo-cluster-admin
subjects:
- kind: User
name: https://kubectl-auth-dev.cip-core.local/auth/idp/k8sIdp#RAIMUNDO.NONATO
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
EOF
clusterrolebinding.rbac.authorization.k8s.io/raimundo-cluster-admin created
[root@server1 ~]#
[root@server1 ~]#
[root@server1 ~]#kubectl create -f << EOF
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: antonio-app-loja-admin
namespace: app-loja
subjects:
- kind: User
name: https://kubectl-auth-dev.cip-core.local/auth/idp/k8sIdp#ANTONIO.AUGUSTO
roleRef:
kind: Role
name: app-loja-admin
apiGroup: rbac.authorization.k8s.io
EOF
rolebinding.rbac.authorization.k8s.io/antonio-app-loja-admin created
[root@server1 ~]#

Now you can test your access. There are a few secrets regarding the Tremolo solution, if you need help, drop me a line and I'll show how to break "some of the barriers".

In the second part, I will continue with the implementation of DEX+Gangway+Oauth2, stay tuned.

Thanks for knowledge share and support

support.ssl.com

tremolosecurity.com

kubernetes.io

heptiolabs, now VMware

nginx.com, now F5

DISCLOSURE STATEMENT: These opinions are those of the author. Unless noted otherwise in this post, Kubernetes, Tremolo Security, Heptio Labs or VMware is not affiliated with, nor is it endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are the ownership of their respective owners. This article is © Adonai Costa.