Skip to content

AWX Kerberos Implementation

Kerberos Implementation

You may find that you need to be able to run playbooks on domain-joined Windows devices using Kerberos. You need to go through some extra steps to set this up after you have successfully fully deployed AWX Operator into Kubernetes.

Configure Windows Devices

You will need to prepare the Windows devices to allow them to be remotely controlled by Ansible playbooks. Run the following powershell script on all of the devices that will be managed by the Ansible AWX environment.

Create an AWX Instance Group

At this point, we need to make an "Instance Group" for the AWX Execution Environments that will use both a Keytab file and custom DNS servers defined by configmap files created below. Reference information was found here. This group allows for persistence across playbooks/templates, so that if you establish a Kerberos authentication in one playbook, it will persist through the entire job's workflow.

Create the following files in the /awx folder on the AWX Operator server you deployed earlier when setting up the Kubernetes Cluster and deploying AWX Operator into it so we can later mount them into the new Execution Environment we will be building.

/awx/custom_dns_records.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-dns
  namespace: awx
data:
  custom-hosts: |
    192.168.3.25 LAB-DC-01.bunny-lab.io LAB-DC-01
    192.168.3.26 LAB-DC-02.bunny-lab.io LAB-DC-02
    192.168.3.4 VIRT-NODE-01.bunny-lab.io VIRT-NODE-01
    192.168.3.5 BUNNY-NODE-02.bunny-lab.io BUNNY-NODE-02
/awx/krb5.conf
[libdefaults]
    default_realm = BUNNY-LAB.IO
    dns_lookup_realm = false
    dns_lookup_kdc = false

[realms]
    BUNNY-LAB.IO = {
        kdc = 192.168.3.25
        kdc = 192.168.3.26
        admin_server = 192.168.3.25
    }

[domain_realm]
    192.168.3.25 = BUNNY-LAB.IO
    192.168.3.26 = BUNNY-LAB.IO
    .bunny-lab.io = BUNNY-LAB.IO
    bunny-lab.io = BUNNY-LAB.IO

Then we apply these configmaps to the AWX namespace with the following commands: 

cd /awx
kubectl -n awx create configmap awx-kerberos-config --from-file=/awx/krb5.conf
kubectl apply -f custom_dns_records.yml

  • Open AWX UI and click on "Instance Groups" under the "Administration" section, then press "Add > Add container group".
  • Enter a descriptive name as you like (e.g. Kerberos) and click the toggle "Customize Pod Specification".
  • Put the following YAML string in "Custom pod spec" then press the "Save" button
    Custom Pod Spec
    apiVersion: v1
kind: Pod
metadata:
  namespace: awx
spec:
  serviceAccountName: default
  automountServiceAccountToken: false
  initContainers:
    - name: init-hosts
      image: busybox
      command:
        - sh
        - '-c'
        - cat /etc/custom-dns/custom-hosts >> /etc/hosts
      volumeMounts:
        - name: custom-dns
          mountPath: /etc/custom-dns
  containers:
    - image: quay.io/ansible/awx-ee:latest
      name: worker
      args:
        - ansible-runner
        - worker
        - '--private-data-dir=/runner'
      resources:
        requests:
          cpu: 250m
          memory: 100Mi
      volumeMounts:
        - name: awx-kerberos-volume
          mountPath: /etc/krb5.conf
          subPath: krb5.conf
  volumes:
    - name: awx-kerberos-volume
      configMap:
        name: awx-kerberos-config
    - name: custom-dns
      configMap:
        name: custom-dns

Job Template & Inventory Examples

At this point, you need to adjust your exist Job Template(s) that need to communicate via Kerberos to domain-joined Windows devices to use the "Instance Group" of "Kerberos" while keeping the same Execution Environment you have been using up until this point. This will change the Execution Environment to include the Kerberos Keytab file in the EE at playbook runtime. When the playbook has completed running, (or if you are chain-loading multiple playbooks in a workflow job template), it will cease to exist. The kerberos keytab data will be regenerated at the next runtime.

Also add the following variables to the job template you have associated with the playbook below: 

---
kerberos_user: [email protected]
kerberos_password: <DomainPassword>

You will want to ensure your inventory file is configured to use Kerberos Authentication as well, so the following example is a starting point: 

virt-node-01 ansible_host=virt-node-01.bunny-lab.io
bunny-node-02 ansible_host=bunny-node-02.bunny-lab.io

[virtualizationHosts]
virt-node-01
bunny-node-02

[virtualizationHosts:vars]
ansible_connection=winrm
ansible_port=5986
ansible_winrm_transport=kerberos
ansible_winrm_scheme=https
ansible_winrm_server_cert_validation=ignore
#[email protected] #Optional, if you define this in the Job Template, it is not necessary here.
#kerberos_password=<DomainPassword> #Optional, if you define this in the Job Template, it is not necessary here.

Usage of Fully-Quality Domain Names

It is critical that you define Kerberos-authenticated devices with fully qualified domain names. This is just something I found out from 4+ hours of troubleshooting. If the device is Linux or you are using NTLM authentication instead of Kerberos authentication, you can skip this warning. If you do not define the inventory using FQDNs, it will fail to run the commands against the targeted device(s).

In this example, the host is defined via FQDN: virt-node-01 ansible_host=virt-node-01.bunny-lab.io

Kerberos Connection Playbook

At this point, you need a playbook that you can run in a Workflow Job Template (to keep things modular and simplified) to establish a connection to an Active Directory Domain Controller via Kerberos before running additional playbooks/templates against the actual devices.

You can visualize the connection workflow below:

graph LR
    A[Update AWX Project] --> B[Update Project Inventory]
    B --> C[Establish Kerberos Connection]
    C --> D[Run Playbook against Windows Device]

The following playbook is an example pulled from https://git.bunny-lab.io

Playbook Redundancies

I have several areas where I could optimize this playbook and remove redundancies. I just have not had enough time to iterate through it deeply-enough to narrow down exact things I can remove, so for now, it will remain as-is, since it functions as-expected with the example below.

Establish_Kerberos_Connection.yml
---
- name: Generate Kerberos Ticket to Communicate with Domain-Joined Windows Devices
  hosts: localhost
  vars:
    kerberos_password: "{{ lookup('env', 'KERBEROS_PASSWORD') }}"  # Alternatively, you can set this as an environment variable
    # BE SURE TO PASS "kerberos_user: [email protected]" and "kerberos_password: <domain_admin_password>" to the template variables when running this playbook in a template.

  tasks:
    - name: Generate the keytab file
      ansible.builtin.shell: |
        ktutil <<EOF
        addent -password -p {{ kerberos_user }} -k 1 -e aes256-cts
        {{ kerberos_password }}
        wkt /tmp/krb5.keytab
        quit
        EOF                
      environment:
        KRB5_CONFIG: /etc/krb5.conf
      register: generate_keytab_result

    - name: Ensure keytab file was generated successfully
      fail:
        msg: "Failed to generate keytab file"
      when: generate_keytab_result.rc != 0

    - name: Keytab successfully generated
      ansible.builtin.debug:
        msg: "Keytab successfully generated at /tmp/krb5.keytab"
      when: generate_keytab_result.rc == 0

    - name: Acquire Kerberos ticket using keytab
      ansible.builtin.shell: |
        kinit -kt /tmp/krb5.keytab {{ kerberos_user }}                
      environment:
        KRB5_CONFIG: /etc/krb5.conf
      register: kinit_result

    - name: Ensure Kerberos ticket was acquired successfully
      fail:
        msg: "Failed to acquire Kerberos ticket"
      when: kinit_result.rc != 0

    - name: Kerberos ticket successfully acquired
      ansible.builtin.debug:
        msg: "Kerberos ticket successfully acquired for user {{ kerberos_user }}"
      when: kinit_result.rc == 0