#SSH Key rotation! In some circles ssh key rotation is terrifying and considered a massive headache. In security circles the lack of key rotation is a ripe target to compromise ALL the things. How do we solve this?
Well you can write a bash script that is going to go through a list of hosts and then do an ssh-copy-id to each host, but this is not going to invaliadate the old keys it is just going to dd the new ones. Only solving half of the issue. None the less for the initial setup of the solution this command is pretty nice none the less so here it is. ssh-copy-id -i ~/.ssh/mykey user@host
https://www.ssh.com/ssh/copy-id

Soultion anisible. Ansible is nothing new, it has been around for a while but it is surprising the number of folks that do not use it. Here is some info for ansible http://docs.ansible.com/. The feature that we are going to focus on is the authorized_key module http://docs.ansible.com/ansible/latest/authorized_key_module.html

Lab setup: SSH key already setup on each of the hosts that you are going to change. Ansible running on your laptop, which is where we are going to ssh from. Ansible inventory of the hosts with the shared ssh key. – for my lab i am using a bunch of physical raspberry pis, because I like them and it makes me feel like a tiny mad scientist having a mini cluster of pi’s running on my desk. There were days that I would rather have a beuwolf cluster of machines running but hey you gotta make do. https://www.youtube.com/watch?v=8LsxmQV8AXk - switch to linux

Here is an inventory example - simple enough

[pi]
192.168.1.6 user=pi
192.168.1.7 user=pi
192.168.1.8 user=pi
192.168.1.9 user=pi

Ansible is going to generate a new ssh key at a different location than my primary key because right now I am labbing this out and don’t want to break all the things. It is easy enough to use your central location, once we are out of lab stage. A future implementation is going to have a KMS of sorts so I can securely store all the keys but right now lets do this insecurely and we can discuss the best practice later. Vault-> https://www.vaultproject.io/docs/secrets/ssh/index.html

Great we have an inventory and our ssh key on each of the hosts.. Lets say hello world to our hosts to make sure the lab is configured

ansible pi -a “/bin/echo hello world” -i hosts -u pi –private-key="~/.ssh/id_rsa

192.168.1.6 | SUCCESS | rc=0 >>
hello world 
92.168.1.7 | SUCCESS | rc=0 >>
hello world 
92.168.1.8 | SUCCESS | rc=0 >>
hello world 
92.168.1.9 | SUCCESS | rc=0 >>
hello world

Great it worked, next lets change the key and hope to not break everything. We are going to create an ansible playbook with a few tasks in it. First task is to generate a new ssh-key locally at our new location.

name: "Set up authorized_keys for the root user"
  hosts: pi
  user: pi

  tasks:
  - name: Create new ssh key-pair
    local_action: command ssh-keygen -t rsa -N "" -q -f ~/test/id_rsa

Next we will have a task that takes our new keyfile and pushes it to the hosts, using an exclusive command. The exclusive property is important because without it, we can retain the old keys. Or if we wanted to have multiple keys deployed we can do multiple steps, the first being the exclusive push and then appending our keys as we go.

   - name: Set up authorized_keys for the pi user
    authorized_key: user=pi key="{{ item }}" state=present exclusive=yes 
    with_file:
      - ~/test/id_rsa.pub

Finally we have two steps to move the newly generated keys to our super secret archive, I have not worked this one out yet but cleanup is needed somewhere.

  - name: move key-pair  
    local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak
    run_once: true

  - name: move key-pair  
    local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak
    run_once: true

putting it all together we get

- name: "Set up authorized_keys for the root user"
  hosts: pi
  user: pi

  tasks:
  - name: Create new ssh key-pair
    local_action: command ssh-keygen -t rsa -N "" -q -f ~/test/id_rsa
  
        
  - name: Set up authorized_keys for the pi user
    #authorized_key: user=pi key="{{ item }}" state=present exclusive=yes 
    authorized_key: user=pi key="{{ item }}" state=present exclusive=yes 
    with_file:
      - ~/test/id_rsa.pub

  - name: move key-pair  
    local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak
    run_once: true

  - name: move key-pair  
    local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak
    run_once: true

Lets fire it up and change some keys!

$ ansible-playbook -i hosts playbook/ssh-key-push-revoke-old.yml

PLAY [Set up authorized_keys for the root user] ********************************

TASK [setup] *******************************************************************
ok: [192.168.1.9]

TASK [Create new ssh key-pair] *************************************************
changed: [192.168.1.9 -> localhost]

TASK [Set up authorized_keys for the pi user] **********************************
changed: [192.168.1.9] => (item=ssh-rsa xxxxx mikefettis@MAC2093.local)

TASK [move key-pair] ***********************************************************
changed: [192.168.1.9 -> localhost]

TASK [move key-pair] ***********************************************************
changed: [192.168.1.9 -> localhost]
        to retry, use: --limit @playbook/ssh-key-push-revoke-old.retry

PLAY RECAP *********************************************************************
192.168.1.9               : ok=1    changed=1    unreachable=0    failed=0

It worked! We rotated keys and we are more secure than we were an hour ago!!!

there are a ton of links i used diggning through this so I will include them all here https://derpops.bike/2014/06/07/ssh-key-rotation-with-ansible/ http://vicendominguez.blogspot.kr/2014/05/ansible-generating-pub-key-file-and.html

ansible docs: http://docs.ansible.com/ansible/latest/playbooks_intro.html http://docs.ansible.com/ansible/latest/authorized_key_module.html

manually ssh-copy-id https://www.ssh.com/ssh/copy-id