How to Log Key Value Changes in Consul

What is Consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure. It provides several key features:

For more information you can visit their Official Page or Github Repository.

In scope of this blogpost we will only focus on Key/Value Storage feature. By default, Consul doesn’t provide a mechanism for you to audit when a key/value is changed or what was the change, but if you keep your application configuration or feature flags for your application you may want to see logs about key/value changes.

What are Consul Watches

Watches are a way of specifying a view of data (e.g. list of nodes, KV pairs, health checks) which is monitored for updates. When an update is detected, an external handler is invoked. A handler can be any executable or HTTP endpoint. As an example, you could watch the status of health checks and notify an external system when a check is critical. According to official documentation we can use Consul Watches to trigger a handler when any change is detected in key/value pairs, which means we can create a shell script which logs the differences between states each time it gets triggered [1].

Logging the Key/Value Changes

Setup

First we need to download docker image and executable binary for consul to run commands from our local. To download the binary you can visit here. As of this writing the latest version is 1.8.0, so I will use that one.

Now we create a local Consul server in development mode with following command:

➜  ~ consul agent -dev

We started the server in dev mode because first we will construct the script, then we will test it on a non-dev setup. After this point, when you go to localhost:8500, you will see Consul UI. Under Key/Value tab we can create whatever we want. For the sake of simplicity I only created following keys:

serhat
├── key1=value1
└── key2=value2

From my local I can see values of each key:

➜  ~ consul kv get serhat/key1
value1
➜  ~ consul kv get serhat/key2
value2

Watch Mode With Agent

After the initial setup when you run following command you should get a json showing all keys and base64 encoded values:

➜  ~ consul watch -type=keyprefix -prefix=serhat
[
    {
        "Key": "serhat/key1",
        "CreateIndex": 48,
        "ModifyIndex": 48,
        "LockIndex": 0,
        "Flags": 0,
        "Value": "dmFsdWUx",
        "Session": ""
    },
    {
        "Key": "serhat/key2",
        "CreateIndex": 49,
        "ModifyIndex": 49,
        "LockIndex": 0,
        "Flags": 0,
        "Value": "dmFsdWUy",
        "Session": ""
    }
]

However, this is a one-time command, what we need is some kind of daemon to keep watching all the time for any change. For this purpose we will use consul agent with watch config and a script for logging. The script will initialize a git repository in a directory and get this output to a file in that directory. Then everytime something changes the script will get the diff by running git diff and log the changes to another file. For the shell script I begin by initializing a local repo if it doesn’t exist already:

#!/usr/bin/env sh

# if folder does not exist create it
# and initialize a git repo for the first run
if [ ! -d "/tmp/consul/kvs/${1}/" ]; then
        mkdir /tmp/consul/kvs/${1}/
        pushd /tmp/consul/kvs/${1}/
        touch kvs.txt
        consul watch -type keyprefix -prefix ${1} > kvs.txt
        git init
        git config user.email "consul@serhat.dev"
        git config user.name "consul"
        git add kvs.txt && git commit -m 'init kvs.txt'
        popd
fi

If the directory is there, it means we have a state of all the key/values in /tmp/consul/kvs/${1}/kvs.txt file (we also check log file directory and create it if doesn’t exist). Then we need to do a check to see if there is a change for what we watch. Consul will trigger our script when any key is changed but we want to check only a specific prefix (serhat in this case), that’s why we need to make sure the script is triggered by a change in the watched prefix before taking any actions. Also we use ${1} argument to make it generic, instead of hardcoding a static keyprefix in our script.

if [ ! -d "/tmp/consul/log/" ]; then
    mkdir -p /tmp/consul/log/
fi

FLAG=0
pushd /tmp/consul/kvs/${1}/
consul watch -type keyprefix -prefix ${1} > kvs.txt
git diff --no-ext-diff --quiet --exit-code || FLAG=1

If the flag is 0 then the key/value change is not our concern, if it is 1 then there is something changed under our keyprefix. So we continue by:

if [ $FLAG == 0 ]; then
        echo "$(date) : No Change" >> "/tmp/consul/log/consul_watch.log"
        popd
        exit 0
fi

if [ $FLAG == 1 ]; then
        # first we trim quotes and commas then grep for Key and get what it is
        KEY=`git diff | sed -e 's/\"//g' | sed -e 's/\,//g' | grep Key | awk '{ print $2 }'`

        # to get the old and new values we do the same and decode base64 values
        OLD=`git diff | sed -e 's/\"//g' | sed -e 's/\,//g' | grep Value | awk -F'Value: ' '{print $2}' | head -1 | base64 -D`
        NEW=`git diff | sed -e 's/\"//g' | sed -e 's/\,//g' | grep Value | awk -F'Value: ' '{print $2}' | tail -1 | base64 -D`

        if [ "$KEY" != "Key:" ]; then
                echo $(date) >> "/tmp/consul/log/consul_watch.log"
                echo "Change detected for key: ${KEY}" >> "/tmp/consul/log/consul_watch.log"
                echo "Old Value: ${OLD}," >> "/tmp/consul/log/consul_watch.log"
                echo "New Value: ${NEW}" >> "/tmp/consul/log/consul_watch.log"
                echo " " >> "/tmp/consul/log/consul_watch.log"
        fi
        git add kvs.txt
        git commit -m "Update kvs.txt"
        popd
        exit 0
fi

Finally the handler script would look like this:

#!/usr/bin/env sh

# if folder does not exist create it
# and initialize a git repo for the first run
if [ ! -d "/tmp/consul/kvs/${1}/" ]; then
        mkdir -p /tmp/consul/kvs/${1}/
        pushd /tmp/consul/kvs/${1}/
        touch kvs.txt
        consul watch -type keyprefix -prefix ${1} > kvs.txt
        git init
        git config user.email "consul@serhat.dev"
        git config user.name "consul"
        git add kvs.txt && git commit -m 'init kvs.txt'
        popd
fi

if [ ! -d "/tmp/consul/log/" ]; then
    mkdir -p /tmp/consul/log/
fi

FLAG=0
pushd /tmp/consul/kvs/${1}/
consul watch -type keyprefix -prefix ${1} > kvs.txt
git diff --no-ext-diff --quiet --exit-code || FLAG=1

if [ $FLAG == 0 ]; then
        echo "$(date) : No Change" >> "/tmp/consul/log/consul_watch.log"
        popd
        exit 0
fi

if [ $FLAG == 1 ]; then
        # first we trim quotes and commas then grep for Key and get what it is
        KEY=`git diff | sed -e 's/\"//g' | sed -e 's/\,//g' | grep Key | awk '{ print $2 }'`

        # to get the old and new values we do the same and decode base64 values
        OLD=`git diff | sed -e 's/\"//g' | sed -e 's/\,//g' | grep Value | awk -F'Value: ' '{print $2}' | head -1 | base64 -D`
        NEW=`git diff | sed -e 's/\"//g' | sed -e 's/\,//g' | grep Value | awk -F'Value: ' '{print $2}' | tail -1 | base64 -D`

        if [ "$KEY" != "Key:" ]; then
            echo $(date) >> "/tmp/consul/log/consul_watch.log"
            echo "Change detected for key: ${KEY}" >> "/tmp/consul/log/consul_watch.log"
            echo "Old Value: ${OLD}," >> "/tmp/consul/log/consul_watch.log"
            echo "New Value: ${NEW}" >> "/tmp/consul/log/consul_watch.log"
            echo " " >> "/tmp/consul/log/consul_watch.log"
        fi
        git add kvs.txt
        git commit -m "Update kvs.txt"
        popd
        exit 0
fi

Now we will run a Consul server (which is also an agent to itself) on our local to see if it works. We will also pass following json file as config to the Consul agent and make it trigger our script whenever it detects a key value change:

{
  "datacenter": "dc1",
  "data_dir": "/path/to/your/data/dir",
  "watches": [
    {
      "type": "keyprefix",
      "prefix": "serhat/",
      "args": ["/path/to/your/watch_handler.sh", "serhat"]
    }
  ]
}

To start the server use following command:

consul agent -server -bootstrap -ui -client=0.0.0.0 -bind='{{ GetPrivateIP }}' -config-file=conf.json

Now create some key/values under serhat keyprefix and tail the log file, after this point whenever you update an existing key you will see an output similar to this one:

➜  ~ tail -f /tmp/consul/log/consul_watch.log
Sun Jul 26 19:28:57 +03 2020 : No Change
Sun Jul 26 19:29:54 +03 2020
Change detected for key: serhat/key1
Old Value: value1,
New Value: new value

Sun Jul 26 19:30:05 +03 2020
Change detected for key: serhat/key1
Old Value: new value,
New Value: newest value

Sun Jul 26 19:30:11 +03 2020
Change detected for key: serhat/key2
Old Value: value2,
New Value: it works!

Note 1: This is not a production grade setup, if you want to run Consul on production please have a look at production checklist prepared by Hashicorp [2]. However, if you already have a running Consul setup, you should be able to enhance it easily by adding this script and watch section to your config file.

Note 2: Demo code mentioned above is tested on a Macbook, if you are trying to make it work on a Linux machine you might need to tweak some of the commands (for example instead of base64 -D you should use base64 -d).

References

[1] - https://www.consul.io/docs/agent/watches.html

[2] - https://learn.hashicorp.com/consul/datacenter-deploy/production-checklist