We use the Instana APM for our application monitoring. In Instana, we have 2 tenants, one for Production and one for Non-Prod (everything else). Due to these 2 tenants, we found that there are certain configurations that we really want to “promote” from one to the other. One such configuration is the endpoint configuration for the microservices.
The story here is that we need to test out a service’s endpoint configuration in our Non-Prod instance, and then move that configuration up to Production when we are happy with it. Instana does not have anything out-of-the-box that can do this, but they do have a set of APIs. We use these APIs to accomplish this task.
We use Jenkins as our build/deployment tool, and it is a nice kind of Swiss-Army knife tool to use for many different tasks, including the one we want to execute. To drive this tool, we created a simple Git repo to house some Jenkinsfiles and a JSON configuration file. The JSON file holds the “current” state of the endpoint configurations. One Jenkins job will gather the configuration from a target tenant, and save that to the JSON file. The other Jenkins job will take the configuration in the JSON file and publish that to a target Instana tenant.
Storing the configuration file in a Git repo allows us to both follow a kind of SDLC process, and to provide diffs to use to double-check that the changes the tool will make are what we expect. Let’s dig in.
The “Gather” Job
First, we need to gather the configuration from the tenant we want.
#!groovy
//will check if git branch exists, create if no. Pull down configs and check in the json file.
def prodApi = 'https://prod-example.instana.io/api'
def nonProdApi = 'https://nonprod-example.instana.io/api'
pipeline {
agent any
environment {
API_TOKEN = credentials('instana-endpoint-config-key')
}
stages {
stage('git branch'){
steps {
script {
echo "checking if branch ${BRANCH} exists"
try {
sh "git branch -a | grep '${BRANCH}'"
sh "git checkout ${BRANCH}"
}
catch (err) {
echo "branch does not exist, creating"
sh "git checkout -b ${BRANCH}"
}
}
}
}
stage('get rules'){
steps {
script {
def url = (env.TENANT == 'nonprod' ? nonProdApi : prodApi)
def text = sh(
script: "curl -H 'authorization: apiToken ${API_TOKEN}' ${url}/application-monitoring/settings/http-endpoint",
returnStdout: true
)
def rules = readJSON text: text
rules.each { rule ->
def service = readJSON text: sh(
script: "curl -H 'authorization: apiToken ${API_TOKEN}' '${url}/application-monitoring/services;id=${rule.serviceId}'",
returnStdout: true).trim()
rule.serviceName = service.label
rule.remove('serviceId')
}
writeJSON json: rules, file: 'endpointRules.json', pretty: 2
}
}
}
stage('push'){
steps {
script {
sshagent(['git-auth'], {
sh 'git add endpointRules.json'
sh "git commit -m 'rules added by Jenkins from ${BUILD_URL} ran by ${currentBuild.getBuildCauses('hudson.model.Cause$UserIdCause')[0].userName}'"
sh "git push -u origin ${BRANCH}"
})
}
}
}
}
}
There is a standalone Jenkins job that uses this Jenkinsfile, and expects 2 parameters, a branch name and the tenant. First, we check if the branch named exists. We create the branch if it doesn’t already exist, and then check it out. Next, in the get rules stage, we actually gather the rules from the tenant, and save them to the JSON file. Finally we commit the changes and push to origin.
The “Publish” Job
The next Jenkins job is a multi-branch pipeline job, listening to changes in the git repo.
#!groovy
//will deploy to the non-prod tenant for task branches, and prod tenant for master.
def prodApi = 'https://prod-example.instana.io/api'
def nonProdApi = 'https://nonprod-example.instana.io/api'
def rules
pipeline {
agent any
environment {
API_TOKEN = credentials('instana-endpoint-config-key')
}
stages {
stage('get current rules'){
steps {
script {
def url = (env.BRANCH_NAME == 'master' ? prodApi : nonProdApi)
def text = sh(
script: "curl -H 'authorization: apiToken ${API_TOKEN}' ${url}/application-monitoring/settings/http-endpoint",
returnStdout: true
)
rules = readJSON text: text
rules.each { rule ->
def service = readJSON text: sh(
script: "curl -H 'authorization: apiToken ${API_TOKEN}' '${url}/application-monitoring/services;id=${rule.serviceId}'",
returnStdout: true)
rule.serviceName = service.label
rule.remove('serviceId')
}
}
}
}
stage('reduce'){
steps {
script {
def newRules = readJSON file: 'endpointRules.json'
rules = newRules - rules
echo "update that will be run: ${rules}"
if (rules.isEmpty()){
echo "will skip updating because rules already match"
}
}
}
}
stage('update'){
when {
expression { !rules.isEmpty() }
}
steps {
script {
def url = (env.BRANCH_NAME == 'master' ? prodApi : nonProdApi)
rules.each { rule ->
def serviceResult = readJSON text: sh(
script: "curl -H 'authorization: apiToken ${API_TOKEN}' '${url}/application-monitoring/services?nameFilter=${rule.serviceName}'",
returnStdout: true)
if(serviceResult.totalHits == 0) {
serviceResult = null
} else if (serviceResult.totalHits == 1) {
serviceResult = serviceResult.items[0]
} else {
serviceResult = serviceResult.items.find { it.label == rule.serviceName }
}
if(!serviceResult) {
error "Cannot find service ${rule.serviceName}"
}
rule.serviceId = serviceResult.id
sh "curl -f " +
"-H 'authorization: apiToken ${API_TOKEN}' " +
"-H 'Content-Type: application/json' " +
"-X PUT " +
"-d '${rule.toString()}' " +
"'${url}/application-monitoring/settings/http-endpoint/${rule.serviceId}'"
}
}
}
}
}
}
This job also gets the current rules from a tenant, but here it compares them to what is in the JSON file. If the rules all match, then there is nothing to do, and the job says so. If the rules do not match, the job will update the target tenant with each new/updated rule.