Playing Healthchecks

Healthchecks in the Scala Play Framework

The Play framework for Scala is “marketed” as a lightweight framework used all over for folks building microservices and other quick and nimble services. While it may be fast and opinionated, it definitely does not set you up for success in a microservice world.

Microservice Necessities

Microservices need a few things to survive. Pertinent to this topic, they need to be observable. This means that they need to expose certain pieces of information about themselves that would allow an operator to determine if an instance of the service is healthy or not. If an instance is not healthy, the operator should take action to either restore the instance to a healthy state, or replace the instance with a new (hopefully healthier) instance.

Kubernetes specifically (and you must be using Kubernetes to say you’re doing microservices, right?) breaks down these monitoring probes into two categories, a Readiness probe, and a Liveness probe. Yes, technically there is also the startup probe, but we are not concerned with that one at the moment. The Readiness probe tells Kubernetes (the operator) that the service is ready to do work, in other words it is ready to be enrolled in a load balancer. The Liveness probe tells Kubernetes that the service is alive.

If a service instance fails the Readiness probe, Kubernetes will remove it from the load balancer, which should prevent the instance from getting new work, allowing the instance to finish work it currently has and become healthy enough to take on new work again.

If a service instance fails the Liveness probe, Kubernetes will terminate it, and restart it.

Healthcheck

What does the Scala Play framework give you for adding healthchecks to your service? Nothing. Nothing at all. You can perhaps find some vague notes online on adding Dropwizard metrics libraries for getting metrics, and that happens to come with a healthcheck component as well, but that’s it. Nothing out of the box. Nothing even from a supported plugin or anything like that.

I opted to go the Dropwizard route, because that’s really the only thing I could find information for.

Dropwizard

Firstly, we need to add dependencies to our project. We need metrics-core and metrics-healthchecks:

"io.dropwizard.metrics" % "metrics-core"         % "4.2.23",
"io.dropwizard.metrics" % "metrics-healthchecks" % "4.2.23"

Then update the application configuration to enable the healthcheck. We use an APM tool, so I did not enable any JVM or other instrumentation that is available.

metrics {
  enabled = true
  jvm = false
  logback = false
  healthchecks {
    enabled = true
  }
}

In my case, the only dependency I really want to check in my healthcheck is our Database for now, so I then added a check for that

package infrastructure

import com.codahale.metrics.health.HealthCheck
import scala.concurrent.duration._
import scala.concurrent.{Await, ExecutionContext, Future}
import play.api.db.DBApi


class DatabaseHealthCheck (db: DBApi)(implicit val ec: ExecutionContext) extends HealthCheck {
  private val timeoutDuration = 5.seconds
  override def check(): HealthCheck.Result = {
    val database = db.database("default")
    val future: Future[Boolean] = Future {
      database.withConnection { conn =>
        conn.isValid(1)
      }
    }
    try {
      if (Await.result(future, timeoutDuration)) {
        return HealthCheck.Result.healthy()
      } else {
        return HealthCheck.Result.unhealthy("default db is not healthy")
      }
    } catch {
      case _: java.util.concurrent.TimeoutException =>
        return HealthCheck.Result.unhealthy("default db is not healthy, timeout while obtaining db connection")
    }
  }
}

This will check the “default” database to make sure it can connect to it, waiting up to 5 seconds for a connection to return and be valid. If no connection is returned in that time, or the connection that comes back is not valid for whatever reason, the healthcheck will respond as unhealthy.

Next I need to register the endpoint. Inside my REST controller I created a healthcheck registry (not the best practice, but I didn’t really have a better place to construct this and register the DB healthcheck), and then added an endpoint to respond with the healthcheck status.

import com.codahale.metrics.health.HealthCheckRegistry
import infrastructure.DatabaseHealthCheck
import scala.collection.JavaConverters._
import scala.concurrent.{ExecutionContext, Future}
import play.api.libs.json.Json
import play.api.db.DBApi

...

class RESTController @Inject()(db: DBApi)(implicit val ec: ExecutionContext) extends InjectedController {
  val healthCheckRegistry = new HealthCheckRegistry()
  healthCheckRegistry.register("database", new DatabaseHealthCheck(db))

  ...

  def healthcheck: Action[AnyContent] = Action.async { _ =>
    Future {
      val healthCheckResults = healthCheckRegistry.runHealthChecks()
      val resultsAsJson = healthCheckResults.asScala.map { case (name, result) =>
        Json.obj(
          "name" -> name,
          "isHealthy" -> result.isHealthy,
          "message" -> result.getMessage
        )
      }

      if (resultsAsJson.forall(_("isHealthy").as[Boolean])) {
        Ok(Json.obj("status" -> "healthy", "checks" -> resultsAsJson))
      } else {
        logger.warn("healthcheck results ${healthCheckResults.toString()}")
        ServiceUnavailable(Json.obj("status" -> "unhealthy", "checks" -> resultsAsJson))
      }
    }
  }

  ...
}

Expose that endpoint in routes:

GET         /healthcheck                                 controllers.RESTController.healthcheck

And now my service has a healthcheck that I can then use for my Kubernetes readiness probes!