Fintrospect

overview

Fintrospect is a Scala web-framework with an intelligent HTTP routing layer, based on the Finagle RPC framework from Twitter. Via a shared contract, it provides a simple way to implement fast web service endpoints and HTTP clients which are:

• Type-safe : auto-marshalls all request parameters/bodies into the correct types (including primitives + JSON/XML etc...)
• Auto-validating : the presence of required and optional request parameters and bodies are checked before entering service-layer code
• Auto-documenting : runtime generation of endpoint documentation such as Swagger JSON or web sitemap XML. Generates JSON Schema for example object formats to be included in these API docs.
• Uniform : reuse the same contract to define both incoming or outgoing Finagle HTTP services. This also allows extremely low effort fake servers to be created

Additionally, Fintrospect provides a number of mechanisms to leverage these routes:

• Easily build type-safe HTTP responses with a set of custom builders for a wide variety of message formats:
  • JSON: Argo, Argonaut, Circe, GSON, Jackson, Json4S, Play JSON, Spray JSON
    • Auto-marshaling of case classes instances to/from JSON (for Argonaut/Circe/Json4S/Play).
    • Implement simple PATCH/PUT endpoints of case class instances (Circe only).
  • Native implementations of XML, Plain Text, HTML, XHTML
  • MsgPack binary format
• Serve static content from the classpath or a directory
• Template View support (with Hot-Reloading) for building responses with Mustache or Handlebars
• Anonymising headers for dynamic-path based endpoints, removing all dynamic path elements. This allows, for example, calls to particular endpoints to be grouped for metric purposes. e.g. /search/author/rowling becomes /search/author/{name}
• Interacts seamlessly with other Finagle based libraries, such as Finagle OAuth2
• Utilities to help you unit-test endpoint services and write HTTP contract tests for remote dependencies

broad concepts

The main concepts in play:

• RouteSpec: defines the self-describing contract of an HTTP endpoint
• ServerRoute: the binding of a RouteSpec to a Finagle Service implementing some business-logic, to create an inbound HTTP endpoint
• RouteClient: the binding of a RouteSpec to a Finagle Service representing an HTTP client, to create a simple function for making outbound HTTP calls
• ParameterSpec: defines the acceptable format for a request parameter (Path/Query/Header/Form-field). Provides the auto-marshaling mechanic for serializing and deserializing objects to and from HTTP messages
• BodySpec: similar to ParameterSpec, but applied to the body of an HTTP message
• RouteModule: defines a set of ServerRoute instances which are grouped under a particular request path. These modules can be combined and then converted to a Finagle Service and attached to a Finagle HTTP server. Each module provides an endpoint under which it's own runtime-generated documentation can be served (eg. in Swagger format)
• ResponseSpec: defines the characteristics of a possible response generated by a ServerRoute

regarding finagle

Since Fintrospect is built on top of Finagle, it's worth acquainting yourself with its concepts, which can be found here.

&tldr; Finagle Primer

1. Finagle provides protocol-agnostic RPC and is based on Netty
2. It is mainly asynchronous and makes heavy usage of Twitter's version of Scala Futures
3. It defines uniform Service and Filter interfaces for both client and server APIs that are effectively a single method...

   Service: def apply(request : Request) : Future[Response] Filter: def apply(request : RequestIn, service : Service[RequestOut, ResponseIn]) : Future[ResponseOut] 

4. Filters can be chained together and then applied to a Service, which results in another Service. This is useful to apply layers of functionality such as caching headers, retry behavior, and timeouts.

a note on style

• The code in this guide has omitted imports that would have made it read more concisely. The sacrifices we make in the name of learning... :)

installation

Fintrospect is intentionally dependency-lite by design - other than Finagle, the core library itself only has a single non-org.scala dependency.

To activate some optional features, additional dependencies may be required - these are shown in the subsequent sections depending on the use-case.

core

Add the following lines to build.sbt - the lib is hosted in Maven Central and JCenter:

resolvers += "JCenter" at "https://jcenter.bintray.com" libraryDependencies += "io.fintrospect" %% "fintrospect-core" % "15.1.0" 

message formats

For consuming  request parameters  in a type-safe way or using the relevant  HTTP response builders add the following:

templating

For utilizing the templating engines listed below, add the following:

request parameters & bodies

Fintrospect broadly abstracts the various parts of an HTTP Request Path, Header, Query and Body behind a common interface to which the main functions are:

1. Retrieve a valid value from an incoming request in a type-safe way using <--() or from()
2. Bind a specified value into an outgoing request in a type-safe way using -->() or of()

Parameters are created in a uniform way using the objects Path, Header, Query, FormField and Body. The general form for definition is as follows, although since Path and Body parameters are always required, the middle step is omitted:

<parameter location class>.<required|optional>.<param type>("<name>") 

Descriptions can be attached to these definitions for documentation purposes. Note the retrieved type for the optional param:

val anniversary = Header.required.localDate("anniversary", "the date you should not forget! format: yyyy-mm-dd") val myAnniversary: LocalDate = age <-- request val age = Header.optional.int("age", "your age") val age: Option[Integer] = age <-- request val ohDear = Body.xml("soapMessage", "This isn't Simple") val ohDearDear: Elem = anniversary <-- request 

There are convenience methods for a standard set of "primitive" types, plus extensions for such as native Scala XML, Forms and JSON.

Additionally, there is another form for parameters which can appear multiple times in a request - simply insert the *() method in the chain:

val kidsBirthdays = Query.required.*.localDate("birthdays", "the dates you should not forget! format: yyyy-mm-dd") val ourKidsBirthdays: Seq[LocalDate] = kidsBirthdays <-- request 

forms

These represent a slightly special case: you first need to retrieve the form from the request, and then the fields from the form.

val name = FormField.required.string("name") val isMarried = FormField.optional.boolean("married") val form = Body.form(name, isMarried) val myForm = form <-- request val myName = name <-- myForm val iAmMarried = isMarried <-- myForm 

working with custom parameter types

Custom parameter and body types can be implemented by defining a ParameterSpec or BodySpec and passing an instance instead of calling the <param type> method in the form above. These Spec objects define:

• name and description of the entity type being handled
• for Parameters: the higher level ParamType of the on-the-wire representation. For custom formats, this is StringParamType, although ObjectParamType should be used for JSON appearing in an request body
• for Bodies: the higher level ContentType of the on-the-wire representation.
• functions representing the serialization and deserialization from the String format that comes in on the request. Note that we are only concerned with the happy case on-the-wire values. These throw exceptions if unsuccessful - these are caught by the request validation mechanism and turned into a rejected BadRequest (400) response which is returned to the caller.

An example for a simple domain case class Birthday:

case class Birthday(value: LocalDate) val birthdayAsAQueryParam = Query.required(ParameterSpec.localDate().map(Birthday(_), (b:Birthday) => b.value), "DOB") val birthdayAsABody = Body(BodySpec.string(ContentTypes.TEXT_PLAIN) .map(s => Birthday(LocalDate.parse(s)), (b:Birthday) => b.value.toString), "DOB") 

usage of JSON libraries

Fintrospect comes with binding support for several JSON libraries, each represented by static instances of JsonFormat on a class named after the library in question. When defining a Body or Parameter, if required you can also specify the library format for it to use (else it will default to the bundled Argo JSON library) - and if this is done centrally then you can switch out JSON libraries with only a single line of code change.

val jsonFormat = Argonaut.JsonFormat val exampleObject = jsonFormat.obj("fieldName" -> json.string("hello")) val json = Body.json(Option("my lovely JSON object"), exampleObject, Argonaut) val body: Json = json <-- request 

Notice that in the above we specified an example of the JSON message. This is not mandatory, but allows the generation of JSON Schema to be included in the auto-generated API documentation.

Additionally, in the case of some JSON libraries that provide auto marshaling and unmarshalling to case class instances, you can remove the JSON step altogether:

case class Email(address: String) val email = Body(Argonaut.bodySpec[Email](Option("an email address")), Email("jim@example.com")) val retrieved: Email = email <-- request 

defining routes

A RouteSpec() call starts to define the specification of the contract (in terms of the required parameters) and the API follows the immutable builder pattern. Apart from the path-building elements (which terminate the builder), all of the "builder-y" calls here are optional, as are the descriptive strings (used for the auto-documenting features). Here's the simplest possible REST-like example for getting all employees in a notional system:

RouteSpec().at(Method.Get) / "employee" 

Notice that the request routing in that example was completely static? If we want an example of a dynamic endpoint, such as listing all users in a particular numerically-identified department, then we can introduce a Path parameter:

RouteSpec("list all employees in a particular group").at(Method.Get) / "employee" / Path.integer("departmentId") 

... and we can do the same for Header and Query parameters; both optional and mandatory parameters are supported, as are parameters that can appear multiple times.:

RouteSpec("list all employees in a particular group") .taking(Header.optional.boolean("listOnlyActive")) .taking(Query.required.*.localDate("datesTakenAsHoliday")) .at(Method.Get) / "employee" / Path.integer("departmentId") 

Moving onto HTTP bodies - for example adding an employee via a HTTP Post and declaring the content types that we produce (although this is optional):

RouteSpec("add employee", "Insert a new employee, failing if it already exists") .producing(ContentTypes.TEXT_PLAIN) .body(Body.form(FormField.required.string("name"), FormField.required.localDate("dateOfBirth"))) .at(Method.Post) / "user" / Path.integer("departmentId") 

... or via a form submission and declaring possible responses:

RouteSpec("add user", "Insert a new employee, failing if it already exists") .body(Body.form(FormField.required.string("name"), FormField.required.localDate("dateOfBirth"))) .returning(Created -> "Employee was created") .returning(Conflict -> "Employee already exists") .at(Method.Post) / "user" / Path.integer("departmentId") 

using routes

As can be seen above, there are several stages to defining a route. Here is the complete construction lifecycle:

1. Create a RouteSpec with a name and description
2. Add details of parameters, any body, media-types and possible responses
3. Finalize the RouteSpec with a call to at(). This creates an UnboundRoute.
4. Continue to add static or dynamic Path parameters to the URL structure (creating UnboundRoute<n> instances).

Once the final UnboundRoute has been created (with all of it's Path parts declared), it represents an HTTP contract, which can then be bound to:

1. an HTTP server Service if you wish to serve that contract to other systems.
2. an HTTP client Service if you wish to consume that contract from a remote system.

server routes & modules

A RouteSpec needs to be bound to a standard Finagle Service to receive requests, in order to create a ServerRoute. Since Finagle services are very lightweight, we can create a new instance of the Service for every request, and bind the RouteSpec to a factory method which receives the dynamic Path parameters and returns the Service. Other parameters can be retrieved directly in a type-safe manner from the HTTP request by using <--() or from() method on the parameter declaration.

validation

The presence and format validity of ALL parameters which are attached to a RouteSpec is verified by Fintrospect before requests make it to this bound Service, so no validation code is required. The response returned to the client is:

• Not Found 404: if there are any Path params which are missing or invalid (all are required)
• Bad Request 400: if there are any Header, Query, or Body params are missing (required only) or invalid

simple example

val holidays = Query.required.*.localDate("datesTakenAsHoliday") val includeManagement = Header.optional.boolean("includeManagement") def findEmployeesOnHoliday(departmentId: Integer) = Service.mk[Request, Response] { request => vale holidayDates: Seq[LocalDate] = holidays <-- request val includeManagementFlag: Option[Boolean] = includeManagement <-- request val response = Response(Ok) val baseMsg = s"Everyone from department $departmentId was at work on $holidayDates" response.contentString = baseMsg + (if (includeManagementFlag.getOrElse(false)) "" else ", even the management") Future(response) } val route = ServerRoute[Request, Response] = RouteSpec() .taking(holidays) .taking(includeManagement) .at(Method.Get) / "employee" / Path.integer("departmentId") bindTo findEmployeesOnHoliday 

modules

A Module is a collection of ServerRoute that share a common URL context, which is built up from the Root object. Add the routes and then convert into a standard Finagle Service object which is then attached in the normal way to an HTTP server.

def listEmployees(): Service[Request, Response] = Service.mk(req => Future(Response())) Http.serve(":8080", RouteModule(Root / "employee") .withRoute(RouteSpec("lists all employees").at(Method.Get) bindTo listEmployees) .toService ) 

Modules with different root contexts can also be combined with one another and then converted to a Service:

RouteModule(Root / "a").andThen(RouteModule(Root / "b")).toService 

self-describing Module APIs

A big feature of the Fintrospect library is the ability to generate API documentation at runtime. This can be activated by passing in a ModuleRenderer implementation when creating the RouteModule and when this is done, a new endpoint is created at the root of the module context (this location is overridable) which serves this documentation.

Bundled with Fintrospect are:

• Swagger (1.1 and 2.0) JSON, including JSON Schema models
• A simple JSON format
• Sitemap XML format

Other implementations are pluggable by implementing the ModuleRenderer trait - see the example code for a simple XML implementation.

val service = RouteModule(Root / "employee", Swagger2dot0Json(ApiInfo("an employee discovery API", "3.0"))).toService Http.serve(":8080", new HttpFilter(Cors.UnsafePermissivePolicy).andThen(service)) 

Note above the usage of the Finagle CorsPolicy filter, which will allow the services to be called from a Swagger UI - without it, the server will reject any cross-domain requests initiated inside a browser.

security

Module routes can be secured by adding an implementation of the Security trait - this essentially provides a filter through which all requests will be passed. An ApiKey implementation is bundled with the library which returns an 401 Unauthorized HTTP response code when a request does not pass authentication.

RouteModule(Root / "employee") .securedBy(ApiKey(Header.required.string("api_key"), (key: String) => Future(key == "extremelySecretThing"))) 

client routes

A RouteSpec can also be bound to a standard Finagle HTTP client Service and then called as a function, passing in the parameters which are bound to values by using the -->() or of() method. The client marshalls the passed parameters into an HTTP request and returns a Twitter Future containing the response. Any required manipulation of the Request (such as adding timeouts or caching headers) can be done in the standard way by chaining a Filter to the client Service. Note that Content-Type headers for posted HTTP bodies is already handled by the bound Body instance.:

val employeeId = Path.integer("employeeId") val name = Query.required.string("name") val client: RouteClient = RouteSpec() .taking(name) .at(Get) / "employee" / employeeId bindToClient Http.newService("localhost:10000") val response: Future[Response] = client(employeeId --> 1, name --> "") 

super-cool feature time: reuse of HTTP contracts

Because the RouteSpec objects can be used to bind to either a Server or a Client, we can be spectacularly smug and use them on both sides of an HTTP boundary to provide a type-safe remote contract, to either:

1. Auto-generate fake HTTP server implementations for remote HTTP dependencies. In this case, defining the RouteSpec as part of an HTTP client contract and then simply reusing them as the server-side contract of a testing fake - see the /clients example code
2. Use a shared library approach to define a contract and the data objects that go across it for reuse in multiple applications, each of which import the shared library. This obviously binary-couples the applications together to a certain degree, so utmost care should be taken, backed up with sufficient CDC-style testing to ensure that the version of the contract deployed is valid on both ends.

building http responses

It's all very well being able to extract pieces of data from HTTP requests, but that's only half the story - we also want to be able to easily build responses. Fintrospect comes bundled with a extensible set of HTTP Response Builders to do this. The very simplest way is by using a ResponseBuilder object directly...

ResponseBuilder.toFuture( ResponseBuilder.HttpResponse(ContentTypes.APPLICATION_JSON).withCode(Status.Ok).withContent("some text").build() ) 

However, this only handles Strings and Buffer types directly. Also bundled are a set of bindings which provide ResponseBuilders for handling content types like JSON or XML in a set of popular OSS libraries. These live in the io.fintrospect.formats package. Currently supported formats are in the table below:

The simplest (least concise) way to invoke an auto-marshaling (ie. typesafe) ResponseBuilder is along the lines of:

Xml.ResponseBuilder.HttpResponse(Status.Ok).withContent(<xml>lashings and lashings of wonderful</xml>).toFuture 

... although with tiny bit of implicit magic, we can use custom status methods on the builders and then convert the ResponseBuilder to a Future[Response], you can reduce this to the rather more concise:

import io.fintrospect.formats.Xml.ResponseBuilder._ val responseViaImplicits: Future[Response] = Ok(<xml>lashings and lashings of wonderful</xml>) 

These ResponseBuilders also support AsyncStream[T], so you can build services which can stream responses in a typesafe way.

taking advantage of auto-marshalling

controlled mode

Some of the JSON libraries (Circe, Argonaut, Json4S, Play) supported by Fintrospect support auto-marshaling of Scala Case class instances directly to JSON without any custom conversion code needing to be written. This is supported by encode() and decode() methods present on the relevant Fintrospect JsonFormat format instance (e.g. io.fintrospect.formats.Circe.JsonFormat). Generally, these are very simple to use:

case class EmailAddress(address: String) import io.circe.generic.auto._ import io.fintrospect.formats.Circe.ResponseBuilder._ Status.Ok(Circe.JsonFormat.encode(EmailAddress("dev@fintrospect.io") 

The auto-marshaling functionality of these JSON libraries requires implicit parameters in order to make it work. This requirement is echoed in the signatures of the relevant Fintrospect encode() and decode() methods, but unless you're going to be providing custom encoder/decoder instances, you can get away with just importing the relevant implicit params from the parent lib, as in the example above.

full-auto mode

Fintrospect also contains filters which allow you to abstract away the HTTP Request/Response entirely. In this example, the Circe.Filters.AutoInOut filter converts the Service[Request, Response] to a Service[EmailAddress, ReversedEmailAddress], auto-converting the case class objects in and out of the request/response. The returned status code in the Response is 200, but this is overridable:

import io.circe.generic.auto._ import io.fintrospect.formats.Circe import io.fintrospect.formats.Circe.Auto._ case class ReversedEmailAddress(sserdda: String) val domainSvc = Service.mk[EmailAddress, ReversedEmailAddress] { email => Future(ReversedEmailAddress(email.address.reverse)) } val httpSvc: Service[Request, Response] = Circe.Auto.InOut(domainSvc) 

cross-field validation

Further to the process of retrieving them from the request, there exists the problem of validating that the passed parameters are actually logically valid when passed together. For example, a date range is only valid when the start date is before the end date.

For this purpose, you can use an Extractor - a trait which provides a single method <--?() to return one of 2 states: Extracted(Option(value)) and ExtractionFailed(parameter) for missing or invalid values. These constructs can be used inside a for comprehension to provide cross-field validation, and eventual creation of a target object. Below is a service that implements this logic - note the use of a predicate and a failure reason which provides the logic for the check:

case class DateRange(startDate: LocalDate, endDate: Option[LocalDate]) val range: Extractor[Request, DateRange] = Extractor.mk { (request: Request) => for { startDate <- Query.required.localDate("start") <--? request endDate <- Query.optional.localDate("end") <--?(request, "end date invalid", _.isAfter(startDate.get)) } yield DateRange(startDate.get, endDate) } val route = RouteSpec().at(Get) bindTo Service.mk { req: Request => range <--? req match { case Extracted(Some(dates)) => Ok(dates.startDate + " ->" + dates.endDate) case ExtractionFailed(sp) => BadRequest(sp.mkString(", ")) } } 

(Although we are calling get() on the Options in the above yield - normally a VBT (Very Bad Thing), in this case we are safe to do so since any missing or invalid mandatory parameters short-circuit the comprehension.)

Extractable is modular, so instances can be embedded inside each other for comprehensions to build object graphs from the incoming request.

The above example can be further simplified by use of the built-in RequestFilters.ExtractableRequest filter to transform the input:

 RequestFilters.ExtractableRequest(range).andThen(Service.mk[DateRange, Response] { dateRange => ... }) 

templating

Templates are applied by using a custom RenderView filter to convert View instances into standard Http Responses. Simply implement the View trait and then put a matching template file onto the classpath, and chain the output of the model-creating Service into the filter. You can do this for entire modules by making the RouteModule itself generified on View by using the templating Filter as a Module-level filter:

case class ViewMessage(value: String) extends View val showMessage = Service.mk[Request, View] { _ => Future(ViewMessage("some value to be displayed")) } val renderer = if(devMode) MustacheTemplates.HotReload("src/main/resources") else MustacheTemplates.CachingClasspath(".") val webModule = RouteModule(Root / "web", new SiteMapModuleRenderer(new URL("https://root.com")), new RenderView(Html.ResponseBuilder, renderer)) .withRoute(RouteSpec().at(Get) / "message" bindTo showMessage) 

redirects

After Form posts, it might be desirable to return an HTTP redirect instead of a View in the case of success. For this purpose, use an instance of the View.Redirect class. The location and the status code (default 303) are configurable:

val redirect = Service.mk[Request, View] { _ => Future(View.Redirect("https://my.server/myRoute")) } 

Available implementations of the TemplateRenderer are (see the relevant implementation of Templates):

• cached from the classpath
• cached from the filesystem
• hot-reloading from the filesystem

Similarly to how the ResponseBuilder codecs work, no 3rd-party dependencies are bundled with Fintrospect - simply import the extra SBT dependencies as required:

static content

Static files can easily be served from the either the Classpath or a Directory by using an instance of StaticModule with an appropriately injected ResourceLoader:

val cpModule = StaticModule(Root / "public", ResourceLoader.Classpath("package/path")) val dirModule = StaticModule(Root / "public", ResourceLoader.Directory("file/dir/path")) 

Note that due to security concerns, ResourceLoader.Classpath should be configured to use a package which does NOT overlap with class assets.

testing

routes

Provided trait Testing introspect Route can be used to unit test your routes, as in the simple example below:

object EchoRoute { val route = RouteSpec().at(Method.Get) / Path.string("message") bindTo( (message: String) => Service.mk { req: Request => Future(PlainText.ResponseBuilder.OK(message)) }) } class EchoRouteTest extends FunSpec with Matchers with TestingFintrospectRoute { override val route = EchoRoute.route describe("Echo") { it("bounces back message") { responseFor(Request("hello")).contentString shouldBe "hello" } } } 

test http server

The TestHttpServer is convenient for attaching routes during development, or can be used as a scaffold to provide automatically generated fake servers for downstream dependencies - simply complete the stub implementation of the server-side and fire it up. This works especially well if you are utilizing custom serialization formats (such as one of the auto-marshaling JSON libraries), as there is absolutely no marshaling code required to send back objects over the wire from your stub.

val route = RouteSpec().at(Get) / "myRoute" bindTo(() => Service.mk {r => Future(Response(Status.Ok))}) new TestHttpServer(9999, route).start() 

examples

The Fintrospect example code hosted here:

• /circe- using the Circe library for auto-marshaling of case classes to JSON
• /clients- an example of using HTTP routes as a client and creating really easy fake remote servers for testing
• /custom formats- creating a custom message format with it's own Response Builder: HipsterXML
• /custom parameters- creating a custom serialization format for request parameters and bodies
• /extended- the library application. Example used on the Github project README
• /formvalidation- shows form validation, with feedback given to the user
• /oauth- shows the usage of plugging in an OAuth2 library to change the input request type
• /msgpack- using the MsgPack binary message format, including auto-marshaling of case classes in/out of responses
• /streaming- shows streaming of responses in a typesafe way
• /strict contenttypes- shows how to serve multiple content types from a single route
• /templating- shows the usage of plugging in Mustache templating library to change the output response type
• /validation- shows cross-field validation of inputs taken from the request

quickstart repo

If you want to dive in head-first, there is example source project of a Fintrospect application with most of the features hosted in this Github Repo

cookbook

This contains a set of cookbook recipes that may be useful to see how particular features are activated.

Eventually, these will be documented with explanations on the website, but for the moment, they hopefully are still useful