REST APIs and restaurants [Take II*]

[Level C3]

 - "... and sauteed with the finest aromatic saffron - we import directly from our suppliers in Tibet - and garnished with the finest quality ocra from the heart of Morocco and ..."

- : ".. hmmm, bu..."

- : " and Sir, the King Shrimps day fresh from our suppliers in Belgium mildly cooked with a hint of lime zest and aromatic Thai basil snuggled with a swirl of Cornish Creme Fraiche and blossoms of Burmese jasmine and ..."

- : "... I mean, ..."

- : " and I should remind you that our head chef has used a special blend of exotic spices, marinated for 42 minutes exact, to create a sensual oriental effect of ... "

- : "I REALLY DON'T CARE! This stuff is awful!"

⚑  ⚑  ⚑

In more than one way a REST API is similar to a restaurant. They both hide a very complex process of accomplishing a service (serving food or data) behind a nice, warm and welcoming presentation. And we need to bear in mind that presentation is the key: no matter what goes on behind the scene in the kitchen, the food and its presentation is the key. And your API is the menu, the door to the services provided by the server. Whatever goes on in the kitchen (your server's internal domain) is not the concern of your customers as long as the output and presentation is good.

Commonly, the perspective in which we look at our services is wrong. They are server-centric. [I have talked before about the definition of server (and client) and their interaction] In brief, when we talk about the server, this is what comes in mind (as per CSDS):

This view is useful when building a server. It highlights inherent complexity of the server and how it should hide its complexity behind its API. But it is way too easy to get attached to the bird's eye view of the server where all internals of the server is visible.

And it is too easy to forget that this is how the client sees the server:

Client perspective of a server, where it sees a black box (here it's blue!) and
the gateway to the server is through its API.

When talking of the domain in an API, we are very likely to mean the server's private domain - its internals and complex process. Reality is server's internal domain does not surface beyond the API layer. API Layer is responsible to define resources which in turn define a coherent public domain composed of API and its messages - abstracted away from server's internal domain. If the server is an online retail API, I can create, edit or cancel orders. The fact that this is achieved through contact to many suppliers and their own custom processes is not a concern of the client.

In fact, in the same way that kitchen of a restaurant and its dining area are completely abstracted away, server's public domain is an abstraction on the top of server's internal domain. This is good for the chef and the customer. It is true the food that is served comes out of the kitchen but the whole process of preparation is irrelevant for the customer. 

Why talking of server's public domain is important? Because we commonly look down on the public domain. When we talk of the DTO messages passed to and fro, we regard them as View Models so they are not even proper models, they are just DTOs, no behaviour just some property bag. We use a mapper to strip down the sensitive data from our real models and spit out the view models to the client. And when we talk about DDD, we only mean server - we think view models do not deserve the love of care of DDD.

But this results in a brittle API that has to be changed by every change of the server. It becomes a cut down version of the server's internal API. We all know examples of good and bad API but too many changes in an API is a sign of bad design usually from Server Chauvinism. So always do a separate DDD for your server's public domain. It might sound strange since I am in favour of no-business-logic-in-api-layer. Well, it is true - we do not implement any business logic in the API Layer but the biggest responsibility of API Layer is talking HTTP. This translates to defining resources in REST.

So practice DDD in your API Layer. Define its aggregate roots, entities and value types - it is simplistic to think such concepts can be entirely hidden from the clients. Specify the commands and queries - with commands usually being the payload of the PUT, DELETE and POST and queries usually expressed in the URL itself to make them cache friendly.

What we see in the above picture define server's public domain: it is API (set of resources), input messages (commands and queries) and output messages (entities and value types). The case of queries - as we said - is a bit special since it is normally expressed in URL. This has the benefit of cacheability in contract with sending a payload (using other methods) which not suitable for caching. As an example, here is twitter's API for getting status timelines:

Get /1.1/statuses/user_timeline.json?screen_name=twitterapi&count=2

Defining parameters as query strings is useful as they can also imply optionality. One downside to this approach is the fact that ordering of parameters cannot be expressed in query string and while any combination can be valid, semantically equal combinations will be considered different resources and cached separately. For example, screen_name=twitterapi&count=2 and screen_name=twitterapi&count=2 are semantically equal while they are considered different resources. This reduces effectiveness of caching for GET requests. For this reason, URL segments separated by / are superior although in a big API can result in prohibitive complexity of routing.

Items returned in this API are Entities. it is evident by having an ID (id_str) which is returned as part of data.

Looking at what each timeline contains, we can see list of urls, where each url returned is a Value Type. It has three properties: url, indices and expanded_url.

In the case of commands, we have a similar situation. Looking at an example in twitter, tweeting (sending a tweet) is achieved using this command:

POST /1.1/statuses/update.json

status=Maybe%20he%27ll%20finally%20find%20his%20keys.%20%23peterfalk

Perhaps a better implementation would have been to omit the update.json totally - as this naming has an RPC smell. Using singular or plural is mainly down to taste - although I prefer singular. The command can be expressed by the simple class below:

Not all commands have to have a payload. Deleting a tweet can get send an ID in the URL with the DELETE command to /1.1/statuses/update.json although actual twitter API uses a POST instead.

Resources are basically all that can be accessed through the API although a more useful definition is the grouping around identities. For example Flickr API defines cameras, people, photos, photo comments, etc. It is true that Flickr is not considered a good example of a REST API when it comes to HTTP semantics, however, it does a good job in defining a coherent domain represented by resources while hiding its internals. A simple example is how it sends you previous and next photo instead of letting the client guess what these are based on the internal implementations.

In brief, it is responsibility of the API to create a coherent set of resources that can layout a cohesive picture in front of the client.

Common Anti-Patterns

As we said, the main danger is to have a server-centric perspective and neglect the public domain of your API.

While performing appropriate-sized DDD exercise is important, we need to be careful not expose too much of the server. One common anti-pattern is exposing server's internal to outside world. OData is a RESTful protocol which defines semantics for querying (mainly tabular) data store.

GET /api/products?$filter=name eq 'Gizmo 3'&$orderby=price desc&$top=1

While OData itself is not necessarily prescribing exposing server's internals, most of the implementations are actually built straight on the top of an RDBMS exposing its internal schema and and potentially allowing ad-hoc changes using SQL-like commands rather than domain-rich commands. Netflix who initially warmed up to the idea of exposing its films database to the public, reached the conclusion this is not such a RESTful idea and retired its OData service.

Another issue is the server-centric design: server assuming how its services will be used. I believe (Sorry Roy!) HATEOAS touches the subjects discussed here. When we talk of Hypermedia As The Engine OF Application State, which application do we mean? OK, let's consider Twitter's API. How can you think same hypermedia can equally drive the state of a twitter client, a game that posts scores on twitter, 4Square app that declares you a mayor on twitter or a twitter realtime map showing location of tweets using the firehose data pipe? The relationship from clients to servers are not one-to-one any more and effectiveness of HATEOAS is questionable. Application is defined by the client and not the server, as such server's hypermedia cannot be the engine of the application.

At the end of the day, none of us want to sound like the pompous chef who bores the customers with self-flattery. Server serves, so design it for servant-hood.


* this post was originally posted slightly differently under the name: "Server Chauvinism - API's public domain"

Using PocoHttp to consume Classic OData services

Introduction

[Level T2] In the last post, I introduced PocoHttp which I have been working on. As I promised there, we can use it to access classic OData services in addition to as much as OData support we have in ASP.NET Web API. OData functionality currently provided in PocoHttp is limited at the moment but I will be building upon.

[UPDATE: PocoHttp package v0.1 is not available in NuGet. Just type in the package manager console PM>Install-Package PocoHttp or search for PocoHttp]

Why OData?

During the last  few years, Microsoft has invested a lot in OData. Some other technologies have also been built upon it such as RIA services which is heavily used in the Silverlight world. So Microsoft has been pushing OData and marketing it as an open protocol but reality is outside Microsoft, adoption has been nill. I believe this has to do with issues regarding query syntax and paylaod.

Firstly, as we talked about before, community believes that OData services are not RESTful as query syntax is unique to OData (Darrel explains here). It also exposes the bowels of the server to the client which does not respect REST's client-server model.  Personally I believe there is a place for some of the simpler query constructs of OData such as top and skip that could be implemented by non-OData services as well.

And payload is limited to specific data structures in XML and JSON. As for XML payload, I think AtomPub as data wrapping format is heavy and inefficient - not to say that it has not been realised of recent. Sending metadata to the client that consumes the data every second is not a very good idea - metadata was used once to build the client but the client is eternally doomed to receive that information until the day sun stops shining (there might be a way to turn it off but that is not the default design). So it is like a web service that sends its WSDL all the time.

JSON payload is definitely lighter - it contains some metadata but nothing to be concerned about. The JSON format has changed between versions 1 and 2 of OData and server could be sending V1 or V2 (etc) depending on the content. There is DataServiceVersion custom header that can be used to get the version number from the response but I think as a canonical HTTP implementation, it should have been implemented as part of content-type:

Content-Type: application/json;DataServiceVersion=2.0

Or

Content-Type: application/odata2.0+json

Anyhow, that is not a big problem and I will be using JSON format in this sample. You can ask for JSON using an Accept header or appending $format=json.

OK, with all this so why OData then? For two reasons.

First of all, ASP.NET team are working to support alternative formats that do not have all the extra metadata: plain JSON or XML. So OData is going to live much longer in an improved implementation.

Second reason is there is already a considerable adoption inside the Microsoft community. Many teams while moving to new shiny Web APIs might still have an existing OData services that they need to use along with the new ones. 

Why use PocoHttp to access OData?

While OData has the problems we talked about, it is not short of one thing: tools and SDKs. You can easily use DataService<T> to consume OData services and the API even allows for automatic generation of data models for Entity Framework. You are fully abstracted from OData itself while using its all benefits.

So why would you ever want to use anything other than the existing tools to comsume OData services? Well, a few reasons.

With adoption of Web API increasing (which can be seen from the enthusiasm in the community), you will most likely start using Web APIs alongside OData ones. You would be building custom handlers to take care of authentication, tracing, caching, auditing, exception handling, etc and start to rely on them. But you would not be able to use these them with existing client tools since you need a client that is built upon the new HTTP pipeline in System.Net.Http.dll. That is PocoHttp!

Also you might want to harness the extreme powers that the client has in terms of queries it can run (which we touched in the last post) and limit them to a few handful essential but safe constructs - protecting your database and services. Then again that is one of the goals of PocoHttp.

New sample on the block

So I have created this sample on the PocoHttp that connects to OData sample services exposing Northwinds database. 

This basically sets up PocoClient to retrieve data in JSON format. Since the version of the JSON could be V1 or V2, the code needs to be able to support both.

Show us some code

OK! Let's look at request setup. There are two ways to request for JSON: by setting the Accept header or by appending $format=json:

// 1
pocoConfig.RequestSetup = (request) => request.Headers.Add("Accept", "application/json");

// 2
pocoConfig.RequestSetup = (request) => request.RequestUri = new 
   Uri(request.RequestUri.ToString() + "&$format=json",UriKind.Relative);

PocoConfiguration is a set of configuration parameters which is passed to the client constructor. One the parameters is RequestSetup which is an Action<HttpRequestMessage> that allows for the request to be modified before sending it to the server.

On the way back we use ResponseReader function to read the content into the objects. It receives the response, formatters and element types and must return objects it has read (will be IEnumerable<T>)

public Func<HttpResponseMessage, IEnumerable<MediaTypeFormatter>, Type, Task<object>> 
   ResponseReader { get; set; }

So this is a property sitting on the PocoConfiguration. PocoHttp provides a default implementation but since the data we get back is wrapped inside OData top level envelopes, we have to provide our own implementation. We need to define Payload<T> types that represent the structure of the data we are receiving. I have defined one for V1 and one for V2.


The gist of what we need to do is:

1. Create a generic type of the entity using MakeGenericType
2. Call non-generic ReadAsAsync passing the type
3. Use reflection on the result to read off the properties to unwarp and get to the entities we need

For example, the code for V1 looks like this:

private static object ReadFromV1(HttpResponseMessage response, 
 IEnumerable<MediaTypeFormatter> formatters, Type type)
{
 var outermostType = typeof(PocoHttp.Samples.ClassicOData.V1.Payload<>)
  .MakeGenericType(type);
 var result = response.Content.ReadAsAsync(outermostType, formatters).Result;
 return outermostType.GetProperty("d").GetValue(result, null);
}

In the example, I have used only Take and Skip but you can use direct Where clauses as well.

Conclusion

While ASP.NET Web API is the future of providing services and data, there is a considerable number of existing OData services that still need to be consumed. 

While existing OData client tools expose full features of OData abstracting the client from HTTP detailed, we outlined two reasons why you would want to use PocoHttp instead to consume these services: re-using HTTP pipeline handlers and limiting the client in terms of queries it can run so that the.

Introducing PocoHttp: Consuming HTTP data services

Introduction

[Level T2] PocoHttp is a non-opinionated Open Source .NET library for seamless consumption of HTTP data services using a familiar IQueryable<T> interface. Version 0.1 of the library is now available on GitHub - NuGet package to follow soon.

This post explains the background and motivation for this library as well as its usage and possible future directions.

Motivation and background

ASP.NET Web API exposes full richness of the HTTP spec. As such, many new avenues have been opened for creating HTTP client-server applications. Having said that, with HTTP spec aimed to be Touring-Complete, there is a client burden involved in implementing clients capable of deep protocol coherency.

ASP.NET Web API has made it easy to achieve protocol coherency as part of the HTTP pipeline - modelled as Russian Dolls (see Part 6 of the series).

With a lot of WCF services (that commonly were serving domain's value objects) soon to be moved to Web API, there is a requirement for abstracting HTTP-level aspects for seamless consumption of such services. PocoHttp is designed to provide an IQueryable<T> interface which is familiar to developers who have been working with ORMs such as Entity Framework, NHibernate, etc.

WCF Data Services are able to provide this seamless communication but they

• Can only generate AtomPub payloads
• As such no content negotiation or ability to generate plain XML or JSON (it does honour Accept header but returns OData specific format)
• Use OData query syntax which is deemed by many in the community not RESTful since it assumes non-HTTP syntax coherence in the form of query string parameters on the client
• Fully expose the data to the outside world

Hence there is a need for a .NET client library to take advantage of new HTTP features exposed in System.Net.Http to be able to consume HTTP data services whether developed in ASP.NET Web API or in any other platform. PocoHttp rises up to that challenge.

Minimal example

Before going much further into details, it might be useful to present a minimal usage of how to use it. Example below is from the PocoHttp samples (which hosts a minimal server too) available from GitHub.

var pocoClient = new PocoClient()
        {
            BaseAddress = new Uri("http://localhost:12889/api/")
        };
var list = pocoClient.Context<Car>() // calling "/api/Cars"
 .Take(1).ToList(); // getting the first item
Console.WriteLine(list[0]);

As can be seen above, pocoClient is initialised with a BaseAddress (Note the trailing slash which is mandatory for correct URI composition) and then a generic context of Car type is queried. PocoClient assumes (using the naming convention setup) that the Car should be exposed at BaseAddress + "Cars" so it makes an HTTP request and turns the result into Car objects.

PocoHttp's Grammar model

Basically PocoHttp translates IQueryable<T> queries into HTTP semantics (query string parameter in case of GET or a query criteria object in the body in case of POST) using a grammar and a set of naming convention settings - and then turns the result into .NET types using media type formatting provided by the ASP.NET Web API.

Currently, two built-in grammars are provided: OData and Pagination. OData implementation does not implement full features of OData (see below). Grammar can be implemented to translate the queries into other providers such as MongoDB or other custom syntaxes.

Normally, a grammar modifies the request by adding query string parameters. While it is not recommended, a grammar can modify the request to turn it into an HTTP POST and send a payload containing the criteria. The reason this is not recommended is that using POST method for GETting resources is not REST-friendly and responses to POST results cannot be cached while caching results of calls to data services can be very useful.

OData implementation

Current implementation of OData grammar supports below features:

• Where: current implementation supports direct property expression. Complex property expressions (such as Car.Make.StartsWith("M")) not yet supported
• AND, OR, greater than, lesser than, lesser than or equal, greater than or equal, not equal in WHERE expressions
• Take
• Skip
• OrderBy
• OrderByDescending 

Exposing an HTTP service in ASP.NET Web API (so that it can be consumed in PocoHttp) is simple: you just need to return IQueryable<T> from your action and decorate the action with [Queryable] attribute.

Having said that, while this feature is currently available in ASP.NET Web API RC, the future of OData support on Web API is a bit unclear as the Queryable attribute has been removed in ASP.NET Web API source code and it might not ship with RTM. As far as we can find out from ASP.NET team, OData support will be implemented using OData libraries and might ship out of band after the RTM. So the team is committed to full OData support but the timeline is unclear. As for now, you can happily use Queryable attribute in ASP.NET Web API RC.

As for PocoHttp, I am committed to implement full OData query syntax in the future releases.

Using PocoHttp against existing AtomPub OData services

Nothing prevents us to consume classic OData services that return AtomPub by just adding AtomPub media type formatter to the formatters. This will allow you to take advantage of all the niceties of HttpClient while using your existing OData services.

I will be having a post on this soon.

Disclaimer

As I initially said, this is a non-opinionated framework. You can expose your full domain model through an IQueryable<T> interface out to the client but this is not necessarily a good thing. Exposing bowels of your server and data to the outside world is an anti-pattern and could be a costly mistake. For example you can easily expose your database to the outside world allowing this query:

pocoClient.Context<Car>()
 .Where(car => car.Make.EndsWith("L"))

turn into this SQL statement that can bring down the server by having to look into each record:

SELECT * FROM CAR WHERE MAKE LIKE '%L'

Also free form Linq queries enable the client to select data using criteria which involves columns that do not have indices upon.

Pagination grammar

Pagination grammar is a simple non-OData syntax which is REST-friendlier. An example of the syntax is /api/Cars?skip=100&count=20.

Pagination vocabulary only supports 4 constructs:

1. Skip: normally represented by skip
2. Take: normally represented by count
3. OrderBy: normally represented by order
4. OrderByDescending: normally represented by orderDesc

All above representations can be changed by the client. A typical server implementation supporting these parameters will be

public IEnumerable<Car> Get(int skip, int count)
{
    return _repository.Skip(skip).Take(count);
}

As it can be seen, server implementation only supports skip and count and ignores the OrderBy and OrderByDescending.

Overriding naming convention and routing

There are times that you might want to override default route:

pocoClient.Context<Person>()
 .Where(p => p.Name == "Ali"); // calls /api/Persons?$filter=(Name eq 'Ali')

while you might want it to go to /api/Employees.

You have to ways to achieve this:

1. To decorate your entity with EntityUriAttribute. In this case [EntityUri("Employees")]
2. Pass the full URI as below when creating context:

pocoClient.Context<Car>("http://server/api/Employees")
 .Where(car => car.Make.EndsWith("L"))

Conclusion

In this post, we have introduced PocoHttp which is an emerging open source client library for consuming HTTP data services.

PocoHttp can be used to consume ASP.NET Web API services where IQueryable<T> is returned, existing OData services or any other HTTP data service built in other platforms - provided the grammar is implemented for the query syntax.