Thursday 28 June 2012

Using PocoHttp to consume Classic OData services


[Level T2] In the last post, I introduced PocoHttp which I have been working on. As I promised there, we can use it to access classic OData services in addition to as much as OData support we have in ASP.NET Web API. OData functionality currently provided in PocoHttp is limited at the moment but I will be building upon.

[UPDATE: PocoHttp package v0.1 is not available in NuGet. Just type in the package manager console PM>Install-Package PocoHttp or search for PocoHttp]

Why OData?

During the last  few years, Microsoft has invested a lot in OData. Some other technologies have also been built upon it such as RIA services which is heavily used in the Silverlight world. So Microsoft has been pushing OData and marketing it as an open protocol but reality is outside Microsoft, adoption has been nill. I believe this has to do with issues regarding query syntax and paylaod.

Firstly, as we talked about before, community believes that OData services are not RESTful as query syntax is unique to OData (Darrel explains here). It also exposes the bowels of the server to the client which does not respect REST's client-server model.  Personally I believe there is a place for some of the simpler query constructs of OData such as top and skip that could be implemented by non-OData services as well.

And payload is limited to specific data structures in XML and JSON. As for XML payload, I think AtomPub as data wrapping format is heavy and inefficient - not to say that it has not been realised of recent. Sending metadata to the client that consumes the data every second is not a very good idea - metadata was used once to build the client but the client is eternally doomed to receive that information until the day sun stops shining (there might be a way to turn it off but that is not the default design). So it is like a web service that sends its WSDL all the time.

JSON payload is definitely lighter - it contains some metadata but nothing to be concerned about. The JSON format has changed between versions 1 and 2 of OData and server could be sending V1 or V2 (etc) depending on the content. There is DataServiceVersion custom header that can be used to get the version number from the response but I think as a canonical HTTP implementation, it should have been implemented as part of content-type:
Content-Type: application/json;DataServiceVersion=2.0
Content-Type: application/odata2.0+json
Anyhow, that is not a big problem and I will be using JSON format in this sample. You can ask for JSON using an Accept header or appending $format=json.

OK, with all this so why OData then? For two reasons.

First of all, ASP.NET team are working to support alternative formats that do not have all the extra metadata: plain JSON or XML. So OData is going to live much longer in an improved implementation.

Second reason is there is already a considerable adoption inside the Microsoft community. Many teams while moving to new shiny Web APIs might still have an existing OData services that they need to use along with the new ones. 

Why use PocoHttp to access OData?

While OData has the problems we talked about, it is not short of one thing: tools and SDKs. You can easily use DataService<T> to consume OData services and the API even allows for automatic generation of data models for Entity Framework. You are fully abstracted from OData itself while using its all benefits.

So why would you ever want to use anything other than the existing tools to comsume OData services? Well, a few reasons.

With adoption of Web API increasing (which can be seen from the enthusiasm in the community), you will most likely start using Web APIs alongside OData ones. You would be building custom handlers to take care of authentication, tracing, caching, auditing, exception handling, etc and start to rely on them. But you would not be able to use these them with existing client tools since you need a client that is built upon the new HTTP pipeline in System.Net.Http.dll. That is PocoHttp!

Also you might want to harness the extreme powers that the client has in terms of queries it can run (which we touched in the last post) and limit them to a few handful essential but safe constructs - protecting your database and services. Then again that is one of the goals of PocoHttp.

New sample on the block

So I have created this sample on the PocoHttp that connects to OData sample services exposing Northwinds database. 

This basically sets up PocoClient to retrieve data in JSON format. Since the version of the JSON could be V1 or V2, the code needs to be able to support both.

Show us some code

OK! Let's look at request setup. There are two ways to request for JSON: by setting the Accept header or by appending $format=json:

// 1
pocoConfig.RequestSetup = (request) => request.Headers.Add("Accept", "application/json");

// 2
pocoConfig.RequestSetup = (request) => request.RequestUri = new 
   Uri(request.RequestUri.ToString() + "&$format=json",UriKind.Relative);

PocoConfiguration is a set of configuration parameters which is passed to the client constructor. One the parameters is RequestSetup which is an Action<HttpRequestMessage> that allows for the request to be modified before sending it to the server.

On the way back we use ResponseReader function to read the content into the objects. It receives the response, formatters and element types and must return objects it has read (will be IEnumerable<T>)

public Func<HttpResponseMessage, IEnumerable<MediaTypeFormatter>, Type, Task<object>> 
   ResponseReader { get; set; }

So this is a property sitting on the PocoConfiguration. PocoHttp provides a default implementation but since the data we get back is wrapped inside OData top level envelopes, we have to provide our own implementation. We need to define Payload<T> types that represent the structure of the data we are receiving. I have defined one for V1 and one for V2.

The gist of what we need to do is:
  1. Create a generic type of the entity using MakeGenericType
  2. Call non-generic ReadAsAsync passing the type
  3. Use reflection on the result to read off the properties to unwarp and get to the entities we need
For example, the code for V1 looks like this:

private static object ReadFromV1(HttpResponseMessage response, 
 IEnumerable<MediaTypeFormatter> formatters, Type type)
 var outermostType = typeof(PocoHttp.Samples.ClassicOData.V1.Payload<>)
 var result = response.Content.ReadAsAsync(outermostType, formatters).Result;
 return outermostType.GetProperty("d").GetValue(result, null);

In the example, I have used only Take and Skip but you can use direct Where clauses as well.


While ASP.NET Web API is the future of providing services and data, there is a considerable number of existing OData services that still need to be consumed. 

While existing OData client tools expose full features of OData abstracting the client from HTTP detailed, we outlined two reasons why you would want to use PocoHttp instead to consume these services: re-using HTTP pipeline handlers and limiting the client in terms of queries it can run so that the.

Monday 25 June 2012

Introducing PocoHttp: Consuming HTTP data services


[Level T2PocoHttp is a non-opinionated Open Source .NET library for seamless consumption of HTTP data services using a familiar IQueryable<T> interface. Version 0.1 of the library is now available on GitHub - NuGet package to follow soon.

This post explains the background and motivation for this library as well as its usage and possible future directions.

Motivation and background

ASP.NET Web API exposes full richness of the HTTP spec. As such, many new avenues have been opened for creating HTTP client-server applications. Having said that, with HTTP spec aimed to be Touring-Complete, there is a client burden involved in implementing clients capable of deep protocol coherency.

ASP.NET Web API has made it easy to achieve protocol coherency as part of the HTTP pipeline - modelled as Russian Dolls (see Part 6 of the series).

With a lot of WCF services (that commonly were serving domain's value objects) soon to be moved to Web API, there is a requirement for abstracting HTTP-level aspects for seamless consumption of such services. PocoHttp is designed to provide an IQueryable<T> interface which is familiar to developers who have been working with ORMs such as Entity Framework, NHibernate, etc.

WCF Data Services are able to provide this seamless communication but they
  • Can only generate AtomPub payloads
  • As such no content negotiation or ability to generate plain XML or JSON (it does honour Accept header but returns OData specific format)
  • Use OData query syntax which is deemed by many in the community not RESTful since it assumes non-HTTP syntax coherence in the form of query string parameters on the client
  • Fully expose the data to the outside world
Hence there is a need for a .NET client library to take advantage of new HTTP features exposed in System.Net.Http to be able to consume HTTP data services whether developed in ASP.NET Web API or in any other platform. PocoHttp rises up to that challenge.

Minimal example

Before going much further into details, it might be useful to present a minimal usage of how to use it. Example below is from the PocoHttp samples (which hosts a minimal server too) available from GitHub.

var pocoClient = new PocoClient()
            BaseAddress = new Uri("http://localhost:12889/api/")
var list = pocoClient.Context<Car>() // calling "/api/Cars"
 .Take(1).ToList(); // getting the first item

As can be seen above, pocoClient is initialised with a BaseAddress (Note the trailing slash which is mandatory for correct URI composition) and then a generic context of Car type is queried. PocoClient assumes (using the naming convention setup) that the Car should be exposed at BaseAddress + "Cars" so it makes an HTTP request and turns the result into Car objects.

PocoHttp's Grammar model

Basically PocoHttp translates IQueryable<T> queries into HTTP semantics (query string parameter in case of GET or a query criteria object in the body in case of POST) using a grammar and a set of naming convention settings - and then turns the result into .NET types using media type formatting provided by the ASP.NET Web API.

Currently, two built-in grammars are provided: OData and Pagination. OData implementation does not implement full features of OData (see below). Grammar can be implemented to translate the queries into other providers such as MongoDB or other custom syntaxes.

Normally, a grammar modifies the request by adding query string parameters. While it is not recommended, a grammar can modify the request to turn it into an HTTP POST and send a payload containing the criteria. The reason this is not recommended is that using POST method for GETting resources is not REST-friendly and responses to POST results cannot be cached while caching results of calls to data services can be very useful.

OData implementation

Current implementation of OData grammar supports below features:
  • Where: current implementation supports direct property expression. Complex property expressions (such as Car.Make.StartsWith("M")) not yet supported
  • AND, OR, greater than, lesser than, lesser than or equal, greater than or equal, not equal in WHERE expressions
  • Take
  • Skip
  • OrderBy
  • OrderByDescending 

Exposing an HTTP service in ASP.NET Web API (so that it can be consumed in PocoHttp) is simple: you just need to return IQueryable<T> from your action and decorate the action with [Queryable] attribute.

Having said that, while this feature is currently available in ASP.NET Web API RC, the future of OData support on Web API is a bit unclear as the Queryable attribute has been removed in ASP.NET Web API source code and it might not ship with RTM. As far as we can find out from ASP.NET team, OData support will be implemented using OData libraries and might ship out of band after the RTM. So the team is committed to full OData support but the timeline is unclear. As for now, you can happily use Queryable attribute in ASP.NET Web API RC.

As for PocoHttp, I am committed to implement full OData query syntax in the future releases.

Using PocoHttp against existing AtomPub OData services

Nothing prevents us to consume classic OData services that return AtomPub by just adding AtomPub media type formatter to the formatters. This will allow you to take advantage of all the niceties of HttpClient while using your existing OData services.

I will be having a post on this soon.


As I initially said, this is a non-opinionated framework. You can expose your full domain model through an IQueryable<T> interface out to the client but this is not necessarily a good thing. Exposing bowels of your server and data to the outside world is an anti-pattern and could be a costly mistake. For example you can easily expose your database to the outside world allowing this query:

 .Where(car => car.Make.EndsWith("L"))

turn into this SQL statement that can bring down the server by having to look into each record:

Also free form Linq queries enable the client to select data using criteria which involves columns that do not have indices upon.

Pagination grammar

Pagination grammar is a simple non-OData syntax which is REST-friendlier. An example of the syntax is /api/Cars?skip=100&count=20.

Pagination vocabulary only supports 4 constructs:

  1. Skip: normally represented by skip
  2. Take: normally represented by count
  3. OrderBy: normally represented by order
  4. OrderByDescending: normally represented by orderDesc

All above representations can be changed by the client. A typical server implementation supporting these parameters will be

public IEnumerable<Car> Get(int skip, int count)
    return _repository.Skip(skip).Take(count);
As it can be seen, server implementation only supports skip and count and ignores the OrderBy and OrderByDescending.

Overriding naming convention and routing

There are times that you might want to override default route:
 .Where(p => p.Name == "Ali"); // calls /api/Persons?$filter=(Name eq 'Ali')
while you might want it to go to /api/Employees.

You have to ways to achieve this:

  1. To decorate your entity with EntityUriAttribute. In this case [EntityUri("Employees")]
  2. Pass the full URI as below when creating context:

 .Where(car => car.Make.EndsWith("L"))


In this post, we have introduced PocoHttp which is an emerging open source client library for consuming HTTP data services.

PocoHttp can be used to consume ASP.NET Web API services where IQueryable<T> is returned, existing OData services or any other HTTP data service built in other platforms - provided the grammar is implemented for the query syntax.

Using the CachingHandler in ASP.NET Web API


[NOTE: Please also see updated post and the new framework called CacheCow
This class has been removed from WebApiContrib code and NO LONGER SUPPORTED]

[Level T3] Caching is an important concept in HTTP and comprises a sizeable chunk of the spec. ASP.NET Web API exposes full goodness of the HTTP spec and caching can be implemented as a message handler explained in Part 6 and 7. I have implemented a CachingHandler and contributed the code to the WebApiContrib in GitHub.

This post will have two sections: first a primer on HTTP caching and then how to use the handler. This code uses ASP.NET Web API RC and with .NET 4 (VS 2010 or 2012).


NOTE: This topic is fairly advanced and complex. You do not necessarily need to know all this in order to use CachingHandler and are welcomed to skip it but more in-depth knowledge of HTTP could go a long way. 

Caching is a very important feature in the HTTP. RFC 2616 extensively covers this topic in section 13. Review of all the spec is beyond the scope of this post but we will briefly touch on the subject.

First of all let's get this straight: this is not about putting an object in memory as we do with HttpRuntime.Cache. In fact server does not store anything (more on this below), it only tells the client (or mid-stream cache or proxy servers) what can be cached, how long and validates the caching.

Basically, HTTP provides semantics and mechanism for the origin server, client and mid-stream proxy/cache servers to effectively reduce traffic by validating the version of the resource they have against the server and retrieve the resource only if it has changed. This process is usually referred to as cache validation.

In HTTP 1.0, server would return a LastModified header with the resource. A client/user agent could use this value and send it in the If-Modified-Since header in the subsequent GET requests. If the resource was not changed, server would respond with a 304 (Not Modified) otherwise the resource would be sent back with a new LastModified header. This would be also useful in PUT scenarios where a client sends a PUT request to update a resource only if it has not changed: a If-Unmodified-Since header is sent. If the resource has not changed, server fulfils the request and sends a 2xx response (usually 202 Accepted) otherwise a 412 (Precondition Failed) is sent.

For many reasons (including the fact that HTTP dates do not have milliseconds) it was felt that this mechanism was not adequate. In HTTP 1.1, ETag (Entity Tag) was introduced which is an opaque Id in the format of a quoted string that is returned with the resource. ETags can be strong (refer to RFC 2616 for more info) or weak in which case they start with w/ such as w/"12345". ETag works like a version ID for a resource - if two ETags (for the same resource) are equal, it means those two versions of the resource are the same.

ETag for the same resource could be different according to various headers. For example a client can send an Accept-Language header of de-DE or en-GB and server would send different ETags. Server can define this variability for each resource by using Vary header.

Cache validation for GET and PUT requests are similar but instead will use If-Match (for PUT) and If-None-Match (for GET).

Well... complex? Yeah, pretty much and yet I have not covered some other aspects and the edge cases. But don't worry! You will be abstracted from a lot of it if you use the CachingHandler.

Using CachingHandler right out of the box

OK, using the CachingHandler is straightforward especially if you go with the default settings. You can have a look at the CarManager sample in the WebApiContrib project. CarManager.Web sample demonstrate server side setup of the caching and CarManager.CachingClient for making calls to the server and testing various caching scenarios.

All you have to do is to create an ASP.NET Web API project and add the CachingHandler as a delegating handler as we learnt in Part 6:


And you are done! Now let's try the server with some scenarios. I would suggest that you use the CarManager sample, otherwise create a simple handler and implement GET, PUT and POST on it.

Now let's make some request and look at the response. I would suggest using Fiddler or Chrome's Postman to send requests and view the response.

So if we send a GET request:

GET http://localhost:8031/api/Cars HTTP/1.1
User-Agent: Fiddler
Host: localhost:8031

We get back this response (or similar; some headers removed and body truncated for clarity):

HTTP/1.1 200 OK
ETag: "54e9a75f2dbb4edca672f7a2c4a73dca"
Vary: Accept
Cache-Control: no-transform, must-revalidate, max-age=604800, private
Last-Modified: Thu, 21 Jun 2012 23:35:46 GMT
Content-Type: application/json; charset=utf-8


So we see the ETag header here along with important caching headers. Now if we make another GET call, we get back the same response but ETag and last modified stays the same.

Generating same ETag is fine (showing our CachingHandler is doing something) but we have not yet seen any caching. That is where client has to do some work to do. It has to use If-None-Match with the ETag to conditionally ask for the resource: if it matches server, it will get back 304 but if not, server will return the new resource:
GET http://localhost:8031/api/Cars HTTP/1.1
User-Agent: Fiddler
Host: localhost:8031
If-None-Match: "54e9a75f2dbb4edca672f7a2c4a73dca"
Here we get back 304 (Not modified) as expected:
HTTP/1.1 304 Not Modified
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
ETag: "54e9a75f2dbb4edca672f7a2c4a73dca"
Server: Microsoft-IIS/8.0
Date: Sun, 24 Jun 2012 07:34:29 GMT

A typical server that can use CachingHandler

CachingHandler makes a few RESTful assumptions about the server for effective caching. Some of these assumptions can be overridden but generally not recommended. These assumptions are:

  • HTTP verbs (POST/GET/PUT/DELETE for CRUD operations) are used to modify resources - and not RPC-style resources (such as POST /api/Cars/Add). This is the most fundamental assumption.
  • All resources are to be modified through the same HTTP pipeline that implements caching. If a resource modified outside the pipeline, cache state needs to be updated by the same process.
  • Resource are organised in a natural cache-friendly manner: invalidation of related resources can be done with a minimal setup (more details upcoming).

Defining cache state

As we said, this has nothing to do with HttpRuntime.Cache! Unfortunately ASP.NET implementation makes it really confusing between the HTTP cache (where the resource gets cached on the client or midstream cache servers) and server caching (when the rendered output gets cached on the server).

Cache state is a collection of data that helps keep track of each resource along with its last modified date and ETag. It might initially seem that for a resource, there exists a single of such pieces of information. But as we touched upon above, a resource can have different representations each of which needs to be stored separately on the client while they will most likely invalidated together. 

For example, resource /api/disclaimer can exist in multiple languages as such client has to cache each representation separately but when disclaimer changes, all such representations need to be invalidated. That will require a storage of some sort to keep track of all this data. Current implementation comes with an in-memory storage but in a web farm scenario this needs to be a persistent store.

Cache state storage and cache invalidation

So we do not need to store the cache on the server, but we DO need to store ETag and various states on the server. If we only have a single server, this state can be stored in the memory. In a web farm (or even web garden) scenario a persisted store is needed. This store is called Entity Tag Store and is represented by a simple interface:

public interface IEntityTagStore
 bool TryGetValue(EntityTagKey key, out TimedEntityTagHeaderValue eTag);
 void AddOrUpdate(EntityTagKey key, TimedEntityTagHeaderValue eTag);
 bool TryRemove(EntityTagKey key);
 int RemoveAllByRoutePattern(string routePattern);
 void Clear();

We need to have an implementation of this interface so that the cache management can be abstracted away from controllers and instead done in the DelegatingHandlers.

Introducing some concepts (you may skip and come back to it later)

This store will keep track of the Etag (and related state stored in TimedEntityTagHeaderValue) based on an Entity Tag key which is calculated based on the resource URI and content of the important headers (which their list will be on the Vary header). In-Memory implementation for a single server is provided out of the box with CachingHandler - I would be creating a SQL-Server implementation of it very soon; watch this space.

It is important to note that change in the resource will most likely invalidate all forms of the resource so all permutations of important headers will be invalidated. So invalidation is usually performed at the resource level. Sometimes several related resources can be represented as a RoutePattern. By default, URI of a resource is its RoutePattern.

Also in some cases, change in a resource will invalidate linked resources. For example, a POST to /api/cars to add a car will invalidate /api/cars/fastest and /api/cars/mostExpensive. In this case, "/api/cars/*" can be defined as the linked RoutePattern (since /api/cars does not qualify for /api/cars/*).

Some assumptions in CachingHandler

  1. Re-emphasising that resource can only change through the HTTP API (using PUT, POST and DELETE verbs). If resources are to be changed outside the API, it is responsibility of the application to use IEntityTageStore to invalidate the cache for those resources.
  2. If no Vary header is defined by the application, CachingHandler creates weak ETags.
  3. Change in the resource will invalidates all forms of the resources (all permutations of important header values)

CarManager sample

I have considered a pretty complex and interrelated routing and caching requirement for the sample to display what is possible. Resources available are:

  1. /api/Car/{id}: GET, PUT and DELETE
  2. /api/Cars: GET and POST
  3. /api/Cars/MostExpensive: GET
  4. /api/Cars/Fastest: GET
So creating a car would invalidate cache for 2, 3 and 4. Updating car with id=1 invalidates 2,3 and 4 in addition to the car itself at /api/Car/1. Deleting a car will invalidate 2, 3 and 4.

The client sample (CarManager.CachingClinet) displays how to call the server with headers to validate the cache.

Configuring CachingHandler

Best place to start is the CarManager.Web sample to give an idea on how various setup can be used to configure complex caching requirements. Basically the points below can be used to configure the CachingHandler


A list of request header names can be passed that will define the vary headers. If no vary header is passed, system only generates weak ETags. Also optionally an implementation of IEntityTagStore otherwise by default InMemoryEntityTagStore will be used.


This is an opportunity to provide linked resource for a resource.

 Other customisation points

CachingHandler provides properties in the form of functions with default implementation that can be changed  to override the default behaviour which we will cover in the upcoming post on "Extending CachingHandler".


CachingHandler is a server-side DelegatingHandler which can be used to abstract away caching from individual ApiControllers so that controller logic can be coded without having to worry about caching. 

The  code is hosted on GitHub (currently sitting at my fork waiting to be merged merged!) and comes with both a client and server sample - CarManager.Web and CarManager.CachingClient.

Thursday 14 June 2012

ASP NET Web API Series - Part 9: MediaTypeFormatter on the client


[Level T3] In part 5 we talked about MediaTypeFormatter on the server. We have another side of the story - perhaps not as big - on the client. MediaTypeFormatters will helps us converting our ViewModels/DTOs to and from HTTP content. HttpClient exposes full goodness of server side formatting as we have seen in part5. This post is going to be short and simple - especially if you have read the server-side post.

Why should I care?

You should care if you want to do anything useful consuming an HTTP API from a .NET client - and not Javascript. 

As we discussed on the server-side post, MediaTypeFormatter marries the world of HTTP (headers and body) to the world of .NET (strongly-typed objects). It will be very likely that you will be sending some strongly typed objects across the wire and expecting the same back. MediaTypeFormatter will help you achieve that.

ObjectContent and ObjectContent<T>

ASP.NET Web API abstracts the HTTP message's body and its body-related headers as HttpContent class - this applies to both request and response. So what are body-related headers? Here is a list:
  • Allow
  • Content-Disposition
  • Content-Encoding
  • Content-Language
  • Content-Length
  • Content-Location
  • Content-MD5
  • Content-Range
  • Content-Type
  • Expires
  • Last-Modified
As we have highlighted, Content-Type is the key header here - as for the media type formatting. These are in fact the same headers that can appear in each part of a Multipart message. As we covered briefly before, in a multi-part message, each part gets its own share of the body and body-related headers.

While HttpContent generally abstracts HTTP body as a stream, ObjectContent which inherits from HttpContent abstracts it as a strongly typed object - especially the generic type ObjectContent<T>

Creating an ObjectContent<T>

ObjectContent<T> has two constructors:

public ObjectContent(T value, MediaTypeFormatter formatter) : this(value, formatter, null)

public ObjectContent(T value, MediaTypeFormatter formatter, string mediaType) : base(typeof(T), value, formatter, mediaType)

We are supposed to pass the value of generic type T and pass the actual MediaTypeFormatter we want to format the content. The content then can be used to be sent to the server, for example in a post:

var content = new ObjectContent<foo>(new Foo(), new JsonMediaTypeFormatter);
var httpClient = new HttpClient();
httpClient.PostAsync(url, content);

Since it is the client who initiates the HTTP call, it would know what format to send the request body in. So the content-negotiation does not apply here. Having said that, I would have liked to be able to POST a request without having to decide on a MediaTypeFormatter and let Web API to choose a default formatter for me. I have not been able to find this in the framework yet.

Choosing a default media type formatter is not that hard in fact. All we have to do is to create a new MediaTypeFormatterCollection and then choose the JsonFormatter or XmlFormatter property.


ASP.NET Web API comes with a non-generic collection for MediaTypeFormatter. Unlike most collections which are a bag/list of their items, MediaTypeFormatterCollection has some knowledge and understanding of the formatters.

For starter, constructor of this collection creates the collection with these three default formatters:

public MediaTypeFormatterCollection() : this(CreateDefaultFormatters())

private static IEnumerable<MediaTypeFormatter> CreateDefaultFormatters()
    return new MediaTypeFormatter[] { new JsonMediaTypeFormatter(), 
          new XmlMediaTypeFormatter(), 
          new FormUrlEncodedMediaTypeFormatter() };

Also it has three properties: JsonFormatter, XmlFormatter and FormUrlEncodedFormatter. With using the parameterless constructor, each of these properties will point to one of the formatters constructed. However, if we create our own IEnumerable<MediaTypeFormatter> and pass to constructor, collection will find these formatters and assign the properties. I call these common formatters since they are bread and butter of the web.

Formatting objects into HTTP content (Request)

As we saw earlier, we can directly create an ObjectContent<T> or do that as part of POST/PUT:

client.PostAsync<Foo>(uri, new Foo(),
 new MediaTypeFormatterCollection().JsonFormatter);

This is not really much more complex than that.

Reading object from HTTP content (response)

This is where ASP.NET Web API provides on the client its full media type formatter selection on the server. Normally you would use ReadAsAsync<T> on the HttpResponseMessage. For example:

Foo result = client.GetAsync(uri)

As you can see, no media type formatter had to be passed and we used the organic media type formatter selection within the framework. How? Well, parameterless ReadAsAsync<T> is in fact an extension method that creates a new MediaTypeFormatterCollection (as we saw above) and pass as a set of formatters:

public static Task<T> ReadAsAsync<T>(this HttpContent content)
    return content.ReadAsAsync<T>(new MediaTypeFormatterCollection());

This in turn will call other methods and in the end, FindReader method of the MediaTypeFormatterCollection will be used to select a suitable formatter based on type and Contet-Type header.

Example: using Rotten Tomatoes web API

Rotten Tomatoes is a popular website for movie reviews. So if you are a movies fan like me, and you are into Web API, you might be tempted to expose the search movie search functionality in your application/mashup.

So what you need is an API key. It is free to acquire and you just need a Mashery ID and register for an API key in Rotten Tomatoes here. After you got your api key, you need to append this key as a query string parameter to all your calls.

So let's imagine we want to search films having "matrix" word in them. I have done that and here is the JSON.

Next step is to create classes to represent your JSON. You are welcome to do that by hand but there is a useful online tool there that can do that for you: json2csharp.

So here is what it has generated:

public class ReleaseDates
    public string theater { get; set; }
    public string dvd { get; set; }

public class Ratings
    public string critics_rating { get; set; }
    public int critics_score { get; set; }
    public string audience_rating { get; set; }
    public int audience_score { get; set; }

public class Posters
    public string thumbnail { get; set; }
    public string profile { get; set; }
    public string detailed { get; set; }
    public string original { get; set; }

public class AbridgedCast
    public string name { get; set; }
    public string id { get; set; }
    public List<string> characters { get; set; }

public class AlternateIds
    public string imdb { get; set; }

public class Links
    public string self { get; set; }
    public string alternate { get; set; }
    public string cast { get; set; }
    public string clips { get; set; }
    public string reviews { get; set; }
    public string similar { get; set; }

public class Movy
    public string id { get; set; }
    public string title { get; set; }
    public int year { get; set; }
    public string mpaa_rating { get; set; }
    public int runtime { get; set; }
    public string critics_consensus { get; set; }
    public ReleaseDates release_dates { get; set; }
    public Ratings ratings { get; set; }
    public string synopsis { get; set; }
    public Posters posters { get; set; }
    public List<AbridgedCast> abridged_cast { get; set; }
    public AlternateIds alternate_ids { get; set; }
    public Links links { get; set; }

public class Links2
    public string self { get; set; }
    public string next { get; set; }

public class RootObject
    public int total { get; set; }
    public List<Movy> movies { get; set; }
    public Links2 links { get; set; }
    public string link_template { get; set; }

As you can see, it does a very good job in converting them but there are some naming conventions that you might have to go back and re-visit (In C# we use Pascal-Casing). Also Movy really needs to be Movie but for our purpose it is just fine.

So let's open Visual Studio 2010 and create a new console application project. Then add a code file and paste the codes above in there.

Then in the "Package Manager Console", type

PM> Install-Package System.Net.Http.Formatting
This should add all client libraries you need. Then we just need to write

var httpClient = new HttpClient();
   var rootObject = httpClient.GetAsync("[YOUR_API_KEY]")

You should have the data correctly deserialised back to your objects. Well, not quite, there is a problem since the API uses legacy content-type "text/javascript" for the JSON instead of  "application/json". With the technique we learnt back in Part 7 and 8, we can create a message handler to change the legacy content type to the correct one and add it to the pipeline:

public class LegacyJsonMediaTypeConverterDelegatingHandler : DelegatingHandler
 protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  return base.SendAsync(request, cancellationToken)
   .ContinueWith(task =>
                 var httpResponseMessage = task.Result;
        if (httpResponseMessage.Content.Headers.ContentType.MediaType == "text/javascript")
         httpResponseMessage.Content.Headers.ContentType.MediaType = "application/json";
                 return httpResponseMessage;

Now all we have to do is to add the handler to the pipeline and make the call:

static void Main(string[] args)

 var httpClient = new HttpClient(new LegacyJsonMediaTypeConverterDelegatingHandler() { InnerHandler = new HttpClientHandler()});
 var rootObject = httpClient.GetAsync("")



ObjectContent<T> is an important class in ASP.NET Web API which abstracts the HTTP content (or a part within a multipart message) which includes its body and body-related headers.

MediaTypeFormatter on the client is as important as on the server. For sending requests, since client is the initiator of the request, it needs to explicitly set the formatter. For processing response data, however, a suitable formatter is selected based on type of the object to be read and content-type.

Friday 8 June 2012

What I think coupling is ...


[Level C3] This post is a follow-up to the question/discussion-point Darrel Miller has started here. The question is that in the REST world, what is coupling and how we can achieve de-coupling. Although this post can be read independently, it is best to start with reading his post first. [This discussion carries on in the next related post]


REST has a strong focus on decoupling client and server. With REST awareness and adoption increasing, the challenge to define new best practices has now become more apparent. This is particularly important in the light of how much nowadays can be achieved in the browser - a previously limited client. Rise of the Single Page Applications (SPAs) is a testament to the popularity of creating a rich-client in what was previously called thin-client. Diversity of available clients and their capabilities have - in a way - forced us towards REST and achieving de-coupling of client and server. 

But recently, some have felt that too much power and control has shifted towards the client and the server has been reduced to mere a data provider. One of such groups is ROCA that defines a set of best practices that are in clear opposition to SPA paradigms. I actually meant to have a post on ROCA (which I hopefully will soon) but I suppose this post can be a primer as the issue at question is relevant.


Darrel defines coupling as "a measure of how changes in one thing might cause changes in the other". I would very much go with this definition but I would like to expand upon.

Coupling is a software design (logical) and architecture (physical) anti-pattern. Loose-coupling is on the other hand  a virtue that allows different compartments (intentionally avoiding using bloated words module or component) of the system to change independently. 

In the world of code (logical), we use Single-Responsibility principle (S from SOLID) to decouple pieces of code: a class needs to have a single reason to change. Arguably, rest of the SOLID principles deal with various degrees of decoupling. For example D is for not baking the dependency.

On the other hand, in SOA (physical) we loosely-couple the services. Services can be dependent on other services but they would be able to maintain a reasonable level of functionality if other services go down. The key tasks in building a successful SOA is defining service boundaries. Another important concept is achieving cohesion: keeping all related components of the service in the same service and resisting to break a service into two compartments when boundary is weak.

Defining boundary and achieving cohesion

Coupling in other words is baking knowledge of something in a compartment where it is concern of another compartment.

When I buy stuff from Amazon and it is delivered when I am not at home, they leave a card with which I can claim my package at the depot. The card has just an Id in the form of barcode. It does not have the row and shelf number my package is kept at the depot. If it had, it might have made it easier for the clerk to use the numbers on the card to fetch the package. But this way I am baking the knowledge of location into something it does not need to know and what if they had to change the location of the package? That is why the clerk zaps the barcode and location is shown in the system. They can happily change the location as long as the Id does not change.

On the other hand, the depot does not need to know what is in the package - it is not its concern. If it did, it might have been helpful in rare scenarios but that is not worth considering.

So the key to defining boundary and achieving cohesion is to understand whose concern it is for an abstraction. We will represent 3 models here: server-concern, client-concern and mixed-concern. In brief, it depends.


In this case, only server needs to know about an abstraction. The problem happens when the concept oozes out to the client when client gets to know server implementation details (exposing server bowels).



A typical example is getting the list of most recent contacts:
GET /api/contacts?lastUsedMoreThan=2012-05-06&count=20
GET /api/contacts/mostRecent 
In the first case, client gets to know that server keeps a value of lastUsed for contacts. This prevents server to optimise the algorithm by also including number of times we have used the contact. However, in the second case server hides its implementation so the change can be implemented without breaking the client.


In this scenario, the abstraction is purely a client concept. The problem happens when server starts to make decisions for the client.



A typical example is pagination:
GET /api/contacts/page/11
GET /api/contacts?skip=200&count=20
Number of pages available and number of each record in a page is a client concern. This kind of detail will be different on the iPhone from the desktop from the tablet (and what tablet). In the first case, server will make a decision on the number of records per page (and even understands the page as a resource) while in the second case, knowledge of pagination is confined to the client since it is its concern.


In this scenario, both server and client need to work in accord for the feature to work. The problem happens when server or client assumes that the other definitely implements the abstraction.



A typical example is HTTP caching. For HTTP caching to work, client and server need to work in tandem. Server needs to return with each resource, a Cache-Control header, an ETag or a LastModified and the client needs to use these parameters in its future conditional requests with If-Modified-Since or If-None-Match. 

No assumption in implementation on the other side
helps both sides to carry on working without the feature

However, if server does not provide caching or the client does not use and respect the caching parameters from the server, system does not break - albeit it can result in an inferior or suboptimal experience.

Diversity and compromise

While we can have 3 above models, there are cases where same feature can be implemented differently. 

Let's bring an example from engineering. Where do you put AC/DC power transformer? My PC has the transformer in its power supply (item 6) which is equivalent of Client-Concern. My laptop uses a power adaptor and has no mains supply which is equivalent of Server-Concern. On the other hand, my electrical toothbrush has divided the transformer into both charger element and also inside the toothbrush so it works by magnetic induction (analogous to mixed-concern). This is clearly a compromise but the thinking behind it is make the toothbrush waterproof and safe.


Darrel's definition is excellent but we have expanded upon it since the change (maintenance) is one facet of the same concept. Others are concern (requirement) and knowledge of that concern in a compartment (implementation).

We discussed three models: server-concern, client-concern and mixed-concern. Each of these are valid patterns but they each come with its own anti-pattern to be aware of. So in short: it depends.

Tuesday 5 June 2012

ASP.NET Web API Series - Part 8: MessageHandler on the Client


[Level T2] So far we have only covered server-side features while Microsoft's ASP.NET Web API has excellent client-side features. It is true that server-side requires a lot of care and attention but as I have pointed out in the past, implementing REST-style application usually involves a client burden. In this post, we will cover Russian Doll model on the client (for more background have a look at part 6).

Client arsenal in the age of Web API

Before Microsoft HTTP history was re-written in ASP.NET Web API, there were 3 ways to make an HTTP request:

  1. High-level WebClient. While lacked some tuning features, it presented a useful tool capable of HTTP, FTP and file access.
  2. Mid-level HttpWebRequest: This was used most often and presented abstractions at the right level.  
  3. Close-to-the-metal Windows socket. While you could, I cannot think of a reason why you would.

While these could still prove useful, Microsoft has designed and built HttpClient class and it is intended to be used as the main client API in the Web Stack. If you read Mike Wasson's excellent post on Message Handlers you will know why: it has been built using the same abstractions as the server side of Web API. This enables lego-esque pipeline of various message handlers (as we have seen before) each to add a nice little feature to the pipeline.


First of all HttpClient is heavily asynchronous - well that is an understatement, it is purely asynchronous.

Using HttpClient is simple:

var client = new HttpClient();
var asyncTask = client.GetAsync("");
asyncTask.ContinueWith(task =>
                       .ContinueWith(readTask => Console.WriteLine(readTask.Result));
Console.WriteLine("Waiting ...");

or even simpler:

string result = client.GetAsync("")

or even simpler:

var result = client

There is not much magic in using HttpClient. Basically you may use

  • GetAsync: analogous to HTTP GET
  • PostAsync: analogous to HTTP POST
  • PutAsync: analogous to HTTP PUT
  • DeleteAsync: analogous to HTTP DELETE
  • GetStringAsync, GetByteArrayAsync and GetStreamAsync if all you care is the Content
  • SendAsync: general method if you want to have full control on constructing an HttpRequestMessage or want to use other/custom verbs.

Specifying an HttpMessageHandler

When you create an HttpClient using its parameterless constructor, in fact this is what you are doing:

public HttpClient() : this(new HttpClientHandler())

So basically the default constructor creates an HttpClientHandler to handle the all leg work for passing the request over to the server and then construct an HttpResponseMessage based on the result.

HttpClientHandler in fact uses HttpWebRequest and HttpWebResponse behind the scene (discussion here) but translates the communication to ASP.NET-agnostic model of the Web API, a.k.a. HttpRequestMessage and HttpResponseMessage.

The the other two constructors we can pass an HttpMessageHandler. This message handler has to be the handler that ultimately returns the result - usually from the server over the wire. Well, I said usually since nothing stops us to put a full-fledged HttpServer in there: this means the request and response will not even touch the wire and will run fully in-memory. Pedro Felix shows us how to do this here.

HTTP pipeline on the client

What if we do not want to mess around with all the whole communication pipeline and we just want to use a DelegatingHandler? Well, not so difficult, all we have to do is to create a delegating handler and set its InnerHandler to an HttpClientHandler.

Here we look our minimal scenario where we have delegating handler that outputs Uri into the Trace:

public class LoggingHandler : DelegatingHandler
 public LoggingHandler()
   InnerHandler = new HttpClientHandler();

 protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  return base.SendAsync(request, cancellationToken);

class Program
 static void Main(string[] args)
  var client = new HttpClient(new LoggingHandler());
  var result = client.GetStringAsync("").Result;

We can see the trace output in the Visual Studio output window.

Real-world client message handlers

Admittedly, there are far less implementation of message handlers on the client perhaps since most early adopters of ASP.NET Web API have been server-focused geeks. Yet, there are some implementations out there.

Henrik explains how to build an OAuth message handler and use it to build a twitter client. This sample can be downloaded from here.

I am hoping to build a CachingHandler on the client using ASP.NET Web API. If all goes well, this should be the topic of the next post.


ASP.NET Web API offers the same wealth of HTTP pipeline abstractions (known as Russian Doll model) on the client as on the server. HttpClient is the main tool to make HTTP calls and it - by default - uses HttpClientHandler for all its communications. If you need to provide a delegating handler (or a series of delegating handlers) you can pass it to the constructor while setting the InnerHandler to an instance of HttpClientHandler.

Sunday 3 June 2012

FizzBuzz coding challenge, the functional way

[Level T2]

I think most of you guys have either heard about the popular FizzBuzz coding challenge or actually been asked to solve during an interview. However, for those of you that have not heard it, here it is (or at least a version of it):
Ask the user to enter two integers: start of the range and end of the range. Then for every integer in the range, output "Fizz" if the integer is dividable by 3, output "Buzz" if it is dividable by 5, output "FizzBuzz" if dividable by 15 otherwise output the number itself.
Unless one is a novice programmer, implementing this is not really hard. Adding stress of the interview, unfamiliarity of the location and the machine you are given for the challenge, it probably can represent a moderate-difficulty task for the interview if you have heard about it for the first time. But like said by many before, always ask developers to write some code during interview. Be it as simple as this challenge.

C#, a functional language?

No, I know it is not. Yet, it has a lot of the elements of functional programming since v 3.0 with Func and Action. PTL has made it even more functional since not only a function can be abstracted as Action or Func, its result and state can be abstracted and encapsulated as Task<T>.

Until recently, I have been unaware of the power of the functional programming that is possible in C#. And the reason is you have to think functional first to be able to do functional. It might surprise you that in fact Javascript (not particularly known as a functional language) inspired me to start thinking functionally - but that's another story for another day.

So I have picked up a simple scenario to gradually build up using functional C#. If you are a fluent Haskell, F# or Erlang developer perhaps this post is not for you - unless you would like to see the functional features of C#. So let's start with the conventional approach.

FizzBuzz, the conventional procedural way

As I said before, solving FizzBuzz challenge the conventional way is almost trivial (assuming we have captured user input):

private static void ConventionalApproach(int start, int end)
 for (int i = start; i <= end; i++)
  if(i % 15 == 0)
  else if (i % 3 == 0)
  else if (i % 5 == 0)


The only point to note is that the condition to divide by 15 must be written first, obviously, since every number dividable to 15 is also dividable by 3 and 5.

We might actually improve the "performance" by sparing the division to 15 so that division by 15 is defined by division by 3 and division by 5 which the output will be "Fizz" + "Buzz" => "FizzBuzz".

private static void ConventionalApproachImprovedPerf(int start, int end)
 for (int i = start; i <= end; i++)
  bool wroteAny = false;
  if (i % 3 == 0)
   wroteAny = true;
  if (i % 5 == 0)
   wroteAny = true;
  if (!wroteAny)


This sort of detail is the kind of stuff your interviewers are expected to see in your solution.

What a functional solution could look like?

Functional approach lays out all new possibilities and as such, I think everyone might come up with a different solution. But I would like to present my solution which looks like this:

var chain = ChainOfResponsibility<int>
 .Start(i => i.IsDividableBy(15), i => "FizzBuzz".OutputLine())
 .Then(i => i.IsDividableBy(3), i => "Fizz".OutputLine())
 .Then(i => i.IsDividableBy(5), i => "Buzz".OutputLine())
 .Else(i => i.OutputLine());

Enumerable.Range(rangeStart, rangeEnd+1).ToList()
 .ForEach(x => chain.Run(x));

Well. This looks a lot more readable. Let's see how to build it.

Abstracting out

OK, now let's first of all think about what we have just done. We used if statements a lot. So what is an if statement:
"if" is an expression that produces a boolean value. That boolean value is fed to an action - if it has a value of true.
So in pseudo-code terms:

if ( a_condition )

More specifically for our case since we have an integer input to we can re-write this as:

if ( a_condition(input) )

So we can say that the condition is Predicate<int> and the action is an Action<int>. Predicate<T> is actually nothing but Func<T, bool> which means that it takes an input of type T and returns a boolean.

Chain of responsibility

Chain of responsibility is a design pattern that can be thought of as a series of if-else whereby the first one to fulfil the condition will short-circuit the path and the rest of conditions will not be checked. 

The first implementation above (which contains a series of if and if-else) can be thought of as a chain of responsibility.

In our functional approach we implement a chain of responsibility as below (note the return this statement leading to fluent API making our code more readable):

public class ChainOfResponsibility<T>
 private List<ConditionalAction<T>> _chain = new List<ConditionalAction<T>>();

 private ChainOfResponsibility(ConditionalAction<T> conditionalAction)

 private class ConditionalAction<TInput>
  public Predicate<TInput> If { get; set; }
  public Action<TInput> Do { get; set; }

 public static ChainOfResponsibility<T> Start(Predicate<T> condition, Action<T> action)
  return new ChainOfResponsibility<T>(new ConditionalAction<T>()
   If = condition, 
   Do = action

 public ChainOfResponsibility<T> Then(Predicate<T> condition, Action<T> action)
  _chain.Add(new ConditionalAction<T>()
   If = condition,
   Do = action
  return this;

 public void Run(T input)
  foreach (var conditionalAction in _chain)

 public ChainOfResponsibility<T> Else(Action<T> action)
  return Then((i) => true, action);

This is a simple implementation which defines a generic type of T for inputs of conditions and actions. Now in order to make our code even more readable, let's define some static extension for IsDividable and Ouput for console:

public static class Int32Extensions
 public static bool IsDividableBy(this int a, int b)
  return a%b == 0;

public static class ObjectExtensions
 public static void Output(this object o)
 public static void OutputLine(this object o)

Now to use the code, all we have to write is:

var chain = ChainOfResponsibility<int>
 .Start(i => i.IsDividableBy(15), i => "FizzBuzz".OutputLine())
 .Then(i => i.IsDividableBy(3), i => "Fizz".OutputLine())
 .Then(i => i.IsDividableBy(5), i => "Buzz".OutputLine())
 .Else(i => i.OutputLine());

Enumerable.Range(rangeStart, rangeEnd+1).ToList()
 .ForEach(x => chain.Run(x));

This code is not necessarily shorter or more performant. But it is much more readable/maintainable and the intention of the developer and underlying logic is apparent just be reading the code as a sentence.

Getting the user input

if you have ever worked with a console app and tried to get the user input from the standard input, you definitely have experienced how clumsy the implementation can be.

Now we want to do this in a simple yet generic fashion using functional programming.

Our logic can be simplified and represented as below:
  • A type: type of the value to be cast from string input
  • A validation function to verify the value
  • A conversion function
  • and A message to be shown in case the value is not in the correct format.
So all of this can be simplified as the function below:

private static T GetUserInput<T>(string message, 
 Predicate<string> validation, Func<string, T> conversion)
 while(true) // ideally we must allow an escape route for the user but they can just close the application if they are tired so we just create an endless loop
  string value = Console.ReadLine();
  if (validation(value))
   return conversion(value);

So we can define the logic as below. Note that the validation of the end range makes sure this value is larger than the range start:

int tempInt = 0;
int rangeStart = GetUserInput<int>("Please enter start of the range (positive numbers):",
 s => int.TryParse(s, out tempInt) && tempInt >= 0,
 s => int.Parse(s));

int rangeEnd = GetUserInput<int>("Please enter end of the range (positive and larger than start):",
 s => int.TryParse(s, out tempInt) && tempInt > rangeStart, // note the "Closure" of rangeStart
 s => int.Parse(s));


C# is not a functional language but contains many of the functional languages constructs. Lambda expressions, Func and Action, etc help us to be able to write a more readable code by composing functions.

Functional C# is definitely fun!