Media type: how much can you cram into a single token?

[Level C4]

Introduction

This post discusses the problems associated with the use of a single token as media type (usually as the main value of the Content-Type header in HTTP response or Accept header in request) to describe all attributes of the content.

Motivation and background

This has been bugging me for a while. But recently I engaged in a discussion on twitter with Glenn Block @gblock and the rest of the REST enthusiast community on the options in versioning RESTful services. There are generally 2 camps: those advocating using Content Negotiation for versioning (putting version number in Content-Type header) and those preferring to stick to classic resource based versioning (including version number in the URL). Regardless of which one is better, MediaType lacks the richness required to express a media type and adding version information to a media type is not possible considering current status of the media type.

One of the main problems associated with the use of media type is its current implementation in various systems is key based, i.e. it involves matching all or none of the media type. As we will see this causes considerable problems in effective consumption of media types.

Media Type

Media type has been described in various RFCs (main one being RFC 2046) while historically these have been limited what is known as MIME types. RFC 4288 defines the procedure for registering the media types describing a formal process which needs to be followed to publicly register.

Registering a media type for a public API is all well and good but as described by this book, use of private APIs far exceeds use of public ones and registering all media types exposed within private APIs is impractical and unwarranted.

Also with popularity of REST-based APIs, there are going to be more and more service endpoints exposed. If all such services are to define new media types, we would have an explosion of media types rendering current implementation of content negotiation 

Media type is a case of an extreme semantic mix-up. A single token has been used to express many different facets of a media type. In fact the semantic space with all its axes will contain many useful points yet industry currently uses a very sparse set of points defined as media type values. Rest of this space is unusable - as such a very inefficient solution.

We will now have a look at facets/axes.

1- Human-illegibility

This is the lowest and least specific level of semantic definition of a media type. It is very simple: content of a media type can be read by a human (for example text/plain, application/xml or application/json) or the data is meant for the machine comprehension or rendering (for example image/png or video/mpeg)

Having this information separate to the actual media type can help tools such as Fiddler to decide whether they can display text of the content whose media type is unknown to the tool. Media types initially used "text" to denote such information (e.g. text/xml or text/javascript) but these have been replaced with 

2- Formatting

This is the most common and important axis of a media type information which informs the tools/clients which parser/interpreter/renderer to use for consuming such content. text/plain, application/xml, application/json, image/png or video/mpeg are all examples of such use of the media type. 

There are several known vendor-specific media types in this space such as application/vnd.ms-excel.

3- Schema

This is a further specialisation of the formatting. Common examples include application/rss+xml or application/hal+json. Basically these mean that in terms of formatting, they are the same as their parent (application/xml or application/json) yet they follow a superset schema. Use of + sign - as far as I know - is not canonical and is merely a convention followed by the industry to add schema to the established formats. Comprehension of this convention would be crucial to correct interpretation of the media type without the need for having a dictionary of all possible values, however, I believe most tools we have at the moment lack such features.

4- Domain/Vendor specific

This is where we see most of the expansion in the media type space. Basically you could output your own media type via your private API. Since you will be the main consumer of the API, integration could be easy but it is very common for private APIs to go public - especially if they are successful. An example of such media types can be found here.

5- Versioning

Versioning is the highest aspect of a media type which is normally added to Domain-specific media types. This is a popular solution to the Web API versioning problem.

For example, you could have application/mydomain.customer.1.1 as opposed to application/mydomain.customer or application/mydomain.customer.1.0

So where is the problem?

Basically information gets lost.

First problem is that clients might be interested in a lower order of these aspects of media type while in order to consume the resource, they are forced to comprehend higher order and extract the axes they are interested in. For example, a tool such as fiddler could be only interested in only whether it could display the information for the end user as plain text. A client capable of consuming XML and deserialising to objects is only interested at knowing whether it is XML while it might be represented with a media type which is essentially XML but has a different value. On the other hand, if a server uses HAL to send domain objects/view models to the client, either it has to use the standard application/hal+json or use the domain level name of the media type (with or without a version).

Another problem is that the content negotiation process will become more complex. In the lack of a standard in defining multi-axial media types, most systems implement a dictionary based rule on content negotiation as such maintaining list of possible content types becomes a burdensome task.

A solution

Basically I believe we can solve this by keeping the common media types but use media type extensions in the Content-Type header (or in the Accept header). For example:

Content-Type: application/xml; human-illegible=true; domain-name=customer; domain-version=1.1

This will ensure that existing clients and servers will not break while new clients and servers can use new extensions for content negotiation and more loosely coupled resource consumption. I will try to expand upon this idea in another post.

Conclusion

Cramming as much as information into a single token and then try parsing that one token is not a good idea especially when it comes to media type which is the communication bridge between loosely coupled world of HTTP clients and servers.

Media type token value covers 5 different aspects of the resource and separating the concerns of breaking these aspects into their own tokens can result in more robust and decoupled systems.

ASP.NET Web API Series - Part 5: MediaTypeFormatter explained

Introduction

[Level T2] MediaTypeFormatter is an exciting concept introduced in the ASP.NET Web API which will enable seamless conversion of HTTP data to/from.NET types. This post reviews the concepts and basic usage of MediaTypeFormatter in the ASP.NET Web API pipeline. This is an area of Web API which is being actively developed so the content of this post might be updated to reflect the changes - but this post at the time of publishing is based on the latest source code available.

Background

HTTP abstracts a resource (identified by a URI which is commonly a URL) from its representation. A resource e.g. an employee detail can be identified by /employees/123. An HTTP agent (client) and a server engage in content negotiation to decide on the best format it can be represented. For example, a client can express its wishes to receive employee detail in plain text (by specifying content-type header of text/plain), RTF, XML, JSON or even image.

On the other hand, ASP.NET Web API has also abstracted away parameters or result of an action form its representation. While ASP.NET MVC had this feature for input parameters, return type should have been an instance of ActionResult hence controller had to make a decision on the format of resource by returning ContentResult, JsonResult, etc.

MediaTypeFormatter as we will see will bridge the gap between these two abstractions.

What is Media Type?

As you all probably know, media type refers to the value of the content-type header within an HTTP request and response. Media types allow agent (client) and server to define the type of the data sent in the HTTP body (payload). It is also used within the accept header in the request to allow content negotiation, i.e. client notifies the server of the media types it accepts/prefers. I will need to have a separate post on content negotiation but as for now, this little introduction suffices.

There are standard media types as listed in the Wiki link. There is no limitation on the media types and you can come up with your own media types but these media types usually start with X-.

A request or response does not have to have a single media type. It can mix the media types but in this case it has to use multipart content-type (value of the content type will be multipart/mixed) so that each part defines its content type.

What is MediaTypeFormatter?

Media type formatter is the bridge between the HTTP world of URI fragments, headers and body on one side, and the controller world of actions, parameters and return types.

Tower Bridge of ASP.NET Web API

A media type formatter in brief:

1. Has a knowledge of one or more media type (for example text/xml and application/xml both refer to the same structure which is XML) and tells Web API which content types it supports (for the HTTP world)
2. Tells Web API whether it can read or write a type (for Controller world)
3. Has an understanding of encoding/charset which is passed in the HTTP header and can read accordingly
4. It will be given a stream to read (from request) or write (to response)
5. Its work usually (but not always) involves serialisation (at the time writing to the response) or deserialisation (at the time of reading from the request)
6. Inherits abstract class MediaTypeFormatter

MediaTypeFormatter class

MediaTypeFormatter class in the ASP.NET Web API is an abstract class providing base services for various media type formatters.

Important properties and methods include (more informative as code):

public abstract class MediaTypeFormatter
{

 // properties
 public Collection<MediaTypeHeaderValue> SupportedMediaTypes { get; private set; }
 
 public Collection<Encoding> SupportedEncodings { get; private set; }
 
 public Collection<MediaTypeMapping> MediaTypeMappings { get; private set; }
 
 // methods
 public virtual Task<object> ReadFromStreamAsync(Type type, Stream stream, HttpContentHeaders contentHeaders, IFormatterLogger formatterLogger)
 {
  // to be overriden by base class
 }

 public virtual Task WriteToStreamAsync(Type type, object value, Stream stream, HttpContentHeaders contentHeaders, TransportContext transportContext)
 {
  // to be overriden by base class
  }

 public abstract bool CanReadType(Type type);

 public abstract bool CanWriteType(Type type);

}

Things to note above are:

• As with the rest of the Web API, MediaTypeFormatter fully supports Async using TPL. Having said that, most implementations of  MediaTypeFormatter run synchronously as they involve serialisation which is safe as a synchronous operation. 
• SupportedMediaTypes defines a list of media type headers that an implementation supports. For example application/xml and text/xml
• MediaTypeMappings is an interesting concept where a media type formatter can define its preference for a particular media type based on a value in the request (query string, URI fragment, HTTP header). A typical example is existence of x-requested-with header which signals the AJAX based request hence JSON is the preferred content-type.

How ASP.NET Web API uses formatters?

Media type formatters are global formatters sitting in the Formatters property of HttpConfiguration. If you are using ASP.NET hosting (IIS, Cassini, etc) then you may use GlobalConfiguration.Configuration to access the instance of HttpConfiguration containing Formatters. If you are using Self-Hosting, then you would be creating a HttpSelfHostConfiguration object which you will use its Formatters property.

This snippet will output all formatters that are setup by default in the ASP.NET Web API:

foreach (var formatter in config.Formatters)
{
 Trace.WriteLine(string.Format("{0}: {1}", 
  formatter.GetType().Name,
  string.Join(", ", formatter.SupportedMediaTypes.Select(x=>x.MediaType))
  ));
}

And here is the output (at the time of writing this blog):

JsonMediaTypeFormatter: application/json, text/json
XmlMediaTypeFormatter: application/xml, text/xml
FormUrlEncodedMediaTypeFormatter: application/x-www-form-urlencoded
JQueryMvcFormUrlEncodedFormatter: application/x-www-form-urlencoded

This list is very much likely to be extended by the time ASP.NET Web API is shipped. 

You might be surprised to see two different media type formatters targeting the same media type. But that is very normal: media type formatters compete for becoming the formatter of choice! If ASP.NET Web API find two formatters for the same content type, it will pick the first one so it is very important to add formatters in the right order.

Writing a simple BinaryMediaTypeFormatter

Now, we want to get our hands dirty and implement a useful formatter that is not currently provided by the ASP.NET Web API. This formatter will be able to formatter application/octet-stream media type in the HTTP world to the byte[] type in the controller world.

Let's imagine we have a controller that calculates SHA1 hash of the small binary data posted to it (this is a good practice for large streams):

public class BinaryController : ApiController
{
 [HttpPost]
 public string CalculateHash(byte[] data)
 {
  using(var sha1 = new SHA1CryptoServiceProvider())
  {
   return Convert.ToBase64String(sha1.ComputeHash(data));       
  }
 }
}

In our implementation, we use synchronous approach, although in this case it is safe to use asynchronous as there is no serialisation taking place. However, since this is intended only for small payloads, context switching of the asynchronous TPL has more overhead - in any case turning this code into asynchronous is very easy: an alternate implementation supporting async can be found here.

public class BinaryMediaTypeFormatter : MediaTypeFormatter
{

 private static Type _supportedType = typeof (byte[]);
 private const int BufferSize = 8192; // 8K 

 public BinaryMediaTypeFormatter()
 {
  SupportedMediaTypes.Add(new MediaTypeHeaderValue("application/octet-stream"));
 }

 public override bool CanReadType(Type type)
 {
  return type == _supportedType;
 }

 public override bool CanWriteType(Type type)
 {
  return type == _supportedType;
 }

 public override Task<object> ReadFromStreamAsync(Type type, Stream stream, 
  HttpContentHeaders contentHeaders, IFormatterLogger formatterLogger)
 {
  var taskSource = new TaskCompletionSource<object>();
  try
  {
   var ms = new MemoryStream();
   stream.CopyTo(ms, BufferSize);
   taskSource.SetResult(ms.ToArray());
  }
  catch (Exception e)
  {
   taskSource.SetException(e);
  }
  return taskSource.Task;
 }

 public override Task WriteToStreamAsync(Type type, object value, Stream stream, 
  HttpContentHeaders contentHeaders, TransportContext transportContext)
 {
  var taskSource = new TaskCompletionSource<object>();
  try
  {
   if (value == null)
    value = new byte[0];
   var ms = new MemoryStream((byte[]) value);
   ms.CopyTo(stream);
   taskSource.SetResult(null);
  }
  catch (Exception e)
  {
   taskSource.SetException(e);
  }
  return taskSource.Task;
 }
}

Using BinaryMediaTypeFormatter

Now let's use our formatter. You need a REST console of your choice (Chrome REST console, REST Sharp library, Fiddler) to send this request:

POST http://localhost:7777/api/Binary HTTP/1.1
User-Agent: Fiddler
Host: localhost:7777
content-type: application/octet-stream
Content-Length: 14

This is a test

Since we have not yet added our formatter, we get back this error:

No MediaTypeFormatter is available to read an object of type 'Byte[]' from content with media type 'application/octet-stream'.

So we just need to add our formatter:

config.Formatters.Add(new BinaryMediaTypeFormatter());

And we will get back this response:

HTTP/1.1 200 OK
Content-Length: 30
Content-Type: application/json; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
Date: Sat, 28 Apr 2012 12:09:44 GMT

"pU2I4GYS2CC8O+cod8dPJXtWGxk="

As you can see, the response content type is application/json. I have explained in my post Part 1 why it is the case: JsonMediaTypeFormatter is the default media type formatter. Now we know why it is the case: it is the first item in the collection (see above).

Conclusion

Media type formatter is the bridge between the HTTP world of URI, headers and body on one side, and the controller world of actions, parameters and return types. In ASP.NET Web API, it is represented by abstract class MediaTypeFormatter. Order of formatters in the Formatters property of HttpConfiguration is important when ASP.NET Web API has to choose between two formatters supporting the same media type.