Tuesday, 13 August 2019

What is fluentd and its support in PerfIt


After a few years of several contending products and tools in the tracing middleware space, it seems tthe industry has settled on fluentd. I believe this is probably because it has maintained technology-neutrality when it comes to ultimate destination of metrics and logs, i.e. storage/visualisation/search/alerting aspect.

The journey had started with Etsy’s statsd back in 2012. It allowed for collection of metrics mainly using UDP which started an important if not revolutionary trend. On the other hand, Zipkin project came out of internal works in Twitter inspired by Google’s Dapper paper.

Zipkin had poised to become the universal Open Source distributed tracing solution. It certainly inspired the OpenTracing initiative but the more OpenTracing grew, the more it dented the status of Zipkin, as OpenTracing improved on some of the shortcomings of Zipkin that were kept in order to maintain backward compatibility. I actually worked with Zipkin community and its amazing community leader, Adrian Cole, to bring Zipkin support to Azure Event Hubs. I also started adding support for Zipkin to the PerfIt project (I did not get a chance to use it in anger and I doubt anyone else used it). Regardless of its merits, drawbacks and its future, it couples middleware with storage and search and is not a pure middleware.

Another contender was LogStash which then morphed into Beats and was the default choice for collection of metrics and logs in Elasticsearch. But then again, these technologies are optimised for use with Elasticsearch and Kibana.

Zipkin had poised to become the universal Open Source distributed tracing solution. It certainly inspired the OpenTracing initiative but the more OpenTracing grew, the more it dented the status of Zipkin, as OpenTracing improved on some of the shortcomings of Zipkin that were kept in order to maintain backward compatibility. I actually worked with Zipkin community and its amazing community leader, Adrian Cole, to bring Zipkin support to Azure Event Hubs. I also started adding support for Zipkin to the PerfIt project (I did not get a chance to use it in anger and I doubt anyone else used it). Regardless of its merits, drawbacks and its future, it couples middleware with storage and search and is not a pure middleware.

Another contender was LogStash which then morphed into Beats and was the default choice for collection of metrics and logs in Elasticsearch. But then again, these technologies are optimised for use with Elasticsearch and Kibana.



fluentd is ubiquitous

Now, with fluentd being a default choice with Kubernetes, it understandably has gained wide adoption. Having said that, I believe it says more about fluentd than Kubernetes as there is so much to love about fluentd since it:
  • Is simple and straightforward
  • Is Open Source
  • Supports many inputs/transports (HTTP, UDP, file)
  • Supports common output types (Elasticsearch, s3, kafka)
  • Supports many formats/parsers (csv, json, nginx)
  • Supports transformations and buffering
  • Does not try to solve high availability and instead provides probes so that infrastructure can monitor
  • Provides extension points using the plugin model
So simply, it can accept logs, metrics and traces from your applications, while running as a Kubernetes service, a sidecar to your application or a simple deamon, and then push them to your destination of choice so your application does not have to worry about buffering, retry, destinations, etc. Using UDP which is essentially fire and forget, you completely decouple trace collection from your code - practically zero to little overhead.

While it initially targeted as a Big Data transformation middleware, it is clear now that its strength will be in the observability space. That is why I believe it will be a glue that will be binding producers and consumers of observability data for the coming years. And that is the main reason I felt I had to support it in PerfIt.

PefIt and fluentd

For those of you who are not familiar with PerfIt, I started the project more than 6 years ago for instrumenting .NET systems - in the absence of similar tool. Initially it was meant only for custom Windows performance counters but gradually grew to support ETW and finally with .NET Core to support other forms of transport such as Azure Event Hubs.
Now I support fluentd using UDP transport.

Getting started

I have blogged in the past on using PerfIt but essentially you create an instrumentor and instrument a piece of code:
var si = new SimpleInstrumentor(new InstrumentationInfo()
{
    Description = "Instruments your code!",
    Name = "general",
    CategoryName = "my app",
    InstanceName = "Sleep100"
});

si.Instrument(() =>
{
    Thread.Sleep(100);
});
You can also achieve this using AOP with attributes in ASP.NET Web Api or Core MVC.

You can register tracers which are essentially outputs/sinks for your captured metrics. Currently Azure EventHubs, Zipkin and fluend are supported - the latter is the new addition:
// add the tracer after creating instrumentor - sends UDP datagrams to localhost on port 5160
si.Tracers["fluentd"] = new UdpFluentdTracer("localhost", 5160);
That is really it! This means all your *sampled* traces will be sent to your fluentd. Easy peasy...

Saturday, 18 May 2019

Why WebAssembly Matters?

WebAssembly as a specification-first technology, is probably one of the few innovations of the web where the spec is not a retrospective documentation of the technology. WebAssembly community group started in 2015 and with its version 1.0, nowadays it feels like it has crossed from an experimental status into a production-ready first-generation technology - let alone the same also being claimed on its website.

On the browser, execution of any application code other than javascript is immediately reminiscent of the pile of technologies of the kind, that nowadays have little significance other than their historical ones: Java applets, Adobe Flash and Microsoft Silverlight. It has been proved over and over that Web Always Wins.

This has recently sparked conversations and debates on the community on the merits, use cases and typical scenarios where you would use WebAssembly (and especially Blazor in the .NET community, although Blazor adds a server-side scenario which simply is not WebAssembly).
WebAssembly

The classic view is that WebAssembly will allow C, C++ and Rust developers to be able to join the web’s cool party. For some, it is a game changer: it will revolutionise Web Application development - potentially even sideline Javascript. For others, it has (albeit limited) use-cases for high-computation gaming. Some propose that AI on the browser could be a growing market and enhance the capabilities of today and tomorrow web sites. And yet others feel it will die off as it is competing with web, and the web always wins.

Of course, it is different to compiled-code technologies of the past. Unlike them, it is an Open standard adopted equally by all major players. It does not need a plugin to run - since, simply as it were, the plugin is distributed along with the browser.

There is a level of truth to many of these statements: it will shine in special case scenarios such as gaming and AI. But that is NOT why I think it matters, and I am writing these lines. But I believe the main potential has nothing to do with gaming or AI: it is an incidental feature of the WebAssembly and in order to explain it I have to bring example from the web.

Do you remember Ajax? Ajax was a hack by Microsoft Outlook team to load emails asynchronously as they arrive - instead of reloading the page. Now it is impossible to think of the web without Ajax.

Hence, I claim that the most important feature of WebAssembly is its Security (an incidental feature): the fact that it runs on the same sandbox as the browser’s Javascript. In fact, the C, C++ and Rust impact is just a minor distraction: companies will build (and already have) adaptors for high level languages (such as Java, C#, Python) to interact according to the WebAssembly specification.

WebAssembly will revitalise desktop app development

When was the last time you installed an unknown app on your desktop (Windows or Mac)? In the early 2000s, viruses almost completely killed desktop application market for everyone but the major vendors. For many smaller vendors, still the only way to reach escape velocity and conquer mass market is to provide native mobile apps or web-based application and after years of gaining customers’ trust, one might install their desktop app. Who would have just tried Evernote purely as a desktop application?

Apple’s iOS (and then Android) with its rigorous app governance and opt-in ACL policy created a sandbox for the apps to harness them so that they ask for access on resources they absolutely need. This was not an after-thought, but a design decision from day one. Major Desktop Operating Systems do have concepts of kernel mode vs user mode and file ACL but they are not designed for comprehensive opt-in ACL. And that is why even Mac app store feels pretty deserted with tumbleweed rolling on the street, let alone Windows 8/10 whose app store is more like a joke.

I strongly believe WebAssembly will become a vehicle for desktop application delivery. First, it is web-based, allowing the same levels of discoverability, ease of registration, monetisation/subscription and seamless update mechanism. Second, it is 100% secure (or at least as secure as websites we visit everyday) and there is no more a concept of installation. Third, it will make software rental very easy and will allow for the growth of SaaS for compute-intensive applications.

While Electron applications provided ease of developing desktop applications using Web toolkit, WebAssembly will turn this approach on its head and bring native performance to the web.

Web is ubiquitous and WebAssembly will conquer desktop applications.