XML or JSON, and that is not the question

So in last couple of days, our .NET community has showed some strong reactions to the announcements in the ASP.NET team stand-up. While Ruby and more recently node community are known to endless dramas on arguably petty issues, it felt that .NET community was also capable of throwing tantrums. For those who are outside .NET community or have not caught up with the news, .NET/ASP.NET team have decided to revert the project.json (JSON) in #DotNetCore back to *.csproj/*.vbproj (XML) and resurrect msbuild. So was it petty in the end?

Some believed it was: they argued all that was changed was the format of the project file and the drama associated with it was excessive. They also pointed out that all the goodness of project.json would be ported to the familiar yet different *.csproj. I call this group the loyalists:

@palantirza @terrajobst @tourismgeek @aliostad I'm amazed that you picked a technology based on the format of the project file.

— Isaac Abraham (@isaac_abraham) May 12, 2016

On the other hand, some were upset by the return of the msbuild to the story of .NET development. This portion of the community were arguing that +15-year-old msbuild does not have a place in the modern development. They have been celebrating death of this technology not knowing it was never really dead - I call them msbuild-antagnoists. The first group (loyalists), on the other hand, were flagging that the msbuild would be improved and the experience would be modernised.

That PR replacing MSBuild with Make doesn’t seem so foolish now does it?

— Colin Scott (@AbstractCode) May 11, 2016

Now there were another group of people were frustrated that this decision had been made despite the community feedback and solely based on the feedback of “some customers” behind the closed doors. I call them OSS-apologetics and their main issue was the seemingly lack of weight of the community feedback when it comes to the internal decisions that Microsoft takes as a commercial enterprise - especially in the light of the fact that project.json was announced almost 2 years ago and it was very late to change it.

@terrajobst @tourismgeek @isaac_abraham @aliostad And there it is: yeah, it's not REALLY OSS, and the community doesn't matter.

— Palantir (@palantirza) May 12, 2016

Now there were yet another group that had invested time and effort (==money?) in building projects and tooling (some of which commercial) and they felt that the rug has been pulled from underneath them and all those hours gone to waste - for the lack of a better phrase I call them loss-bearers. And they were even more upset to see that their loss was accounted as a learning process:

I always tell developers not to worry when their code gets thrown away. Learning is the most important thing.

— David Fowler (@davidfowl) May 11, 2016

Obviously there is not a great answer for them but it is usually said that it is a very minor part of the whole community who have been living on the bleeding edge and knew it could be coming any minute, as mentioned on the stand-up:

Where do I stand?

I stand somewhere in between. Cannot quite agree with the loyalists since it is not just the question of format. On the other hand, I do not bear any losses since I had decided long time ago that I will skip the betas and pick it up when the train of changes slows down - something not yet in sight.

But I do not think any of the above captures the essence of what has been happening recently. I am on the belief that this decision along with the previous disrupting ones have been important and shrewd business decisions to save the day and contain losses for Mircosoft as a commercial platform - and no one can blame Microsoft for doing that.

I had warned times and times again that the huge amount of change in the API and tooling and no easy migration path will result in dividing the community into a niche progressive #DotNetCore minority and the mainstream commercial majority who would stay on .NET Fx and need years (not months) to move on to #DotNetCore - if at all. And this potentially will create a Python-vs-3-like divide in the community.

@jeremydmiller @Cranialstrain @demisbellot fragmentation (some staying with 4.5 for long time) and then demise of .Net as we know it.

— Ali Kheyrollahi (@aliostad) July 27, 2015

The cross from the old .NET to the new #DotNetCore (seemingly similar on the surface yet wildly different at heart) would not be dissimilar to the cross between VB6 to .NET. And what makes it worse is that unlike then, there are many viable alternatives OSS stacks (back then there was only Java and C/C++). This could have meant that the mainstream majority might in fact decide to try an altogether different platform.

So Microsoft as a business entity had to step in and albeit late, fix the few yet key mistakes made at the start and alongside the project during the last 2 years:

• ASP.NET team to make platform/language decisions and implement features with clever tricks rather the .NET Fx baking such features in the framework itself. An example was Assembly Neutral Interfaces.
• Ignoring the importance upgrade path for the existing projects and customers
• Inconsistent, confusing and ever changing layering of the stack
• Poor and conflicting scheduling messages
• Using Silverlight’s CoreCLR for ASP.NET resulting in dichotomy of the runtime, something that as far as I know has no parallel in any other language/platform. In the most recent slides I do not see CoreCLR being mentioned anymore yet it might be there. If it is, it will stay a technical debt to be paid later.

All in all it has been a rough ride both for the drivers and the passengers of this journey but I feel now the clarity and cohesion is back and long-standing issues have been addressed now.

Where could I be wrong?

My argument naturally brings these counterarguments:

• Perhaps had ASP.NET team not pushed the envelope this far by single-handedly crusading to bring modern ideas and courageous undertakings such as cross-platform, we would be having .NET 5 now instead of #DotNetCore.
• By carrying baggage from the past (msbuild), Microsoft is extending the lifespan its stacks which in the short term will be beneficial to the corporate but since it is not a clean break, in the long term results in dispersion of the community and a need for another redeux.

Hard to answer these arguments since one is a hypothetical situation and the other looks well into the uncertainty of the future. I will leave it to the readers to weigh the arguments.

Last word

It is not possible to hide that none of this has been without casualties. Some confidence lost, community at times upset and overall has not been all rosy as far as the Microsoft’s image in its OSS ventures goes. I did mention old and new Microsoft coming head-to-head, which might not be correct but as Satya Nadella said, culture does not change overnight.

Thank you Microsoft, nine months on ...

Level [C1]

I felt that almost a year after my blog post Thank you Microsoft and so long, it is a right time to look back and contemplate on the decision I made back then. If you have not read the post, well in a nutshell, I decided to gradually move towards non-Microsoft technologies - mainly due to Microsoft's lack of innovation, especially in the Big Data space.

A few things have changed since then. TLDR; Generally it really feels I made the right decision to diversify. Personally, I have learnt a lot and at the same time I had a lot of fun. I have built a bridge to the other side and I can easily communicate with non-Microsoft peers and translate my skills. On the technology landscape, however, there has been some major changes that makes me feel having a hybrid skillset is much more important than a complete shift to any particular platform.

I will first look at the technology landscape as it stands now and then will share my personal journey so far in adopting alternative platforms and technologies.

A point on the predictions

OK, I made some predictions in the previous post such as "In 5 years, all data problems become Big Data problems". Some felt this is completely wrong - which could be - and left some not very nice messages. At the end of the day, predictions are free (and that is why I like them) and you could do the same. I am sharing my views, take it as it is worth for you. I have a track record of making predictions, some came true and some did not. I predicted a massive financial crash in 2011 which did not happen and lead to one of the biggest bull markets ever (well my view is they pumped money into the economy and artificially made the bull market) and I lost some money. On the other hand back in 2010 I predicted in my StackOverflow profile something that I think it is called Internet Of Things now, so I guess I was lucky (by the way, I am predicting a financial crash in the next couple of months). Anyway, take it easy :)

Technology Horizon

The New Microsoft

Since I wrote the blog post, a lot has changed, especially in Microsoft. It now has a new CEO and a radically different view on the Open Source. Releasing the source of a big chunk of the .NET Framework is harbinger of a shift whose size is difficult to guess at the moment. Is it mere a gesture? I doubt it, this adoption was the result of years of internal campaign from the likes of Phil Haack and Scott Hanselman and it has finally worked its way up the hierarchy.

But adopting Open Source is not just a community service gesture, it has an important financial significance. With the rate of change in the industry, you need to keep an army of developers to constantly work and improve products at this scale. No company is big enough on its own to build what can be built by an organic and healthy ecosystem. So crowd-sourcing is an important technique to improve your product without paying for the time spent. It is probably true that the community around your product is the real IP of most cloud platforms and not so much the actual code.

Microsoft is also relinquishing its push strategy towards its Operating System and to be honest, I am not surprised at all. Many have talked about the WebOS but reality is we have already had it for the last couple of years. Your small smartphone or tablet come to life when they are connected - enabling you to do most of what you can do on the laptop/pc with the only limitation being the screen size. On the other hand, Microsoft has released the web version of the office and to be fair it is capable of doing pretty much everything you can do in the desktop versions, and sometimes it does it better. So for the majority of consumers, all you need is the WebOS. It feels that the value of a desktop operating system becomes of less and less importance when most of the applications you use daily are web-based or cloud-based.

Cloud and Azure

I have been doing a lot of Azure both at work and outside it. Apart from HDInsight, I think Azure is expanding at a phenomenal rate in both feature and reliability and this is where I feel Microsoft is closing in the Innovation Gap. It is mind-blowing to look at the list of new features that are coming out of Azure every month.

Focusing mainly on the PaaS products, I think future of Azure in terms of adoption by the smaller companies is looking more and more attractive compared to AWS which has traditionally been IaaS platform of choice. Companies like Netflix have built all their software empire on AWS but they had an army of great developers to write the tooling and integration stuff.

All-in-all I feel Azure is here to stay and might even overtake AWS in the next 5 years. What will be a decider is the innovation pace.

Non-Hadoop platforms

A recent trend that could change the balance is the proliferation of non-Hadoop approaches to Big Data which will favour Microsoft and Google. With Hadoop 2.0 trying to abstract away even more the algorithm from the resource management, I think there is an opportunity for Microsoft to jump in and plug-in a whole host of Microsoft languages in a real way - it was possible to use C# and F# before but no one really used it.

Microsoft announced the release AzureML which is the PaaS offering of Machine Learning on the Azure Platform. It is early to say but it looks like this could be used for smaller-than-big-date machine learning and analysis. This platform is basically productionising of the Machine Learning platform behind the Bing search engine.

Another sign that the Hadoop's elephant is getting old is Google's announcement to drop MapReduce: "We invented it and now we are retiring it". Basically in-memory processing looks more and more appealing due to the need for quicker feedback cycle and speeding up processes. Also it seems that there is resurgence of focus towards in-memory grid computing, perhaps as a result of Actor Frameworks popularity.

In terms of technologies, Spark and to a degree Storm are getting a lot of traction and the next few months are essential to confirm the trend. These still very much come from a JVM ecosystem but there is potential in building competitor products.

Personal progress

MacBook

This is the first thing I did after making the decision 9 months ago: I bought a MacBook. I was probably the farthest thing away from being an Apple fanboy, but well it has put its hooks in me too now. I wasn't sure if I should get a Windows laptop and run a Linux VM on it, or buy a MacBook and run Windows VM. Funny enough, and despite my presumptions, I found the second option to be cheaper. In fact I could not find an Windows UltraBook with 16 GB of RAM and that is what I needed to be able to comfortably run a VIM. So buying a 13.3" MacBook pro proved both economical (in the light of what you get back for the money) and the right choice - since you want your VM to be your secondary platform.

Initially I did not like OSX but it helped me to get better at using the command line - be it the OSX variant of Linux commands. Six months on, similar to what some of my twitter friends had said, I don't think I will ever go back to Windows.

I have used Mac for everything apart from using Visual Studio and occasional Visio - also using some Azure tools had to be on Windows. I think I now spend probably only 20% of my time in Windows, the rest in Linux (Azure VM) and OSX.

Linux, Shell scripting and command line

I felt like an ignorant to find out the wealth of command line tools at my disposal in OSX and Linux. Find, Grep, Sort, Sed, tail, head, etc just amazing stuff. I admit, for some there might be windows equivalent that I have not heard of (which I doubt) but it really makes the life so easier to automate and manage your servers. So been working on understanding services on Linux and OSX, learning about Apache and how to configure it... I am no expert by any stretch but it has been fun and learnt a lot.
And yes, I did use VIM - and yes, I did find it difficult to exit it the first time :) I am not mad about it, I just have to use it on Linux VMs I manage configs, etc but cannot see myself using it for development - at least anytime soon.

Languages

As I said the, I had decided to start with some JVM languages. Scala felt the right choice then and with knowing more about it now, even more so. It is powerful yet all the wealth of Java libraries are at your fingertip. It is widely adopted (and Clojure the second candidate not so much). Erlang probably not appropriate now and go is non-JVM. so I am happy with that decision.

Having said that, I could not learn a lot of it. Instead I had to focus on Python for a personal NLP project - well, as you know most NLP and data science tools are on Python. I had to learn to code, understand its OOP and functional side, its versioning and distribution and finally above all being able to serve REST APIs (using Flask and RESTful-Flask) for interop with my other C# code.
My view on it? Python is simple and has a nice built-in support for important data structures (list, map, tuple, etc) making it ideal for working with the data. So it is a very useful language but it is not anywhere near as elegant as Scala or even C#. So for complex stuff, I would still rather coding in C#, until I properly pick up Scala again. I am also not very comfortable with distributing non-compiled code - although that is what we normally do in JavaScript (minimising aside), perhaps another point of similarity between these two.

Apart from these, I have still been doing a ton of C#, as I had predicted in the previous blog post. I have been working on a Cloud Actor Mini-Framework called BeeHive which I am currently using myself. I still enjoy writing C# and am planning to try out Mono as well (.NET on OSX and Linux). Having said that, I feel tools and languages best to be used in their native platform and ecosystem, so I am not sure if Mono would be a viable option for me.

Conclusion

All-in-all I think by embracing the non-Microsoft world, I have made the right decision. A new world has been suddenly opened up for me, a lot of exciting things to learn and to do. I wish I had done this earlier.

Would I think I will completely abandon my previous skills? I really doubt it: The future is not mono-colour, it is a democratised hybrid one, where different skillsets will result in cross-pollinisation and producing better software. It feel having a hybrid skill is becoming more and more important, and if you are looking to position yourself better as a developer/architect, this is the path you need to take. Currently cross-platform/hybrid skills is a plus, in 5 years it will be a necessity.

Thank you Microsoft, and so long ...

[Level C1]

Ideas and views expressed here are purely my own and none of my employer or any related affiliation.

Yeah, I know. It has been more than 13 years. Still remember how it started. It was New Years Eve of the new millennium (yeah year 2000) and I was standing on the shore of the Persian Gulf in Dubai. With my stomach full of delicious Lebanese grilled food and a local newspaper in hand looking for jobs. I was a frustrated Doctor, hurt by the patient abuse and discouraged by the lack of appreciation and creativity in my job. And the very small section of medical jobs compared to pages, after pages, after pages of computer jobs. Well, it was the dot.com boom days, you know, it was the bubble - so we know now. And that was it. A life-changing decision to self-teach myself network administration and get MCSE certificate.

Funny thing is I never did. I got distracted, side-tracked by something much more interesting. Yeah programming, you know, when it gets its hooks in you, it doesn't leave you - never. Hence it started by Visual Basic, and then HTML, and obviously databases which started from Microsoft Access. Then you had to learn SQL Server, And not to forget web programming: ASP 3.0. I remember I printed some docs and it was 14 pages of A4. It had everything I needed to write ASP pages, connect to database, store, search, everything. With that 14 pages I built a website for some foreign Journalists working in Iran: IranReporter - It was up until a few years ago. I know it looks so out-dated and primitive but for a 14-page documentation not so bad.

Then it was .NET, and then move to C# and the rest of the story - I don't bore you with. But what I knew was that the days of 14-page documentation were over. You had to work really hard to keep up. ASP.NET Webforms which I never liked and never learnt.  With .NET 3.0 came LINQ, WCF, WPF and there was Silverlight. And there was ORMs, DI frameworks, Unit Testing and mocking frameworks. And then there was Javascript explosion.

And for me, ASP.NET Web API was the big thing. I have spent best part of the last 2 years, learning, blogging, helping out, writing OSS libraries for it and then co-authoring a book on this beautiful technology. And I love it. True I have not been recognised for it and am not an MVP - and honestly not sure what else could do to deserve to become one - but that is OK, I did what I did because I loved and enjoyed it.

So why Goodbye?

Well, I am not going anywhere! I love C# - I don't like Java. I love Visual Studio - and eclipse is a pain. I adore ASP.NET Web API. I love NuGet. I actually like Windows and dislike Mac. So I will carry on writing C# and use ASP.NET Web API and Visual Studio. But I have to shift focus. I have no other option and in the next few lines I try to explain what I mean.

Reality is our industry is on the cusp of a very big change. Yes, a whirlwind of the change happened already over the last few years but well, to put it bluntly I was blind.

So based on what I have seen so far I believe IN THE NEXT 5 YEARS:

1. All data problems become Big Data problems

If you don't like generalisations you won't like this one too - when I mean all I mean most. But the point is industries and companies will either shrink or grow based on their IT agility. The ones shrinking not worth talking about. The ones growing will have big data problems if they do not have them yet. This is related to the growth of internet users as well as realization of Internet of Things. And the other reason is the adoption of Event-Based Architecture which will massively increase the amount of data movement and data crunching.

Data will be made of what currently is not data. Your mouse move and click is the kind of data you would have to deal with. Or user eye movement. Or input from 1000s of sensors inside your house. And this stuff is what you will be working on day-in day-out.

Dear Microsoft, Big Data solutions did not come from you. And despite my wishes, you did not provide a solution of your own. I love to see an alternative attempt and a rival to Hadoop to choose from. But I am stuck with it, so better embrace it. There are some alternatives emerging (real-time, in-memory) but again they come from the Java communities.

2. Any server-side code that cannot be horizontally scaled is gonna die

Is that even a prediction? We have had web farms for years and had to think of robust session management all these years. But something has changed. First of all, it is not just server-stickiness. You have to recover from site-stickiness and site failures too. On the other hand, this is again more about data.

So the closer you are to the web and further from data, the more likely that your technology will survive. For example, API Layer technologies are here to stay. As such ASP.NET Web API has a future. Cannot say the same thing for middleware technologies, such as BizTalk or NServiceBus. Databases? Out of question.

3. Data locality will still be an issue so technologies closer to data will prevail

Data will still be difficult to move in the years to come. While our bandwidth is growing fast, our data is growing faster so this will probably be even a bigger problem. So where your data is will be a key in defining your technologies. If you use Hadoop, using Java technologies will make more and more sense. If you are Amazon S3, likelihood of using Kafka than Azure Service Bus. 

4. We need 10x or 100x more data scientists and AI specialists

Again not a prediction. We are already seeing this. So perhaps I can sharpen some of my old skills in computer vision.

So what does it mean for me?

As I said, I will carry on writing C#, ASP.NET Web API and read or write from SQL Server and do my best to write the best code I can. But I will start working on my Unix skills (by using it at home) and pick up one or two JVM languages (Clojure, Scala) and work on Hadoop. I will take Big Data more seriously. I mean real seriously... I need to stay close to where innovation is happening which sadly is not Microsoft now. From the Big Data to Google Glass and cars, it is all happening in the Java communities - and when I mean Java communities I do not mean the language but the ecosystem that is around it. And I will be ready to jump ships if I have to.

And still wish Microsoft wakes up to the sound of its shrinking, not only in the PC market but also in its bread and butter, enterprise.

PS: This is not just Microsoft's problem. It is also a problem with our .NET community. The fight between Silverlight/XAML vs. Javascript took so many years. In the community that I am, we waste so much time fighting on OWIN. I had enough, I know how to best use my time.