Douglas Purdy

We just completed the US leg of the roadshow.  Thanks to everyone who attended.  We really hit our stride in Mountain View (it is home court for me), although the NYC and Chicago stops offered some really great customer interactions that we are still talking about.

We’ll be doing a keynote at the European VC Summit (Guy Kawasaki is the MC which I am excited about) and then it is off to Asia, TechEd US, and back to Europe.

We will be taping at least one of those events, which I will post here.

In the meantime, you can get the PDF (PPT is too big for WP to upload and I am too tired to SSH in to the server and fix it).

We are taking the MIX10 Services Powering Experiences keynote on the road.

New York, NY – May 12, 2010
Chicago, IL – May 14, 2010
Mountain View, CA – May 18, 2010
Shanghai, China – June 1, 2010
Tokyo, Japan – June 3, 2010
Reading, United Kingdom – June 15, 2010
Paris, France – June 17, 2010

One full day of OData and Azure fun.

More details at http://www.odata.org/roadshow

See you there…

The OData team is hiring developers and testers.

https://careers.microsoft.com/JobDetails.aspx?ss=&pg=0&so=&rw=1&jid=15692&jlang=EN

https://careers.microsoft.com/JobDetails.aspx?ss=&pg=0&so=&rw=1&jid=11878&jlang=EN

Kent has a version of the MIX Day 2 Keynote with just the “Services Powering Experiences” section posted on the Data DevCenter.

http://msdn.microsoft.com/en-us/data/ff191185.aspx

Miguel has called for us to open source our .NET OData Server implementation.

It is something we have talked about doing in the run up to MIX and something we continue to discuss.

Broadly, I agree with much of Miguel’s argument.  Expect to hear more on this topic from us.

As we explore open sourcing our .NET OData server code, the question that I am pondering is if that would be enough to kick-start an ecosystem of OData services on Unix.

Do we need to have an OData implementation for PHP &| Python as well?

What if we both open sourced our .NET implementation and also spooled up a PHP &| Python project?

If we did decide to do something like this and we could only pick one other language/runtime beyond .NET, which one should it be?

Interested in any and all thoughts on this topic.

If you have seen the MIX Day 2 keynote, you know that we showed off OData on a Palm Pre.

Several of us (myself, JCar and Pablo) had a chance to write code for webOS as part of the demo work — and we really liked the runtime (effectively HTML5) and the device.

If you haven’t done so, it is really worth checking out the webOS SDK and most importantly, Ares, which I consider a herald of things to come.

BTW: I want to personally thank Palm (in particular Mike Abbott) for being supportive of this work.

The keynote that I did around OData, Windows/SQL Azure and “Dallas” is now available.

You can watch it at http://live.visitmix.com/MIX10/Sessions/KEY02.

My Microsoft on the Issues post went live.

http://microsoftontheissues.com/cs/blogs/mscorp/archive/2010/03/16/open-data-for-the-open-web.aspx

If you have every designed an API, you know that it is not a trivial process.

From naming, to versioning, to security, the number of things you have to get right is quite large.

Creating a Web API introduces more complexity; security writ large, versioning writ large, subscriber management, reporting, billing, etc.

At MIX, we will talk about the things that can help you build out Web APIs, including  OData and “Dallas”.

There are also some interesting startups that provide services to help you kickstart your Web APIs.

One of which is WebServius, which I call out because I know one of the founders (they are also part of our BizSpark program).

They provide both free and paid services to enable you to focus on the Web API itself (still not an easy task), while letting them worry about security, monitoring, etc.

It is exactly these sorts of services that will help get more and more Web sites to offer Web API.

Over time, more and more developers will realize that it is the Web API, not the Web site, that truly captures and exposes real value.

The Twitter API is one of the first evidence points of this movement.

I long for the day when I can pivot and access any piece of information on the Web in tools like Excel or DabbleDB.

If you run a Web site, why not start today?

Byte magazine played an instrumental role in my youth, particularly in how I learned about computing and programming.  I still remember reading about the introduction of the Macintosh in Byte (26 years ago).  I remember hunting for the “Balloon” issue when I finally understood the power of Smalltalk.  Every month I dreamed I was Jerry in the Chaos Manor.  I remember (or think I do) seeing the below ad in Byte which likely had more impact on my life than anything else in print.

One of the voices that emerged in Byte, particularly in the move to the Web, was Jon Udell.  I have had the opportunity to interact with Jon over the years.  I was happy to learn we hired him.  I bounce ideas off of him from time to time to ensure that I have the benefit of his experience and insights.

Jon and I both share a passion around what I call “Open Data for the Open Web”.  Based on his blog, he appears to be as excited as I am about OData:

• OData is grease to cut data friction
• OData for collaborative sense-making
• Producing and consuming OData feeds: An end-to-end example

Reading my recent posts, I hope you can see the potential around the Open Data Protocol (OData) as it makes its way into more and more of our and others products.

Pablo does a nice job of outlining a key business scenario in OData: The Movie, but I want to use this post to outline a personal scenario that is crying out for OData.

In doing so, I hope that you will join me in pushing for more Web sites/services to expose their your data over open protocols.

[Note:  The high order bit for me is Open Data.  OData is a mechanism that we have found some success using to achieve this goal.  It is important not to confuse mechanism with goal.  I am not confused and readers should not be either.]

Scenario: Monthly Net Worth

I have an Excel worksheet to calculate net worth.  It compute this number as a way of smoothing out betas and ensuring that I am on track toward our financial goals.  This “app” consists of a bunch of tabs with financial information (stock, salary, bank accounts, etc.), with macros to create roll-ups and then charts to report.

You may wonder why I don’t use Quicken or one of the many other financial tools out there.  The answer is simple.  For me, these applications are chains; they do not let me interact with my data in the flexible, transparent and empowering way that Excel does.

This is the exact reason that I see many business being run from Excel rather than packaged software.  It is also the reason that many enterprise IT shops have what is called an “Excel/Access Problem” (business units/departments building “applications” like rabbits that are not under management).

I only have one issue with my solution: I have to screen scrape all of my data.

I screen scrape stock information.  I copy and paste from a number of different locations.  I automated what I could, but in the end, the data acquisition cost is high, very high.  I will pay that cost, however, because the power that I get from Excel is worth more to me.

Now that Excel (via PowerPivot) supports OData, I see light at the end of the tunnel.  What I now need are feeds.  OData feeds from my brokerage.  OData feeds from Microsoft.  OData feeds from my bank.  OData feeds from the California and US governments. 

With feeds like that and a “data workbench” like Excel, you can control your financial destiny like never before.  It is this empowerment that I personally crave and it is this empowerment that is at the heart of my personal vision.

Call to Action

My good friend James Conard, is always hammering on me to have a clear call to action (I should hammer on him to update his blog).

If you work in the financial industry: Please push to expose your data via an HTTP-based open protocol like OData.  I think it would be interesting to consider how to tunnel OFX through OData.  I am going to follow-up with some our teams internally about it.

If you work in government agencies like the IRS & SSA in the US: Ditto.

If you are would like to use Excel to access this kind of data:  Tell your bank, brokerage, local government official about OData (or something like it) and tell them you want it.

A Closing Note…

There are a host of what you may consider “altruistic” scenarios for OData.  I don’t want those to get lost in the self-interest that drives this scenario and post.  I’ll be writing a lot more about these scenarios in the near future.  I just happened to be running my “Worth Report” (interesting name that, particularly for the philosophical minded), so it was top of mind.

The title of this post is not mine.

It is Bret Taylor’s.

Bret, of Google Maps and more importantly FriendFeed fame, is now at Facebook working closely with some of the best Microsoft alums I know.

Back in 2008, he was on to something, something important.

How do you discover a given dataset, particularly a common dataset that should be like “air” for developers?

Once you find it, what are the legal requirements to access it?

Once you can legally access it, what is the mechanism to access it?  Do you have to screen scrape it?  You would be surprised at the amount of screenscraping you need to do for even datasets you pay for.  Jon Udell captured some of my personal frustration around this in 2006 here.

Of course, if you are a dataset provider, you have the inverse of these questions.

Bret called his solution to these problems, DataWiki.

I call it “Dallas”.

There is, however, a key difference between Bret’s concept of the DataWiki and “Dallas” that is best highlighted by a Steward Brand quote:

Information Wants To Be Free. Information also wants to be expensive.

I do not think you can ignore this tension and any “data as a service” like “Dallas” needs to internalize this deeply in both its technical architecture and business strategy.

With that said, I think of “Dallas” as an important example and (I hope) success story of the Open Data vision that many of us at Microsoft share.

Maybe Bret will get his DataWiki after all…

At PDC 2009, I had an opportunity to sit down with Keith and Woody to talk about SQL Server Modeling (nee “Oslo”) and OData, among other topics.

I enjoyed doing the podcast.  Keith/Woody were great hosts.

You can listening at http://tinyurl.com/deepfried43.

.

The below diagram highlights all the products that have shipped or announced that support OData.

This is a very impressive list and there are more in the pipeline.

One of the questions I often hear is “That is great, but what is the scenario?”

I’ll admit that I tend to think the scenario(s) should be self-evident, but I am very close to the technology.

In order to answer this question, Pablo put together a video of a concrete, real-world scenario that should resonate well with even the most jaded cynic.

Watch OData: The Movie Now

BTW:  One of the things that we are looking at going is adding support in SQL Azure for OData.  Create a database and get a non-code OData service that you can access from any platform/language over HTTP.  If you are interested in this feature, please let use know:  Vote for OData Support in SQL Azure.

As noted on the OData.org site, WebSphere eXtreme Scale uses the OData protocol.

Billy Newport, an IBM Distinguished Engineer, was interviewed recently on why they selected a RESTful data service as the API and how OData helped.

The article: http://searchsoa.techtarget.com/news/article/0,289142,sid26_gci1379765,00.html

More product details: http://www.ibm.com/developerworks/websphere/downloads/xs_rest_service.html 

It is great to see that developers, regardless of platform/language, have a simple way to consume these services.

I’ll make one interesting note about this implementation.

As near as I can tell, Billy’s team implemented OData without ever talking to anyone at Microsoft.

I suspect they used the protocol documents we have online (these define the protocol with even greater precision that many standard specifications I have seen) and a HTTP trace tool.

Having been involved in distributed computing/protocol integration work for a long time, that is quite an achievement.

It could speak to simplicity of the protocol (it is just conventions/extensions over HTTP/AtomPub), the quality of the documentation or the intelligence/patience of the IBM team.

Likely it was all of these.

I am spending a lot of time on the Open Data Protocol (OData).

Many of us at Microsoft that believe that this protocol can help usher in a more open and programmable Web.

A protocol like this is a prerequisite for the broader “Infobus” and “Information Liberation” vision that I often talk about.

Envision a world where every application/Web property exposes its data (actually your data) in a way that you can easily query it in rich tools like Excel/Numbers or write your own mash-up/custom application.

A world were government data is transparent, queryable and accessible to any citizen.

A world where you can you can ask a question and know: “There’s a feed for that”.

We are just starting, but we (Microsoft) already has an impressive list of OData producers/consumers coming online (including SharePoint, SQL Server 2008, IIS/ASP.NET, etc.) and there are more in the pipeline that we will announce this year.

We are working as hard as we can to get OData support on as many platforms (both client and server/service) as we can, so a developer on any platform can both consume and produce these feeds.

We are begin to engaging partners, consumers and even competitors in a more structured way to see how we can work together to build up an ecosystem of open data services.

To make this vision a little more concrete, let’s look at a couple of screenshots.

Below is a third-party tool called LinqPad.  LinqPad recently added support for OData, which is demonstrated below.  The most interesting thing is the data service that I am accessing.  The City of Edmonton, Canada is exposing datasets as OData feeds at http://data.edmonton.ca/.  That lets tools that understand OData, like LinqPad, access this information in rich ways.

Also, notice the two other data services in the tool.  These point to District of Columbia and New America Foundation data at http://ogdisdk.cloudapp.net/.

This next screenshot is of Excel (via the PowerPivot plug-in) accessing the same data service.

Just to prove that this is all open, accessible and available to non-Microsoft clients/tools, see the below.  This is the same query that we are executing in LinqPad, but in Chrome and on the address bar.

Of course, you can access all this information using PHP, Java, JavaScript or .NET language using our OData clients.

Further, we are beginning to have conversations with key technical leaders in other companies/organizations about adding support in other platforms/languages/products.

We are excited about the possibilities here and think there is a real opportunity to usher in a world where open data is not only possible, but pervasive.