Douglas Purdy

If you have seen my MIX keynote, you know that we showed off OData on a Palm Pre.

Several of us (myself, JCar and Pablo) had a chance to write code for webOS as part of the demo work — and we really liked the runtime (effectively HTML5) and the device.

If you haven’t done so, it is really worth checking out the webOS SDK and most importantly, Ares, which I consider a herald of things to come.

BTW: I want to personally thank Palm (in particular Mike Abbott) for being supportive of this work.

The keynote that I did around OData, Windows/SQL Azure and “Dallas” is now available.

You can watch it at http://live.visitmix.com/MIX10/Sessions/KEY02.

My Microsoft on the Issues post went live.

http://microsoftontheissues.com/cs/blogs/mscorp/archive/2010/03/16/open-data-for-the-open-web.aspx

If you have every designed an API, you know that it is not a trivial process.

From naming, to versioning, to security, the number of things you have to get right is quite large.

Creating a Web API introduces more complexity; security writ large, versioning writ large, subscriber management, reporting, billing, etc.

At MIX, we will talk about the things that can help you build out Web APIs, including  OData and “Dallas”.

There are also some interesting startups that provide services to help you kickstart your Web APIs.

One of which is WebServius, which I call out because I know one of the founders (they are also part of our BizSpark program).

They provide both free and paid services to enable you to focus on the Web API itself (still not an easy task), while letting them worry about security, monitoring, etc.

It is exactly these sorts of services that will help get more and more Web sites to offer Web API.

Over time, more and more developers will realize that it is the Web API, not the Web site, that truly captures and exposes real value.

The Twitter API is one of the first evidence points of this movement.

I long for the day when I can pivot and access any piece of information on the Web in tools like Excel or DabbleDB.

If you run a Web site, why not start today?

Byte magazine played an instrumental role in my youth, particularly in how I learned about computing and programming.  I still remember reading about the introduction of the Macintosh in Byte (26 years ago).  I remember hunting for the “Balloon” issue when I finally understood the power of Smalltalk.  Every month I dreamed I was Jerry in the Chaos Manor.  I remember (or think I do) seeing the below ad in Byte which likely had more impact on my life than anything else in print.

One of the voices that emerged in Byte, particularly in the move to the Web, was Jon Udell.  I have had the opportunity to interact with Jon over the years.  I was happy to learn we hired him.  I bounce ideas off of him from time to time to ensure that I have the benefit of his experience and insights.

Jon and I both share a passion around what I call “Open Data for the Open Web”.  Based on his blog, he appears to be as excited as I am about OData:

• OData is grease to cut data friction
• OData for collaborative sense-making
• Producing and consuming OData feeds: An end-to-end example

Reading my recent posts, I hope you can see the potential around the Open Data Protocol (OData) as it makes its way into more and more of our and others products.

Pablo does a nice job of outlining a key business scenario in OData: The Movie, but I want to use this post to outline a personal scenario that is crying out for OData.

In doing so, I hope that you will join me in pushing for more Web sites/services to expose their your data over open protocols.

[Note:  The high order bit for me is Open Data.  OData is a mechanism that we have found some success using to achieve this goal.  It is important not to confuse mechanism with goal.  I am not confused and readers should not be either.]

Scenario: Monthly Net Worth

I have an Excel worksheet to calculate net worth.  It compute this number as a way of smoothing out betas and ensuring that I am on track toward our financial goals.  This “app” consists of a bunch of tabs with financial information (stock, salary, bank accounts, etc.), with macros to create roll-ups and then charts to report.

You may wonder why I don’t use Quicken or one of the many other financial tools out there.  The answer is simple.  For me, these applications are chains; they do not let me interact with my data in the flexible, transparent and empowering way that Excel does.

This is the exact reason that I see many business being run from Excel rather than packaged software.  It is also the reason that many enterprise IT shops have what is called an “Excel/Access Problem” (business units/departments building “applications” like rabbits that are not under management).

I only have one issue with my solution: I have to screen scrape all of my data.

I screen scrape stock information.  I copy and paste from a number of different locations.  I automated what I could, but in the end, the data acquisition cost is high, very high.  I will pay that cost, however, because the power that I get from Excel is worth more to me.

Now that Excel (via PowerPivot) supports OData, I see light at the end of the tunnel.  What I now need are feeds.  OData feeds from my brokerage.  OData feeds from Microsoft.  OData feeds from my bank.  OData feeds from the California and US governments. 

With feeds like that and a “data workbench” like Excel, you can control your financial destiny like never before.  It is this empowerment that I personally crave and it is this empowerment that is at the heart of my personal vision.

Call to Action

My good friend James Conard, is always hammering on me to have a clear call to action (I should hammer on him to update his blog).

If you work in the financial industry: Please push to expose your data via an HTTP-based open protocol like OData.  I think it would be interesting to consider how to tunnel OFX through OData.  I am going to follow-up with some our teams internally about it.

If you work in government agencies like the IRS & SSA in the US: Ditto.

If you are would like to use Excel to access this kind of data:  Tell your bank, brokerage, local government official about OData (or something like it) and tell them you want it.

A Closing Note…

There are a host of what you may consider “altruistic” scenarios for OData.  I don’t want those to get lost in the self-interest that drives this scenario and post.  I’ll be writing a lot more about these scenarios in the near future.  I just happened to be running my “Worth Report” (interesting name that, particularly for the philosophical minded), so it was top of mind.

The title of this post is not mine.

It is Bret Taylor’s.

Bret, of Google Maps and more importantly FriendFeed fame, is now at Facebook working closely with some of the best Microsoft alums I know.

Back in 2008, he was on to something, something important.

How do you discover a given dataset, particularly a common dataset that should be like “air” for developers?

Once you find it, what are the legal requirements to access it?

Once you can legally access it, what is the mechanism to access it?  Do you have to screen scrape it?  You would be surprised at the amount of screenscraping you need to do for even datasets you pay for.  Jon Udell captured some of my personal frustration around this in 2006 here.

Of course, if you are a dataset provider, you have the inverse of these questions.

Bret called his solution to these problems, DataWiki.

I call it “Dallas”.

There is, however, a key difference between Bret’s concept of the DataWiki and “Dallas” that is best highlighted by a Steward Brand quote:

Information Wants To Be Free. Information also wants to be expensive.

I do not think you can ignore this tension and any “data as a service” like “Dallas” needs to internalize this deeply in both its technical architecture and business strategy.

With that said, I think of “Dallas” as an important example and (I hope) success story of the Open Data vision that many of us at Microsoft share.

Maybe Bret will get his DataWiki after all…

At PDC 2009, I had an opportunity to sit down with Keith and Woody to talk about SQL Server Modeling (nee “Oslo”) and OData, among other topics.

I enjoyed doing the podcast.  Keith/Woody were great hosts.

You can listening at http://tinyurl.com/deepfried43.

.

The below diagram highlights all the products that have shipped or announced that support OData.

This is a very impressive list and there are more in the pipeline.

One of the questions I often hear is “That is great, but what is the scenario?”

I’ll admit that I tend to think the scenario(s) should be self-evident, but I am very close to the technology.

In order to answer this question, Pablo put together a video of a concrete, real-world scenario that should resonate well with even the most jaded cynic.

Watch OData: The Movie Now

BTW:  One of the things that we are looking at going is adding support in SQL Azure for OData.  Create a database and get a non-code OData service that you can access from any platform/language over HTTP.  If you are interested in this feature, please let use know:  Vote for OData Support in SQL Azure.

As noted on the OData.org site, WebSphere eXtreme Scale uses the OData protocol.

Billy Newport, an IBM Distinguished Engineer, was interviewed recently on why they selected a RESTful data service as the API and how OData helped.

The article: http://searchsoa.techtarget.com/news/article/0,289142,sid26_gci1379765,00.html

More product details: http://www.ibm.com/developerworks/websphere/downloads/xs_rest_service.html 

It is great to see that developers, regardless of platform/language, have a simple way to consume these services.

I’ll make one interesting note about this implementation.

As near as I can tell, Billy’s team implemented OData without ever talking to anyone at Microsoft.

I suspect they used the protocol documents we have online (these define the protocol with even greater precision that many standard specifications I have seen) and a HTTP trace tool.

Having been involved in distributed computing/protocol integration work for a long time, that is quite an achievement.

It could speak to simplicity of the protocol (it is just conventions/extensions over HTTP/AtomPub), the quality of the documentation or the intelligence/patience of the IBM team.

Likely it was all of these.