Douglas Purdy

The “checkbox” to expose a SQL Azure database as OData service is now live.

https://www.sqlazurelabs.com/

You can see a quick demo of this functionality in my MIX keynote.

I want to personally thank Mike Pizzo, David Robinson, and Jack Greenfield for their work on this service over the last couple of weeks.

Great work guys!

The keynote that I did around OData, Windows/SQL Azure and “Dallas” is now available.

You can watch it at http://live.visitmix.com/MIX10/Sessions/KEY02.

My Microsoft on the Issues post went live.

http://microsoftontheissues.com/cs/blogs/mscorp/archive/2010/03/16/open-data-for-the-open-web.aspx

I was asked this question based on my Emacs.NET (2010) post, so I thought I respond with a screenshot from the current CTP.

“Intellipad” is a .NET application that is used to write a .NET application: (large portions of) itself.

Most of the commands are in IronPython.  Further, the tests are in IronPython.  You can run them from the command buffer in the tool (>>>Test(“TestName”)) and they ship in the CTP.

Beyond IronPython, “Intellipad” supports “modes” for “M”, T-SQL, and lots more.  Of course, since the tool is completely extensible, you can add whatever language you want, as I did with MUrl.

BTW:  In case it is not clear, this is all still in CTP, the team is still working on it, so stuff can change.  I know the “M” and “Intellipad”/”Quadrant” teams believe strongly in text as a core way of “modeling” software, so you can expect that principle to be honored.

Byte magazine played an instrumental role in my youth, particularly in how I learned about computing and programming.  I still remember reading about the introduction of the Macintosh in Byte (26 years ago).  I remember hunting for the “Balloon” issue when I finally understood the power of Smalltalk.  Every month I dreamed I was Jerry in the Chaos Manor.  I remember (or think I do) seeing the below ad in Byte which likely had more impact on my life than anything else in print.

One of the voices that emerged in Byte, particularly in the move to the Web, was Jon Udell.  I have had the opportunity to interact with Jon over the years.  I was happy to learn we hired him.  I bounce ideas off of him from time to time to ensure that I have the benefit of his experience and insights.

Jon and I both share a passion around what I call “Open Data for the Open Web”.  Based on his blog, he appears to be as excited as I am about OData:

• OData is grease to cut data friction
• OData for collaborative sense-making
• Producing and consuming OData feeds: An end-to-end example

Reading my recent posts, I hope you can see the potential around the Open Data Protocol (OData) as it makes its way into more and more of our and others products.

Pablo does a nice job of outlining a key business scenario in OData: The Movie, but I want to use this post to outline a personal scenario that is crying out for OData.

In doing so, I hope that you will join me in pushing for more Web sites/services to expose their your data over open protocols.

[Note:  The high order bit for me is Open Data.  OData is a mechanism that we have found some success using to achieve this goal.  It is important not to confuse mechanism with goal.  I am not confused and readers should not be either.]

Scenario: Monthly Net Worth

I have an Excel worksheet to calculate net worth.  It compute this number as a way of smoothing out betas and ensuring that I am on track toward our financial goals.  This “app” consists of a bunch of tabs with financial information (stock, salary, bank accounts, etc.), with macros to create roll-ups and then charts to report.

You may wonder why I don’t use Quicken or one of the many other financial tools out there.  The answer is simple.  For me, these applications are chains; they do not let me interact with my data in the flexible, transparent and empowering way that Excel does.

This is the exact reason that I see many business being run from Excel rather than packaged software.  It is also the reason that many enterprise IT shops have what is called an “Excel/Access Problem” (business units/departments building “applications” like rabbits that are not under management).

I only have one issue with my solution: I have to screen scrape all of my data.

I screen scrape stock information.  I copy and paste from a number of different locations.  I automated what I could, but in the end, the data acquisition cost is high, very high.  I will pay that cost, however, because the power that I get from Excel is worth more to me.

Now that Excel (via PowerPivot) supports OData, I see light at the end of the tunnel.  What I now need are feeds.  OData feeds from my brokerage.  OData feeds from Microsoft.  OData feeds from my bank.  OData feeds from the California and US governments. 

With feeds like that and a “data workbench” like Excel, you can control your financial destiny like never before.  It is this empowerment that I personally crave and it is this empowerment that is at the heart of my personal vision.

Call to Action

My good friend James Conard, is always hammering on me to have a clear call to action (I should hammer on him to update his blog).

If you work in the financial industry: Please push to expose your data via an HTTP-based open protocol like OData.  I think it would be interesting to consider how to tunnel OFX through OData.  I am going to follow-up with some our teams internally about it.

If you work in government agencies like the IRS & SSA in the US: Ditto.

If you are would like to use Excel to access this kind of data:  Tell your bank, brokerage, local government official about OData (or something like it) and tell them you want it.

A Closing Note…

There are a host of what you may consider “altruistic” scenarios for OData.  I don’t want those to get lost in the self-interest that drives this scenario and post.  I’ll be writing a lot more about these scenarios in the near future.  I just happened to be running my “Worth Report” (interesting name that, particularly for the philosophical minded), so it was top of mind.

The title of this post is not mine.

It is Bret Taylor’s.

Bret, of Google Maps and more importantly FriendFeed fame, is now at Facebook working closely with some of the best Microsoft alums I know.

Back in 2008, he was on to something, something important.

How do you discover a given dataset, particularly a common dataset that should be like “air” for developers?

Once you find it, what are the legal requirements to access it?

Once you can legally access it, what is the mechanism to access it?  Do you have to screen scrape it?  You would be surprised at the amount of screenscraping you need to do for even datasets you pay for.  Jon Udell captured some of my personal frustration around this in 2006 here.

Of course, if you are a dataset provider, you have the inverse of these questions.

Bret called his solution to these problems, DataWiki.

I call it “Dallas”.

There is, however, a key difference between Bret’s concept of the DataWiki and “Dallas” that is best highlighted by a Steward Brand quote:

Information Wants To Be Free. Information also wants to be expensive.

I do not think you can ignore this tension and any “data as a service” like “Dallas” needs to internalize this deeply in both its technical architecture and business strategy.

With that said, I think of “Dallas” as an important example and (I hope) success story of the Open Data vision that many of us at Microsoft share.

Maybe Bret will get his DataWiki after all…

At PDC 2009, I had an opportunity to sit down with Keith and Woody to talk about SQL Server Modeling (nee “Oslo”) and OData, among other topics.

I enjoyed doing the podcast.  Keith/Woody were great hosts.

You can listening at http://tinyurl.com/deepfried43.

.

The below diagram highlights all the products that have shipped or announced that support OData.

This is a very impressive list and there are more in the pipeline.

One of the questions I often hear is “That is great, but what is the scenario?”

I’ll admit that I tend to think the scenario(s) should be self-evident, but I am very close to the technology.

In order to answer this question, Pablo put together a video of a concrete, real-world scenario that should resonate well with even the most jaded cynic.

Watch OData: The Movie Now

BTW:  One of the things that we are looking at going is adding support in SQL Azure for OData.  Create a database and get a non-code OData service that you can access from any platform/language over HTTP.  If you are interested in this feature, please let use know:  Vote for OData Support in SQL Azure.

I am spending a lot of time on the Open Data Protocol (OData).

Many of us at Microsoft that believe that this protocol can help usher in a more open and programmable Web.

A protocol like this is a prerequisite for the broader “Infobus” and “Information Liberation” vision that I often talk about.

Envision a world where every application/Web property exposes its data (actually your data) in a way that you can easily query it in rich tools like Excel/Numbers or write your own mash-up/custom application.

A world were government data is transparent, queryable and accessible to any citizen.

A world where you can you can ask a question and know: “There’s a feed for that”.

We are just starting, but we (Microsoft) already has an impressive list of OData producers/consumers coming online (including SharePoint, SQL Server 2008, IIS/ASP.NET, etc.) and there are more in the pipeline that we will announce this year.

We are working as hard as we can to get OData support on as many platforms (both client and server/service) as we can, so a developer on any platform can both consume and produce these feeds.

We are begin to engaging partners, consumers and even competitors in a more structured way to see how we can work together to build up an ecosystem of open data services.

To make this vision a little more concrete, let’s look at a couple of screenshots.

Below is a third-party tool called LinqPad.  LinqPad recently added support for OData, which is demonstrated below.  The most interesting thing is the data service that I am accessing.  The City of Edmonton, Canada is exposing datasets as OData feeds at http://data.edmonton.ca/.  That lets tools that understand OData, like LinqPad, access this information in rich ways.

Also, notice the two other data services in the tool.  These point to District of Columbia and New America Foundation data at http://ogdisdk.cloudapp.net/.

This next screenshot is of Excel (via the PowerPivot plug-in) accessing the same data service.

Just to prove that this is all open, accessible and available to non-Microsoft clients/tools, see the below.  This is the same query that we are executing in LinqPad, but in Chrome and on the address bar.

Of course, you can access all this information using PHP, Java, JavaScript or .NET language using our OData clients.

Further, we are beginning to have conversations with key technical leaders in other companies/organizations about adding support in other platforms/languages/products.

We are excited about the possibilities here and think there is a real opportunity to usher in a world where open data is not only possible, but pervasive.