Douglas Purdy

I was asked this question based on my Emacs.NET (2010) post, so I thought I respond with a screenshot from the current CTP.

“Intellipad” is a .NET application that is used to write a .NET application: (large portions of) itself.

Most of the commands are in IronPython.  Further, the tests are in IronPython.  You can run them from the command buffer in the tool (>>>Test(“TestName”)) and they ship in the CTP.

Beyond IronPython, “Intellipad” supports “modes” for “M”, T-SQL, and lots more.  Of course, since the tool is completely extensible, you can add whatever language you want, as I did with MUrl.

BTW:  In case it is not clear, this is all still in CTP, the team is still working on it, so stuff can change.  I know the “M” and “Intellipad”/”Quadrant” teams believe strongly in text as a core way of “modeling” software, so you can expect that principle to be honored.

At the end of 2007, we started hiring for a project that we referred to internally as “Emacs.Net”.

I often get asked about this project and it is a recurring search leading to this blog, so I thought I would give an update on what happened since we staffed up that team.

The short story is that we gave this project an official unofficial name: “Intellipad”.  This tool was included in the first “Oslo” CTP and continues to ship in the latest SQL Server Modeling CTP.

We designed “Intellipad” to work standalone as a lightweight companion to “Quadrant” and VS.  In addition, a core scenario was in situ use within “Quadrant” itself.

You can see three “Intellipad” views in “Quadrant” below (from the current CTP).  One shows “M”.  Another shows the T-SQL generated from the “M”.  The last shows “Intellipad” as the query editor in the table view.

The ‘Quadrant” integration is something we debuted at PDC this year, but the fact that we were going there shouldn’t have been a surprise.  First, I told anyone that asked and second the name is the best clue.  “Intellipad” is short for “Intellisense Workpad”.  A workpad is the name that we give for a view inside of “Quadrant”.

BTW:  One of the main developers of “Intellipad” and “Quadrant” has an impressive side-project that you may want to check out: SmallBasic.

[Update:  If you want to see what you can do with “Intellipad”, check out how we extend it for MUrl.]

Byte magazine played an instrumental role in my youth, particularly in how I learned about computing and programming.  I still remember reading about the introduction of the Macintosh in Byte (26 years ago).  I remember hunting for the “Balloon” issue when I finally understood the power of Smalltalk.  Every month I dreamed I was Jerry in the Chaos Manor.  I remember (or think I do) seeing the below ad in Byte which likely had more impact on my life than anything else in print.

One of the voices that emerged in Byte, particularly in the move to the Web, was Jon Udell.  I have had the opportunity to interact with Jon over the years.  I was happy to learn we hired him.  I bounce ideas off of him from time to time to ensure that I have the benefit of his experience and insights.

Jon and I both share a passion around what I call “Open Data for the Open Web”.  Based on his blog, he appears to be as excited as I am about OData:

• OData is grease to cut data friction
• OData for collaborative sense-making
• Producing and consuming OData feeds: An end-to-end example

Reading my recent posts, I hope you can see the potential around the Open Data Protocol (OData) as it makes its way into more and more of our and others products.

Pablo does a nice job of outlining a key business scenario in OData: The Movie, but I want to use this post to outline a personal scenario that is crying out for OData.

In doing so, I hope that you will join me in pushing for more Web sites/services to expose their your data over open protocols.

[Note:  The high order bit for me is Open Data.  OData is a mechanism that we have found some success using to achieve this goal.  It is important not to confuse mechanism with goal.  I am not confused and readers should not be either.]

Scenario: Monthly Net Worth

I have an Excel worksheet to calculate net worth.  It compute this number as a way of smoothing out betas and ensuring that I am on track toward our financial goals.  This “app” consists of a bunch of tabs with financial information (stock, salary, bank accounts, etc.), with macros to create roll-ups and then charts to report.

You may wonder why I don’t use Quicken or one of the many other financial tools out there.  The answer is simple.  For me, these applications are chains; they do not let me interact with my data in the flexible, transparent and empowering way that Excel does.

This is the exact reason that I see many business being run from Excel rather than packaged software.  It is also the reason that many enterprise IT shops have what is called an “Excel/Access Problem” (business units/departments building “applications” like rabbits that are not under management).

I only have one issue with my solution: I have to screen scrape all of my data.

I screen scrape stock information.  I copy and paste from a number of different locations.  I automated what I could, but in the end, the data acquisition cost is high, very high.  I will pay that cost, however, because the power that I get from Excel is worth more to me.

Now that Excel (via PowerPivot) supports OData, I see light at the end of the tunnel.  What I now need are feeds.  OData feeds from my brokerage.  OData feeds from Microsoft.  OData feeds from my bank.  OData feeds from the California and US governments. 

With feeds like that and a “data workbench” like Excel, you can control your financial destiny like never before.  It is this empowerment that I personally crave and it is this empowerment that is at the heart of my personal vision.

Call to Action

My good friend James Conard, is always hammering on me to have a clear call to action (I should hammer on him to update his blog).

If you work in the financial industry: Please push to expose your data via an HTTP-based open protocol like OData.  I think it would be interesting to consider how to tunnel OFX through OData.  I am going to follow-up with some our teams internally about it.

If you work in government agencies like the IRS & SSA in the US: Ditto.

If you are would like to use Excel to access this kind of data:  Tell your bank, brokerage, local government official about OData (or something like it) and tell them you want it.

A Closing Note…

There are a host of what you may consider “altruistic” scenarios for OData.  I don’t want those to get lost in the self-interest that drives this scenario and post.  I’ll be writing a lot more about these scenarios in the near future.  I just happened to be running my “Worth Report” (interesting name that, particularly for the philosophical minded), so it was top of mind.

The title of this post is not mine.

It is Bret Taylor’s.

Bret, of Google Maps and more importantly FriendFeed fame, is now at Facebook working closely with some of the best Microsoft alums I know.

Back in 2008, he was on to something, something important.

How do you discover a given dataset, particularly a common dataset that should be like “air” for developers?

Once you find it, what are the legal requirements to access it?

Once you can legally access it, what is the mechanism to access it?  Do you have to screen scrape it?  You would be surprised at the amount of screenscraping you need to do for even datasets you pay for.  Jon Udell captured some of my personal frustration around this in 2006 here.

Of course, if you are a dataset provider, you have the inverse of these questions.

Bret called his solution to these problems, DataWiki.

I call it “Dallas”.

There is, however, a key difference between Bret’s concept of the DataWiki and “Dallas” that is best highlighted by a Steward Brand quote:

Information Wants To Be Free. Information also wants to be expensive.

I do not think you can ignore this tension and any “data as a service” like “Dallas” needs to internalize this deeply in both its technical architecture and business strategy.

With that said, I think of “Dallas” as an important example and (I hope) success story of the Open Data vision that many of us at Microsoft share.

Maybe Bret will get his DataWiki after all…

At PDC 2009, I had an opportunity to sit down with Keith and Woody to talk about SQL Server Modeling (nee “Oslo”) and OData, among other topics.

I enjoyed doing the podcast.  Keith/Woody were great hosts.

You can listening at http://tinyurl.com/deepfried43.

.

http://www.theatlantic.com/doc/201001/american-decline

I am repeatedly impressed with the writing in the Atlantic.  So much so, that it joins the Economist as the only two periodicals to grace my Kindle.

James Fallows recent article with the same title as this post is a compelling read.  Although the reasons may not be what you expect.

The first reason is summarized best in a John Adams quote found in the article.

Remember, democracy never lasts long. It soon wastes, exhausts, and murders itself. There never was a democracy yet that did not commit suicide.

It is worthwhile to consider the track record of democracy as long-term governing mechanism.  If you decide to undertake that, decoupling individual “freedom/liberty” from “democracy” during the process could provide new insights.

The second reason is the policy point around the importance of the public sector as “capital collector and director” (my words).

Robert Atkinson, the director of the Information Technology and Innovation Foundation, in Washington, has written that several times per century, a “transformational wave” of new technologies ripples through the economy and creates new opportunities and wealth. In the past, these have included mass-production systems, modern chemicals, aviation, and so on. Today the economically important technologies include genomic knowledge, information technologies like the Internet, and the geospatial information, from the GPS network, that is built into everything from dashboard navigators to the climate-change-monitoring systems that measure the size of glaciers or extent of forests. Private companies now create the jobs and wealth in each field, but public funds paid for the original scientific breakthroughs and provided early markets.

It couldn’t have been otherwise, Atkinson says. The scale of investment was too vast. The uncertainty of payoff was too great. The risk that profits and benefits would go to competitors who hadn’t made the initial investment was too high. The difference between promising and dead-end technologies was too hard to predict—especially decades ago, when work in all these fields began. So each started as a public program: the Internet by the Pentagon, the Human Genome Project by the National Institutes of Health, and the GPS network by the Air Force, which still operates it. The government could not have created Google, but Google could not have existed without government efforts to establish the Internet long before the company’s founders were born. This pattern—public investment and standard-setting, followed by private industrial growth—has been consistent through the years, Atkinson said, which is what worries him now. “Our companies and entrepreneurs are matchless in their power to adapt,” he said. “We lead in many categories the private economy can handle by itself. But where you need any public-private coordination, we’ve become handicapped. I worry that our companies can adapt, but our system can’t.”

The below diagram highlights all the products that have shipped or announced that support OData.

This is a very impressive list and there are more in the pipeline.

One of the questions I often hear is “That is great, but what is the scenario?”

I’ll admit that I tend to think the scenario(s) should be self-evident, but I am very close to the technology.

In order to answer this question, Pablo put together a video of a concrete, real-world scenario that should resonate well with even the most jaded cynic.

Watch OData: The Movie Now

BTW:  One of the things that we are looking at going is adding support in SQL Azure for OData.  Create a database and get a non-code OData service that you can access from any platform/language over HTTP.  If you are interested in this feature, please let use know:  Vote for OData Support in SQL Azure.

As noted on the OData.org site, WebSphere eXtreme Scale uses the OData protocol.

Billy Newport, an IBM Distinguished Engineer, was interviewed recently on why they selected a RESTful data service as the API and how OData helped.

The article: http://searchsoa.techtarget.com/news/article/0,289142,sid26_gci1379765,00.html

More product details: http://www.ibm.com/developerworks/websphere/downloads/xs_rest_service.html 

It is great to see that developers, regardless of platform/language, have a simple way to consume these services.

I’ll make one interesting note about this implementation.

As near as I can tell, Billy’s team implemented OData without ever talking to anyone at Microsoft.

I suspect they used the protocol documents we have online (these define the protocol with even greater precision that many standard specifications I have seen) and a HTTP trace tool.

Having been involved in distributed computing/protocol integration work for a long time, that is quite an achievement.

It could speak to simplicity of the protocol (it is just conventions/extensions over HTTP/AtomPub), the quality of the documentation or the intelligence/patience of the IBM team.

Likely it was all of these.

I am spending a lot of time on the Open Data Protocol (OData).

Many of us at Microsoft that believe that this protocol can help usher in a more open and programmable Web.

A protocol like this is a prerequisite for the broader “Infobus” and “Information Liberation” vision that I often talk about.

Envision a world where every application/Web property exposes its data (actually your data) in a way that you can easily query it in rich tools like Excel/Numbers or write your own mash-up/custom application.

A world were government data is transparent, queryable and accessible to any citizen.

A world where you can you can ask a question and know: “There’s a feed for that”.

We are just starting, but we (Microsoft) already has an impressive list of OData producers/consumers coming online (including SharePoint, SQL Server 2008, IIS/ASP.NET, etc.) and there are more in the pipeline that we will announce this year.

We are working as hard as we can to get OData support on as many platforms (both client and server/service) as we can, so a developer on any platform can both consume and produce these feeds.

We are begin to engaging partners, consumers and even competitors in a more structured way to see how we can work together to build up an ecosystem of open data services.

To make this vision a little more concrete, let’s look at a couple of screenshots.

Below is a third-party tool called LinqPad.  LinqPad recently added support for OData, which is demonstrated below.  The most interesting thing is the data service that I am accessing.  The City of Edmonton, Canada is exposing datasets as OData feeds at http://data.edmonton.ca/.  That lets tools that understand OData, like LinqPad, access this information in rich ways.

Also, notice the two other data services in the tool.  These point to District of Columbia and New America Foundation data at http://ogdisdk.cloudapp.net/.

This next screenshot is of Excel (via the PowerPivot plug-in) accessing the same data service.

Just to prove that this is all open, accessible and available to non-Microsoft clients/tools, see the below.  This is the same query that we are executing in LinqPad, but in Chrome and on the address bar.

Of course, you can access all this information using PHP, Java, JavaScript or .NET language using our OData clients.

Further, we are beginning to have conversations with key technical leaders in other companies/organizations about adding support in other platforms/languages/products.

We are excited about the possibilities here and think there is a real opportunity to usher in a world where open data is not only possible, but pervasive.