Douglas Purdy

Byte magazine played an instrumental role in my youth, particularly in how I learned about computing and programming.  I still remember reading about the introduction of the Macintosh in Byte (26 years ago).  I remember hunting for the “Balloon” issue when I finally understood the power of Smalltalk.  Every month I dreamed I was Jerry in the Chaos Manor.  I remember (or think I do) seeing the below ad in Byte which likely had more impact on my life than anything else in print.

One of the voices that emerged in Byte, particularly in the move to the Web, was Jon Udell.  I have had the opportunity to interact with Jon over the years.  I was happy to learn we hired him.  I bounce ideas off of him from time to time to ensure that I have the benefit of his experience and insights.

Jon and I both share a passion around what I call “Open Data for the Open Web”.  Based on his blog, he appears to be as excited as I am about OData:

• OData is grease to cut data friction
• OData for collaborative sense-making
• Producing and consuming OData feeds: An end-to-end example

Reading my recent posts, I hope you can see the potential around the Open Data Protocol (OData) as it makes its way into more and more of our and others products.

Pablo does a nice job of outlining a key business scenario in OData: The Movie, but I want to use this post to outline a personal scenario that is crying out for OData.

In doing so, I hope that you will join me in pushing for more Web sites/services to expose their your data over open protocols.

[Note:  The high order bit for me is Open Data.  OData is a mechanism that we have found some success using to achieve this goal.  It is important not to confuse mechanism with goal.  I am not confused and readers should not be either.]

Scenario: Monthly Net Worth

I have an Excel worksheet to calculate net worth.  It compute this number as a way of smoothing out betas and ensuring that I am on track toward our financial goals.  This “app” consists of a bunch of tabs with financial information (stock, salary, bank accounts, etc.), with macros to create roll-ups and then charts to report.

You may wonder why I don’t use Quicken or one of the many other financial tools out there.  The answer is simple.  For me, these applications are chains; they do not let me interact with my data in the flexible, transparent and empowering way that Excel does.

This is the exact reason that I see many business being run from Excel rather than packaged software.  It is also the reason that many enterprise IT shops have what is called an “Excel/Access Problem” (business units/departments building “applications” like rabbits that are not under management).

I only have one issue with my solution: I have to screen scrape all of my data.

I screen scrape stock information.  I copy and paste from a number of different locations.  I automated what I could, but in the end, the data acquisition cost is high, very high.  I will pay that cost, however, because the power that I get from Excel is worth more to me.

Now that Excel (via PowerPivot) supports OData, I see light at the end of the tunnel.  What I now need are feeds.  OData feeds from my brokerage.  OData feeds from Microsoft.  OData feeds from my bank.  OData feeds from the California and US governments. 

With feeds like that and a “data workbench” like Excel, you can control your financial destiny like never before.  It is this empowerment that I personally crave and it is this empowerment that is at the heart of my personal vision.

Call to Action

My good friend James Conard, is always hammering on me to have a clear call to action (I should hammer on him to update his blog).

If you work in the financial industry: Please push to expose your data via an HTTP-based open protocol like OData.  I think it would be interesting to consider how to tunnel OFX through OData.  I am going to follow-up with some our teams internally about it.

If you work in government agencies like the IRS & SSA in the US: Ditto.

If you are would like to use Excel to access this kind of data:  Tell your bank, brokerage, local government official about OData (or something like it) and tell them you want it.

A Closing Note…

There are a host of what you may consider “altruistic” scenarios for OData.  I don’t want those to get lost in the self-interest that drives this scenario and post.  I’ll be writing a lot more about these scenarios in the near future.  I just happened to be running my “Worth Report” (interesting name that, particularly for the philosophical minded), so it was top of mind.

The title of this post is not mine.

It is Bret Taylor’s.

Bret, of Google Maps and more importantly FriendFeed fame, is now at Facebook working closely with some of the best Microsoft alums I know.

Back in 2008, he was on to something, something important.

How do you discover a given dataset, particularly a common dataset that should be like “air” for developers?

Once you find it, what are the legal requirements to access it?

Once you can legally access it, what is the mechanism to access it?  Do you have to screen scrape it?  You would be surprised at the amount of screenscraping you need to do for even datasets you pay for.  Jon Udell captured some of my personal frustration around this in 2006 here.

Of course, if you are a dataset provider, you have the inverse of these questions.

Bret called his solution to these problems, DataWiki.

I call it “Dallas”.

There is, however, a key difference between Bret’s concept of the DataWiki and “Dallas” that is best highlighted by a Steward Brand quote:

Information Wants To Be Free. Information also wants to be expensive.

I do not think you can ignore this tension and any “data as a service” like “Dallas” needs to internalize this deeply in both its technical architecture and business strategy.

With that said, I think of “Dallas” as an important example and (I hope) success story of the Open Data vision that many of us at Microsoft share.

Maybe Bret will get his DataWiki after all…

At PDC 2009, I had an opportunity to sit down with Keith and Woody to talk about SQL Server Modeling (nee “Oslo”) and OData, among other topics.

I enjoyed doing the podcast.  Keith/Woody were great hosts.

You can listening at http://tinyurl.com/deepfried43.

.

http://www.theatlantic.com/doc/201001/american-decline

I am repeatedly impressed with the writing in the Atlantic.  So much so, that it joins the Economist as the only two periodicals to grace my Kindle.

James Fallows recent article with the same title as this post is a compelling read.  Although the reasons may not be what you expect.

The first reason is summarized best in a John Adams quote found in the article.

Remember, democracy never lasts long. It soon wastes, exhausts, and murders itself. There never was a democracy yet that did not commit suicide.

It is worthwhile to consider the track record of democracy as long-term governing mechanism.  If you decide to undertake that, decoupling individual “freedom/liberty” from “democracy” during the process could provide new insights.

The second reason is the policy point around the importance of the public sector as “capital collector and director” (my words).

Robert Atkinson, the director of the Information Technology and Innovation Foundation, in Washington, has written that several times per century, a “transformational wave” of new technologies ripples through the economy and creates new opportunities and wealth. In the past, these have included mass-production systems, modern chemicals, aviation, and so on. Today the economically important technologies include genomic knowledge, information technologies like the Internet, and the geospatial information, from the GPS network, that is built into everything from dashboard navigators to the climate-change-monitoring systems that measure the size of glaciers or extent of forests. Private companies now create the jobs and wealth in each field, but public funds paid for the original scientific breakthroughs and provided early markets.

It couldn’t have been otherwise, Atkinson says. The scale of investment was too vast. The uncertainty of payoff was too great. The risk that profits and benefits would go to competitors who hadn’t made the initial investment was too high. The difference between promising and dead-end technologies was too hard to predict—especially decades ago, when work in all these fields began. So each started as a public program: the Internet by the Pentagon, the Human Genome Project by the National Institutes of Health, and the GPS network by the Air Force, which still operates it. The government could not have created Google, but Google could not have existed without government efforts to establish the Internet long before the company’s founders were born. This pattern—public investment and standard-setting, followed by private industrial growth—has been consistent through the years, Atkinson said, which is what worries him now. “Our companies and entrepreneurs are matchless in their power to adapt,” he said. “We lead in many categories the private economy can handle by itself. But where you need any public-private coordination, we’ve become handicapped. I worry that our companies can adapt, but our system can’t.”

The below diagram highlights all the products that have shipped or announced that support OData.

This is a very impressive list and there are more in the pipeline.

One of the questions I often hear is “That is great, but what is the scenario?”

I’ll admit that I tend to think the scenario(s) should be self-evident, but I am very close to the technology.

In order to answer this question, Pablo put together a video of a concrete, real-world scenario that should resonate well with even the most jaded cynic.

Watch OData: The Movie Now

BTW:  One of the things that we are looking at going is adding support in SQL Azure for OData.  Create a database and get a non-code OData service that you can access from any platform/language over HTTP.  If you are interested in this feature, please let use know:  Vote for OData Support in SQL Azure.

As noted on the OData.org site, WebSphere eXtreme Scale uses the OData protocol.

Billy Newport, an IBM Distinguished Engineer, was interviewed recently on why they selected a RESTful data service as the API and how OData helped.

The article: http://searchsoa.techtarget.com/news/article/0,289142,sid26_gci1379765,00.html

More product details: http://www.ibm.com/developerworks/websphere/downloads/xs_rest_service.html 

It is great to see that developers, regardless of platform/language, have a simple way to consume these services.

I’ll make one interesting note about this implementation.

As near as I can tell, Billy’s team implemented OData without ever talking to anyone at Microsoft.

I suspect they used the protocol documents we have online (these define the protocol with even greater precision that many standard specifications I have seen) and a HTTP trace tool.

Having been involved in distributed computing/protocol integration work for a long time, that is quite an achievement.

It could speak to simplicity of the protocol (it is just conventions/extensions over HTTP/AtomPub), the quality of the documentation or the intelligence/patience of the IBM team.

Likely it was all of these.

I am spending a lot of time on the Open Data Protocol (OData).

Many of us at Microsoft that believe that this protocol can help usher in a more open and programmable Web.

A protocol like this is a prerequisite for the broader “Infobus” and “Information Liberation” vision that I often talk about.

Envision a world where every application/Web property exposes its data (actually your data) in a way that you can easily query it in rich tools like Excel/Numbers or write your own mash-up/custom application.

A world were government data is transparent, queryable and accessible to any citizen.

A world where you can you can ask a question and know: “There’s a feed for that”.

We are just starting, but we (Microsoft) already has an impressive list of OData producers/consumers coming online (including SharePoint, SQL Server 2008, IIS/ASP.NET, etc.) and there are more in the pipeline that we will announce this year.

We are working as hard as we can to get OData support on as many platforms (both client and server/service) as we can, so a developer on any platform can both consume and produce these feeds.

We are begin to engaging partners, consumers and even competitors in a more structured way to see how we can work together to build up an ecosystem of open data services.

To make this vision a little more concrete, let’s look at a couple of screenshots.

Below is a third-party tool called LinqPad.  LinqPad recently added support for OData, which is demonstrated below.  The most interesting thing is the data service that I am accessing.  The City of Edmonton, Canada is exposing datasets as OData feeds at http://data.edmonton.ca/.  That lets tools that understand OData, like LinqPad, access this information in rich ways.

Also, notice the two other data services in the tool.  These point to District of Columbia and New America Foundation data at http://ogdisdk.cloudapp.net/.

This next screenshot is of Excel (via the PowerPivot plug-in) accessing the same data service.

Just to prove that this is all open, accessible and available to non-Microsoft clients/tools, see the below.  This is the same query that we are executing in LinqPad, but in Chrome and on the address bar.

Of course, you can access all this information using PHP, Java, JavaScript or .NET language using our OData clients.

Further, we are beginning to have conversations with key technical leaders in other companies/organizations about adding support in other platforms/languages/products.

We are excited about the possibilities here and think there is a real opportunity to usher in a world where open data is not only possible, but pervasive.

As you may know, my vision is all about giving people the power to create, access and share their data as they will.

Although I work at Microsoft, I love to see other companies making progress on technologies that I believe soundly support this vision.

Recently, I have been using a product by FileMaker (owned by Apple) called Bento.

There is both a Mac and iPhone version.  You can sync your “database” (called a library in Bent0) between your Mac and iPhone.  You can also share your libraries with any Mac on your local subnet – like iTunes – via Bonjour.

I could nitpick features I want and lament what I consider a powerful platform play Apple could execute on, but in general I have nothing but praise, great praise, for this product.

If you own an iPhone or a Mac, I really encourage you to check it out. 

Great work Bento team!

Before we gave “M” an official external codename, we called it “D” internally.

Back in the “D” days, our team started working with MSR on a number things related to the language.

We recently released some of that work on the MSR site: Dminor.

Dminor is a data-modeling language based on M, the data-modeling language of Microsoft Oslo. Dminor provides extended compile-time checking of code, making use of an SMT (Satisfiability Modulo Theories) solver.

You can read more about it at http://whigmaleerie.spaces.live.com/blog/cns!C6149B019D236BF5!846.entry.

You can download it at http://research.microsoft.com/en-us/downloads/cd60cdb0-353f-48b3-81d7-177621eba1bf/default.aspx.

Congrats to Andy and company…

Every January 1st, since 2007, I have started the New Year by jumping in a large body of water.

Each year there is a different cohort of people that I am with.

Back in 2007 it was Don Box, Mike Abbott, and Brad Lovering, among others.

Last year, it was Dennis Pilarinos.

This year it is all me in the waters of the Pacific Ocean, although my brother waded a little to take pictures, for which I am thankful.

Now if you think this ritual is strange, I can add to that by talking about the importance of water purification and honoring the archetype of the Roman god Janus (where January comes from).

But I will forgo that and just show some pictures.

Next year I am going surfing.

This is a passage from Thus Spake Zarathustra (Chapter 14 — same title as this post) that resonated with me recently.

It is interesting to think about interactions with people that you either consciously or unconsciously wish to punish for a perceived slight.

Inspired ones they resemble: but it is not the heart that inspireth them—but vengeance. And when they become subtle and cold, it is not spirit, but envy, that maketh them so.

Their jealousy leadeth them also into thinkers’ paths; and this is the sign of their jealousy—they always go too far: so that their fatigue hath at last to go to sleep on the snow.

In all their lamentations soundeth vengeance, in all their eulogies is maleficence; and being judge seemeth to them bliss.

But thus do I counsel you, my friends: distrust all in whom the impulse to punish is powerful!

In the end, these interactions diminish you.

Nietzsche has a recommendation on how to ensure that you don’t fall prey to the above.

It can be found in Chapter 51 (On Passing-By): “Where one can no longer love, there should one—PASS BY!”

I normally tweet things like this, but I have been reading Thus Spake Zarathustra and Beyond Good and Evil for what seems the 21th time (they should be read together, I have found) and this post really spoke to me.

From EcoGeek.org

Lori Marino at Emory University is taking a scientific approach to determining how human dolphi[n]s are. She’s simply running them through an MRI and measuring the complexity of their brains. The result, unsurprisingly, is that dolphins are extremely smart. Their brains, according to Emory, are more complex than any other non-human brain, beating out Chimpanzees for the title.

The question that post raises is how this fact should impact the way that treat dolphins and the ethics associated with that.

My question is a little more in depth, as I would love to question the fundamental values that led us to believe that (generally) humans are more valuable than dolphins.

As absurd as it may seem, ask yourself the question, “Why am I more valuable than a dolphin?” and then follow the chain down to your axiomatic values.

You may think that has a simple answer, but under careful scrutiny you end up with a teleological question and those sorts of questions are very hard indeed.

That aside, I think it is wonderful that we are getting some quant that we can use to determine the intelligence of a given non-human species.

WCF Data Services supports two data sources out of the box:  an EF data source and a CLR data source.

This lets you expose an OData service from any database that supports EF as well as arbitrary CLR objects.

There are many reasons to need more than this, particularly if your data is not relational or you can’t afford to have CLR types floating around all over the place.

A canoncial example of this scenario is the SharePoint data model.

In order to support SharePoint, we adedd a dynamic data source provider model that we call IDSP internally.

This provider model consists of a number of interfaces that you can use to expose any data source as an OData service.

One of the PMs on the WCF Data Services team, AlexJ, is blogging about this at now.

http://blogs.msdn.com/alexj/archive/2010/01/04/creating-a-data-service-provider-part-1-intro.aspx

Worth checking out…

I went through iTunes tonight to determine my favorite music of the 2000s.

Based on number of tracks played and ratings, here is what what I found.

Top Albums

1. Songs for the Deaf by Queens of the Stone Age (Best track: First It Giveth)
2. In Keeping Secrets Of Silent Earth: 3 by Coheed and Cambria (Best track: A Favor House Atlantic)
3. FutureSex / LoveSounds by Justin Timberlake (Best track: What Goes Around…/…Comes Around)
4. Collision Course by Linkin Park & Jay-Z (Best track: Dirt Off Your Shoulder/Lying From You)
5. I Brought You My Bullets You Brought Me Your Love by My Chemical Romance (BT: Skylines and Turnstiles)
6. Three Cheers for Sweet Revenge by My Chemical Romance (BT: Thank You for the Venom)
7. Stadium Arcadium by Red Hot Chili Peppers (BT: Dani California)
8. Morning View by Incubus (BT: Wish You Were Here)
9. Renegades by Rage Against the Machine (BT: I’m Housin’)
10. Blood Mountain by Mastodon (BT: Blood and Thunder [Live])

Live Album of the 2000s

How the West was Won by Led Zeppelin

Live Single of the 2000s

Bold as Love (Live) by John Mayer

Artist of the 2000s:

My Chemical Romance

Best Cover of the 2000s

Imagine by A Perfect Circle

Imagine there’s no Heaven

It’s easy if you try

No hell below us

Above us only sky

Imagine all the people

Living for today

Imagine there’s no countries

It isn’t hard to do

Nothing to kill or die for

And no religion too

Imagine all the people

Living life in peace

You may say that I’m a dreamer

But I’m not the only one

I hope someday you’ll join us

And the world will be as one

Imagine no possessions

I wonder if you can

No need for greed or hunger

A brotherhood of man

Imagine all the people

Sharing all the world

You may say that I’m a dreamer

But I’m not the only one

I hope someday you’ll join us

And the world will live as one

I just completed Stoic Warriors.

This book was notable for two reasons.

First, it was the first book at I read on the Kindle device.

I have read several books on the iPhone with the Kindle app, which I enjoy (small screen and all), but the reading experience on the device itself was great, modulo one thing — it needs a backlight for nightime reading.

Second, this is the last book that I will complete this year and it will likely have the most impact of any book in 2009.

In short, the book outlines Stoic philosophy through the lens of the needs of today’s military forces. For example, what does Stoism tells us about anger, grief, loss, etc. — all emotions and feelings that are most manifest in military situations where life and limb hang in the balance.

The book covers basic Stoic philosophical teachings, largely through Cicero and Seneca, but a sizable portion of the work covers ‘applied Stoism’ through military experiences in WWI, WWII, Vietnam and the Gulf Wars.

The book has flaws. The final chapter should have likely kicked off the work as the notion of a human community makes war and its horror much more profound. The points about Buddhism and Stocism should have been explored. I think there was maybe a paragraph on it throughout the whole book — but I think the connection is profound. Lastly, the author seems to be an Aristotlian but doesn’t come out and say it.

I wrote a big wrap-up paragraph here, but the Wordpress iPhone app ate it, so I will close by saying that I recommend the book, especially if you are one that holds (as I do), that practical philosophic inquiry is not about Truth (whatever that is), but about ways of thinking about the world that help us survive and even thrive in a hostile environment.

I just looked at my most played songs that were released in 2009 (not my most played songs of the year).

New Fang: Them Crooked Vultures

Mind Eraser, No Chaser: Them Crooked Vultures

See You: Dinosaur Jr.

Catch & Release: Silversun Pickups

Best I Ever Had: Drake

Panic Switch: Silversun Pickups

Stillborn (Acoustic Version): Black Label Society

Check My Brain: Alice In Chains

Black Gives Way to Blue (Piano Mix): Alice In Chains

Run This Town (feat. Rihanna & Kanye West): Jay-Z

The true king can walk among his subjects unarmed, unguarded at night, so it is said.

Of this, I am of two minds.

My Romance, the desire for true brotherhood and love of the human family, wishes this to be true.

How long I have wanted such a world!

My Reason tells me that the king would be cut down in seconds.

How long have we lived in such a world and how much longer still?

Now, the real question is which world is best for humanity?

Every Communist must grasp the truth, “Political power grows out of the barrel of a gun.” Our principle is that the Party commands the gun, and the gun must never be allowed to command the Party. Yet, having guns, we can create Party organizations, as witness the powerful Party organizations which the Eighth Route Army has created in northern China. We can also create cadres, create schools, create culture, create mass movements. Everything in Yenan has been created by having guns. All things grow out of the barrel of a gun. According to the Marxist theory of the state, the army is the chief component of state power. Whoever wants to seize and retain state power must have a strong army. Some people ridicule us as advocates of the “omnipotence of war”. Yes, we are advocates of the omnipotence of revolutionary war; that is good, not bad, it is Marxist. The guns of the Russian Communist Party created socialism. We shall create a democratic republic. Experience in the class struggle in the era of imperialism teaches us that it is only by the power of the gun that the working class and the laboring masses can defeat the armed bourgeoisie and landlords; in this sense we may say that only with guns can the whole world be transformed. We are advocates of the abolition of war, we do not want war; but war can only be abolished through war, and in order to get rid of the gun it is necessary to take up the gun.

The (WCF) Data Services team just released the an update to .NET FX 3.5 for the OData.

New features in this release are:

· Projections: This ADO.NET Data Services URI format has been extended to express projections (i.e. you can now work with a subset of the properties of an entity).  This release includes both server and client library (including LINQ support) support for projections.  We’ve done a fair amount of work in this space to support roundtripping projected values, working with anonymous types, etc.  We’ll create a subsequent series of posts to describe this feature.

· Data Binding:  The data services client library for the .NET Framework 3.5 SP1 has been extended to support two-way data binding.

· Row Count: One scenario we heard a ton of feedback on after shipping V1 of ADO.NET Data Services in the .NET Framework 3.5SP1 is the ability for the a client of a data service to determine the total number of entities in a set without having to retrieve them all.  To address this need, we have extended the data services addressing scheme to allow a client to obtain this type of information without having to download all the entities in a set.

· Feed Customization (aka “Web Friendly Feeds”): A common ask we have received is to provide the ability to customize how entities are mapped into the various elements of an AtomPub feed.  This feature does just that by providing a data service author declarative control over how the data service runtime maps the properties of an entity (e.g. a Customer, Order, etc) to the elements of a feed.

· Server Driven Paging (SDP): This one is best described by example.  If you had a data service that exposes photos, you likely want to limit the total number of photos a single request to the service can retrieve because the total collection of photos may be very large.  This feature allows a service author to set per collection limits on the total number of entities returned for each request.  In addition to limiting the number of photos returned per request, the server provides the client a “next link” which is simply a URI specifying how to continue retrieving the rest of the entities in the collection not returned by the first request.  For those familiar with AtomPub, this feature adds support for AtomPub <link rel=”next” …> elements to the data service runtime.

· Enhanced BLOB Support: This feature enhances the BLOB support provided in V1 to enable data services to stream arbitrarily large BLOBs, store binary content separate from its metadata, easily defer the loading of BLOB content when its metadata is requested, etc.

· Request Pipeline: We have started to expose events throughout the data services server request processing pipeline.  For this release we’ll expose request level events and in future we’ll look to expose more fine grained events based on your feedback.  The goal of exposing our processing pipeline is to allow services further transparency into a data service such that a service author can do things such as setting HTTP response cache headers, wrapping interceptor processing and data service request processing in a single transaction, etc.

· New “Data Service Provider” Interfaces for Custom Provider Writers: As the data services runtime has evolved, so has the number of ways people want to plug data into the data service framework.  In V1, two methods (Entity Framework and arbitrary .NET classes) were supported to enable a data service to interact with various diverse data sources.  To address another class of environments and data sources we have introduced a way to write a “custom” provider for those cases when the previous two provider models don’t meet your needs.