Douglas Purdy

I am spending a lot of time on the Open Data Protocol (OData).

Many of us at Microsoft that believe that this protocol can help usher in a more open and programmable Web.

A protocol like this is a prerequisite for the broader “Infobus” and “Information Liberation” vision that I often talk about.

Envision a world where every application/Web property exposes its data (actually your data) in a way that you can easily query it in rich tools like Excel/Numbers or write your own mash-up/custom application.

A world were government data is transparent, queryable and accessible to any citizen.

A world where you can you can ask a question and know: “There’s a feed for that”.

We are just starting, but we (Microsoft) already has an impressive list of OData producers/consumers coming online (including SharePoint, SQL Server 2008, IIS/ASP.NET, etc.) and there are more in the pipeline that we will announce this year.

We are working as hard as we can to get OData support on as many platforms (both client and server/service) as we can, so a developer on any platform can both consume and produce these feeds.

We are begin to engaging partners, consumers and even competitors in a more structured way to see how we can work together to build up an ecosystem of open data services.

To make this vision a little more concrete, let’s look at a couple of screenshots.

Below is a third-party tool called LinqPad.  LinqPad recently added support for OData, which is demonstrated below.  The most interesting thing is the data service that I am accessing.  The City of Edmonton, Canada is exposing datasets as OData feeds at http://data.edmonton.ca/.  That lets tools that understand OData, like LinqPad, access this information in rich ways.

Also, notice the two other data services in the tool.  These point to District of Columbia and New America Foundation data at http://ogdisdk.cloudapp.net/.

This next screenshot is of Excel (via the PowerPivot plug-in) accessing the same data service.

Just to prove that this is all open, accessible and available to non-Microsoft clients/tools, see the below.  This is the same query that we are executing in LinqPad, but in Chrome and on the address bar.

Of course, you can access all this information using PHP, Java, JavaScript or .NET language using our OData clients.

Further, we are beginning to have conversations with key technical leaders in other companies/organizations about adding support in other platforms/languages/products.

We are excited about the possibilities here and think there is a real opportunity to usher in a world where open data is not only possible, but pervasive.

As you may know, my vision is all about giving people the power to create, access and share their data as they will.

Although I work at Microsoft, I love to see other companies making progress on technologies that I believe soundly support this vision.

Recently, I have been using a product by FileMaker (owned by Apple) called Bento.

There is both a Mac and iPhone version.  You can sync your “database” (called a library in Bent0) between your Mac and iPhone.  You can also share your libraries with any Mac on your local subnet – like iTunes – via Bonjour.

I could nitpick features I want and lament what I consider a powerful platform play Apple could execute on, but in general I have nothing but praise, great praise, for this product.

If you own an iPhone or a Mac, I really encourage you to check it out. 

Great work Bento team!

Before we gave “M” an official external codename, we called it “D” internally.

Back in the “D” days, our team started working with MSR on a number things related to the language.

We recently released some of that work on the MSR site: Dminor.

Dminor is a data-modeling language based on M, the data-modeling language of Microsoft Oslo. Dminor provides extended compile-time checking of code, making use of an SMT (Satisfiability Modulo Theories) solver.

You can read more about it at http://whigmaleerie.spaces.live.com/blog/cns!C6149B019D236BF5!846.entry.

You can download it at http://research.microsoft.com/en-us/downloads/cd60cdb0-353f-48b3-81d7-177621eba1bf/default.aspx.

Congrats to Andy and company…

WCF Data Services supports two data sources out of the box:  an EF data source and a CLR data source.

This lets you expose an OData service from any database that supports EF as well as arbitrary CLR objects.

There are many reasons to need more than this, particularly if your data is not relational or you can’t afford to have CLR types floating around all over the place.

A canoncial example of this scenario is the SharePoint data model.

In order to support SharePoint, we adedd a dynamic data source provider model that we call IDSP internally.

This provider model consists of a number of interfaces that you can use to expose any data source as an OData service.

One of the PMs on the WCF Data Services team, AlexJ, is blogging about this at now.

http://blogs.msdn.com/alexj/archive/2010/01/04/creating-a-data-service-provider-part-1-intro.aspx

Worth checking out…

The (WCF) Data Services team just released the an update to .NET FX 3.5 for the OData.

New features in this release are:

· Projections: This ADO.NET Data Services URI format has been extended to express projections (i.e. you can now work with a subset of the properties of an entity).  This release includes both server and client library (including LINQ support) support for projections.  We’ve done a fair amount of work in this space to support roundtripping projected values, working with anonymous types, etc.  We’ll create a subsequent series of posts to describe this feature.

· Data Binding:  The data services client library for the .NET Framework 3.5 SP1 has been extended to support two-way data binding.

· Row Count: One scenario we heard a ton of feedback on after shipping V1 of ADO.NET Data Services in the .NET Framework 3.5SP1 is the ability for the a client of a data service to determine the total number of entities in a set without having to retrieve them all.  To address this need, we have extended the data services addressing scheme to allow a client to obtain this type of information without having to download all the entities in a set.

· Feed Customization (aka “Web Friendly Feeds”): A common ask we have received is to provide the ability to customize how entities are mapped into the various elements of an AtomPub feed.  This feature does just that by providing a data service author declarative control over how the data service runtime maps the properties of an entity (e.g. a Customer, Order, etc) to the elements of a feed.

· Server Driven Paging (SDP): This one is best described by example.  If you had a data service that exposes photos, you likely want to limit the total number of photos a single request to the service can retrieve because the total collection of photos may be very large.  This feature allows a service author to set per collection limits on the total number of entities returned for each request.  In addition to limiting the number of photos returned per request, the server provides the client a “next link” which is simply a URI specifying how to continue retrieving the rest of the entities in the collection not returned by the first request.  For those familiar with AtomPub, this feature adds support for AtomPub <link rel=”next” …> elements to the data service runtime.

· Enhanced BLOB Support: This feature enhances the BLOB support provided in V1 to enable data services to stream arbitrarily large BLOBs, store binary content separate from its metadata, easily defer the loading of BLOB content when its metadata is requested, etc.

· Request Pipeline: We have started to expose events throughout the data services server request processing pipeline.  For this release we’ll expose request level events and in future we’ll look to expose more fine grained events based on your feedback.  The goal of exposing our processing pipeline is to allow services further transparency into a data service such that a service author can do things such as setting HTTP response cache headers, wrapping interceptor processing and data service request processing in a single transaction, etc.

· New “Data Service Provider” Interfaces for Custom Provider Writers: As the data services runtime has evolved, so has the number of ways people want to plug data into the data service framework.  In V1, two methods (Entity Framework and arbitrary .NET classes) were supported to enable a data service to interact with various diverse data sources.  To address another class of environments and data sources we have introduced a way to write a “custom” provider for those cases when the previous two provider models don’t meet your needs.

I tweeted this, but I think it deserves a full post.  This a great example that leverages many of the different aspects of SQL Server Modeling (”M”-based DSLs, “M”, Modeling Services, etc.) to enable a better experience around application development/management.

One of the major new WCF features in .NET 4 is the Routing Service which is a configurable WCF-based service that supports content-based routing and protocol bridging. The content based routing capability in .NET 4 allows for WCF to perform message filtering based on content contained in either the SOAP headers or within the message body. For instance, if a company has two different versions of the same service deployed, the client application can call into a central routing service. The routing service can then forward the message to the correct backend service based on information it extracts from the incoming message, such as version number. The routing service also supports error handling routing that can automatically resend the message to another destination endpoint in the event of an error.

Unfortunately, the default routing service is not the easiest to manage or configure. Fortunately, the recently released SQL Server Modeling CTP makes it easy to build model driven application and the .NET 4 router configuration is a good modeling candidate. As such, the RouterManager sample which is available for download from MSDN Code Gallery, shows how this can be accomplished using Visual Studio 2010 Beta 2 and the SQL Server Modeling CTP which was released in conjunction with PDC09. This is the first in a series of articles that will cover the architecture and concepts used in the RouterManager sample application to create a fully model driven content based router.

The SQL Server Modeling CTP can be used to create a domain specific language (DSL) that can make the routing configuration of the WCF routing service much simpler. This is accomplished by first defining a language that is human readable, which will provide IT Operations and others within an organization who are not expert developers with an easier way to interact with applications. The CTP’s modeling language features make it easy to define the structure of the configuration data in a human readable representation that can be shared within an application’s modules, tiers and tools or between even between applications. Also included, as part of the CTP, is the SQL Server Modeling Services, which provides functionality to store, access and share both the physical and metadata representation of the model. The Modeling CTP also provides a visual tool called “Quadrant” for viewing and interacting with models and model data.

http://blogs.msdn.com/dmcat/archive/2009/11/16/model-driven-content-based-routing-using-sql-server-modeling-ctp-part-i.aspx

I don’t know if these work against the latest CTP, but you can ask Ceyhun Ciper.

http://www.sixpairs.com/intellipad/mgraph-visualization-intellipad-plugin.aspx

“MUrl” is updated to the latest CTP (thanks SNinja and the rest of the Intellipad team).

There is a good readme in the zip file that outlines how to build, install and use it.

Download it now. 

[Updated on 2010/1/4 to fix a bug in installation]

Twitter.murl

How the credential information is stored

AAguiar asked me a question related to my EDM 3.0: Bring EDM and “M” together post that I wanted to answer here.

@douglasp In .edmx I could add attributes in my own namespace to enrich my model. How is that going to work with ‘M’?

In the latest CTP you can add attributes/annotations for “M” types, extents, fields, functions.

Below is a simple example.  Note the SQL that is generated by default, but you can control that as well.

I am on vacation this week, so that means that I am writing as much code as I can.

One of my little projects on .NET (I am spending a bunch of time writing iPhone code — updates to LocoFoto and something new) is an update to MUrl sample that I used as part of the MIX09 talk with Sells.

I had hoped to post it tonight, but I am running into some issues that the “Intellipad” team is helping with.

Hopefully tomorrow…

Speaking of “Intellipad”, make sure you check out the latest version of “Quadrant” in the CTP.

We have folded the Intellipad control into “Quadrant” in many ways that I think really adds to the experience (the query bar being the most obvious).