Douglas Purdy

As noted on the OData.org site, WebSphere eXtreme Scale uses the OData protocol.

Billy Newport, an IBM Distinguished Engineer, was interviewed recently on why they selected a RESTful data service as the API and how OData helped.

The article: http://searchsoa.techtarget.com/news/article/0,289142,sid26_gci1379765,00.html

More product details: http://www.ibm.com/developerworks/websphere/downloads/xs_rest_service.html 

It is great to see that developers, regardless of platform/language, have a simple way to consume these services.

I’ll make one interesting note about this implementation.

As near as I can tell, Billy’s team implemented OData without ever talking to anyone at Microsoft.

I suspect they used the protocol documents we have online (these define the protocol with even greater precision that many standard specifications I have seen) and a HTTP trace tool.

Having been involved in distributed computing/protocol integration work for a long time, that is quite an achievement.

It could speak to simplicity of the protocol (it is just conventions/extensions over HTTP/AtomPub), the quality of the documentation or the intelligence/patience of the IBM team.

Likely it was all of these.

I am spending a lot of time on the Open Data Protocol (OData).

Many of us at Microsoft that believe that this protocol can help usher in a more open and programmable Web.

A protocol like this is a prerequisite for the broader “Infobus” and “Information Liberation” vision that I often talk about.

Envision a world where every application/Web property exposes its data (actually your data) in a way that you can easily query it in rich tools like Excel/Numbers or write your own mash-up/custom application.

A world were government data is transparent, queryable and accessible to any citizen.

A world where you can you can ask a question and know: “There’s a feed for that”.

We are just starting, but we (Microsoft) already has an impressive list of OData producers/consumers coming online (including SharePoint, SQL Server 2008, IIS/ASP.NET, etc.) and there are more in the pipeline that we will announce this year.

We are working as hard as we can to get OData support on as many platforms (both client and server/service) as we can, so a developer on any platform can both consume and produce these feeds.

We are begin to engaging partners, consumers and even competitors in a more structured way to see how we can work together to build up an ecosystem of open data services.

To make this vision a little more concrete, let’s look at a couple of screenshots.

Below is a third-party tool called LinqPad.  LinqPad recently added support for OData, which is demonstrated below.  The most interesting thing is the data service that I am accessing.  The City of Edmonton, Canada is exposing datasets as OData feeds at http://data.edmonton.ca/.  That lets tools that understand OData, like LinqPad, access this information in rich ways.

Also, notice the two other data services in the tool.  These point to District of Columbia and New America Foundation data at http://ogdisdk.cloudapp.net/.

This next screenshot is of Excel (via the PowerPivot plug-in) accessing the same data service.

Just to prove that this is all open, accessible and available to non-Microsoft clients/tools, see the below.  This is the same query that we are executing in LinqPad, but in Chrome and on the address bar.

Of course, you can access all this information using PHP, Java, JavaScript or .NET language using our OData clients.

Further, we are beginning to have conversations with key technical leaders in other companies/organizations about adding support in other platforms/languages/products.

We are excited about the possibilities here and think there is a real opportunity to usher in a world where open data is not only possible, but pervasive.

As you may know, my vision is all about giving people the power to create, access and share their data as they will.

Although I work at Microsoft, I love to see other companies making progress on technologies that I believe soundly support this vision.

Recently, I have been using a product by FileMaker (owned by Apple) called Bento.

There is both a Mac and iPhone version.  You can sync your “database” (called a library in Bent0) between your Mac and iPhone.  You can also share your libraries with any Mac on your local subnet – like iTunes – via Bonjour.

I could nitpick features I want and lament what I consider a powerful platform play Apple could execute on, but in general I have nothing but praise, great praise, for this product.

If you own an iPhone or a Mac, I really encourage you to check it out. 

Great work Bento team!

Before we gave “M” an official external codename, we called it “D” internally.

Back in the “D” days, our team started working with MSR on a number things related to the language.

We recently released some of that work on the MSR site: Dminor.

Dminor is a data-modeling language based on M, the data-modeling language of Microsoft Oslo. Dminor provides extended compile-time checking of code, making use of an SMT (Satisfiability Modulo Theories) solver.

You can read more about it at http://whigmaleerie.spaces.live.com/blog/cns!C6149B019D236BF5!846.entry.

You can download it at http://research.microsoft.com/en-us/downloads/cd60cdb0-353f-48b3-81d7-177621eba1bf/default.aspx.

Congrats to Andy and company…

WCF Data Services supports two data sources out of the box:  an EF data source and a CLR data source.

This lets you expose an OData service from any database that supports EF as well as arbitrary CLR objects.

There are many reasons to need more than this, particularly if your data is not relational or you can’t afford to have CLR types floating around all over the place.

A canoncial example of this scenario is the SharePoint data model.

In order to support SharePoint, we adedd a dynamic data source provider model that we call IDSP internally.

This provider model consists of a number of interfaces that you can use to expose any data source as an OData service.

One of the PMs on the WCF Data Services team, AlexJ, is blogging about this at now.

http://blogs.msdn.com/alexj/archive/2010/01/04/creating-a-data-service-provider-part-1-intro.aspx

Worth checking out…

The (WCF) Data Services team just released the an update to .NET FX 3.5 for the OData.

New features in this release are:

· Projections: This ADO.NET Data Services URI format has been extended to express projections (i.e. you can now work with a subset of the properties of an entity).  This release includes both server and client library (including LINQ support) support for projections.  We’ve done a fair amount of work in this space to support roundtripping projected values, working with anonymous types, etc.  We’ll create a subsequent series of posts to describe this feature.

· Data Binding:  The data services client library for the .NET Framework 3.5 SP1 has been extended to support two-way data binding.

· Row Count: One scenario we heard a ton of feedback on after shipping V1 of ADO.NET Data Services in the .NET Framework 3.5SP1 is the ability for the a client of a data service to determine the total number of entities in a set without having to retrieve them all.  To address this need, we have extended the data services addressing scheme to allow a client to obtain this type of information without having to download all the entities in a set.

· Feed Customization (aka “Web Friendly Feeds”): A common ask we have received is to provide the ability to customize how entities are mapped into the various elements of an AtomPub feed.  This feature does just that by providing a data service author declarative control over how the data service runtime maps the properties of an entity (e.g. a Customer, Order, etc) to the elements of a feed.

· Server Driven Paging (SDP): This one is best described by example.  If you had a data service that exposes photos, you likely want to limit the total number of photos a single request to the service can retrieve because the total collection of photos may be very large.  This feature allows a service author to set per collection limits on the total number of entities returned for each request.  In addition to limiting the number of photos returned per request, the server provides the client a “next link” which is simply a URI specifying how to continue retrieving the rest of the entities in the collection not returned by the first request.  For those familiar with AtomPub, this feature adds support for AtomPub <link rel=”next” …> elements to the data service runtime.

· Enhanced BLOB Support: This feature enhances the BLOB support provided in V1 to enable data services to stream arbitrarily large BLOBs, store binary content separate from its metadata, easily defer the loading of BLOB content when its metadata is requested, etc.

· Request Pipeline: We have started to expose events throughout the data services server request processing pipeline.  For this release we’ll expose request level events and in future we’ll look to expose more fine grained events based on your feedback.  The goal of exposing our processing pipeline is to allow services further transparency into a data service such that a service author can do things such as setting HTTP response cache headers, wrapping interceptor processing and data service request processing in a single transaction, etc.

· New “Data Service Provider” Interfaces for Custom Provider Writers: As the data services runtime has evolved, so has the number of ways people want to plug data into the data service framework.  In V1, two methods (Entity Framework and arbitrary .NET classes) were supported to enable a data service to interact with various diverse data sources.  To address another class of environments and data sources we have introduced a way to write a “custom” provider for those cases when the previous two provider models don’t meet your needs.

I tweeted this, but I think it deserves a full post.  This a great example that leverages many of the different aspects of SQL Server Modeling (”M”-based DSLs, “M”, Modeling Services, etc.) to enable a better experience around application development/management.

One of the major new WCF features in .NET 4 is the Routing Service which is a configurable WCF-based service that supports content-based routing and protocol bridging. The content based routing capability in .NET 4 allows for WCF to perform message filtering based on content contained in either the SOAP headers or within the message body. For instance, if a company has two different versions of the same service deployed, the client application can call into a central routing service. The routing service can then forward the message to the correct backend service based on information it extracts from the incoming message, such as version number. The routing service also supports error handling routing that can automatically resend the message to another destination endpoint in the event of an error.

Unfortunately, the default routing service is not the easiest to manage or configure. Fortunately, the recently released SQL Server Modeling CTP makes it easy to build model driven application and the .NET 4 router configuration is a good modeling candidate. As such, the RouterManager sample which is available for download from MSDN Code Gallery, shows how this can be accomplished using Visual Studio 2010 Beta 2 and the SQL Server Modeling CTP which was released in conjunction with PDC09. This is the first in a series of articles that will cover the architecture and concepts used in the RouterManager sample application to create a fully model driven content based router.

The SQL Server Modeling CTP can be used to create a domain specific language (DSL) that can make the routing configuration of the WCF routing service much simpler. This is accomplished by first defining a language that is human readable, which will provide IT Operations and others within an organization who are not expert developers with an easier way to interact with applications. The CTP’s modeling language features make it easy to define the structure of the configuration data in a human readable representation that can be shared within an application’s modules, tiers and tools or between even between applications. Also included, as part of the CTP, is the SQL Server Modeling Services, which provides functionality to store, access and share both the physical and metadata representation of the model. The Modeling CTP also provides a visual tool called “Quadrant” for viewing and interacting with models and model data.

http://blogs.msdn.com/dmcat/archive/2009/11/16/model-driven-content-based-routing-using-sql-server-modeling-ctp-part-i.aspx

I don’t know if these work against the latest CTP, but you can ask Ceyhun Ciper.

http://www.sixpairs.com/intellipad/mgraph-visualization-intellipad-plugin.aspx

“MUrl” is updated to the latest CTP (thanks SNinja and the rest of the Intellipad team).

There is a good readme in the zip file that outlines how to build, install and use it.

Download it now. 

[Updated on 2010/1/4 to fix a bug in installation]

Twitter.murl

How the credential information is stored

AAguiar asked me a question related to my EDM 3.0: Bring EDM and “M” together post that I wanted to answer here.

@douglasp In .edmx I could add attributes in my own namespace to enrich my model. How is that going to work with ‘M’?

In the latest CTP you can add attributes/annotations for “M” types, extents, fields, functions.

Below is a simple example.  Note the SQL that is generated by default, but you can control that as well.

I am on vacation this week, so that means that I am writing as much code as I can.

One of my little projects on .NET (I am spending a bunch of time writing iPhone code — updates to LocoFoto and something new) is an update to MUrl sample that I used as part of the MIX09 talk with Sells.

I had hoped to post it tonight, but I am running into some issues that the “Intellipad” team is helping with.

Hopefully tomorrow…

Speaking of “Intellipad”, make sure you check out the latest version of “Quadrant” in the CTP.

We have folded the Intellipad control into “Quadrant” in many ways that I think really adds to the experience (the query bar being the most obvious).

[Updated:  I re-recorded and added a voice over.]

The below screencast runs ~2 minutes and worth watching if you care about interacting with data (and who doesn’t).

What you see are 3 key features of “Quadrant” in the latest CTP:

• Access to SQL Azure databases
• Creating custom views (including master detail) by mashing views together
• See the “markup” of any view in Quadrant and steal it/change it (just like view source in a Web browser)

You also get to see the infinite canvas and the zoom features indirectly.

I am thinking about posting a couple more of these 2 minute demos over the course of the next couple weeks.

If you want to see something more in-depth now, check out the “Quadrant” PDC talk or Quadrant Overview video.

[Updated:  Download Video @ http://www.douglaspurdy.com/wp-content/uploads/quadrant1.swf]

One of the biggest things we have done in recent months is start to bring EDM and “M” together into one data model.

I could spend a bunch of time talking about the design principles behind that work, but I want to be as concrete as I can and get down to something that help you (if you are a .NET developer) today.

In a nutshell, “M” replaces the EDMX XML file that you are no doubt familiar with if you have used either the Entity Framework or Data Services.

Said differently, we have a nice human-friendly DSL that replaces a human-hostile XML file.

In addition, this nice, warm and friendly DSL can both generate your database and the EF classes you need to program against the database.

Before I talk about how this works, I want to say one thing:

EF and DS are at the knee of the adoption curve and we are not going to do anything to slow down that adoption; so rest assured that we are committed bring EDM and “M” together in a way that places the upmost importance on existing and future investments in EF and DS.

With that said, let’s look at a simple example (if you want to follow along, download the bits first)…

1. Look at all the “Oslo” VS projects that we have in the CTP bits (the name landed too late to get the bits updated).

2.  Let’s create a simple “M” model

3. Set the connection string in the project properties

4. Deploy (this will create the database and the CLR classes you need)

5. Write some code against the generated classes

6. Run the App

8. Under the covers (you can look in the obj folder to see what we generate)

9. Really under the covers, especially if you don’t like to use an IDE (you know who you are)

This uses a command-line option for m.exe and tool that will generate the EF classes for you.

I am fairly sure that someone on the language team is going to throw something at me when they read this.

When Sells and I talked about this feature at PDC, it resulted in at least two separate rounds of applause from the audience and I know it was a highlight (one of many) at the Box/Anderson anchor session.

You can get more details on this at http://msdn.microsoft.com/en-us/library/ee713219(VS.85).aspx.

http://msdn.microsoft.com/en-us/library/ee730343.aspx

This is one of the best articles that I have read about the Microsoft data and modeling history and strategy. 

Kraig wrote this piece and I love it.

Below is at the end of the doc that talks about modeling…

Data Development Futures: Modeling

If we take a step back from the whole arc of the discussion we’ve had in this article, we can see a definite trend toward greater levels of abstraction. The earliest data access solutions for SQL Server, like DB-Library and ESQL for C, were sufficient though quite rudimentary. Technologies like ODBC created an abstraction layer above the proprietary APIs of different databases, a model that OLE DB, ADO, and ADO.NET continue to follow.

Most recently, the Entity Framework has gone a step further to create an additional abstraction layer not over the data access API but over the structure of a relational database itself. Similarly, Data Services transform any number of diverse data sources into something accessible through a simple REST-based exchange protocol. (In fact, Microsoft expects that using such protocols will become increasingly popular, as it allows data providers and consumers to evolve independently from their programming model.)

Taken as a whole, this trend can be described as a trend toward modeling, the act of creating representations of real-world concepts that are internally translated into the representations that computer systems (like database engines) inherently understand.

What’s key here is that modeling itself is something that takes place outside of the database, in large part because data often spans multiple databases and multiple formats. Thus what matters is the richness of the pipeline between what’s in the data store and what runs outside that store.

Increasing that richness is the purpose of Microsoft’s next wave of investments beyond the Entity Framework and Data Services (see Entity Framework futures and Data Services futures), namely a body of technologies collectively called the SQL Server Modeling CTP (Community Technology Preview). Rather than replacing existing data development methods, these technologies introduce new ways of working with SQL Server databases along with greater availability of metadata alongside the data itself.

Metadata—or information about data and applications—is the key to the next advances in developer productivity with data-oriented applications. Modeling, in other words, is a recognition that data about an application, and data about data, is just as important as the application and data themselves. And the components of the SQL Server Modeling CTP, shown in Figure 10 as a projection from Figure 9, are the ways in which Microsoft is beginning to explore this new territory.

Figure 10: Future technologies in the SQL Server Modeling CTP.

The code name “M” language is like a more manageable (though more limited) form of Transact-SQL, the language normally used to describe data schema and values. It’s also closely aligned with the purpose of the EDMX files used in the Entity Framework to implement the conceptual layer (or Entity Data Model). The "M" language is, in fact, being developed as a textual language that also implements the Entity Data Model, and will serve alongside the XML-based EDMX dialect. Furthermore, "M" includes powerful support for the creation of domain-specific languages (DSLs) in which application logic, data access layers, and even an application’s user interface can be easily defined by even non-developers.

SQL Server Modeling Services, for its part, deals with enterprise-wide metadata and the kinds of applications that enterprises are interested in building around that metadata. And the code name “Quadrant” tool provides a visual means of interacting with relational data in ways that have traditionally been either very difficult or have necessitated a custom application of some kind. Many “forms over data” applications that have to date been written using the other data access technologies we’ve seen can be quickly assembled directly within “Quadrant”.

I love the Zune HD.  I love the Zune desktop software – little touches like the below.

This is such good work and it shows the quality of (cool) work that Microsoft is capable of doing.

All of our PDC talks are available.  You can get both videos and slides from the links.

If you have been skeptical of our recent renaming of “Oslo”, I encourage you to actually watch the videos, download the bits and then make up your mind.

More importantly, if you are a .NET developer, please take a look at what we are doing around EF and DS – especially the OData protocol.

Data/Modeling “Anchor Session”

Data Programming and Modeling for the Microsoft .NET Developer

Coming in VS 2010

Evolving ADO.NET Entity Framework in .NET 4 and Beyond

ADO.NET Data Services: What’s new with the RESTful data services framework

Developing REST Applications with the .NET Framework

SQL Server Modeling (nee “Oslo”)

Microsoft Project Code Name “M”: The Data and Modeling Language

Building Data-Driven Applications Using Microsoft Project Code Name "Quadrant" and Microsoft Project Code Name "M"

SQL Server Modeling Services: Using Metadata to Drive Application Design, Development and Management

Using SQL Server Modeling

The ‘M’-Based System.Identity Model for Accessing Directory Services

When we announced at the components of “Oslo” would ship in a future version of SQL Server, a common question was “Does that mean that I can only use these components with SQL Server?”.

I know it is hard for some people to hear the name “SQL Server” and think anything more than “relational engine”, but it is so much more.

The reality is that SQL Server, for lack of a better brand, is our overall data/information management offering that spans more scenarios than you can shake a stick at.

In addition, this offering is used through the entire Microsoft platform, from Windows, to SharePoint, to System Center, to Dynamics.

If there is data involved, you can bet components of the “SQL Server” are being used.

What does this mean for our modeling technologies? Only good things.

This means that our approach to modeling will get widespread adoption throughout the Microsoft internal and external ecosystem that uses SQL Server.

Want to see an example of an end-to-end scenario?

Go http://microsoftpdc.com/Sessions/KEY01 and jump to 1:57:00 in the presentation.

That is SQL Server Modeling in action.

Visual Studio and System Center working together over a common model (written in “M”) in stored in SQL Server Modeling Services.

This capability will add tremendous value to customers and it is brought to you through SQL Server and our investments in “Oslo”.

Want to see another example?

Go watch the video for The ‘M’-Based System.Identity Model for Accessing Directory Services.

Again, this is going to add tremendous value to customers and it is brought to you through SQL Server and our investments in “Oslo”.

To but a bow on this, go read: Oslo transforms into the underlying application model of the cloud.

At PDC, after I explained the above, most people understood what we were doing with SQL Modeling, but two questions seemed to remain.

First, “How do I use these technologies in my application?”

Second, “This seems like a bunch of Microsoft internal stuff, can I just get the DSL stuff?  Oh, and please don’t say SQL Server around me”.

The former question is straightforward to answer.

If you use a database or XML to drive your application, our modeling technologies can help you a great deal.

Most ISVs that we talk with are already “model-driven” using either a database or XML files, but they effectively roll their own modeling platform.

SQL Modeling provides modeling capabilities that these ISVs can leverage, just like Microsoft “ISVs” are doing today.

These same capabilities that make model-driven applications possible, also help with applications that deal with lots of operational data.

This is one of the core reasons that we have brought “M” and EDM together and started to leverage the Entity Framework and Data Services.

You can model your application’s domain in “M” and get a wonderful conceptual model that you program against in VS/.NET.

The latter question is straightforward too.

Ignore the name SQL Server and check out the bits.

If we are doing something stupid and we are blocking you from using the DSL part of “M” in the way you want, send us the feedback.

I think you will find that enabling a DSL ecosystem is an important aspect of what we are trying to do.

“M” is a language for data – and we ship our implementation language with the Microsoft data offering – which for better or worst, we call SQL Server.

Lars is after me to come up with a better name for SQL Server/SQL Azure – which I am working on…

"What’s in a name? That which we call a rose
By any other name would smell as sweet." – Romeo and Juliet

Our team did a lot of work prior to and during PDC to ensure that we were clearly explaining the value of the data & modeling technologies we are delivering for customers.

One of the things that we really want to get clear was the value of the ADO.NET Data Services protocol, independent of our .NET implementation. 

This was especially important to do as we are folding much of the protocol implementation into WCF proper which will enable WCF Core Services and WCF RIA Services to implement the protocol as well (this resulting in us renaming ADO.NET Data Services to WCF Data Services).

In addition, it was very hard to talk about SharePoint, Excel, etc. implementing the “ADO.NET Data Services Protocol” – as many of us on the team can attest.

Lastly, we believe that this protocol gives us, our customers, and even our competitors an opportunity to unlock data sources throughout the enterprise and the Internet.

The most notable example of this is the “Dallas” service we announced at PDC.

The name “ADO.NET Data Services Protocol” got in the way of explaining this opportunity.

As such, we decided to give a name to the protocol – Open Data Protocol or OData for short.

We had already documented the protocol fully, but this naming event gave us the perfect opportunity to do two other things:  release the protocol specification under the OSP and start a conversation with the community about rolling the features of the protocol into AtomPub or other appropriate standards.

The name landed in the nick of time for PDC, so we are still getting all of our resources lined up behind the new name.

You can read an overview at odata.org, which we’ll be updating with more material soon.

Based on the reaction that we got at PDC, all three moves (the rename, the OSP, and the promise of community engagement toward standardization) was very welcomed and the promise seemed clear to everyone I spoke with.

As Jon Udell says, “OData is grease to cut data friction”.

One of the things that I am most excited about is the opportunity for this protocol (and whatever evolves out of the engagement with the community) to be used as the the basis on the “InfoBus”.

Pablo Castro (we call him Mr. OData now) is going to take point on our engagement with the community and you’ll be hear more from him soon (we are all taking a little break from the run-up to PDC right now).

The “Repository” now has an official name: SQL Service Modeling Services.

This name continues our tradition of naming functional components beyond the core engine with the services moniker.

Other such services are Reporting Services, Analysis Services, Integration Services, etc.

Shosh will be talking more about it at PDC at SQL Server Modeling Services: Using Metadata to Drive Application Design, Development and Management.

We are still working on the official names for “Quadrant” and “M”.  We have some good candidates, but we didn’t get to close for PDC.

Just as a reminder:

“Oslo” –> SQL Service Modeling

The components of the SQL Server Modeling are:

• “M” is a highly productive, developer friendly, textual language for defining schemas, queries, values, functions and DSLs for data
• “Quadrant” is a customizable tool for interacting with large datasets stored in SQL Server databases
• SQL Service Modeling Services (nee “Repository”) is a SQL Server role for the the secure sharing of models between applications and systems

I just saw a final draft of the System.Identity PDC slides.

System.Identity is built by the Active Directory team.

It is one of the first uses of SQL Server Modeling (nee “Oslo’) that we are talking about publically (you will see others, but this is the first announced Microsoft project that is ready to have a PDC session).

If you are interested in how a real and big ISV (they are considered an ISV for us) uses SQL Server Modeling, this session should be of interest to you.

The ‘M’-Based System.Identity Model for Accessing Directory Services