Douglas Purdy

We are looking for to this year’s PDC.  In addition to the keynotes and bits we are delivering, we will be featuring various data and modeling technologies in the below sessions.  I think we have a couple more “data” sessions to be published, but this should give you the general idea.  See you in LA.

The “data & modeling keynote” with Don and Chris

Data Programming and Modeling for the Microsoft .NET Developer

Astoria and EF talks

ADO.NET Data Services- What’s new with the RESTful data services framework

Evolving ADO.NET Entity Framework in .NET 4 and Beyond

“Oslo” talks

Building Data-Driven Applications Using Microsoft Project Code Name "Quadrant" and Microsoft Project Code Name "M"

Microsoft Project Code Name “Repository”: Using Metadata to Drive Application Design, Development, and Management

Microsoft Project Code Name “M”- The Data and Modeling Language

I was just reading William Vambenepe’s So long Oslo post and thought I would comment here.

While we are aligning “Oslo” more closely with our data platform assets as I discussed in my On Oslo post, that does not mean that this work is any less important for DSI/Dynamic IT scenarios.

We spent a great deal of time with the System Center team before this move.  Since the move, we have spent more – a lot more.

Part of that is some things coming out of incubation in the management space that I am not going to talk about.

Another part is that having a modeling language, a set of common models and a repository as part of the underlying data platform makes it easier to get some of this work done.

Net: the “Oslo” technologies have and will continue to play an important role in enabling DSI/Dynamic IT scenarios.

I haven’t posted anything about “Oslo” since ~May.

Before I mention what we have been up to, I want to give a little recap.

Back in September of 2008 (previous to PDC 2008), I wrote a post titled, What is Oslo?.  Beyond my typical hyperbole, this post remains a fairly accurate description of what “Oslo” is and how we describe it.  That said, this description does not outline how the project developed prior or after PDC 2008.

Back in September 2007, we announced “Oslo” at the SOA & BP Conference in Redmond.  “Oslo” was the name that we gave to a multiyear, multiproduct effort to simplify the application development lifecycle by enhancing .NET, Visual Studio, Biztalk and SQL Server.

As we ran toward PDC 2008, we discovered two important things.  First, using the name “Oslo” to talk about a new version of Biztalk, a new tool, a new workflow engine, a new … really confused customers.  Second, it was possible for us to roll out a bunch of “Oslo” technologies in already established ship vehicles rather than creating a separate “Oslo” wave.

Based on this, we made two decisions.  We started using the term “Oslo” for only the the modeling platform pieces of the overall vision.  In addition, we would roll out a bunch of technologies in the .NET 4.0 wave.  So when you hear about things like WF 4.0, WCF 4.0,  “Dublin”, MEF, the unified XAML stack – all of those things were part of “Oslo” at some stage.

I feel good about our ability to deliver solid value to customers as soon as we can, while at the same time continuing to develop new innovative technologies (“M”, “Quadrant”, etc.) to make a greater impact in the future.

The only thing that I feel bad about is that we kept the “Oslo” name around so long (you will see that change at the next PDC), which has continued to be a confusing point for customers (“I thought Oslo was your new SOA platform”).

Now that we are all caught up, I want to give a short update on what is going on with “Oslo” (the modeling platform) today.

During the 10 months since the last PDC, it has become increasingly clear to us that the modeling platform is aligned in a deep and fundamental way with the data programmability stack (ADO.NET, EF/EDM, Astoria, etc.).

Why?

The fundamental focal point of “Oslo” has always been the notion of (meta)data stored within SQL Server or another database.  If you look at the Repository, it has always been “just a SQL Server database” containing application metadata.  Likewise, “M” and “Quadrant” having their roots in making this particular database easier to use.

With this in mind, we made a decision to merge the Data Programmability team (EDM, EF, Astoria, XML, ADO.NET, and tools/designers) and the “Oslo” team (“Quadrant”, Repository, “M”) together.

What does this mean for you (.NET developers)?  You are going hear more about how “M”/EF/EDM align.  How our VS tools relate to “Quadrant”.  How this notion of “model-driven software” evolves with the existing .NET FX investments.

More to be reveled at PDC.  Speaking of which, note the Data Programming and Modeling for the Microsoft .NET Developer session with Don and Chris as speakers.  See you there.

Jeff Pinkston is the GPM of the Modeling Languages team.

We call him Pinky.

http://tinyfinger.blogspot.com/

I really want Jeff to be the ‘voice of the Microsoft “M” implementation’ in the community and blogosphere.

Looks like he is taking me up on the challenge.

If you are paying attention to “M” language specification, you will notice some very interesting additions in the latest draft, particularly if you are someone that likes to add the “meta” prefix to your words.

In short, DLan included the language model and the infoof operator in the spec.

The January CTP doesn’t do any of this yet, but the bits in our tree have had it for sometime and it should make it into the next CTP.

For those of you of a practical bent, this lets you do things like the below (how we would use something like this in our tooling is fairly obvious).

module Test {

    import Language.Catalog;

    type Person {
        Name : Text;
        Age : Integer;
    }

    People : Person*;

    Children() { People where Age < 18 }

    Documentation : {
        About : Language.Catalog.Declaration,
        Description : Text
    }*;

    Documentation {
    {
        About = infoof(Person).Declaration,
        Description = "Describes people"
    },
    {
        About = infoof(People).Declaration,
        Description = "Contains people"
    },
    {
        About = infoof(Children).Declaration,
        Description = "Extracts young people"
    },
  }
}

As part of rolling out the “M” Specification Community, DLan published the draft of the language spec that marries MSchema and MGrammar together.

MGraph was always in the MSchema spec, so we now have a unified language specification for “M”.

Read it here

We used the terms MSchema, MGrammar and MGraph at PDC to help people understand what we had in the bits at the time, but we have always known that we will have a single language.

That is one of the downsides of engaging in this effort so early with the community, but we judged that being more open and less buttoned up than we normally like was the right trade off.

Likewise, the language specification (even the most recent version) needs a lot of work, but we are blessed with a number of seasoned language spec authors both on my team and externally, so we should land in a fine place.

We have two more milestones before PDC and the Microsoft implementation of the language is converging to a point where we have “One M”.

We have a bunch of new things that we will reveal at PDC (with some peeks in CTPs prior) and the unified specification is evidence of our progress on that journey.

Continuing the conversation with Jean-Jacques.

So if it is not too much to ask, could please clarify the M3 layer of Oslo or, possibly what are the extensible mechanisms you provide at the M3 level to support this kind of concept.

Our M3 (M4M) doesn’t make assertions about implementation.

We do make assertions about our data model at this layer however.

This is layer is, of course, extensible via annotations, specialization, and instantiation.

I have to think more about it, but I think we could make the same claim as MOF (“[s]uffice it to say MOF 2.0 with its reflection model can be used with as few as 2 levels and as many levels as users define”), as we just added a new reflection model to “M” (which will be part of the next CTP, I believe).

DLan can answer that definitively.

What you outline “service implemented as orchestration with one implementation that integrates N operations” is a foundational scenario for what we call the services domain. 

If you download and install the SDK, you can see some early work in that that domain (in the models folder).

I talked to Keith Short the other day and he is going to join this conversation to talk concretely on how we are using “M” to model software artifacts and systems.

As you may know, Keith is is very passionate and knowledgeable about employing the technologies like “M” to model the exact sorts of systems you are interested in.

BTW: I absolutely love conversations like this.  We have started talking about Oslo with the community (customers, partners and competitors) very early to ensure that we are getting the right level of feedback and involvement.  In this day and age, developing something as broad as this requires the involvement of everyone you can muster.  The only constant for us is a goal (make it easier to design, build, and manage applications) and a set of principles around approach (data-driven, dynamic, transparent, etc.).

Charles Young hits the nail on the head in his comments over on Jean-Jacques’ blog.

http://www.ebpml.org/blog/181.htm#IDComment18452505

They are split up over a host of comment blocks, but I could not agree with him more.

“Oslo” is a platform for building and interacting with models (fancy term for data, metadata, and metametadata).

There are a host of applications of this technology.

Writing cloud applications is one.

Writing REST Services is one. 

Writing a Web application hosted in IIS is another. 

Writing down an application architecture independent of the implementation is another.

In fact, the use of this approach is not limited to design, build and managing software systems and applications.

The thing I am really excite about is what sort of things are going to emerge from third-party developers and users when these capabilities are broadly available as part of the platform.

JJD responded.  I’ll do the same.

Productivity

I think we are really talking past each other here.  “Oslo” is just as valuable as an approach in the cloud as on-premise.  In fact, the nice thing about the cloud is that most of the management and some of the application infrastructure is already model-driven, so it makes the adoption easier.

The thing I find most interesting is that you think we are not being ambitious enough.  I think that is the first time that claim as has leveled in our direction.  I don’t know quite how to respond, as our team is often considered much too ambitious.

DSLs

The distinction between vDSL and tDSL is purely stylistic and frankly totally uninteresting.

Perhaps not to you, but this is key value to many customers.  The ability for a user, architect, developer or administrator to view/edit the same data via N pluggable and easily creatable textual or visual projections is a important capability of the system.  How much of an application is actually just providing different representations and transforms over the same data?  In my experience a great deal.  Modulo validation and workflow logic, it may be the big spend across a given application.

If you think you need all this machinery to CRUD or to convert a 3-line code snippet into an HTTP request, that’s quite overkill.

The examples I used at MIX were to make concrete to developers writing REST services some of the benefits of this approach in terms of developer productivity.  I didn’t show how easy it was the manage the service.  I didn’t show how easy it was to design the service, etc.  That was not the goal for this audience.  That said, I think you are seriously downplaying the productivity win for a developer writing and consuming these services.  Go watch the demo and then trying writing the same client and service in C# using the existing frameworks.  It is still early in the development of this particular domain and we have a lot more to flush out, but the results to date are promising.

Which product more than Oslo could afford not to deal with this heterogeneity.

“Oslo” is all about embracing heterogeneity.  We assuming nothing about the underlying runtimes.  In addition, we are doing more to encourage cross-vendor and cross-platform involvement in this approach than anything I have seen, certainly on the language front.  See the “M” Specification Community.

BTW, I am sorry, but I don’t see any hi-REST in the samples that you are providing. Hi-REST involves HATEOAS at a minimum and there is no evidence of HATEOAS in anything that I have seen.

Don defined the term Hi-REST back in 2006.  It did not include HATEOAS.  That said, I am just going to side-step this whole thing – MService is an application of “M” to define HTTP services.

This leads me to a very important point.  MService is actually just “M”.  The thing that provides the “HTTP-ness” is the MService runtime.  The MService runtime is a thin “adapter” over the .Net Framework that binds the data description of the service definition written in “M” to the machinery that does the actual computation.  This is the same architecture as what we use in WF and WCF.  In those frameworks, there is a programming model (CLR types or XAML) that builds an in-memory tree (ServiceDescription in the WCF case and WorkflowDescription in the WF case) that is evaluated by a runtime, respectively.

“Oslo” simply provides languages, tools, and runtimes to make the creation and usage of these programming models and resulting data descriptions more tractable to implementers and certainly more appealing and usable to users (be that end-users, architects, developers or ITPros).  “Oslo” is the logical extension of our work with CLR attributes and XAML.

With that in mind, we need to separate the platform aspect of “Oslo”:  languages, tools, and runtimes for building programming models/data descriptions from the “domains” (Web, Service, Database, …) that leverage these capabilities.

Strictly speaking, “Oslo” says nothing about a particular domain (how to write a service, how to build a Web applications, etc.) will work, save for the domain of writing domains (I hope that is meta enough for you ).

What “Oslo” does do is embrace an approach, provide mechanisms to do enable this approach, and then we work with teams all over Microsoft to get them on board.

Stitching

If you provide a good transformation framework to go from the programming model to these concrete models maybe this approach could yield some benefits. I would like to suggest that it would be better to focus on a "DSL connector framework" that would allow you to deploy a programming model defined and managed by Oslo into a variety of architecture layers. I spoke with Nikhil yesterday who mentioned the "metadata pipeline" in .Net RIA services, I think that this is a very important concept and ultimately you could see some integration points between the pipeline and Oslo.

One of the important aspects of our approach is that runtimes read the data description directly from either the database or some other persistent store (a file for instance).  That is to say, we do not do code generation.  We have found that approach to be very limiting toward our goals and that it ultimately leads to “drift” between the different layers of the system.  That is not to say that you couldn’t do it in our system, but it is not the center of our approach and we try to actively look for ways not to do it.  I need to be careful here, as we do generate SQL and XAML, but that is a one-way transformation, just like a typically compiler, rather than like a round-tripping modeling framework.

I have spoke to Nikhil and others about the metadata pipeline in Alexandria.  The ideal thing for me would be that Alexandria uses “M” for the definition of the business domain, inclusive of business constraints, etc. rather than XAML or CLR attributes.  In addition, there is a huge opportunity for totality of the programming model itself to be described using “Oslo” – and at a much higher level (as you no doubt want).  We have the same set of conversations with lots of other teams in the company.

I was just reading Where is Oslo going? over at ebPML.org.

I normally try to send email when I see posts like these (I have an open email thread with Ayende and Alex, for example) as having a blog conversation is hard and most of the time there is some sort of miscommunication involved, but I think I am going to just start replying with posts, so everyone gets the data.

So let’s talk first on the approach that the Oslo team is taking to drive its direction and requirements. If I understand it correctly, they are using "FTA". … IMHO, it is not a good start.

We are not using FTA to drive anything.  FTA was the name of an app that was being built on top of “Oslo” back in the days when “Oslo” == Modeling + WF 4 + what is now Dublin + what is now .Net Services.  I should get Adam to take that blog down.

In case you have not noticed MOP has already happened, all the annotations in Java or C# is MOP layered on top of OO. But MOP layered on top of OO does not provide a clean separation between Architecture and Programming Models.

I couldn’t agree more.  This is the reason that we are doing “Oslo” in the first place.  Having to use CLR attributes in WCF was a painful thing for many us and “Oslo” is largely a logical extension of our work in the space.

It is probably Doug Purdy who is answering these questions.

Nope, but I will take credit for the blah blah blah.

But frankly I am a bit scared by this sentence [“Oslo” will bring together a connected view of today’s models, which are often built in vertical, isolated silos.], MOP is not about stitching together the programming models across the layers of an architecture, MOP is the other way around, it is about creating and deploying a unified programming model onto all the layers of an architecture (whatever this architecture is, SOA, WOA, EDA…).

You are misunderstanding this sentence and it you are forgetting about an important aspect of this technology: managing the application.  Applications are in silos today (how do you think about the apps on a box when you are managing them?), the applications themselves are composed of different silos (the presentation silo, the middle-tier silo, the data silo, etc.).  Our hope is to make it easier to design, develop and manage the applications across all of these silos.  In the limit, we will have a unified model for all aspects of the application.  Pragmatically, there are going to be N models (legacy silos, silos that are “opaque”, etc.) and we want to have some level of support for all of these.

Oslo can’t eat its own dog food: "EF and EDM are important technologies for Microsoft. “Oslo” full embraces EF/EDM as a primary mechanism for “Oslo”-based runtimes to access the repository. "

What makes you think this means we don’t eat our own dogfood?  We absolutely do.  This is statement around how .Net developers can access the models in a repository (which is just a database).

Doug Purdy touts 10X improvement as a great achievement, he obviously has not done any homework on what IT needs and Cloud Computing provides today in terms of a programming model that abstracts architecture and provides already 10X or more

I’ll ask for the data backing this claim.  I don’t think I am the one that hasn’t done their homework.

"M lets developers build out domain-specific languages (DSLs) relatively easily" is no longer the problem, Microsoft has DSL tools for that, the question that the Oslo team should ask itself is does M stands for Model or Metamodel?

The current DSL tools (the DSL Toolkit) are for visual DSLs, not for textual DSLs.  We want an architecture that supports both both textual and visual DSLs operating over the same model.  Speaking of model, M is for model and metamodel and metametamodel.  Since my Smalltalk days, I have grown tired of using the meta prefix, as it often confuses more than helps the conversation with developers.

It seems that the Oslo team is heavily (and emotionally) involved in REST, actually let me rephrase that, involved in both lo-REST and CRUD-REST. That’s pretty pathetic for a team that think they are working on models.  One of the key foundation of MDE is precisely to avoid CRUDing at the programming level.

We are involved (emotionally and otherwise) in anything that works for our customers.  We did this little thing called WS-*, which has model written all over it.  Some customers love that.  Some customers love CRUDing over their models.  It turns out that MService was geared to Web developers that just want do something simple like Rails.  That doesn’t mean we don’t love and support all the other ways of doing building services.  BTW:  We do Hi-REST in MService.

…completely missing the mark on MOP.

I like many aspects of your MOP formulation.  I do not like the explicit transformation, but otherwise it seems reasonably sound at a high-level after skimming it.  The example you give of WCF attributes is actually one of the key driving factors of our work.

At MIX09, we announced the formation of the “M” Specification Community.

You can read all about it at http://www.douglaspurdy.com/2009/03/20/m-specification-community/.

We have just started working on our first substantial issue: Unicode support.

The mailing list is open to the public, so you can monitor the spec development process.

To subscribe to the mailing list, send an email message to listserv@discussms.hosting.lsoft.com with the following body content:

SUBSCRIBE MSPECCOMMUNITY <insert email address> <insert first and last name>

In order to post, you need to agree to the Participation Agreement.

[Update:  If you want to use a feed reader to monitor the mailing list:  http://discussms.hosting.lsoft.com/SCRIPTS/WA-MSD.EXE?RSS&L=MSPECCOMMUNITY&v=2.0]

Prereading: Part I, Part II

Based on all the conversations I have had internally and externally, I wanted to get two additional pieces of data out about “M”.

First, we didn’t have time to show this at MIX and this is still in “being thought a lot about state”, so expect it to change.

Invoke is used to call extern code in CLR and SQL runtimes.

Second, when we talk about a repository – it is just a SQL database with a couple of additional tables in it called the “catalog”.

The catalog stores the M4M models as well as the M2SQL information.

What that means is that you can turn any database into one that supports “M” or you can create new repositories as easily as you create a new database in SQL.

No fuss, no muss.

I have had more than one of my followers on Twitter send me email or tell me in person that they love reading what I share from Google Reader, but they can’t stand going from Twitter to FriendFeed and then to the link.

You have to keep your audience happy, so I wrote rdr2twt. The below shows the key logic and I have attached the source below. Of course, I use MUrl for most of this (I found a bug – one of many I need to fix – in the Uri syntax – no support for ‘~’ – which I fixed in the grammar in this project).

static void Main(string[] args)
{
    ServicePointManager.Expect100Continue = false;

    var formatter = new Atom10FeedFormatter();

    formatter.ReadFrom(XmlReader.Create("your-shared-items-feed"));

    var feed = formatter.Feed;

    var lastUpdate = GetLastUpdate();

    var items = from item in feed.Items
                where Compare(item, lastUpdate)
                select item;

    foreach (SyndicationItem item in items)
    {
        Tweet(item);
    }

    if (items.Count() > 0)
    {
        SetLastUpdate(GetTimestamp(items.ElementAt(0)));
    }
}

Source: rdr2twt.zip

Kris “The real deal” Horrocks is blogging now.  His second post is quite good.

http://blogs.msdn.com/krisho/archive/2009/03/24/oslo-is-running-my-house.aspx

During our MIX09 talk, Chris and I showed how to take an instance of a “M” language (in this case MUrl) and transform the resulting M values into XML, SQL, and in-memory CLR objects.

The attached zip contains all the things you need (along with the Jan. CTP) to do that same thing:

• Create an image file from M language definition (CompileGrammar.cmd)
• Transform an instance of that language to XML (CompileXml.cmd)
• Transform an instance of that language to SQL (CompileM.cmd & LoadIntoDatabase.cmd)
• Transform an instance of that language to a in-memory graph (MUrlRuntime folder)

Before you ask, yes, we are looking at strongly-typed mappings to CLR types, although there are some interesting things we may be able to do with dynamic.

We are still thinking hard about that.

A good intuition about how we see this building out is to look at System.XML — in particular the reader/writer, document, and serializer support.

M to N Sample

The video for our MIX09 talk was just posted at http://videos.visitmix.com/MIX09/T11F.

You can see the demos live with some color commentary that may make things a little more clear.

From what I can recall of the session, the interaction between Chris and I was good.

There is always a danger when you do two speakers that the interplay will obscure the material.

I think we managed the balance fine, but I am open to feedback.

If you have questions about the session, comment here or send me email at douglasp@microsoft.com.

One thing I want to make clear, especially to people new to "M", the MService example I showed in my first post uses a database to store all values.

In fact the exact SQL for the People extent is below.

There is no O/R mapping to write.  There is no O/X mapping to write (notice that we are using “M” on the wire).

As I said previously, our hope is that we can build a language and associated set of runtimes that we can use to ensure that application developers can focus on the domain.

To make that concrete:  less time on mappings/transformations and repeating validation rules – more time on business problems and (hopefully) your friends/family.

This morning we announced the formation of the “M” Specification Community (MSC) at MIX09.

The MSC is a group of individuals and organizations throughout the software industry working together to develop the “M” language specification.

Our goals with the MSC are two-fold: 

1. Ensure the broadest possible support for “M” with a number of “M” implementations across multiple platforms and excellent interoperability with existing standards/approaches.
2. Ensure that we have the best possible language that can span the presentation, services and data domains and is well informed by the XML, modeling, and Web communities.

The MSC is a key part of our broader strategy to move the industry toward model-driven development and unlocking the value that this approach provides to our customers and partners.

This community will function primarily through a public discussion list, just like other public specification efforts, with the goal of a good interchange of ideas that will lead us to the best possible language with the broadest industry support.

The MSC has already shown a great deal of promise toward our goals as two key open-source technical leaders will serve as “charter members” of the community:  James Clark (Technical Lead of the XML WG and Editor of the XSLT 1.0 Specification) and Ed Merks (Technical Lead of the Eclipse Modeling Framework and PMC (Project Management Committee) Lead for the top level Eclipse Modeling Project).

You can find more details about MSC on the “Oslo” Developer Center (http://msdn.microsoft.com/oslo/msc).

We made an explicit decision to be “low-key” about the formation of this community.

We want it to be about getting the best technical solution and the benefits that comes with that focus.

If you are interested in helping out, please join us.

This is the second of three posts about our MIX talk/announcements.

In addition to “MUrl” (which works on the CTP bits today), we showed off an early look at “MService”.

This is the the next rev of the “MService” prototype that we showed at PDC.

The key and very important difference is that “MService” is no longer an external DSL like what we showed at PDC.

Rather “MService” is just “M” that a runtime uses to provide RESTful services.

Let’s take a look at how it works.

We start off with a simple domain (Person).  This, of course, could be as complex as you want (have a look at the models.mproj in the SDK), but I want keep it small.

We then add two Get functions; one that yields all the values of People and the other than returns a specific value.

So far this is all “M” that you would see in the CTP, but now we are going to do something new.

The below shows an early look into some new  “M” language features we expect to have in place in a future CTP.

Don’t get wrapped up in the keywords, etc. (although they are quite nice), the key thing is that you will be able to do this in “M”.

As you see, we now have described a complete, albeit trivial, RESTful service.

All we need to do is save this file in a vroot with the “MService” runtime installed.

That let’s us do things like this.

and this

One of the cool things “MService” supports is OPTIONS to get metadata.  We use the new infoof keyword for that.

That allows a client to get all the metadata for the service.

That let’s do interesting things like build this test form.

 and even flow validation information to the browser client

You can fully expect for “MService” to leverage the full power of all the .Net runtimes we have available today, include WCF, ADO.Net Data Services, ASP.Net, etc.

In fact, we call “M” and all “M” based languages — .Net DSLs.

The key thing we are doing is letting the developer stay focused on their domain and then the runtime behavior emerges from the domain description.

Lastly, I want to be clear that this is all early thinking and this could and likely will change as we learn more (just as “MService” has changed between PDC and MIX).

One of the things we are explicitly doing with “M” and “Oslo” is engaging the community very early and very openly to work together on this technology.

Our goal is to greatly simplify how people design, develop and manage applications.

We need your help to do it.

This is the first of three posts about our MIX talk/announcements.

Today we showed off “MUrl”, which is a DSL and runtime for interacting with HTTP services.

We think “MUrl” has a lot of potential and we are still working on it, but we didn’t want that to stop you from getting your hands on it.

As such, you can download the “MUrl” source code today from the “Oslo” Developer Center.

We are doing this for three reasons:  we want you to have something that will make your life a little easier, we want to provide concrete proof of how “M” can be used to make your life easier, and we want to show how you can write your own “M”-based DSLs and runtimes.

I will be clear that “MUrl” is a work in progress and you are going to find things that don’t work the way you want/expect.  You do have the source code however.  Enjoy.