WSE 3.0 RTM

WSE 3.0 will be available on MSDN for download this Monday, 7th November to coincide with the Visual Studio 2005 launch. This also includes two completely re-written Hands on Labs, a swath of samples in C# and VB along with updated whitepapers.

Congratulations to the team!

UsernameToken Security

Matt Powell links to an article by Keith Brown on UsernameToken security. Worth reading.

WSE 2.0 Hands On Lab Updated

The Hands On Lab materials for WSE have been updated for the WSE 2.0 SP2 release and now include VB.NET code as well as C#. If you're looking to get started with WSE 2.0, these provide some great grounding material.

WSE 2.0 SP2 Golden

Service Pack 2 is final and available on MSDN.

There was one additional fix between the Pre-Release and the final build involving policy processing for clients that do not send the WS-Addressing headers. In this case, request policy was being applied correctly but response policy would not be applied.

Please report any issues you find on the product to wsefeed.

WSE 2.0 on .NET FX 2.0 - Update

.NET Framework 2.0 is still a moving target for the WSE team but things have stabilized a lot recently. We made a few tweaks inside the WSE 2.0 SP2 PreRelease plus got a few bugs fixed in the framework itself. The net result is that we believe that WSE 2.0 SP2 will run correctly on .NET Framework 2.0. Note that this only applies to 32bit mode - there's no support in WSE2 for 64bit execution.

WSE 2.0 SP2 PreRelease

A pre-release builds of WSE 2.0 Service Pack 2 is now available. The download links are:

• WSE 2.0 SP2 PreRelease Full Product


• WSE 2.0 SP2 PreRelease Runtime Product

Here's the content from the readme file:

Core product changes:

• A new compatibility section is used to select the wire format on the sending
  side. The mode attribute tells WSE runtime to generate a message which will be
  compatibable to a particular release of WSE. By default, the mode is WSE2RTM. It can be WSE2RTM, WSE2SP1 WSE2SP2 and so on. On the receiving end, a particular version of WSE runtime will be able to accept all types of wire format in all its previous releases. In a request-response message exchange, a server will generate a response message which is compatible with the request message. The server will, by default, still use the compatibility section to generate its response if it cannot determine the compatibility mode based on the request message.
  
• A new implementation of the Kerberos token based on SSPI interfaces is included in this release. The new token is named KerberosToken2. Please see the reference documentation for more details.
  
• The TokenIssuer element in the KerberosToken security token assertion is
  no longer supported on the receiving end.
  
• The SecurityTokenManager no longer throws exception in LoadTokenFromXml() when token.IsCurrent returns false. WSE security input filter will continue checking that and throw exception if token.IsCurrent returns false.
  
• The inclusion of unencrypted Username tokens in a message may represent a security vulnerability. The SecurityTokenServiceClient class will now automatically encrypt any Username tokens included in a request. Similarly, the SecurityTokenService class will automatically encrypt any username tokens included in a response.

  The following methods have been added to the token issuing framework:

  protected virtual void SecurityTokenServiceClient.EnforceRequestUsernameTokenEncryption().
  Called from EnforceRequestPolicy(), this method enforces the requirement that any Username tokens in an RST message must be encrypted. The issuerToken is used as the encrypting token. This method will throw an exception if it cannot encrypt the UsernameTokens in an RST message.
  Override this method to suppress this behavior.

  protected virtual void SecurityTokenService.VerifyRequestUsernameTokenEncryption().
  Called from VerifyRequestPolicy(), this method verifies that tokens in an RST message are encrypted. This method will throw an exception if it encounters an unencrypted UsernameToken in an RST message.
  Override this method to suppress this test.

  protected virtual void SecurityTokenService.EnforceResponseUsernameTokenEncryption().
  Called from EnforceResponsePolicy(), this method enforces the requirement that any Username tokens in an RSTR message must be encrypted. The ResponseEncryptingToken is used as the encrypting token. This method will throw an exception if it cannot encrypt the UsernameTokens in an RSTR message.
  Override this method to suppress this behavior.

  protected virtual void SecurityTokenServiceClient.VerifyResponseUsernameTokenEncryption().
  Called from VerifyResponsePolicy(), this method verifies that tokens in an RSTR message are encrypted. This method will throw an exception if it encounters an unencrypted UsernameToken in an RSTR message.
  Override this method to suppress this test.
  

• A new method, protected virtual void SecurityTokenServiceClient.ClearRequestSoapContext(), has been added. This method was added to fix a bug in which successive requests for a SecurityContextToken, made through a single instance of SecurityContextTokenClient, would fail. The problem was caused by a failure to clear security elements and tokens from the soap context after a request was made. In the event that a sub-class of SecurityTokenServiceClient requires that security elements or tokens in the soap context be preserved from one request to another, the new behavior may be suppressed by overriding the ClearRequestSoapContext method.
  
• SoapService will now send back an empty SoapEnvelope back if the soap method it is invoking returns null for a request/response scenario.
  
• EncryptedData.Decrypt will only support decryption to one and only one xml element. It throws a security fault otherwise.
  
• In the SoapHttpOutputChannel.Send method, if the response has an unsupported content type, such as text/html, the response stream will be read and stored in the exception text.
  
• Two new properties, SimpleDisplayName and FriendlyDisplayName, have been added to the X509Certificate class.
  
• The default Label used in DerivedKeyToken has changed from "WS-SecureConversation" to "WS-SecureConversationWS-SecureConversation".
  
• If an incoming message contains multiple security tokens with envelope signature inside those tokens, the server was returning a security fault. This is now fixed.
  
• If WSE SOAP messaging stack is used over HTTP/HTTPS transport, a simple WSE805 exception was thrown when the response content type was not supported. Now if the response stream is readable, WSE runtime will read from response stream and throw a WSE805 exception with a more detailed error information read from the response stream.
  
• WSE configuration section will now allow whitespace or comments as child nodes for the following configuration elements: diagnostics/trace, diagnostics/policyTrace, diagnostics/detailedErrors, referralCache/cache, security/x509, security/limits.
  
• When a security context token was deserialized, WSE runtime will retrieve a token from its cache based on its globally unique Identifier. The token retrieved from the cache sometimes have a different Id than the token received from the incoming message. If that happens, WSE runtime would previously fail to verify a signature or decrypt the message. It is now fixed as it will assign the Id of the token from the incoming message to the token retrieved from the cache.
  
• The built-in SecurityContextToken service would previously cache a newly issued security context token instance before those properties defined in the IIssuedToken interface are set. Now it is fixed so that those properties are set before the newly issued token is cached.
  
• WSE runtime would previouly always generate a relative token reference in calculating message signature. When the token is not sent with the message, the receiver will return a security fault. It is now fixed so that an absolute token reference will be generated in the case when the security token is not in the message.
  
• A server fault was thrown previously when a server tries to verify a signature or decrypt a message based on a username token which uses plaintext password option or no password option and does not have nonce and/or created element. The exception is now changed to be a client fault.
  
• SoapService instances configured to automatically issue SecurityContextToken (SCT) security tokens no longer re-use the same instance of the SCT issuer. With this service pack, a new instance is created per request. This instance is accessible via the SoapService.AutoIssueSCTService property. If the old behavior
  is required, there are two work-arounds:
  
  
  
  1. override the SoapService.AutoIssueSCTService property to return a singleton instance of a SecurityContextTokenService, or
     
  
  
  2. create a wrapper SecurityContextTokenService that delegates to a singleton instance and register it through configuration.
     
  
  
  
  

Visual Studio tool integration changes:

• The default value for http Routing handler type is updated.
  
• WSE setting tool can support VS C++ project.
  
• WSE setting tool would always prompt users for confirmation when the Cancel button was clicked. Now it is fixed so that the tool will prompt user only when some changes are made by users.
  
• The Security Settings Wizard can support creating Policy files for remote service.
  

WseWsdl2 tool changes:

• The WseWsdl2 tool now properly generates proxy classes when an Web service uses a Guid type.
  
• If an input schema contains a type definition which is derived from another complex type, the proxy class would generate a class definition for that derived type with incorrect namespace. This is now fixed.
  

As always, please report any issues you have with the pre-release to the WSE Feedback mail address.

UsernameToken - SendHashed

Scott Watermasysk posts about UsernameToken hashed passwords and Julia Lerman adds comments in which she says "the database needs to store clear text" although the recommended method is to "store a user's password as a hash, or even better a salted hash" in the database. I've thought about these posts a bit, here are my comments, hopefully they'll be useful for people trying to use these schemes.

First, some basics. The word password, when used with the UsernameToken means password or password equivalent. There is absolutely no requirement that the value you use to construct the UsernameToken at the client is exactly what the user typed in. The only absolute requirement is that the value used in the constructor at the client matches the value returned by your UsernameToken.AuthenticateToken method. Thus the simplest case where you want to use plaintext passwords in the database and directly compare the value in the token to that in the database.

Now, the UsernameToken has three options for password management associated with it, each of which are defined in the OASIS WSS UsernameToken Profile:

1) SendNone. This is obvious, the password is never sent on the wire with the token. Not a lot of use unless you're just authenticating with usernames, clearly that's not terribly secure.

2) SendPlainText. Again, this is hopefully obvious and maps to the wsse:PasswordText option in the token profile. When you use this option the plaintext form of the password supplied on the UsernameToken constructor is included in the token on the wire. Clearly, using this option presents some security issues: either the token itself must be encrypted for the target service or the whole message exchange must occur on a secure channel, for example SSL.

3) SendHashed. Not so obvious. The one thing it isn't is a simple hash of the password provided on the constructor. Instead it maps to the wsse:PasswordDigest option in the token profile. This hashes 3 values from the UsernameToken instance together: created time, nonce and password. This process binds the password value to other elements in the UsernameToken instance: every time the UsernameToken is used to send a message, the created time and nonce are updated and the hash value is recomputed. Maybe we'll rename this option to "SendDigest" in the next release.

Now back to Scott and Julias posts, specifically storing either password hashes or salted hashes in the database. These are indeed good models for storing passwords at the server since the password values themselves are not actually retained. However, neither a plain hash of a password or a salted hash of a password in these models is the same as the SendHashed option of the UsernameToken.

Let's consider the two server password models separately and see what will work if you want to use SendHashed.

First, the server stores a simple hash of the password, for example SHA-1. The client knows (a) the plain password (b) the nonce (c) the created time. In contrast the server knows (a) the hashed password (b) the nonce (c) the created time. Using SendHashed alone just won't work since it's not the same hash as the server uses and the UsernameToken.AuthenticateToken method cannot return the same password value that the client used on the constructor. However, if the client hashes the password before constructing the UsernameToken then both sides know the same thing - the hashed password equivalent.

Second, the salted hash model. The client knows (a) the plain password (b) the nonce (c) the created time. In contrast the server knows (a) the hashed password (b) the nonce (c) the created time and (d) the salt. Note that the client doesn't know the salt and thus hashing the password before passing it to the UsernameToken constructor won't work. There's no way in this model for the client and server to have shared knowledge of the same password or password equivalent so SendHashed won't work. The only option with salted hashes is to use SendPlainText as follows: at the server, your modified UsernameToken.AuthenticateToken method takes the supplied password and hashes it with the salt. It then compares the result with the value in the database, if the two match, it returns the password provided in the UsernameToken allowing authentication to succeed.

I hope this sheds a little light on this topic, I'm happy to hear your comments and suggestions if there's something I've missed or something you'd like to see changed in the next version of WSE.

WSE 2.0 on .NET Framework 2.0

The WSE team recently ran a series of test passes with WSE 2.0 on .NET Framework 2.0 Beta 1 (aka Whidbey or VS 2005). We tried both the shipping version of WSE 2.0 and a recompiled version, but experienced a number of problems with both.

We know for sure that the soap.tcp transport is completely broken - incoming connections generally fail with a SocketException during Accept processing. We've also run into a number of problems associated with ASMX classes.

We're working with the relevant parties on the Framework team to try to resolve these issues, we believe that most of them are caused by changes made to the Framework itself, not by problems in the WSE codebase. Of course, .NET Framework 2.0 is still beta software and this is therefore to be expected despite all the effort that goes into trying to ensure backward compatibility early in the development cycle.

Current product plans are that WSE 2.0 will be supported on .NET Framework 2.0 when the Framework releases. Until then, we can't make any guarantees on whether the products will work together.

Sorry folks :-(

What's going on in WSE land?

The plate is full right now for the WSE team, we have three pieces of work underway concurrently keeping everyone on the team busy (along with all those summer vacations, of course).

First up is the C# implementation of the WS-I compliance test tools. I doubt many people know that this is also developed by the WSE team. Right now, the focus is on completing the implementation and testing of various Basic Profile and Basic Security Profile assertions. The BSP assertions right now are limited to examining the message without unravelling the signatures and encryptions; further on down the road we're expecting that we'll be linking to tools to WSE itself in order to perform more detailed verification of secure messages.

Second is WSE 2.0 Service Pack 2. Like SP1, this will include bug fixes for reported problems, so if you have a bug that is causing you problems make sure that you report it either on the WSE public newsgroup or to the feedback e-mail address in the documentation. Unlike SP1, SP2 will also include some new functionality in the form of additions to the built-in tokens and token issuers in the product. One of these will be a replacement for the existing Kerberos token in WSE 2.0 that will more closely track the Kerberos Token Profile being developed by the OASIS WSS Technical Committee. We'd like to see the folks on the TC accelerate the finalization of this profile as we're beginning to see demand for a more interoperable form. Unfortunately, I can't say anything about the other addition at the present time.

Thirdly, we're progressing through the planning milestone for WSE 3.0. We're pretty much locked down on the plan of record for the products features and timing, we have a bunch of design work and some prototyping in progress as well. At the present time, I can't announce the details of the product but we believe that we've struck the right balance in what was a very tangled web of requirements, several of which were incompatible. What I can say is that we plan to adopt a more open approach to delivering public builds of the product for everyone to work with and provide feedback against. Once we have an initial build complete, we're hoping to release incremental builds on a very regular schedule up to the point of final release. This is like the Visual Studio Community Drops, only that, being WSE, we plan to go a little faster! We're adopting some modifications to our existing engineering process that we believe will allow us to meet those goals and execute more smoothly this time around.

WSE 2.0 Service Pack 1 - Update

Several people have asked how soon this will be released. It should have been early last week but an interoperability problem came to our attention and needed to be investigated prior to release of the service pack. That investigation is now complete and no changes were needed to the service pack; as a result, it should appear within next few days. Now I can go back to my vacation... :-)

WSE 2.0 Service Pack 1

Service Pack 1 will be released very shortly. We've tried to address the major pain points that people found after the product was released. Here's a list of the changes from the readme file:

Core product changes:

• WSE no longer throws an exception due to a SOAP message being signed by a KerberosToken security token in a time zone east of GMT.
  
• The value of the maxTokens attribute for the <limits> element specifies the maximum number of security tokens that may be contained within an incoming SOAP message. Previously, only natively supported security tokens counted towards the limit.
  
• All members of the ElementList class now use by value parameters. Previously, the Remove and IndexOf methods used by reference parameters.
  
• When a SOAP request does not contain a SOAPAction HTTP header, WSE throws an Microsoft.Web.Services2.Addresssing.AddressingFormatException. Previously, a System.NullReferenceException was thrown.
  
• WSE now generates WSDL files with unique operations when a Web service is built using the WSE Messaging API and it contains multiple Web service methods with identical names. Previously, WSE generated an incorrect WSDL document for this scenario.
  
• WSE no longer throws an exception during WSDL generation for Web
  services built using the WSE Messaging API that contain a method with an System.Xml.XmlElement parameter.
  
• The GetBodyObject method of the Microsoft.Web.Services2.SoapEnvelope class now supports System.Data.DataSet objects.
  
• Invalid policy documents that contain multiple <Role> or <SubjectName> claims in a <SecurityToken> assertion will now fail to load.
  
• SecurityContextToken and UsernameToken security tokens with the same key as a cached security token can now be cached using the CacheSecurityToken method. Previously, the security token would not be cached.
  
• A new protected virtual method, named RaiseDispatchFailed, has been added to the Microsoft.Web.Services2.Messaging.SoapTransport class. Any class derived from SoapTransport can raise the Microsoft.Web.Services2.Messaging.SoapTransport.DispatchFailed event by calling this method.
  
• A new flag named IncludeReferenceProperties has been added to the Microsoft.Web.Services2.Security.SignatureOptions enumeration. This flag can be used to specify that all elements in the SoapContext.Addressing.Destination.ReferenceProperties.AnyElements collection are signed. This flag is only valid for outgoing SOAP messages.
  
• When dispatching a SOAP message based on an Microsoft.Web.Services2.Addressing.EndpointReference, the wsu:Id, SOAP actor and mustUnderstand attributes are now ignored.
  
• While retrieving an x.509 certificate, WSE is now capable of using an RFC3280 compliant algorithm to generate a key identifier for X.509 certificates that do not have the Key Identifier extension. To enable this algorithm, add an <x509> element to the receiving application's configuration file and set the useRFC3280 attribute to "true". The default value for the useRFC3280 attribute, which is new for SP1, is "false". Setting the attribute to "false" specifies that WSE uses an algorithm that is not RFC3280 compliant, as it did prior to SP1.
  
• After processing the SOAP envelope through all the input filters, WSE runtime will remove the SOAP header element if it no longer contains any child elements.
  
• When a security token request does not have an Microsoft.Web.Services2.Policy.AppliesTo (RequestSecurityToken.AppliesTo is null), WSE no longer adds an AppliesTo. Previously, WSE would default the destination to the URL for the security token service.
  

Visual Studio tool integration changes:

• The type attribute for settings on security token managers and binary security token managers is now an optional setting in the WSE Settings 2.0 tool. when the type attribute is not specified, the built-in security token managers are used, such as Microsoft.Web.Services2.Security.Tokens.X509SecurityTokenManager. Previously, it was required.
  
• The Time Tolerance in Seconds setting on the Security tab of the WSE Settings 2.0 tool now updates the <ttlInSeconds> element in the application's configuration file, even if one already exists. Previously, the value was only inserted into the <ttlInSeconds> element when the element was not currently in the configuration file.
  
• The X509Certificates... link on the TokenIssuing tab of the WSE Settings 2.0 tool now opens the certificate store location specified on the Security tab. Previously, it would always open the CurrentUser store.
  
• The Security Settings Wizard can now be used on Microsoft Windows 2000 to select, add, and view X.509 certificates. Previously, the Select Certificate, Add Certificate, and View Certificate buttons were not functional on Windows 2000.
  
• A new check box named Use RFC3280 has been added to the Security tab in the WSE Settings 2.0 tool that enables you to specify that WSE generates RFC3280 compliant key identifiers when an X.509 certificate does not contain the Key Identifier extension. When set, the Policy Wizard generates RFC3280 compliant PKI too.
  

WseWsdl2 tool changes:

• The WseWsdl2 tool now properly generates proxy classes when an Web service exposes multiple operations with the same name but with unique action names. The tool will now create a proxy correctly for this scenario by setting the new UniqueMethodName property of the SoapMethodAttribute class to a unique name.
  
• Proxy classes will now be generated for services containing relative URI values for SOAP action names.
  
• Generated proxy classes for Web service methods that use a DataSet type for a parameter or return value, now use the DataSet type as the Web service method. Previously, the proxy class would generate the System.Xml.XmlElement type for DataSet types.
  
• When a SoapService has an SoapActor attribute applied to it, an EndpointReference with a Via reference property is required to send a SOAP message to the Web service. To build a SoapClient that calls a Web service that has an SoapActor attribute applied to it, use the new WseWsdl2 tool command line parameter, name, and specify the logical name that is specified in the Web service's SoapActor attribute. The generated SoapClient proxy class will correctly address the Web service with a fully specified EndpointReference. For an example of how to specify the EndpointReference without using the WseWsdl2 tool, see the TcpSyncStockService QuickStart sample.
  
• When a WSDL contains a schema definition that is registered with an empty namespace and if any of the message part elements contain a Type attribute the tool will no longer throw an exception.
  
• The WseWsdl2 tool can now process WSDL documents that have nested <import> elements. That is, WSDL documents that contain an <import> element that references a WSDL document, which in turn has an <import> element that references another WSDL document, and so forth. To specify the level of recursion that the WseWsdl2 tool will go through to download the imported WSDL documents, two new command line parameters have been added to SP 1: noRecursion and recursionLevel. To specify how many levels of recursion you want the tool to go through, use the recursionLevel followed by an integer representing the number of levels or documents you want the tool to go through. The default is 3. If you do not want the tool to download WSDL documents specified in <import> elements, use the noRecursion command line parameter. If both parameters are used, the noRecursion parameter takes precedence.
  
• Like the WSDL.exe tool, the WseWsdl2 tool now generates methods to communicate with a Web service asynchronously. That is, for every Web service method, a BeginXXX and EndXXX method are generated. The XXX in the name of the method is the name of the synchronous method. Use the BeginXXX method to initiate an asynchronous call and the EndXXX method to complete it.
  

X509 Certificate Tool changes:

• When choosing a certificate, the X509 Certificate tool will also display the RFC3280 compliant PKI for that certificate.
  

DispatchFailed

When trying to understand why your application doesn't seem to be processing the messages you are sending, it can be helpful to detect and monitor message dispatch failures from the WSE2 messaging layer. This is the purpose of the static SoapTransport.DispatchFailed event. You can hook this event anywhere in your application and have your callback invoked when a transport fails to dispatch an incoming message to a registered input channel. Here's the code to do this:

[MTAThread]
static void Main(string[] args)
{
SoapTransport.DispatchFailed +=
new SoapTransport.DispatchFailedEventHandler(OnDispatchFailed);

...

}

static void OnDispatchFailed(object sender, SoapTransport.DispatchFailedEventArgs e)
{
Console.WriteLine("Dispatch Failed!");
}

The sender is the transport that failed to dispatch and the DispatchFailedEventArgs has a property called Message that is the SoapEnvelope that could not be dispatched.

Note that this event is not fired by the built-in HTTP transport that is hosted under ASP.NET. It is fired by soap.tcp and soap.inproc. Custom transports that use the SoapTransport.DispatchMessage method to dispatch incoming messages will also fire this event.

Soap.Udp 0.1

This is the first release of a custom WSE 2.0 transport that I have been writing to support SOAP over UDP, including IPv4 multicast. You can download the source here.

And yes, I know it's not perfect (that's why it's release 0.1), but hopefully it will illustrate some concepts for those that are writing other custom transports.

Note that the source code is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License.

Tracing Confusion

It seems all kinds of people are asking questions about the use of tracing tools with the WSE 2.0 release. Benjamin mentions it and points to an entry from Christoph Schittko that has some more background.

Christoph is correct in noting that WSE 2.0 is restrictive regarding matching the <wsa:To> header with the URL that the request arrives on. This is a security restriction: the policy engine in WSE 2.0 uses the <wsa:To> to select policy - allowing random headers here would allow an attacker to attempt to select different, possibly weaker, policy than the service developer intended.

Christian Weyer has posted one possible workaround as a comment:

EndpointReference epr = new EndpointReference(new Uri("http://localhost/AsymEncryptPolicyService/AsymEncryptService.asmx"));
epr.Via = new Uri("http://localhost:7777/AsymEncryptPolicyService/AsymEncryptService.asmx");
ws.Destination = epr;

This ensures that (a) the <wsa:To> header has the correct value for the final destination (b) the request is routed through the tracing tool.

OK, so now you're asking "Sure, but this workaround means I have to change my code!" That's a fair comment, so here's another solution to try: configure a referralCache at the client and place the appropriate entries in it so that the request is redirected to the tracing tool automatically. For example:

<r:ref xmlns:r="http://schemas.xmlsoap.org/ws/2001/10/referral">
<r:for> <r:exact>http://localhost/AsymEncryptPolicyService/AsymEncryptService.asmx</r:exact>
</r:for>
<r:if>
</r:if>
<r:go> <r:via>http://localhost:7777/AsymEncryptPolicyService/AsymEncryptService.asmx</r:via>
</r:go>
<r:refId>uuid:09233523-345b-4351-b623-5dsf35sgs5d6</r:refId>
</r:ref>

When you're finished tracing, clear the referralCache or remove it from your config file. Check the product documentation for more details on the entry format and configuration of the referralCache itself.

WS-RM in WSE 2.0?

There have been comments in a couple of recent articles (for example, here and here) that state that WSE 2.0 implements WS-ReliableMessaging. Before everyone gets too excited, I have to say that these articles are not correct; WS-ReliableMessaging is not available in the WSE 2.0 release. It is, however, something that we're considering for WSE 3.0.

WSE 2.0 Shipped

Finally. Get it here.

Final WSE 2.0 Bug Bash

Today is the day. Build 2.0.4089.0 is the victim. The leads will triage bugs as they arrive and the development team is going to pair program for the day, whacking bugs as they are triaged. We're hoping we can stay on top of this one.

XP Applied

I am spending some time catching up on Agile Development and Extreme Programming trying to match those practices against our development model for WSE. Faced with a number of rather trivial bugs against WSE 2.0 we tried a practical experiment with Extreme Programming's Pair Programming model this evening.

Rather than have an individual developer fix a number of bugs and then go to another developer for a code review, we sat 2 developers together and told them to spend a couple of hours working on bugs together. They cleared off 15 in 2 hours.

An interesting experiment with excellent results.

Namespaces

In .NET parlance, WSE is a Library, not a Platform. The primary distinguishing characteristic is that between WSE major versions there can be breaking API changes, breaking wire-format changes and there is no guarantee of backward compatibility. Just because WSE 1.0 has a class called Microsoft.Web.Services.SoapContext doesn't guarantee that WSE 2.0 will have that class, that it will have the same interface or that methods with the same name will have the same behaviour.

Given this, how do you deal with the adoption of WSE versions over time? Can different versions co-exist on the same machine? What about the same AppDomain? What happens when that application you built 6 months ago with WSE 1.0 needs to talk to a second service that is built with WSE 2.0? How do you host the same service accessible using different versions of WSE? These are some of the questions we've been investigating as part of our Application Building exercises as we get closer to the release of WSE 2.0.

We had always planned for Side-by-Side existence of different major versions of WSE on the same machine. This is why we moved away from placing configuration data in the machine.config file and instead pushed the model of declaring the WSE config section handler inside an applications config file. However, hosting different versions of WSE inside a single AppDomain takes things a step further. Are there any runtime conflicts? Can you reference both assemblies in the same Visual Studio project? How do you have a single source file that uses classes from both versions?

Hosting both versions in a single AppDomain turned out to be relatively trivial from a technical perspective: there are a couple of conflicts including the configuration section name, trace file names and so on, but our use of other resources such as Thread Local Storage or X509 Certificate Stores were non-issues. Where things got tricky was the user experience.

Visual Studio presently does not allow you to reference two assemblies with the same file name (but different strong names) in the same project. To make things work, we had to encapsulate the code using different WSE versions into separate projects so that each only referenced a single version of WSE. Whilst this works, and some may prefer it as a matter of design, it's not as simple as we'd like: we're still left with the fact that by re-using the namespace across different versions we may be misleading people into assuming that the classes exhibit the same behaviour or that they can be passed between the different versions.

WSE major versions are not simply incremental feature additions, they are different libraries. As such, they deserve different names. The classes should have different names to reflect the fact that they may have different behaviour. The assemblies should have different names to reflect that they are different. Giving the classes different names (by changing the namespace) and renaming the assemblies also gives new options for developers: now you can reference the two (or more) different assemblies from a single Visual Studio project, even from a single class in that project[1].

As a result of these investigations and a 2 hour find/replace game in an editor, WSE 2.0 is now Microsoft.Web.Services2 and the config section is called <microsoft.web.services2>.

This change allows us to enable the following side-by-side scenarios:

• Single server application running multiple services, each accepting requests using a different wire format[2].
• Single machine running multiple server applications, each using a different wire format.
• Single client application sending requests with different wire formats.
• Single service accepting requests over one wire format and sending requests using a different wire format.

We will also ensure that the toolsets from different versions of WSE will co-operate with each other when you use multiple versions in the same project.

Scenarios that we are explicitly not supporting are:

• Single service accepting requests using multiple wire formats - there's no simple way to sniff the messages to determine which version they are using and there are other conflicts from the Soap processing model. The services must have different endpoints.
• Relay accepting messages over one wire format and translating to another wire format. The WSE 2.0 router will not be able to translate messages to WSE 1.0 wire format; the potential deltas in the wire formats make this difficult. Of course, if the incoming message is signed, such a translation would likely break the signature.
• We are not supporting sharing class instances between versions. For example, you cannot host a WSE 1.0 SoapFilter inside the WSE 2.0 Pipeline.

For customers using WSE we think this change gives you greater flexibility in developing and deploying your WSE-enabled applications over time. It also crystalises the fact that different versions of WSE are different libraries and need to be considered as such.

[1]This gets even easier with the Whidbey version of the .NET Framework.

[2]This scenario is not supported if the services are written using ASMX; it is supported for WSE's own Messaging stack. We are not enabling co-existence of the multiple SoapExtensions used in the ASMX case for a single AppDomain, you must use different virtual directories for this scenario. Of course, some of you may find ways to make this work.

Code Complete

We declared Code Complete on WSE 2.0 just over a week ago. This is one of the major milestones on the road to shipping the product and states that we have finished all the feature work for the product. The team is now in what is generally called "the Endgame" - the place in the product cycle where we close everything else down and then kick it out of the door. What happens in the Endgame? Quite a few things, the main ones for WSE are:

• The entire team engages in a Security Review of the product. We spend time identifying potential threats against the product, confirming those threats and fixing or mitigating them. There's no strict schedule on this step, we often pull in people outside the team for assistance and it can generate some additional workitems that need to be implemented before the product ships.
  
• People inside and outside the team engage in Application Building using the product. This is a sanity check involving taking several possible applications, building them and verifying that there are no obvious feature issues. For example, I'm building WS-Discovery on a soap.udp transport to test the transport and channels part of the product.
  
• Stress runs continue and we move to long-haul runs to seek out those nasty little bugs that prevent the product from being stable over time.
  
• We expect to go to at least one more interop test before we ship, testing our implementation of the current Committee Draft of the OASIS WSS specification with other vendors. We will also complete another review of all the WS-* specifications that we've implemented. Of course, these activities can generate some more bugs that must be fixed before release.
  
• We perform at least one more "Bug Bash" where everyone on the team and as many as possible outside the team takes the current build and spends a solid day installing it, reading the docs, running the samples, writing code and then raising new bugs.
  
• The development team hammers away at all the outstanding bugs on the product until we reach "Zero Bug Bounce", aka ZBB.
  
• The QA team re-tests and verifies all the resolved bugs to confirm they are fixed and test for any regressions. When they are done, we hit "Zero Bugs Resolved", aka ZBR.
  

Only when all the above are complete will the product be almost ready to ship. We will generate at least one "Release Candidate" and let it bake for a while, usually about a week, whilst we stress test it and people work with it. If any bugs are found we evaluate them and decide whether they need to be fixed. Fixing a bug against the Release Candidate resets the timer; we generate a new candidate and start over.
So whilst Code Complete is definitely a major milestone worth passing the beers around for, it doesn't mean we are done. Indeed, the Endgame can often be the most intense and stressful part of the cycle; this is why I've been relatively quiet lately.

Comment Roundup I

A quick roundup of recent comments on entries in the blog.

Sundeep asks about UsernameToken Key Generation:

The label is "WS-Security". What difference is there between it's UTF-8 representation and its ASCII representation ? My understanding is that ASCII is a subset of UTF-8 and since the string is already in ASCII, it's already in UTF-8.

The token specified in Zulu time using xsd:DateTime format is similarly an ASCII string. Why do you warn us to put it in UTF-8 format?

Clearly, you see some difference. Could you please explain it?

My comments are generally an echo of the material in the WS-Trust specification. With particular regard to the label, WSE 2.0 will allow users to set their own label and thus conversion to UTF-8 may be important.

Rich Salz asks about SignedXml:

Any chance of putting back support for standard XML canonicalization? One of our customers found out the hard way -- from their customers -- what WSE 1.0 SP1 only supports exclusive canonicalization.

Sorry Rich, at this time we're not planning to add support for standard ( a.k.a. Inclusive) canonicalization back into WSE 1.0 or WSE 2.0. This form of canonicalization has some limitations and the WSS specification strongly recommends the use of Exclusive canonicalization.

Gia asks about Recipient:

I'm curious to hear what sort of business scenarios do you have in mind that make it essential to be able to select pipeline, target service and policies based on a full set of reference properties instead of single wsa:To header :)

It's no so much about selecting different pipelines or policies as much as about being able to select a specific instance of the target service. The ReferenceProperties in an EndpointReference can behave like cookies if they are reflected back and forth correctly. By way of an example, consider a simple P2P chat program, where every conversation is represented by an instance of the same receiver. By placing information related to the sender in the ReferenceProperties it would be possible to direct messages to different receiver instances (appearing as windows on the display) and thus keep conversations separate. In the same fashion, WS-RM defines Sequences with Identifiers and managing this in a similar means might be desirable. This could be generalized to business conversations between two endpoints. In the end, it becomes a means of maintaining state between two active instances. Now, I'm not saying you *must* use this approach, only that it is there if you needed it.

David ask about SecurityContextToken:

Have you looked at the Caching Application Block the patterns and practices team develops as a means to cache the tokens on the server? That way there would not be another provider interface for caching things, but people could just use the providers that are developed for the caching block already.

Yes, we did look at this pattern. If fact, the design we arrived at does not generally expose the cache object itself at all, only the Add operation. Caches are held within SecurityTokenManagers and the handling of them is therefore the responsibility of the STM, so you should be able to use whatever pattern you prefer.

Channels

In a previous entry I mentioned that we had added a new, low-level programming model to the WSE 2.0 release, this entry gives some early, overview, information on what that programming model is and how it works.

public interface ISoapChannel
{
    void Close();
    bool Closed { get; }
    EndpointReference Endpoint { get; }
}

public interface ISoapInputChannel : ISoapChannel
{
    IAsyncResult BeginReceive(...);
    SoapEnvelope EndReceive(IAsyncResult ar);
    SoapEnvelope Receive();
}

public interface ISoapOutputChannel : ISoapChannel
{
    IAsyncResult BeginSend(SoapEnvelope msg, ...);
    void EndSend(IAsyncResult ar);
    void Send(SoapEnvelope msg);
}

public interface ISoapTransport
{
    ISoapInputChannel GetInputChannel(EndpointReference epr, ...);
    ISoapOutputChannel GetOutputChannel(EndpointReference epr, ...);
}

As you can see from the above code, the new model is exclusively channel and EndpointReference based. Sending a message involves retrieving an ISoapOutputChannel to the target endpoint and then calling the Send method. Receiving a message involves retrieving an ISoapInputChannel for a local endpoint and then calling Receive. There will be a single place to obtain a channel from, but this is presently in flux; expect something like a SoapChannelManager class.

As with the Tech Preview release, we've continued the approach of trying to avoid leaking transport-level semantics up into even this level of the programming model; there's no notion of the TCP-level listen / accept model in the channels layer. Further, a channel can be used to either send or receive, but not both, and there's no access to the network layer itself - you cannot retrieve a socket handle or an MSMQ handle from a channel.

We learned a lot in this area from experimenting with the Tech Preview and from Kevin's work implementing a transport for MSMQ. In particular we realised that the bi-directional capabilities of HTTP and TCP that we'd leaked up into the Tech Preview programming model caused problems when the transport wasn't capable of providing such a connection[1].

Channels are, however, directly bound to transports via the TransportAddress of the EndpointReference object that they relate to. This has two effects: (1) to receive or send over different transports you need two channels (2) transport cross-over cannot occur. The second point is important in that we don't want allow messages to be dispatched to an input channel when that message didn't arrive on the transport associated with the channel.

Following on from the above is the fact that the fundamental message dispatching takes place at the transport level in current builds, not in the old SoapReceiver.DispatchMessage method of the Tech Preview. Each transport has a collection of input channels ordered by EndpointReference; incoming messages are deserialized and the addressing headers are loaded. The transport then selects the correct input channel to deliver the message to. A common base class, SoapTransport, provides helper methods for individual transports to perform the dispatch.

You may also have noticed that channels do not have Pipelines associated with them. Channels are presently an abstraction of the network transport for sending and receiving messages - only limited WS-Addressing processing occurs as part of SoapEnvelope deserialization to allow for dispatching, all other filters presently remain as part of the Pipeline. Looking forward to the future, I can see a model where the current filters are replaced by "filter channels" and a message is passed through a set of channels rather than the Pipeline, this migration could be simplified with a simple adapter class.

The current Pipeline, along with channels, is used by the SoapSender / SoapReceiver layer to process the headers in messages. SoapSender / SoapReceiver have been re-plumbed to use the new channel model. Adding a SoapReceiver to the SoapReceivers collection triggers the creation of an ISoapInputChannel that the collection then posts a BeginReceive on. When a message appears on the channel it is dispatched to the appropriate SoapReceiver.

As you can see, this is a fairly fundamental change to the WSE 2.0 Messaging plumbing - it provides a stronger foundation for the current feature set and for several items that we have planned for the next release. There's also a little more synergy, at least conceptually, with the low-level Indigo programming model.

[1] You can, however, imagine a channel implementation that implements both ISoapInputChannel and ISoapOutputChannel to provide an ISoapDuplexChannel. We're still debating how useful this would be in practice given differences in transport semantics.

SignedXml

WSE 1.0 and WSE 2.0 Tech Preview both include a version of System.Security.Cryptography.Xml.SignedXml. We snapped this code, along with other dependencies, for the WSE 1.0 release when we discovered a blocking issue for processing WS-Security based signatures. We left our clone of the class largely untouched in the product so that we could manage the delta with the base version. Since then, we've noticed that quite a few people appear to be using the WSE class instead of the system one, this is not something that we recommend. If you're one of these people, now is the time to break your dependency on it as it will be removed from the WSE 2.0 final release.

WSE Positioning

Omri has made a definitive post on WSE positioning viz Indigo. We hope this makes things clearer.

WSE as a Conversation

Dave Bettin writes:

"...this is what is great about the "Conversational" mode of WSE, it allows Microsoft to leverage the knowledge they are gaining from implementing a first class service programming model ("Indigo") and push it to us now."

Indeed, WSE is not just about delivering implementations of the latest Web Services specifications, a part of it is about holding a conversation with our customers (hopefully, that means you!). The short product cycle[1], combined with the "breaking changes allowed" model, allows us to listen to feedback and make significant changes from release to release, even from Tech Preview to release. It also allows us to better understand the feature set that customers need and the ways they use the product. In short, we try to listen, learn and react quickly.

As for WSE taking lessons from Indigo, that's true, but just as true is Indigo taking lessons from WSE. What we learn from customers about WSE, we try to also feedback into the broader Indigo team to assist in their decision-making.

[1] OK, no smart remarks about WSE 2.0.

IPasswordProvider Madness

Benjamin says that he doesn't like the IPasswordProvider interface in WSE 1.0. I agree with him, it is a little wierd. WSE 2.0 Tech Preview eliminates the interface but the mechanism still remains if you're not using the built-in Windows Authentication model.

We arrived at this model a long while back because of the nature of the PasswordDigest mechanism in the UsernameToken. In the case where a digest is passed we didn't want the application developer to have to understand the algorithm required to calculate the digest and to remember to verify the digest. However, we didn't make a great job of factoring this so that the password is only passed across when it's absolutely needed.

The 2.0 Tech Preview improved things a bit with the SecurityTokenManager model, but still retained the fundamental requirement to hand the password off to WSE. Since the UsernameToken in 2.0 exposes a public ComputePasswordDigest method, we can probably refactor some more to improve this. The encapulated algorithm still requires that the application hands the password into a piece of framework code, if you don't like that then you're probably going to have to do the work yourself if you want to use PasswordDigest.

V.Last++

Indigo is finally getting some public scrutiny.

The message that WSE is not Microsoft's long-term Web Services solution is plain for everyone to see. This is good. We have worked hard to ensure that we've been transparent with our customers regarding the positioning for WSE: 2+1 years of support and breaking changes between releases, but not everyone seems to have understood.

WSE isn't V.Next, it's primarily V.Last++ with a sprinkling of V.Next--.

It's not over though. WSE isn't going away anytime soon. We're not going to dump it or the customers that have invested in it. We're going to continue doing what we've always done: delivering interoperable implementations of the next generation of Web Services capabilities on an accelerated schedule for customers to experiment with and take tactical advantage of.

There's a buzz circulating around the WSE team: the shipping light has been seen. It's still a little way off, but nevertheless clearly visible. It's been a while since it was last seen, but such is the price of a genuine effort to achieve interoperability.

The Spin Zone

News.com has a "Perspective" article in their "GUTS" (Get Up To Speed) section. Scotg has already given his thoughts on it in post called "Random Speculation". I thought I'd add some of my own.

Technical details about the demo that was shown haven't exactly been widely broadcast.

Here's the big picture.

The following WS-* specifications were used:

• WS-Addressing
  
• WS-Security with multiple token types
  
• WS-Trust
  
• WS-Federation, both Passive and Active profiles
  
• WS-Reliable Messaging
  
• WS-Transactions
  

There were 10 different nodes in play, ranging from a browser client, an application client, 3 Security Token Services, 4 WS-RM nodes, 2 database nodes, distributed transactions using WS-TX between both applications and transaction co-ordinators, plenty of .NET and Java code running on Windows and Linux machines. We layered WS-Security over WS-RM over WS-TX in the same SOAP messages (Secure, Reliable, Transacted).

There was even a fancy monitor application that graphically showed the message flows as they occured because we'd instrumented all the nodes using SOAP. Unfortunately, that wasn't shown at the event.

IBM implemented half of the nodes and Microsoft the other half. In the process of building the demo, both sides probably implemented all the nodes - the choice of how it was hooked up on stage was big company politics and marketing, not a technical limitation.

Despite all this, Bill Robbins says "That makes one think that the whole thing was thrown together rather quickly."

I wish that were true - this "non-event" took too much of my life this summer.

More Interop Fun Next Week

Are we serious about interoperability? You better believe it. For Keith and myself, it's the high-order bit.

WS-Policy in WSE 2.0

In my opinion, the importance of this feature in WSE 2.0 can't be overstated. I like programming models, but the power of the policy model to rid your business code of infrastructure setup is very important, not least because it can shield you from all those breaking changes that I keep making! Ingo Rammer's article on MSDN is good place to get started.

Redistribution and Patching

Mike Vernal has a write-up of some counter-arguments for the recent CyberInsecurity paper in which he discusses the effect of a SQL-based virus against computers across the world. The SQL Slammer virus was compounded by the need to track redistribution of components of SQL Server, not just the server product itself. The Windows world has a model of redistribution using "merge modules" - shrink-wrapped components of a larger application that customers can redistribute. The trouble with this model is that when a merge module is installed the package code associated with it doesn't get registered on the system and so detecting installation is made more difficult, likewise patching if the redistributable is detected. On WSE, we're concerned about this problem as it affects our ability to service indirect customers - those that have our assembly installed as part of someone else's product.

We're going to take some pro-active steps to try to mitigate this in the final 2.0 release but, like the dependency Microsoft has on people running Windows Update, we're still dependent on people that redistribute our product having plans in place to monitor for security bulletins and then be able to redistribute any patches in a timely manner to their customers. This is true whether the redistribution is via an internal IT department or an ISV. If you're going to redistribute, make sure you take account of this and plan appropriately. Alternatively, don't redistribute WSE but have the customer install it directly as a pre-requisite - that way at least the customers IT department has a chance to get into the loop and prepare themselves.

Web Services Interoperability

We've just spent all of last week interop testing OASIS Web Services Security scenarios with a group of other vendors. By the end of the week, we had successful message exchanges with everyone on the list for as many of the seven scenarios that they supported. Some vendors, I'm sad to say, could only do the most trivial of scenarios.

Some articles on the web frustrate the hell out of me, particularly when I know of several large customers using our Web Services technology for transactions worth millions of dollars. That same software is interoperating with another vendors Web Services software in order to deliver the overall solution. I wish this kind of article would drill down some more with specific examples, but maybe there's a reason they don't do that.