Managed World

Hello, everybody, I have released the first version of a rudimentary managed wrapper around Transactional NTFS to Code Gallery on MSDN: http://code.msdn.microsoft.com/txfmanaged.

The wrapper in its current form is very basic but should give developers a good idea on how simple it is to use Transactional NTFS from managed code. Feel free to go grab it, play around with it (there are some sample screencasts using the wrapper on the Developer Meet Server Show), and feel free to use it any way you wish.

Enjoy!

Howdy all, I will be at Tech Ed Developer this week doing several things. First of all, I will be giving two talks: the first on Application Compatibility in Windows Server 2008 outlining the top issues developers need to be aware of when developing for it, and second, a talk on Transactional NTFS as it pertains to Windows Server 2008.

Also, I will be "manning" the Ask The Experts booth for Windows Server 2008. So if you want to chat about Windows Server 2008 (or just say hello), make sure to drop on by (I will have the hours I plan on working the booth posted there at the booth).

Here's a rundown:

• Monday, 3pm - [INF203] WS08 For Developers - App Compat, Tent 1
• Thursday, 4:30pm - [INF307] WS08 For Developers - Transactional NTFS

I'm also hoping to attend some sessions myself. These will mostly be around C++ (TLA323, TLA302, TLA404, etc.).

So, if you're going to be at the conference, please, drop on by the Ask The Experts booth for Windows Server 2008 and say hello!

Jim Johnson has a new post about how to use TxF with the new model in Vista RC1. You will obviously need the RC1 bits to mess around with it.

I'm hoping to expand on this topic in the future to show some code samples of using it not only by itself, but also using it with SQL and the like. I'll also have some "context" posts around that describing the architecture and relationships around TxF so you can better understand the articles.

Until then, go check out Jim's post :).

A friend of mine made a good comment on my last post about TxF: "what the heck does TxF do and why should developers care?" I hope to answer that question in this post. If you've never heard of TxF, read on. If you have and still don't "grok it", read on. If you just happen to love the wonderful language that is expelled from my tranquil appendages as they dance over the keyboard, then by all means, read on as well :).

So, what is TxF? TxF is the "nick name" for a new technology coming with Vista and Longhorn Server: Transactional NTFS. You may have read about or used logging file systems in the past, and you may have used journaling file systems in the past as well. But TxF is a fully-transactional file system, all built on top of NTFS. And when I say "fully-transactional", I mean it supports full ACID transactions on the file system itself.

Why should you care about TxF? You should care about TxF because the transactions that operations are contained in are not only specific to the file system. When working with TxF, there is full support for the Distributed Transaction Coordinator as well. This means that now I can coordinate transactions between the file system and SQL. Heck, this also means that I can coordinate transactions between the file system and Windows Communication Foundation.

Just stop for a second, and think about the scenarios that this opens up to us as developers. Let's look at a Document Management System, for example. If I were to submit a new version of a file into the system, the database may have to store a hash of the file as well as a pointer to the file on the file system. The problem is that we have to now guarantee that the hash of the file in the database in identical to the version of the file on the file system (I'm of course assuming that we aren't just shoving everything into blob fields).

If we update the file on the file system first, there are at least two situations that can occur that we might have to deal with: 1) If someone tries to pull down the file before the database has been updated, then the hash comparison will fail and the file will falsely be identified as corrupt. 2) If the database update fails for any reason at all, we now have to write extra code to roll back the file manually, and in the meantime, users pulling down the file could have it falsely identified as corrupt again.

So, let's change the order. Let's say we update the database first and then the file. Unfortunately, we're not in a better position, only reversed from before. Either way we go, there is more code for us to write if we want a stable system. This is where TxF can come in to make our systems more stable and robust. We can use TxF's ability to coordinate with DTC to manage all of this for us.

First of all, it would be completely Atomic. By using DTC for our database update, and TxF for our file update, they can coordinate with each other to guarantee that either they both succeed or none of them do. That's right, if the database update fails, the file system update will be rolled back, and vice versa. The transaction is also Consistent. At the same time, the file system update is also Isolated. After we update the file and before we commit the transaction, every one outside of the transaction will still see the old version of the file (how cool is that?!?!?). And the system is Durable. If we commit the transaction and the system crashes, when it restarts, the file will be guaranteed to be updated. And all of this is handled by TxF (well, handled by the new Kernel Transaction Manager, to be specific).

That's just the beginning of the story though. Imagine this on a distributed scale. We can use TxF to manage file operations across multiple machines. We can also coordinate TxF with WCF using System.Transactions :D. So, let's say I wrote a Music Download service. As a user, I want to ensure that my credit card is not charged unless I download the file completely. This is a scenario that becomes a lot easier with TxF. If my credit card was not able to be charged, I should be not be able to keep the file. On the other hand, if I was not able to download the file successfully, my credit card should not be charged (disclaimer: there are some obvious security holes to work through in this example :P).

By adding transactional support into the file system, we are also hoping to make your lives easier in the long run. You no longer have to write extra code to self-manage the file system when dealing with distributed systems. By leveraging TxF and System.Transactions, all you basically need to do is code against "The Happy Path." You just have to ensure that if anything fails, all of it is rolled back (which is ultimately handled by the system). There are a couple of sayings I like to keep in mind as a developer: 1) The fastest running code is code that doesn't run at all (I believe I got this one from Rico M.) and 2) Code that isn't written is code that doesn't have to be tested. Easier lives through transactions :).

It's not all glitz and glamour though. In my last post on TxF, I discussed some of the "down side" of TxF as the API exists today. There were two "features" that are not implemented that I didn't mention. Mainly, at this time, we do not have support for transactions on streams, or transactions when dealing with encryption. The latter, I am told, is because of some interesting security challenges that we need to iron out first.

One of the challenges I have ahead of me as a Technical Evangelist is "spreading the word" of TxF. Even before I joined Microsoft, I had only heard about it because of a couple of things: 1) the job posting for the job I interviewed for and 2) A video about TxF made by Channel 9. There really isn't a lot of information out there on TxF. And the information you can find is largely based on Vista Beta 2 or earlier, meaning, it is now quite out of date. 

When chatting with Carlos about it, he said (and I quote): "I read a lot of blogs and that's the first I've ever heard of it." I'm pretty sure he's not the only one that would have this same reaction. Time to spread the word, my friends :). I only hope I can get a fraction of you out there as excited about TxF as I am. If I can do this, then hopefully the technology will speak for itself and word of it will spread fairly quickly.

[Usual Disclaimer: my opinions in no way reflect or represent the opinions of my employer]

You'll probably hear me start cheering for TxF after a couple of posts because it is a great technology, but at this point in the ballgame, let's take a step back and take a sobering look at the state of TxF as it exists in post-Beta 2 Vista.

As I've been getting up to speed with TxF, I've realized that there is very little up-to-date information on TxF available (except for an internal presentation that was given at an internal conference). Most likely, any blog posts you'll find on the internet regarding TxF and how to use it are no longer accurate because of the newly revamped API. The extent of up-to-date information even available internally is constrained to a single presentation by the PM and the header files for the various APIs. Of course, it is one of my responsibilities to contribute to the amount of content that is available, so I suppose you could consider it job security if I do my job right :).

TxF itself, is a VERY cool technology. And to put together a managed code example of how to use the new KTM to manage transactions across files simply on the local file system is nearly trivial (you can expect to see a Screencast on Channel 9 from me on just how to do this). However, the truly interesting and compelling scenarios (I believe) don't exist simply with only files on the local file system. To me, the interesting scenarios are when you are managing transactions that span SQL and TxF, or WCF and TxF, or heck, even SQL + WCF + TxF. However, when we get to these scenarios, the "story" breaks down a bit.

If you are a C/C++ developer, you're fine, you're clear, you're out of the woods, you've been cleared for a safe landing, you can use this stuff full-force out of the box. If you are a managed code developer on the other hand, buckle your safety belt folks, we're going in for an emergency landing. To me, this part of the story, frankly, SUCKS (and trust me, we know it sucks). You heard it correctly folks, as of the post-Beta 2 API, there is no longer any Managed API into TxF. If you want to use TxF from C# (and have it work with SQL/WCF via DTC), you will be following the brown-bricked road into P/Invoke and Marshal.QueryInterface land.  

The removal of the managed API can partially be attributed to the removal of the implicit transaction model, which brings up another unfortunate side-effect. When the implicit transaction model existed, you could leverage TxF with little to no code changes. You could basically instantiate a TransactionScope, have KTM enter the transactional context (EnterTransactionScope/ExitTransactionScope in kernel32.dll), and then utilize the normal File class APIs. Since the transaction model was implicit, the files would (you got it) _implicitly_ be enrolled in a KTM transaction that is being coordinated with the TransactionScope.

With the implicit model having been ousted now, there are a series of new transacted API calls that exist in kernel32. For example, there is now a CreateFileTransacted, OpenFileTransacted, etc. and they all have almost the exact same interface as their corresponding un-transacted siblings (the only real difference being that the *Transaction API calls accept some additional transaction-oriented parameters). As you can see, it is very much an explicit programming model now. Unfortunately, this means that if you have an existing product, you will have to change any file system calls necessary to leverage the new Transacted APIs. It's not too bad, but it still sucks in my humble opinion. Of course, for any managed code developers out there, it sucks more because instead of using the File class in System.IO, you now have to P/Invoke into kernel32 to get at all the Transacted APIs.

In my opinion, the challenge this presents us as a company is that if we want our managed developers out there to use this, we absolutely CAN'T expect them to use it as it exists today. I think if we tried to do that, adoption would fall flat on its face. My current thoughts are that we can perhaps release a publicly-available community wrapper (maybe on GotDotNet, CodePlex, or something else) that people can use to easily interact with TxF with very little effort. I also believe that for even that to be successful, it can't be a "totally new" API. We should leverage the existing look and feel of APIs like System.Transactions and System.IO.File to provide the TxF functionality. I'd imagine this is one effort that I'm going to try to champion and to get out there for all of you. Heck, if not for you, I'll just do it selfishly just for me since I don't want to be constantly context-switching down into P/Invoke, COM madness when writing my screencasts, demos, and other TxF code.

Don't get me wrong, there has been a lot of cool work done. And all of the reasons I've heard on why the implicit model was "broken" (or at least that implementation of an implicit model) make absolute sense. It is just another example of what you have to go through when you have to worry about backwards compatibility, app-compat, etc. Yes, in this case, it has given us a bit more work to do. But I can live with that :).

Why am I making this post? Am I a pot-stirrer? I don't believe so. I simply want to get the truth out there so you all know that there are some warts we need to remove on the product. More importantly, we are recognizing these warts and are actively engaged at trying to remedy them. I truly hope this does not dissuade you from trying it out as it is certainly very cool stuff and shows a lot of potential moving forward (which I hope you'll see as I dive into it with all of you :D).