Managed World

You know, I don't know Robert Scoble that well, so I wasn't really wanting to chime in to respond to this post he recently made. In fact, the closest I've ever been to Scoble is that I'm now working in his old office, that's it. But after reading Jeff Sandquist's response, I have to chime in and disagree with Scoble as well.

I won't pretend to know whether Scoble's ego is out of control or not*, but Scoble, you are just wrong. I vehemently disagree with the sentiment that a blog has to be public to be a "blog". My wife blogs, is it public? No. Why? Because it deals with issues like fertility problems and other struggles. Even though it is not "public", it is available to be linked _within a smaller community_.

You can't find any posts from Google, MSN Search, or any other external search engine. However, all her posts are discoverable by other members of the community that are facing similar issues. She is getting a lot of the benefits from public blogging while being able to limit her audience to a support group that would be able to understand and sympathize with her.  I fully support her doing so, and encourage her to continue. Would I call it a blog? ABSOLUTELY. Within her community, they all gain the benefits of blogs and accomplish all five "things" that Scoble discusses (via the use of internal tools). After blogging for almost three years, I like to think at least that I perhaps "get it" more than the average Joe.

In the comments on Jeff's post, Scoble makes the following statements:

You just gave them their own name "internal blogs." Over on my blog you called them "private blogs."
Go see a trademark lawyer. You know that "Vista" is not the same thing as "Windows Vista."

Scoble, once again, you are just wrong. I don't believe this is the same thing as comparing "Vista" and "Windows Vista." Comparing "private blogs" to "public blogs" is more like comparing "wool sweater" and "polyester sweater": they are BOTH sweaters!!! They simply use a different "medium".

In my opinion, it is pretty short-sighted to say that "private blogs" or not "blogs." Think of the use of blogging within support groups: like for people suffering from HIV/AIDS, recovering Alcoholics, etc. They can have a way to communicate "publicly" within their own support group yet still remain "private" to the outside world. In a way, it is almost like comparing the "private", "protected" and "public" keywords in a programming language. Alas, I won't go down that road of comparison though :).

So please Robert, step back and look at the bigger picture. By taking the perspective you are, you are only hurting the topic that you say you are an "expert" at. If you really love blogging as much as you do, please open your eyes to the different ways blogging can be used to help make this a better world, not just the ways that "you want" to use them. Remember, you != everyone.

*although statements like "Blogging is something I'm a weeeee bit of an expert on" in this response post certainly make it seem that way. With how relatively young blogging is compared to other forms of communication, I don't believe that any single person can be considered an "expert" at it.

A friend of mine made a good comment on my last post about TxF: "what the heck does TxF do and why should developers care?" I hope to answer that question in this post. If you've never heard of TxF, read on. If you have and still don't "grok it", read on. If you just happen to love the wonderful language that is expelled from my tranquil appendages as they dance over the keyboard, then by all means, read on as well :).

So, what is TxF? TxF is the "nick name" for a new technology coming with Vista and Longhorn Server: Transactional NTFS. You may have read about or used logging file systems in the past, and you may have used journaling file systems in the past as well. But TxF is a fully-transactional file system, all built on top of NTFS. And when I say "fully-transactional", I mean it supports full ACID transactions on the file system itself.

Why should you care about TxF? You should care about TxF because the transactions that operations are contained in are not only specific to the file system. When working with TxF, there is full support for the Distributed Transaction Coordinator as well. This means that now I can coordinate transactions between the file system and SQL. Heck, this also means that I can coordinate transactions between the file system and Windows Communication Foundation.

Just stop for a second, and think about the scenarios that this opens up to us as developers. Let's look at a Document Management System, for example. If I were to submit a new version of a file into the system, the database may have to store a hash of the file as well as a pointer to the file on the file system. The problem is that we have to now guarantee that the hash of the file in the database in identical to the version of the file on the file system (I'm of course assuming that we aren't just shoving everything into blob fields).

If we update the file on the file system first, there are at least two situations that can occur that we might have to deal with: 1) If someone tries to pull down the file before the database has been updated, then the hash comparison will fail and the file will falsely be identified as corrupt. 2) If the database update fails for any reason at all, we now have to write extra code to roll back the file manually, and in the meantime, users pulling down the file could have it falsely identified as corrupt again.

So, let's change the order. Let's say we update the database first and then the file. Unfortunately, we're not in a better position, only reversed from before. Either way we go, there is more code for us to write if we want a stable system. This is where TxF can come in to make our systems more stable and robust. We can use TxF's ability to coordinate with DTC to manage all of this for us.

First of all, it would be completely Atomic. By using DTC for our database update, and TxF for our file update, they can coordinate with each other to guarantee that either they both succeed or none of them do. That's right, if the database update fails, the file system update will be rolled back, and vice versa. The transaction is also Consistent. At the same time, the file system update is also Isolated. After we update the file and before we commit the transaction, every one outside of the transaction will still see the old version of the file (how cool is that?!?!?). And the system is Durable. If we commit the transaction and the system crashes, when it restarts, the file will be guaranteed to be updated. And all of this is handled by TxF (well, handled by the new Kernel Transaction Manager, to be specific).

That's just the beginning of the story though. Imagine this on a distributed scale. We can use TxF to manage file operations across multiple machines. We can also coordinate TxF with WCF using System.Transactions :D. So, let's say I wrote a Music Download service. As a user, I want to ensure that my credit card is not charged unless I download the file completely. This is a scenario that becomes a lot easier with TxF. If my credit card was not able to be charged, I should be not be able to keep the file. On the other hand, if I was not able to download the file successfully, my credit card should not be charged (disclaimer: there are some obvious security holes to work through in this example :P).

By adding transactional support into the file system, we are also hoping to make your lives easier in the long run. You no longer have to write extra code to self-manage the file system when dealing with distributed systems. By leveraging TxF and System.Transactions, all you basically need to do is code against "The Happy Path." You just have to ensure that if anything fails, all of it is rolled back (which is ultimately handled by the system). There are a couple of sayings I like to keep in mind as a developer: 1) The fastest running code is code that doesn't run at all (I believe I got this one from Rico M.) and 2) Code that isn't written is code that doesn't have to be tested. Easier lives through transactions :).

It's not all glitz and glamour though. In my last post on TxF, I discussed some of the "down side" of TxF as the API exists today. There were two "features" that are not implemented that I didn't mention. Mainly, at this time, we do not have support for transactions on streams, or transactions when dealing with encryption. The latter, I am told, is because of some interesting security challenges that we need to iron out first.

One of the challenges I have ahead of me as a Technical Evangelist is "spreading the word" of TxF. Even before I joined Microsoft, I had only heard about it because of a couple of things: 1) the job posting for the job I interviewed for and 2) A video about TxF made by Channel 9. There really isn't a lot of information out there on TxF. And the information you can find is largely based on Vista Beta 2 or earlier, meaning, it is now quite out of date. 

When chatting with Carlos about it, he said (and I quote): "I read a lot of blogs and that's the first I've ever heard of it." I'm pretty sure he's not the only one that would have this same reaction. Time to spread the word, my friends :). I only hope I can get a fraction of you out there as excited about TxF as I am. If I can do this, then hopefully the technology will speak for itself and word of it will spread fairly quickly.

I know Dave, Tom, and Boyd had stated that they were pleasantly surprised that the announcements coming out at GameFest tomorrow hadn't been leaked. Well, close but no cigar (if this is indeed the announcement they were talking about). Of course, considering that it is already "tomorrow" on the east coast (and many of you won't read this post until "tomorrow" anyway), it's probably close enough :).

So, check out that link above :).

As a hobbyist developer myself, I have to admit that I did entertain thoughts in the back of my mind that it perhaps wasn't going to get any easier for us. Well, I'm happy to say that it looks like that thought was totally false. Not only was that thought wrong, it was WAY wrong. Microsoft (according to the Joystiq article) will release a free version of XNA Game Studio inline with their current Express products. So, for those of you enjoying writing games with totally free tools like Visual C# Express, you will be able to apparently continue doing so when XNA is released.

That isn't the most exciting part though. From what it sounds like, Microsoft is indeed going to support hobbyist game development on XBox Live Arcade. Of course, that isn't free. However, it currently appears that it will be fairly cheap, just $99/year. This could be HUGE. I mean HUUUUGGGGEEE. This opens the door for a Community-Powered XBLA, all in Managed Code nonetheless!!! Oh my, I'm getting tingles just talking about it :).

Imagine being able to write a game for your 360 yourself, AND be able to share it with friends. Heck, I can't imagine what the possible ramifications for the game industry are. If done properly, it could be history all over again. It could be "the next" Shareware model that will see a level of innovation not truly seen since the days of Commander Keen, Wolfenstein, and Doom. Will the next iD grow out of this new model. Could this be the next opportunity for hobbyists to be able to make a living of their game programming. Okay, I won't get carried away here, I can't imagine the same boom happening that happened in the 80s/90s. But geez, in some ways it could be even better.

I, for one, will HAVE to check this out when I can get my hands on it. Oh boy, good luck getting me to sleep tonight! Thanks guys!

(Andy, I can't wait to see you start writing about this stuff :P).

Update: I found this Microsoft PressPass Announcement

[Usual Disclaimer: my opinions in no way reflect or represent the opinions of my employer]

You'll probably hear me start cheering for TxF after a couple of posts because it is a great technology, but at this point in the ballgame, let's take a step back and take a sobering look at the state of TxF as it exists in post-Beta 2 Vista.

As I've been getting up to speed with TxF, I've realized that there is very little up-to-date information on TxF available (except for an internal presentation that was given at an internal conference). Most likely, any blog posts you'll find on the internet regarding TxF and how to use it are no longer accurate because of the newly revamped API. The extent of up-to-date information even available internally is constrained to a single presentation by the PM and the header files for the various APIs. Of course, it is one of my responsibilities to contribute to the amount of content that is available, so I suppose you could consider it job security if I do my job right :).

TxF itself, is a VERY cool technology. And to put together a managed code example of how to use the new KTM to manage transactions across files simply on the local file system is nearly trivial (you can expect to see a Screencast on Channel 9 from me on just how to do this). However, the truly interesting and compelling scenarios (I believe) don't exist simply with only files on the local file system. To me, the interesting scenarios are when you are managing transactions that span SQL and TxF, or WCF and TxF, or heck, even SQL + WCF + TxF. However, when we get to these scenarios, the "story" breaks down a bit.

If you are a C/C++ developer, you're fine, you're clear, you're out of the woods, you've been cleared for a safe landing, you can use this stuff full-force out of the box. If you are a managed code developer on the other hand, buckle your safety belt folks, we're going in for an emergency landing. To me, this part of the story, frankly, SUCKS (and trust me, we know it sucks). You heard it correctly folks, as of the post-Beta 2 API, there is no longer any Managed API into TxF. If you want to use TxF from C# (and have it work with SQL/WCF via DTC), you will be following the brown-bricked road into P/Invoke and Marshal.QueryInterface land.  

The removal of the managed API can partially be attributed to the removal of the implicit transaction model, which brings up another unfortunate side-effect. When the implicit transaction model existed, you could leverage TxF with little to no code changes. You could basically instantiate a TransactionScope, have KTM enter the transactional context (EnterTransactionScope/ExitTransactionScope in kernel32.dll), and then utilize the normal File class APIs. Since the transaction model was implicit, the files would (you got it) _implicitly_ be enrolled in a KTM transaction that is being coordinated with the TransactionScope.

With the implicit model having been ousted now, there are a series of new transacted API calls that exist in kernel32. For example, there is now a CreateFileTransacted, OpenFileTransacted, etc. and they all have almost the exact same interface as their corresponding un-transacted siblings (the only real difference being that the *Transaction API calls accept some additional transaction-oriented parameters). As you can see, it is very much an explicit programming model now. Unfortunately, this means that if you have an existing product, you will have to change any file system calls necessary to leverage the new Transacted APIs. It's not too bad, but it still sucks in my humble opinion. Of course, for any managed code developers out there, it sucks more because instead of using the File class in System.IO, you now have to P/Invoke into kernel32 to get at all the Transacted APIs.

In my opinion, the challenge this presents us as a company is that if we want our managed developers out there to use this, we absolutely CAN'T expect them to use it as it exists today. I think if we tried to do that, adoption would fall flat on its face. My current thoughts are that we can perhaps release a publicly-available community wrapper (maybe on GotDotNet, CodePlex, or something else) that people can use to easily interact with TxF with very little effort. I also believe that for even that to be successful, it can't be a "totally new" API. We should leverage the existing look and feel of APIs like System.Transactions and System.IO.File to provide the TxF functionality. I'd imagine this is one effort that I'm going to try to champion and to get out there for all of you. Heck, if not for you, I'll just do it selfishly just for me since I don't want to be constantly context-switching down into P/Invoke, COM madness when writing my screencasts, demos, and other TxF code.

Don't get me wrong, there has been a lot of cool work done. And all of the reasons I've heard on why the implicit model was "broken" (or at least that implementation of an implicit model) make absolute sense. It is just another example of what you have to go through when you have to worry about backwards compatibility, app-compat, etc. Yes, in this case, it has given us a bit more work to do. But I can live with that :).

Why am I making this post? Am I a pot-stirrer? I don't believe so. I simply want to get the truth out there so you all know that there are some warts we need to remove on the product. More importantly, we are recognizing these warts and are actively engaged at trying to remedy them. I truly hope this does not dissuade you from trying it out as it is certainly very cool stuff and shows a lot of potential moving forward (which I hope you'll see as I dive into it with all of you :D).

If you haven't already, you must go checkout InfoCenter (being developed by a coworker of mine). I want it, I want it bad. A new toy to play with. Gimme gimme gimme :). 

I figured I would use this blog not only to talk about some of the cool technologies I'm currently working with, but to also increase the transparency into what the role of Technical Evangelist is actually like here at Microsoft. Keep in mind, of course, that I'm new to this position and hence am still getting up to speed with the position (and will be getting up to speed for a while).

One of the things I love so far about the role of a Technical Evangelist is all the different skills you get to use and all the different people you get to interact with. Today, I believe, is a good example of that. So, let's chat about my day :).

Today, I came into the office planning on doing some coding with TxF (Transactional NTFS). I was happily coding away when I hit a bit of a snag (more on that in an upcoming post) and shot off an email with some questions to a Product Manager in the Enterprise Services area. He was able to answer some of my questions and is going to follow up with others on answering the others.

What I find interesting about this in hindsight though, is that I actually knew of the Product Manager not through my boss, not through my direct coworkers, but through another Product Manager that I'm working with that gave me his name. It only drives home the point even more that sometimes it's not what you know, it's who you know. If what I've seen and experienced in my first couple of weeks here is any indication, this saying is especially true for a Technical Evangelist. And it's not only true when it comes to Product Teams (more on that to come).

Once I shot off the email to the PM, I went to an internal site to start doing some networking with technical contacts for ISVs that I want to pursue to adopt TxF (what do you know, it's that whole "who you know" thing again). What's interesting is that being a Technical Evangelist is not only technical, it is also part sales and part marketing. If you're not willing to be a "shmoozer" or get out there to build up your professional network (meaning, you would rather site in your office and write code), then the Technical Evangelist position is definitely not for you. Luckily, for me, I like this whole aspect of the job, so I'm definitely happy so far.

So, now I've done some coding, started to build up some internal ISV networking support, and I still have many other things to do. Funny, reliving my day like this makes me realize that I'm kind of like a single processor computer (okay, now you have to KNOW I'm a geek after comparing myself to a computer). You know, while I "multitask", it's not _actually_ multitasking. It's more that I switch from thread to thread getting a bit done on each everytime while still trying to minimize the amount of context switching so I can continue to be productive. If only there was a human equivalent to being Multi-Core or Multi-Proc, *then* I'd be set. Anyways.....

After all of this, I switched over to start working on a powerpoint deck that is on my "deliverables list" for my boss. I figure I can dedicate a good chunk of time to this while I wait to hear back from the PM regarding my DTC questions. This powerpoint is a powerpoint roughly titled "Longhorn Server For Developers" and will outline some of the major changes/enhancements coming in Longhorn Server. To me, this is a win-win. It provides an internal/external resource that isn't really distilled in one place yet, and it gives me exposure to a bunch of new features to better build up my mental "road map" of what is coming (and to see if there are particular features that I'm very excited about and want to cover).

In my research so far, it is amazing how many resources are out there available (even _just_ on Microsoft's intranet). It's definitely a lot of information to take in. The challenge I've seen so far though is that so much information I have seen on Longhorn Server is really dedicated to the ITPro crowd. What is it about Longhorn Server that the Developers should care about. It's a challenge, that's for sure.

In that challenge, lies yet another challenge though. How do you take a Core OS feature, give it glitz and glamour, and make it sexy. The .NET 3.0 features? Yes, I can see how you can make those sexy. But, let's say Remote Differential Compression. How do you take something like that and make it sexy for the general development community? Yes, you can make it exciting for the hardcore geeks out there (goodness knows that I almost had a heart attack out of sheer joy when I heard those three words mentioned together). But how do you make it exciting to average Joe Developer (dare I say Mort without risking getting flamed)? It's a tough one. For sure, it will take creativity. Yet, that's another reason that I'm loving this job so far :).

Basically, in a nutshell, I get paid to write code, talk to people and get them excited about the code I'm writing, be creative, work with cutting edge technologies every day, AND blog about it all to boot? Yup, that's what I get paid to do :). There are the other things like commitments and goals that I'll talk about in a future post though (which are cool in of themselves since it gives you measurable items to achieve during a year (and to get reviewed against at the end of the year)).

Yup, that was my day today: Code, Chat, Email, Network, Research, Blog :D.