Names and places are changed to protect…well, to protect against any personal liability. You can guess the rest.
The first thing you will note about a project like this is that the actual end user and/or business requirements are not really defined until you are at least 75% through the development of the software that is supposed to support the requirements. An important thing to highlight here is that the actual end user requirements and the business requirements are not necessarily the same. No, it would be folly to think that they were.
Another fine point is when the PM in charge of the project, and the lead developer who built the code for the project, both leave the day after the initial deployment of the software in question to production. And leave it to you to manage it. And, by the way, the software doesn’t actually work quite right and needs to be fixed. And no, I’m making this up.
And let’s compound the fun by making the software part of a deployment process where the development team is not allowed to have any physical or remote access to the servers in UAT or Production. And, also, let’s add that this is the first deployment of the group into a particular environment (so, there’s Production in the current environment, and then there’s Production in the new environment) which compounds everything because you haven’t had the chance to ‘kick the tires’ previously to shake out all of the configuration related issues that always come up.
And, finally, let’s add the fact that the new Production environment is ‘clustered.’ This means different things in different situations, but for now, let’s just say that it means that there are multiple virtual machines (the number of which is potentially always in flux), all of which have the application deployed. And each machine runs one or more Windows Services to manage the application. Oh, and finally, finally, let’s point out that the application was designed under the assumption that one and only one machine would be running it at any time, meaning that no single machine has any idea that there might be other active machines trying to do the same things at any given point in time.
And did I mention that at the core of this application is a vended application that we have no control over (in terms of changing code, at the very least)?
Some might think I was making a scenario like this up. Though the details are sometimes better or sometimes worse, this isn’t actually completely out of the ordinary. Okay, it isn’t good. Having the PM and Lead Developer leave the day after it was first deployed to production was a nice little event to deal with. They both left voluntarily (i.e., they weren’t canned), and I don’t blame either of them for it, but it makes for some interesting issues. As much of a jackass as I am, I’m very little inclined to blame the prior regime for any problems, as I don’t think it comes across well, and it’s sort of irrelevant. What matters is dealing with the current situation and dealing with it. On the other hand, some of the problems are, quite simply, due to oversights. In this particular scenario, the fact that it was being deployed into a ‘clustered’ environment was not really thought out. The ‘solution’ designed assumed it could be handled by shutting down the Windows Service(s) on all but one machine when deployed. Anyone with any operational experience knows this is a non-starter. It’s a clustered environment where there are multiple active machines, not one where there is software that automatically handles having one machine be active with the rest passive, as you have when running Microsoft Cluster Server, for instance. But, I digress.
The question is: what the hell do you do if you find yourself in this position?
The Magic Happens Here…Maybe
The first thing you need to do is to determine, what really is the critical business need?
What helps here is to find out who the actual end users are who are on the critical path, and to determine what they really need to work. When coming into a software project ‘blind', so to speak, there are certain things you can do. As primitive as it might sound, a great starting point is to watch an end user use the application to get a sense of what their expectations are.
The idea is pretty simple, but I think worth describing. I’m Johnny New Developer, and I’m given a fairly significantly sized Visual Studio solution. It’s always fun to go through the source code ‘cold’, so to speak, but even when the code is well-written (as the core code, the code that provides the central functionality of the application, for this app was), it isn’t always clearly evident what the starting point is. I mean, obviously, if the App is a Windows Form app (as this one is), you can start from Main, and go from there, but that doesn’t really tell you the crucial parts of the software, where it is most likely to fail.
The fact that at the core lies a vended app brings up another point (or 12), so let me digress….
TDD sucks, but even if it didn’t, it wouldn’t matter
I’ve often commented on why I really dislike TDD, but it’s worth expounding upon, because this particular project helps to explain why.
TDD tells you to write tests before you write functioning code (there are endless debates about how rigorous you need to be about this. The people who argue in favor of total rigorosity (yes, I made that up) are wrong. But I digress). The idea is straightforward, which is why is sounds so cool. Before writing the code that actually implements functionality, writing tests naturally forces you to consider basic questions like “what do I actually want this code to do” and “once I figure out what I actually want the code to do, what’s the easiest, simplest way to implement it” and so forth. Nothing about this is inherently bad, in fact, in general, it is inherently good.
But what TDD doesn’t tell you is where to start, and even worse, it focuses on the class level. What should this class do? What random edge cases do I need to consider? Unless you are building a framework (where you don’t know necessarily how it might be used, and so need to define an API that considers all legitimate possibilities), this is wasted effort.
Where what is commonly called BDD (no one really agrees what to call it, or what it exactly means, from what I can tell, but I digress) is vastly superior by orders of magnitude is that your testing is guided by the business requirements of the software project, which should be defined by, well, the business. Moreover, when you are building an application, *you* define how the application works.
To use a stupid example: suppose if someone enters “-74” in a textbox, it will cause your class function to blow up. TDD would tell you that you need to write tests to determine what will happen if “-74” is an input to your function. What if someone enters “Q”? You need to write a test for that (and to be clear, I don’t mean to imply you need to write tests for every single possible input, that would be silly, but that you should write tests for ‘types’ of bad input). And so on and so forth.
If you look at it from a BDD standpoint, you don’t need to know how your function will respond to bad input, because you write ‘Specs’ (a fancy word for ‘Tests’ that has some implications) that prevents your application from accepting bad input in the first place, often through simple RegEx validation (and yes, I know that the idea that RegEx is easy is funny, but I digress).
More importantly, from a BDD standpoint, you understand that the important functionality of your code crosses classes, functionality which you can control at the application level. TDD can tell you how classes will function in isolation, but they don’t really tell you how your code works across classes, and when it comes to testing an application, this matters most of all. BDD focuses your attention on the functional/business requirements of your software, exactly where it is needed.
Once you start working with vended applications, TDD really sucks because it doesn’t focus on the often unavoidable interactions your application have with those vended applications. BDD ‘Specs’ more naturally focus on how the various parts fit together, which is often times where real-world applications break down. It’s not a *bad* thing to know that *if* a vended application gives back output A, my Application will do B. And mocking all that out (or stubbing or whatever) isn’t evil. But more often than not, you want and need to know what the actual output from a vended Application is.
And with BDD, you can easily separate ‘Specs’ that are ‘integration-like’ from those that aren’t. With TDD, integration testing is something above and beyond, but, especially when you are working with vended applications, the integration tests often become the most important ones. BDD tests “does my application actually work” whereas TDD tests “given how well I’ve thought out the possibilities, do my class functions work as I expect?” The latter is nice to know, the former is what actually matters.
Where the magic happens, maybe (again)
Once you understand that the integration tests are the ones that ultimately matter, you can then focus in on the entry points into the vended application that can cause your application to fail.
There is no ‘magic’ without knowing what those entry points are and how the vended application will perform, but some simple points are obvious. If you are in a ‘clustered’ environment, are those entry points idempotent? If they are, great, but if they aren’t, sometimes you have to fall back to a simplistic data driven solution, where only one server is designated as being active, and so no action will be performed unless performed by the designated server. Though it feels massively hacky to do this, it is often the easiest way out of a tough situation.
Every situation is different, but if you are forced into trying to support a software project that you otherwise know nothing about, work with the end users. Watch how the application works. Find out what the critical entry points are and focus on them.
Or, you could quit. I don’t recommend that course of action, but in the end, you have to do what you have to do.