When you are setting up a new system, or troubleshooting a problem with any system, do you use something to compare the problem system to in order to find the irregularities that might be the problem? If you do use a comparison or baseline reference, how do you know it is correct? Really, these are very tricky questions and to some extent you have to take it on faith that your baseline system is correct. But what it it isn’t or at least it isn’t entirely correct?
I have a system that I administer that consists of seven segments of data that have to be processed. I am in the process of setting up a new version of the system that is two releases newer than what we use for production. I have a test or development copy of the production system that I have been testing these processes against. It’s great because they can both use the same data source that is static and therefore I should be able to get the exact same results.
For two of the processes this worked great almost right away. For the next one it required running some auxiliary processes to populate some of the files correctly. And then it was fine and this also helped the next process. One of the remaining three processes is entirely different than the others so we will ignore it. The last two processes seemed to have the errors in roughly the same segments of data.
So I broke it down to as small a segment of data as I could and then I started comparing how the data was being retrieved from the data source and verifying that both systems appear to be retrieving the correct data. So then I moved on to how that data was being loaded into the target systems to verify that both systems appear to be loading in the same manner.
Here’s where it gets really frustrating. If the problem had to do with the new version and how it was processing and loading the data then I should see a similar problem in all the processes and none of them should appear to be reconciled. But I now have four processes that reconcile. If the problem were related to how the commands to retrieve the data were being interpreted, because that too is a new version, then when that was tested there should have been differences in the data coming back. While there were, it was numerical data that was statistically insignificant.
So what if the test system isn’t up to date and it is what’s wrong and not the new version? Annoying, but what if?
A comparison of the version against the production system and it turns out that the segment that is incorrect against the test system is correct against production. It also turns out that there is a segment of production that is incorrect against the new version. This is a historical segment and could be accounted for by a couple of things that have changed in the processing. However, one of those remaining processes reconciled completely against the production system.
For my situation, after a few more tests as confirmation, this means that the new version is working as expected and I can move on to the next step in the testing for upgrade process. In general what it means is that sometimes our assumptions about our baseline information can be wrong.
Posted in 
Tags:
