I stumbled across tomorrow's SQL Server 2005 and XML webcast almost by accident.
Further information is available at MSDN Webcast: Optimizing and Troubleshooting XML Applications at the Server (Level 300). It's at 10:00 Pacific time.
The SQL Server 2005 and XML webcasts from Michael Rys and Shankar Pal have been good listening.
This post dissects out factors that can play a part in achieving effective and meaningful bug reports and product suggestions. It seems to me that several of the factors could benefit product quality, if a more proactive approach were taken by Microsoft.
The following lists the factors which I see as potentially impacting whether an individual or group become effective product testers. The current process seems ad hoc and fairly random, perhaps because nobody is responsible for action on these factors. If somebody is supposed to be actively addressing these issues they are remarkably less visible than I would hope and expect.
- Get the willing and motivated tester(s) in the program.
- Get the product to the tester.
- Get the tester to install the product.
- Get the tester to use the product.
- Get the tester to submit a first bug report.
- Get the tester to submit a first quality bug report.
- Get the tester of a single bug report to submit multiple bug reports.
Get the willing and motivated tester(s) in the program. - Fundamental to this is getting the willing tester connected to the product that needs testing. It seems to me that the current system of recruiting beta testers is inefficient. I remember trying, via BetaPlace, several times to get on beta programs that interested me. For whatever reason(s), the automated system at Beta Place failed to connect me to the product(s) I wanted to test. I know that others have had the same frustrating experience. I don't now have to fill in the survey on BetaPlace very often but it stills seems to be using the banal "one size fits all" form. Perhaps a more customised form might not achieve better recruitment. But it seems to me that it must be worth a try. More proactive recruitment would perhaps help too.
Get the product to the tester. - Downloads work, even if they are excruciatingly slow for downloads of between 0.5GB and 3.7GB. Improvements in bandwidth when a CTP is released would certainly be welcome.
Get the tester to install the product. - If the tester can't successfully install the product the amount of testing, other than of the installer, that gets done is zero. Therefore I see it as crucial that the installer for CTPs work 100% or very close to it. I have sympathy for installer developers because the bar for "acceptable quality" is very high. On the other hand I also have sympathy with users who waste hours trying to install a product because the installer doesn't do the necessary. One of the problem areas here is the lack of timely newsgroup support for users with setup problems. Sometimes individuals are highly visible in supporting users. When they are visible they are usually very effective and helpful. At least as often the support for install is delayed or absent. It is no surprise, in that setting, that potential beta testers decide that they have better things to do with their time. A response several days later is unlikely to keep a potential tester motivated.
Get the tester to use the product. - This seems to be an entirely passive process. I don't recall any encouragement to get started. A beta or CTP appears and then ... nothing. Simple things like more effectively communicating new pieces of functionality could help. Sometimes that information is present but it is buried in lengthy poorly structured and outdated readme documents. This does not make a helpful contribution to getting the tester actively using the program under test. The information to stimulate interest needs to be presented clearly to beta testers. Get their interest and you are more likely to get their activity and cooperation.
Get the tester to submit a first bug report. - My guess is that maybe 90% of beta testers in any one program never submit a bug. If the bug reporting rate at the MSDN Product Feedback Center is anything to go by the problem may be significantly more serious than that. Giving product to beta testers who never report a bug may have marketing seeding value but it makes no contribution to product quality. Sitting back and waiting for reports, which seems to be the historical norm, is surely an approach that can be bettered. A basic action would be to proactively publicise the Product Feedback Center. I haven't seen it happen. Surely a little initiative would suggest ways to get a substantial number of beta testers from the "never reported a bug" category to the active reporter of a bug?
Get the tester to submit a first quality bug report. - Getting a beta tester to submit a first bug is a useful step. Better still is to get beta testers to submit quality bug reports. There is remarkably little information available to help beta testers know what is wanted or not. That seems to be an opportunity missed. I know that I have asked explicitly about what types of reports were sought and have received no answer. If, as an MVP, such a question is ignored, what chance do the 90% have?
Get the tester of a single bug report to submit multiple bug reports. - If even a fraction of the "never submitted a bug" people could be encouraged to report a bug that would be progress. But I suggest that transforming "reporters of a single bug" into "reporters of multiple bugs" is also important. Again, oddly, the process seems to be passive, as if the encouragement of submission of multiple quality bugs were a matter of indifference to Microsoft. I find that deeply puzzling. Added to this odd inaction are the active disincentives provided by flawed reporting tools such as the MSDN Product Feedback Center. See my post on that topic at The MSDN Product Feedback Center is broken and needs to be fixed. If the tools for reporting get in the way and prove irritating and inefficient to use it can be no surpise that many testers don't bother to use them.
I suspect that a key factor in these omissions is that issues such as these are not the responsibility of any one individual. If there was a person or persons responsible for making progress on these issues surely there would be progress. Hopefully, as a result, the testing process would be more thorough and effective and we, as customers, would have better quality products to use.
One of the issues which needs to be examined when considering the value to Microsoft of external product testing is where the value in a product lies. It is unrealistic, given commercial realities, to expect Microsoft to assign significant resources to support effective external testing where no return can be expected. If external testing provides the most useful testing of the value-generating aspects of a product ... or provides testing of those aspects in ways not susceptible to automated testing ... it is an issue that Microsoft cannot afford to ignore.
As mentioned in Part 1, Towards more effective and more customer-centric CTP and beta testing - Part 1, automated testing of SQL Server 2005 and other complex products is essential. But it is not sufficient. I am concentrating on the external testing that I believe is necessary and which, I believe, needs to be improved.
The quality of the core engine of SQL Server 2005 is impressive, yet Microsoft is prepared to give it away due, no doubt, to the presence in the marketplace of "free" products such as MySQL. So, although the database engine itself may be the basis for the value in the product, the value does not now lie directly in the basic database engine functionality.
Among the issues which are particularly appropriate for external testing and which contribute to the value of the product are usability and manageability.
If, for example, a new version exhibits substandard usability and/or manageability then, at the risk of stating the obvious, there is potential for commercial damage.
Usability is important at several levels. For example, at the user interface level the following questions are examples of those that can arise. Does the product behave in the way that the customer expects? Does it do what the customer needs? These broad questions break down into myriad individual choices or perceptions.
Microsoft employees, as a generalisation, don't think as users do. Any assumptions or beliefs about how customers will respond to new features or changes in existing features must be tested externally.
Manageability is another potentially important commercial distinguishing factor. If a product can cut down staff time spent on activities repeated multiple times per day or per week ... compared to previous versions or to competitor products ... the cost savings to the customer can be substantial. The potential for cost savings adds value to the product.
Testing of usability and manageability requires exposure of the product to external testers. The question then arises as to how those external testers can be persuaded to test effectively and to provide useful feedback which enables the usability and manageability of the product to be improved. Issues which impact on effective external testing are the subject of Part 3.
Installing Visual Studio Standard Beta 2 on the same machine as SQL Server 2005 April CTP seems able to alter settings so that Report Manager does not work. Windows Server 2003 RTM.
Running the following command from the command line,
solves the problem.
Installing Visual Studio Team System Beta 2 on Windows 2003 SP1 did not have the same problem when attempting to run Report Manager.
Report Builder on April CTP failed on first use on two Windows 2003 RTM machines.
The problem was easily solved. Running Report Builder a second time, while making no other change, resulted in it running successfully.
Report Builder ran successfully first time on a Windows 2003 SP1 machine.
If you are interested in testing Express Manager, then April CTP doesn't provide an opportunity to test any new Express Manager features.
It appears that the development team is currently focussing on getting SQL Server Management Studio "just so". With finite developer time that means that Express Manager will continue with its fairly basic functionality at least during the lifetime of April CTP.
It was interesting to read yesterday's press release, Microsoft Delivers Latest Visual Studio 2005 and SQL Server 2005 Releases and other material about Microsoft's commitment to continuing testing of SQL Server 2005 using Community Technology Previews.
For me, that was both good and expected news. I see CTPs as a way of Microsoft attempting more effective product development in an increasingly complex environment. In my view, a crucial component of such effective product development is customer-centric product testing.
The increasing complexity of software and potentially exponential increase of usage scenarios means that a conventional approach to beta testing is simply inadequate.
If you watched some of the Channel 9 videos about the SQL Server 2005 team you will have seen the banks of machines on which automated testing takes place. Necessary but not sufficient, particularly as the commercial value of database products moves away from the database engine towards human and usability issues such as manageability. To effectively test those commercially crucial aspects you need human beings doing the things that human beings do.
All this is taking place against the background of an ongoing culture change in Microsoft. Microsoft needs the community and, arguably, has been slow in recognising that and finding the most effective ways to respond to that changing climate. The existence of CTPs is itself a symptom or evidence of that cultural shift. I expect to see the cultural shift move much further ... because I believe that it is in Microsoft's long-term commercial interests that it does move significantly further in a customer-centric direction. Microsoft employees will need, in my expectation, to develop new aspects of customer awareness. For some that will be a demanding change.
Among the issues I hope to discuss in future posts on this topic are the following:
- Where does product value lie?
- Customers have finite and limited time
- Communication with testers
- Providing incentives to testers
- Maintaining interest of testers and avoiding disincentives
- Efficient bug reporting mechanisms
- Newsgroup support
The approach I will take is based on an assumption that Microsoft has a commercial imperative to produce the best software possible in a competitive marketplace. To do that it needs to carry out the best pre-release product testing possible. One aspect of such testing is effective customer-centric product testing. To do that effectively any CTP program has to make efficient and effective use of customer time. Why? Because customer time is finite. If you use it ineffectively or waste it then it's gone and the quality of the final product may suffer. Time is finite, so it's imperative how to look at how it can best be used.
In my opinion, there is signficant scope for improvement on several aspects and the future posts will explore some existing issues and how improvement can be made. More effective and more customer-centric product testing is something I would expect to lead to improved Microsoft products. Both the community and company benefit from that.
It is already on BetaPlace. Hopefully it will be on MSDN soon.
April CTP of SQL Express failed to install on Windows XP SP1 for me. See my lengthy post at April CTP SQL Express Install Saga - Serious Setup Bug? for details.
Very disappointing.
I subsequently tried a further uninstall then an install of Client Tools only as a workaround, someone having suggested that Shared Memory had been disabled by default on a Windows 2003 install which he had carried out. The client tools installed successfully (Shared Memory wasn't disabled in SQL Configuration Manager) but the subsequent attempt at installing April CTP of SQL Server Express failed at the same point with the same error dialog.
I have April CTP runing happily with Visual Studio Beta 2 on Windows 2003. However, since many people who are likely to want to explore April CTP may be using Windows XP SP1, it would be good if there is a workaround.
SQL Server 2005 Express Edition is available for download, SQL Server Express Edition April CTP.
Currently only one link to the download itself, this one, is working.
Read the uninstall instructions, Uninstalling Express Editions, carefully. The order in which components of .NET Framework 2.0, SQL Server and Visual Studio from releases are uninstalled is crucial to a clean uninstall.
The SQL Server 2005 Developer Edition isn't visible on either BetaPlace or MSDN yet.
Since Visual Studio 2005 Beta 2 is now available and SQL Server 2005 Express April CTP is likely to be available soon, I have updated my advice on installation for users and potential users of SQL Server 2005 Express. This is written in order to help them avoid commonly occurring problems with uninstallation and installation, due to version incompatibility of essential components.
The updated advice is located at Choosing a version of SQL Server 2005 Express (Updated for Visual Studio Beta 2).
MSDN subscribers can now download Visual Studio Beta 2 including, I assume, a new version of SQL Server 2005 Express. Since the downloads are presented as single DVD images it will be several hours before I can confirm personally that SQL Server Express is included in the download.
There is currently no sign of the eagerly awaited of April CTP of SQL Server 2005 Developer Edition on either BetaPlace or on MSDN.
It seems that April CTP is very close. According to Mark Bower, a Microsoft consultant in the UK, Visual Studio Beta 2 and SQL Server 2005 April CTP are being prepped for download, Visual Studio 2005 Beta 2 - coming to a download site near you!.
I wish I could avoid this involuntary temptation to hold my breath!
Today there is the fourth in a series of ten SQL Server 2005 webcasts. Today's webcast is on the topic of security. There are many new aspects in security in SQL Server 2005. It's definitely a topic that you need to know about if you are going to be using SQL Server 2005.
The webcast takes place at 09:00 Pacific Time.
To register, navigate to TechNet Webcast: SQL Server 2005 Series (Part 4 of 10): Securing Your SQL Server (Level 200).
Thinking a few months ahead, I have been considering the issue of how best to display SQL Server 2005 Integration Services packages for teaching or training purposes. I particularly had screen shots in mind. Which layout of components will work best on screen or on paper in a teaching setting?
That led me on to ask how we can best lay out Integration Services components on screen to help a learner understand the process for which the displayed components are a metaphor.
So far the data on which to base any view is pretty limited. I have seen several of the SQL Server Integration Services team perform in webcasts and similar demos. The dominant approach seems to be top down, with a hint of left to right. I found the top down metaphor interesting but a little surprising too. Computer screens are wider horizontally than vertically so more information can be captured on a single screen shot with a more left to right approach. I did notice a bit of moderately frantic scrolling going in some webcasts. It seems to me that if we can keep that to a minimum the learning experience might be better.
The only other data point I have on this is my own approach. I tend to start top left and work down and across. It struck me that that is similar to how I write text in English. For me that is a pretty natural approach.
But is top down, left to right a natural approach in other cultures? Do Arabs and Jews exploring SSIS think in terms of starting from the right of the screen? Do Chinese think predominantly vertically?
I guess that for me, with a target audience likely to be predominantly English speaking that the top down, left to right approach makes most sense. Since, I guess, my audience will find that a natural visual metaphor.
I wonder if a semi-standard visual metaphor ... perhaps top-down, left to right ... adopted by Microsoft webcasters, trainers and book authors might help users more quickly and more effectively grasp what the visual components of an SSIS package represent.
On 13th April there is a webcast to take you beyond the basics in SQL Server 2005 Reporting Services, visit MSDN Webcast: Advanced Reporting with SQL Server 2005 Reporting Services (Level 200) for more details and to sign up.
Today there is a webcast on using XML Web Services to connect to SQL Server 2005, MSDN Webcast: Using Web Services To Connect to SQL Server 2005 from Your Smart Client Applications (Level 200).
If you were wondering why the blitz of SQL Server 2005 and Visual Studio 2005 webcasts in April then you need to be aware that Visual Studio Beta 2 and SQL Server April CTP are not far away (see Visual Studo 2005 Beta 2 and SQL Server 2005 Beta 2 webcasts in April 2005 and Next CTP available soon...).
The discussion among Donald Farmer and others about data quality, Business logic and data integration with SQL Server Integration Services has made me think a little about the hidden problems of data consistency (more precisely, inconsistency) that SQL Server 2005 Integration Services is going to bring to the surface in many enterprises which hitherto have, very possibly, not had to consider these issues. Or at least have not had to take them very seriously, outside the relatively narrow world of the DBA.
The problems of data inconsistency have been widespread. But, because of the lack of joined-up data, the problem has been, I suspect, poorly recognised and often ignored. When tools such as Integration Services enter wider use the problems are going to surface. And may bite.
Let me try to explain a little of what I mean, using a very simple example.
One of the purposes of normalisation in relational databases is to reduce or avoid the risk of data being inconsistent by holding a single piece of data only once. So far, so obvious I guess for many readers of SQLJunkies.com.
Now consider the simplified situation of customer information being held in a SQL Server database and in an Excel spreadsheet, the latter perhaps at departmental level. The information is about the same customers but has been created independently or has been maintained independently, perhaps having at one time had a common source.
A moment's thought will likely show you that we have returned (in a wider context) to the problem of data consistency (or potential inconsistency) which normalisation was intended to mitigate. There is nothing ... at least nothing very reliable ... to prevent the data in SQL Server and Excel being inconsistent.
What happens when Integration Services is used to source information from the Excel spreadsheet and the SQL Server table? It all depends on how much of the potential inconsistency has been created in practice.
Of course, the situation described is simplistic. In the real world the potential for data inconsistency as Integration Services allows us to move to a world of "joined-up data" is enormous.
How are such problems going to be solved? It seems to me that the problem can be attacked at various levels. I will briefly mention one proactive approach and one retroactive.
One proactive approach is to try to repeat the lessons of normalisation at a business or enterprise level by reducing or avoiding the duplication of data. Departmental data independence may come to be seen as a signficant disadvantage to a business's efficiency in handling data. If departmental Excel spreadsheets are to be discouraged or banned, those affected will demand an efficient substitute. Replication to a departmental instance of SQL Server Express? A SharePoint site? The possibilities are varied. ... It has crossed my mind that Microsoft Office servers (for Office 12?) mentioned in the rumour Web sites may be intended to address just such problems.
But what about the huge amounts of, for want of a better term, "non-joined-up data" that already exists? How can we improve its consistency? How can we do for semantics what XML has done for syntax? Can we ever achieve, even in a single enterprise, a "semantic web"? It's only recently that it has occurred to me that the arrival of Integration Services might make an enterprise-level semantic web a highly desirable project.
Michael Rys, Musings on XML, XQuery and more..., presented a webcast last night containing an overview of XML in SQL Server 2005. For me, one of the most interesting aspects was the brief glimpse of some of the new functionality in the April CTP.
It was almost midnight when I was watching the webcast and the only detail that now sticks in my mind is the addition, in April CTP, of a new XML namespace syntax. I won't attempt to reproduce it here, since I will almost certainly get some aspect of the new syntax wrong.
It's been several weeks since Microsoft first mentioned April CTP on the public newsgroups. If they are now using it in public demos, I guess it can't be too far away.
By the way, if you want to watch a recorded version it should be available in the next 24-48 hours at MSDN Webcast: Managing XML Data on the Database with SQL Server 2005 and Visual Studio 2005 (Level 300).