Quality Frog Questioning Software. Ben Simo's software testing blog. Context-driven software testing.

Statistics Unique Visitors: Total Unique Visitors: Visitors Out: Total Visitors Out: 45 167 3021 3021

Software Development Life Cycle Explained
2007-06-24 08:42:00
A couple weeks back, there was some discussion on the software-testing group about the use of the term "SDLC" on resumes. (Matt Heusser posted some excerpts from this conversation here.) My warning flags go up when people claim to have a "full understanding" of the SDLC. I sometimes see this as an indication that someone may not be as experienced as they claim. The "SDLC" will vary from one company to another and even one project to another. SDLC is a process documentation term -- and there are many differing processes used to develop software. Its not how people talk in the real world.I recently re-stumbled upon a description of the SDLC that seems to be fairly common across many companies and projects. Perhaps this is what people are referring to when they claim full knowledge of the SDLC. :)Software doesn’t just appear on the shelves by magic. That program shrink-wrapped inside the box along with the indecipherable manual and 12-paragraph disclaimer notice actually came to you by ...
Development  Life  Software  Software Development 

How many load generators do I need?
2007-06-23 19:55:00
How many load generators do I need to run a [insert number here] user load test on a web application?I am often asked how many load generators are needed for a load test with a certain number of simulated users.My answer: It depends.It depends on the system under test. It depends on your test tool. It depends on your specific script. It depends on your load generation hardware.There is no straight forward answer to this question. There is no formula that can be used to extrapolate an answer. There is no one-size-fits-all rule of thumb. Some tool vendors will attempt to provide an answer, but they are wrong. I once spent a half hour arguing with a tool vendor support representative that claimed that I could run 200, and no more than 200, simulated users per load generator regardless of what those simulated users did or what hardware I used to host them. I had successfully simulated 1200 users with this tool for one script but could not simulate 50 users with another script. The number I ...

How many load generators do I need?
2007-06-23 19:55:00
How many load generators do I need to run a [insert number here] user load test on a web application?I am often asked how many load generators are needed for a load test with a certain number of simulated users.My answer: It depends.It depends on the system under test. It depends on your test tool. It depends on your specific script. It depends on your load generation hardware.There is no straight forward answer to this question. There is no formula that can be used to extrapolate an answer. There is no one-size-fits-all rule of thumb. Some tool vendors will attempt to provide an answer, but they are wrong. I once spent a half hour arguing with a tool vendor support representative that claimed that I could run 200, and no more than 200, simulated users per load generator regardless of what those simulated users did or what hardware I used to host them. I had successfully simulated 1200 users with this tool for one script but could not simulate 50 users with another script. The number I ...

How many load generators do I need?
2007-06-23 19:55:00
How many load generators do I need to run a [insert number here] user load test on a web application?I am often asked how many load generators are needed for a load test with a certain number of simulated users.My answer: It depends.It depends on the system under test. It depends on your test tool. It depends on your specific script. It depends on your load generation hardware.There is no straight forward answer to this question. There is no formula that can be used to extrapolate an answer. There is no one-size-fits-all rule of thumb. Some tool vendors will attempt to provide an answer, but they are wrong. I once spent a half hour arguing with a tool vendor support representative that claimed that I could run 200, and no more than 200, simulated users per load generator regardless of what those simulated users did or what hardware I used to host them. I had successfully simulated 1200 users with this tool for one script but could not simulate 50 users with another script. The number I ...

Marching Through Hell
2007-06-21 08:01:00
There is a Churchill quote about when you're marching through hell, just keep marching. Well, that's basically what we're doing. We need to slow this down and get it right.- Charles BurbridgeCFO, Los Angeles Unified School DistrictA couple weeks ago, the Los Angeles Unified School District's new computer system incorrectly paid over 32,000 employees. That's almost a third of the district's employees. And the trouble started this past January.Glitches in Los Angeles Payroll System Spark Furor, Education WeekL.A. Unified payroll snafu's costs soaring, L.A. TimesI've been involved in some software projects with a "hell" phase in the SDLC. I've worked on projects that failed. I've been involved with projects that went millions over budget. However, I'm thankful that I have never worked on a project for which tens of thousands of production issues were reported in a handful of months.Through the replacement of our aging financial, human resources, payroll andprocurement syst ...
Hell 

Marching Through Hell
2007-06-21 08:01:00
There is a Churchill quote about when you're marching through hell, just keep marching. Well, that's basically what we're doing. We need to slow this down and get it right.- Charles BurbridgeCFO, Los Angeles Unified School DistrictA couple weeks ago, the Los Angeles Unified School District's new computer system incorrectly paid over 32,000 employees. That's almost a third of the district's employees. And the trouble started this past January.Glitches in Los Angeles Payroll System Spark Furor, Education WeekL.A. Unified payroll snafu's costs soaring, L.A. TimesI've been involved in some software projects with a "hell" phase in the SDLC. I've worked on projects that failed. I've been involved with projects that went millions over budget. However, I'm thankful that I have never worked on a project for which tens of thousands of production issues were reported in a handful of months.Through the replacement of our aging financial, human resources, payroll andprocurement systems, the Di ...
Hell 

Marching Through Hell
2007-06-21 08:01:00
There is a Churchill quote about when you're marching through hell, just keep marching. Well, that's basically what we're doing. We need to slow this down and get it right.- Charles BurbridgeCFO, Los Angeles Unified School DistrictA couple weeks ago, the Los Angeles Unified School District's new computer system incorrectly paid over 32,000 employees. That's almost a third of the district's employees. And the trouble started this past January.Glitches in Los Angeles Payroll System Spark Furor, Education WeekL.A. Unified payroll snafu's costs soaring, L.A. TimesI've been involved in some software projects with a "hell" phase in the SDLC. I've worked on projects that failed. I've been involved with projects that went millions over budget. However, I'm thankful that I have never worked on a project for which tens of thousands of production issues were reported in a handful of months.Through the replacement of our aging financial, human resources, payroll andprocurement syst ...
Hell 

Crapper Automation
2007-06-20 04:37:00
Mechanization best serves mediocrity.- Frank Lloyd WrightAutomation in public restrooms is becoming commonplace. The restrooms at my workplace have recently been remodeled and automated. The lights turn on and off automatically. The toilets flush automatically. Soap dispenses automatically. Faucets turn on and off automatically. Paper towels dispense automatically. All of the above appear to be good candidates for automation. Automation in the restroom is supposed to improve cleanliness, reduce maintenance, and cut supply costs. However, I am not certain that this is true in implementation. I have found the following bugs in our newly automated water closets.I have to walk about ten feet into the restroom in the dark before the lights come on.It has been reported that if one spends too much time seated on the throne, the lights will turn off leaving the occupant to finish their business in the dark.The paper towel dispensers spit out towels when someone walks by the dispenser.T ...
Automation 

Crapper Automation
2007-06-20 04:37:00
Mechanization best serves mediocrity.- Frank Lloyd WrightAutomation in public restrooms is becoming commonplace. The restrooms at my workplace have recently been remodeled and automated. The lights turn on and off automatically. The toilets flush automatically. Soap dispenses automatically. Faucets turn on and off automatically. Paper towels dispense automatically. All of the above appear to be good candidates for automation. Automation in the restroom is supposed to improve cleanliness, reduce maintenance, and cut supply costs. However, I am not certain that this is true in implementation. I have found the following bugs in our newly automated water closets.I have to walk about ten feet into the restroom in the dark before the lights come on.It has been reported that if one spends too much time seated on the throne, the lights will turn off leaving the occupant to finish their business in the dark.The paper towel dispensers spit out towels when someone walks by the dispenser.T ...
Automation 

Crapper Automation
2007-06-20 04:37:00
Mechanization best serves mediocrity.- Frank Lloyd WrightAutomation in public restrooms is becoming commonplace. The restrooms at my workplace have recently been remodeled and automated. The lights turn on and off automatically. The toilets flush automatically. Soap dispenses automatically. Faucets turn on and off automatically. Paper towels dispense automatically. All of the above appear to be good candidates for automation. Automation in the restroom is supposed to improve cleanliness, reduce maintenance, and cut supply costs. However, I am not certain that this is true in implementation. I have found the following bugs in our newly automated water closets.I have to walk about ten feet into the restroom in the dark before the lights come on.It has been reported that if one spends too much time seated on the throne, the lights will turn off leaving the occupant to finish their business in the dark.The paper towel dispensers spit out towels when someone walks by the dispenser.T ...
Automation 

Installing Windows Vista
2007-06-16 23:42:00
I have not yet made the jump to Windows Vista on any of my personal machines.We have installed Vista on some machines in the test lab where I work. When we started playing with a Vista beta, Vista was not allowed on our network but our customers were using Vista; demanding that we test our products on Vista. There were numerous technical and political battles that had to be fought to get Vista installed in our test lab.We quickly discovered that Microsoft's minimum system requirements are just the minimum to install the OS. I can't imagine any user being happy with Vista on a machine that just met the minimum requirements.We encountered incompatible DVD drives from a major PC manufacturer. We followed Microsoft's instructions for copying the DVD to a hard drive for installation from the hard drive instead of the DVD only to have the Vista install inform us that it cannot be installed as Microsoft instructed.We've had to call Microsoft for permission to reinstall failed installa ...
Vista  Windows  Windows Vista 

Installing Windows Vista
2007-06-16 23:42:00
I have not yet made the jump to Windows Vista on any of my personal machines.We have installed Vista on some machines in the test lab where I work. When we started playing with a Vista beta, Vista was not allowed on our network but our customers were using Vista; demanding that we test our products on Vista. There were numerous technical and political battles that had to be fought to get Vista installed in our test lab.We quickly discovered that Microsoft's minimum system requirements are just the minimum to install the OS. I can't imagine any user being happy with Vista on a machine that just met the minimum requirements.We encountered incompatible DVD drives from a major PC manufacturer. We followed Microsoft's instructions for copying the DVD to a hard drive for installation from the hard drive instead of the DVD only to have the Vista install inform us that it cannot be installed as Microsoft instructed.We've had to call Microsoft for permission to reinstall failed installation ...
Vista  Windows  Windows Vista 

Installing Windows Vista
2007-06-16 23:42:00
I have not yet made the jump to Windows Vista on any of my personal machines.We have installed Vista on some machines in the test lab where I work. When we started playing with a Vista beta, Vista was not allowed on our network but our customers were using Vista; demanding that we test our products on Vista. There were numerous technical and political battles that had to be fought to get Vista installed in our test lab.We quickly discovered that Microsoft's minimum system requirements are just the minimum to install the OS. I can't imagine any user being happy with Vista on a machine that just met the minimum requirements.We encountered incompatible DVD drives from a major PC manufacturer. We followed Microsoft's instructions for copying the DVD to a hard drive for installation from the hard drive instead of the DVD only to have the Vista install inform us that it cannot be installed as Microsoft instructed.We've had to call Microsoft for permission to reinstall failed installa ...
Vista  Windows  Windows Vista 

Modeling the Windows Calculator: Part 2
2007-06-16 01:28:00
Adding Basic ValidationsIn the previous post, I created a simple model for starting and stopping the Windows calculator, and for switching between standard and scientific modes. I then created the code needed to execute that test and ran a test that hit each of the defined actions once.As the next step, I reran the test with the MBTE configured to capture GUI object information as it executes. This created a checkpoint table that I then ran through a script that removes duplicates and combines rows that are the same for multiple states. I also manually reviewed this table to verify that the reported results are as I expected. I made some tweaks to the table based on my expectations. I can then use this checkpoint table as input for the next test execution. You may view the edited checkpoint file using the link below.The checkpoint table is one of two table formats that I use for defining test oracles. I call the other format a state table. The state tables contain one validatio ...
Calculator  Modeling  Windows 

Modeling the Windows Calculator: Part 2
2007-06-16 01:28:00
Adding Basic ValidationsIn the previous post, I created a simple model for starting and stopping the Windows calculator, and for switching between standard and scientific modes. I then created the code needed to execute that test and ran a test that hit each of the defined actions once.As the next step, I reran the test with the MBTE configured to capture GUI object information as it executes. This created a checkpoint table that I then ran through a script that removes duplicates and combines rows that are the same for multiple states. I also manually reviewed this table to verify that the reported results are as I expected. I made some tweaks to the table based on my expectations. I can then use this checkpoint table as input for the next test execution. You may view the edited checkpoint file using the link below.The checkpoint table is one of two table formats that I use for defining test oracles. I call the other format a state table. The state tables contain one validatio ...
Calculator  Modeling  Windows 

Modeling the Windows Calculator: Part 2
2007-06-16 01:28:00
Adding Basic ValidationsIn the previous post, I created a simple model for starting and stopping the Windows calculator, and for switching between standard and scientific modes. I then created the code needed to execute that test and ran a test that hit each of the defined actions once.As the next step, I reran the test with the MBTE configured to capture GUI object information as it executes. This created a checkpoint table that I then ran through a script that removes duplicates and combines rows that are the same for multiple states. I also manually reviewed this table to verify that the reported results are as I expected. I made some tweaks to the table based on my expectations. I can then use this checkpoint table as input for the next test execution. You may view the edited checkpoint file using the link below.The checkpoint table is one of two table formats that I use for defining test oracles. I call the other format a state table. The state tables contain one validatio ...
Calculator  Modeling  Windows 

Modeling The Windows Calculator: Part 1
2007-06-15 02:57:00
I have received a number of requests for some sample models. Based on a question I received a couple weeks ago, I'd like to create a test model for the Windows Calculator. The Windows calculator contains some things that are very simple to model as a state machine (such as switching between standard and scientific modes) and other things that do not have clear distinguishable states (such as performing the actual calculations).I plan to model the calculator a piece at a time in a series of blog posts. I welcome your input.I will start by modeling the obvious states that I see in the Calculator's user interface.At the highest level, I can partition the Calculator's behavior into two states: running or not running. Next, the calculator has two major modes of operation: standard and scientific. After a little experimentation, I see that if I stop the calculator it will return to the previous mode when it is restarted. These transactions can be modeled as follows:calc.standard -> calc. ...
Calculator  Modeling  Windows 

Modeling The Windows Calculator: Part 1
2007-06-15 02:57:00
I have received a number of requests for some sample models. Based on a question I received a couple weeks ago, I'd like to create a test model for the Windows Calculator. The Windows calculator contains some things that are very simple to model as a state machine (such as switching between standard and scientific modes) and other things that do not have clear distinguishable states (such as performing the actual calculations).I plan to model the calculator a piece at a time in a series of blog posts. I welcome your input.I will start by modeling the obvious states that I see in the Calculator's user interface.At the highest level, I can partition the Calculator's behavior into two states: running or not running. Next, the calculator has two major modes of operation: standard and scientific. After a little experimentation, I see that if I stop the calculator it will return to the previous mode when it is restarted. These transactions can be modeled as follows:calc.standard -> calc.sci ...
Calculator  Modeling  Windows 

Modeling The Windows Calculator: Part 1
2007-06-15 02:57:00
I have received a number of requests for some sample models. Based on a question I received a couple weeks ago, I'd like to create a test model for the Windows Calculator. The Windows calculator contains some things that are very simple to model as a state machine (such as switching between standard and scientific modes) and other things that do not have clear distinguishable states (such as performing the actual calculations).I plan to model the calculator a piece at a time in a series of blog posts. I welcome your input.I will start by modeling the obvious states that I see in the Calculator's user interface.At the highest level, I can partition the Calculator's behavior into two states: running or not running. Next, the calculator has two major modes of operation: standard and scientific. After a little experimentation, I see that if I stop the calculator it will return to the previous mode when it is restarted. These transactions can be modeled as follows:calc.standard -> calc. ...
Calculator  Modeling  Windows 

Sapience In The Age Of Automation
2007-06-12 11:30:00
Society has unwittingly fallen into a machine-centered orientation to life, one that emphasizes the needs of technology over those of people, thereby forcing people into a supporting role, one for which we are most unsuited. Worse, the machine-centered viewpoint compares people to machines and finds us wanting, incapable of precise, repetitive, accurate actions. Although this is a natural comparison, and one that pervades society, it is also a most inappropriate view of people. It emphasizes tasks and activities that we should not be performing and ignores our primary skills and attributes -- activities that are done poorly, if at all, by machines. When we take the machine-centered point of view, we judge things on artificial, mechanical merits. The result is continuing estrangement between humans and machines, continuing and growing frustration with technology and with the pace ans stress of a technologically centered life.- Donald A. NormanThings That Make Us Smart:Defending Human At ...
Automation 

Sapience In The Age Of Automation
2007-06-12 11:30:00
Society has unwittingly fallen into a machine-centered orientation to life, one that emphasizes the needs of technology over those of people, thereby forcing people into a supporting role, one for which we are most unsuited. Worse, the machine-centered viewpoint compares people to machines and finds us wanting, incapable of precise, repetitive, accurate actions. Although this is a natural comparison, and one that pervades society, it is also a most inappropriate view of people. It emphasizes tasks and activities that we should not be performing and ignores our primary skills and attributes -- activities that are done poorly, if at all, by machines. When we take the machine-centered point of view, we judge things on artificial, mechanical merits. The result is continuing estrangement between humans and machines, continuing and growing frustration with technology and with the pace ans stress of a technologically centered life.- Donald A. NormanThings That Make Us Smart:Defending Human At ...
Automation 

Sapience In The Age Of Automation
2007-06-12 11:30:00
Society has unwittingly fallen into a machine-centered orientation to life, one that emphasizes the needs of technology over those of people, thereby forcing people into a supporting role, one for which we are most unsuited. Worse, the machine-centered viewpoint compares people to machines and finds us wanting, incapable of precise, repetitive, accurate actions. Although this is a natural comparison, and one that pervades society, it is also a most inappropriate view of people. It emphasizes tasks and activities that we should not be performing and ignores our primary skills and attributes -- activities that are done poorly, if at all, by machines. When we take the machine-centered point of view, we judge things on artificial, mechanical merits. The result is continuing estrangement between humans and machines, continuing and growing frustration with technology and with the pace ans stress of a technologically centered life.- Donald A. NormanThings That Make Us Smart:Defending Human At ...
Automation 

Performance: Investigate Early, Validate Last
2007-06-11 00:47:00
Performance and load testing is often viewed as something that has to be done late in the development cycle with a goal of validating that performance meets predefined requirements. The problem with this is that fixing performance problems can require major changes to the architecture of a system. When we do performance testing last, it is often too late or too expensive to fix anything.The truth is that performance testing does not need to happen last. Load test scripting is often easier if we wait until the end, but should we sacrifice quality just to make testing easier?Scott Barber divides performance testing requirements and goals into the following three categories:Scalability -- extremely technicalStability -- mostly technicalSpeed -- fuzzy: some technical, other usabilitySpeed is where things get fuzzy. Some speed requirements are quite definable, quantifiable and technical; others are not.- Scott BarberScott says that hard measurable requirements can usually be defined for sca ...
Performance 

Performance: Investigate Early, Validate Last
2007-06-11 00:47:00
Performance and load testing is often viewed as something that has to be done late in the development cycle with a goal of validating that performance meets predefined requirements. The problem with this is that fixing performance problems can require major changes to the architecture of a system. When we do performance testing last, it is often too late or too expensive to fix anything.The truth is that performance testing does not need to happen last. Load test scripting is often easier if we wait until the end, but should we sacrifice quality just to make testing easier?Scott Barber divides performance testing requirements and goals into the following three categories:Scalability -- extremely technicalStability -- mostly technicalSpeed -- fuzzy: some technical, other usabilitySpeed is where things get fuzzy. Some speed requirements are quite definable, quantifiable and technical; others are not.- Scott BarberScott says that hard measurable requirements can usually be defined for sca ...
Performance 

Performance: Investigate Early, Validate Last
2007-06-11 00:47:00
Performance and load testing is often viewed as something that has to be done late in the development cycle with a goal of validating that performance meets predefined requirements. The problem with this is that fixing performance problems can require major changes to the architecture of a system. When we do performance testing last, it is often too late or too expensive to fix anything.The truth is that performance testing does not need to happen last. Load test scripting is often easier if we wait until the end, but should we sacrifice quality just to make testing easier?Scott Barber divides performance testing requirements and goals into the following three categories:Scalability -- extremely technicalStability -- mostly technicalSpeed -- fuzzy: some technical, other usabilitySpeed is where things get fuzzy. Some speed requirements are quite definable, quantifiable and technical; others are not.- Scott BarberScott says that hard measurable requirements can usually be defined for sca ...
Performance 

Free Internet: Flushing The Web
2007-06-09 16:45:00
Although we understand that there's a lot of crap on the web, we also believe strongly in providing equal opportunity access to all our users. - Google The folks at Google are working hard to provide innovative services to the world. A couple months ago, they announced the Beta release of their free in-home wireless broadband service: Google TiSP.Google TiSP promises free, fast, and easy to install wireless broadband service. Sorry, this services is only available in the United States and Canada.Check it out here.What is TiSP?Toilet Internet Service Provider When things go wrong with TiSP, they go very, very wrong. Let's leave it at that.Happy Flushing! Now where'd I put my WiiHelm? Ben Simo QualityFrog.com Questioning Software℠ ...
Internet  Web 

Free Internet: Flushing The Web
2007-06-09 16:45:00
Although we understand that there's a lot of crap on the web, we also believe strongly in providing equal opportunity access to all our users. - Google The folks at Google are working hard to provide innovative services to the world. A couple months ago, they announced the Beta release of their free in-home wireless broadband service: Google TiSP.Google TiSP promises free, fast, and easy to install wireless broadband service. Sorry, this services is only available in the United States and Canada.Check it out here.What is TiSP?Toilet Internet Service Provider When things go wrong with TiSP, they go very, very wrong. Let's leave it at that.Happy Flushing! Now where'd I put my WiiHelm? Ben Simo QualityFrog.com Questioning Software℠ ...
Internet  Web 

Free Internet: Flushing The Web
2007-06-09 16:45:00
Although we understand that there's a lot of crap on the web, we also believe strongly in providing equal opportunity access to all our users. - Google The folks at Google are working hard to provide innovative services to the world. A couple months ago, they announced the Beta release of their free in-home wireless broadband service: Google TiSP.Google TiSP promises free, fast, and easy to install wireless broadband service. Sorry, this services is only available in the United States and Canada.Check it out here.What is TiSP?Toilet Internet Service Provider When things go wrong with TiSP, they go very, very wrong. Let's leave it at that.Happy Flushing! Now where'd I put my WiiHelm? Ben Simo QualityFrog.com Questioning Software℠ ...
Internet  Web 

Bad Messages
2007-06-09 02:30:00
I regularly spend a great deal of time tracking down the root cause of software errors -- both on and off the job. Much of the investigation effort could have been avoided if I were not presented with incomplete or incorrect error messages. The text of error messages appears to be commonly overlooked by software developers and testers.One of my early test automation development tasks was to fix the error messages in a test tool. The tool was used to validate the structure of data exchanged between computer systems. In nearly all cases, this tool displayed an error dialog window stating that the data was not valid. This tool gave the user (a tester) no information about what was wrong with the data. The lack of detail in these messages required that the user manually examine the data -- bit by bit -- with a protocol analyzer. This turned investigation of errors reported by an automation tool into a tedious manual task. The tool encountered a problem but did not report the source of the ...

Bad Messages
2007-06-09 02:30:00
I regularly spend a great deal of time tracking down the root cause of software errors -- both on and off the job. Much of the investigation effort could have been avoided if I were not presented with incomplete or incorrect error messages. The text of error messages appears to be commonly overlooked by software developers and testers.One of my early test automation development tasks was to fix the error messages in a test tool. The tool was used to validate the structure of data exchanged between computer systems. In nearly all cases, this tool displayed an error dialog window stating that the data was not valid. This tool gave the user (a tester) no information about what was wrong with the data. The lack of detail in these messages required that the user manually examine the data -- bit by bit -- with a protocol analyzer. This turned investigation of errors reported by an automation tool into a tedious manual task. The tool encountered a problem but did not report the source of the ...