In the software development community automated test have been there more and more. I myself started coding in PHP without a single test and just ran the app like QA would have done to see if it worked. Over the past decade this has really changed for me. I’ve lived through a flux where everything needed to be tested through and through to only integration tests. Currently I think we all agree that automated tests are something that is kind of a requirement when talking about serious software development. Note; “serious” I’m not talking about hacks and one of experiments.
So now the question becomes; When are we too obsessive with our test? Or, when are we just not covering the code enough?
A; you should at least test! B; make sure you’ve covered all the important parts with unit tests. C; don’t shy away from integration test on different parts of your system. D; make sure system tests are there but don’t over do it! E; use TDD as it will change the way you code and improve your results. F; Use coverage tools to make sure you’re on the right track. But don’t see them as the ultimate answer.
Overall; just make sure you balance test coverage with an eye on keeping it DRY. Don’t just do one type of test, cover the different levels with a sensible test suite. If you’re using libraries or external services that your integration tests are sane and apply SoC where you know integrations are covered by the library itself. Make sure your not just writing test because you have to but write them as they can help you during development. And don’t forget, TDD/BDD is your friend!
I know testing is a subject that many do not comprehend completely so here is a summary of what types of testing you can do. Traditionally there are two categories of tests. White box testing involves intimate knowledge of the code. Basically White box testing tests of all the inner working of the code behave as expected. Black box testing is it’s counterpart by leaving the testing with just the outside of the mechanism to test. Black box testing focuses on the perceived functionality from the outside of the mechanism without caring about the internal workings. By using these two categories there are several testing strategies that a developer or QA (Quality Assurance) person could take.
Unit testing is considered the deepest form of testing and is a white box method that is almost exclusively used by developers. Developer create a function and specify the different scenarios for that function and test if the function returns the expected outcome for those scenarios.
Integration tests are created by either developers or QA staff to validate the interactions between systems. This may be black or white box depending on what level of knowledge is required to test the interactions between the systems. scenarios will be tested involving both systems.
System testing involves testing a system end to end. Because of this the QA department is most likely responsible for this type of testing and it’s a classic example of black box testing. Scenarios are most likely based on user stories and are driven by product design. Scenarios are complex and although in the last decade this has been automated more and more it is still being done by hand in many places. Technically it is the most complicated to do as the computer needs to act like it is a human.
Acceptance testing is the last stage for any product going out to market and is mostly done by QA but has also been done by UX people or even by marketing departments. It is the part where you sit down with a user and let them work with the system before it is shipped or released. Alpha and Beta programs/releases, users visiting your office to test locally or just handing it out to somebody outside of product development within the company are all ways to do it. It’s not something very technical though so I won’t be going into this any further beyond this point.
Beyond the type of testing for developers there is also the question; when? As a developer you could code it and then start to test it or you could turn it the other way around. TDD (Test Driven Development) is a methodology where a developer will write test before the code exists. It allows the developer to see if the code does as specified while coding it. I’m not going to talk about it too much as that’s another post on itself.
I talked briefly about how QA could and should do a lot of these tests. In practice, for smaller companies, the developers also carry a lot of that weight.
Unit testing’s larges advantage is that because it’s so low level a problem within a method can be easily identified and resolved. This low level targeting also is very useful when applying TDD as a developer. When working on a method you can just create the scenarios that fit the requirements for that specific method and then make your tests pass on a micro level one by one.
Optimizing code can also be done with relative ease as once the expected input and output are in the test the internals of the method can be changed quite easily without changing it’s workings.
Edge cases on a higher level can be a real bitch. For a simple scenario with 5 steps you could run into 4 or five edge cases on each level. Testing this all on a higher level would increase the scenario load a lot. By testing these edge case on the lower levels you keep the tests DRY (Don’t Repeat Yourself) while still keeping an eye on the possible issues.
Last but not least unit tests can change they way you code! WHAT!? Yes, by creating unit tests you define a set of functionality on such a level that the amount of tests will explode if your methods grow out of proportions. A method with over 20 lines of code will have tons of tests and it will become messy quickly. Unit tests set you to think about the structure of your code, applying a DRY mindset where you won’t use the same code twice in a method or even think again about creating that 4 level deep if-else statement. Main reason; It’s a bitch to test large methods like that and keep an overview.
Unit tests are only for testing units. Basically you do not want to test anything outside of the method. You would want to apply SoC (Separation of Concerns) to make sure you are only testing the method and not anything else. By doing so a method that goes to a database and returns a record you will have to stub out the database response and return a predicted response that is then processed further by the method. This makes for fast test as it’s just testing the internal logic of the method. But you have to make sure you have some sort of integration testing to make sure it still works with the database. This is especially important when systems are updated.
Unit test in no way guard against changes of other methods. Making changes to a system or doing some major refactoring will require you to thread lightly. But as long as you have integration tests in place that should be less of a problem. But only unit tests won’t cut it.
The core reason integration testing is done is to make sure systems and even layers within the same system work together properly. This allows for changes to be made easily and areas that need adapting to be identified within a heartbeat. Tests will fail if things change and these types of tests monitor these changes closely.
Because of their nature the tests are very useful when doing upgrades to systems. Like changing the version of a database or updating a framework that you’re using. These upgrades are normally extremely tricky but with these integration test you can find the problem areas, deprecations and other issues fairly easily making the process of upgrading doable.
Interestingly enough the only major disadvantage is that testing integration goes against SoC and creates problems related to that. Integration tests are more fragile than unit tests but therefore are also more useful. In certain areas however you would not want to test things. When using a pattern like active record while using a framework you wouldn’t like to test that integration. Frameworks like that have ample tests to make sure that it works correctly and if the API changes of that library your other test will very likely fail showing the issue. The main reasons for not testing these is because it’s not DRY and you can end up testing the same thing over and over again.
Also integration tests are slower than unit test as they need to pass through more levels of code. Making sure that you only test the correct set of interactions allows your test suite to stay as lean as possible while making sure you still have everything covered.
System testing is technically the last step into the automated testing puzzle. If you can automate this, do so! It’s more reliable than their human counterparts and are faster in doing so. These tests are the last safeguard against any issues that you might have in your system. There might be a lot of down-sides to these types of test but they can really help your quality control in the long run.
The biggest problem with automated system tests is that they are very fragile. If just a single element on a website changes it’s name or the styling is changed and the CSS class is replaced it could break. Especially in a system that is ever changing and evolving this could reduce development speed considerately.
Next to fragile system tests are also complex as scenarios on the highest level have a lot of steps. In many cases you would also like to avoid stubbing of methods to make sure it is a white box test. This will make your test less DRY but will ensure that the coverage is correct. Finding a balance with this is very difficult and I’ve seen horrible examples of test and implementations that went against everything you would think to be correct. But that’s just the world of system testing. Also a reason why this is something when done fully should be the responsibility of the QA department.
Because of the high level nature of these tests they run through the full stack of code. Next to that a user needs to be simulated and the app completely ran and build. This makes the test really slow compared to unit and integration tests. My advise is to not run these tests in a TDD environment but to run them before your commit and during your deployment.
Test coverage is the percentage your code is covered by test. There are a lot of tools out there that can monitor this and integrate with your test suite to determine what lines are hit during the tests and how many times they are hit. These coverage tools do have an impact on the speed of a run in your test suite. But this is worth it. Knowing what is covered and what is not becomes so much easier using a tool like this it’s worth the wait.
The level of coverage have been a fierce discussion that has been going on. Some say it has to be 100%. Other feel that there can be some leeway in that number. From my personal experience I know that 100% is not always possible in every setup. Some library configuration files or initializers are hard to test, especially when they contain different information for production compared to development or test environments. So personally I don’t feel 100% should be the holy grail. It should however cover all the systems code and you should strive for full coverage if you can.
In some cases you may need to take coverage with a grain of sand as well. It may be that the coverage you are getting back from your coverage library is high but the actual quality of that coverage is poor. I don’t feel that coverage in that way is the only thing you should look at. I could write a set of system test that would cover the whole system but it would be very fragile. The same goes for writing any one level of tests without the other as together they make sure not only the single methods work as they should but also the higher level functionality does as required.
I see coverage tools best used as a method to find gaps in your coverage. To make sure you have everything covered with enough tests, even the edge cases.
Most people think you should test but time and again I’ve heard people that have used arguments for and against it. Here are some arguments;
We should only focus test on our core code to save time and money: On almost any project this comes up at least once or twice. By trying to save cost this way you’re opening yourself up to other problems. Not covering your basis makes the QA process so much more expensive that the saved the time in development. Having a good test suite can both save time and money on the long run as for a new release you won’t have to test the whole system in detail over and over again. It still needed to have human eyes on it, but not as many as you would have without test.
We don’t have to test this, it’s going to be taken out soon: But that doesn’t mean that you should not test it. As it’s going in production users will get in contact with it. Users that may be confronted with errors and unexpected behavior. Errors, especially when dealing with new systems, can kill the user’s experience and can result in the user loosing confidence in the system or might prone the user to not coming back. For startups this is killing! Don’t take that risk.
This needs to be refactored later, we don’t need test: Refactoring, good! Although refactoring without tests is going to be harder as test also document the requirement that methods had before. Re-doing or refactoring becomes easier simply by having tests in place to guide the new code to what it should be doing. Tests hold certain knowledge about the functionality that you need during development. Not writing those tests means that knowledge is lost.
We need to cover everything with system tests to make sure everything is covered: Although a noble goal I feel that covering every possible scenario might be too much. In a 5 step process there might be 3 possible outcomes per step making it 243 possible different scenarios. This is overkill! Make sure your main path’s are covered and if you really need try covering all the scenarios with 45 tests maximally, but preferably with less. Main reason, more tests means more maintenance and as system test are fragile you can lose a lot of time with them when there are so many of them.
We already have unit test covering everything, why would we need system test: Unit testing is the core of test for a developer. But the system you’re building is not for developers, it’s for users. System test add that layer to the tests that connects it to the user and what their goal is with the system.
What to test?
This may be the ultimate question for many and where most of the discussion is happening. You’ve read the arguments people throw at it and in my opinion it’s about finding the right balance. Making your test cover enough without slowing you down. Finding that balance is all about what the team working on the project feels comfortable with. It’s important for the team to agree upon how for testing should go. Personally I find balance in unit test for back-end functions and everything data related. Integration tests help a lot with single actions that use multiple methods but confine a single purpose. System test for the positive paths through the system without touching too many edge cases and making sure I don’t test the same thing over and over again as it’s always part of the flow. I know it’s considered a sin, but I do stub in system test. There are only so many times you can log in as a user to check if the log in form is working.
Tests should also be there not just to make sure it works but can be used as requirement documentation as well. By using BDD (Behavior Driven Development) to develop as a form of TDD you can turn your tests into documentation about how your code should behave. The documentation might be a “side-effect” from the test but in the long run they are of create importance as they help transform the system to the next state. By either refactoring or changing the test and then changing the code you have a helping hand guiding you through legacy code. Reading your own code from years ago you may understand why this can be of great use.
The main thing to keep in mind is to keep sharp. Don’t forget to test but also always wonder if, how and what to test. When doing that your code should be able to surprise you in the future instead of giving you headaches.