Automated integration tests are slow and tend to result with false negatives. We show by collected data and probabilistic arguments that false negatives are an inherent quality of integration tests. It follows that it is impossible to solve the problem by adjusting tests. We argue that the testing machinery and processes must include strategies to deal with false negatives.
We lay out a general, and tool independent strategy to achieve this. We also demonstrate the feasibility with our own implementation (open sourced and freely available). A very welcome side effect of our approach is that we could lower the testing time for our projects from 1–2 hours to about three minutes!
Skills
None but interest in the matter of testing and particular automated integration testing.
Lernziele
* Spread the understanding that the no zero probability of false negatives of integration tests does have a practical impact on testing.
* There are solutions to the problem. However, writing "better" tests doesn't help. Solutions must be implemented in the CI process.
* Show one possible way how to implement a solution/process.
// Referenten
//
Dr. Thomas Schank
@DrTom21
is a Senior Software Engineer at the Zurich University of the Arts. He is the lead architect for the Madek project. Thomas designed, implemented and runs the CI infrastructure at the ZHdK. He received his Ph.D. in computer science from the University of Karlsruhe. He has given numerous talks at international conferences and workshops.
//
Max Albrecht
@EINS78
is a Frontend Developer at the Zurich University of the Arts and mainly working on the Madek project. Apart from that he is interested in free/open ecosystems of any kind. He provided input for the UI design aspects of Cider-CI.