Hydra Image Processor Consistency Testing:
Though a full test framework is not implemented in HIP at this time, we have implemented a snapshot comparison framework in the 'testing' branch, which allows for the verification of filter consistency across library versions and platforms. When combined with manual and automatic tests, snapshot verification is a powerful and scalable method for maintaining and validating filter results as changes are made to the codebase. In addition, we have outlined a testing roadmap for implementing automated filter and HIP framework tests. The roadmap also provides a protocol for filter writers to follow when validating their own filters.
The data snapshots use a set of simulated images from (2D-5D) generated in a deterministic manner (stored in the Testing/Images directory). The images are used as input for each HIP filter and the outputs of all individual filter-type variations are stored as a snapshot. Each snapshot output can be compared exactly across version changes in the HIP library, with any output differences indicating a possible consistency error was introduced. Given a stable and verified baseline for comparison, this process guarantees that filter outputs remain consistent across versions, platforms (Linux/Windows), and scripting engines (Matlab/Python). Fixes or modifications to filters will be detected as inconsistencies, which should then be validated on a per-filter basis. In future we hope to integrate this snapshot comparison into a full unit-test and continuous integration framework.
The code for capturing snapshots is available in src/MATLAB/+Test/captureSnapshot.m and src/Python/Test/captureSnapshot.py. Individual snapshot outputs are stored as MATLAB *.mat files for convenience, and can be compared using src/MATLAB/+Test/compareSnapshots.m