br
This presentation focuses on SciUnit, a Pythonic framework for data-driven unit testing. SciUnit is used to create a domain specific model review package. The package can then be applied across models. For each model you create a sciunit.Model
for a sciunit.Capability
that will be judged by a sciunit.Test
.
my_model = MyModel(**my_args) # Instantiate a class that wraps your model of interest.
my_test = MyTest(**my_params) # Instantiate a test that you write.
score = my_test.judge() # Runs the test and return a rich score containing test results and more.
SciUnit contributions 2012-2021:
ISSE conference paper from 2014 and an active repository. Heavy users in neuroscience (NeuronUnit)
Test classes are data-agnostic
from cosmounit import PositionTest, VelocityTest, EccentricityyTest # Cosmounit is an external library.
and test instances encode the data you want a model to recapitulate.
from . import saturn_data # Hypothetical library containing Saturn data.
position_test = PositionTest(observation=saturn_data.position)
velocity_test = VelocityTest(observation=saturn_data.velocity)
eccentricity_test = EccentricityTest(observation=saturn_data.eccentricity)
Orbital models can predict any planet, but we are interested in Saturn:
from cosmounit import PtolemyModel, CopernicusModel, KeplerModel, NewtonModel
ptolemy_model = PtolemyModel(planet='Saturn')
copernicus_model = CopernicusModel(planet='Saturn')
kepler_model = KeplerModel(planet='Saturn')
newton_model = NewtonModel(planet='Saturn')
from saturn_suite.suites import saturn_motion_suite
saturn_motion_suite.judge([ptolemy_model, copernicus_model, kepler_model, newton_model])
Tests could be location specific or a theoretical result.
from . import measles_data # Hypothetical library containing measles data.
from epiunit import CCSTest, AgeAtInfTest # epiunit is an external library.
ccs_test = CCSTest(observation=measles.ccs)
age_at_inf_test = AgeAtInfTest(observation=measles.Nigeria)
seas_test = SeasonalityTest(observation=measles.Nigeria)
and packaged up into a location specific test suite:
nigeria_epi_suite = sciunit.TestSuite([ccs_test, age_at_inf_test, seas_test])
Models could be differentiated by version, features, type, etc...
from enod_package import EmodModel
from tsir_package import TSIRModel, DynaMICEModel
emod_model = EmodModel(location='Nigeria')
tsir_model = TSIRModel(location='Nigeria')
mice_model = DynaMICEModel(location='Nigeria')
from epi_suite.suites import nigeria_epi_suite
nigeria_epi_suite.judge([emod_model, tsir_model, mice_model])
Models do not need to be capable across the entire suite.
Every model has a capability it aims to implement:
class ProducesNumber(sciunit.Capability):
"""An example capability for producing some generic number."""
def produce_number(self):
"""The implementation of this method should return a number."""
raise NotImplementedError("Must implement produce_number.")
Each model may have a unique method for that particular capability:
class ConstModel(sciunit.Model, ProducesNumber):
"""A model that always produces a constant number as output."""
def __init__(self, constant, name=None):
self.constant = constant
super(ConstModel, self).__init__(name=name, constant=constant)
def produce_number(self):
return self.constant
and create a model instance:
const_model_37 = ConstModel(37, name='Constant Model 37')
A sciunit.Test
class must contain:
generate_prediction
to get model predictioncompute_score
to compute a sciunit.Score
## Example test class
class EqualsTest(sciunit.Test):
"""Tests if the model predicts the same number as the observation."""
required_capabilities = (ProducesNumber,) # The one capability required for a model to take this test.
score_type = sciunit.scores.BooleanScore # This test's 'judge' method will return a BooleanScore.
def generate_prediction(self, model):
return model.produce_number() # The model has this method if it inherits from the 'ProducesNumber' capability.
def compute_score(self, observation, prediction):
score = self.score_type(observation['value'] == prediction) # Returns a BooleanScore.
score.description = 'Passing score if the prediction equals the observation'
return score
## Example test instance
# create test instances
equals_37_test = EqualsTest({'value': 37}, name='=37') # Test model output equals 37.
equals_1_test = EqualsTest({'value': 1}, name='=1') # Test model output equals 1.
# create test suite
equals_suite = sciunit.TestSuite([equals_1_test, equals_2_test, equals_37_test], name="Equals test suite")
# run suite
score_matrix = equals_suite.judge(const_model_37)
=1 | =37 | |
---|---|---|
Constant Model 37 | Fail | Pass |
Complete score types in SciUnit are:
BooleanScore
: true or falseZScore
: standardized difference from the meanCohenDScore
: normalized difference between two meansRatioScore
: ratio of two numbersPercentScore
: float between 0 and 100FloatScore
: a floatIncomplete score types are also included (NoneScore
, TBDScore
, NAScore
, InsufficientDataScore
).
SciUnit does not include statistical tests (ex. Kolmogorov–Smirnov test, Cramér–von Mises test) for comparing distributions.
SciUnit is only one way of tackling this problem. It requires support and time to write the tests but they can be wrapped around the final model. Test can be applied across models.
Paper reproduction:
References: