Complex System Theory as an Approach to Building Evaluation Theory

Morell, J.A. American Evaluation Association 2002

We use program theory to shape evaluation. We ask stakeholders to articulate program theory. We search the literature to discover relationships between action and result. We choose variables based on program theory. We execute methodologies and interpret findings based on program theory. These activities assume that critical elements of a system can be identified, and that relationships among those elements can be articulated. Although we certainly believe in interactions, feedback loops, and unpredictable environments, we have faith that to some reasonable degree, relationships among critical variables can be defined, and will remain stable over time. Without a doubt, our faith is often rewarded by well designed programs and valid, useful evaluation. Often, though, either the program, or its evaluation, or both, do not work out as expected. The usual prescription to remedy failure is to do what we have always done, but to try harder. We must do a better job of understanding how a program really works. We must put more work into assuring fidelity of program reality to program theory. We must change the variables we measure, or do a better job of measurement, or use a more powerful mix of methodologies. But there is another possible explanation for failure. It may be that our theories miss the true dynamics that explain system behavior. This would be the case for evaluation settings that are best described with the principles of Complex System Theory (CST). For instance, in CST:

High level order and complex behavior can emerge from simple interactions among numerous individual actors.
Rapid transitions from one stable condition to another are possible.
Equally rapid transitions are possible between stable and unstable behavior.
Systems evolve and co-evolve as they range across differently shaped fitness landscapes.
Adaptability to new circumstance is explained in terms of the scale of organizational sub-units who are seeking to maximize their individual functioning.
Susceptibility to outside influence can vary over a wide range, from insensitivity to almost all influence, to hair trigger response to the smallest input.
Diverse patterns of system behavior move toward a single predictable pattern.
Auto catalytic interactions fuel mutually reinforcing behavior among parts of a system.

None of these concepts are familiar parts of our theory, yet any can hold the key to why a program behaves as it does. This session will challenge the audience with the possibility that there are times when program theory can be strengthened by incorporating the principles of CST. Our purpose is to sensitize the evaluation community to CST, and to begin a dialogue about CST's implications for how we develop and use theory to guide evaluation.