System Behavior in the Programs We Evaluate: Implications for Theory, Measurement, and Methodology Morell, J.A. American Evaluation Association 2003
|
| There seems to be a growing sense among evaluators that while logic modeling and program theory are the foundation of good evaluation, our efforts in these areas are unsatisfying. No matter how extensive the model building exercise, and no matter how many stakeholders are involved, and no matter how much previous experience and data are brought to bear, models don't seem to help as much as they should. If we were to try harder, could we build better models and more powerful program theory? We believe the answer to this question is NO, IF by "trying harder" we mean putting more effort into doing things as we have always done them. We believe the nub of the problem is that the programs we evaluate often exhibit behaviors that are a function of system dynamics. These behaviors are quite different, (often in radical and surprising ways), from the behaviors we expect based on our usual logic models and program theory. Because those theories drive our measures, our methodologies, and our relationships with stakeholders, failure to appreciate system behavior is leading to our weak evaluations. We propose a panel to explore these issues, and point the way to solutions. To achieve these objectives we will combine multiple views of system behavior, an illustrative case study, and heavy audience participation. Our goal is for participants to leave the panel with new and useful insight about how to develop logic models and how to apply those models to make decisions about measurement, methodology, and interaction with stakeholders. |