In table 1 we see that EDI was never really implemented very well, which implies that as an innovation in electronic commerce it was not tested. In this case we don't know if it would have worked or not. The data also tell us that our model is weak because it has not accounted for many reasons why change may take place. For instance: There is noteworthy change in suppliers' data input accuracy and "OEM production time", even though no useful change took place in "shipping by suppliers", "improper deliveries by suppliers", or "OEM delivery time".
In table 2 we observe that EDI was implemented, that errors were reduced, but that improper or late deliveries were unaffected. We also observe, however, that considerable favorable change took place in the OEM's behavior - both production time and delivery time improved. On one hand these data show us again how weak the model is because "downstream" variables changed in spite of no change in what were hypothesized as upstream, critical path activities. On the other hand, the model saves us from making an incorrect inference. If we had data only for EDI implementation and OEM behavior, we would conclude (incorrectly) that our experiment in SCI succeeded.
This pattern of results also illustrates the value of models in guiding continuous improvement. Here we know that EDI did work in terms of having the proximate effect of decreasing data errors, but that better data accuracy is not, by itself, sufficient to affect shipping delays or improper shipments. The model lets us identify where we need to look for further improvement.
The pattern in table 3 gives us great confidence that SCI worked as intended. First, all metrics improved as expected. Second, the pattern of results was plausible, i.e. it made sense in terms of what we know about SCI. We all realize that no model can identify all relevant variables, and that the further one gets from "proximate effects", the more complicated gets the world, and the greater the number of unanticipated factors which may influence events. And what do we see? Large changes in suppliers ability to ship as planned, and lesser changes in the OEM's ability to ship as planned. This makes sense because the OEM's behavior is only partially determined by suppliers' behavior.
The lesson in this example is that because we have a model we can understand what happened. Without the model we would not have been directed to collecting metrics on degree of EDI implementation, input errors, improper deliveries to the OEM, shipping delays to the OEM, and production time. We would only have data on delivery time to the end user, and thus, little ability to use the data in a constructive fashion.
Tables 1-3 also illustrate several other important points about the value of models. First, metrics can differ in terms of scale, precision or their specific meaning. Observe how "extent of EDI implementation" is measured in the same way in tables 2 and 3, but in a different way in table 1. (Compare italics for variable 1 in table 1 with the corresponding definitions in tables 2 and 3.) Both methods are valid in that they accurately reflect "extent of EDI implementation", but they certainly differ in degree of precision, and in the type of statistical analysis that can be conducted. While ratio scales of measurement and high degrees of precision are always desirable, these concepts are different from the four critical measurement criteria discussed earlier (validity, reliability, salience and ease of collection). A metric may be precise and not valid or reliable (we have all seen meaningless data carried to four decimal places), reliable but qualitative (e.g. experts agree on categorizing into a few broad categories), and so on. Thus "scale quality" and "precision" must be seen as fifth and sixth attributes of measures, and added into the mix of trade-offs when models and metrics are developed.
As an example of these trade-offs, consider, a company's information system may make it easy to accurately determine the number of shipping delays per month, but not the number of shipping delays per week. Do we care enough about the added precision of weekly data to add to the burden of a special data collection effort to get the data weekly? Are we willing to put up with increased risk of unreliable data due to a new and unproved data collection mechanism? Without careful consideration of specific context and needs for data, these questions are unanswerable. But left to chance, incorrect answers may cause great harm to the proper collection of powerful data.
To see how meaning can differ, compare the definition of "data accuracy" (variable 2) and "OEM delivery time" (variable 6) in the tables. (Note the italicized text in the definitions in table 3.) These are all useful ways to measure accuracy and lateness, but they do not mean exactly the same thing across all three scenarios.
A second insight from comparing our hypothetical results is that models are never likely to account for all relevant factors. One problem is that fixed models cannot ever fully specify an open system. Over and above this theoretical problem, models are only as good as people's imperfect ability to identify relevant metrics. Finally, each element added to a model requires the investment of time, effort, and money to collect data and to assure its quality.
Two approaches can help bolster the inherently limited power of models. The first is to embed model-based data collection in a qualitative analysis of events and circumstances. As an example, consider table 3. Do we really want to believe that improvement in suppliers' ability to ship affected the OEM's ability to ship? We would feel a lot more certain about it if we had a set of interviews with key people within the OEM, all of whom said something like: "One of our big problems in satisfying our customers was always that we could never really trust our suppliers' shipping schedules. Since this EDI stuff started, that problem has pretty much been solved, and because of that we have gotten a lot better with our own internal planning."
On the other hand, we might not trust the quantitative data if all interview responses were along the following lines: "Not being able to rely on our suppliers' deliveries was always a problem for us, but it never really got in the way or our delivery to our customers. Sure we had to scramble a lot more than we wanted, but we always knew what we had to do to get our shipments out. I can only think of a few times in the last year or so when problems with delivery from our suppliers really caused us to miss a production deadline."
A second approach is to be ready and willing to add to a model as events unfold. To extend the previous example, imagine that the interview results came half way through an eighteen month Pilot. In such a situation, it may be worth the effort to add a metric to the model and to specifically measure the OEM's internal planning.
As a final point about models, it is important to realize that models are context dependent. Models can differ a great deal depending on the people involved, their needs for data, and the way a project unfolds. As an example of how context can lead to different models, consider figures 2 and 3.
Figure 3: Metric Model for the ECOTS Supply Chain Integration Pilot
Figure 1 is a model developed for the Manufacturing Assembly Pilot (MAP), an effort to integrate an automotive supply chain, starting with a tier-one automotive supplier (Johnson Controls), and extending four levels deep. Figure 2 is a model developed for Electronic Commerce for Oshkosh Truck Suppliers (ECOTS), an effort to establish electronic commerce relationships between Oshkosh Truck and six of its direct suppliers.
Both pilots are similar in important ways. Both are in the automotive industry, and both focus on material flow, i.e. the ordering or scheduling of parts. Despite these similarities, the models used are very different. The MAP model (figure 1) shows a clear chain from three immediate changes wrought by SCI (faster information, more complete information, and more accurate information) to a variety of outcomes at intermediate levels, to the business outcomes that people really care about. It does not, however, detail internal changes within Johnson Controls, nor does it articulate which data elements are being collected from which participants. Levels of change are not as clear in the Oshkosh model (figure 2), but the Oshkosh model does detail internal changes in the OEM, and it does state which data are coming from which companies. There are considerable differences in the actual metrics specified in each model. Finally, style, form, and layout, are very different.
The reason for these differences are rooted in the realities of what stakeholders in each Pilot cared about, what data were available, the number of elements in each model, and a variety of other factors that led to specific models for specific circumstances. Both though, are real models that were (are) used to evaluate the effectiveness of real industry Pilot projects in SCI. Each in their own way is equally useful. Neither Pilot would have good evaluation if it had only a laundry list of metrics without a model of how each metric fits into an overall system.
Metrics and Model Development as an Iterative Process
For the sake of clarity of presentation, I have broken the treatment of "metrics" and "models" into separate sections, and thus left the false impression that these tasks are independent. They are in fact, highly interdependent, and should be developed in an iterative and concurrent manner. Models are useless unless they are populated with good metrics, i.e. metric which jointly optimize validity, reliability, salience, and ease of collection. Metrics in turn, are useless unless they can inform a relevant model (i.e. one that speaks to critical business problems that may be affected by SCI), and map changes in critical business and technological processes that may affect business problems. In engineering terms, the challenge is to jointly optimize the quality of the metrics and the quality of the model.
Developing Your Own Metrics and Models
In light of the need for metrics and models to be context-specific, what are the general guidelines that can help a group develop useful metrics and models for their specific circumstances? Those guidelines fall into two categories - development process and meta-models.
Development process
In terms of development process, metric and model development must be viewed as any complex design problem wherein good process is characterized by the principles of integrated product - process design (IPPD). These principles, along with examples of applications to SCI, are presented in table 4.
| Table 4: Design Principles Applied to Metric and Model Development |
| Principle |
Example of Application to Metric and Model Building for SCI |
| Information and Work Flow |
1- Non-value added data and flow should be eliminated to minimize inaccuracy and redundancy.Its easy to imagine a situation where an exercise in metric and model building for evaluation of an SCI initiative becomes confounded with a more general effort for building as-is and to-be models for a business process reengineering exercise. While model building for evaluation and for BPR are related, they are not one in the same. For instance, "rework and rejects" (in figure 3) may require a great deal of tracking and reporting that is important for a BPR team but not for an SCI evaluation team. Efficient design would keep these BPR reporting issues outside the efforts of the evaluation group. |
| 2- Necessary information and other outputs of work should go directly to the point of action in a timely manner |
Consider the example used to show the importance of supporting qualitative data. The people doing that interviewing may very likely not be the same as those charged with model maintenance. Unless information flowed efficiently between the two groups, model developers may not realize (or realize too late) that they should modify their plans so as to specifically assess the OEM's internal planning processes. |
| 3- Decisions should be traceable to the source |
Again consider figure 2. Its easy to imagine a data analysis exercise toward the end of the process looking at relationships between "on-time shipments", "producing more accurate material requirements", and "inventory turns", and wondering: "Why ever did we think these would be correlated?" Without knowledge of the answer, it would be very difficult to decide whether to actually do the analysis as originally intended, or whether it still made sense to look at the relationship at all. |
| 4- All parties to the development process should use a common vocabulary |
The evaluations depicted in tables 1, 2, and 3 all collect data on a variable called "delivery time, OEM to customer", but does this mean we can compare or combine the data? Without a good data dictionary, its impossible to answer this question. |
| Organization |
|
| 5- People should be grouped by interrelated tasks |
To build on the example provided with principle #2, data collection and model building are very different activities, and people competent at one may not be at the other. For instance, the ability to pull stakeholders together and get them to reach consensus on metrics and models requires a very different set of skills than does the ability to plan and execute a data collection exercise. Why assume that people who are good at the first will also be good at the second? |
| 6- Product-focused leaders should be appointed for all important development projects |
Again to invoke the example of model building and data collection/analysis. Two "products" are involved, the first a model based on high quality metrics, and the second a workable data collection and analysis plan. Because these require different participants and different sets of expertise, it makes sense for the two tasks to have different leaders. |
| 7- Appropriate coordinating mechanisms should be developed to manage interfaces |
Consider two important connections between the tasks of model building and data collection. One connection is through the continuous improvement process cited above. More important is that model building, metric development, and data collection are best carried out within a framework of integrated product/process development. Data analysts, for instance may understand that high quality measurement scales are important for the analysis of some variables, while purely qualitative data would suffice for others. A group without this expertise, charged with metric and model development, may define metrics in such a way that high quality measurement is done when it is not needed, and low quality measurement is done when better measures would help a lot |
| 8- All stakeholders affected by a decision should be represented in the decision. |
Its easy to put metrics into a model. Its hard for busy people doing real business in corporate settings to collect data. Without involving the business people, any effort at data collection is likely to fail |
| 9- Hierarchy should be minimized with decisions made at the point of action |
One example is technical decisions about data analysis. While the general themes of an analysis plan need to be agreed upon at a high level (e.g. we agree that we should assess both costs and benefits, but neither have to be defined strictly in terms of dollars), decisions about data manipulation and specific techniques should be left to trusted experts. |
| 10- Single points of primary responsibility and authority are required for each major task. |
To build on the data analysis example in the principle #9, while several people may be involved in data analysis, there must be a clearly defined leader of the analysis team. That person must have both authority over the team, and responsibility to assure that what the team does meets the needs of all stakeholders |
| 11- Reward systems should reinforce organizational objectives. |
The objective of an SCI Pilot is not to build good models, to have good metrics, or to do sophisticated data analyses. Those tasks are important only if they further objectives of the Pilot, which are likely to be goals such as helping to build a business case for SCI, or to develop guidelines to help others engage in similar activities. |
| Product Development |
|
| 12- A structured design process should be developed and used |
Metrics and models are only a small part of SCI activities, which are likely to include actually doing business process reengineering and introducing new technology into business settings. Schedule priorities for these activities are likely to override the scheduling needs of metric developers, who themselves have many complex tasks to perform. To assure success in this complex process of coordination, it is important to have clearly defined, and publicly known, schedules for all major milestones |
| 13- The voice of the customer should be the focus throughout the process |
To build on the example for principle 11, customers for an SCI pilot are likely to be groups who care about what models and metrics can tell them about SCI, and not in the models and metrics per se. Consider a choice of two metrics which differ in the degree to which companies can exercise instrumental control over them., e.g. "rework and rejects due to internal process control", and "engineering changes due to changes in customer's requests". All else being equal, the first metric would be more useful to customers than the second. |
| 14- Stable strategic targets should be developed early in any design project. |
Two reasonable choices for an exercise in SCI may be to improve: 1- responsiveness to existing customers, or 2- the ability of trading partners to engage in agile, virtual partnerships so as to attract new customers. Once this decision is made by stakeholders, teams engaged in models, metrics and analysis must continually work at orienting their decisions in ways that will support the strategic goal of the overall exercise |
| 15- Design history should be captured and used. |
Any particular arrow drawn in figures 2 and 3 were based on a specific rationale as to why one metric might affect another. While these examples are in the automotive industry, several SCI pilots are beginning in the aerospace industry. Those aerospace teams could save considerable labor (or at least glean considerable wisdom) from easy access to knowledge of why these specific choices were made in the automotive pilots |
The scale and criticality of model and metric development should determine the rigor with which these principles are applied. To see why contrast the following two cases.
Case #1:
Three people are developing an assessment of a small scale experiment in which technical product data is enclosed in mime-enabled messaging software and sent to a single supplier. The object of the exercise is to determine whether this process has any cost or speed advantages over sending faxes or express mailing disks.
Case #2
Representatives of two large OEMs and many common suppliers are working on an effort to revamp internal systems and to put both EDI technology and supportive business process in place so that time and costs are reduced for all participants in the participating network of trading partners.
While good design principles are needed in both cases, these two efforts differ greatly in factors such as the time it will take to do the development, difficulty of managing the process, consequences of failure, diversity of group members and their vested interests, complexity of the model, and number of metrics. Case two certainly requires a more rigorous design process than does case one.
Meta-models
While it may be true that models for evaluating SCI are unique to their specific purposes, it is also the case that all specific models can be seen as derivative of an over-arching model of system functioning based on general systems theory. By using basic concepts from system theory it is possible to assure that appropriate specific metrics are chosen. While there is not a one to one correspondence between specific models and a general systems model, that general model works well as a heuristic to guide metric development. This notion is illustrated in table 5 which presents specifics of the MAP and ECOTS models, and maps them into a set of characteristics that describe open systems.
| Table 5: Systems Theory as a Guide to Model Development |
| Systems Theory Concept |
Application to Models of SCI |
| Importation of energy |
The emphasis here is on inputs into a system that move across organizational boundaries. The challenge for builders of SCI models is to assure that all such inputs are accounted for, particularly since they may interact with each other. Are we doing EDI only, or are we doing EDI and also sharing a data base on engineering change notices? If both, then both must be included in the model. |
| Throughput |
The notion here is on the reorganization of input. The key is to make sure that all relevant reorganization is accounted for in the model. As an example, the MAP model (figure 1) articulates two starting points and 3 paths to reduced inventory through the actions of SCI. If for instance, "faster information" also affected inventory, we would have an under-specification, and hence a limited evaluation. |
| Output |
As with input and throughput, incomplete specification leads to limited understanding. The MAP model, for instance (figure 2) is silent on the subject of relationships between EDI and the ability of a company to capture new business. If new business is a possible consequence of EDI, the model will blind us to important outcomes |
| Cycles |
The notion here is that as elements of a system change, they affect each other. For instance both the MAP and ECOTS models (figures 2 and 3) specify metrics which should make suppliers more desirable business partners to their customers. But neither model accounts for possible consequences of that change. If we think such a change is unlikely, the models are fine as they are. But if such changes are possible, the models have kept us from collecting important information |
| Entropy, homeostasis and coordination |
All.three of these concepts refer to a system's efforts to maintain organization (and grow) in the face of powerful forces for disorganization. The ECOTS model (figure 3) for instance, hypothesizes a trade-off of one maintenance method over another, i.e. the substitution of a buffer (inventory) for better process control. Good model development in SCI will identify these dynamics and assure that the most important ones are included |
Relation Of Metrics And Models To The Business Case For Supply Chain Integration
One of the chief reasons for evaluating innovations in SCI is to help build a business case for change, i.e. to provide information that will help others institute similar changes. For this process to be effective it is important to appreciate the difference between metrics and models on the one hand, and a business case on the other. Metrics and models help tell the story of what happened during an exercise in SCI. A business case is a story that helps business people to commit to action. The two stories are not, and cannot be, the same. The challenge is one of approximation, i.e. to design a system of models and metrics that is as close as possible to a business case. Figure 4 represents this challenge.
Why can't a business case be completely coincidental with metrics and a model?
Because:
- No matter how good the metrics, some people may not believe the data. As an example, it is a common belief that small manufacturers cannot do cost effective, integrated EDI. Despite convincing case examples to the contrary, the old belief will not die.
- Metrics may not be salient to decision makers. As an example, an executive may believe that sales are made on the basis of price to the exclusion of response time to an order. In such a situation, data which shows how SCI shortens order response time may be believed, but not considered important.
- Metrics may not touch on important issues such as risk, consequences for functioning in other parts of the company, or the ability to raise capital to invest in technology.
- Metrics and models assume a view of rational decision making which may not prevail in the real world of business decision making. This is not to say that business people are irrational. It is to say that commitments of resources are made for complicated reasons, and that factors that enter into decisions interact with each other in subtle and often non-linear ways. There is an extremely interesting and useful body of research on how information affects decision making in real-world settings. A good entry into this literature can be found in:
- The very process of presenting people with data assumes that the people receiving the data have the freedom of action required to act on the information. This is not always the case, as most decision makers, no matter how seemingly powerful their role, often have to build consensus within a coalition prior to committing resources.
The challenge is to develop models and metrics that will, to the greatest degree possible, result in a business case for SCI. The keys to success are to include appropriate decision makers in the metric and model design team, and as the data collection process unfolds, to continually educate key people on the value of what is being discovered. The key to good mental health is to accept that given the constraints and complexity of good business practice, models and metrics should not equal a business case for SCI.
Conclusion
Manufacturing faces a growing challenge to shorten cycle times, reduce costs, and respond more quickly to markets. Supply chain integration is a critical element in meeting this challenge. It is also, however, a risky endeavor which entails large-scale changes in business process and technology, and which can only be implemented during limited windows of opportunity. Because of the risk, scale of change and limited opportunity, high quality information is needed to encourage action and to improve the probability of success. That information can come only from a collaboration between social scientists who understand data and modeling, and the executives, managers and staff who are responsible for assuring the vitality of their businesses. A chief goal of that collaboration must be the deployment of SCI testing models which have both methodological power and practical value.
Jonathan A. Morell, Ph.D.
|
