How to Evaluate a Conference

This document provides a map of how I approach the task of evaluating conferences. It deals with three broad issues. Who provides data? What data do we need? How should the data be collected? 

Attendees

Knowledge transfer and networking People attend conferences to acquire technical and business information, and to find out about useful products and services. They also go to network. With respect to the first two objectives it is important to make a distinction between two types of knowledge.

Instrumental knowledge is knowledge that is put to use in some reasonably well defined and specific manner. Examples: 1- Need to buy an EDI translator. Use knowledge of prices and specifications to choose a vendor. 2- Learned about doing on-line business with the government. Use that knowledge to seek new business opportunities.

Conceptual knowledge is knowledge that has an impact on how people think about issues, but does not have an unambiguous impact on a specific action. Examples: 1- Learned about cases of Web-based sales. Change notions about what markets it may be practical for my business to serve. New perspective plays a part in later decisions about company direction. 2- Experience provided a sense that organizational issues are important when implementing new technology. Attend workshop on socio-technical methods of change management. Details forgotten, but workshop leaves an appreciation of the importance of including organizational issues in change management.

 Context determines whether knowledge is put to specific uses. Take the case of learning about the cost and specifications of EDI translators. Imagine that I am not under pressure to do EDI and thus do not use the information for any immediate purpose. Time passes. Costs and specifications change. A potential new customer wants me to do EDI as a condition of doing business. The aged information cannot be directly applied, but it did contribute to my belief that EDI may be feasible for my company. As a result I decide to pursue the business opportunity.

Unless evaluation assesses both kinds of knowledge acquisition the true value of the meeting cannot be known. Its hard to quantify the value of conceptual knowledge, but it is possible to get a general sense of how much such knowledge transfer took place during a meeting. Judgements about the impact of a conference would be misleading without this information.

With respect to the numerous informal interactions that take place during conferences, another distinction comes into play -- getting to know people, and knowledge transfer. The latter is important because it is quite possible that the most important information learned at a meeting came from informal conversations.

Attendees expectations: An important dimension of conference evaluation must be how it met expectations from the point of view of the attendee. No matter what the stated reason for a meeting, people bring a unique set of interests and needs to such events. There are three reasons why it is important to evaluate a conference from the users' point of view.

• The knowledge is useful for planning other meetings. As an example, organizers may think the reason for a meeting is to make people aware of how to use EC to do business with the federal government. Perhaps, though, many attendees saw the meeting as a convenient way to learn about EC in general, regardless of their intentions vis a vis the federal government. This knowledge provides insight on how the conference was advertised and on how potential government suppliers see the importance of EC.
• The information is needed to develop a complete view of the value of the meeting. To continue the example above, consider two evaluation approaches, one based only on the sponsors' stated reason for the meeting, i.e. use of EC with the federal government; and one that included both the stated reason and attendees' expectations. The first approach would fail to detect a great deal value that people may have derived from the meeting.
• The data are useful in interpreting evaluation results. Consider a situation where attendees reported dissatisfaction with the meeting. It is important to know if the dissatisfaction derived from their walking into the event with unrealistic expectations.

Finally it is important to recognize the distinction between what people got out of a meeting and what they wanted to get. Its entirely possible that a conference was extremely valuable, but not for the reasons that motivated initial attendance. This information is important to conference organizers because it provides feedback about how the conference was advertised, and about any disjunction between the meeting as planned and the meeting as delivered. An overview of data collection from the attendees' point of view appears as Table 1.

Indicators/measurement: While it is premature to work out the details of how all necessary data will be measured, it is useful to have a sense that the necessary data can be obtained. The key is to lead respondents through a logical sequence of questions that will help them remember their experience at the meeting and their reactions to it. I envision either a questionnaire or an interview along the general lines shown in Table 2. (Please regard Table 2 only as an illustration. It is most certainly not a definitive view of how an interview or a questionnaire would be designed.)

Exhibitors

Exhibitors show at conferences for five reasons:

• Business leads
• Solidify existing customer relationships
• Derive information on markets and customer needs
• Intelligence on the competition
• Raising their corporate profile

Its not practical to ask people to quantify these goals, but it is practical to ask people which goals were important how well they think the goals were met. The general logic of the questioning is shown in Table 3. As with the questions for attendees, please regard Table 3 only as an illustration of an approach for getting data from conference exhibitors.

Sponsors

Because producing a conference is difficult, time consuming and expensive, organizations do it only when they believe that useful purposes will be served. Evaluation from the sponsors' point of view should proceed based on a logic outlined in Figure 1. Data should be collected by means of either individual or group interviews with a few key people in each sponsoring organization. Questions to those people should include expectations and opinions about the following.

• Number and type of attendees
• Number and type of exhibitors
• Financials
• Content
• Impact on key individuals (e.g. agency heads, representatives of trade associations)
• Impact on groups (e.g. the sponsoring organization, economic development groups)

Contextual Information

The data collection outlined above is all focused directly on what impact a conference has on particular groups. In order to derive maximum value from the evaluation, certain contextual information is also needed, as illustrated in Table 4.

Methods of Data Collection

Without knowing a great many context-specific details, it is impossible to specify a methodology. Based on experience, however, a few guidelines can be articulated.

• Most data collection should take place approximately 3 -- 6 weeks after the conference. Some time is needed for attendees to settle back into a routine where they might apply their new knowledge, but too long a delay will dim their memories for detail.
• Depending on results from the first round of post-conference data collection, it may be wise to follow selected cases for a longer period of time.
• If at all possible, pre-conference data on expectations should be collected from at least a sample of people who registered.
• If interviews are used, they should be conducted via telephone. In all likelihood data collection will be a combination of telephone interviews and questionnaires. Interviews with key stakeholders in sponsoring agencies should be in-person.
• Respondents should be given some incentive to respond. The incentive usually offered is a summary of results. Another possibility is a set of recommendations about the topic of the conference, derived from the evaluation data

The choice of a sampling method will depend on the total number of attendees, the variety of subgroups of people at the conference, and the need for interviews as opposed to questionnaires. The need for sampling will rise with the number of people and the desirability of questionnaires. The need for stratified sampling will increase with the variety of subgroups represented by the attendees.