|
|||||||||||||||
 | |||||||||||||||
 |
|
||||||||||||||
|
|||||||||||||||
 |
|
 |
 | Home | | About CCOHS | | Reports |
|
||||||
|
Program Evaluation and Performance Measurement Study |
| January 26, 2006 NG342 |
|
Presented to - Présenté à Canadian Centre for Occupational Health and SafetyTNS Canadian Facts Social and Policy ResearchKPMG LLP
Ronald B. Robinson Douglas A. Smith Geoff Golder Brenda Sharpe |
|
TABLE OF CONTENTS Chapter Page D. Performance Measurement framework A. Mandate, Objectives and Key Results Commitments IV. Program Evaluation Findings D. Cost-Effectiveness and Program Alternatives V. Proposed Performance Measurement Framework A. Context for Performance Measurement B. Features of the Current Approach to Performance Measurement C. Proposed Foundation for Future Performance Measurement D. Proposed Focus for CCOHS' Strategic Performance Measurement and Reporting A. CCOHS Plays an Important Role B. CCOHS activities have strong links to workplace changes C. A Stronger Focus on Results is Needed in CCOHSS' Strategic Performance Measurement Appendix: Survey Questionnaires I. Executive SummaryA. IntroductionThe Canadian Centre for Occupational Health and Safety (CCOHS or the Centre) was created in 1978, by the Canadian Centre for Occupational Health and Safety Act, which mandates CCOHS to promote the fundamental right of Canadians to a healthy and safe working environment. CCOHS operates under a tripartite council consisting of representatives from business, labour and government (federal, provincial and territorial). CCOHS contracted with TNS Canadian Facts, Social and Policy Research and KPMG to carry out a program evaluation and an assessment of performance measurement options. This study has reviewed a large number of aspects of CCOHS performance. Results from this study are intended to provide the foundation for the development of a new performance measurement system that will see CCOHS continue to achieve its core objective of promoting workplace health and safety. This study includes two closely related components. These components are:
B. Current Status of CCOHSCCOHS operates as a source for unbiased technical information and expertise to support the efforts of governments, labour organizations, employers, and individual Canadians to improve workplace safety and health. Transfer of current knowledge essential for workplace participants to effectively address the specific health and safety issues in their workplaces in a form readily usable by them has been a unique feature of this service. CCOHS fulfils this mandate through a portfolio of free and priced products and services that draw upon a core collection of occupational safety and health information and the application of information management technologies. Partnerships with leading national and global institutions on occupational health and safety have enabled CCOHS to base its services on current global knowledge. The key products and services offered by the Centre are the free Inquiries Service and web site and information products and services for which it charges. CCOHS also supports national and international initiatives promoting workplace health and safety through collaborative projects with a wide range of partners, and delivers education and training services as well, including e-learning. CCOHS' services are shaped in accordance with the Guiding Principles outlined by its tripartite Council of Governors and represent the consensus of the parties at the Council. CCOHS has three main sources of funding/revenue - voted public appropriations, supplementary estimates, and revenues from external sources (cost recovery revenues). The past decade has also seen changes in the composition of CCOHS revenue sources. There have been changes in the balance between product sales and revenues from collaborative projects and other sources. The share of revenues coming from product sales has decreased and the proportionate reliance on collaborative projects, publications and other revenue has increased. For the current fiscal year, the CCOHS operating budget is $8.8 million. Of this total, appropriations from the Main Estimates account for $4.54 million (52%), contributions from provinces and territories account for $0.45million (4%) and cost recovery revenue accounts for the remaining $3.9 million (44%). The level of cost recovery achieved by CCOHS has declined somewhat in recent years but continues to be very high compared to that achieved by comparable information agencies in Canada and internationally. The ability of CCOHS to create new revenue-generating services and to adapt existing services to respond to evolving needs of Canadians is heavily restricted by its small funding base, which also has to support Canada-wide free-of-charge services. The financial viability and sustainability of services has been a continuing challenge to CCOHS. CCOHS must keep current its core information collection and the information technologies that the collection, the Centre's web site and many of its priced products rely on. This continuing development is necessary if CCOHS is to continue to provide up-to-date and comprehensive OHS information to users. The Centre needs to develop an updated strategy and set of priorities for selecting projects and partners, focusing on activities that will keep its products, services, and distribution channels at the leading edge. However, the operating environment of CCOHS makes this difficult. Costs continue to rise whereas Main Estimates funding is constant. Cost recovery has been pursued aggressively but there appear to be limits to pushing this farther. This is, especially the case in light of the survey data in this report indicting that higher fees may inhibit the ability of the public to access CCOHS products and services. C. Program EvaluationA key component of this review was an evaluation of the operations of CCOHS, including an assessment of the relevance of the program and the effectiveness and efficiency of the program given its mandate, and its positioning within the mosaic of Canada's existing structure and the framework of workplace health and safety programmes from governments, compensation boards and others.. The ultimate objective of CCOHS is to improve workplace health and safety. This objective involves CCOHS being recognized as a national clearinghouse for health and safety information and as the recognized leader in the provision of occupational health and safety (OHS) information to the OHS community. This community includes workers, their unions, employers, governments and related OHS professionals. The evaluation assessed the extent to which CCOHS has achieved this recognition with leading members of the OHS community. The activities of CCOHS are related to the promotion of occupational health and safety. The benefits are improved workplace conditions that ultimately lead to fewer illnesses, fewer injuries, and fewer deaths. Reduced time loss and increased productivity are tied to these benefits. To the extent that CCOHS information is used to reduce workplace risks, the ultimate social benefits can be very high. Current research on the value of lives saved and accidents prevented indicate that Canadian society would assign a very high value to such reductions in workplace risks. It is not possible to establish the number of injuries, illnesses and deaths prevented, let alone attribute any changes in numbers to CCOHS activities directly. Many factors influence these variables and determining the independent impact of CCOHS is not feasible. The extent of the improvements in workplace conditions, attributed by users to the information disseminated by CCOHS, is used as a proxy measure of the health and safety impact of CCOHS. Our analysis, based on interviews and surveys, found that the intended effects on workplace conditions do occur. Current funding decisions for organizations like CCOHS that reduce workplace risks were made when the estimated benefits of risk reduction were lower than current estimates. New evidence on higher payoffs to risk reduction implies that society should increase the resources devoted to reducing workplace risks. The economic rationale for government support of programs and activities has two sources. These are:
Support for CCOHS is consistent with both of these sources of rationale for the program. There will be many types of information that have a value to society exceeding their costs of production that will not be produced if firms have to recover all of their costs in doing so. This is the essence of the market failure issue for information as it applies to CCOHS. Providing the information essential for addressing health and safety concerns in the workplace to workers and managers, is a powerful and cost-effective mechanism, a strategy parallel to regulation and enforcement by governments in the area of promoting occupational health and safety. The causal linkages from activities to outputs to program impacts and finally to overall objectives are clear and have been confirmed by the results of this evaluation. CCOHS provides information through a variety of products and services, some of which are available for free. Surveys were conducted of purchasers of CCOHS products, users of CCOHS Inquiries Services and users of the CCOHS web site. Some of the key findings include:
D. Performance Measurement frameworkPerformance measurement is a tool to enable better management or performance. In this context, performance management is the means by which performance against objectives is reviewed, using appropriate performance measurement information, and decisions made regarding direction, required actions and resource allocations. CCOHS' current strategic performance measurement activity is primarily used to support:
In reviewing the structure and content of performance reporting in these areas it was apparent to us that the Centre's strategic performance measurement and reporting is more concerned with operational activities and their outputs, and less so with results achieved. We believe that CCOHS would benefit from a more strategic approach to performance measurement and reporting, that overlays (and complements) the existing operational focus. To that end, we propose that CCOHS utilize a logic model summarizing the focus of, and relationships between, CCOHS' Activities, Reach (i.e., targeted user groups, intermediaries and partners), Outputs, anticipated Key Results and longer-term Impacts. This logic model can then provide the basis for the selection of strategic performance measures and a results-based focus in its performance reports to the Council of Governors and Parliament. The logic model - presented in Exhibit I-1 - proposes four inter-dependent Key Results:
This shift to a greater results orientation in CCOHS' strategic performance measurement should provide the Council of Governors and external stakeholders with a better understanding of the extent to which intended results are being achieved or are influenced by intervening barriers or enabling factors. We have also proposed, as a starting point for further development by CCOHS management, a set of performance measures based on our understanding���drawing on the findings from the evaluation���as to the key success factors for the proposed Key Results in the logic model. The suggested performance measures combine measures of reach among targeted user groups, outputs produced and results achieved, as well as measures of the Centre's ability to generate cost recovery revenues and manage operating costs. The data required for these measures would need to come from combinations of internally generated data, such as cost, revenue and sales tracking data; and external sources, such as surveys of users and potential users of CCOHS' products and services, and feedback from stakeholders. We recommend that CCOHS use the proposed logic model and associated performance measures as the basis for a revised performance measurement framework to support performance reporting to Parliament in the annual Departmental Performance Report, and to the Council of Governors. We believe that this alternative approach will provide a clearer focus on Key Results and, by inference, the impact on occupational health and safety in the work place. The proposed logic model and associated strategic performance measures presented in this report provide a starting point for this framework. Further review and development of the framework by CCOHS' management will be necessary to ensure that it closely reflects the organization's key priorities and practicalities involved in collecting or generating the required performance data. Exhibit I-1 CCOHS Logic Model II. IntroductionA. Project ObjectivesThe Canadian Centre for Occupational Health and Safety (CCOHS or the Centre) was established to promote the basic right of all Canadians to a healthy and safe workplace. CCOHS operates under a tripartite council with representatives from workers and their unions, employers and governments. CCOHS contracted with TNS Canadian Facts and KPMG LLP to carry out this program evaluation and performance measurement study:
B. Methodologies1. Program evaluationIn order to meet the objectives of this study, a broad range of data sources have been used. A number of surveys were conducted, including: a survey of 262 current product customers; a survey of 226 users of the Inquiries Service; and a survey of 513 users of the CCOHS web site. We also conducted interviews with key informants representing business, labour and government. More details on the surveys are provided below. Two interactive surveys were conducted as part of this evaluation. The incentive for each was a random draw to win a single year's subscription to one of CCOHS's core products. Pop-Up Web-site Visitor survey: Pop-up invitations to complete a survey were placed on the following pages on the CCOHS site: CCOHS Home: http://www.ccohs.ca The survey, which was hosted on the TNS Interactive survey site, was available from July 4 to July 19, 2005. It should be kept in mind that this is a self-selecting sample. Customer survey: An invitation to participate in an online survey was e-mailed by CCOHS to 2,495 customers from their database (1,299 paying customers and 1,196 Inquiries clients) on July 7, 2005. E-mails bounced from 219 addresses, for a total of 2,276 successful broadcasts. CCOHS also e-mailed two reminders. The survey was available online until July 22, 2005. The sample file identified four product/service types:
Web-site visitor survey: 513 visitors completed the survey over the course of the field period. Customer survey: Completions were achieved with 488 respondents (262 paying customers and 226 Inquiries customers) for a response rate of just over 20%, which is a typical response rate for a customer survey in a self-completion mode. The client Internet survey questionnaire (English and French) and the CCOHS web site pop-up survey questionnaire (English and French) are appended to this report. 2. Performance measurement frameworkOur work on the development of a proposed new CCOHS performance measurement involved:
This strategic performance measurement framework provides the starting point for more detailed assessment and development by CCOHS management. The framework proposed in this report is based on the knowledge we gained during the conduct of the current and prior program evaluations of CCOHS. More extensive involvement in the development and finalization of the framework will be necessary to ensure that it truly reflects CCOHS' strategic directions, any constraints on its ability to act, and the way in which it works. C. Structure of This ReportWe provide details of the current status of CCOHS, including its mandate, corporate objectives and structure in Chapter III of this report. Chapter IV describes our findings for the program evaluation of CCOHS and discussion of issues relating to the relevance of CCOHS, program delivery, impacts and effects, cost-effectiveness and program alternatives. Findings from our assessment of performance measurement at CCOHS and proposed new measurement framework are provided in Chapter V. In the final chapter, Chapter VI, we summarize the key conclusions from the evaluation. Recommendations have been included in Chapters IV and V, following on from the program evaluation and performance measurement analysis, respectively. III. Current Status of CCOHSThis chapter of our report provides the context for the subsequent presentation of findings relating to the program evaluation and the assessment of performance measurement options. It summarizes key aspects of the Centre's mandate and key characteristics of its performance. Up until 1989, CCOHS was a fully funded government department with an annual operating budget of $10 million (approximately $14.2 million in 2005 dollars). In 1987, the Centre implemented a limited level of cost recovery aimed at recovering some of the "out of pocket" expenses associated with the production of CDs. In 1989 the Centre was directed by Treasury Board to become financially self-sufficient through cost recovery by 1991. As a result, substantial reductions in staffing and activities resulted since full self-funding was not possible. The current funding environment for CCOHS is one in which revenues from the Main Estimates and from cost recovery revenues provide most of the funding of CCOHS. For the current fiscal year, the CCOHS operating budget is $8.8 million. Of this total, appropriations from the Main Estimates account for $4.54 million (52%), contributions from provinces and territories account for 0.45 million (4%) and cost recovery revenue accounts for the remaining $3.9 million (44%). Based on the findings outlined in this report, we believe that CCOHS is at the limit of what it can achieve through cost recovery without harming its core operations. In fact, our report indicates that simply maintaining current cost recovery revenues will be difficult. Users of priced products are now making a significant contribution to the funding of core activities. The rationale for public support for core CCOHS activities is that the outputs are valuable to society because they reduce workplace risks. It appears unlikely that these services can be provided without continuing government support. A. Mandate, Objectives and Key Results CommitmentsCCOHS is an independent departmental corporation under Schedule II of the Financial Administration Act and is accountable to Parliament through the Minister of Labour. The Centre's mandate, powers and authorities are defined in the Canadian Centre for Occupational Health and Safety Act, as summarized in Exhibit III-1. The focus in the Act is on the promotion of health and safety in the Canadian workplace, and the reduction or elimination of fatalities, injuries and illnesses. CCOHS has chosen to fulfill this mandate by operating as "a source for unbiased technical information and expertise to support the efforts of governments, labour organizations, employers and individual Canadians to improve workplace safety and health". Governance is provided by a tripartite Council of Governors representing government, labour and employers. The CCOHS also operates under the Official Languages Act and, consistent with its national mandate, provides services in both French and English. The implicit expectation in the mandate and powers is that society benefits from improved levels of workplace safety and health, and that, by functioning as an information clearinghouse and disseminator, CCOHS contributes to the achievement of such improvements. The cost to Canada from workplace injuries and fatalities continues to be high���some estimates indicate that annual costs may be as high as $20 billion. As a result, even small improvements in safety performance can have significant payoffs. New evidence on higher payoffs to risk reduction cited in this report implies that society should increase the resources it devotes to risk reduction.
B. Products and ServicesCCOHS maintains a portfolio of both free and priced products and services that draw upon a core collection of occupational safety and health (OHS) information and the application of information management technologies. It also undertakes a range of collaborative projects that allows the Centre to remain on the leading edge of OHS knowledge. The areas in which CCOHS provides products and services are as follows: 1. Inquiries ServiceThe Inquiries Service provides a bilingual free-of-charge person-to-person occupational health and safety information resource for Canadians. The Centre responds to telephone and e-mail inquiries, and all inquiries are kept confidential. The information provided is unbiased and considered relevant to workers, employers, unions and governments alike. Personalized service is given to each inquirer, and the Centre provides a verbal or written response depending on the nature of the inquiry. Usually material is sent directly to the inquirer and referrals are made to other organizations as required. Approximately 14,000 telephone and written inquiries are responded to annually through the Inquiries Service. In recent years the service has been extended to include a self-serve model using the Internet. "OSH Answers" is a collection of over 3,000 frequently asked questions and answers concerning occupational health and safety that can be accessed via the CCOHS website 24 hours a day. The OSH Answers service was accessed approximately four million times by Internet users during 2004-2005. 2. Performance AreasThe Canadian Centre for Occupational Health and Safety provides a variety of information products and services designed to promote safer and healthier workplaces. These products and services are organized by CCOHS into the following eight categories, including a brief description of the purpose of each:
CCOHS has an extensive suite of databases available to support its information services, from which a wide variety of OHS products and services are made available in various electronic formats (CD/DVD-ROM, Internet, Intranet, as well as hardcopy). These databases are used in a variety of work place applications such as developing personal protective programs, conducting workplace risk assessments, assisting emergency responders, developing first aid and medical programs, developing training, meeting regulatory compliance, proper storage, handling and transporting chemicals and for research. IV. Program Evaluation FindingsA. Rationale and Relevance1. The role of CCOHSThe evaluation of the rationale for the programs and activities of CCOHS is based on a review of the initial mandate as it relates to current activities. The mission of CCOHS is to promote the fundamental right of Canadians to a healthy and safe working environment. As a national institute, CCOHS undertakes a wide range of activities to achieve its objectives. These activities include the following:
These activities generate a flow of information. By its nature, the impacts of information may be widespread and difficult to attribute to the source activities of CCOHS. The provision of information is obviously a valuable activity. However, tracing out all of the ultimate impacts and effects of information provision is a difficult task for all kinds of information, not just for health and safety information. The ultimate objective of CCOHS is to improve workplace health and safety. The activities described above involve CCOHS being recognized as a national clearinghouse for health and safety information and as the recognized leader in the provision of useful occupational health and safety (OHS) information to the OHS community. This community includes workers, their unions, employers, governments and related OHS professionals. This evaluation assesses the extent to which CCOHS has achieved this recognition with leading members of the OHS community. It is not possible to establish the number of injuries, illnesses and deaths prevented, let alone attribute any change in numbers to the activities of CCOHS directly. Many factors influence these variables and as a result, it is not possible to determine the independent impact of CCOHS. The extent of improvement in workplace conditions, attributed by users to the information disseminated by CCOHS, is used as a measure of the health and safety impact of CCOHS. Our analysis, based on interviews and surveys, found that the intended effects on workplace conditions do occur. 2. Potential benefits of CCOHS activitiesThe activities and products of CCOHS consist of information that is related to the promotion of occupational health and safety. The benefits that potentially flow from this consist of improved workplace conditions that ultimately lead to fewer illnesses, fewer injuries and fewer deaths. Reduced time loss and increased productivity are tied to these benefits. Linking improved workplace safety to any particular CCOHS product or activity does not seem feasible. However, to the extent that CCOHS information is used to promote workplace change, the ultimate social benefits and increases to productivity can be very high. The information generated by CCOHS is distributed in a series of products that users purchase and through services made available to the public at no cost. How "valuable" are these products and services? From an economic perspective, the products that are paid for must have a value to purchasers of at least the purchase price otherwise they would not purchase the product. However, the value to society of the paid products is measured as the value of the improvement in occupational health and safety that can be attributed to them. This value to society can exceed by a large margin the private value (the amount paid for the CCOHS product) because the information, once purchased, can be used to produce safety benefits for many workers. The value to society of the activities and products that are provided with no fee is also measured as the value of the improvement in occupational health and safety that can be attributed to them. Even though users pay no explicit price, the value to society can still be very high. There is an extensive economics literature on the benefits of policies to make workplaces safer. These benefits come from reduced risks of injuries, illness or death and improved productivity, reduced absenteeism, increased competitiveness of Canadian enterprises and reduced health-care costs. It is important to note, however, that the benefits of the activities of CCOHS to reduce risks have many points of comparison in other government functions. That is, there are many analogies to the question of how to determine the benefits to Canadians of reduced health and safety risks in the workplace. The most recent research results on workplace risks and the value of measures to reduce these risks indicate that the benefits of reducing these risks are much higher than had previously been estimated. Gunderson and Hyatt (2001) provide new Canadian evidence on the benefits of reducing workplace risks. Their evidence indicates that the benefits of reducing workplace risks are more than four times higher than earlier estimates. Leigh (2001) has provided related results for the United States More specifically, the Gunderson and Hyatt estimates suggest that the benefits to society of reducing the risks that would lead to one fatality are as high as $13 million while the benefits of reducing the risks that would lead to one non-fatal injury are approximately $20,000. 3. Injuries, fatalities and program rationaleThe original rationale for CCOHS was to serve as a national centre that would contribute to reductions in occupational injuries and illnesses. This core rationale remains valid. Many sources of data could be used to support this assertion. The most frequently cited data in the literature on occupational health and safety are data dealing with occupational injuries and deaths. The standard data sources relate occupational injuries and fatalities to the number of workers employed (the population at risk for occupational hazards). For Canada in 2003, the employed population consisted of 15.8 million persons. Of this total, approximately 349,000 suffered an injury requiring time loss from work. Occupational fatalities totaled 963. Exhibit IV-1 shows comparative data on occupational injuries and fatalities for 2003 (the latest year for which published data are available), 1998 and 1970. There have been significant reductions in injuries and fatalities relative to numbers of workers but the numbers remain high. The reductions likely result from a variety of factors including changing technologies, better educated workers and industry initiatives together with occupational health and safety policies and programs, including the activities of CCOHS. Exhibit IV-1 Occupational Injuries and Fatalities in Canada
Source: Human Resources Development Canada, Occupational Injuries and their Cost in Canada, 1993-1997, 1999, and Statistical Analysis: Occupational Injuries and Fatalities, Canada, 1994-1998, 2000. For 2003: Work Injuries and Diseases National Work Injuries Statistics Program, Association of Workers' Compensation Boards of Canada, 2001-2003 and Labour Force Survey2003, Statistics Canada. Small and achievable reductions in risk can generate large benefits to society. An overall perspective on program rationale as it relates to injuries and fatalities can be provided by combining the data in Exhibit IV-1 with the data on the value of reducing the risks of deaths and injuries cited earlier. Note that these values were $13 million per fatality and $20,000 per non-fatal injury. Combining these data with actual fatality and injury data for 2003 from Exhibit IV-1 produces a dollar value of almost $19.5 billion from reducing the risks of all of these accidents to zero. Put differently, the cost to Canadian society of the risks leading to injuries and fatalities in 2003 was approximately $19.5 billion. This analysis relates directly to the CCOHS rationale. If CCOHS activities, for example, were to lead to a 1% reduction in these risks, 3,448 injuries and 9.6 fatalities would be avoided. The research outlined above indicates that Canadian society would assign a value of $194 million to this reduction. 4. Economic aspects of program rationaleThe economic rationale for government support of programs and activities has three sources. These are:
Support for CCOHS is consistent with all of these sources of rationale for the program. The development of a formal rationale for a program such as CCOHS is the first step in the evaluation process. In the economic efficiency case, the evaluation seeks to identify market failures and then relate program activities to the market failure. That is, for the rationale to fit well, the program must undertake activities targeted to rectifying the original market failure. The evaluation has the task of determining whether program activities as they actually take place are plausibly linked to the underlying market failure that provides the rationale for the program. In the case of CCOHS, this chapter focuses on the failure of markets to provide adequate amounts of information related to occupational health and safety as the primary source of program rationale. This type of market failure is discussed in the context of the extensive literature in applied economics on the problems of information as a commodity. In a market context, there are many ways by which users of information are able to avoid paying either entirely or an amount equal to the full value they receive from its use. The excess of the amount that a user of information would have been willing to pay relative to what is actually paid is called a spillover benefit. For example, worker A may pay to acquire safety information. Worker B then observes what worker A does to work more safely and follow A's example. In this case, the benefits to B of using the information paid for by A constitute a spillover benefit. Related to this example, it is clear that the safety information (used by A and B) is costly to produce and disseminate. In the above example, only one of the beneficiaries of this information contributes to its cost of production. In this context, there will be many types of information that have a value to society exceeding their costs of production that will not be produced if firms have to recover all of their costs in doing so. This is the essence of the market failure issue for information as it applies to CCOHS. There is a related source of program rationale that focuses on the role of information related to occupational health and safety regulatory requirements. Workplaces are complex and differ from each other in a variety of ways. Governments enforce health and safety regulations but inspection is costly and even well-trained inspectors will not be as sensitive to specific workplace issues as are workers in that workplace. Providing the information essential for improving workplace health and safety to workers, managers and employers is a powerful and cost-effective mechanism and constitutes a strategy parallel to regulation and enforcement by governments in the area of promoting occupational health and safety. Thus, the activities of CCOHS relate directly to its mandate and to the economic and related sources of rationale for the program. CCOHS promotes occupational health and safety. It acts as a national centre for the production and dissemination of health and safety information used by a large number of Canadian workers. The causal linkages from activities to outputs to program impacts and finally to overall objectives are clear and have been confirmed by the results of this evaluation. B. Program Delivery1. Program resourcesThe resources available to CCOHS are used to support the range of activities and products described below. Many of the resources of CCOHS support both the products that are sold and the services provided to Canadians without a fee. These services draw on the information base and expertise of CCOHS, some of which is generated in producing the products that are sold. There are, in other words, common costs associated with the paid products and the free services. Calculations elsewhere in this report show that some of the costs of providing the free services are shared with the paid products. That is, in the absence of the paid products, it would cost more to provide the free services than is now the case. Based on interviews conducted with key informants, there is widespread support for the continued provision of free services; particularly non-technical information provided to individual workers. 2. Activities and productsThe activities and products of CCOHS can be summarized in the following categories:
a) Inquiries Service The Inquiries Service answers questions in two ways ��� through its person-to-person information service and through its Internet service (OSH Answers). TNS Canadian Facts conducted a survey of 226 current users of the Inquiries Service. Of the 226 respondents, 39% said that they had used the Inquiries Service at least once a month over the previous twelve months. We asked respondents how many individuals within their organization access the information provided by CCOHS. These answers showed substantial use: approximately 13% of respondents indicated that more than 100 individuals in their organization access CCOHS information. The information obtained from the CCOHS's Inquiries Service by respondents is put to a number of uses, the most common of which are:
The survey responses indicate that most (74%) of the uses of CCOHS information relate to either current or future changes to the workplace that may result in improved workplace health and safety. This study assessed key performance indicators such as timeliness, ability of staff and courtesy of staff, and the overall satisfaction with the service to provide a series of measures against which to measure the quality of the service. Respondents rated the CCOHS Inquiries Service highly for all indicators. Ratings for CCOHS are summarized in Exhibit IV-2. Exhibit IV-2 Level of Satisfaction with CCOHS Inquiries Service
Source: Data for CCOHS is from a survey of users conducted by TNS Canadian Facts. Respondents were also asked about the importance of CCOHS in meeting their occupational health and safety requirements. This included a series of questions about the extent to which information products from other organizations could have been used, the relative usefulness of these other products and whether there are occupational health and safety products not offered by CCOHS that would be useful to respondents. The main findings to emerge from these questions are:
Other products provided by CCOHS for which there is no charge include The Health and Safety Report. This report is a monthly electronic newsletter that has provided current occupational health and safety news since its inception in January 2003. The Health and Safety Report is another product that serves CCOHS' mandate to provide unbiased, relevant information to all Canadians that supports responsible decision-making and promotes safe and healthy working environments. TNS Canadian Facts did not survey readers at this time but CCOHS did conduct a readership survey of its 8,000 subscribers that was published in June 2005. Key findings of the readership survey conducted by CCOHS are as follows:
b) Information products sold by CCOHS CCOHS markets a wide range of products in a variety of formats. The major products of CCOHS are described in Chapter III. The information products sold by CCOHS consist of pay for use web products, pay for use DVD/CDs and print publications. TNS Canadian Facts conducted a survey of 262 current purchasers of these products. Of the 262 respondents, more than half had been purchasing the product or service for more than three years. We asked respondents how many individuals within their organization access the information purchased from CCOHS. These answers showed substantial use: approximately 12% of respondents indicated that more than 100 individuals in their organization access the information purchased from CCOHS. The CCOHS products purchased by respondents are put to a number of uses, the most common of which are:
Respondents who purchased CCOHS products or services were asked to rate CCOHS products in terms of a series of performance indicators such as timeliness, clarity, usefulness and the overall satisfaction with the service to provide a series of measures against which to measure the quality of the service. Respondents rated CCOHS products quite highly for all indicators. Ratings for CCOHS products are summarized in Exhibit IV-3. Exhibit IV-3 Level of Satisfaction with CCOHS Products (% of respondents indicating either very satisfied or satisfied)
Source: Data for CCOHS is from a survey of users conducted by TNS Canadian Facts. The results in Exhibit IV-2 and IV-3 show very high levels of user satisfaction. In order to provide a better assessment of the level of satisfaction of CCOHS clients, a benchmark was needed. The Treasury Board and the Canadian Centre for Management Development have developed a Common Measurement Tool (CMT) that is intended for use by federal departments and agencies in assessing their level of client satisfaction and for comparing their results to those of other departments and agencies providing similar services. Benchmarks of client satisfaction used for comparing the level of satisfaction with CCOHS products and services for this report appear in Citizens First 2000 (Erin Research). Erin Research asked survey participants to rate services they had used in the past year using a five point scale where 1 is "very poor," 5 is "very good," and 3 is "neutral." A total of 17 federal departments and agencies were rated in this way with a mean score of 61. According to Citizens First 2000 service providers should assess their performance against other providers of similar services rather than against overall aggregates. In order to do this TNS used Health Canada Information Services and Information Services (in general) as the basis for comparison; the mean score for these two services was 56. We believe these provide a similar product/service to CCOHS ��� namely information. This benchmark information is only available for assessing the overall level of satisfaction. Other key performance indicators assessed in the CMT include timeliness, knowledgeable/competent staff, and courtesy of service. Citizens First summarizes the results of ratings made by 6,040 respondents and aggregates ratings of federal, provincial and municipal services. Comparisons made with CCOHS should be viewed with some caution since different types of services are being compared and the Erin Research results are not contemporaneous with our 2005 surveys. Nevertheless, in the absence of other benchmarks they serve to illustrate the comparative level of satisfaction with CCOHS products and services with regard to timeliness, ability of staff and courtesy of staff. CCOHS exceeds a sample of Federal departments on these measures by substantial amounts. In terms of value for money, 96% of respondents who could answer this question said that CCOHS products were above average value for money or average value. Approximately one third of all respondents to this question put CCOHS in the category of above average value for money. Respondents were also asked about the importance of CCOHS in meeting their occupational health and safety requirements. This included a series of questions about the extent to which products from other organizations could have been used, the relative usefulness of these other products and whether there are occupational health and safety products not offered by CCOHS that would be useful to respondents. The main findings to emerge from these questions are:
These findings suggest that CCOHS information is highly regarded by its varied clients. In comparison with earlier evaluation work for CCOHS, the paid products results continue to show high satisfaction results for all measures. For the Inquiries Service, most of the measures compare favourably with the results of the previous evaluation. The only exception that CCOHS may wish to pursue deals with the overall satisfaction rating for the Inquiries Service (in Exhibit IV-2). Although still above the level for the sample of Federal departments, the score is lower than in previous evaluations. Note that the overall satisfaction score is the result of separate question, not an average of other scores. Note also that the different surveying formats (telephone in 2000 versus internet survey in 2005) may be related to these results. c) Web site The CCOHS web site is extremely active. Data collected by CCOHS indicate approximately 1.4 million annual unique visitors to the main site, logging over 3 million sessions. In addition, there are four million annual requests for information to OSH Answers. Traffic measures by themselves do not reveal the potential impacts of web site use. Our survey of the users of the CCOHS web site, however, provides additional perspective on the potential impact. Survey respondents indicated that the main reason (in order of frequency of mentions) they came to the CCOHS web site were to:
Respondents said that they were relatively successful in finding what they required on the CCOHS web site. Among respondents, 41% found what they wanted on the CCOHS site, another 6% found related useful information, only 2% said they did not find what they wanted and 50% said they were still looking at the time of the web survey. The information that respondents found on the CCOHS web site was being used most frequently for the following purposes:
As noted above, attributing actual health and safety improvements to CCOHS is difficult. However, indirect evidence is provided from responses to the web survey. Respondents were asked how many people in their organization could or would benefit from the CCOHS information. Among respondents, 26% said that 1-10 persons would benefit, 36% said between 11 and 100 persons, 20% said 101 to 500 persons would benefit and 19% said that more than 500 would benefit. More details on the reach of CCOHS are provided below. The most frequently mentioned job responsibilities of respondents providing this information were safety (28%), industrial hygiene/medical (14%) and regulatory compliance (6%). d) Interviews with Key Informants Interviews were conducted with 20 key informants representing a range of CCOHS stakeholders, including business, labour, government, and non-governmental organizations. Respondents belonged primarily to Canadian organizations, although one international respondent was included. Most interviewees (85%) were direct users of CCOHS products and services, and six were past or present members of the Council of Governors. Respondents were first asked to describe the CCOHS products and services used within their organization. The most frequently cited products used were OSH Answers, CHEMINFO, MSDS products, Inquiries Services, and the website in general. A small number of respondents had contracted CCOHS to do specialized work for them (i.e. creating an OHS website, specialized publications). Others reported using the CD-ROMs, database subscriptions, mailing list for OHS professionals, ergonomic guides, poison control centre databases, and links to international websites. CCOHS products and services were normally used as reference tools to answer client or employee questions, but were also used in provincial/territorial libraries, and in support of regulatory enforcement work, product development, best practice development and training sessions. A few interviewees reported using CCOHS products and services in support of research projects. Most interviewees reported sourcing OHS materials from other organizations when CCOHS did not itself offer the necessary product. These materials were primarily safety publications, including books, magazines (e.g. OHS Canada), health and safety journals, and videos (e.g. Summit). Respondents also accessed information related to standards, regulations, codes of practice, best practices, electronic learning, and toxicity databases. These materials were accessed directly from the companies that produce them, through other provincial health and safety organizations, Canadian Standards Association (CSA), CGSB Canadian Government Standards, and American organizations such as National Institute for Occupational Safety and Health (NIOSH) and Occupational Safety and Health Administration (OSHA). Generally, organizations' use of CCOHS products and services has been constant over time, as only one-third of respondents reported a change. Of these, about half commented that their usage of CCOHS products and services had increased, while the rest felt it had declined. Those who reported increased use of the products and services offered varying reasons for this change, including an increased awareness of and interest in the products and services CCOHS provides. Decreases in use were principally attributed to increased fees or to the increased accessibility of free information over the Internet. Most respondents felt that CCOHS' products and services had met their organizations' needs quite well, one describing them as "cheap, efficient, good reference points." Others noted that the CCOHS products and services are credible, unbiased, and good resources for employees. As CCOHS is a neutral source of information, organizations do not have to validate its products and services, in contrast with information received from organizations perceived to be less independent. Generally, CCOHS products and services are perceived to address needs not met by other information sources. In particular, the listserve of OHS professionals, chemical products (i.e. MSDS Management Service, CHEMpendium), and databases are resources unique to CCOHS. CCOHS is also perceived to have a "single source advantage," in that a multitude of different products and services are available through CCOHS, whereas other organizations might only offer a certain type of product. A small number of respondents noted that free information is increasingly available on-line. Respondents also provided suggestions for improvements to the products and services offered by CCOHS, centering on issues of cost, new products or services, and operations:
In general, interviewees believed CCOHS has a strong reputation within Canada. OHS experts, professional leadership, unions, workers and employers with health and safety committees tend to be quite familiar with CCOHS. This high level of awareness was attributed primarily to CCOHS' own efforts, such as partnerships it has forged with other organizations (i.e. Safe Communities Foundation), and the high level of visibility maintained by key CCOHS officers. Some respondents, however, commented that "provincial health and safety organizations and CCOHS are the best kept secret at the workplace level," as the general Canadian public tends not to be aware of CCOHS. In particular, interviewees mentioned that awareness of CCOHS is quite low in Canada's North and West. Within Canada, CCOHS is best known for the "easy to access, clear, accurate and useful" information it provides. In fact, one respondent enthusiastically described the centre as a "national asset." Respondents noted that CCOHS does a good job of synthesizing research findings into unbiased information products that are useful and easy to understand for health and safety professionals, employers and workers alike. CCOHS also possesses a strong international reputation. It is perceived as one of the world's leading sources of OHS information, with a higher level of recognition than many other OHS national institutions. According to one respondent, this strong reputation "speaks to the credibility of the product and respect for the institutional model." Europe, US, Mexico, Brazil, South Africa were all noted specifically as users of CCOHS products and services. A small number of respondents were either unsure about CCOHS' international reputation, or did not think it is very well known. Based on interview responses, it appears that at high levels, CCOHS has a strong international reputation, but at the lower echelons of management, international OHS professionals are frequently unfamiliar with CCOHS. Internationally, CCOHS is known for two things: the quality of the products and services available, and the international accessibility of its work. The information available through CCOHS "based on the best knowledge from the best institutions around the world," and CCOHS is known as a "one-stop shop for international regulations" and links to international websites. CCOHS is also distinguished by its international collaborative work, and the online accessibility of its products, including particularly its Spanish translations. All interviewees agreed that CCOHS has contributed strongly to the development and dissemination of knowledge, one noting that CCOHS is the "best source of information on established risks to workers' health." CCOHS is also known as a source of information on non-regulated topics, such as shift work. By providing accessible information products that synthesize recent research, including international research sources (e.g. WHO, UN), CCOHS has clearly increased the amount of information available, and has therefore contributed to the growth of knowledge of OHS among Canadians. Most respondents felt, however, that CCOHS has not contributed to the growth of research in OHS. Some interviewees noted that in the past, CCOHS had been actively engaged in research, but that more recently it has gravitated towards being primarily an information provider. It was suggested that by increasing the level of knowledge, CCOHS is likely indirectly contributing to the growth of research. Many believed that conducting pure research is outside CCOHS' mandate, and were therefore not concerned by the perception that CCOHS has not actively contributed to a growth in OHS research. CCOHS is generally believed to have improved workplace safety practices. By making information about workplace hazards and best practices available to companies, workplace health and safety committees, activists and unions, CCOHS has helped workplaces modify their safety practices, and has generally improved awareness of workplace health and safety issues. Some evidence of this positive impact was cited, including a decreased injury rate, increased enforcement of health and safety regulations, and improved quality of life. A few respondents, however, pointed out that since workplace safety is a combination of many factors, it is difficult to isolate CCOHS' contribution. Most respondents felt that CCOHS has been quite effective in facilitating partnerships among workers, unions and government with respect to OHS. This success was attributed primarily to two factors: CCOHS' tripartite board structure, and its dissemination of information.
A small number of respondents argued that, although it may facilitate high-level partnerships, CCOHS has not affected relationships at the workplace level. The issue of charging for products and services is important for CCOHS because of the nature of the information and the nature of the organization. Respondents were therefore asked several questions regarding the policy of charging for CCOHS products and services About half the interviewees believed that the current charges for products and services limit the accessibility of the information; the rest believed it has little impact. Those who felt the charges limit accessibility perceived the cost barrier to be most significant for workers, small and medium sized employers, health and safety committees, unions and students. These users frequently have only modest budgets, and may not be able to afford the information they need. Others felt that the current prices attached to CCOHS products and services do not inhibit accessibility, as government, larger companies, associations, and other organizations can usually afford the information. Furthermore, since so much information provided by CCOHS is free, that those who require more specialized information are generally in the position where they can afford to pay for it. Nearly all interviewees felt that increasing existing fees or adding new fees would inhibit the accessibility of CCOHS' products and services. They suggested that users with fixed budgets might be unable to afford the increased prices, and workers would become even more dependent on employers for access to this information. Many believed the reaction of users would be to look elsewhere for cheaper products, or to stop using the products altogether. These respondents argued that since CCOHS is a national, publicly funded service, its first priority should be accessibility. A small number of respondents, however, felt that charging more for products whose costs are not currently covered would have little effect on accessibility. They felt consumers would not react negatively to justifiable price increases, and that those purchasing this type of specialized information (i.e. databases) accessible only through CCOHS would likely continue to pay for it, even at higher rates. The general opinion among respondents is that it is "critical" for CCOHS to provide an extensive range of services for free. Many pointed out that CCOHS has a mandate to provide Canadians with current and accessible information on occupational health and safety, and argued that, to satisfy this mandate, CCOHS must provide free access to its information products. Others felt that CCOHS products and services should be provided for free for the sake of workers, unions and small businesses who require this information in order to create and maintain safe and healthy work environments. In terms of establishing a proper balance between for-fee and for-free products, the dominant opinion was that products for workers and small businesses should be freely accessible, while larger companies may be charged for products and services. Most believed that specialized collaborative services should also be provided on a for-fee basis. In particular, items of direct benefit to workers, small business and the general public should be free, such as "tip sheets," WHMIS information and the CCOHS pocket guides. One respondent suggested that to determine which information should be free, CCOHS could ask itself "what can benefit workers on the shop floor?" Any corresponding products or services should then be offered free of charge. Others maintained that all information should be provided for free, arguing that CCOHS "can't sacrifice [its] principles to raise money." These respondents pointed out that not everyone who needs the information would be able to afford it (i.e. SMEs), and that it is difficult to distinguish between those who can and cannot afford to pay. Some interviewees were concerned that many workplaces would forego the information altogether, thereby limiting CCOHS' ability to positively influence workplace health and safety. One respondent suggested that CCOHS could address the issue of cost by charging only for hard copies of information (i.e. paper products, CD-ROMs), while making all information available for free electronically. Some interviewees noted that the issue of cost-recovery raises fundamental questions regarding CCOHS' mandate, objectives, and target market. If CCOHS' objective is to reach workers, increasing fees could inhibit the achievement of this mandate. On the other hand, if CCOHS' objective is to reach companies, increasing fees might be an acceptable action. Most interviewees felt that government funding, both provincial and federal, should be key in covering the costs not covered by sales. In particular, it was felt that the federal government should take primary responsibility for funding the centre. Many argued that since the federal government had created CCOHS and had provided it with a mandate, the federal government should therefore be responsible for ensuring that CCOHS has the necessary funds to operate. Some also noted that the federal government has a duty to "safeguard the health and safety of working Canadians." It was believed that the only way for CCOHS to meet its mandate effectively would be if it had some steady source of funding upon which it could rely, such as a federal government endowment. For the sake of comparison, one respondent commented that in many countries, of a similar size as Canada, the level of annual funding for a national OHS organization is between $50 and $100 million dollars. Interviewees agreed that provincial governments should also make some contribution to CCOHS, but at a lower level than the federal government. It was observed that some provincial governments already contribute to CCOHS' operations, and should continue to do so at the same level. Others felt that the provinces could probably afford to contribute more. A small number of respondents suggested analysing the extent to which each province uses CCOHS, and creating contribution agreements that reflect the levels of usage. A small number of respondents advocated pursuing partnerships with business and industry, through collaborative arrangements. This was primarily because these respondents thought it highly unlikely that government, either federal or provincial, would increase its contributions. C. Impacts and EffectsThe logical links relating CCOHS programs and activities to improved workplace outcomes are clear. CCOHS assembles and disseminates information on health and safety. Some of this information is directed to workers who are dealing with a workplace issue and seek information from the Inquiries Service and/or the web site. The results of this evaluation confirm that many individual workers use CCOHS to provide them with assistance on a specific workplace health or safety issue. There is strong support for the continuation of this free service. The CCOHS information products that are sold also have direct application in the workplace. Our survey results show frequent use of these products by many people in the organisations that purchase them. The entire range of workplace partners-workers, their unions, employers and governments use CCOHS information products to promote occupational health and safety. We base the assertion of widespread use of CCOHS information on several sources of evaluation evidence. This evidence consists of survey data from customers, users of the Inquiries Service, users of the web site and key informants. Evidence from these sources provides a consistent picture of widespread use of CCOHS health and safety information. Although these data cannot be linked to accident or illness data, the widespread and frequent use of CCOHS information implies that there will be important workplace impacts. The interviews with key informants confirm the data from the surveys that imply important workplace impacts of CCOHS. Interviews were conducted with 20 key informants from a range of organizations representing business, labour, government and non-governmental organizations. Key informants were chosen based on their involvement in the OHS community internationally and within Canada. Ideally, the analysis of program impacts would relate the logical program linkages to actual outcome data. We know that the use of appropriate information on safe workplace practices will reduce risks. This evaluation has generated substantial data on the use of CCOHS information. Exhibit IV-4 summarizes this information on the reach of CCOHS in the workplace. These data indicate that there can be little reasonable doubt about its widespread use. However, in our view, any attempt to directly link specific information sources with specific reductions in occupational accidents and illness would not be credible. Exhibit IV-4 Reach of Information Within Organizations Using CCOHS
D. Cost-Effectiveness and Program AlternativesFrom the point of view of its customers, the products sold by CCOHS represent good value for money and can be described as cost-effective investments. The savings in time costs for health and safety professionals who would have to access information in other ways appear to be substantial. According to respondents, the costs of CCOHS products compare favourably with those of competing products. Like most other information providers, CCOHS has substantial fixed costs associated with developing and maintaining data bases. These fixed costs are necessary to generate and keep current the information that is sold in the form of CCOHS products and to provide the information base for the Inquiries Service and the web site services that are provided free of charge. Again as is the standard case with information, the high fixed costs of producing information stand in contrast to the relatively low marginal cost of distributing the information. From a public sector perspective, the further question related to cost-effectiveness is the extent to which CCOHS represents a cost-effective use of the tax dollars provided to it. This question is a difficult one on which to provide a definitive answer because this approach to cost-effectiveness requires information on the value of the complete set of services provided by CCOHS. This evaluation indicates that the value of these services is likely to be high. There are many users of the substantial quantities of occupational health and safety information produced and disseminated by CCOHS. It is our assessment that CCOHS produces very large quantities of useful information for a relatively modest public expenditure. However, the attribution problem, discussed earlier, means that we cannot pinpoint cost-effectiveness in terms of societal value for money. That is, we know only the expenditure on CCOHS, not the value of its services in dollar terms.
V. Proposed Performance Measurement FrameworkA. Context for Performance MeasurementOur approach to the development of performance measurement is based on the view that performance measurement is a tool to enable better performance management. In this context, performance management is the means by which performance against objectives is reviewed, using appropriate performance measurement information, and decisions made regarding direction, required actions and resource allocations. Related and supporting elements include alignment of lower-level programs with the overall strategic direction, risk management systems and knowledge management processes. Performance measurement is critical to the functioning of the performance management framework. An effective performance measurement system should, ideally, be able to:
The development of performance measurement systems should also be responsive to the requirement of the RMAF (Results-Based Management and Accountability Framework) policy to monitor how policies, programs and initiatives are progressing, and to use such information for reporting on the achievement of planned results and outcomes. As such, the performance measurement strategy should provide managers with knowledge they require in order to manage their policy, program or initiative on an ongoing basis. It can provide reassurance that outcomes are unfolding as expected, or can serve as an early warning that the planned results are not occurring (which could lead to a decision for additional research to determine root causes and contributory factors). B. Features of the Current Approach to Performance MeasurementCCOHS' current strategic performance measurement and reporting activity is primarily used to support:
In reviewing the structure and content of performance reporting in these areas it was apparent to us that the Centre's strategic performance measurement and reporting is more concerned with the performance of activities and their outputs and less so with results achieved. In other words, it provides information and statistics on activities undertaken or in progress and outputs generated, but only limited information relating to the results that flow from the dissemination of OHS information and provision of OHS training, and application of such knowledge in the Canadian workplace. This point is illustrated in the following section, when the current approach is compared to a proposed new framework for performance measurement. C. Proposed Foundation for Future Performance MeasurementWe believe that CCOHS would benefit from a more strategic approach to performance measurement and reporting, that overlays (and complements) the more operational focus of reporting related to the thirteen Performance Areas listed above and the activity/output orientation in the annual performance reports to Parliament. This approach should provide CCOHS' management and stakeholders with a clearer understanding of the way in which it utilizes the resources available to delivers on the Centre's mandate to enhance the physical and mental health of the working people. To that end, we propose that CCOHS utilize a logic model summarizing the focus of, and relationships between, CCOHS' Activities, Reach (i.e., targeted user groups, intermediaries and partners), Outputs, anticipated Key Results and longer-term Impacts. This logic model can then provide the basis for the selection of strategic performance measures and a greater results-based focus in its performance reports to the Council of Governors and Parliament. Logic models are used to summarize the overall rationale and design of programs, and to facilitate the development of performance measurement and evaluation strategies. An effective logic model is one that summarizes, and demonstrates, causal relationships between resources deployed, activities performed and results to be achieved. Exhibit V-1 presents the proposed logic model for CCOHS, which is based on insights from the program evaluation analysis and our review of the current performance reports. The logic model proposes four inter-dependent Key Results:
These proposed outcomes are achieved through the operation of a technical infrastructure for the assembly, storage and distribution of CCOHS' five different types of Outputs:
If the current set of Strategic Outcomes covered in the DPR/RPP process is overlaid on the proposed logic model (as shown in Exhibit V-2) it is apparent that more of the Centre's reporting to external stakeholders focuses on operational activities, with proportionately less attention paid to outputs and key results. There is also some overlap between the current strategic outcomes, particularly for: Provide Canadians with information and Accessibility and availability of information for Canadians. Exhibit V-1 CCOHS logic model
Exhibit V-2 Current focus in CCOHS' strategic performance reporting
D. Proposed Focus for CCOHS' Strategic Performance Measurement and ReportingA shift to a results orientation in CCOHS' strategic performance measurement should provide the Council of Governors and external stakeholders with a better understanding of the extent to which intended results are being achieved or are influenced by intervening barriers or enabling factors. Typically, organizations have to deal with the issue of "how few" performance measures are needed, not "how many". Experience with performance measurement systems at other public and private organizations suggests that the ideal is to have no more than about 3���5 distinct performance measures for each intended Key Result. These measures may be quantitative or qualitative. Development of such a system for CCOHS can be facilitated by a number of actions by the Centre's management to further develop or refine the logic model and associated suite of performance measures to ensure they capture the central essence of CCOHS's operations and intended results. These steps involve:
A further consideration in the selection of results measures is the need to demonstrate, to the extent possible, the contributions that CCOHS activities make to the application of OHS information and knowledge to improve workplace practices in Canada. Performance measurement at the operational level is more likely to be concerned with the performance of key processes and activities (that underpin or drive the achievement of the intended Key Results), and outputs generated. In other words, operational performance management is more focused on those things that management is able to control while strategic performance management is more concerned with the results achieved and overall direction of the business. Exhibit V-3 proposes a starting point for the development and confirmation of a suite of strategic performance measures for CCOHS, based on the Key Results proposed in the logic model, and the types of critical success factors that appear to be shaping CCOHS' performance. The CSFs and proposed measures presented in Exhibit V-3 are based on the findings from our analysis and need further critical review and development, drawing upon management's knowledge and understanding of the way CCOHS' business works in practice, to ensure that performance measurement and reporting "measures what really matters". The suggested performance measures combine measures of reach among targeted user groups, outputs produced and results achieved, as well as measures of the Centre's ability to generate cost recovery revenues and manage operating costs. The data required for these measures would need to come from combinations of internally generated data, such as cost, revenue and sales tracking data; and external sources, such as surveys of users and potential users of CCOHS' products and services, and feedback from stakeholders. Further development of the performance measurement framework also needs to take the ease, or difficulty (and costs), of collecting and compiling the desired information. We recommend that CCOHS use the proposed logic model and associated performance measures as the basis for a revised performance measurement framework to support performance reporting to Parliament in the annual Departmental Performance Report, and to the Council of Governors. We believe that this alternative approach will provide a clearer focus on Key Results and, by inference, the impact on occupational health and safety in the work place. The proposed logic model and associated strategic performance measures presented in this report provide a starting point for this framework. Further review and development of the framework by CCOHS' management will be necessary to ensure that it closely reflects the organization's key priorities and practicalities involved in collecting or generating the required performance data.
Exhibit V-3 Starting point for a new CCOHS strategic performance measurement framework
VI. Key ConclusionsA. CCOHS Plays an Important RoleThe core objective of CCOHS is to provide Canadians with information about occupational health and safety that is trustworthy, comprehensive, and intelligible. The information facilitates responsible decision-making, promotes improvements in the workplace, increases awareness of the need for a healthy and safe working environment, and supports occupational health and safety education and training. To meet this objective, CCOHS maintains a portfolio of both free and priced products and services that draw upon a core collection of occupational safety and health information and the application of information management technologies. To be effective on a continuing basis, the Centre must remain on the leading edge of workplace health and safety knowledge. B. CCOHS activities have strong links to workplace changesThe activities and products of CCOHS consist of information that is related to the promotion of occupational health and safety. The benefits that potentially flow from this consist of improved workplace conditions that ultimately lead to fewer illnesses, fewer injuries and fewer deaths. Reduced time loss and increased productivity are tied to these benefits. An evaluation design that relies on linking increased workplace safety to any particular CCOHS product or activity is not feasible. However, satisfaction with and use of information provided by CCOHS is a good proxy indicator of likely improvements in workplace safety. All of the interview and survey data that we collected support the conclusion that here is a high degree of satisfaction with and widespread use of CCOHS products and services. To the extent that CCOHS information is used to promote workplace change, the ultimate social benefits and increases in productivity can be very high. C. A Stronger Focus on Results is Needed in CCOHS' Strategic Performance MeasurementCCOHS' current strategic performance measurement and reporting activity is primarily used to support reporting to the Council of Governors, via periodic reports by the President/CEO, and to external stakeholders and Parliament, via the Annual Report, Performance Report and Report on Plans and Priorities. The information reported to these audiences is more concerned with the performance of activities and their outputs and less so with results achieved, in terms of putting OHS information in the hands of Canadians and having that information applied in the workplace. We believe that CCOHS would benefit from a more strategic approach to performance measurement and reporting, and builds on the thirteen Performance Elements used by the Centre for resource planning and budgeting, and activity-based performance monitoring. To that end, we have proposed a logic model summarizing the focus of, and relationships between, CCOHS' Activities, Reach (i.e., targeted user groups, intermediaries and partners), Outputs, anticipated Key Results and longer-term Impacts. This logic model then provides the basis for a proposed new strategic performance measurement framework intended to provide CCOHS' management and stakeholders with a clearer understanding of its progress in achieving anticipated results and delivering on its mandate to enhance the physical and mental health of the working people. The proposed strategic performance measurement framework provides a starting point for further work by CCOHS' management to confirm the anticipated Key Results, and factors that have to be managed most closely to enable the targeted results to be achieved, and select and refine the associated performance measures. This iterative process of review and refinement of the framework is also necessary to ensure that it truly reflects the strategic directions and priorities of the Centre and is ultimately integrated into the way its management plan and report on performance.
|
|
Important Notices and Disclaimers ©CCOHS, 2008 |
|
|
Technical Support |
|
|
Client Services Products & services info |
|
|
Inquiries Service Answers to workplace questions |
|
|
Webmaster Website feedback |
