Designing a Program Scorecard for Continuous Improvement

The purpose of an office of institutional effectiveness is to collect, manage and distribute evidential data that is used to inform decision-making. If the data is not meaningful, accessible or complete, it will not be useful for anything other than compliance. So, the location of this office needs to be in an organizational area where there's a culture that encourages, supports, and engages in evidence-informed decision-making—that pulls together data across the institution, its programs and units and combines it with key performance indicators which can provide information for growth and improvement. 

In this vein, institutional effectiveness is no longer just an office but rather a holistic practice as we shift into a new era of institutional effectiveness. According to Transforming Data into Meaningful Information in SCUP’s Planning for Higher Education Journal, “institutional success depends on long-term sustainability, connecting investments with outcomes, and creating a culture that is data-informed”. An academic program review can jumpstart this process by providing necessary quantitative and qualitative feedback regarding program strength, health, and existence.

To do this well, you might consider designing a program scorecard.

What Is a Program Scorecard in Higher Education?

A program scorecard in higher education is a performance metric used to evaluate academic programs at an institution against a selected set of criteria. This data can then be analyzed and those insights used to inform curriculum revision and strategic decision-making for academic programming moving forward, as well as other planning processes across the institution.

This concept is not new. The balanced scorecard, first pioneered in commercial organizations, has since been utilized in the nonprofit sector as a way to better standardize performance evaluations and drive data-informed decisions. A similar practice has been developed and applied in higher education in the form of program scorecards.

Given the many challenges facing higher education, what does an effective program scorecard look like today and how does it support institutional effectiveness and improve institutional performance?

The Importance of Designing an Effective Program Scorecard in Higher Education

Designing a program scorecard in higher education is important because it supports your integrated planning process and provides a comprehensive view of your institution. This is a critical step in your academic planning process. Both academic and administrative data is needed to make this comprehensive. If not, you risk making uninformed or isolated decisions that can negatively impact your effectiveness.

This is one way to ensure you’re gathering the data needed to make these decisions. It is not always a “one size fits all” solution, but whether you’re starting from scratch or re-evaluating one that already exists, your scorecard should always aim to accomplish the following:

Serve the needs of all programs

You need to account for all academic programming at your institution from the undergraduate to the graduate level. Consistency is key in order to gather data on program performance and ensure you are holistic in your decision-making.

Evaluate the sustainability of your programs as a whole

Is this program meeting expectations when it comes to the number of students enrolled or credit hours generated? While this data isn’t the full picture of a program’s effectiveness, these can be key indicators of a program’s relevance and sustainability.

Provide effectiveness of student achievement

Are your students successfully progressing through the program? Are they meeting program outcomes? Are they satisfied with the program and its impact on their future career? Taking a closer look at your student performance and student feedback on their experience is critical to understanding where you may need to improve to serve your students better.

Look at your institution in context

Look beyond the proverbial walls of your institution. You need to take into account your peer institutions in higher education, your competition, as well as the labor market. Every institution is a part of and feeds into a broader ecosystem, and these external factors can impact your planning for the future as much as what’s happening internally.

Account for the student experience from beginning to end

From the moment a student enrolls in a program, to when they graduate, and everything in between - your program review should be comprehensive and have metrics covering the scope of the entire student journey. 

Five Components of a Program Scorecard

There are two main elements to consider when reviewing your academic programs: data categories and the metrics used to measure them. In their work, University of Maryland Global Campus used the following five data categories and their corresponding metrics:

Student Learning

Measured through student assessments, course pass rates, and retention

Effectiveness

Measured through graduation rates, student satisfaction, and capstone survey results

Sustainability

Measured through program headcount, new student enrollment, average class size, and credit hours generated

Market

Measured through enrollment trends at other online institutions and labor market data on related careers

Competitive

Measured through a growth-share matrix according to national trends

You’ll notice that these five categories align with the goals outlined above and what they should accomplish. It can apply to any program, is comprehensive in its assessment, and also looks externally - beyond the institution - to determine impact and need.

Design Your New Program Scorecard to Support Institutional Effectiveness

Identifying the right metrics to review your academic programs will give you the quantitative and qualitative data needed to improve your programming and demonstrative institutional effectiveness. Now that you have the components to make this change, what should be your next steps?

1. Engage stakeholders in a data needs assessment and SWOT Analysis of their program

2. Collaborate on a design of a data dictionary to provide a consistent language for talking about data 

3. Partner with a technology platform to support data collection and analysis

You may also want to lean on some of the practices applied by the University of Southern Mississippi as part of their reorganization process. USM converted data into easily consumed, visual information to actionize change. In addition, UMGC - whose components we highlighted above -  have found that their scorecard provides much needed information to keep their program current and competitive.

If you need a place to start, check out our free Program Assessment Toolkit! It includes a Program Scorecard template based on the work done at UMGC, and outlines the data you need to support your program review process and build a foundation for an improved integrated planning process at your institution.