Cultivating a Climate of Data-Informed Decision-Making at UMass Amherst

Even before COVID-19 became a key part of our vocabulary this spring, data-informed decision-making has been a trending topic of conversation among college and university leaders.

Last fall, AIR, EDUCAUSE and NACUBO issued a joint statement encouraging institutions to commit to the use of analytics to make better strategic decisions. The events of the past few months have put a fine point on the important role data and analytics play in empowering campus leaders as they chart their best path forward.

However, important as it is, advancing an institutional analytics program is not easy and it can help to hear from those whose journey is already underway.

I recently asked Barb Chalfonte, Ph.D., Executive Director of Strategic Analytics, University Analytics & Institutional Research (UAIR) at University of Massachusetts Amherst (UMA), to share insights from UMA’s experience undertaking this hard work.

Darren Catalano: What was the catalyst for prioritizing and building an analytics capability on campus?

Chalfonte: As an IR office, we had a lot of data for and about the University but it wasn’t in the most usable formats for people — it was data but not information. The data were presented in tables within PDF and excel files, relatively hard for users to manipulate and understand the data and there were no visualizations. While we were able to do our own internal analyses and create custom reports, there was no vehicle to distribute the data broadly and access to the raw data was not (and still isn’t) practical for all users. In addition to these practical and strategic reasons to develop our analytics capability, an overarching principle in the university’s strategic plan aspires to, “Instill a culture of evidence at all levels that applies the best possible information and analysis to decisions.”

Catalano: What are some of the cultural considerations when embarking on this type of initiative?

Chalfonte: First, there needs to be strong senior leadership support to embark on a project of this magnitude as well as sufficient investment in the necessary financial and human resources. This work doesn’t happen “for free” so it needs to be prioritized relative to the work that is already ongoing. The support of senior leadership is absolutely necessary for such work to get the attention it needs and to ensure that the right people are involved.
Second, there needs to be a culture of collaboration, particularly between the IR and IT functions as well as with other data SMEs. Bringing together these groups who understand both technology and how data is used at the institution from the very start of the initiative is critical to developing a platform that will be relevant and responsive to the decision-making needs of the campus.
Finally, the use of an analytics platform needs to be embedded into the regular business of the University to promote the expectation that people will be making data-augmented decisions. To achieve this, the initial roll-out of the platform needs to be customized to a particular campus’s culture and data needs.

Catalano: What are realistic expectations for others considering an analytics initiative? How long did the initial implementation take to start realizing value?

Chalfonte: When we launched the initiative we had a radically understaffed but also radically talented IR office. Despite the understaffing, we were able to move from building out the technical back-end to developing and validating data models to initial delivery of dashboards in ten months. Because we included both highly-used data (enrollment data) as well as a long-desired analyses (student movement amongst colleges and majors) on the dashboards, the initial roll-out of our first set of dashboards was very well received.

Catalano: What challenges did you encounter and how did you overcome them?

Chalfonte: That sounds past tense! We are constantly encountering challenges and working to overcome them. The challenges come in multiple forms such as: working to translate our internal algorithms and data with HelioCampus; discovering data issues that require larger business process decisions; and, unleashing a flood of “we want more” requests upon the initial roll-out of the dashboards. We address these various challenges by having a regular and ongoing relationship with our HelioCampus colleagues; connecting with SMEs on campus to help them understand the implications of our findings; and having formal and informal communications with campus stakeholders about our work and how requests for modifications or more information are being addressed.

Catalano: How is your institution using these advanced analytics capabilities? What are the initial use cases that you prioritized for the initiative?

Chalfonte: At its core, our analytics platform helps the campus community find information in an easily understandable format and allows conversation and decision-making to happen using standardized data. Many data elements that were hard, tedious, or time-consuming to obtain and use in the past are now easily accessible and understandable.

Our first analysis effort, beyond data visualization, is to develop a revenue/enrollment projection model for our on-campus undergraduate student population. That will be followed by similar models for graduate and online student groups. From a student success perspective, we are developing dashboards to explore possible inequities in the outcomes of programs and courses.

Catalano: What do you now know that you wish you’d have known when you first began this work?

Chalfonte: Although we knew that data validation would be a significant effort, we did not expect it to be the Herculean amount of work that it has proved to be. In fact, the term “data validation” diminishes the work involved which has translated into reverse engineering every data element and associated business processes. That takes a radically talented staff (see above) and an incredible amount of collaboration with people within the university as well as with our HelioCampus partners.

Some other things we wish we’d known about earlier include the importance of some of the decisions made around technology as well as consequences of decisions about naming conventions for data being imported into our platform. Those are both examples of decisions you have to make early in the process before you have a sufficiently robust understanding of the longer-term repercussions.

Catalano: How does having advanced analytics capabilities impact your ability to respond to a crisis like COVID-19?”

Chalfonte: The value of our analytics capacity became quickly apparent during the COVID-19 crisis in two ways: First, in seeing the excellent skills that various offices have been cultivating to work with and display their information; and second, in being able to pull and analyze data far quicker than previously to answer important questions in tight timeframes. Shortly after we shifted to emergency remote teaching, our faculty senate voted to broadly offer a pass/fail grading option. My colleague was able to quickly develop dashboards to track student adoption of this grading option and to assess any excessive course withdrawals relative to last year.

Our UAIR office and our colleagues in Administration & Finance and Research & Engagement were also able to get critical curricular, building space, and laboratory research data integrated quickly into reports, forms, and dashboards that let campus groups explore complex scenarios to support their decision making for the fall. Seeing these things happen rapidly and in support of critical work made all of the work leading up to this seem “worth it.”

Catalano: What’s next for your analytics program?

Chalfonte: We have several efforts that are happening in parallel:

• Developing revenue/enrollment projections,
• Integrating data from multiple sources into our analytics platform to support groups who are, for example, responding to online student enrollment and engagement or addressing summer melt,
• Continuing to develop and deploy dashboards that support broad campus decisions, and
• Identifying key offices and people who are needed to participate in governance.

To view the original article, please refer to the following link: https://www.ecampusnews.com/it-leadership/2020/06/12/cultivating-a-climate-of-data-informed-decision-making-at-umass-amherst/