Evaluation, Higher Education

Outstanding Education?

Nearly twenty years ago, when my children were just getting started in elementary school, I attended a community meeting about the proposed school budget.  I live in a very small town (smaller now, with the regional demographic shifts), so such meetings were an important part of the democratic process. We came together to discuss the details of the budgets before heading to any votes.  At that time, I recall two dominant themes – what do we need to invest in to create a great educational experience and how do we keep the costs down so we do not price people out of our community.  These themes, of course, turned into questions about must haves vs. nice to haves.  Like any town, we had differing opinions about what that meant, but we generally came to some consensus, or at least voted to approve a budget.

Today, I am a member of that school board, and we are still having the same conversations. Our shrinking enrollments and aging tax base have made the conversations a little more strident, but it is still mostly good conversation.  We are spending our time trying to define quality educational experiences that will help our students thrive, without creating an overwhelming cost burden for the town. As I listen to the concerns of my neighbors, and try to keep us from speaking in the hyperbole that so easily divides, I find my mind turning to my job as provost.  We, too, are facing shrinking enrollments (the drop in K-12 enrollments has a necessary consequence in higher education) and a tax base concerned with their ability to support everything the state needs.  As the challenges have descended upon us, we have done a lot of speaking in hyperbole.  Perhaps, it is time to have some honest, if difficult, conversations.

Let’s start those conversations by asking ourselves to identify the necessary components of an outstanding undergraduate education.  No one sets providing mediocre education as a goal, of course, so we want to set the bar high.  However, we do not usually think carefully about what we mean by “outstanding” or how we might achieve it.  Instead, we wait for it to emerge from our offerings.

Because higher education is built on the idea that faculty expertise is our greatest resource, we have a habit of deferring to that expertise at all times.  Much of this habit is a good one.  It would be foolish to hire people who have deep expertise in their disciplines and then tell them what to teach.  No innovation will happen under those circumstances. So we try to create an environment that encourages faculty innovation and hope that this will help us discover excellence.

However, the result of too much deference to this expertise is generally curricular sprawl.  New options or concentrations or majors and courses pop up on a regular basis.  They reflect emerging interests or fields, or sometimes a momentary trend.  These additions to the curriculum are rarely accompanied by a reduction in other offerings, because, well there is a good argument to be made for any course or any major.  Frankly, good arguments are a specialty of higher education. The sprawl is fine until we see sustained dips in enrollment. Then we are faced with low-enrolled courses, degrees, or majors, and the removal process awaits.  We are not good at this part, so we try to avoid the question, or speak in the language of outrage, and try not to eliminate anything.

But, eliminate we must.  Enrollments have made the decision for us.  So has the proportion of our costs that states are willing to fund. The time for avoidance is over. However, we still want to rely on the insights of our faculty.  So, as we take these necessary steps, instead of starting the conversation with dollar figures, we should start by coming together to define the components of an outstanding undergraduate education.

There is a lot to consider, and it isn’t just the number of courses or majors. For example, we might want to look at how we have defined our degrees (BA vs. BS vs. BFA) and the proportion of each in our catalog.  Doing so might uncover a tendency toward more professionally oriented degrees (or the opposite), and reveal that for the students we serve and the faculty expertise we have, we see this as a priority.  At the same time, we might want to take a close look at the differences between options within a major and answer the question of whether that level of specialization really benefits our graduates. Perhaps some things could be a little less career focused.

Then there is the general education curriculum or liberal arts core.  What role is it playing in the overall vision of an outstanding undergraduate education?  Are students encountering varied ideas or are they mastering key skills or some combination of the two?  Is it organized developmentally? Does it support the major? We know we must provide general education, but have we set it up in a way that promises to support critical habits of mind in our graduates?

What pedagogies should we feature? Are there approaches to teaching that every student should experience? If so, how do we organize schedules to support those, pedagogies while keeping the balance of offerings in view?  Is it possible to design schedules around encounters with critical pedagogies, without privileging one approach?

Then there is that very tricky question: How will our graduates be different from the graduates of any other college or university? This is a difficult to answer, of course, because much of what is promised in higher education really is the same everywhere. We are all trying to support graduates who have a reasonable grasp of the world around them and the potential to thrive in an environment where change is a constant. Nevertheless, we have different students, different faculty, and different expertise. Surely, we have a unique point of view that can help shape the decisions we must make about what we offer.

If you read any news about higher education, you will encounter a long list of mergers, financial challenges, closures, and other worrisome trends.  No one in the northeast is immune (well, no one but Harvard and Yale).  It is a scary time, but I think, if we try to come to a consensus about the qualities of an outstanding undergraduate education, we might just start to see what the path forward could be.

 

 

 

Agency, Evaluation, Innovative Pedagogies

Reflection vs. Evaluation

Well, it is December and we are racing toward the end of the semester. As students complete term papers, prepare for final exams or presentations or performances, faculty are making room in the schedule for teaching evaluations.  These evaluations are generally short questionnaires that ask students to give an assessment of the effectiveness of the teaching they just experienced. It is an opportunity to give feedback, which is to the good, but most are constructed in a way suggests expertise where it does not exist (students are not instructional designers, nor will they have depth of knowledge of the discipline), and there is well-documented evidence that they reflect cultural biases throughout.  So, why do them at all?  Good question.

As currently constructed at my university (and at all of the universities where I have taught), there is little value in this exercise.  We have made the whole process about evaluation instead of about learning.  We have also cast our students as consumers, who then provide ratings (stars?) of our work, without really helping them reflect on their learning. What if we reimagined teaching evaluations as course reflections? Instead of using them to tally the effectiveness of a faculty member, they could become a mechanism for collaborative course construction. Instead of seeking an ill-informed critique, we could invite our students to share what they’ve learned from us and give us suggestions for future iterations of the course.

Here’s what it might look like.

Dear Students,

At the end of each semester, I gather information about your experiences in my classes so that I can get a better understanding of what is working well and what new ideas I should explore. Please take a few minutes to reflect on what you have learned in this class and then answer the questions below thoughtfully and honestly.

  1. What was the most interesting or most important thing you learned in this class?  

Why?

    • It provided a foundation for this or another class that I will take.
    • It connected to important topics beyond this course.
    • It helped me see things from a perspective other than my own.
    • Other (please explain).
  1. What was the least interesting or least important thing you learned in this class?

Why?

    • It was too foundational/I’ve encountered it in several other classes.
    • It seemed like a tangent that was not relevant to the class.
    • Other (please explain).
  1. Considering the course overall, were there ideas or assignments that you think will help you succeed in other classes at the university? Please explain your answer.
  2. Considering all opportunities for feedback on your understanding of the material (tests, quizzes, presentations, papers, group work, etc.), which did you find most helpful? Please explain your answer.
  3. Is there an opportunity for feedback on your work that you would like to see added to this course?
  4. Considering things like grading criteria, timing of assignments, or overall organization, do you have any suggestions that you think might improve this course?
  5. Do you have any additional comments that I should consider?

Thank you for your feedback and good luck in your studies.

What I like about this structure is that it invites students to participate in the evolution of the course, instead of asking for some kind of score for performance. By using the first person in the opening paragraph, the faculty are given agency, suggesting that they are fully committed to this dialogue with their students.  It also suggests that students are speaking directly to that faculty member, not some unknown administrator who will then evaluate the professor. 

Moving in this direction, faculty can use the information to learn how students are experiencing their teaching and respond as they deem appropriate.  For example, maybe the thing that students identified as unimportant, was in fact very important.  Perhaps some reframing needs to take place.  Or, maybe several students felt the need for a presentation to be included in the course.  Digging into why would be a good next step.  No doubt some students will ask for extra credit. If the answer is no, then being clear about why not might be a good thing to discuss in the next class.

I also like that this is a disaster for quantitative summaries.  While the current scales from 1 to 5 may be helpful for creating graphs and charts, and they do provide some sense of the instruction in terms of extremes (outside of university norms), in reality they do next to nothing for teaching.  Mostly, they inspire defensiveness. I’m not worried about losing those statistical summaries, because the extremes are easily captured in the syllabi, sample assignments, and peer observations. I’d rather cultivate the reflective practice that this qualitative approach implies.

As one of the people who reads faculty portfolios in their applications for tenure, I am most interested in seeing how faculty respond to student feedback. The most compelling thing that can be included in any tenure packet is a narrative about how one’s teaching has evolved and why.  Evidence of change over time should include sample complaints and sample praise found in these course reflections. If the examples are followed by explanations of how things changed as a result, then I feel confident that I will know enough to fairly review the candidate. I will also know that I have a professor devoted to good teaching.

Let’s drop the ratings model and focus on learning about our teaching. Let’s try to foster an environment where we take student voices to heart, without ceding our expertise.  Let’s listen carefully to concerns and ideas, and work to grow in our profession. Let’s be reflective educators.

 

 

 

Evaluation, Higher Education

Root Causes

Since many institutions of higher education set preparation for and engagement with life-long learning as a goal, it is fitting that I should take my own continued education to heart. Recently, I enrolled in a course called Policy Design and Delivery: A Systematic Approach to strengthen my skills at both developing and analyzing policy proposals.  Motivated by a sense that the many “solutions” to higher education’s problems do not represent a clear and thoughtful analysis of the contexts in which those problems arise, I am searching for a better understanding of how to craft a good policy.  It has been illuminating so far.

Without even using the tools in my policy design course, though, I can see that many of our education policies mistake correlations for causes. For example:

  • Students who participate in co-curricular activities have higher retention rates than those who do not, so we push to require co-curricular activities.  Sounds good, but maybe the students who are committed to staying at a university are the ones who decide to get involved.
  • Students who attend classes regularly are more likely to be successful than those who do not, so let’s adopt attendance tracking technologies and pair them with nudge technology and get students into class.  Attendance matters of course, but not attending is often a decision that no amount of nudging will change. Perhaps attendance is the mark of a student who wants to succeed.
  • And for today’s discussion: Students who commit to a true full-time schedule are most likely to complete their studies…otherwise known as 15 to finish.

Complete College America has invested a lot in the 15 to finish initiative, and for good reason. First of all, we have been confusing our students about the meaning of full-time.  Federal financial aid rules set full-time status at 12 credit hours per semester. This status opens up access to housing and many grants.  A student might logically conclude that 12 credits per semester is sufficient for timely completion of a four-year degree.  Unfortunately, it doesn’t add up. State, national, and accreditation standards define an undergraduate degree as a having a minimum of 120 credit hours (there are some regional differences, but this is the generally accepted standard). The true number is 15 credits per semester.  This is important for us to communicate, so the slogan is a good start.

Second, there is a growing body of research surrounding the notion of “momentum.” Simply put, students who are engaged in their majors early and earn those 15 credits per semester tend to remain motivated to get to the finish line. This is usually linked to guided pathways, which help to limit the chances that students enroll in courses that do not support progress in their degree plan. Success and engagement propel students forward.  Those on a slower path have a greater tendency to stop out, or grow discouraged and lose sight of the finish line.  So, in an ideal world, a truly full-time schedule with a mix of general education and major coursework from the first semester of enrollment is the best approach to degree completion.

Great, but here is where it starts to get problematic. Once we clear up the mystery of degree requirements so that all students understand the 15 to finish concept, there are still myriad reasons why 15 credits per semester is not possible for a significant number of students.

  1. A student may need to work on some foundational skills in their first year of college and the better path to success is 12 or 13 credits, rather than 15.  For example, if a student is behind in their math skills, they might enroll in a 4-credit course in math (rather than the more typical 3-credit general education math course).  To get their schedule just right, with adequate time to pay attention to those math skills, it might be best to stop at somewhere between 13 credits.  If all goes well, the end of the first year might leave that student with 27-29 credits. This is technically behind (below 30 credits) and penalties follow. In this case, the most obvious penalty is their continued status as a first year student. Registration priorities are tied to the number of credits earned.  Higher numbers go first.  A student who took this slightly slower path is at a higher risk of not getting into the next class in a set of requirements because they are still registering with first year students.
  2. A student may be enrolled in a highly competitive program that requires very challenging foundation courses, and opts for a slightly lower number of credits to help manage their time and attention.  For example, science, nursing and pre-med students are likely to take this option.  They might be enrolled in two lab sciences (8 credits), math (3 credits or 4), and humanities (3 credits). If the math class is 4 credits, they will be on track for the 15.  If not, they will be behind and suffer the same penalty as the student with some foundational needs.
  3. A student may have to work while in college (or raise children or report for military service).  The rational decision is to take a lighter load, but the penalties abound.  A part-time student will not be eligible for many grants, will not receive the benefits of the bundling discount (charging the same price for 12-18 credits) and will suffer the registration penalty because they have not yet made it to the status of sophomore or junior or senior.  This means that a hard working part-time student could be in school for eight years, steadily working toward a degree, and never be recognized as deserving of the benefits of higher class standing and never receive any financial support. Now that is a real disincentive to completion.

So, what’s the point. Well, if 15-to-finish were just a catchy slogan it would be of no real concern.  Indeed, some of the things I have described are things that we can fix locally by reimagining our registration priorities and focusing on part-time tuition support. But there are now trends toward additional financial aid strategies being tied to the 15 credits per semester (See New York’s Excelsior Program as a start).  As the nation discusses free tuition, the nuances I have described are frequently missed. Many of these proposals are built around that ideal full-time student.  Yet many of the students who would benefit from this tuition break will not qualify because they will not be able to complete 15 credits per semester.  Again, the penalties can be tremendous.

As for cause and correlation, well it is obviously true that completing 15 credits per semester is correlated with higher graduation rates. But the root causes of student success, which lead to the ability to actually complete those 15 credits, are far more complex than the credit story.  We need to step back and look at the conditions that are driving those behaviors so that the policies we design do not continue to disadvantage those who need our help the most.

 

 

 

 

Change, Evaluation, Higher Education

The Pace of Change

It is the end of another academic year, and as we move through award ceremonies, research presentations, and finally commencement, I take the time to look at my to-do list from last fall.  It is a bit deflating to see all of the things I didn’t complete.  I expect some of this to happen, after all, not all of my plans were good ones. A few things actually got done, some were re-imagined, a few were abandoned, and some just didn’t get the attention they needed to come to fruition.  It isn’t all bad, but I confess to being a bit disappointed in myself.

Then I remember, higher education is designed to slow the pace of change.  While we are great places for advancing knowledge (yes, new discoveries and inventions do come from higher education), we are best at slow deliberation.  We analyze cultural patterns large and small and try to see them in context, rather than jumping to conclusions.  We look at small changes in forecasting models for weather or economics, tweaking them slightly each year to get closer to a better predictor, and then analyze the results of those changes.  We reflect upon the past to try to divine how we got to this moment.  Change is not something we’re avoiding, it is something we’re vetting.

So here I am, an academic with an administrative role. I understand the care with which my colleagues approach change and I share their suspicions about the innovation of the week.  The brakes they are putting on in the form of more questions, more input, more research are justified.  However, I also spend my time looking at the whole organization and the whole student experience, and I see patterns of successes and failures that are calling for us to move a little faster. I feel the push/pull of the deliberative mindset and the urgency of responding to areas for improvement.

Take, for example, the way this generation of learners is coming to us.  It is well-documented that their experience of reading is very different from that of the generations before them.  (See “The Fall and Rise of Reading” by Steven Johnson in the Chronicle of Higher Education). It isn’t that students can’t read, it’s just that they really haven’t had to grapple with critical reading. The books read and tests taken prior to coming to college are all about short forms, summaries, and highlights.  And of course, there’s the endless interaction on the Internet to reduce the time spent with texts. Reflective reading of long form texts is just not what they are used to doing.  We know this to be true, yet we haven’t reviewed the literature on how to teach critical reading, and then incorporate into our classes.

Maybe we think this isn’t our job. High school was supposed to do it, so just pile on the readings and the students will get it eventually.  But they don’t.  We have to adjust our teaching strategies, and quickly, because we’re losing too many to this gap in skills. Even worse, we are diminishing the conversations we’re having in our classes because we’re not really expecting students to do the reading anymore.  This is a terrible spiral, but the good news is we can stop it from happening. But we have to act, and sooner rather than later.

And then there is the issue that really made me sigh this morning.  After repeated reports on who struggles to succeed at my university, I concluded that the at-risk group is any student who had less than an 85 average in high school.  I learned this two years ago and started a conversation about advising strategies to address the at-risk group. At that time, I used the words “intrusive advising” which is a term found in much of the advising literature. Several of my colleagues objected to the term, so we moved to the idea of enhanced advising.  I brought together a group to develop a protocol and nothing happened.

Then I appointed some faculty members to investigate ways that we might develop an advising protocol for those students.  Like all good faculty members, they went out and talked to their peers. While they found out a few good things about how to support faculty as advisors (and I will work to support those findings), in reality, enhanced advising was set aside in favor of better advising for all.  This is a good idea, but it will take too long to identify and scale those improvements.  Meanwhile, those at-risk students are left with no direct support.

I just got an updated report on at-risk students and it is still students who earned less than an 85 average in high school.  The difference in retention rates for this group is at least 10% lower than those at 85 or above, and the differences in graduation rates are even more stark.  And there’s plenty of literature about how to support these students, so, I’m feeling an urgency.

So, I’m left pondering ways to balance the deliberation with the urgency.  I do respect the reflective and thoughtful nature of my colleagues, but when I keep the larger patterns of student success (or lack thereof) in view, the pace of change is just too slow.  I’m going to have to find a better balance, a better way to move the deliberation along just a little faster.  Because, what I don’t want to do is have this on the unfinished list again next year.

 

DeVos, Evaluation, Higher Education

Under Construction: Peer Review

On Monday, Education Secretary Betsy DeVos, released a list of policy proposals as part of negotiated rule making related to the Higher Education Act.  While I have many questions about the goals of these revisions and how they might impact those most susceptible to education scams, I am intrigued by the part of the list that addresses accreditation.

Several of the policies suggested appear to be a direct assault on regional accreditors.  The reasons that they are a target are complicated, but one such reason seems to be a sense that they are creating barriers for innovation in education.  This argument is fraught with contradictions and for-profit motives, to be sure, but the role of accrediting bodies that are wedded to traditional non-profit educational institutions is not unworthy of review.

Over the past 20 years, regional accreditors have truly transformed from evaluation criteria based on things universities have (faculty, libraries, students, and buildings) to things universities do (retention, graduation, learning/degree outcomes, and graduation placements).  This transformation was, in part, provoked by the emergence of for-profit, online education and our need to grapple with how to evaluate learning in these differing environments.  We resisted, argued, and then re-imagined our responsibilities. We maintained our right to define the parameters of “quality” education, but we no longer argue that we can’t assess our efforts.

As we all became (relatively) comfortable with program assessment and the focus on outcomes, our accrediting bodies asked us to improve our approaches and make it a regular part of what we do.  In the process, we opened the door to developing quality online education because we articulated what our graduates should know.  Whatever the environment, similar degrees should have similar outcomes.  This is the best possible outcome of a robust peer review process. Relying on faculty and administrators from other universities to look at what we do and provide educated and responsible feedback works in this system.  We understand what we’re looking at because there is a lot of common ground. Let’s be honest, this change was hard.  We didn’t like doing it, but here we are.

Now we face something new and the peer review system is going to struggle to define its role again.  In recent years there have been closures of large for-profit and small non-profit educational institutions and, along with those closures have come questions about the oversight provided by accreditors. The withdrawal of accreditation is powerful. It will close an institution, so it is imperative that these bodies develop good ways of monitoring finances. As participants in peer review accreditation processes, we are going to have to figure out reasonable questions to ask that can strengthen evaluation of less traditional and traditional educational institutions alike.

During an accreditation visit, when a university or college is struggling financially, peers are asked to offer feedback. This can be problematic. Most of us from the non-profit world can read a budget and see shortfalls in funding, but we are less prepared to provide insight into how to recover from that shortfall.  Evaluating quality of education and the supporting infrastructures of faculty governance and transparency are things we are comfortable with.  Developing plans for capturing market shares and differentiation of our educational “products” does not come easily to us.

At the heart of this challenge is not that we don’t value or support innovation, something DeVos seems to think we are doing, but that we see innovation in terms of learning, not in terms of money.  We are not blind to changing work landscapes, we respond to those all the time, but we do so on the assumption that we are meeting a societal need, not a bottom line need.  Now we are faced with considering some of the questions that a market oriented evaluation might raise. We will argue and resist, but then we must figure it out for three very important reasons.

Reason 1: Identifying a problem (financial or otherwise) during peer review will not be helpful if we don’t offer some guidance about how to recover. Ours is a collaborative process where we learn from each other.  Ignoring the learning that needs to take place in the financial category makes peer evaluation a threat, not a helpful process.  We have to figure out how to handle this part as a true mentoring opportunity.

Reason 2: We need to continue to impact the definitions of quality and viability, just like we did in the early conversations about online degrees.  By being part of that conversation, we made sure that the measures for online learning were not narrowly evaluated on content delivery, but on learning experiences. We need to shape the questions surrounding financial viability the same way, so that we don’t end up with profit as a primary goal. We want reasonable comparisons to be made between for-profit and non-profit institutions, comparisons that keep students at the center of all we do.

Reason 3: Education is not about widgets.  Students are complex and require nuanced educational experiences that reflect understanding their unique backgrounds and needs.  And the focus of our degrees must anticipate not the job of the moment, but the long-term skills and habits of mind that our students will need as the landscape of work continuously changes. Our traditional, non-profit structures facilitate that nuanced, long-term thinking in ways that are not supported in organizations that produce quarterly reports. This is the thinking valued by regional accreditors associated with these types of educational organizations.

Higher education in the United States has long been built on peer review. It is our strength and it is what allows us to be innovative. One look at the inventions, discoveries, and even changing degree titles will tell the story of responsible, not reactive, innovation. This is supported by the integrity of our peer review processes and we need to hold onto them, because we know our context best.

Yet we are in for some hard work as we reshape the questions we are asking of each other.  Demographics have changed the context for non-profit and for-profit organizations alike.  Finances will have to be considered. But, the questions we ask must reflect our commitment to creating excellent learning environments for all. If they do, the value of our accrediting processes will remain strong.