Focus Groups: Continuous Improvement for a Professional Development Evaluation Toolkit

Introduction: by Joellen Killion, senior advisor to Learning Forward

Collecting data for an evaluation of professional learning, the fourth step in an eight-step process, can be a relatively simple process of survey administration. Yet, as Greece Central School District leaders discovered, a survey does not capture the authentic voice of participants. Surveys are expedient for gathering data from large populations of participants, and efficient in terms of time and cost. Surveys are the most common way districts collect data. Including some open-ended questions is one way to gather that voice yet maintain some efficiency in the data collection process.

Greece Central School District opted for another, more time- and personnel-intensive data collection: the focus group. Many evaluators acknowledge the value of focus groups, yet dread the challenges of managing and coordinating time, people and volume of data. Yet, the conversations that are stimulated by a strong set of consistent questions across multiple groups promote the process of reflection on practice and engage people in hearing others’ perspectives, two benefits that can foster engagement and ownership for a professional learning program. It is through these conversations that participants can voice their authentic perspectives and provide insights to program managers about strengthening the program. Focus groups offer opportunities to dig deeper into a response to gain greater clarity.

Collecting data through focus groups presents the challenges these authors identify, and more. Analyzing the data for trends and themes, extracting patterns or findings in the data, and adding recommendations are all part of the data analysis and interpretation steps of the evaluation process. These are essential to be able to share and use results of the evaluation, yet they generally require additional time because the responses are qualitative rather than quantitative. While software programs that translate speech to text and conduct semantic analyses alleviate some of the data-management and -analysis burden, analyzing focus group data still requires human review for thoroughness and accuracy.

Determining how to collect data is a decision made after the evaluation questions are clearly defined. As Greece Central School District evaluates its long-standing and successful New Teacher Induction Program, it makes sense to use focus groups. By using focus groups as well as surveys, the district can collect additional data against which to check survey results and to gain deeper insights into why the program works and how it can be strengthened.

Those considering the use of focus groups should weigh the benefits as well as the costs. This blog describes how one district realized the benefits and addressed the costs associated with using focus groups for data collection in their evaluation.

— Joellen Killion

Greece Central focuses on the voice of the participants

At Greece Central School District, we’re proud to have a thriving Professional Learning Center that offers professional development by teachers, for teachers. One of our priorities is to provide a comprehensive system of support for our newest teachers. With this in mind, we designed a New Teacher Induction Program that includes a 3-day Summer Academy, two years of formal mentoring and continued coaching from district Teacher Leaders in Years 3 and 4. New teachers also have access to targeted professional learning programs designed to support them in completing required multimedia portfolios, generally due in their 4th year of teaching.

Caption: A new teacher shares her portfolio progress with a fellow new teacher during one of the portfolio work sessions attended by new teachers and mentors.

With the goal of engaging multiple stakeholders in our professional development program evaluations and promoting teacher voice, we seek collaboration from a number of groups — our Professional Learning Center Director and policy board, our district superintendent and cabinet, our curriculum leaders and principals, and our teachers’ union leadership. These groups help us to identify, create and implement our professional learning program evaluation tools.

To measure the impact of New Teacher Induction, we have typically used feedback surveys administered after the Summer Academy, and during the year we administer surveys about mentoring support. More recently, we have added focus groups to our repertoire of evaluation strategies.

Caption: New Teacher Induction 2018: A Breakout EDU session, using core resources for new teachers: Meeting the Needs of Diverse Learners and Why Didn’t I Learn This in College by Paula Rutherford. This session received very high ratings on Greece Central’s post-NTI Academy survey.

Initially, we “dipped a toe in the water” using Ph.D. students from a local school of education to facilitate them. The students were happy to gain additional experience with this form of data collection, and through our conversations with them, we learned strategies for analyzing and reporting the qualitative data from transcripts and notes from focus group discussions.

What is a focus group?

A focus group is a small group interview used to collect data, often for a program evaluation. It’s a type of qualitative research where a moderator typically poses a few questions to the group — usually around 4-8 people — about their experiences, attitudes or ideas.

Why use a focus group to collect data?

Focus groups are a good way to gather in-depth information from teachers in their own voices. Teachers are often more than willing to share stories and details about their experiences that don’t surface from surveys, which tend to feature close-ended questions.

This year, we conducted our own focus groups with both mentors and new teachers. New teachers met after school during one of their regular portfolio working sessions and we had a former experienced mentor moderate those groups (it wouldn’t have been appropriate to have a current mentor collecting data from current new teachers).

It was more challenging to find common time to meet with groups of mentors, so we held one focus group session during their scheduled weekly meeting. This meant we had more than 20 people in the room at one time — far too many for one focus group. To address this challenge, we had a former mentor, along with our PL Director and our program evaluator, split the large group of current mentors into small, manageable groups. Then we moderated simultaneous focus groups. The three moderators met beforehand to generate the list of questions and discuss how we would facilitate the groups and take notes.

What we learned using focus groups

1. From this experience, we learned that both mentors and teachers had a lot to say about their experiences and were eager to share.

We came to a greater understanding of just how variable the mentor-mentee relationships can be, and learned from their perspectives how we might go about improving the program. We were already aware that time is always a factor and a stressor, but through the use of focus groups, we were able to dig a little deeper and understand more about this. Mentors feel the pressure of having to teach their own classes and devote enough time to their own students yet still reserve enough time for their mentees. Some have difficulty prioritizing the mentoring, given their other responsibilities — especially when the new teacher is doing well. Some new teachers, however, wish their mentors would spend more time with them but didn’t feel comfortable asking for it.

2. We also learned how prepared mentors felt for having difficult conversations.

For many, it was their first year as mentors, and some expressed surprise at having mentees who seemed a bit resistant to mentoring. We learned that mentors appreciated a schedule of topics, an agenda or framework from which to initiate their interactions, and that some were finding the guidance tools given to them by the professional learning director very helpful. They felt they needed guidance for conversations that would move beyond “So, how are things going?” especially when the mentee answered that things were going well. Still, mentors valued and enjoyed the positive and trusting relationships they had forged with their mentees, and they enjoyed the rewarding feeling of being helpful and useful. Finally, mentors sometimes felt that mentees didn’t prioritize their interactions and make the time to meet with them.

3. Conversely and interestingly, we learned from some new teachers that they felt their mentors didn’t prioritize their mentoring role.

They wished their mentors had done more to reach out and spend time with them. Overall though, new teachers expressed appreciation for their mentors and commented on the positive relationships and how helpful they were.

Focus groups required some time and planning, along with the coordination of a number of busy people’s schedules, but we found it time well spent. We feel we are now even better positioned to strengthen our New Teacher Induction program design, having incorporated teacher voice in the process. Focus groups served as a great complement to surveys, and when we created focus group questions based on survey results, we were able to deepen our learning in service to evaluating our New Teacher Induction program. We plan to continue refining our focus group practice in the coming years, as well as add other relevant evaluation tools to measure the impact of New Teacher Induction and our other professional learning programs.

Measuring the Impact of Professional Learning study

Participating in the Measuring the Impact of Professional Learning study gave us space to step back and reflect on current practices, interact with other districts grappling with the same questions, and consider how to expand our toolbox of program evaluation strategies. We found focus groups a refreshing way to interact with professional learning participants. We learned that by being purposeful and choosing our “voices” wisely, we can achieve a representative dynamic of the overall district culture. Surveys are quick and easy to administer, but they miss the depth and detail of how people feel impacted by their experiences with professional learning, and how they feel their students have been impacted by that learning. Being face to face with people tends to make them more thoughtful in their responses, and start to feel connections with each other. These connections may open up new avenues for professional growth and learning as well as collaboration, rendering the focus group so much more than a simple tool for data collection.