Preamble
The Q&R Corner began as a way for MB Herald Digest readers to ask questions about CCMBC and its ministries. The June 2023 column available here responded to questions about the Fall 2022 issue of Direction which highlighted the Rich Janzen and Brad Sumner “Using the Confession of Faith” study.
In my June 2023 column I expressed that my first response to the study was both “thankfulness” and “curiosity” with the hope that we can learn more about what it means to be increasingly faithful to Jesus and healthy in how we and our churches use the gift that we have in our MB Confession of Faith. It is for that reason that I am pleased that Rich Janzen has put in significant effort to provide a response to the June 2023 column.
In my column, I took issue with two statements that were in Rich and Brad’s concluding paragraph describing what appeared to be the key takeaways from the study. I questioned the validity and meaningfulness of the first statement—”The range of pastoral perspectives demonstrates that congregations have no uniform way of thinking about or acting out the Confession of Faith” (157). It sounded to me like an overarching “diagnosis” of our Canadian MB reality. I also questioned another concluding statement that described how the study might help MB denominational leaders “learn to live with diversity and difference” (157). I referred to this as a suggested “treatment plan” in light of the diagnosis—and wondered whether learning to live with diversity and difference might not always be the best response since it depended on the level of diversity present. My concerns were primarily in reference to these two concluding statements.
I did not mean to imply anything about the study’s purpose or intention (neither word shows up in my article in reference to the study or the researchers). I was not questioning the integrity of the researchers or the methodological integrity of the study (although I still have some questions about sample size and such). I am very sorry to Rich and Brad if they experienced my comments as questioning their integrity. I am glad that Rich has responded with the article to clear up any misunderstandings and to let readers hear responses to the concerns I raised.
Thank you to the MB Herald Digest for sharing Rich’s response. I hope that the publication of both columns will nurture more conversation among us about how the MB Confession of Faith can and should function to help us become “healthy disciple-making churches…, faithfully joining Jesus in his mission” (CCMBC Mission statement).
Ken Esau, National Faith & Life Director
A Reply From Rich Janzen
I appreciated the question in the Q&R section of the MB Herald Digest June 2023 issue. The question asked national readers to responsd to the recent Direction issue dedicated to a study about how MB congregations use our Confession of Faith. The question helped to bring out another important perspective in the conversation surrounding this topical issue.
I was co-lead of this study together with Brad Sumner. As with all my research, I welcome engaged and critical reaction to the study process and findings, particularly among those at the heart of the issue. I was therefore pleased to be asked to be co-guest editor of the special Direction issue where the study would not only be summarized, but where we would also hear reaction to the study through a series of seven response articles. National leaders were among the original list of about 20 people invited to respond – they were also invited to add other names to the list in order to gain well-rounded perspectives. While no CCMBC response came at that time, it was encouraging to see Ken Esau’s response now.
Unlike the response articles in Direction where I provided editorial feedback, I had not seen Ken’s response prior to it being published. If I had, I might have helped to clarify some misconceptions of the study in his response. I hope to clear up these now so that further conversation can be as accurately informed as possible.
In my teaching and training on research I often mention this slogan: The main thing is to keep the main thing the main thing. I then explain that in research the main thing is the purpose statement. From it flows decisions about the main research questions, methods, sampling, data analysis, sharing of findings, and more. You also assess the quality of research in light of its stated purpose. Given its central importance, it’s unfortunate that in Ken’s response the study purpose was misrepresented.
In his summary of the research Ken states the study’s intent was to offer a “diagnosis” and then a “treatment plan” related to our MB family and the Confession of Faith. Such a characterization implies that the study’s purpose was deterministic – some sort of expert-driven assessment. It was not. Below is the actual purpose statement that was discussed, revised, and eventually agreed upon by the study’s advisory committee. This statement (and the three main research questions that flow from it) is found both in the Direction lead article and the publicly available summary report slides.
The purpose of this research is to explore the diverse ways that Canadian MB congregations have used and are using the MB Confession of Faith in congregational life. This exploration is seen to be potentially helpful in supporting congregational, provincial, and national leaders’ efforts to facilitate broader engagement with the Confession and to teach MBs how to live with diversity and difference. (p 144)
First notice that this was exploratory (not deterministic) research. It was more inductive than deductive. Its intent was to gain in-depth understanding about something little known. Such exploration lends itself to qualitative methods which are more concerned with interpretive meaning than with objective measurement. Notice also that the use of the research was not to prescribe solutions but rather to facilitate further engagement. And so, if you would like two words to describe this study’s purpose then perhaps “explore” and “engage” could work. Consider how different that is to framing the research as a “diagnosis” and “treatment plan”.
To be clear, I have done my share of deterministic research, typically in the form of summative evaluation. And I’ve designed the research accordingly (always in a collaborative way), including using quantitative methods with multivariate statistical analysis. But this study was not that. And for the sake of integrity, it should be judged according to what its purpose actually was.
When assessing the study, it is also important to understand that it used a community-based approach to fulfill its purpose. By community-based I mean that the research strove to uphold both academic excellence and community relevance and did so in a way that reflected the three hallmarks of this approach: community-driven, participatory, and action-oriented.
It was community-driven in that it was curious pastors (not external researchers) who initiated the study idea and directed the research agenda. Brad Sumner was one of those pastors who stepped up to co-lead the project. And other pastors were involved in other ways (see below). It was these pastors who also found the resources to get the study done.
It was participatory in that the initial pastors invited other stakeholders to be involved. One key mechanism was to facilitate input from a cross-stakeholder advisory committee to guide each step of the research process. This advisory committee included other MBs from across the country: pastors, educators, former and current CCMBC staff, and a National Exec Board member. You can see their names in the summary report slides. Another mechanism was the use of peer researchers and an open forum (but more about these below).
It was action-oriented in that its purpose was pragmatic, not only gaining new insight but also engaging others to react to and use the research results. The Direction special issue was one avenue to encourage this action. Other avenues included an on-line open forum held for all research participants and interested others, and a special presentation to the NFLT.
This approach is important to mention to further make the point that Brad and I were not in the business of making expert pronouncements. Yes, I was invited into the project because of my expertise in community-based research (140+ academically and non-academically funded projects over 30 years).
But my (and Brad’s) role was to facilitate a collaboration of people who together wanted to systematically explore this topic further and invite others with a stake to do the same. You could say that this relatively small study was trying to mirror “the change we wished to see” in trying to better understand each other’s perspectives, even as we navigated the tricky waters that diversity of (sometimes passionate) opinion can bring. Those involved in the study can offer their thoughts about the extent to which we achieved those aspirations. But consider the perspective of Richard Lougheed an MB educator from Quebec and member of the study’s advisory committee, who wrote one of the response articles in Direction: “I was hesitant to agree to participate in the proposed study. However, as I became more involved in the study, I was impressed with the seeming consensus on approach, questions, and conclusions despite considerable theological diversity. That made me surprisingly hopeful that [our] MB background might provide greater understanding to help reconcile theological diversity.” (p 195)
With that as a long-winded backdrop, what does this clearer understanding of the study purpose and approach mean for the five methodological concerns raised in the MB Herald Digest? Can we now see them in a new light?
First, is the concern about sample size. Can 17 interviews be enough for such an exploratory study? The answer is yes. At least if participants were carefully chosen to reflect the range of perspectives of what is trying to be understood. The project advisory committee was key in helping to determine such purposive sampling criteria. Furthermore, in qualitative research the concept of “saturation”, not statistical analysis, is used to guide sample size – saturation being the point where more interviews would not lead to new learning. Would we have learned more with more interviews? Perhaps. Yet the reoccurring themes with thick description beginning to arise out of the 140+ transcript pages suggested that data saturation in answering the study’s three main research questions was close at hand if not already reached. This conclusion has support within the qualitative research literature where it is not uncommon for 17 interviews to be considered sufficient for saturation.
Second, is the concern of generalized conclusions. By now this point should be obvious. Did this study provide a conclusive diagnosis of the entire MB denomination? No. It was never intended to be generalized in that way. Having said that, in qualitative research there is the notion of “transferability”. If you sample well (as I think we did), the findings could well be applicable to the whole. It is up to those in the broader context (in this case other MB congregations) to discern the extent to which that is true. And that is why it is so important to facilitate open conversation following an exploratory study – to better understand how the insights from the 17 congregations are reflective of the denomination as a whole.
We will ask that the conclusion in the online version of Direction be edited to make this point more strongly.
Third, is the concern that finding uniformity should not be surprising. In light of the clarification about the actual study purpose, this point should no longer be a methodological concern. Simply put, yes, we were not surprised that we did not find uniformity. Yes, we did expect diversity of opinion – that being written right in the study purpose statement. Remember that the study aimed to explore more deeply and systematically “the diverse ways that Canadian MB congregations have used and are using the MB Confession of Faith in congregational life.”
Fourth, is the concern of the challenge of identifying Confessional uniformity. Again, this was not the goal of the current study. But as Ken acknowledges, the current study certainly has unearthed questions about the relationship between, and the limits of, uniformity and diversity on Confessional matters. This might warrant future exploration. The summary report slides end with a series of questions that might be helpful to that end. So rather than a concern, this point is perhaps one legacy of the study.
And finally, fifth is the concern of peer researcher bias. In other words, was it appropriate to use other MB pastors as interviewers? Here again I can offer a clear yes. But again, with a caveat – that proper training be done. The Centre for Community Based Research (CCBR) which I lead, has hired, trained, and supported hundreds of peer researchers in its 40-year history (CCBR is a nonprofit organization located on the University of Waterloo campus). Peer researchers have had lived experience on a range of social issues: homelessness, newcomer settlement, mental health and substance use, Indigenous justice, community violence/conflict, and many more. We have involved peer researchers because it can improve the quality of interviews particularly on sensitive topics, where rapport and trust help participants to speak more freely, and where shared lived experience can aid in the probing for deeper insight. There is a growing body of academic literature to support this claim (which I have contributed to) and that also points out cautions and limitations. Training is important to address these cautions. After all, in qualitative research it is the interviewer that is the “instrument”—the quality of the data is dependent on how well the conversation is facilitated to answer the questions. This is in contrast with quantitative research where the survey/questionnaire tool is the “instrument” with data quality dependent on how well the tool was designed and consistently implemented. As I listened (with participants’ consent) to the 20+ hours of interview recordings, I heard first-hand the benefits of having pastor to pastor conversation to deepen the exploratory conversation. So rather than viewing peer researchers as a weakness, they can actually be a strength for a study of this kind and purpose.
Taken together, the points above should allay the specific concerns about the study methodology outlined in the MB Herald Digest article. That is not to say that the study was without limitations (all studies have limitations). Those limitations were clearly laid out in the Direction lead article and the summary report slides. And we certainly invite critique of this study, as all researchers should. But the critique should be based on what the study is, not on a misunderstanding of its purpose.
I will leave it to others with more pastoral and theological insight to comment on how this misunderstanding of the study purpose might have impacted the second half of the MB Herald Digest article which discusses the study findings. Perhaps that response too may need some re-thinking.
One final word. Ken does acknowledge (quite eloquently) some contributions of this study, including that through it we have been “gifted with some information that could help us understand our pastors better and by extension our churches” (p 8). I would like to acknowledge the collective effort of that gift-giving. While the grant from the MB Historical Commission was welcome, it was modest. It paid for a small portion of time of one CCBR staff to help with the technical aspects of the research and a CMU student to conduct a literature review. The remaining effort was entirely in-kind—from advisory committee member participation, to peer researchers who in most cases arranged and conducted multiple interviews, to Brad’s considerable involvement as project co-lead, and of course to the 17 pastors who made the effort to prepare for the interview and took time from their busy schedules to participate in it. My own involvement was entirely voluntary. I estimate the initial grant investment was multiplied ten-fold.
I mention this to direct recognition to where it is due and to add my gratitude to those who risked being curious together and who cared enough about the MB family to be engaged on a challenging topic. They can be assured that the study strove to uphold standards of quality in rigorous research.
I trust this response helps to clear up misconceptions of the study and spark further dialogue in a way that advances our ministry of reconciliation and peace.
Rich Janzen is a member of Waterloo Mennonite Brethren Church. He is Executive Director of the Centre for Community Based Research and Adjunct Associate Professor at Conrad Grebel University College.
