Projects can look dramatically different after delving deeper—that’s one of the biggest learnings for the UBC students who spent a month in Asia this past summer on the Global Seminar, Indonesia: Impact Evaluation Analytics. A UBC Go Global offering, Global Seminars are immersive, small-group courses taught by UBC professors on location around the world.
And after 10 years teaching the seminar, instructors Sean Holowaychuk and Chris Bennett scored a major win this year—garnering an invitation to present their findings, conclusions and recommendations to the Bogor City Administration council members and the opportunity to convene all stakeholders in one room.
“Impact Evaluation Analytics” guides students on valuing and giving accountability to international aid projects. That’s important because “the No. 1 reason these projects fail is a lack of independent evaluation,” says Holowaychuk, a UBC Faculty of Land and Food Systems (LFS) instructor and session lecturer. Bennett, who lives in Indonesia, is an LFS adjunct professor. Both do project evaluation in international development professionally and have been sending UBC undergrads into the field for inclusive, “learning-by-doing” sessions with partner IPB University outside Jakarta since the program’s inception. Their passion for the work comes through to every program participant, former students say.
Scenery on the IPB University campus outside Jakarta: IPB partners with UBC on the Global Seminar in Indonesia (photo: Helena Low)
For one team’s project this year, four students evaluated a women’s collective reviving the local batik art of printing on fabric. Beforehand, the group had settled on a project focus. Then they began talking with the workers in person.
“After one interview, we realized there were more issues at hand,” says Graciela Licardo, an LFS Global Resource Systems major plans to go into sustainable development. “There were the women’s relationships with the villagers, the actual work of building their business, as well as many political aspects. Some of the villagers were against the project. That was a shock, and not something we expected. We got to act as sort of detectives rather than sitting in a classroom trying to analyze the situation from afar.”
A close-up look at the women collective's Batik designs (photo: Helena Low)
Says Helena Low, an LFS Global Resource Systems major: “I assumed we had support from the whole village to do the operations; in reality, they didn’t really like it. They weren’t happy because it was bringing in tourists with no positive effects, but more of a disturbance.”
The most important thing, they soon realized, was going in with an open mind, says Graciela. “There are so many factors you might not be aware of and cultural differences, and it’s key to take in all of the information objectively,” she says. “The second is forming connections with the people involved.”
That’s exactly what the professors hope the students will discover, and do so on their own. A notorious problem in the “tricky business” of impact evaluations is an outside expert swooping in to a community with the message, “It works in my country, so it should work in yours,” says Holowaychuk. He refers to this as the inherent “paternalistic nature” of the process.
Entrance to the Indonesian village, Kampung Batik Cibuluh, where the group's UBC Global Seminar project took place (photo: Helena Low)
“Our greatest accomplishment is not that we go in, write these reports and make these presentations; it’s when we get all the stakeholders together,” he says. “Getting people communicating and in the same room is what facilitates success. It’s when someone turns around and says, ‘Wait, that’s not true,’ and somebody else says, ‘Well, I didn’t know that.’ There are so many factors to consider and there are real needs in the community that need to be addressed.”
So professors and students alike were surprised and delighted when, “engaged and listening,” everyone including high-level university administrators showed up for a community meeting at Bogor city hall to discuss together how to make the batik operation into a thriving entrepreneurial endeavor. That level of buy-in is a welcome first after a decade of relationship building, both instructors say.
Group photo with stakeholders and professors who made it all possible: taken after the UBC student presentation at IPB University in Indonesia (photo: Emily Pennefather)
The overall aim of impact evaluation is to add value to development projects and correct market failures: for example, aid or assistance programs implemented by governments, institutions or NGOs targeting watershed mismanagement upstream of local villages, or challenges with healthcare programs or goat farming operations.
“Our course teaches how to apply accountability,” Holowaychuk explains, “that is, measuring the right things, in the right way, at the right time. Funding agencies, donors and taxpayers want a better understanding of where their money is going, and our role is to provide some evidence of what works. Chris (Bennett) and I have a criteria we’ve developed, the Impact Evaluation Analytics methodology. We give students our methodology, but not a lot of answers.”
Working through their plan was a major focus of the course, recalls Soyoung Jeong, who studies cognitive systems in the Faculty of Arts. “There were big, long meetings and some yelling, but at the end, it worked out.”
Says Julia Evans, a Global Resource Systems student with LFS: We had to figure out: who was impacted? Who was benefitting from the business? Depending on whom it was affecting, some people had totally different perspectives than others on the strengths and weaknesses.”
Adds Holowaychuk: “We are there to listen. We don’t give advice; we’re not technical experts. Our students might say, ‘You need to hire a technical expert because there’s something wrong with the crops; you need to seek advice.’”
Soyoung found that applying her classroom learning from psychology research methods “straight into the field” was the best experience—and one that has given her confidence to go out into the world with a feeling of “independence and ownership” of her views.
Women's collective member Melinda making Batik with the traditional Indonesian tool for applying hot wax (photo: Soyoung Jeong)
In addition to many life learnings (be open to connection; learn how to relinquish control of outcomes; take ownership of what you want to do), the students came to understand the importance of the “why”—one of Holowaychuk’s mantras. “Every time I meet with Sean to discuss a finding,” Helena recalls, “he asks, ‘Why?’ It’s a really important thing to do in our interviews. When I reflect now, we asked an official what he did with the funding. He said he bought a fish pond. Why? ‘It’ll feed the people,’ he said. We should have probed more. Why do the people need that? Who’s eating the fish? Why?”
Just as impactful was bonding with the locals, even with a language barrier and through a translator. The students joined potlucks, and enjoyed moments of connection and bonding over food and laughs.
Next, say the instructors, will be parlaying this year’s findings from the cohort of 23 into the curriculum for next year’s Global Seminar. The students lay a foundation that each consecutive cohort builds upon, says Holowaychuk: “They ask tough questions and hold organizations accountable, which has led to improved outcomes for partner-beneficiaries.”
Whatever happens with the Bogor batik business, the seminar group has succeeded in creating conversation where none existed before. “And ultimately,” says Julia, “we were able to present to the government. To make that impact was very special.”
UBC Global Seminar students after presenting their findings to local government members in Bogor (from left: Graciela Licardo, Soyoung Jeong, Helena Low and Julia Evans (photo: Sunny Das)
Get details about the UBC Global Seminar, Indonesia: Impact Evaluation Analytics.
Read about UBC’s Go Global programs for UBC and international students.
Find out about the UBC Faculty of Land and Food Systems.
Views on the walk to a student lecture, led by UBC's Sean Holowaychuck, at the IPB University campus in Bogor, Indonesia (photo: Julia Evans)