TUFF Cookies: Students aim for policy impact through hard-hitting research, part one

By Kyra Solomon, Brad Parks, Soren Patterson

This blog was re-posted from The First Tranche. View the original post.

Over the last two decades, China has dramatically expanded its overseas development program. AidData’s soon-to-be-published report, Banking on the Belt and Road, demonstrates that China is now outspending the United States on a more than two-to-one basis.

However, China does not disclose its overseas spending according to international reporting standards, leading to a vacuum of reliable information about where, when, and how Chinese government-financed development projects take place, which players are involved in implementation, and what effects these projects have in host countries.

This informational void inspired AidData to develop the Tracking Underreported Financial Flows (TUFF) methodology, which in turn led to the construction of the world’s most comprehensive dataset on Chinese development projects. The latest (2.0) version of the dataset will launch on September 29 alongside the Banking on the Belt and Road report.

The first 1.0 version of the dataset was released in the fall of 2017. Since then, more than 100 undergraduate and graduate research assistants have been engaged in an ambitious effort to extend, expand, and enrich it.

“To comprehensively track the far-flung financial activities of a major world power, we assembled an enterprising group of student researchers with a shared interest in undertaking rigorous, innovative, and impactful research,” said Brooke Russell, an author of the report and Associate Director at AidData, who has overseen data collection and quality assurance procedures at the research lab for more than a decade.

“Our previous dataset resulted in more than 250 academic and policy publications, and we hope this release will fill the evidence gap and become a global public good for researchers and decision makers on all sides,” said Dr. Ammar A. Malik, an author of the report and Senior Research Scientist, who oversees the TUFF team and leads AidData’s China Development Finance Program. Dr. Malik was previously the Director of Research at Evidence for Policy Design (EPoD), a research initiative at Harvard Kennedy School, where he led research-policy collaborations by deploying evidence-based insights and training to improve public policies and leadership.

The new research is timely—as the G7 has worked to develop an alternative to China’s Belt and Road Initiative known as the Build Back Better World (B3W) initiative, the TUFF team has worked diligently to build a stronger evidentiary foundation for analysts and policymakers inside and outside of China.‍

Learning while doing

“I first came across the work of the TUFF team while researching Chinese lending, and was thrilled to see that William & Mary had established a strong reputation in this area of Chinese foreign policy,” said Carolin Helmholz, who graduated from William & Mary in 2018 with a major in Chinese Studies. “By working with the TUFF team, I knew I’d be generating new evidence related to one of the most active topics in foreign policy.”

Research assistants like Helmholz bring to bear diverse backgrounds and interests as they seek to uncover new details and insights about China’s overseas development program.

“Given my focus on STEM during the first half of my college career, working with the TUFF team provided my first opportunity to conduct open-source research. We are capturing and synthesizing on-the-ground details and firsthand observations made by recipient communities and local journalists,” said Helmholz. She said that working on the team has allowed her to both “learn the ins-and-outs of development finance and develop a more nuanced understanding of China’s expanding role as a foreign lender and donor.”

Sailor Miao ’24, a Government and Spanish major, has always been fascinated by the topic of Chinese aid and debt. “I am especially interested in how China spends money in the least-developed countries,” he said.

Research assistants on the TUFF team learn about important international issues, and the data they produce contributes to analysis that impacts media coverage, legislative debate, and public sector decision-making.

“AidData’s work informs and supports the decisions of USAID, the White House, and G7 finance ministers,” said Sam Rofman ’22, a Data Science major.

Seeing how international assistance programs are delivered and communicated to local stakeholders on the ground sheds light on how international diplomacy works in practice. “Documenting where and when  signing and handover ceremonies take place puts into context the various responsibilities of diplomats,” said first-year Andy Shufer ’25. “As someone interested in diplomacy, being able to see in practice things that I’ve learned about has been enlightening.”

The data’s in the details

In order to collect the data, TUFF research assistants follow a transparent, systematic, and replicable set of procedures in three stages. The TUFF methodology involves the identification of project-level data from several different sources; triangulation of additional information and updating of project entries; and careful vetting of all project records to ensure that they are accurate, detailed, and complete.

The first iteration of the methodology was pioneered by a former William & Mary student, Austin Strange ’12, who went on to a Ph.D. from Harvard and now teaches and studies Chinese foreign policy as an Assistant Professor of International Relations at the University of Hong Kong. With Dr. Bradley Parks, AidData’s Executive Director, he is also the coauthor of a forthcoming Cambridge University Press book—entitled Banking on Beijing: The Aims and Impacts of China’s Overseas Development Program—that provides extensive analysis of the 1.0 version of the dataset that was released in the fall of 2017.

The most recent (2.0) version of the TUFF methodology has been reengineered to rely more heavily upon official sources, such as grant and loan agreements published in government registers and gazettes, as well as records stored in the aid and debt information management systems of recipient countries. In total, the 2.0 version of AidData’s Global Chinese Development Finance Dataset draws upon more than 90,000 sources. The average project record in the dataset is based on nearly seven sources, and 89% of project records are underpinned by an official source.

The data collection process can seem daunting at first. But as Shufer noted, “with the amount of support that is provided to each research assistant and the incredible training that we received, I was more than prepared to handle all the tasks I was given.”

“We aim to capture the entire lifecycle of a project—from the initial signing of a financing agreement to the beginning of implementation and then all the way to project completion,” recent graduate Jack Mackey ’21 explained. “I really appreciate the TUFF methodology because it ensures that every data point is scrutinized by several pairs of eyes before it ever appears on china.aiddata.org.”

Rofman, a Management Research Assistant, noted that one key lesson is that preparation and certainty go hand in hand. “It’s crucial that each coder knows the TUFF methodology inside and out and is able to confidently make determinations about projects that will stand up to external scrutiny,” she said.

“Quality Assurance (QA) is the final line of defense. While others will review the project, the QA coder is the last person responsible for making sure that everything is in order,” said Rory Fedorochko ’22, a History and International Relations major. “It can be fun, like a game or puzzle. Sometimes you hit a dead end, and other times you realize that you’ve just uncovered something truly rare or extraordinary.”

TUFF research assistants are not automatons working on an assembly line. They are more like detectives who chase leads and triangulate information from various sources in multiple languages. They scour official records in 160+ countries to identify key transactional details and then seek to document all of their findings in ways that will be easily accessible by and understandable to analysts and policymakers.

“The work we do on the TUFF team is inherently investigative because we’re constantly trying to figure out what kinds of financing are being provided by China, where the money is going, and how projects are being implemented in practice,” said Mackey. “I’ve come to learn that work on the TUFF team is all about the details, and finding those details when they’re not immediately available or apparent can be very rewarding.”

Each project captured in the dataset has its own story. Fedorochko remembers an example from when he was tracking Chinese government-sponsored projects in Montenegro, a small European country with a population of about 620,000. The Bar-Boljare Highway project stood out from the others because it was financed with a $943 million loan, which is roughly equivalent to 20% of Montenegro’s GDP.

“This project has been very controversial, which I documented in the project description. The so-called ‘highway to nowhere’ was not economically feasible, and now the country is heavily in debt. It is rare that a single project stirs so much controversy,” said Fedorochko. “Also, the TUFF team managed to acquire an unredacted version of the loan agreement with China Eximbank, which provided valuable details that would not have otherwise been accessible to the research or policy community. It’s one of the more memorable project records in the database that I helped construct.”

“Whether it’s a loan being repaid with the cash proceeds from cocoa bean sales or a payment being made to North Korea by China to encourage its participation in nuclear negotiations, we track it all,” said Rofman.