Guns, Incels, and Algorithms: Where We Are on Managing Terrorist and Violent Extremist Content Online

Armstrong-Scott, Gabrielle L.; Waldo, Jim

From the document: "Ten years ago, U.S. national security agencies grew concerned about a relatively new and powerful weapon used by terrorists: the World Wide Web. What had begun as an effort to connect end users from across the world to share information and to serve as a force of human liberation, instead began to be used as a tool for destruction of life. Terrorists were exploiting technology companies' lax content moderation policies to recruit new members, spread violent extremist ideology, and plan terrorist attacks. [...] Technology companies and governments have spent the past decade trying to better address the evolving threat of terrorist and violent extremist content online (TVEC). However, there are few studies examining just how effective these efforts have been, where we are today in managing the problem, and wherein lie gaps for improvement. This paper argues that companies' efforts to deal with TVEC have been hampered at the outset by a tendency to define TVEC extremely narrowly. [...] This paper also explores the idea of ethical obligations and norms as an alternative to a legally required definition. On the technical side, this paper finds that even if there was consensus on the legal and ethical questions surrounding TVEC, the technical tools currently available are no panacea. Trade-offs across efficiency, scalability, accuracy, and resilience are persistent. Current technical tools tend to disadvantage minority groups and non-English languages. They are also less robustly implemented across small and non-U.S./European firms, generally either because they are left out of inter-firm initiatives or because they lack resources and capability. This paper does not claim to cover every issue relevant to TVEC; however, it highlights several important gaps that could be addressed by policymakers and tech companies and identifies avenues for future research."

Belfer Center For Science And International Affairs. 2023.