In early 2017, an Ontario Provincial Police programmer wrote a piece of computer code used to scrape conversations from an online chat program.
The algorithm automatically copied and stored the communications of users across dozens of chatrooms — some of them password-protected — nearly constantly for many months that year, the programmer later testified. The mass interception occurred without a warrant, a Crown attorney acknowledged, according to court transcripts.
A new report from the University of Toronto’s Citizen Lab offers a survey of “algorithmic policing” in Canada, and warns that these surveillance and investigative tools have outstripped the country’s legal oversight mechanisms.
Law enforcement agencies, the authors say, acquire and use such tools without the transparency necessary to ensure they don’t run roughshod over constitutional and human rights concerns — including re-entrenching the racial biases that are currently the subject of fierce protest and scrutiny across North America and beyond.
The Citizen Lab authors found instances of both automated surveillance — including the warrantless chat-scraping tool — and of “predictive policing,” a method that attempts to predict where crimes will occur or who might commit them. But they warn that the true scope of the use of these methods is likely unknown, since law enforcement is often unwilling to divulge the specifics of these tools or even acknowledge their existence, citing the need to preserve investigative techniques.
“Undoubtedly, algorithmic policing technologies are here,” said Kate Robertson, a research fellow at Citizen Lab and defence lawyer who co-authored the report.
“We encountered far too many challenges in attempting to learn the full extent to which police agencies are using novel algorithmic technology … and have come away from this project with a distinct sense that more is very likely out there than what we know.”
The chat-scraping tool came to light in a recent court case involving a man who was charged with child pornography offences after the OPP programmer, a civilian employee, passed on information derived from the stored chat data to Waterloo regional police. The programmer, who is also contracted by a U.S. technical college that paid him to develop the chat-based code, testified he created the tool to “memorialize” chatroom activity; the tool was then integrated with a U.S.-based law enforcement portal, ICACCOPS, designed to fight online child exploitation.
The charges were dropped before the judge in the case ruled on a charter challenge by the defence, which argued the tool was an unconstitutional interception of private communications: such interceptions require strict judicial authorization. A Crown lawyer acknowledged the chat-scraping algorithm captured real-time conversations and stored them indefinitely without a warrant, according to a court transcript, but indicated the Crown would argue these communications weren’t considered private.
When asked whether the OPP continues to automatically log and store chat, social media, or other online communications, and whether it does so without a warrant, a spokesperson responded that the Child Sexual Exploitation Unit employs specially trained investigators who “utilize a variety of tools and investigative techniques to identify and locate those individuals intent on the exploitation of children and to locate, identify and ensure the safety of the child victims … Regular consultations and ongoing training are conducted with legal counsel to ensure that investigations are conducted in a way that protects the rights of suspects as well as the safety of victims.
“Any evidence collected by CSEU members is done so in strict accordance with the law,” Bill Dickson added.
The researchers at Citizen Lab decided to survey the extent of algorithmic policing in Canada after noting the widespread use of such tools in other jurisdictions, particularly the U.S. and U.K., Robertson said. These tools can be roughly divided into three categories: algorithms that try to predict where and when crimes might occur; algorithms that try to predict who might commit crimes; and algorithms that automate the collection and analysis of surveillance data on a massive scale.
In the location-focused category, the authors cited a Vancouver Police Department (VPD) machine-learning system called GeoDASH that tries to forecast where and when break-and-enter crimes might occur. Using historical data on type of crime, geographical co-ordinates, date and time, the system generates six location-based forecasts for every two-hour interval throughout the day. Uniformed officers in marked cars are deployed to these “high risk” areas to deter criminal activity.
In addition to the force’s public statements and an interview with a VPD officer, Citizen Lab acquired internal documents describing how the officers engage in “proactive” deterrence activities such as monitoring laneways and identifying known offenders and potential persons of interest entering the area. The force may also alert the local civilian neighbourhood watch. The system was adopted after a six-month trial found a substantial decrease in break-and-enters, according to the force.
On algorithmic policing generally, the Citizen Lab report cites a human rights concern raised by civilians and experts alike: that predictive policing technology, because its algorithms rely on data generated by human police forces and their historical practices, will not only re-entrench existing biases and systemic failings but “math-wash” those flaws by proffering the illusion of computer-based neutrality. Algorithms are only as good as the data they are fed.
“A problem that we kept encountering in our research that there is a documented tendency in humans to assume to some degree that because a computer is giving them information, that it is somehow inherently reliable,” Robertson said.
“By moving criminal justice datasets that are notoriously rife and tainted with historic biases into algorithms … we will entrench, perpetuate and hide” those biases.
Some location-focused predictive policing tools in use in the U.S. have stoked these fears, the authors note, by relying on data more commonly associated with marginalized communities that are already overpoliced, such as public “disturbance” calls and the existence of certain types of housing. The VPD officer interviewed by Citizen Lab said the force, mindful of these discriminatory feedback loops, had created “exclusionary zones” within the GeoDASH system to take account of “sensitive areas” like the intensely marginalized Downtown Eastside neighbourhood. Officers monitor the system for potential overpolicing, with special attention to “socioeconomically or culturally sensitive” areas.
The Citizen Lab authors, however, note that despite the VPD’s efforts to mitigate algorithmic discrimination, some forms of bias may remain. Residents of some communities are more likely to report crimes, and to be believed. Civilian neighbourhood watch groups alert to an elevated risk of crime may be more likely to fall victim to the “out of place” effect, in which residents or police in predominantly white communities become suspicious of racialized individuals.
Fears of algorithmic discrimination become even more heightened in the arena of person-focused predictive policing, technologies that try to identify individuals who might be involved in future crimes. Canadian law enforcement officers interviewed by Citizen Lab expressed doubts about these methods, with one Vancouver officer saying “ethically, it would be reprehensible to move in that direction.”
In the U.S., person-focused predictive policing has relied on data such as purported gang affiliations; the Citizen Lab authors note that labelling someone as “gang-involved” can be based on incorrect, discriminatory assumptions. Other data, such as prior criminal history, can be influenced by over-policing.
The Citizen Lab survey found that Calgary police have a software licence with Palantir, which makes predictive policing tools, but that the force currently uses it to integrate internal data silos, and not for its algorithmic policing capabilities. A Saskatchewan Police Predictive Analytics Lab housed in Saskatoon developed a predictive algorithmic model to identify children and youth at risk of going missing; an officer told Citizen Lab the group intended to expand to address other safety concerns such as repeat and violent offenders, domestic violence and the opioid crisis.
The researchers believe there is more algorithmic policing underway than they or others have discovered: a freedom-of-information request to the Toronto Police Service regarding facial recognition technology, a surveillance tool that relies on algorithms, returned no responsive documents; the Star later revealed the force had been using the technology for over a year.
The researchers lay out a series of recommendations, all of which are animated by a key principle: that “robust standards of transparency and accountability” are needed, given the unprecedented risks and intrusions presented by these tools.