Inside Facebook’s secret rule book for Global political speech

Facebook has drawn criticism for undermining democracy and for provoking bloodshed in societies small and large.

A Facebook logo being displayed in a start-up companies gathering at Paris' Station F, in Paris. (AP File Photo)

In a glass conference room at its California headquarters, Facebook is taking on the bonfires of hate and misinformation it has helped fuel across the world, one post at a time.

The social network has drawn criticism for undermining democracy and for provoking bloodshed in societies small and large.

But for Facebook, it's also a business problem.

The company, which makes about $5 billion in profit per quarter, has to show it is serious about removing dangerous content. It must also continue to attract more users from more countries and try to keep them on the site longer.

How can Facebook monitor billions of posts per day in over 100 languages, all without disturbing the endless expansion that is core to its business? The company's solution: a network of workers using a maze of PowerPoint slides spelling out what's forbidden.

Every other Tuesday morning, several dozen Facebook employees gather to come up with the rules, hashing out what the site's 2 billion users should be allowed to say. The guidelines that emerge from these meetings are sent to 7,500-plus moderators around the world.

The closely held rules are extensive, and they make the company a far more powerful arbiter of global speech than has been publicly recognized or acknowledged by the company itself, The New York Times has found.

The Times was provided with more than 1,400 pages from the rule books by an employee who said he feared the company was exercising too much power, with too little oversight — and making too many mistakes.

An examination of the files revealed numerous gaps, biases and outright errors. As Facebook employees grope for the right answers, they have allowed extremist language to flourish in some countries while censoring mainstream speech in others.

Moderators were once told, for example, to remove fundraising appeals for volcano victims in Indonesia because a co-sponsor of the drive was on Facebook's internal list of banned groups. In Myanmar, a paperwork error allowed a prominent extremist group, accused of fomenting genocide, to stay on the platform for months. In India, moderators were mistakenly told to take down comments critical of religion.

The Facebook employees who meet to set the guidelines, mostly young engineers and lawyers, try to distill highly complex issues into simple yes-or-no rules. Then the company outsources much of the actual post-by-post moderation to companies that enlist largely unskilled workers, many hired out of call centers.

Those moderators, at times relying on Google Translate, have mere seconds to recall countless rules and apply them to the hundreds of posts that dash across their screens each day.

Moderators express frustration at rules they say don't always make sense and sometimes require them to leave up posts they fear could lead to violence. "You feel like you killed someone by not acting," one said, speaking on the condition of anonymity because he had signed a nondisclosure agreement.

Facebook executives say they are working diligently to rid the platform of dangerous posts.

"It's not our place to correct people's speech, but we do want to enforce our community standards on our platform," said Sara Su, a senior engineer on the News Feed. "When you're in our community, we want to make sure that we're balancing freedom of expression and safety."

Monika Bickert, Facebook's head of global policy management, said that the primary goal was to prevent harm, and that to a great extent, the company had been successful. But perfection, she said, is not possible.

The Rules

The guidelines for identifying hate speech, a problem that has bedeviled Facebook, run to 200 jargon-filled, head-spinning pages. Moderators must sort a post into one of three "tiers" of severity. They must bear in mind lists like the six "designated dehumanizing comparisons," among them comparing Jews to rats.

"There's a real tension here between wanting to have nuances to account for every situation, and wanting to have set of policies we can enforce accurately and we can explain cleanly," said Bickert, the Facebook executive.

As detailed as the guidelines can be, they are also approximations — best guesses at how to fight extremism or disinformation. And they are leading Facebook to intrude into sensitive political matters the world over, sometimes clumsily.

Increasingly, the decisions on what posts should be barred amount to regulating political speech — and not just on the fringes. In many countries, extremism and the mainstream are blurring.

In the United States, Facebook banned the Proud Boys, a far-right pro-Trump group. The company also blocked an inflammatory ad, about a caravan of Central American migrants, that was produced by President Donald Trump's political team.

"Facebook's role has become so hegemonic, so monopolistic, that it has become a force unto itself," said Jasmin Mujanovic, an expert on the Balkans. "No one entity, especially not a for-profit venture like Facebook, should have that kind of power to influence public debate and policy."

In India, Chinmayi Arun, a legal scholar, identified troubling mistakes in Facebook's guidelines.

One slide says that Indian law prohibits calls for an independent Kashmir, which some legal scholars dispute. The slide instructs moderators to "look out for" the phrase "Free Kashmir" — though the slogan, common among activists, is completely legal.

Facebook says it is simply urging moderators to apply extra scrutiny to posts that use the phrase. Still, even this could chill activism in Kashmir. And it is not clear that the distinction will be obvious to moderators, who are warned that ignoring violations could get Facebook blocked in India.

'Things explode really fast'

In the absence of governments or international bodies that can set standards, Facebook is experimenting on its own.

"A lot of this would be a lot easier if there were authoritative third parties that had the answer," said Brian Fishman, a counterterrorism expert who works with Facebook.

"Sometimes these things explode really fast," Fishman said, "and we have to figure out what our reaction's going to be, and we don't have time for the U.N."

But the results can be uneven.

Consider the guidelines for the Balkans, where rising nationalism is threatening to reignite old violence. The file on that region, not updated since 2016, includes odd errors. Ratko Mladic, a Bosnian war criminal still celebrated by extremists, is described as a fugitive. In fact, he was arrested in 2011.

The slides are apparently written for English speakers relying on Google Translate, suggesting that Facebook remains short on moderators who speak local languages — and who might understand local contexts crucial for identifying inflammatory speech.

The guidelines, said Mujanovic, the Balkans expert, appear dangerously out of date. They have little to say about ultranationalist groups stoking political violence in the region.

The hate list

Facebook's most politically consequential document may be an Excel spreadsheet that names every group and individual the company has quietly barred as a hate figure.

Moderators are instructed to remove any post praising, supporting or representing any listed figure.

Anton Shekhovtsov, an expert in far-right groups, said he was "confused about the methodology." The company bans an impressive array of American and British groups, he said, but relatively few in countries where the far right can be more violent, particularly Russia or Ukraine.

Countries where Facebook faces government pressure seem to be better covered than those where it does not. Facebook blocks dozens of far-right groups in Germany, where the authorities scrutinize the social network, but only one in neighboring Austria.

For a tech company to draw these lines is "extremely problematic," said Jonas Kaiser, a Harvard University expert on online extremism. "It puts social networks in the position to make judgment calls that are traditionally the job of the courts."

The bans are a kind of shortcut, said Sana Jaffrey, who studies Indonesian politics at the University of Chicago. Asking moderators to look for a banned name or logo is easier than asking them to make judgment calls about when political views are dangerous.

But that means that in much of Asia and the Middle East, Facebook bans hard-line religious groups that represent significant segments of society. Blanket prohibitions, Jaffrey said, amount to Facebook shutting down one side in national debates.

And its decisions often skew in favor of governments, which can fine or regulate Facebook.

The view from Menlo Park

One hurdle to reining in inflammatory speech on Facebook may be Facebook itself. The platform relies on an algorithm that tends to promote the most provocative content, sometimes of the sort the company says it wants to suppress.

Facebook could blunt that algorithm or slow the company's expansion into new markets, where it has proved most disruptive. But the social network instills in employees an almost unquestioned faith in their product as a force for good.

When Su, the News Feed engineer, was asked if she believed research finding that more Facebook usage correlates with more violence, she replied, "I don't think so."

"As we have greater reach, as we have more people engaging, that raises the stakes," she said. "But I also think that there's greater opportunity for people to be exposed to new ideas."

Still, even some executives hesitate when asked whether the company has found the right formula.

Richard Allan, a London-based vice president who is also a sitting member of the House of Lords, said a better model might be "some partnership arrangement" with "government involved in setting the standards," even if not all governments can be trusted with this power.

But at company headquarters, the most fundamental questions of all remain unanswered: What sorts of content lead directly to violence? When does the platform exacerbate social tensions?

Rosa Birch, who leads an internal crisis team, said she and her colleagues had been posing these questions for years. They are making progress, she said, but will probably never have definitive answers.