by Ferre WOUTERS for FactCheckNI (25 January 2019)

Nieuwscheckers is a Dutch project for students in journalism at the University of Leiden. Supervised by lecturers Peter Burger and Alexander Pleijter, students monitor the media looking for statements or articles that make their eyebrows frown. Once such a claim is found, they ask the journalist who published the claim to clarify on which it is based. Students then investigate this and publish their conclusion online.


Journalists are supposed to check facts before publishing their own articles (ex ante fact checking). With Nieuwscheckers, journalism students are checking the claims made by others and after publication (ex post fact checking). Most of the articles on concentrate on unintended mistakes made by journalists that are not meant to mislead their readers. These claims can be categorised as “misinformation”. More recently, Nieuwscheckers has also focused on “disinformation”: the intentional spread of misleading messages to make money or to influence public opinion.

On Nieuwscheckers’ website you can find around 450 fact-checking articles, the first being published on 6 March 2009. A decade later, Burger and Pleijter organised a symposium to celebrate their 10th anniversary. On 24 January 2019, presentations and discussions were held among Dutch, Flemish, and international speakers related to fact-checking initiatives.

The symposium highlighted the current trends and challenges for fact checking in and beyond the Netherlands.

The global fact-checking movement in the post-Trump era

Lucas Graves is director of research for Reuters Institute. He got introduced as a “critical friend of the fact-checking movement”.

Graves began by saying that the “post-Trump era” in the title of his speech makes a much better phrase than the more famous “post-truth era”. The era we live in can be characterised as both a time that has never been so good for facts, with tons of new ways to discover them, and at the same time has never been that bad, with facts being openly challenged.

Graves stressed this new era for facts did not come into being with the presidency of Donald Trump. Ronald Reagan already challenged the integrity of the press in 1980. Newspapers at that time avant-la-lettre fact checked his speeches, but stopped doing this after a couple of weeks, because it turned out it didn’t really bother their readers, Graves explained.

According to Graves, the 1988 presidential campaign of George Bush Sr and Dan Quayle was a turning point in the history of both fact checking and journalism. At that time, the campaign was seen as a low point in the history of American politics. On 19 October that year, ABC News even fact checked every statement made in a advertising video of Quayle.

From then on, presidential campaigns got more and more fact checked by organisations as the Washington Post. The fact-checking movement started as what they saw as a legitimate way of doing journalism, said Graves.

Graves outlined the different types of fact-checking organisations. Most fact checkers are tied with a news organisation, and identify themselves as reporters doing journalism. Others are part of independent NGOs, and see themselves as nerdy experts, wanting claims to reflect statistics. Still others describe themselves as activists, with their fact check website operating as a watchdog organisation to fight populism. Only 22% of fact-checking organisations in the USA are not part of a parent news media organisation, while 48% are in the rest of the world.

Fact checkers are getting more influence in some policy fora and some are working with social media platforms. Graves gave the example of ClaimReview, a fact-checking review that Google basically paid to develop, which takes input from fact checkers. [Editor: FactCheckNI participates in ClaimReview.] Facebook is also collaborating with fact check initiatives, although posts ironically can get more likes when getting labelled as untrustworthy.

Graves argued that the booming of fact checking involves a paradox: it is more easy to objectively fact check when nobody cares about what you are doing, whereas it is more difficult to insist on the original commitment of objectivity when stakeholders and policymakers are involved. [Editor: Fact-checking organisations can subscribe to a Code of Principles that includes transparency of funding, organisation, and methodology.]

The reach of Dutch junk news on Facebook

Nieuwscheckers, together with the Leiden Institute for Advanced Computer Sciences, conducted research (also published in English) on the spread of junk news in the Netherlands. They define junk news as “pulp” Facebook pages and websites getting likes by sharing hoax stories with clickbait titles, but also those with cat pictures and “nice” quotes. Nobody knows about these pages, but they are getting a lot of followers, some even more than the Dutch national news page (NOS), Burger and Pleijter claimed.

To them, two main problems of junk news is, firstly, that it spreads misinformation, and secondly, that it takes attention, and money via advertisements, away from real news sources.

For their research, they collected data (likes, comments, shares) for these Dutch pulp pages on Facebook, and did the same for mainstream media pages. Posts coming from pulp pages on average got more likes compared to mainstream media, and get even twice as much comments and shares.

Their research also reveals that measurements done by Facebook pay off; although the number of pulp pages and junk posts is still increasing, the shares and comments went down since the summer of 2017. At that time, Facebook took measures so that junk news appeared less often on users’ timelines. The question we should ask, Burger and Pleijter suggested, is if we should go further with this and take more measures.

Edo Haveman (Public Policy, Facebook) explained that three main measures were taken: (1) finding and closing fake accounts, (2) collaborating with fact checkers, and (3) releasing campaigns to raise awareness. Haveman stressed that most companies like Facebook have good intentions.

Gert-Jaap Hoekman (Editor, also said there is no need for pessimism. In fact, the reach of big news channels is an upgoing trend in the Netherlands, and together are reaching around 10 million people. Of course there is competition among different media, in terms of content and headliners for example. According to Hoekman, pulp media are successful because they are not going all the way, by not completely going into competition with news media. He added that mainstream media organisations overestimate how many people are looking for news on social media; more likely, people go to social media looking for entertainment.

Google makes it possible for their users to report certain messages, which accordingly will be checked, says Arjan El Fassed (Public Policy, Google). Moreover, pulp stories don’t appear high up in their search results. Along with readers and advertisers, Google is “not happy” with junk news; people have the right to be free to read correct news. We have to be careful, warned El Fassed, to not go too far with these interventions. He said there is a grey zone: “When does tackling junk news violates my right of free opinion and of access to information?”

The audience was not fully satisfied with what Google and Facebook had to say; doubt was expressed that advertisers are not happy with junk news, as they make money by putting advertisements on websites with pulp articles that are spread on social media. Does Google or Facebook know how much money these websites are earning via advertising?

Google, in principle, knows how much they earn, because they are paying these websites. However, El Fassed stated that it would not make this information public. Haveman said that people are making money this way for a really long time, and this will never disappear.

Could they reveal other information and data to journalists or fact checkers, such as the method they use to trace accounts that repeatedly share junk news, for example, or the list of those accounts they block from their server? Both Facebook and Google stressed they would keep investing in projects battling junk news, but explained they could not release their methodology, for this would increase the risk of manipulating their server.

Fact checking in the practice of journalism

There are several initiatives in Flanders and the Netherlands which contribute to include fact-checking practices into the field of journalism.

Make Media Great Again is one such initiative. It introduces a system where so-called “Annotators” review articles, providing suggestions and remarks. By using annotations, highlighted statements that need to be verified, volunteers contribute to the editorial process. As an experiment, they applied this technique to 400 articles on Peter Olsthoorn (Annotator Coach) stressed this is a needed initiative, not because journalists are doing a bad job, but because they are humans and not infallible.

According to Jan Jagers (a fact checker with Knack magazine), Belgium is a country without fact checkers. In contrast to the Netherlands, there exists no fact-checking culture, because in Belgian journalism, transparency and standard procedures traditionally are less important, he argued. Jagers said that the prime minister’s pledge to invest in fact-checking initiatives ironically turned out to be fake news! (The Prime Minister of Belgium, Charles Michel, resigned on 18 December 2018.)

Jagers showed several examples to illustrate the poor quality of fact checking in Belgium. One came from a newspaper doing a fact check of an anti-migration campaign from a right-wing party, by doing an interview with a professor who is known to have left-oriented views. Another fact check was done, not by a journalist or NGO, but by a political party. Jagers concluded that it is time to build an independent Belgian institute for fact checking.

How do those who fact checked feel about being fact checked? It is irritating, Rennie Rijpma (Editor, Algemeen Dagblad) remarked with a smile. However, it is even more irritating to detect misinformation in your own media organisation. Fact checking, and rectifying an article accordingly, is time consuming. It takes about a week, which is really long in times where deadlines for publishing have been reduced from days to minutes. This is not an excuse to ignore fact checking. Correcting articles, as well as being transparent and showing which parts got amended at what time, is important, Rijpma concluded.

Monique Hamers is a lecturer in journalism at the Fontys Hogeschool (College) in Tilburg. With Factory she has built a platform for journalism students to verify news items. By making visualisations and videos, students also pay attention to how to reach people with fact checks on social media. Hamers tells her students that providing context is the most important job of a fact checker. It is no coincidence Hamers was invited to speak on this symposium, since Factory directly influenced the founding of Nieuwscheckers.

Also inspired by Factory, Ria Goris introduced a similar platform for students at the Erasmus University College in Brussels. She found that most conclusions students make after fact checking are grey rather than black or white. Goris also conducted research by interviewing experienced journalists from Belgium. It turned out that journalists themselves confessed that they consider most sources reliable, by default, out of time pressure.

Innovation in fact checking

After talking about journalism, the symposium focused on fact-checking innovations.

Newspaper de Volkskrant publishes their fact checks in the form of a video. In these videos, Maarten Keulemans explains the fact check of a claim in an accessible manner. [This way of communicating fact checks is more attractive and may evoke more attention; however, being able to retrace the sources of facts is lost.] Keulemans presented direct tips for how to make fact-check videos:

  1. directly evoke attention;
  2. maintain a personal approach;
  3. think about how to visualise the statistics;
  4. “f–k alla them rules; follow your feelings”.

Next up was Maarten Schenk, who is said to be the “world’s fastest fact checker” [a claim that needs to be checked!]. His website, which started as a hobby but turned into a real job, debunks the spread of misinformation. Using his own developed Trendolizer engine he looks for trending online content. After checking this content, Schenk can publish his fact-check article really quickly because his articles are almost completely pre-programmed. The only thing he needs to do is to label the content (e.g. as “hoax” or “satire”), to insert the URL of the fake news message, and to add “NOT” into the original title. Then, LeadStories automatically produces a video about the article. This gets published even faster, by using the same pre-programmed format for every video, including funky background music, and a robot voice reading the article out loud. The purpose of these videos is to make people find out if a fake message is satire or a hoax as quickly as possible, and thus completely differs from those of de Volkskrant.

With his software, Schenk can reveal whole networks of websites coming from the same owner. Recently, he collaborated with Nieuwscheckers, and together they dismantled a network existing of mostly Macedonian links spreading anti-islam misinformation. Their discovery attracted a lot of media attention worldwide, and the network later shut down. You can read this detective story here.

Another innovative practice getting more popular is live fact checking. In September 2018, news company NRC live fact checked political debates of the Dutch House of Representatives in the Hague. Journalists Jisca Cohen and Pim van den Dool explained how this worked out. Before starting to check, it is fundamental to prepare yourself on what will be discussed. They did this by consulting experts and reading the political parties’ programs. During the meeting of the House, 15 editors were at work. Some were live blogging what was happening during the discussions, others wrote down checkable statements that were made, and finally some fact checked these claims and immediately posted their conclusions online. The lesson they learned from doing this is that it puts pressure on politicians, who become more cautious in making unverified statements.

The promises of automated fact checking

The last presentation was given by Mevan Babakar, head of automated fact checking for Full Fact, who uses innovative technology to develop automated fact-checking software.

Babakar explained that when doing fact checks, one has no choice but to work in a very targeted way. The types of misinformation, disinformation, and malinformation, out there, online and offline, are very diverse. Behind the spread of these different types lies a bunch of different kinds of motivations. The mechanisms used to distribute types of information are getting more diverse as well. This variety in misinformation also results in different forms of harms.

Full Fact has developed automated tools for fact checkers. One tool records all appearances of a claim that has been fact checked, making it possible to see how much and by whom a claim is made. Another tool relates to live checking. Parliament debates get transcribed and checked by a team of fact checkers. During live checks, Full Fact hands out sources to the public about statements being made.

With automated fact checking, data becomes machine readable. The question then arises if soon fact checks will be done by artificial intelligence on a large scale. Babakar urges we should be cautious. Getting fast data is not enough. She gave the example of homicides in England and Wales, where a huge increase in one year was caused by only one man [see also the FactCheckNI articles about suicide, and the bedroom tax]; AI wouldn’t bother about this detail. Caveats like this can not [yet?] be seen by technology. Likewise, dismantling cherry picking of data and delivering context is difficult with technology and automated fact checking only.

When fact checking, one is collecting evidence to know why and what happened. Babakar stressed the importance of people working as fact checkers: it is very helpful to have someone in society asking, “Where the hell did you got those facts from?”. Although Full Fact is a relatively small team of 18 employees, they are really having a big impact in the UK and on the media landscape.