The Challenge

3789 followers

How can we strengthen the Internet for free expression and innovation? read the brief

Entry

Checkdesk: An Open Verification Toolkit for Journalists in the Digital Age

How can I verify this? That’s the constant refrain of journalists covering fast-moving stories online. Faced with a barrage of videos, tweets and photos from myriad sources, how do you know what’s true? Misinformation spreads like wildfire. Think Sandy, Boston, Syria. Sometimes even trusted news organizations spread it. How can you avoid the trap? Checkdesk can help you. Drawing on Craig Silverman’s Verification Handbook, Checkdesk offers a checklist of functions for checking a video, photo, or tweet. See an amazing new photo on a big story and Checkdesk will take you through the steps you need to check it’s true: reverse image search, reveal EXIF data, view satellite image of location. Checkdesk turbocharges you into the editor’s chair.
Let’s show you an example of how Checkdesk can help you get your job done.

You’re a journalist covering the conflict in Syria. You spot a photo trending on social media. What do you do? You have to establish how credible the image is. Here’s where Checkdesk comes in. It gives you a predefined list of operations from the Verification Handbook to help you verify the photo, available on the same page with a single click. For example:
 
- Reverse image search using Tineye
- Reveal the image EXIF data using FotoForensics.com
- View satellite imagery of the photo’s geolocation so you can identify landmarks
- Cross reference photo location, time and date with weather information using Wolfram Alpha

With these steps conducted, you feel more confident to report the photo accurately. You can share out a link or embed the photo on a blog. And with Checkdesk, you can also share the verification steps you took: our embed feature contains the results of the verification operations alongside the source content just like the health warning that accompanies an AP video script.  

What makes Checkdesk really powerful is that the checklists are customizable and shareable. We have checklists for videos and checklists for tweets, each with different strategies for verification. As new tools and platforms emerge, you can create and share out your own checklists - or adopt them from others. Why not collaborate to find the perfect formula?

Checkdesk then is so much more than a static feature set, it is an open framework of ‘integrations’ with 3rd party services. Think of the long arc as like an app store for verification. Developers with a journalistic mind can find their own ingenious ways to exploit third party tools and build new integrations that you can add to your verification checklists. For example, there is huge potential for collaboration with other tools such as Cryptocat (for end-to-end encrypted communication between journalists and media sources), Rbutr (to cross-reference any other work done to highlight media as misinformation), Alghayma (to archive citizen media and protect it from deletion) and Storyful’s Multisearch (for cross-social search). Further integrations could enable translation of key content or consistency checking of urls over time.

What we propose is not a silver bullet for verifying digital media, but a more structured, easy-to-use and expandable toolkit that makes it easier for you, the journalist, to use the best techniques already out there.
 
Who are the users or target customers of your project, and what have you learned from them so far? Please give specific examples.
 
Our users are journalists and editors working for newswires, major publications, and grassroots news organizations that use social and citizen media as a source for reporting. In our work thus far, we have noted that newsrooms rarely have a standardized procedure for verifying the citizen media they use in reporting, and even more rarely do they have a means of communicating their verification work to their readers and viewers.

Our project aims to add value to journalists by providing robust workflow tools based on best practices outlined in the Verification Handbook and checklist efficacy research conducted by Peter Pronovost. We aim to add value to media consumers by adding key information to the citizen and social media they consume such as the extent to which media has been verified. The need for this is highlighted in the BBC Trust report on BBC coverage of the Arab Spring and in an upcoming Tow Center report on User Generated Content in Broadcast Journalism.

One example of when Checkdesk may have spared a major news organization’s blushes and avoided confusion for readers is the BBC’s misuse of a photo originally taken in Iraq to illustrate a story about a purported massacre in Ghouta, Syria. Checkdesk could have walked BBC journalists through a checklist of verification techniques such as reverse image search that would have shown the image’s original source and ensured it wasn’t misused to show another highly emotive and contentious event.
 
What assumptions are you making in what you propose, and how will you test them?
 
Our proposal makes the following assumptions:
  1. That social and citizen media already are important sources for news publishers covering breaking news stories, and their importance will continue to grow with the widespread use of camera-enabled mobile phones and wireless networks.
  2. That as the above happens, there will also be an increased risk of unintentional misinformation and intentional disinformation spreading rapidly via social and traditional media (see Gupta et al.)
  3. That embedded checklists with 3rd party integrations will help balance the needs of newsrooms to provide rapid responses to ongoing news developments and media sent in by citizen journalists and anyone using social media with the important needs of newsrooms to maintain accuracy. These checklists will help embed the fact checking process within journalists’ regular process and facilitate rapid verification steps such as revealing EXIF data, checking the weather, and others.
 
How will you get your project in front of the necessary people or organizations?
 
Since 2011 Meedan have been building a network of partners across MENA and the English-speaking world, in both newsrooms and academia.  This includes the largest newspaper in Egypt (Al Masry Al Youm), the biggest news publisher in Palestine (Ma’an) and a leading newspaper in Lebanon (Annahar).  These publications have used an early version of Checkdesk in their reporting, as have 3 other MENA-based news organizations and one journalism school. Our early project work has also been presented at 4M conferences in Cairo and Marseille, at Stanford University’s Right to Information Conference, at the 4th Arab Bloggers meeting in Amman, and at the IPI News Innovation Conference in London. Additionally, we have discussed and demonstrated early versions of Checkdesk with the BBC, Al Jazeera, the Guardian, Internews and Austrian public broadcaster ORF.

In contributing to and translating the Verification Handbook we have established a strong relationship with Poynters and the European Center for Journalism. All of the above have validated the need for better tools and training resources for digital journalists working with citizen and social sources, and we are confident that — with Knight support to develop the toolkit — several of the aforementioned would use and advocate for the use of Checkdesk. One opportunity that is of particular interest is working with journalism schools. As our experience working with Birmingham City University has demonstrated, the next generation of journalists will need strong digital media literacy to succeed in the workplace. Checkdesk can provide a safe and instructive context for training verification techniques.
 
What are the obstacles to implementing your idea, and how will you address them?
 
Journalists and editors in today’s newsrooms are strapped for time and money, and in the present culture of instant reporting, accuracy has taken a back seat to speed. The major obstacle we foresee is striking a balance between newsrooms’ increased financial pressure to push out news quickly with the very raison d’etre of journalism, namely, to “ be honest, fair and courageous in gathering, reporting and interpreting information.” In addition to the Verification Handbook, we take lessons from Dr. Atul Gawande’s Checklist Manifesto, which has argued for the use of checklists in (amongst other fields) the medical profession, which faces a similar demand for balance between speed of services and necessity for accuracy and consistency.

By embedding checklists for important verification actions, we aim to improve accuracy while also helping journalists meet deadlines in a timely fashion.  Importantly, our goal is to make these actions easy to carry out. The aim of the checklists is to encourage integrations that facilitate rapid execution of verification steps.  For instance, when determining the veracity of a photo posted on a certain date, journalists can use our checklist to quickly check the weather for that day at that location. To the greatest extent possible, we aim to reduce the number of manual actions a journalist must take to verify a report.

Another important obstacle will be technical implementation. We anticipate two important technical challenges: security and third party integrations. With existing implementations of Checkdesk, we have faced problems w DDOS attacks and spam bots.  Historically, we have addressed those with known security filters, such as Captcha, HoneyPot and Cloudflare.  We also anticipate exploring other security programs like Google's Project Shield, but we will need to address each partner’s needs on a case by case basis.  We will also need to explore third party integrations, which will vary depending on the different types of source content. Some integrations will allow for API level integrations, others may require more creative technical solutions, and others may require walking journalists through more manual steps.
 
How much do you think your project will cost, and what are the major expenses?
 
We anticipate the project will cost approximately $350,000 to develop and implement. The major expenses we anticipate around the software include design, development, quality assurance testing, usability testing, product management and security.  In the area of partnerships, we expect costs related to outreach to potential journalistic outlets, and we hope to fund the work of a third-party academic institution in evaluating the impact and efficacy of the software implementation.
 
How will you develop a set of tools that are useful in short time cycle situations, such as breaking news?
 
Checkdesk is designed to address one of the crucial challenges of breaking news in the digital age: the balance between speed and accuracy. For a major breaking news story happening almost anywhere in the world, it is now common to see content emerging almost instantaneously online, posted by onlookers and citizen journalists. Consider Hurricane Sandy, the ongoing conflict in Syria, and the Boston Marathon bombing. In each of these cases, journalists are faced with a barrage of content, some of which may be useful but much of which may be false - indeed, Gupta et al analyzed that 29% of the most viral content spread on Twitter during the Boston crisis was rumor and fake content.

Checkdesk’s checklist provides ready-to-go steps that a journalist can take to verify some of that content and powerful integrations to perform and record those steps quickly. The software provides tools that can quickly help analyze social content and give that content a clearly-marked and evolving verification status. Readers and journalists alike would see content from social media that is clearly marked as “Verification in progress” accompanied by the various steps that newsrooms have taken to verify the content, and which updates dynamically to show the latest information available.
In ONE sentence, tell us about your project to strengthen the Internet for free expression and innovation.
Verifying breaking news information and debunking rumors in emergencies and conflicts saves lives, and with Checkdesk’s Verification Toolkit we aim to provide both veteran journalists and budding citizen journalists with the tools they need to effectively verify user generated content in critical situations.
Who will benefit from what you propose? What have you observed that makes you think that?
Checkdesk benefits: 1) Digital and citizen journalists >> By providing enhanced, open source tools Checkdesk will better equip journalists and citizen journalists to verify digital media quickly and efficiently. This is a major challenge for many newsrooms and digital journalists, as is evidenced by the chaotic reporting witnessed during times of crisis or breaking news stories, and the all too common social media hoaxes that lead to confusion, damaged reputations, or heightened tensions in emergencies or conflicts; and 2) Media audiences and emergency responders >> The public, and emergency responders, benefit from improved accuracy and reliability in media coverage, which is especially valuable in breaking news and emergency situations (Hurricane Sandy, Boston marathon bombing, Japan earthquake).
What progress have you made so far?
We have equipped six leading news publishers and grassroots media collectives in five countries across the Middle East with a proof-of-concept for Checkdesk, funded by Sida, the Arab Partnerships Fund, and the International Press Institute. Built in Drupal, the platform as it stands enables you to gather and embed rich social media content into a virtual ‘desk’, cross-reference it for consistency, and annotate that content with a verification stamp. For the next iteration, we have a product team in place furnished with a compelling product roadmap that will make the app more open access and more mobile, to support a broader range of journalists.
What would be a successful outcome for your idea or project?
Success means: 1) Journalists at media companies around the world using Checkdesk to verify links and debunk misinformation online for the benefit of readers; and 2) Developers building on Checkdesk to add their own tools and checklists.
Who is on your team, and what are their relevant experiences or skills?
Meedan’s team consists of developers, designers, journalists, researchers and project managers who are all passionate about supporting collaboration between newsroom and technology innovators. Since 2011 we have worked with many of the Middle East’s leading newsrooms to implement the Checkdesk project, which in 2012 won the International Press Institute’s News Innovation Challenge. In 2014 we contributed a case study to the Verification Handbook, and are presently translating the book into Arabic as part of our ongoing support for providing high quality training resources for journalists and citizen journalists in the Middle East. Beyond the Meedan team, we also have a dedicated network of partner organizations that are motivated and committed to defining and implementing rigorous and ethical online journalism.
Location
San Francisco, CA, USA // Cairo, Egypt

Comments

Join the conversation and post a comment.

Tom Trewinnard

May 14, 2014, 12:59PM
Amazing insights here into approaches to UGC verification taken in some newsrooms:

"On the issue of verification though, Claire said that she was “shocked” when she was doing her interviews about how lax the approach was. She was still hearing from grizzled veterans that “gut instinct” was important when verifying information, and they wouldn’t be using even the simplest techniques like reverse image search. Fergus said the tools to verify UGC shouldn’t slow down a story if they are properly integrated into your workflows. As he pointed out: “The quickest verification on a breaking news story happens when someone still has the device in their hand.”

Claire did report one anonymous interview subject saying that people “groan” in the office when the topic of verification comes up. It seems so odd – you wouldn’t dream of not verifying other types of journalistic information. One of the problems she identified is that most news managers these days won’t have been on a breaking news desk for five years, as they’ve moved ranks up the organisation. Which means they’ve never experienced running a breaking news desk in the social media era."

http://martinbelam.com/2014/the-use-and-verification-of-ugc-in-breaking-news-panel-session-at-ijf14/

Josh Meyer

April 08, 2014, 11:24AM
Thanks Tom (and team) for submitting such a fascinating proposal, and for engaging those in the comments section. (And it's great to see all of these good questions and comments). It seems to be a very powerful and timely toolkit, especially for journalists and others seeking to verify information on the fly. Can you say a bit more about how you hope to use the experience gained from equipping the six Middle East news publishers and grassroots media collectives with a proof-of-concept for Checkdesk when building out this new product team and product roadmap? E.g. how will it help make the app more open access and more mobile, allowing it to support a broader range of journalists? Many thanks!

Tom Trewinnard

April 08, 2014, 18:15PM
Hi Josh, thanks for the great questions! Our partners in the MENA have been really great for a number of reasons: They've discussed at length with us the challenges they face with regard to verification and the risks of misinformation shared online (i.e. they've validated the need for better tools to deal with this), and they've given tons of extremely useful feedback on what we've developed thus far. This feedback, along with great new resources such as the Verification Handbook, have directly influenced the roadmap, and your question touches on a couple of the key issues: opening access and mobile.

Re: Open access - right now the Checkdesk code is up on GitHub (https://github.com/meedan/meedan-checkdesk) for any newsroom hackers that want to take a look: what we envision with this proposal is a toolkit that's more quickly and easily accessible to any journalist that wants to investigate a URL (A YouTube video, tweet, photo, news report etc.): we want any journalist sitting anywhere in the world to be able to drop a link into Checkdesk and have a smart checklist with integrated tools at their fingertips. This is embeddable on any CMS that a journalist or newsroom might be using - so we're trying to allow journalists to add this verification step to their existing reporting workflows. One of the key takeaways from our work in MENA is that we need to build tools that are lightweight and optimized enough to function even in areas without access to high-speed broadband. This also gets us to your second point:

Re: Mobile - Right now it's pretty difficult to do verification work on a mobile device: there's lots of copying and pasting, lots of switching between browser tabs, and lots of thumb gymnastics as you document what steps you've taken and where it got you. Our proposal here makes verification more mobile friendly by breaking it down into small steps (or micro tasks) that can be carried out by making the integration aspect of the checklists do the heavy lifting. This also ties into research conducted with our MENA partners - mobile usage in the region (and around the world) is growing rapidly and the next generation of media tools will have to account for that reality.

Hope this answers your questions!

Josh Meyer

April 10, 2014, 15:28PM
Excellent. Many thanks Tom

Rebekah Monson

April 05, 2014, 13:29PM
Great proposal, Tom. Thanks for providing additional info in the comments too.

Tom Trewinnard

April 06, 2014, 13:15PM
Thanks Rebekah!

Kyle Davis

April 02, 2014, 17:51PM
But, does Checkdesk actually work?

Would "Checkdesk" be able to identify if any submissions of this Knight News Challenge are related to / or masking for larger organizations versus unique individual ideas and submissions?

Tom Trewinnard

April 02, 2014, 18:54PM
Hi Kyle!

Interesting question - Checkdesk alone couldn't identify such a thing, but would be usfeul for someone looking to dig deeper. Checkdesk is more a tool that would help someone with their investigation by:

- providing a checklist of steps that may be helpful (eg. Reverse image search on the photos used in the submission)
- integrations with 3rd party services that make carrying out those steps very simple
- and a log of what steps have been carried out and what questions remain about a link.

Once a user has gone through these steps then they're able to assign a status to a link ("Verified", "False" etc.) This would work for a submission here, but our main use case ideas are around verifying a YouTube video, a photo, or a tweet etc.

Regarding organizations vs individuals - I think the emphasis of the News Challenge is on ideas before all else. I'm sure there are lots of great and worthy ideas posted by both. As I hope is very clear from the entry and my profile - I work for tech non-profit Meedan and this submission (worked on by me and several of my colleagues) is a bold vision for new technological development of Checkdesk.

Kyle Davis

April 05, 2014, 13:35PM
Thanks for the insight... Tom!

Tom Trewinnard

March 28, 2014, 13:47PM
Here's a really interesting collection of mini-interviews with journalists who got their break in the industry as fact checkers: http://www.poynter.org/latest-news/mediawire/245134/journalists-remember-their-first-fact-checking-jobs/

"Fact-checking as an intern at Philadelphia Weekly really drove home for me that readers expect accuracy, and it doesn’t matter whether it’s the spelling of a band’s name or a description of a neurological disorder — you’d better have it right."

Tom Trewinnard

March 20, 2014, 13:15PM
Cross-posting some interesting thoughts from the great people who contribute to Storyful's Open Newsroom: https://plus.google.com/u/0/108344897173120412718/posts/1KgFrktaz8a

Josh Stearns

March 19, 2014, 22:20PM
I featured CheckDesk in my collection of verification tools at http://VerificationJunkie.com a few months ago because it was one of the most well thought out and holistic verification tools I had seen created for newsrooms. I appreciated the combination of tech tools with on the ground people power.

Tom Trewinnard

March 20, 2014, 12:33PM
Cheers Josh, we really appreciate your input!

An Xiao Mina

March 24, 2014, 01:17AM
Thanks, Josh, for featuring Checkdesk! Hopefully we can take it to the next level, with greater integration with journalists' existing workflows and many of the learnings we've gathered from our work with the Verification Handbook and the methods you've highlighted at Verification Junkie.

Shane Greenup

March 19, 2014, 20:54PM
I spend so much of my time steeped in misinformation and faulty reporting, that I love the idea of journalists always having easy access to a powerful toolkit which can help them fact check things before they publish.

If only all publications out there held truth and accuracy to the high ideals they are meant to!

Tom Trewinnard

March 20, 2014, 12:33PM
Thanks Shane! As I mentioned above, we're really keen to integrate rbutr into the toolkit.

Shane Greenup

March 20, 2014, 18:57PM
And we're happy to help you out with anything you need to make rbutr more useful to your needs! This is pretty much exactly what I hoped rbutr would be used for. Helping content creators avoid promoting misinformation (and also, hopefully, using it to counter misinformation when they write their own responses!)

Enrique Piracés

March 19, 2014, 15:25PM
Agreed with the others, this is very interesting. Would love to see how this can be integrated with other tools . Hint, hint.

Tom Trewinnard

March 19, 2014, 15:42PM
Thanks Enrique! We're really interested in hearing about what tools journalists, citizen journalists and activists are using for verification and debunking . A key part of our idea is to create an open framework for newsroom hackers and devs to integrate all kinds of tools that might be useful in the verification and reporting process. We're checking out secureApp Generator!

Enrique Piracés

March 20, 2014, 12:45PM
Excellent. Let's talk soon. I'm also pointing other people interested in verification to this entry so the can chime in as well.

Eliot Higgins

March 19, 2014, 15:10PM
Really great idea, something sorely lacking at the moment.

Tom Trewinnard

March 19, 2014, 15:27PM
Thanks Eliot! What integrations would be useful in your verification work?

Britain Eakin

March 19, 2014, 10:23AM
This is really great. You must tell me more about this Tom!

Tom Trewinnard

March 19, 2014, 15:16PM
Thanks Britain! We're keen to learn from you and your colleagues' experience of digital journalism and what tools are lacking - especially with regard to verification.
Login
Close
Login to News Challenge
 
or