Introduction:
The increasing use of generative AI has ignited curiosity among companies — both small and large — about its potential to streamline internal and external communications, enhance productivity and more.
This excitement over generative AI has been documented by Gartner which in 2022 said generative AI was at the "peak of inflated expectations" in the hype cycle and predicted its plateau to be in the next two to five years. For newsrooms, it seems as though the use of AI tools like OpenAI's ChatGPT has just begun, as found in a survey by the World Association of News Publishers (WAN-IFRA, 2023).
The study found that half of the 101 newsroom staff members surveyed have worked with AI tools to carry out tasks including text creation, topic ideation, translation, text correction and content creation (2023). With newsrooms facing staffing shortages and increasing concerns about the reliability of news, the introduction of these tools is bound to change the way reporters perform their jobs, forcing this industry to find ways to leverage AI technology (Pew Research Center, 2021, & Northwestern Now, 2022).
Hypothesis:
In this report, researchers aimed to explore the potential of a generative AI-powered chatbot to streamline communication and simplify the editing process in a local newsroom in New Jersey. The researchers developed an MVP with the objective of assisting reporters by answering questions and providing guidance before their stories go to a human editor for review.
The hypothesis was that integrating a ChatGPT-powered chatbot with a specific newsroom knowledge base could guide journalists in their reporting and provide them with newsroom guidelines, ultimately making the editor's job more efficient and freeing up time for other newsroom tasks.
Background:
ChatGPT is a chatbot developed by OpenAI designed to provide human-like responses. However, it has certain limitations because its performance relies heavily on pre-trained knowledge. Users have reported that ChatGPT sometimes "hallucinates data," struggles to assess source bias and lacks the ability to attribute information (Dhiman, 2023). To ensure journalistic integrity,
these limitations need to be considered when implementing generative AI in newsroom operations.
Generative AI has enabled businesses to create their own chatbots, which has garnered increasing interest since 2016 (He & Xin, 2021). Chatbots are optimized to provide users with quick and convenient 24/7 conversational support. Modern chatbots can be divided into two main categories: rule-based and AI-based (Adamopoulou & Moussiades, 2020). Rule-based chatbots typically feature simpler interfaces and rely on "if-else statements" to handle question-answer pairs. The research in this report focuses on AI-based bots. This approach allows the chatbot to answer queries that are not explicitly defined, offering more flexibility than rule-based bots. The report will further explain the methods employed, including approaches that enable summarization, keyword extraction and answer queries by searching through PDFs.
Problems and Audience:
In the newsroom being studied in this test — a New Jersey-based local newsroom — staff cuts over the past decades have resulted in a shortage of editors. As a reporter on the breaking and local news team at this newsroom, I’m personally involved in this study. Our team consists of over 50 reporters who generate content daily, but we only have one or two editors on shift at any given time. Inspired by the way companies utilize chatbots to address FAQs I wondered if a similar application could be developed specifically for newsrooms. But before proceeding, I needed to understand the difficulties reporters in my newsroom face when accessing information and determine the type of information they often seek. This is what I discovered:
In the preliminary research, I conducted a survey with 22 breaking and local news reporters who have varying levels of experience ranging from less than one year to over 10 years.
Demographics
I asked them three questions:
1.) On a scale of 1 to 10, how likely are you to use an AI-powered chatbot daily to help answer questions that may arise while you're reporting?
2.) What information would you like the chatbot to be trained on?
3.) Is it difficult to find this information or obtain answers to these questions currently?
The average response to the first question was a 6 out of 10, indicating that there may be room for an internally developed, knowledge base-trained chatbot in our local newsroom. This number also suggests that many reporters may be skeptical about the use of AI.
The last two questions were open-ended and the results showed that reporters often struggle to find information for newsroom-specific questions.
Reporters highlighted three main categories of information they would like to see included in a chatbot: AP style guidelines, instructions for filing an Open Public Records Act and contact information for sources that are frequently reached out to, such as county prosecutor's offices, along with headline writing tips. Currently, this information is dispersed across separate PDF documents within the newsroom, requiring reporters to locate, retrieve and scan through them to find answers to their queries. Or, they can ask an editor for this information, which, in theory, can slow down newsroom operations and be difficult when you have a limited number of editors. Searching on Google is not a viable option as it lacks knowledge of our newsroom-specific information.
Based on the information gathered, the target audience for this research is reporters across local newsrooms who have limited access to editors and need a centralized platform where they can seek answers to FAQs. Although the information is not missing from our newsroom, it is not stored in a manner that is easily accessible and suited for the fast-paced, variable work schedules of reporters. This is where a generative AI chatbot can come in.
Initial Plan:
When developing the "Newsroom Editor Chatbot" to test the hypothesis, there were several essential attributes I aimed to incorporate. Here is the initial vision:
Integration with generative AI and a specific knowledge base:
To create a chatbot capable of answering a wide range of reporter queries, it needed to be generative-based. To address newsroom-specific questions, the chatbot had to be integrated with the knowledge and documents that journalists in the researched newsrooms frequently sought.
User-friendly interface:
As depicted in the product interface sketch, the bot had to be accessible for mobile devices, laptops, and maybe through Slack. Slack is a commonly used communication tool in my newsroom. Integrating the chatbot into Slack would likely increase user engagement.
Response generation:
For effectiveness, the chatbot would need specific instructions that tell it to first consult the provided knowledge base when answering user queries. If no relevant information is found, it would then turn to ChatGPT for a response. This approach addresses limitations of ChatGPT, such as the lack of training on internal data and giving incorrect answers. As a result, the chatbot would be capable of answering general questions like "Is this numeral written correctly according to AP Style?" as well as specific queries such as "What are the newsroom guidelines for reporting on sexual assault?”
Memory and conversational abilities:
In addition, I envisioned the chatbot having the ability to store information, enabling it to remember user queries and provide follow-up answers. Furthermore, to avoid repetitive responses and create a more human-like interaction the chatbot should possess conversational capabilities.
The Journey Pt. 1: Methods
After hours of watching YouTube tutorials and asking ChatGPT for programming advice, I discovered two main ways that people have used to develop a generative AI-powered chatbot with a specific knowledge base. The first method involved coding in Python.
Python experience:
First, I attempted to code a ChatGPT-powered chatbot using Flowise, a framework that enables the development of language model-driven applications. With the help of a Github repository I found, I successfully built a chatbot that encompassed all the desired components and functioned like how I wanted. I was able to run it on a local server. But, I encountered a challenge: deploying the chatbot for others to try out. Because I couldn’t figure out how to deploy the bot, I decided to explore free software options with user-friendly interfaces that didn't require coding. This search led me to discover Botpress.
The code:
Development environment:
The problem:
Botpress experience:
Botpress describes itself as software that allows users to “build ChatGPT chatbots surprisingly fast,”— and that’s exactly what I was able to do. I uploaded documents into the application to serve as the bot’s primary knowledge base. These documents included PDFs of my newsroom's AP style guide, reporting guides, and more. Adding the knowledge base was as simple as uploading documents into the software and telling the bot to pull information from those documents before asking the generative AI to answer a query. Botpress’s underlying framework is ChatGPT, enabling the creation of a generative chatbot.
Botpress development environment:
Knowledge base:
Some adjustments had to be made to my original idea while I was exploring Botpress. Botpress offered Slack integration, but privacy concerns prevented me from integrating it into my newsroom's Slack channel. Because I was unable to get permission from my mangers to do this, instead, I decided to embed the chatbot on my website and distribute it that way. Although the chatbot was conversational, the software didn't support the storage of past responses and information.
The Journey Pt. 2: Privacy and Concerns of the Unknown
With my final chatbot ready, uploaded to my website and awaiting feedback from the newsroom, concerns quickly arose among my editors and managers. They expressed worries about the capabilities of ChatGPT and privacy.
While I believed that I was providing Botpress with non-sensitive information, my managers pointed out that our newsroom-specific style guide, ethical standards etc. were not meant to be shared outside the newsroom. They also raised concerns about whether the information provided to the generative AI software would be stored. These concerns had not crossed my mind until our discussion, and with the guidance of my managers, I once again scaled back the chatbot's capabilities. Instead of containing newsroom-specific information, the new bot would only provide general information relevant to any reporter working in New Jersey.
As one of my managers put it, "You made it too real!"
The original chatbot and its directions prior to meeting with my managers:
Final Product:
After undergoing these two pivots, the final version of my chatbot consisted of a reduced knowledge base. I removed our newsroom-specific documents and included only two documents: general AP style guidelines and instructions on filing an Open Public Records request in New Jersey. These were two areas that reporters frequently sought guidance on, according to the preliminary analysis. Because this knowledge base is less specific, in theory ChatGPT should still be able to provide answers without it; but for the purpose of the field test and to address privacy concerns expressed by my managers, these were the only documents I decided to include.
Updated knowledge base:
Testing the bot:
Results: I managed to gather feedback again from the same 22 reporters. Following discussions with my managers, who raised concerns about privacy I included questions aimed at gauging concerns about AI. The full Google Forms survey and its responses can be found in the appendix, but I will highlight two key results that are essential for addressing my hypothesis.
Overall experience using the chatbot:
- Very Satisfied: 8
- Satisfied: 10
- Neutral: 3
- Dissatisfied: 1
These findings suggest that integrating a ChatGPT-powered chatbot with a specific newsroom knowledge base can guide journalists in their reporting and provide information on newsroom guidelines.
Extent to which the chatbot could enhance workflow and reduce style mistakes:
- To a great extent: 12
- Somewhat: 7
- Not at all: 3
Additionally, the survey showed diverse attitudes toward AI in newsrooms. Seven respondents reported a significant decrease in their concerns or fears and four expressed an increased level of concern or fear after using the bot. These questions were included to gauge journalists' attitudes and identify potential areas for discussion before implementing AI tools.
Recommendations and Conclusion:
The survey results confirm my hypothesis about the potential value of a generative AI-powered chatbot as a resource for reporters to enhance their workflow and provide newsroom-specific guidance. However, certain limitations were identified due to the constraints of ChatGPT's boundaries and privacy concerns. Future research should focus not on the effectiveness of using generative AI chatbots in newsrooms, but rather on safeguarding information and ensuring privacy when utilizing such applications.
Following my conversations with my managers about the chatbot's prior knowledge base, I now understand that there is still much unknown about AI in newsrooms. It is crucial for newsrooms to explore the ethical implications before implementation. This realization and work led me to be invited to join the AI research team at my newsroom, where topics such as data privacy and security, developing an AI toolkit for journalism, ethics and accuracy are being investigated.
This experience serves as a personal "French Fry Moment" as I recognize the value of making mistakes and thoroughly assessing scenarios before delving into new technologies that can have both benefits and detriments for myself and others in my field. It is through learning from these mistakes that we can ensure the responsible and impactful integration of AI in newsrooms. References
Adamopoulou, E., & Moussiades, L. (2020). An Overview of Chatbot Technology. Artificial Intelligence Applications and Innovations, 584, 373–383. doi: 10.1007/978-3-030-49186-4_31
Dhiman, B. (2023). Does Artificial Intelligence help Journalists: A Boon or Bane? [Preprint]. TechRxiv. https://doi.org/10.36227/techrxiv.22649620.v1
He, J., & Xin, C. (2021). Developing an AI-Powered Chatbot to Support the Administration of Middle and High School Cybersecurity Camps. Journal of Cybersecurity Education, Research and Practice, 2021(1), Article 6.
INMA. (2023, June). News organizations embrace codes of conduct, use of AI. Retrieved from https://www.inma.org/blogs/newsroom-initiative/post.cfm/news-organisations-embrace-codes-of-conduct-use-of-ai
Newspapers close, decline in local journalism. (2022, June 30). Northwestern Now. Retrieved from https://news.northwestern.edu/stories/2022/06/newspapers-close-decline-in-local-journalism/
Pew Research Center. (2021, July 13). U.S. newsroom employment has fallen 26% since 2008. Retrieved from https://www.pewresearch.org/short-reads/2021/07/13/u-s-newsroom-employment-has-fallen-26-since-2008/
WAN-IFRA. (2023, May). New GENAI survey. Retrieved from https://wan-ifra.org/2023/05/new-genai-survey/
Wired. (2023). About Generative AI Policy. Retrieved from https://www.wired.com/about/generative-ai-policy/
Appendix: Full survey results (Google Form)
1. Role in the newsroom:
- All 22 respondents: Reporters
2. How long have you been working in news?
- 2 respondents: Less than a year
- 6 respondents: 1-3 years
- 5 respondents: 4-6 years
- 4 respondents: 7-10 years
- 5 respondents: More than 10 years
3. Overall experience using the chatbot:
- Very Satisfied: 8
- Satisfied: 10
- Neutral: 3
- Dissatisfied: 1
Did the chatbot effectively answer your newsroom specific queries?
- Yes: 12
- No: 3
-Partially: 7
5. Frequency of issues or misunderstandings while interacting with the chatbot:
- Often: 4
- Occasionally: 10
- Rarely: 6
- Did not encounter: 2
6. Accuracy of the chatbot in understanding and responding to general questions:
- Accurate: 14
- Inaccurate: 5
- Did not encounter: 3
7. Extent to which the chatbot could enhance workflow and reduce style mistakes:
- To a great extent: 12
- Somewhat: 7
- Not at all: 3
8. Likelihood of using the chatbot as a resource in daily workflow if accessible on Slack:
- Very Likely: 9
- Likely: 8
- Neutral: 3
- Unlikely: 2
9. On a scale of 1 to 5, please rate the extent to which interacting with the chatbot influenced your level of concern or fear about the use of AI in newsrooms
- Decreased concern or fear significantly: 7
- Neutral: 8
- Increased concern or fear significantly: 4
10. On a scale of 1 to 5, please rate the extent to which interacting with the chatbot influenced your belief in the potential benefits of using AI in newsrooms?
- Strongly believe AI would be beneficial: 10
- Somewhat believe: 9
- Strongly believe AI would not be beneficial: 1
11. Limitations or gaps in the knowledge base of the chatbot:
- Several people mentioned limitations in providing specific newsroom-related information and requested more detailed guidance in certain areas like AP style specific rules.
Additional Feedback:
- Some respondents suggested expanding the knowledge base to include ethical guidelines.
- Others mentioned improving conversational abilities so the chatbot can build off of prior responses and operate more human-like
コメント