Generative AI on JSTOR: What we’re learning from early usage, real-world applications, and user feedback
Based on a presentation titled “Navigating Generative AI: Early Findings and Implications for Research, Teaching, and Learning” given at the Coalition for Networked Information (CNI) on March 26, 2024. The presentation is available in full here.
Advances in technology have always been an accelerator of ITHAKA’s mission to improve access to knowledge and education, and the advent of generative AI is no different. With the release of JSTOR’s AI-powered research tool in beta, we are delivering on our promise to collaborate with our community to use generative AI responsibly and transparently in order to empower researchers and learners to expand and deepen their work.
As of this writing, 22,234 active users at 5,437 institutions in 149 countries have used the tool since its launch in August 2023. We are actively learning from usage, real-world applications, and user feedback in a limited beta environment to build out capabilities thoughtfully, intentionally, incrementally, and most importantly–based on evidence. This blog post delves into the rationale, construction, and current functionality of JSTOR’s beta generative AI tool, as well as the key themes and insights that have emerged from our analysis of its usage.
As of June 18, 2024 (click to enlarge)
The development of JSTOR’s generative AI tool
Our approach
The massive user adoption of ChatGPT in late 2022 signaled a significant shift on the horizon in user behavior and expectations. People were drawn to the idea of conversational interaction made possible by generative AI. We recognized the transformative potential of this shift and wanted to understand its impact on the scholarly community, and to explore how it might be used responsibly and thoughtfully to advance our mission of making knowledge more accessible. The world was changing quickly, and we knew we needed to act fast to help our community navigate this transformation.
To start, in spring 2023, Ithaka S+R, a strategy and research consultancy from ITHAKA and peer service to JSTOR, announced a multi-year collaborative research project focused on helping colleges and universities understand the impact of these technologies and chart a productive path in the use of generative AI in higher education. Around the same time, on JSTOR, we kicked off an ambitious project to bring an AI-powered research tool to the platform in beta by the fall of 2023.
The rationale behind developing JSTOR’s generative AI tool is rooted in a commitment to democratizing research, enhancing educational experiences, and supporting scholarly endeavors across various academic levels. The guiding principles of this initiative include using generative AI in ways that uphold JSTOR’s core values of equitable access and scholarly integrity; supporting education by enhancing user experience without replacing the educational process; helping the academic community make the best use of these capabilities; and sharing what we learn throughout. We opted for a limited beta approach to maximize learning and iterate in a low-risk environment.
Building the tool
The development of JSTOR’s generative AI research tool was a rapid and intensive process aimed at quickly delivering a functional beta to users to kickstart the learning process. Driven by a team of engineers, data scientists, designers, and researchers, the initial phase involved meticulous planning to identify the core user problems that this technology could uniquely solve for JSTOR’s users. Following this, the team embarked on a rapid development cycle, quickly building the tool to ensure a swift beta launch. This agile approach was complemented by a user-centric design methodology, where feedback from users was incorporated even before the beta release to refine the tool’s functionalities and ensure it met user needs effectively.
The technological backbone of the tool relies on advanced AI models, primarily a combination of OpenAI’s GPT-3.5 and smaller open-source models. At the time that we were building the first version in the summer of 2023, these models were chosen for their robustness and ability to handle various text analysis tasks. Those choices allowed us to move quickly while actively exploring and leaving the door open to alternatives. This area is very fast moving, and as new LLMs are created, we have the ability to substitute different models based on the mix of variables, such as accuracy, cost, and capacity. The tool employs sophisticated vectorization techniques, which transform questions and text into vectors to enable precise semantic searches within documents. Custom prompts are another crucial aspect, carefully crafted to ensure that the AI generates accurate and relevant responses, enhancing the overall user experience.
Current functionality
In its current form, JSTOR’s generative AI tool is designed to provide users with a range of powerful functionalities for interacting with the full text of a single item, including:
- Custom summaries: The tool can provide a summary directly related to the user’s search query, as well as a general summary of the document, allowing users to quickly grasp the content and relevance of academic texts.
- Recommended topics: The tool can recommend related topics based on the document’s content, enabling users to explore additional research paths and deepen their understanding of the subject matter.
- Related content: The tool can display documents that are conceptually similar to the current text being viewed, helping users to find additional relevant resources quickly and efficiently.
- Question and answer (Q&A): Users can ask specific questions about the document, and the tool generates answers using the document’s content to ensure accuracy and relevance. This feature is particularly useful for users seeking detailed information on specific aspects of a document, including topics and concepts.
How the Q&A feature works (click to enlarge)
We encourage users to provide feedback on the performance of the tool, with options to rate the quality of responses and provide open commentary about their experience. This feedback helps us make continuous improvements.The tool also maintains transparency by linking back to the original text sections used to generate answers, ensuring users can verify the information provided (see screenshot below). We’ve also engaged subject matter experts from a range of academic disciplines to conduct in-depth, ongoing evaluations of the tool’s output to further ensure accuracy.
The tool maintains transparency by linking back to the original text sections used to generate answers, ensuring users can verify the information provided (click to enlarge)
Key learnings
Improved research efficiency
The tool has been praised for its ability to streamline the research process.
A faculty member noted, “I can do in a day what it used to take me four or five days… AI isn’t doing any of the intellectual work for me… It’s not thinking things for me; it’s showing me where to look.”
This highlights how the tool aids researchers in quickly finding relevant materials, allowing them to focus on deeper analysis and comprehension. Additionally, users are spending significantly more time on JSTOR with the AI tool, with session durations and page visits more than doubling compared to non-beta users. This heightened engagement indicates that the tool is making the research process more interactive and appealing.
Another user shared, “I find that I’m spending more time capturing and downloading the AI summaries than downloading PDFs… I will then download a few of the PDFs that seem appealing.”
This feedback underscores how the tool is changing the way users interact with academic content, making it more efficient and effective. Researchers can spend more time engaging deeply with the most relevant and salient texts to their inquiry, rather than wading through the sometimes overwhelming number of search results to identify them in the first place. The tool also keeps users from having to spend large amounts of time reading articles only to find that they are not relevant to their research. This is particularly valuable for the many articles in the humanities and social sciences that do not include abstracts.
Enhanced engagement and interaction
Distribution of tool usage from launch in August 2023 to the present, including Q&A (dark blue), custom summaries (red), related content (orange), and recommended topics (teal), as of June 18, 2024 (click to enlarge)
Evidence of enhanced engagement and interaction is readily apparent from the distribution of tool usage illustrated above. Of the four main features of the tool—summary, topics, related content, and Q&A—summary and Q&A are used more frequently, with the Q&A feature becoming the most highly used over time. This tells us that while users are engaging with all of the capabilities, they are primarily drawn to the conversational aspect the Q&A feature delivers. In this way, they can “talk to the document” and engage in a kind of “conversation” with the text. Moreover, they can trust and verify that the responses the tool provides are credible and accurate by clicking on the footnotes pointing back to the original sections of the text from which the answer was generated.
Because this initial evaluation takes place between users and a non-sentient research tool, there are no “dumb questions.” There’s also no limit to the number of questions users can ask, the outputs are easy to read and understand, and the responses are provided immediately. We believe that the transparency and traceability created by these guardrails relieve some of the uncertainty and doubt researchers may feel when beginning new research, thus freeing them up to take more creative risks as they explore innovative lines of inquiry while remaining grounded in the literature.
Accessibility and the democratization of research
The AI tool caters to a wide range of researchers, from novice students to experienced scholars. This versatility makes it a valuable resource across different academic disciplines and levels of expertise.
For novice researchers, the AI tool helps in finding and evaluating materials for relevance and understanding complex academic texts. This is particularly helpful for high school and undergraduate students who are still developing effective research skills and can benefit from the tool’s ability to make scholarly content more accessible. The tool prepares them to engage more deeply with the material, much like knowing the plot of a Shakespeare play before reading it enhances comprehension. Helping early career researchers find the “right” material faster and facilitating understanding about why it’s relevant and useful to them reduces frustration, wasted time, and energy that’s better spent exploring the text and developing and testing hypotheses against what they’ve found.
A student volunteered, “Honestly, time is everything… Wasted time and difficulty levels will go down significantly because of this. I mean, again, JSTOR boasts that we have 1,000s of articles, which is awesome; I love JSTOR. But sifting through those articles is kind of hellish.”
These early research experiences are particularly significant because they play a critical role in shaping student attitudes and perceptions of their own abilities and interests related to academic research. By reducing logistical friction while students are being exposed to research methodologies and academic writing—possibly for the first time—JSTOR’s generative AI research tool can lower the barrier to entry and allows students to learn and develop their skills unimpeded, iteratively, and unselfconsciously. In this way, generative AI is truly generative, not in and of itself, but by the nature of the interchange that takes place between the user and the tool, as well as between the user and their instructor and classmates as they collectively master research skills that will benefit them in their academic careers and beyond.
A librarian remarked, “It would make it easier for them [students] to use the database and get information out of it, lowers the scariness factor in the database, it can be frustrating when you’re searching, lowering the annoyance factor as well.”
Faculty, PhD students, and other more advanced researchers use the tool to gather evidence efficiently and explore complex conceptual relationships across disciplines and content types. It’s a generative tool with the potential to supplant bandwidth-sapping tasks so researchers can do the deep work of understanding the material, forming hypotheses, and building a point of view that ultimately expands the collective academic perspective and discourse on a given topic. This democratizes research by providing support that is often only available to those with dedicated research assistants. By providing essential research assistance, JSTOR’s generative AI tool has the potential to level the playing field for faculty, researchers, and students at institutions with limited resources. In turn, the democratization of research can foster a more inclusive academic environment.
Educational applications and implications for deeper learning
Instructors find the tool valuable for supporting both the execution of student research and the development of critical research skills.
Another librarian observed that “It makes research interactive, which is great… I think it would change how I teach, making it a little less introductory…”
Not only does the tool help instructors teach students how to find and evaluate research material, but it also allows students to self-serve, move beyond rudimentary barriers in the research process, and delve deeper into their arguments, interpretations, and evidence. In this way, the tool enables educators to provide more personalized learning experiences adapted to the unique needs, preferences, and interests of each student, an increasingly difficult task given their growing responsibilities and expectations.
More generally, the tool also supports the development of dynamic and responsive curricula, allowing instructors to curate reading lists, develop more relevant and engaging course materials and assignments, and perhaps even offer new perspectives on traditional subjects, sparking more captivating and impactful discussions in the classroom.
Future developments and enhancements
JSTOR’s generative AI tool is poised to transform academic research, teaching, and learning by providing innovative features that enhance efficiency, user engagement, accessibility, and even the impact of teaching and research as a whole.
So what’s next for generative AI on JSTOR? The tool’s potential for translation and adaptability for different reading levels means that more users worldwide can access and understand academic content. This not only promotes global knowledge-sharing, but also supports users with different learning needs and preferences. Additionally, the tool’s potential to generate summaries and other content in multiple formats, such as audio and video, could significantly enhance accessibility for users with disabilities or those with different learning styles and preferences. JSTOR plans to expand the tool’s capabilities to cover broader document scopes, such as entire books and other content types, which will further enhance its utility. We are also actively exploring the possibility of extending the tool into more areas of the platform and research workflow. Continuous user feedback and iterative development will ensure the tool evolves to meet the real-time and ever-changing needs of the academic community.
Have you tried JSTOR’s generative AI tool? Share your experiences and thoughts with us. How do you think AI can further enhance academic research, teaching, and learning? We look forward to continuing the conversation and sharing how your feedback is directly influencing the evolution of generative AI on JSTOR.
If you’re interested in exploring generative AI on JSTOR, you can join the waitlist for the beta here.
Related content
- Recording of original presentation
- Explore generative AI on JSTOR
- Approaching generative AI together
- Empowering research with generative AI on JSTOR
- From Berkeley to beta: How students helped bring generative AI to JSTOR
- Empowering educators, engaging students: How one teacher used JSTOR’s GenAI research tool to boost student engagement, teach key skills, and streamline workflows
About the authors
