Better Search, Better Research
Improvements to the JSTOR search are on their way. Since March 2013, we’ve been testing a new Beta Search, which promises improved search results and faster response times. Based on positive feedback from the community, we’re planning to introduce the first set of changes to the main platform beginning in September. With these changes, we’ll improve the relevance of search results with minimal modifications to JSTOR’s interface at this time.
Changes being released:
- Improved search results relevance
- Faster search performance
- Small interface modifications to the search results page, primarily changing the appearance (but not functionality) of sorting options
- Removal of the little-used features "Recent Searches" and RSS feeds for search results, for redevelopment
These initial search changes will be rolled out over a number of days. We expect no interruption for people using the site. Beta Search will continue to be available for testing and review while we implement these improvements to the main interface. We continue to welcome feedback on the Beta interface and features, which we'll incorporate into ongoing improvements to the search forms.
About the Beta Search
Beta Search is a completely new JSTOR search, providing an easier-to-use interface and a new search engine. A key goal of the new Beta Search is to provide better results for simple queries. Beta Search includes:
- A new interface, letting you narrow down or broaden your searches more easily
- Improved relevance rankings, giving you results that more closely match your search terms
- New features, including auto-suggested search terms and spell-checking
- Enhanced search results view, previewing the article or book details directly from the search results list
The new Beta Search is presented in parallel with the existing search options. You can try it via the link on www.jstor.org or from a standard search results page. It's important to note that the Beta Search is not a finished product; you may encounter bugs and see changes to the interface and features during the beta period.
Beta Search Overview
We are optimizing the Beta Search to address several areas of improvement.
Better Results for “Known-Item” Searches
The Beta Search includes both better matching and ranking of results in known item searches, when you’re looking for a specific article and may type in a few words from the title, author name, and journal. The new search compares the search keywords against a range of bibliographic fields looking for the best combination of matches. The fields evaluated include the document title, author names (with greater weight on surnames), year of publication, start page, and journal title.
The new search engine builds a better query from a searcher’s input by simultaneously searching across multiple fields and assigning more weight to matches that occur in more significant fields (such as title, abstract, and author names). It also prioritizes matches that occur in close proximity when multiple terms are provided. The key factors are the relative frequency of the matched terms in a specific document versus all content on JSTOR, the weight of the fields in which a match occurred, and the relative proximity of matched terms in a multi-term search. Other factors considered include the content type (for instance, books and research articles are weighted slightly higher than other types) and the year of publication.
Easier Evaluation of Search Results
A new preview “flyout” is available for each item returned in a list of search results. The preview is activated by rolling over the item in the list and clicking on the highlighted region or arrow symbols. A preview is then displayed in the right-most column that contains citation information, access links, an abstract (if one is available), extracted key terms, and highlighted search snippets. The goal is to provide information to help evaluate the relevance of the item without having to click through to each individual article.
New Ways to Explore Topics
A major development included with the Beta Search is the ability to explore search results using topics assigned at the article level. Unlike standard searching on JSTOR where searches can be focused within disciplines assigned at the journal level, the Beta Search uses sophisticated text analysis techniques to automatically assign one or more topics to an article. The goal is to help searchers find relevant content that may be outside of their main disciplinary area.
For example, if an article from a journal in the Mathematics discipline contained text similar to that found in an article from the Education discipline, the article would be tagged with a topic associated with each of these disciplines. In addition, each topic is assigned a relative weight indicating how strongly the article reflected core content in the topic. A document is generally associated with two or three topics, sometimes as many as five. The relative weights assigned to each topic influence the item’s position in search results when one or more values are selected from the topic facet.
Smarter Search Engine
New in the Beta Search are convenience features such as auto-completion and search suggest (spell-checking). These capabilities are based on prior searches run on JSTOR by all users. The new search engine is also more forgiving of searches that may not be perfectly constructed (missing parentheses, quotation marks, etc.) and in many cases can accurately interpret and return relevant results.
A Note on the Technical Development
The topic models were produced using the Latent Dirichlet Allocation (LDA) method developed by David M. Blei, Andrew Y. Ng, and Michael I. Jordan (2003). Our specific implementation involves an adaptation of the core LDA method called Labeled LDA, wherein we start with a predefined set of topics (based on the JSTOR discipline taxonomy) rather than letting the topics emerge naturally from the algorithm. Our implementation of the Labeled LDA method was developed by Sean Gerrish during an internship with JSTOR. The Labeled LDA implementation models are built using core content from each JSTOR discipline, which is then used to classify all content in the corpus based on its relative similarity to the topic models.
Frequently Asked Questions
Why is it called "Beta Search"?
This is a beta version of a completely new JSTOR search application. Searchers may run into occasional bugs and will also see changes to the new search during the beta period, including interface enhancements and refinements to the search algorithm that affects results. We feel that real-life use by searchers will give us critical feedback that we can’t adequately capture during internal testing.
What do you plan to change during the beta period?
We have several items we are planning to address during the beta, including:
- Accessibility support: The first Beta release has not been optimized for use with screen readers or other assistive technologies.
- Open URL support in the preview flyout: We're adding links to your library’s website to help you find items not fully available to you on JSTOR
- Additional facets for advanced filtering
- Support field searching (author, title, etc.) without code: In this initial release, it’s necessary to use field abbreviations to conduct these searches.
- Support for using the Beta Search on mobile devices
We also plan to adjust the interface and search algorithms, informed by usage analytics and the feedback we receive from those who try the new search.
Will this affect library usage reports?
During the Beta period, librarians may see changes to the number of searches reported in the JSTOR usage reports. We are logging all searches conducted via the Beta Search, but they will not be added to JSTOR or COUNTER usage reports at this time. The faceted interface to the Beta Search works much differently than the standard searches (each facet selection counts as a search) and we are concerned about the potential for inflating usage reporting. We’ll be speaking with librarians about the best way to provide these data in the future and may incorporate them into our reporting at a later date.