During the 2003-2004 academic year, over thirty colleges, universities and museums have been participating in Artstor’s “test” phase. The goal of Artstor testing was to assess our progress in building the Artstor image collections and software tools. Our institutional test partners – fourteen in the fall semester of 2003, and over thirty-five in the spring of 2004 – were invited to participate on the basis of a range of considerations, but mostly because of their ability and willingness to bring diverse perspectives to bear on key aspects of Artstor’s development. Users at these test institutions have had continuous access to Artstor for use in teaching and scholarship. Institutional staff — including librarians, visual resources professionals, and instructional and information technology staff — have been working closely with Artstor to assess the testing experience. We are grateful to all these individuals for their willingness to help understand their varied interests and needs, and to provide useful guidance for improving all aspects of Artstor. A full list of our test partners is at the bottom of this page.

What follows is a brief interim report of the results of this testing. When the test phase is completed, we will update these findings and post them on the website.

Technology
Technology is an important aspect of any digital library delivery system, and the test findings led us to improve the functionality, usability, and performance of ARTstor’s technology and software. We experienced problems launching the software in some Macintosh environments. As of April, the system works on the OS10 operating system, and we anticipate that the difficulties we experienced with the earlier OS9 system will be resolved by June. We have also developed several approaches to “interoperating” with institutional digital collections and software (e.g., image management systems, learning management courseware, and other digital resources). To facilitate access for our participating institutions, we are now pursuing several approaches to authentication and authorization of end users. Most importantly, testing confirmed our belief that it is essential that ARTstor develop software that is tailored to the considerable variety of needs of its different users. Because we have chosen to build our own software rather than work only with existing software products, we believe that we have been better able to improve and adjust that software to reflect user comments and criticisms in a timely manner.

Feedback during testing also suggested that we build additional user-friendly tools to expand the possible uses of ARTstor’s content. In one specific example, our testers assisted us in designing an off-line viewer that allows the use of ARTstor content in the classroom, with no reliance on Internet connectivity. This off-line viewer was created in response to tester concerns about relying on a network connection to present images in class and by the widespread need to make presentations (in a classroom or elsewhere) without connecting to the Internet. We are very grateful to James Madison University for its generosity in allowing ARTstor to build from their code in their widely adopted MDID viewer to create the initial ARTstor Offline Viewer.

Finally, because ARTstor is hosted, launched and distributed in a way that enables users to work in an active software environment (i.e., dragging and dropping images from one saved group to another), testing allowed us to see how we needed to enhance our technological infrastructure. We improved server capacity and the load-balancing system, and also reconfigured the application server to enhance speed and performance.

Usability Testing
This fall, the Digital Knowledge Center (DKC) at Johns Hopkins University worked closely with ARTstor on usability testing, including conducting one-on-one tests with users at many of the test institutions. Because of the forty-two ARTstor usability tests, and the detailed findings that the DKC reported as a result, we learned about specific barriers, which we were able to remove or ameliorate. The majority of this work will be completed prior to ARTstor’s launch this summer.

Users
During the testing phase, we learned a great deal about the various needs of potential ARTstor users – especially about the range of potential users. While we believe it is clear that ARTstor provides value to individuals studying Art History and related fields, it also hopes to meet the broader need for a campus-wide image resource that is accessible to scholars, teachers and students across the humanities and beyond. We are also encouraged by the fact that our test phase indicated that ARTstor’s software provides the right tools for most users of digital images and is sufficiently user-friendly to enable individuals who are relatively new to digital technologies to feel comfortable using ARTstor.

Content
Feedback also indicated that the ARTstor Charter Collection is broad enough to engage a reasonably wide audience, and to support a good range of teaching needs, as well as deep enough in some areas to support aspects of scholarly research. ARTstor continues to seek new collections to provide greater breadth as well as the depth that scholars will require. One of ARTstor’s greatest challenges in building collections is balancing the need to add a great many more images with the desire to maintain appropriately high standards of image and data quality. Various users at different test institutions championed both “sides” of this equation – some calling for more images even at lower quality and others preferring that we include only the highest quality images in ARTstor. We do not believe there is one “right” path in this area; meanwhile, we are encouraged by the fact that so many users have urged us to strike a sensible balance, and to retain our core commitment to enhancing the quality of images and data over time.

ARTstor Licensing Agreement
Licensing digital content — especially images of art — is extremely complicated. Balancing intellectual property requirements with the educational and research needs of the non-profit educational community is a challenge. Based on concerns expressed by some of the fourteen initial test institutions, we made some significant revisions to our user agreements. We held a productive half-day meeting in December 2003 with user agreement representatives from our Fall test institutions in an effort to solve common issues. The final outcome, we believe, is an agreement that balances the needs and interests of ARTstor users, of their respective institutions, and of the many institutions that help ARTstor to build its collections. The vast majority of our new test institutions signed the revised agreement with scarcely any comments or questions. Given ARTstor’s strong commitment to work with all non-profit educational institutions in order to foster the use of digital media in teaching and learning, we regard the general acceptability of this new user agreement as a very important outcome of ARTstor testing.

What ARTstor Offers
We have deliberately designed ARTstor in a way that differs in many ways from other online information resources. ARTstor is relatively unusual in providing three separate but integrated components. First, it consists of a very large and constantly growing online database of searchable images and accompanying conformation. Second, it has embedded the database in a software “tool-kit” that enables the user to search and make active use of the collections. Finally, ARTstor also provides a restricted environment through its own network that is accessible only to members of non-profit institutions. By providing a restricted network (not accessible publicly through the open internet) and by seeking to balance the interests of content providers and users, we believe that ARTstor can play an important role in working through these complex intellectual property issues in a way that individual institutions – on their own- might find extremely difficult to accomplish.

These three inter-related components and services make ARTstor very different from a straightforward online digital database purchased by a library. During the testing period, we have begun to learn about ARTstor’s “spillover” effects, including how the availability of such a resource encourages certain forms of inter-disciplinary work and begins to help with a very powerful integration of the critical related process of teaching, learning, and research.

While we have only begun to observe and assess the impact that ARTstor will have for users, we also should note how the testing period has been instructive with regard to the collaborations that have been explored and have begun to flourish across organizational boundaries; librarians, visual resources professionals, and instructional and information technology staff have come together to support the use of ARTstor. While ARTstor “lands” in the library, we will be interested to see if – or how – the broad applicability of ARTstor will be a catalyst for new organizational collaborations.

Conclusion
The preliminary findings from this test phase have taught us many important lessons – about almost every aspect of ARTstor. Most of the improvements we have made would not have been possible without the commitment of our testing partners, to whom we are very grateful. We believe that the contributions of all these partners – as well as the professional relationships we have been created or built upon through this process – will make ARTstor a much stronger and more useful resource for all its users.

ARTstor’s Test Partners

  • Art Institute of Chicago
  • Bowling Green State University
  • Bryn Mawr College
  • Carleton College
  • Columbia University
  • Connecticut College
  • Dallas Museum of Art
  • Emory University
  • Getty Research Institute
  • Harvard University
  • Hunter College
  • James Madison University
  • Johns Hopkins University
  • Los Angeles County Museum of Art
  • Metropolitan Museum of Art
  • Minneapolis College of Art and Design
  • National Gallery of Art
  • New York University
  • Northwestern University
  • Ohio State University – Main Campus
  • Pennsylvania State University
  • Perpich Center for Arts Education
  • Princeton University
  • Roger Williams University
  • Sarah Lawrence College
  • Smith College
  • Smithsonian American Art Museum
  • Trinity University
  • University of California – San Diego
  • University of Miami
  • University of Michigan – Ann Arbor
  • University of Oregon
  • University of Notre Dame
  • University of Virginia
  • Walters Art Museum
  • Wesleyan University
  • William Paterson University of New Jersey
  • Williams College, Sterling & Francine Clark Art Institute