1 Introduction
Cultural heritage conveys evidence of culture and society, and is a precious resource for education, research, and appreciation. However, non-moveable heritage sites such as historic townscapes and landscapes, archaeological sites and remains, and standing buildings and monuments are subject to geo-location constraints and erosions caused by natural forces (Herbert, 1995). In recent years, the interest has been growing in applying digital technology to documenting and preserving cultural heritage (Garoufallou, Siatri, & Balatsoukas, 2008). Digital libraries (DLs) have played an important role in digitalizing and preserving cultural heritage materials, providing users with access to digital surrogates of these precious materials from anywhere at any time. DLs have thus contributed to the solutions to both heritage preservation and geo-location independent access. This, in turn, makes DLs agents for promoting local history and cultural heritage to a wider audience (Tait et al., 2016).
Dunhuang is a small town in Northwest China. It is well-known worldwide for the nearby Mogao Caves (also called Mogao Grottoes) which is a world heritage site listed by the United Nations Educational, Scientific and Cultural Organization (UNESCO). The Mogao Grottoes “consist of 492 caves containing over 45,000 m2 of murals, 2,415 painted sculptures, and more than 4,000 Apsaras statues” (Lu & Pan, 2011, p.2) that reflect the cultural interchange within the oriental areas via the Silk Road from the 4th to the 14th century (Lu & Pan, 2011). It has great historical, scientific, and artistic values for the study of ancient Chinese, the central Asian history, Buddhism, and ancient art (Lu & Pan, 2011). Aiming for preserving and promoting the precious Dunhuang heritage using digital tools, the Digital Dunhuang project has been undertaken by Dunhuang Research Academy with the support of the Andrew W. Mellon Foundation (Wang et al., 2015; Zhou, 2011). The Mogao Cave Panorama DL is one of the first end-user facing products in the Digital Dunhuang project that makes digital images of Dunhuang heritage accessible online. It is thus critical to understand how it serves the end-users and how it can be further improved from the users’ perspectives. The findings will not only benefit the Panorama DL, but also all end-user oriented digital presentations of Dunhuang heritage, such as interactive media presentations or 3D immersive exhibitions. The methods used in this study also provide a reference for future evaluations of digital platforms of cultural heritage in Dunhuang and beyond.The Dunhuang Mogao Cave Panorama DL is available on the Dunhuang Research Academy website（1）((1)http://en.dha.ac.cn/). It includes 17 scenes: the surroundings of the academy, the exhibition center outside the Grottoes, and 14 caves. The caves were reconstructed in 360-degree panoramic views to support users’ seamless explorations (Kenderdine, 2013). Users can control the views and navigate through the DL using mouse clicks and movements. Most of the scenes can be zoomed in and out for flexible examinations in different resolutions. Users are also allowed to navigate through the caves by following several prepared paths, which are highlighted as “green footsteps” in the photos. The scenes also have audio introductions in Mandarin Chinese. Figure 1 shows a screenshot of the homepage of the DL.
Figure 1. Homepage of the Mogao Cave Panorama DL© of Dunhuang Research Academy. |
This study focuses on evaluating the Mogao Cave Panorama Digital Library from the end-user point of view. Four groups of end-users were recruited, based on their professions, to conduct usability tests with nine information seeking tasks designed specifically for the Dunhuang heritage. Findings of this study will help reveal how users interact with the Panorama DL. User friendly and problematic areas will be identified, based on which suggestions can be made for designers and developers in Digital Dunhuang projects to improve this DL and other digital presentations of Dunhuang heritage. In addition, as a case study on user evaluation of cultural heritage DLs, this work showcases the methodology, from task design, participant sampling to quantitative and qualitative data analysis, which can be adopted and/or adapted by future studies in related topics.
2 Literature Review
2.1 Digital Libraries on Cultural Heritage
Digitization of cultural heritage treasures has attracted remarkable attention in recent years (Hampson et al., 2012). A number of DLs and online repositories of cultural heritage sites have been established worldwide, with the help of computational technologies. HistoricPlacesLA（2）((2)www.historicplacesla.org/)is an ongoing program for identifying significant historic resources throughout the city of Los Angeles. In this collection, each historic site is labeled with geographic coordinates indicating its location and the period of its existence. Besides, rich metadata is provided to describe the landscape and related stories of a site. The DL also features a set of flexible search functions, supporting fine-grained searching and filtering criteria.
The Eternal Egypt（3）((3)www.eternalegypt.org/) is an online digital museum exhibiting culture heritage in Egypt. The collections contain rich information and support innovative accessing methods. For example, users can view high-resolution images, three-dimensional or virtual reality reconstructions of Egyptian monuments and antiquities, as well as panoramic images. Users can follow an interactive map or a timeline guide to navigate through the digital museum. In addition, entities exhibited are connected by hyperlinks such that users can also learn the complex relationships among them. More professionally oriented is Ancient Locations（4）((4)www.ancientlocations.net.), an evolving online collection of ancient archaeological sites. The collection records historic sites existed prior to 476 CE in the Old World (end of the Western Roman Empire) and prior to 1,492 CE in the New World (re-discovery of the New World) that are of archaeological interests. The number of places included is still increasing.
Without any doubt, all these digital technologies in the aforementioned DLs can be made use by the Digital Dunhuang project. In fact, an immersive exhibition of Cave 220 using virtual reality technique and 360-degree panoramic projection has been held in China and overseas (Kenderdine, 2013).
2.2 Usability Evaluation of Cultural Heritage Digital Libraries
Notwithstanding their irreplaceable advantages in preserving culture heritage, DLs also need careful designs and evaluation to ensure their usefulness and values. There are a number of studies on evaluating the usability of cultural heritage DLs and exploring ways of improvement. For example, Garoufallou, Siatri, and Balatsoukas (2008) evaluated the usability of VeriaGrid, an online virtual map DL focusing on the provision of information about cultural heritage in Veria, Greece. Their findings indicated that, although novel features such as recommended points of interests were added into the system, users still encountered difficulties in locating texts describing the heritage sites in the DL. Suggestions were then proposed to address this issue accordingly. Likewise, Jeng (2008) investigated the New Jersey Digital Highway, a DL built for demonstrating the history and culture of the state of New Jersey. Their findings identified issues in navigation and layout of the DL. Barak et al. (2009) examined the usability of MOSAICA, a Web 2.0-based DL for the preservation and presentation of cultural heritage in Europe. They applied online questionnaires and semi-structured interviews to obtain evidence of the usefulness, ease of use, and attractiveness of the DL. The authors reported positive views toward MOSAICA’s usage.
Smith and Rowley (2012) analyzed websites of nine public libraries in the Greater Manchester area in UK for the Web presence of local heritage, on the aspects of usability, content, aesthetics, and Web 2.0 services. Together with interviews with local study librarians, their findings suggest that, among other things, researchers and practitioners should continue developing knowledge of users’ expectations and information seeking behaviors, which can inform decision making on preservation practice, strategy, and policies (Smith & Rowley, 2012). In their framework of “information representation design,” Abbas et al. (2016) also argued that, in the spirit of user-centered design, understanding of users’ information behaviors and needs is crucial for designing information systems to support users’ information seeking tasks.
The Dunhuang Mogao Cave Panorama DL is the first panorama image collection of the Dunhuang heritage available online for public access from anywhere at any time. It is also one of the first of its kind for stone cave temples and murals. It was released to the public in 2015 and has yet to be systematically evaluated by end users. This study aims to fill this gap by exploring its performances with regard to usability and user experience. Specifically, we aim to answer the following research questions (RQ):
RQ1: To what extent does the Mogao Cave Panorama DL facilitate information seeking tasks in different levels?
RQ2: How does the Mogao Cave Panorama DL perform in each criterion of the user evaluation?
3 Method
Usability tests with a think aloud protocol were conducted in this study, followed by post-test interviews. A set of information-seeking tasks were designed for the participants to complete with the Panorama DL during the usability tests.
3.1 Participants
As different users may have different purposes when using the DL, four groups of representative end-users were identified for this study. The first two groups are users with professional purposes, such as artists or designers who seek materials
to stimulate their creation (group 1), and school teachers who look for examples or evidence for the courses they teach (group 2). The other two groups are users with non-professional purposes, such as the general public who seeks background information of the Mogao Caves for entertainment or for trip preparations (group 3), and students who want to learn new knowledge about this splendid cultural heritage (group 4). Participants in this study were then recruited according to these considerations, through the authors’ professional and personal networks.
to stimulate their creation (group 1), and school teachers who look for examples or evidence for the courses they teach (group 2). The other two groups are users with non-professional purposes, such as the general public who seeks background information of the Mogao Caves for entertainment or for trip preparations (group 3), and students who want to learn new knowledge about this splendid cultural heritage (group 4). Participants in this study were then recruited according to these considerations, through the authors’ professional and personal networks.
Fourteen participants in total joined this study. There were six males and eight females, aged between 18 and 29 with an average age of 23. The occupations of the participants diverged, ranging from college students (6 of them), administrative clerks (4), school teachers (2) to multimedia professionals (2). Half of the participants used the Internet for 6-10 years, while the others used over 10 years. All participants had used the panoramic function in Google Map.
3.2 Usability Tests
We selected the evaluation criteria of the usability tests with considerations of 1) the usability evaluation model adapted from Jeng (2005), and 2) the importance and definition of the website’s interactivity (Liu, 2003). The framework of evaluation criteria is shown in Figure 2 , which includes the aspects of effectiveness, efficiency, satisfaction, and interactivity (Jeng, 2005). Effectiveness and efficiency account for the system’s capability in providing content and facilitating information seeking. Satisfaction involves the DL’s support of offering a smooth and pleasant user experience (Jeng, 2005). Specifically, the following criteria of satisfaction are included: ease of use, organization of information, labeling, visual appearance, content, and error correction (Bevan, Carter, & Harker, 2015; Dani et al., 2015; ISO, 1998; Jeng, 2004; Pant, 2015).
Figure 2. Framework of usability evaluation criteria, adapted from (Jeng, 2005). |
The aspect of interactivity refers to the DL’s performance in engaging users and facilitating two-way communication between users and the DL (Liu, 2003). Specific criteria of interactivity include: active control, two-way communication, and synchronicity (Alba et al., 1997; Jensen, 1999; Liu & Shrum, 2002; Wu, 1999). Interactivity is regarded as important for cultural heritage DLs because online visitors would naturally expect that the digitized historical sites can preserve as much the visiting experiences of the real sites as possible. Therefore, DLs with better interactivity can help make the virtual visiting experiences more engaging and richer, through facilitating multi-dimension perceptions, observations, and manipulations (if possible) (Tait et al., 2016).
As users need to perform relevant tasks in the usability tests, we designed nine information seeking tasks in four categories of different cognitive complexity levels (Spool, 1999), with considerations of the contexts of Mogao Caves as well as panorama representations of cultural heritage, including colors, shapes, structures, and themes (Shiaw, Jacob, & Crane, 2004).
Tasks 1 to 3 belong to simple fact tasks that require users to locate given items or to identify specific information:
Task 1: Does the collection include any caves that are central-pillared (which means there is pillar in the middle of the cave)?
Task 2: Please locate one item that is constructed in the Northern Wei, Tang, and Yuan Dynasty respectively.
Task 3: In the collection you are working with, how many caves can you find displaying in the Digital Exhibition Center?
Tasks 4 and 5 are in the category of comparison of facts tasks which require participants to conduct comparisons of simple facts or information:
Task 4: Please try to compare the central statue in Cave 275 and the first statue on the right in Cave 322, and see if the right hand of the first statue (the central one in Cave 275) is damaged.
Task 5: Among the caves mentioned in task 3, list two which are decorated with the Thousand-Buddha motif (shown in Figure 3 ):
Figure 3. Example of Thousand-Buddha motif© of Dunhuang Research Academy. |
Tasks 6 to 8 are judgment tasks that require the participants to first locate needed information and then formulate an opinion based on analysis of the information:
Task 6: You would like to know more about the artistic style (e.g. color and shape) of those Flying Apsaras (an example shown in Figure 4 .) from the Mogao cave arts. Name one cave from which you can find such information?
Task 7: Other than Buddha stories and Chinese mythology, you would like to know more about how people lived in ancient China (e.g. their economic, military, political status, and activities) from the wall paintings and statues. Which item(s) will you go for?
Task 8: Which of the following theme(s) best describe the paintings on the roof of Cave 249? A). War; B). Chinese mythology; C). Hunting; D). Music performing.
Finally, task 9 is the comparison of judgments task. It is the most complicated category, involving both comparisons and judgments:
Task 9: Please identify and describe in general if there is/are any difference(s) between the artistic styles of the cave arts in the Tang and Yuan Dynasty?
In the usability testing, efficiency of the DL was measured by time (in seconds) spent in completing the information seeking tasks (i.e. task completion time). Effectiveness of the DL in supporting information seeking was measured by the proportion of tasks that were successfully completed（5）((5)A task was considered to be successfully completed if the participant was able to finish it without committingany fatal errors such as giving obviously wrong answers or giving up the task.)(i.e. task success rate).
During the usability test, each participant conducted all the nine tasks. They were instructed to follow the think aloud protocol. The test sessions were audio recorded and then transcribed into text afterwards. All the participants were voluntarily recruited and no monetary payments were provided.
Figure 4. Example of Flying Apsaras© of Dunhuang Research Academy. |
3.3 Post-test Interview
After the usability tests, the participants were asked to answer a set of post-test questions in an interview. The participants were asked to rate the DL in the aforementioned criteria of usability. The interview also included open-ended questions for soliciting detailed explanations behind the ratings. The actual questions are presented in Table 1 . All the rating questions were on a five-point Likert scale, ranging from difficult/unclear/disagree (point 1) to easy/clear/agree (point 5). The interviews were audio recorded and then transcribed into text for analysis.
Table 1 Interview questions on evaluation criteria. |
| Criterion | Question |
|---|---|
| Ease of use | Was the DL easy to use? Why or why not? |
| Organization of information | How do you think about the organization of information on the DL? Why? |
| Labeling | Do you think the categories of the DL are clearly labeled? Why or why not? |
| Visual appearance | Do you think the DL is visually attractive? Why or why not? |
| Content | What do you think about the accuracy of the information on the DL? Why? |
| Error correction | Can you recover from mistakes easily? Please give an example. |
| Active control | When exploring the site, do you have a lot of control over what you can do? |
| Two-way communication | Do you think the DL allows two-way communication between you and the DL? |
| Synchronicity | Do you think the DL is fast in responding to your requests? Please give an example. |
4 Results
4.1 Quantitative Results
To measure the effectiveness and efficiency of the DL, we counted the number of participants who were not able to successfully complete a task and the average time used by the participants for completing each task. The results are shown in Table 2 . Tasks in the simple fact and comparison of facts categories (tasks 1-5) resulted in an average success rate of 87.1%, while tasks involving judgment and comparison of judgments yielded an average success rate of 76.8%. The result suggests that the DL was better in supporting tasks in lower complexity levels than those in higher complexity levels.
Table 2 Task completion statistics. |
| Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Task 7 | Task 8 | Task 9 | Avg. | |
|---|---|---|---|---|---|---|---|---|---|---|
| Avg. time (seconds) | 157.2 | 21.6 | 99.6 | 28.2 | 118.6 | 108.5 | 80.2 | 86.3 | 110.9 | 90.0 |
| #. Failures | 1 | 0 | 4 | 0 | 4 | 2 | 4 | 7 | 0 | 2.44 |
To compare the effectiveness of the DL in supporting different user groups, we calculated the task success rate of each participant which is defined as the number of successful tasks divided by the total number of tasks (i.e. 9). The results are shown in Table 3 , which seem to suggest that the DL worked very well for multimedia professionals, followed by teachers and college students. To further understand the difference across users, we also compared task success rates between genders, age groups, experience with the Internet, and interests in cultural heritage. An obvious difference was found between genders, with males outperforming females by more than 10%. Longer history of Internet use seems to have helped a little too. These observations seem to suggest that experience and competency on IT skills might benefit the use of this DL. However, further work with a larger sample of users in various groups is needed for a systematic cross group comparison.
Table 3 Comparison of task success rate among participants. |
| Dimension | Groups | Success rate |
|---|---|---|
| Occupation | Admin clerks | 75.03% |
| College students | 83.35% | |
| Multimedia professionals | 94.45% | |
| Secondary school teachers | 83.35% | |
| Gender | Female | 77.80% |
| Male | 88.90% | |
| Age | 18-20 | 83.35% |
| 25-30 | 81.96% | |
| Experience with the Internet | 10 years | 84.14% |
| 6-10 years | 80.97% | |
| Interests in cultural heritage | Somewhat uninterested | 81.50% |
| Neural and somewhat interested | 83.35% |
The rating scores on perceived satisfaction and interactivity in post-task interview are aggregated across participants and presented in Table 4 . The results indicate that the DL offers pleasant experience by providing aesthetic appearance (median = 4), guidance for correcting errors (median = 4), and active and instant controls (median = 4); whereas it is quite weak in providing two-way communication with users (median = 2).
Table 4 Participant ratings on perceived satisfaction and interactivity. |
| Dimension | Rating (five-point Likert scale) | |
|---|---|---|
| Median | Mean | |
| Ease of use | 3 | 3.2 |
| Organization of information | 3 | 3.4 |
| Labeling | 3 | 3.4 |
| Visual appearance | 4 | 3.6 |
| Content | 3 | 3.4 |
| Error correction | 4 | 3.9 |
| Active control | 4 | 3.8 |
| Two-way communication | 2 | 1.9 |
| Synchronicity | 4 | 4.0 |
Correlation analyses among task completion time, success rate, user satisfaction, and interactivity reveal that satisfaction and interactivity are significantly correlated (Spearman’s rank-order correlation coefficient ρ = 0.864, p = 0.000). In other words, the higher rates on interactivity participants gave, the higher satisfaction levels they would have as well. Besides, the results also show a positive and moderate correlation between task completion time and success rate, indicating that the more time spent on completing the tasks, the higher chances the user could succeed in the tasks (Pearson’s correlation coefficient r = 0.592, p = 0.026).
4.2 Qualitative Results
In order to explore the factors affecting usability of the DL in more depth, we analyzed the participants’ think-aloud and the post-task interview transcripts. Major findings are summarized as follows.
4.2.1 Efficiency
Evidence shows that sometimes responses from the DL were slow, resulting in participants spending longer time in completing the information seeking tasks. For example, in task 8 which required participants to identify the themes of the roof painting of a given cave, one participant had the experience:
“...waiting half a minute for this page to load or the graphics to pop up, and this is the third time I refresh this page, but it just gets stuck there...” (Participant No. 04)
One reason for this glitch might be the conditions of users’ hardware or network. Because the visual materials in the DL are of high resolution, it may require a fast network speed for transmission and a fast client computer for rendering the visual elements. The fact that users’ network conditions and accessing devices may vary should be taken into consideration in DL design, and multiple choices of image quality (i.e. file size) can be provided.
4.2.2 Effectiveness
Statistics have shown that higher-level tasks (those in judgment and comparison of judgments categories) tend to result in lower success rates. This could be at least partially attributed to the higher cognitive load demanded for completing these tasks. Nonetheless, we can still find evidence of system deficiency in supporting higher-level information seeking tasks from users’ think-aloud utterances. For example, in task 5 when participants were asked to find the caves with the Thousand-Buddha motif, some participants complained:
“...the wall paintings of many caves are too tightly squeezed together, which make it quite difficult to pick out specific items or themes...” (Participant No. 02)
“...they (the arts in the cave) are not labeled with any textual descriptions... (I) find exploring the caves a matter of hardship and suffering...” (Participant No. 03)
These comments suggest that the layout and arrangement of the digital images could have room for improvement, and more descriptions and labels are desirable, in both facilitating users’ information seeking processes, but also improving user experience in navigation.
4.2.3 Ease of Use
As navigation in a panorama DL heavily relies on its view control, whether the control function is easy to use can affect users’ experience with the DL to a large extent. There appears to be several issues of view control in users’ think aloud discourses. The problem of adjusting the position and rotation speed of the view control was mentioned quite often:
“...When I try to move left, it (the view) goes right; and when I want to look right, it (the view) goes left. Everything just goes reversely...” (Participant No. 06)
“...it is really annoying that the view always keeps rotating in a very high speed...” (Participant No. 06)
Another participant who also complained about the view control claimed that the contradictive direction controls of the DL to his preferred ways made him sometimes confused and move in the wrong way.
Other problems include insufficient instructions on the view control, no “back” buttons for returning to previous pages, and inconsistency of the interface.
4.2.4 Organization of Information
There were different views among participants regarding the DL’s way of organizing information. Some participants appreciated the organization structure ordered by cave numbers and dynasties:
“...the items on the main page are organized in terms of dynasty, which is quite clear...” (Participant No. 04)
“...most of the items on the main page are well organized in terms of cave number and dynasty...” (Participant No. 13)
Others, however, argued that the organization of information on this DL could be improved by arranging the exhibits in a mini navigation map which could mark the caves and the surroundings in a way aligned with the geographic configurations in the real-world. In this way, all the exhibits could be linked together, allowing a more intuitive and straightforward navigation:
“...I’d suggest to applying the mini-map fast travelling function to the whole collection, so caves and caves are more interrelated, and helps to provide us with a more consistent tour...” (Participant No. 09)
4.2.5 Labeling
From the participants’ think-aloud discourses and comments on the DL’s performance on labeling, it is found that most of the labeling issues were related to the mini-map available in the scene, “Digital Exhibition Center.” The mini-map contained clickable green hotspots that are designed to guide users to corresponding exhibits. However, the hotspots were not labeled and thus users did not have the information about which exhibits a hotspot would lead them to:
“...the mini-map of the Digital Exhibition Center is kind of raw and not labeled, I have to spend quite a lot of time to click on those green spots and see what they really are...” (Participant No. 10)
Besides, participants complained that many of the cave arts presented in the DL were unlabeled or labeled unclearly. As a result, users with limited relevant knowledge were not able to understand the meanings of the arts. For example, one participant commented on the labeling system in the DL:
“...most of the caves arts are not labeled. It very much depends on my first impression to identify the contents or themes of those wall paintings and statues...” (Participant No. 11)
4.2.6 Visual Appearance
The application of panoramic photos for reconstructing the caves in virtual space makes the DL visually appealing to the participants. The visual appearances of the DL were regarded as highly detailed, of superior quality, high resolution and realism. Some participants (e.g. No. 03 and No. 06) found the sunlight effects on the main page “quite advanced” and “looks like real,” as the angle of shinning changed as soon as the participants moved their mouse to change the view.
However, the visual appearance of the homepage was criticized by participants, for a large part of the page was covered with thumbnail images representing the exhibits. Besides, the image effects of the homepage were not consistent with other pages, reducing the consistency of the DL and visual attractiveness.
4.2.7 Content
Most participants agreed that the content on the DL was authoritative and accurate. Nonetheless, some participants commented that, as they had never been to Dunhuang, and they had very little knowledge of the caves to tell if the virtual caves are in fact the same as the real ones.
4.2.8 Error Correction
In general, the DL performed well on this criterion. One issue that might cause unexpected errors was the default option of opening new tab when an item was clicked. While some participants liked this design for easy return to previous pages, others complained that having too many tabs might result in accidental closings of some tabs.
4.2.9 Interactivity
Participants agreed that the DL provided good human-computer interaction experiences in viewing the exhibits. However, they complained that most interactions were only one-directional:
“...most of the time it is me who inputs, and I don’t see there’s any feedback at all...” (Participant No. 02)
“...I think I’m all the way exploring the site alone, and there’s no support from the system...” (Participant No. 05)
Most participants mentioned that it would be desirable to have more two-way communication between users and the DL, as well as between users.
5 Discussion
The results of quantitative data analysis help answer research questions Q1 (To what extent does the Mogao Cave Panorama DL facilitate information seeking tasks in different levels?). The success rates on information seeking tasks of high complexity levels were lower than those on tasks of low complexity levels. This may imply that the DL lent limited support to relatively complex information seeking tasks. One possible way to improve could be to provide more navigation support and meaningful labels to facilitate users’ exploration with the DL. In addition, more textual descriptions about the background, themes, and interpretations of the caves and the arts (i.e. murals, statues) can be added to help users learn new knowledge during the search and interaction process.
The qualitative data analysis helps answer research question Q2 (How does the Mogao Cave Panorama DL perform in each criterion of the user evaluation?). Although the DL performed generally well in the usability tests, issues were identified in most evaluation criteria, suggesting room for future improvements. For example, system responses were sometimes slow (efficiency); it was not easy to adjust mouse positions and rotation speeds (ease of use); the hotspots in the mini-map lack semantic labels (labeling); and bi-directional communication in the system was weak (interactivity).
Suggestions can be made to address these issues: 1) options can be provided to make the image quality adjustable according to users’ hardware and network conditions, so that timely responses from the DL can be ensured regardless of different accessing conditions; 2) to improve ease of use, the navigation system may provide users with ways to adjust mouse movement direction and speed; 3) navigation can be facilitated by a mini-map that situates the caves into their surroundings; 4) the original mini-map with hotspots can incorporate labels such as titles or cave numbers, for easy reference by the users; 5) considering some users may have little background knowledge about Dunhuang, the DL can add introductions to cultural heritage materials, which will also make the DL more attractive and interesting; 6) to help users recognize the authority and accuracy of this DL, information on the development of the DL could be added, such as the image creation process (e.g. technologies used in preserving and representing the caves), and background of the parties who created the images and those who developed the DL itself; and 7) interactivity can be enriched, by possibly creating a virtual character that can guide and support users in navigation. Alternatively, more communication channels (e.g. chatting forum, mailing lists, social tagging, and Weibo account) can be established to enhance library-to-user and user-to-user discussions, as well as timely updates with the latest news about the DL.
6 Conclusion and Future Work
DLs provide a strategic solution for culture heritage preservation and access. The usability and user experience of DLs are critical for the success of this strategy. In this paper, we presented a user evaluation of the Mogao Cave Panorama DL, aiming to identify issues that end-users may encounter when using such DLs. Usability tests were conducted based on a framework of evaluation criteria and nine information seeking tasks designed for the Dunhuang cultural heritage. The results help identify the strengths of the panorama DL in visual appearance and error correction, as well as areas for improvement, particularly in user control and two-way communication. From the results, we proposed suggestions for further improving this DL. It is our hope that, after the improvement, the DL can better convey the educational, research, and artistic values of the Dunhuang Mogao Grottoes, and at the same time offering a multipurpose and user-customized access experience.
The evaluation framework, the designed information seeking tasks, and the interview questions can be adopted or adapted in evaluating other DLs and digital platforms of Dunhuang heritage. Very recently, the Digital Dunhuang project released the e-dunhuang（6）((6)www.e-dunhuang.com) digital platform that presents digital images and panorama views of 30 Mogao Caves, with textual descriptions and search functions. Some of the suggestions in this study were adopted and realized in this new digital platform such as providing introductory information for the materials, allowing adjusting mouse movement direction, and adding labels to small pictures on top of panorama views. Currently, we are conducting user evaluation of this new digital platform. Future research will include fine turning of the evaluation instruments and enlarging the sample size to include more user groups such as researchers, school students, and oversea users, toward the goal of enhancing research and education values of Dunhuang heritage.
Author Contributions
X. Hu (xiaoxhu@hku.hk, corresponding author) proposed the research problems, guided the literature review, designed the research framework and the methods, and co-wrote the manuscript. E.M.Y. Ho (h1287872@connect.hku.hk) carried out the usability tests and interviews, and co-wrote the manuscript. C. Qiao (cqiao@connect.hku.hk) searched the literature, analyzed the data, and co-wrote the manuscript. The authors discussed details of the study throughout the process, and refined the manuscript together.
The authors have declared that no competing interests exist.