User Loyalty Articles

During the two cycles of the iterative design process the use of appropriate design that suits the topic, focus on the topic, delivery of content and the creation of a positive atmosphere were identified as the most essential aspects for a valuable user experience. The prerequisite for these aspects to become the most essential was that the lower levels of user needs, usability and availability, were fulfilled.

Four pages had personalisation features implemented, the start page, the news page, the idea-rating page and the start page of the tutorial. While the tutorial did not have enough visitors to be of interest in this study, the personalisation system of the idea-rating page lacked content and therefore could not work as planned. The findings show that the news page and the start page were better perceived by users that had the personalisation features enabled (B+users). The accuracy and coverage of the news page personalisation system could be confirmed. B+users perceived the website as more trustworthy, were more interested in new features of the site and were more likely to recommend the site. This indication of increased user loyalty was confirmed by data that showed that B+users were visiting the site more frequently. Therefore, the hypothesis, that it is possible by utilising dynamic personalisation to increase user loyalty to a site, is confirmed.

Discussion and future work

The findings after the first cycle of the iterative design process depicted that users demanded an appropriate visual design that suits the topic and the findings of the first survey that 23.4% of the users liked the site because of the atmosphere having parts, which overlapped from the visual design. This was shown by the descriptions of atmosphere in the second survey in which 41% of users related the atmosphere to the visual design. The reason that the visual design made an impact was because aesthetic fidelity was achieved.

After the first cycle of the iterative design process it became clear that the participation of users would be less than expected and the concept had to be changed. Even though, the site already had the main features such as a forum and the possibility to submit own articles, the understanding of the audience was missing, which is crucial for developing an online community (Preece et al. 2004).

Therefore the goal of the website shifted to providing news and knowledge about the topic. The development was concentrated on creating more focus on the topic by providing background information and on developing an appropriate visual design. The main problem at this stage of the process was the lack of contribution of users to the content of the site. The idea section should bring more focus on the topic and give users another possibility to add content while the tutorial section was developed to guide users through the process of idea creation.

Feedback from forums in which problems with the structure of the start page of the second version of the site were mentioned were addressed by a test in which the aesthetic aspect and the scanability was tested. The test, with participants who were new to the interface, was conducted with the assumption that if users of a forum who were new to the interface were able to identify problems with the structure of the start page, the participants would be able to do that as well. The findings suggested that a more structured start page would decrease the aesthetic aspect of the page and therefore have a negative impact on the user experience while the scanability was only slightly faster on a better-structured site. During the progress of this research, comments of long-term users about the lack of structure on the start page showed that the test did not produce valid results for long-term users. It confirmed Dillon’s et al. (2001) findings that ‘evaluations based on single experiences with technology do not provide stable estimates of long-term (or even medium-term) usability.”

Certain limitations of this research must be acknowledged. Due to the complexity of user experience (Olsen, 2003) the essential aspects for the user experience, which were found, might not apply in general, especially given the specific topic and the resulting target audience. The focus on the German speaking countries, and as a result, the cultural background of users might have had an impact on the findings as well.

Second, the period of time for the field study, two months, might also be a limitation. The short period of time did not allow to establish an online community of any size in which personalised user rankings and identity played a significant role. Even though, the descriptions of the atmosphere of long-term users confirmed that it was possible to establish a “sense of community ” (Blanchard et al. 2001).

Third, because the final survey relies on data from a small group of people, five b+ user and five b- user, their personal interests in the website could have played a role in their judgements.

The result that four out of five B+ user could name the same favourite news category as in their user profile, while only one of the five B- user could, suggests that personalisation helped to create personal preferences or helped to identify them. The knowledge of a certain preference could make a user more sensitive to changes on a site and therefore have a negative impact on their loyalty.

Furthermore, the delivery of personalised content could limit the users perception of the variety of information on the site and therefore have a negative long-term effect on loyalty. Thus, future researchers should examine whether that the findings of dynamic personalisation can have an impact on user loyalty is consistent over a longer period of time. The creation of a personalisation lifecycle, which begins by creating a user profile and ends in de-personalisation, could be the framework for this future research.

The results of the final survey show that B+user consider the website as more trustworthy. Similarity, or in the case of this research content similar to the users preference can increase trust. Fogg (2003) identified credibility, which consists of trust and expertise, as a key factor for persuasive webdesign. The findings suggest that the use of personalisation can enhance the impact of persuasive webdesign. While persuasive webdesign only played a role in establishing credibility of the website. Future research should focus on the impact that personalisation can have in combination with persuasive webdesign.

The second survey was conducted three weeks after the first survey.

The second-survey questions about the loyalty of users were derived from indices used in the industry (Heater, pers. com. Feb. 19, 2005; Stratigos, pers. com. Feb. 19, 2005). To understand how efficiently the personalisation system works one has to measure the accuracy and the coverage (Mobasher et al. 2001)(Sheperd et al. 2001). Accuracy measures how precisely the personalisation system can deliver content that the user likes and coverage measures how likely the user is to accept the content.

Questions about the credibility of the site were important because without credibility a website cannot ‘persuade users to change their attitudes or behaviours’ (Fogg 2003, p. 148).

Unlike the first survey, the second survey was not promoted, except on the site itself and through emails to members. This was done because this survey was designed for long-term users of the site.

Data from the five most active B+users and the five most active B-users who completed the survey were evaluated, because this group of users experienced the site over a longer period of time. Every participant was a site member for at least 15 days, had at least seven visits to the site and read at least six news items distributed on different visits. During the time they were members the sum of all visits of the B+user group was 166 while the B-users visited 152 times.

In the first section of the survey participants had to rate aspects of every site in which personalisation features were included. The part of the survey in which users had to rate the tutorial was not evaluated because an insufficient number of users did the tutorial.

Following the Software Usability Measurement Inventory (SUMI) (Kirakowski 1994), which is a method of measuring software quality from the user’s point of view, a survey was prepared. The goal was to get more detailed information about the user experience the site provided and some indications of what impact that had on loyalty. The questions about usability (q. no. 3, 4) were implemented to reassure that no usability problems existed, because these problems would have direct impact on user satisfaction (Froekjmr et al. 2000).

Furthermore, the survey had questions which could be categorised into satisfaction, loyalty, website characteristics and background information. The survey was promoted in several forums and one had to become a member and visit every personalised page at least twice on the website before the link to the survey became visible. 64 members participated in the survey but only 10 of these were site members before the survey was announced. Therefore, the overall findings of the survey represent an extended first impression. They could only deliver an indication of the impact of personalisation features because even though new members had to visit every site a second time, these features provide valuable results for users over a longer period of time.

The survey confirmed the previous findings that the site was perceived well by users. Overall users with personalisation features enabled rated the site slightly higher than users without personalisation features. Of all participants, 69.2% were in the age of the target audience and nearly the half (31 participants) were using the internet more than 15 hours every week and were therefore considered as internet-savvy.

The response to the question (q. no. 7) in which participants had to characterise the site showed that only 5.4% named attributes that were considered as not intended by the designer and therefore negative. 94,6% of the selected attributes described the site as affectionate, personal, pleasant, unusual, exciting, bizarre, structured, warm and beautiful. This shows that aesthetic fidelity has been achieved. A surprising finding was that 23.4% rated ‘Atmosphere’ below ‘Content’ with 39.1%, as the most liked aspect of the site (q. no. . This result is positive because atmosphere describes ‘an interesting, often exotic, effect’ (Webster’s New World Dictionary 1988) and is considered a general emotional quality (Webster’s New World Thesaurus 1997) that can have an impact on the mood of the viewer (The Hutchinson Encyclopaedia 2003). That shows that the site can make an impact on users’ feelings. Even though it does not show what atmosphere the site creates. The high rating on ‘Content’ is positive because content is the ‘classic way to increase loyalty on the web’ (Nielsen 1997, 1999) and a way to attract new customers to a site (Wurster et al. 1999). This high rating confirmed that the chosen content for the website was appropriate.

A negative finding was that there was a gap between the overall impression and the perceived value a user got from visiting the site (q. no. 1,2). Even though the ‘overall impression’ is a mix of various elements and ‘what the site provides’ can be understood as related to content, it showed that the site generated expectations which it did not completely meet. This was repeated in the two questions about usability of the site (q. no.3,4). The structure of the site was rated better than how the users rated how they could navigate on the site.

Despite these results, more than half of the participants voted either good or very good in all four questions.

The question (q. no. of whether a user would return to the site was also answered positively by 31.5% of participants who said they would definitely return and 57.4% who thought they would return. The data of their user profiles showed that 33.3% of members who said they would definitely return and 80.6% of users who said they thought they would return did not return and log in. Therefore this question seemed to be unnecessary in the second survey.

The survey showed that users could be considered as overall satisfied. The usability and structure of the site was well perceived and aesthetic fidelity has been achieved. Questions that needed further investigation were about the kind of atmosphere that the site creates and if the site meets the user’s expectations. Furthermore the impact of personalisation features on the site had to be examined in detail.

A poll, on the beta site in which 13 members participated, showed that 12 members or 92% liked the beta site more than the alpha site, while one member or 8% liked the site as much as the alpha site (refer to Appendix G).

The positive feedback from the forums and the poll showed that no fundamental changes of the visual design were necessary any more.

The second poll on the alpha site had showed a demand for background information about ideas. While the knowledge section already provided background information, an on-line tutorial for two creative techniques was implemented to add an act experience to the site (Schmitt 2004) and guide users through the idea creation process. This tutorial also had personalisation features on the first page. The amount of information that was shown was controlled by the average time a user spent looking at a page of the website. If a user spent less time than the average of the other users he did not see extended explanations for the tutorial. Furthermore, an example picture with slogans according to his interests was presented to increase his motivation.

Even so, the qualitative data that was gathered until this point in time were from people who did not visit the site over a period of time. The positive feedback indicated that the fundamental needs of users – availability and usability – were fulfilled, which means that the site had no fundamental problems that could harm the user experience. The quantitative data that was gathered from members of the site confirmed the issues described in the qualitative data. This could be because the site was promoted mainly in the forums where feedback was gathered. The visitor statistics and days users registered -suggested that many members came from these forums and therefore the qualitative and quantitative data were similar.

After the second version of the site was launched and promoted in forums, 27 people gave feedback. The main problem with the alpha site was that the design was inappropriate for the topic. The feedback from the second version of the site showed that this issue was solved. While in the feedback for the alpha site only five people said they liked the design, the beta site feedback had 12 people with positive comments about the design. Only one person said that he did not like the design.

In the feedback (refer to Appendix F), two issues were named by several people. Five people said that the home page needed a better structure and two people said that the length of the sentences decreased the readability. The problem with the readability was an obvious issue and was fixed immediately by limiting the text length of every sentence. A test , based on Dillon’s et al. (2001) testing methodology for aesthetics and usability, was conducted to determine how fast a home page with more structure could be scanned and what impact this version had on the impression participants got from the design of the site. Four versions of the home page were used to do that. The test was conducted on a computer and the six participants chosen were not regular users of the website and therefore had no routine in navigating the layout.

Every participant was tested separately. In the first step participants had to give a rating on the look of the home page in its original form and in the more structured form. Both versions that were used in this step had news in a different order than in the next step. The design was rated on a 10-point scale with 10 as the best. In the second step, two versions with different news were used. The participants had to find a certain word in the headlines of both versions while the time was clocked by the instructor who made a note of how long it took them to find the word and point to it. The word they had to find was at the same location in both versions. Every second participant received the two versions in reverse order to assure valid findings, because participants might compare the second version to the first when they rated the design or might have already some routine to scan the layout when they had to find the word in the second version of the second step.

The design of the structured version was rated 14.85% – worse than the design of the unstructured version – while it was only scanned 2.03% faster (refer to Appendix L). This result was expected even though the small difference in the time it took a participant to scan both versions were an unexpected finding. The minimal difference in time it took users to scan both versions would not justify a change of the site design and therefore a decrease of the aesthetic fidelity (Norman 2002), which is “the degree to which users feel the target impression intended by designers who developed the web pages” (Park et al. 2004).

Due to the low amount of user participation and, as a result, the relatively small amount of content, it was decided to use content personalisation in this research. Collaborative filtering would have been the wrong choice as it finds a profile of another user that is most similar to the users, but without many members it is unlikely to find peers that are similar enough to provide valuable results (Adomavicius et al. 2003). Furthermore, the news section has only a small amount of new information every day, which reduces the options for success of collaborative filtering even more. Another reason for using content personalisation were the findings of Shepherd et al. (2001) that users do not enjoy too detailed information filtering, therefore the news is only filtered according to news categories.

The personalisation features were implemented in the start page, the news page and the page on which the user rates ideas. Every member was identified by his or her login and password. Because A/B testing was used, every second member that registered surfed the site with personalisation features enabled (B+user) while the rest surfed the site without them (B-user).

The implicit data for the personalisation system derived from predefined PHP variables (PHP Group, 2003), which save user data that is sent to the server, and the tracking of users’ browsing behaviour. The system is based on a user profile in which all data is saved and every site that has personalised content on it is controlled by the user profile. The user profile consists of data for identification (Identification Profile), system data such as ip address or id (System Profile), self-categorised data such as age or gender (Socio-economic Profile) and data about the actions of the user on the site (Interaction Profile)(Schubert et al. 2000)(refer to Appendix J). Every time a user loads a page his user profile is updated in the appropriate category of interest. The start page is excluded because it shows all categories of content in five modules and therefore does not represent a specific interest. Over a period of time a profile of the users’ interest in the site is created. According to this profile the ratio of each content category of the start page is adjusted. If one category was not visited yet a link is shown with a call to visit this category while B-users did not see this feature. [P4] This is done to create awareness of existing features on the site and thereby increase the user experience (McMullin 2004). Furthermore, the start page shows special features and articles according to the operating system of the user and every user is greeted according to the time of the day and the time he has not[P5] visited the site. (pic1)

On the news page the news posts are sorted by starting with news from the category the user preferred most in the past. B-users see the news items according to the order in which they were published. The news is categorised according to the five news groups that were determined in the user-perception test. These five groups have subgroups with more specific topics (refer to Appendix I). When users click on the link to the news source their profile is updated in the corresponding category. To prevent the profile being updated several times when the news has more than one link, the user profile logs every news item a user has already read and compares this data with the incoming data from every update. Only the news that was posted since the user’s last visit or in the past six hours is sorted according to the user’s profile. This is done because changing all news in real time could confuse a user and harm the user experience. To sort the news the ratio of the viewed news of a user to the total amount of news in this category was calculated. This had to be done because not every category had the same amount of news and therefore the pure data in the user profile did not have to represent the user’s interest.

The user rank was also derived from their activity on the site and favourite news category. The activity is the sum of all values of the content categories in the user profile. (pic user profile structure)

The user rank feature is the only personalisation feature that uses explicit data to a minimal degree. The gender that users assign to themselves is used to give the title the right grammatical form.

Ideas are also categorised (refer to Appendix K). On the idea-rating page a B+ user will see an idea from a category in which he rated an idea before if he did not already rate all ideas of this category. These 100%, which represent all ratings he made, are divided according to the sum of all ratings in each category. Therefore if a user rates ideas from a particular [EB6] category highly, it is more likely that they will get more ideas from this category to rate next. In this personalisation feature no ratio is necessary because the ratings control the level of interest.

The personalisation features have multiple goals. In the short term they should provide users with a better user experience on the website. The long-term goal is to give the website a personal touch and thereby establish an emotional bond between the user and the website, which results in increased loyalty.

After the site was put on line it was promoted in several forums(refer to Appendix E) which were selected according to the target audience. In the forums a thread was opened in the appropriate subcategory starting with a short description of the site and a call for feedback about the site.

28 people responded with their opinion about the site (refer to Appendix F). The first and most important finding was that the concept was considered a ‘good concept’ or ‘unusual’, which are both positive outcomes. The second finding was that five people stated that they like the design, but also five people said that it was not appropriate for the content of the site. Even though only five people supported this point of view, it was the only issue, besides the positive feedback, that was supported by several people.

The first poll in which 17 members participated, confirmed that the site was well received (refer to Appendix G). The second poll, in which 15 members participated, showed a demand for background information about ideas and that the site should deal more with the subject ‘ideas’ overall (7 votes). The second poll also confirmed that the design of the site did not suit the subject (6 votes).

Even though the majority of the feedback was not very detailed, the findings indicated that the visual design of the site was considered inappropriate. The site should also become more focussed on the topic ‘ideas’ by providing background information.

Despite the positive feedback, the number of members increased more slowly than expected.

First time visits to the site reached their peak usually on the day or the day after a forum thread was opened (refer to Appendix E).

A logfile analysis showed that the most visited page of the alpha site was the start page, which was also the news page. It was discovered that the news was the reason for users to visit the site again. That the news was the most popular part of the site was confirmed in November when the go.php page was implemented. This page decrypts encrypted links to articles that were posted in the news and became the most visited in the site.

In the competitor analysis (refer to Appendix B), websites with similar concepts were rated according to different criteria. The first key finding was that the main competitor of ideatemple was ideen-box.de because it was the only website of that kind in German and it had a high search-engine ranking (Google first result when people search for the German word for ideas, ‘ideen’). Another benefit of the site is its large amount of content in the form of ideas and the possibility of asking a lawyer copyright-related questions. The second key finding was that all examined sites did not have community features, with the exception of globalideasbank.org , but its forum lacked categories and had only a very small group of visitors. The third key finding was that no website provided background information or tutorials that were related to ideas except ideafinder.com, which lacked the possibility of submitting one’s own ideas. The fourth key finding was that the aesthetic and creative aspect was not considered in the websites that were compared.

While ideatemple did not have the resources to compete with the rich content of ideen-box.de, in the short term it was possible to implement community features, background information about ideas and develop an attractive and easy-to-use design that suited the subject.

The key findings of the literature review and the competitor analysis form the foundation of this research and the concept of the first version of the site is based on these findings.

As the internet is an active medium (Chak 2003, p.29) in which customers decide how they interact, it was important to give users various ways to contact the organisation behind the website and meet the demand for user-to-user interaction. The concept of the website included the possibility of getting assistance by email, using the chat feature to get live support, submit their own articles, submit their own news, submit their own ideas, or use the chat or forum to enter into a dialogue with other users. The news section also offered the possibility of writing comments to a news post. These features were included to adhere to the concept of establishing an online community (Rheingold 1993; Preece et al. 2003). A community offers the possibility of establishing relationships and improving the value of information (Tartaglia et al. 2005) and can generate much of the content of a site (Kotha 1998).

In addition creation of content by customers increases their involvement and enhances the credibility of a website (Kotha 1998). Another beneficial aspect of communities is that they produce loyal visitors. Studies show that active community members visit a site up to nine times more frequently (Sterne 2002, p. 180).

Online communities can be categorised as communities of relationship, communities of interest, communities of transaction, and communities of fantasy (Armstrong et al. 1996). The community that was created in this research was primarily a community of interest, in which people who are interested in the topic, ideas and creativity, can meet and exchange. A secondary goal was to establish a community of relationship because increased social interactions among users generate a ‘switching cost’ (Ginsburg et al. 2004) or barrier. Positive interpersonal relationships are viewed as positive switching barriers (Julander et al. 2003) because users would have to leave these interactions behind if they were to defect to another website.

The open source CMS PHP-Fusion was selected because it had several features like a forum, options to submit news and submit articles already implemented. The reason for using open-source software was that it is free to anyone to do modifications on the source code. PHP-Fusion is under the GNU Public Licence (1991) and therefore everyone is permitted to edit the source code.

The content for the site had to be interesting, unusual and related to ideas and innovations, therefore the weblog awards winner of 2004, engadget.com, which is a technology weblog and its sources, were used to gather news for the site. Using the format of weblogs as a news source had several advantages. The main advantage is that they are written by amateurs and not much effort is needed to create a weblog and therefore they can deal with much more specific content than traditional journalists who have to focus on attracting a crowd of people (Rosen 2003). The growing popularity of weblogs shows in a study that in 2004 the weblog readership increased by 58% in the US and 27% of ‘on-line Americans’ read weblogs (BBC 2005).

An example of a weblog that demonstrates the variety of content that is available is the weblog, ‘A star from Mosul’, in which a 16-year-old girl writes about her daily life in post-Saddam Iraq (Abdullah, 2005).

Attractive graphic design is a key element in making interfaces more enjoyable to use (Shneiderman 2004), which enhances the user experience. The importance of consistency (Brannon et al. 2000) and design is crucial for the user experience (Tedeschi 1999) (Lynch et al. 2004). Before the final design was developed several other possibilities were considered (refer to Appendix C). In order to unify the design (Graham 2001, p.84) a three-column grid was used (Graham 2001, p. 112). The decision to use ‘cold colours’ was made because of their relaxing effect on the viewer (Birren 1984, p81). Blue also stands for ‘a sense of balance, calm, and security’ (Lu, 2004), which are attributes that seemed appropriate, as ideas can be very personal (WorldNet, Definition ‘idea’). The minimal use of controls in the top navigation with the only options to choose ‘Home’ or ‘Forum’ emphasised the community features (Hearst 2005). A system for news categories was implemented which should become personalised in the beta phase. Additional fields to add links to news were implemented in the news administration to increase the functionality. To give guests a reason to become a member, the link and the download section were filled with content and became a members-only section. Furthermore, the news arena panel, with additional news for members, was implemented.

The initial concept was focused on creating an aesthetically designed website (sight experience), establishing an online community to provide users with the opportunity of a co-experience and to create switching barriers through social interactions. Furthermore, the website should motivate users to engage creatively (think experience) in one of the seven forum categories (refer to Appendix D). The main goal was to establish a loyal user base for the field study.

Figure 3: First version of the website.

The website which was created for this project has the title ‘Ideatemple’. It is in German and because it deals with ideas and creativity it focuses on a specific target audience.

The target audience consists of male and female adults between 18 and 30 years. They have completed higher education and understand German. Moreover, they are interested in innovation and are involved in a creative profession or hobby. They are extroverted, open minded, curious and tolerant people who like to socialise. If one would have to name a typical job it would be artist or, in the Myers-Briggs Type Indicator, the person would be an ENTP (Extravert, Intuitive, Thinking, Perceiving) type (Naomi 1999, p.17).