User Loyalty Articles

During the two cycles of the iterative design process the use of appropriate design that suits the topic, focus on the topic, delivery of content and the creation of a positive atmosphere were identified as the most essential aspects for a valuable user experience. The prerequisite for these aspects to become the most essential was that the lower levels of user needs, usability and availability, were fulfilled.

Four pages had personalisation features implemented, the start page, the news page, the idea-rating page and the start page of the tutorial. While the tutorial did not have enough visitors to be of interest in this study, the personalisation system of the idea-rating page lacked content and therefore could not work as planned. The findings show that the news page and the start page were better perceived by users that had the personalisation features enabled (B+users). The accuracy and coverage of the news page personalisation system could be confirmed. B+users perceived the website as more trustworthy, were more interested in new features of the site and were more likely to recommend the site. This indication of increased user loyalty was confirmed by data that showed that B+users were visiting the site more frequently. Therefore, the hypothesis, that it is possible by utilising dynamic personalisation to increase user loyalty to a site, is confirmed.

Discussion and future work

The findings after the first cycle of the iterative design process depicted that users demanded an appropriate visual design that suits the topic and the findings of the first survey that 23.4% of the users liked the site because of the atmosphere having parts, which overlapped from the visual design. This was shown by the descriptions of atmosphere in the second survey in which 41% of users related the atmosphere to the visual design. The reason that the visual design made an impact was because aesthetic fidelity was achieved.

After the first cycle of the iterative design process it became clear that the participation of users would be less than expected and the concept had to be changed. Even though, the site already had the main features such as a forum and the possibility to submit own articles, the understanding of the audience was missing, which is crucial for developing an online community (Preece et al. 2004).

Therefore the goal of the website shifted to providing news and knowledge about the topic. The development was concentrated on creating more focus on the topic by providing background information and on developing an appropriate visual design. The main problem at this stage of the process was the lack of contribution of users to the content of the site. The idea section should bring more focus on the topic and give users another possibility to add content while the tutorial section was developed to guide users through the process of idea creation.

Feedback from forums in which problems with the structure of the start page of the second version of the site were mentioned were addressed by a test in which the aesthetic aspect and the scanability was tested. The test, with participants who were new to the interface, was conducted with the assumption that if users of a forum who were new to the interface were able to identify problems with the structure of the start page, the participants would be able to do that as well. The findings suggested that a more structured start page would decrease the aesthetic aspect of the page and therefore have a negative impact on the user experience while the scanability was only slightly faster on a better-structured site. During the progress of this research, comments of long-term users about the lack of structure on the start page showed that the test did not produce valid results for long-term users. It confirmed Dillon’s et al. (2001) findings that ‘evaluations based on single experiences with technology do not provide stable estimates of long-term (or even medium-term) usability.”

Certain limitations of this research must be acknowledged. Due to the complexity of user experience (Olsen, 2003) the essential aspects for the user experience, which were found, might not apply in general, especially given the specific topic and the resulting target audience. The focus on the German speaking countries, and as a result, the cultural background of users might have had an impact on the findings as well.

Second, the period of time for the field study, two months, might also be a limitation. The short period of time did not allow to establish an online community of any size in which personalised user rankings and identity played a significant role. Even though, the descriptions of the atmosphere of long-term users confirmed that it was possible to establish a “sense of community ” (Blanchard et al. 2001).

Third, because the final survey relies on data from a small group of people, five b+ user and five b- user, their personal interests in the website could have played a role in their judgements.

The result that four out of five B+ user could name the same favourite news category as in their user profile, while only one of the five B- user could, suggests that personalisation helped to create personal preferences or helped to identify them. The knowledge of a certain preference could make a user more sensitive to changes on a site and therefore have a negative impact on their loyalty.

Furthermore, the delivery of personalised content could limit the users perception of the variety of information on the site and therefore have a negative long-term effect on loyalty. Thus, future researchers should examine whether that the findings of dynamic personalisation can have an impact on user loyalty is consistent over a longer period of time. The creation of a personalisation lifecycle, which begins by creating a user profile and ends in de-personalisation, could be the framework for this future research.

The results of the final survey show that B+user consider the website as more trustworthy. Similarity, or in the case of this research content similar to the users preference can increase trust. Fogg (2003) identified credibility, which consists of trust and expertise, as a key factor for persuasive webdesign. The findings suggest that the use of personalisation can enhance the impact of persuasive webdesign. While persuasive webdesign only played a role in establishing credibility of the website. Future research should focus on the impact that personalisation can have in combination with persuasive webdesign.

Start page

Even though in the overall category B+users rated the start page better than B-users, the categories of usability and importance were rated better by B-users, while structure was rated equally by both groups.

News page

The news page was rated overall and in structure and usability better by B+users, while B-users rated the importance slightly better.

Ideas page

B-users rated the ideas page as better in all four categories.

The survey further showed that all five B+users ‘always find something interesting in the news’, while B-users where less likely to find something of interest. B+users and B-users are both equally worried about their privacy on the Internet and B+users responded slightly more positive that the site was credible and personal. More B+users responded positively to the two questions that indicated their loyalty. B+users also were able to identify with the site slightly better. Expectations were equally met, even though the rating shows that they were not exceeded. While B-users rate the content of the site slightly better, B+users perceive the site as more trustworthy than do B-users.

The key findings are that pages which rely heavily on personalisation (start page, news page) are perceived better by B+users. The ideas page could not use the personalisation system that was implemented as it was planned because of its low amount of content (44 ideas at the time of the survey) distributed over 16 categories. Therefore user could not create profiles that represent their preferences.

The sum of all ratings of the first three sections show that the news page was the best-perceived section of the site followed by the start page and then the ideas page.

The impact of the news page personalisation on users is confirmed by the positive rating that B+ users ‘always find something interesting in the news’.

During the two months of the field study, the sum of all visits to the news page by users in both groups 396. The difference is that B+users visited 279 links to news while B-users visited only 159. This data confirms the statement that B+users were more likely to find something interesting in the news.

The effectiveness of the news page personalisation system was confirmed by the fact that four out of the five B+users named the same category of news as their favourite, according to their user profiles. An interesting aspect is that only one of the B-users named the category that was the highest ranked in their user profile. This could mean that the delivery of the most favoured news as the first news every time a B+user came back to the site has helped them to develop a concrete preference.

The fact that B+users could identify slightly better with the site and that the news page personalisation system delivered content similar to their preferences explains why the site is clearly more trustworthy in the eyes of B+users (Fogg, 2003, p. 123)

The importance of trust for loyalty (Koufaris et al.2002; Vatanasombut et al. 2004; Kim 2000) might have played a role in that more B+users responded positively to the two questions that indicated their loyalty to the site (Stratigos, pers. com. Feb. 19, 2005; Heater, pers. com. Feb. 19, 2005).

The evaluation of the question about how users would describe the atmosphere showed that the atmosphere was perceived as very positive.

Both questions, the question about the atmosphere and the question about what users miss on the site, were also used to evaluate certain aspects of the site.

Two B+users characterised the atmosphere of the site as ‘personal’ , two others said it was ‘dynamic’ and two said they enjoyed the feeling of being in a group, even though one mentioned that the start page should have a better structure. On the question of whether there was something missing, one B+user would like to see several improvements in the ideas section. Three B-users characterised the atmosphere as ‘pleasant’ and one emphasised that he enjoyed the community on the site.

On the question of whether something was missing, two B-users would like to see improvements in the ideas section while one would like to see more structure on the start page. A look at the results of the survey in general shows that the atmosphere on the site is related to the use of warm colours. Of 21 members who gave a comment on it, nine members mentioned the positive impact of the chosen colours on the atmosphere.

Five members said the atmosphere of the website was friendly, while four members said that the atmosphere was personal and they enjoyed the feeling of being in a community.

15 members responded to the question of whether there was something missing.

While eight members said they did not miss anything, five members would like to see improvements in the ideas section. One member said the start page was too unstructured, while two members mentioned this in the question about the atmosphere of the site. Therefore, this issue is not solved and will have to be addressed in the future.

An interesting finding is that users in the B+users and B-users group tended to associate atmosphere more with community and a personal touch, while other users, who used the site over a shorter time associated atmosphere more often with the impression they got from the colors and the visual design. Of all 10 members in the B+ and B- user group only one member mentioned the positive effect of colour.

For the general evaluation of tracking data only data from members who visited the site more than once was considered as valid. This was done to exclude the data of members who joined for the first survey and did not return.

The tracking data showed that B+users with two or more visits looked at an average of 1.47 news items each time they visited while B-users looked at 0.97 news items every visit. Over all, in the time from December 15 to February 15 B+users visited 439 news links and B-users 208. That was done during 299 visits of B+users and 215 by B-users (fig1). Over all, B+users were more likely to find something interesting in their news than B-users.

The run of the curve that shows the frequency of visits is similar between B+users and B-users, but a B+user was more likely to return after three visits.

On average, during these two months, B+users visited 28.09% times more often than B-users.

The second survey was conducted three weeks after the first survey.

The second-survey questions about the loyalty of users were derived from indices used in the industry (Heater, pers. com. Feb. 19, 2005; Stratigos, pers. com. Feb. 19, 2005). To understand how efficiently the personalisation system works one has to measure the accuracy and the coverage (Mobasher et al. 2001)(Sheperd et al. 2001). Accuracy measures how precisely the personalisation system can deliver content that the user likes and coverage measures how likely the user is to accept the content.

Questions about the credibility of the site were important because without credibility a website cannot ‘persuade users to change their attitudes or behaviours’ (Fogg 2003, p. 148).

Unlike the first survey, the second survey was not promoted, except on the site itself and through emails to members. This was done because this survey was designed for long-term users of the site.

Data from the five most active B+users and the five most active B-users who completed the survey were evaluated, because this group of users experienced the site over a longer period of time. Every participant was a site member for at least 15 days, had at least seven visits to the site and read at least six news items distributed on different visits. During the time they were members the sum of all visits of the B+user group was 166 while the B-users visited 152 times.

In the first section of the survey participants had to rate aspects of every site in which personalisation features were included. The part of the survey in which users had to rate the tutorial was not evaluated because an insufficient number of users did the tutorial.

Following the Software Usability Measurement Inventory (SUMI) (Kirakowski 1994), which is a method of measuring software quality from the user’s point of view, a survey was prepared. The goal was to get more detailed information about the user experience the site provided and some indications of what impact that had on loyalty. The questions about usability (q. no. 3, 4) were implemented to reassure that no usability problems existed, because these problems would have direct impact on user satisfaction (Froekjmr et al. 2000).

Furthermore, the survey had questions which could be categorised into satisfaction, loyalty, website characteristics and background information. The survey was promoted in several forums and one had to become a member and visit every personalised page at least twice on the website before the link to the survey became visible. 64 members participated in the survey but only 10 of these were site members before the survey was announced. Therefore, the overall findings of the survey represent an extended first impression. They could only deliver an indication of the impact of personalisation features because even though new members had to visit every site a second time, these features provide valuable results for users over a longer period of time.

The survey confirmed the previous findings that the site was perceived well by users. Overall users with personalisation features enabled rated the site slightly higher than users without personalisation features. Of all participants, 69.2% were in the age of the target audience and nearly the half (31 participants) were using the internet more than 15 hours every week and were therefore considered as internet-savvy.

The response to the question (q. no. 7) in which participants had to characterise the site showed that only 5.4% named attributes that were considered as not intended by the designer and therefore negative. 94,6% of the selected attributes described the site as affectionate, personal, pleasant, unusual, exciting, bizarre, structured, warm and beautiful. This shows that aesthetic fidelity has been achieved. A surprising finding was that 23.4% rated ‘Atmosphere’ below ‘Content’ with 39.1%, as the most liked aspect of the site (q. no. . This result is positive because atmosphere describes ‘an interesting, often exotic, effect’ (Webster’s New World Dictionary 1988) and is considered a general emotional quality (Webster’s New World Thesaurus 1997) that can have an impact on the mood of the viewer (The Hutchinson Encyclopaedia 2003). That shows that the site can make an impact on users’ feelings. Even though it does not show what atmosphere the site creates. The high rating on ‘Content’ is positive because content is the ‘classic way to increase loyalty on the web’ (Nielsen 1997, 1999) and a way to attract new customers to a site (Wurster et al. 1999). This high rating confirmed that the chosen content for the website was appropriate.

A negative finding was that there was a gap between the overall impression and the perceived value a user got from visiting the site (q. no. 1,2). Even though the ‘overall impression’ is a mix of various elements and ‘what the site provides’ can be understood as related to content, it showed that the site generated expectations which it did not completely meet. This was repeated in the two questions about usability of the site (q. no.3,4). The structure of the site was rated better than how the users rated how they could navigate on the site.

Despite these results, more than half of the participants voted either good or very good in all four questions.

The question (q. no. of whether a user would return to the site was also answered positively by 31.5% of participants who said they would definitely return and 57.4% who thought they would return. The data of their user profiles showed that 33.3% of members who said they would definitely return and 80.6% of users who said they thought they would return did not return and log in. Therefore this question seemed to be unnecessary in the second survey.

The survey showed that users could be considered as overall satisfied. The usability and structure of the site was well perceived and aesthetic fidelity has been achieved. Questions that needed further investigation were about the kind of atmosphere that the site creates and if the site meets the user’s expectations. Furthermore the impact of personalisation features on the site had to be examined in detail.

A poll, on the beta site in which 13 members participated, showed that 12 members or 92% liked the beta site more than the alpha site, while one member or 8% liked the site as much as the alpha site (refer to Appendix G).

The positive feedback from the forums and the poll showed that no fundamental changes of the visual design were necessary any more.

The second poll on the alpha site had showed a demand for background information about ideas. While the knowledge section already provided background information, an on-line tutorial for two creative techniques was implemented to add an act experience to the site (Schmitt 2004) and guide users through the idea creation process. This tutorial also had personalisation features on the first page. The amount of information that was shown was controlled by the average time a user spent looking at a page of the website. If a user spent less time than the average of the other users he did not see extended explanations for the tutorial. Furthermore, an example picture with slogans according to his interests was presented to increase his motivation.

Even so, the qualitative data that was gathered until this point in time were from people who did not visit the site over a period of time. The positive feedback indicated that the fundamental needs of users – availability and usability – were fulfilled, which means that the site had no fundamental problems that could harm the user experience. The quantitative data that was gathered from members of the site confirmed the issues described in the qualitative data. This could be because the site was promoted mainly in the forums where feedback was gathered. The visitor statistics and days users registered -suggested that many members came from these forums and therefore the qualitative and quantitative data were similar.

After the second version of the site was launched and promoted in forums, 27 people gave feedback. The main problem with the alpha site was that the design was inappropriate for the topic. The feedback from the second version of the site showed that this issue was solved. While in the feedback for the alpha site only five people said they liked the design, the beta site feedback had 12 people with positive comments about the design. Only one person said that he did not like the design.

In the feedback (refer to Appendix F), two issues were named by several people. Five people said that the home page needed a better structure and two people said that the length of the sentences decreased the readability. The problem with the readability was an obvious issue and was fixed immediately by limiting the text length of every sentence. A test , based on Dillon’s et al. (2001) testing methodology for aesthetics and usability, was conducted to determine how fast a home page with more structure could be scanned and what impact this version had on the impression participants got from the design of the site. Four versions of the home page were used to do that. The test was conducted on a computer and the six participants chosen were not regular users of the website and therefore had no routine in navigating the layout.

Every participant was tested separately. In the first step participants had to give a rating on the look of the home page in its original form and in the more structured form. Both versions that were used in this step had news in a different order than in the next step. The design was rated on a 10-point scale with 10 as the best. In the second step, two versions with different news were used. The participants had to find a certain word in the headlines of both versions while the time was clocked by the instructor who made a note of how long it took them to find the word and point to it. The word they had to find was at the same location in both versions. Every second participant received the two versions in reverse order to assure valid findings, because participants might compare the second version to the first when they rated the design or might have already some routine to scan the layout when they had to find the word in the second version of the second step.

The design of the structured version was rated 14.85% – worse than the design of the unstructured version – while it was only scanned 2.03% faster (refer to Appendix L). This result was expected even though the small difference in the time it took a participant to scan both versions were an unexpected finding. The minimal difference in time it took users to scan both versions would not justify a change of the site design and therefore a decrease of the aesthetic fidelity (Norman 2002), which is “the degree to which users feel the target impression intended by designers who developed the web pages” (Park et al. 2004).

Due to the low amount of user participation and, as a result, the relatively small amount of content, it was decided to use content personalisation in this research. Collaborative filtering would have been the wrong choice as it finds a profile of another user that is most similar to the users, but without many members it is unlikely to find peers that are similar enough to provide valuable results (Adomavicius et al. 2003). Furthermore, the news section has only a small amount of new information every day, which reduces the options for success of collaborative filtering even more. Another reason for using content personalisation were the findings of Shepherd et al. (2001) that users do not enjoy too detailed information filtering, therefore the news is only filtered according to news categories.

The personalisation features were implemented in the start page, the news page and the page on which the user rates ideas. Every member was identified by his or her login and password. Because A/B testing was used, every second member that registered surfed the site with personalisation features enabled (B+user) while the rest surfed the site without them (B-user).

The implicit data for the personalisation system derived from predefined PHP variables (PHP Group, 2003), which save user data that is sent to the server, and the tracking of users’ browsing behaviour. The system is based on a user profile in which all data is saved and every site that has personalised content on it is controlled by the user profile. The user profile consists of data for identification (Identification Profile), system data such as ip address or id (System Profile), self-categorised data such as age or gender (Socio-economic Profile) and data about the actions of the user on the site (Interaction Profile)(Schubert et al. 2000)(refer to Appendix J). Every time a user loads a page his user profile is updated in the appropriate category of interest. The start page is excluded because it shows all categories of content in five modules and therefore does not represent a specific interest. Over a period of time a profile of the users’ interest in the site is created. According to this profile the ratio of each content category of the start page is adjusted. If one category was not visited yet a link is shown with a call to visit this category while B-users did not see this feature. [P4] This is done to create awareness of existing features on the site and thereby increase the user experience (McMullin 2004). Furthermore, the start page shows special features and articles according to the operating system of the user and every user is greeted according to the time of the day and the time he has not[P5] visited the site. (pic1)

On the news page the news posts are sorted by starting with news from the category the user preferred most in the past. B-users see the news items according to the order in which they were published. The news is categorised according to the five news groups that were determined in the user-perception test. These five groups have subgroups with more specific topics (refer to Appendix I). When users click on the link to the news source their profile is updated in the corresponding category. To prevent the profile being updated several times when the news has more than one link, the user profile logs every news item a user has already read and compares this data with the incoming data from every update. Only the news that was posted since the user’s last visit or in the past six hours is sorted according to the user’s profile. This is done because changing all news in real time could confuse a user and harm the user experience. To sort the news the ratio of the viewed news of a user to the total amount of news in this category was calculated. This had to be done because not every category had the same amount of news and therefore the pure data in the user profile did not have to represent the user’s interest.

The user rank was also derived from their activity on the site and favourite news category. The activity is the sum of all values of the content categories in the user profile. (pic user profile structure)

The user rank feature is the only personalisation feature that uses explicit data to a minimal degree. The gender that users assign to themselves is used to give the title the right grammatical form.

Ideas are also categorised (refer to Appendix K). On the idea-rating page a B+ user will see an idea from a category in which he rated an idea before if he did not already rate all ideas of this category. These 100%, which represent all ratings he made, are divided according to the sum of all ratings in each category. Therefore if a user rates ideas from a particular [EB6] category highly, it is more likely that they will get more ideas from this category to rate next. In this personalisation feature no ratio is necessary because the ratings control the level of interest.

The personalisation features have multiple goals. In the short term they should provide users with a better user experience on the website. The long-term goal is to give the website a personal touch and thereby establish an emotional bond between the user and the website, which results in increased loyalty.

Following these findings the focus of the next version of the website shifted from establishing an online community to becoming a source for news and knowledge about the topic ideas and creativity. Therefore, the experience design focused on enhancing the usefulness of the news section and the visual design was redone to give the website a more desirable look. The low amount of user participation also influenced the choice of the personalisation systems which were used. To focus more on the topic of the web site and to satisfy the demand for background information the knowledge and the idea section were added.

The first step to enhance the news section was to give users another option to see the latest news. Therefore an RSS-feed was implemented. Furthermore, a user-perception test, with different icons for news categories, was conducted (Syarief et al. n. d.),. Eight participants in the target audience had to group icons in four groups and then assign category names to each of them. Every test was conducted separately. The evaluation showed that the category icons for the categories Design and Technology were inappropriate. Furthermore, two comments were made that the icon of the category Mobile ‘looks like an oven’, indicating that this category icon should also be changed. Another finding was that instead of four groups as was planned, five groups were better for the news categories. In a repeat test all of the three participants identified all icons properly (refer to Appendix H). A new idea section in which user can rate ideas or submit their own ideas offered a more concrete possibility than the forum for user to engage creatively.

To increase the credibility of the website, a privacy policy according to the ‘Online-Datenschutz-Prinzipien (transl: Online-Privacy-Principles) and an ‘Impressum’, which is a duty for commercial sites and recommended for non-profit sites under the German law, were written (Teledienstgesetz 2001, §6). Furthermore, detailed information about the background and people who run the site was added (Fogg, 2003, p.173). Content for the knowledge section and gallery was added and the news administration section was enhanced, making it easier to include pictures. Moreover, a section with a new citation and a new picture every day was added to the start page to give users another reason, besides the daily news, to visit the site on a regular basis.

The visual design of the beta site was build in contrast to the alpha site, using a spectrum of harmonic warm colours (Crüger 2004) with strong hues to make the site look exciting and active (Mahnke 1996, p.71). Another reason for this choice of colour was that warm colours increase brain activity (Birren 1984, p. 80) and Birren claims that extroverted people prefer warm colours over cold colours (Birren 1984, p. 119). The monk in the banner should give the site a face and communicate happiness. Faces are good ways to communicate emotions because these emotions are understood by everyone regardless of their background (Morand, 2001) and faces play a ‘major role in human interaction and nonverbal communication’ (Lien 1998). To establish a better brand recognition a logo was created. The first letter of idea symbolises a temple and the Asian font Wonton, together with the monk, give the site an Asian touch. The angular font for ‘temple’ is a strong contrast in shape to ‘idea’ and symbolises a foundation for ideas. The playful Asian font suits the topic ideas because ideas are not tangible (Boas 2003).The layout was changed from three-column to two-column layout because the site should focus more on the usefulness of its main features, which would be the news and the ideas section. An example of successful design for usefulness is the search engine Google. Because of its focus on usefulness it is one of the leaders in providing a valuable user experience (Hurst 2002).

After the site was put on line it was promoted in several forums(refer to Appendix E) which were selected according to the target audience. In the forums a thread was opened in the appropriate subcategory starting with a short description of the site and a call for feedback about the site.

28 people responded with their opinion about the site (refer to Appendix F). The first and most important finding was that the concept was considered a ‘good concept’ or ‘unusual’, which are both positive outcomes. The second finding was that five people stated that they like the design, but also five people said that it was not appropriate for the content of the site. Even though only five people supported this point of view, it was the only issue, besides the positive feedback, that was supported by several people.

The first poll in which 17 members participated, confirmed that the site was well received (refer to Appendix G). The second poll, in which 15 members participated, showed a demand for background information about ideas and that the site should deal more with the subject ‘ideas’ overall (7 votes). The second poll also confirmed that the design of the site did not suit the subject (6 votes).

Even though the majority of the feedback was not very detailed, the findings indicated that the visual design of the site was considered inappropriate. The site should also become more focussed on the topic ‘ideas’ by providing background information.

Despite the positive feedback, the number of members increased more slowly than expected.

First time visits to the site reached their peak usually on the day or the day after a forum thread was opened (refer to Appendix E).

A logfile analysis showed that the most visited page of the alpha site was the start page, which was also the news page. It was discovered that the news was the reason for users to visit the site again. That the news was the most popular part of the site was confirmed in November when the go.php page was implemented. This page decrypts encrypted links to articles that were posted in the news and became the most visited in the site.