INFORMATION SYSTEMS IN MANAGEMENT
Information Systems in Management (2013) Vol. 2 (2) 146−158
CHATBOTS FOR CUSTOMER SERVICE ON HOTELS’ WEBSITES MIROSŁAWA LASEK a), SZYMON JESSA b) a)
Department of Information Systems and Economic Analysis, Faculty of Economic Sciences, University of Warsaw b) Denise Systems sp. z o.o.
In this article we present an analysis of implementations of a chatbot - a program which simulates an intelligent conversation with webpage visitors, dedicated to hotels and guesthouses (hotel chatbot, in short: HC). We obtained unique data from five various webpages exhibiting various configurations, containing a total of 17413 user statements in 4165 conversations. HC informative function was confirmed for more than 56% of the conversations. Moreover, 63% of users prefer to interact with HC if it suggests at least one clickable option to choose as an alternative to typing. The results indicate that the implementation of speech synthesis increases the percentage of users who decide to start a conversation with the HC and it may have a positive impact on the percentage of users that book rooms online. Keywords: Reservation Systems, Hotel Chatbots, Hotels' Websites, Touristic Services, Human - Computer Interaction, Economic Value of Chatbots
1. Introduction The term chatbot relates to a computer application with which users can conduct a dialogue in natural language as if it was conducted with another person . The most widely described prototype program, which falls into this category, is Eliza, which provided “the responses of a nondirectional psychotherapist in an initial psychiatric interview” [29, 30]. Eliza's success is often explained by the fact that users unequivocally anthropomorphized and somehow set up a relationship with the program, often highly emotional . The term "Eliza effect" describes
this human tendency to assign chatbots the attributes of human intelligence . Nevertheless, the truth is that Eliza and most chatbots use various tricks to simulate intelligent behavior and to make an impression of a speaking human being. This goal distinguishes it from programs that also use natural language interfaces, but have more advanced algorithms used for text analysis and reasoning, while the aspect of human-like naturalness of communication is marginalized . Tricks used in chatbots are based on algorithms that use textual pattern matching rules and dialogue management rules . Perhaps the most surprising fact is that these simple methods can induce “powerful delusional thinking in quite normal people” . It may be due to the fact that in the history of mankind speaking concerned only the human species, although the idea of speaking with other beings has accompanied people since the dawn of history and finds its expression in literature and religions . Despite this, we unknowingly anthropomorphize computers and treat them as social actors, though to a limited extent . Chatbots are looking for their place in e-business . Some of them are used on websites as shop assistants , helping to choose products from the company's offer and causing company’s sales growth. In some cases, chatbot authors report a 18% increase in purchases, and others indicate that 15% of people who chatted with sales assistants made purchases . Chatbots are designed to assist users in navigation on the site, to limit the amount of clicks and to shorten the time to reach the desired information or product. Chatbots also give other benefits, in terms of building social relationships with customers, increasing customer confidence in the company and strengthening a customer's emotional bond with the company [8, 17, 23]. More and more industries relocate their marketing, sales and maintenance services to the Internet. One of them is the hotel industry, where it is estimated that 75% of purchases are made through on-line booking systems . A new tool that supports the hotel industry is a chatbot (HC) offered by Denise Systems. It aims to meet the rigorous demands of the mass market: HC implementation in the new hotel must be made as simple and automatic as possible. In result, the implementation process is reduced to the following: the hotel owner gives details on the form, chooses the chatbot look, specifies its location on the screen and pastes it on the website. HC has been programmed in order to perform specific business goals. It is not a human-oriented simulation and it can instruct the user about the scope of its functions. HC does not aim to win the Loebner , it is solely a targeted marketing tool. Knowledge base was limited to the hotel industry and tourism topics, therefore it does not contain elements of so-called "small talk". HC gives more information about the hotel than a user can find on a website, it helps users to navigate the site, promotes the building and surroundings of the hotel and gathers valuable marketing data from users. In this article we present an analysis of conversations between users and HC. We carried out our analyses in such a way that the results can be compared with Max – male chatbot at Heinz Nixdorf Museums Forum in Paderborn (Germany),
and partly with the results concerning other chatbots: Sgt. Blackwell (male chatbot at Cooper Hewitt Museum in New York), Talk-Bot (robot chatbot available online), Bill (male chatbot available online) and Kathy (female chatbot available online). 2. Database of conversations Data were collected from 4 websites of guesthouses (A, B) and hotels (D, E) that implemented their own HC, and from the HC demonstrational website of a fictional hotel (website C). All implementations are almost identical: the knowledge base is in Polish language and differs in ca. 100 responses about the hotel, which represent approximately 7% of all responses in the HC knowledge base. Location of the HC on the webpage and its appearance may vary from one implementation to another. All conversations were recorded with date, time, user input and HC answers. Logs do not contain information whether the user input has been written or selected by clicking an option. If user input corresponded to the expression accessible in option, it was assumed that the option was clicked. Inputs were tagged with categories by the most recent HC version available when writing this article. However even the latest HC was unable to find any suitable category for 1221 (10%) inputs (for comparison: Max was unable to recognize 25% of inputs ). Among these 1221 inputs, 881 contained unique content, of which ca. 300 (1.7%) were assumed random keystrokes (in Max: 3.1% ). Among others, some inputs were in English and Russian, there were also system commands, URLs, typos and spelling errors. There was also a very small group of expressions which indicated deficiencies in the knowledge base, for instance: several questions about maids, virtual tour, room sizes, vouchers, possible discount negotiations, availability in a given date, job/internship offers, insults, vulgar behavior, sexual comments and orders given to the system (like “raise your hand” etc.). In total, the database analysed contains 17413 user inputs, of which 12126 were written, and 5287 were selected by clicking an option. Of all the user responses only 5957 expressions contained unique content: 4955 expressions were used only once and 1002 were repeated (12458 times). If the system is able to interpret the 17% (1002) of inputs repeated during conversations, it will thus be able to interpret 70% of all users responses. This indicates that a chatbot designed for a particular field with limited topics and questions range doesn’t need to have a large knowledge base, and in most situations it will work properly. For comparison, the unique content chatbot responses were 1341, of which 351 were used once and 990 were repeated. The system uses a limited number of responses, therefore it was tested for repetitions that occurred during one conversation. We observed that: 2633 (63%) conversations contained repeated expressions, and 1612 (39%) conversations had more than one repetition of an expression. Repeated statements were concerning: prices 148
presentation, standard rooms descriptions, descriptions of single rooms and double rooms, room equipment and responses in situations when the HC was unable to determine the category for user input. Average user input length was 11 characters, and for HC answer – 80 characters (for comparison: on Polish IRC network, most responses consist of 10 characters). 18% of all registered responses were threecharacter expressions, mainly “yes” – 778 occurrences, and “no” – 777 occurrences. Similar results concern the IRC network, where the words “no”, “it” and “yes” are the most popular as a one-word expressions . The fact that chatbot responses are much longer in comparison to the user responses, was observed as well by Max authors . 3. Conversation length and duration Conversations were conducted via a web browser, and users were identified by cookies. Conversation duration is counted from the first user’s expression to the last statement of a chatbot. The pause between the user’s statements cannot be longer than 15 minutes – if the same user re-enters anything after 15 minutes, a new conversation start is assumed. The average conversation length: 4.2 user inputs, 118 seconds. On average, the longest conversations observed on website C – 7.7 user inputs per conversation and about 6 minutes (368 seconds) average duration. It may result from the fact that HC on this website is a demonstrational implementation, which serves for testing and presentation of the HC itself. Note, that chatbot on website D has few conversations, therefore the averages may significantly change in the future (this also might impact the standard deviation of number of user inputs, greater than in other implementations). It is interesting that the website A has mainly very short conversations. This may be related to the construction of this webpage, because HC is placed on top, and almost all of its pages requires vertical scrolling, which causes the HC disappears from the screen. It may be the main cause of short talks on website A. In comparison with other chatbots: (1) Sgt. Blackwell – 4 inputs/conversation , (2) Max – 22.60 inputs/conversation, duration: 13 minutes [16, 20], (3) Extempo Sales Assistant – 15 inputs/conversation, duration: 12 minutes (in 90% of conversations) , (4) TalkBot - 22.67 inputs/conversation , (5) Bill – 24.88 inputs/conversation , (6) Kathy – 31.63 inputs/conversation . Conversations with HC are relatively short in comparison with other chatbots. One reason for the discrepancy may be that the HC is only an addition to the hotel website, and other compared chatbots stand mostly in the foreground. Looking at the website C, where the HC appears in foreground, it may be assumed that the role of the chatbot – in foreground or in background, affects the length of the conversation. Another important reason may be that other chatbots often have a very extensive knowledge base for small talks which is not present in HC. Despite this, we registered 44 (1%) conversations 149
which were longer than 0.5 hour. Sheryl Brahnam has examined that conversations conducted with the female chatbot Kathy are longer than those conducted with male chatbot Bill and Talk-Bot robot. On the website D we observed the opposite – conversations are longer than those on A, B and E webpages, which use the female chatbot. Only the implementation of webpage C has longer conversations, but its chatbot can be female or male. Max authors think, that the online conversations will be longer than those conducted with Max, because “interlocutors were probably sitting in front of their computers having a private chat. (…) In the museum, the users are standing in front of Max and his answers are spoken out loud” . 4. Typing and pointing HC gives the user the choice of entering a reply, or selecting an answer by clicking it from displayed options. In this way, the HC is getting closer to the realization of the idea that “the best way to facilitate Human Computer Interaction is by allowing users <>” . From the total 6607 clickable responses displayed, users clicked 2670. Additionally, it was verified that 2617 (63%) talks were initiated by clicking an option displayed in the HC welcome message. This indicates that users are much more likely to interact with a chatbot when they can click a prepared response. Users who chose to click, are more likely to continue the conversation, if the next response also gives them this option of communication. Users who chose to click, more often abandoned their conversation than users who chose to write. Users who chose to write, often abandoned conversations, if a chatbot displayed them options to click. Conclusions are as follows: (1) specifying the option to click increases the chance that users will interact with the chatbot, probably because clicking requires usage of the mouse and clicks, and therefore is easier to do than typing in the text; (2) chatbot should adapt its interface to the user, and if the user prefers to click, then chatbot should guarantee him another option, and if the user prefers to write, then chatbot should restrict displayed options; (3) ultimately, the options should be attractive enough to draw the user into the conversation and convince the potential customer to write, which is even more engaging than clicking. There are two categories of statements, which more often resulted in clicking: (1) questions such as where did the user find the information about the hotel: internet, media, friends; (2) presentation of pricing and room choice request: single room, double room etc. A huge preference to click the option in the short statements containing maximum three options was observed. This may indicate that users prefer options over which they do not have to think long and do not need to read or analyze the context of the whole expression.
5. One-input conversations analysis We registered many conversations, which were very short: 1508 (36%) conversations consist of only one user input and 2506 (60%) conversations had no more than two user inputs. The best results are on website C, where only 34% of conversations last no more than two user inputs (for comparison: Max – 47% - no more than 2 user inputs ). Among all implementations, the least successful is website A, where nearly half of the conversations consist of only one user’s input. The reason for this may be the previously mentioned need for vertical scrolling to view the content of the page – prices, booking form, or gallery, and it hides the HC. It is also interesting that as many as 24% of users engage in conversation with the implementation of the test page (website C) lasting at least 10 user inputs. In the case where the first input results from a click, and the HC response does not give the subsequent option to click, 38% talks were continued and 62% were ended. It means, that 789 users did not decide to go on writing, when the HC didn’t display any further clickable options, and that’s the case in 52% of all one-input conversations. In total there were 255 responses (used 1508 times!), however subsequent responses contain much less occurrences. Six most common responses contain over 50% of the endings. These are also the most common expressions occurring as the first HC response. Responses without options causes frequent resignations from the conversation. The option scheme is proposed to continue discussions in a certain direction. If users do not have their own idea, they will often follow the chatbot direction . And if a chatbot does not propose an option, the conversation ends just before the user writes anything. 6. Conversation content analysis One of the tasks of the HC is to provide answers to questions concerning the hotel’s offer. We examined the extent to which all users enjoy the chatbot as a tool for obtaining information about the hotel. To this purpose, we have counted users questions of a given category and conversations in which at least one user utterance was a question about the hotel/products/attractions. While analyzing a single response, we ignored the context of conversation (for instance the user answered “yes” or clicked an option to access information about the hotel without actually writing the question). The results should be interpreted that at least 56% of the conversations contained users questions about the hotel and/or its offer. This kind of queries constituted 7.2%, and for comparison – Max received 2.3% of queries concerning the museum. In addition, we tested the interest of users towards the chatbot. It turns out that 12% of the conversations include questions of users concerning the chatbot. The number of such queries is a total of 7.2% (1260). For comparison, Max received 14.6% of such queries . This means that users 151
are also interested in knowing the HC, though to a lesser extent than Max. This is probably due to the fact that HC preferred and suggested topics related to the hotel and refused to talk about unrelated topics. We deliberately distinguished “who is it” category from “what is it”, because the first query assumes a person, and the other – a thing. Majority of users perceived chatbot as a person, which may be related to “Eliza effect” . This effect of anthropomorphism can have negative consequences and “generate strong negative reactions from the part of the user” . Further part of this publication explores these issues deeply. 7. Human-Chatbot relation The creators of Max noticed that people are likely to use human-like communication strategies of beginning/ending conversations like greeting and farewell . We examined whether users greet or say farewell to the HC. Both Max and HC greet the user, therefore, according to the authors of the Max, greetings can be triggered by the greeting of the agent. Results of other chatbots: (1) Max - greetings – 57.6% conversations , (2) Max - farewell – 29.8% conversations , (3) Sgt. Blackwell - greetings – 5.6% , (4) Sgt. Blackwell - closing – 1.8% . These results show that HC has more greetings than Sgt. Blackwell and similar amount of farewells. Max has the biggest number of greetings and farewells. Among all implementations, the biggest number of greetings - 31% of conversations with the greetings, had implementations C and D. In addition, we analyzed the users statements concerning their relation to HC - whether they liked it (e.g.: "I like you", "You're cool," "You are very pretty!") or evaluated negatively its characteristics (e.g.: "but you're ugly"). The evaluations ignored behavior regarding sex and vulgar and offensive behaviors that are examined later in this publication. However, statements evaluating the system as stupid or bad (e.g.: "You are stupid") were taken into account. Altogether, statements including positive and negative evaluation constitute about 1%, which is two times less than reported by the Max authors (i.e. 2%) . However, for Max the ratio of statements of positive evaluation to negative evaluation is 1.6:1 (51:32) , the HC's ratio is 1.3:1 (100:79). In addition, the amount of conversations has been measured, which contain an evaluation. Most conversations with assessments occurred in implementations C and D, which have also the longest conversations. This indicates that the assessment takes place in subsequent stages of the conversation and it takes time for user to begin to express his assessment. The most positive evaluation concerned the appearance and in second place – the liking. On the other hand, it was the intelligence of a chatbot that was assessed negatively the most often (e.g.: "you are stupid") and on second place - its appearance.
8. Abuse and sex talks In a society, the aim of abuse is to cause some form of suffering to the recipient. Although chatbots do not feel, they constitute the aim of such attacks [4, 9]. We have observed that people are much easier to behave in this way in relation to the robot/computer/chatbots than to other people. Especially when user recognizes that he is talking to chatbot, and not the real man or woman – then clearly changes the way of behaving towards chatbot . It is possible that this is a mere curiosity of users and their desire to test the chatbot. Often this is also the behavior of users represented by the minors who would otherwise never have dared to expose adults at anything so offensive . This could be also explained by the human tendency to dominance and being “rude” which reflects a relationship in which users want to be – as a higher race, where the user (man) is the master, and the chatbot (computer/robot), the slave . HC politely guides users on topics related to the offer of the hotel, and if necessary, it explains its limits and explains in what it can be helpful. The goal is to raise users’ awareness of the possibilities of using HC. By engaging users in conversation, HC disperse their potential aggression and improve the quality of the conversation with the chatbot. This seems to be a good approach to be used in chatbots . Some researchers complain that the negative reactions of users are ignored in the literature concerning chatbots . To increase the research value of this publication, we carried out the analysis in this direction. Assuming that this category of statements contains vulgar, indecent and insulting vocabulary addressed to the chatbot (e.g. “I do not know this, and you still tell me that you fucking know it”, “what the fuck are you telling me?”, “fuck off,” etc.), we measured the frequency of appearing such a vocabulary in conversations with the HC. It turned out that the numbers are relatively low – i.e. only 2.3% of all statements gathered in 4% of all conversations contained vulgar vocabulary. Most verbal abuses were observed in implementation D – 5.8%, then C – 3%. The cause may be similar as in the case of evaluations of the system – conversations lasting longer encourage users to go beyond the main functions of the chatbot. Report on the chatbot Sgt. Blackwell confirms it, because it also has short conversations (4 inputs/conversation) and only 3% of the inputs with abuses. Another reason may be the form of video chatbot – the only male character is implemented on the webpage D. For comparison, in other chatbots the percentage of such statements of this nature is higher and ranges from 3% to almost 7%: (1) Sgt. Blackwell – 3% inputs , (2) Max – 5.4% inputs , (3) Kathy – 6.72% inputs , (4) Bill – 6.50% inputs , (5) Talk-Bot – 6.01% inputs . Chatbots certainly must be designed to deal with abuse caused by users. Their success may depend greatly on their ability to handle users’ verbal abuse . Chatbots also offer the possibility of trying various scenarios of conversations and behaviors, including those relating to sexual behavior. The most important thing is that all these attempts are entirely
safe, do not hurt anyone and are often the only opportunity to experience a variety of roles for the user. This causes the appearance of a series of conversations concerning sexual expressions . Statements containing erotic and pornographic expressions, included those on sexual proposals addressed to the chatbot (e.g.: “shall we shag now?”, “Let’s meet for sex”) or other vocabulary related to sex (e.g.: “orgasm”, “show spa and tits”, “are you gay?”). Percentage of statements of this nature ranges from 0.5% in Max  (which is publicly available at the museum and you have to speak in public to communicate with Max) to more than 18% for female online chatbot Kathy (the others: Bill – 9.76% inputs, Talk-Bot – 2.49% of inputs ). Sexual expressions addressed to HC measured only 1.8%. This may indicate that the context of a real company, a hotel, which perhaps will be visited by a user, has a greater influence on the polite behavior among users. Another explanation may be the chatbot construction assuming always returning to the main topic concerning hotel offer and not allowing for small talks, as it is a case for other chatbots. In addition, the results do not confirm unambiguously the assumption that more conversations containing sexual expressions concerned chatbots with female embodiment . It is true that most conversations about sex were observed on webpage C – 7% and this is the female chatbot, nevertheless the male chatbot on webpage D experienced almost the same number of statements concerning sexual expressions. To sum up, we would like to present the opinion of Peter Wallis, who writes: „the problem is not to make a machine that is accepted, but to make it behave itself once it is accepted as an actor in the appropriate social context. (…) Abuse is the fore runner to actual harmful action and as such leaves space for individuals to change their anti-social behaviour. Whereas humans and puppies are hardwired to know what these second order behaviours mean, Aibo’s and chatbots need to be told. This is the challenge, I believe, that stands between us and the creation of effective human-machine conversation.” . 9. Economic value It is very difficult to measure the economic value of chatbots. One of the methods of measuring their value in business is an analysis of changes in the sales in online stores. For hotels, one can measure the change in the percentage of webpage visitors that booked rooms online. As the tourist market is a seasonal market, the data should be compared to the year in which the chatbot was not present on the website. Unfortunately, obtaining such data is difficult and takes time. Due to the lack of access to necessary data of all hotels, we were able to examine the economic importance of only one HC. We checked the conversion rate, which is the amount of webpage visitors compared to customers that made bookings online through special form. Additionally, we also examined what was the effect of implementation of the speech synthesis for the chatbot. We compared the periods 154
before the implementation of speech synthesis and after its implementation. The results show that in 2011, when the HC was installed on the webpage, conversion was substantially higher than a year earlier. This applies to both the chatbot without speech synthesizer (in the period March-April) and with speech synthesizer (in the period May-August). The measurements were made on this website, which had a relatively small number of visitors, therefore the results should be treated with reserve. This kind of experiment should be conducted on a wider scale to measure accurately the effects of HC on the amount of reservations. Nevertheless, it seems that using the HC may cause increase in revenues, and such a hypothesis is definitely worth further investigation. Examining the impact of the speech synthesizer in the period from May to August, it turned out that after the implementation of speech synthesis, conversations parameters didn’t change – the number a user statements in one conversation remained unchanged (the difference was at 0.03 level of expression). However, the percentage of visitors that started a conversation with a chatbot also increased by about 5%. And this indicates that the speech synthesizer encourages users to start a conversation, but has no significant impact on the conversation parameters. If, indeed, speech synthesizer has only such an impact, it could be applied only to the first welcome message of the chatbot. Moving on – the synthesizer could be easily replaced by any recording, the aim of which would be to attract attention and inspire visitors to start a conversation with HC. These aspects of the implementation of the HC should be explored in future. As a tool for providing information to users, the HC supports the hotel functioning in the field of customer service . On the other hand, analysis of conversations with HC allows owners to know the needs of customers and their opinions on the object. During its functioning, the HC allowed hotel owners to detect several gaps on their webpages, e.g.: the distance to the beach and airport, availability of towels; it also pointed out that users were mostly interested in double rooms and collected a number of opinions on the webpage and hotel. Some of the gathered information is difficult to obtain in any other way. One of the unique HC features is that it can talk with all visitors simultaneously, and even if they do not execute the purchase, it will store valuable information – why they didn’t choose the offer. From a business perspective, this information is priceless. 10. Summary In science it often happens that the findings are pursuing other findings, and the whole world waits long for their implementation in everyday life. Thanks to our analysis of HC we have the opportunity to provide an additional amount of data from the implementation of chatbots in the "real world" for several deployments simultaneously. The results show that the HC fills its function of an information tool. In addition, increasing the percentage of booking online in one of the hotels 155
 Brahnam S. (2005) Strategies for handling customer abuse of ECAs, in Proceedings of the INTERACT 2005 workshop Abuse: The darker side of Human-Computer Interaction, Rome, Italy, pp. 62-67.  Bogdanovych A., Simoff S., Sierra C., Berger H. (2005) Implicit training of virtual shopping assistants in 3D electronic institutions, in Proceedings of the IADIS International Conference: e-Commerce 2005, IADIS Press, Portugal, pp. 50-57.  Chai J., Budzikowska M., Horvath V., Nicolov N., Kambhatla N., Zadrozny W. (2001) Natural Language Sales Assistant - A Web-Based Dialog System for Online Sales, in Proceedings of the 13th Innovative Applications of Artificial Intelligence Conference, IAAI’01, Seattle, WA, pp. 19-26.  De Angeli A. (2006) On Verbal Abuse Towards Chatterbots, in Proceedings of the CHI 2006 workshop on Misuse and Abuse of Interactive Technologies, Montreal, Quebec, Canada.  De Angeli A., Brahnam S. (2008) I hate you! Disinhibition with virtual partners, Interacting with Computers, 20(3), pp. 302-310.  De Angeli A., Carpenter R. (2005) Stupid computer! Abuse and social identities, in Proceedings of the INTERACT 2005 workshop Abuse: The darker side of HumanComputer Interaction, Rome, Italy.  De Angeli A., Johnson G. I., Coventry L. (2001) The unfriendly user: exploring social reactions to chatterbots, in Proceedings of the International Conference on Affective Human Factor Design, London, pp. 467-474.  Hughes L. (2006) The Eliza Effect: Conversational Agents and Cognition, available at: http://www.laurahughes.com/art/elizaeffect.pdf (accessed 27 September 2011).  Jessa Sz. (2004) Czy chatterboty nas rozumieją?, Software 2.0 Extra, 10/2004, pp. 16-20, available at: http://sdjournal.org/magazine/1251-sztuczna-inteligencja.  Jessa Sz., Jędruch W. (2010) Przetwarzanie wyrażeń języka naturalnego w wyrażenia logiczne - system Denise, in Przedsięwzięcia i usługi informacyjne. Praca zbiorowa Katedry Architektury Systemów Komputerowych KASKBOOK, (Ontologie w opisie scenariuszy usług), Gdańsk, pp. 75-87.  Kuligowska K. (2010) Koszty i korzyści implementacji wirtualnych asystentów w przedsiębiorstwach oraz ich znaczenie dla rozwoju gospodarki elektronicznej, rozprawa doktorska, Wydział Nauk Ekonomicznych Uniwersytetu Warszawskiego, Warszawa.  Kuligowska K., Lasek M. (2011) Virtual assistants support customer relations and business processes, The 10th International Conference on Information Management, Gdańsk.  Kopp S., Gesellensetter L., Krämer N., Wachsmuth I. (2004) A conversational agent as museum guide - design and evaluation of a real-world application, in Proceedings of Intelligent Virtual Agents (IVA 2005), Berlin, Germany, Volume 3661, pp. 329-343.
 Loebner Prize, (2011), available at: http://www.loebner.net/Prizef/loebner-prize.html (accessed 27 September 2011).  Mewes D., Heloir A. (2009) The Uncanny Valley, available at: http://embots.dfki.de/doc/seminar_ss09/writeup%20uncanny%20valley.pdf (accessed 27 September 2011).  Nass C., Steuer J., Tauber E. (1994) Computers are social actors. Human Factors in Computing Systems, in CHI ´94 Conference Proceedings, New York, pp. 72-78.  Pfeiffer T., Liguda C., Wachsmuth I., Stein S. (2011) Living with a Virtual Agent: Seven Years with an Embodied Conversational Agent at the Heinz Nixdorf MuseumsForum, in: S. Barbieri , K. Scott, & L. Ciolfi (eds.), Proceedings of the ReThinking Technology in Museums 2011 - Emerging Experiences. Limmerick: think creative & the University of Limerick, pp. 121-131.  Reeves B. (2004) The Benefits of Interactive Online Characters, available at: http://www.sitepal.com/pdf/casestudy/Stanford_University_avatar_case_study.pdf (accessed 27 September 2011).  Robinson S., Traum D., Ittycheriah M., Henderer J. (2008) What would you ask a conversational agent? observations of human-agent dialogues in a museum setting, in Proceedings of the 5th International Conference on Language Resources and Evaluation.  Saarine L. (2001) Chatterbots: Crash Test Dummies of Communication. Master's Thesis, UIAH Helsinki, available at: http://mlab.uiah.fi/~lsaarine/bots/ (accessed 27 September 2011).  Shieber S. (1993) Lessons from a Restricted Turing Test, available at: http://www.eecs.harvard.edu/shieber/Biblio/Papers/loebner-rev-html/loebner-revhtml.html (accessed 27 September 2011).  Wallis, P. (2005) Believable conversational agents: introducing the intention map, available at: http://nlp.shef.ac.uk/dqa/wallis05-3.pdf (accessed 27 Sep. 2011).  Wallis P. (2005 b) Robust normative systems: what happens when a normative system fails?, in Proceedings of the INTERACT 2005 workshop Abuse: The darker side of Human-Computer Interaction, Rome, Italy.  Weizenbaum J. (1976) Computer power and human reason: from judgment to calculation, W. H. Freeman & Co., NY, USA.  Weizenbaum J. (1966) ELIZA – A computer program for the study of natural language communication between man and machine, Communications of the ACM, 10(8), pp. 36–45.