Research Methodology
Description This assignment should present the first draft of your proposal, and yourbibfile containing all (possibly) relevant papers you have collected. Yourproposal should have your name, your supervisor, and your tentative thesistitle, followed by the sections listed below.Introduction Briefly describe what your topic is, why it is important or interesting, and what the specific research questions are.
Literature review Place your project in the context of the literature. Cite references appropriately, and reference them using latex/bibtex.
Project plan What are the different pieces you will need to do in your project. Break itdown and give a timeline. Allow at least 4 weeks for writing the thesis.
Evaluation How do you plan to evaluate your research project? Describe this in somedetail, perhaps offering some alternative options.
With the increase volume in World Wide Web, search has become one of the most efficient ways to find out the required information for a user. Various search engines or search services have been launched to help users find information stored on World Wide Web, inside corporate networks, or on personal computers. Information is retrieved from different websites using information retrieval techniques and normally information is retrieved from the websites title, URL and a snippet. To convey information to users, people advertise their websites and this is what we will consider in search results for our project, advertisement.
Currently advertisements are displayed on commercial search engines but sometimes they are irrelevant to what user searches for which becomes annoying. The problem is the paid search which is most used prevalent business model for searching on the Web. World most popular search engines, such as Google, Bing, Baidu and Yahoo have adopted paid search globally and also some of the site-specific webs such as Cnn.com have already begun using the paid search model. According to a research Google received 99% of its $3.1 billion revenue from paid search, while Yahoo received 84% of its $3 billion and AOL received 12% of its 1 billion revenues in 2014.The effectiveness of a search engine depends on its sponsored and unsponsored results. A search engine can increase its revenue by increasing the visibility of paid advertisement but this approach would reduce overall quality of the web engine and eventually helps in decreasing of its users. The search engine company chooses the best way to advertise by balancing its business goals, revenue and users satisfaction.
To achieve our objective in this project we will investigate the relevancy, numbers and positions of advertisement to find the best presentation design of them in search result. The relevancy is all about getting the right information in front of the right user rather than showing irrelevant information by using paid ads. Irrelevant information can be used by giving different types of wrong information such as wrong tags, wrong displayed text and redirection to different webs as well. In other word relevancy can be defined as the information which is able to answer the questions of the users. We will also investigate different types of advertisement, such as SERP’s ads and landing page ads, their positions and the number of advertisement available. The SERP’S ads are categorized to different way, google is using different methods to show the advertisement including inline SEO, listing ads with product rating, rich snippets, data highlighter. While the advertisement inside the page have different positions to display advertisements. The most commonly is the top of the page, which is said to be more successful in CRT rather than lower and middle of the page. The position of the advertisement creates a great impact on the user and also helps to catch the user interaction. We will perform some tests and eye-tracking technology for our findings and later consider user satisfaction using post task and experiment questionnaires. We will ask group of test users to search for some information and then using the eye tracking technology we can keep track of their eye movement. According to their movement we can get information for best presentation of advertisements. Users satisfaction will help us understand if the displayed advertisements were relevant to what they were looking for.
Title: Finding the best presentation of advertisements in search results The introduction structure:
- What is the problem?three problems ( some advertisement irrelevant & number of das & the position of the advertisements (over the page-fold)
- Relevant vs irrelevant of the advertisements
- Different numbers of the advertisements (one vs two or more)
- the position of the advertisements (the top of the SERPs vs over the page-fold)
- In term of relevant
- In term of numbers of ads
- In term of the ads position
16-The good, the bad, and the random: an eye-tracking study of ad quality in web search
17- What do exploratory searchers look at in a faceted search interface? 18- What do you see when you're surfing?: using eye tracking to predict salient regions of web pages 19- Learning user interaction models for predicting web search result preferences 20- A semantic approach to contextual advertising 21- What are you looking for?: an eye-tracking study of information usage in web search 22- Search User Interface Design(book)The research description
Title: Finding the best presentation of advertisements in search results Area of research: Information retrieval/search engine, Human-computer interaction Pre-requisite knowledge/background: Interest in information seeking behaviour on web search results using an eye-tracker. Not strong programming skill is required, but some ability of web programming would be very helpful. It is recommended (but not mandatory) to have a course of COSC 1182/1183 – Usability engineering, before starting this project.
Description: In the previous research, studies have confirmed information seeking behaviour is affected by several factors such as the number and position of relevant document, screen size, and rank order. However, the many of previous studies normally considered only organic search results, which are consist of title, URL, and snippet. In this project, we will consider another search results, advertisements. Current commercial search engines generally display some advertisements on the top of search results. In some cases, the advertisements are very useful such as if you are looking for an hotel in Melbourne, but those are sometimes very uninformative if the advertisements do not include the information users are looking for. In this project, we will investigate the relevancy, numbers and positions of advertisement to find the best presentation design of them in search results. We will adopt A/B test and/or eye-tracking technology for this study and consider user satisfaction using post-task and post-experiment questionnaires. Through this project, a student will understand general interactive information retrieval research, and especially information seeking behaviour, and will have knowledge of Usability Engineering such as experimental design and statistical data analysis, and eye tracking technology as well.
Solution
INTRODUCTION With the increase volume in World Wide Web, search has become one of the most efficient ways to find out the required information for a user. Various search engines or search services have been launched to help users find information stored on World Wide Web, using information retrieval techniques. To convey information to users, people advertise their websites. Given the increasing importance of customer-based targeting using data science, the research topic is chosen as “advertisements”.
Currently advertisements are displayed on commercial search engines but sometimes they are irrelevant to what user searches for which becomes annoying. World most popular search engines, such as Google, Bing, Baidu and Yahoo have adopted paid search globally and also some of the site-specific webs such as Cnn.com have already begun using it. According to a research Google received 99% of its $3.1 billion revenue from paid search, while Yahoo received 84% of its $3 billion and AOL received 12% of its 1 billion revenues in 2014.The effectiveness of a search engine depends on its sponsored and unsponsored results. A search engine can increase its revenue by increasing the visibility of paid advertisement but this approach would reduce overall quality of the web engine and eventually helps in decreasing of its users. The search engine company chooses the best way to advertise by balancing its business goals, revenue and users’ satisfaction. So, we want to find best presentation of ads in web search engine so that the user will benefit with the relevant information and can save his time, on the other hand the website will get new users, target proper audience and increase their sales.
To achieve our objective in this project we will investigate the relevancy, numbers and positions of advertisement to find the best presentation design of them in search result. The relevancy is all about getting the right information in front of the right user rather than showing irrelevant information by using paid ads. Irrelevant information can be used by giving different types of wrong information such as wrong tags, wrong displayed text and redirection to different webs as well. In other word relevancy can be defined as the information which is able to answer the questions of the users. We will also investigate different types of advertisement, such as SERP’s ads and landing page ads, their positions and the number of advertisement available. The SERP’S ads are categorized to different way, google is using different methods to show the advertisement including inline SEO, listing ads with product rating, rich snippets, data highlighter. While the advertisement inside the page have different positions to display advertisements. The most commonly is the top of the page, which is said to be more successful in CRT rather than lower and middle of the page. The position of the advertisement creates a great impact on the user and also helps to catch the user interaction.
We will perform some tests and eye-tracking technology for our findings and later consider user satisfaction using post task and experiment questionnaires. We will ask group of test users to search for some information and then using the eye tracking technology we can keep track of their eye movement. According to their movement we can get information for best presentation of advertisements. Users satisfaction will help us understand if the displayed advertisements were relevant to what they were looking for.
LITERATURE REVIEW
In this section, we introduce some of the background knowledge necessary for conducting this experiment. Three general lines of study should be considered:generalbehaviour on laptop and desktop and eye-tracking, mobile web search and the effects of relevant and irrelevant ads.
Eye-Tracking Studies:
Eye-tracking device has been widely utilized in understanding user behavior on both sponsored search (Jacob &Karn, 2003; Rayner, 1998) and organic search results .Since the eye tracker can record users’ real-time eye movement information on SERPs, it helps researchers better understand how users examine results. Granka et al. and Joachims et al. are among the first works that start this line of research on organic SERPs and they all found that there exists position bias during users’ examination processes. There is a connection between the length of contextual snippet and user performance by using eye tracking techniques. Recently, Liu et al. indicated that the examination process of search results might have two steps (skim and read) and proposed a two-stage examination model based on the eye-tracking analysis. Eye-tracking experiments also help analyze the users’ evaluation on search results. For example, Rayner, 1998, 2009 showed that the personal style (economic or exhaustive) and the result relevance can affect search result evaluation. Poole & Ball, 2006 predict the relevance of documents by using the gaze data. Besides, eye-tracking devices are used to investigate the relationship between eye movements and mouse movements, such as (Aula, Majaranta, &Räihä, 2005; Buscher, Dumais, &Cutrell, 2010; Dumais, Buscher, &Cutrell, 2010; Cutrell& Guan, 2007; Granka et al., 2004)Wang et al. found that different verticals might create examination biases on the eye movement behavior for both vertical and other results on SERPs. Lagun et al. performed a similar study on mobile devices and found that both vertical relevance and positions have impacts on users’ attentions.
The eye-movement behaviours just described have several implications for understanding search behaviour. Goldberg and Kotval (1999) found that more fixations indicates less effective searching, and that the optimal scanpath in a search task exhibits a short fixation duration and less hesitation. In addition, more fixations on a particular area of interest implies that the information there is more important than that in other areas (Poole, Ball, & Phillips, 2005), whereas a longer average fixation duration is an indication of task complexity (Just & Carpenter, 1976; Rayner, 1998).
User Interaction on Small Screens:
A few researchers have investigated explicit user interaction with small screens on web search results pages, although these studies did not record eye movements. For example, Jones et al. (2003) compared users’ abilities among three kinds of interfaces: mobile phone-sized, handheld computer-sized (PDA), and conventional desktop. They found that users take more time to complete tasks and exhibit lower task success rates on smaller screens. They suggested several guidelines for the design of small screens to improve user search performance such as sufficient information in a search result, some marker to indicate that a link will display a small-screen-optimized page, and preprocessing conventional web documents for small devices.
Despite the interest in web search behaviour, only a few eye-tracking studies have been conducted on small screens. Drewes et al. (2007) investigated gaze interaction for controlling applications on a handheld device using dwell time and gaze gestures. Further, Nagamatsu et al. (2010) investigated a remote gaze tracker for mobile devices with stereo gaze tracking. Recently, text interaction and reading performed on an actual mobile touch screen device was analyzed by Biedert et al. (2012). However, we could find no investigation that used eye tracking to study user behaviour in web searches on small devices.
The research presented here extends previous studies with the aim of understanding the differences in user search performance and behaviour on the web with respect to two differently sized screens and two task types— informational and navigational. As well as considering differences in behaviour, we compare relationships between search performance and behaviour across the two sizes of screens. Furthermore, we suggest some improvements that may enable the design of a better presentation method for search results pages on small screens.
Effects of relevant and irrelevant advertisements:
This section focuses on literature on the retrieval effectiveness of Web search engines. General literature on retrieval effectiveness as well as a general discussion on the concept of relevance is omitted. While most retrieval effectiveness tests use precision as a retrieval measure, there are some studies that use other metrics. There are studies that test search engines for availability of Web documents (Stock & Stock, 2000) or their ability to retrieve homepages (Hawking &Craswell, 2005).
The number of queries used in the studies varies greatly. Especially the oldest studies (Chu & Rosenthal, 1996; Ding &Marchionini, 1996; Leighton & Srivastava, 1999) use only a few queries (5 to 15) and are, therefore, of limited use (see Buckley & Voorhees, 2000). Newer studies use at least 25 queries, some 50 or more. In older studies, queries are usually taken from reference questions or commercial online systems, while newer studies focus more on the general users’ interest or mix both types of questions. There are studies that deal with a special set of query topics (e.g., business, see Gordon & Pathak, 1999), but we see a trend in focusing on the general user in search engine testing.
Regarding the number of results taken into account, most investigations only consider the first 10 or 20 results. This has to do with the amount of work for the evaluators but also with the general behaviour of search engine users. These users only seldom view more than the first one or two results pages (with usually 10 results on each page). Therefore, a cut-off value of 10 or 20 appears reasonable. Usually, the biggest and most popular search engines are tested, sometimes in comparison to newer or language-specific search engines.
An important question is how the results should be judged. Most studies use relevance scales (with three to six points). Griesbaum’s studies (2004, 2002) use binary relevance judgements with one exception: results can also be judged as “pointing to a relevant document” (i.e., the page itself is not relevant but has a hyperlink to a relevant page). This is done to take into account the special nature of the Web. However, it seems problematic to judge these pages as (somehow) relevant, as pages could have many links, and a user then (in bad circumstances) has to follow a number of links to access the relevant document.
Véronis (2006) uses a five-point relevance scale to judge results. This study also shows that neither of the engines tested receives a good overall relevance score. The author concludes that “the overall grades are extremely low, with no search engine achieving the ‘pass’ grade of 2.5” (Véronis, 2006) The best search engines are Yahoo and Google (both 2.3), followed by MSN (2.0). The results from these studies show that search engines have problems in producing satisfying results lists.
PROJECT PLAN
In the previous research, studies have confirmed information seeking behaviour is affected by several factors such as the number and position of relevant document, screen size, and rank order. However, the many of previous studies normally considered only organic search results, which are consist of title, URL, and snippet.
In this project, we will consider another search results, advertisements. Current commercial search engines generally display some advertisements on the top of search results. In some cases, the advertisements are very useful such as if you are looking for an hotel in Melbourne, but those are sometimes very uninformative if the advertisements do not include the information users are looking for. In this project, we will investigate the relevancy, numbers and positions of advertisement to find the best presentation design of them in search results. We will adopt A/B test and/or eye-tracking technology for this study and consider user satisfaction using post-task and post-experiment questionnaires. Through this project, a student will understand general interactive information retrieval research, and especially information seeking behaviour, and will have knowledge of Usability Engineering such as experimental design and statistical data analysis, and eye tracking technology as well.
Experimental Design:
In this section, we explain the experimental design and our procedure. Using an eye-tracking instrument, mouse movement logging, and other instruments, we recorded gaze and click data from 32 participants as they interacted with web search results.
To study search behaviour, we will measure fixations and other gaze behaviour, click patterns, scrolls, andsimilar interactions.
Participants will be subjected to various tasks in groups and individual manner. All the durations and patterns will be measured.
The objective of this research is to conduct a small-scale search engine retrieval effectiveness study using a representative query sample, distributing relevance judgments across 32 jurors.
The goal is to improve methods for search engine retrieval effectiveness studies. The research questions are:
Which search engine performs best when considering the first 10 results?
Is there a need to use graded judgments in search engine retrieval effectiveness studies?
Relevant vs irrelevance of the advertisements.
What should be the different numbers of the advertisements (one vs two or more).
What should be the position of the advertisements (the top of the SERPs vs over the page-fold).
EVALUATION
Based on the data collected from a controlled experimentation, we will systematically analyse userattention both at page level and result level.
Most servers provide logs recording the number of visits, unique visits, referral address, browserclient, session information, etc. Many sites have log analysis software such as WebTrends or Click Tracks to analyse the server logs. Traffic from search engines can be derived from serverlogs and is one benchmark that can be used to determine success of a search results.
Depending on high-level metrics such as revenue and conversion rate is nearly impossible since these are influenced by many other factors also. After enhancing search or making configuration changes, there can be an expectation of increased revenue or conversion rate.
Search Contribution to Transactions
IOW, the percentage age transactions that are taking place through search. This metric is an indicator of how many transactions are taking place when visitors find products of interest through search. The other ways users can find products are:
Through a SEO/SEM landing page.
Those that are taking place through a visitor’s manual browsing of the product catalogue/categories by clicking through an advertisement .
Advertisement campaigns type.
All of the above will combine to give you a 100% score. This score will tell where most of your transactions are coming from. It is an indicator of whether search is a driver of transactions.
@ARTICLE {Mind 2 (3):4 (2003),
author = “Robert J. K. Jacob \& Keith S. Karn”,
title = “Eye tracking in human-computer interaction and usability research: Ready to deliver the promises”,
journal = “Philpapers”,
year = “2003”
}
@ARTICLE {,
author = “Keith Rayner”,
title = “Eye Movements in Reading and Information Processing”,
journal = “Psychological Bulletin”,
year = “1998”,
volume = “124”,
number = “3”,
pages = “372-422”
}
@ARTICLE {,
author = “Rayner K”,
title = “Eye movements and attention in reading, scene perception, and visual search.”,
journal = “Q J ExpPsychol (Hove)”,
year = “2009”,
volume = “62”,
number = “8”,
month = “aug”
}
@ARTICLE {,
author = “G Buscher, ST Dumais, E Cutrell”,
title = “The good, the bad, and the random: an eye-tracking study of ad quality in web search”,
journal = “Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval”,
year = “2010”,
pages = “42-49”,
month = “aug”
}
@ARTICLE {,
author = “Edward Cutrell, Zhiwei Guan”,
title = “What are you looking for?: an eye-tracking study of information usage in web search”,
journal = “roceeding CHI ’07 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems”,
year = “2007”,
pages = “407-416”,
month = “apr”
}
@ARTICLE {,
author = “BING PAN, THORSTEN JOACHIMS, LAURA GRANKA”,
title = “Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search”,
journal = “ACM Transactions on Information Systems”,
year = “2004”,
volume = “25”,
number = “2”,
month = “apr”
}
@ARTICLE {,
author = “Helena X. Wang, Jeremy Freeman, Elisha P. Merriam”,
title = “Temporal eye movement strategies during naturalistic viewing”,
journal = “Journal of Vision”,
year = “2012”,
volume = “12”,
number = “1”,
pages = “1-27”
}
@ARTICLE {,
author = “Joseph H. Goldberg, Xerxes P Kotval”,
title = “Computer interface evaluation using eye movements: Methods and constructs”,
journal = “International Journal of Industrial Ergonomics”,
year = “1999”,
volume = “24”,
number = “6”,
pages = “631-645”,
month = “oct”
}
@ARTICLE {,
author = “Alex Poole and Linden J. Ball”,
title = “Eye Tracking in Human-Computer Interaction and Usability Research:”,
year = “2005”
}
@ARTICLE {,
author = “HeikoDrewes , Alexander De Luca, Albrecht Schmidt”,
title = “Eye-gaze interaction for mobile phones”,
journal = “Mobility ’07 Proceedings of the 4th international conference on mobile technology, applications, and systems and the 1st international symposium on Computer human interaction in mobile technology”,
year = “2007”,
pages = “Pages 364-371”
}
@ARTICLE {,
author = “Takashi Nagamatsu, Michiya Yamamoto, Hiroshi Sato”,
title = “MobiGaze: development of a gaze interface for handheld mobile devices”,
journal = “CHI EA ’10 CHI ’10 Extended Abstracts on Human Factors in Computing Systems”,
year = “2010”,
pages = “3349-3354”
}
@ARTICLE {,
author = “Ralf Biedert, Georg Buscher, ArmanVartan§”,
title = “Reading and Estimating Gaze on Smart Phones”,
year = “2012”
}
@ARTICLE {,
author = “Stock, M. \& Stock, W.G.”,
title = “Retrievaltestmit Known Item Searches”,
journal = “Internet-SuchwerkzeugeimVergleich”,
year = “2000”,
volume = “15”,
number = “11”,
pages = “23-31”
}
@ARTICLE {,
author = “Heting Chu, Marilyn Rosenthal”,
title = “Search Engines for the World Wide Web”,
year = “1996”
}
@ARTICLE {,
author = “Wei Ding, Gary Marchionini”,
title = “A comparative study of web search service performance”,
journal = “Proceedings of the ASIS Annual Meeting”,
year = “1996”,
volume = “33”,
pages = “136-140”,
month = “dec”
}
@ARTICLE {,
author = “H. Vernon Leighton JaideepSrivastava”,
title = “First 20 precision among World Wide Web search services (search engines)”,
journal = “Journal of the Association for Information Science and Technology (JASIST) i”,
year = “1999”,
month = “jul”
}
@ARTICLE {,
author = “Chris Buckley, M. Voorhees”,
title = “Evaluating evaluation measure stability”,
journal = “SIGIR ’00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval”,
year = “2000”,
pages = “33-40”,
month = “jul”
}
@ARTICLE {,
author = “Gordon, M. and Pathak, P.”,
title = “Finding Information on the World Wide Web: The Retrieval Effectiveness of Search Engines.”,
journal = “Information Processing and Management”,
year = “1999”,
volume = “35”,
pages = “141-180”
}
@ARTICLE {,
author = “Veronis”,
title = “With which key figures can the quality of search engines be measured?”,
year = “2006”
}