Research Methdology Comparisons

Research Methdology Comparisons

Mail Survey
Online Data Collection
Telephone Interviews
In-Person Interviews
Focus Groups
Disk By Mail
Continuous Tracking
Panels

Mail Survey
A mail survey uses questionnaires sent to respondents through the mail. The questionnaire may be on paper or on computer disk or faxed to participants. The respondent fills it out and returns it. Mail surveys often include some incentive or premium.

Advantages

Less costly: Mailing out a bunch of questionnaires is usually the cheapest way to do research.
Can show visuals: Pictures, drawings or graphics can be included in the mailing. These are sometimes helpful to explain or show concepts being researched.
Can do some complex tasks: Rank-Ordering lists or sorting items is possible when using mail surveys.

Disadvantages

Low response rates: While there are some ways to enhance response, mail questionnaires are typically returned by less than one in five recipients. Bias may occur since the respondents can be largely self-selected.
Limited for unaided awareness or open-ended questions: These types of questions are difficult to ask, and the quality of data can be compromised.
Turnaround: Mail-outs typically take weeks or months. Phone surveys can be completed in days or even overnight.
Uncertainty about respondent qualifications: Who really filled out the questionnaire? A clerk or secretary instead of the targeted executive?
Exposure to competition: Of particular concern if researching new or confidential concepts. If a competitor gets your questionnaire, they can replicate your research by mailing copies to a similar population and processing the results. At the very least, they'll know what you're asking.
Data limitations: Many mail-out questionnaires are returned with incomplete or obscure answers. These can't be clarified or probed. The usual action is to discard partials.
Length restrictions: Without special care such as incentives or advance contact by phone, longer mail-out questionnaires are usually not returned. Also, questionnaires that look long or complex reduce response rates.

Online Data Collection
The Internet and online services can be used to conduct a full range of research types. Using specialized software, a questionnaire is programmed and put on a Web page for respondents to fill out. Respondents may be recruited over the phone, by e-mail or by putting a link on a relevant home page.

Advantages

Moderate cost. Data collection costs are reduced.
Speed. Allows for quick collection of large numbers of interviews.
Non-intrusive. Respondents can answer at their convenience.
Access. Unique or hard-to-find respondents can be accessed.

Disadvantages

Sample bias. While a growing number of people have Internet access or use online services, many still don't. Non-users can't be represented in surveys.
Non-response. It's easier to ignore a survey online than to ignore an interviewer on the phone. People may look and then choose not to complete the survey.
Security. If the survey reveals sensitive information, it is more available to competitors than if a telephone interview is used.

Telephone Interviews
Data are collected via structured interviews collected over the telephone. The interviewer records qualified respondents' answers on a questionnaire or at a computer terminal. Computer-aided telephone interviewing (CATI) helps the interviewer by managing skip patterns and quotas. Completed interviews are then pipelined directly to a database. The interviewer uses a guide to provide interview direction and ensure important topics are covered. The interviewer may also add follow-up questions to probe and clarify respondents' answers and obtain in-depth information.

Advantages

Speed: As mentioned earlier, phone interviews can be conducted very quickly. A large phone center can generate many interviews within a short period.
Moderate cost: Phone interviews are less expensive than in-person interviews, but more costly than mailed questionnaires.
More personal: The telephone has some of the attributes of a face-to-face interview. Respondent qualifications can be ensured, responses can be probed and clarified, and questionnaires completely filled out, even if a callback is required. Rapport creates greater cooperation.
Confidential. Sponsorship or even the geographic origin of the survey can be well disguised.
Quality control. Non-interruptive monitoring allows for close supervision of the interviewing process.

Disadvantages

Limitations on length. Depending on the subject and audience, phone interviews longer than 30 minutes can be difficult or impossible.
No visuals. While techniques can sometimes be combined (mail the concept drawings, ask the questions by phone), showing visuals, doing demos or doing other hands-on procedures is normally not possible by phone.
Cooperation barriers. As more telemarketing and phone surveys are conducted, respondent cooperation is sometimes a problem. New technologies such as caller ID tend to increase these barriers.

In-Person Interviews
Data can be collected by sending researchers to conduct face-to-face interviews. As the questions are answered, the researcher records the responses on a questionnaire, enters them into a computer or tapes the interview. Door-to-door interviewing is one version of this tool. In-depth, pre-recruited personal interviewing is another.

Advantages

In-person: The respondent can see and be seen by the researcher. These interactions can be valuable in building rapport, eliciting cooperation and encouraging candor.
Depth or length of interview: An hour-long interview in a respondent's home or office is expensive but usually pleasurable for the respondent. An hour interview on the phone can be too tedious. Probing or in-depth questions are generally more effective or easier to ask in person.
Show and tell: A wide variety of visuals can be used-demonstrations, feature trade-outs (see Conjoint analysis), catalogs, videos, computer screens, concepts.

Disadvantages

Expensive: While showing some resurgence, door-to-door interviewing has almost become extinct due largely to its high cost compared to telephone interviewing.
Quality control: Supervising an in-person interview is more difficult than monitoring telephone interviews.
Slower: It usually takes more time to individually find and interview respondents in person.
Less respondent anonymity: Some sensitive subjects may actually be better researched by phone than in person. There may be more pressure to give conventional or socially acceptable responses when being interviewed face-to-face.

Focus Groups
Focus group research uses group discussions to learn about a topic. Eight to fourteen respondents are typically recruited for each session. Two or more focus groups are usually conducted to determine which results are consistent and, therefore, reliable. Different locations may be used to balance or minimize regional bias. Most focus groups are conducted in special focus group rooms so the process can be observed and video recorded. Groups can also be conducted online. Focus group discussions can last from about 30 minutes to two and one-half or even three and one-half hours. Focus group members receive an honorarium or "co-op" for their participation.

Advantages

In-depth: Focus groups can get at information that cannot be obtained in other ways. Participants not only discuss their own opinions, they also have an opportunity to react to the ideas of others. The give and take among focus group participants provides a powerful dynamic missing from individual interviews.
Hands-on: There is also opportunity for extensive direct investigation. Products can be tried out, food tasted, concepts reviewed and reactions observed and probed.
Timely: Immediate. Post-group debriefings with moderators and clients can create shared insight and reduce selective hearing or premature conclusions.

Disadvantages

Misused: There is often a temptation to consider focus group results a substitute for survey data. Focus group research investigates the nature of attitudes and motivations, not their frequency in the population.
Misinterpreted: Even experienced researchers can be challenged in determining which findings can be generalized from focus groups. Since they are so immediate, naive observers may be inclined to extract their own conclusions — perhaps reflecting their own bias.
Poorly managed group dynamics. Conducting focus groups requires skill, insight and experience. Not everyone should be a moderator.

Disk By Mail
An emerging technique with application in some markets. A computer disk with a self-administered questionnaire programmed on it is mailed to respondents. They answer the questions on screen and return the completed disk. Often, a screening phone call is required to ensure that the respondent has the necessary equipment.

Advantages

Fun for participants: While the novelty may wear off eventually, most computer users enjoy disk-by-mail interviews. Cooperation and response rates approach 80%.
Good for long questionnaires: Disk-by-mail works when long interviews are needed. Some questions can be answered faster on a computer than when read over the phone.
Fits special research designs: More sophisticated techniques benefit from being administered on a computer. Advanced conjoint or trade-out techniques are one example.

Disadvantages

Requires equipment: Some respondents may lack the skills or equipment to do disk-by-mail. This technique is best for business or technical markets.
Slow turnaround: Since calling, sending, filling out and returning are all required, disk-by-mail may take a few weeks longer than a phone interview.
A security risk: In spite of protection, an expert could copy your questionnaire and send it to a competitor — just as with other mailed-out questionnaires. And the possibility of a computer virus may inhibit some from participating.
More expense: Because of the multiple steps involved, disk-by-mail can be more costly than one-step methods.

Continuous Tracking
Ongoing services are used to track performance continuously: daily, weekly, monthly or quarterly. Real-time tracking measures performance factors such as product satisfaction, technical and customer service satisfaction, company or product awareness, and lost business. Reports are produced on a periodic basis (which may be as often as daily).

Advantages

Up-to-the-minute feedback. Can talk to customers about recent experiences. Impressions are fresh and responses are detailed.
Fast results. Data turnaround is rapid and results are quickly available for decision making.
Fixing problems. Real-time tracking can provide an opportunity for timely solutions to customer dissatisfaction.

Disadvantages

Organization needed. In order to best take advantage of an ongoing monitor, lists of potential respondents need to be transmitted on a daily or weekly basis — consistently and accurately. Often this will require dedicating internal IS resources during the start of a project.
Long start-up time. Real-time tracking studies often take longer to set up than normal studies due to the high level of coordination required between the client and research vendor.
Stable questionnaire. The questionnaire must be kept consistent across waves in order to make comparisons. The key issues must be identified before research begins.

Panels
Usually consist of groups of people who supply information on a regular basis. Questions typically focus on subjects such as purchasing behavior, purchase intention or advertising awareness. In some cases, panels are recruited simply to provide a group of qualified respondents who can quickly give input for questions about consumer preference or new product development. Panels can be used as a source of respondents for in-person interviews, telephone interviews or central location tests.

Advantages

Quick or continual access: Panels offer ready and ongoing access to information. Since panelists are pre-recruited, researchers have a group of respondents available for questions on changes in product preferences or perceptions of an industry. And, since the panel can be revisited, changes can be measured over time.
Less costly: Panels provide the opportunity to inexpensively develop a continuing or long-range view of the market, without needing to conduct multiple surveys.
Shorter surveys: Respondents are familiar with the process and have background information on the product or subject.

Disadvantages

Panel conditioning: One difficulty in using a panel is ensuring it remains representative of the market and members do not become professional opinion givers. Panelists must be rotated out and new ones added regularly.
Quality control. Quality can be affected when panelists who are supposed to keep daily records of their purchases, activities or perceptions sometimes fail to and end up reporting biased or inaccurate recollections.

cesselogo

CESSE '06 Attendees
Download a copy of
Fundraising During
Anniversary Years

Adobe PDF

PowerPoint Slideshow