IPOR Telephone Survey Procedures and Capabilities

Sample design

Most IPOR surveys employ random-digit-dial (RDD) telephone samples of land-line telephones. RDD enables households with listed as well as unlisted phone numbers to be sampled. In RDD sampling the last two digits of the phone number are generated randomly, so any number in a group specified by the first 8 digits (area code, exchange, and two more digits) could be selected. The exact location of the sampled household is not known with RDD though the zip code can be roughly approximated (and respondents are usually asked their zip code). For more exact location a sample of only listed numbers is required.

Use of a geographic information system (GIS) is an important part of survey research, particularly in sample design. IPOR's approach is to maintain a GIS framework throughout a survey that incorporates telephone calling areas, census units, and other areas relevant to the study.

Cell phones have typically not been included in RRD surveys but increasingly they will need to be included. They are easily excluded since they are in different blocks of telephone numbers than land-line phones. Reasons for not including them have been low cooperation rates (directly or indirectly cell phone users pay for calls they receive), poor ability to regionally locate households, and higher costs. However in recent years the percentage of cell phone only households has been increasing rapidly. Current estimates are that between 11 and 14% of households use only cell phones, with young adults, minorities, and renters having much higher percentages. IPOR now recommends that cell phones be integrated into samples for studies of ethnic minorities and/or young adults. This usually requires a financial or gift card incentive to compensate the cell phone user, and strict adherence to relevant laws (e.g. only manual dialing is allowed). These requirements will result in higher costs due to the incentives, additional geographic screening, and additional GIS analysis required to derive sample weights needed to make RDD and cell phone samples equivalent.

Human subjects

All IPOR surveys have to be reviewed by FIU's IRB. If proposal comes from another university or institution, that IRB will usually also do a review. For telephone surveys the IRB review is done at the same time the grant or contract is accepted or signed (or on proposal submission if required by a funding agency). Since questionnaires are usually revised during pretesting, a research modification form is submitted after the pretest. For studies not involving a full board review, FIU's IRB review procedure is very fast.

Survey fielding

IPOR uses an in-house programming system that takes an easily revisable markup script for the interview and is able to produce after any revision a revised Ci3 CATI program, text or HTML printout of the questionnaire, SPSS code for full labeling, missing value assignment, multiple response coding, and preliminary frequency data consistency check (it also generates HTML and Perl code for an internet survey, but for security reasons we prefer to outsource these). This system is in operation during qualitative/cognitive testing of the survey instrument, weekly incremental datasets provided to PIs during the survey, and final dataset review/coding with the PIs.

As noted above, survey instruments are programmed into IPOR's Sawtooth Software Ci3 Computer Assisted Telephone Interviewing (CATI) software package. The CATI system ensures skip patterns are followed correctly, that recorded responses fall within the valid range of options, and that there is internal consistency across answers. The CATI system also controls the sample and calling schedule, returning cases to the original interviewer when possible, and routing scheduled appointments to available interviewers. It also routes calls to the appropriate bilingual interviewers and refusal converters. Throughout the data collection period, a cumulative data set is generated by the CATI system on a daily basis along with dispositions of the most recent call attempts (e.g., whether a screener interview was completed, an answering machine was reached, the number is a business, etc.), the number of call attempts across cases, and preliminary cooperation and response rate reports.

In order to maximize cooperation and response rates, and to ensure the representativeness of the sample, a flexible calling schedule is used that allows respondents to complete the interview when most convenient for them. Each sampled phone number receives at least 7 call attempts before it is retired. Households that are screened and identified to have qualified respondents, as well households where screening is not yet completed but information is provided to an interviewer to suggest that the household qualifies, will receive a higher number of call attempts. For each sampled case, calls vary by day of the week and times of day. Respondents also always have the opportunity to complete interviews via call back at a later more convenient time.

Throughout the field period, IPOR closely monitors the sample to ensure that a sufficient number of cases are in the field and being called. In order to ensure that each case receives an adequate number of calls and that call attempts for each case is varied by day of week and time, randomly sorted phone numbers are gradually released in batches to be called by interviewers. This gradual release allows for a close monitoring and control of the sample. In addition to releasing new phone numbers gradually, the CATI system ensures that, in any given time period, call-backs to appointments and previously busy/answering machine/no answer numbers have priority and are completed before any new numbers are introduced. The CATI system cycles call-backs on a schedule appropriate to the situation for each case. The objective of the call system is to make sure each number in the working sample is close to having its chance of resulting in an interview nearly exhausted before new numbers are introduced. Together these steps ensure that the right amount of sample is in the field at any point in time and across the field period.

Throughout the field period, IPOR and project PIs meet via conference call on a regular basis to review field progress, yield, and disposition reports, as well as GIS mapping of these results across the survey area.

Interviewing procedures

All IPOR interviewers are held to the same professional standards and paid accordingly. On any survey about a third to a half of the interviewers will be college students who have to meet the same standards as the others and receive the same pay.

Newly hired interviewers undergo at least six hours of initial training (mostly done one-on-one with an experienced interviewer-trainer). During these sessions, interviewers are taught the fundamentals of standardized interviewing such as how to introduce studies to respondents, gain respondent cooperation, read interview scripts, record answers, and conduct follow-up probes. All interviewers, experienced as well as new, undergo a general training for the survey, which provides study-specific information. Before conducting their first interview in the study, interviewers practice doing mock interviews with staff members. Then in dyads, interviewers role-play the parts of the interviewer and respondent in order to become familiar with all queries and procedures. IPOR will work with the PIs to develop materials for interviewers including responses to frequently asked questions and refusal conversion scripts tailored for the study. New interviewers will be on probationary status for the study and under close supervision and evaluation. Interviewer supervision is done with a system that allows supervisors to listen in to both sides of an interview unknown to the interviewer or respondent (respondents are advised at the outset of the interview that this might occur). Additionally, as disposition reports and preliminary data sets are analyzed for quality control each week, the performance of each interviewer is evaluated. When appropriate, interviewers receive additional feedback and training.

Refusal conversions: While several steps (e.g., short friendly introductions, bilingual interviewers, incentives for some surveys) can be taken to minimize the likelihood that informants and respondents refuse to complete the screener or main interview, refusals do frequently occur. Cases in which a refusal occurs are forwarded to a supervisor who reviews the case's call history (i.e., number and times called, days of week, outcomes of earlier calls) and the interviewer's notes on why the respondent refused. After being "held" (i.e., not called) for a few days, a refusal conversion is attempted by an experienced interview and refusal converter. Each case received a maximum of up to 2 refusal conversion attempts. Only those cases in which a respondent or telephone gatekeeper explicitly indicates that they do not wish to be called back and/or asked to have their number removed from our list are not recontacted.

Interview language: almost all IPOR surveys are done in English and Spanish, and a number in Haitian Creole also. The survey instrument is translated (a back-translated if required) and closely reviewed by project staff and interviewers. Additionally, the translated instruments will be pre-tested internally to ensure that the question content is the same across languages, and that no biases emerge in the telephone conversation context. This pre-testing is conducted by experienced bilingual interviewers in conjunction with bilingual members of the research team. This review helps ensure comparability across languages and appropriateness of key terminology and phrases.

Before hiring new bilingual interviewers, experienced bilingual interviewers test them to make sure they met the needs and standards of IPOR and the study. Versions of the survey instrument in each language is loaded into the CATI system and when the respondent first answers, the interviewer switches immediately to the appropriate language to conduct the interview. If the interviewer does not speak the language, the call is transferred by the CATI system to another interviewer who does. Once the respondent had been identified as a speaker of a particular language, the system directs his or her callbacks to an interviewer who speaks that language.

Cooperation and response rates

These are indicators of data quality and level of field success, and have become of increasing importance as caller ID, answering machines, and other technologies make it easier for people to not cooperate with phone surveys. Large numbers of product-advertising surveys have also taxed the patience of the public (though the no-call list has alleviated this for those who subscribe to it and gives research surveys [exempt from no call] a better chance to make their case). On IPOR surveys the caller ID and initial introductory statement by the interviewer identify the survey as a university-based research survey, and IPOR works with its clients to make its surveys as concise and interesting as possible to respondents. Nonetheless declining cooperation and response rates are a problem and have to be monitored carefully.

The cooperation rate is the ratio of the number of interviews completed to the number of eligible households (e.g., a residential household with a respondent who qualifies for the study) where a contact (i.e. an adult member of the household was reached) was made. The response rate is the ratio of completed interviews to the number of cases fielded minus those cases that are determined not to fall within the parameters of the target population (e.g., faxes, nonworking numbers, business numbers, households that do not have a qualifying person present). IPOR's cooperation rates range from 40 to 70% based on amount of effort, incentives, and interview length/interest. Response rates are more problematic, primarily due to caller ID and answering machines people use for screening. Since these are always more in number than identified eligible households, the maximum response rate that can be achieved for surveys where all residential households are eligible is usually below 25%. This is a brief introduction to a complex topic: The American Association for Public Opinion Research (AAPOR) provides the definitions and standards for cooperation and response rates used in the survey research field

Bias from non-cooperation/response occurs in two ways. First, the demographic characteristics of non-responders may be different than those who do respond (for example, men and young adults are less likely to respond). This can be compensated for by weighting sample proportions by census proportions. IPOR can do the census (using latest available ACS estimates) and GIS work to prepare survey weights for a data set. Second, a more serious bias may result when people make their decision on whether to do a survey based on their feeling about the survey content. In a phone survey they do not know what the questions are before answering, but they are told what the survey is about. Here skilled interviewers, a well tested and designed introduction, and good cooperation rates are essential.

Data cleaning, formats, and analysis

The CATI system and careful pretesting of programmed instruments, along with well-trained interviewers, produce very clean and consistent data sets. Since preliminary data sets can be produced daily, checking goes on during the survey and data sets are ready to use on completion of interviewing. IPOR has software and facility to code open-ended questions provided time and funding are provided to do it.

IPOR's CATI system produces data sets in SPSS format. Using the StatTransfer program, these can be translated into any standard research format incorporating all labeling, weights, etc. Data sets can also be provided in any standard GIS format, with zip codes as the smallest unit for RDD samples. Households from listed samples can be located to the census block group level or within any geographic boundaries provided by the client.

IPOR is capable of doing many types of statistical or GIS analysis of data if requested and budgeted. This can range from simple frequencies/demographic crosstabulations to multivariate multilevel analyses incorporating GIS-based census data.