Methodology of Zandan Poll

(Community Matters) received a note from a friend who took exception with Peter Zandan’s poll’s finding that Austinites satisfaction w/ our city is high and not dependent on socio-economics – that Hispanics are even happier than Whites. The friend who wrote to challenge this finding is close to the ground in Austin’s Hispanic community so I definitely wanted to check out 

Peter is a long time friend & mentor. I admit to discussing some questions and issues prior to the poll going to the field and to having access to results prior to its release. I don’t recall any discussion about the issue of Hispanic satisfaction, however. Caught Peter on his way out of town this morning and an associate was kind enough to summarize the poll’s methodology. The methodology looks really sound – much more so than most polls. [Peter’s reminded me – the poll is of Austin metropolitan citizens – not targeted to voters.]

High-level Summary

The Zandan Poll took great care to be as representative as possible of the Austin population.  In order to do that, an online panel was utilized, a very acceptable and standard practice for modern survey research  (see note below).  The Poll includes large numbers of respondents in each target group (i.e., 184 Hispanics).  Additionally, in order to ensure representativeness, the results were weighted to match latest Census information. Weights for most polls are as high as 1 person being “weighted up” to represent 7 people in the final results.  The weights in the Zandan Poll are extremely modest in comparison, for instance Hispanics have a weight of 1.4 (details for other groups shown below).

Detailed Summary

There are three aspects of sampling that are critical to the validity of any poll:

1.     Sample source

2.     Raw sample composition

3.     Weighted sample composition

For the 2014 Zandan Poll:

1.       Sample Source:  The objective was to be as inclusive as possible.  Over time, researchers have grown concerned that telephone interviewing is no longer as representative as online interviewing, especially among certain demographic groups who do not maintain landlines, move often, or have opted into do-not-call lists.  The Zandan Poll was conducted using an online panel.  The Austin Internet project (http://austintexas.gov/sites/default/files/files/Telecommunications/The_Austin_Internet_and_Global_Citizens_Project.pdf) found that 94% of people in Austin have Internet access at home, compared to the nation as a whole at 78% (Nielsen).  This is not including those who have access to the Internet via smartphones, where Hispanics have more likelihood of having a smartphone than the general population (Pew Research, http://www.pewinternet.org/2012/03/01/nearly-half-of-american-adults-are-smartphone-owners/).  12% of the respondents took the Zandan Poll on a mobile device.

2.     Raw sample composition:  Almost all polls are weighted (described next).  However, to ensure validity, it is critical that the raw number of respondents in any given cell is robust.  In the Zandan Poll, for instance, there are 184 respondents who indicate they are Hispanic.  While this is slightly below the targets from the Census, it is a strong underlying number.

3.     Weighted sample composition:  In polling, the data is weighted based on key variables so the resulting data is representative of all groups in the right proportions.  The measure of how much weighting is needed is easy to evaluate.  For any group, it is simply the weighted number of respondents divided by the unweighted number.  Said another way, you are calculating a ratio for how many people each underlying person in the data set represents.

Most polls have larger weights than were required for the Zandan Poll.  The recent Texas Tribune allowed weights as high as 7.0 in recent polling reports.  (http://s3.amazonaws.com/static.texastribune.org/media/documents/utttpoll-201402-methodology.pdf).   The Zandan Poll average weights by select subgroups weighted are:

·         Hispanics, 1.4

·         Household income < $25K, 1.6

·         Age 65+, 0.8

·         18 to 24 year olds, 1.3

While these groups were indeed weighted up, the size of the weight was not overly large and the unweighted base was substantive in its own right.

Dever, Rafferty and Valliant (2008) found that “statistical adjustments can reduce, if not eliminate, coverage bias in [Internet surveys]” by using sufficient demographic variables in the post-stratification weighting, including education and children in household (both of these included in Zandan Poll weighting).  Between the large Internet penetration in the Austin area, the ability to weight to Census on key variables, plus scholarly research showing the validity of this approach, the concern of excluding potential non-Internet respondents is reduced.  Additionally, studies have shown that respondents, when given the choice, widely prefer an online study to a phone-based one (Hogg, 2002).