FAQs: Sampling, quotas and weighting
Frequently asked questions. Click on any question to see the answer.
Sampling and quotas
SQU01 What is sampling in a research context?
Sampling has two meanings in research situations. First, where the views of an overall population are measured by asking for feedback from a representative sample of the whole. If an exact proportional match cannot be made with the sample, data analysis techniques can be used to weight the responses to match the exact proportions of the overall population (see SQW01 below).
Second, sampling can also refer to taking feedback from a defined sub-sample of a larger population. The sample group could be people living in a particular area or from certain age groups. In Demographix, use of our panel management system, Panelwise, allows you to easily sample sub-groups of a panel based on details given in a registration survey (see SQU02 below).
SQU02 How do I use Demographix to sample a sub-group of a panel?
Our panel management system, Panelwise, allows you to build up a panel of regular survey respondents, and use a Registration Survey to record details about panellists (such as age bracket, sex, region, salary range, etc). When sending out emails to panellists, our system lets you specify that the emails only go to those who fulfil certain criteria in the way they have answered the registration survey. This is done through an Include/Exclude tab in the email creation system.
SQU03 What are quotas and are they important?
A quota is where you specify how many responses from certain types of respondents you want taken, and when quotas are full to take no more responses. Quotas were very important to researchers when polling techniques involved human intervention to collate data such as manually entering results from printed opinion surveys completed by respondents on paper or over the phone. However, online surveys greatly reduce the cost to researchers, because respondents can fill in the surveys themselves and data is increasingly low-cost to store. Nowadays, researchers can gather as much data as they like, and then use sophisticated filtering and weighting tools to analyse data in depth. NOTE: As Demographix has no limit on the number of responses a survey can take (i.e. there is no capping, above which extra costs kick in)you can have as many responses as you want.
SQU04 Can I screen out respondents who don't fulfil criteria for taking part in a survey?
Yes, you can create Exit pages that are conditional on the criteria on which you want to screen out participants. For example, all males (so only female respondents are included), or all people under a certain age in a certain area. If the survey respondent fulfils the criteria to see this Exit page, they will be given a notification of why they are not being included, and then exited from the survey. Click here to read how to create an Exit page (FAQ SCL06).
SQU05 Can I use Exit pages to define quotas?
Yes, Demographix staff can dynamically change a survey to exit respondents after some quotas have been filled. You will need to talk to our Helpdesk about special custom coding for your specific needs.
Introduction to Weighting
SQW01 What is weighting?
In data analysis, researchers often want to revise the data generated by a survey so that the profile of the respondents matches the profile of an overall population. For example, a survey taken in a specific region gets a 45% response rate from males and 55% from females, but we know that the actual population of that region is 60% male and 40% female. The responses of the females would need to be "weighted" to make them proportionate to their representation in the population as a whole.
SQW02 Can you give a simple example of how weightings are calculated?
An institution has a known total population of 8,000 males and 2,000 females, which can be called the Universe (U). A survey sample, designated S, is taken where 500 males and 300 females respond. Clearly, the proportions are out of sync, so weighting is used. First, we need to calculate mathematical formulae that can weight the response data to be equivalent to the population as whole. Weight (calculated U divided by S) is a measurement of what a sample unit represents in the above example, males are weighted at 16 - so, 1 male response in the survey is equivalent to 16 males in the population, while 1 female response is equivalent to 6.67 females in the overall population. The Weight Factor is the value used to calculate the weighted data of questions in a survey. Real values of response levels are increased (or decreased) by this factor, and then gross or nett means are calculated. You can view these formulae in a table here: Reference weights calculation table.
SQW03 Can you give a simple example of how a question's responses are affected by weighting?
Using the weight factors calculated in the above FAQ, we have prepared illustrative examples of how these weights can be applied to the results of a simple Y/N question. You can view these here: Reference applied weighting tables.
SQW04 What data do I need in order to use the Demographix weighting tools?
In order to use the Demographix weighting toolset, you will need to supply Universe figures that correspond to the sample of your survey. For instance, you have a survey which you are looking for representative feedback from specific age groups in Scotland. You will need to know the general population breakdown in Scotland for those age groups (the Universe figures). This might be: Under 16: 22%; 16-32: 27%; 33-48: 28%; 49-65: 16%; 66+: 7%. Those figures can be used to weight against an age question in the survey using the same age breaks.
NOTE: It is important to emphasise that in order to use our weighting tools, you will need a full set of Universe figures that exactly match the question/s in the survey that you wish to weight your sample with. Every option specified in a question in the survey to be used for weighting will need equivalent Universe figures.
SQW05 Will I be able to use two factors to weight against, such as sex and regions?
Yes, so long as you have a Universe data breakdown for all the possible variables in combination of those two questions. If there are three regions (A, B and C) and two sex options (M and F), then you will need Universe figures for AM, AF, BM, BF, CM and CF categories. This is called a Data Weighting Table and all the cells (whether a single-column for a single question, or a table for a dual-question) must add up to 100%.
NOTE: It is important to emphasise that in order to use our weighting tools, you will need a full set of Universe figures that exactly match the question/s in the survey that you wish to weight your sample with. In the case of multiple questions, a grid for every combination is needed for the Universe figures. Also, note that any cell with zero in it will result in data in the resulting weighted analysis also being zero, whether there was data or not in the original response database.
Using the Weighting Tools
WTL01 How do I create a Weights table?
To use the Weighting tool in Demographix, click on the Analysis of Responses link beneath the survey name. The Weighting tool is the fifth tab (after Derived Variables, at right). Click on this. The tab will show if any Weights tables have so far been defined for this survey, and if none so far, there will be only a brown "Create a New Weighting" button. Click this to activate the "Configure Weighting" pop-up, which allows you to choose one or two questions to define your weighting table. Once selected, the pop-up will show three sub-tables:
TABLE 1: SURVEY DISTRIBUTION will show what percentage of the respondents fall into each of the categories.
TABLE 2: DESIRED DISTRIBUTION allows you to enter the desired breakdown, or to select an existing table.
TABLE 3: RESPONDENT WEIGHTS will automatically calculate the weighting that will be applied to each respondent's data set. Finally, choose the "Save Reference Weights Table" option in the Table 2 dropdown menu, and press the "Go" button.
WTL02 How do I apply weighting tables to my survey data?
After you've created a Weighting table, it will appear in a list on the Weights tab when you next access the Analysis of Responses. The list is headed "Weightings defined for this survey", and shows the name of the weights table, the text of the question or questions used to define the table, and two actions associated with each table: EDIT and VIEW ANALYSIS. Use either of these to apply weighting.
1. Click EDIT and the Configure Weighting tool will pop-up again, visually reminding you of what the Weighting breakdown involves. Here you can click the brown button at right: "Apply these weights to survey analysis", and this will close the configuration tool and generate a weighted analysis pop-up (with the word "WEIGHTED" as a watermark in the background).
2. Alternatively, click VIEW ANALYSIS to go directly to the weighted data.
WTL03 How can I view and compare weighted data alongside unweighted data?
You simply generate two pop-ups and then manually put these side by side. First, create the Weighted data version of the analysis, then click on Analysis of Responses again to get the original unweighted data. Resize the windows of the two pop-ups so that they can be placed next to each other on screen, and use the vertical scroll buttons to co-ordinate and compare question data.
WTL04 What analysis tools can I use on Weighted data?
A weighted analysis allows you to use all the standard data analysis tools, such as showing/hiding sample sizes, toggling between nett and gross figures, reordering bar charts as highest first and lowest first, applying keywords to write-in (open ended/verbatim) questions, and applying values to create average calculations in questions. Basic filtering can be activated by clicking on bar charts in the analysis – but this restricts you to just one answer, not multiple answer filters. The full crosstab toolset is available under the separate Crosstab tab.
WTL05 What reporting tools can I use on Weighted data?
The same reporting tools are available for weighted data as for the Analysis of Responses. You can download reports in Excel or PDF formats. Any filtering, page hiding, sample size hiding, answer re-ordering, or applied tools will all be reflected in the reports generated. PDF reports name the weighting table used, and highlight the question(s) on which weighting has been applied.