How Are Polls Made? – AHA (2024)

FromGI Roundtable 4: Are Opinion Polls Useful? (1946)

The modern polls usually take great care to select and to interview a small part or “sample” of the population. In his respect they differ from older and less scientific ways of feeling the public pulse. To ask questions of a majority of the country’s adult population, or of even a majority in a large city, would be too big and costly a job. It is usually possible to interview only a very small part of the people. But the part or sample of the population which is finally interviewed must be made up of all types—it must represent the same kinds of people who make up the entire population. This is the crucial point, because an accurate “miniature” of the population should reflect the opinions f the whole population.

Those who uphold the reliability of opinion polls say that if an accurate miniature, consisting for example of 2,500 persons, is interviewed at random, the chances are 99 out of 100 that the answers will vary no more than about 3 percent from the opinions of the whole population. Likewise, they claim that when an accurate sample of 1,000 persons is interviewed, the chances are about 99 out of 100 that the answers will not differ by more than 4 to 5 percent from the answers of the whole population.

So the first requirement of a trustworthy poll is the selection of an accurate sample or miniature of the population. A homely illustration of sampling might be that of a cook making soup. He stirs the broth thoroughly and then dips out a spoonful to test the contents of the whole kettle. He doesn’t need to drink all or most of the soup to judge the taste. A spoonful will do the job. Likewise, the grain inspector chooses a test sample from the contents of a bin or carload. Crop estimates by the federal government are not made by counting, for example, all ears of corn grown in the United States—that would be impossible, of course—but by judging yields on the basis of fair samples.

How many are questioned?

The size of the sample used in opinion polling naturally affects the accuracy of the results. Many nation-wide surveys are being made on the basis of some 2,500 to 5,000 interviews. If figures are to be presented by states also, or for the different groups in the sample—as, for example, members of labor unions—then the total sample must be increased to assure a big enough sample of each of these parts or subgroups. A sample of 2,500 may be large enough for nation-wide figures, for instance, but if figures from the same survey for each of the forty-eight states were reported, they would be based on samples which average only a little more than 50. The samples for each state would probably be too small.

It is an old rule that the smaller the sample the greater the chances of error. Nevertheless, thesizeof the sample is not as, important to experienced pollers as therepresentativenessof the sample. A large sample, carelessly chosen, can lead to greater error than a properly selected small sample. The main question to the poller is, therefore: Is the sample a good cross section of the population? In other words, does the sample include the various types of people who make up the whole population? This is the key point.

How is the sample set up?

There are several ways in which the sample may be set up. The two most commonly used by polling organizations to get a proper cross section of the population are known as (1) the “controlled sample” method and (2) the “area sampling” method.

Controlled sample. The controlled sample is more commonly used. The samples are carefully set up or “controlled” so that they include all the different types of people—butcher, baker, candlestickmaker, and so forth—that make up the whole population. Each interviewer is assigned the exact number and types of persons he is to question.

What types of people should be included to get a cross section of public opinion? Why doesn’t everyone react in the same way to public questions? Does a poor man think differently about politics than a rich man? If so, a sample should have both poor men and rich men. Are people in the South likely to think differently about some current issues than people in the North or on the West Coast? If so, people from all sections of the country should be included in a national sample. Are farmers likely to size up public problems differently from city people? If so, then farmers should be interviewed as well as city dwellers.

The polling organizations have studied this problem for many years and applied mathematical techniques. They haven’t found out all the answers, but they believe that among the main influences that make a person what he is and cause one person to think differently about current issues than another are:

1. The section of the country he lives in—East, South, North, Middle West, or West.
2. The type of community he lives in—big city, small city, village, or farm.
3. His standard of living—poor, average, or wealthy.
4. Whether he is a man or woman.
5. How old he is.

In addition, interviewers are usually told to find out how much schooling each person has and what his race and religion are. These points also have a bearing on a person’s views. A poller must consider all these factors and many more.

At any rate, before the sample can be made up accurately, facts about the population must be known. Figures must be gathered from the latest census reports and other sources so that it can be known what the make-up of the miniature or cross section should be. When the facts and figures have been collected, then the sample can be arranged so that the same percentage of men and women, different age groups, economic classes, people who live in the big cities or on farms can be included.

A National Cross Section or Sample

Total polled: 2,523

If 20 percent of the people in the United States are farmers, then the sample question must consist 20 percent of farmers. If 30 percent of the people live in cities, of one million or over, then 30 percent of the interviews should be made among residents of large cities. If 53 percent of the people are 40 or over, then that percentage of persons past 40 years of age should be interviewed.

Notice the table on page 11. It shows the percentages for a national sample of adults used in the fall of 1944 by the National Opinion Research Center of Denver, Colorado. The number of persons in the sample was 2,523.

After the sample has been made up, each interviewer is given a list or order which calls for answers to questions from certain types of people. When the answers obtained by all the interviewers are added up, say the upholders of public opinion polls, they should reflect the opinion of the entire population.

The interviewers find out from each person interviewed what his occupation is, how much schooling he has had, whether he has a telephone or owns a car, and other facts. These figures, after they are added up for the whole sample, can be compared with the known facts in the entire population about occupations or maybe about telephone and car; ownership. If the percentage of car and telephone owners in the sample, for instance, is about the same as the percentage of car and telephone owners in the whole population, then the sample checks with one accuracy test.

The area sample. The area sample is coming into greater favor each year. It calls for .a cross section of areas rather than of persons. In this case, facts about different types of areas such as counties, townships, or election districts are gathered from the census and other sources. For example, the polling organization gathers figures on counties that contain big cities, those that include medium sized cities, those that have only small towns within their borders, and those that are mainly made up of farms. These figures are collected for the East, West, and other parts of the country. Some of each type of county are picked to make up the sample. A number of small areas such as townships and sections in the rural counties and blocks in the cities are chosen so that the sum of the areas is an accurate sample of all the areas in the country.

Interviewers are instructed to call on every household or perhaps every third or fourth household in the sections or blocks that are finally selected. This method has been devised because it allows the interviewer little or no choice among the persons whom he is to visit and therefore has certain advantages over the use of the controlled sample. Also, some observers believe that it tends to avoid bias and mistakes resulting from the use of the interviewer’s personal judgment.

How are the questions phrased?

Selecting a good sample is very important, but careless phrasing of questions and the use of poorly trained and supervised interviewers can ruin the poll no matter how scientifically the sample has been made up.

The questions must be neutral. A “loaded” question which suggests an answer may cause serious error in the results. For example, take the question, “Should employers be forced to negotiate with union labor?” Is it a fair question? Couldn’t many meanings be read into this question simply because of the word “force”? Would you know for sure what the answers to this question really meant? Likewise the blunt question, “Do you read any books?” would yield meaningless answers because it is too general, it allows for too many interpretations, and it might hurt the pride of persons who are interviewed.

Questions must be clear, so that people cannot misunderstand their meanings. For example, in the course of one survey it was found that the word “salvage” meant many things during the war to different people. To some it suggested “paper and tin can drives.” To a few it had to do with “bringing the boys home from the front.” Therefore, to have used that word in a final survey would have resulted in error.

The questions must deal with matters which the people who are interviewed can be expected to answer properly. Complicated and technical questions, or those dealing with subjects on which they have little information, may yield meaningless answers.

As a result of these and other requirements of a good set of questions, the polling organizations should pre-test their questions by trying them out on a small number of persons rather than relying entirely on the judgment of their own staff. Only after the questions have been properly tested can they be incorporated, in a final questionnaire. Many polling surveys are faulty because the questions have not been carefully worded and have not been tested.

What kinds of questions are asked?

Exact forms of the questions depend also on the type of information which the survey proposes to gather. The most common form is one which requires only a yes or no answer. It is particularly useful when issues have been before the people and when they have probably already formed an opinion about them.

For instance, the American Institute of Public Opinion (Gallup Poll) asked the simple question, “Do you believe that war bonds are a good investment?” Answers were:

Yes 91%

No 5%

Undecided 4%

Open or free questions which allow the persons interviewed to express themselves freely and at length are sometimes used, especially when it is important to find out the various lines of thinking which are current on an issue. For instance, workers in certain industries might be interviewed and allowed to express themselves freely on the conditions under which they work, in an effort to find out the most important personnel problems in the industry from the employees’ angle.

Another form is the so-called multiple-choice question, or questions grouped into a check list. Here the persons interviewed are allowed to make a choice among a number of answers. The multiple-choice questions are useful, for example, when it is important to know how strongly people feel about a problem; how well they are informed about an issue, or which viewpoint they hold among those listed.

Here is an example of this form of question as it was used by the National Opinion Research Center:

“Which of the following statements comes closest to describing how you feel, on the whole, about the people who live in Germany (Japan)?”

1. The German (Japanese) people will always want to go to war to make themselves as powerful as possible.
2. The German (Japanese) people may not like war, but they have shown that they are too easily led into war by powerful leaders.
3. The German (Japanese) people do not like war. If they could have the same chance as people in other countries, they would become good citizens of the world.

One experienced member of a polling organization has summed up the objectives of questions on a typical issue as follows

1. Questions which will discover how many persons have heard of the issue, read of it thought about it.
2. Questions to get at the direction of popular thinking about the issue.
3. Questions to bring out the intensity with which people are thinking about the issue—how strongly they feel about it.
4. Yes-no questions to find out which side of the fence the people are on.

How are interviewers trained and supervised?

Most of the modern polling surveys are based on personal interviews. The Gallup Poll, for example, employs from 800 to 1,000 interviewers throughout the country.

Personal interviews are considered preferable to surveys made through the mails because some classes of persons are more likely to answer by mail than are others. It has been found, for instance, that persons in the upper economic levels are more likely to answer by mail than those on lower levels. As a result, the sample cannot be controlled as closely when mail questionnaires are used as is possible when personal calls are made.

But good interviewing is an art which calls for careful training and supervision. Bias, and hence error, can result if the interviewer isn’t neutral or if he influences the answers of the person he is questioning. Interviewers are usually picked on the recommendation of local leaders. Their social, economic, and political leanings are studied in an effort to select those who are likely to do their work objectively and honestly. A check on the interviewer is usually made in the course of the analyses of polling data to find out if he followed instructions exactly and how his work compares with that of the others.

Next section:Why Do Polls Get Different Results?