Human Systems Integration Division @ NASA Ames

The FAA pilot knowledge test is a multiple-choice assessment tool designed to measure the extent to which applicants for FAA pilot certificates and ratings have mastered a corpus of required aeronautical knowledge. All questions that appear on the test are drawn from a database of questions that is made available to the public. An ongoing concern about this practice is that the availability of the test questions may encourage students to focus their study on memorizing test questions. To investigate this concern, we created our own database of questions that differed from FAA questions in four different ways. Our first three question types were derived by modifying existing FAA questions: (1) rewording questions and answers; (2) shuffling answers; and (3) substituting different figures for problems that used figures. Our last question type posed a question about required knowledge for which no FAA question currently exists. Forty-eight student pilots completed one of two paper-and-pencil knowledge tests that contained a mix of these experimental questions. The results indicate significantly lower scores for some question types when compared to unaltered FAA questions for which participants had prior access.