respondents, including the use of automated collection techniques or other forms of information technology (IT) (Federal Register, 2013a,b,c). In total, VA received approximately 300 comments from individuals and veterans’ advocacy groups (Ciminera, 2015b); however, it is not clear if any changes were made to the questionnaire in response to those public comments. Similarly, no information is available on how the pilot testers were selected, the details of their experiences, or the changes made to the questionnaire following the pilot phase.

VA conducted usability testing in October 2013, and it included a human factors analysis by its Office of Informatics and Analytics (Ciminera, 2015a,b). VA contracted with two private-sector firms for web-based implementation and IT support for the registry and to design the required database architecture so that the information could be accessed, stored, linked to other database systems, and extracted. The web-based format allows for real-time performance monitoring and quality improvement initiatives to be part of the system architecture. VA stated that this capacity makes it possible to monitor weekly metrics such as the numbers of new registrants and registrant user status, to monitor the registry’s status for accessibility, and to log help desk calls that provide technical assistance to users (Montopoli, 2016). VA indicated that it has also enhanced the integration of the AH&OBP Registry database with the health care and enrollment data available in the Veterans Health Administration (VHA) databases.

In addition to the web-based version of the questionnaire, VA has implemented a version that can be accessed with Android, iPad, iPhone, and Windows Phone platforms. This mobile format was introduced on September 1, 2015, and had been used by more than 16,000 individuals through September 15, 2016 (representing approximately 10% of all individuals who accessed the registry as of that time). However, in response to the committee’s request for updated information on using these platforms to complete the questionnaire, VA responded that this information is not available (VA, 2022a).

Once participants submit their responses to the registry questionnaire, the data are saved on a VA server, regardless of whether the individuals are active-duty service members or veterans.¹ As of January 2021, VA’s Airborne Hazards and Burn Pits Center of Excellence (AHBPCE) had taken custodianship of the registry and data because VA judged that this would allow for better lines of ownership, support, and flexibility. VA has stated that this new administrative home will put the registry in closer proximity to patients and the airborne hazards care field and will improve the ability to make changes to the registry and support research priorities, including requests for access to the data.^1,2 The Department of Defense (DoD) has access to the AH&OBP Registry information through the Individual Longitudinal Exposure Record (ILER; described in Chapter 4) (VA, 2022a).

Several IT and system updates have been implemented since the AH&OBP Registry launched. These include changes to the database platform, enhancing the VHA staff portal to make VA health care users’ registry data more easily accessible to VA providers, adding capabilities for ad hoc reporting, creating a “data mart”³ in VHA’s Corporate Data Warehouse for internal analyses of the raw registry data, and integrating with eBenefits⁴ to allow access to the registry from the eBenefits website (Lezama, 2016). However, the committee cannot evaluate whether these enhancements to the VHA staff portal make it easier for VA providers to access AH&OBP Registry data. Additionally, VA has made some improvements to data collection elements of the AH&OBP Registry, such as incorporating racial and ethnic background information from VA Corporate Data Warehouse files and adding VHA and DoD identifiers to facilitate linking with other datasets.⁵ Date of death, if applicable, has been added from the VA Vital Status File, which is updated quarterly (VA, 2022a).

___________________

¹ Personal communication, Dr. Eric Shuping, director, Post-9/11 Era Environmental Health Program, Health Outcomes Military Exposures, VA. November 30, 2021.

² Personal communication, Dr. Eric Shuping, director, Post-9/11 Era Environmental Health Program, Health Outcomes Military Exposures, VA. July 27, 2021.

³ A data mart is a subset of a data warehouse focused on a specific set of information.

⁴ “eBenefits is a joint VA/DoD web portal that provides resources and self-service capabilities to Veterans, Service members, and their families to research, access and manage their VA and military benefits and personal information” (VA, 2014a).

⁵ Personal communication, Dr. Eric Shuping, director, Post-9/11 Era Environmental Health Program, Health Outcomes Military Exposures, VA. January 21, 2021.

___________________

⁶ Personal Communication—Open Burn Pit Registry/Airborne Hazards Exposure Assessment Working Group charter, Dr. Nicolas Lezama, deputy chief consultant, Post-Deployment Health Services, Patient Care Services, VHA. February 18, 2013.

⁷ Skip patterns are a question or series of questions that are associated with a conditional response. For example, if a respondent answers that they have “never smoked,” questions pertaining to duration or frequency of smoking would not be applicable to them and would be skipped.

___________________

⁸ Personal communication, Dr. Eric Shuping, director, Post-9/11 Era Environmental Health Program, Health Outcomes Military Exposures, VA. May 7, 2021.

⁹ Although the national launch date for the registry was June 19, 2014, data were collected and recorded beginning in May 2014 (VA OIG, 2022).

and it still explicitly asks two questions (the first a question about frequency of drinking during the year and the second about frequency of drinking in an average week). The current response options allow for gradations of alcohol consumption during a week, an improvement over the initial response options which grouped consuming alcohol 1 day per week with consuming it every day, but assume consistency of behavior in weekly alcohol consumption over the last year. This is the only question on alcohol consumption. If this information is to be useful in informing other factors that may influence exposure and disease relationships, the question will require substantial revision.

___________________

¹⁰ Several terms are used for the optional health evaluation that may be requested by veterans and service members who respond affirmatively to question 7.1.B. The questionnaire uses the term “free health evaluation” and “exam” as of October 26, 2021, but previously the terms “health concerns” and “health examination” (in VHA Directive 1307, August 19, 2019), and “registry examination” (in VHA Directive 1308, March 25, 2022) were used. The committee has opted to use the term health evaluation for the clinical examination and discussion between the veteran and health provider that are initiated through the AH&OBP Registry questionnaire.

___________________

¹¹ Other locations, including Syria and Uzbekistan have also been added to the list of eligible locations beginning on or after September 11, 2001. Egypt was added as an eligible location for deployments beginning on August 2, 1990. Other locations such as Yemen are also being considered.

obtain the account, an individual must meet one of several requirements: have a DoD Common Access Card with an accessible reader; have a Defense Finance and Accounting Service myPay account; be a veteran, dependent of a veteran, or survivor of a veteran; or be registered in the Defense Enrollment Eligibility Reporting System (DEERS). The DEERS account provides secure, self-service identification that is used to access several accounts and password-protected websites as well as to access VA eBenefits. A registry help desk is available for service members and veterans who are experiencing difficulties registering for an account or accessing the online questionnaire (VA, 2022d). The help desk phone line connects callers directly with a live operator (no automated directory or ability to leave a voice message), although there may be a wait until an operator is available. No information was available to the committee on whether or how these access requirements affect participation in the registry.

Figure 3-2 shows the steps a service member or veteran must complete to enroll in the AH&OBP Registry and become a participant. A factsheet on VA’s website presents the process in 14 steps (VA, 2019b). After logging onto the registry portal, an individual must consent to participate by acknowledging having read the consent form “End User Agreement and Notice of Privacy Practices.” Those who do not consent do not move to the next phase. An individual who does not initially consent to participate may reconsider at any time. As of February 1, 2022, the committee found that 21,010 individuals had logged into the registry but had not consented (the eligibility of these individuals is unknown) and had ended their participation at this stage. Those individuals who consent to participate move on to the automated eligibility review. Eligibility to participate is confirmed using the Veterans

Affairs/Department of Defense Identity Repository (VADIR) database, which contains information derived from and subject to the accuracy of information in DoD’s Defense Manpower Data Center (DMDC; see Chapter 2).

Eligibility is confirmed based on service in at least one eligible country and the dates of deployment. If an individual is determined to not be eligible for the AH&OBP Registry because of, for example, missing or incorrect information in DoD records, the individual may request a manual review by VA. In the manual review individuals begin by entering contact information and details about those deployments (information on dates, country, base names, or provinces as applicable) that they believe make them eligible. The individuals must certify the information as correct to the best of their knowledge before submitting. A VA email is provided for individuals to contact with questions or if they need assistance (VA, 2021b). The manual eligibility review uses copies of DD 214 forms, orders, awards, the veterans information system,¹² and the health eligibility center to determine whether an individual is actually eligible. As of January 3, 2022, 59,061 registrants had initially been determined to not be eligible, and 54% (n = 32,043) of them requested a manual review (VA, 2022a). Of the registrants who requested a manual review, 26,256 (83%) were determined to be eligible, 4,170 were determined to be ineligible, and review is pending for 1,617 (1,147 were notified that additional documentation was needed) (VA, 2022a). Information from manual reviews of eligible deployment information are not updated in VADIR or DMDC because DMDC’s policy is that the information must be provided by each service (Army, Air Force, Navy, Marines, and Coast Guard) and not by individuals (VA, 2022a). Once a request is received, the first step of the manual review process by VA is to access ILER (see Chapter 4) for records of deployment. If no record is found, the potential registrant is sent an email stating that more documentation is needed. A second email reminder is sent if there is no response to the first email request for more documentation (VA, 2022a). Individuals who are determined to be eligible can then access and complete the self-assessment questionnaire. The eligibility confirmation process serves as a preliminary verification check of potential participants before they spend time completing the lengthy questionnaire.

Not everyone who begins the questionnaire completes it in a single sitting. Respondents can save their current progress and return at a later date to continue or complete it. Until an individual clicks submit, the questionnaire is not considered complete. However, once an individual clicks submit, their information cannot be updated or

___________________

¹² The veterans information system is a web-based query application that provides a consolidated view of comprehensive eligibility and benefits data from across the Veterans Benefits Administration (VBA) and DoD. It provides access to information from VBA’s corporate Rating Board Automation 2000, Compensation and Pension Benefits Delivery Network for veterans, and VBA’s Beneficiary Identification and Locator System, and Veterans Affairs/Department of Defense Identity Repository (VADIR) (VA, n.d.).

changed, including for individuals who have subsequent medical diagnoses or military exposures of interest. However, as of March 29, 2022, service members who have had new eligible deployment segments after submission or service members and veterans who have served in locations made eligible (such as Uzbekistan, Syria, and Egypt) may add these segments and answer the nine associated deployment-related questions for them. No other questions in the questionnaire are asked again nor can prior responses be updated.

After submitting the questionnaire, a participant can request to schedule a registry-associated health evaluation at his or her local VA medical center. A total of 30,049 AH&OBP Registry health evaluations were completed between June 2014 and February 1, 2022 (see Chapter 7). Not surprisingly, those who responded affirmatively to question 7.1.B of wanting to see a provider were about three times more likely to actually receive the health evaluation than those who responded that they did not want to see a provider (15.6% vs 4.7%, respectively).

Changes to AH&OBP Registry Eligibility and Enrollment Since 2014

Since the registry’s inception, several changes to it have been made or proposed by VA or Congress. Changes to the eligibility criteria affect the size of the eligible population, the representativeness of participants to the entire eligible population, and the generalizability of findings from participants to the entire population. For example, as of August 2021, registry eligibility had been broadened to include post-9/11 deployments to Syria and Uzbekistan, and deployments to Egypt beginning in 1990. These deployment segments could not be entered on the questionnaire prior to 2022, so participants who submitted the self-assessment questionnaire before this time were unable to add these deployments. Individuals who were considered ineligible to participate before these countries were added (approximately 300 for Syria and Uzbekistan) and who did not have other deployments that would have made them eligible have been kept in review status until the AH&OBP Registry system is able to accept these deployments as valid (VA, 2022a). Syria was added as an eligible deployment location for post-9/11-related operations because the environment is similar to Iraq and Afghanistan and there was documented use of burn pits and combat operations.¹³ Egypt was added in response to a congressional directive. VA stated that those who deploy to the newly added eligible countries (Uzbekistan, Syria, or Egypt) should be able to enter and complete these deployment segments in section 1.1 as if they had been deployed to the original countries. That capability launched on June 7, 2022, and those who are in the pending status for Uzbekistan, Egypt, and Syria will be moved to eligible once the transition occurs (VA, 2022b). The VA office responsible for overseeing and maintaining the registry, Health Outcomes Military Exposures (HOME), has stated that following the activation of this capability, it will “monitor for patterns and have the sustainment team troubleshoot if it seems that many in the cohort are not being automatically enrolled” (VA, 2022b). Federal legislation calling for adding new countries to those already eligible is being considered.

Another example of changes to AH&OBP Registry enrollment concerns § 704 of PL 116-92, which states:

Automatic enrollment in the registry with an opt-out option is a very different mechanism than the current voluntary opt-in method. Automatic enrollment does not necessarily mean that individuals who would not have otherwise been interested in the AH&OBP Registry will complete the self-assessment questionnaire, so it appears that this policy will likely increase the number and rate of noncompletions while providing no discernable benefit. When asked how it plans to respond to this directive, VA stated that post-deployment health assessment and reassessment forms (which are administered by DoD) have been updated in their electronic formats effective September 2020 to include questions on whether a service member was exposed to burn pits. Based on responses to these exposure

___________________

¹³ Personal communication, Dr. Eric Shuping, director, Post-9/11 Era Environmental Health Program, Health Outcomes Military Exposures, VA. April 2, 2021.

questions, a health care provider will determine whether exposure occurred and provide the service member with instructions for enrolling in the AH&OBP Registry. This process will not automatically enroll service members, but they will be informed about participating (VA, 2022a).

Motivators and Barriers to Participation

In 2020, VA contracted for a qualitative assessment of the AH&OBP Registry in order to improve the understanding of barriers to, motivators for, and experiences of enrolling in the registry (VA, 2020, 2021c). This qualitative assessment was based on feedback from 12 in-depth interviews and focus groups consisting of 42 veterans and service members, including registry completers and noncompleters. The committee received a full report on the in-depth interviews and two slide decks that summarized the findings and recommendations from its work. No information on how the members of the focus groups were selected was provided. On the basis of these in-depth interviews,

• veterans learn about the AH&OBP Registry from other veterans and from VA and DoD communications, but generally not from their physician,
• some veterans believe that their participation in the registry or the optional health evaluation affects the availability of resources for other veterans who may be more in need,
• veterans recommended streamlining communications and using QR codes to help facilitate access to the registry; and
• there are still technical issues with the AH&OBP Registry.

Several motivations for engaging with the registry were reported, including personal reasons, having had respected others (e.g., medical providers, commanding officers, veteran peers) tell them to register, and altruism. During the in-depth interviews, veterans said that they participated to create a record of military service exposures that could then be used to support disability claims or have a mechanism to learn about possible health effects compensation should it become available. Other motivating factors included contributing to the registry for altruistic purposes and having someone hear about their experiences, especially for women and racial/ethnic minority populations.

Several themes emerged from the focus groups on barriers for participation in or completion of the self-assessment questionnaire (for those who began it). To begin, incorrect or misunderstood information about the purpose of the registry or the information it collects was reported by the in-depth interview and focus group participants. Although participating in the registry was simple for some who viewed their experience of the optional health evaluation positively, more often the comments or experiences was negative. For example, some in-depth interview and focus group participants felt that those who had the most severe exposures or health concerns were eligible or deserving; others thought that only those who had burn pit exposures were eligible for the registry, not those who had served more generally. Other concerns that led to barriers for participation were related to perceptions of VA, not hearing about the registry from reliable medical providers, or not seeing a need for it if the veteran already had been assessed and granted disability status. Additionally, some veterans thought that they had already completed the questionnaire or health evaluation, even if they had not (perhaps because of completing other similar questionnaires).

A third category of concern involved technical difficulties with the registry, including not having an obvious help desk, pages being slow to load, database problems such as accurately determining eligibility, and inaccurate deployment records (see Chapter 2). Veterans in the interviews noted that the website did not highlight the ability to check eligibility for the registry and said that veterans should be informed about the estimated completion time prior to logging in. Veterans also said that the notice denying eligibility needed to be revised in light of technical flaws in the system and errors in the deployment record, especially for those individuals who logged on and who were found to be ineligible to participate based on prepopulated deployment data (see Eligibility and Enrollment section earlier in this chapter). Interview and focus groups participants encouraged VA to provide more robust,

real-time technical support for those participating in the registry (e.g., a ticketing system or troubleshooting point of contact, rather than referring them to the website or generic 1-800 numbers) and to address or help those individuals who need assistance (for example, disabled veterans, those with cognitive impairment).

A common theme from the registry participant in-depth interviews was that the registry questionnaire was long and repetitive and would be improved if the reasons why the information was being collected were clear. They also did not understand why so many questions were asked that were not directly about deployment exposures to airborne hazards or burn pits (VA, 2020). Other recommendations included noting a need for clear instructions for the questionnaire; these instructions include (1) information about eligibility criteria related to time and place of service, not just exposure to burn pits, (2) clarifying that active-duty service members cannot add new deployments (the committee notes that participants cannot update any portion of their previous registry responses once submitted but new eligible deployments may be added), and (3) telling veterans that they can print or save registry responses after completion. Focus group participants noted that the registry was “triggering” unwanted memories from deployments that had been “blocked out” (VA, 2021c). The recommendations to overcome barriers and increase participation in the AH&OBP Registry were as follows:

• conduct outreach to VA providers (presumably to make them knowledgeable about the AH&OBP Registry and to encourage participation),
• continue to be inclusive of all airborne hazards,
• combat the incorrect perception that those who had less severe perceived health effects should not complete the registry because veterans with more severe disabilities or health needs would then not receive services, and
• provide assistance to veterans when registering (VA, 2020).

This qualitative work identified several areas in which the registry could be improved, including communications about the registry to veterans (see Chapter 9) and the instructions for the registry itself. Given how recently this assessment was conducted, it is not clear to what extent those recommendations have been integrated into the AH&OBP Registry.

ANNUAL MAINTENANCE AND COSTS

Developing and maintaining a registry is not cost-free to either participants or sponsors. As of July 1, 2022, more than 317,000 participants had completed and submitted the questionnaire, and over 130,000 individuals (see Table 2-11) had enrolled but did not complete or submit the AH&OBP Registry questionnaire. Using the updated findings of 38 minutes as the median time needed for completion of the questionnaire portion (not including section 1.1), which did not include the time needed for verifying or manually entering eligible deployment segments, a very conservative estimate is that participants have invested a minimum of 200,000 person-hours in providing information for the AH&OBP Registry on questions other than deployment history. This does not include the time invested by the more than 30,000 participants who received the AH&OBP Registry health evaluation between June 2014 and February 2022 and the additional time they spent contacting and scheduling the health evaluation, traveling to the VA facility for evaluation, waiting to be seen or the time of the evaluation itself and any associated laboratory tests, and discussing or following-up on results with primary care providers.

VA did not provide total developmental and operational costs from the time of the registry’s inception or the compartmentalized costs to develop the questionnaire and the web-based platform and interface (VA, 2022a). However, some cost estimates were given for specific activities and functions, such as adding or expanding inclusion criteria on a country level. The total bulk annual IT costs for the first 6 years of registry operation were estimated to be $1.1 million. These costs were presented in three categories: analysis, reporting, staff education, and congressionally mandated reports ($350,000); development ($400,000); and sustainment ($350,000). These estimates appear to have been excerpts from VA reports submitted to OMB (VA, 2021d) and do not include the cost of contracts for information collection.

___________________

¹⁴ Exposure to soot, ash, smoke, or fumes from oil-well fires in the 1990–1991 Gulf War were not included in this count, as the question was not asked of every participant.

this exposure. For questions 1.2.B–D and 1.2.H–I, there were 1,190,888 eligible segments representing 278,645 individual participants. For example, ever having been near a burn pit was reported on at least one deployment by 91% of participants in the committee’s dataset (a decrease from 96% as reported in the initial assessment of the first 46,404 participants), but for only 60% of deployment segments. An example with a time component response was exposure to dust storms, defined as at least 1 day per month during deployment, which was reported by 83% of individual participants (similar to the 85% of early participants who reported the dust storms exposure). In the initial assessment 89% of participants who served in the 1990–1991 Gulf War reported that they were exposed to soot from oil-well fires, but this exposure was now reported by 63% of all eligible individuals and for only 31% of eligible deployment segments.

The lack of variability in responses to key exposure questions and deployment experience is one factor that limits the analyses that are possible using these items (see Chapter 5). Given that the registry was promoted with a focus on burn pit exposures, it is not surprising that virtually all participants reported having this exposure. In effect, the high levels of reporting the exposures in Table 3-3 are not surprising; nearly anyone who was deployed experienced them at some point. Moreover, there are no follow-up questions to capture more granular details.

Only one health outcome was reported by at least 80% of participants (how often do you snore, 86.5%). Regular insomnia or trouble sleeping in the past 12 months was reported by 79.0% of participants, and experiencing neurologic problems was reported by 68.4% of participants. Several health outcomes were reported by less than 5% of participants: emphysema (1.1%), chronic bronchitis (2.6%), constrictive bronchiolitis (0.8%), idiopathic pulmonary fibrosis (0.2%), coronary artery disease (1.7%), angina pectoris (1.2%), and heart attack (1.3%).

Twenty-one different types of health conditions are asked about on the self-assessment questionnaire (sections 2.2 and 2.4) (Table 3-4): allergies, asthma, emphysema, chronic bronchitis, chronic obstructive pulmonary disease (COPD), other lung condition, constrictive bronchiolitis, idiopathic pulmonary fibrosis, hypertension, coronary artery disease, angina, myocardial infarction, other heart condition, insomnia, neurologic condition, immune condition, liver condition, chronic multi-symptom condition, frequent snoring, sleep apnea, and cancer. A diagnosis of cancer was based only on an affirmative response to question 2.4.A. The total number of reported health conditions was calculated for each participant. Because “Other lung condition” is asked as a gateway question for the questions about constrictive bronchiolitis and idiopathic pulmonary fibrosis, if a participant reported yes to “Other lung condition” and had idiopathic pulmonary fibrosis or constrictive bronchiolitis but no other lung condition, this may have resulted in over counting. Only 2.2% of participants reported none of the 21 health conditions, but the proportion who reported none was higher among later participants than early participants (2.3% vs 1.3%, respectively). Similarly, very few participants (<1.0% of early or late participants) selected having 12 or more health conditions. Only 8 of 278,645 total participants (2 early and 6 late) reported 20 of 21 possible health outcomes. No participant reported all 21 possible health outcomes. The median number of health conditions for both early and late participants was 4, and the mean was similar at 4.6 and 4.5, respectively. Early participants selected slightly more health conditions on average than late participants, but these frequency counts do not account for variations in severity of reported conditions.

Questions with High Item Nonresponse Rates

Respondents are required to answer every question in the questionnaire, but for most questions they can also indicate that they “Don’t know” or select “I do not wish to answer” (refusal) on most questions. During the initial assessment analysis, VA indicated that there was an “anomaly” in an earlier version of the questionnaire which allowed respondents to skip questions; the anomaly was subsequently corrected and only affected a small number of responses. As such, nonresponse analyses examine completely missing, reported “Don’t know,” and reported refusal responses individually and combined.

Without further adjustments for missing data (such as multiple imputation), high nonresponse rates¹⁵ reduce the usefulness of these questions for analyses and may indicate questions with a high risk of nonresponse bias. A high nonresponse rate on a particular question may also indicate that the question is difficult to answer or requires recalling details that are not salient or may not be deeply encoded in memory, or it may indicate difficulty in comprehension. For those who used a mobile device to complete the questionnaire, some nonresponses may be due to bad design or formatting on the device. Consistent with the initial assessment, nonresponse rates remain high on many questions. Across all questions, the average nonresponse rate is about 10%, with many questions having nonresponse rates greater than 15%. The highest nonresponse rate is for the question “Do you send or receive emails?” at 74% missing; this question has since been removed from the questionnaire.

To better understand nonresponse on the AH&OBP Registry questionnaire, the committee first examined the percentage of 31 core exposure and disease questions answered by each participant (from sections 1.2, 1.3, 1.4, 2.2, 2.3, and 2.4). Table 3-5 shows the breakout of nonresponse to these questions. Approximately 90% of participants answered at least 80% of these key exposure and health questions.

Nonresponse rates were calculated for each question by combining the nonsubstantive responses (the number of “Don’t know,” refused, and missing responses) and dividing by the total number of respondents eligible to

___________________

¹⁵ In the field of survey design and research, “nonresponse rates” generally refer to unit nonresponse, which is defined as the failure of a sample of those selected to participate to answer any questions in the survey. Because participation in the AH&OBP Registry is based on voluntary opt-in design, the unit nonresponse is more difficult to determine. In this chapter, nonresponse rates refer to item nonresponse rates, meaning question-level nonresponse.

answer the question. Table 3-6 shows the nonresponse rate for each question except for “Check all that apply” questions. The committee used 15% as the cutoff for determining a high item nonresponse rate. Whereas responding affirmatively to an exposure at least once across deployment segments was considered affirmative exposure for an individual, this method does not work for determining a summary of nonresponse to these questions across an individual’s deployment segments. One possibility would be to report the percent of individuals with any item nonresponse on any deployment; however, this might yield high rates of nonresponse among those with many deployments. Another possibility may be to calculate a deployment-specific item nonresponse rate for each registry participant, that is, the percentage of deployments with any item nonresponse for each individual and average this percent across individuals. Instead, Table 3-6 presents nonresponse rates for deployment-specific questions 1.2.A–1.2.I at the deployment level (n = 1,190,888).

The questions with the highest nonresponse rates have to do with exposures, many of which have nonresponse rates that exceed 15%. That is, the most critical questions to be completed for an exposure registry are the least likely to be answered. Question 1.2.A, which asks about exposures to oil-well fires during the 1990–1991 Gulf War, has a deployment-level nonresponse rate of 68%. Question 1.2.I, which asks about sewage ponds, has missing responses for 43% of deployments, with 80% of the missing values being from “Don’t know” responses. Among the other exposure questions with high nonresponse rates was Question 1.2.D (whether a respondent was near a burn pit on each deployment), which had a nonresponse rate of 27%. In other words, for 27% of deployments, the response was don’t know (69% of the missing responses), refused (1% of the missing responses), or missing (29% of the missing responses) for this key question of being near a burn pit. Question 1.2.G (the number of hours per day during the deployment that smoke/fumes from a burn pit entered the work site or housing area) had a nonresponse rate of 20%. Question 1.3.B (the number of days exposed to smoke from heavy weapons) had a nonresponse rate of 23%. Item nonresponse rates were high on questions for potential exposure outside of military service (e.g., asbestos, occupational categories, living in a home with elevated radon levels; sections 5 and 6 of the self-assessment questionnaire) and thus may have been viewed as outside of the focus of the registry. For example, question 5.5.A, which assesses exposure to asbestos at work, has a nonresponse rate of 36%, with about 97% of the missing attributed to “Don’t know” responses. In general, “Don’t know” responses constitute a large portion of the item missing values.

As Table 3-7 shows, certain deployment characteristics may have a large effect on whether a registry participant is likely to answer questions about having had certain exposures, such as to burn pits. Certain deployment characteristics maximize participants’ likelihood of clearly knowing whether they were exposed to a burn pit,

NOTES: Questions with * indicate the number of eligible responses was based on number of deployments (versus individuals); questions with + indicate those that have nonresponse rates of 15% or greater.

thus lowering their chances of responding “Don’t know.” Those individuals who served in the Air Force or had deployments that occurred in the Combined Joint Task Force–Operation Inherent Freedom/Operation Freedom’s Sentinel (OIR/OFS) era, to countries other than Iraq or Afghanistan, or that lasted less than 30 days are more likely to answer “Don’t know” or to not answer questions about burn pit exposure.

The committee examined nonresponse to the deployment-specific exposure questions for participants who had multiple deployments. Three exposure questions that are asked of all deployments (questions that are not restricted to Gulf War era deployments and not part of a skip pattern) were examined: exposure to burn pits (1.2.D), deployment hours outside (1.2.H), and deployment near sewage ponds (1.2.I). As shown in Table 3-8, among those participants with two or more deployments, 16.8% had at least one deployment missing all three of these exposure questions and 1.9% had responses missing to all three of these exposure questions on all deployments. Thus, the most critical exposure information concerning those with multiple deployments is at least partially missing for about one in six participants.

A previous analysis of registry data found that nonresponse rates were significantly higher for deployment segments that were asked about later in the questionnaire than those that were asked about earlier, a pattern that suggests that the nonresponse may be due to respondent fatigue (Gasper and Kawata, 2015a). There were several problems with the registry data that may explain the high missing rates for certain questions. For example, at the deployment level a high rate of nonresponse was for “exposure to oil well fires” (1.2.A). This question was only asked on approximately 64,000 eligible deployments in the 1990–1991 Gulf War era and had a missing rate of 65%. One possible explanation for this high level of missing response is that deployments were not correctly identified as Gulf War era in VADIR. To examine this possibility, using the dates for each deployment a flag was created for whether a deployment occurred in the 1990–1991 Gulf War era (defined as deployment start prior to April 6, 1991) and cross-tabulated with responses to 1.2.A. Of the 46,769 deployments with missing data on the

question, 98% of the missing responses were not flagged to be in the 1990–1991 Gulf War era. This pattern suggests there may be some issue with the criteria used by the registry program to determine whether a deployment is 1990–1991 Gulf War or the skip logic for this question. That is, VADIR and DMDC do not reliably identify eligible deployments of this era, which in turn affects the logic used by the AH&OBP Registry to accurately determine participant eligibility, particularly from this era.

An analysis found the nonresponse rate to Question 1.2.D (ever having been near a burn pit on deployment) was higher for deployments to bases where burn pits did not exist than on those where they did, suggesting that some of the “don’t know” responses may actually be “not exposed” (Gasper and Kawata, 2015b). The committee conducted a similar analysis using two different methods and found that for both methods, participants had indicated considerably higher proportions of reported exposure and “don’t know” responses than would be expected in locations where incinerators were operating and the burn pit had ceased or where there was no documented burn pit (see Tables 2-17 and 2-18 in Chapter 2). This supports the prior suggestion that “don’t know” responses may actually be “not exposed” or that the wording of the questions is not clear.

Health Outcomes

Most questions on health outcomes have a nonresponse rate of 15% or less, with a few exceptions. Question 2.2.3.I, which asks about the frequency of stopping of breathing during sleep, has a nonresponse rate of 39%, 99% of which are missing (rather than “don’t know” or refused). This question (and 2.2.3.H) has the response option of “Don’t know” on the questionnaire, but this response option does not appear in the data, and instead these questions have unusually high rates of completely missing data. It is possible that this option was miscoded as “missing” instead of “don’t know.” Question 2.2.3.C (occurrence of problems of the immune system) has a nonresponse rate of 20%, with 94% of those attributable to “Don’t know.” The lower nonresponse rate on most questions assessing health outcomes compared with exposures may indicate that respondents know or are aware of many health diagnoses prior to completing the registry questionnaire. Two questions that assess health beliefs with regard to exposures encountered during deployment(s) (3.B and 3.C) also have high nonresponse rates (32% and 34%, respectively). For both of these questions, 97% of the nonresponses are “Don’t know.” The reasons for high nonresponse rates for these health outcome questions may be because the respondent may not have knowledge of them (e.g., stopping breathing while asleep; having “problems of the immune system”) or may want to attribute them to causal events (e.g., being sick because of something that they breathed during deployment).

In addition, some questions have skip patterns detailed in the questionnaire, but the skip patterns were not coded in the data, necessitating cleaning and adding them before nonresponse analyses could be conducted. For example, question 2.5.I (smokeless tobacco frequency) should not have been asked if a participant answered “No” to question 2.5.H (smokeless tobacco use). The questionnaire did not automate which questions were asked of respondents based on the responses to question 2.5.H, which resulted in an inflated nonresponse rate for question 2.5.I before the data were cleaned to include only those eligible to answer the question. Similarly, skip patterns were not implemented for questions 2.4.D and 2.4.F (second and third cancers).

___________________

¹⁶ The committee notes that these figures may be misleading as 14.7% of values were missing for sex for early participants vs 1.0% for late participants. VA provided additional information on this variable. VA uses three sources of data to provide information on gender; the primary source is the AH&OBP Registry (although the questionnaire does not include a question on gender, the registry is linked to VA data sources with that information). If data from the AH&OBP Registry are incomplete, the data are matched with VHA data (Corporate Data Warehouse S-Patient Table) for the information. The final method used to identify missing gender information is through matching with the Veterans Affairs/Department of Defense Identity Repository (VADIR). Using the three sources has reduced the number of missing values for the gender variable and is the reason for the change that was observed (VA, 2022e).

¹⁷ Age at questionnaire completion was missing for 4% of late participants versus less than 0.1% for early participants.

¹⁸ Unit component comparisons were based on participants from the June 3, 2021, extract as this variable was not included in the February 1, 2022, extract.

from the early participants on unit component (active duty or reserve/National Guard) and continue to greatly over-represent active duty.

INITIAL ASSESSMENT RECOMMENDATIONS

The NASEM committee responsible for the initial assessment of the AH&OBP Registry offered nine unnumbered recommendations (see Box 1-1 in Chapter 1). Four of these recommendations relate to information that has been covered in this chapter; the other five recommendations are presented in the following chapters. The reassessment committee’s Statement of Task requires, “The report should additionally address how VA has done in implementing the recommendations offered in the 2017 NASEM report,” which the committee does next and at the end of each other chapter (chapters 5, 7, and 9) for which initial assessment recommendations are relevant. Each of the initial assessment recommendations given below is followed by VA’s verbatim response to it—received in September 2020 and noted to have been extracted from a report to OMB on VA’s responses to the initial assessment committee’s report, including each recommendation (the full report was not made available to the committee)—and then the current committee’s assessment of VA’s response and any new information that has been presented. The section is in response to the reassessment committee’s Statement of Task and is not to be interpreted as an endorsement of the initial assessment committee’s recommendations. The remainder of the report presents specific findings based on the assessment of the current committee and, where findings are related to an initial assessment recommendation, whether the reassessment committee concurs with all or parts of that recommendation.

Initial Assessment Recommendation: The committee recommends that once VA clarifies the intent and purpose of the registry, it develop a specific plan for more seamlessly integrating relevant VA and DoD data sources with the registry’s data with the goals of reducing future participant burden, increasing data quality by restructuring questions to minimize recall and other biases, and improving the usefulness of the registry database as an information source for health care professionals and researchers.

VA Response: DoD and VA are working on the Individual Longitudinal Exposure Record (ILER) which will become part of the Electronic Health Record (EHR). Additionally, Cerner [Cerner Corporation, a health IT

company], the developer of the EHR has already taken the registry exams and templated them to pull data from records already available. It is hoped that the focus groups and review of the questionnaire would assist with question restructure as needed.

Reassessment Committee Response: VA has not clarified the purpose or aims of the AH&OBP Registry. Rather, it continues to state that it can serve five purposes: etiologic research, population health surveillance, facilitating access to and use of health care to improve clinical care, as a conduit for communications with participants and veterans, and supporting VA policies and processes. Chapters 5–9 explore and assess the AH&OBP Registry’s ability to fulfill those purposes. Many of the identified issues with the registry, including those with the self-assessment questionnaire, stem from VA not having made the recommended structural changes to reduce participant burden. The few changes made to the questionnaire do not address the need to clarify the intent and purpose of the registry. Additionally, findings from focus groups indicated that incorrect or misunderstood information about the purpose of the registry or the information it collects were barriers for participation or completion of the self-assessment questionnaire (for those who began it). For example, some in-depth interview and focus group participants said that they felt that those who had the most severe exposures or health concerns were eligible or deserving; others thought that only those who had burn pit exposures were eligible for the registry, not those who had served more generally (VA, 2020). Moreover, a common theme in the registry participant in-depth interviews was that the registry questionnaire was long and repetitive and would be improved if the reasons for why the information was being collected were clear. They also did not understand why so many questions were asked that were not directly about exposure to airborne hazards or burn pits during deployment (VA, 2020). Clarifying the purpose of the registry and the purpose for collecting the information would likely mitigate confusion concerning eligibility or the desire to participate.

The committee is aware of several updates and data linkages made to the AH&OBP Registry for improved data collection since the initial assessment of the AH&OBP Registry was published in 2017. These have included IT and system updates, such as changes to the database platform, enhancements to the VHA staff portal, adding capabilities for ad hoc reporting, linkage with VHA’s Corporate Data Warehouse to incorporate racial and ethnic and other selected demographic information and to better integrate registry data with health care and VHA enrollment data, adding VHA and DoD identifiers to facilitate linking with other datasets, and, where applicable, adding date of death from the VA Vital Status File. For example, the committee’s February 1, 2022, registry dataset included only a binary deceased variable but not year of death. Among the 278,645 participants, 1,257 (0.5%) were determined to be deceased.

Although these are all positive steps, none has addressed any of the major underlying issues of the registry and, specifically, the questionnaire. Participant burden may have lessened on the deployment verification section for deployments of 30 days or less, but on the whole it appears unchanged if not worse, based on nonresponse analyses. Participants continue to encounter issues with the platform, including the speed of loading and the ability to save before moving on, which the committee also observed during its live demonstration of the questionnaire. The few changes made to specific questions, such as on alcohol consumption and interest in the optional health evaluation, may have clarified the wording and intention of the question, but without validation and other measures the effects on data quality are unknown. It is unclear how any of the VA improvements listed to date could be construed as reducing participant burden or helping to minimize recall, information, or selection biases, the latter of which would be key to improving its usefulness for health care professionals or researchers.

Other recommendations from the recent qualitative evaluation focused on the clarity of instructions for the questionnaire. Specific revisions include (1) information about eligibility criteria related to time and place of service, not just exposure to burn pits; (2) clarification that active-duty service members cannot add new deployments (the committee notes that as of March 29, 2022, participants could add new eligible deployment segments); and (3) a reminder to print or save registry responses after completion (VA, 2021c, 2022b).

VA’s direct response to this recommendation focused on ILER, which appears to be its response to the part of the recommendation that called for VA to develop a specific plan for more seamlessly integrating relevant VA and DoD data sources with the registry’s data, but this does not address how the registry should be used or clarify its purposes. ILER’s features and its ability to supplement or improve information collected by the AH&OBP Registry

is described in the following chapters. In short, ILER is a data compiler, and completeness, errors, or other flaws in the administrative and clinical data being compiled in ILER will continue to be present and affect accuracy.

Initial Assessment Recommendation: The committee recommends that alternative means of completing the questionnaire, such as a mail-in form or via a computer-assisted phone interview, be offered in order to ensure that the subset of eligible persons who do not use or are not facile with the Internet have the opportunity to participate in the registry.

VA Response: We have not built a paper copy of the AHOBPR. We will assist any veteran or service member who cannot get on a computer to complete. This can be done via phone, printing a questionnaire for mail in, or in person. Only two have needed direct assistance in last 2 years.

Reassessment Committee Response: VA did not adequately address this recommendation, as no alternative means have been developed or offered to complete the self-assessment questionnaire, although a helpline is active. As long as the AH&OBP Registry is in operation, having a mechanism for those without Internet access to be able to record their interest or concern in airborne hazards exposures and to request a health evaluation remains important. Although a web-based survey may confer benefits over more traditional methods of mail surveys, in-person interviews, or computer-assisted telephone interviewing, not all eligible persons have access to a web-enabled device such as a computer or smart phone. Although not stratified by era of military service, a 2019 report by the Federal Communications Commission found that only 78.4% of veterans have 10 Mbps/3Mbps mobile LTE broadband coverage (FCC, 2019), which would potentially exclude participation from part of the eligible population, likely those veterans who have lower incomes, less education, and who live in rural areas or areas with poor broadband coverage. Moreover, technical difficulties with the registry, including not having an obvious help desk, continue to be a concern for those who begin the participation process. Interview and focus-group participants encouraged VA to provide more robust, real-time technical support for those participating in the registry (e.g., a ticketing system or troubleshooting point of contact rather than referring them to the website or generic 1-800 numbers), and to address or help those individuals who need assistance (for example, disabled veterans and those with cognitive impairment).

Initial Assessment Recommendation: The committee recommends that VA involve external survey experts experienced in web-based instruments in any restructuring of the registry questionnaire.

VA Response: A contract has been developed to assist VA in responding to items 8 [once VA clarifies the intent and purpose of the Registry. . .] and 9 [involve external survey experts. . .] and will let in FY 21. Specific language in the contract is that the Contractor shall convene a panel of experts to evaluate and analyze current web-based Airborne Hazards and Open Burn Pit Registry questionnaire and resulting information from interview process.

• Include subject matter experts (SMEs) with experience in Survey Methodology and Survey Design including online-surveys, web-based communications, environmental epidemiology, occupational medicine, pulmonary medicine, and clinical and basic research.
• Include SMEs familiar with VA environmental exposure programs.
• Emphasis is on the registry question phraseology, respondent comprehension, web-based questionnaire construct, validity, reliability and response bias.
• After panel convenes, produce report within 30 days.

Reassessment Committee Response: VA did not respond to this recommendation until 2021. During its deliberations the committee was made aware that the contract for the work responding to this recommendation had been awarded to Westat, and the associated report was completed in August 2022, after the reassessment committee’s formal deliberations were finished (VA, 2022g). Westat conducted a comprehensive review including a question-by-question assessment by an expert panel and a series of qualitative interviews of veterans that included cognitive and usability interviews with nine or fewer veterans for each. The recommendations from that expert panel were

focused on improving the registry to meet two of VA’s stated main objectives for the registry: to provide data to help clinicians administer the clinical exam, and to collect data for researchers about airborne hazards exposure (VA, 2022g). The expert panel for Westat is not a consensus committee as is the National Academies’ reassessment committee and does not include members who have served on or who are currently serving on the National Academies’ assessments of the AH&OBP Registry. Although the Westat contract addresses the initial committee’s recommendation, the following chapters address how implementing this action at this late stage after more than 317,000 individuals have participated is unlikely to create substantial improvements in the registry such that it can be used for the five purposes VA intends.

Initial Assessment Recommendation: The committee recommends that VA evaluate whether and how registrants who did not complete the questionnaire differ from those who did, analyze the determinants of non-completion, and use this information to formulate strategies to encourage registrants to finish and submit their responses and improve the completion rate for future participants.

VA Response: Analysis was done in late March 2019 where 170,693 participants who finished the registry were compared to 90,125 who did not (registrants). [Table 3-10] below shows where the registrant group stopped within the questions. In the group 39% did not enter any data at all while 37% stopped within the first section. It is difficult to do data analysis on the group that did not start in that very little data has been collected on them. A primary purpose of the focus groups would be talking to veterans and service members in the first two categories to determine why they stopped.

Reassessment Committee Response: VA did not address this recommendation in its response, although the committee is aware that VA has sent targeted messages through the registry to noncompleters encouraging them to revisit and submit the questionnaire (see Chapter 9). Because the registry collects demographic and military characteristics through linkages to VA sources, these characteristics could be compared between full and partial completers. The reassessment committee attempted to conduct such an analysis but was not provided with the demographic or military characteristics of noncompleters. A further comparison could examine differences on a question-by-question basis. The committee was only provided with the last section viewed (see Table 3-10) and no responses of noncompleters to any of the questions and therefore was unable to discern whether noncompleters were affirming the same exposures or health outcomes or not responding to questions at the same rates as full completers. VA responded to requests to provide question responses for partial completers that itemized data on noncompleters is not available, but only the last section viewed is available (VA, 2022e). However, the committee

TABLE 3-10 Last Section View for Noncompleters

observes that if respondents are able to save their responses and come back at a later time to continue, question-level responses of noncompleters should be available. One of the primary reasons for conducting comparisons with noncompleters is to determine if they are different from completers so that such information may inform strategies that might be used to encourage participation for those with under-represented characteristics.

Recently, VA contracted for a qualitative assessment using focus groups related to barriers to completion of the questionnaire. Focus group participants were supportive of VA providing more robust, real-time support for those participating in the registry (e.g., a ticketing system or troubleshooting point of contact, and not just referring them to the website or generic 1-800 numbers), and also of VA addressing or helping those individuals who need assistance to participate (for example, disabled veterans and those with cognitive impairment) (VA, 2021c).

SYNOPSIS

Very little has changed in the AH&OBP Registry in the 5 years since the initial assessment. Despite a recommendation from the initial committee to shorten the questionnaire and ease the response burden for participants, few questions have been changed and an evaluation of the questionnaire has been started only during the writing of this report. The changes to the few questions that have been completed were not motivated by the initial assessment and were not pretested, and it is unclear how their changes will benefit the registry. The major change to the deployment segments that was made with the goal of reducing respondent burden—prefilling an “I don’t wish to answer” response for deployment segments of 30 days or fewer—is instead likely to further reduce the quality of data by increasing missingness on these segments.

About 10% of eligible veterans have completed the registry questionnaire to date, providing approximately 200,000 person-hours of time after verifying eligible deployment segments and exposures on those deployments alone. Participation in the registry is time-consuming and burdensome, and there appears to be a lack of clarity among those in the target population about who is eligible for the registry. In recent qualitative work, many noted that they thought eligibility required exposure to open burn pits, rather than serving in any of the eligible areas or conflicts. Others were concerned that participation would reduce resources that may be needed for those with higher levels of need. This potential for confusion over eligibility is reflected in the questionnaire responses. Virtually all participants report at least one exposure to a burn pit at some point during one of their deployments, with an average of 5 to 6 different types of exposures reported across deployments.

Many individuals who start the questionnaire fail to complete it in its entirety, either through breaking off altogether or through answering “don’t know” or refusing to answer a question. Despite the high rates of affirmation when aggregating across all deployment segments, the highest rates of missing data are on questions about specific types of exposures on individual deployments, limiting the usefulness of these questions in assessing the risk associated with exposure at specific locations even under perfect representation. Other questions with high rates of missing data have vague, confusing, or undefined terms (e.g., “problems with the immune system”) or ask for details about a health outcome that the individual may not know (e.g., number of times the participant stops breathing at night). This lack of completeness on questions (an average of 10% missing across all items) means that information on exposures during eligible deployments is incomplete and that, in addition, answers to items may be inaccurate (reporting the same exposures during all deployments, for example). This is a separate issue from the approximately 40% of individuals who do not complete the survey questionnaire (partial completers). Taken together, these issues result in a high likelihood of both selection and nonresponse bias.

There were modest differences between early (those who participated during the first year of registry operations) and late participants (those who participated on or after August 1, 2015). Compared with the early participants, late participants had a higher percentage of women, were older, and were more likely to identify as Black or Hispanic. Later participants were more likely to have served in the Air Force and Navy and less likely to have served in the Army, although there was little difference in the proportion who served in the Marine Corps or Coast Guard. Late participants had fewer eligible deployment segments and shorter cumulative deployment durations than early participants, and more time had elapsed between military service and participation in the registry for late participants than for early participants. Although later participants were somewhat more representative of the full eligible population on some characteristics overall, the combined set of participants were still notably

different from the eligible population on a number of important characteristics, notably age, service branch, and unit component. Additionally, the committee did not have the necessary information available to examine two-way combinations of veteran characteristics (e.g., age and sex) of the eligible population, limiting the comparisons to only those overall, nor did the committee have data available to compare for key exposure variables.

Although the initial assessment committee recommended that there be a systematic evaluation of the questionnaire by subject-matter and methodological experts, that the burden of responding to the questionnaire be reduced by deleting nonessential sections, and that additional modes be developed for participants to complete the questionnaire, VA chose not to implement these recommendations. During the reassessment committee’s deliberations, VA hired outside firms to conduct methodological work on the questionnaire, but this has not manifested in changes to the questionnaire. The modest questionnaire changes have not yielded an easier-to-complete questionnaire. Altogether, little has changed in the registry questionnaire or processes in the 5 years since the initial assessment was completed.

In response to the recommendation offered by the initial assessment committee that VA clarify the intent and purpose of the registry, VA has stated that the AH&OBP Registry has five purposes:

• To support etiologic research (Chapter 5),
• To perform population health surveillance (Chapter 6),
• To improve clinical care (Chapter 7),
• To support VA policies and processes (Chapter 8), and
• To communicate with veterans and other stakeholders (Chapter 9).

The committee will use chapters 5–9 to assess whether the AH&OBP Registry is meeting or has the ability to meet each of those stated purposes. Before the committee examines each of those purposes in depth, the next chapter—Chapter 4—examines several sources of information in addition to the AH&OBP Registry that are currently used to assess airborne hazards, including burn pits, and toxic exposures related to deployment. The capabilities of those alternative sources to perform each intended purpose are compared and contrasted with the AH&OBP Registry in chapters 5–9.

REFERENCES