The research problem of investigating the role of system effectiveness in the acquisition process over 60 years does not fit into a traditional dissertation-like process. The answers to the research questions are qualitative, not quantitative. The data is the literature. For example, what was the subject and when was the document published? Gathering and analyzing literature that went back before 1958 requires a different form of a literature review; hence after some trial and error, the structured literature review concept was adopted for the subject research. Towards the end of the literature search, the need for a more detailed analysis process became apparent. The structured literature review was vital in determining the patterns in the literature. However, the structured literature review did not provide a methodology to aggregate the perceived patterns into a central concept or theory. Grounded theory methods were selected to meet this need because they facilitate developing the patterns in the data into a defensible theory. Finally, assembly of the timelines led to the inclusion of historiography techniques to assist with the analysis.
There are four essential elements to developing a structured literature review and grounded theory analysis:
The following subsections discuss the first three elements. The Results Section presents the findings.
The techniques are sequential and recursive. Each pass through the data builds off the last pass, refining and distilling the observations into a central theme.
2.1. The Research Question(s)
The aim of the research is to assess the role of system effectiveness relative to the acquisition and sustainment process over a period that exceeds 60 years. The changes are a given. For example, sustainment was not a consideration in the beginning, so what were the factors that led to its inclusion? How did that change the role of system effectiveness? Sustainment as a process in acquisition appeared as a theme, along with reliability and maintainability amongst others. Were there noticeable patterns in the themes? Taken together, these three points form a set of questions that can investigate the system effectiveness domain over the period of interest.
What factors led to the change in the role of System Effectiveness?
What themes began to emerge with the changing role?
What were noticeable patterns of change?
The structured literature review presents the data organized into a set of timelines that supports answering the research questions. Finally, the domain of inquiry is the examination of the literature in the context of the timeline using grounded theory. The outcome is in the form of factors, themes, and patterns that emerge from the literature analysis over time.
2.2. The Structured Literature Review
The structured literature review served two purposes in this study. First, the search protocol identified material related to system effectiveness facilitating the development of an organized database. Second, the structured literature review served as the first filter in identifying potential patterns for the grounded theory analysis.
Figure 1 describes the overall literature search process. The scoping study of
Figure 1 identified possible sources to search.
Table 2 presents the various literature categories used in the search.
Table 1 presents the list of sources used. Furthermore, the scoping study helped to limit the keywords used in the literature search.
Table 3 lists prospective keywords developed from several sources, the primary source being the paper written by Tillman, Hwang, and Kuo [
19]. The Tillman, Hwang, and Kuo paper was a known entity and used in the scoping study of
Figure 1. The paper, written in 1980, surveyed the literature and identified 89 references specific to system effectiveness. The paper also described the main system effectiveness models developed to that point in time. Finally,
Table 4 presents the final list used in the protocol.
The focus of the search was on primary literature or original reports and secondary literature, which describes or summarizes the original writings. Additionally, important is the category of the literature. What is its source?
Table 2 lists the various literature categories used in the search:
The order of search was:
Grey literature is unpublished or not published commercially [
27]. Because the development of System Effectiveness was primarily a government effort, the majority of the literature retrieved fell into the Grey category.
The initial searches used different browsers and search engine combinations. The Google search engine was picked as the best option for this research because it had an excellent search string feature, and Google Scholar is a bonus. Additionally, the Chrome browser has a better download feature.
The literature retrieval process used three steps.
The use of a focused search string simplified the building of the database. Storing of the results was in folders named for the keywords. All filtering was manual, and sources identified but not available were not included in the database.
Figure 2 uses “records” as a general term to cover papers, books, and reports.
TITLE-ABS ((“System Effectiveness” AND ((“keyword”)) |
The issue of using “system” vice “systems” is essential. The use of “systems” provides lots of unfocused results, most of which are not usable. On the other hand, the use of “system” provides more focused results that are usable.
The desire to conduct as complete a search as possible drove the selection of sources to search (
Table 1). Unfortunately, most 1950s and 1960s materials exist only in microfiche format, and COVID-19 restrictions limited access to archived materials. The search of
Table 1 covered all sources listed. However, the primary focus was on the government column. The Defense Technical Information Center (DTIC) changed the public interface to use the Google search engine early in the research phase. This change had two undesired effects. First, the early searches were not repeatable, and the Google search engine provided few results. Fortunately, DTIC has a research portal that provides good results with the search string and the snowball search discussed below. Unfortunately, the portal is not available to the general public.
The second step was a snowball search [
28] using the reference section of the selected papers. This search produced another 52 unique papers that were retrievable. Finally, the search string was also employed in a general web search, resulting in three conference proceedings found in
Google Books unavailable from other sources. There were numerous references to conference proceedings as a significant source of information. However, few were available electronically and library access was nonexistent because of the COVID crisis.
Table 3 and
Table 4 present the list of keywords considered and selected, respectively. Many of the keywords of
Table 3 not used were tested but returned results not germane to system effectiveness. For example, serviceability is too broad even with filtering. The final list is composed of words that provided focused results relevant to the research. The primary focus of the search was thematic. What was the paper’s subject, and how did it relate to system effectiveness? The specific focus was on papers that addressed the theory, application, or programmatic issues.
The focus in examining search returns was title relevance, abstract relevance, and paper content in that order. In addition, the search return had to demonstrate relevance to system effectiveness, the Department of Defense, and the acquisition and sustainment process.
Table 3.
Prospective Key Words.
Table 3.
Prospective Key Words.
Tillman, Hwang, and Kuo [19] | Other Sources |
---|
Reliability | Sustainment |
Availability | Tactical availability |
Operational readiness | Readiness |
Repairability | Acquisition |
Maintainability | Mission Reliability |
Serviceability | Cost Effectiveness |
Design adequacy | Operational Availability |
Capability | Mission analysis |
Dependability | Measures of effectiveness |
Human performance | Measures of performance |
Environmental effects | |
Table 4.
Selected Key Words.
Table 4.
Selected Key Words.
Reliability | Maintainability |
Availability | Operational availability |
Operational readiness | Readiness |
Dependability | Mission reliability |
Design adequacy | Cost effectiveness |
Capability | Mission analysis |
Measures of effectiveness | Measures of performance |
2.3. The Domain of Inquiry: Grounded Theory and Coding the Data
McCall and Edwards [
29] have identified three methodologies associated with grounded theory: classic grounded theory, pragmatic grounded theory, and constructivist grounded theory. The discussion of the differences among these methodologies is beyond the scope of this paper. However, the study that this paper is reporting on used the pragmatic grounded theory approach [
20].
The following reasons are the basis for selecting this approach: First, it recognizes the literature as the phenomena to be studied. Second, it takes an interpretive approach that allows the development of a more profound understanding of the literature and the evolution of an abstract theory. Resultant theories are the researcher’s interpretations of causal mechanisms. Third, the role of the researcher is that of an interpreter. However, this approach recognizes the researcher’s personal experience and knowledge as a factor.
The data sampling process is a back-and-forth effort that results in substantial memo writing and diagramming to identify and incorporate the data into manageable sets. The technique employs three distinct methods: open coding, axial coating, and selective coding. These sequential processes take the researcher through the steps to develop the data patterns (open coding) and examine the derived patterns for causality (axial coding). Axial coding confirms relationships between categories or bounds their applicability. Selective coding is about determining which category embodies the characteristics of the previously derived patterns. This category becomes the core category and represents the resulting theory. The overall procedure is recursive and proceeds until the sequence results in a candidate theory.