Designing the Archive for SHRP 2 Reliability and Reliability-Related Data (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

87 C h a p t e r 8 The objective of the testing discipline is to verify that the SHRP 2 Archive functions as designed. The areas tested and their objectives are as follows: • Functionality—verify that the functional requirements are satisfied; • Performance—verify that the performance requirements are satisfied; • Availability—verify that the procedures designed to guar- antee the desired availability function correctly; • Scalability—verify that the scalability requirements are satisfied; • Maintainability—verify that the code that supplies the SHRP 2 Archive functionality is supported by a compre- hensive unit test suite that gives confidence that system functionality is not broken by code changes; • Integration—verify that the SHRP 2 Archive functions in the required target environments; and • Security—verify that necessary software security updates have been applied. Note that the test plan provided in this chapter was based on the latest version (Version 0.3) of the document at the time of developing the final report. 8.1 Development testing Strategy Every story/feature that is developed must be tested. The team will run two instances of the Archive system on two servers: (1) production server, and (2) development server. The development server will be used to test the newly imple- mented stories/features based on the test plan provided below. Once the development server passes the tests, the pro- duction server will be updated with the latest changes. The team will use the Bugzilla platform to keep track of the bugs and enhancement stories. 8.1.1 Unit Testing 8.1.1.1 Objective Produce an application that is maintainable. This objec- tive is met by giving confidence to development engineers that unanticipated side effects of code changes will be detected before a code update is released into production. Unit testing is an industry standard way of doing precisely this. Unit testing is used to test individual units of the Archive source code. Because the team is following the agile approach, the programmers are mandated to 1. Write unit tests for every class/module written, and 2. Verify that updated code does not break any preexisting unit test. Note that the written tests should not cross the unit/class boundaries. For example, the unit test code should not try to interact with the database. In this case, a mock object has to be created and used. The details of the test depend on the module/class that is being tested. 8.1.1.2 Items to Be Tested All code modules must have unit tests. Below are some addi- tional guidelines that are useful to construct a complete unit test suite for a class. • Test any SHRP 2 unit that sends a request to the database. Make sure the requests and the output at the client side are displayed correctly. Errors, if any, must be caught by the corresponding plugin and logged or shown to the admin- istrator only, not to the end user. • Test if any errors are shown while executing database queries. Test Plan

88 • Test data integrity while creating, updating, or deleting data in the database. • Check that incorrect values entered in the form fields are handled gracefully. • Check that invalid latitude/longitude values are handled gracefully. • Check that ranking an artifact results in the corresponding value adjustment in the database. • Check that an attempt to plot nonnumeric values is han- dled gracefully. • Check that every graph type in the visualization displays as expected. • Test PHP code that retrieves data for visualization. • Test Java code that parses a CSV string. • Test Java code that validates column types. 8.2 System tests 8.2.1 Functionality Tests 8.2.1.1 Objective Verify that the functional requirements are satisfied. They include 1. System access control—only valid users are permitted access, determined by their roles; 2. Searching the Archive based on the spatial and functional (i.e., metadata) areas; 3. Uploading an artifact and specification of relevant meta- data; 4. Visualization of previously loaded artifacts; and 5. Downloading previously loaded artifacts. 8.2.1.2 Test Cases 8.2.1.2.1 Userid And PAssword 1. Verify that the userid, L13ATestUser, and password, L13AsECURITYiSaWESOME!, can log in. 2. Verify that the userid, L13ATestUser, and password, L13AsECURITYiSaWESOME!, can log in. Note that userids are not case-sensitive. 3. Verify that the userid, L13ATestUser, and password, L13AsECURITYiSaWESOME!, cannot log in. 4. Verify that the userid, L13ATestUser, and password, L13aSecurityIsNotAwesome, cannot log in. 5. Verify that a user’s login credentials are cleared when the browser is restarted. Steps a. Log into the SHRP 2 Archive. b. Close the browser used to log in. c. Restart the browser. d. Go to the SHRP 2 Archive URL. e. Verify that the user must log in again. 8.2.1.2.2 Project L02 Search criteria (values subject to change), by project: Project L02 Returns: Orange 5 over Atlanta, GA Orange 7 over Washington, DC Orange 9 over San Diego, CA Orange 10 over Philadelphia, PA Orange 12 over San Francisco, CA Blue report over Lake Tahoe, CA 8.2.1.2.3 Project L03 Search criteria (values subject to change), by project: Project L03 Returns: Orange 3 over Los Angeles, CA Orange 3 over San Diego, CA Orange 5 over Houston, TX Orange 5 over Jacksonville, FL Orange 9 over San Francisco, CA Orange 20 over Minneapolis, MN Blue report over state of Georgia 8.2.1.2.4 Project L05 Search criteria (values subject to change), by project: Project L05 Returns: Orange 2 over Knoxville, TN Blue report over Detroit, MI Blue report over state of Washington 8.2.1.2.5 FinAL rePorts Search criteria (values subject to change), non–data set types: final reports Map returns: Blue report over Lake Tahoe, CA Blue report over San Diego County, CA Blue report over Atlanta, GA Blue report over Knoxville, TN Blue report over state of Virginia Blue report over New York City, NY List returns (values subject to change): (26) Artifacts, first three are San Diego case study New York City case study Task 3: Technical Memorandum on User Engagement . . .

89 8. Time interval from SHRP 2 Landing page, Explore by Focus Area/Project tab; navigate from Capacity Focus Area to Renewal Focus Area meets UIR objective. 9. Time interval from SHRP 2 Landing page, Explore by Focus Area/Project tab; navigate from Renewal Focus Area to Safety Focus Area meets UIR objective. 10. Time interval from SHRP 2 Landing page, Explore by Focus Area/Project tab; navigate from Safety Focus Area to Other meets UIR objective. 11. Time interval from SHRP 2 Landing page, Explore by Focus Area/Project tab; navigate from Safety Focus Area to Home meets UIR objective. 8.2.2.2.2 sPeciFic LocAtion 1. Starting from SHRP 2 Landing page, Search tab, select the orange circle #11 above San Francisco, CA; 11 bubbles appear within the time interval UIR b. 2. Starting from SHRP 2 Landing page, Search tab, with orange circle #11 above San Francisco, CA, selected and 11 bubbles visible, click the bubble closest to the orange circle; a bubble containing 101NB Palo Alto to SR92 Jan 5-31, 2009, appears within the time interval UIR 1.b. 8.2.2.2.3 ArtiFAct PAge 1. Starting from the artifact page located at http://shrp2 archive.org/?attachment_id=848, go to the Data tab and select the date range from 07/01/08 to 07/02/08; the query result shows up within the time interval UIR 1.c. 2. Verify that ingested artifacts appear in search results within UIR 2 time frame. The steps are as follows: a. Ingest Artifact A2 into the SHRP 2 Archive. b. As the administrator, process the artifact. c. Start a timer and wait for the time interval UIR 2 to pass. d. Complete a SHRP 2 search, and verify that Artifact A2 appears in the result set. 3. Verify that deleted artifacts do not appear in search results within the UIR 3 time frame. The steps are as follows: a. Delete Artifact A2 from the SHRP 2 Archive. b. Start a timer and wait for the time interval UIR 3 to pass. c. Complete a SHRP 2 search, and verify that Artifact A2 does not appear in the result set. 8.2.3 Availability Tests 8.2.3.1 Objective Verify that the procedures designed to guarantee 99% of avail- ability function correctly: 1. Time required to detect that login is not functional is limited to no more than 5 min. 8.2.1.2.6 ArtiFAct Upload Artifact A1 and assign the following metadata. Steps: 1. Go to the Upload a New Artifact wizard in the My Profile dialog. 2. Select File→click Browse, and use chooser to locate and load the artifact. 3. Select Save and Continue. 4. Complete the Set Metadata dialog: a. Set Description to “Test Artifact.” b. Set Project to “Project L15—Innovative IDEA Projects.” c. Set Artifact Type to “Final Report.” d. Set Locations to “Sacramento, California.” e. Click Save and Continue. 5. When the Publish Artifact dialog appears, click Submit. 8.2.2 Performance Tests 8.2.2.1 Objective Verify that the performance requirements are satisfied: 1. UI responsiveness (UIR) as follows: a. Basic features return in less than 3 s for 90% of requests; b. Search results return in less than 5 s for 90% of requests; c. Queries on the Data tab return in less than 30 s per GB of content, 90% of the time; d. Visualization returns in less than 5 s per GB of content; and e. Ingestion completes in less than 30 s per GB of content. i. Assuming an upload feed of 3 MB/s or more. 2. Newly ingested artifacts appear in search results within 4 h of upload. 3. Deleted artifacts no longer appear in search results within 4 h of deletion. 8.2.2.2 Test Cases 8.2.2.2.1 time intervAL 1. Time interval from SHRP 2 Login page to SHRP 2 Land- ing page meets UIR objective. 2. Time interval from SHRP 2 Landing page to Artifact page meets UIR objective. 3. Time interval from Artifact page, Metadata tab, to Dis- cussion tab meets UIR objective. 4. Time interval from Artifact page, Discussion tab, to Home tab meets UIR objective. 5. Time interval from SHRP 2 Landing page, Home tab, to Search Archive tab meets UIR objective. 6. Time interval from SHRP 2 Landing page, Search Archive tab, to Explore by Focus Area/Project tab meets UIR objective. 7. Time interval from SHRP 2 Landing page, Explore by Focus Area/Project tab; navigate from Reliability Focus Area to Capacity Focus Area meets UIR objective.

90 2. Time required to detect that database is no longer com- municating is limited to no more than 5 min. 3. Time required to fail over from primary server to backup server completes in less than 4 h. 4. Time required to rebuild a failed database is less than 2 h. 5. Data loss is limited to no more than the artifacts loaded in the last 24 h. 8.2.3.2 Test Cases 8.2.3.2.1 connection issUes 1. Verify that the login monitoring process detects the inabil- ity to log in within the time frame listed in the objectives above. The steps are as follows: a. Stop the WordPress Service process (which processes login requests). b. Start a timer. c. Verify that a notification is received which informs the administrators that the login function has failed within the time frame described in the Objective sec- tion above. 2. Verify that the database connection monitoring process detects the inability to connect to the database. The steps are as follows: a. Stop the database process. b. Start a timer. c. Verify that a notification is received which informs the administrators that the database is no longer support- ing connections. 8.2.3.2.2 server FAiLUre 3. Verify that the server failover process completes within the time frame described in the objectives above. The steps are as follows: a. Stop the primary server. b. Start a timer. c. Verify that the backup server takes over within the time frame described in the Objective section. A backup server takeover is successful if a user can log in and per- form Test Case 8.2.1.2. 4. Verify that the database reconstruction process can com- plete a database rebuild within the time frame described in the objectives above. The steps are as follows: a. Perform the database reconstruction process and con- nect it to a trial SHRP 2 Archive server. b. Successfully perform all the functional tests described in Subsection 8.2.1, Functionality Tests. 5. Verify that an artifact loaded more than 24 h ago is included in a backup server’s Archive inventory. The steps are as follows: a. Load Artifact A3 into the primary SHRP 2 server. b. Wait 24 h. c. Verify that Artifact A3 is present in the artifact list in the backup server. 8.2.4 Scalability Tests 8.2.4.1 Objective Verify that the system’s scalability requirements are satisfied: 1. Five concurrent users observe the performance targets described in the Performance Testing section above. 2. Artifacts up to 2.5 GB can be ingested into the system in the time frame described in the Performance Testing sec- tion. Artifacts larger than 2.5 GB are rejected. 8.2.4.2 Test Cases 1. Run five instances in parallel of the tests described in Sub- section 8.2.2, Performance Tests. Verify that the perfor- mance requirements are satisfied. 2. Verify that Artifact A4, which is 2.5 GB in size, can be ingested into the Archive. 3. Verify that Artifact A5, which is greater than 2.5 GB in size, is rejected within the ingestion process. 8.2.5 Maintainability Tests 8.2.5.1 Objective Verify that the code that supplies the SHRP 2 Archive function- ality is supported by a comprehensive unit test suite that gives confidence that system functionality is not broken by code changes. This is accomplished by inspecting each class to ver- ify that unit tests exist that 1. Verify that invalid values of each incoming parameter are detected and further processing is prevented; 2. Verify positive operation of the class with at least one test case; and 3. Verify invocation of each “catch” block in at least one test case. 8.2.5.2 Test Procedure Verify unit test existence. The steps are as follows: 1. Procure the source code for the project along with the test cases. 2. For each class, inspect the unit tests and verify a. There are test(s) that verify validity of code processing the incoming parameters. b. There is at least one test that verifies positive operation. c. There is at least one test case for each “catch” block, assuming that the block can be reached with a

91 combination of input parameter values and/or mock object behavior. 8.2.6 Integration Tests 8.2.6.1 Objective Verify that the SHRP 2 Archive functions in the required target environments. As this is a web application, the supported inte- gration environments are 1. Internet Explorer (IE) 9 on Windows 7, 2. Safari 5.1 on Mountain Lion (10.8), 3. Firefox 25.0 on Windows 7, and 4. Chrome 30.0 on Windows 7. Test Procedure 1. Run the tests listed in Subsections 8.2.1, Functionality Tests, and 8.2.2, Performance Tests, using IE9 on Windows 7. 2. Run the tests listed in Subsections 8.2.1, Functionality Tests, and 8.2.2, Performance Tests, using Safari 5.1 on Mountain Lion (10.8). 3. Run the tests listed in Subsections 8.2.1, Functionality Tests, and 8.2.2, Performance Tests, using Firefox 25.0 on Windows 7. 4. Run the tests listed in Subsections 8.2.1, Functionality Tests, and 8.2.2, Performance Tests, using Chrome 30.0 on Windows 7. 8.2.7 Security Tests 8.2.7.1 Objective Verify that necessary software security updates have been applied. Vulnerabilities described in the National Vulnera- bility Database must be addressed within 90 days of a fix being produced. The following environments must be updated: 1. CentOS, 2. Java—OpenJDK, 3. Apache Tomcat, 4. WordPress, 5. MySQL, and 6. Lucene and Solr. Test Procedure For each of the software modules listed in the Objective section, 1. Identify recently discovered vulnerabilities by searching http://web.nvd.nist.gov/view/vuln/search. 2. Apply remediation within the time frame described in the Objective section. 8.2.8 Visual GUI Testing 8.2.8.1 Objective Verify the visual appeal of all graphical user interface elements of the Archive. 8.2.8.2 Test Procedure Check all the pages and GUI elements (e.g., containers, menus, buttons) for size, position, width, length, and acceptance of characters or numbers. GUI elements that have to be checked are as follows: 1. Homepage a. Slider on the home page b. Latest artifacts on the Recent Artifacts list. Long titles and descriptions should be truncated. 2. Top menus 3. Metadata page a. Metadata page and position of the text when the arti- fact metadata contains a lot of information b. Metadata map c. Leaflet credentials. These have to be viewable on the map. d. Metadata information. Check the metadata informa- tion to make sure it is consistent. e. Data tab’s left and right containers f. Text on the filter drop-down menus. This text has to be readable. 4. Grid view a. Scroll bar on the grid page b. Scroll bar on the filter page when the user enters too many filtering criteria 5. Graph view 6. Map view 7. Discussion tab a. Location of the rating stars 8. Ingestion page 9. User profile page 10. Search Archive page a. Leaflet credentials. These have to be viewable on the map. b. Filter check boxes c. Search results on the map d. List results 8.2.9 Testing Automation Functionality and performance test cases were automated using the Selenium framework. The testing code was written

92 in Python. After each modification (according to the stories submitted through the Bugzilla platform), the testing team ran the code to make sure other elements of the system were not affected. The code is written in a way that it can be run against different browsers (i.e., Mozilla Firefox, Google Chrome, and Internet Explorer). 8.3 List of artifacts Needed to run the test plan Table 8.1 lists the artifacts needed to run the test plan. Table 8.1. Artifacts Needed to Run Test Plan Artifact Number Test Relevant Characteristics A1 8.2.1.2.6 Used to demonstrate artifact upload function A2 8.2.2.2.3 Used to demonstrate artifact upload time Used to demonstrate artifact deletion A3 8.2.3.2.2 Used to demonstrate that artifact upload propagates to backup server A4 8.2.4.2 2.5 GB in size A5 8.2.4.2 Greater than 2.5 GB in size